JMB575 SATA multiplier just works !!

zupet · Post by **zupet** » Wed Jul 20, 2022 8:17 am

port multipler just works.
I ran basic two disk raid0 read write test.

#two disk raid0 write test
odroid@server:/media$ sudo dd if=/dev/zero of=/dev/md0 bs=512k oflag=direct status=progress                                                       
30523025+0 records in
30523024+0 records out
16002855206912 bytes (16 TB, 15 TiB) copied, 58996.3 s, 271 MB/s

Code: Select all

#two disk raid0 read test
odroid@server:/media$ sudo dd if=/dev/md0 of=/dev/null bs=512k iflag=direct status=progress
30523024+0 records in
30523024+0 records out
16002855206912 bytes (16 TB, 15 TiB) copied, 58356.7 s, 274 MB/s

I tested raid1 also, but forgot to add --assume-clean option and it ran background resync while test. I found out after two days.

after flushing datas on other two disks, I'll run four disk raid0, raid1, raid5 test.

any requests ? these are WD 8TB SMR disks. It does good sequential read/write, but not good at random or heavy file system IOs.

tkaiser · Post by **tkaiser** » Wed Jul 20, 2022 7:39 pm

zupet wrote: ↑
Wed Jul 20, 2022 8:17 am
any requests ?

dmesg output please. It's about whether RK3568 and the PM negotiate CBS or FIS-based-switching.

And if there's something wrt SATA below /sys/kernel/debug/ output would be also great!

zupet · Post by **zupet** » Wed Jul 20, 2022 11:53 pm

tkaiser wrote: ↑
Wed Jul 20, 2022 7:39 pm
dmesg output please. It's about whether RK3568 and the PM negotiate CBS or FIS-based-switching.

And if there's something wrt SATA below /sys/kernel/debug/ output would be also great!

dmsg.txt: (60.55 KiB) Downloaded 10 times

Code: Select all

[    6.294355] ahci fc800000.sata: flags: ncq sntf pm led clo only pmp fbs pio slum part ccc apst 
[    6.294371] ahci fc800000.sata: port 0 is not capable of FBS

it looks like CBS.

Code: Select all

odroid@server:~$ sudo ls -l /sys/kernel/debug/
total 0
drwxr-xr-x   4 root root 0 Jul 15 15:47 asoc
drwxr-xr-x  12 root root 0 Jul 18 07:17 bdi
drwxr-xr-x   7 root root 0 Jan  1  1970 block
drwxr-xr-x   3 root root 0 Jan  1  1970 cec
--w-------   1 root root 0 Jan  1  1970 clear_warn_once
drwxr-xr-x 516 root root 0 Jul 15 15:47 clk
-rw-r--r--   1 root root 0 Jan  1  1970 debug_enabled
drwxr-xr-x   2 root root 0 Jan  1  1970 device_component
-r--r--r--   1 root root 0 Jul 15 15:47 devices_deferred
drwxr-xr-x   2 root root 0 Jan  1  1970 dma_buf
drwxr-xr-x   4 root root 0 Jan  1  1970 dri
drwxr-xr-x   2 root root 0 Jan  1  1970 dw-hdmi
drwxr-xr-x   2 root root 0 Jan  1  1970 dynamic_debug
-rw-r--r--   1 root root 0 Jul 15 15:47 fault_around_bytes
drwxr-xr-x   2 root root 0 Jan  1  1970 fcc00000.dwc3
drwxr-xr-x   2 root root 0 Jan  1  1970 fd000000.dwc3
-r--r--r--   1 root root 0 Jan  1  1970 gpio
drwxr-xr-x   2 root root 0 Jul 15 15:47 hid
drwxr-xr-x   2 root root 0 Jan  1  1970 iio
drwxr-xr-x   9 root root 0 Jul 15 15:47 iodomain
drwxr-xr-x   4 root root 0 Jul 15 15:47 mali0
drwxr-xr-x   2 root root 0 Jan  1  1970 memblock
drwxr-xr-x   2 root root 0 Jul 15 15:47 mmc0
drwxr-xr-x   3 root root 0 Jul 15 15:47 mmc1
drwxr-xr-x   3 root root 0 Jan  1  1970 mtd
drwxr-xr-x   5 root root 0 Jul 15 15:47 opp
drwxr-xr-x   4 root root 0 Jan  1  1970 pinctrl
drwxr-xr-x  11 root root 0 Jul 15 15:47 pm_genpd
drwxr-xr-x   2 root root 0 Jul 15 15:47 pm_qos
drwxr-xr-x   2 root root 0 Jan  1  1970 provoke-crash
drwxr-xr-x   5 root root 0 Jul 15 15:47 pvtm
-r--r--r--   1 root root 0 Jan  1  1970 pwm
drwxr-xr-x   2 root root 0 Jan  1  1970 ras
drwxr-xr-x  46 root root 0 Jan  1  1970 regmap
drwxr-xr-x  29 root root 0 Jan  1  1970 regulator
drwxr-xr-x   2 root root 0 Jul 15 15:47 rkrga
-rw-r--r--   1 root root 0 Jul 15 15:47 sched_debug
-rw-r--r--   1 root root 0 Jul 15 15:47 sched_features
-r--r--r--   1 root root 0 Jul 15 15:47 sleep_time
drwxr-xr-x   3 root root 0 Jan  1  1970 stmmaceth
-r--r--r--   1 root root 0 Jul 15 15:47 suspend_stats
dr-xr-xr-x   3 root root 0 Jan  1  1970 tracing
drwxr-xr-x   6 root root 0 Jan  1  1970 usb
-r--r--r--   1 root root 0 Jan  1  1970 wakeup_sources

I can't find any sata related debug output. any suggestions?

tkaiser · Post by **tkaiser** » Thu Jul 21, 2022 4:42 am

zupet wrote: ↑
Wed Jul 20, 2022 11:53 pm
Code: Select all
[    6.294355] ahci fc800000.sata: flags: ncq sntf pm led clo only pmp fbs pio slum part ccc apst 
[    6.294371] ahci fc800000.sata: port 0 is not capable of FBS
it looks like CBS.

Unfortunately yes. AFAIK (based on DT contents) I would believe it's this controller inside the recent RK SoCs: https://www.synopsys.com/dw/ipdir.php?ds=dwc_sata_ahci

So FIS-based switching should work (also mentioned in the TRMs) but maybe it's a driver or DT thing.

I've looked into Rockchip's most recent BSP code drop and the relevant message is here: https://github.com/radxa/kernel/blob/st ... ci.c#L2359 (should be the same with 4.19?)

zupet · Post by **zupet** » Thu Jul 21, 2022 10:05 am

FBS is not supported on 5.18 mainline release either.

started 4 disk raid0 write test, it looks like maximum transfer rate is capped at 300MB/s.

changing to 4 disk raid1 write test, it dropped to 62MB/s * 4 = 248MB/s total. interesting..

4 disk raid5 shows worst result, 41MB/s wirte. iostat reports first three disk doing read and write jobs, fourth one does only write (parity).
raid5 resync showed three 38MB/s reads, one 38MB/s write, total 152MB/s.

2 pairs of raid0 /dev/md0, /dev/md1 copy showed 73.0MB/s = total 146MB/s.
both arrays show 295MB/s single array fill test.

PigLover · Post by **PigLover** » Thu Jul 21, 2022 1:05 pm

Doing raid on a port multiplier is not likely to be a satisfying experience. Awesome tech to connect additional drives - but trying to multiplex near-simultaneous transfers on them may work but the transfer rates will disappoint.

tkaiser · Post by **tkaiser** » Thu Jul 21, 2022 2:18 pm

PigLover wrote: ↑
Thu Jul 21, 2022 1:05 pm
but trying to multiplex near-simultaneous transfers on them may work but the transfer rates will disappoint.

That's why getting FIS-based switching to work is part of the game since with CBS crappy performance 'by design' is all that could be expected.

tkaiser · Post by **tkaiser** » Thu Jul 21, 2022 2:27 pm

zupet wrote: ↑
Thu Jul 21, 2022 10:05 am
FBS is not supported on 5.18 mainline release either.

While it should have worked already on 3.2.1: https://youtu.be/wkpo8amhQW8?t=190

zupet · Post by **zupet** » Fri Jul 22, 2022 12:16 am

4 disk raid-z is slightly better.
it's little bit fast because raidz does not read back from disk while writing. mdadm raid5 does read some data from disk when writing new data. (why? overlapped chunks?)

Code: Select all

odroid@server:~$ sudo zpool create Test -f -o ashift=12 -O atime=off -O recordsize=128K -O xattr=sa -m /media/Test raidz wwn-0x5000c500c2f120fa wwn-0x5000c500b9cf5922 wwn-0x5000c500aaacd1d1 wwn-0x5000c500ac84b489

zpool creation parameter

Code: Select all

odroid@server:~$ iostat -d -m 60
Linux 4.19.219-odroid-arm64 (server)    07/21/22        _aarch64_       (4 CPU)
.....
Device             tps    MB_read/s    MB_wrtn/s    MB_dscd/s    MB_read    MB_wrtn    MB_dscd
mmcblk1           0.00         0.00         0.00         0.00          0          0          0
sda             104.37         0.00        41.02         0.00          0       2461          0
sdb             122.90         0.00        40.87         0.00          0       2451          0
sdc              98.43         0.00        41.02         0.00          0       2461          0
sdd             159.67         0.00        40.61         0.00          0       2436          0

60 second average

Code: Select all

odroid@server:~$ dd if=/dev/zero of=/media/Test/temp.dat bs=1M count=1M status=progress
37898682368 bytes (38 GB, 35 GiB) copied, 307 s, 123 MB/s

just fit for 1GBE.

tkaiser · Post by **tkaiser** » Mon Jul 25, 2022 4:15 pm

I've asked Rockchip about the CBS vs. FBS issue (though in the context of RK3588) and they responded with

需要配置hpriv->flags |= AHCI_HFLAG_YES_FBS;

and 2 patches:

Code: Select all

From dfdc21c0e6618ee66a3c2223c1b6b3061f79d05c Mon Sep 17 00:00:00 2001
From: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Date: Tue, 7 Dec 2021 14:39:11 +0800
Subject: [PATCH 1/2] phy: rockchip: naneng-combphy: Add php clock for rk3588

Signed-off-by: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Change-Id: I30998193795213dbd15e12ac56f4f81b461921e2
---
 drivers/phy/rockchip/phy-rockchip-naneng-combphy.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c b/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c
index 61c56b64aea1..5d6a4c861b67 100644
--- a/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c
+++ b/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c
@@ -794,6 +794,7 @@ static const struct rockchip_combphy_grfcfg rk3588_combphy_grfcfgs = {
 static const struct clk_bulk_data rk3588_clks[] = {
 	{ .id = "refclk" },
 	{ .id = "apbclk" },
+	{ .id = "phpclk" },
 };
 
 static const struct rockchip_combphy_cfg rk3588_combphy_cfgs = {
-- 
2.17.1

Code: Select all

From 123a2587939000f01f1987a0e82255aa2c3801c4 Mon Sep 17 00:00:00 2001
From: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Date: Tue, 7 Dec 2021 14:44:00 +0800
Subject: [PATCH 2/2] arm64: dts: rockchip: rk3588: Add compatible for sata

Signed-off-by: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Change-Id: Icbd198976183743210eb34df448a49a38d17e5d5
---
 arch/arm64/boot/dts/rockchip/rk3588.dtsi  | 2 +-
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/boot/dts/rockchip/rk3588.dtsi b/arch/arm64/boot/dts/rockchip/rk3588.dtsi
index 2aaeec40b6b5..934092659ecf 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588.dtsi
@@ -670,7 +670,7 @@
 	};
 
 	sata1: sata@fe220000 {
-		compatible = "snps,dwc-ahci";
+		compatible = "rockchip,rk-ahci", "snps,dwc-ahci";
 		reg = <0 0xfe220000 0 0x1000>;
 		clocks = <&cru ACLK_SATA1>, <&cru CLK_PMALIVE1>,
 			 <&cru CLK_RXOOB1>, <&cru CLK_PIPEPHY1_REF>,
diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
index 35be4eec5bcf..08b9a93e8f62 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
@@ -3330,7 +3330,7 @@
 	};
 
 	sata0: sata@fe210000 {
-		compatible = "snps,dwc-ahci";
+		compatible = "rockchip,rk-ahci", "snps,dwc-ahci";
 		reg = <0 0xfe210000 0 0x1000>;
 		clocks = <&cru ACLK_SATA0>, <&cru CLK_PMALIVE0>,
 			 <&cru CLK_RXOOB0>, <&cru CLK_PIPEPHY0_REF>,
@@ -3346,7 +3346,7 @@
 	};
 
 	sata2: sata@fe230000 {
-		compatible = "snps,dwc-ahci";
+		compatible = "rockchip,rk-ahci", "snps,dwc-ahci";
 		reg = <0 0xfe230000 0 0x1000>;
 		clocks = <&cru ACLK_SATA2>, <&cru CLK_PMALIVE2>,
 			 <&cru CLK_RXOOB2>, <&cru CLK_PIPEPHY2_REF>,
-- 
2.17.1

Haven't checked yet how this could be applied/adapted for RK3568 (BSP and mainline kernel).

zupet · Post by **zupet** » Tue Jul 26, 2022 10:26 am

tkaiser wrote: ↑

Mon Jul 25, 2022 4:15 pm

I've asked Rockchip about the CBS vs. FBS issue (though in the context of RK3588) and they responded with

需要配置hpriv->flags |= AHCI_HFLAG_YES_FBS;

and 2 patches:

Code: Select all

From dfdc21c0e6618ee66a3c2223c1b6b3061f79d05c Mon Sep 17 00:00:00 2001
From: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Date: Tue, 7 Dec 2021 14:39:11 +0800
Subject: [PATCH 1/2] phy: rockchip: naneng-combphy: Add php clock for rk3588

Signed-off-by: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Change-Id: I30998193795213dbd15e12ac56f4f81b461921e2
---
 drivers/phy/rockchip/phy-rockchip-naneng-combphy.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c b/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c
index 61c56b64aea1..5d6a4c861b67 100644
--- a/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c
+++ b/drivers/phy/rockchip/phy-rockchip-naneng-combphy.c
@@ -794,6 +794,7 @@ static const struct rockchip_combphy_grfcfg rk3588_combphy_grfcfgs = {
 static const struct clk_bulk_data rk3588_clks[] = {
 	{ .id = "refclk" },
 	{ .id = "apbclk" },
+	{ .id = "phpclk" },
 };
 
 static const struct rockchip_combphy_cfg rk3588_combphy_cfgs = {
-- 
2.17.1

Code: Select all

From 123a2587939000f01f1987a0e82255aa2c3801c4 Mon Sep 17 00:00:00 2001
From: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Date: Tue, 7 Dec 2021 14:44:00 +0800
Subject: [PATCH 2/2] arm64: dts: rockchip: rk3588: Add compatible for sata

Signed-off-by: Yifeng Zhao <yifeng.zhao@rock-chips.com>
Change-Id: Icbd198976183743210eb34df448a49a38d17e5d5
---
 arch/arm64/boot/dts/rockchip/rk3588.dtsi  | 2 +-
 arch/arm64/boot/dts/rockchip/rk3588s.dtsi | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/arch/arm64/boot/dts/rockchip/rk3588.dtsi b/arch/arm64/boot/dts/rockchip/rk3588.dtsi
index 2aaeec40b6b5..934092659ecf 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588.dtsi
@@ -670,7 +670,7 @@
 	};
 
 	sata1: sata@fe220000 {
-		compatible = "snps,dwc-ahci";
+		compatible = "rockchip,rk-ahci", "snps,dwc-ahci";
 		reg = <0 0xfe220000 0 0x1000>;
 		clocks = <&cru ACLK_SATA1>, <&cru CLK_PMALIVE1>,
 			 <&cru CLK_RXOOB1>, <&cru CLK_PIPEPHY1_REF>,
diff --git a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
index 35be4eec5bcf..08b9a93e8f62 100644
--- a/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
+++ b/arch/arm64/boot/dts/rockchip/rk3588s.dtsi
@@ -3330,7 +3330,7 @@
 	};
 
 	sata0: sata@fe210000 {
-		compatible = "snps,dwc-ahci";
+		compatible = "rockchip,rk-ahci", "snps,dwc-ahci";
 		reg = <0 0xfe210000 0 0x1000>;
 		clocks = <&cru ACLK_SATA0>, <&cru CLK_PMALIVE0>,
 			 <&cru CLK_RXOOB0>, <&cru CLK_PIPEPHY0_REF>,
@@ -3346,7 +3346,7 @@
 	};
 
 	sata2: sata@fe230000 {
-		compatible = "snps,dwc-ahci";
+		compatible = "rockchip,rk-ahci", "snps,dwc-ahci";
 		reg = <0 0xfe230000 0 0x1000>;
 		clocks = <&cru ACLK_SATA2>, <&cru CLK_PMALIVE2>,
 			 <&cru CLK_RXOOB2>, <&cru CLK_PIPEPHY2_REF>,
-- 
2.17.1

Haven't checked yet how this could be applied/adapted for RK3568 (BSP and mainline kernel).

great.

hardkernel's 4.19 has similar "clk_bulk_data rk3568_clks[]" and compatible = "snps,dwc-ahci"; settings.

https://github.com/hardkernel/linux/blo ... phy.c#L622
https://github.com/hardkernel/linux/blo ... .dtsi#L494

mainline 5.18 does not have "static const struct clk_bulk_data rk3588_clks[]" variable and already using rockchip's ahci driver. compatible = "rockchip,rk3568-dwc-ahci", "snps,dwc-ahci";

https://github.com/tobetter/linux/blob/ ... phy.c#L557
https://github.com/tobetter/linux/blob/ ... 8.dtsi#L12

I'll change image back to 4.19 and try to patch it.

zupet · Post by **zupet** » Tue Jul 26, 2022 11:00 am

zupet wrote: ↑
Tue Jul 26, 2022 10:26 am
https://github.com/hardkernel/linux/blo ... phy.c#L622
https://github.com/hardkernel/linux/blo ... .dtsi#L494

https://github.com/hardkernel/linux/blo ... rm.txt#L11

there is no devicetree for rk-ahci on 4.19.

zupet wrote: ↑
Tue Jul 26, 2022 10:26 am
https://github.com/tobetter/linux/blob/ ... phy.c#L557
https://github.com/tobetter/linux/blob/ ... 8.dtsi#L12

https://github.com/tobetter/linux/blob/ ... m.yaml#L54

it look like rockchip,rk3568-dwc-ahci = snps,dwc-ahci on 5.18.

I should search for rockchip,rk-ahci

zupet · Post by **zupet** » Wed Jul 27, 2022 10:23 am

I removed AHCI_HFLAG_YES_FBS check from kernel source and force FBS mode.
and it's working. need test but it booted, all disk recognized, read/write works.

ran mdadm sequential read/write test for minutes.

4 disk mdadm raid0 write showed 425MB/s, on four disks. (oflag=direct 298MB/s)
4 disk mdadm raid0 read showed 482MB/s. on four disks.(iflag=direct 119MB/s)

4 disk mdadm raid1 write showed 98MB/s, 392MB/s on four disks. (oflag=direct 106MB/s)
4 disk mdadm raid1 read showed 185MB/s, over 185MB/s on one disk. (iflag=direct 189MB/s)

4 disk mdadm raid5 write showed 138MB/s, 180MB/s on four disks. (oflag=direct 49MB/s)
4 disk mdadm raid5 read showed 308MB/s, 308MB/s on four disks. (iflag=direct 84MB/s)
4 disk mdadm raid5 resync 210MB/s read + 70MB/s write, total 280MB/s

raid5 used all disks for read !! why not on raid1 ?

I should test CBS again. mdadm raid does not like oflag=direct/iflag=direct for some tests.
and make zfs-dkms work with modified kernel.

zupet · Post by **zupet** » Wed Jul 27, 2022 11:40 am

back to unmodified 5.18.0-odroid-arm64.

Code: Select all

ahci fc800000.sata: port 0 is not capable of FBS

4 disk mdadm raid0 write showed 207MB/s, on four disks. (oflag=direct 298MB/s)
4 disk mdadm raid0 read showed 169MB/s. on four disks. (iflag=direct 120MB/s)

4 disk mdadm raid1 write showed 48MB/s, 192MB/s on four disks. (oflag=direct 52MB/s)
4 disk mdadm raid1 read showed 185MB/s, 185MB/s on single disk. (iflag=direct 187MB/s)

4 disk mdadm raid5 write showed 105MB/s, 140MB/s on four disks. (oflag=direct 43.5MB/s)
4 disk mdadm raid5 read showed 103MB/s, 103MB/s on four disks. (iflag=direct 83.4MB/s)

tkaiser · Post by **tkaiser** » Wed Jul 27, 2022 3:02 pm

zupet wrote: ↑
Wed Jul 27, 2022 10:23 am
I removed AHCI_HFLAG_YES_FBS check from kernel source and force FBS mode.

Thank you! So we know the SATA implementation supports FIS-based switching and it's just a driver/DT problem preventing FBS getting negotiated

raid5 used all disks for read !! why not on raid1 ?

Since mdraid is designed that way? IMO nobody should use mdraid's raid1 (better do btrfs raid1 or a zMirror) but if you want to stick with mdraid always use raid10 instead: https://archive.ph/5F7Lb (this works even with just 2 disks since almost a decade).

Do you have by accident a bunch of SSDs lying around since with those HDDs that perform really worse with random I/O it will be hard to measure FBS benefits anyway?

Then a test with fio, no mdraid and mixed write/read loads on all disks would give a better picture.

But with HDDs a 'use case' where the difference could be seen is a RAIDz resilver with tons of live data and a ZFS version not already implementing sequential resilver. Though with spinning rust the differences might not be that noticeable anyway...

zupet · Post by **zupet** » Wed Jul 27, 2022 5:01 pm

tkaiser wrote: ↑
Wed Jul 27, 2022 3:02 pm
Thank you! So we know the SATA implementation supports FIS-based switching and it's just a driver/DT problem preventing FBS getting negotiated

now I'll find way to add and select "snps,rk3568-ahci" with AHCI_HFLAG_YES_FBS flag. simple and clean.

tkaiser wrote: ↑
Wed Jul 27, 2022 3:02 pm
Do you have by accident a bunch of SSDs lying around since with those HDDs that perform really worse with random I/O it will be hard to measure FBS benefits anyway?

I have few nvme but no sata SSDs. I'll test it out if I'm lucky enough to access SSDs. there was a change few weeks ago, but it's on service.

zupet · Post by **zupet** » Thu Jul 28, 2022 2:09 pm

I'm back to hardkernel's 4.19 kernel. with custom FBS forced kernel, zfs performance doubled.

4 disk raidz

sequential 100GB file write 209MB/s
sequential 100GB file read 211MB/s
scrub 400MB/s

3 disk raidz resilver

resilvering 100GB data showed 274MB/s from zpool report, iostat showed 182MB/s read 91MB/s write.

tkaiser · Post by **tkaiser** » Thu Jul 28, 2022 11:22 pm

zupet wrote: ↑
Thu Jul 28, 2022 2:09 pm
zfs performance doubled.

That's great to hear. Now that we as community (mostly you) done the work to show the benefits it's time to hand over to SBC manufacturer to figure out how to patch kernel/DT to negotiate FIS-based switching only when available (there are SATA PMs like JMB321 that are only capable of CBS). At least SBC manufacturer can contact Rockchip support so now they should do their part of work.

resilvering 100GB data showed 274MB/s from zpool report, iostat showed 182MB/s read 91MB/s write.

That's exactly one of those use cases where even with spinning rust a performance boost is to be expected: mixed read/write workloads. And the best: it's not a synthetic benchmark but some real-world task that should be finished ASAP to save the user's ass once a failed disk needs to be replaced

zupet · Post by **zupet** » Mon Aug 01, 2022 12:18 am

tesed with 6 WD 12TB disks. PM can handle 5 disks, last one is used for swap and run resilver.

raid5/zfs write performance was not fas as expected. I think it's bottlenecked on software side. (to avoid write holes ? I'll try zfs cache.)
raid0 write/read and raid5 read showed good result. zfs resilver on FBS was great, fast as raid0 bandwidth.
JM858 PCIe 5 port SATA controller showed better performance, but JM575 can free PCIe.

raid0 - CBS
100GB write 291 MB/s (oflag=direct 299 MB/s)
100GB read 245 MB/s (iflag=direct 337 MB/s)
raid0 - FBS
100GB write 413 MB/s (oflag=direct 348 MB/s)
100GB read 519 MB/s (iflag=direct 385 MB/s)

raid1 - CBS
100GB write 57.4 MB/s (oflag=direct 27.8 MB/s)
100GB read 206 MB/s (iflag=direct 203 MB/s)
raid1 - FBS
100GB write 90.1 MB/s (oflag=direct 91.9 MB/s)
100GB read 203 MB/s (iflag=direct 203 MB/s)

raid5 - CBS
100GB write 149 MB/s (oflag=direct 47.3 MB/s)
100GB read 251 MB/s (iflag=direct 333 MB/s)
raid5 - FBS
100GB write 201 MB/s (oflag=direct 73.5 MB/s)
100GB read 527 MB/s (iflag=direct 376 MB/s)

5 disk raidz - CBS
100GB write 76.0 MB/s
100GB read 120 MB/s
100GB scrub 163 MB/s
100GB resilver 130 MB/s
5 disk raidz - FBS
100GB write 201 MB/s
100GB read 189 MB/s
100GB scrub 264 MB/s
100GB resilver 397 MB/s

4 disk raidz - FBS
100GB write 206 MB/s
100GB read 219 MB/s

3 disk raidz - FBS
100GB write 209 MB/s
100GB read 245 MB/s

2 disk raidz - FBS
100GB write 162 MB/s
100GB read 152 MB/s

2 disk mirror - FBS
100GB write 146 MB/s
100GB read 194 MB/s

tkaiser · Post by **tkaiser** » Tue Aug 02, 2022 4:55 pm

I took your RAIDz numbers and added it to the end of RK3588/Rock 5B SATA section: https://github.com/ThomasKaiser/Knowled ... 5B.md#sata

It should serve as a hint for SBC manufacturers using RK3566, RK3568 and RK3588 (ODROID M2) that when exposing SATA they also should get things straight software-wise and enable FIS-based switching since other than public perception SATA PMs are fine for attaching a bunch of spinning rust to a single SATA port... but surely not with CBS

tkaiser · Post by **tkaiser** » Tue Aug 02, 2022 5:06 pm

@zupet: I guess you knew the request would be coming...

Have you already tried to attach the JMB575 to the JMS585 in the M.2 slot?

M1 + 1 x JMS585 + 6 x JMB575 = 30 SATA devices connected

zupet · Post by **zupet** » Tue Aug 02, 2022 8:24 pm

tkaiser wrote: ↑
Tue Aug 02, 2022 5:06 pm
@zupet: I guess you knew the request would be coming...

Have you already tried to attach the JMB575 to the JMS585 in the M.2 slot?

M1 + 1 x JMS585 + 6 x JMB575 = 30 SATA devices connected

https://ko.aliexpress.com/item/1005004417694518.html

sorry. I just ordered this one today.

tkaiser · Post by **tkaiser** » Tue Aug 02, 2022 9:10 pm

zupet wrote: ↑
Tue Aug 02, 2022 8:24 pm
https://ko.aliexpress.com/item/1005004417694518.html

Sorry, the page you requested can not be found:(

zupet · Post by **zupet** » Tue Aug 02, 2022 9:40 pm

tkaiser wrote: ↑
Tue Aug 02, 2022 9:10 pm

zupet wrote: ↑
Tue Aug 02, 2022 8:24 pm
https://ko.aliexpress.com/item/1005004417694518.html

Sorry, the page you requested can not be found:(

https://www.aliexpress.com/item/1005004417694518.html

how about this link? product is

"IOCREST M.2 PCIe3.0 to 8 Ports SATA 6G Multiplier Controller Card B/M key NGFF Each Port Arrive 6Gbps with 2 Mini SAS Interface"

you can find it at IOCREST Official Store.

ODROID

JMB575 SATA multiplier just works !!

JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Re: JMB575 SATA multiplier just works !!

Who is online