[SOLVED] Problem with 2x USB3 Hard Drives (ODROID-XU4)

[rottobotics]

I'm trying to rsync between two WD Red hard drives. Both drives are in externally powered enclosures. I have an issue if both drives are in USB3 enclosures and connected to USB3 ports. The rync will start, get through a few dozen files, freeze, and then report:

Code: Select all

Read-only file system (30) rsync error: error in file IO (code 11) at receiver.c(378) [receiver=3.2.3]

since the drives are remounted as read-only.

Doesn't Work:
USB3 Enclosure on USB3 port & USB3 Enclosure on USB3 port
USB3 Enclosure on USB3 port & USB3 Enclosure on USB3 port, both connected through a powered USB3 hub

Works:
USB3 Enclosure on USB3 port & USB2 Enclosure on USB3 port
USB3 Enclosure on USB3 port & USB3 Enclosure on USB2 port

I've cleaned the contacts on both USB3 ports. I've also tried a different power supply.

Issue happens on both:
Armbian 23.02.2 Bullseye with Linux 5.4.230-odroidxu4
Armbian 22.11.1 Bullseye with Linux 5.4.225-odroidxu4

dmesg below:

Code: Select all

[  604.577345] xhci-hcd xhci-hcd.8.auto: xHCI host controller not responding, assume dead
[  604.578891] usb 4-1.2: cmd cmplt err -108
[  604.578910] xhci-hcd xhci-hcd.8.auto: HC died; cleaning up
[  604.667363] scsi host2: uas_eh_device_reset_handler FAILED to get lock err -19
[  604.667487] blk_update_request: I/O error, dev sdc, sector 3156931072 op 0x0:(READ) flags 0x80700 phys_seg 32 prio class 0
[  604.667904] blk_update_request: I/O error, dev sdc, sector 3156931328 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
[  604.668190] sd 2:0:0:0: rejecting I/O to offline device
[  604.668208] blk_update_request: I/O error, dev sdc, sector 3156931072 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[  604.687294] blk_update_request: I/O error, dev sdb, sector 1528161936 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[  604.687407] blk_update_request: I/O error, dev sdb, sector 1528163984 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[  604.687440] blk_update_request: I/O error, dev sdb, sector 1528165008 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[  604.687475] blk_update_request: I/O error, dev sdb, sector 1528166032 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[  604.687505] blk_update_request: I/O error, dev sdb, sector 1528167056 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[  604.687535] blk_update_request: I/O error, dev sdb, sector 1528169744 op 0x1:(WRITE) flags 0x0 phys_seg 94 prio class 0
[  604.687584] Buffer I/O error on device sdb1, logical block 191020962
[  604.687623] Buffer I/O error on device sdb1, logical block 191020963
[  604.687637] Buffer I/O error on device sdb1, logical block 191020964
[  604.687652] Buffer I/O error on device sdb1, logical block 191020965
[  604.687668] Buffer I/O error on device sdb1, logical block 191020966
[  604.687683] Buffer I/O error on device sdb1, logical block 191020967
[  604.687698] Buffer I/O error on device sdb1, logical block 191020968
[  604.687714] Buffer I/O error on device sdb1, logical block 191020969
[  604.687729] Buffer I/O error on device sdb1, logical block 191020970
[  604.687745] Buffer I/O error on device sdb1, logical block 191020971
[  604.688270] blk_update_request: I/O error, dev sdb, sector 1528062552 op 0x1:(WRITE) flags 0x4000 phys_seg 128 prio class 0
[  604.720060] Aborting journal on device sdb1-8.
[  604.720359] Buffer I/O error on dev sdb1, logical block 0, lost sync page write
[  604.720424] Buffer I/O error on dev sdb1, logical block 121667584, lost sync page write
[  604.720478] JBD2: Error -5 detected when updating journal superblock for sdb1-8.
[  604.720512] EXT4-fs (sdb1): I/O error while writing superblock
[  604.720533] EXT4-fs error (device sdb1): ext4_journal_check_start:61: Detected aborted journal
[  604.720550] EXT4-fs (sdb1): Remounting filesystem read-only
[  604.720654] Buffer I/O error on dev sdb1, logical block 0, lost sync page write
[  604.720691] EXT4-fs (sdb1): I/O error while writing superblock
[  604.727560] EXT4-fs error (device sdb1) in ext4_reserve_inode_write:6006: Journal has aborted
[  604.727605] Buffer I/O error on dev sdb1, logical block 0, lost sync page write
[  604.727620] EXT4-fs (sdb1): I/O error while writing superblock
[  604.727627] EXT4-fs error (device sdb1): mpage_map_and_submit_extent:2631: comm kworker/u16:3: Failed to mark inode 49290501 dirty
[  604.727657] Buffer I/O error on dev sdb1, logical block 0, lost sync page write
[  604.727670] EXT4-fs (sdb1): I/O error while writing superblock
[  607.253235] Buffer I/O error on dev sdc1, logical block 488144896, lost sync page write
[  607.253251] JBD2: Error -5 detected when updating journal superblock for sdc1-8.
[  607.253257] Aborting journal on device sdc1-8.
[  607.253270] Buffer I/O error on dev sdc1, logical block 488144896, lost sync page write
[  607.253278] JBD2: Error -5 detected when updating journal superblock for sdc1-8.

[ruppi]

The high current consumption of the WD Red Hard drive seems to affect the power supply(5V Power) of the ODROID-XU4. In ODROID-XU4, it's an inherent limit.
What Power Supply are you using?
If possible, try Power Supply 6A or higher.
How about connecting USB Enclosure one by one after booting up on ODROID-XU4?

[rottobotics]

The enclosures are externally powered. I've also tried connecting both through an externally powered USB hub. If it were an external power supply issue, I would probably see the device reboot. It could be an insufficient internal supply for the USB controller, which would make more sense because it seems to be throughput dependent.

[rottobotics]

I found this thread:
viewtopic.php?f=184&t=42630

even setting /sys/class/block/sdb/queue/max_sectors_kb and /sys/class/block/sdc/queue/max_sectors_kb to 8 which causes slow (~8-12MB/s) transfer speeds, the issue still occurs.

[rottobotics]

Both devices are using the uas driver, I'm going to try using usb-storage instead.

Code: Select all

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/2p, 5000M
        |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=uas, 5000M
        |__ Port 2: Dev 4, If 0, Class=Mass Storage, Driver=uas, 5000M

The enclosures use an ASM bridge

Code: Select all

Bus 001 Device 002: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge, ASM1153E SATA 6Gb/s bridge

[rottobotics]

I was able to get the drive to use usb-storage by modifying armbianEnv.txt

Code: Select all

$ cat /boot/armbianEnv.txt 
board_name=xu4
usbstoragequirks=0x2537:0x1066:u,0x2537:0x1068:u,0x174c:0x55aa:u

$ sudo update-initramfs -u

Code: Select all

/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/2p, 5000M
        |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M

which allowed the rync to finish for the first time with both drives using USB3 at 72.69M bytes/sec.

write speed from /dev/zero:

Code: Select all

$ sync; dd if=/dev/zero of=tempfile bs=1M count=1024; sync
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.8953 s, 98.6 MB/s

[rottobotics]

If I open the enclosure, the chip-set is actually an ASM225CM

[rottobotics]

rysnc of ~100G completed successfully at 77.32M bytes/sec between two USB3 drives on USB3 ports, will rename the subject of this thread as solved

[ruppi]

I'm glad to hear the good news.