XU4Q USB3 high load

Post Reply
texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

XU4Q USB3 high load

Unread post by texcor » Sat Jul 06, 2019 9:41 pm

Hi,

New user of hardkernel products, I'm using an XU4Q now.
Or rather, I'm trying to replace my old setup with this one.
I'm having a lot of trouble in the USB3 layer (running Ubuntu 18.04.2 LTS with kernel 4.14.127-164).

Does XU4Q really works with USB3 under high load?

Many freeze or FS errors like

Code: Select all

[13500.957200] xhci-hcd xhci-hcd.3.auto: WARN Wrong bounce buffer write length: 1024 != 0
[13517.067744] xhci-hcd xhci-hcd.3.auto: WARN Wrong bounce buffer write length: 1024 != 0
or

Code: Select all

[12445.425122] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12450.800995] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12451.008763] usb 4-1.2: device not accepting address 5, error -62
[12456.432878] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12461.808891] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12462.016623] usb 4-1.2: device not accepting address 5, error -62
[12467.440845] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12472.816674] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12473.024414] usb 4-1.2: device not accepting address 5, error -62
[12478.448491] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12483.824427] xhci-hcd xhci-hcd.3.auto: Timeout while waiting for setup device command
[12484.032132] usb 4-1.2: device not accepting address 5, error -62
[12484.037391] usb 4-1.2: USB disconnect, device number 5
[12484.052271] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00
[12484.052406] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 66 02 61 e8 00 00 20 00
[12484.052495] print_req_error: I/O error, dev sda, sector 1711432168
[12484.057795] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x01 driverbyte=0x00
[12484.057849] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x35 35 00 00 00 00 00 00 00 00 00
[12484.057906] print_req_error: I/O error, dev sda, sector 1954915053
[12484.063564] XFS (sda1): metadata I/O error: block 0x74859aed ("xlog_iodone") error 5 numblks 64
[12484.063672] print_req_error: I/O error, dev sda, sector 1954915117
[12484.063735] XFS (sda1): metadata I/O error: block 0x3d89f218 ("xfs_buf_iodone_callback_error") error 5 numblks 8
[12484.072144] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1232 of file fs/xfs/xfs_log.c.  Return address = 0xbf31bb78
[12484.078856] XFS (sda1): metadata I/O error: block 0x74859b2d ("xlog_iodone") error 5 numblks 64
[12484.088432] XFS (sda1): Log I/O Error Detected.  Shutting down filesystem
[12484.097044] XFS (sda1): xfs_do_force_shutdown(0x2) called from line 1232 of file fs/xfs/xfs_log.c.  Return address = 0xbf31bb78
[12484.103842] XFS (sda1): Please umount the filesystem and rectify the problem(s)
[12484.111284] XFS (sda1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
[12484.111659] XFS (sda1): xfs_imap_to_bp: xfs_trans_read_buf() returned error -5.
[12484.168156] XFS (sda1): Failing async write on buffer block 0x3d9fe860. Retrying async write.
[12484.168170] XFS (sda1): Failing async write on buffer block 0x3dba0cc8. Retrying async write.
[12484.168181] XFS (sda1): Failing async write on buffer block 0x3db32470. Retrying async write.
[12484.168191] XFS (sda1): Failing async write on buffer block 0x3e1cbb10. Retrying async write.
[12484.168204] XFS (sda1): Failing async write on buffer block 0x3db53388. Retrying async write.
[12484.168215] XFS (sda1): Failing async write on buffer block 0x3da36808. Retrying async write.
[12484.168225] XFS (sda1): Failing async write on buffer block 0x3e272f50. Retrying async write.
[12484.168238] XFS (sda1): Failing async write on buffer block 0x3d89f900. Retrying async write.
[12484.168246] XFS (sda1): Failing async write on buffer block 0x3e26bf90. Retrying async write.
[12484.168255] XFS (sda1): Failing async write on buffer block 0x3d8a3680. Retrying async write.
[12484.223566] sd 0:0:0:0: [sda] Synchronizing SCSI cache
[12484.223715] sd 0:0:0:0: [sda] Synchronize Cache(10) failed: Result: hostbyte=0x01 driverbyte=0x00
or

Code: Select all

[5.662797] usb 4-1: reset SuperSpeed USB device number 2 using xhci-hcd
What I have done:
- Use separate power for external HDD
- Use multiple strong PSU 5V 5A+
- Use strong and custom build cables
- Use multiple HDD
- Downclock CPUs
- Disable USB UAS
- Use different `max_sectors_kb` queue
- Cool CPUs to stay below 50°C

Current interrupts status:

Code: Select all

root@hostname:~# cat /proc/interrupts 
           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
 49:          0          0          0          0          0          0          0          0  COMBINER 187 Edge      mct_comp_irq
 50:     297960          0          0          0          0          0          0          0     GICv2 152 Edge      mct_tick0
 51:          0     162606          0          0          0          0          0          0     GICv2 153 Edge      mct_tick1
 52:          0          0      82730          0          0          0          0          0     GICv2 154 Edge      mct_tick2
 53:          0          0          0      41115          0          0          0          0     GICv2 155 Edge      mct_tick3
 54:          0          0          0          0    1347032          0          0          0     GICv2 160 Edge      mct_tick4
 55:          0          0          0          0          0    1038494          0          0     GICv2 161 Edge      mct_tick5
 56:          0          0          0          0          0          0     561472          0     GICv2 162 Edge      mct_tick6
 57:          0          0          0          0          0          0          0     323214     GICv2 163 Edge      mct_tick7
 60:       1312          0          0          0          0          0          0          0     GICv2  85 Edge      12c20000.serial
 62:          0          0          0          0          0          0          0          0     GICv2  89 Edge      12c70000.i2c
 63:          0          0          0          0          0          0          0          0       PMU  43 Edge      s3c2410-rtc alarm
 64:          0          0          0          0          0          0          0          0       PMU  44 Edge      s3c2410-rtc tick
 68:          0          0          0          0          0          0          0          0     GICv2  74 Edge      101d0000.watchdog
 69:          0          0          0          0          0          0          0          0     GICv2 192 Edge      arm-pmu
 70:          0          0          0          0          0          0          0          0     GICv2 193 Edge      arm-pmu
 71:          0          0          0          0          0          0          0          0     GICv2 194 Edge      arm-pmu
 72:          0          0          0          0          0          0          0          0     GICv2 195 Edge      arm-pmu
 73:          0          0          0          0          0          0          0          0  COMBINER  10 Edge      arm-pmu
 74:          0          0          0          0          0          0          0          0  COMBINER  56 Edge      arm-pmu
 75:          0          0          0          0          0          0          0          0  COMBINER 134 Edge      arm-pmu
 76:          0          0          0          0          0          0          0          0  COMBINER 154 Edge      arm-pmu
 77:          0          0          0          0          0          0          0          0     GICv2 144 Edge      10830000.sss
 78:    9902636          0          0          0          0          0          0          0     GICv2  92 Edge      12ca0000.i2c
 79:          0          0          0          0          0          0          0          0     GICv2  93 Edge      12cb0000.i2c
 80:          1          0          0          0          0          0          0          0     GICv2 103 Edge      ehci_hcd:usb1, ohci_hcd:usb2
 81:          4          0          0          0          0          0          0          0     GICv2 128 Edge      11000000.codec
 82:          0          0          0          0          0          0          0          0     GICv2 123 Edge      drm_g2d
 83:          0          0          0          0          0          0          0          0     GICv2 107 Edge      dw-mci
 84:   18750299          0          0          0          0          0          0          0     GICv2 109 Edge      dw-mci
 85:          0          0          0          0          0          0          0          0     GICv2  77 Edge      13400000.pinctrl
103:          0          0          0          0          0          0          0          0     GICv2 110 Edge      13410000.pinctrl
104:          0          0          0          0          0          0          0          0     GICv2  78 Edge      14000000.pinctrl
105:          0          0          0          0          0          0          0          0     GICv2  82 Edge      14010000.pinctrl
106:          0          0          0          0          0          0          0          0     GICv2  79 Edge      3860000.pinctrl
107:          0          0          0          0          0          0          0          0     GICv2 142 Edge      3880000.adma
108:         27          0          0          0          0          0          0          0     GICv2  66 Edge      121a0000.pdma
109:          0          0          0          0          0          0          0          0     GICv2  67 Edge      121b0000.pdma
110:          0          0          0          0          0          0          0          0     GICv2  65 Edge      10800000.mdma
111:          0          0          0          0          0          0          0          0     GICv2 101 Edge      spi-s3c64xx
112:          0          0          0          0          0          0          0          0     GICv2 138 Edge      12d10000.adc
114:          0          0          0          0          0          0          0          0     GICv2 126 Edge      drm_mixer
115:          0          0          0          0          0          0          0          0     GICv2 116 Edge      drm_rotator
116:          0          0          0          0          0          0          0          0     GICv2 117 Edge      13e00000.video-scaler
117:          0          0          0          0          0          0          0          0     GICv2 118 Edge      13e10000.video-scaler
118:          0          0          0          0          0          0          0          0     GICv2 121 Edge      11f50000.jpeg
119:          0          0          0          0          0          0          0          0     GICv2 200 Edge      11f60000.jpeg
120:       7843          0          0          0          0          0          0          0     GICv2  97 Edge      10060000.tmu
121:       1322          0          0          0          0          0          0          0     GICv2 215 Edge      10064000.tmu
122:        791          0          0          0          0          0          0          0     GICv2 216 Edge      10068000.tmu
123:        566          0          0          0          0          0          0          0     GICv2 217 Edge      1006c000.tmu
124:        266          0          0          0          0          0          0          0     GICv2 247 Edge      100a0000.tmu
125:          0          0          0          0          0          0          0          0  COMBINER 197 Edge      10a60000.sysmmu
126:          0          0          0          0          0          0          0          0  COMBINER 178 Edge      10a70000.sysmmu
127:          0          0          0          0          0          0          0          0  COMBINER  60 Edge      14650000.sysmmu
128:          0          0          0          0          0          0          0          0  COMBINER  16 Edge      13e80000.sysmmu
129:          0          0          0          0          0          0          0          0  COMBINER  18 Edge      13e90000.sysmmu
130:          0          0          0          0          0          0          0          0  COMBINER 180 Edge      12880000.sysmmu
131:          0          0          0          0          0          0          0          0     GICv2 218 Edge      12890000.sysmmu
132:          0          0          0          0          0          0          0          0     GICv2 220 Edge      128a0000.sysmmu
133:          0          0          0          0          0          0          0          0  COMBINER 218 Edge      128c0000.sysmmu
134:          0          0          0          0          0          0          0          0  COMBINER 182 Edge      128d0000.sysmmu
135:          0          0          0          0          0          0          0          0  COMBINER 158 Edge      128e0000.sysmmu
136:          0          0          0          0          0          0          0          0  COMBINER  32 Edge      11d40000.sysmmu
137:          0          0          0          0          0          0          0          0  COMBINER  34 Edge      11f10000.sysmmu
138:          0          0          0          0          0          0          0          0     GICv2 201 Edge      11f20000.sysmmu
139:          0          0          0          0          0          0          0          0  COMBINER  50 Edge      11200000.sysmmu
140:          0          0          0          0          0          0          0          0  COMBINER  69 Edge      11210000.sysmmu
141:          0          0          0          0          0          0          0          0  COMBINER  26 Edge      14640000.sysmmu
142:          0          0          0          0          0          0          0          0  COMBINER  24 Edge      14680000.sysmmu
143:          0          0          0          0          0          0          0          0     GICv2 251 Edge      11800000.mali
144:          0          0          0          0          0          0          0          0     GICv2 106 Edge      11800000.mali
145:          1          0          0          0          0          0          0          0     GICv2 149 Edge      11800000.mali
146:          1          0          0          0          0          0          0          0  exynos4210_wkup_irq_chip   7 Edge      hdmi
147:          0          0          0          0          0          0          0          0  exynos4210_wkup_irq_chip   4 Edge      s2mps11
148:          0          0          0          0          0          0          0          0   s2mps11  10 Edge      rtc-alarm0
149:       3102          0          0          0          0    4789653          0          0     GICv2 104 Edge      xhci-hcd:usb3
150:        259          0          0          0          0          0    6469442          0     GICv2 105 Edge      xhci-hcd:usb5
151:          0          0          0          0          0          0          0          0  exynos4210_wkup_irq_chip   5 Edge      extcon_usb3_0
152:          1          0          0          0          0          0          0          0  exynos4210_wkup_irq_chip   1 Edge      extcon_usb3_1
153:          0          0          0          0          0          0          0          0  exynos4210_wkup_irq_chip   4 Edge      extcon_usb3_1
154:          0          0          0          0          0          0          0          0  exynos4210_wkup_irq_chip   3 Edge      power key
IPI0:          0          0          0          0          0          0          0          0  CPU wakeup interrupts
IPI1:          0          0          0          0          0          0          0          0  Timer broadcast interrupts
IPI2:    1213448     542227     193750     100452   23785853    7348401    3938460    2486870  Rescheduling interrupts
IPI3:     275588     220807     271385    2687753        869       1379       1026        983  Function call interrupts
IPI4:          0          0          0          0          0          0          0          0  CPU stop interrupts
IPI5:     316266      24963      10082       8219     342643     553158      60757      24215  IRQ work interrupts
IPI6:          0          0          0          0          0          0          0          0  completion interrupts
Err:          0

User avatar
odroid
Site Admin
Posts: 31363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 50 times
Been thanked: 184 times
Contact:

Re: XU4Q USB3 high load

Unread post by odroid » Mon Jul 08, 2019 10:55 am

Do you use EXT4 file system with multiple USB HDDs?
Did you set "performance" governor?
https://wiki.odroid.com/odroid-xu4/appl ... q_govornor

joy
Posts: 841
Joined: Fri Oct 02, 2015 1:44 pm
languages_spoken: english
ODROIDs: ODROID-C1+, XU4, X
Has thanked: 7 times
Been thanked: 35 times
Contact:

Re: XU4Q USB3 high load

Unread post by joy » Mon Jul 08, 2019 3:29 pm


texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4Q USB3 high load

Unread post by texcor » Mon Jul 08, 2019 3:38 pm

Default CPU governor (ondemand).
Only one USB HDD at same time.
I've use EXT4 or XFS.

Multiple crash happens, repair the catalog works, but there is data loss (inode in use).
Then, filesystem crash with directory catalog completely broken, 2TB of data in the trash.

Never happened with other industrial cards in my possession

User avatar
odroid
Site Admin
Posts: 31363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 50 times
Been thanked: 184 times
Contact:

Re: XU4Q USB3 high load

Unread post by odroid » Mon Jul 08, 2019 5:27 pm

Please try the performance CPU governor to narrow down root causes.

texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4Q USB3 high load

Unread post by texcor » Mon Jul 08, 2019 5:44 pm

ok, i'll try that.

I think there are voltage irregularities in the USB bus, or EMI.
I don't understand how increasing this load with governor will fix that

texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4Q USB3 high load

Unread post by texcor » Wed Jul 10, 2019 10:57 pm

Using ext4 with performance CPU governor

Code: Select all

[273650.920016] usb 4-1.2: reset SuperSpeed USB device number 3 using xhci-hcd
[273650.942977] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=0x00
[273650.942986] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 16 40 0e 10 00 00 f8 00
[273650.942993] print_req_error: I/O error, dev sda, sector 373296656
Harddisk works fine.

User avatar
odroid
Site Admin
Posts: 31363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 50 times
Been thanked: 184 times
Contact:

Re: XU4Q USB3 high load

Unread post by odroid » Thu Jul 11, 2019 9:57 am

Can you check with smaller queue size after enabling UAS mode?

Code: Select all

# echo 32 > /sys/class/block/sda/queue/max_sectors_kb
# echo 32 > /sys/class/block/sdb/queue/max_sectors_kb

texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4Q USB3 high load

Unread post by texcor » Fri Jul 12, 2019 1:37 am

Same result with low "max_sectors_kb"

Code: Select all

[375872.014298] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=0x00
[375872.014309] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 2a 80 12 70 00 00 40 00
[375872.014317] print_req_error: I/O error, dev sda, sector 713036400
[376066.977478] usb 4-1.2: reset SuperSpeed USB device number 3 using xhci-hcd
[376067.000408] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=0x00
[376067.000418] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 05 40 2c a8 00 00 40 00
[376067.000426] print_req_error: I/O error, dev sda, sector 88091816
[376091.176764] usb 4-1.2: reset SuperSpeed USB device number 3 using xhci-hcd
[376091.199896] sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=0x00
[376091.199908] sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x28 28 00 66 80 0e 10 00 00 40 00
[376091.199917] print_req_error: I/O error, dev sda, sector 1719668240

User avatar
odroid
Site Admin
Posts: 31363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 50 times
Been thanked: 184 times
Contact:

Re: XU4Q USB3 high load

Unread post by odroid » Fri Jul 12, 2019 8:29 am

Which USB-to-SATA bridge do you use?
Do you have the issue with only one HDD connection?
We couldn't reproduce the issue with multiple heavy dd tests as well as smartctl long test.
viewtopic.php?f=99&t=35107

Have you cleaned the connector?
viewtopic.php?f=147&t=30446&p=218401#p218401

texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4Q USB3 high load

Unread post by texcor » Fri Jul 12, 2019 3:15 pm

There's no USB-to-SATA bridge, hard disk use USB3 direct interface on PCB board.
A few days ago, I tried with two different disks (HD and SSD), which use separate USB-SATA converters. Same result.
Connector are cleaned.

texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4Q USB3 high load

Unread post by texcor » Fri Jul 12, 2019 6:06 pm

https://www.google.com/search?q=odroid+ ... USB+device"

You must to admit boards have a problem.
Sell computers with a real USB3 port as a selling point, but this one is not fully functional.

Agree to refund the purchase price, shipping and customs taxes?
XU4Q is useless, $100 directly in the trash.

User avatar
odroid
Site Admin
Posts: 31363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 50 times
Been thanked: 184 times
Contact:

Re: XU4Q USB3 high load

Unread post by odroid » Mon Jul 15, 2019 10:58 am

We connected two SSDs to XU4 via this bridge boards. SSDs were formatted with EXT4.
https://www.hardkernel.com/shop/usb3-0- ... oard-plus/
And ran some multiple DD tests over several hours on the same Kernel 4.14.127-164 and there was no issue so far.

Can you tell me the brand/model name of the HDD which has a built-in USB3 interface?

User avatar
odroid
Site Admin
Posts: 31363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 50 times
Been thanked: 184 times
Contact:

Re: XU4Q USB3 high load

Unread post by odroid » Tue Jul 16, 2019 8:47 am

Another side note: "noop" scheduler seems to be very worth to try.
viewtopic.php?f=147&t=35642

texcor
Posts: 8
Joined: Sat Jul 06, 2019 9:30 pm
languages_spoken: english,french
ODROIDs: XU4Q
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4Q USB3 high load

Unread post by texcor » Tue Jul 16, 2019 3:59 pm

dd is not a real test environnement, it' sequential.
run multiple rsync locally and networkly with some big and small files..

i've already test all scheduler (fifo/noop, cfq, deadline)

User avatar
odroid
Site Admin
Posts: 31363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 50 times
Been thanked: 184 times
Contact:

Re: XU4Q USB3 high load

Unread post by odroid » Tue Jul 16, 2019 4:23 pm

We will try to setup a similar test condition with rsync.
One more question.
Did you connect two USB 3.0 storage devices for the multiple rsync tests?

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 1 guest