Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Test and fix the Kernel 4.9 features

Moderators: mdrjr, odroid

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby crazyquark » Thu Jul 13, 2017 2:07 pm

Yeah, I think so too. On kernel 3.10 things were mostly stable.
It did happen on 3.10 as well but it was rare and I think, like I said, it started happening after I switched my network to gigabit.
On 3.10 I could workaround it by limiting the download speed on qBittorrent to 1MB/s. But it did happen as well with this HDD, just not very often.
crazyquark
 
Posts: 199
Joined: Thu Jan 15, 2015 4:22 pm
languages_spoken: english, french, romanian
ODROIDs: C1,C1+,C2,XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby bronco » Thu Jul 13, 2017 3:17 pm

odroid wrote:We will do "stress" command test with Samba transfer in parallel.


You as the manufacturer/engineer could do better: start to measure (since you might understand Ohm's law). If it's a voltage drop problem then it's easy to reproduce. And stress is lightweight, better use cpuminer. I recommended to @crazyquark to test with stress since he's able to trigger the well known and old
Code: Select all
usb 4-1.2: reset SuperSpeed USB device number 3 using xhci-hcd

problem with rather light workloads (after switching to Gigabit Ethernet which results in an overall higher board consumption). He should be able to nail the problem down without network being involved with the script from yesterday. But as usual people prefer developing theories over testing ;)

odroid wrote:there have been very few stability issues reported while we've shipped over 5 thousands of CloudShell kits in the past one year.
So I guess it could be related to the Kernel 4.9 software issue probably even we don't know what it is exactly.


You have reports that contain already the solution: viewtopic.php?f=97&t=16912#p110383

PSUs as all other electronics also suffer from aging effects so it's pretty normal that these issues will be more over time. And it's also easy for you to check whether a correlation with kernel 4.9 and higher failure rates is real or not. Grab a powermeter, grab a multimeter, run the same tests with both kernels and draw a nice chart with overall consumption and voltage drops if related.

If 4.9 is more efficient (likely) and if the problem is related to voltage drops (very likely) then of course it will be triggered with 4.9 more often. But it's still Ohm's law and not software.
bronco
 
Posts: 27
Joined: Tue Jul 11, 2017 2:58 pm
languages_spoken: english

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Thu Jul 13, 2017 3:45 pm

I agree.
We need to prove whether the bus reset issue appears only when the voltage-drop happens or not.
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Sat Jul 15, 2017 9:14 pm

as I have said before when I use an external hdd with is powered by odroid usb3 port and do the test I still get the bus reset errors but the hdd doesn't go offline. so maybe the cloudshell is not the main issue but it exaggerates it.
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Sun Jul 16, 2017 11:01 am

Official Ubuntu 16.04 & Kernel 4.9 on eMMC with official 5V/4A PSU.
2TB HDD is connected to the old CloudShell.
Running "stress" to use all 8 cores.
Keep copying 10GB file from/to Windows PC in parallel (two Samba instances)
Keep copying a big file from/to eMMC to/from HDD in parallel.
We've run above test for 3hrs 44min. There is no USB reset issue yet.
Image

The DMM measured voltage on the DC jack is 5.1Volt and average load is 2.54Amp.
HDD SATA power pin shows 4.76~ 4.89 Volt.
We will keep running this test for 24 hours more.

We will perform the same test with the OMV image soon.

I think we can share the test result on Monday or Tuesday because it is already Friday PM 5:00 in Korea.
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Sun Jul 16, 2017 1:44 pm

reza wrote:as I have said before when I use an external hdd with is powered by odroid usb3 port and do the test I still get the bus reset errors but the hdd doesn't go offline. so maybe the cloudshell is not the main issue but it exaggerates it.


Can you tell me the brand/model name of the external HDD?
We found a slightly old Hitachi Travelstar 1TB HDD which has very high in-rush current. We will test it soon.
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Sun Jul 16, 2017 2:11 pm

adata 2tb
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby crazyquark » Sun Jul 16, 2017 3:19 pm

Nice stress test, odd that you cannot reproduce it. On my end it usually happened while I was also reading from the disk, like I said before, while streaming a movie via NFS while all the writing was also taking place.

I accidentally reproduce it on the cloudshell 2 as well but with a very different setup :)

OK, so this weekend I upgraded to cloudshell 2. My setup is a bit complicated but the important part is:
I connected the old HDD to the second USB 3.0 port via an external HDD dock; now this dock was picked up by the usb-storage driver while the cloudshell2 disks were running via uas.
I started copying my old disk to my new setup via 'rsync'. It all worked fine until I started also streaming a movie via NFS... that's when the old HDD connected via usb-storage started getting reset errors and it ended up crashing my uas disk.
So the key ingredients seem to be:
- usb storage
- reading via NFS while also writing
The writing was happening at around 60MB/s

Now, not being able to copy any further my data, I kept getting crashes, I moved my old HDD to a newer enclosure that I knew had better support. And guess what, this time the HDD was picked up by the uas driver and I was able to complete my rsync without a hitch!

Bottom line:
- use NFS for reading and write data from a local process for testing
- make sure the usb-storage driver is used
Last edited by crazyquark on Sun Jul 16, 2017 3:27 pm, edited 1 time in total.
crazyquark
 
Posts: 199
Joined: Thu Jan 15, 2015 4:22 pm
languages_spoken: english, french, romanian
ODROIDs: C1,C1+,C2,XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby crazyquark » Sun Jul 16, 2017 3:26 pm

Sorry for the previous long post. I want to propose an alternative testing strategy:

On Odroid XU4 w/ Cloudshell1 and 2TB disk attached:
- Download a large file via FTP let's say from a remote host at speeds over 2-3MB/s at least.
- On a Linux host, copy a file from Odroid via NFS
crazyquark
 
Posts: 199
Joined: Thu Jan 15, 2015 4:22 pm
languages_spoken: english, french, romanian
ODROIDs: C1,C1+,C2,XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby studioai » Sun Jul 16, 2017 10:56 pm

crazyquark wrote:Sorry for the previous long post. I want to propose an alternative testing strategy:

On Odroid XU4 w/ Cloudshell1 and 2TB disk attached:
- Download a large file via FTP let's say from a remote host at speeds over 2-3MB/s at least.
- On a Linux host, copy a file from Odroid via NFS


I'm experiencing same issue, In my case it occurs when I download files with transmission daemon.

when download or upload speed over 5mb/s USB drive resets.
studioai
 
Posts: 11
Joined: Thu Apr 21, 2016 10:43 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby crazyquark » Mon Jul 17, 2017 5:37 pm

I use qBittorrent, same, the HDD resets or even worse, it crashes when downloads exceed a certain speed.
crazyquark
 
Posts: 199
Joined: Thu Jan 15, 2015 4:22 pm
languages_spoken: english, french, romanian
ODROIDs: C1,C1+,C2,XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Mon Jul 17, 2017 6:27 pm

We performed further tests with various HDDs what we acquired recently and a few different input voltages.
We found that Seagate 2TB and HGST/Hitachi 1TB HDDs are quite sensitive to the input voltage level while WD 1TB/500GB, Samsung Momentum and Toshiba HDDs are working well even with 4.4Volt input.
The Seagate and HGST HDDs are generating the bus reset error when the voltage on the SATA port is lower than 4.7Volt.
It is the main reason why we couldn't reproduce the issue for a long time.
I think the disk input voltage tolerance seems to be different from each model of HDDs even with the same brand or manufacturer.
So please let us know your HDD brand and model name to confirm our test result.

BTW, the typical output voltage on our 5V/4A and 5V/6A PSU is 5.25Volt at light load. When the load is very high, it can be down to 5.05volt.
But some of them can be lower than 5Volt and the SATA port voltage can be lower than 4.7Volt due to the variant characteristics of each PSU.
We could observe 200~400mV of voltage drops by the resistance of FETs in the protection ICs as well as cables/connectors.
It is very worth to measure the voltage of VBUS on the USB connector if you have a DMM.

Other major root causes : There are two protection ICs on the XU4 board.
The first one is NCP372 which is placed near the DC jack to block high-voltage, low-voltage and reverse voltage from the DC plug.
The second one is NCP380 which is placed near the USB 3.0 ports for controlling the load current of the USB devices.
Refer the full schematics of XU4. https://dn.odroid.com/5422/ODROID-XU4/S ... OT1606.pdf

We will try to find a way to lower the resistance such as sharing the USB 3.0 VBUS or bypassing the protection ICs.
For example, @phaseshifter's approach.
viewtopic.php?f=99&t=25813#p181268

Other than that, we found few "S.M.A.R.T." commands could cause the bus reset error when the HDD doesn't support the proper command due to its old ATA firmware version.
But this one is not directly related to this thread.
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby bronco » Mon Jul 17, 2017 8:55 pm

odroid wrote:I think the disk input voltage tolerance seems to be different from each model of HDDs even with the same brand or manufacturer.


Of course :)

But that's only part of the problem since we're still talking about Ohm's law being valid everywhere. So users with a 5V/6A PSU with power lines thinner than yours will experience these voltage drops earlier since voltage available at XU4's DC-IN jack is already lower under load.

BTW: Threads like these viewtopic.php?f=146&t=26121#p184891 are also perfect candidates to check on layer 0 (hardware, voltage, DMM) first. If the whole setup suffers from huge voltage drops and there's a disk that tolerates low voltages might it be possible that in full load situations RTL8153 disappears since being also affected by under-voltage?
bronco
 
Posts: 27
Joined: Tue Jul 11, 2017 2:58 pm
languages_spoken: english

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby studioai » Tue Jul 18, 2017 3:01 pm

I ordered a voltage regulator circuit board and usb 3.0 male and female port.
I'll combine those components and attach to the usb port on XU4.
I'll report the result.
studioai
 
Posts: 11
Joined: Thu Apr 21, 2016 10:43 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Tue Jul 18, 2017 3:10 pm

bronco wrote:If the whole setup suffers from huge voltage drops and there's a disk that tolerates low voltages might it be possible that in full load situations RTL8153 disappears since being also affected by under-voltage?

I don't think so because all the power rails on RTL8153 uses only 3.3Volt.
After applying the RTL815X patch with Kernel 4.9.33 update, the random ETH disappearing issue was gone a few weeks ago.
https://git.kernel.org/pub/scm/linux/ke ... ?h=v4.9.33
Our latest kernel package has 4.9.37 and we will release a new package with 4.9.38 very soon.
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby studioai » Thu Aug 03, 2017 10:15 am

As I mentioned I'm assembling voltage regulator and usb sockets.

meanwhile I upgraded kernel to 4.9.37+ and the Issue never occurred until now.

Is there any change of UAS module in 4.9.37+?
studioai
 
Posts: 11
Joined: Thu Apr 21, 2016 10:43 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Thu Aug 03, 2017 11:43 am

@studioai,
No. There has been no change as far as I remember.
Do you use OMV? or our Ubuntu?
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby studioai » Thu Aug 03, 2017 7:53 pm

@odroid

I'm using dietpi with the kernel maintained by mevericks

BYW it occured last night again! Forget about my previous question please.
studioai
 
Posts: 11
Joined: Thu Apr 21, 2016 10:43 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Fri Aug 04, 2017 10:33 am

I see.
We will try to reproduce this issue from next week.
I think this u-ra's test method can be applied for testing the original CloudShell too.
viewtopic.php?f=146&t=26016&start=250#p198256
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby studioai » Sun Aug 06, 2017 4:36 pm

@odroid

I completed assembling Voltage regulator. I did torrent download test but failed.
I also did the test you suggested but I met a socket error and disconnected from XU4.
It looks like Voltage regulator is not solution of this issue.

Although I failed solve the issue but I figured out that there's no reset messages in log. But same behavior of OS.
(Freezes when try to access the device that occurred error)
studioai
 
Posts: 11
Joined: Thu Apr 21, 2016 10:43 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby bronco » Mon Aug 07, 2017 4:21 am

studioai wrote:Is there any change of UAS module in 4.9.37+?


How does the output of 'lsusb -t' and also 'dmesg' looks like when you run into problems?
bronco
 
Posts: 27
Joined: Tue Jul 11, 2017 2:58 pm
languages_spoken: english

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby studioai » Sat Aug 12, 2017 11:06 am

@bronco

Sorry. I bought a he'd enclosure with power supply. I quit to solve this issue.
studioai
 
Posts: 11
Joined: Thu Apr 21, 2016 10:43 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby rncwnd » Mon Aug 14, 2017 6:32 pm

I have the Cloudshell 1 with two 1GB WD Red. One is internally connected to the SATA poart. The other one is connected to one USB3 port. The enclosure is an ASMedia.
The disks are formatted with BTRFS Raid 1.
Code: Select all
root@odroid:~# lsusb -s 4:4
Bus 004 Device 004: ID 174c:55aa ASMedia Technology Inc. ASM1051E SATA 6Gb/s bridge, ASM1053E SATA 6Gb/s bridge, ASM1153 SATA 3Gb/s bridge


Code: Select all
root@odroid:~# lsusb -t
/:  Bus 06.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Vendor Specific Class, Driver=r8152, 5000M
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/2p, 5000M
        |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
        |__ Port 2: Dev 4, If 0, Class=Mass Storage, Driver=uas, 5000M
/:  Bus 03.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/2p, 480M
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=exynos-ohci/3p, 12M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=exynos-ehci/3p, 480M


When doing a btrfs scrub, this happens:

Code: Select all
[  724.010580] sd 1:0:0:0: [sdb] tag#0 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD IN
[  724.010641] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 00 18 00 00 04 00 00
[  730.075620] sd 1:0:0:0: [sdb] tag#1 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD OUT
[  730.075659] sd 1:0:0:0: [sdb] tag#1 CDB: opcode=0x2a 2a 00 26 b6 57 58 00 00 28 00
[  730.075871] scsi host1: uas_eh_bus_reset_handler start
[  730.078221] sd 1:0:0:0: [sdb] tag#2 uas_zap_pending 0 uas-tag 3 inflight: CMD
[  730.078256] sd 1:0:0:0: [sdb] tag#2 CDB: opcode=0x2a 2a 00 26 b6 57 80 00 00 80 00
[  730.078292] sd 1:0:0:0: [sdb] tag#3 uas_zap_pending 0 uas-tag 4 inflight: CMD
[  730.078322] sd 1:0:0:0: [sdb] tag#3 CDB: opcode=0x2a 2a 00 26 b6 58 00 00 00 80 00
[  730.078355] sd 1:0:0:0: [sdb] tag#4 uas_zap_pending 0 uas-tag 5 inflight: CMD
[  730.078384] sd 1:0:0:0: [sdb] tag#4 CDB: opcode=0x28 28 00 00 00 1c 00 00 04 00 00
[  730.078415] sd 1:0:0:0: [sdb] tag#5 uas_zap_pending 0 uas-tag 6 inflight: CMD
[  730.078441] sd 1:0:0:0: [sdb] tag#5 CDB: opcode=0x2a 2a 00 26 b6 58 80 00 00 80 00
[  730.078471] sd 1:0:0:0: [sdb] tag#6 uas_zap_pending 0 uas-tag 7 inflight: CMD
[  730.078498] sd 1:0:0:0: [sdb] tag#6 CDB: opcode=0x28 28 00 00 00 20 00 00 04 00 00
[  730.161071] usb 4-1.2: reset SuperSpeed USB device number 4 using xhci-hcd
[  730.187194] scsi host1: uas_eh_bus_reset_handler success


Kernel is
Code: Select all
root@odroid:~# uname -a
Linux odroid 4.9.38+ #1 SMP PREEMPT Wed Aug 9 17:56:32 UTC 2017 armv7l GNU/Linux
rncwnd
 
Posts: 22
Joined: Tue Apr 11, 2017 11:18 pm
languages_spoken: english, german
ODROIDs: XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby crashoverride » Mon Aug 14, 2017 7:24 pm

You have to disable UAS for each device (VID/PID) explicitly. The output shows UAS is only disabled on a single device:
rncwnd wrote: |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 2: Dev 4, If 0, Class=Mass Storage, Driver=uas, 5000M
crashoverride
 
Posts: 3058
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby tchiwam » Mon Aug 14, 2017 10:43 pm

Something else I noticed when I stress test CPU and HDD, the normal 3A power brick is not enough, my problems went away with the 5V-4A power brick.
tchiwam
 
Posts: 81
Joined: Wed Dec 30, 2015 4:53 am
Location: Greenland
languages_spoken: english, French, Finnish, Greenlandic(little)
ODROIDs: 5x XU4, 3x C2, 2 cloudshell, 1 VU7, 1 VU7+, UPS1, UPS2, UPS3, USB/SATA, and many other fine bits from Odroid

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby rncwnd » Tue Aug 15, 2017 5:49 pm

crashoverride wrote:You have to disable UAS for each device (VID/PID) explicitly. The output shows UAS is only disabled on a single device:
rncwnd wrote: |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 2: Dev 4, If 0, Class=Mass Storage, Driver=uas, 5000M


tchiwam wrote:Something else I noticed when I stress test CPU and HDD, the normal 3A power brick is not enough, my problems went away with the 5V-4A power brick.



Thanks for your suggestions.
When I first encountered the USB problems I cleaned the ports as described somewhere else in this forum. Then I changed the cable, it's now much shorter and a bit thicker. And I am using the power brick that came with the cloudshell. It has at least 4A.

I disabled uas now completely:
Code: Select all
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/2p, 5000M
        |__ Port 1: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
        |__ Port 2: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M


Then I started btrfs scrub, and guess what:
Code: Select all
[  801.850551] usb 4-1.2: reset SuperSpeed USB device number 4 using xhci-hcd
[  811.337713] xhci-hcd xhci-hcd.2.auto: WARN Event TRB for slot 4 ep 2 with no TDs queued?
[  811.338487] sd 1:0:0:0: [sdb] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[  811.338494] sd 1:0:0:0: [sdb] tag#0 Sense Key : 0x3 [current]
[  811.338501] sd 1:0:0:0: [sdb] tag#0 ASC=0x11 ASCQ=0x0
[  811.338508] sd 1:0:0:0: [sdb] tag#0 CDB: opcode=0x28 28 00 00 9c 07 08 00 08 00 00
[  811.338515] blk_update_request: critical medium error, dev sdb, sector 10225416
[ 1171.376001] sd 1:0:0:0: timing out command, waited 360s


I am running the recent kernel:
Code: Select all
root@odroid:~# uname -a
Linux odroid 4.9.38+ #1 SMP PREEMPT Wed Aug 9 17:56:32 UTC 2017 armv7l GNU/Linux


I'm a bit lost and disappointed right now...
rncwnd
 
Posts: 22
Joined: Tue Apr 11, 2017 11:18 pm
languages_spoken: english, german
ODROIDs: XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby tkaiser » Tue Aug 15, 2017 10:41 pm

rncwnd wrote:I am using the power brick that came with the cloudshell. It has at least 4A.


Well, the next time you should use a PSU with 100A. And disable UAS twice to ensure that it's really disabled! And then try a PSU with 5000A. It won't help since you should keep in mind that your problem is neither software related (the funny 'UAS is evil' campaign still running here) nor amperage related but most probably you either suffer from the usual voltage drop and/or cable/connector issues.

Check 7) here https://forum.armbian.com/index.php?/to ... ment=32340 (what user Kosmatik had to report after being told to fix cable/connector issues by disabling UAS).

BTW: ODROID HC1 is the direct result of fighting both hardware issues XU4 users are constantly running into (under-voltage and cable/contact SuperSpeed troubles)
tkaiser
 
Posts: 93
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby odroid » Wed Aug 16, 2017 12:39 am

@rncwnd

We could reproduce the random XHCI reset issue with the original CloudShell by applying very heavy traffics on USB-to-SATA and USB-to-Ethernet simultaneously.
Refer this link to see how we tested.
viewtopic.php?f=146&t=26016&start=250#p198451

So the USB host device driver still has a stability issue on a non-UAS SATA bridge like GL3321G while the UAS capable JMS578 has no issue.
We will try to find which part of the related drivers cause the issue on the original CloudShell.
User avatar
odroid
Site Admin
 
Posts: 24305
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby Mangasora » Wed Aug 16, 2017 7:31 pm

Hello.I has been in the forum for some time, but I have not written - my English is too bad.
While I was staring at the old kernel, I had no problems with my WD My Passport Essential 1TB hard drive.
With the Android 7 disc also works well.I waited a while to solve the problem, but when it did not happen,I bought a USB3 hub with external power supply.I also bought a USB-SATA adapter with a separate power supply.When the disc is connected directly to XU4, the connection collapses continuously, especially when I watch something from it through Kodi.When the disc is connected through the hub no problem.
I figured I had power problems,but after I connected another drive through a usb to sata adapter with a separate power supply,the situation is repeated.Through usb3 hub works, directly to XU4 not.
Hopefully this helps you find the problem.
Mangasora
 
Posts: 3
Joined: Sun Oct 09, 2016 9:00 pm
languages_spoken: bulgarian;rusian;greek
ODROIDs: maybe XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby rncwnd » Fri Aug 18, 2017 5:50 pm

@odroid, @tkaiser, @crashoverride

Disabling uas on the external USB HDD made it worse!
Had to hard reset the XU4 by unplugging power because it suddenly stopped doing anything except rising the temperature and my blood pressure. :roll:

Btw: I love BTRFS...

edit: at the time the errors happen there was no such thing as "heavy traffic" on the network interface.
rncwnd
 
Posts: 22
Joined: Tue Apr 11, 2017 11:18 pm
languages_spoken: english, german
ODROIDs: XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby tkaiser » Fri Aug 18, 2017 8:04 pm

rncwnd wrote:Disabling uas on the external USB HDD made it worse!


Good to know. BTW I was only making some fun about the 'UAS is evil' campain still running here. How should exchanging a software driver fix hardware issues like cabling/contact issues or under-voltage (which is totally unrelated to provided current, it's all about resistance here and there letting voltage drop below acceptable levels and then you run into USB reset issues as well)
tkaiser
 
Posts: 93
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby rncwnd » Fri Aug 18, 2017 8:14 pm

tkaiser wrote:
rncwnd wrote:Disabling uas on the external USB HDD made it worse!


Good to know. BTW I was only making some fun about the 'UAS is evil' campain still running here. How should exchanging a software driver fix hardware issues like cabling/contact issues or under-voltage (which is totally unrelated to provided current, it's all about resistance here and there letting voltage drop below acceptable levels and then you run into USB reset issues as well)

I know. But sometimes it's the sum of different factors and optimizing each one helps stabilizing the whole thing.
rncwnd
 
Posts: 22
Joined: Tue Apr 11, 2017 11:18 pm
languages_spoken: english, german
ODROIDs: XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby joaofl » Mon Aug 28, 2017 5:45 pm

I have the exact same problem. Have run loads of testing with different cables, USB-SATA bridges, power supplies. This issue happens in all situations.

I was posting and discussing this issue in a parallel thread https://forum.odroid.com/viewtopic.php?f=146&t=26016&start=250#p198256

Found no solution so far.

One thing is for sure (tested it again yesterday to make sure): The issue only happens when using the 4.9 kernel line.. With the 3.10 it works slower, but smooth.

It happens when USB3 (the two ports if possible) and Ethernet (which is also USB3), are put in to heavy load.. Then I keep on getting these resets.

It dont think it is related to the cloudshell, since I dont have it, and I've seen this issue with many other controllers as well (at least 4 different ones, 2 from ASMEDIA). My external HD is a WD MyBook 8TB

I'm really interested on solving this issue, and could help with some testing with the right guidance.

Cheers.
joaofl
 
Posts: 28
Joined: Sat Feb 27, 2016 2:29 am
Location: Portugal, Brasil
languages_spoken: english, portuguese, spanish
ODROIDs: XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby moon.linux » Wed Sep 06, 2017 3:51 pm

@joaofl
Can you people try out some simple patch and provide me the feedback.
Attachments
usbphy.patch.txt
usb phy + clk
(2.33 KiB) Downloaded 18 times
moon.linux
 
Posts: 849
Joined: Thu Oct 02, 2014 11:42 pm
languages_spoken: english

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby tkaiser » Wed Sep 06, 2017 6:48 pm

moon.linux wrote:Can you people try out some simple patch and provide me the feedback.


To ease this for users of Armbian next or OMV -- see https://sourceforge.net/projects/openme ... %20images/ -- it's as easy as
Code: Select all
wget http://kaiser-edv.de/tmp/NumpU8/odroidxu4-kernel-4.9.47-moon.linux.usbphy.tgz
tar xf odroidxu4-kernel-4.9.47-moon.linux.usbphy.tgz
dpkg -i *.deb
reboot


Patch applied flawlessly and changes are present (quick look with dtc tool). But I fear I can't test since all issues I had with XU4 and USB3 in the past were due to connector or underpowering problems.
tkaiser
 
Posts: 93
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby tkaiser » Wed Sep 06, 2017 7:42 pm

tkaiser wrote:I fear I can't test


Well, I did test with your patch applied. And am somewhat impressed now. My usual EVO840 in my usual UAS capable disk enclosures using my usual iozone benchmark:
Code: Select all
JMS567 (XU4)                                         random    random
     kB  reclen    write  rewrite    read    reread    read     write
 102400       4    17669    20165    20257    20383    16982    20178
 102400      16    54516    60047    70987    72652    59637    56508
 102400     512   250916   255465   193677   197900   194514   256698
 102400    1024   284266   290289   249001   263474   256741   294794
 102400   16384   271132   344344   331696   344080   343171   345746
1024000   16384   352032   364478   343505   345327   344443   365380

JMS578 (XU4)                                         random    random
     kB  reclen    write  rewrite    read    reread    read     write
 102400       4    17689    20321    20243    20266    18551    20507
 102400      16    68430    76461    71857    73693    62686    75654
 102400     512   254971   260691   197113   199591   196360   253364
 102400    1024   278828   284385   253822   269164   264648   303429
 102400   16384   276623   353381   334671   346203   346380   355198
1024000   16384   358391   375032   353011   355223   354554   375321

ASM1153 (XU4)                                        random    random
     kB  reclen    write  rewrite    read    reread    read     write
 102400       4    17727    20283    20287    20285    17872    20444
 102400      16    68623    75456    72367    73232    62588    77710
 102400     512   255580   258394   196286   200486   196993   262408
 102400    1024   281646   291620   246851   265401   261619   305942
 102400   16384   278442   355873   341660   354970   354001   362453
1024000   16384   370482   385175   354552   356081   354719   384753


These numbers are way better than anything we've seen before attached to XU4, see for example http://xu4.keltike.de/performance/odroi ... s-support/

Ok, let's take HC1 and re-test without patch first (4.9.46) then with patch applied (4.9.47):
Code: Select all
JMS578 (HC1) 4.9.46 without patch                    random    random
     kB  reclen    write  rewrite    read    reread    read     write
 102400       4    18383    20801    20474    20487    19393    20421
 102400      16    68070    76047    72498    73197    63177    77768
 102400     512   255112   255304   201297   205841   202232   256493
 102400    1024   289082   296909   266883   281516   275388   310520
 102400   16384   284807   372181   359489   382336   381975   372153
1024000   16384   377370   394916   382322   383737   383250   392821

JMS578 (HC1) 4.9.47 with patch                       random    random
     kB  reclen    write  rewrite    read    reread    read     write
 102400       4    17677    20536    20509    20673    19395    20666
 102400      16    66447    76073    71478    74159    64066    76610
 102400     512   226057   258087   203215   207852   204298   258405
 102400    1024   294799   307044   267094   285855   275917   311129
 102400   16384   284902   371441   367828   382426   382487   368945
1024000   16384   377829   392470   382574   383683   383237   394680


Ok, no performance differences but for whatever reasons we're now with Exynos5422 also hitting the magical USB3 400 MB/s barrier :)
tkaiser
 
Posts: 93
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby moon.linux » Wed Sep 06, 2017 7:49 pm

Thanks you but still some internal issue are their in USB 3.0, but this should stabilize little bit now. :D
moon.linux
 
Posts: 849
Joined: Thu Oct 02, 2014 11:42 pm
languages_spoken: english

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby u-ra » Thu Sep 07, 2017 3:09 am

moon.linux wrote:@joaofl
Can you people try out some simple patch and provide me the feedback.


Your patch made no difference for me, but moving to kernel 4.13 did. 4.13 can sustain ~145MiB/s disk transfer over USB3 with 80-100Mbit/s download for as long as I run it.
u-ra
 
Posts: 12
Joined: Tue May 17, 2016 9:06 pm
languages_spoken: english
ODROIDs: C2, XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby joaofl » Thu Sep 07, 2017 3:19 am

moon.linux wrote:@joaofl
Can you people try out some simple patch and provide me the feedback.


Ill give it a try asap and report back here.
joaofl
 
Posts: 28
Joined: Sat Feb 27, 2016 2:29 am
Location: Portugal, Brasil
languages_spoken: english, portuguese, spanish
ODROIDs: XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby moon.linux » Thu Sep 07, 2017 11:27 am

This patch just fix the correct clk needed for usb 3.0...we will like these small fix so end up in mainline.
And look it to stabilize the board.
moon.linux
 
Posts: 849
Joined: Thu Oct 02, 2014 11:42 pm
languages_spoken: english

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Sat Sep 09, 2017 12:37 am

u-ra wrote:
moon.linux wrote:@joaofl
Can you people try out some simple patch and provide me the feedback.


Your patch made no difference for me, but moving to kernel 4.13 did. 4.13 can sustain ~145MiB/s disk transfer over USB3 with 80-100Mbit/s download for as long as I run it.


how did you move to kernel 4.13?
is it on ubuntu?
it is available on archlinuxarm?
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby u-ra » Sat Sep 09, 2017 5:17 am

reza wrote:it is available on archlinuxarm?

Yes, here's the package.
u-ra
 
Posts: 12
Joined: Tue May 17, 2016 9:06 pm
languages_spoken: english
ODROIDs: C2, XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Sat Sep 09, 2017 4:35 pm

I think I had problems with this kernel before and had to go to kernel linux-xu3 before.
I dont remember what it was.
EDIT:
viewtopic.php?f=97&t=27546&p=195414#p195414
this was my issue.
is it fixed for you?
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Sat Sep 09, 2017 4:56 pm

oh I see.
sorry to tell you this but ,your solution is not a solution at all.
you get 80-100 Mbits/s but our issue is with 1000Mbit(~100 MBytes/sec).
and I testes again with linux-arm7.
your ethernet card becomes usb2 and max speed of ETHERNET100 instead of gigabit/s.
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby u-ra » Mon Sep 11, 2017 3:11 am

reza wrote:oh I see.
sorry to tell you this but ,your solution is not a solution at all.
you get 80-100 Mbits/s but our issue is with 1000Mbit(~100 MBytes/sec).
and I testes again with linux-arm7.
your ethernet card becomes usb2 and max speed of ETHERNET100 instead of gigabit/s.


Huh, you're right.
Code: Select all
/:  Bus 05.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 480M
    |__ Port 1: Dev 3, If 0, Class=Vendor Specific Class, Driver=r8152, 480M

My disk usage is mostly local to the XU4, apart from occasional streaming over the network, so that's a compromise I'm willing to accept.
u-ra
 
Posts: 12
Joined: Tue May 17, 2016 9:06 pm
languages_spoken: english
ODROIDs: C2, XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Mon Sep 11, 2017 3:59 am

the issue on this thread usually occurs with high lan usage alongside high disk usage. for example copying a file to odriod and simultaneously copying a file from odroid both via lan.
so limiting the lan speed to 100Mb/s from 1000Mb/s is not a solution. just a workaround to limit the occurrence of the issue. even with that speed the issue occurred for me (but I haven't tested with kernel 4.13) .
if for you the issue if fixed with the kernel that is not for odroid(linux-arm is not for odroid but for some other arm boards) and you have no problem with the speed, more power to you.
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby u-ra » Mon Sep 11, 2017 4:35 am

reza wrote:the issue on this thread usually occurs with high lan usage alongside high disk usage.


Actually, I could reliably hit the issue even with low network traffic (20-30 Mbit/s) with kernel 4.9.

reza wrote:linux-arm is not for odroid but for some other arm boards


That's not true. linux-armv7 package is the upstream 4.13 kernel which supports most ARM boards in a single kernel binary, including XU4.
u-ra
 
Posts: 12
Joined: Tue May 17, 2016 9:06 pm
languages_spoken: english
ODROIDs: C2, XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Mon Sep 11, 2017 5:43 am

u-ra wrote:That's not true. linux-armv7 package is the upstream 4.13 kernel which supports most ARM boards in a single kernel binary, including XU4.


see the link you yourself provided.
https://archlinuxarm.org/packages/armv7h/linux-armv7

and look at the split packages
Split Packages: linux-armv7-ax3, linux-armv7-chromebook, linux-armv7-cubox, linux-armv7-d3plug, linux-armv7-headers, linux-armv7-mirabox, linux-armv7-smileplug

none of them are for xu4.

and if linux-arm is for xu4 then the maintainer messed up because as you saw the usbs are miss-configured on it but not on linux-odroid-xu3. and maybe some other minor issues that are present in that kernel but not seen.
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby u-ra » Mon Sep 11, 2017 7:03 am

The package maintainer didn't mess up anything. The upstream kernel supports a whole bunch of ARM boards in a single generic image, including XU4.
If you look at the PKGBUILD, you can see the split packages differ only in how the final image is constructed. They're all built from the same source and config.

R8152 being limited to High Speed was reported here last year: https://www.spinics.net/lists/linux-usb/msg145831.html
Looks like the problem was never fixed upstream.
u-ra
 
Posts: 12
Joined: Tue May 17, 2016 9:06 pm
languages_spoken: english
ODROIDs: C2, XU4

Re: Unstable HDD(USB3.0) on Cloudshell w/ kernel 4.9

Unread postby reza » Mon Sep 11, 2017 5:04 pm

I am not a developer or anything. what I meant was that I used to have linux-arm installed and I had issues with ethernet speed (as you described) and also graphic driver incompatibility.
If it works ok for you , then you are right ,it is for xu4.
it just didnt work for me on xu4 as I wanted.
btw you mean that the ethernet port miss-configuration is a new bug introduced after kernel 4.9 ?
reza
 
Posts: 58
Joined: Tue Mar 15, 2016 3:40 am
languages_spoken: english
ODROIDs: xu4

PreviousNext

Return to Linux Kernel 4.9 Debugging Party

Who is online

Users browsing this forum: No registered users and 7 guests