Syncing two UAS devices causes Ubuntu to hang

[tobetter]

My workmate, Joshua, had tried a lot and long test and he made a change on the Linux kernel based on his parametric test to change a value in a driver. So I am uploading the Image and kernel modules, so I hope some of you may try it out if it gives reliable performance. Before starting updating with uploaded images, please do back up the original files, specifically Image.gz and meson64_odroidn2.dtb for sure.

Here are the links which you need to download:
Image.gz - https://drive.google.com/open?id=1w3StN ... L4CghOTvRA
meson64_odroidn2.dtb - https://drive.google.com/open?id=1Ek3f2 ... MEse91m6GB
Linux kernel modules - https://drive.google.com/open?id=1NbUph ... -vMS2LBdIF

Please do copy the first two files Image.gz and meson64_odroidn2.dtb into "probably" /media/boot where the original kernel files are stored and please do tar xzvf linux-4.9.182-lib_modules.tar.gz -C /. After copying every files, what you have to do is adding one parameter usb-xhci.tablesize=8 to boot.init, it would be looked like this.

setenv bootargs "root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait rw ${condev} ${amlogic} no_console_suspend fsck.repair=yes net.ifnames=0 elevator=noop hdmimode=${hdmimode} cvbsmode=576cvbs max_freq_a53=${max_freq_a53} max_freq_a73=${max_freq_a73} maxcpus=${maxcpus} voutmode=${voutmode} ${cmode} disablehpd=${disablehpd} cvbscable=${cvbscable} overscan=${overscan} ${hid_quirks} usb-xhci.tablesize=8

Once everything is done, please do reboot then the kernel will use the new value added to the boot argument. If you are able to boot successfully with the kernel, please do set 32 to the max sector size as below before testing the storages. This was already been introduced earlier in this thread as a workaround.

Code: Select all

# echo 32 > /sys/class/block/sda/queue/max_sectors_kb
# echo 32 > /sys/class/block/sdb/queue/max_sectors_kb

What are we doing here?
So basically the parametric test I've mentioned are about two factors, one for the buffer size in USB driver and another is in the block device driver, we've performed multiple combos and eventually, 8 for USB driver and 32 for the block device drive gives better stability but the performance decreased but had run for a couple of days without any errors. Still, the test devices are running with the same factors and other combos as well, at the same time Amlogic is also looking into the same problem and trying a different approach.

Sorry for long instruction instead of giving you the kernel update, since this is a temporary approach it would be grateful if someone can try out this change and share the result if it works with another device as well. Also, you could try out other factors like from 8 to 16 for usb-xhci.tablesize or 32 to 64+ for max_sectors_kb. Whenever you change the value of usb-xhci.tablesize you must reboot your N2.

Please not that this change is not for a user who is using an single USB storage attached to ODROID-N2.

Thank you.

[gahabana]

@fonix232,
In my experience it does not matter which USB to SATA or NVME bridges/adapters you are using. I have not been using UAS and it does not matter - even with USB 3.0 (older) HDDs nto to mention newer USB 3.1/USB 3.2/USB-C interface - with 2 drives system will error sooner or later. Sometimes even with only one usb HDD. Something is wrong either with HW or with SW support.
Everyone is hopeful that Odroid wiht help from chipset manufacturors will deliver a sw fix (either kernel patch or some 'tweak') and until that does happen USB is at best unreliable (for HDD at least) on Odroid-N2.

Maybe it will change and maybe speeds will not go from 300+Mb/second down to USB 2.0 range but we will need to see.

[gregeeh]

@fonix232,
Sometimes even with only one usb HDD. Something is wrong either with HW or with SW support.

And it is not just HDD's that have an issue. My PS3 PlayTV Dual Tuner plugged into the USB 3 port will give errors after just a few minutes of watching FTA TV that only has a transfer rate of about 3500 kb/sec. There's no problem if the PS3 PlayTV Dual Tuner is plugged into the micro USB port it is just the USB 3 ports that have issues.

This has been an on going problem for several months now and it's time something was done to fix it once and for all so we can use what we paid for.

[Andrew Sayers]

My workmate, Joshua, had tried a lot and long test and he made a change on the Linux kernel based on his parametric test to change a value in a driver. So I am uploading the Image and kernel modules, so I hope some of you may try it out if it gives reliable performance.

I'm now running this on my N2, and will let you know the results tomorrow. My system has been fairly stable with just the previous 32 fix (admittedly with a fairly low load), but at least this should tell you whether it makes anything worse

I do have a few questions though...

• do you want to know anything more than whether it works and what the transfer speed looks like?
• is it worth including your instructions in the top post, or do you plan to push this out through the normal channels once it's been tested?
  • if you're planning to push it out, is there anything I can do to help get this unrelated issue with bootini fixed at the same time?
• this sounds like a temporary workaround - what are the chances the performance of this workaround is similar to the performance of the permanent solution?
• do you guys want to start a new (sticky) thread about this whole issue? I'm happy to keep maintaining the top post in this thread either way, but recent posts have shown the title is confusing and people want to see a more official response

[tobetter]

My workmate, Joshua, had tried a lot and long test and he made a change on the Linux kernel based on his parametric test to change a value in a driver. So I am uploading the Image and kernel modules, so I hope some of you may try it out if it gives reliable performance.

I'm now running this on my N2, and will let you know the results tomorrow. My system has been fairly stable with just the previous 32 fix (admittedly with a fairly low load), but at least this should tell you whether it makes anything worse

I do have a few questions though...

Trying to figure out the combos with less IO traffic drops but reliable stability is the first.

[*]do you want to know anything more than whether it works and what the transfer speed looks like?

It would be great if you can share how your set up works at which performance.

[*]is it worth including your instructions in the top post, or do you plan to push this out through the normal channels once it's been tested?

If the way works and if it helps a bit having more stable performance, I will ask to add it for the next kernel update. So making less work for setting up.

[*]if you're planning to push it out, is there anything I can do to help get this unrelated issue with bootini fixed at the same time?

Not sure about the issue, honestly I haven't looked into it so far and I am not able to manage it. But let me look into the thread and try if there is what I can help.

[*]this sounds like a temporary workaround - what are the chances the performance of this workaround is similar to the performance of the permanent solution?

It's a bit hard to say at the moment, we can offer you the best combo which we tested that gives more stability of USB I/O at the same time we have a binary update from Amlogic which could a bit improve the stability which is being tested internally. We can commit later after having some test, sorry about the slow progress.

[*]do you guys want to start a new (sticky) thread about this whole issue? I'm happy to keep maintaining the top post in this thread either way, but recent posts have shown the title is confusing and people want to see a more official response
[/list]

Not sure which is better. I understand since there are a few threads with a similar issue, so maybe it could be better to invite the users to one thread for USB stability issue. IMHO, at the moment, thinking to keep tracking on this post and we could manage the USB problem once the patch is released if a similar issue can happen from other USB devices rather than storage.

[Andrew Sayers]

I've now done a day's testing, and the results are... a bit odd. I wasn't able to capture as much data as I would have liked, but I'll keep trying and get back to you.

I've now run the test described in the top post twice with the new kernel, using tablesize=8 and max_sectors_kb=32. In both cases, /dev/sdb went away after approximately 9500 seconds (+-100). But other USB devices weren't affected and sdb came back after a reboot instead of needing a full power cycle. I haven't tried unplugging it then plugging it back in, but I wouldn't be surprised if that works too. Obviously this is a serious issue, but the behaviour is completely different to the normal bug. I can't rule out the possibility this is an issue with the drive, although frankly it would be quite a coincidence.

I haven't run a test that long on the old kernel before, so I'm now doing that. Given that each test takes nearly three hours, it might be a while before I can give you much detail.

[elatllat]

Anyone try this on the 5.2 kernel?
Is this an issue for non UAS devices? ( I have to add my devices to quirks to get smartctl to work anyway )

[tobetter]

Another update on U-boot, please download the recent update U-boot blob from the link and follow the instruction.
https://github.com/hardkernel/u-boot/re ... droidn2-51

Code: Select all

$ tar xzvf u-boot-odroidn2-51.tar.gz
$ cd sd_fuse
$ sudo ./sd_fusing.sh </dev/path/of/your/memory/card>

In the lab test, this binary update gives a bit better stability but not performance improvement, what we also observed is that an error rate could vary with the type of storage, SSD or USB stick. In case 2+ disks are going to run, I would recommend using 2 for the kernel parameter sg_tablesize which would give a slower performance.

I've asked to prepare the update package and expect to be done soon.

[tobetter]

Linux kernel and U-boot are updated with patches, please do the commands for HK's Ubuntu users.

Code: Select all

$ sudo apt update
$ sudo apt upgrade

Once you have done updating, please add one kernel argument usb-xhci.tablesize=2 to /media/boot/boot.ini. So it would be look like this.

Code: Select all

# Boot Args
setenv bootargs "root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait rw ${condev} ${amlogic} no_console_suspend fsck.repair=yes net.ifnames=0 levator=noop hdmimode=${hdmimode} cvbsmode=576cvbs max_freq_a53=${max_freq_a53} max_freq_a73=${max_freq_a73} maxcpus=${maxcpus} outmode=${voutmode} ${cmode} disablehpd=${disablehpd} cvbscable=${cvbscable} overscan=${overscan} ${hid_quirks} usb-xhci.tablesize=2"

[Andrew Sayers]

Linux kernel and U-boot are updated with patches, please do the commands for HK's Ubuntu users.

Thanks for the updates, I'll try them tomorrow and report back. Today's tests showed that the bug I mentioned in my previous post was also present with the old kernel, but happened about twice as quickly. In other words, the July 1st kernel and tablesize are an improvement but still don't produce a fully stable system. Hopefully the new u-boot will improve things further.

Here's the kernel log from the moment the hard disk fails (this is with the old kernel, but the error from the July 1st kernel is basically the same):

Code: Select all

Jul  3 21:46:37 andrews-house kernel: [16770.701997@0] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1
Jul  3 21:46:37 andrews-house kernel: [16770.707044@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf000 trb-start 00000000cf7d4fc0 trb-end 0000000000000000 seg-start 00000000cf7d4000 seg-end 00000000cf7d4ff0
Jul  3 21:46:37 andrews-house kernel: [16770.707048@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf000 trb-start 00000000cf7d3000 trb-end 00000000cf7d3000 seg-start 00000000cf7d3000 seg-end 00000000cf7d3ff0
Jul  3 21:46:37 andrews-house kernel: [16770.707053@0] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 13
Jul  3 21:46:37 andrews-house kernel: [16770.717784@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf020 trb-start 00000000cf7d4fc0 trb-end 0000000000000000 seg-start 00000000cf7d4000 seg-end 00000000cf7d4ff0
Jul  3 21:46:37 andrews-house kernel: [16770.717788@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf020 trb-start 00000000cf7d3000 trb-end 00000000cf7d3000 seg-start 00000000cf7d3000 seg-end 00000000cf7d3ff0
Jul  3 21:46:37 andrews-house kernel: [16770.717792@0] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 1
Jul  3 21:46:37 andrews-house kernel: [16770.728444@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf060 trb-start 00000000cf7d4fc0 trb-end 0000000000000000 seg-start 00000000cf7d4000 seg-end 00000000cf7d4ff0
Jul  3 21:46:37 andrews-house kernel: [16770.728448@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf060 trb-start 00000000cf7d3000 trb-end 00000000cf7d3000 seg-start 00000000cf7d3000 seg-end 00000000cf7d3ff0
Jul  3 21:46:37 andrews-house kernel: [16770.728484@0] xhci-hcd xhci-hcd.0.auto: ERROR Transfer event TRB DMA ptr not part of current TD ep_index 2 comp_code 6
Jul  3 21:46:37 andrews-house kernel: [16770.739104@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf070 trb-start 00000000cf7d4fc0 trb-end 0000000000000000 seg-start 00000000cf7d4000 seg-end 00000000cf7d4ff0
Jul  3 21:46:37 andrews-house kernel: [16770.739108@0] xhci-hcd xhci-hcd.0.auto: Looking for event-dma 00000000cf7bf070 trb-start 00000000cf7d3000 trb-end 00000000cf7d3000 seg-start 00000000cf7d3000 seg-end 00000000cf7d3ff0
Jul  3 21:47:07 andrews-house kernel: [16800.967917@0] usb 2-1.4: reset SuperSpeed USB device number 5 using xhci-hcd
Jul  3 21:47:38 andrews-house kernel: [16831.932018@1] usb 2-1.4: reset SuperSpeed USB device number 5 using xhci-hcd
Jul  3 21:47:48 andrews-house kernel: [16842.076226@1] usb 2-1.4: reset SuperSpeed USB device number 5 using xhci-hcd
Jul  3 21:48:04 andrews-house kernel: [16858.459982@1] usb 2-1.4: reset SuperSpeed USB device number 5 using xhci-hcd
Jul  3 21:48:04 andrews-house kernel: [16858.559862@0] usb 2-1.4: reset SuperSpeed USB device number 5 using xhci-hcd
Jul  3 21:48:15 andrews-house kernel: [16868.699941@1] usb 2-1.4: reset SuperSpeed USB device number 5 using xhci-hcd
Jul  3 21:48:15 andrews-house kernel: [16868.720356@0] sd 1:0:0:0: Device offlined - not ready after error recovery
Jul  3 21:48:15 andrews-house kernel: [16868.720395@0] sd 1:0:0:0: [sdb] tag#0 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
Jul  3 21:48:15 andrews-house kernel: [16868.720410@0] sd 1:0:0:0: [sdb] tag#0 CDB: Read(16) 88 00 00 00 00 00 25 07 23 20 00 00 00 40 00 00
Jul  3 21:48:15 andrews-house kernel: [16868.720417@0] blk_update_request: I/O error, dev sdb, sector 621224736
Jul  3 21:48:15 andrews-house kernel: [16868.721426@1] sd 1:0:0:0: rejecting I/O to offline device
Jul  3 21:48:15 andrews-house kernel: [16868.726921@1] blk_update_request: I/O error, dev sdb, sector 621224800
Jul  3 21:48:15 andrews-house kernel: [16868.733381@4] sd 1:0:0:0: rejecting I/O to offline device
Jul  3 21:48:15 andrews-house kernel: [16868.738660@4] blk_update_request: I/O error, dev sdb, sector 621224864
Jul  3 21:48:15 andrews-house kernel: [16868.747008@4] sd 1:0:0:0: rejecting I/O to offline device
Jul  3 21:48:15 andrews-house kernel: [16868.750458@4] blk_update_request: I/O error, dev sdb, sector 621224424
Jul  3 21:48:15 andrews-house kernel: [16868.757183@4] sd 1:0:0:0: rejecting I/O to offline device
Jul  3 21:48:15 andrews-house kernel: [16868.762421@4] blk_update_request: I/O error, dev sdb, sector 621224424
Jul  3 21:48:15 andrews-house kernel: [16868.768897@4] Buffer I/O error on dev sdb, logical block 77653053, async page read

I was able to restart the disk simply by pulling out its USB cable and plugging it back in, so whatever's happening here certainly isn't as severe as the original bug.

@tobetter - I'm planning to spend my time tweaking the tablesize and max_sectors_kb values in the latest kernel+u-boot, but I'm happy to look into other things if anything catches your eye in the log above. Also, if I understand correctly, the new recommended workaround is:

1. apt-get update && apt-get upgrade (ignoring the U-boot blob in the link above, which is no longer necessary)
2. manually edit /media/boot/boot.ini as described above
3. set max_sectors_kb as described in the top post

If that's all correct, I'll update the top post again.

Anyone try this on the 5.2 kernel?
Is this an issue for non UAS devices? ( I have to add my devices to quirks to get smartctl to work anyway )

This is is definitely an issue for non-UAS devices. People have reported lots of USB devices with high enough sustained transfer speeds triggering the bug, it's just that UAS disk drives are the most common example (also, I didn't really understand the problem when I chose the thread title ). I'll mention that in the next update to the top post.

[fonix232]

Linux kernel and U-boot are updated with patches, please do the commands for HK's Ubuntu users.

Code: Select all

$ sudo apt update
$ sudo apt upgrade

Once you have done updating, please add one kernel argument usb-xhci.tablesize=2 to /media/boot/boot.ini. So it would be look like this.

Code: Select all

# Boot Args
setenv bootargs "root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait rw ${condev} ${amlogic} no_console_suspend fsck.repair=yes net.ifnames=0 levator=noop hdmimode=${hdmimode} cvbsmode=576cvbs max_freq_a53=${max_freq_a53} max_freq_a73=${max_freq_a73} maxcpus=${maxcpus} outmode=${voutmode} ${cmode} disablehpd=${disablehpd} cvbscable=${cvbscable} overscan=${overscan} ${hid_quirks} usb-xhci.tablesize=2"

Unfortunately for me, the uboot+kernel updates just made it worse. Now even with max_sectors_kb set to 10, I'm getting failures almost immediately after reboot - in fact now roughly ~5 minutes after booting, my storage dies, making it useless as a home media server.

This made me seriously reconsider sending back the whole shebang to HardKernel, and replace it with a H2... Bit more pricy, but less of these quirks.

[tobetter]

Linux kernel and U-boot are updated with patches, please do the commands for HK's Ubuntu users.

Code: Select all

$ sudo apt update
$ sudo apt upgrade

Once you have done updating, please add one kernel argument usb-xhci.tablesize=2 to /media/boot/boot.ini. So it would be look like this.

Code: Select all

# Boot Args
setenv bootargs "root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait rw ${condev} ${amlogic} no_console_suspend fsck.repair=yes net.ifnames=0 levator=noop hdmimode=${hdmimode} cvbsmode=576cvbs max_freq_a53=${max_freq_a53} max_freq_a73=${max_freq_a73} maxcpus=${maxcpus} outmode=${voutmode} ${cmode} disablehpd=${disablehpd} cvbscable=${cvbscable} overscan=${overscan} ${hid_quirks} usb-xhci.tablesize=2"

Unfortunately for me, the uboot+kernel updates just made it worse. Now even with max_sectors_kb set to 10, I'm getting failures almost immediately after reboot - in fact now roughly ~5 minutes after booting, my storage dies, making it useless as a home media server.

This made me seriously reconsider sending back the whole shebang to HardKernel, and replace it with a H2... Bit more pricy, but less of these quirks.

I am sorry to hear it does work worse. Since I understand you have RAID setup, if remember correctly, would you be able to test once after removing usb-xhci.tablesize and not setting max_sectors_kb?

[fonix232]

I am sorry to hear it does work worse. Since I understand you have RAID setup, if remember correctly, would you be able to test once after removing usb-xhci.tablesize and not setting max_sectors_kb?

Yes, I'm running a RAID0 array on two USB3 HDDs (identical hard drives and USB adapters), and I also have an NVMe SSD with a single partition and some symlinks (my docker config folder, which I need if I reinstall the system, some private documents, and the rest is used for temp cache for torrents while they seed). The SSD seems to be doing fine, which is weird because it has probably the highest continuous throughput in reads and writes, and it does use UASP... Could it be that there's also some kind of overflow in relevance to transaction count? RAID0 needs a lot more syncing even on an idle system than a regular direct partition, thus resulting in more frequent, but considerably smaller messages, and in that case it would make sense why RAID arrays die sooner than direct mounts. Just a hunch.

I've tried removing the tablesize parameter and commented out the line responsible for setting max_sectors_kb, in the appropriate autostart file as well. It reset back to 512 on all drives, and after restart it started producing the errors immediately. After a power cycle, and removing the USB drives while running, and repeating the power cycle, I was able to have a working system for about 15 minutes, with only a Plex scan running that affects the disks in question.

This is also a change from previous behaviour, as before the update I never needed a power cycle to restore working conditions, but now I apparently do.

[tobetter]

I am sorry to hear it does work worse. Since I understand you have RAID setup, if remember correctly, would you be able to test once after removing usb-xhci.tablesize and not setting max_sectors_kb?

Yes, I'm running a RAID0 array on two USB3 HDDs (identical hard drives and USB adapters), and I also have an NVMe SSD with a single partition and some symlinks (my docker config folder, which I need if I reinstall the system, some private documents, and the rest is used for temp cache for torrents while they seed). The SSD seems to be doing fine, which is weird because it has probably the highest continuous throughput in reads and writes, and it does use UASP... Could it be that there's also some kind of overflow in relevance to transaction count? RAID0 needs a lot more syncing even on an idle system than a regular direct partition, thus resulting in more frequent, but considerably smaller messages, and in that case it would make sense why RAID arrays die sooner than direct mounts. Just a hunch.

I've tried removing the tablesize parameter and commented out the line responsible for setting max_sectors_kb, in the appropriate autostart file as well. It reset back to 512 on all drives, and after restart it started producing the errors immediately. After a power cycle, and removing the USB drives while running, and repeating the power cycle, I was able to have a working system for about 15 minutes, with only a Plex scan running that affects the disks in question.

This is also a change from previous behaviour, as before the update I never needed a power cycle to restore working conditions, but now I apparently do.

Well...I am trying to be on the track of your problem. I and my staff had tested with two SSD disks or SSD+NVMe which are faster storage device than HDD, so I assumed that they could make more troubles than slow storage such as USB stick. We've tried the different parameter of uxb-xhci.tablesize and max_sectors_kb and better stability are given when usb-xhci.tablesize is low. Having test with SSD and NVMe was fine even USB stick, RAID0 might be a different story. What I don't understand is that you have to power cycle which you didn't before the update, which only changes U-boot.

[nikodemus]

@fonix: Are you sure your power supply can catch up with the demands of your Nas? The Rockpi community had ongoing prolems with usb3 and TKaiser was investigating this by using a raid. (raid is perfect for stress test systems).
All errors he found were underpowering events caused by heavy load from the raid:

https://github.com/ayufan-rock64/linux-build/issues/112
"The 'good' news: utilizing more than one USB3 disk at the same time is the same sh*t show as everywhere else in SBC land: https://www.cnx-software.com/2019/06/24 ... ent-564128"

I am convinced you checked this before, but as you said, its a little surprising that your fastest ssd seems not to have issues.

[rondi7]

@tobetter After reading the links provided above by nikodemus, it appears a problem (and it could easily be THE problem) is a drop in the 5 volts at the USB3 connector during heavy writes. If this is the case--in your lab you could isolate the 5volts on the board to the USB3 connectors (and or the hub) and use an external power supply to power the USB3 connectors. Another suggestion--add a a fairly big ( your engineers can decide what is big )capacitor to the 5 volts close to the USB3 connector --maybe even to the 5v pin on the header.

Earlier in the thread you kept asking about the power adapter, as if you suspected a problem there. I wonder--has anyone tried a 16v 4-6 amp supply (iirc your input power spec is up to 20v). Maybe the standard HK 12v @2amp is not enough?

This from one of the links above:
What looked like ‘UAS problems’ or ‘broken USB3 controller’ was simply underpowering. In the first try both SSDs were powered by the RPi and as such the current limitation on the USB ports kicked in (slightly above 1.1A available for all ports together). Now with one SSD powered externally everything is fine.

[tobetter]

@tobetter After reading the links provided above by nikodemus, it appears a problem (and it could easily be THE problem) is a drop in the 5 volts at the USB3 connector during heavy writes. If this is the case--in your lab you could isolate the 5volts on the board to the USB3 connectors (and or the hub) and use an external power supply to power the USB3 connectors. Another suggestion--add a a fairly big ( your engineers can decide what is big )capacitor to the 5 volts close to the USB3 connector --maybe even to the 5v pin on the header.

Earlier in the thread you kept asking about the power adapter, as if you suspected a problem there. I wonder--has anyone tried a 16v 4-6 amp supply (iirc your input power spec is up to 20v). Maybe the standard HK 12v @2amp is not enough?

This from one of the links above:
What looked like ‘UAS problems’ or ‘broken USB3 controller’ was simply underpowering. In the first try both SSDs were powered by the RPi and as such the current limitation on the USB ports kicked in (slightly above 1.1A available for all ports together). Now with one SSD powered externally everything is fine.

Thanks, I was reading the same and thinking of the way to test. It would be worth to try run with other adaptors as well...I am not sure, maybe wire the power to HDD from somewhere else like PC power and just wire USB 3.0...since they are mentioning of power issue, maybe I would set up new test env with a new external power source for disk drives?

[elatllat]

I use a piece of paper and some tape to cover the 5v on the USB to isolate the SBC from a powered hub, so anyone who is able to reproduce this issue could just do that in about a minute and know if it's a power issue.

[Andrew Sayers]

Some more tests from me:

• with the latest kernel and tablesize=2, my system is stable but read speeds are extremely low
• with the latest kernel and no tablesize, my sdb fails after about 700 seconds of continuous reading from both disks at once
• with the kernel tobetter posted here on July 1st and no tablesize, my sdb fails after about 7,000 seconds of continuous reading

All these tests were done after apt-get update && apt-get upgrade, so the only variables are in Image.gz and meson64_odroidn2.dtb. I assume the first test is stable because of the low speeds, but the other two show the latest kernel (not u-boot etc.) crashes 10 times quicker than the one posted here.

[rondi7]

I use a piece of paper and some tape to cover the 5v on the USB to isolate the SBC from a powered hub, so anyone who is able to reproduce this issue could just do that in about a minute and know if it's a power issue.

I have not gone back to read your previous posts on this thread---What was the result of your testing--are you able to have a Raid with 2 USB3 sticks and have it work for a long time? or perhaps as Andrews 1st post describes, his test case. If both of those work with an external 5 volts then there may be a solution. BUT--is it not true--IF you connect your HDD, SSD or?? to an external powered USB3 hub--you are now going thru 2 hubs and this will decrease your transfer rate because you are only connected to 1 USB3 on the N2.

IF it is a quick sharp drop in the 5v because of the increase in current required, then a correctly sized capacitor from 5v ( or maybe even from the `12v supply) to ground may contain enough storage to keep 5v from sagging. If it is a limitation of the voltage regulator--which changes the 12v to 5v, and 3.3 and ??--- then maybe it is also effecting other circuits--ie the SoC. Again adding a capacitor might fix it and it could easily be done on the GPIO header.

@tobetter--IF this is a problem with the 5 volts and maybe the SoC power (3.3v ?)--your analog (dare I say that here ) engineering folks should have a close look at not only the rails, but also at the USB3 connector, SOC etc when you are making it fail. They would know what size cap to try, and where is best place to put it.

[fonix232]

Well...I am trying to be on the track of your problem. I and my staff had tested with two SSD disks or SSD+NVMe which are faster storage device than HDD, so I assumed that they could make more troubles than slow storage such as USB stick. We've tried the different parameter of uxb-xhci.tablesize and max_sectors_kb and better stability are given when usb-xhci.tablesize is low. Having test with SSD and NVMe was fine even USB stick, RAID0 might be a different story. What I don't understand is that you have to power cycle which you didn't before the update, which only changes U-boot.

I was guessing the same, especially since the issue seems to be throughput-related. Not sure what's so different in my case, I'll try to reinstall the whole board some other day, see if recreating the RAID0 array helps. The immediate issue on boot might be related to that, the software RAID trying to repair all the fallen-out-of-sync blocks, and overloads the bus immediately.

@fonix: Are you sure your power supply can catch up with the demands of your Nas? The Rockpi community had ongoing prolems with usb3 and TKaiser was investigating this by using a raid. (raid is perfect for stress test systems).
All errors he found were underpowering events caused by heavy load from the raid:

https://github.com/ayufan-rock64/linux-build/issues/112
"The 'good' news: utilizing more than one USB3 disk at the same time is the same sh*t show as everywhere else in SBC land: https://www.cnx-software.com/2019/06/24 ... ent-564128"

I am convinced you checked this before, but as you said, its a little surprising that your fastest ssd seems not to have issues.

I'm using a 12V@5A power adapter, not the one supplied with the board. So unless the N2 itself limits the max USB power use to some ridiculously low value. All in all I doubt my disks use more than 5V@1A each.

[tobetter]

Well...I am trying to be on the track of your problem. I and my staff had tested with two SSD disks or SSD+NVMe which are faster storage device than HDD, so I assumed that they could make more troubles than slow storage such as USB stick. We've tried the different parameter of uxb-xhci.tablesize and max_sectors_kb and better stability are given when usb-xhci.tablesize is low. Having test with SSD and NVMe was fine even USB stick, RAID0 might be a different story. What I don't understand is that you have to power cycle which you didn't before the update, which only changes U-boot.

I was guessing the same, especially since the issue seems to be throughput-related. Not sure what's so different in my case, I'll try to reinstall the whole board some other day, see if recreating the RAID0 array helps. The immediate issue on boot might be related to that, the software RAID trying to repair all the fallen-out-of-sync blocks, and overloads the bus immediately.

@fonix: Are you sure your power supply can catch up with the demands of your Nas? The Rockpi community had ongoing prolems with usb3 and TKaiser was investigating this by using a raid. (raid is perfect for stress test systems).
All errors he found were underpowering events caused by heavy load from the raid:

https://github.com/ayufan-rock64/linux-build/issues/112
"The 'good' news: utilizing more than one USB3 disk at the same time is the same sh*t show as everywhere else in SBC land: https://www.cnx-software.com/2019/06/24 ... ent-564128"

I am convinced you checked this before, but as you said, its a little surprising that your fastest ssd seems not to have issues.

I'm using a 12V@5A power adapter, not the one supplied with the board. So unless the N2 itself limits the max USB power use to some ridiculously low value. All in all I doubt my disks use more than 5V@1A each.

Can I have the model name of your disks for RAID?

[nikodemus]

Peak value can be even much higher, see above article by Tkaiser.
Some years ago i had a drive that made trouble no matter whic PC i used though everything was within specification. Only an active hub finally helped. An additional hub sometimes is not a good idea because it can make trouble too, but i had no other chance, because the drive had no connector for external supply.

[nick793]

Just want to echo that I'm having issues as well.

I'm running two RAID1 arrays off my N2 and I'm having reboots every 12 hours or so. The disks are connected as follows:

Array 1 (1TB RAID1):

• 2x 1TB Seagate Barracuda HDD (5400rpm)

• Startech.com 2-bay HDD enclosure (powered)

Array 2 (240GB RAID1):

• 1x 240GB M.2 SSD (Kingston)

• 1x 256GB M.2 SSD (Samsung PM851)

• 2x Startech.com M.2 NGFF to USB3 enclosure

The extra 16GB of space on the Samsung SSD is being used as part of a GlusterFS volume that's replicated across the network. Every few hours the bus resets and the GlusterFS daemon crashes.

How big of a performance hit is the workaround? I'm about to set up a new RAID5 array but if it's back to USB2 then I'm not going to bother for a while until a fix comes along.

[tobetter]

Just want to echo that I'm having issues as well.

I'm running two RAID1 arrays off my N2 and I'm having reboots every 12 hours or so. The disks are connected as follows:

Array 1 (1TB RAID1):

• 2x 1TB Seagate Barracuda HDD (5400rpm)

• Startech.com 2-bay HDD enclosure (powered)

Array 2 (240GB RAID1):

• 1x 240GB M.2 SSD (Kingston)

• 1x 256GB M.2 SSD (Samsung PM851)

• 2x Startech.com M.2 NGFF to USB3 enclosure

The extra 16GB of space on the Samsung SSD is being used as part of a GlusterFS volume that's replicated across the network. Every few hours the bus resets and the GlusterFS daemon crashes.

How big of a performance hit is the workaround? I'm about to set up a new RAID5 array but if it's back to USB2 then I'm not going to bother for a while until a fix comes along.

Does the error also happen when you connect the Array 1 only?

[nick793]

Just want to echo that I'm having issues as well.

I'm running two RAID1 arrays off my N2 and I'm having reboots every 12 hours or so. The disks are connected as follows:

Array 1 (1TB RAID1):

• 2x 1TB Seagate Barracuda HDD (5400rpm)

• Startech.com 2-bay HDD enclosure (powered)

Array 2 (240GB RAID1):

• 1x 240GB M.2 SSD (Kingston)

• 1x 256GB M.2 SSD (Samsung PM851)

• 2x Startech.com M.2 NGFF to USB3 enclosure

The extra 16GB of space on the Samsung SSD is being used as part of a GlusterFS volume that's replicated across the network. Every few hours the bus resets and the GlusterFS daemon crashes.

How big of a performance hit is the workaround? I'm about to set up a new RAID5 array but if it's back to USB2 then I'm not going to bother for a while until a fix comes along.

Does the error also happen when you connect the Array 1 only?

Not sure haven't tried it yet. The errors only became noticeable after I moved all my media to array 1 and started downloading more media to it. Last night after streaming with Plex for a few hours it rebooted. The most direct way I found to trigger the bug is to verify torrents in transmission. If I verify one at a time I have no problems but more than one at a time and my system reboots in minutes.

I can try and see if that impacts it, but right now I'm using array 1 for media storage, and array 2 for most of my docker persistent storage (config files, databases, etc) and Plex transcoding.

I should add that I'm not using any updates/workarounds beyond those that come from apt. If there's something I should try please let me know, but I'm not that skilled with kernel parameters/u-boot so I've avoided touching them until now

[fonix232]

Can I have the model name of your disks for RAID?

Sure. For the RAID0 array I'm using two HGST TravelStar 7 K1000 (HGST721010A9E630). The 250GB SSD is a Samsung PM961 (MZVLW256HEHP-000).

Peak value can be even much higher, see above article by Tkaiser.
Some years ago i had a drive that made trouble no matter whic PC i used though everything was within specification. Only an active hub finally helped. An additional hub sometimes is not a good idea because it can make trouble too, but i had no other chance, because the drive had no connector for external supply.

I've been using these disks for two years now, in the current enclosures. No issues so far, and I've ran them from a 5V@3A adapter and a RasPi, and even a 12V@2A adapter and a router (Mi Router 3G). No issues whatsoever. It's only the N2 that refuses to work properly with it.


@tobetter: Is it possible that because of my RAID0 array's default block size of 64kB, the patching does not work? I'd try to reformat and lower it but ATM have no other storage option and I have some documents on the array I need.

EDIT:

Weird thing happening. Yesterday I did some copy work and moved data to the SSD from the RAID array (preparing to recreate it, just not yet). I rebooted as I was getting the XHCI errors again, and since... Nothing. The server downloaded about 20 gigs since, directly written to the RAID array (I disabled pooling incomplete downloads to the cache drive), and still no crash. No patch/workaround applied, just latest updates from APT - except for kernel, I'm going to download and install 4.9.184-37 right now (currently on 182-35).


EDIT2:

Apparently sequential reading speeds up the process in my case. Downloads are fine, but I can barely stream 10-15 minutes via Plex (no transcoding) before it crashes the device.

EDIT3:

Definitely a sequential read thing. Stopped torrent after an hour of continuous, 7MB/s (not a typo, 7 megabytes per sec, roughly 24GB of download) download which went without problems, waited a bit, started a stream via Plex and it died in 5 minutes... Streaming a different file at each time, so its not a single file causing the issue.

[Soleil]

On power supply side - when I've run into the issue had same thought process about voltage dropping.
Issue happened to me with externally powered USB hub too as well as power supply 12V/6A (72W) - tried with this one:
JOYLIT-1260 - https://www.amazon.fr/dp/B0716XX3S6
Whilst it says to power LED, it was the most powerful one I could quickly come up at the time of testing.
Neither didn't help.

[rondi7]

@Soleil Have you tried this==posted 10 or so posts before this one.
by elatllat » Sun Jul 07, 2019 10:09 am
"I use a piece of paper and some tape to cover the 5v on the USB to isolate the SBC from a powered hub, so anyone who is able to reproduce this issue could just do that in about a minute and know if it's a power issue. "

I'm not sure of his results--he didn't say, but perhaps isolating the 2 5v supplies might help. I have no idea if the N2 5v is connected to the your external powered hub--I would hope not.

[nikodemus]

@fonic: Thank you for taking time testing that much!

@Soleil
The N2 power supply is not questionable. It provides reliable 900 mA for USB3. In addition with the external hub we have enough power to supply the disk (it will not help to cut 5V line between N2 and active hub). But the critical distance still is the cabling between USB and the device. USB provides 5V, the device will get maximum 4,9V or below. USB cables are for data connections, not power source. With some cables i measured a drop to 4,6V with current of 500 mA, with 1000 mA the drop will be even higher (keep in mind Ohms Law). So what we need is a short and thick wiring. Best would be to use y cables.
I am not quite aware, if we have underpowering issues, but it must be clear before futher investigations.

[mad_ady]

It would be nice if there were some cheap device you could plug between the sata power interface and the disk to measure and display instant power and current - like some usb dongles do...

[elatllat]

You'd need a 5GHz oscilloscope for that, way easier to just cover the five bolt pin with paper and tape.

[nikodemus]

I must excuse, what would it bring to cover the 5V line? Where will the power come from?
Besides if you want to check signal quality of the data lines, you need way more than 5Ghz (sampling theoreme).

[fonix232]

@fonic: Thank you for taking time testing that much!

@Soleil
The N2 power supply is not questionable. It provides reliable 900 mA for USB3. In addition with the external hub we have enough power to supply the disk (it will not help to cut 5V line between N2 and active hub). But the critical distance still is the cabling between USB and the device. USB provides 5V, the device will get maximum 4,9V or below. USB cables are for data connections, not power source. With some cables i measured a drop to 4,6V with current of 500 mA, with 1000 mA the drop will be even higher (keep in mind Ohms Law). So what we need is a short and thick wiring. Best would be to use y cables.
I am not quite aware, if we have underpowering issues, but it must be clear before futher investigations.

I'm testing with extra short cables (15cm USB A to C, 3.1 gen2 certified, 10Gbps cables), while the NVMe SSD is connected on a longer, 1m cable of the same brand and make. I've checked official specs, and the NVMe SSD seems to be using 6.1W tops with idle being around ~0.5W, whereas the HDDs go around 4.5W at startup, idling at <1W, and continuous read-write uses around 2W. I can see how the SSD might be causing power issues, especially if it's accessed continuously (it is, since it's used as a cache+docker config storage). I'll try moving docker config to the eMMC, and remove the SSD completely, see if it at least partially fixes my issue.

I've also done some further testing, and it's NOT the reading - I managed to watch (and watch only) roughly 6 hours of stream without a reboot, but then I started a heavier download (multiple files, so multiple disk accesses during write), it died within 10 minutes.

[fonix232]

Has anyone tried using MergerFS instead of a RAID array?

[Nighti]

Hello,
Just to make sure this topic will stay alive: I got my N2 last week and have the same issue.
My setup:
* Mulitple external HDD, powered by own power supply
* Attached directly to USB ports without any additional hubs
* Suppressed the usage of UAS, only usb-storage is being used

Error Description:
While moving files from one drive to another xHCI stops responding:
...
Jul 24 11:33:25 odroid kernel: xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: Host not halted after 16000 microseconds.
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: Non-responsive xHCI host is not halting.
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: Completing active URBs anyway.
Jul 24 11:33:26 odroid kernel: hub 2-1:1.0: hub_ext_port_status failed (err = -108)
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
Jul 24 11:33:26 odroid kernel: xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
...

Lowering the max_sector size seems to help but slows the drives drastically. Looks to me like a kernel/module bug. It would be great if this issue can be prioritized as it makes the whole speed advantage of USB3 and N2 obsolete. Let me know if any additional logs might help.

[Nighti]

@tobetter: I don't have enough posts to send private messages, yet .
Some additional information:
root@odroid[~]$ uname -a
Linux droid 4.9.184-37 #1 SMP PREEMPT Wed Jul 10 02:50:15 -03 2019 aarch64 aarch64 aarch64 GNU/Linux

Board purchased from ameridroid.com two weeks ago.

Right now I went down to max_sectors_kb = 16. Let's see how long it will take till the next fail.

Update:
* sg_tablesize is still unchanged
* Two 8TB external drives connected directly to the board.
Bus 002 Device 005: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 004: ID 1058:25ee Western Digital Technologies, Inc.

Let me know how I can send you other additional information.

[tobetter]

@tobetter: I don't have enough posts to send private messages, yet .
Some additional information:
root@odroid[~]$ uname -a
Linux droid 4.9.184-37 #1 SMP PREEMPT Wed Jul 10 02:50:15 -03 2019 aarch64 aarch64 aarch64 GNU/Linux

Board purchased from ameridroid.com two weeks ago.

Right now I went down to max_sectors_kb = 16. Let's see how long it will take till the next fail.

Update:
* sg_tablesize is still unchanged
* Two 8TB external drives connected directly to the board.
Bus 002 Device 005: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 004: ID 1058:25ee Western Digital Technologies, Inc.

Let me know how I can send you other additional information.

Thanks, I would like to have the mac address of your ODROID-N2, you can grab it with the command ifconfig eth0. Also would be nice you could try out the updated kernel and U-boot, boot with additional kernel argument usb-xhci.tablesize=2 if this improves the stability. I think you can try without changing the default value of max_sectors_kb.
posting.php?mode=quote&f=181&p=263270#pr261105

[Nighti]

Thanks, I would like to have the mac address of your ODROID-N2, you can grab it with the command ifconfig eth0. Also would be nice you could try out the updated kernel and U-boot, boot with additional kernel argument usb-xhci.tablesize=2 if this improves the stability. I think you can try without changing the default value of max_sectors_kb.
posting.php?mode=quote&f=181&p=263270#pr261105

I will pm you the mac address once I have enough forum posts .
Adding usb-xhci.tablesize=2 with the 4.9.184-37 kernel doesn't help. If you want me to try different one, would be nice to have it packaged. Replacing the boot files manually have a high risk to make the server unusable.

I've tried in addition my older USB2.0 2TB HDDs. They are getting USB resets:
Jul 25 00:40:33 odroid kernel: usb 1-1.4.3.2: reset high-speed USB device number 10 using xhci-hcd

Looks to me like the xhci-hcd module is highly unstable for whatever reason.

[elatllat]

...kernel...

care to try 5.3?
viewtopic.php?p=263361&sid=f72f801b1795 ... 50#p263225

[tobetter]

Thanks, are you able to capture the logs from U-boot to kernel if you have USB-to-Serial board?

Could you let me know the model or the product of you USB storage? Does it run with extenral power?

It could let me have the mac address of your N2 if you don't mind.

"Tapatalk wishes you to have fun with ODROID"

[Nighti]

@elatllat: I would prefer to see it fixed for 18.04 LTS. Jumping from one unstable thing to another doesn't really help.

@tobetter:
Mac: "00:1e:06:42:23:cd"

I don't have such a device yet. Do you have more information what is required to capture those logs?

Here are my hard drives currently attached: All are external powered

root@odroid[~]$ lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 1: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 2: Dev 6, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 3: Dev 8, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 4: Dev 3, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 1: Dev 5, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 3: Dev 7, If 0, Class=Hub, Driver=hub/4p, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 6, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 1: Dev 7, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 3: Dev 8, If 0, Class=Hub, Driver=hub/4p, 480M
|__ Port 4: Dev 10, If 0, Class=Mass Storage, Driver=usb-storage, 480M
|__ Port 4: Dev 9, If 0, Class=Mass Storage, Driver=usb-storage, 480M
root@odroid[~]$ lsusb
Bus 002 Device 007: ID 2109:0812 VIA Labs, Inc. VL812 Hub
Bus 002 Device 005: ID 2109:0812 VIA Labs, Inc. VL812 Hub
Bus 002 Device 003: ID 2109:0812 VIA Labs, Inc. VL812 Hub
Bus 002 Device 008: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 006: ID 1058:25ee Western Digital Technologies, Inc.
Bus 002 Device 004: ID 1058:25ee Western Digital Technologies, Inc.
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 009: ID 1058:1021 Western Digital Technologies, Inc. Elements Desktop (WDBAAU)
Bus 001 Device 010: ID 04fc:0c25 Sunplus Technology Co., Ltd SATALink SPIF225A
Bus 001 Device 008: ID 2109:2812 VIA Labs, Inc. VL812 Hub
Bus 001 Device 007: ID 2109:2812 VIA Labs, Inc. VL812 Hub
Bus 001 Device 006: ID 2109:2812 VIA Labs, Inc. VL812 Hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

[elatllat]

@elatllat: I would prefer to see it fixed for 18.04 LTS. Jumping from one unstable thing to another doesn't really help.
...

Ubuntu 18.04 has nothing to do with Kernel 5.3 (you can mix and match most any modern kernel and distribution)
I'd trust the well tested upstream to have fewer bugs than the comparatively untested thousands of changes on the HK branch.
But it's just testing so once the bug is isolated it can be fixed, while you revert to whatever environment you are comfortable with.

[Nighti]

Ubuntu 18.04 has nothing to do with Kernel 5.3 (you can mix and match most any modern kernel and distribution)
I'd trust the well tested upstream to have fewer bugs than the comparatively untested thousands of changes on the HK branch.
But it's just testing so once the bug is isolated it can be fixed, while you revert to whatever environment you are comfortable with.

Disagree. Each distribution/version provides you with kernel support. Means security updates and stability patches will be provided and applied. Once you start using your own kernel, you have to take care of this.
You can mix the kernel versions to some level (as long as the used part stays compatible). In most cases you're stuck at the same kernel version once the distribution version is released.

I understand that you're trying to isolate an issue by trying as many versions as possible. If the actual kernel developer who is looking into this issue think it's helpful to get this data, I'm glad to help. In general it's usually not helpful.

[elatllat]

Ubuntu 18.04 has nothing to do with Kernel 5.3 (you can mix and match most any modern kernel and distribution)
I'd trust the well tested upstream to have fewer bugs than the comparatively untested thousands of changes on the HK branch.
But it's just testing so once the bug is isolated it can be fixed, while you revert to whatever environment you are comfortable with.

Disagree. Each distribution/version provides you with kernel support. Means security updates and stability patches will be provided and applied. Once you start using your own kernel, you have to take care of this.
You can mix the kernel versions to some level (as long as the used part stays compatible). In most cases you're stuck at the same kernel version once the distribution version is released.

I understand that you're trying to isolate an issue by trying as many versions as possible. If the actual kernel developer who is looking into this issue think it's helpful to get this data, I'm glad to help. In general it's usually not helpful.

I know this is a trivial aside but
hardkernel etc are typically a moth behind the official upstream maintainer gregkh on security updates and stability patches, so a cron script may arguably provide better support.

[Nighti]

Yeah, I love this but. It never gets old

[fonix232]

@elatllat: I would prefer to see it fixed for 18.04 LTS. Jumping from one unstable thing to another doesn't really help.
...

Ubuntu 18.04 has nothing to do with Kernel 5.3 (you can mix and match most any modern kernel and distribution)
I'd trust the well tested upstream to have fewer bugs than the comparatively untested thousands of changes on the HK branch.
But it's just testing so once the bug is isolated it can be fixed, while you revert to whatever environment you are comfortable with.

Tried your kernel on 19.04, now the system won't boot.

No offense but there's no point recommending Armbian at every step if they don't even have the kernel version you're referring to as nightly builds.

[elatllat]

@elatllat: I would prefer to see it fixed for 18.04 LTS. Jumping from one unstable thing to another doesn't really help.
...

Ubuntu 18.04 has nothing to do with Kernel 5.3 (you can mix and match most any modern kernel and distribution)
I'd trust the well tested upstream to have fewer bugs than the comparatively untested thousands of changes on the HK branch.
But it's just testing so once the bug is isolated it can be fixed, while you revert to whatever environment you are comfortable with.

Tried your kernel on 19.04, now the system won't boot.

No offense but there's no point recommending Armbian at every step if they don't even have the kernel version you're referring to as nightly builds.

I have never used or recommended Armbian (are you confusing me with igorpec?) I use software directly from kernel.org and directly from debian/ubuntu,fedora/centos,arch/manjaro... (maybe someday upstream uboot)

[fonix232]

Tried your kernel on 19.04, now the system won't boot.

No offense but there's no point recommending Armbian at every step if they don't even have the kernel version you're referring to as nightly builds.

I have never used or recommended Armbian (are you confusing me with igorpec?) I use software directly from kernel.org and directly from debian/ubuntu,fedora/centos,arch/manjaro... (maybe someday upstream uboot)

Ah, sorry, indeed, the comments slipped together. Shouldn't reply to forum posts after 9PM However I did use your 5.3 kernel image, and unfortunately it does not boot I don't have an UART adapter at hand either, so I'm going to re-image the eMMC tomorrow.

[elatllat]

... I did use your 5.3 kernel image, and unfortunately it does not boot I don't have an UART adapter at hand either, so I'm going to re-image the eMMC tomorrow.

Only thing I can think of is I'm using ssh so maybe you judged the "no boot" as a HDMI issue.