USB disconnects randomly

[binhex]

To my second N2, running Ubuntu, there is a USB3 HDD connected. Same disconnects. Even it restarts randomly. I'm getting disappointed with the N2.

interesting!, so i am as you know running CoreELEC and have a USB3 external HDD connected (externally powered) and have no issues with disconnects or random restarts, do you have the same issue with your HDD on CoreELEC?

No restarts only happen on Ubuntu. Disconnects of the HDD under Ubuntu I see sometimes. I log them. Last to where on 9. and 16. of July. Disconnects are not present when using the otg USB port.

I found it also interesting in the beginning, however, I'm getting tired a bit since there is no solution for 3 months now. Almost any USB related problem in this forum is related to this issue described here. Same holds for the Coreelec forum.

i get your frustration, i really do!, i guess i just got lucky with my hardware, in that my tv tuners (all xbox one tuners) and my external hdd all play nicely, i truly hope this issue gets resolved for you.

edit - my concern is that the issue is a hardware one, and thus cannot be fixed, i REALLY hope this isnt the case.

[rosenrot]

To my second N2, running Ubuntu, there is a USB3 HDD connected. Same disconnects. Even it restarts randomly. I'm getting disappointed with the N2.

interesting!, so i am as you know running CoreELEC and have a USB3 external HDD connected (externally powered) and have no issues with disconnects or random restarts, do you have the same issue with your HDD on CoreELEC?

No restarts only happen on Ubuntu. Disconnects of the HDD under Ubuntu I see sometimes. I log them. Last to where on 9. and 16. of July. Disconnects are not present when using the otg USB port.

I found it also interesting in the beginning, however, I'm getting tired a bit since there is no solution for 3 months now. Almost any USB related problem in this forum is related to this issue described here. Same holds for the Coreelec forum.

i get your frustration, i really do!, i guess i just got lucky with my hardware, in that my tv tuners (all xbox one tuners) and my external hdd all play nicely, i truly hope this issue gets resolved for you.

edit - my concern is that the issue is a hardware one, and thus cannot be fixed, i REALLY hope this isnt the case.

So all your devices are connected to the rear ports? Does it work since the fix of tobetter which he came up with recently?

I mean it is rather unlikely that both of my devices show a hardware issue. Both from different suppliers delayed by one month.

However, if so, I would be happy if Odroid would exchange my units.

[binhex]

So all your devices are connected to the rear ports? Does it work since the fix of tobetter which he came up with recently?

all devices are connected via rear port, yes, i also have a usb IR receiver hanging off the back (USB 2 device) and that also works fine.

I mean it is rather unlikely that both of my devices show a hardware issue. Both from different suppliers delayed by one month.

However, if so, I would be happy if Odroid would exchange my units.

I got my unit from odroid uk and i was one of the early ones to receive my unit, so if there was a production problem then i would expect i would be hit by it, with later units having the 'fix' in place.

What i really meant was a design issue with the usb hub in some way, such that it doesn't play nicely with certain usb devices, not that there are 'bad' production runs - i must stress this is PURELY SPECULATION, not saying this is the issue at all.

[tobetter]

@roserot, could you please share the picture of how your USB devices are attached to N2? If you are not willing to be shown by someone else, it's ok. It's also good to know what's the brand and the model of your disk, probably more importantly the USB-to-SATA controller. Since some user has a similar issue with a different product, the problem could depend on the product attached. I am also not considering the case of faulty hardware. By any chance, are you able to measure the voltage at pin #1 and #2 on GPIO header when USB disconnection happens?

[rosenrot]

@roserot, could you please share the picture of how your USB devices are attached to N2? If you are not willing to be shown by someone else, it's ok. It's also good to know what's the brand and the model of your disk, probably more importantly the USB-to-SATA controller. Since some user has a similar issue with a different product, the problem could depend on the product attached. I am also not considering the case of faulty hardware. By any chance, are you able to measure the voltage at pin #1 and #2 on GPIO header when USB disconnection happens?

If you want I can share a picture. However, I plugged the Seagate Backup Plus Portable 4TB with the original cable to one of the rear ports. I don't know about the SATA controller. I can not measure the voltage of the GPIOs while the disconnect happens because this would require some high temporal resolution logging hardware.

Honestly, I would love to solve the issue and to contribute in solving it, however, at some point I have to tell you that it is not my job to do measurements for Odroid. I bought this product and I think I can expect it to work properly when running their OS and a general USB HDD. Instead I do own two N2s now, both having USB issues and no solution to solve it available. If it would be a RPI for 35 USD I would say okay, bad luck. However, it is above 100 USD.

I was really happy to see the N2's power consumption and I wanted to use it as a home server. However, it is not reliable at all.

Sorry for being a little harsh but after 3 months without any solution, I might be allowed to say one or another word. I'm not personal, I just think at some point we should accept that their is an issue and look for a solution that doesn't require the customer to solve it.

@binhex: I got mine as well from odroid uk 2 days after the release. The other one about a month later from another distributor.

@tobetter: My words are not towards you. They are pointed towards Odroid, I don't see any contribution to this topic since some time, at least not officially. I don't know what is going on behind the scene.

The producer - customer relationship should look like this:
Producer: Releases N2
Customer: Buys N2
Customer: Has issue
Producer: Investigates
Producer: Able to reproduce, yes or no.
Producer: Should provide a solution to the customer, independent of the investigation result
Customer: Happy

[mad_ady]

@rosenrot: @tobetter is on the Hardkernel team, so his replies represent official responses.

[gregeeh]

I bought this product and I think I can expect it to work properly when running their OS and a general USB HDD. Instead I do own two N2s now, both having USB issues and no solution to solve it available

I totally agree. We have been very patient with this and as @rosenrot says we have paid for a product that does not work. I like @rosenrot cannot use my N2 until this issue with the rear USB ports is solved 100%. As it is I have wasted my money on a faulty product.

[gmdev]

It's not just the HDD, as it was earlier pointed out, as it's a general USB failure issue.

[rosenrot]

@rosenrot: @tobetter is on the Hardkernel team, so his replies represent official responses.

Oh, I didn't know that. Summarizing, I would be happy for a final solution.

And just to make myself clear. I really like the Hardkernel products and I have some other boards at home. This time the N2 causes me headache but that doesn't change my opinion on the good quality they produce.

[tobetter]

@rosenrot: @tobetter is on the Hardkernel team, so his replies represent official responses.

Oh, I didn't know that. Summarizing, I would be happy for a final solution.

And just to make myself clear. I really like the Hardkernel products and I have some other boards at home. This time the N2 causes me headache but that doesn't change my opinion on the good quality they produce.

Sorry for making you have a headache and glad to know you like Hardkernel products. As of now, keep testing USB issue and arguing with Amlogic to find out the solution. The USB issues I have acknowledged from different users are various and sometimes even myself is confusing which is which since some user use without a problem or just silent. I am trying to bring up the issue as generic USB issue with Amlogic, not just for storage...also looking into power problem in case as someone else has evaluated with multiple disk storages.

[nick793]

I bought this product and I think I can expect it to work properly when running their OS and a general USB HDD. Instead I do own two N2s now, both having USB issues and no solution to solve it available

I totally agree. We have been very patient with this and as @rosenrot says we have paid for a product that does not work. I like @rosenrot cannot use my N2 until this issue with the rear USB ports is solved 100%. As it is I have wasted my money on a faulty product.

Yep, I'm stuck until this is solved. While I can run the device, I'm afraid to because of the potential for data corruption.

Sorry for making you have a headache and glad to know you like Hardkernel products. As of now, keep testing USB issue and arguing with Amlogic to find out the solution. The USB issues I have acknowledged from different users are various and sometimes even myself is confusing which is which since some user use without a problem or just silent. I am trying to bring up the issue as generic USB issue with Amlogic, not just for storage...also looking into power problem in case as someone else has evaluated with multiple disk storages.

@tobetter I completely understand. I'm an electrical engineer myself and I'm sorry that you're stuck in this position.

If it helps, I've only noticed the problem during high bitrate reads (1.5Gbps+). I've been building a RAID5 array for the past week (it seems stuck at a total of 240Mbps total read/write) but the N2 keeps on running smooth! I direct streamed 12Mbps video from Plex last night for hours without issues.

I've only noticed failures during verification of torrents (high sequential reads). At that point the N2 fails, but it seems fine otherwise (only the USB resets from what I can tell, but I run most of my workload off USB so that's why I thought it was rebooting)

If I had to take a guess, I'd take a look at the power that's feeding into your USB high speed transmitters/receivers. If it was a USB device drawing too much power we would see 5V sag on the USB output when your efuses kick in.

Without seeing the schematics, is it possible there might be a decoupling/filtering issue on the input to your high speed USB hub?

[nick793]

Ok I might be wrong, but I just took a look at the schematics you posted for the N2 and a datasheet for the Gensys GL3523 that you're using. I don't have detailed info on it, but it looks like you're feeding your 3.3V analog and digital supplies from the same source without any filtering between the two. Not sure if that matters but I'd look at that supply first.

Also, the 5V_2 pin (pin 42 on the GL3523) is actually labelled as a 5V input for the embedded 1.2V switching regulator in the datasheet (labelled VDDP). However, the pin is shorted to 3.3V - without info regarding the internal regulator, can it handle this? The 5V_1 pin (labelled 5V) can and the datasheet says you can connect it to 3.3V directly, but the VDDP pin doesn't say.

Also your USB reset line is held to your 3.3V rail through a resistor divider. Any glitch on that rail would put the whole controller into reset.

Is it possible during high speed operation (when the 1.2V switching regulator is under high load) that it pulls the 3.3V line down and trips the reset?

[tobetter]

Ok I might be wrong, but I just took a look at the schematics you posted for the N2 and a datasheet for the Gensys GL3523 that you're using. I don't have detailed info on it, but it looks like you're feeding your 3.3V analog and digital supplies from the same source without any filtering between the two. Not sure if that matters but I'd look at that supply first.

Also, the 5V_2 pin (pin 42 on the GL3523) is actually labelled as a 5V input for the embedded 1.2V switching regulator in the datasheet (labelled VDDP). However, the pin is shorted to 3.3V - without info regarding the internal regulator, can it handle this? The 5V_1 pin (labelled 5V) can and the datasheet says you can connect it to 3.3V directly, but the VDDP pin doesn't say.

Also your USB reset line is held to your 3.3V rail through a resistor divider. Any glitch on that rail would put the whole controller into reset.

Is it possible during high speed operation (when the 1.2V switching regulator is under high load) that it pulls the 3.3V line down and trips the reset?

Thanks, let me share the thing with H/W engineer and confirm.

[rosenrot]

Just another detail. I connected an esp8266 wifi uart bridge to the uart connector. While doing so I heard my USB HDD spinning down. So a load of 70mA on the uart 3.3V spins down the USB HDD? Kind of weird to me.

By the way, which schematic for the UART port is correct?

https://wiki.odroid.com/odroid-n2/hardware

Or the one from the schematic? TX RX are flipped.

[fcmts]

I also guess there is some power issue with the USB ports. I have an old USB webcam (Pixart) that I can't make it work on N2, though it works on a RPi 3+.

$ lsusb
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 010: ID 046d:c534 Logitech, Inc. Unifying Receiver
Bus 001 Device 012: ID 093a:2620 Pixart Imaging, Inc.
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

$ LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libv4l/v4l1compat.so fswebcam
--- Opening /dev/video0...
Trying source module v4l2...
/dev/video0 opened.
No input was specified, using the first.
Adjusting resolution from 384x288 to 640x480.
--- Capturing frame...
Timed out waiting for frame!
No frames captured.

The same in Ubuntu or Armbian.
Besides that, it has issues with an USB wireless keyboard+mouse (Logitech shown above).

[fonix232]

Is it possible during high speed operation (when the 1.2V switching regulator is under high load) that it pulls the 3.3V line down and trips the reset?

This would explain why it happens so randomly. I roughly correlated my USB failures to times when the disk suddenly had to wake up and do lots of random seeks - which would be a sudden spike increase in power use, which would trip the reset, and to times when it was consistently used at high speeds, but the usage changes (e.g. watching a TV show and going to the next episode, while downloading torrents). If the disks were ramped up nice and slow at boot, and gradually used (torrents off, just a single video stream), they would go on for days.

And if I get your words right, this would mean a mass recall of the N2.

Wasn't it the same with the N1 too? But the issues there were a bit more prominent (or rather, more internal QA testing was done - it's not exactly the most uncommon use case to have a hard drive or two connected to such an SBC...), and the line was scrapped soon enough.

So Odroid, what are we getting? Hardware replacement of our current N2s with a fixed version, or N3?

[nick793]

Is it possible during high speed operation (when the 1.2V switching regulator is under high load) that it pulls the 3.3V line down and trips the reset?

This would explain why it happens so randomly. I roughly correlated my USB failures to times when the disk suddenly had to wake up and do lots of random seeks - which would be a sudden spike increase in power use, which would trip the reset, and to times when it was consistently used at high speeds, but the usage changes (e.g. watching a TV show and going to the next episode, while downloading torrents). If the disks were ramped up nice and slow at boot, and gradually used (torrents off, just a single video stream), they would go on for days.

And if I get your words right, this would mean a mass recall of the N2.

Wasn't it the same with the N1 too? But the issues there were a bit more prominent (or rather, more internal QA testing was done - it's not exactly the most uncommon use case to have a hard drive or two connected to such an SBC...), and the line was scrapped soon enough.

So Odroid, what are we getting? Hardware replacement of our current N2s with a fixed version, or N3?

I'm hoping I'm wrong and that it's just a kernel issue. I don't have the tools to verify it on my end, so this is all speculation at best.

That being said, I can correlate my reboots/USB failures with a sustained spike in drive reads (SSD or HDD). The most surefire way I've found is using the "verify local data" button in Transmission. If I verify more than one torrent in a row then I can guarantee the USBs will fail within 5 minutes. However, during that time I'm seeing sustained reads around 200MB/s+. When streaming a movie or letting transmission seed the box performs beautifully.

[fonix232]

I'm hoping I'm wrong and that it's just a kernel issue. I don't have the tools to verify it on my end, so this is all speculation at best.

That being said, I can correlate my reboots/USB failures with a sustained spike in drive reads (SSD or HDD). The most surefire way I've found is using the "verify local data" button in Transmission. If I verify more than one torrent in a row then I can guarantee the USBs will fail within 5 minutes. However, during that time I'm seeing sustained reads around 200MB/s+. When streaming a movie or letting transmission seed the box performs beautifully.

I'm using Deluge, and for me even a simple torrent download can make it crash and burn randomly. Although I am using two disks joined into a RAID0 array, which might add to the issue.

[nick793]

Ok on second look, I found where they derive 3.3V from and I'd be shocked if they were able to pull that low enough to reboot the board (or trip the 3.3V reset on the USB). So that's likely out.

But I still don't know if 3.3V is too low a voltage for the GL3523 5V-1.8V switching regulator. That's ultimately a question for Hardkernel's hardware engineers to answer, but it's the only hardware oriented concern I could identify while staring at this thing.

I'm curious to see if the folks dabbling with the 5.3-rc1 kernel are seeing these USB issues though.

[Nighti]

I have issues with various external HDD drives. All of them have their own power supply. So far the xhci-hcd module behave very unstable for whatever reason and I hope it's only a software issue.

[gmdev]

So it's starting to get annoying.
Basically the new board is useless, as it cannot handle basic usb devices , like HDD drive, Tv tuner, etc.
This is the main reason why people bought it in the first place.
@HK: your customers are waiting for an answer.

[odroid]

The external USB storage issue could be significantly reduced once you set the Kernel parameter usb-xhci.tablesize=2.
viewtopic.php?f=181&t=35031&start=50#p261377
We will release new Ubuntu minimal and Mate images in the next week with some other improvements.

We've been looking into the stability issues of the USB camera and tuners. But we need more time to find a firm solution.

[Nighti]

The external USB storage issue could be significantly reduced once you set the Kernel parameter usb-xhci.tablesize=2.
viewtopic.php?f=181&t=35031&start=50#p261377
We will release new Ubuntu minimal and Mate images in the next week with some other improvements.

We've been looking into the stability issues of the USB camera and tuners. But we need more time to find a firm solution.

Thanks for the update. Do you have any idea what this might cause this? If you need any data which helps investigate this issue, please let us know which one and how to get.
The thing is, even if the issue is reduced with some parameters, the N2 can't be used as a file server. It requires a stable environment. If the whole USB devices are either not responsive (usb reset) or getting removed from USB list, server reboot is required. Nobody would like to loose their data because of this issue.
So please let us know if the community can help.

[mad_ady]

@Nighti: When the usb resets don't you get the disks again but with different names? E.g. sda becomes sdb? With normal mounting rules I agree, you can't mount it because the node name changes and the mount point is stuck on the old name. When I had this issue on the xu4 (it would reset even during light reads), I could make it beareble by mounting the disks with autofs by uuid. If the reset happens when disks are idle (I know, it never happens when idle), then autofs recovers gracefully. When usb resets under load, autofs (and nfs) hang. You should be able to recover them without reboot by:

Code: Select all

sudo service autofs restart
sudo service nfs-mountd restart

That should recover your mountpoints.
On the client you may need to restart autofs as well (for nfs shares).
You could script it by reading (tail -f) syslog and when you identify a usb reset, issue the commands automatically and ssh (with keys) into the client(s) to restart autofs.
It may be problematic when writing data (the writing process will probably fail), but should be relatively smooth for reading processes (kodi might absorb it via buffering).
I admit, it's not a fix, but might make the platform more usable in the meantime...

[gmdev]

The external USB storage issue could be significantly reduced once you set the Kernel parameter usb-xhci.tablesize=2.
viewtopic.php?f=181&t=35031&start=50#p261377
We will release new Ubuntu minimal and Mate images in the next week with some other improvements.

We've been looking into the stability issues of the USB camera and tuners. But we need more time to find a firm solution.

This is not a solution, it is a temporary workaround, as you have said, it doesn't fix the issues, just reduces it "significantly".

Under no circumstances can this be a solution as it will randomly lead to data corruption.

[odroid]

I have to agree that.
So we've started a more intensive test to make a rock solid solution.
viewtopic.php?f=181&t=35031&start=100#p263905

[Nighti]

@mad_ady: The issue was that the devices were removed from the USB driver. No reattaching (in software) is possible.
But even if it would be possible, it's not a solution. FS is corrupted in this case and FS repair is forced after the incident. So not really usable.

I think it would make sense to move the discussion to this place: viewtopic.php?f=181&t=35031&start=100#p263905
as all testing and progress is happening there.