Restart xhci-hcd module

[mengxi]

Hi all, is it possible to restart the xhci-hcd module without rebooting the system?

[tobetter]

Hi all, is it possible to restart the xhci-hcd module without rebooting the system?

When do you need to restart the module?
Do you experience that USB devices are blocked?

[mengxi]

Hi all, is it possible to restart the xhci-hcd module without rebooting the system?

When do you need to restart the module?
Do you experience that USB devices are blocked?

We found out of all the Odroid N2 we have, 15% of them have unstable OTG communication. If it works fine at the start of a boot cycle, it would work fine for the rest of the time. You can unplug / replug in our OTG device, or power cycle it with the code you have provided that toggle GPIO6, all without any issue. However, sometimes it would boot, and OTG communication is either very unstable or does not recognize the device at all, this issue can only be resolved by keep rebooting the computer or keep unplug and then re plug in the power, until we get to one of the boot situations where OTG communication is stable. This is not an issue with our OTG device, because the same device would work well on different Odroids.

Here's the dmesg that shows one of the situations where the boot cycle result in the device not connecting at all, I want to restart xhci-hcd because somewhere online they say restarting xhci-hcd would fix the issue.
[ 5.114900] usb 1-2: new high-speed USB device number 3 using xhci-hcd
[ 5.255351] usb 1-2: New USB device found, idVendor=------, idProduct=-------
[ 5.256572] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 5.263867] usb 1-2: Product: ----------
[ 5.268709] usb 1-2: Manufacturer: ------------
[ 6.353616] usb 1-2: USB disconnect, device number 3 <<<<<<<<<<<<<<<<<<<< OTG device keep disconnecting
[ 6.654979] usb 1-2: new high-speed USB device number 4 using xhci-hcd
[ 6.795420] usb 1-2: New USB device found, idVendor==------, idProduct=-------
[ 6.795427] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 6.795430] usb 1-2: Product: ----------
[ 6.795432] usb 1-2: Manufacturer: ------------
[ 10.449598] usb 1-2: USB disconnect, device number 4
[ 10.746899] usb 1-2: new high-speed USB device number 5 using xhci-hcd
[ 10.887337] usb 1-2: New USB device found, idVendor= ------------, idProduct= ------------
[ 10.887341] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 10.887343] usb 1-2: Product: ------------
[ 10.887345] usb 1-2: Manufacturer: ------------
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Eventually it causes error at our OTG device
[ 38.418288] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Recovery port(1) tuning for host cf(xhci_irq)--
[ 38.418439] usb 1-2: USB disconnect, device number 12
[ 38.658953] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Set port(1) tuning for host cf(xhci_hub_control)--
[ 38.718922] usb 1-2: new high-speed USB device number 13 using xhci-hcd
[ 39.121529] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Recovery port(1) tuning for host cf(xhci_irq)--
[ 39.386929] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Set port(1) tuning for host cf(xhci_hub_control)--
[ 39.446966] usb 1-2: device descriptor read/64, error -71
[ 40.465543] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Recovery port(1) tuning for host cf(xhci_irq)--
[ 40.730943] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Set port(1) tuning for host cf(xhci_hub_control)--
[ 40.790947] usb 1-2: device descriptor read/64, error -71
[ 41.010405] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Recovery port(1) tuning for host cf(xhci_irq)--
[ 41.018924] usb 1-2: new high-speed USB device number 14 using xhci-hcd
[ 41.286938] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Set port(1) tuning for host cf(xhci_hub_control)--
[ 41.346963] usb 1-2: device descriptor read/64, error -71
[ 42.513534] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Recovery port(1) tuning for host cf(xhci_irq)--
[ 42.778948] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Set port(1) tuning for host cf(xhci_hub_control)--
[ 42.838953] usb 1-2: device descriptor read/64, error -71
[ 42.946974] usb usb1-port2: attempt power cycle
[ 43.154983] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Recovery port(1) tuning for host cf(xhci_irq)--
[ 43.530969] amlogic-new-usb2-v2 ffe09000.usb2phy: ---Set port(1) tuning for host cf(xhci_hub_control)--
[ 43.590926] usb 1-2: new high-speed USB device number 15 using xhci-hcd
[ 43.590973] usb 1-2: Device not responding to setup address.
[ 43.798945] usb 1-2: Device not responding to setup address.
[ 44.006918] usb 1-2: device not accepting address 15, error -71
[ 44.126918] usb 1-2: new high-speed USB device number 16 using xhci-hcd
[ 44.126955] usb 1-2: Device not responding to setup address.
[ 44.334947] usb 1-2: Device not responding to setup address.
[ 44.542915] usb 1-2: device not accepting address 16, error -71
[ 44.543395] usb 1-2: the parent's name is usb1
[ 44.543458] usb usb1-port2: Device no response
[ 44.548028] usb usb1-port2: unable to enumerate USB device

So this issue is not fixed by unplugging and plugging in our device, and power cycling it by turning off the VBUS doesn't work either. So after the boot process (run level 3 or run level 5) I want to restart the xhci-hcd, is that possible?

[mengxi]

I have also read that EHCI module conflicts with XHCI module. Is it possible to disable ehci on Odroid N2?

[tobetter]

I have also read that EHCI module conflicts with XHCI module. Is it possible to disable ehci on Odroid N2?

Are you trying to run your devices on USB 3.0 or 2.0? And what are the devices or do you know what is the USB controller inside of the device?
What's the current kernel version running now with which OS?
Also what ODROID do you mean by " because the same device would work well on different Odroids."? Other ODROID-N2 or other models?

[mengxi]

I have also read that EHCI module conflicts with XHCI module. Is it possible to disable ehci on Odroid N2?

Are you trying to run your devices on USB 3.0 or 2.0? And what are the devices or do you know what is the USB controller inside of the device?
What's the current kernel version running now with which OS?
Also what ODROID do you mean by " because the same device would work well on different Odroids."? Other ODROID-N2 or other models?

The OTG port is USB 2.0, but online it says XHCI suppose to work with both USB 2.0 and USB 3.0, so disable EHCI could be something worth to try.
Different Odroid N2, we have a lot of them. We put the same OTG device on different Odroid N2s, and observed ~15% cant communicate with the OTG device reliably by power cycling each N2 ~8 times and observing the result.

[mengxi]

I can find out what is the USB controller is running on the OTG device.

The OS is Ubuntu minimal
Linux odroid 4.9.196-63 #1 SMP PREEMPT Thu Oct 17 00:44:03 -03 2019 aarch64 aarch64 aarch64 GNU/Linux

[mengxi]

Again I want to try rebooting the XHCI instead of rebooting the whole OS because rebooting whole OS takes ~30 seconds and if it takes 6 boot cycles to get to one of those boots where OTG communication is consistent on some of the bad Odroid N2s, it would take a whole 3 minutes and not acceptable for us.

[mengxi]

Also we observed that out of the 500 boot cycles we have recorded, around 3% of the time boot never happens, the blue light never turns on, and we have to un-plug and replug in the device manually for it to boot again. So excessive number of boot cycles to enter in a good state of OTG communication is not desirable for us, as it could cause the device needing an operator to manually power cycle the device.

[tobetter]

I have also read that EHCI module conflicts with XHCI module. Is it possible to disable ehci on Odroid N2?

Are you trying to run your devices on USB 3.0 or 2.0? And what are the devices or do you know what is the USB controller inside of the device?
What's the current kernel version running now with which OS?
Also what ODROID do you mean by " because the same device would work well on different Odroids."? Other ODROID-N2 or other models?

The OTG port is USB 2.0, but online it says XHCI suppose to work with both USB 2.0 and USB 3.0, so disable EHCI could be something worth to try.
Different Odroid N2, we have a lot of them. We put the same OTG device on different Odroid N2s, and observed ~15% cant communicate with the OTG device reliably by power cycling each N2 ~8 times and observing the result.

The OTG port with a micro USB port won't work as USB 3.0, so disabling EHCI means no USB on the port.

[tobetter]

Again I want to try rebooting the XHCI instead of rebooting the whole OS because rebooting whole OS takes ~30 seconds and if it takes 6 boot cycles to get to one of those boots where OTG communication is consistent on some of the bad Odroid N2s, it would take a whole 3 minutes and not acceptable for us.

The rebooting XHCI/EHCI means that the driver modules should be changed as a module which I have not tried yet. Let me have a chance to look into it if feasible.

[tobetter]

Also we observed that out of the 500 boot cycles we have recorded, around 3% of the time boot never happens, the blue light never turns on, and we have to un-plug and replug in the device manually for it to boot again. So excessive number of boot cycles to enter in a good state of OTG communication is not desirable for us, as it could cause the device needing an operator to manually power cycle the device.

How do you make a boot cycle? reboot or something else? Do you boot with uSD or eMMC?
What is it intended to make a numerous rebooting test? For stability testing to meet the hardware qualification or for USB issue?

[tobetter]

I can find out what is the USB controller is running on the OTG device.

The OS is Ubuntu minimal
Linux odroid 4.9.196-63 #1 SMP PREEMPT Thu Oct 17 00:44:03 -03 2019 aarch64 aarch64 aarch64 GNU/Linux

Maybe you can let me know what is the USB device?
Does the device cause high bandwidth traffic like camera or video streaming?

[mengxi]

Thanks, let me know if it is feasible.

How do you make a boot cycle? reboot or something else? Do you boot with uSD or eMMC?
We found boot via "reboot" command or boot by unplug and replug the power does the same thing. Boot with eMMC.

What is it intended to make a numerous rebooting test?
So the device is a camera that takes high bandwidth.
We define a "steady state of communication" as the camera stays connected and we are able to get frames from it for 10 minutes without the device disconnecting, at which point we are reasonably sure it will stay connected for a very long time.
A "not steady state of communication" is anything other than a "steady state of communication", which could include the camera disconnecting, or every time we try to connect to the camera it will not give us frames.
For 85% of the N2s we have, it would always be in a "steady state of communication" no matter how many times we boot it. For the other 15%, some times the camera and N2 enters into a "not steady state of communication", the frequency of which depends on the N2. Some are 50% 50%, 80% 20%, etc.
The only fix we currently have, when it is in the "not steady state of communication", is to reboot it (either by "reboot" command or unplug/replug), and hope that on the next boot cycle, it is in the "steady state of communication". The issue is, it takes 30 seconds per reboot, and if we get unlucky, it won't be able to connect to the camera 10 times in a row. which means it takes a whole 5 minutes to boot. To make the matters worse, 3% of the time, after issuing the "reboot" command, the device may not reboot at all, and we have to manually unplug / replug.
We need these devices we ship out to be reliable, because they should work close to 100% of the time. This is why we kind of need to somehow reset the USB controller / software driver on the CPU without actually rebooting, to reduce reboot time and make it more reliable.

[tobetter]

Thank you for the detail.

I am not sure if your OS image has this workaround that assigns a core for USB XHCI. It's easily can be checked with a command cat /proc/cpuinfo | grep xhci and the workaround is applied if you can see the incremental counter happens in any within CPU2 to CPU5. Also, you could try out to increase the CPUFREQ for A73 and use performance CPU governor, please refer to the wiki for the same. I think it's worth to try with board in 15% instable.
https://wiki.odroid.com/odroid-n2/appli ... t_cpu_freq

I do remember that this script could be applied a few months ago.

Code: Select all

#!/bin/sh
echo none > /sys/class/leds/blue:heartbeat/trigger
sleep 30
USB3_IRQ=`cat /proc/interrupts | grep "xhci-hcd:usb1" | awk -F : '{print $1}' | sed s/" "//g`
echo 4 > /proc/irq/${USB3_IRQ}/smp_affinity_list
echo "Affinity set to"
echo `cat /proc/irq/${USB3_IRQ}/smp_affinity_list`

[mengxi]

Will try that when I'm back in the office and let you know the results.

[mengxi]

So the script had no effect, doing the cat ... produces no results before and after running the script.

What I found work to reset the USB OTG cable completely, so eventually the MCU and the OTG device will be able to communicate properly, was to execute the following command:

echo -n "xhci-hcd.0.auto" | tee /sys/bus/platform/drivers/xhci-hcd/unbind
echo reset > /sys/devices/platform/gpio-reset/reset-usb_otg/control
echo -n "xhci-hcd.0.auto" | tee /sys/bus/platform/drivers/xhci-hcd/bind

To execute the 2nd command you need to compile your own meson64_odroidn2.dtb and override the one in the /media/boot/ folder
viewtopic.php?f=177&t=37328#top

After the commands are executed, both the MCU and OTG device will be reset, you can keep trying resetting it until the connection becomes stable.