USB interrupts

[tuxx]

Hi all,

I recently did some tests while preparing my next image for Archphile.

After doing some finetuning with IRQ afffinity settings and set all usb interrupts to be sent to 4th core, I did some measurements with monitorix, listening to music with my usb dac (aune s16), which is the only attached usb device.

The distribution I use is Archlinuxarm. The kernel used is:

Code: Select all

[root@archphile ~]# uname -a
Linux archphile 3.14.79-7-ARCH #1 SMP PREEMPT Thu Oct 13 18:37:53 MDT 2016 aarch64 GNU/Linux

The result of the interrupts is the following:



Number 63, which is the green colored graph, shows the usb interrupts. I believe that the number is huge.

Is there any explanation for this?

Is there any way to improve it?

Many thanks,

Michael

[brad]

I am seeing the same problem, it becomes most noticeable when patching kernel with the realtime patches and enabling threaded IRQ. 2 threads exist for irq 63 (and another 2 on 62 if you run the other usb port on host mode). Depending if there are devices plugged in and how many each threaded irq process will consume anywhere between 6-80% of the cpu.

The issue seems to come from the software dwc USB host controller driver which is quiet complex and is using many sleeps in code to get timings right, I find a small scheduler stall every 4-5 secs or so and a larger one every 20-40 secs or so which hurts realtime performance. I guess the driver has to time correctly but seems like it could be done much more effectively. USB support is now in new mainline kernel and it is using newer versions of dwc host drivers and has much better scheduler support. I plan test the new drivers to see how they perform. Hopefully they minimise the number of usb interrupts & kernel stalls to prove that its possible and give some direction to optimising or updating current drivers.

I was able to reduce the irq smashing by forcing the controller to slow mode 12Mbit, but obviously not as much throughput. Microsoft wireless keyboard / mouse seems to hit harder than an audio interface being plugged in but even with nothing plugged in the driver keeps hitting irq's hard.

Im also interested playing with some setting to give the usb irq host 1 core, the OS 2 cores and my realtime process 1 to see how I go with things.

Some newer patches have recently come through hardkernel source which I am yet to try enabling an npacks option for usb audio, you could try the latest update and modify nrpacks option to see if it helps and improves your test....
viewtopic.php?f=138&t=23467
https://github.com/hardkernel/linux/com ... abc5731d88

[tuxx]

Hi Brad,

thank you very much for your feedback.

What I forgot to mention is that in my case the music reproduction using usb is fine. I was just curious why this huge number of interrupts is happening.

I am playing now with nrpacks command, but i am almost sure that it will not change the result. I will come back with some newer measurements.

EDIT:

nrpacks=1 keeps the number of interrupts the same

I also tested:

nrpacks=8 which increases the number to 14k/sec

[tuxx]

Hi all,

I would really appreciate it if I had some feedback from hardkernel with regards to USB performance and why the interrupts/sec are so high. Is it a bug? Is it a bad driver implementation?

Regards,

Michael

[odroid]

We've observed the issue a couple of year ago on S805/S905 platforms. But we don't know what causes the issue precisely.
We tried the FIQ approach like RPI platform, but we couldn't avoid the negative side effect.
viewtopic.php?f=111&t=16885

USB 2.0 high-speed storage also generates 8K IRQs, but it works well stably.
Is your USB Sound card using the Isochronous protocol? or Bulk protocol?

[crashoverride]

Number 63, which is the green colored graph, shows the usb interrupts. I believe that the number is huge.

Is there any explanation for this?

This appears to be due to the way the DWC USB controller is designed. The design is focused on cost, not performance. The DWC expects the host to handle the complexities of the USB protocol instead of using dedicated hardware. The IRQs should correspond to USB frames that the host must service in real-time. The driver is also very complex because it must handle low level USB tasks that a HCI normally would.

[tuxx]

Hi all after a lot of time.

Is there any update on the odroid usb driver implementation?

A lot of people are interested in this device as a hifi transport with a usb sound card/dac, but as it seems Odroid C2 is not yet an trouble free device for this purpose. If you google about it, you will find several people having bad experiences:

https://volumio.org/forum/usb-waveio-xm ... t5299.html
https://volumio.org/forum/odroid-paraso ... t5524.html
https://volumio.org/forum/odroid-usb-au ... t5618.html
https://forum.libreelec.tv/thread-4852.html
http://www.runeaudio.com/forum/odroid-c2-t3181-250.html

I have done several tests and even if I don't have serious issues with my device, I have pops and clicks from time to time.

It's clear that something is wrong with Odroid C2 and USB. Do you intend to find a solution to this? I really hoped to see an improvement until today but unfortunately I am really disappointed seeing zero interest on you supporting this device for audio.

Many thanks,

Michael

[brad]

Hi all after a lot of time.

Is there any update on the odroid usb driver implementation?

Yes a new driver has been rewritten for the C2 in upstream kernel and my initial testing seemed to suggest that the interrupt sleeps do not block other processes running on the cpu.

Mainline kernel is now very close hopefully (months) and many hope to see some improvements in interrupt handles for USB. Additionally some work has been done into hardware audio, video and GPU. We need to wait a little more time.

[DigiBC]

Unfortunately, it's almost impossible to use SDR (software defined radio) hardware like the USRP's plugged to the USB ports of the C2 so far. Maybe such bad results aren't surprising since SDR's demand an even higher level of stability and performance of the USB communication than audio hardware, especially in terms of data rates.

My "old" U3 perform very well with SDR's - even better than some PC systems - but I would like to move on to the C2...

So, I'm very much looking forward to those improvements of the driver and the overall interrupt handling!
However, it remains to be seen if there are hardware limitations that'll play a role in the end...

[rooted]

Unfortunately, it's almost impossible to use SDR (software defined radio) hardware like the USRP's plugged to the USB ports of the C2 so far. Maybe such bad results aren't surprising since SDR's demand an even higher level of stability and performance of the USB communication than audio hardware, especially in terms of data rates.

My "old" U3 perform very well with SDR's - even better than some PC systems - but I would like to move on to the C2...

So, I'm very much looking forward to those improvements of the driver and the overall interrupt handling!
However, it remains to be seen if there are hardware limitations that'll play a role in the end...

Have you tried using the OTG port instead?

[DigiBC]

Have you tried using the OTG port instead?

Many thanks for your tip, but using the OTG port isn't working either. In my case the results show no difference between the USB ports.

[rooted]

Have you tried using the OTG port instead?

Many thanks for your tip, but using the OTG port isn't working either. In my case the results show no difference between the USB ports.

It was worth a try. Have you tried a powered hub? Tried hub OTG?

I doubt it will help anything but it's slightly possible.

[tuxx]

It's nice to read that we are expecting changes in USB support with the mainline kernel.

I have been using ODROID C2 for almost a year now for my transport. Most of this time I had no problems even with this situation with interrupts.

I have been upadting archlinuxarm very frequently and the last three months whenever I get these errors:

Code: Select all

[  514.680349] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073bd0c00 status=-104

[ 1019.558455] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073989000 status=-104

[ 4378.269004] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b475c00 status=-104

[ 7838.097688] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b475a00 status=-104

[ 8138.052939] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073aef800 status=-104

I get pops/clicks, something that did not happen in the past. I got these errors too but I could not hear any distrortion/pop/click etc.

As a result, I decided to go back to my old trusted Udoo Quad until usb implementation is fixed.

[tuxx]

I just tested my Odroid C2/MPD combo with mainline krenrl in ArchlinuxARM:

Code: Select all

[root@alarm ~]# uname -a
Linux alarm 4.12.0-rc4-1-ARCH #1 SMP Sun Jun 4 19:13:14 MDT 2017 aarch64 GNU/Linux

I tested with various flac files over samba network. The result was terrible, full of pops and clicks, even with 16/44.1 files. I did not have the time if the problem is caused by a usb or network misbehavior however.

With regards to the interrupts, I did not measure them, but giving several cat /proc/interrupts, I was given the impression that the situation is even worse comparing to the official hardkernel kernel.

As it seems we are really far away from a functional mainline kernel for audio.

[odroid]

Thank you for sharing the valuable information about the USB host driver in the mainline kernel.
We might need to wait a few more months to see the next LTS Kernel.

[tuxx]

@Odroid,

I am really sorry to say that, but we shouldn't wait anymore! I bought my board in March of 2016. How much longer should I wait for a solution from hardkernel? It's totally clear that official support is very poor.

Regards,

Michael

[tuxx]

I just tested my Odroid C2/MPD combo with mainline krenrl in ArchlinuxARM:

Code: Select all

[root@alarm ~]# uname -a
Linux alarm 4.12.0-rc4-1-ARCH #1 SMP Sun Jun 4 19:13:14 MDT 2017 aarch64 GNU/Linux

I tested with various flac files over samba network. The result was terrible, full of pops and clicks, even with 16/44.1 files. I did not have the time if the problem is caused by a usb or network misbehavior however.

With regards to the interrupts, I did not measure them, but giving several cat /proc/interrupts, I was given the impression that the situation is even worse comparing to the official hardkernel kernel.

As it seems we are really far away from a functional mainline kernel for audio.

Back to my tests today. I wrote multiple files on a USB stick (from 16/44.1 up to DXD) and tested. As it seems all the issues I previously had were because of the network and not the USB. That does not mean that USB performance is good. In fact I did not hear in detail, but for sure the situation is much better. When serving these files from the network the pops/clicks are here all the time.

So for the moment, I have to further investigate the network performance with this kernel.

EDIT:

I downgraded to latest stable mainline:

Code: Select all

[root@archphile ~]# uname -a
Linux archphile 4.11.4-1-ARCH #1 SMP Wed Jun 7 19:51:50 MDT 2017 aarch64 GNU/Linux

The situation is the same as with the RC

[brad]

So for the moment, I have to further investigate the network performance with this kernel.

I would be interested to know if you could try the network at 100Mb instead of Gigabit to see if the pops/ clicks disappear

[rubensx]

Hello guys I'm do some tests and post here:

viewtopic.php?f=137&t=27329

May I confirm the same trouble in USB mentioned above, to solve my audio problems I'm running out USB DACs and using a SPDIF out from android and a digital to analog external converter (using the #2 pin from I2s connectors from odroid C2 "almost in heatsync" (the #2 pin is in the middle of hdmi connector location) and don't wire the ground, only the center of COAX SPDIF in to external converter.

But, my RTL-SDR FM radio still is USB and the audio only works when the SDRTouch app is in first plan, if goes to backgroud the pops and glitches appers...


Al files playng from a usb stick drive, or from a SD card .mp3, .flac runs fine and clear, and Spotify are working good... (in background too)...

Maybe this report can help the developers to find this usb bug and understand this error...

If I can help testing, copying logs, creating diagrams or something please tell me!

[tuxx]

So for the moment, I have to further investigate the network performance with this kernel.

I would be interested to know if you could try the network at 100Mb instead of Gigabit to see if the pops/ clicks disappear

I don't know if the procedure below is correct, but limiting the speed with:

Code: Select all

ethtool -s eth0 speed 100 duplex full

the result was even worse!

[brad]

I don't know if the procedure below is correct, but limiting the speed with:

Code: Select all

ethtool -s eth0 speed 100 duplex full

the result was even worse!

I have seen a few ethernet issues (at gigabit only) in various releases of mainline but I thought they were all resolved in 4.11 or rc4 (not that they caused any problems with USB but I thought it was worth a try).

Do you use pulseaudio?

[tuxx]

I just tested with the OTG cable, putting the dac on the mini usb port. I didn't hear the result, I just measured the interrupts and this solution produces a hell of interrupts too.

Using the dac in standard usb port causes approx 8k-10k interrupts/sec in 63:

Code: Select all

cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       

 63:     120960          0          0     476341       GIC  63  dwc_otg, dwc_otg_hcd:usb1

Using the dac in otg usb port causes approx 8k-10k interrupts/sec not only in 62 but in 62 too:

Code: Select all

cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       

 62:     123182          0          0     146213       GIC  62  dwc_otg, dwc_otg_hcd:usb2, dwc_otg_pcd
 63:     124253          0          0     144559       GIC  63  dwc_otg, dwc_otg_hcd:usb1


I can't see any reason that this solution is better comparing with the standard usb ports.


P.S. usb interrupts are sent to CPU3 after boot with a script.

[tuxx]

I managed to reproduce the urb errors, knowing what caused them:

Code: Select all

[ 2177.338241] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b570200 status=-104
[ 2207.107679] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b571c00 status=-104

I am able to get the above:

- when I put only one music file in MPD queue and it ends - if I manually stop mpd I can't reproduce it however
- when I put in queue two music files with different bit depth/sampling rate - the moment the dac changes it's mode the urb warn is generated.

To sum up, in my case at least when these errors are generated it's not the result of audible pops/clicks.