USB interrupts

Moderators: mdrjr, odroid

USB interrupts

Unread postby tuxx » Sun Oct 23, 2016 2:42 am

Hi all,

I recently did some tests while preparing my next image for Archphile.

After doing some finetuning with IRQ afffinity settings and set all usb interrupts to be sent to 4th core, I did some measurements with monitorix, listening to music with my usb dac (aune s16), which is the only attached usb device.

The distribution I use is Archlinuxarm. The kernel used is:

Code: Select all
[root@archphile ~]# uname -a
Linux archphile 3.14.79-7-ARCH #1 SMP PREEMPT Thu Oct 13 18:37:53 MDT 2016 aarch64 GNU/Linux


The result of the interrupts is the following:

Image

Number 63, which is the green colored graph, shows the usb interrupts. I believe that the number is huge.

Is there any explanation for this?

Is there any way to improve it?

Many thanks,

Michael
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby brad » Sun Oct 23, 2016 9:49 am

I am seeing the same problem, it becomes most noticeable when patching kernel with the realtime patches and enabling threaded IRQ. 2 threads exist for irq 63 (and another 2 on 62 if you run the other usb port on host mode). Depending if there are devices plugged in and how many each threaded irq process will consume anywhere between 6-80% of the cpu.

The issue seems to come from the software dwc USB host controller driver which is quiet complex and is using many sleeps in code to get timings right, I find a small scheduler stall every 4-5 secs or so and a larger one every 20-40 secs or so which hurts realtime performance. I guess the driver has to time correctly but seems like it could be done much more effectively. USB support is now in new mainline kernel and it is using newer versions of dwc host drivers and has much better scheduler support. I plan test the new drivers to see how they perform. Hopefully they minimise the number of usb interrupts & kernel stalls to prove that its possible and give some direction to optimising or updating current drivers.

I was able to reduce the irq smashing by forcing the controller to slow mode 12Mbit, but obviously not as much throughput. Microsoft wireless keyboard / mouse seems to hit harder than an audio interface being plugged in but even with nothing plugged in the driver keeps hitting irq's hard.

Im also interested playing with some setting to give the usb irq host 1 core, the OS 2 cores and my realtime process 1 to see how I go with things.

Some newer patches have recently come through hardkernel source which I am yet to try enabling an npacks option for usb audio, you could try the latest update and modify nrpacks option to see if it helps and improves your test....
viewtopic.php?f=138&t=23467
https://github.com/hardkernel/linux/com ... abc5731d88
brad
 
Posts: 599
Joined: Tue Mar 29, 2016 1:22 pm
Location: Australia
languages_spoken: english
ODROIDs: C2

Re: USB interrupts

Unread postby tuxx » Sun Oct 23, 2016 3:54 pm

Hi Brad,

thank you very much for your feedback.

What I forgot to mention is that in my case the music reproduction using usb is fine. I was just curious why this huge number of interrupts is happening.

I am playing now with nrpacks command, but i am almost sure that it will not change the result. I will come back with some newer measurements.

EDIT:

nrpacks=1 keeps the number of interrupts the same

I also tested:

nrpacks=8 which increases the number to 14k/sec
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby tuxx » Mon Nov 07, 2016 5:42 pm

Hi all,

I would really appreciate it if I had some feedback from hardkernel with regards to USB performance and why the interrupts/sec are so high. Is it a bug? Is it a bad driver implementation?

Regards,

Michael
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby odroid » Mon Nov 07, 2016 10:48 pm

We've observed the issue a couple of year ago on S805/S905 platforms. But we don't know what causes the issue precisely.
We tried the FIQ approach like RPI platform, but we couldn't avoid the negative side effect.
viewtopic.php?f=111&t=16885

USB 2.0 high-speed storage also generates 8K IRQs, but it works well stably.
Is your USB Sound card using the Isochronous protocol? or Bulk protocol?
User avatar
odroid
Site Admin
 
Posts: 24590
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: USB interrupts

Unread postby crashoverride » Tue Nov 08, 2016 1:28 am

tuxx wrote:Number 63, which is the green colored graph, shows the usb interrupts. I believe that the number is huge.

Is there any explanation for this?

This appears to be due to the way the DWC USB controller is designed. The design is focused on cost, not performance. The DWC expects the host to handle the complexities of the USB protocol instead of using dedicated hardware. The IRQs should correspond to USB frames that the host must service in real-time. The driver is also very complex because it must handle low level USB tasks that a HCI normally would.
crashoverride
 
Posts: 3077
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: USB interrupts

Unread postby tuxx » Fri Apr 07, 2017 3:16 am

Hi all after a lot of time.

Is there any update on the odroid usb driver implementation?

A lot of people are interested in this device as a hifi transport with a usb sound card/dac, but as it seems Odroid C2 is not yet an trouble free device for this purpose. If you google about it, you will find several people having bad experiences:

https://volumio.org/forum/usb-waveio-xm ... t5299.html
https://volumio.org/forum/odroid-paraso ... t5524.html
https://volumio.org/forum/odroid-usb-au ... t5618.html
https://forum.libreelec.tv/thread-4852.html
http://www.runeaudio.com/forum/odroid-c2-t3181-250.html

I have done several tests and even if I don't have serious issues with my device, I have pops and clicks from time to time.

It's clear that something is wrong with Odroid C2 and USB. Do you intend to find a solution to this? I really hoped to see an improvement until today but unfortunately I am really disappointed seeing zero interest on you supporting this device for audio.

Many thanks,

Michael
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby brad » Sun Apr 09, 2017 4:44 pm

tuxx wrote:Hi all after a lot of time.

Is there any update on the odroid usb driver implementation?



Yes a new driver has been rewritten for the C2 in upstream kernel and my initial testing seemed to suggest that the interrupt sleeps do not block other processes running on the cpu.

Mainline kernel is now very close hopefully (months) and many hope to see some improvements in interrupt handles for USB. Additionally some work has been done into hardware audio, video and GPU. We need to wait a little more time.
brad
 
Posts: 599
Joined: Tue Mar 29, 2016 1:22 pm
Location: Australia
languages_spoken: english
ODROIDs: C2

Re: USB interrupts

Unread postby DigiBC » Thu Apr 20, 2017 5:37 am

Unfortunately, it's almost impossible to use SDR (software defined radio) hardware like the USRP's plugged to the USB ports of the C2 so far. Maybe such bad results aren't surprising since SDR's demand an even higher level of stability and performance of the USB communication than audio hardware, especially in terms of data rates.

My "old" U3 perform very well with SDR's - even better than some PC systems - but I would like to move on to the C2...

So, I'm very much looking forward to those improvements of the driver and the overall interrupt handling!
However, it remains to be seen if there are hardware limitations that'll play a role in the end...
DigiBC
 
Posts: 8
Joined: Thu Aug 07, 2014 2:48 am
languages_spoken: english
ODROIDs: 2 x U3, 2 x C2

Re: USB interrupts

Unread postby rooted » Fri Apr 21, 2017 6:02 am

DigiBC wrote:Unfortunately, it's almost impossible to use SDR (software defined radio) hardware like the USRP's plugged to the USB ports of the C2 so far. Maybe such bad results aren't surprising since SDR's demand an even higher level of stability and performance of the USB communication than audio hardware, especially in terms of data rates.

My "old" U3 perform very well with SDR's - even better than some PC systems - but I would like to move on to the C2...

So, I'm very much looking forward to those improvements of the driver and the overall interrupt handling!
However, it remains to be seen if there are hardware limitations that'll play a role in the end...

Have you tried using the OTG port instead?
User avatar
rooted
 
Posts: 3653
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1
C1+
C2
XU3 Lite
XU4
VU7+
HiFi Shield 2
Smart Power (original)

Re: USB interrupts

Unread postby DigiBC » Sat Apr 22, 2017 2:35 am

rooted wrote:Have you tried using the OTG port instead?

Many thanks for your tip, but using the OTG port isn't working either. In my case the results show no difference between the USB ports.
DigiBC
 
Posts: 8
Joined: Thu Aug 07, 2014 2:48 am
languages_spoken: english
ODROIDs: 2 x U3, 2 x C2

Re: USB interrupts

Unread postby rooted » Sat Apr 22, 2017 6:44 am

DigiBC wrote:
rooted wrote:Have you tried using the OTG port instead?

Many thanks for your tip, but using the OTG port isn't working either. In my case the results show no difference between the USB ports.

It was worth a try. Have you tried a powered hub? Tried hub OTG?

I doubt it will help anything but it's slightly possible.
User avatar
rooted
 
Posts: 3653
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1
C1+
C2
XU3 Lite
XU4
VU7+
HiFi Shield 2
Smart Power (original)

Re: USB interrupts

Unread postby tuxx » Sun Apr 23, 2017 7:26 pm

It's nice to read that we are expecting changes in USB support with the mainline kernel.

I have been using ODROID C2 for almost a year now for my transport. Most of this time I had no problems even with this situation with interrupts.

I have been upadting archlinuxarm very frequently and the last three months whenever I get these errors:
Code: Select all
[  514.680349] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073bd0c00 status=-104

[ 1019.558455] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073989000 status=-104

[ 4378.269004] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b475c00 status=-104

[ 7838.097688] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b475a00 status=-104

[ 8138.052939] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073aef800 status=-104


I get pops/clicks, something that did not happen in the past. I got these errors too but I could not hear any distrortion/pop/click etc.

As a result, I decided to go back to my old trusted Udoo Quad until usb implementation is fixed.
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby tuxx » Wed Jun 07, 2017 9:44 pm

I just tested my Odroid C2/MPD combo with mainline krenrl in ArchlinuxARM:

Code: Select all
[root@alarm ~]# uname -a
Linux alarm 4.12.0-rc4-1-ARCH #1 SMP Sun Jun 4 19:13:14 MDT 2017 aarch64 GNU/Linux


I tested with various flac files over samba network. The result was terrible, full of pops and clicks, even with 16/44.1 files. I did not have the time if the problem is caused by a usb or network misbehavior however.

With regards to the interrupts, I did not measure them, but giving several cat /proc/interrupts, I was given the impression that the situation is even worse comparing to the official hardkernel kernel.

As it seems we are really far away from a functional mainline kernel for audio.
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby odroid » Thu Jun 08, 2017 8:59 am

Thank you for sharing the valuable information about the USB host driver in the mainline kernel.
We might need to wait a few more months to see the next LTS Kernel.
User avatar
odroid
Site Admin
 
Posts: 24590
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: USB interrupts

Unread postby tuxx » Thu Jun 08, 2017 4:23 pm

@Odroid,

I am really sorry to say that, but we shouldn't wait anymore! I bought my board in March of 2016. How much longer should I wait for a solution from hardkernel? It's totally clear that official support is very poor.

Regards,

Michael
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby tuxx » Thu Jun 08, 2017 10:18 pm

tuxx wrote:I just tested my Odroid C2/MPD combo with mainline krenrl in ArchlinuxARM:

Code: Select all
[root@alarm ~]# uname -a
Linux alarm 4.12.0-rc4-1-ARCH #1 SMP Sun Jun 4 19:13:14 MDT 2017 aarch64 GNU/Linux


I tested with various flac files over samba network. The result was terrible, full of pops and clicks, even with 16/44.1 files. I did not have the time if the problem is caused by a usb or network misbehavior however.

With regards to the interrupts, I did not measure them, but giving several cat /proc/interrupts, I was given the impression that the situation is even worse comparing to the official hardkernel kernel.

As it seems we are really far away from a functional mainline kernel for audio.


Back to my tests today. I wrote multiple files on a USB stick (from 16/44.1 up to DXD) and tested. As it seems all the issues I previously had were because of the network and not the USB. That does not mean that USB performance is good. In fact I did not hear in detail, but for sure the situation is much better. When serving these files from the network the pops/clicks are here all the time.

So for the moment, I have to further investigate the network performance with this kernel.

EDIT:

I downgraded to latest stable mainline:
Code: Select all
[root@archphile ~]# uname -a
Linux archphile 4.11.4-1-ARCH #1 SMP Wed Jun 7 19:51:50 MDT 2017 aarch64 GNU/Linux


The situation is the same as with the RC
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby brad » Sat Jun 10, 2017 9:15 am

tuxx wrote:So for the moment, I have to further investigate the network performance with this kernel.


I would be interested to know if you could try the network at 100Mb instead of Gigabit to see if the pops/ clicks disappear
brad
 
Posts: 599
Joined: Tue Mar 29, 2016 1:22 pm
Location: Australia
languages_spoken: english
ODROIDs: C2

Re: USB interrupts

Unread postby rubensx » Tue Jun 13, 2017 10:00 am

Hello guys I'm do some tests and post here:

viewtopic.php?f=137&t=27329

May I confirm the same trouble in USB mentioned above, to solve my audio problems I'm running out USB DACs and using a SPDIF out from android and a digital to analog external converter (using the #2 pin from I2s connectors from odroid C2 "almost in heatsync" (the #2 pin is in the middle of hdmi connector location) and don't wire the ground, only the center of COAX SPDIF in to external converter.

But, my RTL-SDR FM radio still is USB and the audio only works when the SDRTouch app is in first plan, if goes to backgroud the pops and glitches appers...


Al files playng from a usb stick drive, or from a SD card .mp3, .flac runs fine and clear, and Spotify are working good... (in background too)...

Maybe this report can help the developers to find this usb bug and understand this error...

If I can help testing, copying logs, creating diagrams or something please tell me!
rubensx
 
Posts: 52
Joined: Wed Aug 17, 2016 9:53 am
Location: Brazil, São Paulo
languages_spoken: english, portuguese
ODROIDs: C2

Re: USB interrupts

Unread postby tuxx » Tue Jun 13, 2017 8:28 pm

brad wrote:
tuxx wrote:So for the moment, I have to further investigate the network performance with this kernel.


I would be interested to know if you could try the network at 100Mb instead of Gigabit to see if the pops/ clicks disappear


I don't know if the procedure below is correct, but limiting the speed with:

Code: Select all
ethtool -s eth0 speed 100 duplex full


the result was even worse!
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby brad » Wed Jun 14, 2017 5:29 pm

tuxx wrote:
I don't know if the procedure below is correct, but limiting the speed with:

Code: Select all
ethtool -s eth0 speed 100 duplex full


the result was even worse!


I have seen a few ethernet issues (at gigabit only) in various releases of mainline but I thought they were all resolved in 4.11 or rc4 (not that they caused any problems with USB but I thought it was worth a try).

Do you use pulseaudio?
brad
 
Posts: 599
Joined: Tue Mar 29, 2016 1:22 pm
Location: Australia
languages_spoken: english
ODROIDs: C2

Re: USB interrupts

Unread postby tuxx » Sun Jul 16, 2017 9:01 pm

I just tested with the OTG cable, putting the dac on the mini usb port. I didn't hear the result, I just measured the interrupts and this solution produces a hell of interrupts too.

Using the dac in standard usb port causes approx 8k-10k interrupts/sec in 63:

Code: Select all
cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       

 63:     120960          0          0     476341       GIC  63  dwc_otg, dwc_otg_hcd:usb1



Using the dac in otg usb port causes approx 8k-10k interrupts/sec not only in 62 but in 62 too:

Code: Select all
cat /proc/interrupts
           CPU0       CPU1       CPU2       CPU3       

 62:     123182          0          0     146213       GIC  62  dwc_otg, dwc_otg_hcd:usb2, dwc_otg_pcd
 63:     124253          0          0     144559       GIC  63  dwc_otg, dwc_otg_hcd:usb1


I can't see any reason that this solution is better comparing with the standard usb ports.


P.S. usb interrupts are sent to CPU3 after boot with a script.
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby tuxx » Mon Jul 17, 2017 3:43 pm

I managed to reproduce the urb errors, knowing what caused them:

Code: Select all
[ 2177.338241] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b570200 status=-104
[ 2207.107679] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b571c00 status=-104


I am able to get the above:

- when I put only one music file in MPD queue and it ends - if I manually stop mpd I can't reproduce it however
- when I put in queue two music files with different bit depth/sampling rate - the moment the dac changes it's mode the urb warn is generated.

To sum up, in my case at least when these errors are generated it's not the result of audible pops/clicks.
User avatar
tuxx
 
Posts: 38
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2


Return to Issues

Who is online

Users browsing this forum: No registered users and 2 guests