USB interrupts

Moderators: odroid, mdrjr

USB interrupts

Unread postby tuxx » Sun Oct 23, 2016 2:42 am

Hi all,

I recently did some tests while preparing my next image for Archphile.

After doing some finetuning with IRQ afffinity settings and set all usb interrupts to be sent to 4th core, I did some measurements with monitorix, listening to music with my usb dac (aune s16), which is the only attached usb device.

The distribution I use is Archlinuxarm. The kernel used is:

Code: Select all
[root@archphile ~]# uname -a
Linux archphile 3.14.79-7-ARCH #1 SMP PREEMPT Thu Oct 13 18:37:53 MDT 2016 aarch64 GNU/Linux


The result of the interrupts is the following:

Image

Number 63, which is the green colored graph, shows the usb interrupts. I believe that the number is huge.

Is there any explanation for this?

Is there any way to improve it?

Many thanks,

Michael
User avatar
tuxx
 
Posts: 27
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby brad » Sun Oct 23, 2016 9:49 am

I am seeing the same problem, it becomes most noticeable when patching kernel with the realtime patches and enabling threaded IRQ. 2 threads exist for irq 63 (and another 2 on 62 if you run the other usb port on host mode). Depending if there are devices plugged in and how many each threaded irq process will consume anywhere between 6-80% of the cpu.

The issue seems to come from the software dwc USB host controller driver which is quiet complex and is using many sleeps in code to get timings right, I find a small scheduler stall every 4-5 secs or so and a larger one every 20-40 secs or so which hurts realtime performance. I guess the driver has to time correctly but seems like it could be done much more effectively. USB support is now in new mainline kernel and it is using newer versions of dwc host drivers and has much better scheduler support. I plan test the new drivers to see how they perform. Hopefully they minimise the number of usb interrupts & kernel stalls to prove that its possible and give some direction to optimising or updating current drivers.

I was able to reduce the irq smashing by forcing the controller to slow mode 12Mbit, but obviously not as much throughput. Microsoft wireless keyboard / mouse seems to hit harder than an audio interface being plugged in but even with nothing plugged in the driver keeps hitting irq's hard.

Im also interested playing with some setting to give the usb irq host 1 core, the OS 2 cores and my realtime process 1 to see how I go with things.

Some newer patches have recently come through hardkernel source which I am yet to try enabling an npacks option for usb audio, you could try the latest update and modify nrpacks option to see if it helps and improves your test....
viewtopic.php?f=138&t=23467
https://github.com/hardkernel/linux/com ... abc5731d88
brad
 
Posts: 398
Joined: Tue Mar 29, 2016 1:22 pm
Location: Australia
languages_spoken: english
ODROIDs: C2

Re: USB interrupts

Unread postby tuxx » Sun Oct 23, 2016 3:54 pm

Hi Brad,

thank you very much for your feedback.

What I forgot to mention is that in my case the music reproduction using usb is fine. I was just curious why this huge number of interrupts is happening.

I am playing now with nrpacks command, but i am almost sure that it will not change the result. I will come back with some newer measurements.

EDIT:

nrpacks=1 keeps the number of interrupts the same

I also tested:

nrpacks=8 which increases the number to 14k/sec
User avatar
tuxx
 
Posts: 27
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby tuxx » Mon Nov 07, 2016 5:42 pm

Hi all,

I would really appreciate it if I had some feedback from hardkernel with regards to USB performance and why the interrupts/sec are so high. Is it a bug? Is it a bad driver implementation?

Regards,

Michael
User avatar
tuxx
 
Posts: 27
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby odroid » Mon Nov 07, 2016 10:48 pm

We've observed the issue a couple of year ago on S805/S905 platforms. But we don't know what causes the issue precisely.
We tried the FIQ approach like RPI platform, but we couldn't avoid the negative side effect.
viewtopic.php?f=111&t=16885

USB 2.0 high-speed storage also generates 8K IRQs, but it works well stably.
Is your USB Sound card using the Isochronous protocol? or Bulk protocol?
User avatar
odroid
Site Admin
 
Posts: 22295
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: USB interrupts

Unread postby crashoverride » Tue Nov 08, 2016 1:28 am

tuxx wrote:Number 63, which is the green colored graph, shows the usb interrupts. I believe that the number is huge.

Is there any explanation for this?

This appears to be due to the way the DWC USB controller is designed. The design is focused on cost, not performance. The DWC expects the host to handle the complexities of the USB protocol instead of using dedicated hardware. The IRQs should correspond to USB frames that the host must service in real-time. The driver is also very complex because it must handle low level USB tasks that a HCI normally would.
crashoverride
 
Posts: 2565
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: USB interrupts

Unread postby tuxx » Fri Apr 07, 2017 3:16 am

Hi all after a lot of time.

Is there any update on the odroid usb driver implementation?

A lot of people are interested in this device as a hifi transport with a usb sound card/dac, but as it seems Odroid C2 is not yet an trouble free device for this purpose. If you google about it, you will find several people having bad experiences:

https://volumio.org/forum/usb-waveio-xm ... t5299.html
https://volumio.org/forum/odroid-paraso ... t5524.html
https://volumio.org/forum/odroid-usb-au ... t5618.html
https://forum.libreelec.tv/thread-4852.html
http://www.runeaudio.com/forum/odroid-c2-t3181-250.html

I have done several tests and even if I don't have serious issues with my device, I have pops and clicks from time to time.

It's clear that something is wrong with Odroid C2 and USB. Do you intend to find a solution to this? I really hoped to see an improvement until today but unfortunately I am really disappointed seeing zero interest on you supporting this device for audio.

Many thanks,

Michael
User avatar
tuxx
 
Posts: 27
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2

Re: USB interrupts

Unread postby brad » Sun Apr 09, 2017 4:44 pm

tuxx wrote:Hi all after a lot of time.

Is there any update on the odroid usb driver implementation?



Yes a new driver has been rewritten for the C2 in upstream kernel and my initial testing seemed to suggest that the interrupt sleeps do not block other processes running on the cpu.

Mainline kernel is now very close hopefully (months) and many hope to see some improvements in interrupt handles for USB. Additionally some work has been done into hardware audio, video and GPU. We need to wait a little more time.
brad
 
Posts: 398
Joined: Tue Mar 29, 2016 1:22 pm
Location: Australia
languages_spoken: english
ODROIDs: C2

Re: USB interrupts

Unread postby DigiBC » Thu Apr 20, 2017 5:37 am

Unfortunately, it's almost impossible to use SDR (software defined radio) hardware like the USRP's plugged to the USB ports of the C2 so far. Maybe such bad results aren't surprising since SDR's demand an even higher level of stability and performance of the USB communication than audio hardware, especially in terms of data rates.

My "old" U3 perform very well with SDR's - even better than some PC systems - but I would like to move on to the C2...

So, I'm very much looking forward to those improvements of the driver and the overall interrupt handling!
However, it remains to be seen if there are hardware limitations that'll play a role in the end...
DigiBC
 
Posts: 4
Joined: Thu Aug 07, 2014 2:48 am
languages_spoken: english
ODROIDs: 2 x U3, 2 x C2

Re: USB interrupts

Unread postby rooted » Fri Apr 21, 2017 6:02 am

DigiBC wrote:Unfortunately, it's almost impossible to use SDR (software defined radio) hardware like the USRP's plugged to the USB ports of the C2 so far. Maybe such bad results aren't surprising since SDR's demand an even higher level of stability and performance of the USB communication than audio hardware, especially in terms of data rates.

My "old" U3 perform very well with SDR's - even better than some PC systems - but I would like to move on to the C2...

So, I'm very much looking forward to those improvements of the driver and the overall interrupt handling!
However, it remains to be seen if there are hardware limitations that'll play a role in the end...

Have you tried using the OTG port instead?
User avatar
rooted
 
Posts: 3500
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1
C1+
C2
XU3 Lite
XU4
VU7+
HiFi Shield 2
Smart Power (original)

Re: USB interrupts

Unread postby DigiBC » Sat Apr 22, 2017 2:35 am

rooted wrote:Have you tried using the OTG port instead?

Many thanks for your tip, but using the OTG port isn't working either. In my case the results show no difference between the USB ports.
DigiBC
 
Posts: 4
Joined: Thu Aug 07, 2014 2:48 am
languages_spoken: english
ODROIDs: 2 x U3, 2 x C2

Re: USB interrupts

Unread postby rooted » Sat Apr 22, 2017 6:44 am

DigiBC wrote:
rooted wrote:Have you tried using the OTG port instead?

Many thanks for your tip, but using the OTG port isn't working either. In my case the results show no difference between the USB ports.

It was worth a try. Have you tried a powered hub? Tried hub OTG?

I doubt it will help anything but it's slightly possible.
User avatar
rooted
 
Posts: 3500
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1
C1+
C2
XU3 Lite
XU4
VU7+
HiFi Shield 2
Smart Power (original)

Re: USB interrupts

Unread postby tuxx » Sun Apr 23, 2017 7:26 pm

It's nice to read that we are expecting changes in USB support with the mainline kernel.

I have been using ODROID C2 for almost a year now for my transport. Most of this time I had no problems even with this situation with interrupts.

I have been upadting archlinuxarm very frequently and the last three months whenever I get these errors:
Code: Select all
[  514.680349] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073bd0c00 status=-104

[ 1019.558455] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073989000 status=-104

[ 4378.269004] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b475c00 status=-104

[ 7838.097688] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc07b475a00 status=-104

[ 8138.052939] WARN::urb_dequeue:780: urb->hcpriv == NULL! urb = ffffffc073aef800 status=-104


I get pops/clicks, something that did not happen in the past. I got these errors too but I could not hear any distrortion/pop/click etc.

As a result, I decided to go back to my old trusted Udoo Quad until usb implementation is fixed.
User avatar
tuxx
 
Posts: 27
Joined: Fri Mar 11, 2016 6:42 am
languages_spoken: english
ODROIDs: C1+, C2


Return to Issues

Who is online

Users browsing this forum: No registered users and 4 guests