I am seeing the same problem, it becomes most noticeable when patching kernel with the realtime patches and enabling threaded IRQ. 2 threads exist for irq 63 (and another 2 on 62 if you run the other usb port on host mode). Depending if there are devices plugged in and how many each threaded irq process will consume anywhere between 6-80% of the cpu.
The issue seems to come from the software dwc USB host controller driver which is quiet complex and is using many sleeps in code to get timings right, I find a small scheduler stall every 4-5 secs or so and a larger one every 20-40 secs or so which hurts realtime performance. I guess the driver has to time correctly but seems like it could be done much more effectively. USB support is now in new mainline kernel and it is using newer versions of dwc host drivers and has much better scheduler support. I plan test the new drivers to see how they perform. Hopefully they minimise the number of usb interrupts & kernel stalls to prove that its possible and give some direction to optimising or updating current drivers.
I was able to reduce the irq smashing by forcing the controller to slow mode 12Mbit, but obviously not as much throughput. Microsoft wireless keyboard / mouse seems to hit harder than an audio interface being plugged in but even with nothing plugged in the driver keeps hitting irq's hard.
Im also interested playing with some setting to give the usb irq host 1 core, the OS 2 cores and my realtime process 1 to see how I go with things.
Some newer patches have recently come through hardkernel source which I am yet to try enabling an npacks option for usb audio, you could try the latest update and modify nrpacks option to see if it helps and improves your test....viewtopic.php?f=138&t=23467https://github.com/hardkernel/linux/com ... abc5731d88