[Fixed]: full 1gb NIC speed driver now in kernel

Moderators: odroid, mdrjr

[Fixed]: full 1gb NIC speed driver now in kernel

Unread postby mlinuxguy » Tue Jan 06, 2015 9:13 am

UPDATE: The nic issues have been fixed in 3.10.65-44, and my network driver merged in as the default nic driver

Wanting to setup NFS shares to my new Odroid-CI cluster I ran some NIC benchmark tests
Note: all tests were done on a 24-port 1-gb switch with each box connected in my office to that switch

Summary:
Code: Select all
Odroid   to Core i7 server:  26.3 MBytes/sec
Core i7 to Odroid:              0.03 MBytes/sec
RPI to  Core i7 server:        7.32 MBytes/sec
Core i7 to RPI:                  11.2 MBytes/sec

Is there anything that can be done to improve this performance? Apparently I cannot set MTU size to 9k for Jumbo Frames as that gives errors....

The full testing is below:
Code: Select all
TCP tests,  server i7-2600 CPU @ 3.40GHz (1Gb NIC)
# iperf -s -fM
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.118 port 5001 connected with 192.168.1.130 port 58258
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   263 MBytes  26.3 MBytes/sec

# iperf -fM -c 192.168.1.118
Client connecting to 192.168.1.118, TCP port 5001
TCP window size: 0.02 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.130 port 58258 connected with 192.168.1.118 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   263 MBytes  26.3 MBytes/sec

====== TCP switch directions with Odroid-CI as server, same Intel Linux box as client
# iperf -s -fM
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.130 port 5001 connected with 192.168.1.118 port 60745
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-19.5 sec  0.62 MBytes  0.03 MBytes/sec

# iperf -fM -c 192.168.1.130
------------------------------------------------------------
Client connecting to 192.168.1.130, TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.118 port 60745 connected with 192.168.1.130 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-18.0 sec  0.62 MBytes  0.03 MBytes/sec

====== UDP tests:

Odroid as Server:
root@odroidci-1:~# iperf -s -fM --udp
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 0.16 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.130 port 5001 connected with 192.168.1.118 port 58320
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0-10.3 sec  0.88 MBytes  0.09 MBytes/sec  15.659 ms  265/  893 (30%)

Odroid as Client:
root@odroidci-1:~# iperf --udp -fM -c 192.168.1.118
[  3]  0.0-10.0 sec  1.25 MBytes  0.13 MBytes/sec   0.036 ms    0/  893 (0%)

============= RPI tests TCP:
RPI as client:
[root@rpicam ~]# iperf -fM -c 192.168.1.118
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  73.5 MBytes  7.32 MBytes/sec

RPI as server:
[root@rpicam ~]# iperf -s -fM
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.23 port 5001 connected with 192.168.1.118 port 42152
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.2 sec   114 MBytes  11.2 MBytes/sec

Last edited by mlinuxguy on Tue Feb 03, 2015 5:17 am, edited 6 times in total.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: NIC speed tests using IPERF and compared to RPI

Unread postby mdrjr » Tue Jan 06, 2015 12:49 pm

root@odroid:~# iperf -c 192.168.2.1
------------------------------------------------------------
Client connecting to 192.168.2.1, TCP port 5001
TCP window size: 20.7 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.2.130 port 36255 connected with 192.168.2.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 272 MBytes 228 Mbits/sec
root@odroid:~#



And no, there's no Jumbo Frame support.
mdrjr
Site Admin
 
Posts: 11552
Joined: Fri Feb 22, 2013 11:34 pm
Location: Brazil
languages_spoken: english, portuguese
ODROIDs: -

Re: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Tue Jan 06, 2015 1:02 pm

So you can get 228 MBits/sec while I crawl along with 26.3 MBytes/sec at best

My UDP tests showed lots of lost datagrams:
[ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams
[ 3] 0.0-10.3 sec 0.88 MBytes 0.09 MBytes/sec 15.659 ms 265/ 893 (30%)

So I may be hitting that switch issue several others have posted. This is a brand new Rosewell 24-port switch and has no issues under RPI, Linux or Windows.
I'm pulling the source down to look at your network driver...
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: NIC speed tests using IPERF and compared to RPI

Unread postby mdrjr » Tue Jan 06, 2015 1:13 pm

228/8 = 28,5Mbyte/s
While you got 26.3Mbyte/s
mdrjr
Site Admin
 
Posts: 11552
Joined: Fri Feb 22, 2013 11:34 pm
Location: Brazil
languages_spoken: english, portuguese
ODROIDs: -

Re: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Tue Jan 06, 2015 1:24 pm

ah sorry, however that does not explain the speed I get the other direction, Core I7 linux ---> to ---> Odroid CI 0.03 MBytes/sec
seems a tad lower :p
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Wed Jan 07, 2015 2:41 am

I increased the RX and TX ring buffer sizes in the amlogic network driver, rebuilt it all, and tested using IPERF again
With Odroid-CI as client I got the same 26 MBytes/sec transfer

However with Odroid-CI as Server I get:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 394 MBytes 39.3 MBytes/sec

This is up from 0.03 MBytes/sec under the default driver

I will dig into the amlogic network driver some more to see where I can improve it further. I could have the issue listed in the other thread with 1gb negotiation on my switch. Perhaps I could force my driver to 100mb and retest.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mdrjr » Wed Jan 07, 2015 2:54 am

Amazing :)

Great work :)

At 100Mbit the "PHY RESET" issue doesn't happen.
Only at 1000Mbit and only with certain switches :(
mdrjr
Site Admin
 
Posts: 11552
Joined: Fri Feb 22, 2013 11:34 pm
Location: Brazil
languages_spoken: english, portuguese
ODROIDs: -

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Wed Jan 07, 2015 2:59 am

My switch is a gigabit switch so it might be worth me checking his fix to 100mb out.

Note: I set the ring buffers for both to 1024 (up from 128).
There are some other area's where I think this driver is setup for Android and not Linux server style, but I will need to investigate it further. Note I have rebuilt it as a module while I test so I cannot supply someone a uImage without all the driver modules too.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Now 41 Mbytes/sec as server, 28.9 Mbytes as client

Unread postby mlinuxguy » Wed Jan 07, 2015 4:09 am

New tests with ring buffer large enough to hold a packet: 2048
Code: Select all
odroid-CI as server:
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   412 MBytes  41.2 MBytes/sec

odroid-CI as client:
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   289 MBytes  28.9 MBytes/sec


I will dig in further for other optimizations
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: NIC speed tests using IPERF and compared to RPI

Unread postby mikronauts » Wed Jan 07, 2015 4:53 am

Nice work!

mlinuxguy wrote:I increased the RX and TX ring buffer sizes in the amlogic network driver, rebuilt it all, and tested using IPERF again
With Odroid-CI as client I got the same 26 MBytes/sec transfer

However with Odroid-CI as Server I get:
[ ID] Interval Transfer Bandwidth
[ 3] 0.0-10.0 sec 394 MBytes 39.3 MBytes/sec

This is up from 0.03 MBytes/sec under the default driver

I will dig into the amlogic network driver some more to see where I can improve it further. I could have the issue listed in the other thread with 1gb negotiation on my switch. Perhaps I could force my driver to 100mb and retest.
http://Mikronauts.com ... Home of RoboPi, Pi Rtc Dio, Pi Jumper, EZasPi
mikronauts
 
Posts: 225
Joined: Wed Aug 27, 2014 4:28 am
Location: Langley, BC, Canada
languages_spoken: english
ODROIDs: odroid-w, odroid c1

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Wed Jan 07, 2015 10:02 am

New data after more tuning, I think I can get it even higher now that I know more about what their code looks like

Code: Select all
Odroid as Client:
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   332 MBytes  33.1 MBytes/sec

Odroid as Server:
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   474 MBytes  47.3 MBytes/sec
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby onefst250r » Wed Jan 07, 2015 12:22 pm

Would be pretty epic if you could create/post a tutorial on how to make these modifications after you get them tuned.
onefst250r
 
Posts: 2
Joined: Sat Dec 20, 2014 2:45 pm
languages_spoken: english

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Wed Jan 07, 2015 1:17 pm

I'm keeping original code so I can create a diff patch after I'm done
The rx and tx ring buffers gave some performance increase, but now I'm modifying other area's. It's taking a while to understand their driver code and what I can safely modify... not a lot of comments in the code :D
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Wed Jan 07, 2015 5:58 pm

When the Odroid-CI is setup as IPERF server on either of my 1gb switches (one Rosewell 24-port the other a Netgear 8 port) I get random network performance when I reboot.
Odroid as Server means it is receiving data from the client box

At times I can get speeds as high as 50 MB/sec and on other most other reboots I get 0.02 MB/sec.

To confirm the switch is causing the problems I verified I was booted in a 0.02 MB/sec mode (Odroid Server in IPERF), moved the lan cable over to a 100M switch, and re-ran the test. I got 40 MB/sec

I can get this speed on any reboot while connected to the 100M switch. So I'm halting optimizing the network driver until the 1GB switch issue gets fixed.

I tried a few tcpdump captures of the slow mode, but I don't think the odroid is the right host to capture the tcpdump on... it was a mess. I'll try again later with the Intel linux box running the tcpdump and see how Client vs Server mode on the odroid look (retransmissions, etc..)
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mikronauts » Wed Jan 07, 2015 11:57 pm

Umm.. do you mean 40mbit/sec?

I don't see how you can get 40MB/sec on a 100Mbit/sec link.

mlinuxguy wrote:When the Odroid-CI is setup as IPERF server on either of my 1gb switches (one Rosewell 24-port the other a Netgear 8 port) I get random network performance when I reboot.
Odroid as Server means it is receiving data from the client box

At times I can get speeds as high as 50 MB/sec and on other most other reboots I get 0.02 MB/sec.

To confirm the switch is causing the problems I verified I was booted in a 0.02 MB/sec mode (Odroid Server in IPERF), moved the lan cable over to a 100M switch, and re-ran the test. I got 40 MB/sec

I can get this speed on any reboot while connected to the 100M switch. So I'm halting optimizing the network driver until the 1GB switch issue gets fixed.

I tried a few tcpdump captures of the slow mode, but I don't think the odroid is the right host to capture the tcpdump on... it was a mess. I'll try again later with the Intel linux box running the tcpdump and see how Client vs Server mode on the odroid look (retransmissions, etc..)
http://Mikronauts.com ... Home of RoboPi, Pi Rtc Dio, Pi Jumper, EZasPi
mikronauts
 
Posts: 225
Joined: Wed Aug 27, 2014 4:28 am
Location: Langley, BC, Canada
languages_spoken: english
ODROIDs: odroid-w, odroid c1

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Thu Jan 08, 2015 1:01 am

Interesting, I run all my tests with the IPERF options:
server: # iperf -fM -s
client: # iperf -fM -c target_IP

-fM reports it in megabytes
-f, --format
[kmKM] format to report: Kbits, Mbits, KBytes, MBytes

Intel Linux client: still on 1G switch
Odroid: 100M switch
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mikronauts » Thu Jan 08, 2015 1:41 am

Very - perhaps your 100mbps switch is actually a gigabit switch?

40MB/sec needs ~400Mbits/sec+, so literally impossible over a 100Mbps link.

mlinuxguy wrote:Interesting, I run all my tests with the IPERF options:
server: # iperf -fM -s
client: # iperf -fM -c target_IP

-fM reports it in megabytes
-f, --format
[kmKM] format to report: Kbits, Mbits, KBytes, MBytes

Intel Linux client: still on 1G switch
Odroid: 100M switch
http://Mikronauts.com ... Home of RoboPi, Pi Rtc Dio, Pi Jumper, EZasPi
mikronauts
 
Posts: 225
Joined: Wed Aug 27, 2014 4:28 am
Location: Langley, BC, Canada
languages_spoken: english
ODROIDs: odroid-w, odroid c1

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Thu Jan 08, 2015 2:33 am

heh, you are totally correct. Its a white-box switch I picked up on ebay and in fine print on the bottom its listed 10/100/1000.
So out of 3 1GB switches I have it does not show the issue

Netgear 8-port shows issue (GS108)
Rosewill 24-port switch shows issue, as does a friends 16-port Rosewill (RGS-1016 and RGS-1024)
White-box 8-port switch does not show issue
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mikronauts » Thu Jan 08, 2015 3:51 am

TrendNet TEG-S80g gigabit switch also works fine :)

mlinuxguy wrote:heh, you are totally correct. Its a white-box switch I picked up on ebay and in fine print on the bottom its listed 10/100/1000.
So out of 3 1GB switches I have it does not show the issue

Netgear 8-port shows issue (GS108)
Rosewill 24-port switch shows issue, as does a friends 16-port Rosewill (RGS-1016 and RGS-1024)
White-box 8-port switch does not show issue
http://Mikronauts.com ... Home of RoboPi, Pi Rtc Dio, Pi Jumper, EZasPi
mikronauts
 
Posts: 225
Joined: Wed Aug 27, 2014 4:28 am
Location: Langley, BC, Canada
languages_spoken: english
ODROIDs: odroid-w, odroid c1

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Thu Jan 08, 2015 4:46 am

On the problem switches this is what I see with odroid-CI as IPERF server:
tcpdump shows Len=7240 (>MTU) meaning GSO/TSO is in use by client
I disable GSO/TSO on Intel client and still see dup acks
I'm still losing packets somewhere but 'ifconfig' on both client/server are clean, no dropped packets reported

I went back to the driver and under the assumption the driver cannot keep up increased many RX related parms, rebooted and tested again:
Code: Select all
root@odroidci-1:~# iperf -fM -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.130 port 5001 connected with 192.168.1.118 port 38033
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   514 MBytes  51.3 MBytes/sec

But I'm not confident this is a stable configuration... CPU0 is handling all the IRQ's for the network on the odroid and I appear to have some bash process that is also consuming CPU0:
Code: Select all
  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
  948 root      20   0    6680   3376   1284 R  99.8  0.4   9:41.88 bash

This process reappears every time I kill it (kill -9 948) and appears to be doing:
Code: Select all
root@odroidci-1:~# strace -p 948
Process 948 attached
write(2, "\377", 1)                     = 1
read(0, "\377", 1)                      = 1
write(2, "\377", 1)                     = 1
read(0, "\377", 1)                      = 1
write(2, "\377", 1)                     = 1
read(0, "\377", 1)                      = 1
write(2, "\377", 1)                     = 1
read(0, "\377", 1)                      = 1


against /dev/ttys0
Is that due to me not having a console debug cable connected?
if so that is poor design somewhere

Update: I killed the bash getty spawn by commenting out:
# exec /bin/auto-serial-console
in file: /etc/init/auto-serial-console.conf
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Fri Jan 09, 2015 9:15 am

Further analysis of tcdump data while Odroid-CI was in server mode
1 - Packets are sent from Intel Linux I7 (or another Odroid-CI) to the Odroid server
2 - Not all of these packets are received and show up on the Odroid server tcpdump
3 - The 1gb switches are not losing data as I can achieve near line speed on other machines


Notes: tcpdump on the Odroid server will only show packets that make it thru the network stack on the Odroid.
ifconfig shows NO dropped packets nor does netstat, the odroid logs nothing lost, though tcpdump shows it never processed those packets

This means that the Odroid-CI has either:

1 - a buggy network driver
or
2 - a non-optimally coded network driver

My vote is for #2, with TX and RX queue being shared by the same tasklet, on the same processor, under the same IRQ
I see no checking for TASKLET states such as already executing, or scheduled. The way the tasklet is coded it does the TX queue first, then RX queue... all in a single tasklet.

Buggy network driver is still possible as I cannot explain why on some reboots, for a while, I can surpass 50 MB/sec on receive with the Odroid-CI. But it takes many reboots (or reinit of the driver) to get the high speed and then it will later drop down to 0.02 MB/sec on its own.

The missing packets are best explained by the network driver dropping packets, perhaps with IRQ disabled too long.

Additional note: I have 4 Odroid-CI all with the latest kernel, short Cat5e cables (less than a meter) to the switch, all on the same switch. I've also tested 3 other switches, one of which will work a bit more often than the others (probably because its cheap and slow switch even though it does 1GB)
I've also tested Odroid to Odroid and still see the lost packets
Last edited by mlinuxguy on Sun Jan 11, 2015 5:32 pm, edited 1 time in total.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: updated: NIC speed tests using IPERF and compared to RPI

Unread postby mlinuxguy » Fri Jan 09, 2015 4:36 pm

I'm now leaning toward there being a problem with initialization inside the am_internal.ko module, or am_rtl8211f.ko module
When I boot up the odroid-ci and it connects with slow read performance:
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-14.9 sec  0.88 MBytes  0.06 MBytes/sec


I have a script that reload the kernel modules, if I run it 2 to 3 times it will connect to the switch with full performance
Code: Select all
# cat reloadnet.sh
#!/bin/bash
rmmod am_net8218
rmmod am_rtl8211f
rmmod am_internal
sleep 2
modprobe am_internal
modprobe am_rtl8211f
modprobe am_net8218

note: this will only work if the network is built as a loadable module and not built-in as it is now

On the 3rd reloadnet.sh I connected with full performance:
Code: Select all
root@odroidci-1:~# iperf -fM -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.130 port 5001 connected with 192.168.1.118 port 38093
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   442 MBytes  44.2 MBytes/sec


I also have 2 scripts that turn on RPS and XPS (receive packet steering and transmit) which I then run and re-test:
Code: Select all
root@odroidci-1:~# ./set_rps.sh
root@odroidci-1:~# ./set_xps.sh
root@odroidci-1:~# iperf -fM -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.130 port 5001 connected with 192.168.1.118 port 38094
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   544 MBytes  54.4 MBytes/sec


and the transmit side:
Code: Select all
===== switch odroid-ci to client and see how fast we can transmit
root@odroidci-1:~# iperf -fM -c 192.168.1.118
------------------------------------------------------------
Client connecting to 192.168.1.118, TCP port 5001
TCP window size: 0.08 MByte (default)
------------------------------------------------------------
[  3] local 192.168.1.130 port 48498 connected with 192.168.1.118 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   387 MBytes  38.7 MBytes/sec


I will leave this up overnight and see if the connect speed remains at full performance, but it seems promising.
Perhaps I can use these in a network connected cluster after all.

Update at 8hours later
Code: Select all
[  ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   569 MBytes  56.8 MBytes/sec

Still up at full performance, will check periodically
Update at 16hours
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   556 MBytes  55.6 MBytes/sec
Last edited by mlinuxguy on Sat Jan 10, 2015 6:18 am, edited 3 times in total.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby odroid » Fri Jan 09, 2015 5:28 pm

Very weird and interesting method. ;)
Please update the status after 24hours.
User avatar
odroid
Site Admin
 
Posts: 21303
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: [workaround]: NIC speed tests using IPERF

Unread postby bohdans » Sat Jan 10, 2015 8:29 am

Not sure if you saw this: http://forum.odroid.com/viewtopic.php?f=117&t=8096

But I had a huge impact on NFS due to the krb5 module in the kernel.
bohdans
 
Posts: 57
Joined: Tue Dec 30, 2014 4:07 pm
languages_spoken: english
ODROIDs: Odroid-C1

setup scripts: NIC speed tests using IPERF

Unread postby mlinuxguy » Sat Jan 10, 2015 11:33 am

Thanks for the NFS fix information, I made that change

After roughly 24hrs I can report that the NIC performance stays high
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   572 MBytes  57.2 MBytes/sec


I spent some time looking the network driver to see if there was a easy way to force it to re-initialize the PHY without reloading.
Such as a /sys/class/net echo command
However, there isn't one already implemented in the code. I could add one but it will take some effort to see what is safe to call.
(there is NO documentation/comments in the PHY part of the code to let you figure out what is safe)

If HK could get the PHY init bug fixed that would prevent the network module reloads from being necessary.

I would like to recode the driver (am_net8218.c) to be multiple queues for RX and TX and see if I can get near line speed off the driver.
The buggy PHY sure slows down module testing, however.

Notes: if anyone wants to test RPS or XPS here are my scripts
In one terminal setup a watch of your per-processor softirqs
Code: Select all
root@odroid-c3:~# cat watchirq.sh
#!/bin/bash
watch -n1 cat /proc/softirqs


In another terminal window setup RPS (receive packet steering)
Code: Select all
root@odroid-c3:~# cat set_rps.sh
#!/bin/bash
echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
echo 2048 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
echo f > /sys/class/net/eth0/queues/rx-0/rps_cpus


Then setup XPS (for transmit)
Code: Select all
root@odroid-c3:~# cat set_xps.sh
#!/bin/bash
echo c > /sys/class/net/eth0/queues/tx-0/xps_cpus


Now run IPERF tests for both client and server and watch the IRQ count mount for each processor on the RX and TX rows

Further notes:
XPS has a bitmask of 'c' which will be these processors 1100 (think quad core)
RPS has a bitmask of 'f' which will be all processors 1111
This may not be the optimal configuration once I re-write the driver to do multiple queuing
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby arel » Sun Jan 11, 2015 2:02 am

Quick question...o you think that in the near future a RELIABLE ethernet driver will be available...at the moment my C1 does make me craZY; BECAUSE OF THE FACT THAT ETHERNET IS not STABLE OR slow LIKE HELL :-(

I can not use the board at the current state with e.g. ubuntu or OMV. Even if it is cheap that doesnt mean it is worth the price .. as long it has such problems with GB ethernet

Maybe someone from "hardkernel2 could help ... I hardly can wait. the specs of the C1 are GREAT...now software/OS (kernel, drive and so on have to be developed in a more or less stable way...and the RPI KILER is born)

regards
arel
arel
 
Posts: 11
Joined: Sun Jan 11, 2015 1:20 am
languages_spoken: english
ODROIDs: Odroid C1

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Sun Jan 11, 2015 6:40 am

Sure there will be a reliable fast driver, eventually..
The bugs in PHY init need to be worked out, its possible its a timing issue since I can reload the drivers a few times and connect with high speed.
However I adjusted all the existing timers in the PHY with no luck.

I suspect some minor rewrite of the network driver could kick its performance up over the max of 60mb/sec (600mbit) that I can get.
If you can imagine a $35 quad core board that gets near 1gb wire speed on the market, that's what you would have then

Give them time to work out the PHY init issue
As it is they have a remarkable product in a short timeframe, in a bit more time all these issues will get worked out.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Mon Jan 12, 2015 5:01 am

I have recoded the network driver to have separate RX and TX tasks
Resulting RX speed:
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   631 MBytes  63.2 MBytes/sec


Resulting TX speed:
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   391 MBytes  39.1 MBytes/sec


To do still:
1: I do not schedule the TX or RX task to only execute when needed (I check a boolean if something to do)
2: I do not have multiple RX or TX queue's

Still the results are promising
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Mon Jan 12, 2015 1:30 pm

I have a workaround for the poor 1GB nic connection (without rebooting)
I added a network driver module option: phyreset

If I connect with a slow link
Code: Select all
[  5] local 192.168.1.144 port 5001 connected with 192.168.1.118 port 57706
[  5]  0.0-10.0 sec  81.0 MBytes  8.08 MBytes/sec

I echo a 1 to my option: phyreset and it will reset the hardware
Code: Select all
root@odroid-c3:/sys/class/ethernet# ls
cali  count  debug  linkspeed  macreg  mdcclk  phyreg  phyreset  pwol  wol
root@odroid-c3:/sys/class/ethernet# echo 1 > phyreset

note that is: # echo 1 > /sys/class/ethernet/phyreset

It will not come back for about 30 seconds, then re-run the IPERF server test:
Code: Select all
[  4] local 192.168.1.144 port 5001 connected with 192.168.1.118 port 57709
[  4]  0.0-10.0 sec   510 MBytes  51.0 MBytes/sec


The DMESG output will look like this:
Code: Select all
[  511.415421@1] eth0: closed
[  511.417103@1] write mac add to:eda7d2c8: 00 1e 06 34 9b 69 |.\x1e.4.i|
[  511.418593@1] netdev_open
[  511.418606@1] Ethernet reset
[  511.418625@1] NET MDA descpter start addr=eca80000
[  511.426544@1] phy_interface = 0
[  511.426558@1] aml_phy_init:  trying to attach to 0:01
[  511.454676@1] --1--write mac add to:eda7d2c8: 00 1e 06 34 9b 69 |.\x1e.4.i|
[  511.454702@1] --2--write mac add to:eda7d2c8: 00 1e 06 34 9b 69 |.\x1e.4.i|
[  511.454721@1] write mac add to:eda7d2c8: 00 1e 06 34 9b 69 |.\x1e.4.i|
[  511.454742@1] Current DMA mode=0, set mode=621c100
[  511.454772@1] eth0: opened (irq 40).
[  511.455002@1] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  511.496646@3] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
[  515.454624@1] libphy: 0:01 - Link is Up - 1000/Full
[  515.454655@1] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby zehnm » Mon Jan 12, 2015 4:31 pm

Awesome work! I'm looking forward to your updated driver :)
zehnm
 
Posts: 20
Joined: Thu Dec 18, 2014 5:13 pm
languages_spoken: english
ODROIDs: C1

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Tue Jan 13, 2015 9:06 am

I've narrowed it down to switch/NIC auto-negotiation

Is it faulty switches responding to auto-eng? or Bad code in the driver?

If I force auto-negotation I can switch from performance problem to none or back
Code: Select all
dmesg:
[  135.944859@3] libphy: 0:01 - Link is Down
[  138.944854@3] libphy: 0:01 - Link is Up - 1000/Full


I setup a new network driver parameter: /sys/class/ethernet/renegotiate
to force it.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

ethtool -r eth0 (another workaround)

Unread postby mlinuxguy » Tue Jan 13, 2015 12:55 pm

I have a easy way for everyone with 1GB switch problems to initiate auto-negotiation without rebooting or a new driver
I need anyone having that problem to test if this workaround solves their network performance problem

This is not the final fix, just to confirm another auto-negotiation transaction solves everyone's network performance issues

I have a bad switch connection after rebooting:
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.2 sec  63.5 MBytes  6.25 MBytes/sec


# ethtool -r eth0
Code: Select all
[  5] local 192.168.1.144 port 5001 connected with 192.168.1.118 port 57917
[  5]  0.0-10.0 sec   488 MBytes  48.8 MBytes/sec


If this works for everyone it can be a generic workaround while hunting down the root cause
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby odroid » Tue Jan 13, 2015 3:08 pm

We've tried "ethtool -r eth0" with a new hub TP-Link TL-WDR4300 (Atheros chipset).
But it didn't help. :(
Which chip is inside your hub?

Anyway, Time to look into the driver source code.
User avatar
odroid
Site Admin
 
Posts: 21303
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Tue Jan 13, 2015 4:05 pm

I will have to tear it down and see :ugeek:

Do multiple reboots ever get you a full speed connection?
Perhaps the negotiation between my switch and the Oroid-CI only a matter of timing, while other switches fail completely.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby Arthur_Pendrag » Wed Jan 14, 2015 12:47 am

I would like to take a moment to offer a big Thank You to mlinuxguy's efforts to improve this ethernet driver. I was so afraid that we would end up with a situation where we knew there was a problem, but no one with the skills and motivation to fix it would be available to do so. I'd love to dive into it myself, but alas my C skills have almost completely atrophied and I never have gotten around to learning how to write Linux device drivers. I can, however, help test if that would be useful. If driver patches can be put into a github repo I'd be happy to pull them down, apply them, compile, and test performance.
Arthur_Pendrag
 
Posts: 1
Joined: Wed Jan 14, 2015 12:37 am
languages_spoken: english
ODROIDs: C1

Re: [workaround]: NIC speed tests using IPERF

Unread postby MPPS » Wed Jan 14, 2015 6:40 am

Hello,

I am new here in the Forum... I just received my odroid c1... :D

I copied all my Server stuff, powered odroid c1... and got 160kbit/sec... :o

By luck I found this forum thread where people already fighting with this issue... thank you guys...

Running 'ethtool -r eth0' has the same result as rebooting the odroid, so by luck you get good iperf results...
From my Point of view it looks if this connection is not stable, in case of copying data to odroid as a NAS the network connections drops to poor speed from time to time...

My workaround is to use a USB 2.0 Gigabit LAN Adapter with ASIX chip with stable 20MBytes/sec.

(I have got the 8-Port Gigabit Switch from TP-LINK TL-SG1008D.)

Looking forward to have a well performing odroid c1.

regards
mpps
Raspberry Pi as a toy for GPIO and Python
Radxa Rock as a linux desktop
Banana Pi as a NAS and backup storage
Odroid C1 as domain controller and AP
MPPS
 
Posts: 33
Joined: Mon Jan 12, 2015 8:54 pm
languages_spoken: english, german
ODROIDs: C1

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Wed Jan 14, 2015 7:47 am

I appear to have fixed the auto-negotiation problems with my 2 1GB switches
It probably does not fix everyone's switch issues, but I'm sending my updated driver to Hardkernel for testing.

I have rebooted my Odroid-C1 hundreds of times while working on optimizing the driver and have not had a single slow connect since I did a test fix yesterday
Odroid-CI server
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   632 MBytes  63.3 MBytes/sec


Odroid-CI client
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   409 MBytes  40.9 MBytes/sec


I optimized the driver in several ways
    (1) re-arranged and cut out all the bad / extra code in the hot TX and RX interrupt and tasklet code
    (2) created 2 different tasklets, one for TX and another for RX
    (3) adjusted the ring buffers
    (4) tested it using RPS and XPS to gain about 20% boost (receive packet steering and transmit)

Still left to do on the driver
    (1) see if there is any more optimization I can do when using tasklets
    (2) see if I can get other cores besides CPU0 to participate
    (3) recode it using NAPI instead of tasklets and see if there is a boost
    (4) fix any outstanding switch issues
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby ryecoaaron » Wed Jan 14, 2015 8:01 am

Can't wait to try it :)
ryecoaaron
 
Posts: 95
Joined: Tue Dec 23, 2014 6:02 am
Location: Wisconsin
languages_spoken: english
ODROIDs: odroid-c1, odroid-xu4, cloudshell xu4

Re: [workaround]: NIC speed tests using IPERF

Unread postby MPPS » Wed Jan 14, 2015 8:30 am

@mlinuxguy: well done

I have some additional Information...

Set the network speed to 100
> ethtool -s eth0 speed 100 duplex full
iperf shows 11.1 MBytes/sec, reproducible

Set back to Speed 1000
> ethtool -s eth0 speed 1000 duplex full
iperf shows by luck > 30MBytes/sec but this is again not for sure

regards
mpps
Raspberry Pi as a toy for GPIO and Python
Radxa Rock as a linux desktop
Banana Pi as a NAS and backup storage
Odroid C1 as domain controller and AP
MPPS
 
Posts: 33
Joined: Mon Jan 12, 2015 8:54 pm
languages_spoken: english, german
ODROIDs: C1

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Wed Jan 14, 2015 8:43 am

I have some additional Information...

That seems to indicate you have the same issue that I did with my switches and my fix would probably work for you (or fixed driver)

My testing showed If I reset the PHY or ran ethtool -r eth0 (several times if necessary) I would get a high-speed connection

The other thing that needs tested is if a cross-over cable changes anyone's switch problems.... that is part of auto-negotiation
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Wed Jan 14, 2015 3:30 pm

We've tried "ethtool -r eth0" with a new hub TP-Link TL-WDR4300 (Atheros chipset).
But it didn't help. :(
Which chip is inside your hub?


Sadly the RGS-1016 Rosewill switch has giant heat sinks glued over the chips, without destroying it I cannot tell you what chips they are using

The NetGear GS108 I do have information for (and it showed the same issue as the Rosewill)
netgear GS108 8 port 1gb switch
GS108 uses an Altima (acquired by Broadcom in 2000) AC388 switch chip and Broadcom BCM 5014 quad PHY. This combo is used instead of the Broadcom BCM5385 5 port switch chip used in the RX3141, as well as D-Link's DGL-4100 and DGL-4300 gaming routers, and 5 port gigabit switches from Belkin, Linksys, TRENDnet and others.
(fun tidbit: I did find a swelling capacitor on the power side of the netgear.... )
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby MPPS » Wed Jan 14, 2015 4:32 pm

@mlinuxguy
That seems to indicate you have the same issue that I did with my switches and my fix would probably work for you (or fixed driver)


Is there a possibility to get the fix (fixed driver)?
(binary would be comfortable :mrgreen: )

regards
mpps
Raspberry Pi as a toy for GPIO and Python
Radxa Rock as a linux desktop
Banana Pi as a NAS and backup storage
Odroid C1 as domain controller and AP
MPPS
 
Posts: 33
Joined: Mon Jan 12, 2015 8:54 pm
languages_spoken: english, german
ODROIDs: C1

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Wed Jan 14, 2015 4:48 pm

I sent my code to odroid and mdrjr for testing. Your best bet is to wait for them to push it to GIT after they test.
I plan to push it to my own github account but mine will be changing with other features very fast... for a tested driver wait for them to push it

I'm going to try for queues per CPU core, but I am unsure if the DMA in the realtek PHY will support that... nothing like blindly poking at a chipset :D
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Thu Jan 15, 2015 3:53 am

For those willing to test my version of the Odroid-C1 network driver, here is the github repo:
https://github.com/mlinuxguy/odroid-c1-network-driver

Please note that you will copy these files over the ones under:
/usr/src/linux/drivers/amlogic/ethernet/ --> replace them

I expect Hardkernel to push my driver (this version) to their repo in a few days
You can keep testing with my driver as I diverge from their code testing different features
or just stick with theirs once they release mine as it will be stable and probably just get fixes

Notes: Should work with Realtek and Broadcom based 1gb switches, will connect with poor performance to Athero's based switches until we fix that
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby wibrain » Thu Jan 15, 2015 8:39 am

mlinuxguy wrote:For those willing to test my version of the Odroid-C1 network driver, here is the github repo:
https://github.com/mlinuxguy/odroid-c1-network-driver

Please note that you will copy these files over the ones under:
/usr/src/linux/drivers/amlogic/ethernet/ --> replace them

I expect Hardkernel to push my driver (this version) to their repo in a few days
You can keep testing with my driver as I diverge from their code testing different features
or just stick with theirs once they release mine as it will be stable and probably just get fixes

Notes: Should work with Realtek and Broadcom based 1gb switches, will connect with poor performance to Athero's based switches until we fix that



Following this thread...Looking for relief...
I wanted to try your solution, but I don't have any files in the path shown (/usr/src/linux/drivers/amlogic/ethernet/)
/usr/src/ has no linux subfolders etc.

Am I just Noobie Nuts....or maybe a path typo :?:
wibrain
 
Posts: 10
Joined: Thu Jan 15, 2015 8:06 am
languages_spoken: english
ODROIDs: C1.C1+, C2

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Thu Jan 15, 2015 8:53 am

You have to checkout the linux source from GIT
My advice is to not do it (since I had to tell you why you didn't have the source). Building the kernel is fairly complicated, look at their WIKI on the subject:
http://odroid.com/dokuwiki/doku.php?id= ... ing_kernel

Building my driver requires more background knowledge than this thread (or I) will provide.
Someone can either build it and post the kernel modules + uImage to the forums or you can wait for Hardkernel to build it and update your kernel through odroid-utility.sh

The process of getting 1gb switches to all work with the board will mean that both my driver and Hardkernel's version of it will be constantly changing for a while.
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby wibrain » Thu Jan 15, 2015 8:57 am

Good Advice....I will just wait this out. Thanks for the response!
wibrain
 
Posts: 10
Joined: Thu Jan 15, 2015 8:06 am
languages_spoken: english
ODROIDs: C1.C1+, C2

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Thu Jan 15, 2015 10:35 am

To see how badly CPU0 is being hammered I overclocked the Odroid-C1 and re-ran my tests
Code: Select all
# ./showfreq.sh
scaling max freq
1536000
cpuinfo max freq
1536000
cpuinfo cur freq
1536000
cpuinfo avail freq
96000 192000 312000 408000 504000 600000 696000 816000 912000 1008000 1104000 1200000 1296000 1416000 1488000 1536000


Odroid-C1 Server (RPS on)
Code: Select all
[  5] local 192.168.1.144 port 5001 connected with 192.168.1.118 port 39532
[  5]  0.0-10.0 sec   681 MBytes  68.0 MBytes/sec

Odroid-C1 Client (XPS on)
Code: Select all
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   415 MBytes  41.5 MBytes/sec

It is very spiky however, so I check how the interrupt count increases over a 30sec sleep while Odroid eats 10 seconds of IPERF inbound data
Code: Select all
# cat /proc/interrupts > one;sleep 30; cat /proc/interrupts > two;diff one two
2,3c2,3
<  35:      47343          0          0          0       GIC  am_osd_vsync, vsync
<  40:   10236201          0          0          0       GIC  eth0
---
>  35:      49145          0          0          0       GIC  am_osd_vsync, vsync
>  40:   10691400          0          0          0       GIC  eth0
8c8
<  63:    6229273          0          0          0       GIC  dwc_otg, dwc_otg_hcd:usb1
---
>  63:    6466447          0          0          0       GIC  dwc_otg, dwc_otg_hcd:usb1
10,13c10,13
<  92:     109359          0          0          0       GIC  MESON TIMER-F
<  93:          0      35224          0          0       GIC  MESON TIMER-G
<  94:          0          0      26012          0       GIC  MESON TIMER-H
<  95:          0          0          0      28789       GIC  MESON TIMER-I
---
>  92:     113306          0          0          0       GIC  MESON TIMER-F
>  93:          0      36297          0          0       GIC  MESON TIMER-G
>  94:          0          0      26967          0       GIC  MESON TIMER-H
>  95:          0          0          0      29788       GIC  MESON TIMER-I
17,18c17,18
< 110:       8525          0          0          0       GIC  sdhc
< 121:      47342          0          0          0       GIC  rdma, osd_rdma
---
> 110:       8573          0          0          0       GIC  sdhc
> 121:      49144          0          0          0       GIC  rdma, osd_rdma
33c33
< IPI2:       5299       8859      14137       6169  Rescheduling interrupts
---
> IPI2:       5332       8950      14153       6187  Rescheduling interrupts
35c35
< IPI4:          2     965013     483229     368010  Single function call interrupts
---
> IPI4:          2    1020612     529660     370235  Single function call interrupts


The really big ones are:
Code: Select all
           CPU0       CPU1       CPU2       CPU3
<  40:   10236201          0          0          0       GIC  eth0
---
>  40:   10691400          0          0          0       GIC  eth0

<  63:    6229273          0          0          0       GIC  dwc_otg, dwc_otg_hcd:usb1
---
>  63:    6466447          0          0          0       GIC  dwc_otg, dwc_otg_hcd:usb1

Not only was eth0 IRQ count climbing as expected for CPU0, but so was dwc_otg, dwc_otg, hcd:usb1 for some reason
Others on CPU0 were increasing but nothing like these numbers.

Might have to look into why USB1 gets so many interrupts..... could it explain the spiky keyboard behavior many see?
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Thu Jan 15, 2015 5:54 pm

No luck so far in figuring out either of these:

(1) Why is the Interrupt rate for USB1 so high? (it's hammering CPU 0)

(2) How to change ETH0 CPU affinity to a CPU different from CPU 0

On the topic of CPU affinity this platform totally ignores:
# echo f > /proc/irq/40/smp_affinity
(or any CPU ID to it, it just keeps IRQ 40 on CPU 0)

If I can figure out where to do it I will try to fix ETH0 to another CPU and see if that improves network performance more
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Re: [workaround]: NIC speed tests using IPERF

Unread postby mlinuxguy » Thu Jan 15, 2015 8:33 pm

I finally figured out how to set affinity on the various IRQ's
The results say that USB1 is killing CPU 0 performance at least for interrupts
Code: Select all
# cat affinity.sh
#!/bin/bash
echo "moving eth0 to cpu 0,1,2"
echo 1,2 > /proc/irq/40/smp_affinity_list
echo "usb1 to cpu 3"
echo 3 > /proc/irq/63/smp_affinity_list


Notes:
(1) ETH0 will only hit CPU 1, even though I list CPU1 and CPU2
(2) USB1 is moved off CPU0 to CPU3
(3) using RPS and XPS doesn't help much since we are limiting the CPU's that can contribute to steering
(4) you get a huge jump in network performance by getting USB1 off of the same CPU as ETH0 (even without RPS/XPS)
(5) Naive use of smp_affinity would say echo to smp_affinity not smp_affinity_list, but the 2nd is all that works

I will have to experiment more to find the best combination, but that tells us that USB1 is a killer for performance on CPU0

Without RPS but with ETH0 on CPU2 (Odroid-C1 as server)
Code: Select all
[  5] local 192.168.1.144 port 5001 connected with 192.168.1.118 port 39611
[  5]  0.0-10.0 sec   684 MBytes  68.4 MBytes/sec
mlinuxguy
 
Posts: 638
Joined: Thu Feb 28, 2013 10:28 am
languages_spoken: english
ODROIDs: X, X2, XU, XU3, XU4, C1, C1+, C2, USB-IO

Next

Return to Hardware and peripherals

Who is online

Users browsing this forum: Bing [Bot] and 1 guest