H2+ driver bug/wrong setting and ethernet inefficiency

Post Reply
dasc
Posts: 3
Joined: Wed Jul 22, 2020 6:40 pm
languages_spoken: italian, english
ODROIDs: 2x ODROID H2+
Has thanked: 0
Been thanked: 0
Contact:

H2+ driver bug/wrong setting and ethernet inefficiency

Post by dasc »

Hi
I recently bought 2 H2+ and I'm experiencing unexpected issues with network communication
digging further it seems to me there may be something wrong with the Realtek driver
please bear with me because I did several tests and this may be a little long

first of all this is the relevant part of my environment:

odroidnas: ODROID H2+, 2x4GB RAM, with Ubuntu 20.04, driver installed as per odroid wiki using hardkernel ppa, no additional relevant installations, just iperf3 and apt updates
gateway: ODROID H2+, 2x8GB RAM, with Ubuntu 20.04, same as above (only difference is the amount of ram)
mainpc: workstation (i7 3770k) with Ubuntu 18.04
oldnas: some branded nas from a couple years back

switch: NetGear ProSafe GS116e Plus
every test happened with both cat5e and cat6 cables (I switched them and repeated the tests)

iperf3 in every direction (mainpc <-> odroids, mainpc <-> oldnas, odroids <-> oldnas) scores approximately in the 950 Mbits/s, ~830 for the odroids which I understand is expected as of now

every other relevant nas-related communication (smb, nfs, scp, rsync over mount points) though runs at a markedly slower speed, something in the vicinity of ~140Mbit/s or 17 MB/s
this, again, happens in every direction to and from the odroids, while from mainpc to oldnas I get about 700Mbit/s (85MB/s)

I tried the samba settings suggested in the wiki and this indeed helped, I get up to ~480Mbit/s (60MB/s) when the odroid acts as a server, but every other service is left uncovered

This led me to perform additional tests and I encountered really unexpected behaviours

the first one is that iperf3 between the 2 odroids (gateway <-> odroidnas) scores just 110 Mbit/s, this is right after they scored in the ~800Mbit/s against mainpc or oldnas, absolutely no changes, no reboot, same cables same ports, same everything.
And it's extremely repeatable, consecutive calls to iperf3 net the exact same results

In addition to this, I happen to have an USB3.0 Gigabit Ethernet dongle
both odroids, one at a time of course, when binding over the dongle ip, suddenly score 830Mbit/s in iperf3 both against each other AND against mainpc/oldnas
and every network service (smb, nfs, rsync etc) reaches a reasonable ~500Mbit/s - 60MB/s speed, again both against each other and against the other machines
I tried several combination of cables and switch ports and all the tests I did behaved exactly the same regardless

as an additional note it seems that if the embedded ethernet port is turned up (ifconfig enp2s0 up) after the dongle (which apparently runs on a r8152 driver as well, lshw tells me), the embedded port scores ~800Mbit/s on iperf3 as well, odroid against odroid (where otherwise looks capped at 100Mbit/s).
this, tested by calling iperf3 -c <ip_of_the_embedded_card>
still, same cables, same switch ports, same everything just enabling and disabling dongle and embedded card in a specific order

any comment on all of this?
is there a way to obtain that kind of performance without resorting to additional appendages?

User avatar
mad_ady
Posts: 9469
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 604 times
Been thanked: 678 times
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by mad_ady »

So both h2s are connected to the same switch with the embedded ports? What does ethtool say about negociated speeds?

schaze
Posts: 1
Joined: Mon Jul 27, 2020 2:25 am
languages_spoken: english
ODROIDs: 3x odroid h2+ / 16GB RAM
Has thanked: 0
Been thanked: 0
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by schaze »

Hey,

I have the same issues. Noticed it the first time when I was running speed tests for my internet. I have a 1000 MBit/s connection, and tests always only showed about 250 MBit. First I thought my internet provider had issues but, a quick test on another random machine in the net showed full 1GBit results.
I am using Debian Buster also with the DKMS driver and 4.19 Kernel version. Switching cables makes no difference.
I searched for any possible parameters to pass to the drivers, also disabled onboard audio (I read that for another SBC once and was out of options), all with no luck. All 3 odroids show the same slow speeds. What good is it to have 2x 2,5GBit/s ethernet ports if they get no speed... I am actually quite disappointed, I wanted to use my 3 odroids as a ceph storage cluster.

Best regards,
Thomas

User avatar
mad_ady
Posts: 9469
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 604 times
Been thanked: 678 times
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by mad_ady »

What tests did you run? iperf? If yes, with what parameters?

dasc
Posts: 3
Joined: Wed Jul 22, 2020 6:40 pm
languages_spoken: italian, english
ODROIDs: 2x ODROID H2+
Has thanked: 0
Been thanked: 0
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by dasc »

mad_ady wrote:
Fri Jul 24, 2020 3:08 am
So both h2s are connected to the same switch with the embedded ports? What does ethtool say about negociated speeds?
They are negotiated at 1000Mb/s (1Gb/s) as expected (the switch does not support 2.5Gb/s)

Please refer to my first post for the exact numbers and symptoms, but the gist of it is that the only thing that actually goes near the theoretical negotiated speed is iperf3 AND when they don't test one against the other, everything else is significantly slower (less then 1/5)

Tests with a usb3.0 dongle, in addition to everything else, show that it's not a switch-cable-othermachines issue

User avatar
mad_ady
Posts: 9469
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 604 times
Been thanked: 678 times
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by mad_ady »

Ok, so the iperf values look good, but samba/scp/rsync don't. Let's rule out the filesystem as a bottleneck. What source/destination filesystem are you using? NTFS? If yes, what transfer speeds do you get when copying files locally? How is cpu usage looking during the transfer?

dasc
Posts: 3
Joined: Wed Jul 22, 2020 6:40 pm
languages_spoken: italian, english
ODROIDs: 2x ODROID H2+
Has thanked: 0
Been thanked: 0
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by dasc »

mad_ady wrote:
Mon Jul 27, 2020 12:59 pm
Ok, so the iperf values look good, but samba/scp/rsync don't. Let's rule out the filesystem as a bottleneck. What source/destination filesystem are you using? NTFS? If yes, what transfer speeds do you get when copying files locally? How is cpu usage looking during the transfer?
the iperf3 values do not always look good, since odroid vs odroid, under normal circumstances, barely scores above 120Mb/s

Filesystem as bottleneck should already be ruled out by the fact that:
- using the usb3-eth adapter nets better bandwidth usage in every scenario (smb, nfs, scp, etc)
- bringing up the embedded eth AFTER the usb3-eth nets basically the same bandwith usage as the usb3 adapter *but on the embedded board* (again, smb, nfs, scp, etc)

as detailed in my first post.

in any case, both odroids mount a Sabrent Rocket 512GB NVME M.2 module with Ext4 fs, which should be more than capable of handling any continuous write a 1Gb/s lan can throw at them
a 2.5GB file copy ends in about 5s

about cpu usage, when the embedded board runs "slow", it's way below saturation, in the 20% to 30% range

henrikno
Posts: 3
Joined: Mon Dec 07, 2020 10:43 am
languages_spoken: english
Has thanked: 0
Been thanked: 5 times
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by henrikno »

I also had similar performance issues in one direction but not the other (testing with iperf). I used wireshark and saw a lot of retransmits. I also noticed using ethtool -S enp3s0 that rx_missed was high and climbing during iperf.

Code: Select all

     rx_missed: 24938
Googling that led me to some threads about other realtek chips (e.g. r8169) that require disabling ASPM.
https://bugzilla.redhat.com/show_bug.cgi?id=1679140
https://bugs.launchpad.net/ubuntu/+sour ... ug/1880076
https://www.spinics.net/lists/netdev/msg548397.html
Running

Code: Select all

echo "performance" > /sys/module/pcie_aspm/parameters/policy
Improved things a lot for me (added pcie_aspm=performance to kernel options to make it permanent)

For reference I'm running 5.4.60-1-pve

Code: Select all

# ethtool -i enp3s0
driver: r8125
version: 9.003.05-NAPI
Apparently you can also disable this in bios (haven't tried this)
BIOS -> Chipset -> South cluster configuration -> PCI Express configuration -> PCI Express root port(1&2) -> ASPM -> Disable
These users thanked the author henrikno for the post (total 2):
alban (Thu Jan 28, 2021 6:43 am) • domih (Wed Feb 03, 2021 8:04 am)

alban
Posts: 1
Joined: Thu Jan 28, 2021 6:35 am
languages_spoken: French
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by alban »

Tested here for pcie_aspm in performance mode with two H2+ on Proxmox (Kernel 5.4.78-2-pve and driver r8125 version 9.004.01-NAPI)

Code: Select all

On H2+ Proxmox 1 :
iperf3 -s

On H2+ Proxmox 2 :
iperf3 -c IP_Proxmox_1 -i 0 -t 5

Result :
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-5.00   sec  1.37 GBytes  2.35 Gbits/sec    0    731 KBytes
 


To enable it in persistent mode for "Systemd-boot" :
- Add pcie_aspm.policy=performance in the end on the line of /etc/kernel/cmdline file,
- apply changes with pve-efiboot-tool refresh
- reboot
- check Kernel parameters : cat /proc/cmdline # (at the end : pcie_aspm.policy=performance)
- check pcie_aspm : cat /sys/module/pcie_aspm/parameters/policy # [performance]

Related links :
- https://pve.proxmox.com/wiki/Host_Bootloader
- https://access.redhat.com/documentation ... guide/aspm
These users thanked the author alban for the post (total 2):
odroid (Fri Jan 29, 2021 11:45 am) • domih (Wed Feb 03, 2021 8:04 am)

domih
Posts: 410
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, N2+, H2, H2+, C4, HC4 - 1GbE, 2.5GbE, 10GbE, 40+GbE
Location: San Francisco Bay Area
Has thanked: 157 times
Been thanked: 158 times
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by domih »

I just tested Realtek driver r8125-9.003.05 with ASPM disabled and while still providing 2.35 GbE on a 2.5 GbE subnet, it indeed provides 940+ Mbits on a 1 GbE subnet. Yeah!

Very good catch! TYVM to @henrikno and @alban.

I tested the 3 ways to disable ASPM, they all work:

# On the fly:

echo performance | sudo tee /sys/module/pcie_aspm/parameters/policy
cat /sys/module/pcie_aspm/parameters/policy
default [performance] powersave powersupersave

# Adding the option to Grub:

sudo vi /etc/default/grub
.../...
GRUB_CMDLINE_LINUX_DEFAULT="pcie_aspm.policy=performance quiet splash"
.../...

sudo update-grub
sudo reboot

# Disabling ASPM in the BIOS:

BIOS / Chipset / South cluster configuration / PCI Express configuration / PCI Express root port(1&2) / ASPM / Disable
Note: do it for root port 1 and root port 2 which are the two root ports used for LAN.

I will test Realtek driver r8125-9.004.01 next week-end to check if it also fixes the "crazy" CPU consumption with this particular version.
These users thanked the author domih for the post (total 2):
odroid (Wed Feb 03, 2021 9:07 am) • joshua.yang (Thu Feb 25, 2021 9:48 am)

User avatar
meepmeep
Posts: 6
Joined: Tue Feb 02, 2021 2:00 am
languages_spoken: english
ODROIDs: H2+
Has thanked: 2 times
Been thanked: 1 time
Contact:

Re: H2+ driver bug/wrong setting and ethernet inefficiency

Post by meepmeep »

I was just browsing this forum to get a better understanding of my h2+ odroid.
I was a little surprised that a simple ping from odroid to my NAS / fw was always slower than from any other equipment.

Before the "performance" settings :

Code: Select all

Odroid : average 0.700ms
shuttle server / laptop on ethernet : 0.200ms
After :

Code: Select all

Odroid : average 0.200ms
shuttle server / laptop on ethernet : 0.200ms
Yeah ! Thank you !
These users thanked the author meepmeep for the post:
domih (Sat Feb 06, 2021 2:11 am)

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 0 guests