High Latency on RTL8125b

Post Reply
TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

High Latency on RTL8125b

Post by TheBigG »

Hi,
I use the H2+ as a router and notice that on OpenWRT and OPNsense the latency of the NICs is very high compared to a notebook or a turris omnia, but the thoughput is good, so i don't want to miss that ever again.
Image

Code: Select all

64 bytes from 192.168.1.1: seq=0 ttl=255 time=1.008 ms
64 bytes from 192.168.1.1: seq=1 ttl=255 time=1.138 ms
64 bytes from 192.168.1.1: seq=2 ttl=255 time=1.151 ms
64 bytes from 192.168.1.1: seq=3 ttl=255 time=1.019 ms
Does somebody know why and if it is fixable?

User avatar
odroid
Site Admin
Posts: 37793
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1919 times
Been thanked: 1186 times
Contact:

Re: High Latency on RTL8125b

Post by odroid »

Try disabling the ASPM feature in the BIOS settings.
viewtopic.php?p=322560#p322560
These users thanked the author odroid for the post:
TheBigG (Wed Jul 14, 2021 7:31 am)

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

odroid wrote:
Tue Mar 16, 2021 2:40 pm
Try disabling the ASPM feature in the BIOS settings.
viewtopic.php?p=322560#p322560
That did work, didn't found that while searching for latency sorry.

Code: Select all

64 bytes from 192.168.1.1: seq=0 ttl=255 time=0.569 ms
64 bytes from 192.168.1.1: seq=1 ttl=255 time=0.271 ms
64 bytes from 192.168.1.1: seq=2 ttl=255 time=0.369 ms
64 bytes from 192.168.1.1: seq=3 ttl=255 time=0.380 ms
These users thanked the author TheBigG for the post:
odroid (Tue Mar 16, 2021 3:33 pm)

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

Hi @odroid
What I noticed after using the H2+ as a router for some time is that there is a large delay between a link comming up and beeing usable. (sorry for posting it so late but I needed more hardware to be able to savely reproduce this) If I connect my PC to the odroid h2+ it takes 2 or more minutes until I get a DHCP lease from it and the network gets usable. If I put a switch between them which keeps the NIC of the H2+ online and than connect my PC to that switch I get a DHCP lease right away.
Is that another BIOS energy saving thing which can be disabled or more a driver problem?
I can repoduce this with Kernel 5.4 (openwrt), 5.10 (openwrt) and 5.11 (proxmox)

User avatar
odroid
Site Admin
Posts: 37793
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1919 times
Been thanked: 1186 times
Contact:

Re: High Latency on RTL8125b

Post by odroid »

It seems to be worth to try Realtek's vendor driver if you can compile instead of using the stock driver in Kernel 5.11.
viewtopic.php?p=327565#p327565

But, I'm not sure if your very slow DHCP issue is related to the driver itself or not.
These users thanked the author odroid for the post:
TheBigG (Wed Jul 14, 2021 4:33 pm)

User avatar
mad_ady
Posts: 9564
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 604 times
Been thanked: 697 times
Contact:

Re: High Latency on RTL8125b

Post by mad_ady »

@TheBigG sounds more like a spanning-tree setting. The ethernet port is most likely part of a bridge. In order to prevent layer2 loops caused by port flapping a bridge/switch with STP enabled typically puts the port in listening and learning for 30s to allow STP to propagate the link state change and then puts the port in forwarding (passing traffic). You can check the state on the H2 with brctl show.
Try disabling STP in openwrt if you don't have physical loops.
These users thanked the author mad_ady for the post:
TheBigG (Wed Jul 14, 2021 4:33 pm)

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

Thanks for your Input mad_ady. But I don't use a bridge in OpenWRT and in Proxmox it is off.

Code: Select all

root@proxmox:~# brctl show
bridge name     bridge id               STP enabled     interfaces
vmbr0           8000.1673309902d5       no              enp1s0
                                                        enp2s0
                                                        enp4s0
                                                        tap102i0
                                                        tap103i1
vmbr1           8000.8a80d75ace22       no              enp3s0
                                                        vmbr0.2
Thank you Odroid for suggesting the Realtek drivers, I thought I used them with 5.4 but I will compile them on 5.11 and see if it helps.

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

I can confirm that this is fixed with using the r8125 driver instead of r8169.
These users thanked the author TheBigG for the post:
odroid (Thu Jul 15, 2021 4:37 pm)

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

After testing a littlebit more I get strange behaviour from that realtek driver.
If there is load on the NICs the throughtput drops to 0 for a second and than goes normal again and 5 secs later it goes to 0 again for a second.
Image
That did not happen with the r8169 of Kernel 5.11 :?
If I would not have to wait minutes until the connection gets usable after booting my PC I would prefer r8169.

User avatar
odroid
Site Admin
Posts: 37793
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1919 times
Been thanked: 1186 times
Contact:

Re: High Latency on RTL8125b

Post by odroid »

Did you still disable the ASPM feature on PCIe port #1 and #2?

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

yes that is disabled for all 6 ports

User avatar
odroid
Site Admin
Posts: 37793
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1919 times
Been thanked: 1186 times
Contact:

Re: High Latency on RTL8125b

Post by odroid »

I have no idea any more indeed. :(

User avatar
mad_ady
Posts: 9564
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 604 times
Been thanked: 697 times
Contact:

Re: High Latency on RTL8125b

Post by mad_ady »

I'm not 100% sure this is a driver issue. Could you run an iperf3 test between your H2 and a PC connected directly to either port? Do you have a switch connected to the port? If yes, can you check (by doing a packet capture) if it sends Ethernet OAM pause frames?

A drop in sent traffic is typically caused by the upstream system signalling its buffers are full and the sender should psuse.
These users thanked the author mad_ady for the post:
TheBigG (Fri Jul 16, 2021 5:35 pm)

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

Image
[ 4] 62.00-63.00 sec 153 MBytes 1.29 Gbits/sec
[ 4] 63.00-64.00 sec 153 MBytes 1.29 Gbits/sec
[ 4] 64.00-65.00 sec 152 MBytes 1.27 Gbits/sec
[ 4] 65.00-66.01 sec 8.57 MBytes 71.1 Mbits/sec
[ 4] 66.01-67.00 sec 0.00 Bytes 0.00 bits/sec
[ 4] 67.00-68.01 sec 0.00 Bytes 0.00 bits/sec
[ 4] 68.01-69.00 sec 122 MBytes 1.03 Gbits/sec
[ 4] 69.00-70.00 sec 151 MBytes 1.27 Gbits/sec

It doesn't seem to happen with Iperf3 against the h2+ only. But it happens if I run iperf3 + nperf or only nperf.

@mad_ady if you can tell me the exact name of the frame you are looking for so I can match it with wireshark and TCPdump (I googled Ethernet OAM pause frame tcpdump and nothing beside some cisco stuff showed up so I'm not sure sorry) I'm happy to have a look if such frame is sent.

User avatar
mad_ady
Posts: 9564
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 604 times
Been thanked: 697 times
Contact:

Re: High Latency on RTL8125b

Post by mad_ady »

Do you get the same drop with udp traffic as well?

Regarding pause frames - the truth is I had this pause issue with a misbehaving cisco switch. Haven't seen pause frames in other home switches. If there's no filter for it you can exclude iperf/ssh traffic and just capture the rest.
These users thanked the author mad_ady for the post:
TheBigG (Sat Jul 17, 2021 9:14 am)

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

Thank you mad_ady, I found the problem, I maxed out the CPU of the H2 with Iperf2 in UDP mode at 500 MBit/s which resulted in heavy packet loss and at testing further I noticed that the packetloss would not go away even if I spread the load over more cores. That made me look at my client and the problem seems to be the realtek driver for windows and not the one for linux. I switched from the onboard NIC to my old SoNNeT Solo 10 G Thunderbolt 3 adapter and there the packetloss was gone.
BTW it seems like the realtek 2,5 Gbit/s driver is broken for onboard and USB on windows because it also happens with a USB realtek 2,5 gbit/s adapter and this driver https://www.realtek.com/en/component/zo ... 0-software with version 10.45.20. Or better call it the r8125 driver on the h2+ and the windows one don't play nice together because the loss is not there if I use the r8169 driver on the h2+
I can't sadly debug this more because I don't have a endless supply of different 2,5 gbit/s capable hardware.
But maybe somebody would want to take a deeper look at this because it is a problem if r8125 and r8169 are not really stable. I mean if you are not in full controll over the complete enviroment you won't find out about this and even if they do they can't tell somebody, if you want to use this network you need this NIC and this driver version or it will fail.

Atleast it runs in this constalation great.
Image
Image

User avatar
mad_ady
Posts: 9564
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 604 times
Been thanked: 697 times
Contact:

Re: High Latency on RTL8125b

Post by mad_ady »

500Mbps for UDP traffic shouldn't max out the CPU usage on the H2... Or does it max out on your windows host?

Two more tests you could try to be sure it's the windows client:
1. Boot your PC from a live linux USB and install the same driver as on the H2. If iperf works stably, then it's the windows driver's fault
2. You should be able to run an iperf test between the two ports of your H2, provided that one port is in a different namespace. Inspired from here: viewtopic.php?t=39613

Here's how to do that:
A) Connect an ethernet cable between your H2's ports
B) For simplicitly's sake, run these commands on a local session (not through ssh). Replace eth0 and eth1 with the actual interface names:

Code: Select all

ip netns add devicea
ip link set eth1 netns devicea
ip -n devicea addr add 10.0.0.2/30 dev eth1 
ip addr add 10.0.0.1/30 dev eth0

ip -n devicea link set eth1 up 
ip link set eth0 up 

You should be able to ping between interfaces:

Code: Select all

ping -c 3 10.0.0.2
ip netns exec devicea ping -c 3 10.0.0.1
Now you can run iperf3 between them, using the physical link to carry data (use two terminals):

Code: Select all

iperf3 -s 
ip netns exec devicea iperf3 -c 10.0.0.1 -t 100

You should also be able to see traffic with iptraf-ng.

Let us know how it goes!

TheBigG
Posts: 37
Joined: Sat Nov 02, 2013 2:33 am
languages_spoken: german english
ODROIDs: XU, XU4, C1, C1+, C2, H2+
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: High Latency on RTL8125b

Post by TheBigG »

It is pretty complex to run this because my setup is complicated and i don't want to tear down it just for a test.

So this is the result
h2+ port to h2+ port: pass for r8169 and r8125
debian bullseye r8169 (on my workstation) to h2+ r8125: pass
debian bullseye r8125 (on my workstation) to h2+ r8125: pass
windows 10 10.45.20 (on my workstation) to h2+ r8169: pass
windows 10 10.45.20 (on my workstation) to h2+ r8125: drops

So r8169 seems to do something different than r8125, intresting is also that the windows test looks the same with a NETGEAR GS110MX between windows and h2+. So maybe there is something that causes this on a higher layer than I expected.

But for now I'm done, it took me a complete day to just test this properly and I have now a setup which works and even if I would find the cause I couldn't fix it because I have no coding skills.

brad
Posts: 1434
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 N2+ H2 H2+ (64 bit ftw)
Location: Australia
Has thanked: 130 times
Been thanked: 218 times
Contact:

Re: High Latency on RTL8125b

Post by brad »

I have been using opnsense for some time now and it is working quiet well.

I had a similar issue to yourself when cpu power settings enabled in that the CPU frequency would not ramp with just network traffic.

If you are running OPNSense try to disable power efficiency to see if it helps
power.png
power.png (26.73 KiB) Viewed 120 times
I have it disabled in the image above but Hiadaptive works well too. If you cannot find power settings on the OS just to load up the cpu on the H2 to see if the performance increases.

Edit: Just to try to explain a little better I suspect the cpu core frequency directly influences how quickly the NIC's IRQ can operate. At low frequencies (cpu power saving) the NIC operates slower but network traffic alone is not enough to trigger the CPU frequency to increase especially with aggressive power saving options configured. So it ends up looking like your CPU(s) might be maxed out but they are only running 25% frequency.

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 0 guests