Troubles with 10GBe on H2

Post Reply
MiguelA0145
Posts: 4
Joined: Sat Oct 05, 2019 1:34 am
languages_spoken: English
ODROIDs: ODROID C2, ODROID H2
Has thanked: 11 times
Been thanked: 0
Contact:

Troubles with 10GBe on H2

Unread post by MiguelA0145 » Sat Oct 05, 2019 1:58 am

Hello all!
I'd like to preface this with by saying I'm still learning about Linux and SBCs in general so I apologize in advance in case I say / do something stupid :) .

To get to the heart of the problem, I'd like to get a 10 gig NIC running at a good speed on my H2 and a separate tower. I've read through this thread but from what I understood, there hasn't been too much clarification on the tuning of the NIC to reach ~ 9 GB/s. I've also tried following this guide but haven't had much luck yet. Currently, all I'm able to get out of my setup is ~ 7.3 GB/s. Any help would be much appreciated! :D

iperf 3 sending test results :

Code: Select all

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec   258 GBytes  7.38 Gbits/sec    0             sender
[  5]   0.00-300.00 sec   258 GBytes  7.38 Gbits/sec                  receiver

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec   251 GBytes  7.19 Gbits/sec    0             sender
[  5]   0.00-300.00 sec   251 GBytes  7.19 Gbits/sec                  receiver
iperf3 receiving test results :

Code: Select all

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-300.00 sec  90.7 GBytes  2.60 Gbits/sec                  sender
[  5]   0.00-300.00 sec  90.7 GBytes  2.60 Gbits/sec                  receiver

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-300.00 sec   161 GBytes  4.60 Gbits/sec                  sender
[  5]   0.00-300.00 sec   161 GBytes  4.60 Gbits/sec                  receiver
netperf test :

Code: Select all

odroid@ODROID-H2:~$ netperf -T0,0 -C -c
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to localhost () port 0 AF_INET : demo : cpu bind
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

 87380  65536  65536    10.00      11035.55   25.45    25.45    0.756   0.756
Hardware List :
ODROID H2
ADT-LINK R34SF
4GB 2400 RAM
White eMMC
ASUS XC-C100C

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Sat Oct 05, 2019 10:00 am

Hi,

1) You won't reach 9 Gb/s (at the payload level) because "cheap?" 10Gbe cards use 8b/10b line encoding(*). I looked for more info about cards supporting 64b/66b for 10Gbe but did not find any so far. So the max you can reach is 8Gb/s. As a matter of fact, 7.38 Gb/s is a pretty good speed. Be very happy! <<< Note that for the exactly same reason a 1Gbe network delivers payloads at max 0.8Gb/s. In both cases you lose some additional bandwidth through all the components involved (example: PCIe itself has a line encoding).>>> <-- WRONG. This is why I said 7.38Gb/s is pretty good.

(*) 8b/10b line encoding means that each time you send 8 bits of data (the payload), the sending NIC hardware scrambles the data and add 2 bits to allow the receiving NIC to descramble the data and retrieve the actual 8 bits. Why do NICs scramble the data? The bits over the wire are sent as low voltage/high voltage. One means bit = 0, the other means bit = 1. The voltage changes according to a clock. So if you were to send let's say 1,000 bits of the same value (0 or 1) the voltage would stay the same for 1,000 clock cycles. At the hardware level this is a problem because the clock signal and synchronization gets lost. So to avoid that, the NICs scramble the data to guaranty a quite small maximum of a few bits of the same value no matter the actual data being sent. Thus the clock signal is always easy to recognize for the hardware. Again the additional 2 bits are there to allow the unscrambling. So if you reach 7.38 Gbits/sec for your data (the payload), it means that 9+ Gb/sec went over the wire but 20% of were used by all these additional for 2 bits.

See https://en.wikipedia.org/wiki/8b/10b_encoding. Disclosure: I am NOT an expert in this matter, far from it, so I'm probably simplifying and might write things that would make a guru jump for its seat! It just happens that I had to read a lot about this to move ahead in my own projects. You can also google for "10Gbase-t 64b/66b". <<< On my side I concluded that consumer NICs (even the $300/$400 top models) only support 8b10b BUT I may be wrong.>>> I AMD WRONG The 10GBase-T standard indicates that 8b/10b and 64b/66b can be supported. However I guess that there is a difference between a standard saying "can" or "must"...

2) 2.6 Gb/sec and 4.6 Gb/sec: can you tell us what the "separate tower" PC is? Is it an onboard 10Gbe NIC, if so which chipset? If it is via a PCIe card, which one is it and what is the type of the PCIe slot it's plugged in (3 or 2 and how many lanes)? CPU and memory speed on this PC? With different machines you will always see a asymmetry between the send/receiving direction. The thing is that 10Gbe is already faster than for example SATA 3 that tops at 6Gb/s so you need "modern" and "performing" hardware all along the chain: NIC, PCIe, CPU, Memory. Also was something running at the same time on the PC and/or the H2. The IP stack consumes CPU resources, so if the CPU is already doing something else, the network will wait.

Best,

Dominique
Last edited by domih on Sun Oct 06, 2019 2:00 pm, edited 1 time in total.
These users thanked the author domih for the post:
MiguelA0145 (Wed Oct 09, 2019 8:51 am)

User avatar
igorpec
Posts: 398
Joined: Sat Dec 12, 2015 4:34 pm
languages_spoken: english,german,slovene
ODROIDs: XU4, HC1, C2, C1+
Has thanked: 9 times
Been thanked: 29 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by igorpec » Sat Oct 05, 2019 6:02 pm

My two cents. It has to work/works with cheap 10GB NICs:

Code: Select all

[  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.55 MBytes       
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.55 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.63 MBytes       
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.71 MBytes       
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.71 MBytes       
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.83 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes       
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes       
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes
Both directions.

Desktop with stock Ubuntu Disco:
Desktop with integrated AQUANTIA® 10 Gigabit LAN (The same chipset as Asus XG-C100C)
Notebook with stock Debian Buster:
Dell XPS 13 external NIC via TB3
Switch, top cables, 3m each.


I also have this Asus card but only tested with RK3399 board. I could reach max 6.6Gbits/s.
These users thanked the author igorpec for the post (total 2):
domih (Sun Oct 06, 2019 2:25 pm) • MiguelA0145 (Wed Oct 09, 2019 8:51 am)
ARMBIAN - follow on Twitter
linux for ARM development boards with user friendly development tools

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Sun Oct 06, 2019 7:57 am

igorpec wrote:
Sat Oct 05, 2019 6:02 pm
My two cents. It has to work/works with cheap 10GB NICs:

Code: Select all

[  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.55 MBytes       
[  5]   1.00-2.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.55 MBytes       
[  5]   2.00-3.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.63 MBytes       
[  5]   3.00-4.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.71 MBytes       
[  5]   4.00-5.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.71 MBytes       
[  5]   5.00-6.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.41 Gbits/sec    0   1.83 MBytes       
[  5]   7.00-8.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes       
[  5]   8.00-9.00   sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes       
[  5]   9.00-10.00  sec  1.10 GBytes  9.42 Gbits/sec    0   1.83 MBytes
Both directions.

Desktop with stock Ubuntu Disco:
Desktop with integrated AQUANTIA® 10 Gigabit LAN (The same chipset as Asus XG-C100C)
Notebook with stock Debian Buster:
Dell XPS 13 external NIC via TB3
Switch, top cables, 3m each.


I also have this Asus card but only tested with RK3399 board. I could reach max 6.6Gbits/s.
Very interesting, you definitely got our attention. Did you do anything in terms of configuration? Do you get more or less the same speed with and without the switch?
These users thanked the author domih for the post:
MiguelA0145 (Wed Oct 09, 2019 8:51 am)

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Sun Oct 06, 2019 2:25 pm

I had a discussion with a friend of mine (he's in hardware) on Friday evening. He said that 8b10b was 'ancient' meaning was still in usage 10 years ago or more but that all "recent" NICs use 64b/66b. So following this discussion and the post from @igorpec I tested again 1GbE NICs between a TR PC, an i5 PC and the H2. All tries showed around 941Mb/sec (0.941 Gb/s) in any direction no matter which of the 3 systems were involved (h2 <-> tr, h2 <-> i5, tr <-> i5). Theoretical max for payload for 1 GbE NICs is just shy of 0.97 Mb/sec (0.97 Gb/s). Conclusion: (a) the 1GbE NICs I just tried are definitely not using 8b/10b but rather 64b/66b (b) In the other giant thread @RomaT also witnessed 8Gb max, short of other explanation I/we concluded it was because of 8b/10b. (c) BUT it does not compute with these new facts.

I can't test "regular" 10GbE NICs on the H2, I chose another path with Mellanox cards (although I think I'll try with the 2nd port in Ethernet mode.)

Is there anybody on planet Earth with an H2 and 2 x PCs able to test the bandwidth using iperf3 between H2<->PC1, H2<->PC2, PC1<->PC2 using onboard 10GbE or cards?

@igorpec: could you test directly from NIC to NIC without the switch in the middle? Just trying to eliminate one variable. And thank you for your post!
Last edited by domih on Sun Oct 06, 2019 6:00 pm, edited 4 times in total.
These users thanked the author domih for the post:
MiguelA0145 (Wed Oct 09, 2019 8:52 am)

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Sun Oct 06, 2019 4:25 pm

I just ran a "pure Ethernet" 10G test between the H2 an a PC with an i5 by configuring my Mellanox cards as Ethernet cards instead of InfiniBand and iperf3 reports 9.4 Gb/sec in one direction and 9.35 Gb/sec in the other (with min=9.01 and max= 9.43). Yes, my hardware is quite different but Ethernet is Ethernet and no RDMA involved here. There is something that does not compute somewhere for @MiguelA0145 and @RomaT, it's the same chipset in all cases and @igorpec gets 9.4 Gb/sec.

Data with the Mellanox cards in "pure" Ethernet mode:

Code: Select all

# On the PC with an i5 - Mellanox ConnectX-3 in Ethernet Mode (IP: 10.10.20.1)
# -----------------------------------------------------------------------------

domih@i5-a:~$ iperf3 -c 10.10.20.2 --bind 10.10.20.1
Connecting to host 10.10.20.2, port 5201
[  4] local 10.10.20.1 port 60407 connected to 10.10.20.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.10 GBytes  9.43 Gbits/sec    0   1.32 MBytes       
[  4]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
[  4]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec    0   1.32 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.40 Gbits/sec                  receiver

iperf Done.
domih@i5-a:~$ ^C
domih@i5-a:~$ iperf3 -s 10.10.20.1
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.10.20.2, port 45123
[  5] local 10.10.20.1 port 5201 connected to 10.10.20.2 port 45173
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  1.05 GBytes  9.01 Gbits/sec                  
[  5]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   2.00-3.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   4.00-5.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   6.00-7.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   7.00-8.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]   9.00-10.00  sec  1.09 GBytes  9.39 Gbits/sec                  
[  5]  10.00-10.04  sec  44.2 MBytes  9.39 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.04  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.04  sec  10.9 GBytes  9.35 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

iperf Done.

# On the Odroid H2- Mellanox ConnectX-2 in Ethernet Mode (IP: 10.10.20.2)
# -----------------------------------------------------------------------------

domih@h2a:~$ iperf3 -s --bind 10.10.20.2
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 10.10.20.1, port 52149
[  5] local 10.10.20.2 port 5201 connected to 10.10.20.1 port 60407
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  1.05 GBytes  9.06 Gbits/sec                  
[  5]   1.00-2.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   3.00-4.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   4.00-5.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   5.00-6.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   6.00-7.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   7.00-8.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   8.00-9.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]   9.00-10.00  sec  1.09 GBytes  9.40 Gbits/sec                  
[  5]  10.00-10.04  sec  40.9 MBytes  9.39 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.04  sec  10.9 GBytes  9.37 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
^Ciperf3: interrupt - the server has terminated
domih@h2a:~$ iperf3 -c 10.10.20.1 --bind 10.10.20.2
Connecting to host 10.10.20.1, port 5201
[  5] local 10.10.20.2 port 45173 connected to 10.10.20.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.09 GBytes  9.40 Gbits/sec    0   1.22 MBytes       
[  5]   1.00-2.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.22 MBytes       
[  5]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.22 MBytes       
[  5]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.43 MBytes       
[  5]   4.00-5.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.43 MBytes       
[  5]   5.00-6.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.43 MBytes       
[  5]   6.00-7.00   sec  1.09 GBytes  9.38 Gbits/sec    0   1.43 MBytes       
[  5]   7.00-8.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.43 MBytes       
[  5]   8.00-9.00   sec  1.09 GBytes  9.39 Gbits/sec    0   1.43 MBytes       
[  5]   9.00-10.00  sec  1.09 GBytes  9.39 Gbits/sec    0   1.43 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  10.9 GBytes  9.39 Gbits/sec                  receiver

iperf Done.
These users thanked the author domih for the post:
MiguelA0145 (Wed Oct 09, 2019 8:52 am)

User avatar
igorpec
Posts: 398
Joined: Sat Dec 12, 2015 4:34 pm
languages_spoken: english,german,slovene
ODROIDs: XU4, HC1, C2, C1+
Has thanked: 9 times
Been thanked: 29 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by igorpec » Sun Oct 06, 2019 5:06 pm

domih wrote:
Sun Oct 06, 2019 7:57 am
Very interesting, you definitely got our attention. Did you do anything in terms of configuration?
IIRC nothing. The only thing I was playing with is MTU but only temporally and because doing some tests with 2.5Gb cards where changing MTU indeed had effects.
domih wrote:
Sun Oct 06, 2019 7:57 am
Do you get more or less the same speed with and without the switch?
Yes, it's the same (top) speed.

I already tested directly few weeks ago with a purpose to see if my existing cabling is suitable for 10Gb networking. My main connection to server is around 20-30m cable and there I could experience lots of dropped packages so the BW was jumping between 7-9Gbits/s. I expected worse since cabling is really bad and there is one extender in the middle.
Is there anybody on planet Earth with an H2
Sadly no H2 around or anything similar.
These users thanked the author igorpec for the post:
MiguelA0145 (Wed Oct 09, 2019 8:52 am)
ARMBIAN - follow on Twitter
linux for ARM development boards with user friendly development tools

back2future
Posts: 240
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 9 times
Been thanked: 4 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by back2future » Sun Oct 06, 2019 8:33 pm

domih wrote:
Sun Oct 06, 2019 4:25 pm
Data with the Mellanox cards in "pure" Ethernet mode:
Would You mind doing a

Code: Select all

ethtool ens817 

Code: Select all

ethtool -i ens817 
Maybe there is something to suggest, while reading hardware settings.

Edit: Thx, speed limit to lowest possible 100baseT could save up to ~2-4W on external 10gbe cards with low demand hours and multiple cards setups on managed LinkAggregation/Teaming.
Don't know what's the power savings for an internal phy.
MII was a pluggable connector standard, while XGMII nowadays is mostly an on-chip connection on pcb, standardized at 156.25 MHz DDR (312.5 MT/s).
Sometimes seen at ethtool after TP. (XAUI can be an Extender to XGMII (<7cm) for up to 50cm at ic-to-ic connections on pcb traces.)
TP is twisted pair copper wiring and up to 30m-100m depending on wiring standards like cat 6a 6e, cat 7a (or cat8.1@2000Mhz for ~30m server room distances and 8P8C connectors) and electrical capabilities of network devices (FIFOs-> MAC->XGMII(XAUI)->PHY(->Xformer)-> cabling). That's probably where arm_SoC or x64_system independent improvements are possible for network speed?

Code: Select all

netstat -i
shows MTU size

Code: Select all

ethtool -S
InDroppedDma (might be an indicator for unusual packet drops)
Last edited by back2future on Mon Oct 07, 2019 2:03 am, edited 7 times in total.
These users thanked the author back2future for the post:
MiguelA0145 (Wed Oct 09, 2019 8:52 am)

RomaT
Posts: 235
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 42 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by RomaT » Sun Oct 06, 2019 10:57 pm

I have like this
.
link.jpg
link.jpg (86.39 KiB) Viewed 805 times
These users thanked the author RomaT for the post:
MiguelA0145 (Wed Oct 09, 2019 8:52 am)

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Mon Oct 07, 2019 4:43 am

You can also dig deeper if not enough.

1. List the sysctl entries with tcp:

root@h2a:~# sysctl -a | grep tcp

2. List the sysctl entries with the name of the NIC

sysctl -a | grep enp2s0

Replace "enp2s0" with the name of your NIC.

3. List and print the sys fs values for the NIC:

find /sys/bus/pci/devices/0000:02:00.0/* -type f \( -exec echo {} \; -o -exec true \; \) -exec cat {} \;

Replace "02:00.0" with the slot of your NIC.

Maybe there is a better way to do this.

In the examples below I'm using the first onboard NIC of the H2. Compare the values between a NIC that goes full speec and a NIC that does not. There should be at least a few differences. You could check the PCI vendor and device id and the driver in the first place. You never know.

Code: Select all

# Use "ip a" to find the name of your NIC

root@h2a:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
.../...
# First onboard H2 1Gb/E NIC:

2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:1e:06:45:0d:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.70/24 brd 192.168.1.255 scope global noprefixroute enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::bdf4:34b7:f1a3:eb1/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
.../...
3: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 00:1e:06:45:0d:48 brd ff:ff:ff:ff:ff:ff
4: ibp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc fq_codel state UP group default qlen 256
    link/infiniband 80:00:02:08:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:ea:45 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
5: ibp1s0d1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state DOWN group default qlen 256
    link/infiniband 80:00:02:09:fe:80:00:00:00:00:00:00:00:02:c9:03:00:10:ea:46 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff

# List the sysctl entries with tcp:

root@h2a:~# sysctl -a | grep tcp
fs.nfs.nlm_tcpport = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = reno cubic
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = reno cubic
net.ipv4.tcp_available_ulp = 
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_comp_sack_delay_ns = 1000000
net.ipv4.tcp_comp_sack_nr = 44
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_demux = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 0
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 3600
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.tcp_limit_output_bytes = 1048576
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 131072
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_max_tw_buckets = 131072
net.ipv4.tcp_mem = 381429	508573	762858
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_snd_mss = 48
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096	131072	6291456
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_reuse = 2
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096	16384	4194304
net.ipv4.tcp_workaround_signed_windows = 0
sunrpc.tcp_fin_timeout = 15
sunrpc.tcp_max_slot_table_entries = 65536
sunrpc.tcp_slot_table_entries = 2
sunrpc.transports = tcp 1048576

# List the sysctl entries with the name of your NIC.

sysctl -a | grep enp2s0

root@h2a:~# sysctl -a | grep enp2s0
net.ipv4.conf.enp2s0.accept_local = 0
net.ipv4.conf.enp2s0.accept_redirects = 1
net.ipv4.conf.enp2s0.accept_source_route = 1
net.ipv4.conf.enp2s0.arp_accept = 0
net.ipv4.conf.enp2s0.arp_announce = 0
net.ipv4.conf.enp2s0.arp_filter = 0
net.ipv4.conf.enp2s0.arp_ignore = 0
net.ipv4.conf.enp2s0.arp_notify = 0
net.ipv4.conf.enp2s0.bc_forwarding = 0
net.ipv4.conf.enp2s0.bootp_relay = 0
net.ipv4.conf.enp2s0.disable_policy = 0
net.ipv4.conf.enp2s0.disable_xfrm = 0
net.ipv4.conf.enp2s0.drop_gratuitous_arp = 0
net.ipv4.conf.enp2s0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.enp2s0.force_igmp_version = 0
net.ipv4.conf.enp2s0.forwarding = 0
net.ipv4.conf.enp2s0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.enp2s0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.enp2s0.ignore_routes_with_linkdown = 0
net.ipv4.conf.enp2s0.log_martians = 0
net.ipv4.conf.enp2s0.mc_forwarding = 0
net.ipv4.conf.enp2s0.medium_id = 0
net.ipv4.conf.enp2s0.promote_secondaries = 0
net.ipv4.conf.enp2s0.proxy_arp = 0
net.ipv4.conf.enp2s0.proxy_arp_pvlan = 0
net.ipv4.conf.enp2s0.route_localnet = 0
net.ipv4.conf.enp2s0.rp_filter = 2
net.ipv4.conf.enp2s0.secure_redirects = 1
net.ipv4.conf.enp2s0.send_redirects = 1
net.ipv4.conf.enp2s0.shared_media = 1
net.ipv4.conf.enp2s0.src_valid_mark = 0
net.ipv4.conf.enp2s0.tag = 0
net.ipv4.neigh.enp2s0.anycast_delay = 100
net.ipv4.neigh.enp2s0.app_solicit = 0
net.ipv4.neigh.enp2s0.base_reachable_time_ms = 30000
net.ipv4.neigh.enp2s0.delay_first_probe_time = 5
net.ipv4.neigh.enp2s0.gc_stale_time = 60
net.ipv4.neigh.enp2s0.locktime = 100
net.ipv4.neigh.enp2s0.mcast_resolicit = 0
net.ipv4.neigh.enp2s0.mcast_solicit = 3
net.ipv4.neigh.enp2s0.proxy_delay = 80
net.ipv4.neigh.enp2s0.proxy_qlen = 64
net.ipv4.neigh.enp2s0.retrans_time_ms = 1000
net.ipv4.neigh.enp2s0.ucast_solicit = 3
net.ipv4.neigh.enp2s0.unres_qlen = 101
net.ipv4.neigh.enp2s0.unres_qlen_bytes = 212992
net.ipv6.conf.enp2s0.accept_dad = 1
net.ipv6.conf.enp2s0.accept_ra = 1
net.ipv6.conf.enp2s0.accept_ra_defrtr = 0
net.ipv6.conf.enp2s0.accept_ra_from_local = 0
net.ipv6.conf.enp2s0.accept_ra_min_hop_limit = 1
net.ipv6.conf.enp2s0.accept_ra_mtu = 1
net.ipv6.conf.enp2s0.accept_ra_pinfo = 0
net.ipv6.conf.enp2s0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp2s0.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.enp2s0.accept_ra_rtr_pref = 0
net.ipv6.conf.enp2s0.accept_redirects = 1
net.ipv6.conf.enp2s0.accept_source_route = 0
net.ipv6.conf.enp2s0.addr_gen_mode = 1
net.ipv6.conf.enp2s0.autoconf = 1
net.ipv6.conf.enp2s0.dad_transmits = 1
net.ipv6.conf.enp2s0.disable_ipv6 = 0
net.ipv6.conf.enp2s0.disable_policy = 0
net.ipv6.conf.enp2s0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.enp2s0.drop_unsolicited_na = 0
net.ipv6.conf.enp2s0.enhanced_dad = 1
net.ipv6.conf.enp2s0.force_mld_version = 0
net.ipv6.conf.enp2s0.force_tllao = 0
net.ipv6.conf.enp2s0.forwarding = 0
net.ipv6.conf.enp2s0.hop_limit = 64
net.ipv6.conf.enp2s0.ignore_routes_with_linkdown = 0
net.ipv6.conf.enp2s0.keep_addr_on_down = 0
net.ipv6.conf.enp2s0.max_addresses = 16
net.ipv6.conf.enp2s0.max_desync_factor = 600
net.ipv6.conf.enp2s0.mc_forwarding = 0
net.ipv6.conf.enp2s0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp2s0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp2s0.mtu = 1500
net.ipv6.conf.enp2s0.ndisc_notify = 0
net.ipv6.conf.enp2s0.ndisc_tclass = 0
net.ipv6.conf.enp2s0.proxy_ndp = 0
net.ipv6.conf.enp2s0.regen_max_retry = 3
net.ipv6.conf.enp2s0.router_probe_interval = 60
net.ipv6.conf.enp2s0.router_solicitation_delay = 1
net.ipv6.conf.enp2s0.router_solicitation_interval = 4
net.ipv6.conf.enp2s0.router_solicitation_max_interval = 3600
net.ipv6.conf.enp2s0.router_solicitations = -1
net.ipv6.conf.enp2s0.seg6_enabled = 0
net.ipv6.conf.enp2s0.seg6_require_hmac = 0
net.ipv6.conf.enp2s0.suppress_frag_ndisc = 1
net.ipv6.conf.enp2s0.temp_prefered_lft = 86400
net.ipv6.conf.enp2s0.temp_valid_lft = 604800
net.ipv6.conf.enp2s0.use_oif_addrs_only = 0
net.ipv6.conf.enp2s0.use_tempaddr = 2
net.ipv6.neigh.enp2s0.anycast_delay = 100
net.ipv6.neigh.enp2s0.app_solicit = 0
net.ipv6.neigh.enp2s0.base_reachable_time_ms = 30000
net.ipv6.neigh.enp2s0.delay_first_probe_time = 5
net.ipv6.neigh.enp2s0.gc_stale_time = 60
net.ipv6.neigh.enp2s0.locktime = 0
net.ipv6.neigh.enp2s0.mcast_resolicit = 0
net.ipv6.neigh.enp2s0.mcast_solicit = 3
net.ipv6.neigh.enp2s0.proxy_delay = 80
net.ipv6.neigh.enp2s0.proxy_qlen = 64
net.ipv6.neigh.enp2s0.retrans_time_ms = 1000
net.ipv6.neigh.enp2s0.ucast_solicit = 3
net.ipv6.neigh.enp2s0.unres_qlen = 101
net.ipv6.neigh.enp2s0.unres_qlen_bytes = 212992

# Use "lspci" to identify the PCI id of your NIC.

root@h2a:~# lspci
00:00.0 Host bridge: Intel Corporation Device 31f0 (rev 03)
00:02.0 VGA compatible controller: Intel Corporation Device 3185 (rev 03)
00:0e.0 Audio device: Intel Corporation Device 3198 (rev 03)
00:0f.0 Communication controller: Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface (rev 03)
00:12.0 SATA controller: Intel Corporation Device 31e3 (rev 03)
00:13.0 PCI bridge: Intel Corporation Device 31d8 (rev f3)
00:14.0 PCI bridge: Intel Corporation Device 31d6 (rev f3)
00:14.1 PCI bridge: Intel Corporation Device 31d7 (rev f3)
00:15.0 USB controller: Intel Corporation Device 31a8 (rev 03)
00:17.0 Signal processing controller: Intel Corporation Device 31b4 (rev 03)
00:17.1 Signal processing controller: Intel Corporation Device 31b6 (rev 03)
00:17.2 Signal processing controller: Intel Corporation Device 31b8 (rev 03)
00:17.3 Signal processing controller: Intel Corporation Device 31ba (rev 03)
00:1c.0 SD Host controller: Intel Corporation Device 31cc (rev 03)
00:1f.0 ISA bridge: Intel Corporation Device 31e8 (rev 03)
00:1f.1 SMBus: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
01:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev b0)
.../...
# First onboard H2 1Gb/E NIC:

02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
.../...
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)

# List an print the sys fs values for your NIC.
# In this example, I'm listing those of the first H2 onboard NIC.
# Note: even as root there will be a few errors.

find /sys/bus/pci/devices/0000:02:00.0/* -type f \( -exec echo {} \; -o -exec true \; \)  -exec cat {} \;


root@h2a:~# find /sys/bus/pci/devices/0000:02:00.0/* -type f \( -exec echo {} \; -o -exec true \; \)  -exec cat {} \;
/sys/bus/pci/devices/0000:02:00.0/aer_dev_correctable
RxErr 0
BadTLP 0
BadDLLP 0
Rollover 0
Timeout 0
NonFatalErr 0
CorrIntErr 0
HeaderOF 0
TOTAL_ERR_COR 0
/sys/bus/pci/devices/0000:02:00.0/aer_dev_fatal
Undefined 0
DLP 0
SDES 0
TLP 0
FCP 0
CmpltTO 0
CmpltAbrt 0
UnxCmplt 0
RxOF 0
MalfTLP 0
ECRC 0
UnsupReq 0
ACSViol 0
UncorrIntErr 0
BlockedTLP 0
AtomicOpBlocked 0
TLPBlockedErr 0
TOTAL_ERR_FATAL 0
/sys/bus/pci/devices/0000:02:00.0/aer_dev_nonfatal
Undefined 0
DLP 0
SDES 0
TLP 0
FCP 0
CmpltTO 0
CmpltAbrt 0
UnxCmplt 0
RxOF 0
MalfTLP 0
ECRC 0
UnsupReq 0
ACSViol 0
UncorrIntErr 0
BlockedTLP 0
AtomicOpBlocked 0
TLPBlockedErr 0
TOTAL_ERR_NONFATAL 0
/sys/bus/pci/devices/0000:02:00.0/ari_enabled
0
/sys/bus/pci/devices/0000:02:00.0/broken_parity_status
0
/sys/bus/pci/devices/0000:02:00.0/class
0x020000
/sys/bus/pci/devices/0000:02:00.0/config
.../... removed, binary value .../...
/sys/bus/pci/devices/0000:02:00.0/consistent_dma_mask_bits
64
/sys/bus/pci/devices/0000:02:00.0/current_link_speed
2.5 GT/s
/sys/bus/pci/devices/0000:02:00.0/current_link_width
1
/sys/bus/pci/devices/0000:02:00.0/d3cold_allowed
1
/sys/bus/pci/devices/0000:02:00.0/device
0x8168
/sys/bus/pci/devices/0000:02:00.0/dma_mask_bits
64
/sys/bus/pci/devices/0000:02:00.0/driver_override
(null)
/sys/bus/pci/devices/0000:02:00.0/enable
1
/sys/bus/pci/devices/0000:02:00.0/irq
22
/sys/bus/pci/devices/0000:02:00.0/local_cpulist
0-3
/sys/bus/pci/devices/0000:02:00.0/local_cpus
f
/sys/bus/pci/devices/0000:02:00.0/max_link_speed
2.5 GT/s
/sys/bus/pci/devices/0000:02:00.0/max_link_width
1
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/uevent
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/runtime_active_time
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/runtime_active_kids
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/runtime_usage
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/runtime_status
unsupported
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/autosuspend_delay_ms
cat: '/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/autosuspend_delay_ms': Input/output error
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/async
disabled
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/runtime_suspended_time
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/runtime_enabled
disabled
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/power/control
auto
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/uevent
DEVTYPE=PHY
DRIVER=Generic PHY
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/phy_interface
gmii
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/phy_id
0x001cc800
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/runtime_active_time
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/runtime_active_kids
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/runtime_usage
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/runtime_status
unsupported
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/autosuspend_delay_ms
cat: '/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/autosuspend_delay_ms': Input/output error
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/async
disabled
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/runtime_suspended_time
0
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/runtime_enabled
disabled
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/power/control
auto
/sys/bus/pci/devices/0000:02:00.0/mdio_bus/r8169-200/r8169-200:00/phy_has_fixups
0
/sys/bus/pci/devices/0000:02:00.0/modalias
pci:v000010ECd00008168sv000010ECsd00000123bc02sc00i00
/sys/bus/pci/devices/0000:02:00.0/msi_bus
1
/sys/bus/pci/devices/0000:02:00.0/msi_irqs/127
msix
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/uevent
INTERFACE=enp2s0
IFINDEX=2
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/carrier_changes
2
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/carrier
1
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/dev_id
0x0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/carrier_down_count
1
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/proto_down
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/address
00:1e:06:45:0d:47
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/operstate
up
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/phys_switch_id
cat: '/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/phys_switch_id': Operation not supported
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/link_mode
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/dormant
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_length_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_packets
53127
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_carrier_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_dropped
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_missed_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_over_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_aborted_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_crc_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_frame_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_nohandler
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_fifo_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/multicast
577
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_packets
30579
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_window_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_bytes
73318866
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/collisions
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_dropped
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_bytes
2433998
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_heartbeat_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_fifo_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_errors
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/tx_compressed
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/statistics/rx_compressed
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/mtu
1500
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/gro_flush_timeout
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/runtime_active_time
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/runtime_active_kids
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/runtime_usage
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/runtime_status
unsupported
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/autosuspend_delay_ms
cat: '/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/autosuspend_delay_ms': Input/output error
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/async
disabled
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/runtime_suspended_time
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/runtime_enabled
disabled
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/power/control
auto
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/carrier_up_count
1
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/ifalias
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/speed
1000
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/netdev_group
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/phys_port_name
cat: '/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/phys_port_name': Operation not supported
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/ifindex
2
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/broadcast
ff:ff:ff:ff:ff:ff
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/type
1
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/dev_port
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/phys_port_id
cat: '/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/phys_port_id': Operation not supported
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/tx_maxrate
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/xps_cpus
cat: '/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/xps_cpus': No such file or directory
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/byte_queue_limits/limit_min
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/byte_queue_limits/limit
4578
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/byte_queue_limits/limit_max
1879048192
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/byte_queue_limits/hold_time
1000
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/byte_queue_limits/inflight
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/tx_timeout
0/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/xps_rxqs
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/traffic_class
cat: '/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/tx-0/traffic_class': No such file or directory
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/rx-0/rps_flow_cnt
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/queues/rx-0/rps_cpus
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/name_assign_type
4
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/duplex
full
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/addr_assign_type
0
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/addr_len
6
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/tx_queue_len
1000
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/iflink
2
/sys/bus/pci/devices/0000:02:00.0/net/enp2s0/flags
0x1003
/sys/bus/pci/devices/0000:02:00.0/numa_node
-1
/sys/bus/pci/devices/0000:02:00.0/power/runtime_active_time
2948028
/sys/bus/pci/devices/0000:02:00.0/power/wakeup
disabled
/sys/bus/pci/devices/0000:02:00.0/power/wakeup_count

/sys/bus/pci/devices/0000:02:00.0/power/wakeup_abort_count

/sys/bus/pci/devices/0000:02:00.0/power/wakeup_expire_count

/sys/bus/pci/devices/0000:02:00.0/power/runtime_active_kids
0
/sys/bus/pci/devices/0000:02:00.0/power/wakeup_total_time_ms

/sys/bus/pci/devices/0000:02:00.0/power/wakeup_active_count

/sys/bus/pci/devices/0000:02:00.0/power/runtime_usage
1
/sys/bus/pci/devices/0000:02:00.0/power/runtime_status
active
/sys/bus/pci/devices/0000:02:00.0/power/autosuspend_delay_ms
cat: '/sys/bus/pci/devices/0000:02:00.0/power/autosuspend_delay_ms': Input/output error
/sys/bus/pci/devices/0000:02:00.0/power/async
enabled
/sys/bus/pci/devices/0000:02:00.0/power/runtime_suspended_time
0
/sys/bus/pci/devices/0000:02:00.0/power/wakeup_max_time_ms

/sys/bus/pci/devices/0000:02:00.0/power/wakeup_active

/sys/bus/pci/devices/0000:02:00.0/power/runtime_enabled
forbidden
/sys/bus/pci/devices/0000:02:00.0/power/control
on
/sys/bus/pci/devices/0000:02:00.0/power/wakeup_last_time_ms

/sys/bus/pci/devices/0000:02:00.0/remove
cat: '/sys/bus/pci/devices/0000:02:00.0/remove': Permission denied
/sys/bus/pci/devices/0000:02:00.0/rescan
cat: '/sys/bus/pci/devices/0000:02:00.0/rescan': Permission denied
/sys/bus/pci/devices/0000:02:00.0/reset
cat: '/sys/bus/pci/devices/0000:02:00.0/reset': Permission denied
/sys/bus/pci/devices/0000:02:00.0/resource
0x000000000000e000 0x000000000000e0ff 0x0000000000040101
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000a1b04000 0x00000000a1b04fff 0x0000000000140204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000a1b00000 0x00000000a1b03fff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
/sys/bus/pci/devices/0000:02:00.0/resource0
cat: '/sys/bus/pci/devices/0000:02:00.0/resource0': Invalid argument
/sys/bus/pci/devices/0000:02:00.0/resource2
cat: '/sys/bus/pci/devices/0000:02:00.0/resource2': Input/output error
/sys/bus/pci/devices/0000:02:00.0/resource4
cat: '/sys/bus/pci/devices/0000:02:00.0/resource4': Input/output error
/sys/bus/pci/devices/0000:02:00.0/resource4_wc
cat: '/sys/bus/pci/devices/0000:02:00.0/resource4_wc': Input/output error
/sys/bus/pci/devices/0000:02:00.0/revision
0x0c
/sys/bus/pci/devices/0000:02:00.0/subsystem_device
0x0123
/sys/bus/pci/devices/0000:02:00.0/subsystem_vendor
0x10ec
/sys/bus/pci/devices/0000:02:00.0/uevent
DRIVER=r8169
PCI_CLASS=20000
PCI_ID=10EC:8168
PCI_SUBSYS_ID=10EC:0123
PCI_SLOT_NAME=0000:02:00.0
MODALIAS=pci:v000010ECd00008168sv000010ECsd00000123bc02sc00i00
/sys/bus/pci/devices/0000:02:00.0/vendor
0x10ec
/sys/bus/pci/devices/0000:02:00.0/vpd
cat: '/sys/bus/pci/devices/0000:02:00.0/vpd': Input/output error
These users thanked the author domih for the post:
MiguelA0145 (Wed Oct 09, 2019 8:52 am)

brad
Posts: 969
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 16 times
Been thanked: 34 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by brad » Mon Oct 07, 2019 4:28 pm

Which kernel versions are each of you guys using on your machines? Some significant changes to Aquantia drivers have occurred since 4.9 which may impact performance.
These users thanked the author brad for the post:
MiguelA0145 (Wed Oct 09, 2019 8:51 am)

MiguelA0145
Posts: 4
Joined: Sat Oct 05, 2019 1:34 am
languages_spoken: English
ODROIDs: ODROID C2, ODROID H2
Has thanked: 11 times
Been thanked: 0
Contact:

Re: Troubles with 10GBe on H2

Unread post by MiguelA0145 » Wed Oct 09, 2019 1:23 am

Hey guys, sorry for the late reply.

To start off, the hardware on that separate tower I mentioned is as follows :
  • CPU : I3-4160
  • RAM : 4GB Ballistix 1600 MHz (x2)
  • NIC : ASUS XC-C100C (Same as H2) in the x16 lane on the mobo
  • OS : Windows 10
Since I forgot to say this in the first post, my current setup has both the PC and H2 connected via a 10gig switch.

Here are the results for the commands that I saw :

Code: Select all

root@ODROID-H2:/home/odroid# ethtool enp1s0
Settings for enp1s0:
        Supported ports: [ TP ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: g
        Wake-on: d
        Link detected: no
        
----------------------------------------------------------

root@ODROID-H2:/home/odroid# ethtool -i enp1s0
driver: atlantic
version: 2.0.4.0-kern
firmware-version: 3.0.33
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

----------------------------------------------------------

root@ODROID-H2:/home/odroid# netstat -i
Kernel Interface table
Iface      MTU    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg
enp1s0   9600     216       0      0 0       38            0      0      0 BMU
enp2s0   1500     0           0      0 0        0             0      0      0 BMU
enp3s0   1500     3978      0      0 0        610         0      0      0 BMRU
lo           65536   614320   0      0 0        614320   0      0      0 LRU

----------------------------------------------------------

root@ODROID-H2:/home/odroid# ethtool -S enp1s0
NIC statistics:
     InPackets: 216
     InUCast: 1
     InMCast: 35
     InBCast: 180
     InErrors: 0
     OutPackets: 38
     OutUCast: 5
     OutMCast: 32
     OutBCast: 1
     InUCastOctets: 64
     OutUCastOctets: 390
     InMCastOctets: 4201
     OutMCastOctets: 5888
     InBCastOctets: 11819
     OutBCastOctets: 64
     InOctets: 16084
     OutOctets: 6342
     InPacketsDma: 0
     OutPacketsDma: 0
     InOctetsDma: 0
     OutOctetsDma: 0
     InDroppedDma: 0
     Queue[0] InPackets: 0
     Queue[0] OutPackets: 0
     Queue[0] Restarts: 0
     Queue[0] InJumboPackets: 0
     Queue[0] InLroPackets: 0
     Queue[0] InErrors: 0
     Queue[1] InPackets: 0
     Queue[1] OutPackets: 0
     Queue[1] Restarts: 0
     Queue[1] InJumboPackets: 0
     Queue[1] InLroPackets: 0
     Queue[1] InErrors: 0
     Queue[2] InPackets: 0
     Queue[2] OutPackets: 0
     Queue[2] Restarts: 0
     Queue[2] InJumboPackets: 0
     Queue[2] InLroPackets: 0
     Queue[2] InErrors: 0
     Queue[3] InPackets: 0
     Queue[3] OutPackets: 0
     Queue[3] Restarts: 0
     Queue[3] InJumboPackets: 0
     Queue[3] InLroPackets: 0
     Queue[3] InErrors: 0

Code: Select all

root@ODROID-H2:/home/odroid# sysctl -a | grep tcp
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = reno cubic htcp
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = reno cubic htcp
net.ipv4.tcp_available_ulp =
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_comp_sack_delay_ns = 1000000
net.ipv4.tcp_comp_sack_nr = 44
net.ipv4.tcp_congestion_control = htcp
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_demux = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 0
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 3600
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 15
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.tcp_limit_output_bytes = 1048576
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 16384
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 128
net.ipv4.tcp_max_tw_buckets = 16384
net.ipv4.tcp_mem = 43404        57874   86808
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_snd_mss = 48
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 1
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096        87380   134217728
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 0
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_reuse = 2
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096        65536   134217728
net.ipv4.tcp_workaround_signed_windows = 0

----------------------------------------------------------

root@ODROID-H2:/home/odroid# sysctl -a | grep enp1s0
net.ipv4.conf.enp1s0.accept_local = 0
net.ipv4.conf.enp1s0.accept_redirects = 1
net.ipv4.conf.enp1s0.accept_source_route = 1
net.ipv4.conf.enp1s0.arp_accept = 0
net.ipv4.conf.enp1s0.arp_announce = 0
net.ipv4.conf.enp1s0.arp_filter = 0
net.ipv4.conf.enp1s0.arp_ignore = 0
net.ipv4.conf.enp1s0.arp_notify = 0
net.ipv4.conf.enp1s0.bc_forwarding = 0
net.ipv4.conf.enp1s0.bootp_relay = 0
net.ipv4.conf.enp1s0.disable_policy = 0
net.ipv4.conf.enp1s0.disable_xfrm = 0
net.ipv4.conf.enp1s0.drop_gratuitous_arp = 0
net.ipv4.conf.enp1s0.drop_unicast_in_l2_multicast = 0
net.ipv4.conf.enp1s0.force_igmp_version = 0
net.ipv4.conf.enp1s0.forwarding = 0
net.ipv4.conf.enp1s0.igmpv2_unsolicited_report_interval = 10000
net.ipv4.conf.enp1s0.igmpv3_unsolicited_report_interval = 1000
net.ipv4.conf.enp1s0.ignore_routes_with_linkdown = 0
net.ipv4.conf.enp1s0.log_martians = 0
net.ipv4.conf.enp1s0.mc_forwarding = 0
net.ipv4.conf.enp1s0.medium_id = 0
net.ipv4.conf.enp1s0.promote_secondaries = 0
net.ipv4.conf.enp1s0.proxy_arp = 0
net.ipv4.conf.enp1s0.proxy_arp_pvlan = 0
net.ipv4.conf.enp1s0.route_localnet = 0
net.ipv4.conf.enp1s0.rp_filter = 2
net.ipv4.conf.enp1s0.secure_redirects = 1
net.ipv4.conf.enp1s0.send_redirects = 1
net.ipv4.conf.enp1s0.shared_media = 1
net.ipv4.conf.enp1s0.src_valid_mark = 0
net.ipv4.conf.enp1s0.tag = 0
net.ipv4.neigh.enp1s0.anycast_delay = 100
net.ipv4.neigh.enp1s0.app_solicit = 0
net.ipv4.neigh.enp1s0.base_reachable_time_ms = 30000
net.ipv4.neigh.enp1s0.delay_first_probe_time = 5
net.ipv4.neigh.enp1s0.gc_stale_time = 60
net.ipv4.neigh.enp1s0.locktime = 100
net.ipv4.neigh.enp1s0.mcast_resolicit = 0
net.ipv4.neigh.enp1s0.mcast_solicit = 3
net.ipv4.neigh.enp1s0.proxy_delay = 80
net.ipv4.neigh.enp1s0.proxy_qlen = 64
net.ipv4.neigh.enp1s0.retrans_time_ms = 1000
net.ipv4.neigh.enp1s0.ucast_solicit = 3
net.ipv4.neigh.enp1s0.unres_qlen = 101
net.ipv4.neigh.enp1s0.unres_qlen_bytes = 212992
net.ipv6.conf.enp1s0.accept_dad = 1
net.ipv6.conf.enp1s0.accept_ra = 0
net.ipv6.conf.enp1s0.accept_ra_defrtr = 0
net.ipv6.conf.enp1s0.accept_ra_from_local = 0
net.ipv6.conf.enp1s0.accept_ra_min_hop_limit = 1
net.ipv6.conf.enp1s0.accept_ra_mtu = 1
net.ipv6.conf.enp1s0.accept_ra_pinfo = 0
net.ipv6.conf.enp1s0.accept_ra_rt_info_max_plen = 0
net.ipv6.conf.enp1s0.accept_ra_rt_info_min_plen = 0
net.ipv6.conf.enp1s0.accept_ra_rtr_pref = 0
net.ipv6.conf.enp1s0.accept_redirects = 1
net.ipv6.conf.enp1s0.accept_source_route = 0
net.ipv6.conf.enp1s0.addr_gen_mode = 1
net.ipv6.conf.enp1s0.autoconf = 1
net.ipv6.conf.enp1s0.dad_transmits = 1
net.ipv6.conf.enp1s0.disable_ipv6 = 0
net.ipv6.conf.enp1s0.disable_policy = 0
net.ipv6.conf.enp1s0.drop_unicast_in_l2_multicast = 0
net.ipv6.conf.enp1s0.drop_unsolicited_na = 0
net.ipv6.conf.enp1s0.enhanced_dad = 1
net.ipv6.conf.enp1s0.force_mld_version = 0
net.ipv6.conf.enp1s0.force_tllao = 0
net.ipv6.conf.enp1s0.forwarding = 0
net.ipv6.conf.enp1s0.hop_limit = 64
net.ipv6.conf.enp1s0.ignore_routes_with_linkdown = 0
net.ipv6.conf.enp1s0.keep_addr_on_down = 0
net.ipv6.conf.enp1s0.max_addresses = 16
net.ipv6.conf.enp1s0.max_desync_factor = 600
net.ipv6.conf.enp1s0.mc_forwarding = 0
net.ipv6.conf.enp1s0.mldv1_unsolicited_report_interval = 10000
net.ipv6.conf.enp1s0.mldv2_unsolicited_report_interval = 1000
net.ipv6.conf.enp1s0.mtu = 9600
net.ipv6.conf.enp1s0.ndisc_notify = 0
net.ipv6.conf.enp1s0.ndisc_tclass = 0
net.ipv6.conf.enp1s0.proxy_ndp = 0
net.ipv6.conf.enp1s0.regen_max_retry = 3
net.ipv6.conf.enp1s0.router_probe_interval = 60
net.ipv6.conf.enp1s0.router_solicitation_delay = 1
net.ipv6.conf.enp1s0.router_solicitation_interval = 4
net.ipv6.conf.enp1s0.router_solicitation_max_interval = 3600
net.ipv6.conf.enp1s0.router_solicitations = -1
net.ipv6.conf.enp1s0.seg6_enabled = 0
net.ipv6.conf.enp1s0.seg6_require_hmac = 0
net.ipv6.conf.enp1s0.suppress_frag_ndisc = 1
net.ipv6.conf.enp1s0.temp_prefered_lft = 86400
net.ipv6.conf.enp1s0.temp_valid_lft = 604800
net.ipv6.conf.enp1s0.use_oif_addrs_only = 0
net.ipv6.conf.enp1s0.use_tempaddr = 0
net.ipv6.neigh.enp1s0.anycast_delay = 100
net.ipv6.neigh.enp1s0.app_solicit = 0
net.ipv6.neigh.enp1s0.base_reachable_time_ms = 30000
net.ipv6.neigh.enp1s0.delay_first_probe_time = 5
net.ipv6.neigh.enp1s0.gc_stale_time = 60
net.ipv6.neigh.enp1s0.locktime = 0
net.ipv6.neigh.enp1s0.mcast_resolicit = 0
net.ipv6.neigh.enp1s0.mcast_solicit = 3
net.ipv6.neigh.enp1s0.proxy_delay = 80
net.ipv6.neigh.enp1s0.proxy_qlen = 64
net.ipv6.neigh.enp1s0.retrans_time_ms = 1000
net.ipv6.neigh.enp1s0.ucast_solicit = 3
net.ipv6.neigh.enp1s0.unres_qlen = 101
net.ipv6.neigh.enp1s0.unres_qlen_bytes = 212992

----------------------------------------------------------

root@ODROID-H2:/home/odroid# find /sys/bus/pci/devices/0000:01:00.0/* -type f \( -exec echo {} \; -o -exec true \; \) -exec cat {} \;
/sys/bus/pci/devices/0000:01:00.0/aer_dev_correctable
RxErr 0
BadTLP 0
BadDLLP 0
Rollover 0
Timeout 0
NonFatalErr 0
CorrIntErr 0
HeaderOF 0
TOTAL_ERR_COR 0
/sys/bus/pci/devices/0000:01:00.0/aer_dev_fatal
Undefined 0
DLP 0
SDES 0
TLP 0
FCP 0
CmpltTO 0
CmpltAbrt 0
UnxCmplt 0
RxOF 0
MalfTLP 0
ECRC 0
UnsupReq 0
ACSViol 0
UncorrIntErr 0
BlockedTLP 0
AtomicOpBlocked 0
TLPBlockedErr 0
TOTAL_ERR_FATAL 0
/sys/bus/pci/devices/0000:01:00.0/aer_dev_nonfatal
Undefined 0
DLP 0
SDES 0
TLP 0
FCP 0
CmpltTO 0
CmpltAbrt 0
UnxCmplt 0
RxOF 0
MalfTLP 0
ECRC 0
UnsupReq 0
ACSViol 0
UncorrIntErr 0
BlockedTLP 0
AtomicOpBlocked 0
TLPBlockedErr 0
TOTAL_ERR_NONFATAL 0
/sys/bus/pci/devices/0000:01:00.0/ari_enabled
0
/sys/bus/pci/devices/0000:01:00.0/broken_parity_status
0
/sys/bus/pci/devices/0000:01:00.0/class
0x020000
/sys/bus/pci/devices/0000:01:00.0/config
j       D     E              CA  @@          "0) CC @ B                                                                                                                                                        
0 F                                                                 A                                                $$$$
/sys/bus/pci/devices/0000:01:00.0/consistent_dma_mask_bits
64
/sys/bus/pci/devices/0000:01:00.0/current_link_speed
5 GT/s
/sys/bus/pci/devices/0000:01:00.0/current_link_width
4
/sys/bus/pci/devices/0000:01:00.0/d3cold_allowed
1
/sys/bus/pci/devices/0000:01:00.0/device
0x07b1
/sys/bus/pci/devices/0000:01:00.0/dma_mask_bits
64
/sys/bus/pci/devices/0000:01:00.0/driver_override
(null)
/sys/bus/pci/devices/0000:01:00.0/enable
1
/sys/bus/pci/devices/0000:01:00.0/irq
22
/sys/bus/pci/devices/0000:01:00.0/local_cpulist
0-3
/sys/bus/pci/devices/0000:01:00.0/local_cpus
f
/sys/bus/pci/devices/0000:01:00.0/max_link_speed
8 GT/s
/sys/bus/pci/devices/0000:01:00.0/max_link_width
4
/sys/bus/pci/devices/0000:01:00.0/modalias
pci:v00001D6Ad000007B1sv00001043sd00008741bc02sc00i00
/sys/bus/pci/devices/0000:01:00.0/msi_bus
1
/sys/bus/pci/devices/0000:01:00.0/msi_irqs/126
msix
/sys/bus/pci/devices/0000:01:00.0/msi_irqs/129
msix
/sys/bus/pci/devices/0000:01:00.0/msi_irqs/127
msix
/sys/bus/pci/devices/0000:01:00.0/msi_irqs/128
msix
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/uevent
INTERFACE=enp1s0
IFINDEX=4
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/carrier_changes
2
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/carrier
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/dev_id
0x0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/carrier_down_count
1
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/proto_down
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/address
04:d4:c4:63:83:43
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/operstate
down
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/phys_switch_id
cat: '/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/phys_switch_id': Operation not supported
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/link_mode
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/dormant
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_length_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_packets
216
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_carrier_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_dropped
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_missed_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_over_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_aborted_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_crc_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_frame_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_nohandler
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_fifo_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/multicast
35
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_packets
38
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_window_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_bytes
16084
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/collisions
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_dropped
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_bytes
6342
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_heartbeat_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_fifo_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_errors
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/tx_compressed
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/statistics/rx_compressed
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/mtu
9600
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/gro_flush_timeout
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/runtime_active_time
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/runtime_active_kids
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/runtime_usage
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/runtime_status
unsupported
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/autosuspend_delay_ms
cat: '/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/autosuspend_delay_ms': Input/output error
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/async
disabled
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/runtime_suspended_time
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/runtime_enabled
disabled
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/power/control
auto
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/carrier_up_count
1
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/ifalias
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/speed
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/netdev_group
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/phys_port_name
cat: '/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/phys_port_name': Operation not supported
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/ifindex
4
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/broadcast
ff:ff:ff:ff:ff:ff
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/type
1
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/dev_port
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/phys_port_id
cat: '/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/phys_port_id': Operation not supported
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/tx_maxrate
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/xps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/byte_queue_limits/limit_min
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/byte_queue_limits/limit
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/byte_queue_limits/limit_max
1879048192
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/byte_queue_limits/hold_time
1000
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/byte_queue_limits/inflight
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/tx_timeout
0/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/xps_rxqs
00
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-2/traffic_class
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/tx_maxrate
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/xps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/byte_queue_limits/limit_min
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/byte_queue_limits/limit
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/byte_queue_limits/limit_max
1879048192
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/byte_queue_limits/hold_time
1000
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/byte_queue_limits/inflight
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/tx_timeout
0/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/xps_rxqs
00
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-0/traffic_class
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-3/rps_flow_cnt
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-3/rps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-1/rps_flow_cnt
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-1/rps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/tx_maxrate
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/xps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/byte_queue_limits/limit_min
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/byte_queue_limits/limit
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/byte_queue_limits/limit_max
1879048192
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/byte_queue_limits/hold_time
1000
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/byte_queue_limits/inflight
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/tx_timeout
0/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/xps_rxqs
00
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-3/traffic_class
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/tx_maxrate
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/xps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/byte_queue_limits/limit_min
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/byte_queue_limits/limit
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/byte_queue_limits/limit_max
1879048192
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/byte_queue_limits/hold_time
1000
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/byte_queue_limits/inflight
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/tx_timeout
0/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/xps_rxqs
00
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/tx-1/traffic_class
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-2/rps_flow_cnt
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-2/rps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-0/rps_flow_cnt
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/queues/rx-0/rps_cpus
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/name_assign_type
4
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/duplex
full
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/addr_assign_type
0
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/addr_len
6
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/tx_queue_len
13888
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/iflink
4
/sys/bus/pci/devices/0000:01:00.0/net/enp1s0/flags
0x1003
/sys/bus/pci/devices/0000:01:00.0/numa_node
-1
/sys/bus/pci/devices/0000:01:00.0/power/runtime_active_time
2402124
/sys/bus/pci/devices/0000:01:00.0/power/wakeup
disabled
/sys/bus/pci/devices/0000:01:00.0/power/wakeup_count

/sys/bus/pci/devices/0000:01:00.0/power/wakeup_abort_count

/sys/bus/pci/devices/0000:01:00.0/power/wakeup_expire_count

/sys/bus/pci/devices/0000:01:00.0/power/runtime_active_kids
0
/sys/bus/pci/devices/0000:01:00.0/power/wakeup_total_time_ms

/sys/bus/pci/devices/0000:01:00.0/power/wakeup_active_count

/sys/bus/pci/devices/0000:01:00.0/power/runtime_usage
2
/sys/bus/pci/devices/0000:01:00.0/power/runtime_status
active
/sys/bus/pci/devices/0000:01:00.0/power/autosuspend_delay_ms
cat: '/sys/bus/pci/devices/0000:01:00.0/power/autosuspend_delay_ms': Input/output error
/sys/bus/pci/devices/0000:01:00.0/power/async
enabled
/sys/bus/pci/devices/0000:01:00.0/power/runtime_suspended_time
0
/sys/bus/pci/devices/0000:01:00.0/power/wakeup_max_time_ms

/sys/bus/pci/devices/0000:01:00.0/power/wakeup_active

/sys/bus/pci/devices/0000:01:00.0/power/runtime_enabled
forbidden
/sys/bus/pci/devices/0000:01:00.0/power/control
on
/sys/bus/pci/devices/0000:01:00.0/power/wakeup_last_time_ms

/sys/bus/pci/devices/0000:01:00.0/remove
cat: '/sys/bus/pci/devices/0000:01:00.0/remove': Permission denied
/sys/bus/pci/devices/0000:01:00.0/rescan
cat: '/sys/bus/pci/devices/0000:01:00.0/rescan': Permission denied
/sys/bus/pci/devices/0000:01:00.0/reset
cat: '/sys/bus/pci/devices/0000:01:00.0/reset': Permission denied
/sys/bus/pci/devices/0000:01:00.0/resource
0x00000000a1440000 0x00000000a144ffff 0x0000000000140204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000a1450000 0x00000000a1450fff 0x0000000000140204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000a1000000 0x00000000a13fffff 0x0000000000140204
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x00000000a1400000 0x00000000a143ffff 0x0000000000046200
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
/sys/bus/pci/devices/0000:01:00.0/resource0
cat: '/sys/bus/pci/devices/0000:01:00.0/resource0': Input/output error
/sys/bus/pci/devices/0000:01:00.0/resource2
cat: '/sys/bus/pci/devices/0000:01:00.0/resource2': Input/output error
/sys/bus/pci/devices/0000:01:00.0/resource4
cat: '/sys/bus/pci/devices/0000:01:00.0/resource4': Input/output error
/sys/bus/pci/devices/0000:01:00.0/revision
0x02
/sys/bus/pci/devices/0000:01:00.0/rom
cat: '/sys/bus/pci/devices/0000:01:00.0/rom': Invalid argument
/sys/bus/pci/devices/0000:01:00.0/subsystem_device
0x8741
/sys/bus/pci/devices/0000:01:00.0/subsystem_vendor
0x1043
/sys/bus/pci/devices/0000:01:00.0/uevent
DRIVER=atlantic
PCI_CLASS=20000
PCI_ID=1D6A:07B1
PCI_SUBSYS_ID=1043:8741
PCI_SLOT_NAME=0000:01:00.0
MODALIAS=pci:v00001D6Ad000007B1sv00001043sd00008741bc02sc00i00
/sys/bus/pci/devices/0000:01:00.0/vendor
0x1d6a
And last but not least, the kernal version that "uname -r" lists : 5.0.0-29-generic
Last edited by MiguelA0145 on Wed Oct 09, 2019 8:31 am, edited 1 time in total.

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Wed Oct 09, 2019 7:13 am

Without looking into all the details

1) ethtool reports Speed: Unknown! while for @RomaT it reports Speed: 10000Mb/s. Note that @RomaT also experiences speeds less than the expected 9.4 Gb/s.
2) Question: netstat -i reports an MTU of 9600, did you set it yourself or was is the default? What's the MTU on the PC? Can you try with a MTU set to 1500 on both sides? [no particular expectations though] What's running on the other PC, Linux or Windows?

Now it would be great if @igorpec could send the same data and then we can compare field by field.
These users thanked the author domih for the post:
MiguelA0145 (Wed Oct 09, 2019 8:52 am)

MiguelA0145
Posts: 4
Joined: Sat Oct 05, 2019 1:34 am
languages_spoken: English
ODROIDs: ODROID C2, ODROID H2
Has thanked: 11 times
Been thanked: 0
Contact:

Re: Troubles with 10GBe on H2

Unread post by MiguelA0145 » Wed Oct 09, 2019 8:51 am

I had noticed that Unknown field as well! I really am not sure why it reports that, but regardless, I've updated the last post to include the OS, which is Windows 10.

As for the MTU size, I had set that myself as well as a few other parameters in the kernel. I followed one guide linked in the previous thread with @Mullcom, @brad, you (@domih) and everyone else, specifically this one. Right now all those parameters are set by a script since I wanted to be able to compare the differences between the default parameters and the new ones.
The MTU size on the I3 machine is 1500, frankly I had forgotten to try and tune the NIC on that side. I don't have too much time today to do more testing, but as soon as I can, I'll run more tests and update you guys! :D

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Wed Oct 09, 2019 9:46 am

(1) The MTU 9600 aka "jumbo frames" is usually not necessary or even advised, it is indented for specific cases to provide optimization. But in most cases, the default 1500 for Ethernet is adequate. Let us know how it goes with 1500 on both sides.

(2) I'm not expert in networking TCP settings but I did read a lot about these so called optimizations. The thing is that most of these papers (or derived or repeated) were written back in the days, probably around Kernel 2.x. Today's kernel is doing these optimizations for you: cpu affinity, irq_balance, tcp window size and so on is now done automatically by the kernel. I tried plenty of these options once I thought I knew enough about them (and thought I understood what they were doing), never saw a difference (quite probably because I'm not expert) and I quickly discovered that the Kernel had already done the job by simply comparing these values on different machines and seeing that the Kernel came up with different settings depending on the machine. For IRQ balance and CPU affinity it's even done on the fly changing constantly. So I would advise to just let these settings as they are for now.

(3) On the other hand, comparing the values you posted with the ones @igorpec will hopefully post, you might get a clearer picture.

HTH.

User avatar
igorpec
Posts: 398
Joined: Sat Dec 12, 2015 4:34 pm
languages_spoken: english,german,slovene
ODROIDs: XU4, HC1, C2, C1+
Has thanked: 9 times
Been thanked: 29 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by igorpec » Wed Oct 09, 2019 11:09 pm

Code: Select all

ethtool enp13s0
Settings for enp13s0:
	Supported ports: [ TP ]
	Supported link modes:   100baseT/Full 
	                        1000baseT/Full 
	                        10000baseT/Full 
	                        2500baseT/Full 
	                        5000baseT/Full 
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  100baseT/Full 
	                        1000baseT/Full 
	                        10000baseT/Full 
	                        2500baseT/Full 
	                        5000baseT/Full 
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: Unknown
	Supports Wake-on: g
	Wake-on: d
	Link detected: yes

Code: Select all

enp13s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XXXXXXXXX  netmask 255.255.255.0  broadcast XXXXXXXXXXXX
        inet6 fe80::1976:2760:749e:ddaf  prefixlen 64  scopeid 0x20<link>
        ether 70:85:c2:8d:37:ff  txqueuelen 1000  (Ethernet)
        RX packets 15123  bytes 16628028 (16.6 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 384  bytes 76370 (76.3 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Code: Select all

sysctl -a | grep tcp
fs.nfs.nfs_callback_tcpport = 0
fs.nfs.nlm_tcpport = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = reno cubic
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_autocorking = 1
net.ipv4.tcp_available_congestion_control = reno cubic
net.ipv4.tcp_available_ulp = 
net.ipv4.tcp_base_mss = 1024
net.ipv4.tcp_challenge_ack_limit = 1000
net.ipv4.tcp_comp_sack_delay_ns = 1000000
net.ipv4.tcp_comp_sack_nr = 44
net.ipv4.tcp_congestion_control = cubic
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_demux = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_ecn_fallback = 1
net.ipv4.tcp_fack = 0
net.ipv4.tcp_fastopen = 1
net.ipv4.tcp_fastopen_blackhole_timeout_sec = 3600
net.ipv4.tcp_fastopen_key = 00000000-00000000-00000000-00000000
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_fwmark_accept = 0
net.ipv4.tcp_invalid_ratelimit = 500
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_l3mdev_accept = 0
net.ipv4.tcp_limit_output_bytes = 1048576
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_max_reordering = 300
net.ipv4.tcp_max_syn_backlog = 512
net.ipv4.tcp_max_tw_buckets = 65536
net.ipv4.tcp_mem = 189714	252953	379428
net.ipv4.tcp_min_rtt_wlen = 300
net.ipv4.tcp_min_snd_mss = 48
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_notsent_lowat = 4294967295
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_pacing_ca_ratio = 120
net.ipv4.tcp_pacing_ss_ratio = 200
net.ipv4.tcp_probe_interval = 600
net.ipv4.tcp_probe_threshold = 8
net.ipv4.tcp_recovery = 1
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 4096	131072	6291456
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_reuse = 2
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 4096	16384	4194304
net.ipv4.tcp_workaround_signed_windows = 0
sunrpc.tcp_fin_timeout = 15
sunrpc.tcp_max_slot_table_entries = 65536
sunrpc.tcp_slot_table_entries = 2
sunrpc.transports = tcp 1048576

Code: Select all

uname -a
Linux desktop 5.0.0-31-generic #33-Ubuntu SMP Mon Sep 30 18:51:59 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/lsb-release 
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=19.04
DISTRIB_CODENAME=disco
DISTRIB_DESCRIPTION="Ubuntu 19.04"

Code: Select all

modinfo atlantic
filename:       /lib/modules/5.0.0-31-generic/kernel/drivers/net/ethernet/aquantia/atlantic/atlantic.ko
description:    aQuantia Corporation(R) Network Driver
author:         aQuantia
version:        2.0.4.0-kern
ARMBIAN - follow on Twitter
linux for ARM development boards with user friendly development tools

MiguelA0145
Posts: 4
Joined: Sat Oct 05, 2019 1:34 am
languages_spoken: English
ODROIDs: ODROID C2, ODROID H2
Has thanked: 11 times
Been thanked: 0
Contact:

Re: Troubles with 10GBe on H2

Unread post by MiguelA0145 » Sat Oct 12, 2019 12:12 am

So I finally have the test results with both the H2 and PC at 1500 MTU size!
Here's run 1 :

Code: Select all

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec   141 GBytes  4.04 Gbits/sec    0             sender
[  5]   0.00-300.00 sec   141 GBytes  4.04 Gbits/sec                  receiver
[  7]   0.00-300.00 sec   141 GBytes  4.04 Gbits/sec    0             sender
[  7]   0.00-300.00 sec   141 GBytes  4.04 Gbits/sec                  receiver
[SUM]   0.00-300.00 sec   282 GBytes  8.08 Gbits/sec    0             sender
[SUM]   0.00-300.00 sec   282 GBytes  8.08 Gbits/sec                  receiver

iperf Done.
Run 2 :

Code: Select all

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-300.00 sec   142 GBytes  4.06 Gbits/sec    0             sender
[  5]   0.00-300.00 sec   142 GBytes  4.06 Gbits/sec                  receiver
[  7]   0.00-300.00 sec   140 GBytes  4.01 Gbits/sec    0             sender
[  7]   0.00-300.00 sec   140 GBytes  4.01 Gbits/sec                  receiver
[SUM]   0.00-300.00 sec   282 GBytes  8.08 Gbits/sec    0             sender
[SUM]   0.00-300.00 sec   282 GBytes  8.08 Gbits/sec                  receiver

iperf Done.
Looking at these results, it looks like the tuning I had done earlier wasn't exactly worth while, which is strange because when I first started doing this I was capped at ~ 7.3. :?:

EDIT : I was looking at the results I got before and noticed that I wasn't using two ports when using iPerf like I am now. I assume that was one of the biggest limiters for those tests.

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Sat Oct 12, 2019 2:51 am

You now want to compare the settings from @igorpec with yours and write down the differences if any as a starter.

brad
Posts: 969
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 16 times
Been thanked: 34 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by brad » Sat Oct 12, 2019 10:59 am

domih wrote:
Sat Oct 12, 2019 2:51 am
You now want to compare the settings from @igorpec with yours and write down the differences if any as a starter.
I would like to see someone do some preliminary tests with an updated kernel on the N2. Something that includes the changes for updating rx ringbuffer size to reduce packet loss.

https://git.kernel.org/pub/scm/linux/ke ... a1cc417d63

ie elatllat's, tobetter's or armbian mainline variant kernels which are listed on viewtopic.php?f=176&t=33993&start=400

They can be a little complex to configure but its worth the test I believe.

RomaT
Posts: 235
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 42 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by RomaT » Sat Oct 12, 2019 11:27 am

igorpec wrote:
Sat Oct 05, 2019 6:02 pm
Desktop with integrated AQUANTIA® 10 Gigabit LAN (The same chipset as Asus XG-C100C)
As far as I know there slightly different chip - AQN-107, but not AQC-107
AQtion AQC-107 controllers are compliant to the IEEE 802.3bz standard
AQtion AQN-107 adapters are compliant to the IEEE 802.3an standard.
.
root@ :/# inxi -Fxxx
id.jpg
id.jpg (7.98 KiB) Viewed 380 times
.
domih wrote:
Wed Oct 09, 2019 7:13 am
Note that @RomaT also experiences speeds less than the expected 9.4 Gb/s.
I noticed that the testing methodology affects the result.
For example, if such a test, then the result is better:
.
netspeed07.jpg
netspeed07.jpg (47 KiB) Viewed 408 times
.
netspeed05.jpg
netspeed05.jpg (42.99 KiB) Viewed 424 times
.
netspeed04.jpg
netspeed04.jpg (80.75 KiB) Viewed 426 times
.
Despite the fact that nothing has changed, if specify a different methodology, the old result:
.
netspeed06.jpg
netspeed06.jpg (97.72 KiB) Viewed 423 times
These users thanked the author RomaT for the post:
domih (Sun Oct 13, 2019 10:42 am)

back2future
Posts: 240
Joined: Sun Jul 23, 2017 3:19 pm
languages_spoken: english
Has thanked: 9 times
Been thanked: 4 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by back2future » Sat Oct 12, 2019 11:35 pm

brad wrote:
Mon Oct 07, 2019 4:28 pm
Which kernel versions are each of you guys using on your machines? Some significant changes to Aquantia drivers have occurred since 4.9 which may impact performance.
[ One of the best improvements since years has been implementation of data compression hardware, but API to OS is available only for experts. ]
These users thanked the author back2future for the post:
domih (Sun Oct 13, 2019 11:02 am)

domih
Posts: 137
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2.
Has thanked: 42 times
Been thanked: 35 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by domih » Sun Oct 13, 2019 11:01 am

RomaT wrote:
Sat Oct 12, 2019 11:27 am
I noticed that the testing methodology affects the result.
I assume you did it multiple times and it's reproducible.

I assume these tests are with the H370.

It would be interesting to look at the iperf3 source code to see what it does differently when one specifies -n 20G.
RomaT wrote:
Sat Oct 12, 2019 11:27 am
As far as I know there slightly different chip - AQN-107, but not AQC-107
AQtion AQC-107 controllers are compliant to the IEEE 802.3bz standard
AQtion AQN-107 adapters are compliant to the IEEE 802.3an standard.
Yes, the wording is different as well as in the product brief PDFs. Yet, they plastered all over that each supports 10GBase-T and both are supposed to also support 2.5 and 5 GbE.

---

I believe more and more it's just a (stupid?) configuration option somewhere and/or driver bug(s), there is no reason why any of these chipset would not push to 9.4+ GbE. My main reasoning is: it would already be all over the Internet with people saying "buy this card" and "avoid that card like the plague".

RomaT
Posts: 235
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 42 times
Contact:

Re: Troubles with 10GBe on H2

Unread post by RomaT » Sun Oct 13, 2019 1:12 pm

domih wrote:
Sun Oct 13, 2019 11:01 am
I assume these tests are with the H370.
yes, this is on the H370-I.
I sold my ODROID-H2 to recoup new purchases (I do not have enough in it another connector M.2 with two lanes PCIe).
with H370-I, I got a faster PCIe SSD and a 10Gbps network card and an additional SATA controller and also for the video capture card remains a slot PCI-E x16
i.e. two connectors M.2 M with PCIe x4 v3.0, one M.2 E (removing the wifi adapter) with PCIe x2 v3.0 and bus I2C, one slot PCI-E x16
actually for the same money, paying extra a little for the G5400 processor
i.e. H370-I motherboard in the shop price equivalent ODROID-H2 + shipping ;)
especially considering such assemblies with ODROID-H2, i have in a similar chassis, but with faster components,
noise is not more than I did with one 92mm fan on ODROID-H2 (i.e. only in absolute silence can you hear the rustling, It does not prevent me from sleeping)
.
P9271152s.JPG
P9271152s.JPG (204.23 KiB) Viewed 276 times
.
P9271161s.JPG
P9271161s.JPG (392.07 KiB) Viewed 276 times
.
P9271164s.JPG
P9271164s.JPG (186.54 KiB) Viewed 276 times
.
system_h370i.png
system_h370i.png (86.37 KiB) Viewed 264 times
.
h370i_screen01.png
h370i_screen01.png (1.08 MiB) Viewed 264 times

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 1 guest