H2+ NIC with proxmox and Mikrotik

Post Reply
hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

Hi all,

I have bought 2 Odroid H2+ and I really like them for beeing powerfull and for beeing very power efficient. I have installed Proxmox VE (which is a Hypervisor like VMWare vSphere and Hyper-V but based on Debian). I did manage to install the deb package as described so that the NICs do work. I have connected the H2+'s via a Mikrotik Switch. When I am sending data from an H2+ to another Server (10GBit/s Connection) which is connetected to the MIkrotik switch as well I got a speed of about 2.5 GBit/s (2.34 to be exactly, but that is ok). But in the receiving direction I only get around 234MBit/s. It is the same with both H2+ devices. The Switch is telling me on its statistics page that there are RX pauses. I googled that and to what I found that has to do with the device connected to that port and not with the switch(?).

The Proxmox Kernel is Version 5.4.44-2.

What can I do to get the full speed of 2.5Gbit/s in both directions (sending and receiving.

thanks in advance

User avatar
odroid
Site Admin
Posts: 34849
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean, Japanese
ODROIDs: ODROID
Has thanked: 928 times
Been thanked: 759 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by odroid »

We could get ~2.35Gbps for both up & down directions with iperf tests on Ubuntu 20.04 (Kernel 5.4).
We used a local brand ipTIME HG25008 2.5GbE switching hub.
https://forum.odroid.com/viewtopic.php?p=297973#p297973

Can you connect two H2+ boards directly to narrow down root causes?
iperf result was also 2.35Gbps with the direct connection too.
These users thanked the author odroid for the post:
hans.dampf (Wed Jul 15, 2020 6:05 pm)

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

Thanks for your fast reply. I am gonna give it a try and report back.

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

i did the test with the two h2+ directly connected and rebooted both of them. Afterwards I got maximum speed. Then i plugged the h2+es back to the switch. At first I got maximum speed again (from 10Gbit Server). Then I did iperf from one h2+ to the other and again got poor performance.

Perhaps ethtool output helps a litte:

Code: Select all

Settings for enp2s0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseX/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseX/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        MDI-X: Unknown
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000033 (51)
                               drv probe ifdown ifup
I am a bit unsure about "pause frame use: Symmetric Receive-only" because that is the direction causing problems...

mad_ady
Posts: 8316
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 573 times
Been thanked: 434 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by mad_ady »

Ethernet pause frames are typically sent from a device when its input buffers are full to tell sending devices to slow down. See if you can disable it with something like this: https://www.linuxsecrets.com/discussion ... th-ethtool
These users thanked the author mad_ady for the post:
hans.dampf (Wed Jul 15, 2020 6:05 pm)

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

hans.dampf wrote:
Tue Jul 14, 2020 5:48 pm
.../...
I have connected the H2+'s via a Mikrotik Switch
.../...
2.34 to be exactly, but that is ok
.../...
1) Which MikroTik switch model are you using? 10GBase-T or 10G SFP+ with a couple of SFP+/RJ-45 transceivers?

2) 2500 Gbps - 64b/66b line code = 2420 Gbps - IP stack overhead = 2340 Gbps. As you said this is OK!
These users thanked the author domih for the post:
hans.dampf (Wed Jul 15, 2020 6:05 pm)

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

Hi all,

again thanks for your replies. I did test the the ethtool command. When I query the current status with "ethtool -a enp2s0" then I get the following error "Cannot get device pause settings: Operation not supported". The same error I get when I want to set the parameters with "ethtool -A enp2s0 autoneg off rx off tx off.

The switch is a "Mikrotik CRS317-1G-16S+RM Cloud Router Switch" Router. I have connected the h2+es with Mikrotik S+RJ10 10GBit/s SFP+ Copper Modules to the switch. The switch itself is running switch os in Version 2.12 (should be the latest version). The Switch shows the correct link speed of 2.5 Gbit/s. I enabled and disabled flow control (rx/tx) on the switch side of things and tested again after rebooting the h2+es. Nothing changed. That is why I think that the settings are set by the realtek driver. But that is only my guess? What else can I do to help solve this issue?

Thank you

User avatar
rooted
Posts: 7875
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 724 times
Been thanked: 222 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by rooted »

The switch is definitely running SWOS and not ROS? You aren't doing any fancy routing via the switch?

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

hans.dampf wrote:
Wed Jul 15, 2020 6:04 pm
.../...
The switch is a "Mikrotik CRS317-1G-16S+RM Cloud Router Switch" Router.
.../...
OK, I have the same switch but I used it to test SolarFlare 10G SFP+ cards between H2, H2+ and an Asus mobo with an onboard Aqantia 10GbE. I don't remember if I tested 2.5GbE though. I will give it a try in the next few days, but do not wait for me looking for additional troubleshooting. Disclosure: I am not a networking expert!

Two things you could also check: does the switch support multi-G, meaning 1, 2.5, 5 and 10G? Do the MicroTik transceivers support multi-G? I think STH published reviews where they say they do, but maybe they have not checked everything.
These users thanked the author domih for the post:
hans.dampf (Thu Jul 16, 2020 12:04 am)

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

rooted wrote:
Wed Jul 15, 2020 11:54 pm
The switch is definitely running SWOS and not ROS? You aren't doing any fancy routing via the switch?
Yes it is definitly SWOS because I am not an expert in routing and stuff. The SWOS webinterface is much easier to administrate and for my purposes absolutely enough.
Last edited by hans.dampf on Thu Jul 16, 2020 12:05 am, edited 1 time in total.

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

domih wrote:
Wed Jul 15, 2020 11:57 pm
OK, I have the same switch but I used it to test SolarFlare 10G SFP+ cards between H2, H2+ and an Asus mobo with an onboard Aqantia 10GbE. I don't remember if I tested 2.5GbE though. I will give it a try in the next few days, but do not wait for me looking for additional troubleshooting. Disclosure: I am not a networking expert!

Two things you could also check: does the switch support multi-G, meaning 1, 2.5, 5 and 10G? Do the MicroTik transceivers support multi-G? I think STH published reviews where they say they do, but maybe they have not checked everything.
Thanks for the hint. I bought this switch because of the review from sth article. They said only 6com SFPs do not support 2.5 and the rest of the tested SFP's do. According to the mikrotik wiki https://wiki.mikrotik.com/wiki/MikroTik ... lity_table the SFPs support mulit-G. And thank you for checking that out. Looking forward for your feedback.

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

I went through my notes. In the middle of the SolarFlare 10G SFP+ testing, I did (!) test 2.5 GbE between an H2+ and a PC with a 2.5GbE PCIe card.

Settings
- The H2+ is serving (iperf3 -s ...)
- both the H2+ and PC use RJ-45 CAT6-A cables and connect to the SFP+ switch using an Ipolex transceiver adapter. See https://www.ebay.com/itm/NEW-10G-SFP-RJ ... 3018252965
- the PC runs Windows 10 (win10_10039_05252020)
- iperf3 version on Windows 10 is 3.1.3.

Results

h2+-with-PC-over-2.5Gbe.png
h2+-with-PC-over-2.5Gbe.png (48.16 KiB) Viewed 932 times
So it looks like there is an issue in one direction: while the H2+ is receiving. I did not mind much at the time because I was focused on the 10G testing.

1. Do you have the same issue: is it during receiving?
2. Could you try something else than iperf3? e.g. copy back and forth a 10GB file over Samba and compared the resulting speeds?

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Oops, you wrote <<...But in the receiving direction I only get around 234MBit/s...>> in the OP, so that answers question 1. It's interesting but in my case iperf3 was reporting from 12 to 50Gbps! So we should conclude that iperf3 returns a random value, meaning whatever it founds in memory while computing this speed. Hence my 2nd question: try with something else than iperf3...

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

domih wrote:
Thu Jul 16, 2020 12:48 am
Oops, you wrote <<...But in the receiving direction I only get around 234MBit/s...>> in the OP, so that answers question 1. It's interesting but in my case iperf3 was reporting from 12 to 50Gbps! So we should conclude that iperf3 returns a random value, meaning whatever it founds in memory while computing this speed. Hence my 2nd question: try with something else than iperf3...
Thanks for your fast answers. On Debain iperf version 2.12. And I did test with iSCSI and had good read readings and bad write readings. That is why I started testing with iperf.

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

hans.dampf wrote:
Thu Jul 16, 2020 1:00 am
domih wrote:
Thu Jul 16, 2020 12:48 am
Oops, you wrote <<...But in the receiving direction I only get around 234MBit/s...>> in the OP, so that answers question 1. It's interesting but in my case iperf3 was reporting from 12 to 50Gbps! So we should conclude that iperf3 returns a random value, meaning whatever it founds in memory while computing this speed. Hence my 2nd question: try with something else than iperf3...
Thanks for your fast answers. On Debain iperf version 2.12. And I did test with iSCSI and had good read readings and bad write readings. That is why I started testing with iperf.
OK, this is vexing. I'll let you know once I was able to retest.

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

@hans.dampf

On both H2+ could you try:

sudo ethtool -s enp2s0 speed 2500 autoneg off

(enp2s0 or whatever NIC name you ended up with)

and then run iperf3 in both directions?

Reason for asking (short version): this solved the problem on my side between an H2+ and a PC with an Aqantia 10G via the MicroTik switch and Ipolex transceivers.
These users thanked the author domih for the post:
hans.dampf (Fri Jul 17, 2020 5:09 pm)

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Reason for asking (long version)

H2+ (h2a)..: 172.16.25.70 (2.5GbE realtek r8125b)
PC (trx)...: 172.16.25.36 (10GbE Aqantia)

Via Mikrotik CRS317-1G-16S switch and Ipolex transceivers.

h2a runs as iperf3 server.

domih@trx:~$ iperf3 -c 172.16.25.70 --bind 172.16.25.36
Connecting to host 172.16.25.70, port 5201
[ 5] local 172.16.25.36 port 49775 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 85.2 MBytes 714 Mbits/sec 1133 18.4 KBytes
[ 5] 1.00-2.00 sec 89.7 MBytes 753 Mbits/sec 1110 17.0 KBytes
[ 5] 2.00-3.00 sec 90.1 MBytes 756 Mbits/sec 1217 18.4 KBytes
[ 5] 3.00-4.00 sec 88.2 MBytes 740 Mbits/sec 1142 18.4 KBytes
[ 5] 4.00-5.00 sec 84.9 MBytes 712 Mbits/sec 1221 17.0 KBytes
[ 5] 5.00-6.00 sec 87.0 MBytes 730 Mbits/sec 1159 17.0 KBytes
[ 5] 6.00-7.00 sec 85.5 MBytes 717 Mbits/sec 1133 17.0 KBytes
[ 5] 7.00-8.00 sec 87.5 MBytes 734 Mbits/sec 1187 38.2 KBytes
[ 5] 8.00-9.00 sec 90.1 MBytes 756 Mbits/sec 1218 18.4 KBytes
[ 5] 9.00-10.00 sec 84.6 MBytes 710 Mbits/sec 1043 26.9 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 873 MBytes 732 Mbits/sec 11563 sender
[ 5] 0.00-10.17 sec 872 MBytes 720 Mbits/sec receiver


domih@trx:~$ iperf3 -c 172.16.25.70 --bind 172.16.25.36 -R
Connecting to host 172.16.25.70, port 5201
Reverse mode, remote host 172.16.25.70 is sending
[ 5] local 172.16.25.36 port 45575 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 1.00-2.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 5.00-6.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 6.00-7.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 7.00-8.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 8.00-9.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 9.00-10.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.16 sec 2.74 GBytes 2.32 Gbits/sec 2 sender
[ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver

As expected in the context of this thread, the reception on the H2+ went snafu (see resulting speed highlighted in red).

I then used ethtool, ethtool -i, ethtool -S on both NICs (on H2+ and on PC).

Code: Select all

domih@h2a:~$ ethtool enp2s0
Settings for enp2s0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	                        2500baseX/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	                        2500baseX/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 2500Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: Unknown
Cannot get wake-on-lan settings: Operation not permitted
	Current message level: 0x00000033 (51)
			       drv probe ifdown ifup
	Link detected: yes

domih@h2a:~$ ethtool -i enp2s0
driver: r8125
version: 9.003.05-NAPI
firmware-version: 
expansion-rom-version: 
bus-info: 0000:02:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no


domih@h2a:~$ ethtool -S enp2s0
NIC statistics:
     tx_packets: 2120880
     rx_packets: 721106
     tx_errors: 0
     rx_errors: 3745
     rx_missed: 0
     align_errors: 0
     tx_single_collisions: 0
     tx_multi_collisions: 0
     unicast: 720250
     broadcast: 812
     multicast: 44
     tx_aborted: 0
     tx_underrun: 0


domih@trx:~$ ethtool enp70s0
Settings for enp70s0:
	Supported ports: [ TP ]
	Supported link modes:   100baseT/Full 
	                        1000baseT/Full 
	                        10000baseT/Full 
	                        2500baseT/Full 
	                        5000baseT/Full 
	Supported pause frame use: Symmetric
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  100baseT/Full 
	                        1000baseT/Full 
	                        10000baseT/Full 
	                        2500baseT/Full 
	                        5000baseT/Full 
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: Unknown
Cannot get wake-on-lan settings: Operation not permitted
	Link detected: yes


domih@h2a:~$ ethtool enp2s0
Settings for enp2s0:
	Supported ports: [ TP ]
	Supported link modes:   10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	                        2500baseX/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: Yes
	Supported FEC modes: Not reported
	Advertised link modes:  10baseT/Half 10baseT/Full 
	                        100baseT/Half 100baseT/Full 
	                        1000baseT/Full 
	                        2500baseX/Full 
	Advertised pause frame use: No
	Advertised auto-negotiation: Yes
	Advertised FEC modes: Not reported
	Speed: 2500Mb/s
	Duplex: Full
	Port: Twisted Pair
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: on
	MDI-X: Unknown
Cannot get wake-on-lan settings: Operation not permitted
	Current message level: 0x00000033 (51)
			       drv probe ifdown ifup
	Link detected: yes

domih@trx:~$ ethtool -i enp70s0
driver: atlantic
version: 5.4.0-40-generic-kern
firmware-version: 3.1.98
expansion-rom-version: 
bus-info: 0000:46:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

domih@trx:~$ ethtool -S enp70s0
NIC statistics:
     InPackets: 2122180
     InUCast: 2120725
     InMCast: 546
     InBCast: 909
     InErrors: 0
     OutPackets: 731887
     OutUCast: 731807
     OutMCast: 76
     OutBCast: 4
     InUCastOctets: 3091985968
     OutUCastOctets: 982806461
     InMCastOctets: 41268
     OutMCastOctets: 9775
     InBCastOctets: 276615
     OutBCastOctets: 288
     InOctets: 3092303851
     OutOctets: 982816524
     InPacketsDma: 2121692
     OutPacketsDma: 731887
     InOctetsDma: 3083784187
     OutOctetsDma: 979888874
     InDroppedDma: 0
     Queue[0] InPackets: 54
     Queue[0] OutPackets: 88445
     Queue[0] Restarts: 0
     Queue[0] InJumboPackets: 0
     Queue[0] InLroPackets: 0
     Queue[0] InErrors: 0
     Queue[1] InPackets: 14
     Queue[1] OutPackets: 18
     Queue[1] Restarts: 0
     Queue[1] InJumboPackets: 0
     Queue[1] InLroPackets: 0
     Queue[1] InErrors: 0
     Queue[2] InPackets: 87960
     Queue[2] OutPackets: 0
     Queue[2] Restarts: 0
     Queue[2] InJumboPackets: 0
     Queue[2] InLroPackets: 0
     Queue[2] InErrors: 0
     Queue[3] InPackets: 2032744
     Queue[3] OutPackets: 0
     Queue[3] Restarts: 0
     Queue[3] InJumboPackets: 0
     Queue[3] InLroPackets: 0
     Queue[3] InErrors: 0
     Queue[4] InPackets: 891
     Queue[4] OutPackets: 46
     Queue[4] Restarts: 0
     Queue[4] InJumboPackets: 0
     Queue[4] InLroPackets: 0
     Queue[4] InErrors: 0
     Queue[5] InPackets: 0
     Queue[5] OutPackets: 12
     Queue[5] Restarts: 0
     Queue[5] InJumboPackets: 0
     Queue[5] InLroPackets: 0
     Queue[5] InErrors: 0
     Queue[6] InPackets: 29
     Queue[6] OutPackets: 123454
     Queue[6] Restarts: 0
     Queue[6] InJumboPackets: 0
     Queue[6] InLroPackets: 0
     Queue[6] InErrors: 0
     Queue[7] InPackets: 0
     Queue[7] OutPackets: 0
     Queue[7] Restarts: 0
     Queue[7] InJumboPackets: 0
     Queue[7] InLroPackets: 0
     Queue[7] InErrors: 0
I guessed/estimated (rightly or wrongly?) that the line rx_errors: 3745 in ethtool -S enp2s0 described the errors.

After semi-blindly fooling around with properties I went with an hail Mary, let's tell the Aqantia NIC to forget about 10G:

domih@trx:~$ sudo ethtool -s enp70s0 speed 2500 autoneg off

iperf3 then performed at nominal speed in both directions :-)

domih@trx:~$ iperf3 -c 172.16.25.70 --bind 172.16.25.36
Connecting to host 172.16.25.70, port 5201
[ 5] local 172.16.25.36 port 48511 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 282 MBytes 2.37 Gbits/sec 0 837 KBytes
[ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec 0 969 KBytes
[ 5] 2.00-3.00 sec 281 MBytes 2.36 Gbits/sec 0 969 KBytes
[ 5] 3.00-4.00 sec 281 MBytes 2.36 Gbits/sec 0 969 KBytes
[ 5] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec 0 1.05 MBytes
[ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 1.05 MBytes
[ 5] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec 0 1.05 MBytes
[ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec 0 1.05 MBytes
[ 5] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec 0 1.10 MBytes
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec 0 1.10 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec 0 sender
[ 5] 0.00-10.17 sec 2.74 GBytes 2.31 Gbits/sec receiver


iperf Done.
domih@trx:~$ iperf3 -c 172.16.25.70 --bind 172.16.25.36 -R
Connecting to host 172.16.25.70, port 5201
Reverse mode, remote host 172.16.25.70 is sending
[ 5] local 172.16.25.36 port 51659 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 1.00-2.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 2.00-3.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 3.00-4.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 5.00-6.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 6.00-7.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 7.00-8.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 8.00-9.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 9.00-10.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.19 sec 2.74 GBytes 2.31 Gbits/sec 2 sender
[ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver


I then rebooted the PC, the problem came back (as expected), reissued the 'sudo ethtool -s enp70s0 speed 2500 autoneg off' and the problem was fixed again.

@hans.dampf: I do NOT know if this will also fix your problem. My educated correct or incorrect guess/reasoning is that the autonegotiation wholly or partially gets lost in the interaction: NIC -> Transceiver -> Switch -> Transceiver -> Other NIC.
These users thanked the author domih for the post (total 2):
odroid (Fri Jul 17, 2020 8:47 am) • hans.dampf (Fri Jul 17, 2020 5:08 pm)

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Rhaaaahhhhhhhh :cry:

I've setup 2 x H2+ connected via the MicroTik switch and yes I'm seeing the same discrepancy:

domih@h2d:~$ iperf3 -c 172.16.25.70 --bind 172.16.25.73
Connecting to host 172.16.25.70, port 5201
[ 5] local 172.16.25.73 port 33115 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 275 MBytes 2.31 Gbits/sec 0 817 KBytes
[ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec 0 861 KBytes
[ 5] 2.00-3.00 sec 281 MBytes 2.36 Gbits/sec 0 861 KBytes
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec 0 861 KBytes
[ 5] 4.00-5.00 sec 280 MBytes 2.35 Gbits/sec 0 861 KBytes
[ 5] 5.00-6.00 sec 281 MBytes 2.36 Gbits/sec 0 861 KBytes
[ 5] 6.00-7.00 sec 280 MBytes 2.35 Gbits/sec 0 964 KBytes
[ 5] 7.00-8.00 sec 280 MBytes 2.35 Gbits/sec 0 964 KBytes
[ 5] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec 0 964 KBytes
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec 0 964 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver

iperf Done.
domih@h2d:~$ iperf3 -c 172.16.25.70 --bind 172.16.25.73 -R
Connecting to host 172.16.25.70, port 5201
Reverse mode, remote host 172.16.25.70 is sending
[ 5] local 172.16.25.73 port 56325 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 15.0 MBytes 125 Mbits/sec
[ 5] 1.00-2.00 sec 19.2 MBytes 161 Mbits/sec
[ 5] 2.00-3.00 sec 18.4 MBytes 154 Mbits/sec
[ 5] 3.00-4.00 sec 19.7 MBytes 165 Mbits/sec
[ 5] 4.00-5.00 sec 19.2 MBytes 161 Mbits/sec
[ 5] 5.00-6.00 sec 19.1 MBytes 160 Mbits/sec
[ 5] 6.00-7.00 sec 18.8 MBytes 158 Mbits/sec
[ 5] 7.00-8.00 sec 19.4 MBytes 163 Mbits/sec
[ 5] 8.00-9.00 sec 18.9 MBytes 158 Mbits/sec
[ 5] 9.00-10.00 sec 16.1 MBytes 135 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 185 MBytes 155 Mbits/sec 4556 sender
[ 5] 0.00-10.00 sec 184 MBytes 154 Mbits/sec receiver


So based on the experience I had "fixing" the Aqantia 10G on the PC, I attempted the same fix with the Realtek 8125B BUT the ethtool incantations are rejected. Example:

domih@h2d:~$ sudo ethtool -s enp3s0 speed 1000 duplex full autoneg off
# Looks OK I guess, but it does not make any difference.
domih@h2d:~$ sudo ethtool -s enp3s0 speed 2500 duplex full autoneg off
Cannot set new settings: Invalid argument
not setting speed
not setting duplex
not setting autoneg


So turning off autonegotiation and forcing the speed to 2500 is rejected. I guess I'm not using the right "wise" words in the incantation.

So I went with an alternate sneaky attempt to set the speed to 2500 and autonegotiation to no using insmod:

sudo rmmod r8125
sudo lsmod | grep r8
sudo insmod /lib/modules/5.4.0-40-generic/updates/dkms/r8125.ko speed=2500 duplex=1 autoneg=0
sudo lsmod | grep r8

Note: got the syntax for insmod from the r8125-9.003.05.tar.bz2 README file. I guess this file has actually been a useful README in the BCE era.

Waited for the NICs to reinitialize. Did not change much. Same behavior. I guess this sneaky attempt was not so crafty and modinfo told me why, see below. See the part highlighted in red.

domih@h2d:~$ modinfo r8125
filename: /lib/modules/5.4.0-40-generic/updates/dkms/r8125.ko
'version: 9.003.05-NAPI
license: GPL
description: Realtek RTL8125 2.5Gigabit Ethernet driver
author: Realtek and the Linux r8125 crew <netdev@vger.kernel.org>
srcversion: C166E452AA9E582D89D116B
alias: pci:v000010ECd00003000sv*sd*bc*sc*i*
alias: pci:v000010ECd00008125sv*sd*bc*sc*i*
depends:
retpoline: Y
name: r8125
vermagic: 5.4.0-40-generic SMP mod_unload
sig_id: PKCS#7
signer: h2d Secure Boot Module Signature key
sig_key: 3E:55:D7:DA:82:D1:D9:55:B4:A5:DA:88:DD:F9:60:63:58:BB:1F:7E
sig_hashalgo: sha512
signature: 90:CA:68:4C:7D:66:D2:FE:AC:52:11:08:CE:00:17:CC:A9:9D:9D:1D:
DA:F2:F9:72:6E:F4:42:8D:9D:ED:CB:BC:70:31:9F:74:0D:3A:8A:5A:
C5:5F:1E:0A:DE:55:D5:42:D4:97:38:FD:70:72:BA:D4:67:C9:D4:17:
85:28:9B:4A:82:CB:BC:9E:3A:C5:B4:CB:F4:5D:F3:57:02:25:18:56:
E3:17:66:A9:65:76:1F:15:68:46:86:58:79:0D:A9:3D:09:C5:04:13:
F0:71:5B:45:7A:F8:06:50:B1:8D:B3:2C:82:8E:F4:AC:66:B4:8C:0B:
9B:2D:8B:09:44:E7:BD:2F:14:8C:11:A8:65:08:46:C7:7A:DD:24:9D:
95:B4:45:B8:79:34:2B:4E:3F:75:D9:6A:92:BF:7B:39:6E:01:DC:6D:
72:97:94:A9:CA:AD:D4:55:D6:9C:FB:BC:A3:3C:79:84:44:26:EB:B4:
7E:66:E1:11:63:FF:26:4F:57:BE:4A:39:4E:51:63:27:8E:F5:82:D4:
70:C3:58:7C:75:BE:B5:F8:3F:EF:58:50:98:9F:D1:F8:B5:99:EF:DE:
A1:B9:56:75:89:67:04:E0:B5:C2:44:92:DB:13:F4:93:8D:9D:B5:AE:
2C:DE:2E:59:0C:16:12:AB:C6:00:E8:88:70:72:78:A4
parm: speed_mode:force phy operation. Deprecated by ethtool (8). (uint)
parm: duplex_mode:force phy operation. Deprecated by ethtool (8). (uint)
parm: autoneg_mode:force phy operation. Deprecated by ethtool (8). (uint)
parm: advertising_mode:force phy operation. Deprecated by ethtool (8). (uint)

parm: aspm:Enable ASPM. (int)
parm: s5wol:Enable Shutdown Wake On Lan. (int)
parm: s5_keep_curr_mac:Enable Shutdown Keep Current MAC Address. (int)
parm: rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int)
parm: use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
parm: timer_count:Timer Interrupt Interval. (int)
parm: eee_enable:Enable Energy Efficient Ethernet. (int)
parm: hwoptimize:Enable HW optimization function. (ulong)
parm: s0_magic_packet:Enable S0 Magic Packet. (int)
parm: tx_no_close_enable:Enable TX No Close. (int)
parm: debug:Debug verbosity level (0=none, ..., 16=all) (int)

In other words, there is code/script(s) in the system using ethtool to setup the NICs when they are activated. Where? No clue.

CONCLUSION: I'm gonna stop there for a while and I shamelessly pass the buck to the networking experts, wizards and gurus.
Last edited by domih on Fri Jul 17, 2020 10:23 am, edited 1 time in total.
These users thanked the author domih for the post:
hans.dampf (Fri Jul 17, 2020 5:08 pm)

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Finally to answer a possible upcoming question, yes I see the same thing as @hans.dampf: the speed is nominal at about 2.35Gbits/sec in BOTH directions when the 2 x H2+ are connected directly rather than through the switch.

domih@h2d:~$ iperf3 -c h2a.25 --bind 172.16.25.73 -R
Connecting to host h2a.25, port 5201
Reverse mode, remote host h2a.25 is sending
[ 5] local 172.16.25.73 port 45627 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 273 MBytes 2.29 Gbits/sec
[ 5] 1.00-2.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 2.00-3.00 sec 278 MBytes 2.33 Gbits/sec
[ 5] 3.00-4.00 sec 280 MBytes 2.35 Gbits/sec
[ 5] 4.00-5.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 5.00-6.00 sec 279 MBytes 2.34 Gbits/sec
[ 5] 6.00-7.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 7.00-8.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 8.00-9.00 sec 279 MBytes 2.34 Gbits/sec
[ 5] 9.00-10.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec receiver


domih@h2d:~$ iperf3 -c h2a.25 --bind 172.16.25.73
Connecting to host h2a.25, port 5201
[ 5] local 172.16.25.73 port 55171 connected to 172.16.25.70 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 277 MBytes 2.32 Gbits/sec 0 727 KBytes
[ 5] 1.00-2.00 sec 280 MBytes 2.35 Gbits/sec 0 779 KBytes
[ 5] 2.00-3.00 sec 280 MBytes 2.35 Gbits/sec 0 858 KBytes
[ 5] 3.00-4.00 sec 279 MBytes 2.34 Gbits/sec 0 911 KBytes
[ 5] 4.00-5.00 sec 281 MBytes 2.36 Gbits/sec 0 954 KBytes
[ 5] 5.00-6.00 sec 280 MBytes 2.35 Gbits/sec 0 954 KBytes
[ 5] 6.00-7.00 sec 281 MBytes 2.36 Gbits/sec 0 954 KBytes
[ 5] 7.00-8.00 sec 279 MBytes 2.34 Gbits/sec 0 954 KBytes
[ 5] 8.00-9.00 sec 281 MBytes 2.36 Gbits/sec 0 954 KBytes
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec 0 954 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec receiver



And we also know from Hardkernel that the H2+ 100% fluent in talking 2.5GbE on the South Korean switch they are lucky to have.

So I guess we go back to the interaction NIC -> Transceiver -> Switch -> Transceiver -> Other NIC and finding what detail trashes the party.

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

domih wrote:
Fri Jul 17, 2020 9:58 am
....
domih@h2d:~$ sudo ethtool -s enp3s0 speed 2500 duplex full autoneg off
Cannot set new settings: Invalid argument
not setting speed
not setting duplex
not setting autoneg


So turning off autonegotiation and forcing the speed to 2500 is rejected. I guess I'm not using the right "wise" words in the incantation.
Thank you very much domih for all the effort you put into try solving the issue. I am getting the same error when I want to turn autoneg off. The things you did are way beyond of what I was able to do. But sadly the speed issue remains. So anybody suggestions on how to solve this?

Thanks in advance.

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Suggestion: contact the Mikrotik TechSup.

Reason: the auto negotiation is between the Odroid H2+ RTL8125B and the Mikrotik switch via the transceiver.

I went to my Mikrotik switch admin web page (Link Tab) and tried to turn off auto negotiation and set the speed to 2.5G, the problem is that the option is not present. It only provides 10M, 100M, 1G and 10G. See photo below.

In theory, if multi-g, we should also see 2.5G and 5G. Right?

Could explain why sudo ethtool -s enp3s0 speed 1000 duplex full autoneg off was OK but not sudo ethtool -s enp3s0 speed 2500 duplex full autoneg off?

However, why as posted earlier sudo ethtool -s enp70s0 speed 2500 autoneg off worked with the Aqantia 10G on my PC worked at 2.5G without changing anything on the switch is then a mystery.

HTH


20200719_204003.jpg
20200719_204003.jpg (124.07 KiB) Viewed 741 times
These users thanked the author domih for the post:
hans.dampf (Thu Jul 23, 2020 4:22 pm)

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

Sorry for the late answer. I was a little busy. But first of all thank you domih for your support. Meanwhile I have bought a Zyxel XS1930-12HP to solve the performance issue with my h2+ devices. But that made everything worse. With the XS1930-12HP (which fully supports 2.5Gbit/s Standard) I got even worse performance when connecting to the one h2+ and can't even connect to my other h2+. The switch reports several thousand errors on the Ports where the odriod h2+ are connected to. This makes me belive that the drivers from the realtek nic perhaps have something to do with this (see errors with ethtool in the previous posts). Unmanaged Switches and direct Connections seem to work fine (according to other posts in this forum) but managed devices seem to have trouble (mikrotik and zyxel). But I am not a networking expert so that is just a guess and not for sure. So please someone who has much more knowledge of this than me help in soliving this issue. What other information can I provide to help solve this?

Thanks in advance.

mad_ady
Posts: 8316
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 573 times
Been thanked: 434 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by mad_ady »

Can you check what MTU you have set on both ends? Try with something safe like 1500.
These users thanked the author mad_ady for the post:
hans.dampf (Thu Jul 23, 2020 4:47 pm)

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

Hi mad_ady,

thanks for your support. I have checked switch-settings and found out that switch os does not support changing mtu (https://forum.mikrotik.com/viewtopic.php?t=124048) and my h2+ devices have mtu set to 1500

Code: Select all

enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000

Code: Select all

vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
what other infos can i provide?

mad_ady
Posts: 8316
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 573 times
Been thanked: 434 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by mad_ady »

Hmm, I don't know, MTU looks good.

User avatar
rooted
Posts: 7875
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 724 times
Been thanked: 222 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by rooted »

hans.dampf wrote:Hi mad_ady,

thanks for your support. I have checked switch-settings and found out that switch os does not support changing mtu (https://forum.mikrotik.com/viewtopic.php?t=124048) and my h2+ devices have mtu set to 1500

Code: Select all

enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master vmbr0 state UP group default qlen 1000

Code: Select all

vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
what other infos can i provide?
If you ask at the Microtik forum you will likely get some help, just be sure to be as specific as possible since they can be harsh with people who aren't.

I've had plenty of members there be outright arses for little to no reason.

brad
Posts: 1158
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 N2+ H2 H2+ (64 bit ftw)
Location: Australia
Has thanked: 58 times
Been thanked: 105 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by brad »

It might be worth to check any energy efficient modes by

sudo ethtool --show-eee enp2s0

and Disabling eee like this (im not actually sure how eee works on the 2.5G ports)

sudo ethtool --set-eee enp2s0 eee off
These users thanked the author brad for the post:
hans.dampf (Fri Jul 24, 2020 7:51 pm)

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Contact the Mikrotik TechSup a.s.a.p. and return the Zyxel to get back your $800.

If no solution is found with the help of Microtik:

Return the Mikrotik (to get back your $350) and fall back on a "pure" 2.5G switch like the current Trendnet or the upcoming QNAP.

OR:

"Sacrifice" the H2+ NVMe slot to PCIe gen 2 x4 and use SolarFlare cards ($20 on eBay, about the same for 3m DAC cables)(*). I know for a fact that they work. See Yet another exotic way to do 10G networking with the Odroid H2 (viewtopic.php?f=171&t=39174). Your other existing PC(s) can also use SolarFlare cards, otherwise they'll need a 10GBaseT RJ-45 PCIe card (at least PCIe gen2 2 x4, in both cases) and you can reuse the two transceivers you already have + CAT6-A cables.

(*) You still have the eMMC + 2 x SATA III for OS and storage.

HTH
Last edited by domih on Fri Jul 24, 2020 9:39 am, edited 1 time in total.
These users thanked the author domih for the post:
hans.dampf (Fri Jul 24, 2020 8:09 pm)

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Other thought: 10G (especially SFP+) is mature, has been used for about two decades in the Enterprise and Data Center markets. On the other hand 2.5G and 5G came up late for the consumer market or businesses who do not want to replace their CAT-5e cables. The Realtek 8125B is fine. However integration with (recent) 10G hardware might need network expertise like this thread shows. Don't blame it on Realtek though in this case. INTEL also has teething issues, see for example https://www.techpowerup.com/266335/inte ... -confirmed. It reminds me the early days of 1GbE...

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

brad wrote:
Fri Jul 24, 2020 8:31 am
It might be worth to check any energy efficient modes by

sudo ethtool --show-eee enp2s0

and Disabling eee like this (im not actually sure how eee works on the 2.5G ports)

sudo ethtool --set-eee enp2s0 eee off
Thanks for your help. I have checked that, but it did not help...

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

domih wrote:
Fri Jul 24, 2020 9:28 am
Contact the Mikrotik TechSup a.s.a.p. and return the Zyxel to get back your $800.

If no solution is found with the help of Microtik:

Return the Mikrotik (to get back your $350) and fall back on a "pure" 2.5G switch like the current Trendnet or the upcoming QNAP.

OR:

"Sacrifice" the H2+ NVMe slot to PCIe gen 2 x4 and use SolarFlare cards ($20 on eBay, about the same for 3m DAC cables)(*). I know for a fact that they work. See Yet another exotic way to do 10G networking with the Odroid H2 (viewtopic.php?f=171&t=39174). Your other existing PC(s) can also use SolarFlare cards, otherwise they'll need a 10GBaseT RJ-45 PCIe card (at least PCIe gen2 2 x4, in both cases) and you can reuse the two transceivers you already have + CAT6-A cables.

(*) You still have the eMMC + 2 x SATA III for OS and storage.

HTH
Thank you so much for all your support! I will contact MikroTik and did return the Zyxel unit... a "pure" switch is a bit of a problem because I am running proxmox ve on the h2+ with containers and VMs in different VLANs so "pure" switch will make more problems... The NVMe Ports and the SATA ports are in use. So I hope Mikrotik support will help. Otherwise I will have to sell the h2+. But besides of this networking issue I really like my h2+ for their performance and their power efficiency... So I will not give up that fast :D ;)

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

Last edited by hans.dampf on Sat Jul 25, 2020 4:22 am, edited 1 time in total.

User avatar
rooted
Posts: 7875
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 724 times
Been thanked: 222 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by rooted »

hans.dampf wrote:Hi all,

for the records ;)

https://forum.mikrotik.com/viewtopic.php?f=17&t=164110
The requested topic does not exist.

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

rooted wrote:
Sat Jul 25, 2020 2:11 am
hans.dampf wrote:Hi all,

for the records ;)

https://forum.mikrotik.com/viewtopic.php?f=17&t=164110
The requested topic does not exist.
Thank you, obviously I did something wrong in thread creation process. I was able to see the thread when I was logged in. Deleted that thread and created a new one. (see previous post)

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

hans.dampf wrote:
Fri Jul 24, 2020 8:20 pm
domih wrote:
Fri Jul 24, 2020 9:28 am
Contact the Mikrotik TechSup a.s.a.p. and return the Zyxel to get back your $800.

If no solution is found with the help of Microtik:

Return the Mikrotik (to get back your $350) and fall back on a "pure" 2.5G switch like the current Trendnet or the upcoming QNAP.

OR:

"Sacrifice" the H2+ NVMe slot to PCIe gen 2 x4 and use SolarFlare cards ($20 on eBay, about the same for 3m DAC cables)(*). I know for a fact that they work. See Yet another exotic way to do 10G networking with the Odroid H2 (viewtopic.php?f=171&t=39174). Your other existing PC(s) can also use SolarFlare cards, otherwise they'll need a 10GBaseT RJ-45 PCIe card (at least PCIe gen2 2 x4, in both cases) and you can reuse the two transceivers you already have + CAT6-A cables.

(*) You still have the eMMC + 2 x SATA III for OS and storage.

HTH
Thank you so much for all your support! I will contact MikroTik and did return the Zyxel unit... a "pure" switch is a bit of a problem because I am running proxmox ve on the h2+ with containers and VMs in different VLANs so "pure" switch will make more problems... The NVMe Ports and the SATA ports are in use. So I hope Mikrotik support will help. Otherwise I will have to sell the h2+. But besides of this networking issue I really like my h2+ for their performance and their power efficiency... So I will not give up that fast :D ;)
Sorry for the miscommunication by "pure" I meant supports 2.5Gbe as main speed and not downgraded auto-negotiated from 10G, for example: https://www.amazon.com/TRENDnet-2-5GBAS ... B079Z3L6LN which supports VLAN. But first get the support from Microtik.

"...The NVMe Ports and the SATA ports are in use..." For what? Can you detail?
Last edited by domih on Sat Jul 25, 2020 12:44 pm, edited 1 time in total.

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

hans.dampf wrote:
Sat Jul 25, 2020 4:24 am
rooted wrote:
Sat Jul 25, 2020 2:11 am
hans.dampf wrote:Hi all,

for the records ;)

https://forum.mikrotik.com/viewtopic.php?f=17&t=164110
The requested topic does not exist.
Thank you, obviously I did something wrong in thread creation process. I was able to see the thread when I was logged in. Deleted that thread and created a new one. (see previous post)
Same here, link does not lead to an existing Microtik forum topic and I'm not sure about what you mean in your answer to @rooted.

User avatar
rooted
Posts: 7875
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 724 times
Been thanked: 222 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by rooted »

domih wrote:
hans.dampf wrote:
Sat Jul 25, 2020 4:24 am
rooted wrote:
Sat Jul 25, 2020 2:11 am
hans.dampf wrote:Hi all,

for the records ;)

https://forum.mikrotik.com/viewtopic.php?f=17&t=164110
The requested topic does not exist.
Thank you, obviously I did something wrong in thread creation process. I was able to see the thread when I was logged in. Deleted that thread and created a new one. (see previous post)
Same here, link does not lead to an existing Microtik forum topic and I'm not sure about what you mean in your answer to @rooted.
He created a new thread on MikroTik forum and posted an updated link.

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

Oops, missed the OP and looked at the replies instead :-) Thanks.

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

domih wrote:
Sat Jul 25, 2020 12:40 pm
Sorry for the miscommunication by "pure" I meant supports 2.5Gbe as main speed and not downgraded auto-negotiated from 10G, for example: https://www.amazon.com/TRENDnet-2-5GBAS ... B079Z3L6LN which supports VLAN. But first get the support from Microtik.

"...The NVMe Ports and the SATA ports are in use..." For what? Can you detail?
Hi domih,

thanks for the suggestion. I don't understand the trendnet solution. I have bought the ZYXEL XS1930 because it supports 2.5 GBit, what is different to the trendnet switch?

The NVMe Port carries a Samsung 860 Evo Drive and the SATA Ports an WD Gold 8TB each.

Thanks

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

hans.dampf wrote:
Sun Jul 26, 2020 3:12 am
domih wrote:
Sat Jul 25, 2020 12:40 pm
Sorry for the miscommunication by "pure" I meant supports 2.5Gbe as main speed and not downgraded auto-negotiated from 10G, for example: https://www.amazon.com/TRENDnet-2-5GBAS ... B079Z3L6LN which supports VLAN. But first get the support from Microtik.

"...The NVMe Ports and the SATA ports are in use..." For what? Can you detail?
Hi domih,

thanks for the suggestion. I don't understand the trendnet solution. I have bought the ZYXEL XS1930 because it supports 2.5 GBit, what is different to the trendnet switch?

The NVMe Port carries a Samsung 860 Evo Drive and the SATA Ports an WD Gold 8TB each.

Thanks
The MicroTik and Zyxel are both 10G switch that are being said to be multi-g (1,2.5,5G).
The TrendNet is directly 2.5G.
Your problem seems to be finding the right incantations for making the NICs and the switch talk 2.5G. So the TrendNet should be less a road-block for you.

---

Unless installing ProxMox on a 64GB or 128GB eMMC is a big no-no for you, you could use the NVMe for an SFP+ 10G card via PCIe Gen2 x4 adapter and your H2+ will talk 10G with other devices with an SFP+ card or with a 10GBaseT card + transceiver on the MicroTik switch side. As stated earlier I validated this solution on my side, so you would not go blindly there to meet yet another wall.

All solutions are valid, it is more the practicality of making it work on your side that counts here.

You still need to see what the Microtik forum will suggest this upcoming week (beyond the guy bashing Realtek based on 1G past history).

hans.dampf
Posts: 16
Joined: Tue Jul 14, 2020 5:09 pm
languages_spoken: english
ODROIDs: H2+
Has thanked: 11 times
Been thanked: 0
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by hans.dampf »

domih wrote:
Sun Jul 26, 2020 2:32 pm
The MicroTik and Zyxel are both 10G switch that are being said to be multi-g (1,2.5,5G).
The TrendNet is directly 2.5G.
Your problem seems to be finding the right incantations for making the NICs and the switch talk 2.5G. So the TrendNet should be less a road-block for you.

---

Unless installing ProxMox on a 64GB or 128GB eMMC is a big no-no for you, you could use the NVMe for an SFP+ 10G card via PCIe Gen2 x4 adapter and your H2+ will talk 10G with other devices with an SFP+ card or with a 10GBaseT card + transceiver on the MicroTik switch side. As stated earlier I validated this solution on my side, so you would not go blindly there to meet yet another wall.

All solutions are valid, it is more the practicality of making it work on your side that counts here.

You still need to see what the Microtik forum will suggest this upcoming week (beyond the guy bashing Realtek based on 1G past history).
Okay, got your point. Thanks for that. Do you think the Cisco XG350X-24PD is a "downsized" 10GBit Solution or a valid 2.5G Solution. The trendNET solution is missing 2 additional 10G Ports...

domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by domih »

I can't find a "Cisco XG350X-24PD" in Google.

The TRENDnet has 2 x 10G SFP+, you need more?

dkking
Posts: 15
Joined: Sat Feb 02, 2019 12:07 am
languages_spoken: english
ODROIDs: Rock64
Has thanked: 3 times
Been thanked: 1 time
Contact:

Re: H2+ NIC with proxmox and Mikrotik

Post by dkking »

man will we see cheap 2.5gbe switch anytime soon?

MS510TX for $269.99 has 2x 2.5gbe ports and 2x 5gbe and 1 x 10gb SFP+
or
all 10gbe ports SX508M for $529.99 has 8 10gbe ports and 1 x 10gb SFP+ port

Post Reply

Return to “Other OS”

Who is online

Users browsing this forum: No registered users and 1 guest