Yet another exotic way to do 10G networking with the Odroid H2

Post Reply
domih
Posts: 222
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 72 times
Been thanked: 84 times
Contact:

Yet another exotic way to do 10G networking with the Odroid H2

Post by domih » Wed Jun 10, 2020 2:01 pm

Although I'm using Mellanox InfiniBand for my local Intranet(*) where an H2 can communicate from 10Gbe to 14Gbe depending on direction, there are two other way to do 10G with the Odroid H2 if this is what you want to achieve.

(*) see https://magazine.odroid.com/article/net ... odroid-h2/ and viewtopic.php?f=172&t=38711. Yes this is quite exotic (at the beginning), but works very well and it is cheaper than the current RJ-45 10GBase-T solutions you can find (as of this writing).

1. The RJ-45 10GBase-T way: use an M.2 NMVe M key to PCIe x4 adapter(**) and plug in an RJ-45 10G PCIe card. See https://www.newegg.com/p/pl?d=10+Gbe+PC ... sdeptsrh=1 and https://www.amazon.com/s?k=10+Gbe+PCIe+card. This will cost you from $75 to $250+ depending on brand, features and number or ports.

(**) see https://www.ebay.com/itm/PCIe-x4-3-0-PC ... 4398886092

WARNING: not all PCIe cards will work with the PCIe 2.0 x4 available on the H2. Your mileage may vary. Dig into the Odroid forums to leverage the experience of other users.

At this point, if you only have one computer you want to connect to your H2, just connect them directly with a CAT6A (max length 100M = 330ft) and you're done. If you want to use many more computers, you'll need a switch and it will cost you $$$ (from about $800 at best up to +$2500 depending on number of ports and capabilities). Common brands are slowly climbing the 1+ G mountain. Models with 2.5GBase-T, 5GBase-T and 10GBase-T ports are appearing each month. Expensive. IMHO: wait for the no brand models from China, South Korea and Taiwan. Most of them will work as well... because most of them are exactly the same hardware minus the brand added to it.

If you are planning to have a mix of 2.5, 5 and 10G devices for the next few years, RJ-45 10GBase-T is the way to go for minimizing hassles and headaches.

2. A more exotic way is to chose an SFP+ solution.

Cards: You can find still functioning well 10G SFP+ NICs on eBay for less than $20. Example: https://www.ebay.com/itm/SolarFlare-SFN ... 3129025454. The SolarFlare SFN5122F was one the 10 Gbe networking work horse in data centers years ago. There are more recent models if you want to spend more than $20 (but still much less than $75, depending on features you need). Data centers today (the big ones) are standardizing on 100 or 200 Gbe. From these data centers point of view 10G is like an antique from the Merovingian period. But what is trash from some is treasure for others (the others being us here). There is a cornucopia of used 10, 25, 40 Gbe or InfiniBand 56 Gbps networking hardware on eBay. From our point of view, this is the Golden Age of reusing this hardware for individual purpose. Yes there is at least something trickling down in our dear US of A...

SFP+ cables: A 3M SFP+ DAC (Direct Attach Cable) copper cable can be find for cheap on eBay too. See for example: https://www.ebay.com/sch/i.html?_from=R ... =0&_sop=15

WARNING: 10 Gbe DAC cables are not all equal. One brand may or may not work with particular models of NICs. The Molex SFP-H10Gb-Cu3M works fine with the SolarFlare SFN5122F. Note: did the first cable I bought work with the card? Nope! Then I did my home work. The 2nd cable I bought works AOK :-)

Switches: There is also a plethora of 10G SFP+ switches on eBay. Starting from $70 up to $600 and more. Do you home work: check out how many ports are actually 10G SFP+, often the cheap ones will have 2 or 4 x 10G SFP+ and 24 or 48 1G ports. They were used to connect a corporate department computers to the backbone of the corporate Intranet. Or racks to the backbone network in a data center.

WARNING: Another issue to deal going the SFP+ route if the fact that the various forms of the 2.5G or 5G networking (i.e. BaseT) are relatively recent and "old" used 10G hardware you can buy on eBay will know nothing about of these protocols. In other words, they will NOT negotiate 2.5G or 5G making it impossible to connect a 2.5G or 5G device.

I did not want to spend too much time finding the perfect "old" switch so I settled on a very recent one from MicroTik (a Latvian company well-known in the networking business).

MicroTik CRS317-1G-16S+ Product Page
https://mikrotik.com/product/crs317_1g_16s_rm
MSRP: $399.00

Review from STH
https://www.servethehome.com/mikrotik-c ... rm-review/

On Amazon: $349.90
https://www.amazon.com/Cloud-Router-Swi ... B0747TC9DB

On NewEgg: $369.00
https://www.newegg.com/mikrotik-crs317- ... 002R-000D9

On eBay: $328.76 + $22.00 shipping
https://www.ebay.com/itm/Mikrotik-CRS31 ... 3710103713

I was lucky to stumble on a bidding new one which I got for $275 + shipping + tax = $313.66.
.
ebay-bidding.png
ebay-bidding.png (84.88 KiB) Viewed 283 times
Hands-on
See the H2 and switch in the picture shown below.
.
H2-and-SFP+-switch.png
H2-and-SFP+-switch.png (1.46 MiB) Viewed 283 times
See the SolarFlare SFN5122F connected to the H2 via the m.2 / PCIe adapter cable in the picture shown below:
.
SolarFlare SFN5122F.png
SolarFlare SFN5122F.png (1.85 MiB) Viewed 283 times
.
The SFP+ cable connects the SolarFlare SFN5122F to the first port of the switch:
.
switch ports 1 and 2 in use.png
switch ports 1 and 2 in use.png (1.89 MiB) Viewed 283 times
Note the red CAT6-A cable connected to the SFP+ switch. This "magic" is possible thanks too a small piece of hardware (transceiver) which converts back and forth SFP+ signals to RJ-45 signals. This thing was also bought from eBay. See https://www.ebay.com/itm/10G-SFP-RJ45-C ... 4214937752. There are multiple brands and models of such a transceiver. For a general overview of these transceivers, see this article and the related reviews from STH: https://www.servethehome.com/sfp-to-10g ... ers-guide/

With everything setup, one can see the connections in the switch web-based admin:
.
admin-1.png
admin-1.png (107.24 KiB) Viewed 283 times
CONTINUED IN NEXT POST
Last edited by domih on Wed Jun 10, 2020 2:57 pm, edited 1 time in total.
These users thanked the author domih for the post (total 2):
rooted (Wed Jun 10, 2020 2:03 pm) • mad_ady (Wed Jun 10, 2020 2:58 pm)

domih
Posts: 222
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 72 times
Been thanked: 84 times
Contact:

Re: Yet another exotic way to do 10G networking with the Odroid H2

Post by domih » Wed Jun 10, 2020 2:42 pm

CONTINUING FROM PREVIOUS POST.

In the screenshot shown below, the first device is the H2 with the SolarFlare SFN5122F, the second device is a PC whose motherboard has an onboard Aquantia 10G Ethernet.
.
admin-2.png
admin-2.png (70.36 KiB) Viewed 277 times
Here is a detailed view of the Ipolex transceiver allowing the Aquantia AQC-107 10GBase-T to talk with the SFP+ switch:
.
ipolex-1.png
ipolex-1.png (1.75 MiB) Viewed 277 times
ipolex-2.png
ipolex-2.png (1.92 MiB) Viewed 277 times
Let's now see the way the SolarFlare SFN5122F shows on the H2 with lspci:

Code: Select all

domih@h2a:~$ lspci
00:00.0 Host bridge: Intel Corporation Device 31f0 (rev 03)
.../...
01:00.0 Ethernet controller: Solarflare Communications SFC9020 10G Ethernet Controller
01:00.1 Ethernet controller: Solarflare Communications SFC9020 10G Ethernet Controller
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
Let's check whether or not the card does use the 4 lanes (it does, see LnkSta line):

Code: Select all

domih@h2a:~$ sudo lspci -vv -s 01:00.0 | grep Width 
		LnkCap:	Port #0, Speed 5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
		LnkSta:	Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
After configuring the NIC IP settings with nmtui, let's see the result (see enp1s0f1np1 section):

Code: Select all

domih@h2a:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:1e:06:45:0d:47 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.70/24 brd 192.168.1.255 scope global noprefixroute enp2s0
       valid_lft forever preferred_lft forever
    inet6 fe80::7938:96d2:adee:503b/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: enp3s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 00:1e:06:45:0d:48 brd ff:ff:ff:ff:ff:ff
4: enp1s0f0np0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:0f:53:07:89:80 brd ff:ff:ff:ff:ff:ff
5: enp1s0f1np1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:0f:53:07:89:81 brd ff:ff:ff:ff:ff:ff
    inet 172.16.25.69/16 brd 172.16.255.255 scope global noprefixroute enp1s0f1np1
       valid_lft forever preferred_lft forever
    inet6 fe80::af67:8d14:c279:c9da/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
Is the H2 "seeing" the switch? It does:

Code: Select all

domih@h2a:~$ ping 172.16.25.1
PING 172.16.25.1 (172.16.25.1) 56(84) bytes of data.
64 bytes from 172.16.25.1: icmp_seq=1 ttl=255 time=0.188 ms
64 bytes from 172.16.25.1: icmp_seq=2 ttl=255 time=0.141 ms
64 bytes from 172.16.25.1: icmp_seq=3 ttl=255 time=0.326 ms
64 bytes from 172.16.25.1: icmp_seq=4 ttl=255 time=0.142 ms
64 bytes from 172.16.25.1: icmp_seq=5 ttl=255 time=0.335 ms
^C
--- 172.16.25.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4102ms
rtt min/avg/max/mdev = 0.141/0.226/0.335/0.087 ms
Is the card configured for 10G with its link up? It is:

Code: Select all

domih@h2a:~$ ethtool enp1s0f1np1
Settings for enp1s0f1np1:
	Supported ports: [ FIBRE ]
	Supported link modes:   1000baseT/Full 
	                        10000baseT/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: No
	Supported FEC modes: Not reported
	Advertised link modes:  Not reported
	Advertised pause frame use: No
	Advertised auto-negotiation: No
	Advertised FEC modes: Not reported
	Link partner advertised link modes:  Not reported
	Link partner advertised pause frame use: No
	Link partner advertised auto-negotiation: No
	Link partner advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Port: FIBRE
	PHYAD: 255
	Transceiver: internal
	Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
	Current message level: 0x000020f7 (8439)
			       drv probe link ifdown ifup rx_err tx_err hw
	Link detected: yes
Is the H2 "seeing" the PC with the Aquantia? It does:

Code: Select all

domih@h2a:~$ ping 172.16.25.36
PING 172.16.25.36 (172.16.25.36) 56(84) bytes of data.
64 bytes from 172.16.25.36: icmp_seq=1 ttl=64 time=0.534 ms
64 bytes from 172.16.25.36: icmp_seq=2 ttl=64 time=0.296 ms
64 bytes from 172.16.25.36: icmp_seq=3 ttl=64 time=0.250 ms
64 bytes from 172.16.25.36: icmp_seq=4 ttl=64 time=0.264 ms
64 bytes from 172.16.25.36: icmp_seq=5 ttl=64 time=0.299 ms
64 bytes from 172.16.25.36: icmp_seq=6 ttl=64 time=0.259 ms
64 bytes from 172.16.25.36: icmp_seq=7 ttl=64 time=0.254 ms
64 bytes from 172.16.25.36: icmp_seq=8 ttl=64 time=0.326 ms
^C
--- 172.16.25.36 ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7158ms
rtt min/avg/max/mdev = 0.250/0.310/0.534/0.089 ms
It looks like everything is OK. Let's measure the throughput with iperf3:

Code: Select all

domih@h2a:~$ iperf3 -c 172.16.25.36 --bind 172.16.25.69
Connecting to host 172.16.25.36, port 5201
[  4] local 172.16.25.69 port 43329 connected to 172.16.25.36 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.31 MBytes       
[  4]   1.00-2.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.41 MBytes       
[  4]   2.00-3.00   sec  1.09 GBytes  9.32 Gbits/sec    0   1.41 MBytes       
[  4]   3.00-4.00   sec  1.09 GBytes  9.33 Gbits/sec    0   1.41 MBytes       
[  4]   4.00-5.00   sec  1.08 GBytes  9.30 Gbits/sec    0   1.60 MBytes       
[  4]   5.00-6.00   sec  1.08 GBytes  9.31 Gbits/sec    0   1.60 MBytes       
[  4]   6.00-7.00   sec  1.09 GBytes  9.33 Gbits/sec    0   2.16 MBytes       
[  4]   7.00-8.00   sec  1.07 GBytes  9.22 Gbits/sec    1   1.21 MBytes       
[  4]   8.00-9.00   sec  1.06 GBytes  9.15 Gbits/sec    1   1.28 MBytes       
[  4]   9.00-10.00  sec  1.08 GBytes  9.26 Gbits/sec    0   1.34 MBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.8 GBytes  9.29 Gbits/sec    2             sender
[  4]   0.00-10.00  sec  10.8 GBytes  9.28 Gbits/sec                  receiver

iperf Done.
domih@h2a:~$ iperf3 -c 172.16.25.36 --bind 172.16.25.69 -R
Connecting to host 172.16.25.36, port 5201
Reverse mode, remote host 172.16.25.36 is sending
[  4] local 172.16.25.69 port 37157 connected to 172.16.25.36 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  1.09 GBytes  9.39 Gbits/sec                  
[  4]   1.00-2.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  4]   2.00-3.00   sec  1.09 GBytes  9.40 Gbits/sec                  
[  4]   3.00-4.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  4]   4.00-5.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  4]   5.00-6.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  4]   6.00-7.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  4]   7.00-8.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  4]   8.00-9.00   sec  1.10 GBytes  9.41 Gbits/sec                  
[  4]   9.00-10.00  sec  1.10 GBytes  9.41 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec  408             sender
[  4]   0.00-10.00  sec  11.0 GBytes  9.41 Gbits/sec                  receiver

iperf Done.
Yeah! We get a respectable 9.28 Gbits/sec in one direction and 9.41 Gbits/sec in the other.

However, see the high number of segment retries (408) in the reverse test (-R parameter). I will have to do my home work on this one to mitigate the issue. It does not seem to affect much the throughput though.

CONCLUSION: if you are NOT planning to use 2.5G and 5G devices and want to use 10G as soon as possible an SFP+ switch and a mix of SFP+ and RJ-45 NICs might be the solution you are looking for.

With SFP+, it will cost you about $20 (card) + $15 (3m DAC cable) + $360 / 16 (switch) = $57.50 per port and $50 (transceiver) + $15 (CAT6-A cable) + $360 / 16 = $87.50 per port (not counting the price of the 10GBase-T NIC when it is onboard a motherboard). This means that you want to go with an SFP+ NIC when the PC does not have 10G. All you need is a PCIe 2.0 x8 or higher (x4 is OK as shown above). Expect the price of the SFP+ to RJ-45 transceivers to go down.

Other considerations:
- SFP+ DAC cables are cheap enough for 3m or 5m or 7m. For much longer distances, you'll have to go with a fiber cable and corresponding transceiver. In this case, the 100m max length possible with CAT6-A makes the RJ-45 solution better in terms of cost.
- For casual usage, plugging-in an RJ-45 NIC in a PCIe slot is a simple thing to do (plug and play). The consumer products are easy to use, you will not have to perform tweaking in most of the cases. Old data centers cards based on SFP+ provide many options, the SolarFlare Server Adapter User Guide is 393-page long. But for the hands-on in this post, I did not even read it, the card drivers are part of the Linux kernel and I did not need to change settings on the card beyond setting the IP address and mask. It was also plug and play. So it could be more complex to set up an SFP+ solution but it does not have to be. You only need to start tweaking when you start to do specialized things like VMware servicing or esoteric networking configuration. Also in the RJ-45, it would be the same thing with cards costing > $200.

Do you home work by comparing this price per port to a full RJ-45 solution. The RJ-45 10G NICs are expensive and the switches even more expensive. But then again, if you are planning to have a mix of 2.5G, 5G and 10G devices, you will want to use RJ-45 all along and ignore SFP+.

To conclude: which is better for 10G networking? As you might expect, it depends!

HTH.
Last edited by domih on Wed Jun 10, 2020 3:17 pm, edited 1 time in total.
These users thanked the author domih for the post:
mad_ady (Wed Jun 10, 2020 3:00 pm)

User avatar
mad_ady
Posts: 8155
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 568 times
Been thanked: 404 times
Contact:

Re: Yet another exotic way to do 10G networking with the Odroid H2

Post by mad_ady » Wed Jun 10, 2020 3:09 pm

Great networking overview.
Regarding
However, see the high number of segment retries (408) in the reverse test (-R parameter). I will have to do my home work on this one to mitigate the issue. It does not seem to affect much the throughput though.
I think packet drops and retransmits are normal and a part of TCP's exponential backoff network congestion avoidance algorithm. Simply put - TCP will always try (as long as the transmission window allows it) to send more data on a connection until there is packet loss/need for retransmission. When that happens (based on the current implementation of the backoff algorithm) it will reduce its throughput and ramp it up again slowly until another loss happens, and so on. Unlike UDP who will happily congest every link.
So packet loss is normal in the life of a tcp session. You can see it on 1G iperf tests too.
These users thanked the author mad_ady for the post:
domih (Wed Jun 10, 2020 3:20 pm)

domih
Posts: 222
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 72 times
Been thanked: 84 times
Contact:

Re: Yet another exotic way to do 10G networking with the Odroid H2

Post by domih » Wed Jun 10, 2020 3:24 pm

mad_ady wrote:
Wed Jun 10, 2020 3:09 pm
Great networking overview.
Regarding
However, see the high number of segment retries (408) in the reverse test (-R parameter). I will have to do my home work on this one to mitigate the issue. It does not seem to affect much the throughput though.
I think packet drops and retransmits are normal and a part of TCP's exponential backoff network congestion avoidance algorithm. Simply put - TCP will always try (as long as the transmission window allows it) to send more data on a connection until there is packet loss/need for retransmission. When that happens (based on the current implementation of the backoff algorithm) it will reduce its throughput and ramp it up again slowly until another loss happens, and so on. Unlike UDP who will happily congest every link.
So packet loss is normal in the life of a tcp session. You can see it on 1G iperf tests too.
I will have to read more about it because I'm really used to NOT seeing segment retries. With InfiniBand I only saw them when testing a damaged cable. A handful is acceptable but several hundred means potential issue, like for instance what I saw when testing the 2.5 Gbe USB adapters. But once again, I'll have to read about it, contrary to what it may seem, I am definitely NOT a networking guy ;-)

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 0 guests