[HOWTO] Make 2 x H2+ use the full potential of 2 x 2.5 GbE ports = 5 GbE total

Post Reply
domih
Posts: 274
Joined: Mon Feb 11, 2019 4:48 pm
languages_spoken: English, French
ODROIDs: UX4, HC2, N2, H2, C4, H2+
Location: San Francisco Bay Area
Has thanked: 90 times
Been thanked: 104 times
Contact:

[HOWTO] Make 2 x H2+ use the full potential of 2 x 2.5 GbE ports = 5 GbE total

Post by domih »

If you have 2 x H2+ and connect both their ports, you potentially have 5 GbE of bandwidth available as long as you use the two ports concurrently in separate IP sessions.

Right? Yes, but you do have some configuration to do.

Important Notes
(1) This post is NOT about bonding the two ports on each H2+.
(2) Users knowledgeable in networking and routing will not learn anything new here. I'm writing for the newbies like me :-)


Hands-on
I'll use 2 x H2+, one is named h2a and the other is named h2d (note: yes I also have h2b and h2c but they are H2, not H2+).

h2a port 1 is directly connected to h2d port1.
h2a port 2 is directly connected to h2d port2.

The h2a ports are configured with static IP addresses as shown below:

enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
.../...
inet 172.16.25.10/16 brd 172.16.255.255 scope global noprefixroute enp2s0
.../...

enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
.../...
inet 172.16.25.15/16 brd 172.16.255.255 scope global noprefixroute enp3s0
.../...

And for h2d we have:

enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
.../...
inet 172.16.25.20/16 brd 172.16.255.255 scope global noprefixroute enp2s0
.../...

enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
.../...
inet 172.16.25.25/16 brd 172.16.255.255 scope global noprefixroute enp3s0
.../...

According to the cabling of the ports, we will therefore want:
- 172.16.25.10 (h2a) to talk with 172.16.25.20 (h2d) and vice-versa
- 172.16.25.15 (h2a) to talk with 172.16.25.25 (h2d) and vice-versa

So if we were to write some software opening TCP sessions on both enp2s0 and enp3s0 and sending/receiving data, the application would really have a potential of 5 GbE bandwidth available.

Let's prove it using iperf3 (it's already written...)

On the h2a, let's open two terminals and start iperf3 server on enp2s0 and enp3s0.

Terminal 1
domih@h2a:~$ iperf3 -s --bind 172.16.25.10

Terminal 2
domih@h2a:~$ iperf3 -s --bind 172.16.25.15

From h2d, we run:

domih@h2d:~$ iperf3 -c 172.16.25.10 --bind 172.16.25.20
Connecting to host 172.16.25.10, port 5201
[ 5] local 172.16.25.20 port 54759 connected to 172.16.25.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 275 MBytes 2.31 Gbits/sec 0 655 KBytes
.../...
[ 5] 9.00-10.00 sec 280 MBytes 2.35 Gbits/sec 0 902 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.73 GBytes 2.35 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.73 GBytes 2.34 Gbits/sec receiver

Then we run:

domih@h2d:~$ iperf3 -c 172.16.25.15 --bind 172.16.25.25
Connecting to host 172.16.25.15, port 5201
[ 5] local 172.16.25.25 port 51427 connected to 172.16.25.15 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 280 MBytes 2.35 Gbits/sec 0 761 KBytes
.../...
[ 5] 9.00-10.00 sec 281 MBytes 2.36 Gbits/sec 0 891 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec 0 sender
[ 5] 0.00-10.00 sec 2.74 GBytes 2.35 Gbits/sec receiver

Yeah! So far so good. BUT we ran the two tests one after the other we therefore only proved that h2d can talk max 2.5 Gbe to h2a.

We need to run the two tests at the same time. Let's open two terminals on h2d, one will talk to h2a port 1 the other one to h2a port2.

We get:

Terminal 1 (on h2d)

domih@h2d:~$ iperf3 -c 172.16.25.10 --bind 172.16.25.20 -t 20
Connecting to host 172.16.25.10, port 5201
[ 5] local 172.16.25.20 port 45571 connected to 172.16.25.10 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 105 MBytes 877 Mbits/sec 0 414 KBytes
.../...
[ 5] 19.00-20.00 sec 188 MBytes 1.57 Gbits/sec 0 850 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 2.17 GBytes 933 Mbits/sec 0 sender
[ 5] 0.00-20.00 sec 2.17 GBytes 932 Mbits/sec receiver


Terminal 2 (on h2d)

domih@h2d:~$ iperf3 -c 172.16.25.15 --bind 172.16.25.25 -t 20
Connecting to host 172.16.25.15, port 5201
[ 5] local 172.16.25.25 port 60541 connected to 172.16.25.15 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 231 MBytes 1.94 Gbits/sec 0 703 KBytes
.../...
[ 5] 19.00-20.00 sec 175 MBytes 1.47 Gbits/sec 0 1.40 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 3.44 GBytes 1.48 Gbits/sec 0 sender
[ 5] 0.00-20.00 sec 3.44 GBytes 1.48 Gbits/sec receiver


*&^%$#@!

1.48 + 0.933 = 2.413 Gbits/sec which is roughly the max you can hope (because 2.5 GbE - linecode 64b/66b - IP overhead).

The problem comes from the fact that we are using boards with several NICs, this was called multi-homed hosts in the past - meaning in the time it was so leading edge it needed some nice buzzword. So as soon as you have two NICs which are connected and you want to send something over the network the OS (here Linux) needs to be told which NIC to use. Linux does "choose" using its kernel routing table which is automatically created and you have nothing to do with it if you use one NIC or two NICs on different Intranet classes. The latter are 10.n.n.n, 172.16.n.n and 192.168.n.n (with some additional restrictions for 192.168.n.n.)

If we had assigned let's say 10.0.0.10 (and 20) to the enp2s0 NICs and 192.168.1.20 (and 25) to the enp3s0 we would have gotten our parallel 2 x 2.35 Gbits/sec = 4.7 Gbits/sec because Linux would know exactly where to send the 10.n.n.n packets and through which NIC; same thing for the 192.168.n.n packets.

Because in this hands-on all our addresses are in the same subnet 172.16.25.n Linux will fall into a trap when establishing the routing table. Here is the routing table we get:

Code: Select all

domih@h2a:~$ route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         router.asus.com 0.0.0.0         UG    100    0        0 enx00249b4995ac
10.0.0.0        0.0.0.0         255.0.0.0       U     150    0        0 ibp1s0
link-local      0.0.0.0         255.255.0.0     U     1000   0        0 enx00249b4995ac
172.16.0.0      0.0.0.0         255.255.0.0     U     103    0        0 enp2s0
172.16.0.0      0.0.0.0         255.255.0.0     U     104    0        0 enp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 enx00249b4995ac
The mess occurs because the OS ended up with the SAME subnet|ip_address/netmask (here 172.16.0.0/255.255.0.0) assigned to enp2s0 AND enp3s0.

This means that each time we ask the OS to send something to a 172.16.n.n host, the OS will find the first entry 172.16.0.0/255.255.0.0 for enp2s0 and will therefore use that NIC. Got the picture? Even though you tell the OS to send through enp3s0 (e.g. --bind 172.16.25.25 in the iperf3 calls) the low-level OS routing will go through enp2s0 because that's what it finds first in the routine table and there is nothing entry better matching. So in this hands-on we so far have tried to send 2 x 2.5GbE through enp2s0. 2 x 2.5 does not fit in 2.5 so the OS did the best it could: shared the bandwidth of the NIC between the two sessions. Thus 1.48 + 0.933 = 2.413 Gbits/sec.

To fix this, we need to give the OS more information about who is connected to what: in other words, we need to edit the routing table.

To do this easily, we can use: sudo nmtui and use the GUI.

The Edit Connection dialog includes the Routing option. Let's select <Edit...>. We get:

edit-connection.png
edit-connection.png (99.95 KiB) Viewed 124 times
Let's tell the OS that if we want to send something to h2a port 2 (172.16.25.15/32) it has to be through this NIC on h2d (enp3s0):

add-route.png
add-route.png (119.27 KiB) Viewed 124 times
Notes
- The /32 is the mask, here 255.255.255.255 because we are expressing a route to one host.
- Next Hop is the gateway for this route. In this example there is no gateway so we leave it empty.
- Metric: see https://geek-university.com/ccna/routin ... explained/ or https://en.wikipedia.org/wiki/Metrics_(networking) I entered 1500 in the screenshot above because I was thinking about Ethernet metric but this is wrong :-)

We then do the same thing for h2d (enp2s0) which we want to use to send something to h2a port 1.

Deactivate and reactivate enp2s0 and enp3s0 on h2d. The new routine table on h2d is:

Code: Select all

domih@h2d:~$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    600    0        0 wlx7cdd90ebfda8
169.254.0.0     0.0.0.0         255.255.0.0     U     1000   0        0 enp3s0
172.16.0.0      0.0.0.0         255.255.0.0     U     107    0        0 enp2s0
172.16.0.0      0.0.0.0         255.255.0.0     U     108    0        0 enp3s0
172.16.25.10    0.0.0.0         255.255.255.255 UH    1500   0        0 enp2s0
172.16.25.15    0.0.0.0         255.255.255.255 UH    1500   0        0 enp3s0
192.168.1.0     0.0.0.0         255.255.255.0   U     600    0        0 wlx7cdd90ebfda8
172.16.25.10/255.255.255.255 and 172.16.25.15/255.255.255.255 are more a match than 172.16.0.0/255.255.0.0. So the routine table is telling the OS: for 172.16.25.10 as destination, use enp2s0 and for 172.16.25.15 use enp3s0.

Therefore the parallel calls to iperf3 in the two h2d consoles will use their separate ways :-) And yes we get two parallel throughput of 2 x 2.35 Gbits/sec = 4.7 Gbits/sec.

To use iperf3 reverse mode (-R) meaning h2a is sending to h2d with the calls still made on h2d, we also need to add routes on h2a for using enp2s0 to reach 172.16.25.20 and using enp3s0 to reach 172.16.25.25. We then get simultaneously:

domih@h2d:~$ iperf3 -c 172.16.25.10 --bind 172.16.25.20 -t 20 -R
Connecting to host 172.16.25.10, port 5201
Reverse mode, remote host 172.16.25.10 is sending
[ 5] local 172.16.25.20 port 45521 connected to 172.16.25.10 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 217 MBytes 1.82 Gbits/sec
.../...
[ 5] 18.00-19.00 sec 281 MBytes 2.35 Gbits/sec
[ 5] 19.00-20.00 sec 280 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 5.41 GBytes 2.33 Gbits/sec 8 sender
[ 5] 0.00-20.00 sec 5.41 GBytes 2.32 Gbits/sec receiver



domih@h2d:~$ iperf3 -c 172.16.25.15 --bind 172.16.25.25 -t 20 -R
Connecting to host 172.16.25.15, port 5201
Reverse mode, remote host 172.16.25.15 is sending
[ 5] local 172.16.25.25 port 49795 connected to 172.16.25.15 port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 260 MBytes 2.18 Gbits/sec
.../...
[ 5] 19.00-20.00 sec 281 MBytes 2.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-20.00 sec 5.46 GBytes 2.35 Gbits/sec 2 sender
[ 5] 0.00-20.00 sec 5.46 GBytes 2.34 Gbits/sec receiver


HTH

Note to the networking experts, wizards and gurus: if you did read the whole thing, do not hesitate to comment with clarifications and fixes. TIA!

mad_ady
Posts: 8316
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 573 times
Been thanked: 434 times
Contact:

Re: [HOWTO] Make 2 x H2+ use the full potential of 2 x 2.5 GbE ports = 5 GbE total

Post by mad_ady »

What you're saying is correct. The only thing I want to nitpick about is the netmask you used on your interfaces - /16 is waaaay too big/wasteful :)

In real-world scenario your apps won't know how to open multiple connections and balance them. That's the job of the network layer.

Here's how I would do it (not tested) - with equal-cost multipath (https://blog.fhrnet.eu/2019/03/07/ecmp- ... nel%204.18.)

1. Enable ip routing on both h2+ (h2a, h2d)
2. Assign /30 ips on pair interfaces - e.g.
- h2a eth0: 172.16.0.1/30 -- .2/30 eth0 h2d
- h2a eth1: 172.16.0.5/30 -- .6/30 eth1 h2d
You can keep the current addressing, but they're technically in the same lan, but unreachable since not all links are in the same broadcast domain
3. Insert the dummy module (modprobe dummy) and assign two /32 addresses
h2a: 192.168.100.1/32
h2d: 192.168.100.2/32
4. Install either specific routes, or default ecmp routes on each node - like
to 192.168.100.1 via 172.16.0.1 and 172.16.0.5
5. Then run iperf with 192.168.100.1 destination.
These users thanked the author mad_ady for the post:
domih (Mon Jul 20, 2020 4:11 pm)

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 0 guests