Slow data transfer speed over unidirectional connection

[squitel]

Hello,

With a colleague, we are trying to transfer data between two ODroid H2+ over a unidirectional [(one way) Ethernet IP link (using UDP sockets).
Between the two ODroid H2+ we have an IP optical device which physically guarantees that data can only go from "A" to "B" and not from "B" to "A" (unidirectional).
It works well, however we have very limited transfer speeds because we cannot use a packet size greater than the MTU size (1500 by default).
If we try to exceed that limit, the ODroid H2+ "A" still manage to send the packets but the receiver "B" only receives very few.

More details below:

• 1) With the exact same optical device, we successfully made this working at high speed (exceeding the MTU limit for packets size) on other hardware platforms like Raspberry Pi 3/4. We sent big packets (i.e. 65k) and the kernel handled efficiently the split in smaller pieces of data to stay under the MTU size.
  
  2) We added USB3-Ethernet Gigabit adapters (extra lan card on usb) to both ODroid H2+ and we made tests with the iperf command (iperf -c 20.0.0.2 -u -b 800M -n 1000M -i 1). Result: we reached pretty good speeds (840Mbps).
  
  3) We tried to increase, step by step, the MTU size and increase accordingly our packets size but once again, above 1500 bytes, transfer speed dropped down.
  
  4) We also connected a regular Ethernet cable in replacement of our optical device with guarantee the unidirectional link and in this situation we could exceed the MTU size and send packets of 65500 bytes, having thus very good transfer speeds. Note: As this optical device works well on other platforms like Raspberry Pi we can't suspect it; it looks like it's more the fact the the link is strictly unidirectional that induces the problem on ODroid H2+...

Hence the question: does anyone have an idea of what could explain why, in a unidirectional transfer with ODroid H2+, we cannot use a packet size that exceeds the MTU limit whereas in the same time, it works flawlessly on Raspberry Pi platforms for example ?

I know it's a very specific and technical question but thank you in advance for any help.

[mad_ady]

Hmm, interesting question. Having an unidirectional link shouldn't matter with regards to MTU. MTU is there to ensure you're not forcing fragmentation at a lower layer.

I would check:
* mtu size in ifconfig on both ethernet ends
* udp input buffer size in the receiving kernel (somewhere in /proc or /sys)
* tcpdump on the incoming interface - are the udp packets received but dropped because of kernel?
* static arp on both nodes so that the sender knows where the destination is without relying on arp, and so that the destination knows the source is reachable (though it isn't) and doesn't drop traffic from unreachable sources (see rp_filter).

[squitel]

Hi mad_ady,
Thank you very much for answering.

We'll check 1) mtu size, 2) udp input buffer size and 3) tcpdump.
Regarding 4) static arp, it's already done.

I'll post here the results and/or conclusion.

[squitel]

Hi mad_ady,

We check the different things you pointed out:

1) MTU size: confirmed at 1500 (or more depending on tests ran).

2) udp input buffer size: parameters applied as mentioned here: https://github.com/cea-sec/hairgap
in fact we apply the following commands before running our tests:

Code: Select all

sudo sysctl -w net.core.rmem_max=67108864
sudo sysctl -w net.core.rmem_default=67108864
sudo sysctl -w net.core.netdev_max_backlog=10000
sudo sysctl -w net.ipv4.udp_mem="49314528 65752736 3082158"

3) tcpdump: we managed to make some tests this morning.
Command tcpdump issued:

Code: Select all

sudo tcpdump -i enp3s0 udp port 36016 -vv

Files are sent 3 times sequentially at each test below
TEST 1 to TEST 4:
File transfer with our code
Sender = A ; Receiver = B
MTU = 1500 for the used interfaces
TEST 5:
C++ simple code in replacement of our more complex code
Sender = A ; Receiver = B
MTU = 1500 for the used interfaces
TEST 6 to 8:
Sender = A ; Receiver = B
MTU = 4500 for the used interfaces
TEST 9:
Sender = A ; Receiver = B
MTU = 9000 for the used interfaces
Results are listed in my next post(s)...

[squitel]

The tests results are in the attached TransferTests.txt file.

Basically we can see that increasing MTU size above 1500 leads to problems, whatever packet size we use, even if we artificially limit the transfer speed or not.
We made these tests with our proprietary code and as well we a very simple minimal C++ code just handling the file transfer part and saw that the results were exactly the same, so our code does not bring the problem.

On Raspberry Pi 3/4 platforms & Raspbian (Debian based distrib), if we sent 65500 bytes packets with MTU=1500, the kernel handled pretty well the split of our jumbo packets in smaller packets with respect to the MTU size (28 bytes less -> 1472) and overall performance was good.
It seems that on ODroid running Debian 10 it's not the case and that setting the MTU size over 1500 is not correctly handled...

[mad_ady]

Ok, by the looks of it, the sender doesn't seem to send the packets (am I wrong?) in most failed tests. This can be because it can't fragment them...
Here's some documentation about what I think is going on (and also why non-standard MTU is generally not recommended):
https://blog.cloudflare.com/ip-fragmentation-is-broken/
https://stackoverflow.com/questions/973 ... n-a-socket
https://en.wikipedia.org/wiki/Large_send_offload
https://serverfault.com/questions/45984 ... r-then-mtu

I'd try disabling LSO on the sender NIC and seeing if the packets respect the imposed MTU.

[squitel]

Thank you for your suggestions and the very interesting URL!

I disabled LSO on the sender NIC ("A") as you suggest as described here : https://docs.gz.ro/tuning-network-cards-on-linux.html

Code: Select all

sudo ethtool --offload enp3s0 rx off tx off
sudo ethtool -K enp3s0 rx off tx off

Let's check:

Code: Select all

sudo ethtool -k enp3s0 | grep generic-segmentation-offload
generic-segmentation-offload: off
sudo ethtool -k enp3s0 | grep tx-
tx-checksumming: off
        tx-checksum-ipv4: off
        tx-checksum-ip-generic: off [fixed]
        tx-checksum-ipv6: off
        tx-checksum-fcoe-crc: off [fixed]
        tx-checksum-sctp: off [fixed]
        tx-scatter-gather: on
        tx-scatter-gather-fraglist: off [fixed]
        tx-tcp-segmentation: off [requested on]
        tx-tcp-ecn-segmentation: off [fixed]
        tx-tcp-mangleid-segmentation: off
        tx-tcp6-segmentation: off [requested on]
tx-vlan-offload: on
tx-lockless: off [fixed]
tx-gso-robust: off [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-nocache-copy: off
tx-vlan-stag-hw-insert: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
tls-hw-tx-offload: off [fixed]

We then ran basically the same tests as before and saw no change.

As soon as we change the MTU size from 1500 to a higher value "it" becomes crazy, even if we use packets with a size lower than the MTU. In this configuration we only receive the very small (respectively 57 and 42 bytes each) specific packets that we send before and after the "payload packets".

If the MTU = 1500 bytes and packet size is higher than the MTU, then things are better than in the previous case, nevertheless we experiment huge losses and therefore file transfers fail.

We are not sure but it seems to us that the sender NIC ("A") fragments packets even if told not to do so. And on its side, the reveiver NIC ("B") is unable to reassemble these fragments it receives from the "A" side...

[squitel]

Follow up of what I wrote above...

If I try to send big icmp packets (38787 bytes is the maximum value that works here) from the "A" (sender) side to the "B" (receiver) side, I see with tcpdump that packets are fragmented (to lenght 1500) although they are supposed not to be because what is disabled on the "A" (sender) side (with the ethtool commands).

on "A" :

Code: Select all

ping -s 38787 20.0.0.2

while on "B" :

Code: Select all

sudo tcpdump -i enp3s0 icmp -vv
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:20:27.363658 IP (tos 0x0, ttl 64, id 29685, offset 0, flags [+], proto ICMP (1), length 1500)
    20.0.0.1 > 20.0.0.2: ICMP echo request, id 3025, seq 1, length 1480
16:20:27.363702 IP (tos 0x0, ttl 64, id 29685, offset 1480, flags [+], proto ICMP (1), length 1500)
    20.0.0.1 > 20.0.0.2: icmp
16:20:27.363707 IP (tos 0x0, ttl 64, id 29685, offset 2960, flags [+], proto ICMP (1), length 1500)
    20.0.0.1 > 20.0.0.2: icmp
16:20:27.363711 IP (tos 0x0, ttl 64, id 29685, offset 4440, flags [+], proto ICMP (1), length 1500)
    20.0.0.1 > 20.0.0.2: icmp
[...]
^C
14 packets captured
108 packets received by filter
94 packets dropped by kernel

Maybe have I wrongly disabled the LSO on the sender NIC?

[mad_ady]

So, the icmp test is done with mtu 1500, right?
How does tcpdump look like with mtu 1510?

[squitel]

Good question. Yes MTU was 1500.

On "A" (sender):

Code: Select all

ip a
enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500] qdisc pfifo_fast state UP group default qlen 1000

On "B" (receiver):

Code: Select all

ip a
enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

I ran the test again just in case and it shown the same result:

Code: Select all

tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:53:31.777719 IP (tos 0x0, ttl 64, id 28921, offset 0, flags [+], proto ICMP (1), [b]length 1500[/b])
    20.0.0.1 > 20.0.0.2: ICMP echo request, id 3029, seq 1, length 1480
16:53:31.777757 IP (tos 0x0, ttl 64, id 28921, offset 1480, flags [+], proto ICMP (1), length 1500)
    20.0.0.1 > 20.0.0.2: icmp
...

[mad_ady]

Fragmentation looks fine. Is the receiver not responding? Also, does it look the same with a higher mtu?
You can also get more stats with netstat -s. They clear post reboot.

[mad_ady]

Also what do these say on the receiver?

Code: Select all

cat /proc/sys/net/ipv4/ipfrag_high_thresh 
cat /proc/sys/net/ipv4/ipfrag_low_thresh

[squitel]

The receiver does not respond because we are on a strictly unidirectional connection.
When connected with a simple Ethernet cable instead (bidirectional), it answers.

What happens with higher MTU values is interesting...

Allways the same command is issued on the "A" side (sender) :

Code: Select all

ping -s 38787 20.0.0.2

With MTU = 2000 bytes
On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp3s0 icmp -vv
sudo tcpdump -i enp3s0 icmp -vv
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:46:54.864067 IP (tos 0x0, ttl 64, id 40451, offset 0, flags [+], proto ICMP (1), length 1996)
    20.0.0.1 > 20.0.0.2: ICMP echo request, id 3049, seq 1, length 1976
17:46:54.864119 IP (tos 0x0, ttl 64, id 40451, offset 1976, flags [+], proto ICMP (1), length 1996)
    20.0.0.1 > 20.0.0.2: icmp

We see that the fragments look right (1996)


With MTU = 2500 bytes
On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp3s0 icmp -vv
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
18:10:01.846288 IP (tos 0x0, ttl 64, id 9218, offset 37200, flags [none], proto ICMP (1), length 1615)
    20.0.0.1 > 20.0.0.2: icmp
18:10:02.870308 IP (tos 0x0, ttl 64, id 9236, offset 37200, flags [none], proto ICMP (1), length 1615)
    20.0.0.1 > 20.0.0.2: icmp

We see that the fragments are becoming much smaller (1615) than expected!


With MTU = 3000 bytes
On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp3s0 icmp -vv
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:47:26.694325 IP (tos 0x0, ttl 64, id 46108, offset 38688, flags [none], proto ICMP (1), length 127)
    20.0.0.1 > 20.0.0.2: icmp
17:47:27.703190 IP (tos 0x0, ttl 64, id 46180, offset 38688, flags [none], proto ICMP (1), length 127)
    20.0.0.1 > 20.0.0.2: icmp

We see that the fragments are very small (127)!


With MTU = 4000 bytes
On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp3s0 icmp -vv
tcpdump: listening on enp3s0, link-type EN10MB (Ethernet), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

No packets are transmitted!

[squitel]

Also what do these say on the receiver?

Code: Select all

cat /proc/sys/net/ipv4/ipfrag_high_thresh 
cat /proc/sys/net/ipv4/ipfrag_low_thresh

On the receiver:

Code: Select all

cat /proc/sys/net/ipv4/ipfrag_high_thresh
4194304
cat /proc/sys/net/ipv4/ipfrag_low_thresh
3145728

[mad_ady]

I don't know... it's weird, indeed.

[squitel]

You're right, it's weird...

I tried to see if I could reproduce this behaviour when in bidirectional with different MTU values, and things are a bit different...

Allways the same command is issued on the "A" side (sender) :

Code: Select all

ping -s 38787 10.0.0.56
	PING 10.0.0.56 (10.0.0.56) 38787(38815) bytes of data.
	38795 bytes from 10.0.0.56: icmp_seq=1 ttl=64 time=2.36 ms

With MTU = 2000 bytes
On the sender side ("A") :

Code: Select all

--- 10.0.0.56 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 8ms

On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp2s0 icmp -vv
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:01:32.721639 IP (tos 0x0, ttl 64, id 49562, offset 0, flags [+], proto ICMP (1), length 1996)
    10.0.0.55 > 10.0.0.56: ICMP echo request, id 3417, seq 1, length 1976
09:01:32.721697 IP (tos 0x0, ttl 64, id 49562, offset 1976, flags [+], proto ICMP (1), length 1996)
    10.0.0.55 > 10.0.0.56: icmp
09:01:32.721702 IP (tos 0x0, ttl 64, id 49562, offset 3952, flags [+], proto ICMP (1), length 1996)
    10.0.0.55 > 10.0.0.56: icmp
09:01:32.721706 IP (tos 0x0, ttl 64, id 49562, offset 5928, flags [+], proto ICMP (1), length 1996)
    10.0.0.55 > 10.0.0.56: icmp
[...]
25 packets captured
120 packets received by filter
95 packets dropped by kernel

We see that the fragments look right (1996)


With MTU = 3000 bytes
On the sender side ("A") :

Code: Select all

--- 10.0.0.56 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 3ms

On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp2s0 icmp -vv
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:02:38.135827 IP (tos 0x0, ttl 64, id 55577, offset 0, flags [+], proto ICMP (1), length 2996)
    10.0.0.55 > 10.0.0.56: ICMP echo request, id 3421, seq 1, length 2976
09:02:38.135869 IP (tos 0x0, ttl 64, id 55577, offset 2976, flags [+], proto ICMP (1), length 2996)
    10.0.0.55 > 10.0.0.56: icmp
09:02:38.135875 IP (tos 0x0, ttl 64, id 55577, offset 5952, flags [+], proto ICMP (1), length 2996)
    10.0.0.55 > 10.0.0.56: icmp
09:02:38.135879 IP (tos 0x0, ttl 64, id 55577, offset 8928, flags [+], proto ICMP (1), length 2996)
    10.0.0.55 > 10.0.0.56: icmp
[...]
14 packets captured
56 packets received by filter
42 packets dropped by kernel
1 packet dropped by interface

We see that the fragments look normal whereas in unidirectional mode packet size droped to 127 bytes!
Maybe there's a sort of negociation?


With MTU = 9000 bytes
On the sender side ("A") :

Code: Select all

--- 10.0.0.56 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 11ms

On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp2s0 icmp -vv
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
09:03:23.823999 IP (tos 0x0, ttl 64, id 56213, offset 0, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: ICMP echo request, id 3424, seq 1, length 8976
09:03:23.824063 IP (tos 0x0, ttl 64, id 56213, offset 8976, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: icmp
09:03:23.824071 IP (tos 0x0, ttl 64, id 56213, offset 17952, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: icmp
09:03:23.824077 IP (tos 0x0, ttl 64, id 56213, offset 26928, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: icmp
09:03:23.824082 IP (tos 0x0, ttl 64, id 56213, offset 35904, flags [none], proto ICMP (1), length 2911)
    10.0.0.55 > 10.0.0.56: icmp
09:03:23.824190 IP (tos 0x0, ttl 64, id 19490, offset 0, flags [+], proto ICMP (1), length 8996)
    10.0.0.56 > 10.0.0.55: ICMP echo reply, id 3424, seq 1, length 8976
09:03:23.824198 IP (tos 0x0, ttl 64, id 19490, offset 8976, flags [+], proto ICMP (1), length 8996)
    10.0.0.56 > 10.0.0.55: icmp
[...]
15 packets captured
20 packets received by filter
5 packets dropped by kernel

Here again, this time fragments look normal with the "ping -s 38787 10.0.0.56" command.

With this same MTU value, if I now try with greater packet size (65500), this time it gets worst:
On the sender side ("A") :

Code: Select all

ping -s 65500 10.0.0.56
PING 10.0.0.56 (10.0.0.56) 65500(65528) bytes of data.
^C
--- 10.0.0.56 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 159ms

On the receiver side ("B") :

Code: Select all

sudo tcpdump -i enp2s0 icmp -vv
tcpdump: listening on enp2s0, link-type EN10MB (Ethernet), capture size 262144 bytes
11:19:47.792275 IP (tos 0x0, ttl 64, id 36270, offset 0, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: ICMP echo request, id 3495, seq 1, length 8976
11:19:47.792335 IP (tos 0x0, ttl 64, id 36270, offset 8976, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: icmp
11:19:47.792343 IP (tos 0x0, ttl 64, id 36270, offset 17952, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: icmp
[...]
11:26:08.272261 IP (tos 0xc0, ttl 64, id 19228, offset 0, flags [none], proto ICMP (1), length 576)
    10.0.0.56 > 10.0.0.55: ICMP ip reassembly time exceeded, length 556									<--- /!\
        IP (tos 0x0, ttl 64, id 60642, offset 0, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: ICMP echo request, id 3504, seq 2, length 8976
11:26:08.272323 IP (tos 0xc0, ttl 64, id 19229, offset 0, flags [none], proto ICMP (1), length 576)
    10.0.0.56 > 10.0.0.55: ICMP ip reassembly time exceeded, length 556									<--- /!\
        IP (tos 0x0, ttl 64, id 60495, offset 0, flags [+], proto ICMP (1), length 8996)
    10.0.0.55 > 10.0.0.56: ICMP echo request, id 3504, seq 1, length 8976
11:26:08.420757 IP (tos 0x0, ttl 64, id 64572, offset 0, flags [+], proto ICMP (1), length 8996)
[...]
^C
28 packets captured
28 packets received by filter
0 packets dropped by kernel

This time, packets still have a correct size but they are not reconstructed on the receiver side as the sender side displays 100% icmp requets lost and we can see "ICMP ip reassembly time exceeded" messages in the tcpdump output.

I finally tried this "ping -s 65500 10.0.0.56" command with a regular MTU=1500 on both sides and could observe the same (near 100% icmp packet lost and the "ICMP ip reassembly time exceeded" message from time to time).

I don't see if the problem comes from the Realtek NIC or the driver but I guess it's one of these two. Do you share my assumption?

[mad_ady]

Hmm, as long as higher mtu works with a usb dongle, yes, I'd say it's driver related

[squitel]

Once again, thank you very much for your answers!

Just before lunch I could make the "ping -s" tests on two HP DL360 servers in place of the ODroids.
Hereunder is what I observed:

• 1) With an increased MTU size (let's say 5000 bytes), if I send with "ping -s" increasing size packets, I have the same behaviour than with ODroid: the bigger the packets are on the sender, the smaller the packets are received on the receiver (i.e. MTU=5000, packet size=65500 -> on the receiver the size of the packets received is only 840 bytes whereas I would have awaited a value close to 5000). This seems weird to me.

• 2) With a regular MTU value of 1500, if I send with "ping -s" 65500 bytes packets, then the driver+NIC handled successfully the fragmentation on the sender and the reconstruction on the receiver. On Raspberry Pi 3/4 it worked as well but on the ODroid this was not the case. Therefore this problem seems to be related to the "ODroid H2+" platform (more accurately the NIC or its driver I think).

[squitel]

As far as I heard, 2.5GbE infrastructure had no such issue while 1GbE showed glitches with some network switches/routers.

Anyway, the upstream Kernel 5.9 seems to be including a newer RTL8125B driver.
viewtopic.php?p=300484#p300484
Once Kernel 5.9 RC is available in a couple of weeks, it must be worth to try.

[mad_ady]

Regarding performance - why are you changing the MTU? The only benefits (as far as I know) of having a higher MTU is reducing interrupts on the receiving side (though that can be done with interrupt coalescing on the receiving NIC, if supported), and having slightly higher throughput (because of fewer headers in the way). The cost would be - a lost packet wastes more bandwidth with retransmission.
But since H2+'s NIC can do 2.5Gbps (minus overhead) stably with iperf on 1500 MTU, why not using 1500 for your unidirectional application?

Also, as a note - I've noticed UDP can drop packets way below the top link bandwidth in my tests on gigabit lan. So, hopefully your application has some redundancy/retransmission built-in, even on unidirectional links.

[squitel]

Thank you @odroid and @mad_addy for your answers!
We studied the possibility to increase the MTU size to be able to send bigger packets while staying below the MTU value but our preference is to keep a MTU=1500.

We managed to narrow down our problem...
Three experiments with packet size = 1470 bytes and MTU = 1500bytes.

1) Each ODroid's Realtek 2.5Gbps card connected to a 1Gbps switch (bidirectionnal) or to our 1Gbps Ethernet-optical converters (unidirectional)
-> In this case the 2.5Gbps NIC are used at only 1Gbps
With a code written in full Python or another one in "Python + C++"
=> On the receiver side, we fill a buffer much quicker than we can empty it because of performance problems related to the Python GIL ((Global Interpreter Lock).
With a code written in full C++
=> everything is OK, data is correctly transfered, no performance or saturation problem.
With packets greater than 1470 bytes:
=> problem with the Realtek drivers (maybe the upcoming 5.9 kernel would patch this issue).

2) A simple Ethernet cable wired between the Realtek 2.5Gbps of each ODroid (bidirectional)
-> In this case the 2.5Gbps NIC are used at full speed
With a code written in full Python or in "Python + C++" ou in full C++
=> everything is OK, data is correctly transfered, no performance or saturation problem.
With packets greater than 1470 bytes:
=> everything is OK

3) A 1Gbps USB3-Ethernet adapter is connected to each ODroid. Both ODroid are connected by a Ethernet cable (bidirectionnal), to a 1Gbps switch (bidirectionnal) or to our 1Gbps Ethernet-optical converters (unidirectional)
-> In this case the 2.5Gbps NIC are used at only 1Gbps
With a code written in full Python or another one in "Python + C++"
=> On the receiver side, we fill a buffer much quicker than we can empty it because of performance problems related to the Python GIL ((Global Interpreter Lock).
With a code written in full C++
=> everything is OK, data is correctly transfered, no performance or saturation problem.
With packets greater than 1470 bytes:
=> no packet arrives on the receiver but it's not related to our buffer problem, cause remains unknown at this time.

We are convinced that our problem is directly related to the Realtek 2.5Gbps driver.
We'll now rewrite some parts of our code to avoid using Python GIL and thus avoid loosing precious time at each packet on the receiver side.
We'll also pay attention to the availability of 5.9 kernel to check if by chance the Realtek driver would be patched.

If we can keep this thread open for a while, I'll provide you with a feedbak on this subject.

Once again thank you very much for having helped us with our problem.

[squitel]

As said previously, I now come back to provide you with a feedbak on this subject.

After installation/compilation of 5.9.0 RC2 kernel on top of our original Debian 10 (4.19 kernel):

• network speed is now OK and jumbo packets are correctly split into smaller ones according to the MTU size.

• however if we increase the MTU size, then we still experiment packets loss (but as with high end servers using Intel NIC ; note that with optical fiber cards and SFP the MTU size can be increased flawlessly).

No more need to install the Realtek driver if you use 5.9.0 RC2 kernel. Looking forward to seeing it available as LTS (probably with Debian 12?).