Slow copy over network

Post Reply
npelov
Posts: 11
Joined: Thu Dec 24, 2020 10:21 pm
languages_spoken: english
ODROIDs: Odroid H2+
Has thanked: 0
Been thanked: 1 time
Contact:

Slow copy over network

Post by npelov »

I can't figure out why my Odroid H2+ is copying slow over network.
OS: CentOS 7 - I have lots of servers with the same setup
SSD: Kingston 256 GB NVMe - speeds are over 1GB/s, read speed is about 2GB/s
memory: 16G Samsund

I tested network performance with iperf3: 940 Mbit/s download and 941Mbit/s upload. Nothing wrong here (connected to gigabit network).

However when I copy a file to Odroid over samba (MByte/s):
8-9 MB/s using TeraCopy
10-11 MB/s Using windows copy

Copying file from Odroid:
40MB/s using tera copy
50MB/s using windows copy

The cpu usage is about 16-20% to H2+, 50% from H2+. It's not even boosting - stays at 800MHz. I tested boost using gzip - works fine. When I load the cpu it goes to 2.3-2.4 GHz.

Nothing changes when I try to copy to tmpfs instead of ssd.

Downloading from local web site to odroid works fine - 111 MB/s
It's the same for downloading from odroid's web server to a local machine - 111 MB/s

Am I missing something?

User avatar
odroid
Site Admin
Posts: 36930
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1595 times
Been thanked: 1071 times
Contact:

Re: Slow copy over network

Post by odroid »

Very weird.
We could reach over 200MB/s even with HDDs. But we've used only Ubuntu.
https://wiki.odroid.com/accessory/conne ... card#samba

What happens if you change the CPU governor to 'performance'?

mad_ady
Posts: 9250
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 599 times
Been thanked: 622 times
Contact:

Re: Slow copy over network

Post by mad_ady »

Are you copying one big file, or lots of small files?

npelov
Posts: 11
Joined: Thu Dec 24, 2020 10:21 pm
languages_spoken: english
ODROIDs: Odroid H2+
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Slow copy over network

Post by npelov »

I'm copying 1GB file created with /dev/urandom. Same file that I use for web and disk speed tests

npelov
Posts: 11
Joined: Thu Dec 24, 2020 10:21 pm
languages_spoken: english
ODROIDs: Odroid H2+
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Slow copy over network

Post by npelov »

I just tried ubuntu live from usb - same problem. 10 mb/s to H2+, 70MB/s From H2+.
I tested file transfer from Windows to my Xeon E2100 server (also CentOS 7, NVMe) - to: 60MB/s, From 112MBs ... maybe it's the windows that's causing the problem.
I also tried to mount the H2 on my Xeon Server (linux to linux). The speed was awful - to H2: 3MB/s, from H2: <1MB/s. I mounted using cache=loose and the copy went to ~100MB/s both directions. So linux to linux works. However why there is such a big difference between the Xon server and Odroid H2+ when communicating from windows? It's not limited by network speed - that's tested and it's not limited by CPU. There shouldn't be difference between the Xeon server and the Odroid.

mad_ady
Posts: 9250
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 599 times
Been thanked: 622 times
Contact:

Re: Slow copy over network

Post by mad_ady »

You can try a packet capture. See if you get tcp retransmissions - those are usually lost packets. If you get tcp window zero messages, there is somd buffer getting filled at one end that needs to empty to resume transfer...

npelov
Posts: 11
Joined: Thu Dec 24, 2020 10:21 pm
languages_spoken: english
ODROIDs: Odroid H2+
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Slow copy over network

Post by npelov »

Can you give me the command and what to look for. tcpdump?

npelov
Posts: 11
Joined: Thu Dec 24, 2020 10:21 pm
languages_spoken: english
ODROIDs: Odroid H2+
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Slow copy over network

Post by npelov »

@mad_ady THANK YOU!... no need for packet capture. I just eliminated the possible cause. My QNAP NAS was plugged in my Linksys gigabit switch. The odroid is plugged in Asus gigabit switch which is connected to the linksys switch. I moved the odroid to linksys switch and the problem is mostly gone! Copying to Odroid increased from 10 MB/s to 96MB/s, but copying from Odroid to windows is still not filling the 1Gbit - 50-60MB/s (50% cpu, stays at 800MHz).

I'm not sure if this is the effect of any switch chaining or my Asus switch is not doing well. It's quite slow and I had to replace a capacitor on it. The cap was bulging and the switch was restarting. After the change I never had network outage with it, but I never tested it for performance too.

mad_ady
Posts: 9250
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 599 times
Been thanked: 622 times
Contact:

Re: Slow copy over network

Post by mad_ady »

Hmm, strange since the iperf tests went fine (were they over tcp or udp?)... Could be a buffer issue on an intermediate switch... You could also test with a direct cable between your pc and h2, to rule out the switches/network...
These users thanked the author mad_ady for the post:
TurdFurguson (Sun Feb 21, 2021 4:54 am)

jovany
Posts: 14
Joined: Mon Feb 22, 2021 4:35 am
languages_spoken: english
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Slow copy over network

Post by jovany »

odroid wrote:
Thu Feb 18, 2021 12:26 pm
Very weird.
We could reach over 200MB/s even with HDDs. But we've used only Ubuntu.
https://wiki.odroid.com/accessory/conne ... card#samba

What happens if you change the CPU governor to 'performance'?
What is the usual transfer speed via SFTP between H2 and H2+?
I have both computers and H2+ have an Net card installed and running.H2+ computer is configured as router and running Opnsense, on the H2 I'm running Ubuntu 20.04 LTS but when I've tried to transfer 2GB file via SFTP the transfer speed was around 75 MB/s.It is so weird that speed.I was expecting much faster transfer.On both computers I have Samsung SSDs and 16GB of RAM.
Since the H2 have a Gigabit Ethernet port I was expecting speed around 110 MB/s, but it is really much slower.The Ethernet cable which I'm using is CAT 6.

mad_ady
Posts: 9250
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 599 times
Been thanked: 622 times
Contact:

Re: Slow copy over network

Post by mad_ady »

Check if cpu usage for a core is 100%. That would mean that encryption is the bottleneck.
Also, try scp from a ramdisk to a ramdisk (e.g. /dev/shm) to rule out disk/filesystem.
And 75MB/s is about 75% of the gigabit link, which is not bad...

jovany
Posts: 14
Joined: Mon Feb 22, 2021 4:35 am
languages_spoken: english
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Slow copy over network

Post by jovany »

mad_ady wrote:
Mon Mar 01, 2021 4:21 am
Check if cpu usage for a core is 100%. That would mean that encryption is the bottleneck.
Also, try scp from a ramdisk to a ramdisk (e.g. /dev/shm) to rule out disk/filesystem.
And 75MB/s is about 75% of the gigabit link, which is not bad...
I can't check the CPU usage between H2 and H2+ because on H2+ the OS is FreeBSD(because of OPNsense) and seems there is no package "htop", but I tried a transfer between H2 and Raspeberry pi 4 and this is what CPU usage I get:
CPU.png
CPU.png (194.96 KiB) Viewed 126 times
The screenshot is "htop" from RPI 4.

The transfer between H2 and RPI 4 is around 65MB/s, much slower, but I think it depends from the slower ARM CPU and also from the SATA to USB 3.0 controller which is asmedia 1153e.(On RPI 4 I'm booting from Samsung SSD)
I also think the encryption is limiting the speed and also one of my computer is H2 which is with 1 Gigabit ports and the other one computer is H2+ which have 2.5 Gigabit ports and also a slightly faster CPU 1.8 GHz.

One other forum member wrote me his transfer speeds are a little bit faster, but he is using two H2+ computers.
@brad
Did a quick test between Windows -> Linux

Upload to H2+ @ 91Mb/s
Download from H2+ @ 83 Mb/s

The limiting factor here is encryption, in my case using:

File transfer protocol = SFTP-3
Cryptographic protocol = SSH-2
SSH implementation = OpenSSH_8.4p1 Debian-3
Encryption algorithm = AES-256 SDCTR (AES-NI accelerated)

brad
Posts: 1380
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 N2+ H2 H2+ (64 bit ftw)
Location: Australia
Has thanked: 120 times
Been thanked: 181 times
Contact:

Re: Slow copy over network

Post by brad »

I think this will be related to power saving settings in CentOS, can you try to set them to performance or away from any dynamic settings (sorry not sure in CentOS)? I had a similar issue with another product running a different linux distribution and in power saving mode I could only reach around 10Mb/s and cpu at only 800MHz.

Essentially the network transfer was not working the cpu hard enough for it to scale up and provide good network performance when the cpu was idle (as cpu ticks the network interfaces interrupts I believe). Now with high ramp up settings I get around 180Mb/s (iSCSI) and testing samba just now I see 200Mb/s+
These users thanked the author brad for the post (total 2):
TurdFurguson (Tue Mar 02, 2021 3:48 am) • odroid (Tue Mar 02, 2021 12:01 pm)

jovany
Posts: 14
Joined: Mon Feb 22, 2021 4:35 am
languages_spoken: english
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Slow copy over network

Post by jovany »

brad wrote:
Mon Mar 01, 2021 11:17 pm
and testing samba just now I see 200Mb/s+
Seems If I want to achieve much faster speeds I have to use Samba, but what about the security?
Can you access your Opnsense when you are outside for example and using smartphone?
Did you configure some Firewall rules, which are giving access via SSH and did you disabled Web login interface? Because of the security purpose is not good to login with username and password.

brad
Posts: 1380
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 N2+ H2 H2+ (64 bit ftw)
Location: Australia
Has thanked: 120 times
Been thanked: 181 times
Contact:

Re: Slow copy over network

Post by brad »

jovany wrote:
Tue Mar 02, 2021 11:49 pm
brad wrote:
Mon Mar 01, 2021 11:17 pm
and testing samba just now I see 200Mb/s+
Seems If I want to achieve much faster speeds I have to use Samba, but what about the security?
Can you access your Opnsense when you are outside for example and using smartphone?
Did you configure some Firewall rules, which are giving access via SSH and did you disabled Web login interface? Because of the security purpose is not good to login with username and password.
There is quiet a lot to cover there so ill try to make a start.

- If you want to achieve much faster speeds then you have to disable encryption. (same for almost all devices and not restricted to Odroid) This will be a security concern if you are transferring the data over a network you do not trust as anyone with access to the network could potentially capture your data in transit.
- You don't necessarily need to use Samba for high speeds (ie samba with encryption enabled will be slower as well). It could be normal FTP, NFS, HTTP, etc.
- Samba only works well with low latency network connections. ie it is not realty designed for use over the internet.

In regards to opnsense I don't have any access enabled for external interfaces only my local trusted ones . I have ssh disabled but if I need it I can temporarily enable it from the web GUI or I can access via console / HDMI. I have SSL enabled and enforced on the web interface to secure it some more. There is nothing wrong with a username and password if they are managed correctly but for maximum security I would suggest using multifactor authentication - https://docs.opnsense.org/manual/two_factor.html

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 1 guest