Samba slow throughput over OpenVPN

Post Reply
Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Samba slow throughput over OpenVPN

Unread post by Spada » Tue Mar 24, 2020 5:32 am

I am running a Samba server on a HC2 running Ubuntu 18.04, kernel 4.14.173. In the same LAN I have a N2. Writing from N2 to HC2 I get about 500 Mbps. Pretty nice!

The N2 is also an OpenVPN server. I am connecting from outside the network and accesing the HC2 Samba share. The problem is that this way I get about 2-4 Mbps download or upload! Tried with different clients: Windows, Ubuntu and MacOS. There doesn't seem to be any fragmentation.

However, when I am connecting over OpenVPN (UDP), I can see one core on the HC2 staying at 100% and it's samba taking it... When I connect from N2 and get the 500 Mbps, I can see 2-3 cores going to 50-60%, but no higher. I moved the PID to each core, no difference.

Any ideas how I can get more information about the high CPU (e.g. is it network/packets)? Or any other ideas are welcome!

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by o_lampe » Tue Mar 24, 2020 5:51 pm

Is the N2 updated to the latest kernel? They said they found the reason, why USB3.0 write speed (locally) is so slow on N2.
I haven't seen any improvement yet, only short bursts of data transmission followed by long pauses. The average speed isn't faster. (yet)

Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by Spada » Tue Mar 24, 2020 6:26 pm

Kernel on N2 is 4.9 (see below). Not sure if this is the latest or not.

uname -a
Linux N2_living 4.9.196-63 #1 SMP PREEMPT Thu Oct 17 00:44:03 -03 2019 aarch64 aarch64 aarch64 GNU/Linux

User avatar
mad_ady
Posts: 7577
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 397 times
Been thanked: 302 times
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by mad_ady » Tue Mar 24, 2020 9:40 pm

USB3 on N2 shouldn't matter for this issue.
Can you try a bidirectional iperf test between your remote client and:
1. The N2's tunnel ip address
2. The N2's eth0 ip address
3. The HC2's eth0 ip address

I suspect the N2 is doing NAT, to hide your tunnel addressing, right? Or do you have static routes from HC2 back to the N2?

Though nat shouldn't be a bottleneck...

Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by Spada » Tue Mar 24, 2020 10:47 pm

Well this is odd...

[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 9.37 MBytes 7.86 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 9.15 MBytes 7.67 Mbits/sec receiver

Directly to N2. But last week I copied some files directly from N2 with rsync and I could reach 40 Mbps.

User avatar
mad_ady
Posts: 7577
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 397 times
Been thanked: 302 times
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by mad_ady » Tue Mar 24, 2020 11:26 pm

Last week Europe wasn't in lockdown mode... :lol:

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by o_lampe » Wed Mar 25, 2020 6:41 pm

Spada wrote:
Tue Mar 24, 2020 6:26 pm
Kernel on N2 is 4.9 (see below). Not sure if this is the latest or not.

uname -a
Linux N2_living 4.9.196-63 #1 SMP PREEMPT Thu Oct 17 00:44:03 -03 2019 aarch64 aarch64 aarch64 GNU/Linux
4.9.196 is qite old. IIRC it 4.9.216 already.

I get the idea, N2-Ubuntu is a work in progress and they are only halfway through.
How else can you explain, that a shabby oldfashioned 5400RPM HDD is MUCH faster in sequential write than a modern SSD? (Same USB3.0 uasp-SATA adapters)

Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by Spada » Thu Mar 26, 2020 6:42 pm

After a risky kernel update, I am now running 4.19.216 and I get about 90 Mbps (download speed is 100 Mbps). I am more than happy with this.

But this is just on N2. HC2 is still slow. Investigating it further...

Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by Spada » Tue Mar 31, 2020 3:37 am

Ok, HC2 is now offline after I've played with the TCP buffers. Gonna try a hard reboot when I'll get (physically) near it.

In the mean time, I have added the following configuration to my OpenVPN server config:

sndbuf 393216
rcvbuf 393216
push "sndbuf 393216"
push "rcvbuf 393216"

The results are by far better, I am able to reach 80 Mbps on wireless, from my laptop, in another room. If I knew how, I'd close this thread. :)
These users thanked the author Spada for the post:
mad_ady (Tue Mar 31, 2020 4:34 am)

User avatar
mad_ady
Posts: 7577
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 397 times
Been thanked: 302 times
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by mad_ady » Tue Mar 31, 2020 4:37 am

When doing risky config remotely you should use something like (untested!) sudo shutdown -r 300 to force a reboot in 300s if you don't cancel it.

How does the hc2 behave now? TCP RST on ssh connection from LAN? Maybe you can replicate the settings on the N2 to reestablish the link.
Also, what did you change? So we know not to do the same mistake.

Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by Spada » Wed Apr 01, 2020 4:29 am

I changed the TCP receive and send window size and enabled...selective ack, I think.

I did not lock myself out of the HC2. After changing the above TCP parameters, I issued a "reboot". After that, I got "closed connection" when trying to SSH. Ping worked, but for about two days I got rejected ssh connections. After that, it stopped responding to ping and I have no ARP entry... :(

User avatar
mad_ady
Posts: 7577
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 397 times
Been thanked: 302 times
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by mad_ady » Wed Apr 01, 2020 1:35 pm

So you made the changes persistent and rebooted? Hmm... you might need serial (or pulling out the sd to undo the changes)...

Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by Spada » Mon Apr 06, 2020 9:02 pm

Ok, I got someone to reboot my HC2. I made some changes to both OpenVPN server config (increased buffers) and HC2.

OpenVPN server:

sndbuf 26214400
rcvbuf 26214400
push "sndbuf 26214400"
push "rcvbuf 26214400"

HC2:

sysctl -w net.core.rmem_max=26214400
sysctl -w net.core.rmem_default=26214400

The above config is based on: https://medium.com/@CameronSparr/increa ... d167bb1360

The above config changes gives me about 80-90 Mbps to N2 (sweet, I am happy with that) over WiFi, but only 40 Mbps to HC2. Although 40 Mbps is more than I had when I first started this thread, I am still not happy. Mainly, because the a core of the HC2 is spiking to 100% which I guess limits the performance.

Any ideas how to troubleshoot the high core usage due to samba? Go wild, I am willing to test anything.

User avatar
mad_ady
Posts: 7577
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 397 times
Been thanked: 302 times
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by mad_ady » Mon Apr 06, 2020 9:46 pm

Ok - a couple of questions:
1. Which core goes to 100%? A little or a big core? Is it the samba process spiking? Are you testing with top or htop? top can show you CPU usage from kernel threads as well (e.g. interrupts, etc).
2. What governor are you using on the HC2?
3. What iperf3 performance do you have between HC2 and your OpenVPN client (going throigh N2)? How does it compare to a test between the client and N2?
4. Are you copying a big file or lots of small files? Is your disk write performance on your OpenVPN client more than 40Mbps?

Spada
Posts: 26
Joined: Tue Oct 18, 2016 1:23 am
languages_spoken: english
ODROIDs: C2, XU4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Samba slow throughput over OpenVPN

Unread post by Spada » Tue Apr 07, 2020 12:32 am

Ok, so after getting mad that I could not figure it out, I removed Samba and went with NFS. At first, I also got the same poor performance, but no CPU high utilization. After tweaking the TCP values, I am now able to reach a constant 80 Mbps over WiFi, which is more than acceptable to me.

@mad_ady, just to answer your questions:

1. It was 5, but I've played the previous week with it and moved the process to all cores (one at a time, obviously), without any noticeable results.
2. Performance.
3. It was about the same (a little better, but not by much) - meaning about 8-9 Mbps.
4. One big file. Locally, I could copy from N2 to HC2 with about 500 Mbps. The other way round (from HC2 to N2), I got about 700 Mbps (understandable, since N2 has eMMC, while HC2 is using a classic HDD).

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 11 guests