[SOLVED] Help me troubleshoot: very low ssh / sftp throughput

Post Reply
ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

[SOLVED] Help me troubleshoot: very low ssh / sftp throughput

Unread post by ulfben » Sun Feb 02, 2020 9:02 pm

I'm running Ubuntu 18.04.4 LTS (GNU/Linux 4.14.165-172 armv7l) on the Odroid XU4.

My server is suddenly (eg. without any environment or configuration changes except regular apt-get upgrades) providing extremely low throughput over sftp. I've checked that the NIC and network is in fact working properly (406Mbit/sec when running iperf over the LAN, 203Mbit/sec running speedtest-cli over the Internet).

When transferring a big file from the Odroid over SFTP, in the LAN, I get ~1.1Mbit/s. htop shows a single BIG core loaded at max ~20%. Start two streams and they both seem to share that meager performance - ~500-600Kbit/s each, with very little load on the CPU.

I am testing from both the external USB3 drive and internal SD card (both EXT4 filesystems, both clean according to e2fck). The throughput remains consistently low from both sources.

I had a theory that maybe the LUKS-crypto had stopped being accelerated by the hardware, but then I'd see the CPU spike trying to crunch the math, right? So... what's my next step - how do I troubleshoot the low throughput of sshd?

For my own future reference:

Checking that the network card is identified correctly and running at full speed:

Code: Select all

$ sudo ethtool eth0 | grep Speed
        Speed: 1000Mb/s

$ sudo mii-tool eth0
eth0: negotiated 1000baseT-FD flow-control, link ok
Checking the actual performance of the network card:

Code: Select all

$ sudo iperf -c 192.168.1.9
------------------------------------------------------------
Client connecting to 192.168.1.9, TCP port 5001
TCP window size: 43.8 KByte (default)
[...]
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   484 MBytes   406 Mbits/sec
(I tried running iperf in both directions, with very similar results)
Last edited by ulfben on Mon Feb 03, 2020 7:14 pm, edited 1 time in total.

User avatar
mad_ady
Posts: 7375
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 358 times
Been thanked: 258 times
Contact:

Re: Help me troubleshoot: very low ssh / sftp throughput

Unread post by mad_ady » Sun Feb 02, 2020 10:45 pm

Well, 400Mbps on a gigabit lan is pretty slow. You should be getting 900Mbps.
Try perf3 tcp and udp tests and try with fixed bandwidths (e.g. -u -b 500M). See if you are getting considerable packet loss. If yes, start checking cabling.

For ssh - try to isolate the disks from the equation. Try to scp a file from a ramdrive, also locally:
* try scp -vvv for extra debugging
* try dd if=/dev/zero of=/dev/shm/zero bs=1M count=500; scp /dev/shm/zero root@localhost:/dev/shm/zero.1
* are you getting low speed with wget and https as well? wget -O /dev/null https://somewhere/big.iso
These users thanked the author mad_ady for the post:
ulfben (Mon Feb 03, 2020 7:14 pm)

ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Help me troubleshoot: very low ssh / sftp throughput

Unread post by ulfben » Mon Feb 03, 2020 1:49 am

I'm not overly concerned with the 400Mbps LAN transfer speed as that was run from my laptop over WiFi.

I tried running the tests you suggested. Results:

Code: Select all

dd if=/dev/zero of=/dev/shm/zero bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 1.0253 s, 511 MB/s

Code: Select all

sudo scp /dev/shm/zero ulfben@localhost:/dev/shm/zero.1
zero   	92%  464MB  52.3MB/s   00:00 ETAs
Pulling down a big file from the Apache webserver gives the same low performance as over sftp, ~1.10MB/s. (using wget over wifi from my ubuntu laptop)

ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Help me troubleshoot: very low ssh / sftp throughput

Unread post by ulfben » Mon Feb 03, 2020 2:09 am

Found something! I've been connecting via my domain name (someurl.com) which gets routed to my public IP. When connecting directly to the local IP address I see much more reasonable throughput!

HTTPS:

Code: Select all

wget -O /dev/null https://192.168.1.90/MediaLogics.pdf --no-check-certificate
--2020-02-02 18:02:48--  https://192.168.1.90/MediaLogics.pdf
Connecting to 192.168.1.90:443... connected.
    WARNING: certificate common name ‘[redacted].com’ doesn't match requested host name ‘192.168.1.90’.
HTTP request sent, awaiting response... 200 OK
Length: 285315250 (272M) [application/pdf]
Saving to: ‘/dev/null’

/dev/null                     100%[=================================================>] 272.10M  49.2MB/s    in 5.6s
sftp: ~8-6MB/s

These numbers match what I normally get from this setup.

So I guess I have a few different questions now:
- how come routing via my public IP is so extremely slow despite my internet connection being at least 200mbit/s?
- is there anything I can do to have devices in my LAN get the more direct route, despite using the domain name? (I'm using pihole as my primary dns resolver, if that helps. But I guess the HTTPS certificate issue is going to bite me?)

Thanks!

User avatar
mad_ady
Posts: 7375
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 358 times
Been thanked: 258 times
Contact:

Re: Help me troubleshoot: very low ssh / sftp throughput

Unread post by mad_ady » Mon Feb 03, 2020 4:02 am

Ok, I'm not sure exactly what is going on, but could be NAT + some sort of proxy or maybe throughput limitation imposed by the firewall or tc. What router do you have? What os is it running? A packet capture might help on the router side.

Regarding pihole, I think you can expect the same performance when running wget https://wan-ip/huge.pdf
You can force it to use the lan ip by adding the public dns name mapped to your lan ip. Check the dnsmasq man page on how to do it.


svily0
Posts: 4
Joined: Wed Sep 02, 2015 6:29 pm
languages_spoken: english
ODROIDs: XU4, HC1
Has thanked: 0
Been thanked: 0
Contact:

Re: [SOLVED] Help me troubleshoot: very low ssh / sftp throughput

Unread post by svily0 » Tue Feb 04, 2020 2:01 pm

If it's your server and infrastructure - ip route add is your friend, no need for NAT (I think). By the way, I had noticed some years ago that ssh / sftp was rather slow and tried pure ftp and rsync, which both worked at full speed. I just decided there is not enough computing power on my XU4 to do the encryption.

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 7 guests