SOLVED: force ssh to take connections on BIG cores?

Post Reply
ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

SOLVED: force ssh to take connections on BIG cores?

Post by ulfben »

I did some rudimentary experimentation; downloading multi-gigabyte files from the XU4, over SFTP, and checking htop while the streams are active.

It seems like streams that are served by the BIG-cores provides ~9MB/sec (served from an encrypted spinning disk)
Streams serves by the LITTLE cores gives roughly half the throughput ~4MB/sec.


Is there any way for me to force ssh to put its workloads on the BIG cores, without messing with the rest of the system? (eg. I'd prefer to not entirely disable the smaller cores as my server does a lot of low-key work throughout the day)

Thanks!
Last edited by ulfben on Sun Apr 01, 2018 4:25 am, edited 1 time in total.

User avatar
odroid
Site Admin
Posts: 34943
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean, Japanese
ODROIDs: ODROID
Has thanked: 959 times
Been thanked: 775 times
Contact:

Re: Can I force ssh to take connections on BIG cores?

Post by odroid »

Run the SFTP program with 'taskset' command.
Refer this link.
viewtopic.php?f=146&t=26007&p=194453#p194453

mad_ady
Posts: 8333
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 574 times
Been thanked: 439 times
Contact:

Re: Can I force ssh to take connections on BIG cores?

Post by mad_ady »

I managed to get ~50MB/s over sftp with some tweaks (non-encrypted storage). Look into the NAS article in february 2017 odroid magazine article for details.

ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Can I force ssh to take connections on BIG cores?

Post by ulfben »

odroid wrote:Run the SFTP program with 'taskset' command.
Refer this link.
viewtopic.php?f=146&t=26007&p=194453#p194453
Thanks! That seems just right. Any pointers for where to look, to apply taskset to the normal startup procedure? Eg. sshd starts with the machine, and I'm unsure where that happens exactly.
mad_ady wrote:I managed to get ~50MB/s over sftp with some tweaks (non-encrypted storage). Look into the NAS article in february 2017 odroid magazine article for details.
ODROID-Magazine-201702.pdf (for future reference)
ODROID-XU4 with OpenMediaVault (linked from your article. discusses good governor settings and seems relevant)
The article's support thread

Those performance tricks looks great! I've got some experimentation to do. Thanks a ton.

ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Can I force ssh to take connections on BIG cores?

Post by ulfben »

@mad_ady: I'm looking at /etc/init.d/ssh to have it start up as a bigcore task. I can't find anything resembling a ExecStartPost step in here. Am I in the wrong place?

Or put bluntly; what should I do to have SSH brought up on the bigcores after following your article?

mad_ady
Posts: 8333
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 574 times
Been thanked: 439 times
Contact:

Re: Can I force ssh to take connections on BIG cores?

Post by mad_ady »

The ssh daemon is started by /lib/systemd/system/ssh.service. You can do

Code: Select all

sudo cp /lib/systemd/system/ssh.service /etc/systemd/system
sudo vi /etc/systemd/system/ssh.service
And make the changes there. Make sure you configure the cgroups through systemd as well

ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Can I force ssh to take connections on BIG cores?

Post by ulfben »

Thank you kindly!

I have another question, but I'll post it in the support thread for the article as it doesn't directly relate to sshd affinity.

ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: SOLVED: force ssh to take connections on BIG cores?

Post by ulfben »

I have applied the following tweaks from your article:
- the tweaked ondemand governor
- handle USB / networking interrupts on big cores
- used cgroups to run sshd on big cores

Serving from the BIG cores did fix my original concern - all streams are now shuffled at roughly the same speed - ~8MB/s. This is still far slower than the performance numbers I see you guys report - is it likely that LUKS (drive encryption) and ssh encryption overhead would impact performance this much? My encrypted volume is EXT4, if that makes any difference.

An interesting observation:
3 simultaneous streams = ~6-7MB/s each
2 simultaneous streams = ~7-8MB/s each
1 single stream = ~8MB/s

Eg. one stream or three, they all shuffle across the network at roughly the same speed. This suggest I'm CPU starved more than anything, right?


Do you think it would make a difference if I move /usr/lib/openssh/sftpserver to BIG-cores as well? (it seems every stream = one instance of sftpserver and one instance of sshd)

mad_ady
Posts: 8333
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 574 times
Been thanked: 439 times
Contact:

Re: SOLVED: force ssh to take connections on BIG cores?

Post by mad_ady »

I had experimented with a wrapper around sftp-server to pin it to the big cores by pointing subsystem sftp to this:

Code: Select all

adrianp@aldebaran:~⟫ cat /usr/local/bin/sftp-server-cgroup
#!/bin/bash
#move to a specific cgroup
echo $PPID | sudo tee -a /sys/fs/cgroup/cpuset/bigcores/tasks 2>/dev/null 1>/dev/null
echo $$ | sudo tee -a /sys/fs/cgroup/cpuset/bigcores/tasks 2>/dev/null 1>/dev/null
exec /usr/lib/openssh/sftp-server "$@"
But it was unnecessary if you pin the parent ssh in the bigcores cgroup, since it is inherited by the child process as well.

You can check during a filetransfer with htop on which core the process is executing.
Also try reading/writing from non-encrypted storrage to benchmark just ssh performance. Lastly, consider reducing the encryption settings in ssh for an added 10MB/s.

ulfben
Posts: 112
Joined: Thu Jul 20, 2017 11:56 pm
languages_spoken: english
ODROIDs: XU4Q
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: SOLVED: force ssh to take connections on BIG cores?

Post by ulfben »

mad_ady wrote:[...]
Also try reading/writing from non-encrypted storrage to benchmark just ssh performance. [...]
That's good advice. I'll definetly bench an unencrypted drive to see if there's a difference. And then, perhaps, some experimentation on LUKS-settings to increase throughput.

Does the XU4 provide any sort of hardware acceleration for encryption and if so, what algorithms does it prefer? Can I verify if LUKS is making use of the HW for this?


(I am, for future reference, currently running LUKS with:

Cipher name: aes,
Cipher mode: xts-plain64 )

User avatar
odroid
Site Admin
Posts: 34943
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean, Japanese
ODROIDs: ODROID
Has thanked: 959 times
Been thanked: 775 times
Contact:

Re: SOLVED: force ssh to take connections on BIG cores?

Post by odroid »

Refer this WiKi page about the encrypted storage test with a hardware accelerator.
https://wiki.odroid.com/odroid-xu4/soft ... encryption

But it seems not to be much faster than software encryption.
We couldn't find what made bottlenecks.

mad_ady
Posts: 8333
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 574 times
Been thanked: 439 times
Contact:

Re: SOLVED: force ssh to take connections on BIG cores?

Post by mad_ady »

Even if it's the same speed as software it has the benefit of freeing up software resources during encryption which cann be used for something else (e.g ssh)
@odroid: There may be a speed benefit if you enable cryptodev in kernel and recompile openssl/luks with cryptodev support. I meant to try it, but I never had the time. There's an openssl+cryptodev thread for c2 you can follow for ideas.

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 1 guest