OpenMediaVault for XU3 and XU4

[Gornok]

Some more dd

root@OMV:~# dd if=/dev/sda1 of=/dev/null iflag=direct bs=8M count=64

Code: Select all

64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 5.36564 s, 100 MB/s

root@OMV:~# dd if=/dev/zero of=/dev/sda1 oflag=direct bs=8M count=64

Code: Select all

64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 5.90563 s, 90.9 MB/s

Iperf as server and host

root@OMV:~# iperf -s

Code: Select all

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.10 port 5001 connected with 192.168.0.101 port 63659
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec   433 MBytes   363 Mbits/sec

root@OMV:~# iperf -c 192.168.0.101

Code: Select all

------------------------------------------------------------
Client connecting to 192.168.0.101, TCP port 5001
TCP window size: 20.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.10 port 54929 connected with 192.168.0.101 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec   792 MBytes   664 Mbits/sec

My smb. conf file, I've tried FTP as well and getting exactly same transfer as SMB protocol.

smb.conf

Code: Select all

#======================= Global Settings =======================
[global]
workgroup = WORKGROUP
server string = %h server
dns proxy = no
log level = 0
syslog = 0
log file = /var/log/samba/log.%m
max log size = 1000
syslog only = yes
panic action = /usr/share/samba/panic-action %d
encrypt passwords = true
passdb backend = tdbsam
obey pam restrictions = yes
unix password sync = no
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n $
pam password change = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY
guest account = nobody
load printers = no
disable spoolss = yes
printing = bsd
printcap name = /dev/null
unix extensions = yes
wide links = no
create mask = 0777
directory mask = 0777
map to guest = Bad User
use sendfile = yes
aio read size = 16384
aio write size = 16384
null passwords = no
local master = yes
time server = no
wins support = no

#======================= Share Definitions =======================
[Data]
path = /media/19d379b1-3679-4491-adc9-b83cd72c70d5/Data/
guest ok = yes
read only = no
browseable = yes
inherit acls = yes
inherit permissions = no
ea support = no
store dos attributes = no
printable = no
create mask = 0755
force create mode = 0644
directory mask = 0755
force directory mode = 0755
hide dot files = yes
read list =
write list =

iostat while copying 1 GB file to /dev/sda

root@OMV:~# iostat -x 3

Code: Select all

Linux 3.10.82+ (OMV)    01/01/2013      _armv7l_        (8 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           2.11    0.03    5.81    1.74    0.00   90.31

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           9.83     2.07   29.90    1.93   685.06   116.47    50.36     0.35   11.08    1.23  163.49   1.20   3.81
mmcblk0rpmb       0.01     0.00   41.81    0.00   167.25     0.00     8.00     0.00    0.11    0.11    0.00   0.11   0.45
mmcblk0boot1      0.00     0.00    6.34    0.00    25.32     0.00     7.99     0.00    0.45    0.45    0.00   0.44   0.28
mmcblk0boot0      0.00     0.00    6.24    0.00    24.92     0.00     7.99     0.00    0.45    0.45    0.00   0.45   0.28
sda              47.40   708.50    8.42   54.47    34.84  6147.78   196.64     6.84  108.76    0.62  125.47   1.23   7.73

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.98    0.00    2.76    0.89    0.00   95.38

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.67    0.00     4.00     0.00    12.00     0.00    2.50    2.50    0.00   2.50   0.17
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.33    0.00   58.33     0.00  6588.00   225.87     0.68   10.86    0.00   10.86   1.29   7.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.07    0.00   17.40    2.40    0.00   79.13

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.00    0.00  174.33     0.00 20437.33   234.46    19.39   93.96    0.00   93.96   1.04  18.17

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.43    0.00   23.53    5.27    0.00   70.77

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     3.67    0.33  345.00     1.33 40652.00   235.44    43.16  133.81   60.00  133.88   1.11  38.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.86    0.00   24.82    5.36    0.00   68.97

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.33    0.00  420.67     0.00 49561.33   235.63    49.34  109.78    0.00  109.78   1.03  43.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.52    0.00   23.31    4.15    0.00   72.02

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     2.33    0.00  319.00     0.00 37549.33   235.42    33.68  115.51    0.00  115.51   0.99  31.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.56    0.00   24.26    5.37    0.00   69.80

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     2.00    0.00  441.00     0.00 52012.00   235.88    48.74  110.54    0.00  110.54   1.01  44.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.90    0.00   24.32    4.36    0.00   70.43

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.33    0.00  287.33     0.00 33857.33   235.67    32.32  112.49    0.00  112.49   0.97  28.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.34    0.00   24.68    4.94    0.00   70.04

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     3.00    0.00  439.67     0.00 51833.33   235.78    49.21  111.94    0.00  111.94   1.02  44.67

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.99    0.00   23.73    4.13    0.00   71.15

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     0.67    0.00  296.67     0.00 34853.33   234.97    38.27  109.67    0.00  109.67   1.13  33.67

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.52    0.00   23.36    5.16    0.00   70.96

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0rpmb       0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot1      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
mmcblk0boot0      0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00    0.00    0.00   0.00   0.00
sda               0.00     2.00    0.00  428.67     0.00 50581.33   235.99    48.32  126.10    0.00  126.10   1.01  43.50

Thank you for all your help!

[Obihoernchen]

Your network seems to be the bottleneck. You should achieve ~800-900 Mbit/s throughput.
Did you run these tests with performance governor?
Do you use wifi?
What's your client OS? Linux or Windows?

[Gornok]

Yes I have performance governor turned on. I don't use wifi, I've tried those tests again using link between PC and XU4 without any switch, I've also tried different cable. I'm getting similar results as above.

If XU4 is the server

Code: Select all

Transfer     Bandwidth
440 MBytes   369 Mbits/sec

If XU4 is the client

Code: Select all

Transfer     Bandwidth
665 MBytes   550 Mbits/sec

My PC runs a Windows 10.

[deggo]

I am getting similar results to Gornok transferring files from Windows 10. Have tried everything. Yet a friend with almost identical setup gets ~80MB/s.

I also get +70MB/s copying from same Windows 10 PC to another server on my network.

[mopmop]

I would like to install owncloud on my XU4 which runs with openmediavault. To add the repository I need to know the Linux distribution. As far as I know, is openmediavault based on debian. Is it Debian_8.0 or Debian_7.0?

thanks in advance

[Gornok]

So I'm not alone anyone else getting similar transfers on odroid ?

@Mopmop:

It's debian 7.9. You should be able to find the Owncloud plugin for OMV.

[ryecoaaron]

I would like to install owncloud on my XU4 which runs with openmediavault. To add the repository I need to know the Linux distribution. As far as I know, is openmediavault based on debian. Is it Debian_8.0 or Debian_7.0?

OMV 2.x is Debian 7. OMV 3.x will be Debian 8.

[MrDrizz]

Does this only work on emmc? I burned the iso to a SD card but the system won't boot.

[ryecoaaron]

I built the image on SD if that answers your question

[kiomon]

I just installed OMV on a cloudshell and had a couple of questions.

1. What is the proper way of booting up the cloudshell? When I plug it up, It doesn't always power up immediately. The blue light is blinking in a slow pattern but even after 30 mins it's still not loaded. Then sometimes if I just reapply power it boots right up. What am I missing here?

2. The LCD displays the couple of lines of information but then eventually seems to go asleep/ blank with the backlight on. Is there a way to prevent this? Can we install the cloudshell LCD stats program from the wiki over this included display without issue?

[kicker22004]

It should say HI who am I : root something something... Personnaly I want this to stop.. Anyone know where the script is that is running it??

[Obihoernchen]

Hey guys,

I wrote another blog on how to improve your network and usb speed even more
Hope this is useful for some of you: https://obihoernchen.net/1416/odroid-xu ... usb-speed/

[DarkBahamut]

I'm going to have to disagree with that actually. I've run hundreds upon hundreds of network related tests on the XU4 to optimise performance and setting your network interrupt to run on an A15 is generally a performance loss and just isn't a good thing to do. You are better off keeping the interrupt on the A7 and enabling RPS + TPS instead. Lets back this up with some evidence though.

First off, the blog post uses iperf for testing. While iperf is a great tool for simple diagnostic work, it's only useful as a true performance test if your main workload is running iperf all day long. Given this is a OMV thread and no one is likely running iperf much at all, lets put that to one side and lets see what happens when you run some real workloads with these changes.

Test 1: SMB transfer from XU4 (eMMC) to Windows 10 PC (7200RPM HDD). Interrupt on CPU 0, RPS + TPS enabled.

http://i.imgur.com/Qii8h6f.jpg

This is my daily use tweaked setup. Pretty nice performance overall. Transfer starts off at 85MB/s and quickly rises to over 100MB. There is a small drop into the high 90's for a while but it's back to 105MB/s by the end of the transfer. Top shows all CPU's have at least a little bit of headroom to spare which is ideal (critically, notice CPU 0 has lots of idle time). I'd say there couldn't be too many complaints with that level of performance.

Test 2: SMB transfer from XU4 (eMMC) to Windows 10 PC (7200RPM HDD). Interrupt on CPU 4, RPS + TPS disabled.

http://i.imgur.com/XVvDfW5.jpg

So now we're on the A15 core, everything should be super fast! And... uh oh... 65MB/s huh? The transfer starts fast, then gets really slow, then recovers then drops right back to the slow speed again. Welcome to the world of stupid Linux scheduling. Top tells us what we need to know, CPU 4 has 0.0% idle time, it's completely overloaded - and that's with my A15's even running at 2.1GHz. CPU 4 is processing the interrupts but then Samba also gets allocated to that CPU (even though it could pick any big CPU) so quickly the performance falls away as the CPU becomes overloaded.

I wish CPU scheduling had a bit more common sense than this, but unfortunately this is what happens in the real world and this is why allocating the interrupt to the A15 is bad news. You can end up in situations where the interrupt and another heavy thread end up together and you really impact your performance, and as we can see from test 1 it's completely unneeded as the A7 with RPS+TPS enabled can easily handle even the heaviest file transfer with tons of free CPU time left over. iperf can mislead you into thinking certainly tweaks are better, but it's the real tests which show up if they are actually better and when it comes to file transfers keeping interrupt on CPU0 and switching on RPS+TPS is definitely the best option


TLDR: Don't move the interrupt to the A15 and run these commands instead.

Code: Select all

echo 2 | sudo tee /sys/devices/usb.2/12400000.dwc3/xhci-hcd.5.auto/usb6/6-1/6-1:1.0/net/eth0/queues/rx-0/rps_cpus

Code: Select all

echo 2 | sudo tee /sys/devices/usb.2/12400000.dwc3/xhci-hcd.5.auto/usb6/6-1/6-1:1.0/net/eth0/queues/tx-0/xps_cpus

[Obihoernchen]

Hi thank you for your comment!
I'll do some more tests next weekend.
Looking at your results RSP and TPS seem to be quite good.

[meveric]

TLDR: Don't move the interrupt to the A15...

Actually if I understand rps_cpus correctly, this is exactly what you did

Code: Select all

echo 2 | sudo tee /sys/devices/usb.2/12400000.dwc3/xhci-hcd.5.auto/usb6/6-1/6-1:1.0/net/eth0/queues/rx-0/rps_cpus

Code: Select all

echo 2 | sudo tee /sys/devices/usb.2/12400000.dwc3/xhci-hcd.5.auto/usb6/6-1/6-1:1.0/net/eth0/queues/tx-0/xps_cpus

From what I've read about rps_cpus this is a BIT mask and by setting certain bits you tell it what CPUs to use.
echo "2" will result in "02" in fact so you set the following bitmask: 0000 0010 which means you lock it to just one CPU core.
In order to set it to multiple CPUs you should actually use something like:
echo "f" which would result in "0f" and a bitmask of 0000 1111.
Still the topic is rather interesting.
The same docu also says setting xps_cpus is actually useless in a single queue environment, which we have.
Also instead of something like: /sys/devices/usb.2/12400000.dwc3/xhci-hcd.5.auto/usb6/6-1/6-1:1.0/net/eth0/queues/rx-0/rps_cpus (which is different for each OS, and config)
You should use /sys/class/net/eth0/queues/rx-0/rps_cpus. It's much shorter and easier to find anyway.

Here's some docu to that topic:
for RPS: https://www.mjmwired.net/kernel/Documen ... ing.txt#99
for XPS: https://www.mjmwired.net/kernel/Documen ... ng.txt#364
Suggestion on XPS: https://www.mjmwired.net/kernel/Documen ... ng.txt#412 which says on a single queue (tx-0 only) it's useless to configure xps_cpus

Here's also a nice article about this topic: http://syuu.dokukino.com/2013/05/linux- ... speed.html

[DarkBahamut]

True on the XPS, I'd forgotten about that

As for RPS, when I had been testing previously I had wanted to use a setting to use any A7 core but I had real difficultly getting it working. I'm a linux noob so I don't understand the CPU masks that well, but I had read that using 'f' which should work, but every time I tried it I only seemed to get improvements occasionally, so I settled on '2' which locks pack processing to CPU 1, which is still an A7 core. The interrupting CPU remains CPU 0 so it still has the effect of bringing two CPU in play.

The slight issue with 'f' is that most of the time it's great, but not all the time. As we can see from the above images the CPU load balancing isn't very good as often two threads end up on one busy CPU while the others remain idle. 'f' has a chance to run the packet processing on CPU 0 still, which is the same as disabling RPS which hurts performance to the order of 40%. This happens sometimes which is why my test show poor performance on occasional tests. CPU 0 is never at 100% load so the scheduler considers it fair game for the packet processing even though the real performance drops massively (likely some latency or cache issue). The optimal setting is to never let RPS be able to use CPU 0 which I guess would be 'e'. I've run a few quick tests and it's looking nice. I can touch 110MB/sec during the transfers so hopefully that's the solution here.

Thanks for the better location to set it also, I'm pretty rubbish at this . So I guess that means this would be the ideal setting:

Code: Select all

echo e > /sys/class/net/eth0/queues/rx-0/rps_cpus

And some quick and dirty benchmarks.

RPS disabled

http://i.imgur.com/c6dwtPj.png


RPS enabled ('e')

http://i.imgur.com/AVdF8Eu.png


The performance difference is kinda ridiculous really

[mad_ady]

I'll jump in with my measurements
I'm running kernel 4.9 with HMP disabled on my XU4. I haven't yet set process affinity per core groups, but it's on my todo list.
I have set all usb interrupts on cores 4-7 (though they all stick to core 4).

Code: Select all

/bin/sh -c 'for i in `cat /proc/interrupts | grep usb | cut -d ":" -f 1`; do echo "4-7" > /proc/irq/$i/smp_affinity_list; done'
root@aldebarano:~# grep usb /proc/interrupts 
 89:          1          0          0          0          0          0          0          0     GIC-0 103 Edge      ehci_hcd:usb5, ohci_hcd:usb6
139:       1112          0          0          0     725640          0          0          0     GIC-0 104 Edge      xhci-hcd:usb1
140:         24          0          0          0    6123899          0          0          0     GIC-0 105 Edge      xhci-hcd:usb3

I'm using a Seagate portable USB3 drive for my tests with a XFS filesystem exported over NFS3. I was able to write a 10GB file with dd locally with a total write speed of 116MB/s.

Tests - with usb affinity set to CPU4 (big):

Code: Select all

1. copy 10GB over NFS XU4->PC (null0)  -> 110MB/s
2. copy 10GB over NFS XU4->PC (disk)    -> 64MB/s
3. copy 10GB over NFS PC  ->XU4 (disk)  -> 50MB/s

During the tests I had the ondemand governor tuned to ramp-up the cores quickly so I had the big cores @2GHz and the fan would occasionally start

Tests - with usb affinity set to CPU4 (big) and rps_cpus enabled (set to 'e'):

Code: Select all

1. copy 10GB over NFS XU4->PC (null0)  -> 113MB/s
2. copy 10GB over NFS XU4->PC (disk)    -> 62MB/s
3. copy 10GB over NFS PC  ->XU4 (disk)  -> 42MB/s

I'm not sure what is different - it could be that the 4.9 kernel without HMP adds a considerable performance penalty, but my real-world tests are somewhat lower than yours.
I will try samba as well (though I expect poorer performance) and also manually setting affinity per service.

[DarkBahamut]

What difference in performance do you get if the IRQ is on a small CPU then you enabled RPS? Using RPS when the IRQ is already on an A15 doesn't make much difference for me (as the A15 usually has enough CPU capacity anyway). It's on the small CPU's where it helps to get you get upto the full link speed.

Would be interesing to know what core the NFS process is running on, and if running it on a big core helps (if it's not running on one already). With Samba I can only manage about 55MB/s if it's run on the A7's (@ 1.5GHz). Full speed definitely requires the A15's for that and even then the CPU's are close to capacity.

[meveric]

sadly I don't know from where it counts, so it's either "e" or "7" to skip on CPU0 but I'm not sure as I said.
Also maybe it's worth switching the CPU stack completely and do an echo "f0" instead?

Maybe it's also worth looking into RFS settings too?
Can someone try the suggested parameters?

Code: Select all

echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
echo 4096 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt

[Obihoernchen]

btw before and between every test you should drop your caches to have consistent results:

Code: Select all

echo 3 > /proc/sys/vm/drop_caches

And using performance governor for all tests is a good idea as well.

Otherwise you will have a lot of deviation in test runs.
And we should use some real benchmark tool. Windows explorer transfer speed isn't very accurate
Some samba/nfs benchmark would be nice.

@mad_ady 4.x kernel seems to be slower in general. I compared 3.10 with 4.6 a few month ago and couldn't reach 3.10 samba performance.

I plan to do some in depth tests next week. Right now I don't have access to my Odroid

[mad_ady]

Yeah, I'm willing to endure the performance drop and even some usb3 instability, but it gives me h264 hardware encoder which I plan on using

[mad_ady]

I've manually bound some processes to specific cpuset cgroups that allow the processes to run on the big or little cores in the XU4. I've forced smbd and nfs-mountd to run exclusively on the big cores and reran the tests:

Tests - with usb affinity set to CPU4 (big) Samba (on big CPU):

Code: Select all

    1. copy 10GB over SMB XU4->PC (null0)  -> 98MB/s
    2. copy 10GB over SMB XU4->PC (disk)    -> 61MB/s
    3. copy 10GB over SMB PC  ->XU4 (disk)  -> 43MB/s

Tests - with usb affinity set to CPU4 (big) and NFS (on big cores):

Code: Select all

    1. copy 10GB over NFS XU4->PC (null0)  -> 111MB/s
    2. copy 10GB over NFS XU4->PC (disk)    -> 59MB/s
    3. copy 10GB over NFS PC  ->XU4 (disk)  -> 54MB/s

Judging by the performance when reading from XU4 I get about the same for SMB and NFS (98MB/s vs 111MB/s). When writing to a physical disk things change and performance drops (presumably because of my system/disk latency). I will try to test over the holidays with writing to an EMMC on a C2...

Also, it's worth to note that samba has worse performance than NFS in terms of CPU usage. Though the big cores remain at 2GHz in both tests, smbd was hogging a core and making it reach 90-95%, while when transferring with NFS the big cores never exceeded 50%.

Edit2: Also, using SCP (bound to the big cores) gets you an upload/download speed bound to ~40MB/s (which is not bad for encrypted traffic). If you were to tune cyphers and lower encryption you could probably get better performance. The ssh process hogs a CPU during the file transfer.

[DarkBahamut]

Theoretical peak Samba performance on an XU4 at 2GHz is around 130MB/s, though the network would never allowed that of course. At lower transfer speeds (like the 50-60MB reported) smbd still shows high CPU usage, but I bet most of the time is spent in iowaits (aka, cpu is actually idle). I've seen very similar results when limiting network performance with Samba.

I'd be interested in anyone manages to get ssh performance better. I sometimes use SFTP for file transfers (it's >10x faster than samba for small transfers - useful for backing up folders with lots of small files) and I can only get around the 40MB/s mark reported when doing a big transfer. I've tried a few things like recompiling openssh and openssl with all manner of tweaks recommended but I can't make the speed budge at all. It's good enough for my needs, but faster is always better

[mad_ady]

Have you tried recompiling openssh with the None cypher? If you don't like that you can enable some weak cyphers (like arcfour) and you should see some speedup. The problem is you have to force your client to use that cypher. Ideas here: http://serverfault.com/questions/116875 ... on-openssh.

Regarding increasing openssl performance you could try to enable cryptodev support and offload the computation to the kernel. I've tried it for the c2 and had better performance (e.g. 10x faster) with big data blocks. But for this you'll need an openssl with cryptodev, the cryptodev kernel module and you need to configure ssh/openssl to use the cryptodev by default (I wasn't able to do this). Oh, and there's the problem that I'm not completely sure that openssh actually uses openssl. Also the xu4 might have a hardware crypto unit but I don't know if it's supported by the kernel/userspace.
I will try with weaker cyphers and report back.

[mad_ady]

@DarkBahamut: I got 52MB/s with arcfour cipher, so that's a 20% speed boost
Here's how (I set up a different sshd instance, but you could add that the regular ciphers to the Cipher line): http://forum.odroid.com/viewtopic.php?f ... 66#p172358

I haven't tried changing hashing algorithms, but it could be done as well. Also, I have tested only arcfour (52MB/s) and blowfish-cbc (30MB/s). Setting Compression "no" didn't seem to have much impact. Other ciphers may have better values. Speed varied between 50 and 60MB/s.
In terms of convenience you could configure ~/.ssh/config to hold a special entry:

Code: Select all

Host odroid-big
Hostname 192.168.228.203
Port 2222
Ciphers arcfour
Compression no

and just use the name...
Let me know if you get better performance with different settings.

[Obihoernchen]

I tested this a few month ago and arcfour was the fastest cipher as well.
Using netcat+tar should be even faster. But this is totally unencrypted.

[DarkBahamut]

Nice results on the encryption. I tried using cryptodev before but it made no difference to the performance at all. I assume the incoming file transfer isn't using it, but I got a bit stuck trying to work out any way to make it work. The engine seems to be in place and it works with the local openssl benchmarks but the file transfer is always stuck at 40MB/s.

Code: Select all

darkbahamut@odroid:~$ ssh -V
OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016

Code: Select all

darkbahamut@odroid:/opt/openssl-1.0.2j/bin$ ./openssl speed -evp aes-128-cbc
Doing aes-128-cbc for 3s on 16 size blocks: 107168 aes-128-cbc's in 0.06s
Doing aes-128-cbc for 3s on 64 size blocks: 100990 aes-128-cbc's in 0.01s
Doing aes-128-cbc for 3s on 256 size blocks: 99232 aes-128-cbc's in 0.02s
Doing aes-128-cbc for 3s on 1024 size blocks: 32468 aes-128-cbc's in 0.03s
Doing aes-128-cbc for 3s on 8192 size blocks: 17926 aes-128-cbc's in 0.00s
<snip>
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128-cbc      28578.13k   646336.00k  1270169.60k  1108241.07k         infk

[Obihoernchen]

Ok here are some results for you

I did all tests with performance governor and cleared caches between each run.
I did three tests in a row and calculated the average.

My setup looks like this:
- Win 10 machine with big Intel Xeon (no bottleneck here )
- Odroid-XU4 with Debian 8, Kernel 3.10 and Samba
- Multiple USB3 HDDs with ext4 connected to the Odroid-XU4
- I used Lan Speed Test v3.5 for my tests because this is way more accurate than Windows Explorer
- This tool writes and reads a 2GB file to my samba share. That's it.

All values are Send | Receive to Odroid-XU4 in MB/s.
Interrupts only

Code: Select all

# default interrupts (all 0)
echo 0-3 > /proc/irq/103/smp_affinity_list
echo 0-3 > /proc/irq/104/smp_affinity_list
echo 0-3 > /proc/irq/105/smp_affinity_list
64 | 62
--> default

# distribute interrupts across A15 cores
echo 6 > /proc/irq/103/smp_affinity_list
echo 5 > /proc/irq/104/smp_affinity_list
echo 4 > /proc/irq/105/smp_affinity_list
54 | 70
--> slower send, faster receive

# pin interrupts to one A15 cores
echo 4 > /proc/irq/103/smp_affinity_list
echo 4 > /proc/irq/104/smp_affinity_list
echo 4 > /proc/irq/105/smp_affinity_list
53 | 68
--> same

# just for comparison 4-7 is actually the same as 4:
53 | 67
--> same

# pin to another A15 core (5) is the same as well:
53 | 67
--> same

Interrupts, RPS and RFS

Code: Select all

# distribute interrupts across first two A15 cores + F0 > rps_cpus (all A15)
68 | 82
--> way faster
# distribute across first two A15 cores + C0 > rps_cpus (last two A15 cores)
71 | 87
--> even faster
# on first A15 core (4-7) + C0 > rps_cpus (last two A15 cores)
72 | 89
--> fastest
# on first A15 core (4-7) + E0 > rps_cpus (last three A15 cores)
72 | 89
--> same
# on first A15 core (4-7) + F0 > rps_cpus (all A15 cores)
72 | 89
--> same

# on first A15 core (4-7) + E0 > rps_cpus (last three cores)
 + echo 32768 > /proc/sys/net/core/rps_sock_flow_entries
 + echo 4096 > /sys/class/net/eth0/queues/rx-0/rps_flow_cnt
70 | 80
--> worse than without rps_flow_cnt. RFS doesn't seem to be useful for us.

# on first A15 core (4-7) + E0 > rps_cpus (last three cores) + E0 > xps_cpus
72 | 88
--> no difference when using xps_cpus

# on first A15 core (4-7) + FF > rps_cpus (all cores)
76 | 76
--> interestingly faster receive, but slower send

# of first A15 core (4-7) + EF > rps_cpus (all expect first A15, the one handling interrupts)
75 | 85
--> quite good as well!

For comparison: A7 interrupts and RPS

Code: Select all

# default (0-3) + E0 > rps_cpus
56 | 74
--> bad

# default (0-3) + 0F > rps_cpus
59 | 75
--> bad

Conclusion
Pin network and usb interrupts to one A15 core, distribute rps_cpus across the other three A15 cores.
Or distribute rps_cpus across all CPUs expect the A15 interrupt core.
RFS and XPS are bad or useless for Odroid-XU4.

Code: Select all

# Move USB and network irqs to first A15 CPU cluster
# usb2
echo 4 > /proc/irq/103/smp_affinity_list
# usb3
echo 4 > /proc/irq/104/smp_affinity_list
# network
echo 4 > /proc/irq/105/smp_affinity_list
# Enable RPS on all A15 cores expect the interrupt core (4)
echo e0 > /sys/class/net/eth0/queues/rx-0/rps_cpus

On top of that I did some samba tests

Code: Select all

default, w sendfile, w min recievefile size: 72 | 89
default, w/o sendifle, w min recievefile size: 72 | 64
default, w sendfile, w AIO w min recievefile size: 73 | 48
default, w sendfile, w/o min receivefile size: 50 | 89

--> Use sendfile, use min recievefile size, don't use AIO!

Code: Select all

use sendfile = yes
min receivefile size = 16384

[tmihai20]

I read most replies in this topic and from about 2 or 3 blogs about oDroid and OMV, I applied some tweaks, then I had to reinstall OMV (micro SD card went to micro SD card heaven) and I did not have the same good copying speed from and to the oDroid as before. I did not know what I did wrong. Today I read the last part from @Obihoernchen's post (just right above this one) and I solved the slow copying speed from my Windows desktop. I am (at least) in the same spot as before. AIO option in smb.conf does not help oDroid's OMV at all. I had in mine and I removed it. I am now using OMV 3.0.68 and OMV official kernel 3.10.82+(for reference).

[defender]

Hi All!

New image (based on Armbian) with mainline kernel is available: OMV_3_0_71_Odroidxu4_4.9.23.7z

Excellent performance is out of box!
But, image does not support eMMC card, only SD at the moment.
And you have to update to latest kernel 4.9.27 for enable FAN control.

[odroid]

It seems to have a wrong device tree file which lowers the maximum CPU clock.

Please help the OMV developers if you want to improve the OMV performance.
Refer this thread.
http://forum.openmediavault.org/index.p ... post143703

[tmihai20]

Will we get the new kernel through an upgrade?

[odroid]

There seems to be a solution about the clock issue in that thread.

[tmihai20]

If there is anybody with a XU4 not using the latest Armbian based OMV, they should really switch now. There is no more need to tweak pretty much anything (except to add CPU temperature logging and to make the graphs larger). I am using it with a 4 GB NTFS USB3 Seagate Hub HDD and my XU4 never had any better performance than it has now. I got an average copying speed of 95 MB/s from my OMV to my Windows 10 PC (for large files, of course) when there is no other traffic being done that seeding torrent. When there is something downloading the speed drops to about 65 MB/s. I used to have 20 to 12 MB/s before (OMV 2.x upgraded to OMV 3.x). I made a backup of important things (like deluge configs and some other stuff) just to be sure. It is worth the hassle. Average CPU temperature is at 56 degrees Celsius now because it is very very hot in Romania right now and I have not yet put my oDroid into the case that allows using the 80 cm fan.

[xenek]

I agree the latest OMV image(OMV_3_0_85_Odroidxu4_4.9.37) is quite snappy and stable.

[tmihai20]

Today I've had the most pleasant surprise in a long time with OMV on my oDroid. There were some updates to the kernel, for the first time I stared using OMV 3. My oDroid is not running 4.9.57 and I do not have to just reinstall OMV just to get a kernel upgrade. Good job OMV.

The following packages will be upgraded:
hostapd linux-dtb-next-odroidxu4 linux-headers-next-odroidxu4 linux-image-next-odroidxu4 linux-u-boot-odroidxu4-next
sunxi-tools

[mad_ady]

... and you got the wpa2 krack patch in hostapd as well

[tmihai20]

That is actually why I checked for updates. My Ubuntu laptop already got the patch.

Sent from my BTV-W09 using Tapatalk

[TOMillr]

Is there a feasible way to run both OMV and a Plex Media Server on the same XU4? I wouldn't really use the two at the same time but would like to option to have both running at the same time so I can get to it whenever I need to.

Does the XU4 handle this? Any recommendations as to which image would give me the best headless performance (DietPi maybe)?

[odroid]

The OMV supports the Plex media server via extra plugin.
https://wiki.odroid.com/odroid-xu4/soft ... dia_server

[tmihai20]

@TOMillr: I am running Plex on my XU4 with OMV. I did not know about that wikipage, but it basically installs it. Always make a backup of its files. I managed to even update it a lot more often but adding the sources from PMS devday. Just run

echo "deb https://dev2day.de/pms/ jessie main" > /etc/apt/sources.list.d/pms-dev2day.list
sudo apt-get update
sudo apt-get upgrade

Be aware that after updating it with apt get it will not start because /etc/default/plexmediaserver gets renamed after installation. I usually stop the service, do the update and then restart the service it systemd does not start it itself. It took me a while to discover why Plex would not start after an upgrade. If you do not update it, you will get an annoying prompt on the main webpage about upgrading. This can be disabled within the settings, but I had no issues updating it like I said above.

[xenek]

Hi, this helped me learn some things a long while ago. So here's a little back.

Using:

Linux odroidxu4 5.4.199-odroidxu4 #22.05.3 SMP PREEMPT Wed Jun 22 07:29:40 UTC 2022 armv7l GNU/Linux

That is:

OMV6 on Armbian 22.05.4 Odroidxu4 'bullseye',
Version 6.0.30-1 (Shaitan)
Processor ARMv7 Processor rev 3 (v7l)
Kernel Linux 5.4.199-odroidxu4

via Gigabit RTL Ethernet (Realtek adaptor) & Gigabit switch,
with Default Windows11 installation to XU4 (no adjustments or tweaks to Windows 11 Installation),

READS & WRITES 59MB/Second to 80MB/Second To BTRFS partition on USB3 to SATA (WD Red+) Quite variable performance, slow.
READS & WRITES 100MB/Second to 112 MB/Second To XFS partition on USB3 to SATA (WD Red+) Quite consistent performance, fast.
READS & WRITES ~60MB/Second TO 70MB/Second To NTFS intermittent & not stable, varied so unusable as often dropped to zero (possibly due to interference with PC or with odroid or network)
READS & WRITES 95MB/Second to 108 MB/Second To EXT4 partition on USB3 to SATA (WD Red+) Quite consistent performance, fast.

Conclusions: Performance is excellent on default install, however use XFS or EXT4 over BTRFS if choosing a disk partition format. Sadly none of those journalling file systems function easily or natively when using USB disks that you might wish to attach to a Windows computer directly, making OMV and Windows not really interoperable. I tried to get FUSE working for ExFAT but that was quite unreliable and the device shutdown numerous times necessitating a reburn of the SD card, it's unknown if the reliability issues were actually due to fuse however, as I did have an intermittently failing external powered USB Hub briefly at the beginning when trying to mount ExFat disks.

Additional: Odroid usb3 inbuilt Eth & RTL Eth benchmark via iperf at ~944 Mbits/sec
Test done: Using iperf -s -N (on odroid) Using iperf -c 192.168.xx.xx -i 2 -r (on Ubuntu console windows via WSL Windows Subsystem for Linux on Same Win11 PC)

Lastly, the instruction process was a bit messy to take OMV to functional state.

-Get working
-Connect the disks
-Format them (if blank)
-Mount them
-Share them (in the 'disks' section'). Making sure that the share path is set to \ (a backslash) and not eg. \ext4 (or \whateveryounamedthedisk) as the default path means you can't see the files on any disks installed, as you by default probably create the proffered share path. You have to clear it and replace it by pressing the \ on the keyboard, as you can't select that option with the mouse alone.
-sharing a disk (second time to share it) on the SMB/CIFS page, on the 'share' button to the right. This is where you change the Samba settings.

This is the process I used above, to get the different read & write speeds.

[mad_ady]

For NTFS try mounting with the big_writes option. It makes a big difference.

[tmihai20]

I said NTFS good bye and have not had a single issue since. All my drives are EXT4 now. It is not worth having them NTFS for the sake of Windows. There are live Linux distros that can help if you have issues.

Edit: there is no amount of settings that can prevent issues. I have used OMV on XU4 for more than 1 year, maybe even 2 years and at some point there were issues (slow copy speed, slow access, overall bad performance). Linux does not do filesystem check as Windows does.