Samba server performance

Moderators: mdrjr, odroid

Samba server performance

Unread postby midtskogen » Sun May 13, 2018 1:12 am

I'm using an XU4 running Ubuntu as a samba server for several IP cameras. The cameras will write video to the XU4 continuously at roughly 300 KB/s (combined), or 2.4 mbps. So I need a stable network connection. Experience with wifi causes the cameras to drop frames, so a good link is important. I therefore use 100 mbps ethernet, which should be plenty. Still the cameras drop frames, but not always:

When the exported file system is on the emmc card, everything is rock solid.
When the exported file system is on an extern USB disk, there are occational frame drops.
When the exported file system is on an SD card, there are frequent frame drops.

But the SD card is good for write speeds up to 10 MB/s, and if I log into the camera and write to the mounted disk, I achieve write speeds of a few MB/s, much higher than the write speed of the video. So for some reason, the camera is very picky when it writes video. What I can't understand is why the camera should care, or even know, what the physical medium is. Even if latency is an issue when writing to an SD card, I think smbfs or at least the Linux kernel, should hide this through caching. I've tried to increase the write caching both in smbd and in the Linux kernel, but it makes no impact. I've also tried increasing the priority of smbd through renice and chrt, but that doesn't help either.

I could perhaps export the emmc card and then let XU4 copy to the SD card as new files appear, but that is cumbersome (and if the problem is that writing to SD locks up the XU4 sufficiently to make the cameras drop frames, it wont help).

Any suggestions?
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4

Re: Samba server performance

Unread postby moon.linux » Sun May 13, 2018 5:12 am

I would suggest use SSD for better read/write performance on samba share.
moon.linux
 
Posts: 1159
Joined: Thu Oct 02, 2014 11:42 pm
languages_spoken: english

Re: Samba server performance

Unread postby midtskogen » Sun May 13, 2018 6:00 am

SSD probably helps, but I'm trying using a 256 GB SD card now to see if it's possible to move to that, since it will be such a physically compact setup. So far, it's not a 100% success, but I don't really understand why. 300 KB/s should not be much, not even for SD. So I wonder if it rather has something to do with latency (in particular because I can run dd commands on the camera writing to the samba mount at great speed), but if so, that again should be amendable through caching. It's odd that the samba client should notice what the physical medium is, when the physical medium should support at least 10x the transfer speed. Another explanation could be that writing to SD locks up the entire XU4 long enough for the samba client to notice. So I tried to have smbd run with realtime priority with chrt, but that didn't help.

I've tried to ping the cameras from the XU4. The response time is always well below 1 ms, so the network seems fine.
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4

Re: Samba server performance

Unread postby rooted » Sun May 13, 2018 7:42 am

Try setting the governor to performance.
User avatar
rooted
 
Posts: 5583
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: Samba server performance

Unread postby midtskogen » Sun May 13, 2018 3:31 pm

I already had the governor set to performance.

I tried the following:

1. exported a directory on the emmc card (I guess it also could be a ramdisk to save emmc wear), so the cameras will write here through smbd.
2. ran: inotifywait --format "%w%f" -rqme close_write,moved_to /exported/emmc/dir | (while read line; do mkdir -p /sdcard/$(dirname $line); mv $line /sdcard/$line; done)

I.e. the exported emmc directory is monitored for new files, and once one is written, they are moved to the sd card. This actually works. I no longer see frame drops from the camera, or at least they are greatly reduced. I could also try to export a ramdisk, but caching either by Linux or smbd really should have made that unneccessary.

While this "works", it's kludgy and not probably entirely safe. But it should prove that there is no hardware limitation.

This hack could be a case for unionfs, where an upper layer acts as a faster cache for the slower lower layer, but I don't think this is supported?
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4

Re: Samba server performance

Unread postby midtskogen » Sun May 13, 2018 9:38 pm

I tried numerous samba options, I tried to tune kernel parameters, I tried to make a file system wrapper looking at unionfs, overlayfs and aufs, but all in vain. But finally, I think I've managed to solve my problem in a clean way.

My SD card was ext4 formatted. When I tried to reformat using f2fs, the troubles seem to have gone away. To use f2fs on an SD card, for which f2fs was designed, makes a huge difference.

The root filesystem on the emmc card has ext4. If f2fs can make such a difference on an SD card, I wonder whether it has been considered to use f2fs on the emmc card?
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4

Re: Samba server performance

Unread postby rooted » Sun May 13, 2018 10:36 pm

You also tried running your tasks and samba on the big cores as well?
User avatar
rooted
 
Posts: 5583
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: Samba server performance

Unread postby midtskogen » Sun May 13, 2018 10:40 pm

rooted wrote:You also tried running your tasks and samba on the big cores as well?

No, I didn't try to lock any processes to certain cores. Samba forks a lot, so I don't think it would make much sense either.

Anyway, f2fs seems right for an SD card anyway.
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4

Re: Samba server performance

Unread postby rooted » Sun May 13, 2018 10:50 pm

midtskogen wrote:
rooted wrote:You also tried running your tasks and samba on the big cores as well?

No, I didn't try to lock any processes to certain cores. Samba forks a lot, so I don't think it would make much sense either.

Anyway, f2fs seems right for an SD card anyway.
Actually it does, @mad_ady did an article in odroid magazine describing the process.

Read this for some ideas:

https://magazine.odroid.com/article/set ... rpose-nas/
User avatar
rooted
 
Posts: 5583
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: Samba server performance

Unread postby DarkBahamut » Mon May 14, 2018 1:03 am

I've been using F2FS on my SD card for about 18 months and I regularly stream video off my XU4 (via Samba) while writing to the SD card at the same time and I don't think it's ever dropped frames on me, so hopefully you should get good performance with it. I don't use the stock version of Samba available though as I always build my own from source.

I'd also recommend mounting the SD card with the discard option. Will make the initial file deletion take longer but will give you more stable performance during the file writes.
DarkBahamut
 
Posts: 312
Joined: Tue Jan 19, 2016 10:19 am
languages_spoken: english
ODROIDs: XU4, N1

Re: Samba server performance

Unread postby midtskogen » Mon May 14, 2018 2:51 am

Thanks, "discard" added. In my setup stable write is the most important, whereas read and file delete have lower priority.
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4

Re: Samba server performance

Unread postby midtskogen » Mon May 14, 2018 6:38 am

I've now seen the cameras write up to 6 MB/s to the SD card over samba, and it works perfectly. That's 20x the rate when ext4 caused problems, and much more what I could do on an external hard disk over USB.
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4

Re: Samba server performance

Unread postby rooted » Mon May 14, 2018 8:01 am

midtskogen wrote:I've now seen the cameras write up to 6 MB/s to the SD card over samba, and it works perfectly. That's 20x the rate when ext4 caused problems, and much more what I could do on an external hard disk over USB.
By adding discard?
User avatar
rooted
 
Posts: 5583
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: Samba server performance

Unread postby midtskogen » Tue May 15, 2018 2:05 am

rooted wrote:By adding discard?


No, probably just from going from ext4 to f2fs. I think discard is enabled by default.
midtskogen
 
Posts: 92
Joined: Mon Jul 27, 2015 3:57 am
languages_spoken: english norwegian
ODROIDs: U3 XU3-Lite XU4


Return to General Chat

Who is online

Users browsing this forum: No registered users and 5 guests