[SOLVED] HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed

Post Reply
mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

[SOLVED] HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed

Post by mcmanuf »

Hi.
Originally I had OMV4 running for a couple of years, with the OMV4-SBC image from sourceforge. This image is no longer available, but I am 100% sure it was running armbian. Could it be same image as this one hosted here? -> https://dn.odroid.com/temp/OMV_4_Odroid ... HC2.img.xz , I am not sure.

It was working perfectly, but recently my 2.5 HDD died connected to the HC1-SATA suddenly dies.

I thought it was good time to upgrade SW and FW, so I installed Samsung 870 QVO SSD.
Upgraded JMS578 UAS FW to latest (173.*001 -> 173.*002), and installed latest Armbian:
Armbian 21.08 Bullseye : Kernel 5.4.y, Size: 327Mb, Updated: Nov 22, 2021
Then OMV6 script ontop of it.

Everything was working fine, until I started using SAMBA. One of my CPU is maxed, and I only get 40-42MB/s read/write on this new setup.
I compared smb.conf from old and new OMV installation and they are very similar.
Remember old installation I got around 100MB/s read/write on mechanical disk.

Some important notes.
-UAS is enabled according to lsusb.
-iperf3 shows me consistent ~950Mbit/s. So network is good.
-Disk read/write tests with "dd" command shows me >350MB/s read/write to SSD.
-Selected HC1 as optimized board configuration on armbian-config

I am extremely extremely frustrated. I have been troubleshooting for many hours without any results, and for me unexplainable behaviour.
Last edited by mcmanuf on Fri Feb 18, 2022 7:36 am, edited 4 times in total.

User avatar
odroid
Site Admin
Posts: 39106
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 2498 times
Been thanked: 1380 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 50MB/s, when upgrading system/kernel

Post by odroid »

That image in the link is a mirror of the old OMV4 image for XU4/HC1/HC2 devices.
Try that image first since newer OMV5/OMV6 still has some glitches here and there.
https://wiki.odroid.com/odroid-xu4/soft ... stallation

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 50MB/s, when upgrading system/kernel

Post by mcmanuf »

odroid wrote:
Thu Feb 17, 2022 12:56 pm
That image in the link is a mirror of the old OMV4 image for XU4/HC1/HC2 devices.
Try that image first since newer OMV5/OMV6 still has some glitches here and there.
https://wiki.odroid.com/odroid-xu4/soft ... stallation
I have very difficult understanding how OMV5 still has glitches. It is now very mature. Especially related to Samba implementation?
Both OMV4 and kernel 4.14 is end of life or very soon deprecated. So according to you I am stuck on this?

What about installing OMV5 on this image below instead of the Armbian one:
https://odroid.in/ubuntu_20.04lts/XU3_X ... 812.img.xz

Maybe something wrong with armbian image. Is this image maybe "better" than the Armbian one, and has better/alternative optimizations for HC1, or is this image just ubuntu without odroid SBC optimizations?

During research yesterday, I was reading also threads related to IRQ handling, moving IRQ to other cores etc.
Could this also be the culprit, or this was just in the old days of XU4/HC1 days?

And can it really be, that no one in these forums, are running anything newer than OMV4 with fast samba connection?

User avatar
rooted
Posts: 9436
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 758 times
Been thanked: 479 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 50MB/s, when upgrading system/kernel

Post by rooted »

If OMV4 works better why not use it? I don't understand the need to keep a home server on latest software versus well tested stable.

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 50MB/s, when upgrading system/kernel

Post by mcmanuf »

rooted wrote:
Thu Feb 17, 2022 6:16 pm
If OMV4 works better why not use it? I don't understand the need to keep a home server on latest software versus well tested stable.
Security updates is first and main important point. This is also why I change phones, and update my pc operating system. Why should it be different here. My setup is exposed to the internet, So want to make sure I get latest security updates always.

And as I mentioned, both OMV4 and Kernel 4.9/4.14 is nearing EOL. So now was a good time to update, after HDD crash. I just kept postponing it, exactly due to the fears I will waste hours and hours.

Also, are you still using Ubuntu 10 because it just works? You never upgrade your main OS or your phone?

I am not a first mover by any means, and as mentioned OMV5 and kernel 5.4 seems to be bit more "futureproof" regarding updates/security, than the current OMV4 and kernel 4.14.

User avatar
rooted
Posts: 9436
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 758 times
Been thanked: 479 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 50MB/s, when upgrading system/kernel

Post by rooted »

If your server is internet facing I understand wanting to run the latest kernel available, I'm all for security which is why the only thing I have facing internet is wireguard. I don't need to worry about keeping my NAS updated as wireguard is on a different device.

I don't run Ubuntu on my main laptop, I run Manjaro and I run Google Pixel phones for exactly the reason you mention which is security. Ones main OS or phone isn't a NAS however.

@mad_ady usually has some good ideas to speed up network transfers, maybe he will join in.
These users thanked the author rooted for the post:
mcmanuf (Thu Feb 17, 2022 9:15 pm)

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 50MB/s, when upgrading system/kernel

Post by mcmanuf »

I am now extremely confused.

I just reloaded my old perfectly working SD card installation (OMV4/Kernel 4.14) and I get 100% cpu load by smbd process, and only 40MB/s write/read from my new SSD, over gigabit network.

So now it seems the problem is not even the OMV6/Kernel 5.4 which I thought.

So only change to my system:
-Went from 5TB Seagate Mechanical HDD to 8TB Samsung SSD 870 QVO.
-Upgraded FW of JMS578 controller from 178*.001 -> 178*.002

These changes decreased SAMBA speed over LAN from 100MB/s to 40MB/s.

Does anyone have ANY explanation which would make sense, why I am seeing what I am seeing?

I am going crazy.

Dmesq related to disk /dev/sdb (new SSD)
Also is this sentence of concern?
[ 16.725496] sd 1:0:0:0: [sdb] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)

[ 16.144977] usb 4-1: New USB device found, idVendor=152d, idProduct=0578
[ 16.145016] usb 4-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 16.145046] usb 4-1: Product: USB to ATA/ATAPI Bridge
[ 16.145074] usb 4-1: Manufacturer: JMicron
[ 16.145104] usb 4-1: SerialNumber: 0123456789ABCDEF
[ 16.159991] usb 6-1: New USB device found, idVendor=0bda, idProduct=8153
[ 16.160032] usb 6-1: New USB device strings: Mfr=1, Product=2, SerialNumber=6
[ 16.160062] usb 6-1: Product: USB 10/100/1000 LAN
[ 16.160121] usb 6-1: Manufacturer: Realtek
[ 16.160151] usb 6-1: SerialNumber: 000001000000
[ 16.177766] scsi host1: uas
[ 16.178722] scsi 1:0:0:0: Direct-Access JMicron Generic 3102 PQ: 0 ANSI: 6
[ 16.180065] sd 1:0:0:0: Attached scsi generic sg1 type 0
[ 16.327840] usb 6-1: reset SuperSpeed USB device number 2 using xhci-hcd
[ 16.401220] r8152 6-1:1.0 eth0: v1.09.9
[ 16.722953] sd 1:0:0:0: [sdb] 15628053168 512-byte logical blocks: (8.00 TB/7.28 TiB)
[ 16.722998] sd 1:0:0:0: [sdb] 4096-byte physical blocks
[ 16.723534] sd 1:0:0:0: [sdb] Write Protect is off
[ 16.723578] sd 1:0:0:0: [sdb] Mode Sense: 53 00 00 08
[ 16.724484] sd 1:0:0:0: [sdb] Disabling FUA
[ 16.724530] sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[ 16.725496] sd 1:0:0:0: [sdb] Optimal transfer size 33553920 bytes not a multiple of physical block size (4096 bytes)
[ 16.738256] sdb: sdb1
[ 16.741576] sd 1:0:0:0: [sdb] Attached SCSI disk

alignment.PNG
alignment.PNG (16.06 KiB) Viewed 675 times

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s?

Post by mcmanuf »

Ok.
Now I reinserted a 5TB mech HDD instead of the new SSD, and I STILL only get around 40MB/s.

This can only mean, that at some point in time, something has been done on the NAS SW, that decreases the performance in SAMBA from 100MB/s to around 40MB/s.

This could be something related to Odroid FW or kernel, or related to OMv4 updates during the years.

Anyone have any idea where I should start looking?

This is my global settings for Samba /etc/smb.conf:

#======================= Global Settings =======================
[global]
workgroup = WORKGROUP
server string = %h server
dns proxy = no
log level = 0
log file = /var/log/samba/log.%m
max log size = 1000
logging = syslog
panic action = /usr/share/samba/panic-action %d
encrypt passwords = true
passdb backend = tdbsam
obey pam restrictions = no
unix password sync = no
passwd program = /usr/bin/passwd %u
passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
pam password change = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY
guest account = nobody
load printers = no
disable spoolss = yes
printing = bsd
printcap name = /dev/null
unix extensions = yes
wide links = no
create mask = 0777
directory mask = 0777
use sendfile = yes
aio read size = 16384
aio write size = 16384
local master = yes
time server = no
wins support = no
min receivefile size = 16384
write cache size = 524288
getwd cache = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY

User avatar
rooted
Posts: 9436
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 758 times
Been thanked: 479 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s, when upgrading from mech. HDD -> SSD ?

Post by rooted »

Flash the old firmware would be my next step.

User avatar
mad_ady
Posts: 10585
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4 (HC1, HC2), C1+, C2, C4 (HC4), N1, N2, H2, Go, Go Advance, M1
Location: Bucharest, Romania
Has thanked: 643 times
Been thanked: 901 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s, when upgrading from mech. HDD -> SSD ?

Post by mad_ady »

What filesystem are you using on the disk? Are you using ntfs? For ntfs you still need to mount with big_writes option. Is the low performance for read or write? Did you run the dd tests directly on the disk (/dev/sda), or on a mountpoint?
Are you able to test performance with ftp or nfs on the same disk?
These users thanked the author mad_ady for the post:
rooted (Fri Feb 18, 2022 3:12 am)

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s, when upgrading from mech. HDD -> SSD ?

Post by mcmanuf »

I am absolutely losing my mind here.
Now I followed the reference guide to get the 100MB/s for the NAS server, to start completely from scratch.
The image file in the guide, on a new SD card.
https://wiki.odroid.com/odroid-xu4/soft ... ning#samba

This is OMV4, with Kernel 4.9. The same image that odroid used to get ~100MB/s read/write speed via SAMBA.

STILL I ONLY GET 42MB/s speed via SAMBA.
WHAT THE HELL IS GOING ON.

My laptop Ethernet is Intel gigabit I219-LM
I tried external Linksys USB3GIG another chipset, same results.

I downgraded newest JMS578 firmware to older version. Same results.

I am now completely out of ideas.

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s, when upgrading from mech. HDD -> SSD ?

Post by mcmanuf »

mad_ady wrote:
Fri Feb 18, 2022 1:48 am
What filesystem are you using on the disk? Are you using ntfs? For ntfs you still need to mount with big_writes option. Is the low performance for read or write? Did you run the dd tests directly on the disk (/dev/sda), or on a mountpoint?
Are you able to test performance with ftp or nfs on the same disk?
No, using ext4.

FTP gives me decent speeds >95MB/s.

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed

Post by mcmanuf »

I also just followed this example.
https://forum.openmediavault.org/index. ... post260232

I also have a RPI4, and according to this guide, I should be able to max out the 1gbit connection on SAMBA, with the included options.
But again, when testing laptop<->RPI4, I can maximal get ~55MB/s or approx. 485mbit/s. Can there be a limitation on the laptop/windows, on SAMBA speed, being half-duplex or something? Remember I get a full 950mbit on iperf3 testing. So my network is good.

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed - SOLVED

Post by mcmanuf »

After wasting days of troubleshooting doing everything I could think of, I tried one last effort solution.

I wanted to see if there was a limitation in my windows 10 installation on my company PC, which I also use privately.

I booted a Clean Win10-2-GO on same laptop, and I could finally get >100Mb/s read/write over SAMBA. However after approx. 5GB transfer, the speed drops to approx 40-50MB/s before starting to climb again. Could maybe have something to do with SSD DRAM? Anyone knows, would like to know?

This solution proposal will serve as a viable solution in the future for other users. In addition to the common ext4, smb parameters, gigabit switch etc. etc.

Windows definitely has the ability to limit SAMBA transfer. I imagine this is to prevent company network overload.

FINALLY SOLVED.

But please do answer if you a solution to the dropping speed after a certain amounts of gigabytes.

User avatar
odroid
Site Admin
Posts: 39106
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 2498 times
Been thanked: 1380 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed - SOLVED

Post by odroid »

Still very weird.
Do you have the same speed drop issue when you copy several GB files from the HC1 to your laptop?
I want to know which storage device is causing the problem. Laptop storage? or HC1 storage? or both?

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed - SOLVED

Post by mcmanuf »

odroid wrote:
Fri Feb 18, 2022 9:39 am
Still very weird.
Do you have the same speed drop issue when you copy several GB files from the HC1 to your laptop?
I want to know which storage device is causing the problem. Laptop storage? or HC1 storage? or both?
I thought more about it, and have concluded it must be laptop storage, as I was running windows from USB stick. So the USB stick must have some limitation. I was copying big file to and from the USB stick, not the native hdd which has bitlocker.

User avatar
rooted
Posts: 9436
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 758 times
Been thanked: 479 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed - SOLVED

Post by rooted »

mcmanuf wrote:
odroid wrote:
Fri Feb 18, 2022 9:39 am
Still very weird.
Do you have the same speed drop issue when you copy several GB files from the HC1 to your laptop?
I want to know which storage device is causing the problem. Laptop storage? or HC1 storage? or both?
I thought more about it, and have concluded it must be laptop storage, as I was running windows from USB stick. So the USB stick must have some limitation. I was copying big file to and from the USB stick, not the native hdd which has bitlocker.
Not to taint your thread but you were booting and running Windows from USB flash?

mcmanuf
Posts: 18
Joined: Sat Apr 28, 2018 5:03 pm
languages_spoken: english
ODROIDs: HC1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed - SOLVED

Post by mcmanuf »

Yes for the last test, I did that. Helped me find the solution

User avatar
rooted
Posts: 9436
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 758 times
Been thanked: 479 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed - SOLVED

Post by rooted »

mcmanuf wrote:Yes for the last test, I did that. Helped me find the solution
I didn't know this was possible is all, I've been doing it for many years with Linux and somehow missed the memo one could run Windows from USB.

User avatar
mad_ady
Posts: 10585
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4 (HC1, HC2), C1+, C2, C4 (HC4), N1, N2, H2, Go, Go Advance, M1
Location: Bucharest, Romania
Has thanked: 643 times
Been thanked: 901 times
Contact:

Re: HC1- Samba has slowed down from 100MB/s to 40MB/s. Now I cannot get it restored to high speed - SOLVED

Post by mad_ady »

I have an old Windows 10 Gandalf PE on a USB stick for troubleshooting, though I haven't used it in ages. What would be a more modern approach?

Regarding your bottleneck - can you cgeck on windows if you get a core in 100%, or pehaps 100% iops? Maybe you have a DLP (Data leak prevention) app that parses all data transferred to look for stuff to censor.

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 1 guest