SSD/HDD Benchmarking Reminder

Post Reply
crashoverride
Posts: 5705
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 563 times
Contact:

SSD/HDD Benchmarking Reminder

Post by crashoverride »

As a reminder to those testing SSD/HDD performance. By default, a new EXT4 filesystem is "lazy" initialized. This means that the format command completes before the drive is actually done formatting. Formatting then continues in a background task (even across reboots) and can affect performance measurements.
https://www.thomas-krenn.com/en/wiki/Ex ... ialization

tkaiser
Posts: 773
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 2 times
Been thanked: 25 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by tkaiser »

crashoverride wrote:Formatting then continues in a background task (even across reboots) and can affect performance measurements.
Absolutely true but in the context of 'benchmarking' it should be obvious that if a device where such behaviour occurs is used for 'benchmarking' then it's the wrong device anyway since severely bottlenecking any 'performance' measurements later. If creation of an ext4 filesystem takes longer than a minute then please stop the whole 'benchmarking' approach.

120GB EVO on USB3 with an UAS capable disk enclosure:

Code: Select all

root@odroid:/home/odroid# time mkfs.ext4 -E lazy_itable_init=0,lazy_journal_init=0 /dev/sda1
mke2fs 1.43.4 (31-Jan-2017)
/dev/sda1 contains a ext4 file system
	created on Sat Feb 17 12:25:08 2018
Proceed anyway? (y,N) y
Creating filesystem with 29304177 4k blocks and 7331840 inodes
Filesystem UUID: 7b7315a4-8fe2-4fa8-9b1a-887563242152
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (131072 blocks): done
Writing superblocks and filesystem accounting information: done   


real	0m14.066s
user	0m0.024s
sys	0m6.255s
It should also be noted that testing with HDDs and slow SSDs is questionable since what will be tested then if the disk is the bottleneck? And if a SSD is used the limitations the device has should also already be known (e.g. the vast majority of consumer SSDs has a sequential write performance weakness after a certain amount of data has been written and then write performance drops down from maximum to horribly low values. For example out of the 4 SSDs I can currently test with N1 the 'best' one only drops down to 180MB/s, the worst one even arrives at just 60MB/s)

User avatar
rooted
Posts: 9634
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 761 times
Been thanked: 505 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by rooted »

Not sure if it makes a difference since I don't currently have a functional 2.5" but I was digging through the kernel and ran across this which is set to powersave by default:

Code: Select all

echo performance > /sys/module/pcie_aspm/parameters/policy

tkaiser
Posts: 773
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 2 times
Been thanked: 25 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by tkaiser »

rooted wrote:Not sure if it makes a difference since I don't currently have a functional 2.5" but I was digging through the kernel and ran across this which is set to powersave by default:

Code: Select all

echo performance > /sys/module/pcie_aspm/parameters/policy
Idle consumption increases by ~250mW but especially accesses with small blocksizes benefit greatly:

Code: Select all

ODROID N1 PCIe/ASM1061 powersave                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       1     7585     8562     9322     9331     5907     8505
          102400       4    26400    31745    34586    34798    24039    31595
          102400      16    87201    99311   105977   106152    79099    99618
          102400     512   313662   316992   308216   310013   301521   308300
          102400    1024   327748   324230   319738   322929   317812   325224
          102400   16384   368813   369384   385862   390732   390612   379333

ODROID N1 PCIe/ASM1061 performance                            random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       1    15218    19331    23617    23661    10690    18965
          102400       4    49071    65403    79028    79247    39287    64922
          102400      16   137845   168899   185766   186482   116789   166413
          102400     512   326117   332789   324468   326999   317332   328611
          102400    1024   330827   331303   326731   329246   325201   333325
          102400   16384   378331   368429   385870   392127   391348   371753
So HDD users better stay with powersave while SSD owners keen on high random IO performance and with smaller block sizes in general will better choose performance.

tkaiser
Posts: 773
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 2 times
Been thanked: 25 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by tkaiser »

BTW: That's one of the many reasons really everone should stop using inappropriate tools like dd or hdparm to 'benchmark' storage. It's not possible with those tools to determine 'storage performance' since they both test for sequential workloads only and access blocksize matters.

With both dd and hdparm the numbers would've looked exactly identical with ASPM (Active State Power Management) set to either powersave or performance: showing just the same maximum sequential throughput number that is close to irrelevant anyway. It needs real storage benchmark tools like for example fio or iozone to check for storage performance. And no, not bonnie/bonnie++ since this is just another joke: http://www.brendangregg.com/ActiveBench ... nie++.html

crashoverride
Posts: 5705
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 563 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by crashoverride »

I got a 2.5 SATA WD Black HDD and 2.5 Crucial MX500 SSD for testing. While there was nothing remarkable about the HDD performance, I was completely blown away by the SSD performance. All the benchmark numbers posted do not do it justice. Its one of those things you just have to see for yourself.

My "benchmark" unit of measure was "debootstrap". This is heavily read/write intensive and indicative of "real world" performance. The SSD is just insanely fast at it. Running an OS off the SSD was also equally impressive.

TL;DR = Numbers are nice, but actually using it is another thing! ZOMFG!11!1!!!!11! ITS CRAZY FAST! :lol:

crashoverride
Posts: 5705
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 563 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by crashoverride »

rooted wrote:Not sure if it makes a difference
From my preliminary (un-reproduced) tests, it make a significant difference with the WD Black drive. Have not tested SDD.

Code: Select all

echo performance | sudo tee /sys/module/pcie_aspm/parameters/policy

User avatar
rooted
Posts: 9634
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 761 times
Been thanked: 505 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by rooted »

Good to hear it makes a difference to the positive, except power consumption.

tkaiser
Posts: 773
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 2 times
Been thanked: 25 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by tkaiser »

crashoverride wrote:All the benchmark numbers posted do not do it justice.
Well, if it's the usual 'BS benchmarking' attempt with dd or hdparm then of course the numbers don't show reality (since sequential maximum performance is just... irrelevant for most use cases). If you instead look at small block sizes and there especially random IO and espcially writes the numbers already tell directly why HDDs suck and why good SSDs are that superiour:

Code: Select all

iozone -e -I -a -s 100M -r 1k -r 2k -r 4k -r 16k -i 0 -i 1 -i 2
I think comparing HDDs with SSDs is pointless anyway since this is all well known. More interesting would be a comparison SSD vs eMMC since the MMC controller on RK3399 is amazingly fast but eMMC is way more energy efficient compared to any SSD storage solution even if users buy expensive SSDs that show an acceptable idle and active consumption.

tkaiser
Posts: 773
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 2 times
Been thanked: 25 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by tkaiser »

rooted wrote:Good to hear it makes a difference to the positive, except power consumption.
The good news: the ~250mW more only affect idle consumption. When there's happening something IO intensive the numbers do not differ. So it seems possible to use an ugly hack in cpufreq scaling code to switch /sys/module/pcie_aspm/parameters/policy between powersave and performance depending on cpufreq to get the desired behaviour: low idle consumption and top performance when needed. It should depend on io_is_busy though.

User avatar
rooted
Posts: 9634
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 761 times
Been thanked: 505 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by rooted »

tkaiser wrote:
rooted wrote:Good to hear it makes a difference to the positive, except power consumption.
The good news: the ~250mW more only affect idle consumption. When there's happening something IO intensive the numbers do not differ. So it seems possible to use an ugly hack in cpufreq scaling code to switch /sys/module/pcie_aspm/parameters/policy between powersave and performance depending on cpufreq to get the desired behaviour: low idle consumption and top performance when needed. It should depend on io_is_busy though.
Not a bad idea.

tkaiser
Posts: 773
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 2 times
Been thanked: 25 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by tkaiser »

crashoverride wrote:All the benchmark numbers posted do not do it justice.
Let's see. I took a 7.2k HGST 2.5" HDD and connected it to one of the SATA ports. Benchmark used is just

Code: Select all

iozone -e -I -a -s 100M -r 1k -i 0 -i 1 -i 2
I wanted to test with 2K and 4K blocksizes too but since random IO with HDDs is such a sh*t show I had to drop the idea since such a benchmark would take hours with HDDs. I also tested again the orange 16GB eMMC and my EVO840 connected to the other SATA port. For HDDs and SSDs I let the test run with /sys/module/pcie_aspm/parameters/policy set to performance and powersave and collected both numbers each. And I also tested with a cheap 32GB SanDisk Ultra A1 SD card I bought for $15 3 days ago:

Code: Select all

                                                     random    random
                   write  rewrite    read    reread    read     write
  SSD powersave     2085     2003     8678     8743     7339     2431
SSD performance     2061     1994     8710     8606     7908     2523
      16GB eMMC     2051     1965     8682     8694     8491     2507 
     A1 SD card      714      718     3436     3444     2928      440
  HDD powersave      231     2624     5648     5667      168       70
HDD performance      227     3137     8800     8853      170       70 
Since the SSD is bottlenecked by ASM1061 or RK3399 we see identical performance compared to Hardkernel's super fast eMMC. As a reference same SSD tested yesterday connected to a native SATA port of a Clearfog Pro (using a 'NAS SoC': Marvell Armada 385):

Code: Select all

                                                     random    random
                   write  rewrite    read    reread    read     write
EVO840 Clearfog    21853    37308    39815    39753    12597    35440
What do the above numbers tell? Not that much except clearly showing why HDDs should not be used for anything else than cold storage these days: Since their random IO performance horribly sucks. The SSD even if severly bottlenecked is at least 45 times faster when it's about random reads and 35 times faster with random writes (and the eMMC even outperforms the SSD here!). That's what makes the difference with real-world workloads like debootstrapping (or even just a boring 'apt upgrade' with hundred packages to be upgraded).

Even my SD card is 17 to 6 times faster than this HDD (random reads and random writes). But this is due to using a new SD card compliant with A1 performance class: https://forum.armbian.com/topic/954-sd- ... ment-25272

It's 2018, we can buy these things now. Don't fool yourself and buy crappy nonames, PNY, Kingston, Verbatim and this stuff. Check for the A1 logo and check capacity and speed directly after purchase and enjoy way better performance than a HDD even with SD cards! :D

tkaiser
Posts: 773
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 2 times
Been thanked: 25 times
Contact:

Re: SSD/HDD Benchmarking Reminder

Post by tkaiser »

Please forget about the SSD numbers above. Since they looked similar to the eMMC numbers I did a quick check with 'iostat -xNz 1' in another shell just to realize that the SSD was not mounted at all -- silly me :oops:

Correct numbers now also showing clearly the difference between powersave and performance:

Code: Select all

                                                     random    random
                   write  rewrite    read    reread    read     write
SSD performance    14078    17634    20790    20797     7347    17294
  SSD powersave     7206     8205     8132     8187     5600     8154
      16GB eMMC     2051     1965     8682     8694     8491     2507 
     A1 SD card      714      718     3436     3444     2928      440
HDD performance      227     3137     8800     8853      170       70 
  HDD powersave      231     2624     5648     5667      168       70
With performance settings the SSD is now almost 250 times faster than the HDD wrt random writes with 1KB chunks.

Post Reply

Return to “General Chat”

Who is online

Users browsing this forum: No registered users and 2 guests