Toshiba nvme : low performance and high i/o wait

[meepmeep]

I was thinking of moving some of my services to my odroid H2+.
I installed ubuntu 20.04 LTS, added the drivers for realtek .. So far so good.

I'm using an (old - but not very used) nvme disk : a toshiba nvme ssd from my previous Dell XPS.
I installed some services (plex, etc.) and I noticed that i/o wait are easily high :
- goes easily around 20%/30% when playing plex,
- and around 7%/10% when doing administrative task like a massive apt upgrade

As it's not always link to network activity (plex get its data from a NFS mount). I think there is some misconfiguration on my installation.
I run multiple dd check, and I can't get over 240MB/s on write speed. My other server (core i7 6700 on a shuttle box, with a sata SSD) get around 730mb/s.

I disabled the Clock gating and de-empashing thing from bios. No change.

What could be wrong ? I should try with another nvme disk ? Could someone achieve a better performance (at least over my old sata drive) on ubuntu ? Whith wich drive ?

[meepmeep]

Try an 'iozone' test to check the performance.
I'm using a Samsung PM981 256GB and it shows near 1GB/s writing speed.

Code: Select all

odroid@H2:~$ sudo fstrim / -v
/: 6.8 GiB (7258808320 bytes) trimmed

odroid@H2:~$ sudo iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
	Run began: Tue Feb  2 12:31:17 2021

	Include fsync in write timing
	O_DIRECT feature enabled
	Auto Mode
	File size set to 102400 kB
	Record Size 4 kB
	Record Size 16384 kB
	Command line used: iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write     
          102400       4   111210   150252   190920   191881    57285   143327                                                                
          102400   16384  1031261  1054627  1515866  1600858  1584476  1004947

It is also worth to check the NVMe flash storage health.
https://www.percona.com/blog/2017/02/09 ... sh-health/

[fvolk]

For raw hardware throughput speed use (for example on a Samsung Evo 970 Plus with a H2):

Code: Select all

$ hdparm --direct -t /dev/nvme0n1
/dev/nvme0n1:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing O_DIRECT disk reads: 4282 MB in  3.00 seconds = 1426.94 MB/sec

Depending on the software, specific read/write I/O patterns and other things happening in the system at the same time you get of course slower performance in practice.

[fvolk]

My other server (core i7 6700 on a shuttle box, with a sata SSD) get around 730mb/s.

The SATA bus is limited in hardware to 6 Gbit/s or 600 MB/s, see https://en.wikipedia.org/wiki/Serial_ATA
Considering also protocol handling overhead etc, this results for SATA in about ~500MB/s max. transfer speed in practice.

If you see 730MB/s over SATA this includes effects to due temporary caching of data in main memory?

[meepmeep]

Thank you all for theses detailed replies !

Indeed, @fvolk I made some mistake on my other server, the oflag=dsync was not set on the dd command :

Old server

Code: Select all

root@Daenerys:~/bin# dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 5.11331 s, 420 MB/s

Which seems more realistic for this configuration.

on odroid :

Code: Select all

root@odroid:/home/meepmeep# dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 7.93932 s, 270 MB/s

About the hdparm command, it's only about reading operation, and in this case, odroid + toshiba nvme seems correct :

odroid

Code: Select all

root@odroid:/home/meepmeep# hdparm --direct -t /dev/nvme0n1p2
/dev/nvme0n1p2:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing O_DIRECT disk reads: 2972 MB in  3.00 seconds = 990.26 MB/sec

(run multiples times, results are always around 1000MB/sec)

Old server

Code: Select all

root@Daenerys:~/bin# hdparm --direct -t /dev/sda5
/dev/sda5:
 Timing O_DIRECT disk reads: 1514 MB in  3.00 seconds = 504.15 MB/sec

(run multiples times, results are always around 500MB/sec)

With iozone, I'm not sure how to read the result ..
odroid

Code: Select all

root@odroid:/home/meepmeep# iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.489 $
                Compiled for 64 bit mode.
                Build: linux-AMD64
[...]
        Run began: Tue Feb  2 11:23:36 2021

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 102400 kB
        Record Size 4 kB
        Record Size 16384 kB
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    90761   123088    65726    65925    43516   119696
          102400   16384   751169   746117  1281419  1371295  1376478   782232

iozone test complete.

I need to read the second write result (751MB/s) ? the random write ?

The health of the nvme seems correct (5% used, no media error)

Code: Select all

root@odroid:/home/meepmeep# nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning                    : 0
temperature                         : 42 C
available_spare                     : 100%
available_spare_threshold           : 50%
percentage_used                     : 5%
data_units_read                     : 7,974,649
data_units_written                  : 8,076,595
host_read_commands                  : 135,433,671
host_write_commands                 : 117,455,086
controller_busy_time                : 555
power_cycles                        : 189
power_on_hours                      : 768
unsafe_shutdowns                    : 81
media_errors                        : 0
num_err_log_entries                 : 1
Warning Temperature Time            : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1                : 42 C
Thermal Management T1 Trans Count   : 0
Thermal Management T2 Trans Count   : 0
Thermal Management T1 Total Time    : 0
Thermal Management T2 Total Time    : 0

[odroid]

Right. Your random write throughput is around 750MB/s.
Therefore, your storage performance looks fine.

As @fvolk mentioned, I think other background/foreground services could affect the file IO performance.

[meepmeep]

@fvolk and @odroid, could you run my dd command ?

dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output

Because I still don't understand these result (odroid being slower than my sata ssd).

[fvolk]

@fvolk and @odroid, could you run my dd command ?
dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output

Here in my setup /tmp is on a LUKS AES encrypted partition, so this includes the time for encryption and throughput is limited by the CPU core running at 100% and not by the SSD:

2147479552 bytes (2,1 GB, 2,0 GiB) copied, 11,6904 s, 184 MB/s

If you were to do this on an unencrypted partition you would maybe get another side effect, some SSD are smart enough and notice that all data written is all zeroes, so instead of actually writing all zeroes to flash they are TRIMing the blocks - this then measures TRIM performance and not actual write speed.

[odroid]

This is my result.

Code: Select all

odroid@H2:~$ dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 4.33227 s, 496 MB/s

What is model number of your Toshiba NVMe?

[meepmeep]

THNSN5256GPUK NVMe TOSHIBA 256GB.
Taken from a Dell XPS laptop (from 2016 or 2017 I think)

[odroid]

I couldn't find any strange thing in the specification.

Could you test the same 'dd' and 'iozone' test on the Dell XPS laptop?

[meepmeep]

The dell xps is dead

The iozone test looks correct with the 750mb/s. If I find another server with a M.2 nvme slot, i'll make a bench.