Toshiba nvme : low performance and high i/o wait

Post Reply
User avatar
meepmeep
Posts: 6
Joined: Tue Feb 02, 2021 2:00 am
languages_spoken: english
ODROIDs: H2+
Has thanked: 2 times
Been thanked: 1 time
Contact:

Toshiba nvme : low performance and high i/o wait

Post by meepmeep »

I was thinking of moving some of my services to my odroid H2+.
I installed ubuntu 20.04 LTS, added the drivers for realtek .. So far so good.

I'm using an (old - but not very used) nvme disk : a toshiba nvme ssd from my previous Dell XPS.
I installed some services (plex, etc.) and I noticed that i/o wait are easily high :
- goes easily around 20%/30% when playing plex,
- and around 7%/10% when doing administrative task like a massive apt upgrade

As it's not always link to network activity (plex get its data from a NFS mount). I think there is some misconfiguration on my installation.
I run multiple dd check, and I can't get over 240MB/s on write speed. My other server (core i7 6700 on a shuttle box, with a sata SSD) get around 730mb/s.

I disabled the Clock gating and de-empashing thing from bios. No change.

What could be wrong ? I should try with another nvme disk ? Could someone achieve a better performance (at least over my old sata drive) on ubuntu ? Whith wich drive ?

User avatar
odroid
Site Admin
Posts: 36930
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1595 times
Been thanked: 1070 times
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by odroid »

Try an 'iozone' test to check the performance.
I'm using a Samsung PM981 256GB and it shows near 1GB/s writing speed.

Code: Select all

odroid@H2:~$ sudo fstrim / -v
/: 6.8 GiB (7258808320 bytes) trimmed

odroid@H2:~$ sudo iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
	Run began: Tue Feb  2 12:31:17 2021

	Include fsync in write timing
	O_DIRECT feature enabled
	Auto Mode
	File size set to 102400 kB
	Record Size 4 kB
	Record Size 16384 kB
	Command line used: iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
	Output is in kBytes/sec
	Time Resolution = 0.000001 seconds.
	Processor cache size set to 1024 kBytes.
	Processor cache line size set to 32 bytes.
	File stride size set to 17 * record size.
                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write     
          102400       4   111210   150252   190920   191881    57285   143327                                                                
          102400   16384  1031261  1054627  1515866  1600858  1584476  1004947
It is also worth to check the NVMe flash storage health.
https://www.percona.com/blog/2017/02/09 ... sh-health/
These users thanked the author odroid for the post:
meepmeep (Wed Feb 03, 2021 2:19 am)

fvolk
Posts: 571
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 61 times
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by fvolk »

For raw hardware throughput speed use (for example on a Samsung Evo 970 Plus with a H2):

Code: Select all

$ hdparm --direct -t /dev/nvme0n1
/dev/nvme0n1:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing O_DIRECT disk reads: 4282 MB in  3.00 seconds = 1426.94 MB/sec
Depending on the software, specific read/write I/O patterns and other things happening in the system at the same time you get of course slower performance in practice.

fvolk
Posts: 571
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 61 times
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by fvolk »

meepmeep wrote:
Tue Feb 02, 2021 4:37 am
My other server (core i7 6700 on a shuttle box, with a sata SSD) get around 730mb/s.
The SATA bus is limited in hardware to 6 Gbit/s or 600 MB/s, see https://en.wikipedia.org/wiki/Serial_ATA
Considering also protocol handling overhead etc, this results for SATA in about ~500MB/s max. transfer speed in practice.

If you see 730MB/s over SATA this includes effects to due temporary caching of data in main memory?
These users thanked the author fvolk for the post:
meepmeep (Wed Feb 03, 2021 2:19 am)

User avatar
meepmeep
Posts: 6
Joined: Tue Feb 02, 2021 2:00 am
languages_spoken: english
ODROIDs: H2+
Has thanked: 2 times
Been thanked: 1 time
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by meepmeep »

Thank you all for theses detailed replies !

Indeed, @fvolk I made some mistake on my other server, the oflag=dsync was not set on the dd command :

Old server

Code: Select all

root@Daenerys:~/bin# dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 5.11331 s, 420 MB/s
Which seems more realistic for this configuration.

on odroid :

Code: Select all

root@odroid:/home/meepmeep# dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 7.93932 s, 270 MB/s
About the hdparm command, it's only about reading operation, and in this case, odroid + toshiba nvme seems correct :

odroid

Code: Select all

root@odroid:/home/meepmeep# hdparm --direct -t /dev/nvme0n1p2
/dev/nvme0n1p2:
 HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device
 Timing O_DIRECT disk reads: 2972 MB in  3.00 seconds = 990.26 MB/sec
(run multiples times, results are always around 1000MB/sec)

Old server

Code: Select all

root@Daenerys:~/bin# hdparm --direct -t /dev/sda5
/dev/sda5:
 Timing O_DIRECT disk reads: 1514 MB in  3.00 seconds = 504.15 MB/sec
(run multiples times, results are always around 500MB/sec)

With iozone, I'm not sure how to read the result ..
odroid

Code: Select all

root@odroid:/home/meepmeep# iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
        Iozone: Performance Test of File I/O
                Version $Revision: 3.489 $
                Compiled for 64 bit mode.
                Build: linux-AMD64
[...]
        Run began: Tue Feb  2 11:23:36 2021

        Include fsync in write timing
        O_DIRECT feature enabled
        Auto Mode
        File size set to 102400 kB
        Record Size 4 kB
        Record Size 16384 kB
        Command line used: iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4    90761   123088    65726    65925    43516   119696
          102400   16384   751169   746117  1281419  1371295  1376478   782232

iozone test complete.
I need to read the second write result (751MB/s) ? the random write ?

The health of the nvme seems correct (5% used, no media error)

Code: Select all

root@odroid:/home/meepmeep# nvme smart-log /dev/nvme0
Smart Log for NVME device:nvme0 namespace-id:ffffffff
critical_warning                    : 0
temperature                         : 42 C
available_spare                     : 100%
available_spare_threshold           : 50%
percentage_used                     : 5%
data_units_read                     : 7,974,649
data_units_written                  : 8,076,595
host_read_commands                  : 135,433,671
host_write_commands                 : 117,455,086
controller_busy_time                : 555
power_cycles                        : 189
power_on_hours                      : 768
unsafe_shutdowns                    : 81
media_errors                        : 0
num_err_log_entries                 : 1
Warning Temperature Time            : 0
Critical Composite Temperature Time : 0
Temperature Sensor 1                : 42 C
Thermal Management T1 Trans Count   : 0
Thermal Management T2 Trans Count   : 0
Thermal Management T1 Total Time    : 0
Thermal Management T2 Total Time    : 0

User avatar
odroid
Site Admin
Posts: 36930
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1595 times
Been thanked: 1070 times
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by odroid »

Right. Your random write throughput is around 750MB/s.
Therefore, your storage performance looks fine.

As @fvolk mentioned, I think other background/foreground services could affect the file IO performance.

User avatar
meepmeep
Posts: 6
Joined: Tue Feb 02, 2021 2:00 am
languages_spoken: english
ODROIDs: H2+
Has thanked: 2 times
Been thanked: 1 time
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by meepmeep »

@fvolk and @odroid, could you run my dd command ?

dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output

Because I still don't understand these result (odroid being slower than my sata ssd).

fvolk
Posts: 571
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, C4, H2
Has thanked: 0
Been thanked: 61 times
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by fvolk »

meepmeep wrote:
Wed Feb 03, 2021 7:00 pm
@fvolk and @odroid, could you run my dd command ?
dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync; rm -f /tmp/output
Here in my setup /tmp is on a LUKS AES encrypted partition, so this includes the time for encryption and throughput is limited by the CPU core running at 100% and not by the SSD:
2147479552 bytes (2,1 GB, 2,0 GiB) copied, 11,6904 s, 184 MB/s

If you were to do this on an unencrypted partition you would maybe get another side effect, some SSD are smart enough and notice that all data written is all zeroes, so instead of actually writing all zeroes to flash they are TRIMing the blocks - this then measures TRIM performance and not actual write speed.

User avatar
odroid
Site Admin
Posts: 36930
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1595 times
Been thanked: 1070 times
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by odroid »

This is my result.

Code: Select all

odroid@H2:~$ dd if=/dev/zero of=/tmp/output bs=2G count=1 oflag=dsync
0+1 records in
0+1 records out
2147479552 bytes (2.1 GB, 2.0 GiB) copied, 4.33227 s, 496 MB/s
What is model number of your Toshiba NVMe?

User avatar
meepmeep
Posts: 6
Joined: Tue Feb 02, 2021 2:00 am
languages_spoken: english
ODROIDs: H2+
Has thanked: 2 times
Been thanked: 1 time
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by meepmeep »

THNSN5256GPUK NVMe TOSHIBA 256GB.
Taken from a Dell XPS laptop (from 2016 or 2017 I think)

User avatar
odroid
Site Admin
Posts: 36930
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1595 times
Been thanked: 1070 times
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by odroid »

I couldn't find any strange thing in the specification.

Could you test the same 'dd' and 'iozone' test on the Dell XPS laptop?

User avatar
meepmeep
Posts: 6
Joined: Tue Feb 02, 2021 2:00 am
languages_spoken: english
ODROIDs: H2+
Has thanked: 2 times
Been thanked: 1 time
Contact:

Re: Toshiba nvme : low performance and high i/o wait

Post by meepmeep »

The dell xps is dead :(

The iozone test looks correct with the 750mb/s. If I find another server with a M.2 nvme slot, i'll make a bench.

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 0 guests