XU4 eMMC vs SDcard

[tve]

I recently got a XU4 with a fast sdcard and wondered whether an eMMC would really have been faster. Fortunately it turned out that a friend has an XU4 with an eMMC so we ran some comparison tests. This is not super scientific but more than I could find...

My XU4 has a Samsung Pro 16GB Class 10 UHS-3 Micro SDHC Card https://www.amazon.com/gp/product/B014W1ZL3S which cost me $14. It's one of the fastest sdcards I could find, looking mostly at random write speeds.
My friend has a 64GB eMMC sold by hardkernel. If I had bought a 16GB eMMC it would have cost me $34.

First I tried to compare with the `dd` performance listed on hardkernel's page http://www.hardkernel.com/main/products ... 5628174287

sdcard:

Code: Select all

> sudo dd if=/dev/zero of=/tmp/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 13.8632 s, 38.7 MB/s
Here's the read test:
> sudo dd if=/tmp/test of=/dev/null iflag=direct bs=8M
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 9.42691 s, 57.0 MB/s

That's a lot faster than the sdcard used by hardkernel...

eMMC:

Code: Select all

$ sudo dd if=/dev/zero of=/tmp/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 13.9753 s, 38.4 MB/s
$ sudo dd if=/tmp/test of=/dev/null iflag=direct bs=8M
64+0 records in
64+0 records out
536870912 bytes (537 MB) copied, 4.45892 s, 120 MB/s

I then wanted to compare random small I/Os, which are a lot more relevant for many compile-type tasks. For this I resorted to `fio`, which is a very sophisticated disk benchmarking tool with a gazillion options. Here it is doing random 4KB reads and writes with a 75%/25% read/write mix:

sdcard:

Code: Select all

> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=64
fio-2.2.10
Starting 1 process
test: Laying out IO file(s) (1 file(s) / 1024MB)
Jobs: 1 (f=1): [m(1)] [100.0% done] [263KB/95KB/0KB /s] [65/23/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=7102: Fri Sep  2 22:22:36 2016
  read : io=784996KB, bw=1399.9KB/s, iops=349, runt=560775msec
  write: io=263580KB, bw=481308B/s, iops=117, runt=560775msec
  cpu          : usr=1.27%, sys=8.25%, ctx=247792, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=196249/w=65895/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Disk stats (read/write):
  mmcblk0: ios=194508/66175, merge=1797/254, ticks=26266510/9470115, in_queue=35735885, util=100.00%

eMMC:

Code: Select all

Jobs: 1 (f=1): [m(1)] [100.0% done] [7561KB/2495KB/0KB /s] [1890/623/0 iops] [eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=22137: Sat Sep  3 00:16:51 2016
  read : io=784996KB, bw=8311.6KB/s, iops=2077, runt= 94447msec
  write: io=263580KB, bw=2790.8KB/s, iops=697, runt= 94447msec
  cpu          : usr=5.02%, sys=46.55%, ctx=225997, majf=0, minf=21
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=196249/w=65895/d=0, short=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Disk stats (read/write):
  mmcblk0: ios=194962/65703, merge=2272/192, ticks=4469440/1564275, in_queue=6040960, util=100.00%

Quite a speed difference! The numbers I look at most are
read bw=1399.9KB/s and write bw=481308B/s
vs.
read bw=8311.6KB/s and write bw=2790.8KB/s

We then tried sequential read/write bandwidth using fio to see how that compares with the dd test:

sdcard:

Code: Select all

> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=1G --readwrite=rw --rwmixread=75
  read : io=784996KB, bw=16697KB/s, iops=4174, runt= 47013msec
  write: io=263580KB, bw=5606.6KB/s, iops=1401, runt= 47013msec

eMMC:

Code: Select all

  read : io=784996KB, bw=14497KB/s, iops=3624, runt= 54149msec
  write: io=263580KB, bw=4867.7KB/s, iops=1216, runt= 54149msec

And the same test with --bs=16kb:
sdcard:

Code: Select all

  read : io=785776KB, bw=32415KB/s, iops=2025, runt= 24241msec
  write: io=262800KB, bw=10841KB/s, iops=677, runt= 24241msec

eMMC:

Code: Select all

  read : io=785776KB, bw=33857KB/s, iops=2116, runt= 23209msec
  write: io=262800KB, bw=11323KB/s, iops=707, runt= 23209msec

The system CPUs were ~50% busy for these sequential tests.

Sooooo. I'm not what all to make out of this, but so far I observe:

• harkernel uses a poor sdcard for the comparison
• sequential performance is not that far off between the two, dd shows 2x read performance but fio doesn't, I suspect the 2x isn't there in real life, but it's hard to tell
• the eMMC has a clear performance advantage when it comes to random small I/O, I wonder how much of that has to do with flash&controller vs. bus overhead.

I'm sure others will have a different set of observations

[DarkBahamut]

The write speeds are pretty low, that eMMC probably needs fstrim running on it. The read speeds are okay, but could be better with some tweaking.

These are my results with a 32GB eMMC, as you can see they are quite a bit higher than yours using dd.

Code: Select all

darkbahamut@odroid:~$ dd if=/dev/zero of=test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 7.77563 s, 69.0 MB/s
darkbahamut@odroid:~$ dd if=test of=/dev/null iflag=direct bs=8M
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 2.92475 s, 184 MB/s

Edit: I added fio as well just to test. Never used it before so just copied your command. About 10% faster at 4K read/writes than the MMC you test, and miles ahead of the SD card. It should be noted that small block speed is one of the most important areas of performance, which is exactly why SSD's in computers make such a big bigger. It's not because of the sequential performance, it's because the small block random transfers are so fast (about 100 times faster than HDDs reading, even more when writing).

Code: Select all

test: (groupid=0, jobs=1): err= 0: pid=2531: Sat Sep  3 01:02:26 2016
  read : io=784996KB, bw=9134.6KB/s, iops=2283, runt= 85937msec
  write: io=263580KB, bw=3067.2KB/s, iops=766, runt= 85937msec
  cpu          : usr=4.22%, sys=30.57%, ctx=254688, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued    : total=r=196249/w=65895/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: io=784996KB, aggrb=9134KB/s, minb=9134KB/s, maxb=9134KB/s, mint=85937msec, maxt=85937msec
  WRITE: io=263580KB, aggrb=3067KB/s, minb=3067KB/s, maxb=3067KB/s, mint=85937msec, maxt=85937msec

Disk stats (read/write):
  mmcblk0: ios=194116/65596, merge=1732/175, ticks=4078955/1546270, in_queue=5700510, util=100.00%

The take away is versus the SD card on test is that the SD card is at best half the speed, but in most tests it's far slower than half the speed. At the most critical times when IO is at it's slowest the SD card is 6 times slower. Having more modern storage like UFS would be very nice (Maybe in the XU5, eh? ) but I think the eMMC puts up a good showing against the SD card. It's twice the price but more than twice the performance.

EDIT 2: Add the fio small block sequential too. Again much faster than the MMC you tested, and on average 2x-3x the speed of your SD card.

--bs=4k

Code: Select all

 read : io=784996KB, bw=45687KB/s, iops=11421, runt= 17182msec
  write: io=263580KB, bw=15340KB/s, iops=3835, runt= 17182msec

--bs=16k

Code: Select all

read : io=785776KB, bw=89394KB/s, iops=5587, runt=  8790msec
  write: io=262800KB, bw=29898KB/s, iops=1868, runt=  8790msec

[tve]

Thanks for testing! What model eMMC do you have, does the kernel print out some flash model number at boot?

[DarkBahamut]

Mines the 32GB. The kernel doesn't really print anything in terms of a model, though hard kernels product pages says the 32GB model is a Sandisk iNAND extreme chip, but it doesn't say what model exactly. They might be able to clarify.

[tve]

Is there a trick to running fstrim?

Code: Select all

> sudo fstrim /
fstrim: /: the discard operation is not supported

[DarkBahamut]

You need to update your kernel, support for fstrim was only added into quite recently. http://forum.odroid.com/viewtopic.php?f=95&t=23252

It should be in your software updater, or apt-get update then apt-get dist-upgrade should work (I think, been a while since I ran a stock kernel!).

[ouafnico]

Hey guys,

your subject is quite interesting.
What about the lifetime of sdcard or eMMC?
For a linux OS, writing all the time on logs, what could be the most safer to use between both?

[elatllat]

...What about the lifetime of sdcard or eMMC?...

That's going to be hard to test; mainly because sdcards have a wide range of quality, and systems have a wide range of IO.
I have been using the same sdcard in an rpi since 2012 (~4 years) but I know others can die in under a year.

[ouafnico]

...What about the lifetime of sdcard or eMMC?...

That's going to be hard to test; mainly because sdcards have a wide range of quality, and systems have a wide range of IO.
I have been using the same sdcard in an rpi since 2012 (~4 years) but I know others can die in under a year.

That's right

[rooted]

...What about the lifetime of sdcard or eMMC?...

That's going to be hard to test; mainly because sdcards have a wide range of quality, and systems have a wide range of IO.
I have been using the same sdcard in an rpi since 2012 (~4 years) but I know others can die in under a year.

Are you running OE or LE? Since they are RO memory based distributions.

Because I have killed many cards using Debian (XBian) on various Pi's.

[DarkBahamut]

Flash in SD cards tends to be lower quality/endurance, hence the low prices SD cards sell for. NAND die quality generally increases as you move up to eMMC, then up to full SSDs, which again is largely reflected in the price of the storage.

The issue you see with people having SD cards fail very quickly in SBC applications are probably people using cheap TLC based SD cards as OS drives. These SD cards are probably rated for 200-300 P/E cycles maximum. So if you only have a 16GB card with ~200 P/E cycles it won't take long for that to be completely worn out as an OS drive because there are quite significant background writes happening all the time even on a completely idle system (I often see 0.8GB+ per day on an XU4 that's running nothing at all). Now if you have an MLC based SD card then you could expect maybe 10 times the rated endurance, 2000 - 3000 P/E cycles is not uncommon at all. Now suddenly that SD card that might have been worn out in a year as an OS drive might now last 10 years. Given that MLC SD cards are often about double the price of TLC ones, you can quickly see how in these write demanding situations that TLC SD cards can be a complete false economy. Yes they look super cheap and great value, but the endurance is just not there for this application. Use them for mass storage? Perfect. OS drive? Not recommended at all.

This is even before you discuss the performance implications also, as TLC NAND has much lower write performance than MLC NAND, so your system is also slower using such an SD card which isn't good when using one as an OS/application drive.

The final point is trim support. Not all SBC's will do trim on SD cards (nor will all SD cards accept a trim command) which further impacts the life of the cards as it hampers wear leveling and causes the controller to move around data that has been deleted. Generally trim is widely supported on eMMC these days so that additional measure that helps to keep the eMMC healthy for longer.

[ouafnico]

Thanks for your post.
So you do not recommand using sdcard for OS.
I hope emmc sold by hardkernel are good ships to use it with OS for some time

[DarkBahamut]

My view is that the XU4 is a high performance SBC so to me it makes sense to also pair it with a high performance storage options, which in this case the eMMC offers the best performance. That is generalised though so if you only need an XU4 for the CPU grunt only then using eMMC is less important, but if you want it as an all round device then eMMC is the way to go IMO. The eMMC hard kernel use has high quality MLC NAND so should have a very long life expectancy.

That said, there's nothing wrong with using a high quality SD card if you accept the lower performance they comes with. I recently picked up a Samsung Pro 64GB SD card which uses high quality MLC flash and benchmarks at ~92MB/s read and 83MB/s write and it only cost £23 (less than our 32GB eMMC module cost!). I didn't get it for an XU4, but SD cards are compelling given the low price so I can see why people like to use them. Still, I'd always recommend eMMC as the primary OS and application storage for speed, then a big SD card for secondary and overflow storage.

[ouafnico]

Thanks. For my part, i need something durable so I think the emmc will be better.

[atomicODROID]

Awesome testing, thanks for sharing

I know I should be running an eMMC but I stumbled on to some great deals (64GB SanDisk Extreme, tested at 22 write and 93 read for $20). I also found an old bestbuy gift card and decided to try out those crazy Lexar cards which did not disappoint. Read speed is more imporant when running an OS anyway so I'm happy with getting around, sometimes over 100MB/s but I have ordered a few 32GB emmc modules. Main reason I've stayed away from them is swapping SD cards is just so much easier

[ronaldlees]

I recently got a XU4 with a fast sdcard and wondered whether an eMMC would really have been faster. Fortunately it turned out that a friend has an XU4 with an eMMC so we ran some comparison tests. This is not super scientific but ...

This is a little off the topic of your thread title, but if you're going to spend $34, you might want to add $5 to that and use an SSD. I've switched to an inexpensive SSD (~$40 on Amazon for a Kingston SUV400S37 128GB) - and I'm not looking back. Speeds are marvelous, and it truly makes the XU4 seem like my AMD desktop. My speeds, using your "dd benchmark" :

read : 147 MB/s
write: 135 MB/s

IMO - it's no comparison. I don't know how it'll hold up long term, since only have had it a few days. Great so far tho ...

[tve]

> if you're going to spend $34, you might want to add $5 to that and use an SSD. I've switched to an inexpensive SSD (~$40 on Amazon for a Kingston SUV400S37 128GB) - and I'm not looking back.

Thanks for the suggestion! I spent $14, not $34 for the sdcard... But, I'm puzzled, the XU4 doesn't have a sata port, so how do you connect your SSD to it? A USB-SATA adapter? (That's not free either.)

[ronaldlees]

@tve:

You're right, a USB3 to SATA3 connector is needed if you don't already have one. I think they're about 10 bux. The cable is connected to one of the USB3 ports on the XU4. Booting is still done with a cheap uSD, which then mounts rootfs on the SSD. So, booting time isn't changed as much. Once fully booted though - it's off to the races.