XU4 eMMC poor write performance

Post Reply
tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

XU4 eMMC poor write performance

Unread post by tve » Thu Mar 28, 2019 9:32 am

I "upgraded" an XU4 from an sdcard to a 32GB eMMC 5.0 module in July 2017 and I'm having performance issues with it now. It is formatted using ext4, 87% full, and I noticed today that it only did 3-5MB/s writes. After some head-scratching I ran fstrim and that brought it up some. Here are two typical runs:

Code: Select all

> sudo dd if=/dev/zero of=/tmp/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 38.4227 s, 14.0 MB/s
> sudo dd if=/dev/zero of=/tmp/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 59.2406 s, 9.1 MB/s
I also happen to have a 128GB SD Card installed, here's the same on that SD card:

Code: Select all

> sudo dd if=/dev/zero of=/big/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 15.3488 s, 35.0 MB/s
I installed the MMC utils to get some info and it doesn't look like the eMMC is "worn out":

Code: Select all

eMMC Firmware Version: CS1.30
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x01
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x01
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01
(I believe EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x01 means that 0%-10% of reserved blocks are used.)

I'm running Linux version 4.14.35-128 (root@1604_builder_armhf) (gcc version 5.4.1 20160904 (Ubuntu/Linaro 5.4.1-2ubuntu1~16.04)) #1 SMP PREEMPT Fri Apr 20 13:49:49 UTC 2018

Any thoughts on what to try or is this typical performance after a year and a half?

User avatar
odroid
Site Admin
Posts: 32146
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 119 times
Been thanked: 297 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by odroid » Thu Mar 28, 2019 9:41 am

You should have near 100MB/s reading speed.

Update the kernel first. Our Ubuntu 18.04 kernel package should have 4.14.107 at this moment.

Try sudo fstrim / -v and do a performance test again.

BTW, which eMMC module do you have?
https://wiki.odroid.com/accessory/emmc/reference_chart

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Thu Mar 28, 2019 12:59 pm

Thanks for the response! Read speed is great (150MB/s), it's the write speed that sucks. It's a V5.0, i.e. red module.

User avatar
odroid
Site Admin
Posts: 32146
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 119 times
Been thanked: 297 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by odroid » Thu Mar 28, 2019 1:49 pm

I think the write speed should be 40~50MB/s at least on the red eMMC (Sandisk 32GB chipset probably).
Test again after trimming.

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Thu Mar 28, 2019 3:02 pm

fstrim is run weekly. I ran it before doing my tests. Here it is again:

Code: Select all

> sudo fstrim -v /
/: 141 MiB (147861504 bytes) trimmed
> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 95.4729 s, 5.6 MB/s
> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 108.557 s, 4.9 MB/s
> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 85.748 s, 6.3 MB/s
I was running iostat -kx 2 in another shell session. The top 2-second interval I captured (9MB/s):

Code: Select all

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
mmcblk0           0.00     0.00    0.00   14.00     0.00  9982.00  1426.00     6.29  414.14    0.00  414.14  71.14  99.60
All the intervals have >99% util.

I then ran the same tests on the 128MB sdcard (not, I'm not mixing-up sdcard vs eMMC :-):

Code: Select all

> sudo dd if=/dev/zero of=/big/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 14.1006 s, 38.1 MB/s
> sudo dd if=/dev/zero of=/big/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 10.678 s, 50.3 MB/s
> sudo dd if=/dev/zero of=/big/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 12.9243 s, 41.5 MB/s
I guess my eMMC is toast :-(.
Last edited by tve on Fri Mar 29, 2019 2:10 am, edited 1 time in total.

User avatar
mad_ady
Posts: 6552
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by mad_ady » Thu Mar 28, 2019 3:03 pm

Just in case - make sure /tmp is not mounted somewhere where you don't expect it. Can you also show us blkid and mount?

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Thu Mar 28, 2019 3:12 pm

If you look at my dd commandline you'll see I use /test, not /tmp/test.

Code: Select all

> df /test
Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/mmcblk0p2  29925948 26116880   3473548  89% /

> sudo blkid
/dev/mmcblk0p1: SEC_TYPE="msdos" LABEL="boot" UUID="52AA-6867" TYPE="vfat" PARTUUID="3cedfd53-01"
/dev/mmcblk0p2: LABEL="rootfs" UUID="e139ce78-9841-40fe-8823-96a304a09859" TYPE="ext4" PARTUUID="3cedfd53-02"
/dev/mmcblk1p1: UUID="83f32816-49cf-443b-92e5-56f81be4a4f6" TYPE="ext4" PARTUUID="43132247-01"
/dev/mmcblk0: PTUUID="3cedfd53" PTTYPE="dos"
/dev/mmcblk1: PTUUID="43132247" PTTYPE="dos"

> mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,nosuid,relatime,size=952000k,nr_inodes=186720,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=204244k,mode=755)
/dev/mmcblk0p2 on / type ext4 (rw,noatime,block_validity,delalloc,barrier,user_xattr,acl,errors=remount-ro,stripe=32753)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgrou
ps-agent,name=systemd)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/rdma type cgroup (rw,nosuid,nodev,noexec,relatime,rdma)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
tracefs on /sys/kernel/debug/tracing type tracefs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mmcblk0p1 on /media/boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,utf8,errors=remount-ro)
/dev/mmcblk1p1 on /big type ext4 (rw,noatime,data=ordered)
/dev/mmcblk0p2 on /var/lib/docker/overlay type ext4 (rw,noatime,block_validity,delalloc,barrier,user_xattr,acl,errors=remount-ro,stripe=32753)
nsfs on /run/docker/netns/default type nsfs (rw)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=204244k,mode=700,uid=1000,gid=1000)
binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,relatime)
overlay on /var/lib/docker/overlay/02fcdea55f5f3204d76291e3ae7fa4ee848988753f0d5b46aaef40e697f2998e/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/5310b2644f6cbae03faa169bd4765c1b3945161a4d5a444a685a0862895a6e5d/root,upperdir=/var/lib/docker/overlay/02fcdea55f5f3204d76291e3ae7fa4ee848988753f0d5b46aaef40e697f2998e/upper,workdir=/var/lib/docker/overlay/02fcdea55f5f3204d76291e3ae7fa4ee848988753f0d5b46aaef40e697f2998e/work)
shm on /var/lib/docker/containers/7b848f76d242ca75ecd3e4cc7dfba54ce6d98ef38895680838c843ff15cc7468/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /run/docker/netns/7157e22704e5 type nsfs (rw)
overlay on /var/lib/docker/overlay/bad62f0f0373311b234b89d220e31ac3af33bf2f3433bc5f14890dc6d58e764c/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/5310b2644f6cbae03faa169bd4765c1b3945161a4d5a444a685a0862895a6e5d/root,upperdir=/var/lib/docker/overlay/bad62f0f0373311b234b89d220e31ac3af33bf2f3433bc5f14890dc6d58e764c/upper,workdir=/var/lib/docker/overlay/bad62f0f0373311b234b89d220e31ac3af33bf2f3433bc5f14890dc6d58e764c/work)
shm on /var/lib/docker/containers/0975468268f31061778f5cfff0254c46cfaa9f1daea8016876ba9f1e69368d46/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /run/docker/netns/39678e6fac42 type nsfs (rw)
overlay on /var/lib/docker/overlay/06e914429b6bfeff6cdb82d24cb8f0cb3c8341f492cfdd449080e15893ef61b9/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/18dbb0ae9b857695864f1519fd82db164bdeb355b75e7196a9c37673d846d9e6/root,upperdir=/var/lib/docker/overlay/06e914429b6bfeff6cdb82d24cb8f0cb3c8341f492cfdd449080e15893ef61b9/upper,workdir=/var/lib/docker/overlay/06e914429b6bfeff6cdb82d24cb8f0cb3c8341f492cfdd449080e15893ef61b9/work)
shm on /var/lib/docker/containers/35a8c213c245593c3a34322593676b34ac7d86fbb6dd4b02a1a0bba0e4a12330/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /run/docker/netns/dd4d9447edbe type nsfs (rw)
overlay on /var/lib/docker/overlay/e75476602240f20bbfd662da6f095da5c6798a134144c325c86e1173641020aa/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/638ed331272d9b242ec515195aba605dce428e7affec53c860f4018a78458000/root,upperdir=/var/lib/docker/overlay/e75476602240f20bbfd662da6f095da5c6798a134144c325c86e1173641020aa/upper,workdir=/var/lib/docker/overlay/e75476602240f20bbfd662da6f095da5c6798a134144c325c86e1173641020aa/work)
shm on /var/lib/docker/containers/e22ee1d5368f1d1041631168c403857e4259146e80236d1c83d212fd82479439/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /run/docker/netns/156dfb11f34d type nsfs (rw)
overlay on /var/lib/docker/overlay/49e3a4d34dc05b5b2af8e942ee9c4144bcdbeb0186c192036c6bae4615623f36/merged type overlay (rw,relatime,lowerdir=/var/lib/docker/overlay/3cdb6d574dfaf77066b0e23d522a40688903f8e8ed1f953daf5efafd3faa8af0/root,upperdir=/var/lib/docker/overlay/49e3a4d34dc05b5b2af8e942ee9c4144bcdbeb0186c192036c6bae4615623f36/upper,workdir=/var/lib/docker/overlay/49e3a4d34dc05b5b2af8e942ee9c4144bcdbeb0186c192036c6bae4615623f36/work)
shm on /var/lib/docker/containers/8f4187488db1643e203f431b1cb8764f21cbe4f24ad152644721f9c0bc8a8820/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k)
nsfs on /run/docker/netns/775572525e25 type nsfs (rw)

User avatar
odroid
Site Admin
Posts: 32146
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 119 times
Been thanked: 297 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by odroid » Thu Mar 28, 2019 3:54 pm

Your eMMC memory chip seems to be worn out too much or some internal NAND memory cells could be damaged.

Format your eMMC and test again if you don't mind.

User avatar
mad_ady
Posts: 6552
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by mad_ady » Thu Mar 28, 2019 3:58 pm

It may be that it's badly fragmented (since it's 90% full). What color is the emmc? Does it feel very hot?

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Fri Mar 29, 2019 12:59 am

Your eMMC memory chip seems to be worn out too much or some internal NAND memory cells could be damaged.
That's what I was wondering. I ran mmc-util and the flash chip says that fewer than 10% of reserved blocks have been used and that it's far from EOL:

Code: Select all

eMMC Firmware Version: CS1.30
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x01
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x01
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01
Is there any other way to query wear?
Format your eMMC and test again if you don't mind.
That's quite some work given that it's the root drive on a production system... How do I format the eMMC under Linux? If I just do a mkfs.ext4 it doesn't do a low-level format of any sort... Would that even do a "trim" of the entire eMMC?
It may be that it's badly fragmented (since it's 90% full). What color is the emmc? Does it feel very hot?
I's a red eMMC. I can't touch it due to the mounting. I ran e4defrag:

Code: Select all

 Fragmentation score                            27
 [0-30 no problem: 31-55 a little bit fragmented: 56- needs defrag]
 This device (/dev/mmcblk0p2) does not need defragmentation.
I will now defrag anyway and power cycle the system...

User avatar
mad_ady
Posts: 6552
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by mad_ady » Fri Mar 29, 2019 1:05 am

Doesn't seem fragmented. Maybe a fsck could be useful as well, though I doubt it will make much difference.

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Fri Mar 29, 2019 2:09 am

Well, power cycle and defrag made no difference.

But then... success!?

Code: Select all

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 8.10761 s, 66.2 MB/s
> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 11.7774 s, 45.6 MB/s
> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 16.7392 s, 32.1 MB/s
> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 21.4736 s, 25.0 MB/s
After the initial 66MB/s the speed is stabilizing around 30-35MB/s. The trick? mmc sanitize...

One question that remains in my mind: is there a tool to perform a full eMMC erase? Either using a USB adapter or by booting from sdcard?

User avatar
mad_ady
Posts: 6552
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by mad_ady » Fri Mar 29, 2019 2:14 am

What is mmc sanitize? Do you mean fstrim?
For wiping I'd recommend formatting as ext4 and running fstrim on it. Writing zeros on it is not the same as trim.

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Fri Mar 29, 2019 2:29 am

What is mmc sanitize? Do you mean fstrim?
8-) 8-) Install mmc-utils from source (there's an ubuntu package but no binaries for ARM) and run the mmc command. Description for sanitize:
Send Sanitize command to the <device>.
This will delete the unmapped memory region of the device.
I don't have the Jedec eMMC standard, so can't read the exact definition, but I took this to mean that the flash chip performs an erase cycle on all flash blocks that are unused (e.g. have been trimmed).

Given the performance degradation after the sanitization that I have observed I would guess that the flash controller maintains a certain number of erased blocks and that immediately after the sanitize I could run the dd once without hitting that limit, but that subsequent runs caused the controller to erase some blocks and hence the write speed got lowered.
One question that remains in my mind: is there a tool to perform a full eMMC erase? Either using a USB adapter or by booting from sdcard?
I can now also answer my question: mkfs.ext4 has a -E discard option: "Attempt to discard blocks at mkfs time (discarding blocks initially is useful on solid state devices and sparse / thin-provisioned storage). This is set as default." In other words, just performing a mkfs.ext4 does mark the entire eMMC as unused. An mmc sanitize would then in addition perform a full device erase, if my understanding is correct.

Phew... :D :D

User avatar
mad_ady
Posts: 6552
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 169 times
Been thanked: 128 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by mad_ady » Fri Mar 29, 2019 4:16 am

but I took this to mean that the flash chip performs an erase cycle on all flash blocks that are unused (e.g. have been trimmed).
I always thought trim implied the controller would erase those blocks. I wasn't sure when the erase took place - I thought at fstrim time...

I'll see when I get some time I'd like to test my emmc fleet as well...

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Fri Mar 29, 2019 5:28 am

I always thought trim implied the controller would erase those blocks.
I don't think so. I believe it marks the blocks as unused. When to actually erase them is up to the controller. The "sanitize" command forces the controller to erase everything unused, e.g. to ensure that deleted data cannot be recovered. The sanitize command takes minutes (and regular disk access hangs), trim is instantaneous.

indium
Posts: 93
Joined: Thu May 28, 2015 2:27 pm
languages_spoken: english, ukrainian
Location: Ukraine
Has thanked: 0
Been thanked: 1 time
Contact:

Re: XU4 eMMC poor write performance

Unread post by indium » Fri Mar 29, 2019 7:14 am

flash storages get slower, when get almost full. it's not wearing out, it's no room for parallelizing and write speed falls down. If you compared with another module with the same % and absolute amount of left space and found out that it is still significantly faster, then it would be another thing. But I guess, your eMMC is healthy, it's just full. My tablet's eMMC shows the same.

User avatar
odroid
Site Admin
Posts: 32146
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 119 times
Been thanked: 297 times
Contact:

Re: XU4 eMMC poor write performance

Unread post by odroid » Fri Mar 29, 2019 8:50 am

I am learning a great deal of knowledge every day on this forum. :!:

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Fri Mar 29, 2019 2:11 pm

flash storages get slower, when get almost full.
Yup, I knew that. I didn't quite realize that 85% full means that the performance goes to the pits. I expected the threshold to be >90%. This morning I got ~30MB/s, now it's down to 7MB/s. Here's a "how to 10x your eMMC performance":

Code: Select all

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 69.3182 s, 7.7 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 73.223 s, 7.3 MB/s

> sudo fstrim -v /
/: 173.6 MiB (182030336 bytes) trimmed

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 30.1103 s, 17.8 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 40.614 s, 13.2 MB/s

> sudo mmc sanitize /dev/mmcblk0
ioctl: Connection timed out
Could not write 0x01 to EXT_CSD[165] in /dev/mmcblk0

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 10.2051 s, 52.6 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 18.6524 s, 28.8 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 19.6857 s, 27.3 MB/s
Pretty shocking... (I really ran these commands back-to-back!) I wonder whether there's some flag in the eMMC controller one can set to make it perform proactive erases more aggressively when the drive is idle (which does happen quite a bit).
I think the next thing I'll do is move some stuff to the SDcard to free up space and then see how it does.

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Fri Mar 29, 2019 2:38 pm

One more post... I moved a pile of stuff, now the eMMC is 75% full. I again ran the write test, followed by fstrim, more write tests, followed by sanitize, and more write tests...

Code: Select all

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 61.93 s, 8.7 MB/s

> sudo fstrim -v /
/: 1.6 GiB (1685479424 bytes) trimmed

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 13.4277 s, 40.0 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 9.50655 s, 56.5 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 21.5863 s, 24.9 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 17.5671 s, 30.6 MB/s

> sudo mmc sanitize /dev/mmcblk0
ioctl: Connection timed out
Could not write 0x01 to EXT_CSD[165] in /dev/mmcblk0

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 9.93602 s, 54.0 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 8.33066 s, 64.4 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 9.07247 s, 59.2 MB/s

> sudo dd if=/dev/zero of=/test oflag=direct bs=8M count=64
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 9.9469 s, 54.0 MB/s
So it looks like one has to keep 25% free in order to continue enjoying high write speeds?
I wish I had some SSD laying around I could try a similar experiment with.

shebabrained
Posts: 1
Joined: Sat Mar 16, 2019 4:25 pm
languages_spoken: english
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by shebabrained » Mon Apr 15, 2019 10:46 pm

Your eMMC memory chip seems to be worn out too much or some internal NAND memory cells could be damaged.
Update madalin stunt cars 2 new version

tve
Posts: 47
Joined: Sun Jul 13, 2014 4:02 pm
languages_spoken: english
ODROIDs: odroid-u3, odroid-c1, odroid-c1+
Has thanked: 0
Been thanked: 0
Contact:

Re: XU4 eMMC poor write performance

Unread post by tve » Tue Apr 16, 2019 12:58 am

shebabrained wrote:
Mon Apr 15, 2019 10:46 pm
Your eMMC memory chip seems to be worn out too much or some internal NAND memory cells could be damaged.
What makes you say that? Did you see the diagnostic output it produces that says otherwise?

Post Reply

Return to “General Topics”

Who is online

Users browsing this forum: No registered users and 1 guest