SATA - The only numbers that matter

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 4:39 pm

The following attempts to answer very simply the question: How fast is the N1 SATA?

The test numbers are obtained using "hdparm -tT".

PC

Code: Select all

00:17.0 SATA controller: Intel Corporation Sunrise Point-H SATA controller [AHCI mode] (rev 31)

                    |             PC (Skylake Core i5)            |
                    | Cached Reads (MB/s) | Buffered Reads (MB/s) |
WD Black 1TB        |            16249.61 |                137.46 |
Crucial MX500 500GB |            17370.07 |                499.81 |
N1

Code: Select all

01:00.0 IDE interface: ASMedia Technology Inc. ASM1061 SATA IDE Controller (rev 02)

                    |                  ODROID-N1                  |
                    | Cached Reads (MB/s) | Buffered Reads (MB/s) |
WD Black 1TB        |             1145.88 |                137.38 |
Crucial MX500 500GB |             1081.09 |                324.42 |
Dual WD Black 1TB   |                  NA |       135.70 + 125.12 |
1) SATA performance for a HDD is identical on a PC and N1.
2) SATA performance for a SSD is bottlenecked on N1 (likely by ASM1061 controller).
3) Dual SATA on N1 is 1.95 times faster than single SATA.
4) PC's cache (DDR4 memory) performance is 15 times faster than N1.

[edit]
RAID1 (mirror) - Dual WD Black 1 TB (hdparm -tT /dev/md0)

Code: Select all

                    |                  ODROID-N1                  |
                    | Cached Reads (MB/s) | Buffered Reads (MB/s) |
Dual WD Black 1TB   |             1089.32 |                136.73 |

Code: Select all

$ sudo dd if=/dev/md0 of=/dev/null bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 30.6673 s, 140 MB/s

$ sudo dd if=/dev/zero of=/dev/md0 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 37.0118 s, 116 MB/s
[edit2]
RAID0 (stripe) - Dual WD Black 1 TB (hdparm -tT /dev/md0)

Code: Select all

                    |                  ODROID-N1                  |
                    | Cached Reads (MB/s) | Buffered Reads (MB/s) |
Dual WD Black 1TB   |             1177.37 |                272.80 |

Code: Select all

$ sudo dd if=/dev/md0 of=/dev/null bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.1008 s, 267 MB/s

$ sudo dd if=/dev/zero of=/dev/md0 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 18.764 s, 229 MB/s
Last edited by crashoverride on Wed Feb 28, 2018 11:19 pm, edited 4 times in total.

User avatar
odroid
Site Admin
Posts: 32540
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 186 times
Been thanked: 350 times
Contact:

Re: SATA - The only numbers that matter

Unread post by odroid » Wed Feb 28, 2018 5:03 pm

Really appreciate the test result.

How did you test the "Dual WD Black 1TB"?
Did you run hdparm for each HDD in parallel?
Or a soft RAID?

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 5:11 pm

The dual test was "sudo hdparm -t /dev/sda & sudo hdparm -t /dev/sdb"

Of interest is that the interface is reported as "IDE interface" on N1. The driver may not be putting the ASM1061 in AHCI mode. There is no BIOS on N1 to configure this.

User avatar
odroid
Site Admin
Posts: 32540
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 186 times
Been thanked: 350 times
Contact:

Re: SATA - The only numbers that matter

Unread post by odroid » Wed Feb 28, 2018 5:18 pm

I see. You ran it in parallel.

I have no idea how the AHCI mode could be activated.
Or, it could be a just cosmetic issue from the hdparm report.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 5:28 pm

I have a PC using the ASM1062, and its reported as:

Code: Select all

01:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
So its worth investigating if someone has a PC with an ASM1061 based controller what it reports in "lspci". If not, I will try to order a PCIe card with the chipset.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Wed Feb 28, 2018 5:59 pm

crashoverride wrote:So its worth investigating if someone has a PC with an ASM1061 based controller what it reports in "lspci". If not, I will try to order a PCIe card with the chipset.
You can save both money and time if you learn to use tools appropriately (that would include stopping to use 'benchmark' tools like hdparm of course too :lol: )

Code: Select all

lspci -v | grep -i 'driver'
So zero new insights, a couple of wrong conclusions and everyone happy? Great!

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Wed Feb 28, 2018 6:04 pm

crashoverride wrote:4) PC's cache (DDR4 memory) performance is 15 times faster than N1.
Great conclusion! Based on a tool developed in last century by Linux IDE developers that dealt with PCs that were way slower than any recent SBC and hacked together a q&d benchmark mode for their own development needs to deal with insanely slow spinning rust and internal busses.

Code: Select all

root@odroid:~# hdparm -T /dev/sda

/dev/sda:
 Timing cached reads:   2802 MB in  2.00 seconds = 1400.88 MB/sec
How's that possible?

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 6:10 pm

FYI, I am looking for something similar to this but for Asmedia instead of Intel:
http://dreamlayers.blogspot.com/2016/11 ... ch7-m.html

I can't find any ASM1061 register documentation to tell me if there is PCI register to flip on it or not.

User avatar
odroid
Site Admin
Posts: 32540
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 186 times
Been thanked: 350 times
Contact:

Re: SATA - The only numbers that matter

Unread post by odroid » Wed Feb 28, 2018 6:13 pm

@tkaiser,
Don't need to do like that.
Frankly speaking, you are too much offensive and other community members can feel something uncomfortable with you to enjoy this debug party.
I think everybody has to move to your Armbian forum. :(

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Wed Feb 28, 2018 6:27 pm

odroid wrote:Frankly speaking, you are too much offensive and other community members can feel something uncomfortable with you to enjoy this debug party.
Yep, you're right. It's all about having fun here and dealing with stuff that's new to a specific vendor community (like UAS last year). So even wrong conclusions are great (e.g. 'Dual SATA on N1 is 1.95 times faster than single SATA' --> take two MX500 SSD and overall SATA performance is 324.42 * 1.95 == 632.6 MB/s? Obviously wrong conclusion but if that makes the community happy why not?).

Final words: ASM1061 is a SATA controller, not a mixture of an internal IDE and SATA controller and we're not running on a x86 PC trying to be able to be operated in backwards compatible modes to connect peripherals or run operating systems from last century. Simply using lspci to gather information or checking dmesg | egrep -i "ahci|sata| ata|scsi|ncq" is pretty sufficient to get the idea in which mode a disk controller operates (hint: if there's NCQ it can't be IDE). 'Benchmarking' without checking dmesg output constantly is a great recipe for producing numbers without meaning anyway.

Bye :)

elatllat
Posts: 1575
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 24 times
Been thanked: 65 times
Contact:

Re: SATA - The only numbers that matter

Unread post by elatllat » Wed Feb 28, 2018 6:46 pm

tkaiser wrote:...
tkaiser are you intentionally trolling?
you seem like a smart person who is technically correct,
but you consistently degrade others and focus on non-relevant limitations that could be overlooked,
or at least at least addressed with some tact.
Please go edit your posts to show some Courtesy/Integrity/Perseverance/Self-control.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 6:49 pm

I kind of figured someone's head would "explode" which is why I posted the info in a new thread instead of the other thread. ;)

1) hdparm is used because its easily available, reproduceable, and measures what I want to measure excluding things I don't (iozone is a filesystem[1] benchmark). As I stated before, the goal is not to "benchmark" rather to determine if N1 "operates within expectations".
2) It was clearly stated that there is a bandwidth bottleneck. Therefore, I felt it implied knowledge that 2 SSD's would not deliver 632.6MB/s. If this needs to explicitly stated, then I will: 2 SDD's will not deliver 1.95X bandwidth. Only 2 HDDs who's combined bandwidth does not exceed the bottleneck limit will.
3) ASM1061 is indeed a SATA controller that is reporting its PCI Class ID as an IDE controller. This does not occur with ASM1062 on a PC. I want to know why. It is my time and money to "waste". I am not promising any particular outcome. Its simply something I choose to research.

[edit]
I did order a ASM1061 based PCIe card for PC. In addition to the mysterious PCI Class ID, it should also provide exact knowledge if N1 is operating as expected (expectation is to see same hdparm non-cached results).


[1] http://www.iozone.org/
IOzone Filesystem Benchmark
[edit2]
Based on this, it does appear that ASM1061 has both an "IDE" mode and a "AHCI" mode:
http://vip.asus.com/forum/view.aspx?boa ... uage=en-us
" ASM1061 Storage Controller [AHCI Mode]
--------------------------------------
Allows you to enable or disable the ASM1061 storage controller.
Configuration options:[Disabled] [IDE Mode] [AHCI Mode]
Also of interest (to me):
https://github.com/hardkernel/linux/blo ... hci.c#L907

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 7:46 pm

Currently creating a RAID1 (mirror) out of the two WD Black drives. Its reported to take 2+ hrs to complete building it. I will update with info after completion.

[edit]
The speed (133MB/s) is consistent with the number reported by hdparm for single drive performance.

Code: Select all

$ cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb1[1] sda1[0]
      976630464 blocks super 1.2 [2/2] [UU]
      [=>...................]  resync =  8.1% (80043200/976630464) finish=112.7min speed=132533K/sec
      bitmap: 8/8 pages [32KB], 65536KB chunk

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Wed Feb 28, 2018 9:07 pm

elatllat wrote:tkaiser are you intentionally trolling?
Sure. Just another time:

1) SATA as any other somewhat decent protocol allows for dynamic link speed negotiation, enables some features dynamically (like NCQ for example) and provides a primitive CRC mechanism to avoid data corruption on the wire if cabling/contact problems are involved (pretty common issue BTW). Users and especially those that try to benchmark have to know that, they also need to know that dmesg | egrep -i "ahci|sata| ata|scsi|ncq" is needed to diagnose for problems and also for correct benchmark execution. If a SSD only negotiates 3Gbps or even just 1.5Gbps speeds then numbers are obviously affected. You're not testing N1 any more but your SATA cable.

To check for transmission errors one could look at the CRC error counter available with almost all disks as SMART attribute 199. It's mandatory prior to every benchmark run and afterwards to check dmesg output as above and to query SMART attribute 199 of the tested disk (unfortunately some disks do not update the value of this SMART attribute even if CRC errors and retransmits happened and some whole disk families suck due to starting to report CRC errors way too late: https://www.smartmontools.org/wiki/Attr ... rn-Digital )

2) SATA vs. IDE: dmesg tells you what's going on (again: if there's NCQ it can't be IDE), lspci -v tells you what's going on (check 'driver' used), the performance numbers tell you what's going on (the fastest IDE standard allows for 166MB/s as absolute maximum)

3) Testing the wrong things 1): Trying to eliminate side effects is a great idea while trying to use hdparm to eliminate testing the filesystem is still wrong as already explained to one of your community members: his SSD with any filesystem applied will show +385 MB/s with a benchmark suitable to test for SATA bottlenecks while hdparm without any filesystem overhead shows lower numbers. Only possible conclusion: the benchmark tool fails and the already explained reason is simple: hdparm's benchmark mode since being invented 30 years ago uses a block size of just 128KB for its tests (active benchmarking: benchmark the benchmark with eg. iostat).

128KB when defined by hdparm devs last century was a really huge number when HDD capacities were measured in MB and maximum transfer speeds were below 20 or even 10 MB/s! But of course it's too small today and that's why the numbers generated are useless (if you want to check for interface bottlenecks without filesystem influences you need to grab hdparm sources and adjust the 128KB to 16MB -- but why? You need a filesystem anyway and if you did already some benchmarking you know that the filesystem doesn't matter for this sort of test, sequential performance with large blocksizes is the same with every common filesystem).

4) Testing the wrong things 2): Why should only sequential transfer speeds be relevant? The majority of use cases with SBC is about random IO. And this is the only area where SATA on N1 can outperform USB3/UAS and that only if PCIe power management is disabled. Testing with something as inappropriate as hdparm won't be able to show this.

5) Testing the wrong things 3): When only testing for sequential performance why testing only read direction? It's well known that high-speed interface performance in the SBC world can be unbalanced and a real sh*t show, see ODROID-C1 Gigabit Ethernet performance viewtopic.php?f=135&t=20104#p136247 or Allwinner's native SATA implementation that performs somewhat ok-ish in read direction (+200 MB/s) but totally fails in write direction (limited to 45 MB/s max). If you limit your 'testing' to only reading tests you end up not noticing such problems. Testing only one direction is too limited especially when afterwards conclusions are drawn about 'high-speed interface in general'. And please don't think this doesn't apply to N1. One such issue just got fixed yesterday regarding USB3/UAS: https://github.com/hardkernel/linux/pull/343 -- how do you want to test for such issues on the SATA ports with an inappropriate tool like hdparm?

6) Testing the wrong things 4): On SBC it's pretty easy to run into CPU bottlenecks so instead of benchmarking 'the SATA implementation' chances are great to partially test only a CPU bottleneck instead (one single threaded process maxing out one CPU core or similar issues related to memory bandwidth that's always important for such use cases!). This is especially important to take into account on big.LITTLE systems:

Code: Select all

root@odroid:~# taskset -c 0 hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   1498 MB in  2.00 seconds = 749.17 MB/sec
 Timing buffered disk reads: 968 MB in  3.00 seconds = 322.53 MB/sec

root@odroid:~# taskset -c 5 hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   2882 MB in  2.00 seconds = 1441.80 MB/sec
 Timing buffered disk reads: 1094 MB in  3.01 seconds = 364.05 MB/sec
Ignoring this will only result in generating numbers without meaning. If that's the goal, everything's fine! If it's about generating some insights how a new platform works and how we should develop for example IRQ/SMP/HMP affinity settings to get both top performance at lowest (idle) consumption levels at the same time it's important to care about such details and then collecting 'The only numbers that matter' as it's unfortunately done here is simply counterproductive.

7) It's important to do the math correctly (you need to divide KB/s through 1024 and not 1000 to get MB/s for example) and to have in mind how the devices and processes in question work internally (zone bit recording -- watching /proc/mdstat will show that resync performance get slower and slower and slower and the performance itself depends greatly on the block size used when setting up the array)

I'm sorry to focus on the stuff that matters (e.g. exploring and also educating users about the importance of block sizes used when accessing the filesystem and storage layers) and as such seem not to be the appropriate person to join 'debug parties' like this, right?

User avatar
rooted
Posts: 6777
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 220 times
Been thanked: 42 times
Contact:

Re: SATA - The only numbers that matter

Unread post by rooted » Wed Feb 28, 2018 10:04 pm

tkaiser wrote: I'm sorry to focus on the stuff that matters (e.g. exploring and also educating users about the importance of block sizes used when accessing the filesystem and storage layers) and as such seem not to be the appropriate person to join 'debug parties' like this, right?
It's not that you aren't the appropriate person, I personally appreciate the patches you have pointed out and the knowledge you bring to the forum.

The problem is you use words like "stupid" to disagree with someone and this is a friendly place. You are harsh and condescending all the time. I assume it's because English isn't your native language, personally I try to look over it instead of being confrontational. This is hard to do occasionally.

At no time is it okay to refer to what one does as stupid, if you think it's stupid word your disagreement in a friendly manner. Say something like it doesn't make sense to you instead.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 11:27 pm

Updated original post with RAID1 and RAID0 numbers.

Again, this was not about benchmarking. It was about expectations.

Based on the PC vs N1 hdparm numbers, its obvious that there was no point to testing "Dual SSD" (bottleneck). The N1 HDD numbers told me what I should expect from RAID0 and RAID1. The numbers observed from those tests support the findings.

TL;DR = N1 SATA performs as expected. All the numbers are within expected ranges. Obviously, different HDD/SSD will produce different results. The goal stated in the original post was achieved.

User avatar
odroid
Site Admin
Posts: 32540
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 186 times
Been thanked: 350 times
Contact:

Re: SATA - The only numbers that matter

Unread post by odroid » Wed Feb 28, 2018 11:48 pm

@crashoverride,
Thank you for the updates.
Can you please add "direct" parameter in the dd test commands something like this to eliminate the cache effect.

Code: Select all

Write command
  dd if=/dev/zero of=test oflag=direct bs=8M count=64
Read command 
  dd if=test of=/dev/null iflag=direct bs=8M
By the way, we tested a feasibility of two bay NAS based on Exynos-5422 a few weeks ago. Internally, we called it ODROID-HC3 feasibility test. ;)
We connected two 3.5" WD RED HDDs to the XU4 via two separated JMS578 bridge boards.
But the sequential 16KB write performance(~95MB/sec) of mdadm RAID1 was not good relatively. I think the XU4 USB 3.0 hub might be a bottleneck.

Code: Select all

iozone -e -I -a -s 100M -r 4k -r 16384k -i 0 -i 1 -i 2
RAID0

Code: Select all

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    14816    16227    18731    19069     2993     4113
          102400   16384   172634   178978   233742   313906   299428   193228
RAID1

Code: Select all

                                                              random    random
              kB  reclen    write  rewrite    read    reread    read     write
          102400       4    13894    14752    19208    18622      903     1736                                                                
          102400   16384    97890    95713   133900   134253   146238   101798 

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Wed Feb 28, 2018 11:52 pm

Almost forgot to say:

Based on the data collected. I believe the bottleneck lies in the ASM1061 controller. The Intel controller shows the expect ~500MB/s (theoretical max of PCIeX1 Gen2 5GT/s) with the SSD. This means the current N1 SATA controller is operating at 65% (~324MB/s) of the desired performance level. Because of this, I believe its worth testing other SATA controllers for use on N1. Its expected this would increase the final cost of the N1, but it could be justified by increasing SATA performance by up to 35%.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Wed Feb 28, 2018 11:57 pm

crashoverride wrote:

Code: Select all

$ sudo dd if=/dev/zero of=/dev/md0 bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 18.764 s, 229 MB/s
This thread is called 'The only numbers that matter' while in reality only showing how hard it is to avoid most common benchmarking mistakes (also known as 'Casual benchmarking: you benchmark A, but actually measure B, and conclude you've measured C' -- see http://www.brendangregg.com/activebenchmarking.html to understand why that's important)

I don't know what @crashoverride tried to demonstrate with the numbers above. But he's clearly not testing disk or interface performance. dd needs specific flags to be useful for disk 'benchmarks' and these are not used here. So what did he do with the command line above? Instead of writing 1048576 times a 4KB chunk of data to disk he filled the RAM instead which is somewhat challenging on a device with just 4 GB physical RAM. If he would have used count=786432 (3GB) instead I believe his number would've been slightly higher.

It's easy to check for such mistakes: always open one terminal and let watch -n 1 free run (this will show you that dd when reading from /dev/zero without oflag=direct as done by @crashoverride eats up all your RAM instead of writing directly to disk which is most probably what the 'benchmark' should've measured instead). Then in another terminal let an iostat 5 run and in another terminal check for CPU bottlenecks by executing htop. This assures that you benchmark the benchmark not producing only numbers without meaning.

So when you for whatever bizarre reasons want to use dd to measure performance you need to tell the tool to test what you want it to test (when testing SATA or disk it's most probably just that and not RAM?). In the above case that's oflag=direct and with the blocksize chosen (4K) the numbers will be way lower. I would test at least with 128KB block size (that's the same value hdparm uses for reads) or just to be sure better with 16MB instead:

Code: Select all

dd if=/dev/zero of=/dev/md0 bs=4096 count=1048576 oflag=direct
dd if=/dev/zero of=/dev/md0 bs=131072 count=32768 oflag=direct
dd if=/dev/zero of=/dev/md0 bs=16M count=256 oflag=direct
This will always test with the same amount of data (4GB) but this time not filling up the entire DRAM but testing disk performance with various blocksizes. With a slow 2.5" HDD this looks like this for example (22.5 MB/s only with 4KB! With @crashoverride's faster disks in RAID-0 it should be well above 50 MB/s?):

Code: Select all

root@odroid:/mnt# dd if=/dev/zero of=/dev/sdc bs=4096 count=1048576 oflag=direct
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 190.962 s, 22.5 MB/s

root@odroid:/mnt# dd if=/dev/zero of=/dev/sdc bs=128K count=32768 oflag=direct
32768+0 records in
32768+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 46.8641 s, 91.6 MB/s

root@odroid:/mnt# dd if=/dev/zero of=/dev/sdc bs=16M count=256 oflag=direct
256+0 records in
256+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 46.8411 s, 91.7 MB/s
With this slow HDD there are no real differences when accessing the disk with either 128KB or 16MB blocksizes but with SSDs this matters. So always ensure to measure at least with 16MB when you want to check for interface or implementation bottlenecks.

BTW: when testing wrongly (forgetting to use oflag=direct so letting dd both ignore the blocksize while also eating up all DRAM) results are slightly lower:

Code: Select all

root@odroid:/mnt# dd if=/dev/zero of=/dev/sdc bs=4096 count=1048576
1048576+0 records in
1048576+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 48.5649 s, 88.4 MB/s
It's easy to avoid such hassles by using appropriate benchmark tools. Neither dd nor hdparm are amongst them :)

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Thu Mar 01, 2018 12:05 am

odroid wrote:Can you please add "direct" parameter in the dd test commands something like this to eliminate the cache effect.
This is the data for RAID0 (stripe). I will need to rebuild the RAID1 (mirror) to test that later.

Code: Select all

$ sudo dd if=/dev/md0 of=/dev/null bs=8M count=64 iflag=direct
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 1.90344 s, 282 MB/s

$ sudo dd if=/dev/zero of=/dev/md0 bs=8M count=64 oflag=direct
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 1.958 s, 274 MB/s
[edit]
RAID 1 data (on 10GB partitions in the interest of time)

Code: Select all

$ sudo dd if=/dev/md0 of=/dev/null bs=8M count=64 iflag=direct
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 3.71314 s, 145 MB/s

$ sudo dd if=/dev/zero of=/dev/md0 bs=8M count=64 oflag=direct
64+0 records in
64+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 4.08427 s, 131 MB/s
Last edited by crashoverride on Thu Mar 01, 2018 12:22 am, edited 1 time in total.

User avatar
odroid
Site Admin
Posts: 32540
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 186 times
Been thanked: 350 times
Contact:

Re: SATA - The only numbers that matter

Unread post by odroid » Thu Mar 01, 2018 12:18 am

crashoverride wrote:Almost forgot to say:

Based on the data collected. I believe the bottleneck lies in the ASM1061 controller. The Intel controller shows the expect ~500MB/s (theoretical max of PCIeX1 Gen2 5GT/s) with the SSD. This means the current N1 SATA controller is operating at 65% (~324MB/s) of the desired performance level. Because of this, I believe its worth testing other SATA controllers for use on N1. Its expected this would increase the final cost of the N1, but it could be justified by increasing SATA performance by up to 35%.
Our test result was slightly over 380MB/s and the performance increasing rate should be less than 30% probably. :D

Anyway, let's try to estimate the material cost can affect the N1 price.
ASM1061 is around $4 (for 10K order in Korea)
ASM1062 is around $8 (for 10K order in Korea) : We need to increase the PCB dimensions due to larger IC package size. It also needs an extra power block.
As I mentioned in other threads, we may have a serious trouble once we activate two lanes of ASM1062 at Gen2 5GT/s speed.

Other than Asmedia, we may have to consider Marvell PCIe-SATA bridges.
According to my experience with other Marvell SoC solutions (even very long time ago), their price was way too expensive.
But I have no idea about their 88SE917x chipset price at this moment.

Do you guys know any other chipset vendors who make an affordable PCIe-SATA chipset?

Let me be more honest, if we want to start the mass production in May or June, we have to minimize the changes. Sorry about that.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Thu Mar 01, 2018 12:25 am

crashoverride wrote:Based on the data collected. I believe the bottleneck lies in the ASM1061 controller. The Intel controller shows the expect ~500MB/s (theoretical max of PCIeX1 Gen2 5GT/s) with the SSD. This means the current N1 SATA controller is operating at 65% (~324MB/s) of the desired performance level.
Why do you use a '~324MB/s' number instead of the real one for N1? And why should '~500MB/s' be the 'theoretical max of PCIeX1 Gen2 5GT/s' when you tested on an Intel system with 'native SATA' where the SATA controller is part of the chipset (SATA ports attached via PCIe Gen3 with 128b130b encoding resulting in true 6Gbps with 8b10b encoding vs. 5GT/s also with 8b10b encoding and twice the overhead since not 'native SATA' but 'PCIe attached SATA' here)?

The best we have seen in the wild for PCIe Gen2 x1 attached SATA is ~385 MB/s which is exactly what we have with N1 too. Only switching to Gen2 x2 or similar might allow to exceed the 385MB/s barrier.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Thu Mar 01, 2018 12:35 am

Provided RAID1 data in above post.
odroid wrote:Let me be more honest, if we want to start the mass production in May or June, we have to minimize the changes. Sorry about that.
The current SATA performance is more than adequate for my use. All my uses are CPU/GPU bound. If changes to SATA are not suitable, practical, or cost effective, then what we have currently is fine by my standards. The only importance is that it was considered.

This should conclude any and all involvement I have with regard to SATA. Its impossible to have a "calm" discussion about it. After I receive my PCIe card based on the same ASM chip, I will perform the same tests on a PC with the expectation that the numbers are the same as on N1. If all goes as expected, that will be the last piece of data that identifies the bottleneck.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Thu Mar 01, 2018 12:36 am

odroid wrote:Do you guys know any other chipset vendors who make an affordable PCIe-SATA chipset?
Nope. And various tests with those PCIe Gen2 x1 controllers showed that ASM1061 doesn't perform that bad especially when compared to the low-end Marvells:

http://www.tomshardware.de/ssd-motherbo ... 308-4.html
https://de.scribd.com/document/97282353 ... ing-an-SSD

I think exploring alternatives (especially the much more expensive Marvell controllers) is only worth the efforts once we know whether we can exceed 5 GT/s with RK3399 and if you plan to do another RK3399 variant dedicated to storage. Then Marvell controllers could be a good choice since able to be combined with good SATA port multipliers allowing to use fast FIS based switching and not bottlenecked CBS only.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Thu Mar 01, 2018 12:42 am

@tkaiser,

Regretfully, I currently have a lot of active projects. This prevents me from responding in detail to each and every concern. Perhaps some time in the future, we can have a more detailed and involved discussion and investigation with more "best practices". As stated before, this was not intended to be a benchmark. It was "quick and dirty" by design.

elatllat
Posts: 1575
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 24 times
Been thanked: 65 times
Contact:

Re: SATA - The only numbers that matter

Unread post by elatllat » Thu Mar 01, 2018 1:04 am

odroid wrote:...if we want to start the mass production in May or June...
Was the total number of XU4 sold more than the number of MC1/HC1/etc sold? Is there market for a N1 without sata?

crashoverride wrote:...I currently have a lot of active projects. ..
On that subject Fan chirp (easily fixed with passive cooling, or less ideally by increasing the min speed) and video display (can be fixed post production) are the things that would improve the N1 the most (IMO obviously)... would also be nice to know the ddr4 trade-off/feasibility.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Thu Mar 01, 2018 1:11 am

elatllat wrote:Fan chirp
The solution that I would like to do for this is to patch the pwm-fan driver so that it always starts at 100% and then reduces to the set level over an as-yet-undetermined time span.
elatllat wrote:video display
That one is going to be more difficult since there are a lot of others that need to agree on the solution (assuming this is the DRM property+vsync issue).

[edit]
Someone should do a memory bandwidth test/benchmark to ensure we are operating at the expected frequency.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Thu Mar 01, 2018 2:31 am

It doesn't look like ASM1602 offers enough of a performance increase to justify it:
http://blog.kihltech.com/2013/12/storag ... i-9211-8i/

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Thu Mar 01, 2018 3:02 am

crashoverride wrote:It doesn't look like ASM1602 offers enough of a performance increase to justify it:
http://blog.kihltech.com/2013/12/storag ... i-9211-8i/
You're referencing a board that uses 2 x ASM1061 (not ASM1062) so that's just the same performance numbers as everywhere else with ASM1061: http://www.asrock.com/mb/Intel/Z87%20Ex ... ifications

ASM1062 shows better performance than ASM1061 only when making use of 2 Gen2 lanes.

RomaT
Posts: 238
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 44 times
Contact:

Re: SATA - The only numbers that matter

Unread post by RomaT » Thu Mar 01, 2018 12:03 pm

odroid wrote:ASM1061 is around $4
ASM1062 is around $8.
this is not the place to save money, for example, the delivery to me costs from $35 and above
if we talk about a better performance than the USB3.0, it is necessary to use at least two lanes.
odroid wrote:We need to increase the PCB dimensions due to larger IC package size. It also needs an extra power block.
Make two versions with and without SATA, because there is no sense with one lane, can use simple USB3.0-SATA
tkaiser wrote:ASM1062 shows better performance than ASM1061 only when making use of 2 Gen2 lanes.
MCU RK3399 (Odroid-N1) It has 4 PCI-E lanes, three of them are now just hanging in the air, each 2.5GT/s per lane.
Attachments
rk.jpg
rk.jpg (325.46 KiB) Viewed 6250 times

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Thu Mar 01, 2018 2:23 pm

The SATA chips are all 1 or 2 lane.

With 1 lane, it can run at Gen2 (5GT/s).
With 2 lane, it can only run at Gen1 (2.5GT/s). Rockchip retroactively downgraded their PCIe port.

This means that 1 and 2 lane perform the same 1 X 5GT/s = 5GT/s, 2 X 2.5GT/s = 5GT/s.

A 2 lane SATA chip is only useful if there are 2 Gen2 lanes (2 X 5GT/s = 10GT/s). Nobody makes 4 lane SATA chips that are suitable for use.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Thu Mar 01, 2018 3:33 pm

It might be prudent to just wait for RK3399Pro before looking at SATA upgrades for a future Odroid product. Perhaps with the next revision, Rockchip may fix their PCIe interface.

RomaT
Posts: 238
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 44 times
Contact:

Re: SATA - The only numbers that matter

Unread post by RomaT » Thu Mar 01, 2018 4:41 pm

And what if completely abandon SATA, in favor of 4 lanes M.2 PCI-Express compatible with Gen2 ?
then do not need a controller, if necessary SATA, then as usual via USB3.0-SATA ...

User avatar
memeka
Posts: 4370
Joined: Mon May 20, 2013 10:22 am
languages_spoken: english
ODROIDs: XU rev2 + eMMC + UART
U3 + eMMC + IO Shield + UART
Has thanked: 1 time
Been thanked: 39 times
Contact:

Re: SATA - The only numbers that matter

Unread post by memeka » Thu Mar 01, 2018 6:15 pm

@RomaT - latest rockchip datasheet shows the pcie lanes as Gen1 not Gen2.
This means that even if it’s possbile to achieve Gen2 on one lane, as HK did, probably you cannot have Gen2 connection on all 4 lanes.
This is what testing now needs to determine.
Maybe future revisions of the chip will improve this issue and bring back Gen2. Maybe HK can ask RK about this...

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Thu Mar 01, 2018 8:55 pm

RomaT wrote:And what if completely abandon SATA, in favor of 4 lanes M.2 PCI-Express compatible with Gen2 ?
Currently no one (outside of Rockchip) seems to know about overall PCIe performance and reliability with RK3399 and for whatever reasons no one asks those guys that could know (e.g. Theobroma System's beeble available through IRC at #linux-rockchip).

You might find some partially interesting links on CNX comments section: https://www.cnx-software.com/2018/02/06 ... ent-551508

We had some hope to test with another RK3399 device that fully exposes all 4 PCIe lanes (RockPro64) but unfortunately the main dev is running out of spare time and the test setup (using an Samsung Pro 960 NVMe disk via a M.2 adapter) is challenging due to underpowering since this SSD is a beast and needs too much on the 3.3V rail. Only next PCB revision will fix this.

I had some hope to hold another yet not even announced RK3399 thing with M.2 connector in my hands to be able to test but unfortunately China went down two weeks ago due to Chinese New Year (they're still recovering partially from this ;) ): https://browser.geekbench.com/geekbench3/8515294

Besides that M.2 key M would be needed to expose all 4 PCIe lanes and you only find high-end M.2 SSDs with this connector type. Not a great idea to use these expensive SSDs in this scenario since they're made for Gen3 x4 (4 x 8GT/s!) and not for the RK3399 implementation that might be limited to rather boring 5GT/s (maybe more, maybe not -- we simply don't know yet). And even if N1's PCIe attached SATA is in most performance areas no improvement over USB3/UAS attached SATA it's still the better choice for a number of reasons, the most important one: less hassles.

RomaT
Posts: 238
Joined: Thu Oct 23, 2014 4:48 pm
languages_spoken: Russian
ODROIDs: -H2 rev.B, -XU3, -XU4, -C1, -C2, -W, -VU, CloudShell
Location: Perm, Russia
Has thanked: 6 times
Been thanked: 44 times
Contact:

Re: SATA - The only numbers that matter

Unread post by RomaT » Thu Mar 01, 2018 9:34 pm

tkaiser wrote: made for Gen3 x4 (4 x 8GT/s!) and not for the RK3399.
Not all, for example, Toshiba OCZ RD400 - supports both GEN2 and GEN3
Samsung Pro 960 - not support Gen2

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Thu Mar 01, 2018 11:43 pm

RomaT wrote:
tkaiser wrote: made for Gen3 x4 (4 x 8GT/s!) and not for the RK3399.
Not all, for example, Toshiba OCZ RD400 - supports both GEN2 and GEN3
Samsung Pro 960 - not support Gen2
If it supports Gen3 it should support Gen2 and Gen1 (part of the specificitation --> 'link training'). It simply doesn't make a lot of sense to use an expensive SSD that is heavily bottlenecked by the host's PCIe interface wrt both sequential and random IO performance. And I don't think Hardkernel is considering this anyway. BTW: As far as I understood (being a hardware noob!) RK3399's PCIe controller can only operate in x4 mode but not as 2 x x2 or 4 x x1 so the possibilities are rather limited anyway and if the implementation is not able to maintain more than 1 Gen2 link it's even worse...

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Sat Mar 03, 2018 12:57 pm

PCIe SATA card arrived. Tested with same SSD:

Code: Select all

02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

$ sudo hdparm -tT /dev/sdc

/dev/sdc:
 Timing cached reads:   33852 MB in  1.99 seconds = 16993.52 MB/sec
 Timing buffered disk reads: 1130 MB in  3.00 seconds = 376.13 MB/sec
The two things of interest are that its reported as ASM1062 (not ASM1061) SATA (not IDE) controller and the performance is slightly better (376.13 vs 324.42).

I verified through visual inspection of the board that the chip is marked ASM1061. While not conclusive, this supports my theory that we can get better performance on N1 if we can configure the N1's ASM1061 as a "SATA controller" instead of the current "IDE controller". It also suggests that the bottleneck is not a RK3399 SoC limitation. In fact, it may simply be the limit of a 5GT/s link.

[edit]
To explicitly answer the question why the same chip is seen as different devices, I believe (as stated before) the answer lies in that PCIe board has a standard "option rom" that the PC BIOS executes (splash screen seen during boot). This ROM likely configures the device as a "SATA controller" before resuming BIOS boot.

[edit2]
It should go without saying, but the option ROM is x86 and is of no use to the AArch64 based processor on N1.

@odroid,
Is it possible to inquire with Asmedia about what is needed to configure their chip on non-x86 devices?

[edit3]
Also worth mentioning is that the PC is running Ubuntu 16.04 with a 4.4 kernel:

Code: Select all

$ uname -r
4.4.0-116-generic

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Sat Mar 03, 2018 3:21 pm

crashoverride wrote:PCIe SATA card arrived. Tested with same SSD:

Code: Select all

02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

$ sudo hdparm -tT /dev/sdc

/dev/sdc:
 Timing cached reads:   33852 MB in  1.99 seconds = 16993.52 MB/sec
 Timing buffered disk reads: 1130 MB in  3.00 seconds = 376.13 MB/sec
The two things of interest are that its reported as ASM1062 (not ASM1061) SATA (not IDE) controller and the performance is slightly better (376.13 vs 324.42).
I know it won't help you since you're actively rejecting reality but for everyone else: Testing wrongly leads obviously to wrong conclusions.

hdparm was a useful benchmark tool several decades ago, now with faster busses and devices it's just inappropriate. The block size hdparm operates with is 128KB which is too low as already explained several times. The PC you're testing with has a magnitudes higher memory bandwidth, that's why you see 376 MB/s vs. 324 MB/s on the N1 (switch on N1 to little cores that operate with a way lower memory bandwidth and you'll see most probably below 310 MB/s or even less). As soon as you would start to benchmark correctly you would be able to check for the real limitation (even @odroid told you already that your 324 MB/s number is just the result of 'benchmarking gone wrong').

I showed already in this thread that pinning the useless hdparm task to little or big cores makes a difference in reported throughput numbers and that's due to the well known fact that memory bandwidth (almost twice as high when running on big cores) matters with IO as long as the real bottlenecks are not the limit (385 MB/s).

I have an ASM1061 I thought would be an ASM1062 instead in the beginning. Why? Since showing up in Linux as ASM1062. Then I did a firmware update and now it's showing up as ASM1061. Same goes with the device class (IDE vs. SATA). It's all just what the firmware reports to the OS. Unfortunately for whatever reasons on N1 we can not update the firmware (@odroid tried it, I tried it, it doesn't work. @odroid said they might use an SPI protocol analyzer but I think it's not worth the hassles since sequential performance can not change anyway and the 385 MB/s we get everywhere -- yeah on N1 and on your PC! -- will remain. Maybe random IO performance could change but that's something you're not interested in for whatever reasons).

BTW: everything already documented including issues later N1 users might run into: https://forum.armbian.com/topic/6496-od ... ment=50088

On a related note: it's a pretty common 'feature' that SATA controllers show up as different models when queried through software. I have an ASM1053 that will be listed as ASM1153 on some installations while as ASM1053 or ASM1053E on others (depends on some detection logic that is somewhat different in some library somewhere), we deal with JMS578 that are silkscreened as JMS567 on the chip and so on. It's neither a good idea to trust in numbers without meaning nor in some random strings a software spits out.

We need active benchmarking to draw any useful conclusions.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Sat Mar 03, 2018 3:44 pm

tkaiser wrote:Testing wrongly leads obviously to wrong conclusions.
I am not forming conclusions. I am forming theories. Obtaining the information necessary to operate the ASM1061 the same on PC (SATA class) and N1 (IDE class) is the most direct method to proving or disproving the theory. I keep stating repeatedly that I AM NOT BENCHMARKING. The numbers are not being interpreted as "exact". A different testing method yielding different numbers is entirely expected but irrelevant. The numbers from the same tool (hdparm) with the same hardware (asm1061, SSD) are being compared and a hypothesis formed based on the data. If setting the ASM1601 is "SATA controller" mode does not affect the data, then it can be attributed to "128k buffers", PC difference, or space aliens. The "why" is irrelevant at that point since, again, I AM NOT BENCHMARKING. The goal is not to get 'the best number possible'. It is to verify operation. Right now based on my "wrong" tests, there is ample data to suggest that we are not operating as efficiently as possible. Because I AM NOT BENCHMARKING, if changing asm1061 to "SATA controller" mode has no observable impact on performance, what ever performance number is yielded is entire irrelevant. It does not matter if its 2Kbps or 100TB/s since its at that point it is "expected operation".

As a gesture of good faith, @tkaiser, if you can provide exact commands to run and they are not disruptive to my test environment or otherwise prohibitive, I will perform the test and report the results.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Sat Mar 03, 2018 3:55 pm

tkaiser wrote:everything already documented including issues later N1 users might run into: https://forum.armbian.com/topic/6496-od ... ment=50088
Elaborating on that a little bit more. We know since two weeks which SATA limitations are in place with N1 (~385 MB/s per port and overall bandwidth limitation due to PCIe -- that's (not so) surprisingly the same number and limitation as everywhere else around), @rooted discovered an important setting that's available through sysfs (PCIe power management) which heavily affects performance with small block sizes (something that needs appropriate benchmark tools, using broken stuff like hdparm is not even able to report any difference) and it would be interesting whether the way more important performance with small block sizes and especially random IO performance varies depending on ASM1061 firmware updates (see above) and kernel version.

We can't change the maximum sequential performance (since it's a chip / connection bottleneck) but that's not important anyway. There exists no real use case that could make use of more than 385 MB/s sequential transfers with large blocksizes. What really matters is performance with smaller block sizes and especially random IO. All test results we have seen so far are bogus since testing either totally wrong (using hdparm or dd) or only able to generate some baseline numbers (the iozone results provided by me for example). A test or benchmark tool that doesn't reflect reality is useless or only able to test for specific aspects (e.g. maximum sequential transfer speeds --> testing for some bottlenecks).

In reality workloads are different than such primitive benchmarks and there are some challenges that also need different testing methodology. For anyone interested in this: Use fio with specific workload profiles combining reads and writes at the same time to check for real world performance issues. This is especially important with SSDs and TRIM. SATA in contrast to its IDE predecessor supports NCQ https://en.wikipedia.org/wiki/Native_Command_Queuing which is the reason why it's able to squeeze more out of fast storage but also can become challenging with some other tasks like TRIM. So a good test would be to use 2 SSDs connected to the SATA ports and test with some workloads that also include deleting stuff. It's better to test for this now than to start with N1 mass production and get reports from power users about data corruption later.

We (the much larger IT community around) already know that there might be issues with ASM1061 so it's worth to test for this stuff now. I'm out now since dealing with the 'vendor community syndrome' simply sucks too much (c'mon: there exists a reality outside of a vendor community's micro reality! Why not accepting this and relying on 'external' knowledge and experiences? Why the try to reinvent the wheel again every single time?)

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Sat Mar 03, 2018 4:06 pm

crashoverride wrote:If setting the ASM1601 is "SATA controller"
So you're really thinking you're dealing with the difference between IDE and SATA here? As explained already several times and directly to you: IDE doesn't allow for your numbers and with IDE there's no NCQ. Good luck further rejecting reality and drawing weird conclusions. I tried to explain several times which testing methodologies are appropriate and why but it doesn't help. Also I explained several times to you why the numbers you generate look like this (block sizes and memory bandwidth) and how to test for this to confirm it's just this. But it doesn't matter to you...

User avatar
rooted
Posts: 6777
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 220 times
Been thanked: 42 times
Contact:

Re: SATA - The only numbers that matter

Unread post by rooted » Sat Mar 03, 2018 4:22 pm

@tkaiser

He asked and this was your answer, now do you see what I mean about your mannerisms?

[quote="crashoverride]As a gesture of good faith, @tkaiser, if you can provide exact commands to run and they are not disruptive to my test environment or otherwise prohibitive, I will perform the test and report the results.[/quote]

tkaiser wrote:I tried to explain several times which testing methodologies are appropriate and why but it doesn't help. Also I explained several times to you why the numbers you generate look like this (block sizes and memory bandwidth) and how to test for this to confirm it's just this. But it doesn't matter to you...
(I can't figure out why this isn't quoting correctly)
Last edited by rooted on Sat Mar 03, 2018 4:28 pm, edited 3 times in total.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Sat Mar 03, 2018 4:25 pm

tkaiser wrote:So you're really thinking you're dealing with the difference between IDE and SATA here?
Perhaps a change of terminology is needed:

I am proposing that operating in "1602 mode" is more efficient than "1601 mode". The "IDE" and "SATA" monikers are only indicative of the "1601/1602 mode" of operation the chip is in and are otherwise irrelevant.

[edit]
The implication is that ASM1601 and ASM1602 are the same silicon but packaged differently. It is not unheard of to "turn off" silicon features to differentiate a product line that is otherwise the same.

[edit2]
Since it has likely already been lost in discussion, I will reiterate: I am not guaranteeing any specific outcome. Its simply something that should be tested. If it does provide a performance increase, the increase will also be seen in tests that are not "wrong".

[edit3]
It has not been stated but may further clarify the scope. I was hoping there would be an option to set "IDE mode" on the PCIe expansion card. Simply toggling this would also provide the same knowledge as to whether there is a performance difference between "1601 mode" and "1602 mode". Unfortunately, the option ROM only shows a "splash" showing attached devices. It does not offer the ability to configure the card.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: SATA - The only numbers that matter

Unread post by tkaiser » Sat Mar 03, 2018 5:00 pm

crashoverride wrote:The implication is that ASM1601 and ASM1602 are the same silicon but packaged differently.
No, that's just you constantly drawing weird conclusions and rejecting reality. @odroid already explained multiple times that ASM1061 and ASM1062 are different packages. The last time @odroid explained it directly to you and in this thread: viewtopic.php?f=149&t=30307#p216918

Edit: Ah, silly me. That's what you said ;) But doesn't matter since the string lspci outputs doesn't matter, the BIOS setting also doesn't matter and it's about what the driver does and accepting the real limitations we're running into.

What's next? 'Testing' whether ambient temperature matters? Given your testing methodology (yeah, I understood: you're not benchmarking, you're only using benchmark tools to develop fancy theories) you will for sure generate some new numbers that allow you to develop the theory that ambient temperature matters. For the simple reason that you're still using the wrong tool in the wrong mode (a test/benchmark tool that only runs 3 seconds needs to be executed at least twenty times and then average values have to be used and of course decimal places are removed from results since garbage. But with your hdparm tool it's useless anyway since you reject to understand how this tool is limited and ignore how to use more appropriate tools and why -- all explained way too often already within the last two weeks).

Good luck developing more theories and ignoring reality :)

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Sat Mar 03, 2018 7:18 pm


After consulting the above reference material, I have composed the following response:

@tkaiser,
I have read through some of your insightful Armbian posts. What I find encouraging is that despite what a manufacturer told you in some cases, you persevered with your own experimentation. Armbian would surely be just another "RPi distro" had you not taken the initiative to explore beyond "what you were told". I commend this spirit and believe I am emulating the example you set. I too find myself challenging "what I was told". I have also noticed that the experiments you conduct do not always prove you are right. However, you have learned and become more experienced from your successes and failures, and I hope to achieve the same. It would appear these successes and failures are part of what makes Armbian special. I too hope to make N1 special through my own experiment successes and failures.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Sat Mar 03, 2018 9:05 pm

I had an idea to just rip the flash off my PCIe SATA board and replace it on the N1. The N1 schematic shows the flash as U18. Despite my best efforts, I could not locate this chip on N1.

I see no other way than to "just ask Asmedia" about enumerating ASM1061 as 1062 to test this configuration.

User avatar
mad_ady
Posts: 6795
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 216 times
Been thanked: 166 times
Contact:

Re: SATA - The only numbers that matter

Unread post by mad_ady » Sat Mar 03, 2018 9:05 pm

@crashoverride just got a degree in "political correctness" :)

User avatar
odroid
Site Admin
Posts: 32540
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 186 times
Been thanked: 350 times
Contact:

Re: SATA - The only numbers that matter

Unread post by odroid » Sat Mar 03, 2018 11:11 pm

crashoverride wrote:I had an idea to just rip the flash off my PCIe SATA board and replace it on the N1. The N1 schematic shows the flash as U18. Despite my best efforts, I could not locate this chip on N1.

I see no other way than to "just ask Asmedia" about enumerating ASM1061 as 1062 to test this configuration.
U18 SPI flash chip must be mounted near the ASM1061 chip. It is a tiny package instead of much bigger SO-8 package.
Please note that the flash chip was blank because Asmedia didn't supply any flashing utility(even a binary blob) for the Aarch64 Linux platform.
So we've used only the masked-ROM firmware in the ASM1061. We may remove the U18 for the mass production if we can't figure out how to write an external firmware to the SPI flash.

crashoverride
Posts: 4549
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1
Has thanked: 0
Been thanked: 77 times
Contact:

Re: SATA - The only numbers that matter

Unread post by crashoverride » Sun Mar 04, 2018 1:10 pm

odroid wrote:Please note that the flash chip was blank because Asmedia didn't supply any flashing utility(even a binary blob) for the Aarch64 Linux platform.
I had an idea to pass the PCIe device through to a qemu VM running x86 emulation to use the DOS flash util. However, this does not seem to be possible since there is no IOMMU on the RK3399 PCIe device. I only mention it in case others may expand upon it.

There may be other non Aarch64 options available: for example a UEFI utility. The ideal situation is that we simply obtain the register documentation and write our own flash utility.

Post Reply

Return to “General Chat”

Who is online

Users browsing this forum: No registered users and 1 guest