Any HC1/HC2 setup based on the new N1 hardware?

User avatar
rooted
Posts: 8744
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 743 times
Been thanked: 378 times
Contact:

Re: Any HC1/HC2 setup based on the new N1 hardware?

Post by rooted »

tkaiser wrote: Wrt 2 SSDs on N1... while ODROID community is busy doing funny things around the SATA controller on the board there's something that needs real attention: checking behaviour with 2 SSDs, NCQ and TRIM: viewtopic.php?f=149&t=30307 (but I've given up on this, dealing with the 'vendor community syndrome' is nothing worth the time or efforts)
Then why keep pointing it out and bringing it up. I'm really tired of you insulting the entire odroid community!

ASword
Posts: 224
Joined: Fri Aug 04, 2017 12:48 pm
languages_spoken: english
ODROIDs: XU4, HC1, 2x N2
Has thanked: 16 times
Been thanked: 6 times
Contact:

Re: Any HC1/HC2 setup based on the new N1 hardware?

Post by ASword »

Thanks for the great reply, tkaiser. FWIW, the database is a time-series db so its read/write patterns are somewhat different than a typical RDBMS. Most of your observations are quite useful, however.

Data losses with BTRFS: does anyone have any links to help quantify this a bit better? At the moment I'm running the 4.14 kernel on my HC-1 and haven't seen any issues in the past ~3 months. I'm actually quite tolerant to occasional errors in the data, but (obviously) errors resulting in structural problems in the database would be bad.

elatllat
Posts: 1873
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2, C4, N2+, HC4
Has thanked: 63 times
Been thanked: 132 times
Contact:

Re: Any HC1/HC2 setup based on the new N1 hardware?

Post by elatllat »

Basically BtrFS is still maturing so sometimes there are bugs that trash a whole volume(like this but I got something like it on 4.9.30 just after 2017/5/25). It will get more stable but for now it's not stable enough for redhat to use. So if you do want to use it, just be sure you don't depend on it.

tkaiser
Posts: 673
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 5 times
Contact:

Re: Any HC1/HC2 setup based on the new N1 hardware?

Post by tkaiser »

ASword wrote:Data losses with BTRFS: does anyone have any links to help quantify this a bit better?
I won't elaborate on btrfs here since totally irrelevant but on what dealing with storage in the SBC world means: dealing all the time with an insanely high and pretty frustrating amount of both reasons for failure at the hardware layer and ignorance.

When filesystem designers think about storage implementations then they base their decisions on reasonable looking stuff like
1) disks are powered reliably
2) disks are connected reliably (NO cable/contact problems between host and disk corrupting data on the wire)
3) host is powered reliably
4) host DOES NOT corrupt data on its own

The above are the basics all old storage attempts are based on. Since this describes an ideal world that does not exist some precautions have been added over the decades, e.g. ECC memory on the lowest hardware layer or on the lower protocol layers checksums are used to try to verify if data sent over a connection whether it arrives altered or not (CRC checks, part of every low-level protocol but only sometimes available to upper layers -- do a web search for 'SMART attribute 199' for example. If SATA cable/contact problems lead to data corruption between host and disk the disk controller will ask the host to retransmit data -- and vice versa -- and disks should add every event to this 'CRC mismatches' counter available through SMART attribute 199)

In recent years since computers are fast enough filesystem designers also tried to address the above concerns at higher layers. Modern storage attempts like ZFS, btrfs or ReFS for obvious reasons do not trust blindly any more in the data written out to storage arriving there unaltered or remaining intact indefinitely but they check for data integrity (that's the 'checksumming thing'). So if data corruption occured between host and disk or on the disk you're at least able to take notice. If you added the right amount of redundancy in the right way to your storage setup then with these modern approaches even self-healing will happen (of course not with stupid/anachronistic stuff like e.g. mdraid's RAID1!).

But still all these concepts rely on some of the above basics, most importantly that all components are powered appropriately, that a 0 in memory remains a 0 and a 1 remains a 1 and that the administrator of the whole setup takes care that components are interconnected (wired) in a sane way.

When we look at the situation in the SBC world all of the above simply sucks and the 'administrators' of these devices (the end users themselves) neither know nor care.

1) USB disks are powered by the USB ports of a SBC for example --> sh*t show as expected, countless reports that talk about 'UAS is always a problem' or other BS are simply related to the fact that undervoltage or undercurrent in such situations occurs and the disks are not able to operate as expected. Users look into dmesg output and blame some driver reporting the real issue happening layers below.

2) Reliable connectors between disk and host: on SBC still everywhere only the crappy variants are used: internal SATA instead of eSATA, USB3-A instead of USB-C and so on. An awful lot of reports talking about software issues (people looking at dmesg output) is just cable/connector crappiness and some driver layers above having to deal with this mess and spitting out misleading messages via dmesg

3) Host reliably powered? Even this can be a sh*t show as we've seen with all the Micro USB powered SBC around and even with ODROID XU4 so many times. Users are either not willing or able to understand Ohm's law, they look at amperage ratings of their PSU, read 6A and are happy while in reality crappy cables between board and PSU or a crappy PSU itself creates a nice voltage drop under load. Underpowered electronics components usually do not work as expected. Period.

4) Data corruption on the host (happening in memory: DRAM or internal caches). Fortunately not that much of an issue with ODROIDs so far except those rare occassions where ECC memory would prevent a bit flip that happens from time to time with non-ECC memory. But on other SBC this is sometimes a huge problem due to stupid settings. With those cheap Allwinner boards for example we face the situation that both board makers as well as mainline u-boot/linux device maintainers for whatever bizarre reasons on a regular basis use default values for cpufreq/dvfs and DRAM clockspeeds that are too high (or CPU supply voltages too low for a certain clockspeed). Which results in data corruption by design unless settings are adjusted to sane levels. In such a situation without ECC memory the host itself is able to destroy every filesystem you want to use on such a system!

So when you hear about any storage related report generated with any SBC that tries to blame any of the software layers involved better hold on first and think about whether the most basic requirements for storage to work appropriately are met or not (that's the hardware layer below no one is thinking and talking about). Usually these most basic requirements are NOT met and users reporting stuff don't even understand these basics (and to be honest: they should not have to understand these basics but board designers should try to prevent these issues in the first place! The hardware design should take care of such issues and not the user!)

Average users don't understand why powering through Micro USB is bad, they don't understand that looking at amperage ratings is useless since usually voltage drops are the real issue, they don't understand how fragile some connectors are and that a dozen of matings can already destroy some of these connectors leading to all sorts of troubles appearing in dmesg then being interpreted as software issues and spread through the Internet as such...

So now let's have a look how Hardkernel addressed and addresses these issues:

* Fortunately most of their boards are not able to be powered through Micro USB which is a huge plus since it prevents users from connecting crappy phone chargers and using average USB cables with power wires too tiny. But powering through 5V is always problematic. It's the voltage drop issue not a single user wants to understand since focussed on amperage ratings only: https://forum.armbian.com/topic/6171-od ... a-and-sdb/

* So far the only 2 ODROIDs with a really sane powering scheme with storage use cases in mind are the HC2 and the N1. Both use an input voltage well above 5V (wide range input voltage but when using 3.5" disks it has to be 12V) and generate from this variable input voltage stable 5V the connected disks will be fed with.

* Sane dvfs and DRAM clockspeed settings have been developed for every ODROID so far. Only potential improvement: using ECC DRAM (not possible with any of the SoCs chosen for any available ODROID) and downclocking DRAM if you're concered about data integrity in a slightly paranoid way.

* Reliable interconnections between host and disk? Solved only on HC1 and HC2 by eliminating USB3 interconnection troubles but this is still a huge problem on the other boards since the appropriate connectors that reliably connect host and disks and are able to survive thousands of matings would make the whole setup a lot more expensive. That's why we see USB3-A receptacles (bad since awful tiny contacts for the SuperSpeed data lines -- USB-C is at least hundred times more reliable here) and internal SATA connectors instead of eSATA (maximum matings by specs: 50 vs. 5,000)

In other words: Hardkernel performs pretty good in this area and even improved with their latest 2 boards (HC2 and N1 implementing 'input voltage above 5V'). But since the type of connectors/cables affecting host <--> disk connections unfortunately is not the optimal one there's still room for improvements. But this would require educating users about the types of problems they might run into. End users need to know that USB3-A connectors are prone to contact problems, same with internal SATA connectors (though @odroid said to replace those on N1 dev samples with better ones on the production batches), same with the eMMC connector for example: users rip out the eMMC module 50 times to flash eMMC with the SD card adapter, then re-insert the module back in the board. As expected the connector starts to fail after some time and all sorts of weird things happen (looking like software issues of course). The last thing the user remembers he did differently compared to the past was switching from a 3.x kernel to 4.x and boom... there's a new thread about 'switching to mainline kernel bricked my board' in the forum. This can only be solved by educating users why such hardware issues matter.

IMO with the situation on N1 in mind they should also elaborate on why not every SATA device is a great choice. There exist pretty crappy SSDs and even fake ones in the meantime. Users believing into 'SATA is always great' and following the 'always buy as cheap as possible' principle will run into the same issues they run today when buying crappy or fake SD cards: insanely low performance and data losses.

Back to 'reports about $filesystem failing or responsible for data losses'. You find reports on the Internet blaming everything possible and if conclusions look easy most further readers will believe it. Better always question whether the prerequisits for storage to work properly were met or not. If it happened with SBC do not simply trust in. Compared to a PC or server so much more can go wrong and goes wrong with storage and SBC that there's simply no reason to believe into any report prior to knowing all the details.
Last edited by tkaiser on Thu Mar 08, 2018 4:33 pm, edited 1 time in total.

elatllat
Posts: 1873
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2, C4, N2+, HC4
Has thanked: 63 times
Been thanked: 132 times
Contact:

Re: Any HC1/HC2 setup based on the new N1 hardware?

Post by elatllat »

User error and hardware limitations aside, looking at the last 10 LTS Kernel versions I see 16 btrfs bugs and only 1 ext4 documentation fix.

btrfs
4.14.24
-- Fix flush bio leak
4.14.22
-- Fix quota reservation leak on preallocated files
4.14.21
-- fix deadlock in run_delalloc_nocow
-- fix crash due to not cleaning up tree log block's dirty bits
-- fix extent state leak from tree log
-- fix btrfs_evict_inode to handle abnormal inodes correctly
-- fix use-after-free on root->orphan_block_rsv
-- fix unexpected -EEXIST when creating new inode
4.14.20
-- raid56: iterate raid56 internal bio with bio_for_each_segment_all
-- Handle btrfs_set_extent_delalloc failure in fixup worker
4.14.17
-- incremental send, fix wrong unlink path after renaming file
-- fix deadlock when writing out space cache
-- bail out gracefully rather than BUG_ON
-- fix list_add corruption and soft lockups in fsync
-- Fix transaction abort during failure in btrfs_rm_dev_item
4.14.16
-- fix stale entries in readdir

ext4
4.14.21
-- correct documentation for grpid mount option

tkaiser
Posts: 673
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 5 times
Contact:

Re: Any HC1/HC2 setup based on the new N1 hardware?

Post by tkaiser »

elatllat wrote:User error and hardware limitations aside, looking at the last 10 LTS Kernel versions I see 16 btrfs bugs and only 1 ext4 documentation fix.
Sure, btrfs is younger, not that stable and especially not that robust compared to the mature ones like ext4 and XFS. IMO there's no need to discuss this and as already explained I wouldn't put databases on btrfs anyway since you won't benefit from the theoretical advantages any more. I was not talking about btrfs at all before, just talking about why you can't trust in any 'storage related' reports found on the Internet especially when originating from SBC installations without knowing all the details for the simple two reasons:

* an awful lot of times users are suffering from hardware issues in reality when dealing with storage on SBCs
* an awful lot of these users either don't know the real issue or ignore it and report the drama they run into as a software issue related to whatever they remember they changed prior to occurence (new kernel, new filesystem, whatever... simply the last software change they remember or recognize will be blamed for the failure totally ignoring that hardware can be crappy and wears out over time)

BTW: ext4 and XFS vs. the others viewed from a robustness perspective: https://news.ycombinator.com/item?id=11469535 or direct link https://events.static.linuxfound.org/si ... 2016_0.pdf -- slide 3 is the interesting one. With this fuzzing attempt btrfs is dead after 5 seconds while ext4 survives 2 hours. If you're only looking at filesystems from this perspective for whatever reasons you're condemned to use the more mature filesystems forever.

sousaplex
Posts: 20
Joined: Fri Sep 15, 2017 12:20 am
languages_spoken: english
ODROIDs: C2, C4
Has thanked: 2 times
Been thanked: 0
Contact:

Re: Any HC1/HC2 setup based on the new N1 hardware?

Post by sousaplex »

tkaiser wrote:
ASword wrote:Data losses with BTRFS: does anyone have any links to help quantify this a bit better?
I won't elaborate on btrfs here since totally irrelevant but on what dealing with storage in the SBC world means: dealing all the time with an insanely high and pretty frustrating amount of both reasons for failure at the hardware layer and ignorance.

When filesystem designers think about storage implementations then they base their decisions on reasonable looking stuff like
1) disks are powered reliably
2) disks are connected reliably (NO cable/contact problems between host and disk corrupting data on the wire)
3) host is powered reliably
4) host DOES NOT corrupt data on its own

The above are the basics all old storage attempts are based on. Since this describes an ideal world that does not exist some precautions have been added over the decades, e.g. ECC memory on the lowest hardware layer or on the lower protocol layers checksums are used to try to verify if data sent over a connection whether it arrives altered or not (CRC checks, part of every low-level protocol but only sometimes available to upper layers -- do a web search for 'SMART attribute 199' for example. If SATA cable/contact problems lead to data corruption between host and disk the disk controller will ask the host to retransmit data -- and vice versa -- and disks should add every event to this 'CRC mismatches' counter available through SMART attribute 199)

In recent years since computers are fast enough filesystem designers also tried to address the above concerns at higher layers. Modern storage attempts like ZFS, btrfs or ReFS for obvious reasons do not trust blindly any more in the data written out to storage arriving there unaltered or remaining intact indefinitely but they check for data integrity (that's the 'checksumming thing'). So if data corruption occured between host and disk or on the disk you're at least able to take notice. If you added the right amount of redundancy in the right way to your storage setup then with these modern approaches even self-healing will happen (of course not with stupid/anachronistic stuff like e.g. mdraid's RAID1!).

But still all these concepts rely on some of the above basics, most importantly that all components are powered appropriately, that a 0 in memory remains a 0 and a 1 remains a 1 and that the administrator of the whole setup takes care that components are interconnected (wired) in a sane way.

When we look at the situation in the SBC world all of the above simply sucks and the 'administrators' of these devices (the end users themselves) neither know nor care.

1) USB disks are powered by the USB ports of a SBC for example --> sh*t show as expected, countless reports that talk about 'UAS is always a problem' or other BS are simply related to the fact that undervoltage or undercurrent in such situations occurs and the disks are not able to operate as expected. Users look into dmesg output and blame some driver reporting the real issue happening layers below.

2) Reliable connectors between disk and host: on SBC still everywhere only the crappy variants are used: internal SATA instead of eSATA, USB3-A instead of USB-C and so on. An awful lot of reports talking about software issues (people looking at dmesg output) is just cable/connector crappiness and some driver layers above having to deal with this mess and spitting out misleading messages via dmesg

3) Host reliably powered? Even this can be a sh*t show as we've seen with all the Micro USB powered SBC around and even with ODROID XU4 so many times. Users are either not willing or able to understand Ohm's law, they look at amperage ratings of their PSU, read 6A and are happy while in reality crappy cables between board and PSU or a crappy PSU itself creates a nice voltage drop under load. Underpowered electronics components usually do not work as expected. Period.

4) Data corruption on the host (happening in memory: DRAM or internal caches). Fortunately not that much of an issue with ODROIDs so far except those rare occassions where ECC memory would prevent a bit flip that happens from time to time with non-ECC memory. But on other SBC this is sometimes a huge problem due to stupid settings. With those cheap Allwinner boards for example we face the situation that both board makers as well as mainline u-boot/linux device maintainers for whatever bizarre reasons on a regular basis use default values for cpufreq/dvfs and DRAM clockspeeds that are too high (or CPU supply voltages too low for a certain clockspeed). Which results in data corruption by design unless settings are adjusted to sane levels. In such a situation without ECC memory the host itself is able to destroy every filesystem you want to use on such a system!

So when you hear about any storage related report generated with any SBC that tries to blame any of the software layers involved better hold on first and think about whether the most basic requirements for storage to work appropriately are met or not (that's the hardware layer below no one is thinking and talking about). Usually these most basic requirements are NOT met and users reporting stuff don't even understand these basics (and to be honest: they should not have to understand these basics but board designers should try to prevent these issues in the first place! The hardware design should take care of such issues and not the user!)

Average users don't understand why powering through Micro USB is bad, they don't understand that looking at amperage ratings is useless since usually voltage drops are the real issue, they don't understand how fragile some connectors are and that a dozen of matings can already destroy some of these connectors leading to all sorts of troubles appearing in dmesg then being interpreted as software issues and spread through the Internet as such...

So now let's have a look how Hardkernel addressed and addresses these issues:

* Fortunately most of their boards are not able to be powered through Micro USB which is a huge plus since it prevents users from connecting crappy phone chargers and using average USB cables with power wires too tiny. But powering through 5V is always problematic. It's the voltage drop issue not a single user wants to understand since focussed on amperage ratings only: https://forum.armbian.com/topic/6171-od ... a-and-sdb/

* So far the only 2 ODROIDs with a really sane powering scheme with storage use cases in mind are the HC2 and the N1. Both use an input voltage well above 5V (wide range input voltage but when using 3.5" disks it has to be 12V) and generate from this variable input voltage stable 5V the connected disks will be fed with.

* Sane dvfs and DRAM clockspeed settings have been developed for every ODROID so far. Only potential improvement: using ECC DRAM (not possible with any of the SoCs chosen for any available ODROID) and downclocking DRAM if you're concered about data integrity in a slightly paranoid way.

* Reliable interconnections between host and disk? Solved only on HC1 and HC2 by eliminating USB3 interconnection troubles but this is still a huge problem on the other boards since the appropriate connectors that reliably connect host and disks and are able to survive thousands of matings would make the whole setup a lot more expensive. That's why we see USB3-A receptacles (bad since awful tiny contacts for the SuperSpeed data lines -- USB-C is at least hundred times more reliable here) and internal SATA connectors instead of eSATA (maximum matings by specs: 50 vs. 5,000)

In other words: Hardkernel performs pretty good in this area and even improved with their latest 2 boards (HC2 and N1 implementing 'input voltage above 5V'). But since the type of connectors/cables affecting host <--> disk connections unfortunately is not the optimal one there's still room for improvements. But this would require educating users about the types of problems they might run into. End users need to know that USB3-A connectors are prone to contact problems, same with internal SATA connectors (though @odroid said to replace those on N1 dev samples with better ones on the production batches), same with the eMMC connector for example: users rip out the eMMC module 50 times to flash eMMC with the SD card adapter, then re-insert the module back in the board. As expected the connector starts to fail after some time and all sorts of weird things happen (looking like software issues of course). The last thing the user remembers he did differently compared to the past was switching from a 3.x kernel to 4.x and boom... there's a new thread about 'switching to mainline kernel bricked my board' in the forum. This can only be solved by educating users why such hardware issues matter.

IMO with the situation on N1 in mind they should also elaborate on why not every SATA device is a great choice. There exist pretty crappy SSDs and even fake ones in the meantime. Users believing into 'SATA is always great' and following the 'always buy as cheap as possible' principle will run into the same issues they run today when buying crappy or fake SD cards: insanely low performance and data losses.

Back to 'reports about $filesystem failing or responsible for data losses'. You find reports on the Internet blaming everything possible and if conclusions look easy most further readers will believe it. Better always question whether the prerequisits for storage to work properly were met or not. If it happened with SBC do not simply trust in. Compared to a PC or server so much more can go wrong and goes wrong with storage and SBC that there's simply no reason to believe into any report prior to knowing all the details.
This may be my favourite post on this forum.

Post Reply

Return to “General Chat”

Who is online

Users browsing this forum: No registered users and 1 guest