ODROID HC1 Distributed Redundant Home Cloud

Moderators: odroid, mdrjr

ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Thu Mar 22, 2018 5:02 pm

Hi Everybody!

I recently became the happy owner of 8 ODROID HC1 units, each fitted with a 1TB HDD. I would like to run OwnCloud or NextCloud or some other similar software on them, which would allow me to have my own cloud at home, but not just in any fashion, I'd like the cloud storage to be automatically distributed and managed over the 8 ODROIDs and I'd like it to be redundant for the sake of data safety. For example it would be nice to use the 8 1TB HDDs as a 4 TB storage with all data being stored on two drives at all times.

I have very little knowledge about cloud storage, network storage and networking in general, that is why I'd like to request some guidelines about the following:

1. How can I achieve the automatically distributed redundant storage? Does software like OwnCloud, NextCloud, etc. do this automatically or do I need an underlying distributed file system? In either case, please advice which software and/or file system would best suit my needs.

2. I don't really know how to connect the 8 ODROIDs to my LAN. At fist I was thinking to just stick a USB WiFi stick into each, but perhaps that's not the best idea from a network performance point of view, especially because HC1 only has USB 2.0. So what should I do? My router only has 4 Ethernet ports and only 2 of those are available, but I need 8... Also, I would like the ODROID cluster to be easily moved from one room to the next, which means that even if the 8 units are somehow interconnected by cable, the whole cluster should connect to the LAN through WiFi. How would you connect the units together and into the LAN?

I've searched this forum and the internet in general, but people are talking very high-level about these things. I need an explanation broken down to n00b level.

Thank you in advance for any useful info you may be able to share!
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby odroid » Thu Mar 22, 2018 5:44 pm

Refer the GlusterFS articles first to build a distributed storage.
https://magazine.odroid.com/article/exp ... ver-setup/
https://magazine.odroid.com/article/exp ... rformance/

You can find NextCloud configuration in this wiki page.
https://wiki.odroid.com/odroid-xu4/soft ... s_services

We didn't try to use any wireless network to build a cluster computer due to the unstable connection as well as slow speed.
These Gbit Ethernet switches were very affordable devices for our internal cluster computing test.
https://www.bestbuy.com/site/netgear-pr ... Id=8363086
https://www.amazon.com/D-Link-16-Port-G ... B0092KZBCQ
User avatar
odroid
Site Admin
 
Posts: 27959
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Thu Mar 22, 2018 5:52 pm

Thank you!
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Wed Apr 11, 2018 2:14 pm

@odroid you mentioned that you built a cluster test system. What software did you use with it?

I'm wondering about trying OpenMediaVault with OwnCloud plugin, but I'd prefer to do it in a RAID configuration, however the HC1 only has one SATA connector. Can I attach a SATA cable with multiple connectors to the ODROID HC1 and use that cable to attach multiple HDDs to the ODROID HC1?
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby hominoid » Wed Apr 11, 2018 10:55 pm

Un4Seen wrote:...but I'd prefer to do it in a RAID configuration, however the HC1 only has one SATA connector. Can I attach a SATA cable with multiple connectors to the ODROID HC1 and use that cable to attach multiple HDDs to the ODROID HC1?

No, each disk requires it's own SATA interface and connector, RAID will not be possible. Like @odroid said, take a look at GlusterFS, It's a nice approach for your configuration.
hominoid
 
Posts: 164
Joined: Tue Feb 28, 2017 3:55 am
Location: Lake Superior Basin, USA
languages_spoken: english
ODROIDs: C2, XU4, MC1, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Thu Apr 12, 2018 3:51 am

GlusterFS is really nice, but people say that OpenMediaVault has no support for it :(
I did search the internet to see how I could get the two to work together but I've found absolutely nothing :(
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby hominoid » Thu Apr 12, 2018 6:54 am

Un4Seen wrote:GlusterFS is really nice, but people say that OpenMediaVault has no support for it :(
I did search the internet to see how I could get the two to work together but I've found absolutely nothing :(

I have not used GlusterFS yet and I don't use OpenMediaVault but, they might have been implying native support. Both support NFS and Samba so it might be possible to NFS/Samba mount the GlusterFS to OpenMediaValualt and use it that way depending on what/how your going to use OMV. My suggestion would be to setup a small test system and try it. It would give you some further insight on how both worked and which would be best for you if you have to choose one. I don't see anything that OMV offers that couldn't be replicated with individual apps. It seems just to be any easy all-in-one solution. Plus, I have heard some complaints about various issues with it. If a distributed redundent system is what your looking for primarily, GlusterFS with additional apps might be better way to go. There are some threads on OMV in this sub-forum if you haven't explored them already.

Edit:
Here as well viewtopic.php?f=98&t=14907
hominoid
 
Posts: 164
Joined: Tue Feb 28, 2017 3:55 am
Location: Lake Superior Basin, USA
languages_spoken: english
ODROIDs: C2, XU4, MC1, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Thu Apr 12, 2018 3:10 pm

Thank you, @hominoid!
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Fri Apr 20, 2018 6:22 pm

odroid wrote:Refer the GlusterFS articles...

which are missing the 2nd most important information :twisted: ; recovery.
With the understanding that GlusterFS should not be used for redundancy or backup because it requires low latency, and if your nodes are in the same building they are not redundant or backed up.
Also GlusterFS also does not scale well yet so use the more popular Ceph or Lustre if you want to scale.
There are a lot of synchronous options, asynchronous solutions like git-annex and Syncthing require conflict resolution from time to time which is why they are less popular.
Log shipping to read only mirrors are the more common solutions for optimal asynchronous backups commonly used with mariadb or Btrfs. rsync is a 2nd rate solution but old and stable.
Last edited by elatllat on Fri Apr 20, 2018 9:36 pm, edited 8 times in total.
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Fri Apr 20, 2018 6:58 pm

I've put together a system in which I have 8 ODROID HC1 utnits, each fitted with a 1TB HDD. I connect the drives using GlusterFs in a 4x2 distributed, redundant system (meaning that each file is duplicated over 2 disks and that the files are distributed among 4 disks, 4x2=8).

I use a separate ODROID C2 to munt the GlusterFS and to use it. On the C2 I run Samba, NextCloud, FTP server, qBittorrent, etc.

It works, but I have several problems:

1. I can only access the files and folders inside the GlusterFS mount point folder with the linux user who owns the mount point on the ODROID C2. The www-data user (which is default for Apache), cannot access the GlusterFS mount, no matter how I change the mount point's permissions. I tried adding www-data to the group that owns the mount point, I tried adding 777 rights to the mount point, nothing works. I was able to access the GlusterFS only by forcing Apache to run under the user which owns the GlusterFS mountpoint.

2. The GlusterFS is soooo slow. 20 MB/s read, 30 MB/s write speed over Samba from a Windows 10 PC. The Windows PC, the ODROID C2 and the 8 ODROID HC1 units are all connected to the same Gigabit switch via ethernet cable.

3. I can't put the hard disks into idle mode (I can't spin them down). I tried to do it with hdparm and the disks started spinning up, down, up, down infinitely after a while.
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Fri Apr 20, 2018 7:30 pm

Un4Seen wrote:...1...2...3


1. likely fix with a mount option.

2. speed is likely limited by your slowest disk [edit]I mean network[/edit] with your current software config.

3. idle is likely not achievable with synchronous file systems
Last edited by elatllat on Fri Apr 20, 2018 10:00 pm, edited 1 time in total.
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Fri Apr 20, 2018 7:57 pm

1. Yes... but what mount option? I've spent at least a total of 12 hours already searching the internet for this answer and I've tried a "million" things, nothing worked.
2. I have 8 totally identical disks, WD Black, 1TB, 7200 rpm, 32MB cache. They are supposed to be high-performance hard drives with read and write speeds well above 100 MB/s...
3. Yes, I have a suspicion that I might not be able to spin the hard drives down because of GlusterFS. But in theory GlusterFS knows if there are any pending tasks and if there aren't any read/write tasks pending, it could allow the drives to spin down. Theoretically... Depends how it's implemented.
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Fri Apr 20, 2018 9:55 pm

Ideally my digital whims are managed by machine intelligence but practically you should be able to max out your ethernet 125 MB/s (B=b/8) for the first 1 GB (node free RAM) then 62 MB/s (1/2 1Gbps) for the next 7 GB (cluster free RAM) then whatever your disk speed is after that (might be another level if you use lvmcache or the like).

30 MB/s sounds real close to half 62 MB/s so likely it's sending to your nodes in parallel... test some different configurations and alternative software, but keep in mind the only way to get sustained 125 MB/s is to use a distributed file system client (not Samba).
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Fri Apr 20, 2018 9:57 pm

I think I might have found the answer for problem #2:

I ran this command on all 8 HDDs locally to test read speed:
Code: Select all
sudo hdparm -Tt /dev/sda


On disks 2-8 I get results like this:
Code: Select all
/dev/sda:
 Timing cached reads:   1974 MB in  2.00 seconds = 988.26 MB/sec
 Timing buffered disk reads: 350 MB in  3.01 seconds = 116.39 MB/sec


But on disk 1 I get read speeds which are very very small:
Code: Select all
/dev/sda:
 Timing cached reads:   1982 MB in  2.00 seconds = 992.10 MB/sec
 Timing buffered disk reads:  30 MB in 75.86 seconds = 404.93 kB/sec


Now I have no idea what the difference is between cached read and buffered read, but the fact that the buffered read is so much smaller on one of the disks worries me.
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Fri Apr 20, 2018 10:03 pm

you unmounted the disk first, and have plenty of free RAM?
Last edited by elatllat on Fri Apr 20, 2018 10:13 pm, edited 1 time in total.
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Fri Apr 20, 2018 10:10 pm

I also tested write speed with:
Code: Select all
dd if=/dev/zero of=/gfs/brick1/output bs=8k count=10k; rm -f /gfs/brick1/output


and got equally high 100+ MB/s write speeds on all disks:
Code: Select all
10240+0 records in
10240+0 records out
83886080 bytes (84 MB, 80 MiB) copied, 0.51154 s, 164 MB/s


I did not unmount the disks and did not check the free RAM, but I'll do it now.
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Fri Apr 20, 2018 10:17 pm

I've now run the read speed test with unmounted hard disk. On all my HC1 units I have about 150-200MB free memory out of 2GB.
I still get about 1MB/s max buffered read speed on the first unit and over 100 MB/s buffered read speed on the other 7 units.
Strangely the write speed is good (100+ MB/s) on all 8 hard drives...
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby tkaiser » Fri Apr 20, 2018 10:39 pm

Un4Seen wrote:Strangely the write speed is good (100+ MB/s) on all 8 hard drives...


dd and hdparm are no benchmark programs since people usually use them wrongly (with dd you missed a sync call) and also draw the wrong conclusions. I would use a benchmark tool instead (e.g. iozone) and check SMART attribute 199 of the slow disk.
tkaiser
 
Posts: 354
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Fri Apr 20, 2018 10:51 pm

tkaiser wrote:...check SMART attribute 199 of the slow disk.

yah SMART was my next recommendation to.
Mine look like this;
Code: Select all
XU4> smartctl -a /dev/sda | grep RAW_VALUE
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
XU4> smartctl -a /dev/sda | grep 199
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
XU4> smartctl -a /dev/sdb | grep 199
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
XU4> smartctl -a /dev/sdc | grep 199
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Fri Apr 20, 2018 11:22 pm

Thank you! Will check. Is iozone a cmdline tool? I have no poss8bility for GUI.
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Fri Apr 20, 2018 11:27 pm

yes; example
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Sat Apr 21, 2018 3:47 am

I've checked the smart attribute 199 both on the slow drive and on the good drives and I get the same output:

Code: Select all
root@ODROID-HC1-01:~# smartctl -d sat -a /dev/sda | grep RAW_VALUE
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
root@ODROID-HC1-01:~# smartctl -d sat -a /dev/sda | grep 199
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
root@ODROID-HC1-01:~#


I guess this means that the drive seems to be fine... at least form this point of view...

But there's definitely something terribly wrong with one of the drives. I copied 1GB of data (in about 10 files) to the GlusterFS cluster over Samba from Windows, monitoring the write speed. It was just above 30 MB/s on average. The source was an SSD on Windows 10, so it should not affect the speed. Then I tried to copy them back from the GlusterFS to Windows. The read speed was on average 30 MB/s until it got to a file which was stored on the problematic disk (and its pair - because I have redundant storage: 4x2 disks). The read speed dropped quickly to 0 and after some seconds it failed completely. I tried again, it failed again. I know for sure that it stopped reading when it had to read from the problematic drive because the green LED was blinking only on that one drive, the 7 other drives had solid green LED.

This raises some important alarms. First of all, what is wrong with that disk? And why didn't it just get the data from the redundant pair, which seems to work fine?
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Sat Apr 21, 2018 4:00 am

What's the rest of the smart data say for that disk?
Some drives need a firmware update.
RMA it quickly.
I hope you used encryption so you don't have to attempt to zero the drive.
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Sat Apr 21, 2018 4:17 am

Wow! I just came to a huge realization!

I bought these drives together with the ODROID-HC1 units second hand from the other side of the globe. So returning it is out of the question. It was a bargain (truly cheap, if all drives fail, it was still worth it for the HC1s), so I'm not complaining if one fails. The thing is all 8 drives are identical model, but the 7 which behave well were produced in april/may 2017 (just a few days apart), while the one which has problems was made in october 2016. Unsurprisingly exactly this "bad" drive is older. Maybe it just needs a firmware upgrade? Or maybe it got used up? It's not that old though, just 1.5 years old.

I shut down the problematic node from the GlusterFS and the read succeeded. But it still worries me that there's no automatic failover to the redundant pair when one node fails.

Here's the complete smart data:
Code: Select all
root@ODROID-HC1-01:~# smartctl -d sat -a /dev/sda
smartctl 6.5 2016-05-07 r4318 [armv7l-linux-4.14.29+] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     WDC WD10JPLX-00MBPT0
Serial Number:    JR1004D31BXX0M
LU WWN Device Id: 5 000cca 8c8d38416
Firmware Version: 01.01H01
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      2.5 inches
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Apr 20 22:16:47 2018 EEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART Status not supported: Incomplete response, ATA output registers missing
SMART overall-health self-assessment test result: PASSED
Warning: This result is based on an Attribute check.

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (   45) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 151) minutes.
SCT capabilities:              (0x0035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   062    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   040    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   137   137   033    Pre-fail  Always       -       2
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       603
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   040    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   090   090   000    Old_age   Always       -       4456
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       126
191 G-Sense_Error_Rate      0x000a   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       55
193 Load_Cycle_Count        0x0012   059   059   000    Old_age   Always       -       411889
194 Temperature_Celsius     0x0002   240   240   000    Old_age   Always       -       25 (Min/Max 20/48)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
223 Load_Retry_Count        0x000a   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Sat Apr 21, 2018 5:40 am

I'm trying to install iozone to benchmark the HDDs. There is supposed to be a package called "iozone3" but when I try to apt-get install it (yes, I did apt-get update), it can't find it:
Code: Select all
E: Unable to locate package iozone3


To this day I don't really understand how apt-get repositories work, but here is my /etc/apt/sources.list:
Code: Select all
deb http://ports.ubuntu.com/ubuntu-ports/ xenial main universe restricted
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial main universe restricted

deb http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main universe restricted
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-updates main universe restricted

deb http://ports.ubuntu.com/ubuntu-ports/ xenial-backports main restricted
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-backports main restricted

deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security main restricted
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security universe
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security universe
deb http://ports.ubuntu.com/ubuntu-ports/ xenial-security multiverse
deb-src http://ports.ubuntu.com/ubuntu-ports/ xenial-security multiverse


Any idea why it can't find iozone?
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby rooted » Sat Apr 21, 2018 8:22 am

apt search iozone
User avatar
rooted
 
Posts: 4920
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Sat Apr 21, 2018 9:23 am

Yeah, I tried that... it finds nothing :(
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby odroid » Sat Apr 21, 2018 10:28 am

You had to add "multiverse" ppa to install iozone.
User avatar
odroid
Site Admin
 
Posts: 27959
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby tkaiser » Sat Apr 21, 2018 4:55 pm

Un4Seen wrote:I shut down the problematic node from the GlusterFS and the read succeeded. But it still worries me that there's no automatic failover to the redundant pair when one node fails.


Well, that's what you get when you combine toys with IT attempts made for server grade hardware. That's why playing HA or RAID with single board computers is almost always just stupid. Adding tons of unreliable components to increase reliability often simply does not work as you were able to figure out yourself.

HC1 and HC2 are really great devices for single disk NAS operation (adding an USB2 backup disk is fine too). Everything else is just fooling yourself.
tkaiser
 
Posts: 354
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby elatllat » Sat Apr 21, 2018 9:05 pm

tkaiser wrote:...Adding tons of unreliable components to increase reliability...

Works excellently for properly engineered distributed systems like torrent, dns, or with script assistance for dnf/pacman, git, luster, ceph, etc.

One solution is to make your own management script.

Another solution is to use one of the more popular alternatives mentioned above.

Another potential solution is to adjust relevant settings like but not limited to
Code: Select all
gluster volume set shared network.ping-timeout 1

And this;
Code: Select all
storage.health-check-interval 10
elatllat
 
Posts: 971
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Sun Apr 22, 2018 5:05 pm

@tkaiser, I'm familiar with your perspective on ODROID NAS setups, that these are "toys" as you call them and unreliable. I've read this opinion of yours on other forums too. And you may be right, but you never explained why you think these devices can't work reliably in setups like mine. I mean GlusterFS is just a software layer, it doesn't really care what the underlying hardware is. So what is it about ODROIDs that make them unfit for GlusterFS?
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Mon Apr 23, 2018 6:13 am

In the meantime the "bad" hard drive started behaving well. I don't know if it's because I did a heal on the Gluster volume (which should not have anything to do with non-gluster operations) or because I moved the hardware around on my desk (which should again not affect it), but now the "sudo hdparm -Tt /dev/sda" command gives me read speeds of around 130 MB/s, just like on the other 7 drives.

I finally got the results from iozone too.

On the "bad" hard drive:
Code: Select all
Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     1624     2898     9566     9534     7461      977
          102400      16     6970     5902    26193    26204    19053     3352
          102400     512    12532    14461    69635    69677    67641     2011
          102400    1024    30076    75583    71646    71674    70693     9367
          102400   16384    27357    72452    79105    79093    78995    72328


And on another ("good") hard drive (identical model):
Code: Select all
Command line used: iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2
        Output is in kBytes/sec
        Time Resolution = 0.000001 seconds.
        Processor cache size set to 1024 kBytes.
        Processor cache line size set to 32 bytes.
        File stride size set to 17 * record size.
                                                              random    random     bkwd    record    stride
              kB  reclen    write  rewrite    read    reread    read     write     read   rewrite      read   fwrite frewrite    fread  freread
          102400       4     3484     3651    10269    10217     8928     2959
          102400      16     4202     8462    27529    27481    25269     6908
          102400     512    12037    16102    69740    69751    69312    14910
          102400    1024    16782    15279    71522    71547    71345    16099
          102400   16384    15905    14507    79110    79121    79084    18749


I'm not really sure what to make of the results... first of all it seems that the "good" drive is actually suspiciously slow at write (< 20 MB/s) and also both drives are below 80 MB/s both at read and write, which is again strange...
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby tkaiser » Mon Apr 23, 2018 6:28 pm

Un4Seen wrote:I finally got the results from iozone too.


Is this with GlusterFS in place? At least all numbers are meaningless since both too high (random IO) and too low (sequential IO). Such an iozone test is done best on an ext4 since this filesystem honors the -I flag.

Wrt toys vs. server grade it's just that. Some IT concepts deal good with totally unreliable hardware but some concepts require stuff to be working properly (and GlusterFS is one of the latter examples. Also it's weird to do this without ECC memory and at least 10GbE networking)
tkaiser
 
Posts: 354
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Mon Apr 23, 2018 8:38 pm

The iozone results were executed on each ODROID-HC1 individually, on the locally mounted hard drives. File system type is XFS. GlusterFS was running on top of it. I could turn off the GlusterFS volume while testing, I didn't think it would influence anything since it was not processing any read/write requests...

My current suspicion is that I only get 30-40 MB/s read-write speeds with GlusterFS over Samba because the testing Windows machine, the GlusterFS client (ODROID C2) and the 8 GlusterFS nodes (ODROID HC1 units) are all connected to the same Gigabit switch via cable and probably the switch can't keep up. I'll test this by turning off every second GlusterFS node to see if it's faster when it's communicating with fewer machines... although trying to reach the turned off nodes could still influence the performance of the switch...
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby tkaiser » Mon Apr 23, 2018 9:27 pm

You're still ignoring the first answer you got in this thread by @odroid. Low performance is expected and with your setup you do not even have any real benefits from the ressources wasted on implementing redundancy. It's just trashed performance at increased costs.

2 HC2 with each a large 3.5" disk that implement proper backup (I would do below the filesystem layer using btrfs send|receive) will at least double performance and provide at least data protection.

I was involved in a GlusterFS evaluation some years ago with an IBM reseller in germany. The whole project was only targeted at ripping off customers not solving any of their real problems. I told them not wanting to be part of a problem since I prefer providing solutions and at our customers we went then with SuperMicro Xeon boxes with maximum RAM, ZFS, a bunch of cheap 3.5" SAS disks in a smart topology as dual node cluster with one off-site ZFS host receiving snapshots for desaster recovery purposes. Way higher performance at a fraction of costs.
tkaiser
 
Posts: 354
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Tue Apr 24, 2018 3:59 am

I think by "slow speed" odroid was referring to a Wifi-based setup (as opposed to a cable wired setup), I don't see where he stated that performance would be mediocre.
Cost is not a factor here. This is the hardware that I got and I don't have anything else, so this is what I have to make the most of... unless I try to sell the hard disks and buy two larger ones instead...

Anyway, do you remember that one out of 8 hard disks was behaving strangely last week and after a while it magically started working well again? Today I moved the ODROIDs (and the attached hard drives) again on my desk. This is what I did last week before it started working. And now it started not working again. Perhaps that very slight shock as I moved it... maybe it's a connection problem somewhere on hardware level... it's strange...
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Tue Apr 24, 2018 6:32 am

And now the problematic disk is working well again. This time I did not touch anything physically and made no software changes. I may have rebooted the ODROIDs but that should make no difference since I tried that before and it did not fix the problem. I get the feeling that this 1 disk out of 8 has either a connection problem (inside or outside) or is giving me warnings that it's preparing to die. I should probably swap the disks between two odroids to see if it's the disk's fault or the odroid's fault...
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Wed Apr 25, 2018 2:28 pm

@odroid You know what would be awesome? If you made variants of the HC1 and HC2 with 2 SATA ports. That way a single ODROID-HC1/HC2 could be used to attach two large hard disks used in a redundant RAID. That would make for a truly nice NAS! Currently you can only attach one disk and if that disk dies, you say bye-bye to your data...
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby rooted » Wed Apr 25, 2018 2:36 pm

Un4Seen wrote:@odroid You know what would be awesome? If you made variants of the HC1 and HC2 with 2 SATA ports. That way a single ODROID-HC1/HC2 could be used to attach two large hard disks used in a redundant RAID. That would make for a truly nice NAS! Currently you can only attach one disk and if that disk dies, you say bye-bye to your data...
This is what the cloudshell 2 is for.
User avatar
rooted
 
Posts: 4920
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Wed Apr 25, 2018 2:59 pm

Wow! I actually saw that earlier, but the description on the ODROID site is not really understandable and detailed and I did not understand what it was, I thought it's just a plastic box with a front display and a fancy name...

What is it really? Some kind of box with a display into which you somehow plug an ODROID XU4 and some hard disks?
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby rooted » Wed Apr 25, 2018 9:24 pm

Un4Seen wrote:What is it really? Some kind of box with a display into which you somehow plug an ODROID XU4 and some hard disks?


Yes, you plug an XU4 into it. It has two SATA connectors with a hardware raid controller, the display can be used for statistics and such.
User avatar
rooted
 
Posts: 4920
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Wed Apr 25, 2018 9:30 pm

Yepp, I checked it out. It's exactly what I need! Thank you for pointing it out!
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby rooted » Wed Apr 25, 2018 10:51 pm

You're welcome, it's actually a nice device for the price.
User avatar
rooted
 
Posts: 4920
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Wed Apr 25, 2018 11:21 pm

And it's got that geeky cool feeling about it, which is awesome!
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby rooted » Thu Apr 26, 2018 2:41 am

The clear looks super cool but if you are going to have it in a visible place at night you will want to disable the heartbeat and lcd lights (you can on command).

Personally I would go with the smoked version given the choice again.
User avatar
rooted
 
Posts: 4920
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Thu Apr 26, 2018 3:27 am

Yeah, I was just thinking about that... the choice between clear and smoked blue. Personally I would prefer smoked black, but that doesn't exist. I'm not so fond of blue, so it will probably be the clear one. And it doesn't matter what it does at night, because I don't put it into the bedroom :) It will be in the living room where there are already more than 10 ARM devices and they all blink a lot :)
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby rooted » Thu Apr 26, 2018 4:30 am

In person the smoked blue really looks more like steel grey, at least on the Cloudshell 1 I have.
User avatar
rooted
 
Posts: 4920
Joined: Fri Dec 19, 2014 9:12 am
Location: Gulf of Mexico, US
languages_spoken: english
ODROIDs: C1, C1+, C2
XU3 Lite, XU4
N1
VU7+
HiFi Shield 2
Smart Power (original)

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby tkaiser » Thu Apr 26, 2018 4:49 pm

Un4Seen wrote:@odroid You know what would be awesome? If you made variants of the HC1 and HC2 with 2 SATA ports. That way a single ODROID-HC1/HC2 could be used to attach two large hard disks used in a redundant RAID. That would make for a truly nice NAS! Currently you can only attach one disk and if that disk dies, you say bye-bye to your data...


Nope. RAID is not backup. It's 2018 and we still have to write this sentence over and over again.

RAID especially RAID-1 is close to useless if you want data protection since RAID is ONLY about availability. Do backup if you need data protection. Choose filesystems implementing checksumming to get data integrity. Combine both in a reasonable way (e.g. btrfs send|receive sending snapshots not to another disk attached to the local device but to a different one in another room or even building) then you end up with something that protects your data.

RAID-1 is BS for this purpose. Simply try it out using rm -rf / or setting your board on fire. RAID is not backup!

Ah, already forgot: RAID is not backup!!

And BTW: RAID is not backup!!!

The Cloudshell 2 has/had its own problems (USB3/SuperSpeed connection issues in many cases and firmware issues in the beginning) so while RAID-1 is still one of the most bizarre wastes of disks something using the JMS561 directly attached to SuperSpeed lines on a PCB with appropriate firmware flashed would solve at least these two problems and would make accessing 2 disks attached to one USB3 port more convenient.

And HC1 and HC2 have both an USB2 port that is designed to attach your backup disk to if you store important data on the device (many people use it just to backup other stuff so no additional protection layer needed)
tkaiser
 
Posts: 354
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby stmicro » Thu Apr 26, 2018 5:28 pm

Our company has used several CloudShell2 units over a half year for internal Samba server as well as our customers' FTP & ownCloud storage. All of them has two hdds with RAID-1 configuration with HK's official Ubuntu minimal image on quite stable Kernel 4.9.
One of them had a serious data corruption problem a couple of months ago. But it was easily recovered with RAID-setup-key on the CS2. https://wiki.odroid.com/accessory/add-o ... cover_data
So I would like to say the RAID-1 should be a backup solution at least for me.

But, I agree two HC2 units must give us more reliable backup solution with incremental snapshots remote backup via rsync stuff.
stmicro
 
Posts: 225
Joined: Tue Apr 28, 2015 4:23 pm
Location: shenzhen china
languages_spoken: english, chinese
ODROIDs: Many Odroids and Rpis.

Re: ODROID HC1 Distributed Redundant Home Cloud

Unread postby Un4Seen » Thu Apr 26, 2018 6:18 pm

@tkaiser Ok, RAID is not backup... not a complete backup at least. Yes, if you delete your files, they will be deleted from both drives, yes if an asteroid falls on your house, you loose your data. But I'm not trying to prepare against stupidity (deleting your files) or against the impossible (asteroids, aliens, nuclear war). I'm trying to prepare against the probable, which is hard disk failure. RAID 1 protects against that, doesn't it?
Un4Seen
 
Posts: 141
Joined: Thu Apr 10, 2014 6:16 pm
Location: Romania
languages_spoken: english, hungarian, romanian

Next

Return to Projects

Who is online

Users browsing this forum: No registered users and 1 guest