ODROID-H2 Micro Cluster

Post Reply
rlefever
Posts: 1
Joined: Sun Nov 10, 2019 12:14 am
languages_spoken: english
ODROIDs: 0
Has thanked: 0
Been thanked: 3 times
Contact:

ODROID-H2 Micro Cluster

Unread post by rlefever » Wed Jan 22, 2020 2:16 pm

We here at DisComCo Labo are setting up our first Odroid-H2 based microcluster prototype, running K3S for container orchestration and so far, everything is going well.
This beast will consist of 1 Master node (ODROID-H2, 8GB RAM) and 3 Workers (32GB RAM, 2x1TB SATA drives each), so we hope to be running a 96GB RAM, 6TB K3S cluster soon. The experiences gained in this prototype will serve as a basis for our other project: macula.io; a PaaDS (Platform as a Distributed Service) product, in which we will try to create a mesh of similar micro-cluster; eliminating the need for a centralized party.
However there is one bottleneck: the availability of the base SBC...it is so succesful, that it is constantly out of stock!
Best regards

Labo@DisComCo
Attachments
ODROIDH2Cluster001.jpeg
ODROIDH2Cluster001.jpeg (120.28 KiB) Viewed 866 times
These users thanked the author rlefever for the post (total 3):
odroid (Wed Jan 22, 2020 3:25 pm) • rvalle (Wed Jan 22, 2020 4:05 pm) • rwxrwxrwx (Wed Feb 05, 2020 7:00 am)

User avatar
mad_ady
Posts: 7381
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 359 times
Been thanked: 258 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by mad_ady » Wed Jan 22, 2020 3:36 pm

Well, intel has part of the blame for low availability of the cpu...

fvolk
Posts: 330
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, HC1, H2
Has thanked: 0
Been thanked: 17 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by fvolk » Thu Jan 23, 2020 5:49 am

Show the cluster toys thread?
A H2 mini cluster for distributed processing testing:

* One H2 with NVME SSD and more ram (16Gb) as main node (not on picture).
* Multiple diskless H2 worker nodes:
boot via PXE kernel from main node,
mount root fs ro via NFS from main node,
on top runtime rw overlay in ram,
after boot each worker joins work scheduler at main node.
--> all workers in clusters run identical software.
* one HC1 as additional NFS storage

Pro:
* low-power(!), silent(!), put-on-your-desk size
* one worker node: H2 + 8Gb --> ~150 EUR/Node
* one spare 19V notebook power supply split between 3 nodes on picture
* x86 compatible, software "just works"

Con:
* not the fasted
* 5.5x2.1mm power plug of H2 - notebook power supplies usually(?) have 5.5x2.5mm -> need adapter, makes power lines brittle :-/
* software compiled for AVX does not work

upgrade options:
* use one network port per worker for "main" internet traffic and other port solely for worker<-->worker traffic
* Docker images do not work over network filesystem: buy for every worker a cheap used 256Gb NVME as Docker images storage (root fs stays on NFS)
minilab1.jpg
minilab1.jpg (36.14 KiB) Viewed 798 times
minilab2.jpg
minilab2.jpg (48.46 KiB) Viewed 798 times
minilab3.png
minilab3.png (50.53 KiB) Viewed 798 times
These users thanked the author fvolk for the post (total 2):
odroid (Thu Jan 23, 2020 9:26 am) • Mullcom (Tue Feb 11, 2020 7:14 am)

User avatar
Mullcom
Posts: 135
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 73 times
Been thanked: 10 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by Mullcom » Tue Feb 11, 2020 7:17 am

fvolk wrote:Show the cluster toys thread?
A H2 mini cluster for distributed processing testing:

* One H2 with NVME SSD and more ram (16Gb) as main node (not on picture).
* Multiple diskless H2 worker nodes:
boot via PXE kernel from main node,
mount root fs ro via NFS from main node,
on top runtime rw overlay in ram,
after boot each worker joins work scheduler at main node.
--> all workers in clusters run identical software.
* one HC1 as additional NFS storage

Pro:
* low-power(!), silent(!), put-on-your-desk size
* one worker node: H2 + 8Gb --> ~150 EUR/Node
* one spare 19V notebook power supply split between 3 nodes on picture
* x86 compatible, software "just works"

Con:
* not the fasted
* 5.5x2.1mm power plug of H2 - notebook power supplies usually(?) have 5.5x2.5mm -> need adapter, makes power lines brittle :-/
* software compiled for AVX does not work

upgrade options:
* use one network port per worker for "main" internet traffic and other port solely for worker<-->worker traffic
* Docker images do not work over network filesystem: buy for every worker a cheap used 256Gb NVME as Docker images storage (root fs stays on NFS)
minilab1.jpg
minilab2.jpg
minilab3.png
What software are you using?
Have you any information about how to set up something like this for a more beginner?
How well does it doing?
What are you using it for?

Thx for your post it looks great!


Skickat från min SM-G955F via Tapatalk


fvolk
Posts: 330
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, HC1, H2
Has thanked: 0
Been thanked: 17 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by fvolk » Thu Feb 13, 2020 6:36 am

Mullcom wrote:
Tue Feb 11, 2020 7:17 am
Have you any information about how to set up something like this for a more beginner?
What are you using it for?
I use it for development and testing of distributed code (e.g. Dask as in screenshot) and automation on a small but realistic scale.
Nowadays everyone runs multiple VMs or something "in the cloud" somewhere for this, but real stuff in front of you on the desk is real stuff :-)
I started with a Odroid MC1, but the fan noise is annoying and the ARM ports are always a bit late and sometimes with incompatibilities.

I'm afraid this is not a project for a Linux beginner, multiple pieces must work together and even for me with years of Linux experience it took several days of tinkering.
I thought about writing it down step by step while doing a new, clean install (as the current installation is a mess from all the tries), but, well, so far I found no time/motivation.
Also, my choices are probably not popular for others, e.g. no initramfs and a standalone kernel image, no systemd mess, etc.

Approximately:
1) Install Linux on H2.
2) Build and install your own custom Linux kernel.
3) Compile-in all required kernel modules so that you don't need kernel modules to mount and boot the root fs, just a standalone kernel image.
4) Compile-in all required kernel modules for NFS client support, so your root fs can be put on NFS.
5) Setup a NFS daemon on your server, move your root fs to the server, export it.
6) Figure out the kernel commandline to boot your root fs as provided via NFS on server.
7) Disable harddisk boot on client, boot from USB stick with just kernel image, and root fs on server via NFS.
8) On server, setup dnsmasq (or use your network DHCP whatever), configure for PXE boot and configure TFTP for your kernel image.
9) Configure H2 client to PXE boot in UEFI.
10) If this works for the generic rw rootfs case, reconfigure that NFS root provided by server is ro.
11) Replace default /sbin/init with a script that overlays a rw tmpfs in RAM over the ro NFS rootfs provided by the server, this becomes your new / on the client, and then continues normal /sbin/init boot process.

You can find for each of these steps guides on the internet, but putting them all together and make it work is non-trivial.
Good luck!

(and I don't claim this to be the best way, I also PXE+NFS booted the first time in my life, surely there are faster ways to set this whole thing up?)
These users thanked the author fvolk for the post (total 2):
Mullcom (Thu Feb 13, 2020 6:45 am) • a1right (Tue Feb 18, 2020 9:06 pm)

brad
Posts: 1055
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 27 times
Been thanked: 56 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by brad » Mon Feb 17, 2020 7:17 pm

I'm liking everyone's setups :) Here is my grand plan for a micro cluster in its infancy stages at the moment

I have been considering Odroids for creation of a mini home datacentre and currently in the cost analysis / planning stages to see if it would be viable. Being at home it needs to be quiet and low power consumption but reasonably powerful for compiling various software, building images, performing test cases and various CPU dependent tasks which can be distributed to multiple nodes. I don't want to be locked into proprietary solutions so all open source and it will be great for learning and enhancing my skills. I can save on my current cloud costs (storage is the biggest) by migrating existing services back to my own cluster.

H2 High availability SAN cluster over iSCSI (2 nodes)
===========================

- emmc's for boot disks
- at least 1 SSD drive for low level storage on each node using DRBD for block level replication ti the other node.
- 4 port gigabit network card connection to M.2 interface via pcie adaptor (6 ports in total with 2 x onboard).

- 2 or 3 NIC's multipathed to LAN network on each node for client block device access over iSCSI (or maybe just a bonded network)
- 2 or 3 NIC's bonded to a private network for replication between nodes
- Optional 3rd network for a heartbeat only connection.
- Active Active replication setup if possible

Any single node can be destroyed, powered off, rebooted, patched, etc with no loss of data and minimal or temporary loss of service during fail-over.

Worker nodes (Multiple nodes, mix of arm64 and x64 Odroid's for various workloads)
====================================================

- Hopefully H2's can directly boot kernel via PXE/ tftp (and then mount root filesystem over iSCSI) - do I need a bootdisk here to support PXE boot on N2 or can it be handled by BIOS and NIC's?
- C2''s could use small cheap sd cards for bootloader and mount rootfs over iSCSI
- N2's could use spi flash for boot and mount rootfs over iSCSI

Each worker node will probably have its own root filesystem in an individual LUN on SAN although I could use GFS2 for common root filesystem types. Shared or multiuser filesystems will probably handled by a N2 serving NFS and Samba in the work cluster (block storage here is iSCSI LUN or local disk). If I get keen and have GFS2 working I can add another N2 for Distributed node NFS/ Samba and use DLM or DVS for filesystem lock manager be it for NFS or samba.

Backup server with exported snapshots
=================================

- Probably URBackup as backup software (https://www.urbackup.org/)
- Probably in the cloud with compression & duplication enabled on the backup store. (although consider another N2 here using BTRFS or ZFS filesystem on SSD and usb connected HDD's to store duplicate snapshot copies and rotated in and out on regular basis)
- Backups taken regularly of worker and SAN clusters, hourly, daily, etc as per requirements.
- Snapshots of backup file system taken on regular intervals and regular snapshots exported to a 2nd cloud based service or back to me at offsite location.

That covers most of the ideas for the moment.

User avatar
Mullcom
Posts: 135
Joined: Sun Jun 30, 2019 1:41 am
languages_spoken: english
Has thanked: 73 times
Been thanked: 10 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by Mullcom » Mon Feb 17, 2020 8:06 pm

brad wrote:
Mon Feb 17, 2020 7:17 pm
I'm liking everyone's setups :) Here is my grand plan for a micro cluster in its infancy stages at the moment

I have been considering Odroids for creation of a mini home datacentre and currently in the cost analysis / planning stages to see if it would be viable. Being at home it needs to be quiet and low power consumption but reasonably powerful for compiling various software, building images, performing test cases and various CPU dependent tasks which can be distributed to multiple nodes. I don't want to be locked into proprietary solutions so all open source and it will be great for learning and enhancing my skills. I can save on my current cloud costs (storage is the biggest) by migrating existing services back to my own cluster.
Sounds Grate!

I am in to building a Server Rom my self. Looking at how to setup everything to be ballaced and secure if some hardware or net goes down. right now i am having h2 as a router but this might change in future.

i have done some dokumentation.

#Clustering#
OpenHCP - Intel
Open stack
Bliy
Freenas
Beowulf
proxmox - for VM


#SOB#
Odroid H2 - hardkernal
Turris
Clearfog -GT 8k - solidrun
Vim3 https://www.khadas.com/vim

#Related names#
Grid
Cluster


#Systems#
Librenms - allarm om servrar
Bitwarden - security valt
Active directory
Pfsense
Grafana
InfluxDB
Telegraf
UniFi Network Controller
Guacamole
Freenas
Zonealarm

#MAP
Pfsense
|--Proxy for "in/out trafic" -> internet
|--Security
|
|-------Net connection (Cat6 1Gb)
| /=Net connection (Fiber 40Gb)
| | |Freenas (Hardware = H2)
| | |--PFsense
| | |--UniFi Network Controller - wifi
| | |--Grafana,influxdb - grafer
| | |--Guacamole - remoute
| | |
| | \--Backup storage
| | |
| | |---systems config files\
| | |---snapshots----------------|---:- cloud storage
| | |---application mods----/
| | |
| | |
| | |
| \=Net connection (Fiber 40Gb)
|-----Net connection (Cat6 1Gb)
| Freenas (High Performance VM server = razen Traidripper)
| |--windows VM
| |--PFsense
| |
|
|
| Units
|
|-whier
| |-Servers{Net-Vlan:112{Net:xxx.x}}
|
|-Wifi {Net:xxx.xxx.0.0}
| |-mobiles {Net-Vlan:11{Net:xx.x}}
|---+-Computers {Net-Vlan:10{Net:xx.x}}
| \Security
| |----IoT (HomeA){Net-Vlan:10{Net:xx.x}}
| \--video and units{Net-Vlan:88{Net:xx.x}}

fvolk
Posts: 330
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, HC1, H2
Has thanked: 0
Been thanked: 17 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by fvolk » Wed Feb 19, 2020 4:26 am

brad wrote: - Hopefully H2's can directly boot kernel via PXE/ tftp (and then mount root filesystem over iSCSI) - do I need a bootdisk here to support PXE boot on N2 or can it be handled by BIOS and NIC's?
H2 only needs RAM, PXE enabled in UEFI and a network connection to be a worker node.
brad wrote: - C2''s could use small cheap sd cards for bootloader and mount rootfs over iSCSI
I considered to convert my C2 cluster to the same setup like my H2 cluster above.
It should almost be/work the same?
Only requires microSD with bootloader and kernel image -> only thing to update locally/manually is kernel image because there is no PXE?

steps:
1) Move C2 root fs to NFS export at server
2) Port my H2 worker kernel config to C2 patched kernel of similar version.
3) Setup ARM64 chroot on my H2 server with the C2 rootfs to be exported via NFS,
4) Compile all C2 software, including kernel within this ARM64-chroot-on-x86 (viewtopic.php?f=138&t=37284)
5) Copy kernel image to C2 worker nodes...

... but the setup work should be worth it in the long run? Currently I manually update/upgrade/sync all my C2 workers :-/
brad wrote: - N2's could use spi flash for boot and mount rootfs over iSCSI
I thought of a N2 cluster, but due to the various issues of N2 I so far did not buy one - probably I'll skip this generation...
These users thanked the author fvolk for the post:
Mullcom (Wed Feb 19, 2020 4:52 am)

brad
Posts: 1055
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 H2 (64 bit ftw)
Location: Australia
Has thanked: 27 times
Been thanked: 56 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by brad » Wed Feb 19, 2020 10:33 am

fvolk wrote:
Wed Feb 19, 2020 4:26 am

H2 only needs RAM, PXE enabled in UEFI and a network connection to be a worker node.
Thanks I was hoping for this :)
fvolk wrote:
Wed Feb 19, 2020 4:26 am
Only requires microSD with bootloader and kernel image -> only thing to update locally/manually is kernel image because there is no PXE?
I have been able to make the C2 boot from PXE in the past within the uboot environment. Are you using hardkernel or mainline uboot?

If hardkernel uboot I think it needs to be recompiled from source to add tftp / pxe support. Eg viewtopic.php?t=27310

If mainline it should already exist, I was using it 3-4 years ago but have not tried for some time - see viewtopic.php?f=138&t=20869

This is boot script I was using at the time to PCE boot the C2 with mainline uboot but I think now uboot will go off and automatically search for a compiled boor.scr script on tftp server to load boot script. If it can automatically find the boot.scr on tftp we should be able to have boot.src, kernel, dtb, and optional ramdisk all on tftp. The sdcard would just contain the bootloader / mainline uboot, no partitions needed. I need to try this again to see if it works (along with NFS, my example script below was booting OS on SD card but it can be swapped to NFS)

Code: Select all

#MMC Boot via tftp
setenv kernel_addr 11000000
setenv fdt_addr 1000000
setenv initrd_loadaddr 22000000
setenv tftpbootfile Image
setenv tftpfdtfile meson-gxbb-odroidc2.dtb
setenv tftpinitrdfile uInitrd-4.7.0-rc4+
setenv ipaddr 10.1.1.106
setenv serverip 10.1.1.104
setenv fsargs "setenv bootargs root=UUID=e139ce78-9841-40fe-8823-96a304a09859 ro no_console_suspend fsck.fix=yes"
setenv start_odroidc2_mmc 'tftp ${initrd_loadaddr} ${tftpinitrdfile} ;tftp ${kernel_addr} ${tftpbootfile};tftp ${fdt_addr} ${tftpfdtfile};run fsargs ;booti ${kernel_addr} ${initrd_loadaddr} ${fdt_addr}'
run start_odroidc2_mmc

fvolk
Posts: 330
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, HC1, H2
Has thanked: 0
Been thanked: 17 times
Contact:

Re: ODROID-H2 Micro Cluster

Unread post by fvolk » Fri Feb 21, 2020 6:24 pm

brad wrote:
fvolk wrote: H2 only needs RAM, PXE enabled in UEFI and a network connection to be a worker node.
Thanks I was hoping for this :)
Not to confuse, I said H2, I don't know about N2.
brad wrote: I have been able to make the C2 boot from PXE in the past within the uboot environment.
That would be nice if it works almost like on the H2.
This is off-topic here, I made a new thread in the C2 forums: viewtopic.php?f=138&t=37899

Post Reply

Return to “Projects”

Who is online

Users browsing this forum: No registered users and 3 guests