MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post Reply
o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

Hi folks,

sorry if this is pure nonsense,but I had that wet dream to install RAM-disks on every node,then convert them with Glusterfs into a distributed volume.
If we could reroute the swapfile to this RAMdisk, woudn't it be like a single huge RAM-extension?
At least one node could run apps with high memory requirements.
Unfortunately I'm stuck at rerouting the swapfile to a local RAM disk (which wouldn't make sense ofcourse),so I didn't try Glusterfs yet.

Am I a lunatic or is there a chance to do that?
Olaf

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

Have you tried zram first?
Since having a ramdrive shared over network will limit performance to network (1Gbps), which is typically slower than local attached disk. But you're on a mc1, so local disk is slower than network.

In this case you're not a lunatic for trying.
Here's how I'd do it: set up nfs to share /dev/shm on all nodes. Create a file dd if=/dev/zero of=/dev/shm/swap bs=1M count=1500.
Mount all shares on your target node. Run mkswap /media/share3/swap, swapon /media/share3/swap.

Top swap performance will be capped at 1Gbps.
Should one node/networking fail your system will likely hang.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

local disk is slower than network.
That's what I thought too. Especially write speed to SD card can be as low as 8MB/sec ( 22MB/sec max. )
Other SBCs without USB3.0 or EMMC drive would benefit too.
9$+shipping for a 2GB RAM extension sounds reasonable. And it comes with 8 extra-cores as bonus ;)
( as long as Hardkernel has some MC1 solos in stock )

I will definitely try it, once I have sorted out my network issues.
Olaf

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

The original plan may fail, because I remember the mkswap command treats files in a special way and skips fs journalling and apis and uses them as block devices within the filesystem. If it complains, try losetup -a `losetup -f` /media/share3/swap and it should map it to a /dev/loop0. Try running mkswap and swapon with the loop device instead.

If it works, you'll need to tell oom_killer not to kill the nfs process or network related processes.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

Should one node/networking fail your system will likely hang.
"Lets get crazy" #BobRoss_R.I.P.

I just read about RAID setups for SD cards, maybe there's a way to do the same with RAM-Drives? If not, this is my plan B. Eight ore more SD-cards in a RAID 5 or 6 system would be a bit faster and safe. Can we use RAM as write buffer for a RAID system?

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

Well, SD Cards in a RAID implies you are physically attaching them to a node, so expect <40MB/s total throughput. The more disks you have the lower the performance (since software RAID needs to do more disk access than standalone disks).
RAID for performance (e.g. RAID0) works only when the disks are connected to a high-speed bus, so that sum of disk speeds is lower than bus speed. Otherwise the bus will be the bottleneck.
So Ramdrive over network sounds like a saner idea.

legogris
Posts: 7
Joined: Fri Jan 31, 2020 12:05 pm
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by legogris »

Don't run GlusterFS on this device. Users have reported horrible instability and they have officially dorpped 32-bit support: https://www.phoronix.com/forums/forum/s ... it-support

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

Here's how I'd do it: set up nfs to share /dev/shm on all nodes. Create a file dd if=/dev/zero of=/dev/shm/swap bs=1M count=1500.
Mount all shares on your target node. Run mkswap /media/share3/swap, swapon /media/share3/swap.
Ok, I started reading nfs manpages and seems, it needs unique UIDs for all clients. In my case, I think I only have one client?
Could a client also be a nfs-server? It would be nice to use the local RAM first for a RAM-swapfile...

>Mount all shares on your target node.
I think the best way to mount them at startup is to use fstab with the options bg _netdev ?

Just_in_Case, I already have images of all rootfs partitions on external HDD, would I need their boot-partition, too?
Installing nfs_kernel_server changes kernel?

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

nfs-kernel-server doesn't change kernel. The boot partition doesn't need to be exposed through nfs, and I haven't seen (or needed) the uid stuff.
First try it manually. If it works, worry about fstab...
These users thanked the author mad_ady for the post:
o_lampe (Sat Feb 22, 2020 4:43 pm)

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

OK, here's what I did so far:
#Installing nfs-kernel-server on all the nodes ( listed in mc1-hosts.txt )

Code: Select all

 pssh -i -h mc1-hosts.txt -t 0 -p 8 -A -l root apt-get install nfs-kernel-server --yes
Using pssh saves a lot of time :)
#Writing a swapfile of 1GB to /dev/shm
Checking free memory before/after
shared_1GB.jpg
shared_1GB.jpg (39.21 KiB) Viewed 2411 times
Getting error message: 'no space left on device', but it seemed to have worked?

What about the buff/cache changes? Is that expected?

I also used pssh to 'dd' the swap file and edit the /etc/exports file to add the shared directory

Code: Select all

 pssh -i -h mc1-hosts.txt -t 0 -p 8 -A -l root echo '/dev/shm 192.168.xxx.xxx(rw,async,no_subtree_check)' >> /etc/exports
Well, that didn't work, but I could do it one_by_one with

Code: Select all

echo '/dev/shm 192.168.xxx.xxx(rw,async,no_subtree_check)' >> /etc/exports
After adding the shared directory, you have to reread the exports file, but that caused an error:
fsid.jpg
fsid.jpg (12.88 KiB) Viewed 2392 times
I have to figure out, what's about the missing fsid=...

While I'm locally on each node, I also added the nfs-client to the /etc/hosts file

Code: Select all

echo '192.168.xxx.xxx hostname' >> /etc/hosts
Replace hostname with the real hostname of the client

Now getting a bit confused. Should I run mkswap/swapon on all servers?
Is media/share3/swap automatically linked to /dev/shm/swap?

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

You can run the mkswap command on each node or on the master, once mounted.

Now you have to mount everything with something like:

Code: Select all

mount -t nfs -o hard,intr,nodev,nosuid,users,vers=3,rsize=131072,wsize=131072,namlen=255,timeo=20,retrans=5,proto=tcp,rw 192.168.x.x:/shm /media/nodeX
Then try swapon /media/nodeX/swap
If /dev/shm doesn't work, try this instead: https://askubuntu.com/questions/304165/ ... m-disk-nfs

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

I finally got the nerve to try a new way of collecting network-RAM and it is successful!

Basically, I created a ramdisk with tmpfs on each node,
added the mountingpoint to the /etc/exports file on each nfs-server.
Then made a folder for each ramdisk on the client (Odroid N2 with 2GB)
and added the nfs mount to /etc/fstab

Now I have 7 folders with 1.1GB each available :D ( there's even more RAM available on the MC1-nodes )
I'm pretty happy with transfer speeds, but still have to figure out how to combine them into one big RAM-volume.
GlusterFS is no option. Naming all the folders the same, doesn't help either...
8GB-ramdisk.png
8GB-ramdisk.png (300.48 KiB) Viewed 2095 times
I'll do some benchmarks and post them here
Attachments
8GB_fstab.png
8GB_fstab.png (68.46 KiB) Viewed 2094 times

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

104MB/s for creating a remote dummyfile and 86MB/s for copying from one ramdisk to another isn't bad!
8GB_benchmark.png
8GB_benchmark.png (44.97 KiB) Viewed 2085 times

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

Those are normal speeds, but keep in mind that all of them will be capped at 1Gbps.
Next try:

Code: Select all

dd if=/dev/zero of=/media/node1-rd/swapfile bs=1M count=1024
swapon /media/node1-rd/swapfile

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

There's a problem with access permissions. Owner of dd-created files is 'nobody'. I've set chmod to 777 for all nodes ramdisks. Any other options?
swapon.png
swapon.png (47 KiB) Viewed 2031 times

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

Permissions are not an issue. Forgot to tell you to mkswap /media/node1-*/swapfile and then swapon....

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

I think , I made it work. At least with the local ramdisk.
It's a lot work, setting chown+chmod right, using mkswap, losetup and swapon
I'm glad, I only have 8-node cluster...
Tomorrow, I'll do the same things with the remote ramdisks.

Code: Select all

odroid@odroid:~$ sudo mount ramfs
odroid@odroid:~$ sudo -i
root@odroid:~# chown -c -R root /media/ramdisk
root@odroid:~# dd if=/dev/zero of=/media/ramdisk/swapfile bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.37534 s, 1.4 GB/s
root@odroid:~# mkswap /media/ramdisk/swapfile
mkswap: /media/ramdisk/swapfile: insecure permissions 0644, 0600 suggested.
Setting up swapspace version 1, size = 500 MiB (524283904 bytes)
no label, UUID=b0d102a0-e702-40ad-a275-12ca34c62844
root@odroid:~$ chmod 600 /media/ramdisk/swapfile
root@odroid:~# losetup /dev/loop0 /media/ramdisk/swapfile
root@odroid:~# swapon /dev/loop0
root@odroid:~# cat /proc/swaps
Filename				Type		Size	Used	Priority

/dev/loop0                              partition	511996	0	-1
root@odroid:~# 
Last edited by o_lampe on Tue Mar 17, 2020 3:40 am, edited 1 time in total.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

How many loop devices can I use? Or do I losetup all swapfiles to loop0?

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

You don't need to mount from loop files. You can swapon directly from a file. There is a kernel parameter (max_loop?) that you can pass and configure more loop devices if you need to.
I hope you're not keeping the local "swap on a ramdrive". It's such a waste in resources due to overhead. Remote swap is ok given you only have an sd card.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

mad_ady wrote:
Tue Mar 17, 2020 4:47 am
You don't need to mount from loop files. You can swapon directly from a file. There is a kernel parameter (max_loop?) that you can pass and configure more loop devices if you need to.
I hope you're not keeping the local "swap on a ramdrive". It's such a waste in resources due to overhead. Remote swap is ok given you only have an sd card.
I will only have zram-config on my local N2[*]
My ramdrive/swap experiments were made locally, because I've better access to it.

If I get you right, I will have a number of indipendent swapfiles, instead of one big swap?
What If I make only one swapfile with mkswap and use swapon on all other swapfiles (without header/UUID), will that glue them together as one big swap?

[*] On a N2 or other full-featured SBC, I could simply use an EMMC module as swap, but the initial idea is aimed to low-featured MC1/HC1 hardware.
The reason, why I try it on a N2 is the bad combination of my dual SATA adapter with SSD and the NTFS partition. ( IMHO this adapter only makes sense for HDDs )

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

You need to run mkswap on each swapfile to format it as a swap file. You do this only once (when you create the file). Then you enable each one with swapon /path/to/swapfile. Linux can handle many swap files and can ballance the load between them. You don't need (and in your networked ramdrive example - can't have) one big swapfile, but you can have many small ones.
If you need to restart a node you can do swapoff /path/to/maintenance/node/swapfile so it doesn't crash your master system.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

I've read, that tmpfs starts swapping (to SD) automatically, when the volume size isn't fixed and it runs out off RAM.
Ramfs only uses RAM, but can't be fixed size-wise. There is a chance, it locks the nodes.

I will choose tmpfs and hope it will never exceed RAM capacity, or it will slow down dramatically.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe »

OK, I managed to build a 8GB swap, but there are a few issues:

When I reboot the nodes, their ramdisks are empty. I have to create all the swapfiles, change ownership to odroid:root and r/w rights to chmod 666. For peace_of_mind I also restart the nfs-server. I can do that for all nodes at once by using pssh.
When I boot my N2 after the nodes' settings are done, most of the nfs- folders don't show up on the desktop.
Why do some nfs folders show and others don't?
I can see them in the network-GUI but can't mount them. Also using sudo mount -a doesn't work.
It seems the nfs-folders I've put in /etc/exports need some chown/chmod too but I don't know how.

Anyway, those folders who appear on the N2-desktop need some losetup- and swapon treatment every time I boot/reboot before I 'finally' have a remote-RAM-swap.
Nice to see the declining priority. I hope it tells the swap-handler which folder to use first, second a.s.o.
8GB-loop-swap.png
8GB-loop-swap.png (82.62 KiB) Viewed 1715 times
Is it worth it? I don't know, since I haven't found a good usecase to test the swap.

Code: Select all

sudo dd if=/dev/zero ... of=/dev/ram 
Is a good way to fill the local RAM, but doesn't use swap. Opening some more apps finally make use of swap. (don't dare to use a web-browser, the Gigabit port can't handle the extra traffic)
no_swap.png
no_swap.png (214.08 KiB) Viewed 1715 times
These users thanked the author o_lampe for the post:
mad_ady (Thu Mar 19, 2020 5:55 pm)

mad_ady
Posts: 8596
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 579 times
Been thanked: 488 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady »

I'm glad it works.
You can create/format the local swap files on each node on boot.
For nfs I usually use autofs instead of fstab.
To load swap you could use a script on the n2 to test if the file exists.
To stress swap try compiling chrome...

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 0 guests