MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post Reply
o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Wed Jan 29, 2020 9:16 pm

Hi folks,

sorry if this is pure nonsense,but I had that wet dream to install RAM-disks on every node,then convert them with Glusterfs into a distributed volume.
If we could reroute the swapfile to this RAMdisk, woudn't it be like a single huge RAM-extension?
At least one node could run apps with high memory requirements.
Unfortunately I'm stuck at rerouting the swapfile to a local RAM disk (which wouldn't make sense ofcourse),so I didn't try Glusterfs yet.

Am I a lunatic or is there a chance to do that?
Olaf

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Wed Jan 29, 2020 11:42 pm

Have you tried zram first?
Since having a ramdrive shared over network will limit performance to network (1Gbps), which is typically slower than local attached disk. But you're on a mc1, so local disk is slower than network.

In this case you're not a lunatic for trying.
Here's how I'd do it: set up nfs to share /dev/shm on all nodes. Create a file dd if=/dev/zero of=/dev/shm/swap bs=1M count=1500.
Mount all shares on your target node. Run mkswap /media/share3/swap, swapon /media/share3/swap.

Top swap performance will be capped at 1Gbps.
Should one node/networking fail your system will likely hang.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Thu Jan 30, 2020 2:30 am

local disk is slower than network.
That's what I thought too. Especially write speed to SD card can be as low as 8MB/sec ( 22MB/sec max. )
Other SBCs without USB3.0 or EMMC drive would benefit too.
9$+shipping for a 2GB RAM extension sounds reasonable. And it comes with 8 extra-cores as bonus ;)
( as long as Hardkernel has some MC1 solos in stock )

I will definitely try it, once I have sorted out my network issues.
Olaf

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Thu Jan 30, 2020 3:55 am

The original plan may fail, because I remember the mkswap command treats files in a special way and skips fs journalling and apis and uses them as block devices within the filesystem. If it complains, try losetup -a `losetup -f` /media/share3/swap and it should map it to a /dev/loop0. Try running mkswap and swapon with the loop device instead.

If it works, you'll need to tell oom_killer not to kill the nfs process or network related processes.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Fri Jan 31, 2020 2:22 am

Should one node/networking fail your system will likely hang.
"Lets get crazy" #BobRoss_R.I.P.

I just read about RAID setups for SD cards, maybe there's a way to do the same with RAM-Drives? If not, this is my plan B. Eight ore more SD-cards in a RAID 5 or 6 system would be a bit faster and safe. Can we use RAM as write buffer for a RAID system?

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Fri Jan 31, 2020 4:11 am

Well, SD Cards in a RAID implies you are physically attaching them to a node, so expect <40MB/s total throughput. The more disks you have the lower the performance (since software RAID needs to do more disk access than standalone disks).
RAID for performance (e.g. RAID0) works only when the disks are connected to a high-speed bus, so that sum of disk speeds is lower than bus speed. Otherwise the bus will be the bottleneck.
So Ramdrive over network sounds like a saner idea.

legogris
Posts: 3
Joined: Fri Jan 31, 2020 12:05 pm
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by legogris » Wed Feb 05, 2020 11:53 pm

Don't run GlusterFS on this device. Users have reported horrible instability and they have officially dorpped 32-bit support: https://www.phoronix.com/forums/forum/s ... it-support

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Fri Feb 21, 2020 6:05 pm

Here's how I'd do it: set up nfs to share /dev/shm on all nodes. Create a file dd if=/dev/zero of=/dev/shm/swap bs=1M count=1500.
Mount all shares on your target node. Run mkswap /media/share3/swap, swapon /media/share3/swap.
Ok, I started reading nfs manpages and seems, it needs unique UIDs for all clients. In my case, I think I only have one client?
Could a client also be a nfs-server? It would be nice to use the local RAM first for a RAM-swapfile...

>Mount all shares on your target node.
I think the best way to mount them at startup is to use fstab with the options bg _netdev ?

Just_in_Case, I already have images of all rootfs partitions on external HDD, would I need their boot-partition, too?
Installing nfs_kernel_server changes kernel?

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Fri Feb 21, 2020 7:51 pm

nfs-kernel-server doesn't change kernel. The boot partition doesn't need to be exposed through nfs, and I haven't seen (or needed) the uid stuff.
First try it manually. If it works, worry about fstab...
These users thanked the author mad_ady for the post:
o_lampe (Sat Feb 22, 2020 4:43 pm)

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Sat Feb 22, 2020 6:12 pm

OK, here's what I did so far:
#Installing nfs-kernel-server on all the nodes ( listed in mc1-hosts.txt )

Code: Select all

 pssh -i -h mc1-hosts.txt -t 0 -p 8 -A -l root apt-get install nfs-kernel-server --yes
Using pssh saves a lot of time :)
#Writing a swapfile of 1GB to /dev/shm
Checking free memory before/after
shared_1GB.jpg
shared_1GB.jpg (39.21 KiB) Viewed 1930 times
Getting error message: 'no space left on device', but it seemed to have worked?

What about the buff/cache changes? Is that expected?

I also used pssh to 'dd' the swap file and edit the /etc/exports file to add the shared directory

Code: Select all

 pssh -i -h mc1-hosts.txt -t 0 -p 8 -A -l root echo '/dev/shm 192.168.xxx.xxx(rw,async,no_subtree_check)' >> /etc/exports
Well, that didn't work, but I could do it one_by_one with

Code: Select all

echo '/dev/shm 192.168.xxx.xxx(rw,async,no_subtree_check)' >> /etc/exports
After adding the shared directory, you have to reread the exports file, but that caused an error:
fsid.jpg
fsid.jpg (12.88 KiB) Viewed 1911 times
I have to figure out, what's about the missing fsid=...

While I'm locally on each node, I also added the nfs-client to the /etc/hosts file

Code: Select all

echo '192.168.xxx.xxx hostname' >> /etc/hosts
Replace hostname with the real hostname of the client

Now getting a bit confused. Should I run mkswap/swapon on all servers?
Is media/share3/swap automatically linked to /dev/shm/swap?

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Sat Feb 22, 2020 8:20 pm

You can run the mkswap command on each node or on the master, once mounted.

Now you have to mount everything with something like:

Code: Select all

mount -t nfs -o hard,intr,nodev,nosuid,users,vers=3,rsize=131072,wsize=131072,namlen=255,timeo=20,retrans=5,proto=tcp,rw 192.168.x.x:/shm /media/nodeX
Then try swapon /media/nodeX/swap
If /dev/shm doesn't work, try this instead: https://askubuntu.com/questions/304165/ ... m-disk-nfs

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Mon Mar 16, 2020 7:17 pm

I finally got the nerve to try a new way of collecting network-RAM and it is successful!

Basically, I created a ramdisk with tmpfs on each node,
added the mountingpoint to the /etc/exports file on each nfs-server.
Then made a folder for each ramdisk on the client (Odroid N2 with 2GB)
and added the nfs mount to /etc/fstab

Now I have 7 folders with 1.1GB each available :D ( there's even more RAM available on the MC1-nodes )
I'm pretty happy with transfer speeds, but still have to figure out how to combine them into one big RAM-volume.
GlusterFS is no option. Naming all the folders the same, doesn't help either...
8GB-ramdisk.png
8GB-ramdisk.png (300.48 KiB) Viewed 1614 times
I'll do some benchmarks and post them here
Attachments
8GB_fstab.png
8GB_fstab.png (68.46 KiB) Viewed 1613 times

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Mon Mar 16, 2020 7:43 pm

104MB/s for creating a remote dummyfile and 86MB/s for copying from one ramdisk to another isn't bad!
8GB_benchmark.png
8GB_benchmark.png (44.97 KiB) Viewed 1604 times

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Mon Mar 16, 2020 9:26 pm

Those are normal speeds, but keep in mind that all of them will be capped at 1Gbps.
Next try:

Code: Select all

dd if=/dev/zero of=/media/node1-rd/swapfile bs=1M count=1024
swapon /media/node1-rd/swapfile

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Tue Mar 17, 2020 1:56 am

There's a problem with access permissions. Owner of dd-created files is 'nobody'. I've set chmod to 777 for all nodes ramdisks. Any other options?
swapon.png
swapon.png (47 KiB) Viewed 1550 times

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Tue Mar 17, 2020 3:31 am

Permissions are not an issue. Forgot to tell you to mkswap /media/node1-*/swapfile and then swapon....

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Tue Mar 17, 2020 3:35 am

I think , I made it work. At least with the local ramdisk.
It's a lot work, setting chown+chmod right, using mkswap, losetup and swapon
I'm glad, I only have 8-node cluster...
Tomorrow, I'll do the same things with the remote ramdisks.

Code: Select all

odroid@odroid:~$ sudo mount ramfs
odroid@odroid:~$ sudo -i
root@odroid:~# chown -c -R root /media/ramdisk
root@odroid:~# dd if=/dev/zero of=/media/ramdisk/swapfile bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 0.37534 s, 1.4 GB/s
root@odroid:~# mkswap /media/ramdisk/swapfile
mkswap: /media/ramdisk/swapfile: insecure permissions 0644, 0600 suggested.
Setting up swapspace version 1, size = 500 MiB (524283904 bytes)
no label, UUID=b0d102a0-e702-40ad-a275-12ca34c62844
root@odroid:~$ chmod 600 /media/ramdisk/swapfile
root@odroid:~# losetup /dev/loop0 /media/ramdisk/swapfile
root@odroid:~# swapon /dev/loop0
root@odroid:~# cat /proc/swaps
Filename				Type		Size	Used	Priority

/dev/loop0                              partition	511996	0	-1
root@odroid:~# 
Last edited by o_lampe on Tue Mar 17, 2020 3:40 am, edited 1 time in total.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Tue Mar 17, 2020 3:39 am

How many loop devices can I use? Or do I losetup all swapfiles to loop0?

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Tue Mar 17, 2020 4:47 am

You don't need to mount from loop files. You can swapon directly from a file. There is a kernel parameter (max_loop?) that you can pass and configure more loop devices if you need to.
I hope you're not keeping the local "swap on a ramdrive". It's such a waste in resources due to overhead. Remote swap is ok given you only have an sd card.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Tue Mar 17, 2020 3:35 pm

mad_ady wrote:
Tue Mar 17, 2020 4:47 am
You don't need to mount from loop files. You can swapon directly from a file. There is a kernel parameter (max_loop?) that you can pass and configure more loop devices if you need to.
I hope you're not keeping the local "swap on a ramdrive". It's such a waste in resources due to overhead. Remote swap is ok given you only have an sd card.
I will only have zram-config on my local N2[*]
My ramdrive/swap experiments were made locally, because I've better access to it.

If I get you right, I will have a number of indipendent swapfiles, instead of one big swap?
What If I make only one swapfile with mkswap and use swapon on all other swapfiles (without header/UUID), will that glue them together as one big swap?

[*] On a N2 or other full-featured SBC, I could simply use an EMMC module as swap, but the initial idea is aimed to low-featured MC1/HC1 hardware.
The reason, why I try it on a N2 is the bad combination of my dual SATA adapter with SSD and the NTFS partition. ( IMHO this adapter only makes sense for HDDs )

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Tue Mar 17, 2020 4:35 pm

You need to run mkswap on each swapfile to format it as a swap file. You do this only once (when you create the file). Then you enable each one with swapon /path/to/swapfile. Linux can handle many swap files and can ballance the load between them. You don't need (and in your networked ramdrive example - can't have) one big swapfile, but you can have many small ones.
If you need to restart a node you can do swapoff /path/to/maintenance/node/swapfile so it doesn't crash your master system.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Wed Mar 18, 2020 3:26 am

I've read, that tmpfs starts swapping (to SD) automatically, when the volume size isn't fixed and it runs out off RAM.
Ramfs only uses RAM, but can't be fixed size-wise. There is a chance, it locks the nodes.

I will choose tmpfs and hope it will never exceed RAM capacity, or it will slow down dramatically.

o_lampe
Posts: 175
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 11 times
Been thanked: 16 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by o_lampe » Thu Mar 19, 2020 5:18 pm

OK, I managed to build a 8GB swap, but there are a few issues:

When I reboot the nodes, their ramdisks are empty. I have to create all the swapfiles, change ownership to odroid:root and r/w rights to chmod 666. For peace_of_mind I also restart the nfs-server. I can do that for all nodes at once by using pssh.
When I boot my N2 after the nodes' settings are done, most of the nfs- folders don't show up on the desktop.
Why do some nfs folders show and others don't?
I can see them in the network-GUI but can't mount them. Also using sudo mount -a doesn't work.
It seems the nfs-folders I've put in /etc/exports need some chown/chmod too but I don't know how.

Anyway, those folders who appear on the N2-desktop need some losetup- and swapon treatment every time I boot/reboot before I 'finally' have a remote-RAM-swap.
Nice to see the declining priority. I hope it tells the swap-handler which folder to use first, second a.s.o.
8GB-loop-swap.png
8GB-loop-swap.png (82.62 KiB) Viewed 1234 times
Is it worth it? I don't know, since I haven't found a good usecase to test the swap.

Code: Select all

sudo dd if=/dev/zero ... of=/dev/ram 
Is a good way to fill the local RAM, but doesn't use swap. Opening some more apps finally make use of swap. (don't dare to use a web-browser, the Gigabit port can't handle the extra traffic)
no_swap.png
no_swap.png (214.08 KiB) Viewed 1234 times
These users thanked the author o_lampe for the post:
mad_ady (Thu Mar 19, 2020 5:55 pm)

User avatar
mad_ady
Posts: 7925
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 512 times
Been thanked: 351 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post by mad_ady » Thu Mar 19, 2020 5:57 pm

I'm glad it works.
You can create/format the local swap files on each node on boot.
For nfs I usually use autofs instead of fstab.
To load swap you could use a script on the n2 to test if the file exists.
To stress swap try compiling chrome...

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 2 guests