MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Post Reply
o_lampe
Posts: 74
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 4 times
Been thanked: 9 times
Contact:

MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Unread post by o_lampe » Wed Jan 29, 2020 9:16 pm

Hi folks,

sorry if this is pure nonsense,but I had that wet dream to install RAM-disks on every node,then convert them with Glusterfs into a distributed volume.
If we could reroute the swapfile to this RAMdisk, woudn't it be like a single huge RAM-extension?
At least one node could run apps with high memory requirements.
Unfortunately I'm stuck at rerouting the swapfile to a local RAM disk (which wouldn't make sense ofcourse),so I didn't try Glusterfs yet.

Am I a lunatic or is there a chance to do that?
Olaf

User avatar
mad_ady
Posts: 7332
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 346 times
Been thanked: 252 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Unread post by mad_ady » Wed Jan 29, 2020 11:42 pm

Have you tried zram first?
Since having a ramdrive shared over network will limit performance to network (1Gbps), which is typically slower than local attached disk. But you're on a mc1, so local disk is slower than network.

In this case you're not a lunatic for trying.
Here's how I'd do it: set up nfs to share /dev/shm on all nodes. Create a file dd if=/dev/zero of=/dev/shm/swap bs=1M count=1500.
Mount all shares on your target node. Run mkswap /media/share3/swap, swapon /media/share3/swap.

Top swap performance will be capped at 1Gbps.
Should one node/networking fail your system will likely hang.

o_lampe
Posts: 74
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 4 times
Been thanked: 9 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Unread post by o_lampe » Thu Jan 30, 2020 2:30 am

local disk is slower than network.
That's what I thought too. Especially write speed to SD card can be as low as 8MB/sec ( 22MB/sec max. )
Other SBCs without USB3.0 or EMMC drive would benefit too.
9$+shipping for a 2GB RAM extension sounds reasonable. And it comes with 8 extra-cores as bonus ;)
( as long as Hardkernel has some MC1 solos in stock )

I will definitely try it, once I have sorted out my network issues.
Olaf

User avatar
mad_ady
Posts: 7332
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 346 times
Been thanked: 252 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Unread post by mad_ady » Thu Jan 30, 2020 3:55 am

The original plan may fail, because I remember the mkswap command treats files in a special way and skips fs journalling and apis and uses them as block devices within the filesystem. If it complains, try losetup -a `losetup -f` /media/share3/swap and it should map it to a /dev/loop0. Try running mkswap and swapon with the loop device instead.

If it works, you'll need to tell oom_killer not to kill the nfs process or network related processes.

o_lampe
Posts: 74
Joined: Wed Jan 01, 2020 12:35 am
languages_spoken: english german
ODROIDs: MC1 solo
Has thanked: 4 times
Been thanked: 9 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Unread post by o_lampe » Fri Jan 31, 2020 2:22 am

Should one node/networking fail your system will likely hang.
"Lets get crazy" #BobRoss_R.I.P.

I just read about RAID setups for SD cards, maybe there's a way to do the same with RAM-Drives? If not, this is my plan B. Eight ore more SD-cards in a RAID 5 or 6 system would be a bit faster and safe. Can we use RAM as write buffer for a RAID system?

User avatar
mad_ady
Posts: 7332
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 346 times
Been thanked: 252 times
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Unread post by mad_ady » Fri Jan 31, 2020 4:11 am

Well, SD Cards in a RAID implies you are physically attaching them to a node, so expect <40MB/s total throughput. The more disks you have the lower the performance (since software RAID needs to do more disk access than standalone disks).
RAID for performance (e.g. RAID0) works only when the disks are connected to a high-speed bus, so that sum of disk speeds is lower than bus speed. Otherwise the bus will be the bottleneck.
So Ramdrive over network sounds like a saner idea.

legogris
Posts: 3
Joined: Fri Jan 31, 2020 12:05 pm
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: MC1 cluster with GlusterFS on RAMdisk = shared memory ?

Unread post by legogris » Wed Feb 05, 2020 11:53 pm

Don't run GlusterFS on this device. Users have reported horrible instability and they have officially dorpped 32-bit support: https://www.phoronix.com/forums/forum/s ... it-support

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 4 guests