Swap

Post Reply
scythefwd
Posts: 27
Joined: Tue Dec 04, 2018 10:46 am
languages_spoken: english
ODROIDs: xu4 with 32gb emmc
Has thanked: 0
Been thanked: 0
Contact:

Swap

Unread post by scythefwd » Thu Jan 10, 2019 9:26 am

Ok, I created a swap file.. but I'm not loving it. Can I boot off a SD card and safely resize my, now dismounted, root fs on my emmc and create a swap partition? Then I should be able to get rid of the swapfile and just use the partition right?

Thank..

Oh, it's an xu4 running Ubuntu 16.04 and normally booting off the emmc...

User avatar
tobetter
Posts: 4019
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 42 times
Been thanked: 173 times
Contact:

Re: Swap

Unread post by tobetter » Thu Jan 10, 2019 10:38 am

scythefwd wrote:
Thu Jan 10, 2019 9:26 am
Ok, I created a swap file.. but I'm not loving it. Can I boot off a SD card and safely resize my, now dismounted, root fs on my emmc and create a swap partition? Then I should be able to get rid of the swapfile and just use the partition right?

Thank..

Oh, it's an xu4 running Ubuntu 16.04 and normally booting off the emmc...
Are you meaning swap partition? It can be removed from fdisk or gparted and merge/resize to another partition. Before doing this, please check and remove the file system info for the swap partition in /etc/fstab.

Code: Select all

/swapfile                                 none            swap    sw              0       0

scythefwd
Posts: 27
Joined: Tue Dec 04, 2018 10:46 am
languages_spoken: english
ODROIDs: xu4 with 32gb emmc
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by scythefwd » Thu Jan 10, 2019 11:21 am

no, i mean i created a swap file. I would rather use a swap partition. my / partition takes up the full emmc card, so i need to resize it.. which cannot be done with the system online...

User avatar
tobetter
Posts: 4019
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 42 times
Been thanked: 173 times
Contact:

Re: Swap

Unread post by tobetter » Thu Jan 10, 2019 11:53 am

scythefwd wrote:
Thu Jan 10, 2019 11:21 am
no, i mean i created a swap file. I would rather use a swap partition. my / partition takes up the full emmc card, so i need to resize it.. which cannot be done with the system online...
Sorry for the misunderstanding, you would try to create the swap partition on another system.

scythefwd
Posts: 27
Joined: Tue Dec 04, 2018 10:46 am
languages_spoken: english
ODROIDs: xu4 with 32gb emmc
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by scythefwd » Thu Jan 10, 2019 10:37 pm

No probs.. I get where the confusion was lol. I ended up creating a ubuntu sd card, booted off that with the emmc still attached (I'm wary of popping it on and off too many times.. I worry about the tightness of the connector so to speak.. so I do try to avoid it). I was then able to resize the root partition on the emmc card using gparted (I am not a fan of the partitioning tool that comes with this distro, no biggie, thats why we have apt-get right :D ).., created a new swap partition at the end of the emmc card, formated it as swap, then booted to the emmc and fired up gparted again and selected swap on. I then edited my fstab..

Here is a question. I have the fstab entries created and I have rebooted. Doing swapon -s I show the swap partition with a priority of -2 (what does the -2 mean?).. but when I do a mount, I dont see it listed in the output. Is this normal?

User avatar
mad_ady
Posts: 6637
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 185 times
Been thanked: 142 times
Contact:

Re: Swap

Unread post by mad_ady » Thu Jan 10, 2019 10:47 pm

The priority is relevant only if you have several swap partitions/files on different faster/slower storages. It tells the os which to prefer (e.g. zram is preferred before going to disk based swap)

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Swap

Unread post by tkaiser » Fri Jan 11, 2019 2:23 am

scythefwd wrote:
Thu Jan 10, 2019 11:21 am
no, i mean i created a swap file. I would rather use a swap partition.
If you run already kernel 2.6 or later and use ext4 there's absolutely no reason to prefer a swap partition over a swap file (especially on an eMMC where HDD fragmentation won't happen anyway. See last paragraph here: http://lkml.iu.edu/hypermail/linux/kern ... /1690.html

But if I were you I would comment out and delete the swap file and apt install zram-config.

User avatar
mad_ady
Posts: 6637
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 185 times
Been thanked: 142 times
Contact:

Re: Swap

Unread post by mad_ady » Fri Jan 11, 2019 2:29 am

Huh, I never knew swap files bypassed the filesystem journalling. Nice touch :)

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Swap

Unread post by tkaiser » Fri Jan 11, 2019 3:25 am

mad_ady wrote:
Fri Jan 11, 2019 2:29 am
Huh, I never knew swap files bypassed the filesystem journalling.
Well, it's just one and a half decades ago that this doesn't matter any more. And now it's 2019 and we all should've long forgotten about those ancient things like 'traditional swap on disk' since we have zram/zswap...

scythefwd
Posts: 27
Joined: Tue Dec 04, 2018 10:46 am
languages_spoken: english
ODROIDs: xu4 with 32gb emmc
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by scythefwd » Fri Jan 11, 2019 3:40 am

what is zram? I mean that in a technical sense?

Also, is the file vs. partition actually any different in performance? I realize that times change, and old dogs dont like learning new tricks.. does the swap file properly show disk remaining on stuff like df -h? If I'm reading and understanding right (big IF there), zram is a compressed ramdisk? With limited ram like we have on these xu4's.. doesn't blocking the usage of some of it only to be used as an overflow for itself seem a bit.. contradictory?

I mean.. if you're using 2.1gb of ram, then a compressed ramdisk might work.. but if you're using 2.4-3gb of ram.. you're just delaying the inevitable by a few hundred mb.. I dont necessarily intend on using that much ram, but I feel like its a bit of a hax to use ram to back up your ram as swap. Like I said.. it's a big IF on my understanding.. so please educate me more than wiki can..

scythefwd
Posts: 27
Joined: Tue Dec 04, 2018 10:46 am
languages_spoken: english
ODROIDs: xu4 with 32gb emmc
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by scythefwd » Fri Jan 11, 2019 3:48 am

Lol at the decade and a half.. thats 5 years after I stopped using linux daily lol... appreciate the information, and hopefully the further education.

User avatar
meveric
Posts: 10481
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go, H2 (N4100), N2
Has thanked: 17 times
Been thanked: 130 times
Contact:

Re: Swap

Unread post by meveric » Fri Jan 11, 2019 4:16 am

tkaiser wrote:
Fri Jan 11, 2019 3:25 am
And now it's 2019 and we all should've long forgotten about those ancient things like 'traditional swap on disk' since we have zram/zswap...
Sadly that's not always possible. I have applications, that crash/freeze an ODROID if they run on ZRAM, while they just crawl to snail performance running on conventional SWAP.
scythefwd wrote:
Fri Jan 11, 2019 3:40 am
what is zram? I mean that in a technical sense?
As you already found out it's a part of the RAM that is compressed and as such used as a swap partition.
scythefwd wrote:
Fri Jan 11, 2019 3:40 am
With limited ram like we have on these xu4's.. doesn't blocking the usage of some of it only to be used as an overflow for itself seem a bit.. contradictory?
SWAP (any kind of it) is suppose to be a last resort measure not a means of normal runtime.
If your system is swapping, it's already performing outside of it's specification and just tries to avoid a failure.
Still the design of ZRAM is rather intelligent. ZRAM is not used until needed, so if you have 2GB RAM and reserve 500MB for ZRAM, that doesn't mean that you can only use 1.5GB RAM and the rest is for ZRAM, it means you can still use 2GB RAM, but when you require more, part of your RAM will be compressed to "free" some RAM.
The system will only work on the FREE RAM, never on the compressed RAM.
scythefwd wrote:
Fri Jan 11, 2019 3:40 am
I mean.. if you're using 2.1gb of ram, then a compressed ramdisk might work.. but if you're using 2.4-3gb of ram..
Interestingly the compression of RAM is quite efficient. 1GB of RAM may be compressed to just 200~300MB of ZRAM
So it's quite possible to reserve 500MB for ZRAM which may hold up to 2GB of uncompressed RAM. So the total amount of RAM data would be about 3.5 GB if you think of a system with 2GB RAM using 500MB ZRAM (1.5 GB used RAM + 2GB Compressed RAM in a 500 MB ZRAM part).
scythefwd wrote:
Fri Jan 11, 2019 3:40 am
you're just delaying the inevitable by a few hundred mb..
True, but that's the point of SWAP, catching a short spike, to prevent an application from crashing.
scythefwd wrote:
Fri Jan 11, 2019 3:40 am
I dont necessarily intend on using that much ram, but I feel like its a bit of a hax to use ram to back up your ram as swap.
As explained the compression is quite efficient, and RAM is A LOT faster than a harddrive, or even an eMMC or SSD.
Storing 100MB on an SD card swap, may take a long time, in fact some application might to fill RAM a lot faster than you actually can write the data in SWAP.
ZRAM instead compresses pages that are already in RAM, so it only costs a little bit CPU time for compression and decompression.
Also SWAP can NOT be used directly, neither ZRAM nor SWAP are used as an "extension" of RAM. They "park" currently not vital information in the RAM to "free" RAM to work with.
If you have applications A B C D E F G in memory, but G requires more RAM than you have, the system might decide, to store the memory pages used by D in SWAP as D is currently not processing any data and for that "freeing" the part of RAM that was previously used, by D. Now G has enough memory to finish his "work".
Suddenly D wants to do some tasks as well, but the data is no longer in RAM, so it needs to get the DATA out of the SWAP to work on it, for this, for example the data of A is put into swap, the data of D is loaded from the swap to the now free space, and this swapping (hence the name swap) goes on and on, until one process no longer needs the memory.
Doing these swaps of the data on a hard disk of any type is VERY VERY VERY slow.
With ZRAM instead, it's just compressing and decompressing the data that is already in the memory.. it will cost you CPU time for compression and decompression as I said, but it's A LOT faster than reading and writing to an SD/eMMC/HDD/SSD.

Take this as a simplified explanation...
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.

scythefwd
Posts: 27
Joined: Tue Dec 04, 2018 10:46 am
languages_spoken: english
ODROIDs: xu4 with 32gb emmc
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by scythefwd » Fri Jan 11, 2019 8:10 am

appreciate it.. Yes, swap is a measure of last resort. Thanks for the info.. I'm still an old school guy at heart.. bare metal for everything lol. I get that the compression is efficient.. just feels odd to me.

User avatar
rooted
Posts: 6671
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 135 times
Been thanked: 31 times
Contact:

Re: Swap

Unread post by rooted » Fri Jan 11, 2019 9:06 am

Most Android devices use zram and it is quite efficient at what it does.

lazlo
Posts: 143
Joined: Sun Oct 01, 2017 11:32 am
languages_spoken: english
ODROIDs: XU4
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by lazlo » Fri Jan 11, 2019 9:46 am

Personally, I don't like zram because I don't trust the compression of non-ECC RAM. My XU-4 uses a swap file over NFS4 and doesn't have any issues so I stick with it. That has the added benefit of the networking stack doing CRC checks on the packets and the NFS server providing redundancy via RAID10. Other benefits of NFS include saving wear on the SD card I boot from and increased read/write speeds for the filesystems. I am sure it is measurably slower than zram but I don't find it noticeable compared to compiling gcc or rust with 1.5GB of RAM and a zram swap instead of 2GB of RAM and a 2GB swap file.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Swap

Unread post by tkaiser » Fri Jan 11, 2019 4:53 pm

rooted wrote:
Fri Jan 11, 2019 9:06 am
Most Android devices use zram and it is quite efficient at what it does.
It's all about defaults.

Engineers and kernel developers understand this whole issue, developed zram as a rather simple means of improving storage based swap dramatically and it got rolled out as default on billions of Android devices. Of course it works fine (not really great but fine). The same 'changed default' would be needed on Linux. In fact it happens already here and there that zram gets activated on certain platforms or in specific situations * and people usually don't notice it other than their device becoming faster in situations where they run low on memory. Once it's default nobody will talk about it any more but everybody will benefit from it.

We use zram on all Linux based servers except those with ZFS. Usually heavy memory overcommitment of 200% or even 300% (that means with one GB real DRAM 2 or 3 GB configured as ZRAM).



* Fedora 29 activates zram by default on ARM, Armbian does the same since some time, IIRC one distro activates zram on machines with less than 2 GB RAM
Last edited by tkaiser on Fri Jan 11, 2019 7:44 pm, edited 2 times in total.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Swap

Unread post by tkaiser » Fri Jan 11, 2019 4:57 pm

lazlo wrote:
Fri Jan 11, 2019 9:46 am
That has the added benefit of the networking stack doing CRC checks on the packets
All of this happens in the non-ECC DRAM you don't trust in. Why do you trust in this CRC thing then?

lazlo
Posts: 143
Joined: Sun Oct 01, 2017 11:32 am
languages_spoken: english
ODROIDs: XU4
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by lazlo » Sat Jan 12, 2019 6:12 am

tkaiser wrote:
Fri Jan 11, 2019 4:57 pm
lazlo wrote:
Fri Jan 11, 2019 9:46 am
That has the added benefit of the networking stack doing CRC checks on the packets
All of this happens in the non-ECC DRAM you don't trust in. Why do you trust in this CRC thing then?
Wait, what? DDR3 does CRC on data written to and read from it? Are you really sure about that? If so please provide a link so that I may educate myself further.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Swap

Unread post by tkaiser » Sat Jan 12, 2019 9:47 am

lazlo wrote:
Sat Jan 12, 2019 6:12 am
DDR3 does CRC on data written to and read from it?
Huh?

lazlo
Posts: 143
Joined: Sun Oct 01, 2017 11:32 am
languages_spoken: english
ODROIDs: XU4
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by lazlo » Sat Jan 12, 2019 12:33 pm

OK then, I guess I misunderstood your point.

tkaiser
Posts: 672
Joined: Mon Nov 09, 2015 12:30 am
languages_spoken: english
ODROIDs: C1+, C2, XU4, HC1
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Swap

Unread post by tkaiser » Mon Jan 14, 2019 4:05 pm

lazlo wrote:
Sat Jan 12, 2019 12:33 pm
I guess I misunderstood your point.
Networking on the XU4 is implemented using USB. I would believe a single page access traveling to or from your NFS server involves a lot more DRAM access compared to letting zram do the job (filesystem layer, TCP/IP layer, USB layer, no compression happening whatsoever).

Can a CRC check work when we can't trust the RAM? I guess not, since otherwise no ECC RAM would be needed (where integrity check and correction happens a layer below the OS).

What happens if there are bit flips due to using non-ECC memory? Might result in data corruption, application or kernel crash.

Android uses compressed swap on billions of devices (zram), iOS uses compressed memory on billions of devices, macOS uses a similar approach (both more advanced than zram), even Windows 10 implements something similar to zswap (not zram) in the meantime. I bet the amount of devices benefiting from RAM compression but equipped with ECC RAM is below the 0.01% percentage mark.

Almost everywhere compressed memory is default today. Most (un)affected users are those running Linux with distros or OS images that do not switch on more intelligent RAM usage by default.

iOS/macOS: http://newosxbook.com/articles/MemoryPressure.html
Windows 10: https://www.makeuseof.com/tag/ram-compr ... indows-10/

lazlo
Posts: 143
Joined: Sun Oct 01, 2017 11:32 am
languages_spoken: english
ODROIDs: XU4
Has thanked: 0
Been thanked: 0
Contact:

Re: Swap

Unread post by lazlo » Mon Jan 21, 2019 1:49 am

OK, I see what you are trying to say. I will never trust compressed memory on non-ECC RAM because if one bit gets flipped in the compression index then at least two bits (and probably a whole lot more) get flipped inside the compressed region of RAM. I don't care how many devices are using it. I will always see it as a cheap trick to get around the limitations in the way certain products are designed. Why do I trust CRC on my XU4? I don't trust in my XU4 at all. I am of the opinion that it has far too many design limitations for it to be used for anything important. I see my XU4 as fun a toy.

My server is a tool though and that is what I use for all of my critical tasks. And even though it does have 32GB of registered ECC RAM, it also has four swap partitions. One at the end of each HDD.

elatllat
Posts: 1525
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 21 times
Been thanked: 49 times
Contact:

Re: Swap

Unread post by elatllat » Mon Jan 21, 2019 3:08 pm

lazlo wants radeation hardened parts when, amazon, backblaze, SpaceX, etc use distributed cheap parts for better reliability and ROI...

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 2 guests