fsck-ing large fs and 2GB mem

Post Reply
fvolk
Posts: 300
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, HC1, H2
Has thanked: 0
Been thanked: 11 times
Contact:

fsck-ing large fs and 2GB mem

Unread post by fvolk » Fri May 31, 2019 3:27 am

So I had a HC1 running with a 2TB disk for months now without problems as a micro server.

Now, suddenly, processes stopped or crashed - the filesystem became read-only.
Mhh.... ok, maybe the USB-SATA bridge acted up?
Shutdown, reboot and.... mount would not mount the disk because of fs errors.
Ran fsck.ext4, fixed some errors.
Try mount again and... fails again... wtf?
Fsck again and.... lots of errors.
Ok, stop, what's happening here?
Ran long SMART disk on drive - all is right, no bad sectors, disk is ok.
Ran fsck.ext4, does not want do automatic repair, needs to be run in manual mode.
Ok, run fsck.ext4 in manual mode, finds lots of errors, and.... crashes midway through with out of memory.

TL;DR: Somehow fs got errors, SMART ok so not a disk problem, fsck.ext4 run on HC1 became impossible because of lack of main memory.
I physically remove the disk, attach it with an USB adapter to a normal PC, run fsck.... fsck requires almost 5Gb main mem to run through and fix all errors ....wtf?

Everything works now again, but this incident showed me that I can end up with a bad fs that cannot be fixed on the HC1 alone?
I have to physically get to the HC1, and use a PC nearby to fix a buggy fs there.

Has anyone ever had this problem with a large disk attached, the HC1 can use up to 5TB disks and then this would be even worse?
An upgrade to a N2/4GB would also not help to prevent this situation (and has no HC1-like case), so I'm considering to move up to a H2 with 8GB.
...or hope whatever caused the fs error to happen and spread does not happen again :-/

User avatar
mad_ady
Posts: 6808
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 218 times
Been thanked: 167 times
Contact:

Re: fsck-ing large fs and 2GB mem

Unread post by mad_ady » Fri May 31, 2019 3:35 am

Never seen fsck to require so much ram. Did your disk have unusually many files/directories/symlinks/etc?

fvolk
Posts: 300
Joined: Sun Jun 05, 2016 11:04 pm
languages_spoken: english
ODROIDs: C2, HC1, H2
Has thanked: 0
Been thanked: 11 times
Contact:

Re: fsck-ing large fs and 2GB mem

Unread post by fvolk » Fri May 31, 2019 3:39 am

there were docker containers on it, and AFAIK the image layers use lots of hardlinks to avoid file duplication

elatllat
Posts: 1579
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 24 times
Been thanked: 66 times
Contact:

Re: fsck-ing large fs and 2GB mem

Unread post by elatllat » Mon Jun 03, 2019 11:24 am

fsck works up to 15TB for me, then ext4 requires a 64bit CPU to get any bigger.
I did have to make a big swap file for resize2fs though.

what were your fragmentation and free space numbers?

Post Reply

Return to “General Topics”

Who is online

Users browsing this forum: Google [Bot] and 9 guests