[Solved] systemd[174]: Failed to open serialization file: Read-only file system

[mad_ady]

I rebooted a C1 and it failed to come up. I hooked up a serial console and this is what I get:

Code: Select all

...
[    9.942695@1] hdmitx: cec: CEC init
[    9.946192@3] hdmitx: cec: CEC task process
[    9.946321@1] input: cec_input as /devices/virtual/input/input0
[    9.946473@1] hdmitx: cec: hdmitx_device->cec_init_ready:0x1
[    9.946473@1] ALSA device list:
[  �[   10.006761@1] Freeing unused kernel memory: 224K (c0935000 - c096d000)
Loading, please wait...
starting version 229
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... [   10.821583@2] tvmode set to 1080p
[   10.821583@2] 
[   10.831232@2] don't set the same mode as current.
[   10.889215@0] tvmode set to 1080p
[   10.889215@0] 
[   10.889313@0] don't set the same mode as current.
Tue Jan  1 00:00:00 UTC 1980
done.
Begin: Running /scripts/local-premount ... done.
Begin: Will now check root file system ... fsck from util-linux 2.27.1
[/sbin/fsck.ext4 (1) -- /dev/mmcblk0p2] fsck.ext4 -y -C0 /dev/mmcblk0p2 
e2fsck 1.42.13 (17-May-2015)
rootfs: clean, 217896/950272 files, 1787414/3783808 blocks
done.
[   11.120373@0] force enable DISCARD here for ext4 fs
[   11.123199@0] checked enable EXT4 DISCARD here
[   11.124113@0] EXT4-fs (mmcblk0p2): mounted filesystem without journal. Opts: (null)
done.
Begin: Running /scripts/local-bottom ... done.
Begin: Running /scripts/init-bottom ... done.
[   11.617300@0] systemd[1]: System time before build time, advancing clock.
[   11.701848@0] ip_tables: (C) 2000-2006 Netfilter Core Team
[   11.747999@0] systemd[1]: systemd 237 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid)
[   11.764531@0] systemd[1]: De


Welcome to Ubuntu 18.04.3 LTS!

[   11.791996@0] systemd[1]: Set hostname to <spica>.
[   11.820941@1] systemd[174]: Failed to open serialization file: Read-only file system
[   11.823714@1] systemd[1]: (sd-executor) failed with exit status 1.
[   11.829395@1] system!!!!!!] Failed to start up manager, freezing.
[   11.856622@1] systemd[1]: Freezing execution.
[   15.607589@0] sd: mmc_rescan_try_freq: trying to init card at 300000 Hz
[   18.696572@3] Connect Timeout
[   18.696634@3] Device Not Connected/Responding

I'm using the black emmc (16G), and did fsck on both partitions on a separate system without any errors. Not sure what's wrong. Any ideas? The system has been rock solid for a couple of years (migrated to 18.04 about a year ago).

[mad_ady]

Might be related to this snapd update, will try to remove it:
https://www.digitalocean.com/community/ ... ile-system

[mad_ady]

Ok, these are the steps I took to downgrade snapd. I ran them on a Odroid C2, though they could have run on a C1, XU4, N2, whatever can run armhf code:

Code: Select all

root@bellatrix:/media# mount /dev/mmcblk1p2 /media/usb
root@bellatrix:/media# mount -o bind /dev /media/usb/dev 
root@bellatrix:/media# mount -o bind /proc /media/usb/proc
root@bellatrix:/media# mount -o bind /sys /media/usb/sys
root@bellatrix:/media# chroot /media/usb bash
root@bellatrix:/# apt-cache policy snapd                                       
snapd:                                 
  Installed: 2.42.1+18.04                
  Candidate: 2.42.1+18.04                 
  Version table:
 *** 2.42.1+18.04 500                                                       
        500 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main armhf Packages
        100 /var/lib/dpkg/status                      
     2.37.4+18.04.1 500         
        500 http://ports.ubuntu.com/ubuntu-ports bionic-security/main armhf Packages           
     2.32.5+18.04 500             
        500 http://ports.ubuntu.com/ubuntu-ports bionic/main armhf Packages              
root@bellatrix:/# sudo apt-get install snapd=2.32.5+18.04         
Reading package lists... Done                                           
Building dependency tree                                
Reading state information... Done                     
The following packages were automatically installed and are no longer required:
  libdrm-freedreno1 libllvm6.0 libllvm7                                                  
Use 'sudo apt autoremove' to remove them.                                
The following packages will be DOWNGRADED:                                               
  snapd                                                     
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 8 not upgraded.
Need to get 10.0 MB of archives.                                      
After this operation, 705 kB disk space will be freed.                  
Do you want to continue? [Y/n] y                  
Err:1 http://ports.ubuntu.com/ubuntu-ports bionic/main armhf snapd armhf 2.32.5+18.04
  Could not resolve 'ports.ubuntu.com'
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/s/snapd/snapd_2.32.5+18.04_armhf.deb  Could not resolve 'ports.ubuntu.com'
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
root@bellatrix:/# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=55 time=15.9 ms
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 15.997/15.997/15.997/0.000 ms
root@bellatrix:/# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 127.0.1.1
search mdrjr.local
root@bellatrix:/# vi /etc/resolv.conf
root@bellatrix:/# cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8
search mdrjr.local
root@bellatrix:/# sudo apt-get install snapd=2.32.5+18.04
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libdrm-freedreno1 libllvm6.0 libllvm7
Use 'sudo apt autoremove' to remove them.
The following packages will be DOWNGRADED:
  snapd
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 8 not upgraded.
Need to get 10.0 MB of archives.
After this operation, 705 kB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports bionic/main armhf snapd armhf 2.32.5+18.04 [10.0 MB]
Fetched 10.0 MB in 2s (6,030 kB/s)
E: Can not write log (Is /dev/pts mounted?) - posix_openpt (2: No such file or directory)
dpkg: warning: downgrading snapd from 2.42.1+18.04 to 2.32.5+18.04
(Reading database ... 190421 files and directories currently installed.)
Preparing to unpack .../snapd_2.32.5+18.04_armhf.deb ...
Unpacking snapd (2.32.5+18.04) over (2.42.1+18.04) ...
Setting up snapd (2.32.5+18.04) ...
Installing new version of config file /etc/apparmor.d/usr.lib.snapd.snap-confine.real ...
Installing new version of config file /etc/profile.d/apps-bin-path.sh ...
Installing new version of config file /etc/xdg/autostart/snap-userd-autostart.desktop ...
Processing triggers for gnome-menus (3.13.3-11ubuntu1.1) ...
Processing triggers for mime-support (3.60ubuntu1) ...
Processing triggers for desktop-file-utils (0.23-1ubuntu3.18.04.2) ...
Processing triggers for bamfdaemon (0.5.3+18.04.20180207.2-0ubuntu1) ...
Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
root@bellatrix:/# sync
root@bellatrix:/# exit
exit
root@bellatrix:/media#

I can't test whether this fixed it or not (yet) because junior is asleep in the room where C1 lies. Will give it a try tomorrow.
Hope this is helpful for others facing this issue in the future.

[mad_ady]

Ok, reverting snapd did the trick. I also ran sudo apt-mark hold snapd

[danr677]

Very useful, it saved me. The only difference is that I used a freshly written eMMC, with updated with the latest packages. However, I couldn't use Kodi (the main purpose for this Odroid-C1) with the latest image (v3.1), so I used v3.0 and updated it with apt dist-upgrade and it worked.

[mad_ady]

It's strange that so far it only affects C1. Must be related to the old kernel version...

[Nuems]

I encountered the same (or a very similar) problem recently (no serial-to-usb adapter on my C1, so it was hard to debug with just HDMI output). I solved it with a possibly dirty workaround: In the kernel commandline in boot.ini, I changed "ro" (for the rootfs) to "rw". It might cause all kinds of problems, but it seems to work for the moment.

[mdrjr]

Hello Guys,

I've issued a boot.ini update today to fix it.

In case your board is not working, just replace the boot.ini with this file -> https://github.com/mdrjr/c1_bootini/blo ... r/boot.ini

[safeman]

Ok, these are the steps I took to downgrade snapd. I ran them on a Odroid C2, though they could have run on a C1, XU4, N2, whatever can run armhf code:

I can confirm this also works with Odroid-U3+ just now (using the 16.04 LTS system to downgrade snapd on the non-booting 18.04 LTS after system upgrade).

Thanks!

EDIT: I then discovered I needed to put upgrade of snapd on hold, which I did as follows:

Code: Select all

root@odroid:/# sudo apt-mark hold snapd
snapd set on hold.
root@odroid:/# sudo apt-mark showhold
snapd
root@odroid:/# apt-cache policy snapd
snapd:

  Installed: 2.32.5+18.04
  Candidate: 2.42.1+18.04
  Version table:
     2.42.1+18.04 500
        500 http://ports.ubuntu.com/ubuntu-ports bionic-updates/main armhf Packages
     2.37.4+18.04.1 500
        500 http://ports.ubuntu.com/ubuntu-ports bionic-security/main armhf Packages
 *** 2.32.5+18.04 500
        500 http://ports.ubuntu.com/ubuntu-ports bionic/main armhf Packages
        100 /var/lib/dpkg/status