Can't boot after dist-upgrade (odroid-c1-20170221)

Moderators: odroid, mdrjr

Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby tosiara » Thu Apr 06, 2017 5:10 pm

I flashed a fresh ubuntu-16.04.2-minimal-odroid-c1-20170221.img.xz and tried to do dist-upgrade immediately over ssh:

Code: Select all
root@odroid:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  linux-image-3.10.104-186
Use 'apt autoremove' to remove it.
The following NEW packages will be installed:
  linux-image-3.10.104-186
The following packages have been kept back:
  linux-image-c1
The following packages will be upgraded:
  bootini init init-system-helpers makedev nano resolvconf wget
7 upgraded, 1 newly installed, 0 to remove and 1 not upgraded.
Need to get 12.4 MB of archives.
After this operation, 35.3 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf init-system-helpers all 1.29ubuntu4 [32.3 kB]
Get:2 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf init armhf 1.29ubuntu4 [4612 B]
Get:3 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf makedev all 2.3.1-93ubuntu2~ubuntu16.04.1 [24.4 kB]
Get:4 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf resolvconf all 1.78ubuntu4 [51.1 kB]
Get:5 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf nano armhf 2.5.3-2ubuntu2 [174 kB]
Get:6 http://deb.odroid.in/c1 xenial/main armhf linux-image-3.10.104-186 armhf 20170320 [11.8 MB]
Get:7 http://ports.ubuntu.com/ubuntu-ports xenial-updates/main armhf wget armhf 1.17.1-1ubuntu1.2 [281 kB]
Get:8 http://deb.odroid.in/c1 xenial/main armhf bootini armhf 20170303-35 [4166 B]
Fetched 12.4 MB in 7s (1650 kB/s)
Preconfiguring packages ...
(Reading database ... 23547 files and directories currently installed.)
Preparing to unpack .../init-system-helpers_1.29ubuntu4_all.deb ...
Unpacking init-system-helpers (1.29ubuntu4) over (1.29ubuntu3) ...
Processing triggers for man-db (2.7.5-1) ...
Setting up init-system-helpers (1.29ubuntu4) ...
(Reading database ... 23547 files and directories currently installed.)
Preparing to unpack .../init_1.29ubuntu4_armhf.deb ...
Unpacking init (1.29ubuntu4) over (1.29ubuntu3) ...
Setting up init (1.29ubuntu4) ...
Selecting previously unselected package linux-image-3.10.104-186.
(Reading database ... 23547 files and directories currently installed.)
Preparing to unpack .../linux-image-3.10.104-186_20170320_armhf.deb ...
Examining /etc/kernel/preinst.d/
run-parts: executing /etc/kernel/preinst.d/remove-old-on-upgrade 3.10.104-186 /boot/uImage-3.10.104-186
Done.
Unpacking linux-image-3.10.104-186 (20170320) ...
Replacing files in old package linux-image-3.10.104-184 (20170220) ...
Preparing to unpack .../makedev_2.3.1-93ubuntu2~ubuntu16.04.1_all.deb ...
Unpacking makedev (2.3.1-93ubuntu2~ubuntu16.04.1) over (2.3.1-93ubuntu1) ...
Preparing to unpack .../resolvconf_1.78ubuntu4_all.deb ...
Unpacking resolvconf (1.78ubuntu4) over (1.78ubuntu2) ...
Preparing to unpack .../nano_2.5.3-2ubuntu2_armhf.deb ...
Unpacking nano (2.5.3-2ubuntu2) over (2.5.3-2ubuntu1) ...
Preparing to unpack .../wget_1.17.1-1ubuntu1.2_armhf.deb ...
Unpacking wget (1.17.1-1ubuntu1.2) over (1.17.1-1ubuntu1.1) ...
Preparing to unpack .../bootini_20170303-35_armhf.deb ...
Unpacking bootini (20170303-35) over (20170204-33) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu16) ...
Setting up linux-image-3.10.104-186 (20170320) ...

 Hmm. There is a symbolic link /lib/modules/3.10.104-186/build
 However, I can not read it: No such file or directory
 Therefore, I am deleting /lib/modules/3.10.104-186/build


 Hmm. The package shipped with a symbolic link /lib/modules/3.10.104-186/source
 However, I can not read the target: No such file or directory
 Therefore, I am deleting /lib/modules/3.10.104-186/source

Running depmod.
Examining /etc/kernel/postinst.d.
run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 3.10.104-186 /boot/uImage-3.10.104-186
run-parts: executing /etc/kernel/postinst.d/copy_dtb 3.10.104-186 /boot/uImage-3.10.104-186
run-parts: executing /etc/kernel/postinst.d/initramfs-tools 3.10.104-186 /boot/uImage-3.10.104-186
update-initramfs: Generating /boot/initrd.img-3.10.104-186
run-parts: executing /etc/kernel/postinst.d/uImage 3.10.104-186 /boot/uImage-3.10.104-186
run-parts: executing /etc/kernel/postinst.d/uInitrd 3.10.104-186 /boot/uImage-3.10.104-186
run-parts: executing /etc/kernel/postinst.d/unattended-upgrades 3.10.104-186 /boot/uImage-3.10.104-186
Setting up makedev (2.3.1-93ubuntu2~ubuntu16.04.1) ...
Setting up resolvconf (1.78ubuntu4) ...
Setting up nano (2.5.3-2ubuntu2) ...
Setting up wget (1.17.1-1ubuntu1.2) ...
Setting up bootini (20170303-35) ...
update-initramfs: Generating /boot/initrd.img-3.10.104-184
Image Name:   uInitrd
Created:      Thu Apr  6 07:22:23 2017
Image Type:   ARM Linux RAMDisk Image (uncompressed)
Data Size:    4753952 Bytes = 4642.53 kB = 4.53 MB
Load Address: 00000000
Entry Point:  00000000
usermod: user 'odroid' does not exist

A new boot.ini is installed.
Any changes to boot.ini is lost, such as display configuration
For reference your old boot.ini is saved to /media/boot/boot.ini.old
Processing triggers for resolvconf (1.78ubuntu4) ...
root@odroid:~# reboot
ssh: Connection to root@odroid:22 exited: Remote closed the connection

root@router:~# ssh odroid
ssh: Connection to root@odroid:22 exited: Connect failed: Host is unreachable
root@router:~# ping odroid
PING odroid (192.168.1.10): 56 data bytes

The device never comes online. And I can confirm that "odroid" user is really missing, only root was available.

Any one came across such issue with ubuntu-16.04.2-minimal-odroid-c1-20170221.img.xz ?
User avatar
tosiara
 
Posts: 21
Joined: Wed Sep 17, 2014 11:14 pm
languages_spoken: english
ODROIDs: U3, C1

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby odroid » Thu Apr 06, 2017 5:55 pm

How is the blue LED?
Is it flashing?
User avatar
odroid
Site Admin
 
Posts: 22295
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby tosiara » Fri Apr 07, 2017 6:16 pm

No, it is ON and not flashing
User avatar
tosiara
 
Posts: 21
Joined: Wed Sep 17, 2014 11:14 pm
languages_spoken: english
ODROIDs: U3, C1

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby odroid » Fri Apr 07, 2017 6:57 pm

I think some portion of the FAT partition could be corrupted in the update process.
We will check this issue by early next week.
User avatar
odroid
Site Admin
 
Posts: 22295
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby moon.linux » Sat Apr 08, 2017 11:49 pm

I have tried the step at my end and I am not able to reproducer the issue.
Attachments
odroid-c1-miniboot.txt
odroid c1 bootlogs.
(163.2 KiB) Downloaded 14 times
moon.linux
 
Posts: 728
Joined: Thu Oct 02, 2014 11:42 pm
languages_spoken: english

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby tosiara » Wed Apr 12, 2017 4:14 pm

moon.linux wrote:I have tried the step at my end and I am not able to reproducer the issue.

Good idea to hook serial console!
I tried to flash a fresh image once again and did dist-upgrade and, unfortunately, could not reproduce either.

But I noticed the same microSD related error during upgrade as in your log:

Code: Select all
Unpacking linux-image-3.10.104-186 (20170320) ...
Replacing files in old package linux-image-3.10.104-184 (20170220) ...
Preparing to unpack .../makedev_2.3.1-93ubuntu2~ubuntu16.04.1_all.deb ...
Unpacking makedev (2.3.1-93ubuntu2~ubuntu16.04.1) over (2.3.1-93ubuntu1) ...
Preparing to unpack .../resolvconf_1.78ubuntu4_all.deb ...
Unpacking resolvconf (1.78ubuntu4) over (1.78ubuntu2) ...
[  632.911856@1] mmcblk0: r/w command failed, status = 0x80000b00
[  632.912196@1] end_request: I/O error, dev mmcblk0, sector 15564792
[  632.918434@1] Buffer I/O error on device mmcblk0, logical block 1945599
[  632.918676@1] systemd-gpt-auto-generator[3765]: /dev/mmcblk0: failed to probe: Input/output error
Preparing to unpack .../nano_2.5.3-2ubuntu2_armhf.deb ...
Unpacking nano (2.5.3-2ubuntu2) over (2.5.3-2ubuntu1) ...
Preparing to unpack .../wget_1.17.1-1ubuntu1.2_armhf.deb ...
Unpacking wget (1.17.1-1ubuntu1.2) over (1.17.1-1ubuntu1.1) ...
Preparing to unpack .../bootini_20170303-35_armhf.deb ...
Unpacking bootini (20170303-35) over (20170204-33) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for ureadahead (0.100.0-19) ...
Processing triggers for systemd (229-4ubuntu16) ...
[  668.279103@1] mmcblk0: r/w command failed, status = 0x80000b00
[  668.279409@1] end_request: I/O error, dev mmcblk0, sector 15564792
[  668.285571@1] Buffer I/O error on device mmcblk0, logical block 1945599
[  668.292738@3] systemd-gpt-auto-generator[4214]: /dev/mmcblk0: failed to probe: Input/output error
Setting up linux-image-3.10.104-186 (20170320) ...


I know that those kind of errors appear randomly pretty always with almost every microSD card. Could this I/O error cause random kernel corruption during upgrade?
User avatar
tosiara
 
Posts: 21
Joined: Wed Sep 17, 2014 11:14 pm
languages_spoken: english
ODROIDs: U3, C1

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby crashoverride » Fri Apr 21, 2017 7:35 pm

Just adding a "me too" to this. I was hit by this today. My image was trashed with corruption on the ext4 partition. Prior to this condition, there was some very long "unattended upgrade" process that lasted for what seems like an hour. After doing a normal "apt-get upgrade", I saw reference to a kernel 4.x initrd image being created too.

I have been investigating the mmc error. It seems to be a bug somewhere. In my instance, the last 8 sectors of the SD card are not readable.
Code: Select all
Disk /dev/mmcblk0: 14.9 GiB, 16021192704 bytes, 31291392 sectors

Code: Select all
[   21.759854] mmcblk0: r/w command failed, status = 0x80000b00
[   21.760122] end_request: I/O error, dev mmcblk0, sector 31291384
[   21.766286] Buffer I/O error on device mmcblk0, logical block 3911423

(31291392 sectors - 31291384 = 8 sectors)

I inserted the SD card into a Linux PC and resized the file system to leave the last 8 sectors unused. I then shrank the partition to leave the last 8 sectors unused (8 sectors * 512b/sector = 4096b = 4k).

Mmc errors are still reported during boot. However, they no longer occur during normal use (so far). I attribute the errors seen to the kernel and ubuntu looking for GPT backup tables at the end of the media (that are not present).
crashoverride
 
Posts: 2565
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby meveric » Fri Apr 21, 2017 9:17 pm

crashoverride wrote:I have been investigating the mmc error. It seems to be a bug somewhere. In my instance, the last 8 sectors of the SD card are not readable.
Code: Select all
Disk /dev/mmcblk0: 14.9 GiB, 16021192704 bytes, 31291392 sectors

Code: Select all
[   21.759854] mmcblk0: r/w command failed, status = 0x80000b00
[   21.760122] end_request: I/O error, dev mmcblk0, sector 31291384
[   21.766286] Buffer I/O error on device mmcblk0, logical block 3911423

(31291392 sectors - 31291384 = 8 sectors)

I inserted the SD card into a Linux PC and resized the file system to leave the last 8 sectors unused. I then shrank the partition to leave the last 8 sectors unused (8 sectors * 512b/sector = 4096b = 4k).

Mmc errors are still reported during boot. However, they no longer occur during normal use (so far). I attribute the errors seen to the kernel and ubuntu looking for GPT backup tables at the end of the media (that are not present).

I think that's the long standing issue with the SD card support. Apparently it does happen more random nowadays.
The resize script that resizes the rootfs to maximum size kinda causes this issue since it resizes the rootfs into the space of the sectors it can not access.
On my Debian Jessie images, I tried to leave a couple sectors free at the end of the SD card (i believe it was 10 or so) to avoid that issue, but even that is not always working.
Different brands of SD cards, have different sizes of sectors that fail to be addressed it seems.
User avatar
meveric
 
Posts: 6750
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: ODROID-X2,ODROID-U2,ODROID-U3,ODROID-XU-Lite, ODROID-XU3, ODROID-XU3-Lite, ODROID-C1, ODROID-XU4, ODROID-C2

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby tosiara » Fri Apr 21, 2017 9:23 pm

crashoverride wrote:I have been investigating the mmc error. It seems to be a bug somewhere. In my instance, the last 8 sectors of the SD card are not readable.

Interesting is that I have exactly the same issue with those 8 sectors at the end. Reported it separate here: viewtopic.php?f=117&t=26687
User avatar
tosiara
 
Posts: 21
Joined: Wed Sep 17, 2014 11:14 pm
languages_spoken: english
ODROIDs: U3, C1

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby crashoverride » Fri Apr 21, 2017 10:39 pm

In my log are the following errors:
Code: Select all
[  232.642081] mmcblk0: r/w command failed, status = 0x80000b00
[  232.642306] end_request: I/O error, dev mmcblk0, sector 31291384
[  232.649390] quiet_error: 24 callbacks suppressed
[  232.649411] Buffer I/O error on device mmcblk0, logical block 3911423
[  232.664870] systemd-gpt-auto-generator[3840]: /dev/mmcblk0: failed to probe: Input/output error
[  237.083538] mmcblk0: r/w command failed, status = 0x80000b00
[  237.083763] end_request: I/O error, dev mmcblk0, sector 31291384
[  237.090763] Buffer I/O error on device mmcblk0, logical block 3911423
[  237.099602] systemd-gpt-auto-generator[3865]: /dev/mmcblk0: failed to probe: Input/output error
[  238.461151] mmcblk0: r/w command failed, status = 0x80000b00
[  238.461420] end_request: I/O error, dev mmcblk0, sector 31291384
[  238.471576] Buffer I/O error on device mmcblk0, logical block 3911423
[  238.475524] systemd-gpt-auto-generator[3890]: /dev/mmcblk0: failed to probe: Input/output error
[  239.748341] mmcblk0: r/w command failed, status = 0x80000b00
[  239.748557] end_request: I/O error, dev mmcblk0, sector 31291384
[  239.754727] Buffer I/O error on device mmcblk0, logical block 3911423
[  239.762116] systemd-gpt-auto-generator[3920]: /dev/mmcblk0: failed to probe: Input/output error
[  243.452003] mmcblk0: r/w command failed, status = 0x80000b00
[  243.452228] end_request: I/O error, dev mmcblk0, sector 31291384
[  243.459309] Buffer I/O error on device mmcblk0, logical block 3911423
[  243.465573] systemd-gpt-auto-generator[3954]: /dev/mmcblk0: failed to probe: Input/output error

It is clearly evident that "systemd-gpt-auto-generator" is attempting to search for GPTs. Since a GPT is always placed at the end of a volume (in the sectors that cannot be read), it produces errors and brings the entire system to a crawl. I could not find a way to disable this "generator". This led me to just delete it:
Code: Select all
sudo rm /lib/systemd/system-generators/systemd-gpt-auto-generator


[edit]
I copied a file larger than the sd card over the network to the sd card. After claiming all the free space (not including root reserved), there were no mmc errors.

I also disabled the "unattended-upgrades" to prevent it from running:
Code: Select all
sudo dpkg-reconfigure -plow unattended-upgrades
crashoverride
 
Posts: 2565
Joined: Tue Dec 30, 2014 8:42 pm
languages_spoken: english
ODROIDs: C1

Re: Can't boot after dist-upgrade (odroid-c1-20170221)

Unread postby moon.linux » Sat Apr 22, 2017 12:43 pm

it seen that the mmc driver is not able to find the correct block size to read/write the data.

Code: Select all
https://www.freedesktop.org/software/systemd/man/systemd-gpt-auto-generator.html


so we should not disable or remove the systemd feature, but work possible on fixing the driver.
some interesting discussion on the this in below link.

Code: Select all
https://github.com/systemd/systemd/issues/2483
moon.linux
 
Posts: 728
Joined: Thu Oct 02, 2014 11:42 pm
languages_spoken: english


Return to Ubuntu

Who is online

Users browsing this forum: Brian.K, Heise IT-Markt [Crawler] and 6 guests