Kernel panic after upgrade, no boot

Moderators: odroid, mdrjr

Kernel panic after upgrade, no boot

Unread postby Jojo » Wed Nov 29, 2017 5:12 am

Hello together,

once more, I've somehow trashed my system simply by applying upgrades, followed by a reboot. The boot process now hangs with a kernel panic.
Sadly, the kernel panic is not logged into the syslog.

Can someone please help me to solve this? Which information I can provide?

Thanks!
How to ask questions the smart way:
http://www.catb.org/esr/faqs/smart-questions.html
Jojo
 
Posts: 524
Joined: Mon May 18, 2015 12:13 am
Location: Germany
languages_spoken: english, german
ODROIDs: C1, C1+, C2, HC1, HC2, VU8C

Re: Kernel panic after upgrade, no boot

Unread postby meveric » Wed Nov 29, 2017 8:57 am

What image are you using? What Kernel?
In worst case you should be able to flash a new image on a different SD card and copy the file from the bootfs and under /lib/modules and copy them over to your broken image to get it to boot again.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.
User avatar
meveric
 
Posts: 9452
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go

Re: Kernel panic after upgrade, no boot

Unread postby Jojo » Wed Nov 29, 2017 5:24 pm

Hi meveric,

I highly appreciate your willingness to help :) . I gonna donate you another crate of beer, when I am back home...

I am using the official HK Ubuntu image, Kernel version should be 3.14.79-116. At least, this is what I see during the failing boot process (something like "Kernel not tainted" or so...).

In the meantime, I tried to analyze the failing boot process a little bit deeper:
At one point, it says
Code: Select all
[FAILED] Failed to start Load Kernel Modules.

And a little bit later, right before the panic, it says something like:
Code: Select all
BUG failure at security/apparmor/policy.c:160/policy_destroy()!


Does this help in any way?
How to ask questions the smart way:
http://www.catb.org/esr/faqs/smart-questions.html
Jojo
 
Posts: 524
Joined: Mon May 18, 2015 12:13 am
Location: Germany
languages_spoken: english, german
ODROIDs: C1, C1+, C2, HC1, HC2, VU8C

Re: Kernel panic after upgrade, no boot

Unread postby meveric » Wed Nov 29, 2017 6:18 pm

I'm not sure about the module error. Maybe it can't access /lib/modules/ (maybe missing or wrong version).
But I've seen similar output in other Kernel as well, but I think that was part of a systemd message, and had no impact.

Do you have access to the ext4 partition? Can you check if /lib/modules/ is ok?
As I said If you copy all files on bootfs and under /lib/modules from another image you should have a working Kernel again and can try to reinstall the broken Kernel and see if that fix it.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.
User avatar
meveric
 
Posts: 9452
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go

Re: Kernel panic after upgrade, no boot

Unread postby Jojo » Wed Nov 29, 2017 6:55 pm

meveric wrote:I'm not sure about the module error. Maybe it can't access /lib/modules/ (maybe missing or wrong version).
But I've seen similar output in other Kernel as well, but I think that was part of a systemd message, and had no impact.

Do you have access to the ext4 partition? Can you check if /lib/modules/ is ok?
As I said If you copy all files on bootfs and under /lib/modules from another image you should have a working Kernel again and can try to reinstall the broken Kernel and see if that fix it.


Ok, as a first step (a little bit easier and faster than flashing a new image) I downloaded boot- and modules-stuff from here http://deb.odroid.in/c2/pool/main/l/linux-source-3.14.79-116/ and replaced the whole boot partition content and copied the corresponding modules-subdirectory into my local modules directory.
Sadly, this approach did not work (I also tried with the 3.14.79-115 version). Now the question: is this approach equivalent to what you suggested or did I miss something?

EDIT:
I just flashed a fresh image on another SD card, made a complete update/upgrade/dist-upgrade and copied the boot-content and the modules over to the "broken" one. Still not booting :( . So this time it seems not to be related to a broken kernel/modules combination... :(
How to ask questions the smart way:
http://www.catb.org/esr/faqs/smart-questions.html
Jojo
 
Posts: 524
Joined: Mon May 18, 2015 12:13 am
Location: Germany
languages_spoken: english, german
ODROIDs: C1, C1+, C2, HC1, HC2, VU8C

Re: Kernel panic after upgrade, no boot

Unread postby meveric » Wed Nov 29, 2017 8:18 pm

I guess extracting should work too.
Although you're missing the correct uInitrd as this is created only if you install it.
It also means you don't know it's a "working" Kernel, as when you use a new image and know that this is booting, you are sure the Kernel is fine.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.
User avatar
meveric
 
Posts: 9452
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go

Re: Kernel panic after upgrade, no boot

Unread postby Jojo » Wed Nov 29, 2017 9:36 pm

meveric wrote:I guess extracting should work too.
Although you're missing the correct uInitrd as this is created only if you install it.
It also means you don't know it's a "working" Kernel, as when you use a new image and know that this is booting, you are sure the Kernel is fine.

Well, I tried both methods, both didn't work.

In the meanwhile, I came a step further. I moved apparmor out of the way by just renaming it in /etc/init.d/. This lead to not longer having a kernel panic, but the boot process can still not be finished. In contrast, now I see many more errors, please have a look at the picture:
IMG_20171129_133329.jpg
IMG_20171129_133329.jpg (267.2 KiB) Viewed 1252 times

Any idea about that?!
The boot hangs here (blue LED keeps blinking), and I don't get any promt.

In case that an ODROID admin comes across this: this is the fourth time, I trashed my system just by updating and rebooting. Something is telling me, that this is nor normal...
How to ask questions the smart way:
http://www.catb.org/esr/faqs/smart-questions.html
Jojo
 
Posts: 524
Joined: Mon May 18, 2015 12:13 am
Location: Germany
languages_spoken: english, german
ODROIDs: C1, C1+, C2, HC1, HC2, VU8C

Re: Kernel panic after upgrade, no boot

Unread postby odroid » Thu Nov 30, 2017 9:52 am

I've run this system update process almost every day but I couldn't reproduce the issue.
https://wiki.odroid.com/odroid-c2/os_im ... and_system

Blue LED keeps flashing when the Linux kernel is alive regardless of the system init process.
I think some portion of the root file system corrupted.
User avatar
odroid
Site Admin
 
Posts: 29098
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID

Re: Kernel panic after upgrade, no boot

Unread postby meveric » Thu Nov 30, 2017 4:55 pm

The picture above only shows that there's a bunch of stuff that couldn't be started, but if it now doesn't show the panic then maybe it really was an issue with apparmor.
Can you access the log files to see what could be the issue?
Maybe it tells you what's missing. My guess maybe a file got corrupted and causes apparmor to fail and that now affects the entire system.
If even copying the Kernel + modules from a working image does not work, then it's not a Kernel related issue.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.
User avatar
meveric
 
Posts: 9452
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go

Re: Kernel panic after upgrade, no boot

Unread postby Jojo » Thu Nov 30, 2017 7:58 pm

meveric wrote:The picture above only shows that there's a bunch of stuff that couldn't be started, but if it now doesn't show the panic then maybe it really was an issue with apparmor.
Can you access the log files to see what could be the issue?
Maybe it tells you what's missing. My guess maybe a file got corrupted and causes apparmor to fail and that now affects the entire system.
If even copying the Kernel + modules from a working image does not work, then it's not a Kernel related issue.


Hm, I have no idea, which log file could be the correct one to look for the root of these errors :roll: . Is it the syslog?

@odroid
ok, thanks for the information. It was not meant offensive, but I think you agree, that it is kind of frustrating, if one does an update in the way the OS proposes, and everything gets ****ed up (multiple times...). I would really like to find a systematical error in the way I do it, but I really do nothing special. Again, not offensive, I am just getting nervous... ;) . Do you use the graphical update management, or do you use apt-get commands (update/upgrade/dist-upgrade)?
How to ask questions the smart way:
http://www.catb.org/esr/faqs/smart-questions.html
Jojo
 
Posts: 524
Joined: Mon May 18, 2015 12:13 am
Location: Germany
languages_spoken: english, german
ODROIDs: C1, C1+, C2, HC1, HC2, VU8C

Re: Kernel panic after upgrade, no boot

Unread postby meveric » Thu Nov 30, 2017 9:39 pm

Jojo wrote:Hm, I have no idea, which log file could be the correct one to look for the root of these errors :roll: . Is it the syslog?

Yes syslog is probably the first goto to see if something is wrong.
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.
User avatar
meveric
 
Posts: 9452
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go

Re: Kernel panic after upgrade, no boot

Unread postby Jojo » Thu Nov 30, 2017 10:31 pm

OMG... now it's getting even stranger!
What I did was:
- taking the SD-card to my PC
- shrinking down the rootfs partition to 9GB
- making a backup of that "broken" image (dd with bs=1M and count=10000)
- resizing rootfs back to fit the whole disk (~32GB)

Now it boots up again!!! WTH?!
I made a complete update/upgrade/dist-upgrade again and actually the system is at kernel version 3.14.17-116 and is running as stable as before :? .

Fun fact: if I flash the backup of my broken system onto another SD card, it keeps failing as described above. If I have time (and the willingness) I can try to analyze the problem again. But actually I am more than happy, that the system is up again. Although there is some "bad taste" left...
How to ask questions the smart way:
http://www.catb.org/esr/faqs/smart-questions.html
Jojo
 
Posts: 524
Joined: Mon May 18, 2015 12:13 am
Location: Germany
languages_spoken: english, german
ODROIDs: C1, C1+, C2, HC1, HC2, VU8C

Re: Kernel panic after upgrade, no boot

Unread postby meveric » Fri Dec 01, 2017 12:53 am

Witchcraft I tell you witchcraft!!!
Donate to support my work on the ODROID GameStation Turbo Image for U2/U3 XU3/XU4 X2 X C1 as well as many other releases.
Check out the Games and Emulators section to find some of my work or check the files in my repository to find the software i build for ODROIDs.
If you want to add my repository to your image read my HOWTO integrate my repo into your image.
User avatar
meveric
 
Posts: 9452
Joined: Mon Feb 25, 2013 2:41 pm
languages_spoken: german, english
ODROIDs: X2, U2, U3, XU-Lite, XU3, XU3-Lite, C1, XU4, C2, C1+, XU4Q, HC1, N1, Go

Re: Kernel panic after upgrade, no boot

Unread postby Jojo » Fri Dec 01, 2017 1:05 am

meveric wrote:Witchcraft I tell you witchcraft!!!

I tell you something:
- if something works as it always worked, thats ok for me
- if something doesn't work as before, thats still ok and I try to fix it
- if something doesn't work and I can not fix it, thats a pity, but still ok
- but if something didn't work but suddenly it does without I fixed it, THIS drives me crazy :roll: !

Thank you anyway, maybe I'll come back here and post the output of the syslog of my "broken" backup... Lord, I hate when things like that happen...
How to ask questions the smart way:
http://www.catb.org/esr/faqs/smart-questions.html
Jojo
 
Posts: 524
Joined: Mon May 18, 2015 12:13 am
Location: Germany
languages_spoken: english, german
ODROIDs: C1, C1+, C2, HC1, HC2, VU8C


Return to Issues

Who is online

Users browsing this forum: No registered users and 2 guests