Oops: 96000004 [#1] PREEMPT SMP

Post Reply
MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

I am running the Debian 10 netboot_installer on my Odroid HC4. It is headless. Just accessing via SSH.

Sometimes the machine freez. No reaction on ping or ssh. Did not tried to plug in a monitor because I have no "mobile" monitor available.
The blue LED is off. The red LED is permanent red - no blinking.
I am not sure but most of the time a rsync job was running in a tmux. Just a guess.

This happend some times in the last days. After rebooting (power off and on) I could not find a helpfull message in "journalctl -b". Maybe this is the wrong way to read the last log messages?

But today I got an output on my SSH terminal before the freeze.

Code: Select all

Message from syslogd@OLAF at Dec 18 23:44:51 ...
 kernel:[86389.433663] Internal error: Oops: 96000004 [#1] PREEMPT SMP

Message from syslogd@OLAF at Dec 18 23:44:51 ...
 kernel:[86389.476573] Code: f8627821 f100003f 8b000020 9a811001 (f9400020) 
What can I do know about it?

User avatar
odroid
Site Admin
Posts: 37363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1780 times
Been thanked: 1138 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by odroid »

Can you try stock 4.9 kernel to narrow down root causes if you are running 5.9 or 5.10 kernel now?

MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

odroid wrote:
Mon Dec 21, 2020 11:35 am
Can you try stock 4.9 kernel to narrow down root causes if you are running 5.9 or 5.10 kernel now?
I would try if I would knew how.

On a x86/amd64 Debian 10 System I would know in which wikis to look and figuring out myself.

But depending on my not so good experience with the HC4 I do not know how. I use the netboot_default Debian image (by @tobetter if I understand correkt). I know there are some odroid-hc4-ARM-specific modifications in that kernel.
I don't want to brick my system using another unsupported kernel. I don't use C4 (a maker device) but HC4. My HC4 is still in productive use so I need to be carefull.

User avatar
odroid
Site Admin
Posts: 37363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1780 times
Been thanked: 1138 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by odroid »

This pre-made OS image could be flashed to your spare SD card via Etcher.
https://wiki.odroid.com/odroid-c4/os_im ... e/20201214

MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

odroid wrote:
Mon Dec 21, 2020 7:00 pm
This pre-made OS image could be flashed to your spare SD card via Etcher.
https://wiki.odroid.com/odroid-c4/os_im ... e/20201214
This is a bit more then just another kernel! :D

I have no way to do it with this machine. It is in productive state. Sorry I can not help you here. It is not possible to my to shutdown my productive server and run a foreign linux on it for some days with a rsync-job just to find out if the error ocours again or not.

Can you give some informations about what the root of the problem is?

User avatar
odroid
Site Admin
Posts: 37363
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1780 times
Been thanked: 1138 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by odroid »

I have no idea when/why the SMP error appears either.
Can you keep monitoring the CPU and RAM usages via 'htop' on a ssh session while running a heavy rsync job?

MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

Ok, I can do that. But what does mean "heavy"? ;)

Is my problem related to this?
viewtopic.php?p=315350#p315350

elatllat
Posts: 1858
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2, C4, N2+, HC4
Has thanked: 60 times
Been thanked: 132 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by elatllat »

MoonKid wrote:
Sat Dec 19, 2020 8:01 am
.. What can I do ... about it?
What is the output of

Code: Select all

uname -r
? and what is the exact OS.img you are using?
MoonKid wrote:
Tue Dec 22, 2020 10:55 pm
Is my problem related to this?
viewtopic.php?p=315350#p315350
No. (assuming you are not using USB, and the kernel already has that fixed anyway)

If you don't want to test your workload on different OSs,
or build your own kernel,
I can send you a zip of /boot so you can try a different kernel with minimal effort.

or if you can share the minimal steps to reproduce the issue I or others can use that to find a solution for you.

MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

That is the greeting screen when login via SSH including the uname -r.

Code: Select all

Linux OLAF 5.9.0-odroid-arm64 #1 SMP PREEMPT Debian 5.9.14-202012161715~buster (2020-12-16) aarch64
 _   _               _ _                        _ 
| | | | __ _ _ __ __| | | _____ _ __ _ __   ___| |
| |_| |/ _` | '__/ _` | |/ / _ \ '__| '_ \ / _ \ |
|  _  | (_| | | | (_| |   <  __/ |  | | | |  __/ |
|_| |_|\__,_|_|  \__,_|_|\_\___|_|  |_| |_|\___|_|
                                                  
  ___  ____  ____   ___ ___ ____        _   _  ____ _  _   
 / _ \|  _ \|  _ \ / _ \_ _|  _ \      | | | |/ ___| || |  
| | | | | | | |_) | | | | || | | |_____| |_| | |   | || |_ 
| |_| | |_| |  _ <| |_| | || |_| |_____|  _  | |___|__   _|
 \___/|____/|_| \_\\___/___|____/      |_| |_|\____|  |_|  
                                                           

Welcome to Debian GNU/Linux 10 (buster)
Wednesday, 23 December 2020, 02:41:58 PM
Up time:     2 days, 02:11:48
Free memory: 1099080 / 3723616 kB
IP:          192.168.178.40

[!] Please visit 'https://launchpad.net/odroid-image' to report a bug

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Dec 23 12:25:10 2020 from 192.168.178.37
admin@OLAF:~$ uname -r
5.9.0-odroid-arm64
My image? Ask "tobetter". I assumed it was officla Debian 10 - but it is not because it has odroid specific modifications (from tobetter).
In petiboot I typed "netboot_default" and then selected the Debian 10 installation menu point in the petiboot menu. Thats it.

elatllat
Posts: 1858
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2, C4, N2+, HC4
Has thanked: 60 times
Been thanked: 132 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by elatllat »

Did you

Code: Select all

apt update && apt upgrade && shutdown -r now
? (5.9.0 is 16 releases old and EOL)
It the issue persists the 2 options I mentioned (try a mainline kernel or share a test script) are the best ways forward.

MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

Please ask @tobetter for details. I described which image I am using here. Maybe you missunderstand something.
What does it mean 16 relases old? You mean 6.15 or 5.15 is the current fresh release?

Code: Select all

OK:1 http://deb.debian.org/debian buster InRelease
Holen:2 http://deb.debian.org/debian buster-updates InRelease [51,9 kB]
Holen:3 http://security.debian.org/debian-security buster/updates InRelease [65,4 kB]
Holen:4 http://security.debian.org/debian-security buster/updates/main Sources [158 kB]
Holen:5 http://security.debian.org/debian-security buster/updates/main arm64 Packages [251 kB]
Holen:6 http://security.debian.org/debian-security buster/updates/main Translation-en [141 kB]
Holen:7 http://deb.debian.org/debian buster-updates/main Sources.diff/Index [4.672 B]
Holen:8 http://deb.debian.org/debian buster-updates/main arm64 Packages.diff/Index [4.672 B]   
Holen:9 http://deb.debian.org/debian buster-updates/main Sources 2020-12-24-1401.30.pdiff [415 B]  
Holen:9 http://deb.debian.org/debian buster-updates/main Sources 2020-12-24-1401.30.pdiff [415 B]   
Holen:10 http://deb.debian.org/debian buster-updates/main arm64 Packages 2020-12-24-1401.30.pdiff [286 B]
Holen:10 http://deb.debian.org/debian buster-updates/main arm64 Packages 2020-12-24-1401.30.pdiff [286 B]
Holen:11 http://ppa.linuxfactory.or.kr buster InRelease [21,2 kB]
Holen:12 http://ppa.linuxfactory.or.kr buster/main arm64 Packages [79,4 kB]
Holen:13 http://ppa.linuxfactory.or.kr buster/contrib arm64 Packages [518 B]
Es wurden 778 kB in 4 s geholt (218 kB/s).
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
Aktualisierung für 4 Pakete verfügbar. Führen Sie »apt list --upgradable« aus, um sie anzuzeigen.
Paketlisten werden gelesen... Fertig
Abhängigkeitsbaum wird aufgebaut.       
Statusinformationen werden eingelesen.... Fertig
Paketaktualisierung (Upgrade) wird berechnet... Fertig
Die folgenden NEUEN Pakete werden installiert:
  linux-image-5.10.0-odroid-arm64
Die folgenden Pakete werden aktualisiert (Upgrade):
  linux-image-5.9.0-odroid-arm64 linux-image-generic odroid-base tzdata
4 aktualisiert, 1 neu installiert, 0 zu entfernen und 0 nicht aktualisiert.
Es müssen 55,8 MB an Archiven heruntergeladen werden.
Nach dieser Operation werden 121 MB Plattenplatz zusätzlich benutzt.
Möchten Sie fortfahren? [J/n] j
Thei kernel image is loaded from

Code: Select all

Holen:3 http://ppa.linuxfactory.or.kr buster/main arm64 linux-image-5.10.0-odroid-arm64 arm64 5.10.1-202012181454~buster [27,3 MB]
Please also look into the official Debian ressources, too.
https://tracker.debian.org/pkg/linux-signed-arm64

And it is still unclear for me which goal do you want to reach using another kernel? Is this for testing/debuging or do you assume the problem is fixed in that other kernel?

MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

Last night again I had two freezes (rsync from sdb1 to sda1 via ssh-tmux-session). But the tmux session was detached - no output visible.
I did a third run last night and left the session open. Please see the screenshot - rsync on the left, htop on the right.
This time there is no error message.
Image

EDIT:
Here you see a session without tmux. In the upper right (htop) you see the kernel error message.
Image

klippe
Posts: 11
Joined: Thu Jan 28, 2021 3:55 am
languages_spoken: english
ODROIDs: Odroid HC4
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by klippe »

Any solution for the issue described here?

I've just stumped upon exactly the same problem. I'm using Debian 10 installed via netboot with OMV on top of it.

uname -r gives: 5.10.0-odroid-arm64


Seems I can reproduce the issue by simply uploading files to NAS (via SMB/CIFS set up in OMV). From what I see the problem happens when a bigger file (video - a few hundreds of megabytes) is being uploaded.

Here's the output from the ssh session that was running `top` while the system went down.

Code: Select all

top - 22:27:39 up 6 min,  1 user,  load average: 4.20, 1.44, 0.55
Tasks: 215 total,   4 running, 210 sleeping,   0 stopped,   1 zombie
%Cpu(s): 12.1 us, 20.8 sy,  0.0 ni, 48.3 id, 13.4 wa,  0.7 hi,  4.8 si,  0.0 st
MiB Mem :   3635.8 total,    141.6 free,    206.5 used,   3287.8 buff/cache
MiB Swap:    976.0 total,    976.0 free,      0.0 used.   3227.8 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                          
 3072 ext       20   0   63536  17300  14344 R  24.5   0.5   0:16.84 smbd                                                             
 6083 root      20   0   28264  19980  10520 R  13.2   0.5   0:00.40 python3                                                          
  878 root      20   0       0      0      0 S   3.6   0.0   0:00.50 kswapd0                                                          
 4789 root      20   0   13276   8416   5396 S   1.0   0.2   0:00.17 apt-get                                                          
 4639 root      20   0       0      0      0 I   0.7   0.0   0:01.90 kworker/u8:14-events_power_efficient                             
 4646 ext       20   0   10700   3232   2552 R   0.7   0.1   0:00.54 top                                                              
    1 root      20   0  165560   9836   7044 S   0.3   0.3   0:03.32 systemd                                                          
    7 root      20   0       0      0      0 I   0.3   0.0   0:02.00 kworker/u8:0-btrfs-endio-meta                                    
  504 root      20   0       0      0      0 I   0.3   0.0   0:01.23 kworker/u8:3-kcryptd/252:1                                       
 2275 root      20   0  329572  23408   9204 S   0.3   0.6   0:01.79 odroid_homeclou                                                  
 4633 root      20   0       0      0      0 I   0.3   0.0   0:01.66 kworker/u8:8-kcryptd/252:1                                       
 4635 root      20   0       0      0      0 I   0.3   0.0   0:00.97 kworker/u8:10-btrfs-endio-write                                  
    2 root      20   0       0      0      0 S   0.0   0.0   0:00.01 kthreadd                                                         
    3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_gp                                                           
    4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 rcu_par_gp     
                                                      
Message from syslogd@sandworm-nas at Feb  2 22:27:41 ...
 kernel:[  406.549658] Internal error: Oops: 96000006 [#1] PREEMPT SMP

Message from syslogd@sandworm-nas at Feb  2 22:27:41 ...
 kernel:[  406.555019] Code: f8627821 f100003f 8b000020 9a811001 (f9400020) 
Timeout, server sandworm-nas.home not responding.

MoonKid
Posts: 63
Joined: Mon Nov 09, 2020 10:43 pm
languages_spoken: english
Has thanked: 6 times
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by MoonKid »

From my side no clear solution yet.
My rsync jobs are finished now. That is why the situation does nat appear anymore on my side.

My "solution" is to put the HC4 in the trash and get back to a barebone based i686 mini server where Debian run out of the box without any modifications.

klippe
Posts: 11
Joined: Thu Jan 28, 2021 3:55 am
languages_spoken: english
ODROIDs: Odroid HC4
Has thanked: 1 time
Been thanked: 2 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by klippe »

So, I've installed Armbian and seems it's a solution - I was able to successfully upload bunch of small and large files with no issues. Hope it'll stay this way.
These users thanked the author klippe for the post:
igorpec (Thu Feb 04, 2021 9:20 pm)

atwk
Posts: 2
Joined: Thu Feb 18, 2021 2:06 am
languages_spoken: english, german
ODROIDs: HC4
Has thanked: 0
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by atwk »

klippe wrote:
Thu Feb 04, 2021 8:16 pm
So, I've installed Armbian and seems it's a solution - I was able to successfully upload bunch of small and large files with no issues. Hope it'll stay this way.
Is your system still stable with Armbian and OMV? I often get the same error - now and then a message is displayed on the screen:
https://i.ibb.co/ckbRsNZ/DSC-0873.jpg
https://i.ibb.co/cvGDzHg/DSC-0874.jpg
https://i.ibb.co/Xj50FH8/DSC-0875.jpg
https://i.ibb.co/Rph2d6F/DSC-0876.jpg

First I had a ntfs hard disk in it, there were actually no or hardly any errors. Only since I formatted the hard drives with btrfs.

atwk
Posts: 2
Joined: Thu Feb 18, 2021 2:06 am
languages_spoken: english, german
ODROIDs: HC4
Has thanked: 0
Been thanked: 0
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by atwk »

so I then also switched to Armbian - and have had no more problems since then.

brad
Posts: 1418
Joined: Tue Mar 29, 2016 1:22 pm
languages_spoken: english
ODROIDs: C2 N1 N2 N2+ H2 H2+ (64 bit ftw)
Location: Australia
Has thanked: 126 times
Been thanked: 206 times
Contact:

Re: Oops: 96000004 [#1] PREEMPT SMP

Post by brad »

I just seen this and solved a similar issue back last year so if anyone has similar issues on 5.9 / 5.10 kernel try the following. (was never able to reproduce on 5.11)

Edit /etc/sysctl.conf and add this at the bottom

Code: Select all

vm.zone_reclaim_mode=2
vm.min_free_kbytes=45056
Reboot

The parameters have 2 objectives:

- forces the kernel to write out dirty disk buffers sooner under heavy disk writes.
- Attempts to request an amount of physical RAM to be available at all times.

How does this help?

Information being written to disk may be stored in memory until the disk has time to write (ie coming in via network eg samba). This is known as write buffers and in some cases it can fill all available physical RAM. If all physical RAM is consumed by these buffers but the kernel itself needs a little bit more to complete a write operation it has nowhere to go so has no option but to OOPS. Above parameters make it more likely to stop and write out disk buffers earlier before physical RAM is all consumed by them. I suspect this was a bug in 5.9 / 5.10 mainline but likely fixed sometime in 5.10 / 5.11. I could reproduce with BTRFS and EXT4 filesystems at the time.

I never like htop as it provides very limited information but I can see in klippe's top output buffers filled with smb work happening and almost no free RAM which is the ideal scenario for this problem to occur. I enabled debug and captured a stack trace on the console to determine that a critical kernel disk operation had no free physical RAM to allocate in one of the page table tree's at the time.
These users thanked the author brad for the post (total 2):
tobetter (Tue Apr 27, 2021 10:57 pm) • odroid (Wed Apr 28, 2021 10:36 am)

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 0 guests