Hardware watchdog is not working

Post Reply
cjng96
Posts: 17
Joined: Fri Apr 12, 2019 1:31 pm
languages_spoken: english, korean
Has thanked: 2 times
Been thanked: 0
Contact:

Hardware watchdog is not working

Unread post by cjng96 » Thu Apr 25, 2019 11:38 pm

I installed watchdog daemon as following url(of course I know it's for XU4 article)
https://wiki.odroid.com/odroid-xu4/appl ... x_watchdog

and I didn't modify boot.ini since I'm not sure how to modify it. :(

Anyway after installing watchdog daemon, I can see related log message as follows,

Apr 25 13:50:51 odroid watchdog[3574]: String 'watchdog-device' found as '/dev/watchdog'
Apr 25 13:50:51 odroid watchdog[3574]: Variable 'realtime' found as 'yes' = 1
Apr 25 13:50:51 odroid sh[3574]: watchdog: String 'watchdog-device' found as '/dev/watchdog'
Apr 25 13:50:51 odroid sh[3574]: watchdog: Variable 'realtime' found as 'yes' = 1
Apr 25 13:50:51 odroid sh[3574]: watchdog: Integer 'priority' found = 1
Apr 25 13:50:51 odroid watchdog[3574]: Integer 'priority' found = 1
Apr 25 13:50:51 odroid watchdog[3578]: starting daemon (5.15):
Apr 25 13:50:51 odroid watchdog[3578]: int=1s realtime=yes sync=yes load=0,0,0 soft=no
Apr 25 13:50:51 odroid watchdog[3578]: memory not checked
Apr 25 13:50:51 odroid watchdog[3578]: ping: no machine to check
Apr 25 13:50:51 odroid watchdog[3578]: file: no file to check
Apr 25 13:50:51 odroid watchdog[3578]: pidfile: no server process to check
Apr 25 13:50:51 odroid watchdog[3578]: interface: no interface to check
Apr 25 13:50:51 odroid watchdog[3578]: temperature: no sensors to check
Apr 25 13:50:51 odroid watchdog[3578]: no test binary files
Apr 25 13:50:51 odroid watchdog[3578]: no repair binary files
Apr 25 13:50:51 odroid watchdog[3578]: error retry time-out = 60 seconds
Apr 25 13:50:51 odroid watchdog[3578]: repair attempts = 1
Apr 25 13:50:51 odroid watchdog[3578]: alive=/dev/watchdog heartbeat=[none] to=root no_act=no force=no
Apr 25 13:50:51 odroid watchdog[3578]: watchdog now set to 60 seconds
Apr 25 13:50:51 odroid watchdog[3578]: hardware watchdog identity: aml Watchdog
Apr 25 13:50:51 odroid kernel: [ 37.071331@1] meson_wdt ffd0f0d0.watchdog: start watchdog
Apr 25 13:50:51 odroid kernel: [ 37.071350@1] meson_wdt ffd0f0d0.watchdog: set timeout
Apr 25 13:50:51 odroid watchdog[3578]: cannot set scheduler (errno = 1 = 'Operation not permitted')
Apr 25 13:50:51 odroid systemd[1]: Started watchdog daemon.

There is a message,"Operation not permitted" and when I try to crash kernel to check whether watchdog is working or not.
But it seems not to work. :(
Becasue after I make the kernel crash by "echo c > /proc/sysrq-trigger", the system is not reset for 10 minutes.

How can I enable hardware watchdog on ubuntu?

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Mon Oct 07, 2019 10:06 pm

Hello,
Well I am here and I am experiencing the same problem. As well, I followed the XU4 instructions hoping for the best and noticed that there are no entries in the N2 boot.ini for watchdog as there are in the XU4 boot.ini so I manually entered those entries into the N2 boot.ini.
This is a completely new install from the N2 OS image with a couple of apt installs like gawk, locate, htop, so there shouldn't be any conflicts.
I verified watchdog was running thru service watchdog status and then I issued the kill-trigger and there was no restart.
I did a power off reboot and everything restored normally with the watchdog service active so it appears that the watchdog is installed properly.
Maybe it's the kill-test is incorrect?
Any ideas?

User avatar
odroid
Site Admin
Posts: 32508
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 181 times
Been thanked: 349 times
Contact:

Re: Hardware watchdog is not working

Unread post by odroid » Tue Oct 08, 2019 9:01 am

Did you check N2 WDT wiki page? It was written around about five months ago.
https://wiki.odroid.com/odroid-n2/appli ... hdog_timer

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Tue Oct 08, 2019 8:15 pm

Hi,
Thanks for that link. I started a new OS install on the N2 from SD card and installed watchdog according to the posted instructions on the new link, but I still cannot get watchdog to function properly.
Watchdog status shows it's fine and that it restarts on a system power off reboot.
I tried both the pkill -9 watchdog and echo c > /proc/sysrq-trigger
syslog doesn't show anything and /var/log/watchdog/ is empty
I have an XU4 with Ubuntu 16.04.6 with 4.14.5-92 that works properly from the setup of the OP's link.
I took an XU4 that I have 18.04.3 4.14.141-169 installed on and installed watchdog according to the N2 instructions and watchdog works fine on it.
I did a echo c > /proc/sysrq-trigger on it and it all went fine.
Any ideas?

User avatar
odroid
Site Admin
Posts: 32508
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 181 times
Been thanked: 349 times
Contact:

Re: Hardware watchdog is not working

Unread post by odroid » Wed Oct 09, 2019 10:39 am

We will test it tomorrow with the latest N2 Ubuntu image and the latest Kernel 4.9.190-62.

joshua.yang
Posts: 286
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 2 times
Been thanked: 40 times
Contact:

Re: Hardware watchdog is not working

Unread post by joshua.yang » Thu Oct 10, 2019 12:20 pm

Hi.

I have tested Watchdog on N2.
I installed ubuntu-18.04.3-4.9-minimal-odroid-n2-20190806.img.xz and do apt update && apt full-upgrade -y. The kernel version is 4.9.190-62.

I followed XU4 and N2 articles, and these are almost the same to set Watchdog.

This is the Watchdog service results.

Code: Select all

root@odroid:~# service watchdog status
● watchdog.service - watchdog daemon
   Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-10-10 02:46:51 UTC; 30s ago
  Process: 2356 ExecStart=/bin/sh -c [ $run_watchdog != 1 ] || exec /usr/sbin/watchdog $watchdog_options (code=exited, status=0/SUCCESS)
  Process: 2355 ExecStartPre=/bin/sh -c [ -z "${watchdog_module}" ] || [ "${watchdog_module}" = "none" ] || /sbin/modprobe $watchdog_module
 Main PID: 2358 (watchdog)
    Tasks: 1 (limit: 3843)
   CGroup: /system.slice/watchdog.service
           └─2358 /usr/sbin/watchdog -s -v -c /etc/watchdog.conf

Oct 10 02:46:54 odroid watchdog[2358]: still alive after 3 interval(s)
Oct 10 02:46:55 odroid watchdog[2358]: still alive after 4 interval(s)
Oct 10 02:46:56 odroid watchdog[2358]: still alive after 5 interval(s)
Oct 10 02:46:57 odroid watchdog[2358]: still alive after 6 interval(s)
Oct 10 02:46:58 odroid watchdog[2358]: still alive after 7 interval(s)
Oct 10 02:46:59 odroid watchdog[2358]: still alive after 8 interval(s)
Oct 10 02:47:00 odroid watchdog[2358]: still alive after 9 interval(s)
Oct 10 02:47:18 odroid watchdog[2358]: still alive after 10 interval(s)
Oct 10 02:47:19 odroid watchdog[2358]: still alive after 11 interval(s)
Oct 10 02:47:20 odroid watchdog[2358]: still alive after 12 interval(s)
Oct 10 02:47:21 odroid watchdog[2358]: still alive after 13 interval(s)
And this is the configuration file (/etc/watchdog.conf). I just uncommented the watchdog-device = /dev/watchdog line.

Code: Select all

root@odroid:~# cat /etc/watchdog.conf
#ping                   = 172.31.14.1
#ping                   = 172.26.1.255
#interface              = eth0
#file                   = /var/log/messages
#change                 = 1407

# Uncomment to enable test. Setting one of these values to '0' disables it.
# These values will hopefully never reboot your machine during normal use
# (if your machine is really hung, the loadavg will go much higher than 25)
#max-load-1             = 24
#max-load-5             = 18
#max-load-15            = 12

# Note that this is the number of pages!
# To get the real size, check how large the pagesize is on your machine.
#min-memory             = 1
#allocatable-memory     = 1

#repair-binary          = /usr/sbin/repair
#repair-timeout         = 60
#test-binary            =
#test-timeout           = 60

# The retry-timeout and repair limit are used to handle errors in a more robust
# manner. Errors must persist for longer than retry-timeout to action a repair
# or reboot, and if repair-maximum attempts are made without the test passing a
# reboot is initiated anyway.
#retry-timeout          = 60
#repair-maximum         = 1

watchdog-device = /dev/watchdog

# Defaults compiled into the binary
#temperature-sensor     =
#max-temperature        = 90

# Defaults compiled into the binary
#admin                  = root
#interval               = 1
#logtick                = 1
#log-dir                = /var/log/watchdog

# This greatly decreases the chance that watchdog won't be scheduled before
# your machine is really loaded
realtime                = yes
priority                = 1

# Check if rsyslogd is still running by enabling the following line
#pidfile                = /var/run/rsyslogd.pid
And I can check if this configuration is loaded.

Code: Select all

root@odroid:~# watchdog -v
watchdog: String 'watchdog-device' found as '/dev/watchdog'
watchdog: Variable 'realtime' found as 'yes' = 1
watchdog: Integer 'priority' found = 1
If I triggered forced kernel panic,

Code: Select all

echo c > /proc/sysrq-trigger
It reboots promptly.

Code: Select all

[  229.598889@2] sysrq: SysRq : Trigger a crash
[  229.598935@0] Unable to handle kernel NULL pointer dereference at virtual address 00000000
[  229.606049@0] pgd = ffffffc0c8a0c000
[  229.610099@0] [0000000000000000] *pgd=0000000000000000, *pud=0000000000000000
[  229.616464@0] Internal error: Oops: 96000045 [#1] PREEMPT SMP
[  229.622135@0] Modules linked in: cpufreq_ondemand cpufreq_powersave cpufreq_userspace cpufreq_conservative rtc_pcf8563 i2c_meson_master6
[  229.658796@0] CPU: 0 PID: 2337 Comm: bash Not tainted 4.9.190-62 #1
[  229.665034@0] Hardware name: Hardkernel ODROID-N2 (DT)
[  229.670148@0] task: ffffffc0c7e1f000 task.stack: ffffffc0c92ac000
[  229.676222@0] PC is at sysrq_handle_crash+0x28/0x38
[  229.681068@0] LR is at sysrq_handle_crash+0x14/0x38
[  229.685929@0] ------------[ cut here ]------------
[  229.690692@0] WARNING: CPU: 0 PID: 2337 at vmalloc_to_page+0xd8/0xe8
[  229.697013@0] Modules linked in: cpufreq_ondemand cpufreq_powersave cpufreq_userspace cpufreq_conservative rtc_pcf8563 i2c_meson_master6
[  229.733674@0]
[  229.735321@0] CPU: 0 PID: 2337 Comm: bash Not tainted 4.9.190-62 #1
[  229.741560@0] Hardware name: Hardkernel ODROID-N2 (DT)
[  229.746674@0] task: ffffffc0c7e1f000 task.stack: ffffffc0c92ac000
[  229.752742@0] PC is at vmalloc_to_page+0xd8/0xe8
[  229.757337@0] LR is at __show_regs+0xb0/0x3b0
[  229.761671@0] ------------[ cut here ]------------
[  229.766436@0] WARNING: CPU: 0 PID: 2337 at vmalloc_to_page+0xb4/0xe8
[  229.772759@0] Modules linked in: cpufreq_ondemand cpufreq_powersave cpufreq_userspace cpufreq_conservative rtc_pcf8563 i2c_meson_master6
[  229.809419@0]
[  229.811068@0] CPU: 0 PID: 2337 Comm: bash Not tainted 4.9.190-62 #1
[  229.817306@0] Hardware name: Hardkernel ODROID-N2 (DT)
[  229.822420@0] task: ffffffc0c7e1f000 task.stack: ffffffc0c92ac000
[  229.828488@0] PC is at vmalloc_to_page+0xb4/0xe8
[  229.833081@0] LR is at __show_regs+0xb0/0x3b0
[  229.837414@0] pc : [<ffffff80092020dc>] lr : [<ffffff80090860e0>] pstate: 800003c5
[  229.844953@0] sp : ffffffc0c92af620
[  229.848420@0] x29: ffffffc0c92af620 x28: ffffffc0c7e1f000
[  229.853880@0] x27: ffffff8009c02000 x26: ffffff800a058000
[  229.859340@0] x25: ffffffbf00000000 x24: ffffff8009f0c820
[  229.864800@0] x23: ffffffc0c92af9a0 x22: 000000000000001d
[  229.870260@0] x21: ffffff80090860e0 x20: ffffffc0c92af870
[  229.875720@0] x19: ffffff808a7a5fc7 x18: 0000000000000001
[  229.881180@0] x17: 0000007fa3dcc028 x16: ffffff800922de28
[  229.886640@0] x15: ffffffffffffffff x14: ffffff808a7a5fc7
[  229.892100@0] x13: ffffff800a7a5fd7 x12: 000000000000000f
[  229.897560@0] x11: 0000000005f5e0ff x10: 0000000000000378
[  229.903020@0] x9 : 00000000ffffffd0 x8 : ffffff80093982e0
[  229.908480@0] x7 : ffffff800a6601e8 x6 : 0000000000000000
[  229.913940@0] x5 : ffffffc0cf317b98 x4 : 0000000000000001
[  229.919400@0] x3 : 0000000000000000 x2 : 0000000000000001
[  229.924860@0] x1 : ffffff800a783000 x0 : 0000000000000000
[  229.930321@0]
[  229.930321@0] SP: 0xffffffc0c92af5a0:
[  229.935607@0] f5a0  0000001d 00000000 c92af9a0 ffffffc0 09f0c820 ffffff80 00000000 ffffffbf
[  229.943926@0] f5c0  0a058000 ffffff80 09c02000 ffffff80 c7e1f000 ffffffc0 c92af620 ffffffc0
[  229.952247@0] f5e0  090860e0 ffffff80 c92af620 ffffffc0 092020dc ffffff80 800003c5 00000000
[  229.960566@0] f600  00000001 00000000 cf317b98 ffffffc0 ffffffff 0000007f 0a6601e8 ffffff80
[  229.968886@0] f620  c92af640 ffffffc0 090860e0 ffffff80 0000000e 00000000 ffffffc8 00000000
[  229.977206@0] f640  c92af6a0 ffffffc0 0908640c ffffff80 c92af870 ffffffc0 c92af870 ffffffc0
[  229.985526@0] f660  00000000 00000000 00000009 00000000 09202100 ffffff80 00000000 00000000
[  229.993846@0] f680  00000000 ffffffbf 0a058000 ffffff80 00000000 00000000 0a6601e8 ffffff80
[  230.002168@0]
[  230.002168@0] X5: 0xffffffc0cf317b18:
[  230.007453@0] 7b18  00000000 00000000 00000000 00000000 00000000 00000000 8351d802 00000000
[  230.015773@0] 7b38  40362337 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.024093@0] 7b58  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.032413@0] 7b78  00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000000
[  230.040733@0] 7b98  00000007 00000000 00000000 00000000 0910e5a0 ffffff80 00000000 00000000
[  230.049053@0] 7bb8  00000000 01400000 00000000 000a3b4d 00000000 00000000 00000000 00000000
[  230.057373@0] 7bd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.065693@0] 7bf8  00000000 00000000 7b247b24 00000000 00000000 00000000 ffffbb3c 00000000
[  230.074016@0]
[  230.074016@0] X20: 0xffffffc0c92af7f0:
[  230.079386@0] f7f0  c92af8b0 ffffffc0 09145764 ffffff80 09f0c810 ffffff80 c92af850 ffffffc0
[  230.087706@0] f810  c92af8b0 ffffffc0 c92af8b0 ffffffc0 c92af870 ffffffc0 ffffffc8 00000000
[  230.096026@0] f830  c92af8b0 ffffffc0 c92af8b0 ffffffc0 c92af870 ffffffc0 ffffffc8 00000000
[  230.104346@0] f850  c92af8b0 ffffffc0 c92af8b0 ffffffc0 c92af870 ffffffc0 ffffffc8 00000000
[  230.112666@0] f870  00000000 00000000 0a78386d ffffff80 00000001 00000000 00000007 00000000
[  230.120986@0] f890  00000001 00000000 cf317b98 ffffffc0 00000000 00000000 0a6601e8 ffffff80
[  230.129306@0] f8b0  093982e0 ffffff80 ffffffd0 00000000 0000036f 00000000 05f5e0ff 00000000
[  230.137626@0] f8d0  0000000f 00000000 0a7a5fd7 ffffff80 8a7a5fc7 ffffff80 ffffffff ffffffff
[  230.145947@0]
[  230.145947@0] X23: 0xffffffc0c92af920:
[  230.151320@0] f920  0000001d 00000000 c92afd00 ffffffc0 09f0c820 ffffff80 00000000 ffffffbf
[  230.159639@0] f940  0a058000 ffffff80 09c02000 ffffff80 c7e1f000 ffffffc0 c92af9a0 ffffffc0
[  230.167960@0] f960  090860e0 ffffff80 c92af9a0 ffffffc0 09202100 ffffff80 800001c5 00000000
[  230.176280@0] f980  00000001 00000000 cf317b98 ffffffc0 ffffffff 0000007f 0a6601e8 ffffff80
[  230.184600@0] f9a0  c92af9c0 ffffffc0 090860e0 ffffff80 00000007 00000000 08306000 ffffff80
[  230.192919@0] f9c0  c92afa20 ffffffc0 0908b42c ffffff80 0a7a0000 ffffff80 c92afbd0 ffffffc0
[  230.201239@0] f9e0  00000000 00000000 96000045 00000000 00000140 00000000 c7e1f000 ffffffc0
[  230.209560@0] fa00  00000123 00000000 00000040 00000000 96000045 ffffffff 00000000 0000000b
[  230.217881@0]
[  230.217881@0] X28: 0xffffffc0c7e1ef80:
[  230.223253@0] ef80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.231573@0] efa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.239893@0] efc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.248213@0] efe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.256533@0] f000  00000008 00000000 ffffffff ffffffff 00000001 00000000 00000000 00000000
[  230.264853@0] f020  c92ac000 ffffffc0 00000002 00400100 00000000 00000000 00000000 00000000
[  230.273173@0] f040  00000001 00000000 00000005 00000000 ffffbae5 00000000 ca995400 ffffffc0
[  230.281493@0] f060  00000000 00000001 00000078 00000078 00000078 00000000 09c09458 ffffff80
[  230.289814@0]
[  230.289814@0] X29: 0xffffffc0c92af5a0:
[  230.295186@0] f5a0  0000001d 00000000 c92af9a0 ffffffc0 09f0c820 ffffff80 00000000 ffffffbf
[  230.303506@0] f5c0  0a058000 ffffff80 09c02000 ffffff80 c7e1f000 ffffffc0 c92af620 ffffffc0
[  230.311826@0] f5e0  090860e0 ffffff80 c92af620 ffffffc0 092020dc ffffff80 800003c5 00000000
[  230.320146@0] f600  00000001 00000000 cf317b98 ffffffc0 ffffffff 0000007f 0a6601e8 ffffff80
[  230.328466@0] f620  c92af640 ffffffc0 090860e0 ffffff80 0000000e 00000000 ffffffc8 00000000
[  230.336786@0] f640  c92af6a0 ffffffc0 0908640c ffffff80 c92af870 ffffffc0 c92af870 ffffffc0
[  230.345106@0] f660  00000000 00000000 00000009 00000000 09202100 ffffff80 00000000 00000000
[  230.353426@0] f680  00000000 ffffffbf 0a058000 ffffff80 00000000 00000000 0a6601e8 ffffff80
[  230.361746@0]
[  230.363393@0] ---[ end trace 02a79d7d31e85215 ]---
[  230.368162@0] Call trace:
[  230.370762@0] Exception stack(0xffffffc0c92af440 to 0xffffffc0c92af570)
[  230.377348@0] f440: ffffff808a7a5fc7 0000007fffffffff ffffffc0c92af620 ffffff80092020dc
[  230.385322@0] f460: 00000000800003c5 00000000ffffffc8 ffffffc0c92af530 ffffff8009145764
[  230.393295@0] f480: ffffff8009f0c810 ffffffc0c92af4d0 ffffffc0c92af530 ffffffc0c92af530
[  230.401268@0] f4a0: ffffffc0c92af4f0 00000000ffffffc8 ffffffc0c92af530 ffffffc0c92af530
[  230.409241@0] f4c0: ffffffc0c92af4f0 00000000ffffffc8 ffffffc0c92af530 ffffffc0c92af530
[  230.417215@0] f4e0: ffffffc0c92af4f0 00000000ffffffc8 0000000000000000 ffffff800a783000
[  230.425188@0] f500: 0000000000000001 0000000000000000 0000000000000001 ffffffc0cf317b98
[  230.433161@0] f520: 0000000000000000 ffffff800a6601e8 ffffff80093982e0 00000000ffffffd0
[  230.441135@0] f540: 0000000000000378 0000000005f5e0ff 000000000000000f ffffff800a7a5fd7
[  230.449107@0] f560: ffffff808a7a5fc7 ffffffffffffffff
[  230.454135@0] [<ffffff80092020dc>] vmalloc_to_page+0xb4/0xe8
[  230.459768@0] [<ffffff80090860e0>] __show_regs+0xb0/0x3b0
[  230.465141@0] [<ffffff800908640c>] show_regs+0x2c/0x38
[  230.470257@0] [<ffffff80090a5664>] __warn+0xb4/0x118
[  230.475196@0] [<ffffff800943b068>] report_bug+0xb0/0x100
[  230.480482@0] [<ffffff800908b4dc>] bug_handler.part.2+0x24/0x88
[  230.486375@0] [<ffffff800908b57c>] bug_handler+0x3c/0x48
[  230.491662@0] [<ffffff800908461c>] brk_handler+0x7c/0xd0
[  230.496948@0] [<ffffff80090815b8>] do_debug_exception+0x68/0xd0
[  230.502840@0] Exception stack(0xffffffc0c92af7c0 to 0xffffffc0c92af8f0)
[  230.509428@0] f7c0: ffffff800a6601e8 0000007fffffffff ffffffc0c92af9a0 ffffff8009202100
[  230.517401@0] f7e0: 00000000800001c5 00000000ffffffc8 ffffffc0c92af8b0 ffffff8009145764
[  230.525375@0] f800: ffffff8009f0c810 ffffffc0c92af850 ffffffc0c92af8b0 ffffffc0c92af8b0
[  230.533348@0] f820: ffffffc0c92af870 00000000ffffffc8 ffffffc0c92af8b0 ffffffc0c92af8b0
[  230.541322@0] f840: ffffffc0c92af870 00000000ffffffc8 ffffffc0c92af8b0 ffffffc0c92af8b0
[  230.549295@0] f860: ffffffc0c92af870 00000000ffffffc8 0000000000000000 ffffff800a78386d
[  230.557268@0] f880: 0000000000000001 0000000000000007 0000000000000001 ffffffc0cf317b98
[  230.565241@0] f8a0: 0000000000000000 ffffff800a6601e8 ffffff80093982e0 00000000ffffffd0
[  230.573215@0] f8c0: 000000000000036f 0000000005f5e0ff 000000000000000f ffffff800a7a5fd7
[  230.581187@0] f8e0: ffffff808a7a5fc7 ffffffffffffffff
[  230.586215@0] [<ffffff8009082fe0>] el1_dbg+0x18/0x74
[  230.591155@0] [<ffffff80090860e0>] __show_regs+0xb0/0x3b0
[  230.596528@0] [<ffffff800908b42c>] die+0x134/0x1c0
[  230.601296@0] [<ffffff800909ca28>] __do_kernel_fault+0x98/0xa8
[  230.607102@0] [<ffffff800909cf74>] do_page_fault+0x5c/0x3a8
[  230.612648@0] [<ffffff800909d32c>] do_translation_fault+0x6c/0x80
[  230.618715@0] [<ffffff800908138c>] do_mem_abort+0x54/0xb0
[  230.624088@0] Exception stack(0xffffffc0c92afb30 to 0xffffffc0c92afc60)
[  230.630674@0] fb20:                                   ffffff800a619000 0000007fffffffff
[  230.638649@0] fb40: ffffffc0c92afd00 ffffff80094c2f70 ffffffc0c92afb70 ffffff800910d30c
[  230.646622@0] fb60: ffffff8009f18420 0000000100000000 ffffffc0c92afc10 ffffff800910d4b8
[  230.654596@0] fb80: ffffffc0c92afc70 ffffff8009f4fc68 ffffff800a676570 ffffff800a676000
[  230.662568@0] fba0: 0000000000000004 0000000000000000 0000000000000123 0000000000000040
[  230.670541@0] fbc0: ffffff8009c02000 ffffffc0c7e1f000 0000000000000001 0000000000000000
[  230.678515@0] fbe0: ffffffc0cf31b7e0 0000000000000000 0000000000000000 0000000000000000
[  230.686488@0] fc00: 0000000000000000 ffffff800a6601e8 ffffff80093982e0 0000000000000001
[  230.694461@0] fc20: 0000000000000365 0000000000000006 0000000000000007 0000000000000000
[  230.702435@0] fc40: ffffff808a7a5fc7 ffffffffffffffff ffffff800922de28 0000007fa3dcc028
[  230.710408@0] [<ffffff8009082f4c>] el1_da+0x24/0x84
[  230.715263@0] [<ffffff80094c35c0>] __handle_sysrq+0xb0/0x1a8
[  230.720895@0] [<ffffff80094c3b90>] write_sysrq_trigger+0x90/0xa0
[  230.726877@0] [<ffffff80092a8ae0>] proc_reg_write+0x90/0xd0
[  230.732422@0] [<ffffff800922b808>] __vfs_write+0x48/0x118
[  230.737795@0] [<ffffff800922c7fc>] vfs_write+0xac/0x1b0
[  230.742995@0] [<ffffff800922de84>] SyS_write+0x5c/0xc8
[  230.748108@0] [<ffffff80090839c0>] el0_svc_naked+0x34/0x38
[  230.753568@0] pc : [<ffffff8009202100>] lr : [<ffffff80090860e0>] pstate: 800001c5
[  230.761107@0] sp : ffffffc0c92af9a0
[  230.764574@0] x29: ffffffc0c92af9a0 x28: ffffffc0c7e1f000
[  230.770033@0] x27: ffffff8009c02000 x26: ffffff800a058000
[  230.775493@0] x25: ffffffbf00000000 x24: ffffff8009f0c820
[  230.780953@0] x23: ffffffc0c92afd00 x22: 000000000000001d
[  230.786413@0] x21: ffffff80094c2f5c x20: ffffffc0c92afbd0
[  230.791873@0] x19: ffffff800a6601e8 x18: 0000000000000001
[  230.797333@0] x17: 0000007fa3dcc028 x16: ffffff800922de28
[  230.802793@0] x15: ffffffffffffffff x14: ffffff808a7a5fc7
[  230.808253@0] x13: ffffff800a7a5fd7 x12: 000000000000000f
[  230.813713@0] x11: 0000000005f5e0ff x10: 000000000000036f
[  230.819173@0] x9 : 00000000ffffffd0 x8 : ffffff80093982e0
[  230.824634@0] x7 : ffffff800a6601e8 x6 : 0000000000000000
[  230.830093@0] x5 : ffffffc0cf317b98 x4 : 0000000000000001
[  230.835553@0] x3 : 0000000000000007 x2 : 0000000000000001
[  230.841013@0] x1 : ffffff800a78386d x0 : 0000000000000000
[  230.846474@0]
[  230.846474@0] SP: 0xffffffc0c92af920:
[  230.851760@0] f920  0000001d 00000000 c92afd00 ffffffc0 09f0c820 ffffff80 00000000 ffffffbf
[  230.860080@0] f940  0a058000 ffffff80 09c02000 ffffff80 c7e1f000 ffffffc0 c92af9a0 ffffffc0
[  230.868400@0] f960  090860e0 ffffff80 c92af9a0 ffffffc0 09202100 ffffff80 800001c5 00000000
[  230.876719@0] f980  00000001 00000000 cf317b98 ffffffc0 ffffffff 0000007f 0a6601e8 ffffff80
[  230.885040@0] f9a0  c92af9c0 ffffffc0 090860e0 ffffff80 00000007 00000000 08306000 ffffff80
[  230.893359@0] f9c0  c92afa20 ffffffc0 0908b42c ffffff80 0a7a0000 ffffff80 c92afbd0 ffffffc0
[  230.901680@0] f9e0  00000000 00000000 96000045 00000000 00000140 00000000 c7e1f000 ffffffc0
[  230.909999@0] fa00  00000123 00000000 00000040 00000000 96000045 ffffffff 00000000 0000000b
[  230.918321@0]
[  230.918321@0] X5: 0xffffffc0cf317b18:
[  230.923606@0] 7b18  00000000 00000000 00000000 00000000 00000000 00000000 8351d802 00000000
[  230.931926@0] 7b38  40362337 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.940246@0] 7b58  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.948566@0] 7b78  00000000 00000000 00000000 00000000 00000000 00000000 00000001 00000000
[  230.956886@0] 7b98  00000007 00000000 00000000 00000000 0910e5a0 ffffff80 00000000 00000000
[  230.965206@0] 7bb8  00000000 01400000 00000000 000a3b4d 00000000 00000000 00000000 00000000
[  230.973526@0] 7bd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  230.981846@0] 7bf8  00000000 00000000 7b247b24 00000000 00000000 00000000 ffffbb3c 00000000
[  230.990170@0]
[  230.990170@0] X20: 0xffffffc0c92afb50:
[  230.995540@0] fb50  c92afb70 ffffffc0 0910d30c ffffff80 09f18420 ffffff80 00000000 00000001
[  231.003860@0] fb70  c92afc10 ffffffc0 0910d4b8 ffffff80 c92afc70 ffffffc0 09f4fc68 ffffff80
[  231.012180@0] fb90  0a676570 ffffff80 0a676000 ffffff80 00000004 00000000 00000000 00000000
[  231.020500@0] fbb0  00000123 00000000 00000040 00000000 09c02000 ffffff80 c7e1f000 ffffffc0
[  231.028819@0] fbd0  00000001 00000000 00000000 00000000 cf31b7e0 ffffffc0 00000000 00000000
[  231.037140@0] fbf0  00000000 00000000 00000000 00000000 00000000 00000000 0a6601e8 ffffff80
[  231.045460@0] fc10  093982e0 ffffff80 00000001 00000000 00000365 00000000 00000006 00000000
[  231.053779@0] fc30  00000007 00000000 00000000 00000000 8a7a5fc7 ffffff80 ffffffff ffffffff
[  231.062101@0]
[  231.062101@0] X23: 0xffffffc0c92afc80:
[  231.067473@0] fc80  0a676000 ffffff80 00000004 00000000 00000000 00000000 00000123 00000000
[  231.075793@0] fca0  00000040 00000000 09c02000 ffffff80 c7e1f000 ffffffc0 c92afd00 ffffffc0
[  231.084113@0] fcc0  094c2f5c ffffff80 c92afd00 ffffffc0 094c2f70 ffffff80 60000145 00000000
[  231.092433@0] fce0  c92afd00 ffffffc0 094c2f5c ffffff80 ffffffff 0000007f 00000000 00000000
[  231.100753@0] fd00  c92afd10 ffffffc0 094c35c0 ffffff80 c92afd50 ffffffc0 094c3b90 ffffff80
[  231.109073@0] fd20  00000002 00000000 84c5b710 00000055 84c5b710 00000055 00000002 00000000
[  231.117393@0] fd40  c92afeb8 ffffffc0 00000015 00000000 c92afd70 ffffffc0 092a8ae0 ffffff80
[  231.125713@0] fd60  c880a3c0 ffffffc0 c7b03d00 ffffffc0 c92afdb0 ffffffc0 0922b808 ffffff80
[  231.134034@0]
[  231.134034@0] X28: 0xffffffc0c7e1ef80:
[  231.139406@0] ef80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.147726@0] efa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.156046@0] efc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.164366@0] efe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.172686@0] f000  00000008 00000000 ffffffff ffffffff 00000001 00000000 00000000 00000000
[  231.181006@0] f020  c92ac000 ffffffc0 00000002 00400100 00000000 00000000 00000000 00000000
[  231.189326@0] f040  00000001 00000000 00000005 00000000 ffffbae5 00000000 ca995400 ffffffc0
[  231.197646@0] f060  00000000 00000001 00000078 00000078 00000078 00000000 09c09458 ffffff80
[  231.205967@0]
[  231.205967@0] X29: 0xffffffc0c92af920:
[  231.211340@0] f920  0000001d 00000000 c92afd00 ffffffc0 09f0c820 ffffff80 00000000 ffffffbf
[  231.219659@0] f940  0a058000 ffffff80 09c02000 ffffff80 c7e1f000 ffffffc0 c92af9a0 ffffffc0
[  231.227979@0] f960  090860e0 ffffff80 c92af9a0 ffffffc0 09202100 ffffff80 800001c5 00000000
[  231.236299@0] f980  00000001 00000000 cf317b98 ffffffc0 ffffffff 0000007f 0a6601e8 ffffff80
[  231.244620@0] f9a0  c92af9c0 ffffffc0 090860e0 ffffff80 00000007 00000000 08306000 ffffff80
[  231.252939@0] f9c0  c92afa20 ffffffc0 0908b42c ffffff80 0a7a0000 ffffff80 c92afbd0 ffffffc0
[  231.261259@0] f9e0  00000000 00000000 96000045 00000000 00000140 00000000 c7e1f000 ffffffc0
[  231.269579@0] fa00  00000123 00000000 00000040 00000000 96000045 ffffffff 00000000 0000000b
[  231.277900@0]
[  231.279547@0] ---[ end trace 02a79d7d31e85216 ]---
[  231.284313@0] Call trace:
[  231.286914@0] Exception stack(0xffffffc0c92af7c0 to 0xffffffc0c92af8f0)
[  231.293502@0] f7c0: ffffff800a6601e8 0000007fffffffff ffffffc0c92af9a0 ffffff8009202100
[  231.301475@0] f7e0: 00000000800001c5 00000000ffffffc8 ffffffc0c92af8b0 ffffff8009145764
[  231.309448@0] f800: ffffff8009f0c810 ffffffc0c92af850 ffffffc0c92af8b0 ffffffc0c92af8b0
[  231.317421@0] f820: ffffffc0c92af870 00000000ffffffc8 ffffffc0c92af8b0 ffffffc0c92af8b0
[  231.325395@0] f840: ffffffc0c92af870 00000000ffffffc8 ffffffc0c92af8b0 ffffffc0c92af8b0
[  231.333368@0] f860: ffffffc0c92af870 00000000ffffffc8 0000000000000000 ffffff800a78386d
[  231.341341@0] f880: 0000000000000001 0000000000000007 0000000000000001 ffffffc0cf317b98
[  231.349315@0] f8a0: 0000000000000000 ffffff800a6601e8 ffffff80093982e0 00000000ffffffd0
[  231.357288@0] f8c0: 000000000000036f 0000000005f5e0ff 000000000000000f ffffff800a7a5fd7
[  231.365261@0] f8e0: ffffff808a7a5fc7 ffffffffffffffff
[  231.370289@0] [<ffffff8009202100>] vmalloc_to_page+0xd8/0xe8
[  231.375922@0] [<ffffff80090860e0>] __show_regs+0xb0/0x3b0
[  231.381295@0] [<ffffff800908b42c>] die+0x134/0x1c0
[  231.386061@0] [<ffffff800909ca28>] __do_kernel_fault+0x98/0xa8
[  231.391868@0] [<ffffff800909cf74>] do_page_fault+0x5c/0x3a8
[  231.397415@0] [<ffffff800909d32c>] do_translation_fault+0x6c/0x80
[  231.403481@0] [<ffffff800908138c>] do_mem_abort+0x54/0xb0
[  231.408854@0] Exception stack(0xffffffc0c92afb30 to 0xffffffc0c92afc60)
[  231.415441@0] fb20:                                   ffffff800a619000 0000007fffffffff
[  231.423415@0] fb40: ffffffc0c92afd00 ffffff80094c2f70 ffffffc0c92afb70 ffffff800910d30c
[  231.431388@0] fb60: ffffff8009f18420 0000000100000000 ffffffc0c92afc10 ffffff800910d4b8
[  231.439361@0] fb80: ffffffc0c92afc70 ffffff8009f4fc68 ffffff800a676570 ffffff800a676000
[  231.447335@0] fba0: 0000000000000004 0000000000000000 0000000000000123 0000000000000040
[  231.455308@0] fbc0: ffffff8009c02000 ffffffc0c7e1f000 0000000000000001 0000000000000000
[  231.463281@0] fbe0: ffffffc0cf31b7e0 0000000000000000 0000000000000000 0000000000000000
[  231.471255@0] fc00: 0000000000000000 ffffff800a6601e8 ffffff80093982e0 0000000000000001
[  231.479228@0] fc20: 0000000000000365 0000000000000006 0000000000000007 0000000000000000
[  231.487201@0] fc40: ffffff808a7a5fc7 ffffffffffffffff ffffff800922de28 0000007fa3dcc028
[  231.495174@0] [<ffffff8009082f4c>] el1_da+0x24/0x84
[  231.500029@0] [<ffffff80094c35c0>] __handle_sysrq+0xb0/0x1a8
[  231.505662@0] [<ffffff80094c3b90>] write_sysrq_trigger+0x90/0xa0
[  231.511642@0] [<ffffff80092a8ae0>] proc_reg_write+0x90/0xd0
[  231.517189@0] [<ffffff800922b808>] __vfs_write+0x48/0x118
[  231.522561@0] [<ffffff800922c7fc>] vfs_write+0xac/0x1b0
[  231.527761@0] [<ffffff800922de84>] SyS_write+0x5c/0xc8
[  231.532875@0] [<ffffff80090839c0>] el0_svc_naked+0x34/0x38
[  231.538335@0] pc : [<ffffff80094c2f70>] lr : [<ffffff80094c2f5c>] pstate: 60000145
[  231.545873@0] sp : ffffffc0c92afd00
[  231.549340@0] x29: ffffffc0c92afd00 x28: ffffffc0c7e1f000
[  231.554800@0] x27: ffffff8009c02000 x26: 0000000000000040
[  231.560260@0] x25: 0000000000000123 x24: 0000000000000000
[  231.565720@0] x23: 0000000000000004 x22: ffffff800a676000
[  231.571180@0] x21: ffffff800a676570 x20: 0000000000000063
[  231.576640@0] x19: ffffff800a619000 x18: 0000000000000001
[  231.582100@0] x17: 0000007fa3dcc028 x16: ffffff800922de28
[  231.587560@0] x15: ffffffffffffffff x14: ffffff808a7a5fc7
[  231.593020@0] x13: 0000000000000000 x12: 0000000000000007
[  231.598480@0] x11: 0000000000000006 x10: 0000000000000365
[  231.603940@0] x9 : 0000000000000001 x8 : ffffff80093982e0
[  231.609400@0] x7 : ffffff800a6601e8 x6 : 0000000000000000
[  231.614860@0] x5 : 0000000000000000 x4 : 0000000000000000
[  231.620320@0] x3 : 0000000000000000 x2 : ffffffc0cf31b7e0
[  231.625780@0] x1 : 0000000000000000 x0 : 0000000000000001
[  231.631240@0]
[  231.631240@0] SP: 0xffffffc0c92afc80:
[  231.636526@0] fc80  0a676000 ffffff80 00000004 00000000 00000000 00000000 00000123 00000000
[  231.644846@0] fca0  00000040 00000000 09c02000 ffffff80 c7e1f000 ffffffc0 c92afd00 ffffffc0
[  231.653166@0] fcc0  094c2f5c ffffff80 c92afd00 ffffffc0 094c2f70 ffffff80 60000145 00000000
[  231.661486@0] fce0  c92afd00 ffffffc0 094c2f5c ffffff80 ffffffff 0000007f 00000000 00000000
[  231.669806@0] fd00  c92afd10 ffffffc0 094c35c0 ffffff80 c92afd50 ffffffc0 094c3b90 ffffff80
[  231.678126@0] fd20  00000002 00000000 84c5b710 00000055 84c5b710 00000055 00000002 00000000
[  231.686446@0] fd40  c92afeb8 ffffffc0 00000015 00000000 c92afd70 ffffffc0 092a8ae0 ffffff80
[  231.694766@0] fd60  c880a3c0 ffffffc0 c7b03d00 ffffffc0 c92afdb0 ffffffc0 0922b808 ffffff80
[  231.703087@0]
[  231.703087@0] X2: 0xffffffc0cf31b760:
[  231.708373@0] b760  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.716693@0] b780  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.725013@0] b7a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.733333@0] b7c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.741653@0] b7e0  cf317ba0 ffffffc0 00000000 00000000 00000000 00000000 00000000 00000000
[  231.749973@0] b800  00000000 00000000 00000001 00000000 cf31b810 ffffffc0 cf31b810 ffffffc0
[  231.758293@0] b820  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.766613@0] b840  00000000 00000000 0000176f 00000000 00000000 00000000 00000000 00000000
[  231.774938@0]
[  231.774938@0] X28: 0xffffffc0c7e1ef80:
[  231.780306@0] ef80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.788626@0] efa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.796946@0] efc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.805267@0] efe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  231.813586@0] f000  00000008 00000000 ffffffff ffffffff 00000001 00000000 00000000 00000000
[  231.821907@0] f020  c92ac000 ffffffc0 00000002 00400100 00000000 00000000 00000000 00000000
[  231.830226@0] f040  00000001 00000000 00000005 00000000 ffffbae5 00000000 ca995400 ffffffc0
[  231.838546@0] f060  00000000 00000001 00000078 00000078 00000078 00000000 09c09458 ffffff80
[  231.846867@0]
[  231.846867@0] X29: 0xffffffc0c92afc80:
[  231.852240@0] fc80  0a676000 ffffff80 00000004 00000000 00000000 00000000 00000123 00000000
[  231.860560@0] fca0  00000040 00000000 09c02000 ffffff80 c7e1f000 ffffffc0 c92afd00 ffffffc0
[  231.868880@0] fcc0  094c2f5c ffffff80 c92afd00 ffffffc0 094c2f70 ffffff80 60000145 00000000
[  231.877200@0] fce0  c92afd00 ffffffc0 094c2f5c ffffff80 ffffffff 0000007f 00000000 00000000
[  231.885520@0] fd00  c92afd10 ffffffc0 094c35c0 ffffff80 c92afd50 ffffffc0 094c3b90 ffffff80
[  231.893840@0] fd20  00000002 00000000 84c5b710 00000055 84c5b710 00000055 00000002 00000000
[  231.902160@0] fd40  c92afeb8 ffffffc0 00000015 00000000 c92afd70 ffffffc0 092a8ae0 ffffff80
[  231.910480@0] fd60  c880a3c0 ffffffc0 c7b03d00 ffffffc0 c92afdb0 ffffffc0 0922b808 ffffff80
[  231.918800@0]
[  231.920447@0] Process bash (pid: 2337, stack limit = 0xffffffc0c92ac000)
[  231.927120@0] Stack: (0xffffffc0c92afd00 to 0xffffffc0c92b0000)
[  231.933015@0] fd00: ffffffc0c92afd10 ffffff80094c35c0 ffffffc0c92afd50 ffffff80094c3b90
[  231.940988@0] fd20: 0000000000000002 0000005584c5b710 0000005584c5b710 0000000000000002
[  231.948961@0] fd40: ffffffc0c92afeb8 0000000000000015 ffffffc0c92afd70 ffffff80092a8ae0
[  231.956935@0] fd60: ffffffc0c880a3c0 ffffffc0c7b03d00 ffffffc0c92afdb0 ffffff800922b808
[  231.964908@0] fd80: ffffffc0c7b03d00 0000000000000002 ffffffc0c92afeb8 0000005584c5b710
[  231.972882@0] fda0: ffffffc0c92afeb8 ffffff80090d68f8 ffffffc0c92afe40 ffffff800922c7fc
[  231.980855@0] fdc0: 0000000000000002 ffffffc0c7b03d00 0000000000000000 0000005584c5b710
[  231.988828@0] fde0: 0000000000000000 ffffffc0ca86c280 ffffffc0ca86c280 0000000000000002
[  231.996802@0] fe00: ffffffc0c92afe40 ffffff800922c8ec 0000000000000002 ffffffc0c7b03d00
[  232.004775@0] fe20: 0000000000000000 0000005584c5b710 0000005584c5b710 0000005584c5b710
[  232.012748@0] fe40: ffffffc0c92afe80 ffffff800922de84 ffffffc0c7b03d00 ffffffc0c7b03d00
[  232.020721@0] fe60: 0000005584c5b710 0000000000000002 0000000020000000 0000000000000000
[  232.028695@0] fe80: 0000000000000000 ffffff80090839c0 ffffffffffffff1d 00000040c5119000
[  232.036668@0] fea0: ffffffffffffffff 0000007fa3e23bac 0000000000000400 0000000000000000
[  232.044641@0] fec0: 0000000000000001 0000005584c5b710 0000000000000002 0000007fa3eb91a8
[  232.052615@0] fee0: 0000000000000000 0000000000015551 0000005584c5bb60 0000000000000001
[  232.060588@0] ff00: 0000000000000040 0000007fa3f3b700 0000000000000010 0000000000000000
[  232.068561@0] ff20: 0000000000000001 000000000000270f 0000000000000002 0000000000000000
[  232.076535@0] ff40: 000000555db67bf0 0000007fa3dcc028 0000007fa3eb4a70 0000000000000001
[  232.084508@0] ff60: 0000005584c5b710 0000007fa3eb5560 0000000000000002 0000005584c5b710
[  232.092481@0] ff80: 0000000000000002 0000007fa3eb5648 000000555db50000 000000555db3e000
[  232.100455@0] ffa0: 0000005584c58530 0000007ffd0dfcf0 0000007fa3dcf398 0000007ffd0dfcf0
[  232.108428@0] ffc0: 0000007fa3e23bac 0000000020000000 0000000000000001 0000000000000040
[  232.116401@0] ffe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[  232.124373@0] Call trace:
[  232.126974@0] Exception stack(0xffffffc0c92afb30 to 0xffffffc0c92afc60)
[  232.133561@0] fb20:                                   ffffff800a619000 0000007fffffffff
[  232.141535@0] fb40: ffffffc0c92afd00 ffffff80094c2f70 ffffffc0c92afb70 ffffff800910d30c
[  232.149508@0] fb60: ffffff8009f18420 0000000100000000 ffffffc0c92afc10 ffffff800910d4b8
[  232.157482@0] fb80: ffffffc0c92afc70 ffffff8009f4fc68 ffffff800a676570 ffffff800a676000
[  232.165455@0] fba0: 0000000000000004 0000000000000000 0000000000000123 0000000000000040
[  232.173428@0] fbc0: ffffff8009c02000 ffffffc0c7e1f000 0000000000000001 0000000000000000
[  232.181402@0] fbe0: ffffffc0cf31b7e0 0000000000000000 0000000000000000 0000000000000000
[  232.189375@0] fc00: 0000000000000000 ffffff800a6601e8 ffffff80093982e0 0000000000000001
[  232.197348@0] fc20: 0000000000000365 0000000000000006 0000000000000007 0000000000000000
[  232.205321@0] fc40: ffffff808a7a5fc7 ffffffffffffffff ffffff800922de28 0000007fa3dcc028
[  232.213296@0] [<ffffff80094c2f70>] sysrq_handle_crash+0x28/0x38
[  232.219189@0] [<ffffff80094c35c0>] __handle_sysrq+0xb0/0x1a8
[  232.224822@0] [<ffffff80094c3b90>] write_sysrq_trigger+0x90/0xa0
[  232.230801@0] [<ffffff80092a8ae0>] proc_reg_write+0x90/0xd0
[  232.236348@0] [<ffffff800922b808>] __vfs_write+0x48/0x118
[  232.241721@0] [<ffffff800922c7fc>] vfs_write+0xac/0x1b0
[  232.246921@0] [<ffffff800922de84>] SyS_write+0x5c/0xc8
[  232.252035@0] [<ffffff80090839c0>] el0_svc_naked+0x34/0x38
[  232.257496@0] Code: 52800020 b90a1820 d5033e9f d2800001 (39000020)
[  232.263736@0] ---[ end trace 02a79d7d31e85217 ]---
[  232.274788@0] Kernel panic - not syncing: Fatal exception
[  232.274835@0] SMP: stopping secondary CPUs
[  232.278624@0] Kernel Offset: disabled
[  232.282254@0] Memory Limit: none
[  232.291775@0] Rebooting in 5 seconds..
So it looks work well.

Did you modify 2 configuration files as well after installing Watchdog package? (/etc/default/watchdog, /etc/watchdog.conf)
How about results of service watchdog status?

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Thu Oct 10, 2019 6:21 pm

Hi Joshua,
Thanks.

Code: Select all

Linux 18.04.3 odroid 4.9.190-62 

Code: Select all

apt update && apt full-upgrade -y
I am using

Code: Select all

ubuntu-18.04.3-4.9-minimal-odroid-n2-20190806.img.xz
On 32G SD Card

Code: Select all

root@odroid:~# watchdog -v
watchdog: String 'watchdog-device' found as '/dev/watchdog'
watchdog: Integer 'logtick' found = 1200
watchdog: String 'log-dir' found as '/var/log/watchdog'
watchdog: Variable 'realtime' found as 'yes' = 1
watchdog: Integer 'priority' found = 1
watchdog: Integer 'watchdog-timeout' found = 15

Code: Select all

root@odroid:~# service watchdog status
● watchdog.service - watchdog daemon
   Loaded: loaded (/lib/systemd/system/watchdog.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-10-10 04:42:53 EDT; 12min ago
  Process: 2302 ExecStart=/bin/sh -c [ $run_watchdog != 1 ] || exec /usr/sbin/watchdog $watchdog_options (code=exited, status=0/SUCCESS)
  Process: 2301 ExecStartPre=/bin/sh -c [ -z "${watchdog_module}" ] || [ "${watchdog_module}" = "none" ] || /sbin/modprobe $watchdog_modul
 Main PID: 2304 (watchdog)
    Tasks: 1 (limit: 3843)
   CGroup: /system.slice/watchdog.service
           └─2304 /usr/sbin/watchdog -s -v -c /etc/watchdog.conf

Oct 10 04:42:53 odroid watchdog[2304]: interface: no interface to check
Oct 10 04:42:53 odroid watchdog[2304]: temperature: no sensors to check
Oct 10 04:42:53 odroid watchdog[2304]: no test binary files
Oct 10 04:42:53 odroid watchdog[2304]: no repair binary files
Oct 10 04:42:53 odroid watchdog[2304]: error retry time-out = 60 seconds
Oct 10 04:42:53 odroid watchdog[2304]: repair attempts = 1
Oct 10 04:42:53 odroid watchdog[2304]: alive=/dev/watchdog heartbeat=[none] to=root no_act=no force=no
Oct 10 04:42:53 odroid watchdog[2304]: watchdog now set to 15 seconds
Oct 10 04:42:53 odroid watchdog[2304]: hardware watchdog identity: aml Watchdog
Oct 10 04:42:53 odroid systemd[1]: Started watchdog daemon.

Code: Select all

root@odroid:~# cat /etc/watchdog.conf
#ping                   = 172.31.14.1
#ping                   = 172.26.1.255
#interface              = eth0
#file                   = /var/log/messages
#change                 = 1407

# Uncomment to enable test. Setting one of these values to '0' disables it.
# These values will hopefully never reboot your machine during normal use
# (if your machine is really hung, the loadavg will go much higher than 25)
#max-load-1             = 24
#max-load-5             = 18
#max-load-15            = 12

# Note that this is the number of pages!
# To get the real size, check how large the pagesize is on your machine.
#min-memory             = 1
#allocatable-memory     = 1

#repair-binary          = /usr/sbin/repair
#repair-timeout         = 60
#test-binary            =
#test-timeout           = 60

# The retry-timeout and repair limit are used to handle errors in a more robust
# manner. Errors must persist for longer than retry-timeout to action a repair
# or reboot, and if repair-maximum attempts are made without the test passing a
# reboot is initiated anyway.
#retry-timeout          = 60
#repair-maximum         = 1

watchdog-device = /dev/watchdog

# Defaults compiled into the binary
#temperature-sensor     =
#max-temperature        = 90

# Defaults compiled into the binary
#admin                  = root
#interval               = 1
logtick                = 1200
log-dir         = /var/log/watchdog

# This greatly decreases the chance that watchdog won't be scheduled before
# your machine is really loaded
realtime                = yes
priority                = 1

# Check if rsyslogd is still running by enabling the following line
#pidfile                = /var/run/rsyslogd.pid

watchdog-timeout        = 15

Code: Select all

root@odroid:~# cat /etc/default/watchdog
# Start watchdog at boot time? 0 or 1
run_watchdog=1
# Start wd_keepalive after stopping watchdog? 0 or 1
run_wd_keepalive=1
# Load module before starting watchdog
watchdog_module="none"
# Specify additional watchdog options here (see manpage).
watchdog_options="-s -v -c /etc/watchdog.conf"
I do not know how to get the results from echo c > /proc/sysrq-trigger

When I execute echo c > /proc/sysrq-trigger my terminal window just sits. There is no screen dump or anything.

It appears watchdog is functional but I am unable to force it to restart/reboot the system with "echo c > /proc/sysrq-trigger"

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Thu Oct 10, 2019 8:21 pm

Second Test -
I took a new 32G eMMc that I had that was pre-installed with ubuntu.
I did a full-upgrade and installed watchdog per the N2 instructions.
I rebooted.
Verified watchdog was running with service watchdog status.
I then executed echo c > /proc/sysrq-trigger and it worked just fine.
The N2 rebooted and recovered just fine as it should.
The difference I noticed was that after I executed echo c > /proc/sysrq-trigger the heartbeat LED stayed on continuously where before it would go dead. I'm not sure if that's anything other than maybe timing...
At this point because it worked with eMMc and not with SD card then can you verify that?
Thanks

####
Third test -
I burned an image of the eMMc that worked and then I burned that image onto the SD card.
I booted the N2 with the burned image of the eMMc and service watchdog status was good so I executed echo c > /proc/sysrq-trigger.
Once again the heartbeat LED went solid on.
However, there was no reboot.
This test failed.
I am going to use a brand new different SD card and try it again just to rule out a bad SD card.
...1 hour later...==>> A brand newly written and full-upgraded 32G SD card did not work.
I seem to have issues regarding Watchdog and SD cards...
I think.
Now I took the newly created SD card in the last exercise that Watchdog wouldn't reboot and I burned an .img of it.
I took that .img and burned it to an eMMc.
All went smooth, I booted from the eMMc with the Waychdog that wouldn't reboot on the SD card and executed echo c > /proc/sysrq-trigger
and it worked just fine. The .img that wouldn't work on the SD card works fine on the eMMc.
I have since learned and understood that the on or off condition of the heartbeat LED is irrelevant. It's of no consequence.
Tell me what do you think? Something in the bootstrap maybe?
Command "Reboot" works fine so a hot reboot works.
Can you reproduce this? I've also tried it on a second N2 and it's the same failures.

joshua.yang
Posts: 286
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 2 times
Been thanked: 40 times
Contact:

Re: Hardware watchdog is not working

Unread post by joshua.yang » Fri Oct 11, 2019 12:23 pm

I just tested with our 2 MicroSD UHS U1 cards, 8GB and 16GB and those 2 cards all Watchdog test (echo c > /proc/sysrq-trigger) works. :shock:

Which MicroSD card do you use? Brand, capacity?

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Fri Oct 11, 2019 5:24 pm

That surprises me....
Actually that really surprises me.....
I am using SAMSUNG 32G
Can you think of a next step?
###
Well what an adventure....
I write-formatted a SAMSUNG 32G with SD Card formatter.
I downloaded a "Fresh" .img file from the east coast repository.
Burned that .img to the newly formatted SD card.
Did a full-upgrade.
Did an apt-get install watchdog instead of an apt install watchdog.
Followed watchdog procedure to the T.
Issued echo c > /proc/sysrq-trigger
And it still did not reboot.
So I found an 8G Sandisk SD card and quick formatted it. Burned the .img, did the full-upgrade and installed watchdog accordingly.
I issued echo c > /proc/sysrq-trigger and it worked just as it should.
Do you have a SAMSUNG 32G SD card there to try with? Or any SAMSUNG SD card to verify my results?
Thanks.
###
I've reminded myself of something....
I do not have any issues with these SAMSUNG SD cards using watchdog in my XU4's.
In my XU4's, they work perfectly.
They only don't work in my N2's.
Last edited by additude on Fri Oct 11, 2019 9:13 pm, edited 2 times in total.

User avatar
odroid
Site Admin
Posts: 32508
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 181 times
Been thanked: 349 times
Contact:

Re: Hardware watchdog is not working

Unread post by odroid » Fri Oct 11, 2019 7:00 pm

Do you use Petitboot from the SPI flash memory?
Or boot from the SD card directly?

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Fri Oct 11, 2019 8:42 pm

From SD Card directly.

I edited my last message.

BTW I'm using Samsung 32GB Evo Plus Class 10 Micro SDHC 80MB/S (MB-MC32GA)

to add - I've run tests on the SAMSUNG Cards I have and they are all 20+ write and 80+ read speeds. I also pay strict attention to
any possibilities that they are not genuine. All my cards are genuine SAMSUNG product.

Thanks.

User avatar
odroid
Site Admin
Posts: 32508
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 181 times
Been thanked: 349 times
Contact:

Re: Hardware watchdog is not working

Unread post by odroid » Sat Oct 12, 2019 10:02 am

Very weird.
Do you have a USB-UART cable to find what's wrong in raw kernel messages?

Anyway, we will try an EVO + card early next week.
These users thanked the author odroid for the post:
additude (Fri Oct 18, 2019 7:37 pm)

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Sat Oct 12, 2019 7:31 pm

Yea....Very Weird...
I am always subject to these kinds of 1-off problems.
My wife says, "That's Wes" in a positive way like it's part of what I do....
I have a UART here somewhere, but I can't locate it where I am at so maybe it's at the house.
They are inexpensive enough so I'll just order another one just in case, but I really don't know what I'd be looking for.
But I do want to understand what is happening with this.
If it helps I'm willing to mail you one of my SAMSUNG EVO 32G's in case that the one you test performs and doesn't fail.
Would you consider that the .img burn process is somehow interfering?
...and for the record, just in case, I downloaded .img's from both West Coast and East Coast and burned each one and had the same failure.
I expected them to perform equally, but now that's not an assumption.

joshua.yang
Posts: 286
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 2 times
Been thanked: 40 times
Contact:

Re: Hardware watchdog is not working

Unread post by joshua.yang » Tue Oct 15, 2019 2:40 pm

We could reproduce a similar issue with our Samsung PRO Plus SD card.

It triggers kernel panic normally but doesn't take the reboot procedure. It cannot load bootloader properly.
The kernel panic message is shown through UART serial, but in appearance, It looks like it's just disconnected.

Currently, we're looking into this problem and we will post here as long as we resolve it. Sorry for the inconvenience.

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Tue Oct 15, 2019 6:20 pm

Joshua,
In all due respects, the confirmation is awesome!
It's like finding that needle in the haystack.
I'll be anxiously waiting for the results.
Thanks.

###
Just for the record.
I used a 32G SAMSUNG EVO Select that failed.
I used a 32G San Disk Ultra that worked fine.

joshua.yang
Posts: 286
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 2 times
Been thanked: 40 times
Contact:

Re: Hardware watchdog is not working

Unread post by joshua.yang » Thu Oct 17, 2019 10:44 am

Hi.

We've uploaded the new patch that could resolve the issue to our Github repository.
https://github.com/hardkernel/linux/com ... 59ba4e6697

You can build your own kernel with this patch applied. Please refer to this guide: https://wiki.odroid.com/odroid-n2/softw ... ing_kernel
Or, please wait a few days for our Debian repository updated.

Please let us know the results for the patch if you have tested with. :)

joshua.yang
Posts: 286
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 2 times
Been thanked: 40 times
Contact:

Re: Hardware watchdog is not working

Unread post by joshua.yang » Thu Oct 17, 2019 1:41 pm

Our Debian repository is updated.

You can download the new kernel (version 4.9.196-63) using the following command.

Code: Select all

apt update && apt install linux-odroid-n2
Reboot and check if the kernel updated on your system.

Code: Select all

root@odroid:~# uname -a
Linux odroid 4.9.196-63 #1 SMP PREEMPT Thu Oct 17 00:44:03 -03 2019 aarch64 aarch64 aarch64 GNU/Linux
These users thanked the author joshua.yang for the post (total 2):
additude (Fri Oct 18, 2019 7:36 pm) • rooted (Mon Oct 21, 2019 10:32 pm)

additude
Posts: 66
Joined: Sat Dec 02, 2017 8:02 pm
languages_spoken: english
ODROIDs: .
8-XU4's
11-MC1's
6 - N2's
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Hardware watchdog is not working

Unread post by additude » Fri Oct 18, 2019 7:35 pm

Hi Joshua,
Good Job!!
It's working just fine as it should.
I appreciate your expertise.
Thanks.
Wes

joshua.yang
Posts: 286
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
Has thanked: 2 times
Been thanked: 40 times
Contact:

Re: Hardware watchdog is not working

Unread post by joshua.yang » Mon Oct 21, 2019 9:54 am

additude wrote:
Fri Oct 18, 2019 7:35 pm
Hi Joshua,
Good Job!!
It's working just fine as it should.
I appreciate your expertise.
Thanks.
Wes
Glad to hear that. :)

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 0 guests