Kernel panic - not syncing: hung_task: blocked tasks

Post Reply
meglome
Posts: 76
Joined: Sun Jan 12, 2014 4:42 am
languages_spoken: english
Has thanked: 0
Been thanked: 0
Contact:

Kernel panic - not syncing: hung_task: blocked tasks

Unread post by meglome » Fri Feb 03, 2017 12:38 am

I've been getting the Kernel panic - not syncing: hung_task: blocked tasks message a few times a day on a C1+ with kernel 3.10.104-182 on 16.04

I don't have more than this as the output shown on screen but did not log anywhere that I can see. The odroid was locked up with only the message on screen until I powered it off and back on. I've searched all the log files in /var/log and the text is not there. I'll get a picture next time.

Anyway I've seen from other posts that people have done some kernel updates to solve it. I've written an image three times and run all the updates and the error pops up on each at intervals.

What can I do?

Thanks

User avatar
odroid
Site Admin
Posts: 30282
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 3 times
Been thanked: 26 times
Contact:

Re: Kernel panic - not syncing: hung_task: blocked tasks

Unread post by odroid » Fri Feb 03, 2017 1:20 pm

There is a possibility of the unstable power supply.
Which PSU do you use?

A few people "ondemand CPU governor" could cause the instability. Try the performance governor.
http://odroid.com/dokuwiki/doku.php?id= ... erformance

User avatar
tosiara
Posts: 27
Joined: Wed Sep 17, 2014 11:14 pm
languages_spoken: english
ODROIDs: U3, C1
Has thanked: 0
Been thanked: 0
Contact:

Re: Kernel panic - not syncing: hung_task: blocked tasks

Unread post by tosiara » Wed Apr 10, 2019 5:25 pm

Hi,

I have similar issue. After moving the rootfs into a USB flash drive, my board hangs once per week. On the serial console there is this kernel panic:
[899871.649595@2] INFO: task jbd2/sda2-8:167 blocked for more than 120 seconds.
[899871.651135@2] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[899871.659373@2] BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/43
[899871.673752@2] Kernel panic - not syncing: hung_task: blocked tasks
[899871.674495@2] CPU: 2 PID: 43 Comm: khungtaskd Not tainted 3.10.107-190 #1
[899871.681380@2] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[899871.690187@2] [<c0011d74>] (show_stack+0x10/0x14) from [<c068d28c>] (panic+0x9c/0x1ec)
[899871.698167@2] [<c068d28c>] (panic+0x9c/0x1ec) from [<c0095344>] (watchdog+0x248/0x284)
[899871.706149@2] [<c0095344>] (watchdog+0x248/0x284) from [<c004c718>] (kthread+0xb0/0xb8)
[899871.714225@2] [<c004c718>] (kthread+0xb0/0xb8) from [<c000df80>] (ret_from_fork+0x14/0x34)
[899871.722563@3] CPU3: stopping
[899871.725412@3] CPU: 3 PID: 497 Comm: rs:main Q:Reg Not tainted 3.10.107-190 #1
[899871.732696@3] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[899871.741486@3] [<c0011d74>] (show_stack+0x10/0x14) from [<c0013650>] (handle_IPI+0xd4/0x17c)
[899871.749903@3] [<c0013650>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[899871.758681@3] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dc7c>] (__irq_usr+0x3c/0x60)
[899871.767269@3] Exception stack(0xeac5ffb0 to 0xeac5fff8)
[899871.772512@3] ffa0: b51339bc 00000000 00000000 00096280
[899871.780990@3] ffc0: b51338c0 b5afea88 00099c28 b5afea88 00093e00 0008544c 00099c28 000831c4
[899871.789412@3] ffe0: 00081068 b5afea30 00034bf3 00021958 80000030 ffffffff
[899871.796244@0] CPU0: stopping
[899871.799135@0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.107-190 #1
[899871.805887@0] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[899871.814691@0] [<c0011d74>] (show_stack+0x10/0x14) from [<c0013650>] (handle_IPI+0xd4/0x17c)
[899871.823108@0] [<c0013650>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[899871.831885@0] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dac0>] (__irq_svc+0x40/0x70)
[899871.840474@0] Exception stack(0xc096ff70 to 0xc096ffb8)
[899871.845717@0] ff60: 00000000 00000000 36cd0dc8 00000000
[899871.854195@0] ff80: c096e000 c096e000 c0982590 c069b038 c096e000 c09e3ade c0847980 00000000
[899871.862617@0] ffa0: 01000000 c096ffb8 c000f2bc c000f2c0 600f0013 ffffffff
[899871.869456@0] [<c000dac0>] (__irq_svc+0x40/0x70) from [<c000f2c0>] (arch_cpu_idle+0x28/0x2c)
[899871.877995@0] [<c000f2c0>] (arch_cpu_idle+0x28/0x2c) from [<c00701d8>] (cpu_startup_entry+0xf4/0x148)
[899871.887300@0] [<c00701d8>] (cpu_startup_entry+0xf4/0x148) from [<c0935af8>] (start_kernel+0x340/0x34c)
[899871.896674@1] CPU1: stopping
[899871.899521@1] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.107-190 #1
[899871.906273@1] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[899871.915077@1] [<c0011d74>] (show_stack+0x10/0x14) from [<c0013650>] (handle_IPI+0xd4/0x17c)
[899871.923494@1] [<c0013650>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[899871.932271@1] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dac0>] (__irq_svc+0x40/0x70)
[899871.940860@1] Exception stack(0xec69ffa0 to 0xec69ffe8)
[899871.946105@1] ffa0: 00000001 00000000 17c32a80 00000000 ec69e000 ec69e000 c0982590 c069b038
[899871.954582@1] ffc0: ec69e000 c09e3ade c0847980 00000000 01000000 ec69ffe8 c000f2bc c000f2c0
[899871.963001@1] ffe0: 600d0013 ffffffff
[899871.966662@1] [<c000dac0>] (__irq_svc+0x40/0x70) from [<c000f2c0>] (arch_cpu_idle+0x28/0x2c)
[899871.975252@1] [<c000f2c0>] (arch_cpu_idle+0x28/0x2c) from [<c00701d8>] (cpu_startup_entry+0xf4/0x148)
[899871.984565@1] [<c00701d8>] (cpu_startup_entry+0xf4/0x148) from [<00889e84>] (0x889e84)
Sometimes reboot helps, but sometimes it crashes right after booting:
[ 129.130053@2] INFO: rcu_preempt self-detected stall on CPU { 3} (t=2100 jiffies g=1101 c=1100 q=4706)
[ 129.144774@1] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 1, t=2102 jiffies, g=1101, c=1100, q=4706)
[ 132.604154@3] BUG: soft lockup - CPU#3 stuck for 22s! [kworker/3:2:156]
[ 132.648105@3] Kernel panic - not syncing: softlockup: hung tasks
[ 132.650480@3] CPU: 3 PID: 156 Comm: kworker/3:2 Not tainted 3.10.107-190 #1
[ 132.657670@3] Workqueue: events od_dbs_timer
[ 132.661165@3] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[ 132.671138@3] [<c0011d74>] (show_stack+0x10/0x14) from [<c068d28c>] (panic+0x9c/0x1ec)
[ 132.678883@3] [<c068d28c>] (panic+0x9c/0x1ec) from [<c00957c8>] (watchdog_timer_fn+0x234/0x280)
[ 132.687775@3] [<c00957c8>] (watchdog_timer_fn+0x234/0x280) from [<c004f630>] (__run_hrtimer+0x8c/0x108)
[ 132.697310@3] [<c004f630>] (__run_hrtimer+0x8c/0x108) from [<c00500e0>] (hrtimer_interrupt+0x164/0x2cc)
[ 132.706733@3] [<c00500e0>] (hrtimer_interrupt+0x164/0x2cc) from [<c001fea0>] (meson_timer_interrupt+0x74/0x9c)
[ 132.716901@3] [<c001fea0>] (meson_timer_interrupt+0x74/0x9c) from [<c00962a0>] (handle_irq_event_percpu+0x48/0x178)
[ 132.727339@3] [<c00962a0>] (handle_irq_event_percpu+0x48/0x178) from [<c009640c>] (handle_irq_event+0x3c/0x5c)
[ 132.737256@3] [<c009640c>] (handle_irq_event+0x3c/0x5c) from [<c009919c>] (handle_fasteoi_irq+0x104/0x148)
[ 132.746767@3] [<c009919c>] (handle_fasteoi_irq+0x104/0x148) from [<c0095b68>] (generic_handle_irq+0x28/0x38)
[ 132.756762@3] [<c0095b68>] (generic_handle_irq+0x28/0x38) from [<c000ee24>] (handle_IRQ+0x7c/0x8c)
[ 132.765289@3] [<c000ee24>] (handle_IRQ+0x7c/0x8c) from [<c0008458>] (gic_handle_irq+0x40/0x5c)
[ 132.773858@3] [<c0008458>] (gic_handle_irq+0x40/0x5c) from [<c000dac0>] (__irq_svc+0x40/0x70)
[ 132.782172@3] Exception stack(0xeb77fbe0 to 0xeb77fc28)
[ 132.786452@3] fbe0: ebe13500 80070113 eb6c0608 0000476d 00000000 ec8e7000 00000000 00000001
[ 132.795643@3] fc00: ebf19e18 eb9dea80 ebe0d900 ebf19e00 eb9deaac eb77fc28 c0432588 c0697660
[ 132.803923@3] fc20: 60070113 ffffffff
[ 132.806411@3] [<c000dac0>] (__irq_svc+0x40/0x70) from [<c0697660>] (_raw_spin_unlock_irqrestore+0x20/0x48)
[ 132.817657@3] [<c0697660>] (_raw_spin_unlock_irqrestore+0x20/0x48) from [<c0432588>] (dwc_otg_hcd_urb_enqueue+0x1c4/0x1d4)
[ 132.829057@3] [<c0432588>] (dwc_otg_hcd_urb_enqueue+0x1c4/0x1d4) from [<c0433f6c>] (urb_enqueue+0x204/0x27c)
[ 132.838567@3] [<c0433f6c>] (urb_enqueue+0x204/0x27c) from [<c0329df8>] (usb_hcd_submit_urb+0x5c4/0x6e0)
[ 132.848073@3] [<c0329df8>] (usb_hcd_submit_urb+0x5c4/0x6e0) from [<bf10e00c>] (uvc_video_complete+0xb8/0xd4 [uvcvideo])
[ 132.859170@3] [<bf10e00c>] (uvc_video_complete+0xb8/0xd4 [uvcvideo]) from [<c0328ac4>] (usb_hcd_giveback_urb+0xac/0xf4)
[ 132.869740@3] [<c0328ac4>] (usb_hcd_giveback_urb+0xac/0xf4) from [<c0433bd8>] (_complete_in_tasklet+0x18/0x20)
[ 132.879807@3] [<c0433bd8>] (_complete_in_tasklet+0x18/0x20) from [<c0430694>] (isoc_complete_tasklet_func+0x54/0x5c)
[ 132.890378@3] [<c0430694>] (isoc_complete_tasklet_func+0x54/0x5c) from [<c0034cb0>] (tasklet_action+0x98/0x100)
[ 132.900377@3] [<c0034cb0>] (tasklet_action+0x98/0x100) from [<c0034e08>] (__do_softirq+0xf0/0x1b8)
[ 132.909063@3] [<c0034e08>] (__do_softirq+0xf0/0x1b8) from [<c0034f88>] (do_softirq+0x48/0x50)
[ 132.917423@3] [<c0034f88>] (do_softirq+0x48/0x50) from [<c00351ec>] (irq_exit+0x80/0xbc)
[ 132.925312@3] [<c00351ec>] (irq_exit+0x80/0xbc) from [<c000ee28>] (handle_IRQ+0x80/0x8c)
[ 132.933430@3] [<c000ee28>] (handle_IRQ+0x80/0x8c) from [<c0008458>] (gic_handle_irq+0x40/0x5c)
[ 132.942261@3] [<c0008458>] (gic_handle_irq+0x40/0x5c) from [<c000dac0>] (__irq_svc+0x40/0x70)
[ 132.950678@3] Exception stack(0xeb77fdc8 to 0xeb77fe10)
[ 132.954795@3] fdc0: c09e4150 60070113 00000002 000001b9 60070113 00000000
[ 132.964085@3] fde0: 00520800 05b8d800 c0988ddc 00000000 c0982dac 00018bfc 00000001 eb77fe10
[ 132.972319@3] fe00: c001edc4 c0697660 60070113 ffffffff
[ 132.976781@3] [<c000dac0>] (__irq_svc+0x40/0x70) from [<c0697660>] (_raw_spin_unlock_irqrestore+0x20/0x48)
[ 132.987609@3] [<c0697660>] (_raw_spin_unlock_irqrestore+0x20/0x48) from [<c001edc4>] (meson_clk_set_rate+0xd4/0x198)
[ 132.998413@3] [<c001edc4>] (meson_clk_set_rate+0xd4/0x198) from [<c001ef40>] (clk_set_rate+0xb8/0xd8)
[ 133.007244@3] [<c001ef40>] (clk_set_rate+0xb8/0xd8) from [<c043ab08>] (meson_cpufreq_target+0x154/0x238)
[ 133.016765@3] [<c043ab08>] (meson_cpufreq_target+0x154/0x238) from [<c03beb78>] (dbs_check_cpu+0x270/0x278)
[ 133.026594@3] [<c03beb78>] (dbs_check_cpu+0x270/0x278) from [<c03ba910>] (od_dbs_timer+0x90/0x140)
[ 133.035354@3] [<c03ba910>] (od_dbs_timer+0x90/0x140) from [<c0045b6c>] (process_one_work+0x260/0x414)
[ 133.044537@3] [<c0045b6c>] (process_one_work+0x260/0x414) from [<c00464fc>] (worker_thread+0x258/0x438)
[ 133.053992@3] [<c00464fc>] (worker_thread+0x258/0x438) from [<c004c718>] (kthread+0xb0/0xb8)
[ 133.062219@3] [<c004c718>] (kthread+0xb0/0xb8) from [<c000df80>] (ret_from_fork+0x14/0x34)
[ 133.070262@0] CPU0: stopping
[ 133.071542@0] CPU: 0 PID: 1644 Comm: curl Not tainted 3.10.107-190 #1
[ 133.078985@0] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[ 133.088232@0] [<c0011d74>] (show_stack+0x10/0x14) from [<c0013650>] (handle_IPI+0xd4/0x17c)
[ 133.096472@0] [<c0013650>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[ 133.105253@0] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dac0>] (__irq_svc+0x40/0x70)
[ 133.113632@0] Exception stack(0xe9e79d50 to 0xe9e79d98)
[ 133.117974@0] 9d40: 00000001 00000003 3cf00000 00000002
[ 133.127182@0] 9d60: 3850359f 00038503 00000000 c11b8fdc c1382060 00000000 00000000 00000000
[ 133.135504@0] 9d80: 00000003 e9e79d98 c0a81e70 c001a758 200e0113 ffffffff
[ 133.141940@0] [<c000dac0>] (__irq_svc+0x40/0x70) from [<c001a758>] (__sync_icache_dcache+0x38/0x80)
[ 133.151568@0] [<c001a758>] (__sync_icache_dcache+0x38/0x80) from [<c00cb448>] (__do_fault+0x388/0x618)
[ 133.287434@0] [<c00cb448>] (__do_fault+0x388/0x618) from [<c00ce3e0>] (handle_pte_fault+0x70/0x8a8)
[ 133.421116@0] [<c00ce3e0>] (handle_pte_fault+0x70/0x8a8) from [<c00ced40>] (handle_mm_fault+0x128/0x1cc)
[ 133.556604@0] [<c00ced40>] (handle_mm_fault+0x128/0x1cc) from [<c0019e1c>] (do_page_fault+0x11c/0x3bc)
[ 133.690702@0] [<c0019e1c>] (do_page_fault+0x11c/0x3bc) from [<c0008328>] (do_DataAbort+0x38/0x94)
[ 133.823365@0] [<c0008328>] (do_DataAbort+0x38/0x94) from [<c000dc34>] (__dabt_usr+0x34/0x40)
[ 133.954528@0] Exception stack(0xe9e79fb0 to 0xe9e79ff8)
[ 134.082729@0] 9fa0: 00032315 0000038d 00000407 b6865928
[ 134.214658@0] 9fc0: b6f895e8 0000001d b6f875fc 00000004 0ca24d20 00000000 00000000 00000026
[ 134.346607@0] 9fe0: 00000000 be902a40 b6f6f22f b6f6f232 200e0030 ffffffff
[ 134.476275@1] CPU1: stopping
[ 134.603705@1] CPU: 1 PID: 1211 Comm: mysqld Not tainted 3.10.107-190 #1
[ 134.732617@1] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[ 134.864149@1] [<c0011d74>] (show_stack+0x10/0x14) from [<c0013650>] (handle_IPI+0xd4/0x17c)
[ 134.995076@1] [<c0013650>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[ 135.126002@1] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dac0>] (__irq_svc+0x40/0x70)
[ 135.256268@1] Exception stack(0xe6999e98 to 0xe6999ee0)
[ 135.362924@3] SMP: failed to stop secondary CPUs
[ 135.382673@1] 9e80: c142e980 e6998000
[ 135.512357@1] 9ea0: 00000000 00008afb c142e980 e6c81b80 e6c81e60 00000001 e6c81b80 c096c980
[ 135.641852@1] 9ec0: c0983060 e6999f8c 00000000 e6999ee0 c06963d4 c06976a8 60070013 ffffffff
[ 135.771151@1] [<c000dac0>] (__irq_svc+0x40/0x70) from [<c06976a8>] (_raw_spin_unlock_irq+0x20/0x48)
[ 135.901446@1] [<c06976a8>] (_raw_spin_unlock_irq+0x20/0x48) from [<c06963d4>] (__schedule+0x644/0x728)
[ 136.032307@1] [<c06963d4>] (__schedule+0x644/0x728) from [<c005966c>] (sys_sched_yield+0x80/0x88)
[ 136.162634@1] [<c005966c>] (sys_sched_yield+0x80/0x88) from [<c000dee0>] (ret_fast_syscall+0x0/0x38)
[ 136.293329@2] CPU2: stopping
[ 136.419552@2] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.10.107-190 #1
[ 136.547295@2] [<c0014dd4>] (unwind_backtrace+0x0/0xec) from [<c0011d74>] (show_stack+0x10/0x14)
[ 136.677457@2] [<c0011d74>] (show_stack+0x10/0x14) from [<c0013650>] (handle_IPI+0xd4/0x17c)
[ 136.806491@2] [<c0013650>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[ 136.935214@2] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dac0>] (__irq_svc+0x40/0x70)
[ 137.063393@2] Exception stack(0xec6a1fa0 to 0xec6a1fe8)
[ 137.187757@2] 1fa0: 00000002 00000000 0000c420 00000000 ec6a0000 ec6a0000 c0982590 c069b038
[ 137.315079@2] 1fc0: ec6a0000 c09e3ade c0847980 00000000 01000000 ec6a1fe8 c000f2bc c000f2c0
[ 137.441221@2] 1fe0: 60000013 ffffffff
[ 137.563538@2] [<c000dac0>] (__irq_svc+0x40/0x70) from [<c000f2c0>] (arch_cpu_idle+0x28/0x2c)
[ 137.689507@2] [<c000f2c0>] (arch_cpu_idle+0x28/0x2c) from [<c00701d8>] (cpu_startup_entry+0xf4/0x148)
[ 137.816208@2] [<c00701d8>] (cpu_startup_entry+0xf4/0x148) from [<00889e84>] (0x889e84)
I have followed steps to set performance governor http://odroid.com/dokuwiki/doku.php?id= ... erformance and will monitor if that helps. Will update if that helped

User avatar
tosiara
Posts: 27
Joined: Wed Sep 17, 2014 11:14 pm
languages_spoken: english
ODROIDs: U3, C1
Has thanked: 0
Been thanked: 0
Contact:

Re: Kernel panic - not syncing: hung_task: blocked tasks

Unread post by tosiara » Thu Apr 18, 2019 4:24 pm

Unfortunately, that didn't help. The board randomly hangs every week and dumps the same kernel panic into serial console

User avatar
odroid
Site Admin
Posts: 30282
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 3 times
Been thanked: 26 times
Contact:

Re: Kernel panic - not syncing: hung_task: blocked tasks

Unread post by odroid » Thu Apr 18, 2019 4:44 pm

Which USB devices do you connect to C1+?
If you use a UVC camera, consider using the OTG port via OTG-to-Host cable.

Can you double check the current governor?
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 2 guests