BUG: Kernel crashes (khungtaskd, heavy SD load)

Post Reply
Clx
Posts: 50
Joined: Fri Jan 16, 2015 9:23 pm
languages_spoken: english
ODROIDs: ODROID-C1
Contact:

BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by Clx » Wed Feb 18, 2015 11:44 pm

Hi all,

I'm encountering unexpected kernel crashes under heavy SD load. Distro is Ubuntu 14.04 v1.3, default kernel 3.10.67-55 #1 SMP PREEMPT Tue Feb 3

Crash log:

Code: Select all

root@navio-odroid:~# uname -a
Linux navio-odroid 3.10.67-55 #1 SMP PREEMPT Tue Feb 3 12:46:15 BRST 2015 armv7l armv7l armv7l GNU/Linux

root@navio-odroid:~# top

top - 00:19:40 up 46 min,  2 users,  load average: 8.53, 4.03, 6.70
Tasks: 125 total,  13 running, 111 sleeping,   1 stopped,   0 zombie
%Cpu(s): 80.6 us, 19.4 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   1009312 total,   671028 used,   338284 free,    13284 buffers
KiB Swap:        0 total,        0 used,        0 free.   481724 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 4574 odroid    20   0    1644     72      0 R  43.9  0.0   0:18.19 stress
 4581 odroid    20   0    1644     72      0 R  42.3  0.0   0:17.13 stress
 4582 odroid    20   0    1644     72      0 R  41.3  0.0   0:17.61 stress
 4584 odroid    20   0    1644     72      0 R  40.9  0.0   0:16.39 stress
 4570 odroid    20   0    1644     72      0 R  38.0  0.0   0:16.99 stress
 4572 odroid    20   0  132720   8452    184 R  38.0  0.8   0:16.65 stress
 4577 odroid    20   0    1644     72      0 R  37.6  0.0   0:16.09 stress
 4576 odroid    20   0  132720   5800    184 R  36.7  0.6   0:16.07 stress
 4579 odroid    20   0    1644     72      0 R  35.3  0.0   0:16.23 stress
 4583 odroid    20   0    1644     72      0 R  34.3  0.0   0:16.98 stress
 4573 odroid    20   0    2540   1060    120 R   7.6  0.1   0:02.92 stress
   74 root      20   0       0      0      0 S   2.6  0.0   0:25.06 mmcqd/0
 4585 root      20   0       0      0      0 D   1.7  0.0   0:00.44 kworker/u8+
 4586 root      20   0    4604   1128    784 R   0.7  0.1   0:00.05 top
   35 root      20   0       0      0      0 S   0.3  0.0   0:00.47 kthread_hd+
   73 root     -51   0       0      0      0 S   0.3  0.0   0:45.63 irq/110-sd+
   77 root      20   0       0      0      0 S   0.3  0.0   0:00.35 kthread_hd+
root@navio-odroid:~# [ 3480.864774@1] INFO: task stress:4625 blocked for more than 120 seconds.
[ 3480.865751@1] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 3480.873863@1] BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/43
[ 3480.884198@1] Kernel panic - not syncing: hung_task: blocked tasks
[ 3480.888080@1] CPU: 1 PID: 43 Comm: khungtaskd Not tainted 3.10.67-55 #1
[ 3480.894681@1] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[ 3480.903374@1] [<c0011a78>] (show_stack+0x10/0x14) from [<c0633b98>] (panic+0x9c/0x1ec)
[ 3480.911264@1] [<c0633b98>] (panic+0x9c/0x1ec) from [<c008bcc0>] (watchdog+0x254/0x298)
[ 3480.919166@1] [<c008bcc0>] (watchdog+0x254/0x298) from [<c004808c>] (kthread+0xb0/0xb4)
[ 3480.927155@1] [<c004808c>] (kthread+0xb0/0xb4) from [<c000e058>] (ret_from_fork+0x14/0x3c)
[ 3480.935402@3] CPU3: stopping
[ 3480.938265@3] CPU: 3 PID: 4629 Comm: stress Not tainted 3.10.67-55 #1
[ 3480.944702@3] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[ 3480.953378@3] [<c0011a78>] (show_stack+0x10/0x14) from [<c00132b4>] (handle_IPI+0xd4/0x17c)
[ 3480.961714@3] [<c00132b4>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[ 3480.970397@3] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dd3c>] (__irq_usr+0x3c/0x60)
[ 3480.978903@3] Exception stack(0xe6d21fb0 to 0xe6d21ff8)
[ 3480.984115@3] 1fa0:                                     1f5d3e62 00000000 00000001 b6ea34e8
[ 3480.992453@3] 1fc0: 00000000 00000003 000150b4 000150a0 000150a0 0000000d 0000002d 000150b0
[ 3481.000789@3] 1fe0: 00015080 be9c3334 b6de8e71 b6de8e8e 60070030 ffffffff
[ 3481.007561@0] CPU0: stopping
[ 3481.010427@0] CPU: 0 PID: 4631 Comm: stress Not tainted 3.10.67-55 #1
[ 3481.016861@0] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[ 3481.025541@0] [<c0011a78>] (show_stack+0x10/0x14) from [<c00132b4>] (handle_IPI+0xd4/0x17c)
[ 3481.033876@0] [<c00132b4>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[ 3481.042560@0] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dd3c>] (__irq_usr+0x3c/0x60)
[ 3481.051066@0] Exception stack(0xe6da7fb0 to 0xe6da7ff8)
[ 3481.056278@0] 7fa0:                                     00000000 be9c332c b6ea21bc b6ea24c8
[ 3481.064616@0] 7fc0: 00000000 00000001 000150b4 000150a0 000150a0 0000000f 0000002d 000150b0
[ 3481.072952@0] 7fe0: 00015080 be9c3328 b6de8e71 b6de8fa2 60070030 ffffffff
[ 3481.079724@2] CPU2: stopping
[ 3481.082591@2] CPU: 2 PID: 4628 Comm: stress Not tainted 3.10.67-55 #1
[ 3481.089024@2] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[ 3481.097704@2] [<c0011a78>] (show_stack+0x10/0x14) from [<c00132b4>] (handle_IPI+0xd4/0x17c)
[ 3481.106040@2] [<c00132b4>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[ 3481.114724@2] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dd3c>] (__irq_usr+0x3c/0x60)
[ 3481.123229@2] Exception stack(0xe6765fb0 to 0xe6765ff8)
[ 3481.128441@2] 5fa0:                                     00000000 00000000 00000001 b6ea34e8
[ 3481.136780@2] 5fc0: 00000000 00000004 000150b4 000150a0 000150a0 0000000c 0000002d 000150b0
[ 3481.145115@2] 5fe0: 00015080 be9c3328 0000a527 b6de8e56 60070030 ffffffff
To reproduce run the following (requires 2gb of free space):

Code: Select all

stress --cpu 8 --io 4 --vm 2 --hdd 2 --vm-bytes 128M --timeout 2500s

I tried using the stress command without --hdd parameter and it worked fine all night. But with --hdd parameter it crashes in 5-15 minutes.
I've tested a couple of SD cards and it's the same (one card is SanDisk Ultra 8Gb MicroSDHC Class 10 UHS-I SDSDQUI-008G-U46, another one is Transcend unknown model).
Currently I don't have an eMMC module to test with it (but ordered one, so will be able to do that some time soon).

Process temperature during the stress test is ~65C.
Last edited by Clx on Fri Feb 20, 2015 2:11 am, edited 1 time in total.

User avatar
odroid
Site Admin
Posts: 30015
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by odroid » Thu Feb 19, 2015 12:08 pm

We will try the same test just after Lunar New Year Holidays.

Clx
Posts: 50
Joined: Fri Jan 16, 2015 9:23 pm
languages_spoken: english
ODROIDs: ODROID-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by Clx » Fri Feb 20, 2015 2:13 am

odroid wrote:We will try the same test just after Lunar New Year Holidays.
Thanks! I'll do more tests in the meanwhile.

I tried the same stress with the USB flash drive and got a similar result:

Code: Select all

root@odroid:~# uname -a
Linux odroid 3.10.67-55 #1 SMP PREEMPT Tue Feb 3 12:46:15 BRST 2015 armv7l armv7l armv7l GNU/Linux
root@odroid:~#
root@odroid:~# top
top - 02:34:12 up 8 min,  3 users,  load average: 14.04, 4.20, 1.48
Tasks: 138 total,  12 running, 126 sleeping,   0 stopped,   0 zombie
%Cpu(s): 12.1 us,  3.6 sy,  0.0 ni, 83.9 id,  0.4 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:   1009312 total,   386844 used,   622468 free,    25524 buffers
KiB Swap:        0 total,        0 used,        0 free.   129012 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 1867 root      20   0  132720  66256    184 R  40.1  6.6   0:10.29 stress
 1871 root      20   0  132720  19528    184 R  35.1  1.9   0:09.65 stress
 1873 root      20   0    1644     72      0 R  35.1  0.0   0:09.56 stress
 1876 root      20   0    1644     72      0 R  35.1  0.0   0:09.43 stress
 1878 root      20   0    1644     72      0 R  35.1  0.0   0:09.49 stress
 1879 root      20   0    1644     72      0 R  35.1  0.0   0:09.61 stress
 1869 root      20   0    1644     72      0 R  30.0  0.0   0:09.71 stress
 1880 root      20   0    1644     72      0 R  30.0  0.0   0:09.54 stress
 1865 root      20   0    1644     72      0 R  25.0  0.0   0:09.49 stress
 1881 root      20   0    1644     72      0 R  25.0  0.0   0:09.54 stress
 1918 root      20   0    4600   1092    764 R  10.0  0.1   0:00.04 top
 1726 root      20   0    1644     72      0 D   5.0  0.0   0:00.01 stress
    1 root      20   0    3556   2140   1128 S   0.0  0.2   0:05.65 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
    3 root      20   0       0      0      0 S   0.0  0.0   0:00.00 ksoftirqd/0
    4 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0
    5 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:+

root@odroid:~#
root@odroid:~# [  600.930168@1] INFO: task stress:1716 blocked for more than 120 seconds.
[  600.931145@1] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  600.940928@1] BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/43
[  600.980085@1] Kernel panic - not syncing: hung_task: blocked tasks
[  600.980637@1] CPU: 1 PID: 43 Comm: khungtaskd Not tainted 3.10.67-55 #1
[  600.987253@1] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[  600.995926@1] [<c0011a78>] (show_stack+0x10/0x14) from [<c0633b98>] (panic+0x9c/0x1ec)
[  601.003829@1] [<c0633b98>] (panic+0x9c/0x1ec) from [<c008bcc0>] (watchdog+0x254/0x298)
[  601.011733@1] [<c008bcc0>] (watchdog+0x254/0x298) from [<c004808c>] (kthread+0xb0/0xb4)
[  601.019722@1] [<c004808c>] (kthread+0xb0/0xb4) from [<c000e058>] (ret_from_fork+0x14/0x3c)
[  601.027966@0] CPU0: stopping
[  601.030830@0] CPU: 0 PID: 1869 Comm: stress Not tainted 3.10.67-55 #1
[  601.037264@0] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[  601.045942@0] [<c0011a78>] (show_stack+0x10/0x14) from [<c00132b4>] (handle_IPI+0xd4/0x17c)
[  601.054277@0] [<c00132b4>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[  601.062961@0] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dd3c>] (__irq_usr+0x3c/0x60)
[  601.071466@0] Exception stack(0xe8fedfb0 to 0xe8fedff8)
[  601.076679@0] dfa0:                                     00000003 be93847c 00000001 b6e794c8
[  601.085016@0] dfc0: 00000000 00000007 000150b4 000150a0 000150a0 00000004 0000002d 000150b0
[  601.093352@0] dfe0: 00015080 be938468 b6dbfe71 b6dbff76 a00d0030 ffffffff
[  601.100125@3] CPU3: stopping
[  601.102992@3] CPU: 3 PID: 1876 Comm: stress Not tainted 3.10.67-55 #1
[  601.109424@3] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[  601.118104@3] [<c0011a78>] (show_stack+0x10/0x14) from [<c00132b4>] (handle_IPI+0xd4/0x17c)
[  601.126440@3] [<c00132b4>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[  601.135124@3] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dd3c>] (__irq_usr+0x3c/0x60)
[  601.143630@3] Exception stack(0xe8fc7fb0 to 0xe8fc7ff8)
[  601.148842@3] 7fa0:                                     b6e7921c be93847c b6e791b0 b6e794c8
[  601.157180@3] 7fc0: 00000000 b6e79210 000150b4 000150a0 000150a0 0000000b 0000002d 000150b0
[  601.165516@3] 7fe0: 00015080 be938468 b6dbfe71 b6dbff7e a00d0030 ffffffff
[  601.172288@2] CPU2: stopping
[  601.175154@2] CPU: 2 PID: 1881 Comm: stress Not tainted 3.10.67-55 #1
[  601.181587@2] [<c00149dc>] (unwind_backtrace+0x0/0xec) from [<c0011a78>] (show_stack+0x10/0x14)
[  601.190268@2] [<c0011a78>] (show_stack+0x10/0x14) from [<c00132b4>] (handle_IPI+0xd4/0x17c)
[  601.198603@2] [<c00132b4>] (handle_IPI+0xd4/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[  601.207287@2] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dd3c>] (__irq_usr+0x3c/0x60)
[  601.215793@2] Exception stack(0xe913ffb0 to 0xe913fff8)
[  601.221005@2] ffa0:                                     61359735 00000000 00000001 b6e7a4e8
[  601.229343@2] ffc0: 00000000 00000001 000150b4 000150a0 000150a0 00000010 0000002d 000150b0
[  601.237679@2] ffe0: 00015080 be938488 b6dbfe71 0000a52e 600d0030 ffffffff


adbensi
Posts: 80
Joined: Fri Apr 18, 2014 10:29 am
languages_spoken: Português
ODROIDs: U3 and C1
Location: Brazil
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by adbensi » Fri Feb 20, 2015 8:20 am

I will try it on the emmc module.
more closer to solve it ;) thanks all.

Clx
Posts: 50
Joined: Fri Jan 16, 2015 9:23 pm
languages_spoken: english
ODROIDs: ODROID-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by Clx » Sun Feb 22, 2015 8:42 pm

adbensi wrote:I will try it on the emmc module.
more closer to solve it ;) thanks all.
I tried that on eMMC for 30 minutes, but it didn't crash. Maybe it needs more time on eMMC:)

User avatar
streetboy
Posts: 196
Joined: Tue Feb 26, 2013 6:43 pm
languages_spoken: english
ODROIDs: ODROID-X,U2,XU3-Lite
C1, XU4
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by streetboy » Sun Feb 22, 2015 9:31 pm

Hey! Rich people! it is really scary because the eMMC modules are hell expensive for me. :)

User avatar
odroid
Site Admin
Posts: 30015
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by odroid » Mon Feb 23, 2015 7:35 pm

We've tried the SD load test for 30minutes but we couldn't reproduce the issue.
We used our official Sandisk 16GB UHS-1 micro-SD card with the latest Kernel 3.10.69-70.

We will try the same test with some other SD samples tomorrow.
BTW, what is your settings of "setenv disableuhs "disableuhs" in the boot.ini? Did you comment out the "disableuhs" line?

Clx
Posts: 50
Joined: Fri Jan 16, 2015 9:23 pm
languages_spoken: english
ODROIDs: ODROID-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by Clx » Tue Feb 24, 2015 1:37 am

odroid wrote:We've tried the SD load test for 30minutes but we couldn't reproduce the issue.
We used our official Sandisk 16GB UHS-1 micro-SD card with the latest Kernel 3.10.69-70.

We will try the same test with some other SD samples tomorrow.
BTW, what is your settings of "setenv disableuhs "disableuhs" in the boot.ini? Did you comment out the "disableuhs" line?
Thanks!
I've tried both with uhs and without it, the result was the same.

pdenhaan
Posts: 8
Joined: Mon Feb 16, 2015 11:12 pm
languages_spoken: english
ODROIDs: Odroid-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by pdenhaan » Fri Mar 06, 2015 7:21 am

I've got a very similar crash when building Docker images on Arch Linux (kernel 3.10.69, 3.10.70; tried both UHS-1 enabled and disabled). Clearly this is another instance of high SD card load, so that could well be it, but I'm no kernel guru. My SD card is one of the recommended cards - a 16GB Sandisk Extreme.

Code: Select all

[  360.836645] INFO: task docker:213 blocked for more than 120 seconds.
[  360.837535] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  360.845523] docker          D c06c9420     0   213      1 0x00000000
[  360.852057] [<c06c9420>] (__schedule+0x604/0x730) from [<c06ca21c>] (__down_read+0xc8/0xe8)
[  360.860761] [<c06ca21c>] (__down_read+0xc8/0xe8) from [<c0229ec8>] (di_read_lock+0x1c/0x88)
[  360.869207] [<c0229ec8>] (di_read_lock+0x1c/0x88) from [<c0233ff8>] (au_h_path_getattr+0xe8/0x1e0)
[  360.878318] [<c0233ff8>] (au_h_path_getattr+0xe8/0x1e0) from [<c023db74>] (au_lgxattr+0x38/0xdc)
[  360.887268] [<c023db74>] (au_lgxattr+0x38/0xdc) from [<c023e014>] (aufs_getxattr+0x24/0x2c)
[  360.895639] [<c023e014>] (aufs_getxattr+0x24/0x2c) from [<c0253fd8>] (smk_fetch.isra.16+0x68/0x90)
[  360.904947] [<c0253fd8>] (smk_fetch.isra.16+0x68/0x90) from [<c025416c>] (smack_d_instantiate+0x16c/0x290)
[  360.914708] [<c025416c>] (smack_d_instantiate+0x16c/0x290) from [<c010c99c>] (d_splice_alias+0xb0/0xdc)
[  360.924253] [<c010c99c>] (d_splice_alias+0xb0/0xdc) from [<c02329ec>] (aufs_lookup+0xf4/0x20c)
[  360.933026] [<c02329ec>] (aufs_lookup+0xf4/0x20c) from [<c0100ee4>] (lookup_real+0x30/0x4c)
[  360.941529] [<c0100ee4>] (lookup_real+0x30/0x4c) from [<c0101b74>] (__lookup_hash+0x38/0x40)
[  360.950129] [<c0101b74>] (__lookup_hash+0x38/0x40) from [<c0105bd4>] (do_unlinkat+0x7c/0x1c4)
[  360.958825] [<c0105bd4>] (do_unlinkat+0x7c/0x1c4) from [<c000e080>] (ret_fast_syscall+0x0/0x30)
[  360.967668] BUG: using smp_processor_id() in preemptible [00000000] code: khungtaskd/46
[  360.975701] caller is smp_send_all_cpu_backtrace+0x8/0xc4
[  360.981284] CPU: 0 PID: 46 Comm: khungtaskd Not tainted 3.10.69-4-ARCH #1
[  360.988245] [<c0013f90>] (unwind_backtrace+0x0/0xec) from [<c0011908>] (show_stack+0x10/0x14)
[  360.996912] [<c0011908>] (show_stack+0x10/0x14) from [<c02a5ffc>] (debug_smp_processor_id+0xcc/0xe4)
[  361.006186] [<c02a5ffc>] (debug_smp_processor_id+0xcc/0xe4) from [<c0013024>] (smp_send_all_cpu_backtrace+0x8/0xc4)
[  361.016801] [<c0013024>] (smp_send_all_cpu_backtrace+0x8/0xc4) from [<c00942c0>] (watchdog+0x244/0x294)
[  361.026335] [<c00942c0>] (watchdog+0x244/0x294) from [<c004f428>] (kthread+0xb0/0xb4)
[  361.034341] [<c004f428>] (kthread+0xb0/0xb4) from [<c000e118>] (ret_from_fork+0x14/0x3c)
[  361.042580] Backtrace for cpu 0 (current):
[  361.046838] CPU: 0 PID: 46 Comm: khungtaskd Not tainted 3.10.69-4-ARCH #1
[  361.053778] [<c0013f90>] (unwind_backtrace+0x0/0xec) from [<c0011908>] (show_stack+0x10/0x14)
[  361.062483] [<c0011908>] (show_stack+0x10/0x14) from [<c0013084>] (smp_send_all_cpu_backtrace+0x68/0xc4)
[  361.072118] [<c0013084>] (smp_send_all_cpu_backtrace+0x68/0xc4) from [<c00942c0>] (watchdog+0x244/0x294)
[  361.081790] [<c00942c0>] (watchdog+0x244/0x294) from [<c004f428>] (kthread+0xb0/0xb4)
[  361.089749] [<c004f428>] (kthread+0xb0/0xb4) from [<c000e118>] (ret_from_fork+0x14/0x3c)
[  361.097993]
sending IPI to all other CPUs:
[  361.103874] IPI backtrace for cpu 3
[  361.109170] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.10.69-4-ARCH #1
[  361.115942] task: ef274780 ti: ef2a2000 task.ti: ef2a2000
[  361.121503] PC is at arch_cpu_idle+0x28/0x2c
[  361.125929] LR is at arch_cpu_idle+0x24/0x2c
[  361.130358] pc : [<c000ee0c>]    lr : [<c000ee08>]    psr: 600c0013
sp : ef2a3fe8  ip : 01000000  fp : 00000000
[  361.142166] r10: ef2a2000  r9 : c0a48ffa  r8 : c0a48ffa
[  361.147551] r7 : ef2a2030  r6 : c06cdc38  r5 : c09e0590  r4 : ef2a2000
[  361.154237] r3 : 00000000  r2 : 009ac0cc  r1 : 00000000  r0 : 00000003
[  361.160924] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  361.168393] Control: 10c5387d  Table: 2e26c04a  DAC: 00000015
[  361.174296] 
PC: 0xc000ed8c:
[  361.178898] ed8c  e30e0b5c e1a01004 e34c0000 e59f2004 eb0a0bbe e7f001f2 c0a49948 f1080040
[  361.187234] edac  e12fff1e e92d4008 e3a00001 eb0095c0 e3a00000 e8bd4008 ea100bec e92d4008
[  361.195570] edcc  e3a00001 eb100be9 e3a00002 e8bd4008 ea0095b7 ea1ab569 e92d4008 e30938c8
[  361.203907] edec  e34c30a4 e5933080 e3530000 0a000001 e12fff33 ea000000 eb00345d f1080080
[  361.212244] ee0c  e8bd8008 e92d4008 e3a00001 eb01241e e8bd4008 ea009605 e92d4008 f10c0080
[  361.220580] ee2c  eb001116 f10c0080 eafffffe e92d4010 f10c0080 eb001111 ebffff9e e30938c8
[  361.228917] ee4c  e34c30a4 e5933084 e3530000 08bd8010 e8bd4010 e12fff13 e92d4038 e1a04000
[  361.237253] ee6c  f10c0080 eb001105 e30a5f28 ebffff91 e3053658 e34c309e e1a01004 e5932060
[  361.245591] 
LR: 0xc000ed88:
[  361.250193] ed88  e12fff33 e30e0b5c e1a01004 e34c0000 e59f2004 eb0a0bbe e7f001f2 c0a49948
[  361.258529] eda8  f1080040 e12fff1e e92d4008 e3a00001 eb0095c0 e3a00000 e8bd4008 ea100bec
[  361.266866] edc8  e92d4008 e3a00001 eb100be9 e3a00002 e8bd4008 ea0095b7 ea1ab569 e92d4008
[  361.275202] ede8  e30938c8 e34c30a4 e5933080 e3530000 0a000001 e12fff33 ea000000 eb00345d
[  361.283539] ee08  f1080080 e8bd8008 e92d4008 e3a00001 eb01241e e8bd4008 ea009605 e92d4008
[  361.291875] ee28  f10c0080 eb001116 f10c0080 eafffffe e92d4010 f10c0080 eb001111 ebffff9e
[  361.300212] ee48  e30938c8 e34c30a4 e5933084 e3530000 08bd8010 e8bd4010 e12fff13 e92d4038
[  361.308548] ee68  e1a04000 f10c0080 eb001105 e30a5f28 ebffff91 e3053658 e34c309e e1a01004
[  361.316886] 
SP: 0xef2a3f68:
[  361.321488] 3f68  00000003 c001321c fe00110c c09e12c4 ef2a3fa0 fe001100 c0a48ffa c0008470
[  361.329824] 3f88  c000ee08 c000ee0c 600c0013 ffffffff ef2a3fd4 c000dc40 00000003 00000000
[  361.338161] 3fa8  009ac0cc 00000000 ef2a2000 c09e0590 c06cdc38 ef2a2030 c0a48ffa c0a48ffa
[  361.346497] 3fc8  ef2a2000 00000000 01000000 ef2a3fe8 c000ee08 c000ee0c 600c0013 ffffffff
[  361.354834] 3fe8  009ac0cc c0073568 c0a49b4c 008bd4e4 00000000 00000000 00000000 00000002
[  361.363170] 4008  00000000 ef274d00 c09ea908 00000001 00000015 c1560cc0 ef274d00 ee2de380
[  361.371507] 4028  00000000 ef273c80 c09cacc0 c09e10c8 ef2a5f44 ef2a5ea0 c06c9420 00000000
[  361.379843] 4048  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.388181] 
R4: 0xef2a1f80:
[  361.392782] 1f80  c0a48ffa c0008470 c000ee08 c000ee0c 60050013 ffffffff ef2a1fd4 c000dc40
[  361.401119] 1fa0  00000002 00000000 00027962 00000000 ef2a0000 c09e0590 c06cdc38 ef2a0030
[  361.409455] 1fc0  c0a48ffa c0a48ffa ef2a0000 00000000 01000000 ef2a1fe8 c000ee08 c000ee0c
[  361.417792] 1fe0  60050013 ffffffff 00027962 c0073568 c0a49b4c 008bd4e4 00000000 00000000
[  361.426129] 2000  00000000 00000003 00000000 ef274780 c09ea908 00000003 00000015 c1570cc0
[  361.434465] 2020  ef274780 ee2de380 ee2de380 ee511600 c09cacc0 c09e10c8 ef2a3fe4 ef2a3f40
[  361.442802] 2040  c06c9420 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.451138] 2060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.459476] 
R5: 0xc09e0510:
[  361.464077] 0510  c001c1f4 00000000 c001c704 00000000 00000000 00000000 00000000 0000000f
[  361.472414] 0530  0000000f 0000000f 0000000f 00000000 ef20cc00 ef20ccc0 ef20cd80 ef23f100
[  361.480750] 0550  ef20ce40 00000000 00000015 00000001 00000000 00002e7b 000003e8 00000001
[  361.489087] 0570  00000001 00000020 00000001 0007a120 00000000 00000028 00989680 00004e20
[  361.497424] 0590  00000000 00000000 00000000 00000000 00000003 00000001 00000000 00000000
[  361.505760] 05b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.514097] 05d0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.522433] 05f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.530771] 
R6: 0xc06cdbb8:
[  361.535372] dbb8  00000010 00000020 00000040 00000080 00000100 00000200 00000400 00000800
[  361.543709] dbd8  00001000 00002000 00004000 00008000 00010000 00020000 00040000 00080000
[  361.552045] dbf8  00100000 00200000 00400000 00800000 01000000 02000000 04000000 08000000
[  361.560382] dc18  10000000 20000000 40000000 80000000 7570635f 0070755f c09e0538 c09e0530
[  361.568719] dc38  c09e052c c09e0534 0000000f c0038f3c c0038f30 c0038cf8 c0038e8c 00000000
[  361.577055] dc58  c0116b2c c0116c4c 00000000 00000000 00000000 00000000 00000000 00000000
[  361.585392] dc78  00000000 00000000 c0038e5c 00000000 c0116948 00000000 00000000 00000000
[  361.593728] dc98  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.602066] 
R7: 0xef2a1fb0:
[  361.606667] 1fb0  ef2a0000 c09e0590 c06cdc38 ef2a0030 c0a48ffa c0a48ffa ef2a0000 00000000
[  361.615004] 1fd0  01000000 ef2a1fe8 c000ee08 c000ee0c 60050013 ffffffff 00027962 c0073568
[  361.623341] 1ff0  c0a49b4c 008bd4e4 00000000 00000000 00000000 00000003 00000000 ef274780
[  361.631677] 2010  c09ea908 00000003 00000015 c1570cc0 ef274780 ee2de380 ee2de380 ee511600
[  361.640014] 2030  c09cacc0 c09e10c8 ef2a3fe4 ef2a3f40 c06c9420 00000000 00000000 00000000
[  361.648350] 2050  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.656687] 2070  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.665023] 2090  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.673361] 
R8: 0xc0a48f7a:
[  361.677962] 8f78  c06c069c c0a48f90 00000000 c06c07e8 c0a48ef4 00000064 c06c091c c0a1add8
[  361.686299] 8f98  00000000 c05641bc c0a48df4 00000000 00000000 00000000 00000000 00000000
[  361.694636] 8fb8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.702972] 8fd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.711308] 8ff8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.719645] 9018  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.727982] 9038  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.736318] 9058  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.744655] 9078  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.752992] 
R9: 0xc0a48f7a:
[  361.757594] 8f78  c06c069c c0a48f90 00000000 c06c07e8 c0a48ef4 00000064 c06c091c c0a1add8
[  361.765930] 8f98  00000000 c05641bc c0a48df4 00000000 00000000 00000000 00000000 00000000
[  361.774267] 8fb8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.782604] 8fd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.790940] 8ff8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.799277] 9018  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.807613] 9038  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.815950] 9058  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.824286] 9078  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.832624] 
R10: 0xef2a1f80:
[  361.837312] 1f80  c0a48ffa c0008470 c000ee08 c000ee0c 60050013 ffffffff ef2a1fd4 c000dc40
[  361.845649] 1fa0  00000002 00000000 00027962 00000000 ef2a0000 c09e0590 c06cdc38 ef2a0030
[  361.853985] 1fc0  c0a48ffa c0a48ffa ef2a0000 00000000 01000000 ef2a1fe8 c000ee08 c000ee0c
[  361.862322] 1fe0  60050013 ffffffff 00027962 c0073568 c0a49b4c 008bd4e4 00000000 00000000
[  361.870659] 2000  00000000 00000003 00000000 ef274780 c09ea908 00000003 00000015 c1570cc0
[  361.878995] 2020  ef274780 ee2de380 ee2de380 ee511600 c09cacc0 c09e10c8 ef2a3fe4 ef2a3f40
[  361.887332] 2040  c06c9420 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.895668] 2060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  361.904008] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 3.10.69-4-ARCH #1
[  361.910791] [<c0013f90>] (unwind_backtrace+0x0/0xec) from [<c0011908>] (show_stack+0x10/0x14)
[  361.919471] [<c0011908>] (show_stack+0x10/0x14) from [<c001321c>] (handle_IPI+0x13c/0x17c)
[  361.927892] [<c001321c>] (handle_IPI+0x13c/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[  361.936663] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dc40>] (__irq_svc+0x40/0x70)
[  361.945169] Exception stack(0xef2a3fa0 to 0xef2a3fe8)
[  361.950383] 3fa0: 00000003 00000000 009ac0cc 00000000 ef2a2000 c09e0590 c06cdc38 ef2a2030
[  361.958719] 3fc0: c0a48ffa c0a48ffa ef2a2000 00000000 01000000 ef2a3fe8 c000ee08 c000ee0c
[  361.967052] 3fe0: 600c0013 ffffffff
[  361.970705] [<c000dc40>] (__irq_svc+0x40/0x70) from [<c000ee0c>] (arch_cpu_idle+0x28/0x2c)
[  361.979130] [<c000ee0c>] (arch_cpu_idle+0x28/0x2c) from [<c0073568>] (cpu_startup_entry+0xf4/0x150)
[  361.988337] [<c0073568>] (cpu_startup_entry+0xf4/0x150) from [<008bd4e4>] (0x8bd4e4)
[  361.996232] IPI backtrace for cpu 1

[  362.001530] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.69-4-ARCH #1
[  362.008301] task: ef273c80 ti: ef29e000 task.ti: ef29e000
[  362.013862] PC is at arch_cpu_idle+0x28/0x2c
[  362.018288] LR is at arch_cpu_idle+0x24/0x2c
[  362.022717] pc : [<c000ee0c>]    lr : [<c000ee08>]    psr: 60070013
sp : ef29ffe8  ip : 01000000  fp : 00000000
[  362.034526] r10: ef29e000  r9 : c0a48ffa  r8 : c0a48ffa
[  362.039910] r7 : ef29e030  r6 : c06cdc38  r5 : c09e0590  r4 : ef29e000
[  362.046597] r3 : 00000000  r2 : 000342ce  r1 : 00000000  r0 : 00000001
[  362.053284] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  362.060752] Control: 10c5387d  Table: 2e5bc04a  DAC: 00000015
[  362.066656] 
PC: 0xc000ed8c:
[  362.071258] ed8c  e30e0b5c e1a01004 e34c0000 e59f2004 eb0a0bbe e7f001f2 c0a49948 f1080040
[  362.079594] edac  e12fff1e e92d4008 e3a00001 eb0095c0 e3a00000 e8bd4008 ea100bec e92d4008
[  362.087930] edcc  e3a00001 eb100be9 e3a00002 e8bd4008 ea0095b7 ea1ab569 e92d4008 e30938c8
[  362.096267] edec  e34c30a4 e5933080 e3530000 0a000001 e12fff33 ea000000 eb00345d f1080080
[  362.104603] ee0c  e8bd8008 e92d4008 e3a00001 eb01241e e8bd4008 ea009605 e92d4008 f10c0080
[  362.112940] ee2c  eb001116 f10c0080 eafffffe e92d4010 f10c0080 eb001111 ebffff9e e30938c8
[  362.121277] ee4c  e34c30a4 e5933084 e3530000 08bd8010 e8bd4010 e12fff13 e92d4038 e1a04000
[  362.129613] ee6c  f10c0080 eb001105 e30a5f28 ebffff91 e3053658 e34c309e e1a01004 e5932060
[  362.137950] 
LR: 0xc000ed88:
[  362.142552] ed88  e12fff33 e30e0b5c e1a01004 e34c0000 e59f2004 eb0a0bbe e7f001f2 c0a49948
[  362.150889] eda8  f1080040 e12fff1e e92d4008 e3a00001 eb0095c0 e3a00000 e8bd4008 ea100bec
[  362.159225] edc8  e92d4008 e3a00001 eb100be9 e3a00002 e8bd4008 ea0095b7 ea1ab569 e92d4008
[  362.167562] ede8  e30938c8 e34c30a4 e5933080 e3530000 0a000001 e12fff33 ea000000 eb00345d
[  362.175898] ee08  f1080080 e8bd8008 e92d4008 e3a00001 eb01241e e8bd4008 ea009605 e92d4008
[  362.184235] ee28  f10c0080 eb001116 f10c0080 eafffffe e92d4010 f10c0080 eb001111 ebffff9e
[  362.192572] ee48  e30938c8 e34c30a4 e5933084 e3530000 08bd8010 e8bd4010 e12fff13 e92d4038
[  362.200908] ee68  e1a04000 f10c0080 eb001105 e30a5f28 ebffff91 e3053658 e34c309e e1a01004
[  362.209246] 
SP: 0xef29ff68:
[  362.213847] ff68  00000001 c001321c fe00110c c09e12c4 ef29ffa0 fe001100 c0a48ffa c0008470
[  362.222184] ff88  c000ee08 c000ee0c 60070013 ffffffff ef29ffd4 c000dc40 00000001 00000000
[  362.230520] ffa8  000342ce 00000000 ef29e000 c09e0590 c06cdc38 ef29e030 c0a48ffa c0a48ffa
[  362.238857] ffc8  ef29e000 00000000 01000000 ef29ffe8 c000ee08 c000ee0c 60070013 ffffffff
[  362.247194] ffe8  000342ce c0073568 c0a49b4c 008bd4e4 00000000 00000000 00000000 00000002
[  362.255530] 0008  00000000 ef274200 c09ea908 00000002 00000015 c1568cc0 ef274200 ee2de700
[  362.263866] 0028  ee2de700 e9657380 c09cacc0 c09e10c8 ef2a1fe4 ef2a1f40 c06c9420 00000000
[  362.272203] 0048  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.280541] 
R4: 0xef29df80:
[  362.285142] df80  ef29df80 ef29df80 00000000 00000000 ef29df90 ef29df90 ef29dfac ef279ed8
[  362.293479] dfa0  c004f378 00000000 00000000 c000e118 00000000 00000000 00000000 00000000
[  362.301815] dfc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.310152] dfe0  00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[  362.318488] e000  00000000 00000003 00000000 ef273c80 c09ea908 00000001 00000015 c1560cc0
[  362.326825] e020  ef273c80 ef50dc00 ef50da40 e4978b00 c09cacc0 c09e10c8 ef29ffe4 ef29ff40
[  362.335162] e040  c06c9420 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.343498] e060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.351835] 
R5: 0xc09e0510:
[  362.356437] 0510  c001c1f4 00000000 c001c704 00000000 00000000 00000000 00000000 0000000f
[  362.364774] 0530  0000000f 0000000f 0000000f 00000000 ef20cc00 ef20ccc0 ef20cd80 ef23f100
[  362.373110] 0550  ef20ce40 00000000 00000015 00000001 00000000 00002e7b 000003e8 00000001
[  362.381447] 0570  00000001 00000020 00000001 0007a120 00000000 00000028 00989680 00004e20
[  362.389783] 0590  00000000 00000000 00000000 00000000 00000003 00000001 00000000 00000000
[  362.398120] 05b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.406457] 05d0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.414793] 05f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.423130] 
R6: 0xc06cdbb8:
[  362.427732] dbb8  00000010 00000020 00000040 00000080 00000100 00000200 00000400 00000800
[  362.436069] dbd8  00001000 00002000 00004000 00008000 00010000 00020000 00040000 00080000
[  362.444405] dbf8  00100000 00200000 00400000 00800000 01000000 02000000 04000000 08000000
[  362.452742] dc18  10000000 20000000 40000000 80000000 7570635f 0070755f c09e0538 c09e0530
[  362.461078] dc38  c09e052c c09e0534 0000000f c0038f3c c0038f30 c0038cf8 c0038e8c 00000000
[  362.469415] dc58  c0116b2c c0116c4c 00000000 00000000 00000000 00000000 00000000 00000000
[  362.477752] dc78  00000000 00000000 c0038e5c 00000000 c0116948 00000000 00000000 00000000
[  362.486088] dc98  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.494426] 
R7: 0xef29dfb0:
[  362.499027] dfb0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.507364] dfd0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.515700] dff0  00000013 00000000 00000000 00000000 00000000 00000003 00000000 ef273c80
[  362.524037] e010  c09ea908 00000001 00000015 c1560cc0 ef273c80 ef50dc00 ef50da40 e4978b00
[  362.532373] e030  c09cacc0 c09e10c8 ef29ffe4 ef29ff40 c06c9420 00000000 00000000 00000000
[  362.540710] e050  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.549047] e070  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.557383] e090  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.565720] 
R8: 0xc0a48f7a:
[  362.570322] 8f78  c06c069c c0a48f90 00000000 c06c07e8 c0a48ef4 00000064 c06c091c c0a1add8
[  362.578659] 8f98  00000000 c05641bc c0a48df4 00000000 00000000 00000000 00000000 00000000
[  362.586995] 8fb8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.595332] 8fd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.603668] 8ff8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.612005] 9018  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.620342] 9038  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.628678] 9058  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.637015] 9078  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.645352] 
R9: 0xc0a48f7a:
[  362.649954] 8f78  c06c069c c0a48f90 00000000 c06c07e8 c0a48ef4 00000064 c06c091c c0a1add8
[  362.658290] 8f98  00000000 c05641bc c0a48df4 00000000 00000000 00000000 00000000 00000000
[  362.666627] 8fb8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.674963] 8fd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.683300] 8ff8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.691637] 9018  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.699973] 9038  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.708310] 9058  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.716646] 9078  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.724984] 
R10: 0xef29df80:
[  362.729672] df80  ef29df80 ef29df80 00000000 00000000 ef29df90 ef29df90 ef29dfac ef279ed8
[  362.738009] dfa0  c004f378 00000000 00000000 c000e118 00000000 00000000 00000000 00000000
[  362.746345] dfc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.754682] dfe0  00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000
[  362.763018] e000  00000000 00000003 00000000 ef273c80 c09ea908 00000001 00000015 c1560cc0
[  362.771355] e020  ef273c80 ef50dc00 ef50da40 e4978b00 c09cacc0 c09e10c8 ef29ffe4 ef29ff40
[  362.779692] e040  c06c9420 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.788028] e060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  362.796368] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 3.10.69-4-ARCH #1
[  362.803151] [<c0013f90>] (unwind_backtrace+0x0/0xec) from [<c0011908>] (show_stack+0x10/0x14)
[  362.811831] [<c0011908>] (show_stack+0x10/0x14) from [<c001321c>] (handle_IPI+0x13c/0x17c)
[  362.820252] [<c001321c>] (handle_IPI+0x13c/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[  362.829023] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dc40>] (__irq_svc+0x40/0x70)
[  362.837528] Exception stack(0xef29ffa0 to 0xef29ffe8)
[  362.842742] ffa0: 00000001 00000000 000342ce 00000000 ef29e000 c09e0590 c06cdc38 ef29e030
[  362.851079] ffc0: c0a48ffa c0a48ffa ef29e000 00000000 01000000 ef29ffe8 c000ee08 c000ee0c
[  362.859412] ffe0: 60070013 ffffffff
[  362.863065] [<c000dc40>] (__irq_svc+0x40/0x70) from [<c000ee0c>] (arch_cpu_idle+0x28/0x2c)
[  362.871489] [<c000ee0c>] (arch_cpu_idle+0x28/0x2c) from [<c0073568>] (cpu_startup_entry+0xf4/0x150)
[  362.880694] [<c0073568>] (cpu_startup_entry+0xf4/0x150) from [<008bd4e4>] (0x8bd4e4)
[  362.888592] IPI backtrace for cpu 2

[  362.893893] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.10.69-4-ARCH #1
[  362.900661] task: ef274200 ti: ef2a0000 task.ti: ef2a0000
[  362.906224] PC is at arch_cpu_idle+0x28/0x2c
[  362.910649] LR is at arch_cpu_idle+0x24/0x2c
[  362.915077] pc : [<c000ee0c>]    lr : [<c000ee08>]    psr: 60050013
sp : ef2a1fe8  ip : 01000000  fp : 00000000
[  362.926886] r10: ef2a0000  r9 : c0a48ffa  r8 : c0a48ffa
[  362.932270] r7 : ef2a0030  r6 : c06cdc38  r5 : c09e0590  r4 : ef2a0000
[  362.938956] r3 : 00000000  r2 : 00027962  r1 : 00000000  r0 : 00000002
[  362.945644] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  362.953112] Control: 10c5387d  Table: 186ac04a  DAC: 00000015
[  362.959016] 
PC: 0xc000ed8c:
[  362.963618] ed8c  e30e0b5c e1a01004 e34c0000 e59f2004 eb0a0bbe e7f001f2 c0a49948 f1080040
[  362.971954] edac  e12fff1e e92d4008 e3a00001 eb0095c0 e3a00000 e8bd4008 ea100bec e92d4008
[  362.980290] edcc  e3a00001 eb100be9 e3a00002 e8bd4008 ea0095b7 ea1ab569 e92d4008 e30938c8
[  362.988627] edec  e34c30a4 e5933080 e3530000 0a000001 e12fff33 ea000000 eb00345d f1080080
[  362.996963] ee0c  e8bd8008 e92d4008 e3a00001 eb01241e e8bd4008 ea009605 e92d4008 f10c0080
[  363.005300] ee2c  eb001116 f10c0080 eafffffe e92d4010 f10c0080 eb001111 ebffff9e e30938c8
[  363.013636] ee4c  e34c30a4 e5933084 e3530000 08bd8010 e8bd4010 e12fff13 e92d4038 e1a04000
[  363.021973] ee6c  f10c0080 eb001105 e30a5f28 ebffff91 e3053658 e34c309e e1a01004 e5932060
[  363.030311] 
LR: 0xc000ed88:
[  363.034912] ed88  e12fff33 e30e0b5c e1a01004 e34c0000 e59f2004 eb0a0bbe e7f001f2 c0a49948
[  363.043249] eda8  f1080040 e12fff1e e92d4008 e3a00001 eb0095c0 e3a00000 e8bd4008 ea100bec
[  363.051585] edc8  e92d4008 e3a00001 eb100be9 e3a00002 e8bd4008 ea0095b7 ea1ab569 e92d4008
[  363.059922] ede8  e30938c8 e34c30a4 e5933080 e3530000 0a000001 e12fff33 ea000000 eb00345d
[  363.068258] ee08  f1080080 e8bd8008 e92d4008 e3a00001 eb01241e e8bd4008 ea009605 e92d4008
[  363.076595] ee28  f10c0080 eb001116 f10c0080 eafffffe e92d4010 f10c0080 eb001111 ebffff9e
[  363.084931] ee48  e30938c8 e34c30a4 e5933084 e3530000 08bd8010 e8bd4010 e12fff13 e92d4038
[  363.093268] ee68  e1a04000 f10c0080 eb001105 e30a5f28 ebffff91 e3053658 e34c309e e1a01004
[  363.101605] 
SP: 0xef2a1f68:
[  363.106207] 1f68  00000002 c001321c fe00110c c09e12c4 ef2a1fa0 fe001100 c0a48ffa c0008470
[  363.114544] 1f88  c000ee08 c000ee0c 60050013 ffffffff ef2a1fd4 c000dc40 00000002 00000000
[  363.122880] 1fa8  00027962 00000000 ef2a0000 c09e0590 c06cdc38 ef2a0030 c0a48ffa c0a48ffa
[  363.131217] 1fc8  ef2a0000 00000000 01000000 ef2a1fe8 c000ee08 c000ee0c 60050013 ffffffff
[  363.139553] 1fe8  00027962 c0073568 c0a49b4c 008bd4e4 00000000 00000000 00000000 00000001
[  363.147890] 2008  00000000 ef274780 c09ea908 00000003 00000015 c1570cc0 ef274780 ee2de380
[  363.156226] 2028  ee2de380 ee511600 c09cacc0 c09e10c8 ef2a3fe4 ef2a3f40 c06c9420 00000000
[  363.164563] 2048  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.172901] 
R4: 0xef29ff80:
[  363.177502] ff80  00000001 c155f800 00001951 c007d5a4 00000002 c09e10c8 88403690 00000054
[  363.185838] ffa0  00000000 c155f950 ef29e000 00b96000 c155f950 c009b314 00000001 c0acbadc
[  363.194175] ffc0  00000000 ef29e000 c09c9950 600f0093 00b96000 c009c3f8 ef29e000 c09e0590
[  363.202512] ffe0  c06cdc38 ef29e030 000342e0 c0073568 c0a49b4c 008bd4e4 00000000 00000000
[  363.210848] 0000  00000000 00000003 00000000 ef274200 c09ea908 00000002 00000015 c1568cc0
[  363.219185] 0020  ef274200 ee2de700 ee2de700 e9657380 c09cacc0 c09e10c8 ef2a1fe4 ef2a1f40
[  363.227521] 0040  c06c9420 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.235858] 0060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.244195] 
R5: 0xc09e0510:
[  363.248797] 0510  c001c1f4 00000000 c001c704 00000000 00000000 00000000 00000000 0000000f
[  363.257133] 0530  0000000f 0000000f 0000000f 00000000 ef20cc00 ef20ccc0 ef20cd80 ef23f100
[  363.265470] 0550  ef20ce40 00000000 00000015 00000001 00000000 00002e7b 000003e8 00000001
[  363.273807] 0570  00000001 00000020 00000001 0007a120 00000000 00000028 00989680 00004e20
[  363.282143] 0590  00000000 00000000 00000000 00000000 00000003 00000001 00000000 00000000
[  363.290480] 05b0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.298816] 05d0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.307153] 05f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.315490] 
R6: 0xc06cdbb8:
[  363.320092] dbb8  00000010 00000020 00000040 00000080 00000100 00000200 00000400 00000800
[  363.328429] dbd8  00001000 00002000 00004000 00008000 00010000 00020000 00040000 00080000
[  363.336765] dbf8  00100000 00200000 00400000 00800000 01000000 02000000 04000000 08000000
[  363.345102] dc18  10000000 20000000 40000000 80000000 7570635f 0070755f c09e0538 c09e0530
[  363.353438] dc38  c09e052c c09e0534 0000000f c0038f3c c0038f30 c0038cf8 c0038e8c 00000000
[  363.361775] dc58  c0116b2c c0116c4c 00000000 00000000 00000000 00000000 00000000 00000000
[  363.370111] dc78  00000000 00000000 c0038e5c 00000000 c0116948 00000000 00000000 00000000
[  363.378448] dc98  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.386785] 
R7: 0xef29ffb0:
[  363.391387] ffb0  c155f950 c009b314 00000001 c0acbadc 00000000 ef29e000 c09c9950 600f0093
[  363.399723] ffd0  00b96000 c009c3f8 ef29e000 c09e0590 c06cdc38 ef29e030 000342ec c0073568
[  363.408060] fff0  c0a49b4c 008bd4e4 00000000 00000000 00000000 00000003 00000000 ef274200
[  363.416397] 0010  c09ea908 00000002 00000015 c1568cc0 ef274200 ee2de700 ee2de700 e9657380
[  363.424733] 0030  c09cacc0 c09e10c8 ef2a1fe4 ef2a1f40 c06c9420 00000000 00000000 00000000
[  363.433070] 0050  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.441406] 0070  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.449743] 0090  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.458080] 
R8: 0xc0a48f7a:
[  363.462682] 8f78  c06c069c c0a48f90 00000000 c06c07e8 c0a48ef4 00000064 c06c091c c0a1add8
[  363.471018] 8f98  00000000 c05641bc c0a48df4 00000000 00000000 00000000 00000000 00000000
[  363.479355] 8fb8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.487692] 8fd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.496028] 8ff8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.504365] 9018  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.512701] 9038  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.521038] 9058  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.529374] 9078  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.537712] 
R9: 0xc0a48f7a:
[  363.542313] 8f78  c06c069c c0a48f90 00000000 c06c07e8 c0a48ef4 00000064 c06c091c c0a1add8
[  363.550650] 8f98  00000000 c05641bc c0a48df4 00000000 00000000 00000000 00000000 00000000
[  363.558987] 8fb8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.567323] 8fd8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.575660] 8ff8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.583996] 9018  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.592333] 9038  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.600669] 9058  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.609006] 9078  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.617343] 
R10: 0xef29ff80:
[  363.622032] ff80  ef29ffa0 c0008458 c000ee08 c000ee0c 600f0013 ffffffff 000342f7 00000000
[  363.630368] ffa0  00000000 c155f950 ef29e000 00b96000 c155f950 c009b314 00000001 c0acbadc
[  363.638705] ffc0  00000000 ef29e000 c09c9950 600f0093 00b96000 c009c3f8 ef29e000 c09e0590
[  363.647042] ffe0  c06cdc38 ef29e030 000342f8 c0073568 c0a49b4c 008bd4e4 00000000 00000000
[  363.655378] 0000  00000000 00000003 00000000 ef274200 c09ea908 00000002 00000015 c1568cc0
[  363.663715] 0020  ef274200 ee2de700 ee2de700 e9657380 c09cacc0 c09e10c8 ef2a1fe4 ef2a1f40
[  363.672051] 0040  c06c9420 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.680388] 0060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  363.688729] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.10.69-4-ARCH #1
[  363.695516] [<c0013f90>] (unwind_backtrace+0x0/0xec) from [<c0011908>] (show_stack+0x10/0x14)
[  363.704191] [<c0011908>] (show_stack+0x10/0x14) from [<c001321c>] (handle_IPI+0x13c/0x17c)
[  363.712612] [<c001321c>] (handle_IPI+0x13c/0x17c) from [<c0008470>] (gic_handle_irq+0x58/0x5c)
[  363.721383] [<c0008470>] (gic_handle_irq+0x58/0x5c) from [<c000dc40>] (__irq_svc+0x40/0x70)
[  363.729888] Exception stack(0xef2a1fa0 to 0xef2a1fe8)
[  363.735102] 1fa0: 00000002 00000000 00027962 00000000 ef2a0000 c09e0590 c06cdc38 ef2a0030
[  363.743438] 1fc0: c0a48ffa c0a48ffa ef2a0000 00000000 01000000 ef2a1fe8 c000ee08 c000ee0c
[  363.751772] 1fe0: 60050013 ffffffff
[  363.755425] [<c000dc40>] (__irq_svc+0x40/0x70) from [<c000ee0c>] (arch_cpu_idle+0x28/0x2c)
[  363.763849] [<c000ee0c>] (arch_cpu_idle+0x28/0x2c) from [<c0073568>] (cpu_startup_entry+0xf4/0x150)
[  363.773056] [<c0073568>] (cpu_startup_entry+0xf4/0x150) from [<008bd4e4>] (0x8bd4e4)
[  363.781196] Kernel panic - not syncing: hung_task: blocked tasks
[  363.787128] CPU: 0 PID: 46 Comm: khungtaskd Not tainted 3.10.69-4-ARCH #1
[  363.794079] [<c0013f90>] (unwind_backtrace+0x0/0xec) from [<c0011908>] (show_stack+0x10/0x14)
[  363.802757] [<c0011908>] (show_stack+0x10/0x14) from [<c06c0ce8>] (panic+0x9c/0x1ec) 
I can't remember seeing this in older kernel versions. I'll try some older kernels and see what gives.

pdenhaan
Posts: 8
Joined: Mon Feb 16, 2015 11:12 pm
languages_spoken: english
ODROIDs: Odroid-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by pdenhaan » Fri Mar 06, 2015 7:36 am

My oldest 3.10.69 kernel (3.10.69-1, downloaded Feb 21) works fine. The problems started soon after with the -3 version (Feb 26); I don't think I ever ran 3.10.69-2. Was there anything changed around that time that might impact SD card I/O? Docker builds quite reliably cause a kernel panic on any subsequent kernel. This is a bit of a problem, as running a headless Docker server is precisely what I mean to use my C1 for.

Edit: I just rebuilt all my Docker images without a hitch on 3.10.69-1. Quite confident there's a regression somewhere in the later 3.10.69 builds. It could be Arch, I suppose, but that would be quite a coincidence with Ubuntu exhibiting a similar problem above.

User avatar
rooted
Posts: 6435
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by rooted » Fri Mar 06, 2015 8:02 am

You need to disable the hung task reboot option and certainly make sure you're using some type of swap if using -j 4.

As root:

Code: Select all

 
echo 0 > /proc/sys/kernel/hung_task_panic

pdenhaan
Posts: 8
Joined: Mon Feb 16, 2015 11:12 pm
languages_spoken: english
ODROIDs: Odroid-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by pdenhaan » Fri Mar 06, 2015 8:18 am

The Docker image build for which I posted the log doesn't come anywhere near make/gcc. Thanks for the suggestion to disable the hung task reboot option - I hadn't actually tried it so far because I figured disabling it wouldn't change anything about Docker sitting there stuck on cleaning up its images... but anything is worth a shot.

User avatar
rooted
Posts: 6435
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by rooted » Fri Mar 06, 2015 9:36 am

I thought you were building docker, my mistake.

pdenhaan
Posts: 8
Joined: Mon Feb 16, 2015 11:12 pm
languages_spoken: english
ODROIDs: Odroid-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by pdenhaan » Fri Mar 06, 2015 5:12 pm

It's fine, I probably wasn't very clear, your help is very much appreciated.

kinsa
Posts: 40
Joined: Fri Sep 26, 2014 4:32 pm
languages_spoken: english
ODROIDs: U3 C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by kinsa » Sat Mar 07, 2015 8:13 am

I think this has something to do with the SD card or any solid state devices in general as I don't get these errors if I use USB HDDs (with a good power supply, i.e. using a powered hub).

My new SD card doesn't crash as quickly as compared to my old one. So I guess what we're seeing here is the effect of aging on SD cards.
42

pdenhaan
Posts: 8
Joined: Mon Feb 16, 2015 11:12 pm
languages_spoken: english
ODROIDs: Odroid-C1
Contact:

Re: BUG: Kernel crashes (khungtaskd, heavy SD load)

Unread post by pdenhaan » Mon Mar 09, 2015 10:37 pm

kinsa wrote:I think this has something to do with the SD card or any solid state devices in general as I don't get these errors if I use USB HDDs (with a good power supply, i.e. using a powered hub).
That's my experience as well. After moving my Docker image & volume directory to a USB-connected SSD, the latest kernel now runs without apparent stability problems.
My new SD card doesn't crash as quickly as compared to my old one. So I guess what we're seeing here is the effect of aging on SD cards.
There's no doubt that the C1 is much happier with some SD cards than with others, but I'm not sure that ageing is at the root of the problem in this thread, for three reasons:
  • The symptom is a hung I/O operation, not a CRC error or silent data corruption.
  • A kernel downgrade completely fixes the problem as far as I've been able to ascertain. That would be unlikely for ageing.
  • My SD card was bought brand new for the C1. I doubt I've been building enough Docker images to wear it out already...
Even if your new SD card is exactly the same brand and model as the old one, the hardware may be different. SD card manufacturers are notorious for making fairly regular changes; as long as capacity and general performance characteristics remain the same, they don't necessarily update model numbers.

Post Reply

Return to “Hardware and peripherals”

Who is online

Users browsing this forum: No registered users and 1 guest