Real time kernel 3.10.70-rt74 (PREEMPT_RT), finally working

[Clx]

Hi all!

Together with mlinuxguy, adbensi and kinsa we've managed to build a working real time (fully preemptive) kernel.
It is overall stable (I couldn't crash it with stress for a few hours now), but it has few minor issues:
1) Low built-in Ethernet performance - USB-WiFi and USB-Ethernet work fine.
2) Poor RT performance on some SD cards - works great with eMMC module and some SD cards too.

The source code is available on Github - https://github.com/Clouded/linux-rt-odr ... 10.70-rt74
Basically, it's a fork of Hardkernel's Linux for Odroid-C1 with:
- rt-patch applied
- few minor fixes (manually applied patch rejects, made a few spinlocks static, fixed a timer typo)
- fixes by kinsa (sdhc and aufs)
- some configuration (enabled PREEMPT_RT, periodic timer ticks, performance governor default)

DOWNLOADS:

[UPDATE 23 MAR 2015]

Debs of kernel with modules and headers:
linux-image-3.10.72-rt77_20150319_armhf.deb
linux-headers-3.10.72-rt77_20150319_armhf.deb

Built with make-kpkg, installs smoothly on Ubuntu (dtbs included and installed automatically) and should work on other distros too.
To install download the package and run:

Code: Select all

sudo dpkg -i linux-image-3.10.72-rt77_20150319_armhf.deb

Patched source code is available here:
https://github.com/Clouded/linux-rt-odr ... 10.72-rt77

Cyclictest results:

Code: Select all

odroid@odroid:~$ sudo cyclictest --smp -p99 -m -n
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.15 0.37 0.47 1/223 1790          

T: 0 ( 1778) P:99 I:1000 C:4501611 Min:      6 Act:    8 Avg:    9 Max:      42
T: 1 ( 1779) P:99 I:1500 C:3001053 Min:      6 Act:    8 Avg:    8 Max:      36
T: 2 ( 1780) P:99 I:2000 C:2250775 Min:      7 Act:    8 Avg:    8 Max:      38
T: 3 ( 1781) P:99 I:2500 C:1800608 Min:      6 Act:    8 Avg:    9 Max:      26

[mlinuxguy]

"Scheduling while atomic" indicates that you've tried to sleep somewhere that you shouldn't - like within a spinlock-protected critical section or an interrupt handler

When I did the RT patch port to the Odroid-U I also had to fix spin_lock calls that were missed by making them raw_spinlock()
-static DEFINE_SPINLOCK(unwind_lock);
+static DEFINE_RAW_SPINLOCK(unwind_lock);
(perhaps none of those got missed under recent RT patches)

I don't know that this was done correctly:

kernel/sched/core.c.rej looks like this:

Why did you leave out the: system_state != SYSTEM_RUNNING

I'll try applying the RT patches after my new Odroid-C1 comes in, I dedicated the test one to the basement geothermal monitoring...

[Clx]

"Scheduling while atomic" indicates that you've tried to sleep somewhere that you shouldn't - like within a spinlock-protected critical section or an interrupt handler

When I did the RT patch port to the Odroid-U I also had to fix spin_lock calls that were missed by making them raw_spinlock()
-static DEFINE_SPINLOCK(unwind_lock);
+static DEFINE_RAW_SPINLOCK(unwind_lock);
(perhaps none of those got missed under recent RT patches)

I don't know that this was done correctly:

kernel/sched/core.c.rej looks like this:

Why did you leave out the: system_state != SYSTEM_RUNNING

I'll try applying the RT patches after my new Odroid-C1 comes in, I dedicated the test one to the basement geothermal monitoring...

@mlinuxguy, thanks for suggestions!

Is there a rule to determine which lock should be spin_lock and which raw_spin_lock?
Maybe spin_locks in tsdemux\etc should be raw_spin_locks?
https://github.com/Clouded/linux-rt-odr ... 085dcc3a81

The original file used to create rt-patch did not have system_state != SYSTEM_RUNNING, so I thought it should be leaved, but I also tried with it and it's not the case of the problem.

[Clx]

I also tried with NO_HZ disabled, the error looks a little bit different now:

Code: Select all

[   79.751468@0] ------------[ cut here ]------------
[   79.751472@0] kernel BUG at kernel/rtmutex.c:875!
[   79.751476@0] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[   79.751492@0] Modules linked in: rtl8192cu rtlwifi rtl8192c_common joydev aml_i2c ads7846 spidev spicc nls_cp437 vfat fat
[   79.751498@0] CPU: 0 PID: 83 Comm: mmcqd/0 Tainted: G        W    3.10.65-rt69 #1
[   79.751502@0] task: ef2e5d80 ti: eeafe000 task.ti: eeafe000
[   79.751510@0] PC is at rt_spin_lock_slowlock+0x70/0x270
[   79.751513@0] LR is at rt_spin_lock_slowlock+0x58/0x270
[   79.751518@0] pc : [<c063ebdc>]    lr : [<c063ebc4>]    psr: 600f0193
[   79.751518@0] sp : eeaffbc8  ip : ef2e5d81  fp : 00000000
[   79.751520@0] r10: c0935d40  r9 : 00000000  r8 : ef2e5d80
[   79.751523@0] r7 : fe001100  r6 : eeae0e40  r5 : eeafe000  r4 : eeabbaf0
[   79.751526@0] r3 : ef2e5d80  r2 : 00000000  r1 : ef2e5d80  r0 : 00000000
[   79.751530@0] Flags: nZCv  IRQs off  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[   79.751534@0] Control: 10c5387d  Table: 2929c04a  DAC: 00000015
[   79.751536@0]
[   79.751536@0] PC: 0xc063eb5c:
[   79.751545@0] eb5c  e3a00000 e8bd81f0 e3e0006d e8bd81f0 e92d4ff0 e24dd03c e1a0300d e1a04000
[   79.751554@0] eb7c  e3c35d7f e3a02000 e3a03001 e3c5503f e5958010 e58d202c e5cd3034 eb00031d
[   79.751563@0] eb9c  e5943008 e1a00004 e1a01008 e3a02000 e3530000 02843004 05843004 05843008
[   79.751572@0] ebbc  e3a03001 ebe8df39 e2509000 1a000069 e594300c e3c33001 e1580003 1a000000
[   79.751580@0] ebdc  e7f001f2 e2886e45 e3a0a002 e286600c e28d7004 e1a00006 eb000330 e5983000
[   79.751589@0] ebfc  e1a00006 e5883004 e5953010 e583a000 eb000291 e1a03009 e1a01007 e1a00004
[   79.751598@0] ec1c  e1a02008 ebe8e0df e3500000 0a000000 e7f001f2 e1a00004 e1a01008 e1a02007
[   79.751607@0] ec3c  e3a03001 ebe8df19 e3500000 1a00002e e5943004 e5932020 e1540002 1a00003b
[   79.751608@0]
[   79.751608@0] LR: 0xc063eb44:
[   79.751617@0] eb44  e1a00005 eb000331 e5941010 e5816000 f57ff05f eaffffd4 e3a00000 e8bd81f0
[   79.751626@0] eb64  e3e0006d e8bd81f0 e92d4ff0 e24dd03c e1a0300d e1a04000 e3c35d7f e3a02000
[   79.751635@0] eb84  e3a03001 e3c5503f e5958010 e58d202c e5cd3034 eb00031d e5943008 e1a00004
[   79.751643@0] eba4  e1a01008 e3a02000 e3530000 02843004 05843004 05843008 e3a03001 ebe8df39
[   79.751652@0] ebc4  e2509000 1a000069 e594300c e3c33001 e1580003 1a000000 e7f001f2 e2886e45
[   79.751661@0] ebe4  e3a0a002 e286600c e28d7004 e1a00006 eb000330 e5983000 e1a00006 e5883004
[   79.751670@0] ec04  e5953010 e583a000 eb000291 e1a03009 e1a01007 e1a00004 e1a02008 ebe8e0df
[   79.751679@0] ec24  e3500000 0a000000 e7f001f2 e1a00004 e1a01008 e1a02007 e3a03001 ebe8df19
[   79.751680@0]
[   79.751680@0] SP: 0xeeaffb48:
[   79.751689@0] fb48  00000001 ef2e6060 00000000 ef2e5db8 c14aaef8 ef2e6060 00000000 ef2e6060
[   79.751698@0] fb68  c063ebe0 00000000 c000e098 00000000 e7100000 c000de0c 00000000 ef2e5d80
[   79.751706@0] fb88  00000000 ef2e5d80 eeabbaf0 eeafe000 eeae0e40 fe001100 ef2e5d80 00000000
[   79.751715@0] fba8  c0935d40 00000000 ef2e5d81 eeaffbc8 c063ebc4 c063ebdc 600f0193 ffffffff
[   79.751724@0] fbc8  918e9948 630f630e ef4cd280 c0643b1c eeaffbf4 c005204c 00000001 ef4cd280
[   79.751732@0] fbe8  c092cec0 c14aaec0 ef4cd280 00000000 00000001 c0946a01 00000000 eeabba00
[   79.751741@0] fc08  eeabbaf0 eeae0e40 fe001100 0000006e c09aa6bc c0935d40 00000000 c04664d8
[   79.751750@0] fc28  00000000 600f0193 c092e000 c063f828 c063f828 00000000 eeae0e40 fe001100
[   79.751751@0]
[   79.751751@0] IP: 0xef2e5d01:
[   79.751760@0] 5d00  00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000
[   79.751768@0] 5d20  0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751776@0] 5d40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751784@0] 5d60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751792@0] 5d80  00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001
[   79.751801@0] 5da0  00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000
[   79.751809@0] 5dc0  00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012
[   79.751818@0] 5de0  4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000
[   79.751826@0] 5e00  00000000 00000000 00691a40 00000000 00005f84 00000000 3abe19e8 00000000
[   79.751828@0]
[   79.751828@0] R1: 0xef2e5d00:
[   79.751836@0] 5d00  00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000
[   79.751844@0] 5d20  0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751852@0] 5d40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751860@0] 5d60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751868@0] 5d80  00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001
[   79.751877@0] 5da0  00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000
[   79.751885@0] 5dc0  00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012
[   79.751894@0] 5de0  4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000
[   79.751895@0]
[   79.751895@0] R3: 0xef2e5d00:
[   79.751903@0] 5d00  00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000
[   79.751912@0] 5d20  0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751920@0] 5d40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751927@0] 5d60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.751936@0] 5d80  00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001
[   79.751944@0] 5da0  00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000
[   79.751953@0] 5dc0  00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012
[   79.751961@0] 5de0  4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000
[   79.751963@0]
[   79.751963@0] R4: 0xeeabba70:
[   79.751971@0] ba70  00000000 00000000 00000000 00000000 ef2520c0 00000004 eeae0f80 00000000
[   79.751979@0] ba90  c0998d10 00000000 00000000 ef256080 00000000 00000000 00000000 00000000
[   79.751987@0] bab0  00000000 00000000 00000000 00000000 00000000 eeabbe00 00000000 00000000
[   79.751996@0] bad0  00000000 00000000 00000000 00000000 00000000 00000000 eeaffd80 00000000
[   79.752004@0] baf0  042c042b eeabbaf4 eeabbaf4 ef2e5d81 00000000 00000000 00000000 00000002
[   79.752012@0] bb10  00000000 00000000 00000000 00000000 00000000 00000000 00000001 eeabbc00
[   79.752021@0] bb30  00000000 00000000 00000000 00000000 00000000 00000000 0000000d e6240000
[   79.752029@0] bb50  00000000 0517341b eeae3280 63686473 5f64735f 5f6c6c61 736e6970 736e6900
[   79.752031@0]
[   79.752031@0] R5: 0xeeafdf80:
[   79.752039@0] df80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752047@0] dfa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752055@0] dfc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752063@0] dfe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752072@0] e000  00000002 00010003 00000001 00000000 ef2e5d80 c094ec24 00000000 00000015
[   79.752080@0] e020  ee1c2200 00000000 c0946a98 ef2e5d80 c14aaec0 eeafe000 c0949ef8 eeaffe4c
[   79.752088@0] e040  eeaffe00 c063df28 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752096@0] e060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752098@0]
[   79.752098@0] R6: 0xeeae0dc0:
[   79.752107@0] 0dc0  76697264 ff007265 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
[   79.752116@0] 0de0  ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
[   79.752124@0] 0e00  00000003 00000000 ef2ec780 eeae0dc0 ef2ec811 00000000 00000000 00000000
[   79.752133@0] 0e20  00000000 3ccb37e1 eeae0d00 00000000 00000000 a1ff0008 00003962 00000000
[   79.752141@0] 0e40  c04664bc eeabba00 00000000 00000000 c04683f8 ef2e6e00 0000006e 00002020
[   79.752150@0] 0e60  00000000 00000001 c086d12c eeae1280 00000000 00000000 00000000 00000000
[   79.752158@0] 0e80  00000001 00000001 00000000 00000000 00000000 00000000 ffffffff ffffffff
[   79.752167@0] 0ea0  ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
[   79.752168@0]
[   79.752168@0] R7: 0xfe001080:
[   79.752176@0] 1080  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752184@0] 10a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752192@0] 10c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752200@0] 10e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752209@0] 1100  00000001 000000f8 00000002 000003ff 00000000 000000a0 000003ff 00000003
[   79.752217@0] 1120  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752225@0] 1140  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752233@0] 1160  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752234@0]
[   79.752234@0] R8: 0xef2e5d00:
[   79.752243@0] 5d00  00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000
[   79.752251@0] 5d20  0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752259@0] 5d40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752267@0] 5d60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752275@0] 5d80  00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001
[   79.752283@0] 5da0  00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000
[   79.752292@0] 5dc0  00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012
[   79.752300@0] 5de0  4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000
[   79.752302@0]
[   79.752302@0] R10: 0xc0935cc0:
[   79.752310@0] 5cc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752318@0] 5ce0  00050005 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752326@0] 5d00  00000000 00000000 00000000 00000000 00000000 00000000 ef269680 00000000
[   79.752334@0] 5d20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752343@0] 5d40  0000006e 0000006e 00000000 00060000 c095b328 ef202780 00000000 c0946be8
[   79.752352@0] 5d60  00000000 00000001 c092f908 c008f2cc eeae0e40 00000000 00000020 00000000
[   79.752360@0] 5d80  00000000 00001a00 00000000 00000000 00001a01 80001a00 c000f550 00000000
[   79.752368@0] 5da0  4e0c4e0c 00000000 00000000 00000000 00000000 00000000 34023402 c0935dbc
[   79.752371@0] Process mmcqd/0 (pid: 83, stack limit = 0xeeafe238)
[   79.752374@0] Stack: (0xeeaffbc8 to 0xeeb00000)
[   79.752380@0] fbc0:                   918e9948 630f630e ef4cd280 c0643b1c eeaffbf4 c005204c
[   79.752385@0] fbe0: 00000001 ef4cd280 c092cec0 c14aaec0 ef4cd280 00000000 00000001 c0946a01
[   79.752391@0] fc00: 00000000 eeabba00 eeabbaf0 eeae0e40 fe001100 0000006e c09aa6bc c0935d40
[   79.752397@0] fc20: 00000000 c04664d8 00000000 600f0193 c092e000 c063f828 c063f828 00000000
[   79.752403@0] fc40: eeae0e40 fe001100 0000006e c09aa6bc c0935d40 c008c14c eeabba4c 00000000
[   79.752408@0] fc60: eeafe000 c0935d40 eeae0e40 00000000 fe001100 ef217c40 00000000 e8ab0510
[   79.752414@0] fc80: 00000000 c008c2f8 00020000 c0935d40 0000006e c008f3c0 c092c1a4 c008ba5c
[   79.752420@0] fca0: 00000100 c000f0b0 fe00110c c0946be0 eeaffcd0 c000845c c0052750 c063f828
[   79.752426@0] fcc0: 600f0013 ffffffff eeaffd04 c000dd40 00000000 a00f0013 00000001 042b042b
[   79.752432@0] fce0: eeabbaf0 eeaffd80 ef217800 eeabbaf0 ef217c40 00000000 e8ab0510 00000000
[   79.752437@0] fd00: 00000001 eeaffd18 c0052750 c063f828 600f0013 ffffffff eeabbaf0 c063f1bc
[   79.752443@0] fd20: eeabba00 c0467f94 eeafe000 ef2e5d80 eeaffd4c c005360c ef217800 eeaffd80
[   79.752449@0] fd40: 00000000 ef217800 00009456 c0376fe8 0000000d ef217800 eeaffd80 eeaffd90
[   79.752455@0] fd60: c0930400 c03770e0 00000000 eeaffdb8 ef217800 00000005 c0930400 c03771d0
[   79.752460@0] fd80: 00000000 eeaffdb8 00000000 00000000 00000000 00000000 eeaffd98 eeaffd98
[   79.752466@0] fda0: c0376ea8 00000000 eeaffe04 eeaf4540 eeaffe04 c03830ac 0000000d e6240000
[   79.752472@0] fdc0: 00000000 00000000 00000000 00000000 00000195 00000005 00000000 00000000
[   79.752477@0] fde0: 00000000 eeaffd80 eec90800 c0383d88 eeafe000 00000000 eeaffed4 c063ed94
[   79.752483@0] fe00: eeafe000 ef2e5d80 eeafe000 ef2e5d80 00000000 c002cfa4 eeafe000 ef2e5d80
[   79.752489@0] fe20: 00000000 c002cfa4 eeafe000 ef2e5d80 eeafe000 c002d0c8 eeafe000 ef217800
[   79.752495@0] fe40: ef217bd4 eeafe000 eeaf44a0 eeaf464c eeaf4540 ef217bf4 eeaffed4 c037793c
[   79.752501@0] fe60: e8ab0af8 eeaf4448 eeaf4554 00000000 ef2e5d80 c0048430 eeaffe78 eeaffe78
[   79.752507@0] fe80: eeaf441c e8ab0af8 eea12f40 eeaf441c eec90800 eeaf4550 e8ab0af8 eeaf441c
[   79.752512@0] fea0: eeaf4424 c0384e40 e8ab03b0 eeaf4554 00000000 eeaf4400 00000000 00000000
[   79.752518@0] fec0: ee332750 e8ab0af8 ee332720 c01e9794 e8ab03b1 ee332720 e8ab0af8 eec90800
[   79.752524@0] fee0: e8ab0af8 122c8001 eeaf4400 eeaf4400 00000001 eeaf441c eeaf4424 c0385958
[   79.752530@0] ff00: c01e9a4c eea12f40 eeafe000 ef217800 eea12f40 c002d0c8 eeafe000 eeaf441c
[   79.752536@0] ff20: e8ab0af8 eea12f40 eeafe000 00000000 00000001 122c8001 eeaf4424 c03865a4
[   79.752541@0] ff40: 00000000 eea9fce8 00000000 eeaf441c c03864f0 00000000 00000000 00000000
[   79.752547@0] ff60: 00000000 c0047dc4 ffffffff 00000000 ffffffff eeaf441c 00000000 00000000
[   79.752553@0] ff80: eeafff80 eeafff80 00000000 00000000 eeafff90 eeafff90 eeafffac eea9fce8
[   79.752558@0] ffa0: c0047d2c 00000000 00000000 c000e218 00000000 00000000 00000000 00000000
[   79.752563@0] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[   79.752569@0] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 ffffffff ffffffff
[   79.752584@0] [<c063ebdc>] (rt_spin_lock_slowlock+0x70/0x270) from [<c04664d8>] (aml_sdhc_irq+0x1c/0x294)
[   79.752596@0] [<c04664d8>] (aml_sdhc_irq+0x1c/0x294) from [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180)
[   79.752605@0] [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180) from [<c008c2f8>] (handle_irq_event+0x78/0x98)
[   79.752615@0] [<c008c2f8>] (handle_irq_event+0x78/0x98) from [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178)
[   79.752623@0] [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178) from [<c008ba5c>] (generic_handle_irq+0x24/0x38)
[   79.752633@0] [<c008ba5c>] (generic_handle_irq+0x24/0x38) from [<c000f0b0>] (handle_IRQ+0x68/0x90)
[   79.752641@0] [<c000f0b0>] (handle_IRQ+0x68/0x90) from [<c000845c>] (gic_handle_irq+0x3c/0x5c)
[   79.752648@0] [<c000845c>] (gic_handle_irq+0x3c/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[   79.752651@0] Exception stack(0xeeaffcd0 to 0xeeaffd18)
[   79.752655@0] fcc0:                                     00000000 a00f0013 00000001 042b042b
[   79.752661@0] fce0: eeabbaf0 eeaffd80 ef217800 eeabbaf0 ef217c40 00000000 e8ab0510 00000000
[   79.752666@0] fd00: 00000001 eeaffd18 c0052750 c063f828 600f0013 ffffffff
[   79.752674@0] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f828>] (_raw_spin_lock+0x14/0x50)
[   79.752681@0] [<c063f828>] (_raw_spin_lock+0x14/0x50) from [<c063f1bc>] (rt_spin_lock_slowunlock+0xc/0x18)
[   79.752689@0] [<c063f1bc>] (rt_spin_lock_slowunlock+0xc/0x18) from [<c0467f94>] (aml_sdhc_request+0x2d0/0x304)
[   79.752698@0] [<c0467f94>] (aml_sdhc_request+0x2d0/0x304) from [<c0376fe8>] (mmc_start_request+0x138/0x140)
[   79.752707@0] [<c0376fe8>] (mmc_start_request+0x138/0x140) from [<c03770e0>] (mmc_wait_for_req+0x68/0xd4)
[   79.752714@0] [<c03770e0>] (mmc_wait_for_req+0x68/0xd4) from [<c03771d0>] (mmc_wait_for_cmd+0x84/0x94)
[   79.752722@0] [<c03771d0>] (mmc_wait_for_cmd+0x84/0x94) from [<c03830ac>] (get_card_status+0x70/0x84)
[   79.752730@0] [<c03830ac>] (get_card_status+0x70/0x84) from [<c0383d88>] (mmc_blk_err_check+0x328/0x520)
[   79.752738@0] [<c0383d88>] (mmc_blk_err_check+0x328/0x520) from [<c037793c>] (mmc_start_req+0x180/0x358)
[   79.752745@0] [<c037793c>] (mmc_start_req+0x180/0x358) from [<c0384e40>] (mmc_blk_issue_rw_rq+0x394/0xa08)
[   79.752753@0] [<c0384e40>] (mmc_blk_issue_rw_rq+0x394/0xa08) from [<c0385958>] (mmc_blk_issue_rq+0x4a4/0x520)
[   79.752760@0] [<c0385958>] (mmc_blk_issue_rq+0x4a4/0x520) from [<c03865a4>] (mmc_queue_thread+0xb4/0x138)
[   79.752770@0] [<c03865a4>] (mmc_queue_thread+0xb4/0x138) from [<c0047dc4>] (kthread+0x98/0xa0)
[   79.752779@0] [<c0047dc4>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c)
[   79.752784@0] Code: e594300c e3c33001 e1580003 1a000000 (e7f001f2)
[   81.239321@0] ---[ end trace 0000000000000002 ]---
[   81.239325@0] Kernel panic - not syncing: Fatal exception in interrupt
[   81.239332@1] CPU1: stopping
[   81.239338@1] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G      D W    3.10.65-rt69 #1
[   81.239356@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   81.239365@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[   81.239373@1] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[   81.239380@1] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[   81.239382@1] Exception stack(0xef2a9f98 to 0xef2a9fe0)
[   81.239386@1] 9f80:                                                       00000001 00000000
[   81.239392@1] 9fa0: 0000ec3e 00000000 ef2a8000 ef2a8000 c0945fd0 c06434ec c09aa6a4 c09aa6a4
[   81.239397@1] 9fc0: 00000001 ef2a8000 00000000 ef2a9fe0 c000f54c c000f550 60010013 ffffffff
[   81.239406@1] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000f550>] (arch_cpu_idle+0x24/0x2c)
[   81.239416@1] [<c000f550>] (arch_cpu_idle+0x24/0x2c) from [<c0069ce0>] (cpu_startup_entry+0xa8/0x148)
[   81.239427@1] [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) from [<0082fa64>] (0x82fa64)
[   81.239431@2] CPU2: stopping
[   81.239437@2] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G      D W    3.10.65-rt69 #1
[   81.239449@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   81.239457@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[   81.239464@2] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[   81.239471@2] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[   81.239473@2] Exception stack(0xef2abf98 to 0xef2abfe0)
[   81.239476@2] bf80:                                                       00000002 00000000
[   81.239482@2] bfa0: 0000d498 00000000 ef2aa000 ef2aa000 c0945fd0 c06434ec c09aa6a4 c09aa6a4
[   81.239488@2] bfc0: 00000001 ef2aa000 00000000 ef2abfe0 c000f54c c000f550 60000013 ffffffff
[   81.239496@2] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000f550>] (arch_cpu_idle+0x24/0x2c)
[   81.239505@2] [<c000f550>] (arch_cpu_idle+0x24/0x2c) from [<c0069ce0>] (cpu_startup_entry+0xa8/0x148)
[   81.239512@2] [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) from [<0082fa64>] (0x82fa64)
[   81.239516@3] CPU3: stopping
[   81.239521@3] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G      D W    3.10.65-rt69 #1
[   81.239533@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   81.239541@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[   81.239548@3] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[   81.239555@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[   81.239557@3] Exception stack(0xef2adf98 to 0xef2adfe0)
[   81.239561@3] df80:                                                       00000003 00000000
[   81.239566@3] dfa0: 0000c68e 00000000 ef2ac000 ef2ac000 c0945fd0 c06434ec c09aa6a4 c09aa6a4
[   81.239572@3] dfc0: 00000001 ef2ac000 00000000 ef2adfe0 c000f54c c000f550 60070013 ffffffff
[   81.239580@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000f550>] (arch_cpu_idle+0x24/0x2c)
[   81.239589@3] [<c000f550>] (arch_cpu_idle+0x24/0x2c) from [<c0069ce0>] (cpu_startup_entry+0xa8/0x148)
[   81.239596@3] [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) from [<0082fa64>] (0x82fa64)

[mlinuxguy]

I would go back to your previous and fix all the:

Code: Select all

-	spin_lock(..whatever..);
+	raw_spin_lock(..whatever..);

An example patch:
http://comments.gmane.org/gmane.linux.rt.user/8310

Get that working before changing to noHz as the tickless change ripples out with lots of effects.

[Clx]

I would go back to your previous and fix all the:

Code: Select all

-	spin_lock(..whatever..);
+	raw_spin_lock(..whatever..);

An example patch:
http://comments.gmane.org/gmane.linux.rt.user/8310

Get that working before changing to noHz as the tickless change ripples out with lots of effects.

No, I changed from NO_HZ to the normal tick system. The NO_HZ was default in the kernel.
It seems like the problem is caused by aml_sdhc, will look into that...

[mlinuxguy]

You can look at my old 3.8.rt patch set:
https://github.com/hardkernel/linux/com ... beaf50e7e7
search down for something like: include/asm-generic/cmpxchg-local.h
and see my changes to spin_locks().

If that compare link above doesn't work start at the main GIT tree and hit compare:
https://github.com/hardkernel/linux/tre ... d-3.8.y-rt

[Clx]

You can look at my old 3.8.rt patch set:
https://github.com/hardkernel/linux/com ... beaf50e7e7
search down for something like: include/asm-generic/cmpxchg-local.h
and see my changes to spin_locks().

If that compare link above doesn't work start at the main GIT tree and hit compare:
https://github.com/hardkernel/linux/tre ... d-3.8.y-rt

Thanks, the link works! I'm currently analyzing it.

Are you sure that all spin_locks should be changed to raw_spin_locks? That would work, but would also create a lot of non-preemptible sections, that may affect latencies. I thought that there should be some rule which sections should be spin_locked and which raw_spin_locked.

[mlinuxguy]

What I did was find all the missing hunks, fix those.
Then I looked at the crash stacks, found any spin_lock() and converted them to raw_spin_lock().
Once done I had a stable kernel, it would crash when I did things like UART, so I would go off to those routines and find spin_locks() there to convert.

It was a iterative process, I ended up leaving a few normal spin_locks behind....

[Clx]

What I did was find all the missing hunks, fix those.
Then I looked at the crash stacks, found any spin_lock() and converted them to raw_spin_lock().
Once done I had a stable kernel, it would crash when I did things like UART, so I would go off to those routines and find spin_locks() there to convert.

It was a iterative process, I ended up leaving a few normal spin_locks behind....

Understood, thanks! I'll try to fix what I can. Maybe there will be a ready rt-kernel before your odroid-c1 arrives:)

[Clx]

Converting spinlock_t mrq_lock to raw_spinlock_t mrq_lock in aml_sd (d7c9f83a8c6afc25a4196f265ad337438524c4ad) seems to fix the previous issue, but now it's complaining about something different...

Code: Select all

[   82.641480@2] BUG: scheduling while atomic: swapper/2/0/0x00000002
[   82.641483@2] Modules linked in:
[   82.641484@0]
[   82.641485@2] Modules linked in:
[   82.641487@2] Preemption disabled at:
[   82.641490@0] [<  (null)>]   (null)
[   82.641492@2] Preemption disabled at:[<  (null)>]   (null)
[   82.641492@0]
[   82.641493@2]
[   82.641498@2] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.10.65-rt69 #1
[   82.641501@0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.65-rt69 #1
[   82.641521@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   82.641531@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   82.641541@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0)
[   82.641549@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0)
[   82.641557@0] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0)
[   82.641565@2] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0)
[   82.641573@0] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94)
[   82.641581@2] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94)
[   82.641588@0] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270)
[   82.641595@2] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270)
[   82.641605@0] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   82.641613@2] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   82.641621@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   82.641629@2] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   82.641638@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   82.641646@2] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   82.641657@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   82.641666@2] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   82.641674@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   82.641682@2] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   82.641690@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   82.641698@2] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   82.641708@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   82.641716@2] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   82.641726@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314)
[   82.641732@2] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04)
[   82.641736@2] bad: scheduling from the idle thread!
[   82.641740@2] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W    3.10.65-rt69 #1
[   82.641750@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   82.641761@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0057bb8>] (dequeue_task_idle+0x1c/0x28)
[   82.641770@2] [<c0057bb8>] (dequeue_task_idle+0x1c/0x28) from [<c0051afc>] (dequeue_task+0xb8/0xc4)
[   82.641779@2] [<c0051afc>] (dequeue_task+0xb8/0xc4) from [<c063df84>] (__schedule+0x598/0x5c0)
[   82.641788@2] [<c063df84>] (__schedule+0x598/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94)
[   82.641795@2] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270)
[   82.641804@2] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   82.641813@2] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   82.641822@2] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   82.641832@2] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   82.641840@2] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   82.641849@2] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   82.641858@2] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   82.641865@2] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04)
[   82.761472@0] BUG: scheduling while atomic: swapper/0/0/0x00000002
[   82.761476@0] Modules linked in:
[   82.761480@0] Preemption disabled at:[<  (null)>]   (null)
[   82.761481@0]
[   82.761487@0] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.10.65-rt69 #1
[   82.761509@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   82.761521@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0)
[   82.761532@0] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0)
[   82.761541@0] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94)
[   82.761549@0] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270)
[   82.761561@0] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   82.761570@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   82.761579@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   82.761591@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   82.761600@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   82.761609@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   82.761619@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   82.761631@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314)
[   82.781466@0] BUG: scheduling while atomic: swapper/0/0/0x00000002
[   82.781471@1] BUG: scheduling while atomic: swapper/1/0/0x00000002
[   82.781473@1] Modules linked in:
[   82.781474@0]
[   82.781475@1] Modules linked in:
[   82.781477@1] Preemption disabled at:
[   82.781479@0] [<  (null)>]   (null)
[   82.781481@1] Preemption disabled at:[<  (null)>]   (null)
[   82.781482@0]
[   82.781483@1]
[   82.781486@0] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.10.65-rt69 #1
[   82.781490@1] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W    3.10.65-rt69 #1
[   82.781500@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   82.781510@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   82.781518@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0)
[   82.781526@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0)
[   82.781534@0] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0)
[   82.781542@1] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0)
[   82.781550@0] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94)
[   82.781557@1] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94)
[   82.781564@0] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270)
[   82.781571@1] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270)
[   82.781579@0] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   82.781587@1] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   82.781595@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   82.781603@1] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   82.781612@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   82.781620@1] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   82.781629@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   82.781638@1] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   82.781646@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   82.781654@1] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   82.781662@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   82.781670@1] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   82.781678@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   82.781687@1] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   82.781695@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314)
[   82.781701@1] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04)
[   82.781704@1] bad: scheduling from the idle thread!
[   82.781708@1] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G        W    3.10.65-rt69 #1
[   82.781717@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   82.781729@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c0057bb8>] (dequeue_task_idle+0x1c/0x28)
[   82.781737@1] [<c0057bb8>] (dequeue_task_idle+0x1c/0x28) from [<c0051afc>] (dequeue_task+0xb8/0xc4)
[   82.781746@1] [<c0051afc>] (dequeue_task+0xb8/0xc4) from [<c063df84>] (__schedule+0x598/0x5c0)
[   82.781754@1] [<c063df84>] (__schedule+0x598/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94)
[   82.781762@1] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270)
[   82.781771@1] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   82.781779@1] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   82.781789@1] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   82.781798@1] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   82.781806@1] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   82.781815@1] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   82.781824@1] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   82.781830@1] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04)
[   96.501355@0] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 1
[   96.501362@0] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.10.65-rt69 #1
[   96.501384@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   96.501396@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c0632e84>] (panic+0x98/0x1f8)
[   96.501407@0] [<c0632e84>] (panic+0x98/0x1f8) from [<c008b614>] (watchdog_timer_fn+0xd8/0x270)
[   96.501418@0] [<c008b614>] (watchdog_timer_fn+0xd8/0x270) from [<c004b628>] (__run_hrtimer.isra.15+0xc8/0x148)
[   96.501427@0] [<c004b628>] (__run_hrtimer.isra.15+0xc8/0x148) from [<c004c008>] (hrtimer_interrupt+0x148/0x31c)
[   96.501436@0] [<c004c008>] (hrtimer_interrupt+0x148/0x31c) from [<c001c6a0>] (meson_timer_interrupt+0x68/0x98)
[   96.501444@0] [<c001c6a0>] (meson_timer_interrupt+0x68/0x98) from [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180)
[   96.501452@0] [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180) from [<c008c2f8>] (handle_irq_event+0x78/0x98)
[   96.501462@0] [<c008c2f8>] (handle_irq_event+0x78/0x98) from [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178)
[   96.501470@0] [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178) from [<c008ba5c>] (generic_handle_irq+0x24/0x38)
[   96.501479@0] [<c008ba5c>] (generic_handle_irq+0x24/0x38) from [<c000f0b0>] (handle_IRQ+0x68/0x90)
[   96.501486@0] [<c000f0b0>] (handle_IRQ+0x68/0x90) from [<c000845c>] (gic_handle_irq+0x3c/0x5c)
[   96.501494@0] [<c000845c>] (gic_handle_irq+0x3c/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[   96.501497@0] Exception stack(0xc092fe08 to 0xc092fe50)
[   96.501502@0] fe00:                   00000001 00000000 00000097 00000096 c0a601f4 c092e000
[   96.501509@0] fe20: c092fecc c0979f80 c0949ef8 c0a601d0 00000000 00177000 00004e1f c092fe50
[   96.501512@0] fe40: c063f7c4 c063f7f4 80000013 ffffffff
[   96.501522@0] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f7f4>] (_raw_spin_lock+0x40/0x50)
[   96.501529@0] [<c063f7f4>] (_raw_spin_lock+0x40/0x50) from [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270)
[   96.501540@0] [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   96.501550@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   96.501559@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   96.501569@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   96.501577@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   96.501586@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   96.501597@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   96.501609@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314)
[   96.501614@3] CPU3: stopping
[   96.501620@3] CPU: 3 PID: 34 Comm: ksoftirqd/3 Tainted: G        W    3.10.65-rt69 #1
[   96.501633@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   96.501641@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[   96.501648@3] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[   96.501654@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[   96.501657@3] Exception stack(0xef2e5d48 to 0xef2e5d90)
[   96.501662@3] 5d40:                   00000000 00000000 00000099 00000096 c0a601f4 ef2e4000
[   96.501668@3] 5d60: ef2e5e0c c0979f80 ef2bc780 c0a601d0 0493e000 00177000 00000400 ef2e5d90
[   96.501672@3] 5d80: c0052750 c063f7f4 80000013 ffffffff
[   96.501679@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f7f4>] (_raw_spin_lock+0x40/0x50)
[   96.501686@3] [<c063f7f4>] (_raw_spin_lock+0x40/0x50) from [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270)
[   96.501695@3] [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   96.501704@3] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   96.501713@3] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c003639c>] (call_timer_fn.isra.21+0x24/0x80)
[   96.501720@3] [<c003639c>] (call_timer_fn.isra.21+0x24/0x80) from [<c0036628>] (run_timer_softirq+0x230/0x278)
[   96.501730@3] [<c0036628>] (run_timer_softirq+0x230/0x278) from [<c0030b60>] (do_current_softirqs+0x128/0x224)
[   96.501738@3] [<c0030b60>] (do_current_softirqs+0x128/0x224) from [<c0030c8c>] (run_ksoftirqd+0x30/0x50)
[   96.501746@3] [<c0030c8c>] (run_ksoftirqd+0x30/0x50) from [<c004fdbc>] (smpboot_thread_fn+0x254/0x26c)
[   96.501756@3] [<c004fdbc>] (smpboot_thread_fn+0x254/0x26c) from [<c0047dc4>] (kthread+0x98/0xa0)
[   96.501764@3] [<c0047dc4>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c)
[   96.501768@2] CPU2: stopping
[   96.501773@2] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W    3.10.65-rt69 #1
[   96.501785@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[   96.501793@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[   96.501800@2] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[   96.501807@2] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[   96.501810@2] Exception stack(0xef2abe38 to 0xef2abe80)
[   96.501813@2] be20:                                                       00000001 00000000
[   96.501819@2] be40: 00000098 00000096 c0a601f4 ef2aa000 ef2761dc ef2abe8c ef275d80 ef2abe8c
[   96.501825@2] be60: 00000002 ef275801 00000000 ef2abe80 c063f7c4 c063f7f4 800f0013 ffffffff
[   96.501832@2] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f7f4>] (_raw_spin_lock+0x40/0x50)
[   96.501839@2] [<c063f7f4>] (_raw_spin_lock+0x40/0x50) from [<c063ec8c>] (rt_spin_lock_slowlock+0x180/0x270)
[   96.501848@2] [<c063ec8c>] (rt_spin_lock_slowlock+0x180/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc)
[   96.501856@2] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374)
[   96.501866@2] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108)
[   96.501875@2] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84)
[   96.501884@2] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c)
[   96.501892@2] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20)
[   96.501902@2] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148)
[   96.501909@2] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04)
[   96.765997@0] SMP: failed to stop secondary CPUs

[Clx]

Last issue was solved by disabling frequency scaling completely. Well, it's not really needed in RT kernel anyway, but I also had to disable thermal monitor due to the dependency. Will try to enable it partially. Also, /media/boot cannot be mounted, not sure why... Currently system works without it.

The kernel has been working for almost an hour now without an error.

[mlinuxguy]

Awesome work!
If what you have turns out to be stable, run some of the rt latency tests
https://rt.wiki.kernel.org/index.php/Cyclictest

You can preserve your kernel uImage for both standard kernel and RT, copy them back and forth to the boot partition to switch between
run 'cyclictest' under each kernel and report your latencies.

[adbensi]

Very nice job,
I would like to test preempt rt on C1 too...
Did you have the source on github?
Congratulations!

[Clx]

Very nice job,
I would like to test preempt rt on C1 too...
Did you have the source on github?
Congratulations!

The source is here - https://github.com/Clouded/linux-rt-odroid-c1
Even though the kernel runs, it's broken and requires further research.

Disabling cpu frequency scaling results in kernel not being able to load it's own modules from /lib/modules/ (the ones that I compiled together with the kernel and copied there).
Depmod prints error on every module there, like they are all broken.
Didn't have time lately to work on that, but will try to do something in a couple of days.

[adbensi]

Thanks Mikhail,
I will try it too,
Best Regards

[adbensi]

root@odroid:~# uname -a
Linux odroid 3.10.65-rt69 #1 SMP PREEMPT RT Wed Feb 11 06:24:28 ACDT 2015 armv7l armv7l armv7l GNU/Linux

root@odroid:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
35: 38004 0 0 0 GIC am_osd_vsync, vsync
40: 0 1536 0 0 GIC eth0
47: 0 0 0 0 GIC keypad
60: 1033 0 0 0 GIC sdio
62: 0 0 0 0 GIC dwc_otg, dwc_otg_hcd:usb2, dwc_otg_pcd
63: 0 0 0 5055975 GIC dwc_otg, dwc_otg_hcd:usb1
89: 1 0 0 0 GIC amhdmitx
92: 119287 0 0 0 GIC MESON TIMER-F
93: 0 118608 0 0 GIC MESON TIMER-G
94: 0 0 118706 0 GIC MESON TIMER-H
95: 0 0 0 115550 GIC MESON TIMER-I
99: 1 0 0 0 GIC mmc_in
101: 1 0 0 0 GIC mmc_out
107: 0 0 0 0 GIC uart_b_ttyS2:
110: 15042 0 0 0 GIC sdhc
121: 38003 0 0 0 GIC rdma, osd_rdma
122: 0 0 0 0 GIC uart_ao_ttyS0:
178: 0 0 0 0 GIC audiodsp_mailbox
182: 0 0 0 0 GIC ge2d irq
183: 0 0 0 0 GIC amhdmitx-aocec
192: 0 0 0 0 GIC Mali_GP
193: 0 0 0 0 GIC Mali_GP_MMU
194: 0 0 0 0 GIC Mali_PP_Broadcast
196: 0 0 0 0 GIC Mali_PP0
197: 0 0 0 0 GIC Mali_PP0_MMU
198: 0 0 0 0 GIC Mali_PP1
199: 0 0 0 0 GIC Mali_PP1_MMU
IPI0: 0 0 0 0 CPU wakeup interrupts
IPI1: 0 0 0 0 Timer broadcast interrupts
IPI2: 28359 70613 68972 33179 Rescheduling interrupts
IPI3: 6 8 7 8 Function call interrupts
IPI4: 126 66 109 4 Single function call interrupts
IPI5: 0 0 0 0 CPU stop interrupts
IPI6: 0 0 0 0 CPU backtrace
Err: 0

Test case: clock_nanosleep(TIME_ABSTIME), Interval 10000 microseconds,. 10000 loops, no load.
cyclictest -t1 -p 80 -n -i 10000 -l 10000
policy: fifo: loadavg: 0.89 1.21 1.10 1/316 2025
T: 0 ( 2025) P:80 I:10000 C: 10000 Min: 11 Act: 13 Avg: 19 Max: 43

Test case: clock_nanosleep(TIME_ABSTIME), Interval 10000 micro seconds,. 10000 loops, 100% load.
cyclictest -t1 -p 80 -n -i 10000 -l 10000
policy: fifo: loadavg: 1.10 0.99 1.02 3/316 2028
T: 0 ( 2027) P:80 I:10000 C: 10000 Min: 10 Act: 23 Avg: 19 Max: 56

Test case: POSIX interval timer, Interval 10000 micro seconds,. 10000 loops, no load.
cyclictest -t1 -p 80 -i 10000 -l 10000
policy: fifo: loadavg: 0.81 0.81 0.93 1/315 2030
T: 0 ( 2030) P:80 I:10000 C: 10000 Min: 29 Act: 36 Avg: 45 Max: 103

Test case: POSIX interval timer, Interval 10000 micro seconds,. 10000 loops, 100% load.
cyclictest -t1 -p 80 -i 10000 -l 10000
policy: fifo: loadavg: 0.82 0.59 0.78 4/316 2033
T: 0 ( 2032) P:80 I:10000 C: 10000 Min: 29 Act: 35 Avg: 45 Max: 103

Test case: clock_nanosleep(TIME_ABSTIME), Interval 500 micro seconds,. 100000 loops, no load.
cyclictest -t1 -p 80 -i 500 -n -l 100000
policy: fifo: loadavg: 1.78 0.93 0.88 2/315 2035
T: 0 ( 2035) P:80 I:500 C: 100000 Min: 8 Act: 12 Avg: 14 Max: 74

Test case: clock_nanosleep(TIME_ABSTIME), Interval 500 micro seconds,. 100000 loops, 100% load.
cyclictest -t1 -p 80 -i 500 -n -l 100000
policy: fifo: loadavg: 1.22 0.93 0.88 2/315 2037
T: 0 ( 2037) P:80 I:500 C: 100000 Min: 8 Act: 20 Avg: 13 Max: 53

Test case: POSIX interval timer, Interval 500 micro seconds,. 100000 loops, no load.
cyclictest -t1 -p 80 -i 500 -l 100000
policy: fifo: loadavg: 1.39 1.06 0.93 1/315 2039
T: 0 ( 2039) P:80 I:500 C: 100000 Min: 25 Act: 44 Avg: 35 Max: 153

Test case: POSIX interval timer, Interval 500 micro seconds,. 100000 loops, 100% load.
cyclictest -t1 -p 80 -i 500 -l 100000
policy: fifo: loadavg: 1.08 1.03 0.93 2/315 2041
T: 0 ( 2041) P:80 I:500 C: 100000 Min: 25 Act: 28 Avg: 35 Max: 5448

root@odroid:~# cyclictest -p 80 -t5 -n
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 1.41 1.40 1.13 4/321 2020

T: 0 ( 2001) P:80 I:1000 C: 628676 Min: 9 Act: 27 Avg: 20 Max: 82
T: 1 ( 2002) P:80 I:1500 C: 419118 Min: 9 Act: 17 Avg: 19 Max: 81
T: 2 ( 2003) P:80 I:2000 C: 314338 Min: 9 Act: 30 Avg: 18 Max: 67
T: 3 ( 2004) P:80 I:2500 C: 251470 Min: 9 Act: 23 Avg: 19 Max: 72
T: 4 ( 2005) P:80 I:3000 C: 209558 Min: 9 Act: 22 Avg: 19 Max: 75

root@odroid:~# cyclictest -h 100 -q -i 1000 -l 1000000 -n -p 90 -t
# /dev/cpu_dma_latency set to 0us
# Histogram
000000 000000 000000 000000 000000
000001 000000 000000 000000 000000
000002 000000 000000 000000 000000
000003 000000 000000 000000 000000
000004 000000 000000 000000 000000
000005 000000 000000 000000 000000
000006 000000 000000 000000 000000
000007 000000 000000 000000 000000
000008 000351 000004 000225 000090
000009 297081 123482 109351 099271
000010 421718 383068 493099 576951
000011 152816 243205 239785 198354
000012 048019 112962 073045 053963
000013 012317 053596 028244 016136
000014 008699 017104 011050 009815
000015 006982 010354 006703 006332
000016 005526 007509 004723 004354
000017 004255 006108 003797 003547
000018 003916 005847 003807 003346
000019 004148 005450 004355 004007
000020 004184 004859 004220 003980
000021 003817 004113 003118 003212
000022 003446 003358 002618 002641
000023 003027 002913 002194 002266
000024 002579 002407 001841 001828
000025 002096 002020 001554 001557
000026 001808 001762 001299 001387
000027 001646 001617 001113 001227
000028 001439 001482 000913 001086
000029 001246 001294 000691 000957
000030 001196 001100 000522 000789
000031 001045 000961 000446 000654
000032 001002 000802 000332 000513
000033 000928 000664 000268 000416
000034 000817 000550 000194 000336
000035 000679 000409 000132 000255
000036 000625 000273 000099 000173
000037 000487 000188 000062 000120
000038 000433 000125 000041 000097
000039 000344 000119 000030 000077
000040 000276 000075 000024 000060
000041 000249 000057 000019 000036
000042 000188 000039 000016 000035
000043 000116 000037 000011 000021
000044 000097 000023 000009 000028
000045 000077 000014 000008 000015
000046 000055 000017 000007 000015
000047 000058 000008 000004 000008
000048 000041 000005 000005 000007
000049 000031 000008 000004 000008
000050 000028 000001 000006 000004
000051 000014 000000 000003 000003
000052 000014 000001 000003 000003
000053 000017 000001 000001 000005
000054 000015 000002 000001 000001
000055 000012 000001 000002 000004
000056 000006 000000 000000 000001
000057 000005 000000 000000 000001
000058 000008 000001 000000 000000
000059 000002 000001 000000 000002
000060 000004 000001 000003 000001
000061 000003 000001 000000 000000
000062 000000 000002 000002 000000
000063 000002 000000 000000 000000
000064 000000 000000 000000 000000
000065 000002 000000 000000 000001
000066 000002 000000 000001 000001
000067 000000 000000 000000 000001
000068 000001 000000 000000 000000
000069 000001 000000 000000 000000
000070 000001 000000 000000 000001
000071 000000 000000 000000 000000
000072 000001 000000 000000 000001
000073 000000 000000 000000 000000
000074 000000 000000 000000 000000
000075 000000 000000 000000 000000
000076 000001 000000 000000 000000
000077 000000 000000 000000 000000
000078 000000 000000 000000 000000
000079 000001 000000 000000 000000
000080 000000 000000 000000 000000
000081 000000 000000 000000 000000
000082 000000 000000 000000 000000
000083 000000 000000 000000 000000
000084 000000 000000 000000 000000
000085 000000 000000 000000 000000
000086 000000 000000 000000 000000
000087 000000 000000 000000 000000
000088 000000 000000 000000 000000
000089 000000 000000 000000 000000
000090 000000 000000 000000 000000
000091 000000 000000 000000 000000
000092 000000 000000 000000 000000
000093 000000 000000 000000 000000
000094 000000 000000 000000 000000
000095 000000 000000 000000 000000
000096 000000 000000 000000 000000
000097 000000 000000 000000 000000
000098 000000 000000 000000 000000
000099 000000 000000 000000 000000
# Total: 001000000 001000000 001000000 001000000
# Min Latencies: 00008 00008 00008 00008
# Avg Latencies: 00010 00011 00010 00010
# Max Latencies: 00079 00062 00066 00072
# Histogram Overflows: 00000 00000 00000 00000
# Histogram Overflow at cycle number:
# Thread 0:
# Thread 1:
# Thread 2:
# Thread 3:

[mlinuxguy]

Nice! I'll have to free up some time to patch my kernel with the RT patch set and do some testing
Been too busy porting python stuff to the C1....

[adbensi]

Nicely, python support is very very important thanks a lot!

On the 3.10.67 network driver is very stable,
I will update this driver of the 3.10.65-rt and put it on the github,
but if Your port are newest kernel, please post here

I am very happy with this board.

[adbensi]

Huhuuuu patches-3.10.67-rt71.tar.xz available

[Clx]

Huhuuuu patches-3.10.67-rt71.tar.xz available

Applied over 3.10.67 from Hardkernel's Linux Github repo - https://github.com/Clouded/linux-rt-odr ... 10.67-rt71
Haven't seen errors as on the previous one even without sdhc and freq scaling fixes. Will be testing it tomorrow.

[adbensi]

Thanks,

I updated today with your tips was very easy
I notice different performances between FULL RT and DESKTOP kernel. Is the same with you?

With full RT Kernel:

root@odroid:~# ping www.ufscar.br
PING medusa.ufscar.br (200.9.84.70) 56(84) bytes of data.
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=1 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=2 ttl=56 time=5005 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=3 ttl=56 time=1001 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=4 ttl=56 time=17.6 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=5 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=6 ttl=56 time=16.1 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=7 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=8 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=9 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=10 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=11 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=12 ttl=56 time=1000 ms
--- medusa.ufscar.br ping statistics ---
21 packets transmitted, 20 received, 4% packet loss, time 24058ms
rtt min/avg/max/mdev = 16.132/1055.138/5005.960/971.807 ms, pipe 2

root@odroid:~# uname -a
Linux odroid 3.10.67-rt71 #3 SMP PREEMPT RT Fri Feb 13 04:00:02 ACDT 2015 armv7l armv7l armv7l GNU/Linux

==============================================================================
With Desktop Kernel:

root@odroid:~# ping www.google.com.br
PING www.google.com.br (216.58.222.35) 56(84) bytes of data.
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=1 ttl=53 time=22.4 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=2 ttl=53 time=23.3 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=3 ttl=53 time=23.1 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=4 ttl=53 time=24.6 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=5 ttl=53 time=23.4 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=6 ttl=53 time=23.3 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=7 ttl=53 time=31.0 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=8 ttl=53 time=30.0 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=9 ttl=53 time=23.8 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=10 ttl=53 time=21.6 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=11 ttl=53 time=21.7 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=12 ttl=53 time=23.4 ms
--- www.google.com.br ping statistics ---
12 packets transmitted, 12 received, 0% packet loss, time 11016ms
rtt min/avg/max/mdev = 21.697/24.357/31.045/2.892 ms

root@odroid:~# uname -a
Linux odroid 3.10.67-55 #1 SMP PREEMPT Tue Feb 3 12:46:15 BRST 2015 armv7l armv7l armv7l GNU/Linux

[robroy]

Once you guys have everything working, could you zip up the kernel files for me and post them? Maybe add MIDI support too That way, I can publish an updated version of my Ubuntu Studio image (Smooth Groove) for the benefit of artistic ODROIDians!

[adbensi]

My build works with network delay,
my be Clx have different results... I will try isolate the problem and then zip one image

[Clx]

My build works with network delay,
my be Clx have different results... I will try isolate the problem and then zip one image

Maybe there was something wrong with www.ufscar.br?
I didn't test the network yet...

[Clx]

Network testing:

Code: Select all

root@odroid:~# ping google.com -c 10
PING google.com (173.194.122.206) 56(84) bytes of data.
64 bytes from 173.194.122.206: icmp_seq=1 ttl=57 time=16.0 ms
64 bytes from 173.194.122.206: icmp_seq=2 ttl=57 time=13.7 ms
64 bytes from 173.194.122.206: icmp_seq=3 ttl=57 time=13.6 ms
64 bytes from 173.194.122.206: icmp_seq=4 ttl=57 time=15.7 ms
64 bytes from 173.194.122.206: icmp_seq=5 ttl=57 time=13.7 ms
64 bytes from 173.194.122.206: icmp_seq=6 ttl=57 time=20.1 ms
64 bytes from 173.194.122.206: icmp_seq=7 ttl=57 time=13.7 ms
64 bytes from 173.194.122.206: icmp_seq=8 ttl=57 time=13.8 ms
64 bytes from 173.194.122.206: icmp_seq=9 ttl=57 time=13.7 ms
64 bytes from 173.194.122.206: icmp_seq=10 ttl=57 time=14.1 ms

--- google.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 13.673/14.862/20.179/1.963 ms
root@odroid:~# Linux odroid 3.10.67-rt71 #1 SMP PREEMPT RT Thu Feb 12 19:33:00 MSK 2015 armv7l armv7l armv7l GNU/Linux

Results of a quick cyclictest

Code: Select all

odroid@odroid:~$ sudo cyclictest -l1000000 -m -n -a0 -t1 -p99 -i400 -q
# /dev/cpu_dma_latency set to 0us

# Total: 001000000
# Min Latencies: 00006
# Avg Latencies: 00008
# Max Latencies: 00046
# Histogram Overflows: 00000
# Histogram Overflow at cycle number:
# Thread 0:

[Clx]

I've updated the first post with a link on compiled kernel and modules. I'd appreciate any testing feedback.

[adbensi]

Very nice, I will try it,
thanks again

[adbensi]

The same issue
Please, did You can to do diff of the file attached with Your system?
and theses options :

root@odroid:~# cat /proc/irq/63/smp_affinity
2

PowerTOP 2.5 Overview Idle stats Frequency stats Device stats Tunables
>> Bad NMI watchdog should be turned off
Bad VM writeback timeout
Bad Autosuspend for USB device USB Receiver [Logitech]
Bad Autosuspend for USB device Basic Optical Mouse [Microsoft]
Bad Using 'ondemand' cpufreq governor
Good Autosuspend for USB device USB2.0 Hub [1-1]
Good Autosuspend for USB device DWC OTG Controller [usb1]
Good Autosuspend for USB device DWC OTG Controller [usb2]
Good Wake-on-lan status for device ip6tnl0
Good Wake-on-lan status for device eth0
Good Wake-on-lan status for device sit0

I get the same problem with Your pre-build kernel too. My kernel are correct, the problem are in my system..

root@odroid:~# uname -a
Linux odroid 3.10.67-rt71 #1 SMP PREEMPT RT Thu Feb 12 19:33:00 MSK 2015 armv7l armv7l armv7l GNU/Linux
root@odroid:~# ping http://www.ufscar.br
PING medusa.ufscar.br (200.9.84.70) 56(84) bytes of data.
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=1 ttl=56 time=462 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=2 ttl=56 time=898 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=3 ttl=56 time=999 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=4 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=5 ttl=56 time=19.2 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=6 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=7 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=8 ttl=56 time=17.0 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=9 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=10 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=11 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=12 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=13 ttl=56 time=18.5 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=14 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=15 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=16 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=17 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=18 ttl=56 time=880 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=19 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=20 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=21 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=22 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=23 ttl=56 time=18.2 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=24 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=25 ttl=56 time=1000 ms
^C
--- medusa.ufscar.br ping statistics ---
26 packets transmitted, 25 received, 3% packet loss, time 29508ms
rtt min/avg/max/mdev = 17.016/814.006/1008.979/363.517 ms, pipe 2
root@odroid:~#

[Clx]

The same issue
Please, did You can to do diff of the file attached with Your system?
and theses options :

root@odroid:~# cat /proc/irq/63/smp_affinity
2

PowerTOP 2.5 Overview Idle stats Frequency stats Device stats Tunables
>> Bad NMI watchdog should be turned off
Bad VM writeback timeout
Bad Autosuspend for USB device USB Receiver [Logitech]
Bad Autosuspend for USB device Basic Optical Mouse [Microsoft]
Bad Using 'ondemand' cpufreq governor
Good Autosuspend for USB device USB2.0 Hub [1-1]
Good Autosuspend for USB device DWC OTG Controller [usb1]
Good Autosuspend for USB device DWC OTG Controller [usb2]
Good Wake-on-lan status for device ip6tnl0
Good Wake-on-lan status for device eth0
Good Wake-on-lan status for device sit0

I get the same problem with Your pre-build kernel too. My kernel are correct, the problem are in my system..

root@odroid:~# uname -a
Linux odroid 3.10.67-rt71 #1 SMP PREEMPT RT Thu Feb 12 19:33:00 MSK 2015 armv7l armv7l armv7l GNU/Linux
root@odroid:~# ping http://www.ufscar.br
PING medusa.ufscar.br (200.9.84.70) 56(84) bytes of data.
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=1 ttl=56 time=462 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=2 ttl=56 time=898 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=3 ttl=56 time=999 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=4 ttl=56 time=1000 ms
--- medusa.ufscar.br ping statistics ---
26 packets transmitted, 25 received, 3% packet loss, time 29508ms

Sysctl is attached.

Code: Select all

odroid@odroid:~# cat /proc/irq/63/smp_affinity
8

Code: Select all

   Bad           VM writeback timeout
   Bad           Autosuspend for USB device 802.11n WLAN Adapter [Realtek]
   Bad           Using 'ondemand' cpufreq governor
   Good          Wireless Power Saving for interface wlan0
   Good          Autosuspend for USB device USB2.0 Hub [1-1]
   Good          Autosuspend for USB device DWC OTG Controller [usb1]
   Good          Autosuspend for USB device DWC OTG Controller [usb2]
   Good          Wake-on-lan status for device ip6tnl0
   Good          Wake-on-lan status for device eth0
   Good          Wake-on-lan status for device sit0
   Good          Wake-on-lan status for device wlan0

What are you trying to tune something with a PowerTop?

[Clx]

I was stressing the system with rt-kernel currently and got this:

Code: Select all

[2700.100921] Kernel panic - not syncing: Watchdog detected ha

I know that's a hard lockup, but the message is cut, so I don't know in what case it happened. Kernel log has no messages past that time.

[UPDATE]

Also, right after running powertop I've stumbled upon this:

Code: Select all

[  106.842239@3] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=2100 jiffies g=619 c=618 q=343)
[  106.871293@0] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 0, t=2102 jiffies, g=619, c=618, q=343)
[  169.892252@3] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=8405 jiffies g=619 c=618 q=374)
[  169.921759@0] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 0, t=8407 jiffies, g=619, c=618, q=374)
[  232.942258@3] INFO: rcu_preempt self-detected stall on CPU { 3}  (t=14710 jiffies g=619 c=618 q=387)
[  232.971178@0] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 0, t=14712 jiffies, g=619, c=618, q=387)
[  242.202586@2] INFO: task kworker/2:1:44 blocked for more than 120 seconds.
[  242.282408@2] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[  242.295041@2] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: khungtaskd/58
[  242.305269@3] IPI backtrace for cpu 3
[  242.305286@3]
[  242.305360@3] CPU: 3 PID: 91 Comm: irq/63-dwc_otg_ Not tainted 3.10.67-rt71 #1
[  242.305409@3] task: ef572680 ti: eead2000 task.ti: eead2000
[  242.305488@3] PC is at __schedule+0x44/0x5c0
[  242.305554@3] LR is at get_parent_ip+0x10/0x2c
[  242.305617@3] pc : [<c0636418>]    lr : [<c005277c>]    psr: 60070113
[  242.305617@3] sp : eead3ed0  ip : 00000000  fp : eead3f1c
[  242.305654@3] r10: 00000001  r9 : eead2000  r8 : eeb33ae0
[  242.305698@3] r7 : eead2000  r6 : c093ea98  r5 : 00000000  r4 : ef572680
[  242.305739@3] r3 : ef572680  r2 : 00000001  r1 : eeb33ae0  r0 : 00000000
[  242.305789@3] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  242.305835@3] Control: 10c5387d  Table: 2da4404a  DAC: 00000015
[  242.305861@3]
[  242.305861@3] PC: 0xc0636398:
[  242.306015@3] 6398  eaffff99 e92d4800 e3e01102 e28db004 e3a02082 ebffff94 e3700c02 13a00000
[  242.306157@3] 63b8  e8bd8800 e52db004 e3a02082 e28db000 e28bd000 e8bd0800 eaffff8b e92d4ff0
[  242.306297@3] 63d8  e28db020 e24dd02c e59f6598 e1a0200d e3c27d7f e59f3590 e3c7703f e1a09007
[  242.306438@3] 63f8  e50b3034 e50b303c e50b3040 e50b3030 e50b3044 e50b3038 e3a00001 ebe870df
[  242.306578@3] 6418  ebef1b8a e51b4034 e1a05000 e7968100 ebe97a7e e5993004 e0848008 e5987450
[  242.306719@3] 6438  e3c33101 e3530001 0a000004 e5973230 e3530000 1a000001 e1a00007 ebffd6b5
[  242.306859@3] 6458  ebef1b7a e51b403c e30dcfa0 e7963100 e34cc093 e59c1004 e0843003 e593257c
[  242.306999@3] 6478  e3110080 e2822001 e583257c 0a000004 e5983528 e3530000 0a000001 e2880c05
[  242.307025@3]
[  242.307025@3] LR: 0xc00526fc:
[  242.307165@3] 26fc  0a000014 e59f205c e3540000 e284101f a1a01004 e1a012c1 e204301f e5922000
[  242.307305@3] 271c  e7922101 e1a03332 e3130001 0a000009 e1a01007 e50b501c e50b4018 eb179613
[  242.307445@3] 273c  e5953008 e24b201c e59f101c e5930018 eb00d0dc ea000001 e1a01007 eb17960b
[  242.307585@3] 275c  e24bd014 e8bd88f0 c063c4e4 c0052ffc e92d4818 e28db00c e1a04000 eb00976c
[  242.307725@3] 277c  e3500000 0a000002 e3a04000 e1a00004 eb009767 e1a00004 e8bd8818 e59f30d4
[  242.307865@3] 279c  e92d4830 e1a05000 e28db00c e5933000 e3530000 e1a0300d e3c34d7f e3c4403f
[  242.308005@3] 27bc  e5943004 1a000028 e3530000 aa000020 eb0695f8 e3500000 08bd8830 e59f3098
[  242.308145@3] 27dc  e5933000 e3530000 18bd8830 e59f008c e3001b3e e59f2088 e59f3088 e8bd4830
[  242.308170@3]
[  242.308170@3] SP: 0xeead3e50:
[  242.308309@3] 3e50  00000003 c00133e0 fe00110c c093ebe0 eead3e88 fe001100 eeb33ae0 c0008474
[  242.308448@3] 3e70  c005277c c0636418 60070113 ffffffff eead3ebc c000dd40 00000000 eeb33ae0
[  242.308586@3] 3e90  00000001 ef572680 ef572680 00000000 c093ea98 eead2000 eeb33ae0 eead2000
[  242.308724@3] 3eb0  00000001 eead3f1c 00000000 eead3ed0 c005277c c0636418 60070113 ffffffff
[  242.308863@3] 3ed0  eead2000 ef572680 c0923ec0 c0923ec0 c0923ec0 c0923ec0 c0923ec0 c0923ec0
[  242.309001@3] 3ef0  00000000 0ec8b426 eead2000 ef572680 00000000 eead2000 00000000 eeb33ae0
[  242.309139@3] 3f10  c092ba00 00000001 eead3f2c c0636a94 00000001 eeb33ac0 eeb33ac0 c008c55c
[  242.309277@3] 3f30  c092ba50 c008c9f4 00000000 c008c910 c008c45c ef279d60 00000000 eeb33ac0
[  242.309302@3]
[  242.309302@3] FP: 0xeead3e9c:
[  242.309439@3] 3e9c  00000000 c093ea98 eead2000 eeb33ae0 eead2000 00000001 eead3f1c 00000000
[  242.309579@3] 3ebc  eead3ed0 c005277c c0636418 60070113 ffffffff eead2000 ef572680 c0923ec0
[  242.309718@3] 3edc  c0923ec0 c0923ec0 c0923ec0 c0923ec0 c0923ec0 00000000 0ec8b426 eead2000
[  242.309855@3] 3efc  ef572680 00000000 eead2000 00000000 eeb33ae0 c092ba00 00000001 eead3f2c
[  242.309993@3] 3f1c  c0636a94 00000001 eeb33ac0 eeb33ac0 c008c55c c092ba50 c008c9f4 00000000
[  242.310130@3] 3f3c  c008c910 c008c45c ef279d60 00000000 eeb33ac0 c008c45c 00000000 00000000
[  242.310265@3] 3f5c  00000000 00000000 c0047df0 ffffffff 00000000 00000000 eeb33ac0 00000000
[  242.310403@3] 3f7c  00000000 eead3f80 eead3f80 00000000 00000000 eead3f90 eead3f90 eead3fac
[  242.310428@3]
[  242.310428@3] R1: 0xeeb33a60:
[  242.310565@3] 3a60  00000000 12fabfc8 c094c57c 00000000 00000000 81240002 000034a7 00000000
[  242.310704@3] 3a80  31627375 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.310840@3] 3aa0  00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.310978@3] 3ac0  c008c364 ef417600 00000000 00000000 c02d0db4 ef572680 0000003f 00002080
[  242.311113@3] 3ae0  00000008 00000002 ef41765c eeb34280 00000000 00000000 00000000 00000000
[  242.311243@3] 3b00  eeb33b80 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.311371@3] 3b20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.311512@3] 3b40  20435744 2047544f 746e6f43 6c6c6f72 00007265 00000000 ffffffff ffffffff
[  242.311537@3]
[  242.311537@3] R3: 0xef572600:
[  242.311670@3] 2600  00000000 00000000 00000000 00000000 ef2f7888 00000000 00000020 00000000
[  242.311802@3] 2620  0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000
[  242.311931@3] 2640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.312059@3] 2660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.312192@3] 2680  00000001 00000000 eead2000 00000003 00208040 00000000 00000000 00000001
[  242.312327@3] 26a0  00000001 00000031 00000078 00000031 00000032 c063cb1c 00000400 00400000
[  242.312464@3] 26c0  00000001 00000000 00000000 ef5726cc ef5726cc 00000000 6a814bb8 00000038
[  242.312596@3] 26e0  0a62d090 00000014 00000000 00000000 00000000 00000000 00000002 00000000
[  242.312621@3]
[  242.312621@3] R4: 0xef572600:
[  242.312755@3] 2600  00000000 00000000 00000000 00000000 ef2f7888 00000000 00000020 00000000
[  242.312887@3] 2620  0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000
[  242.313015@3] 2640  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.313144@3] 2660  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.313276@3] 2680  00000001 00000000 eead2000 00000003 00208040 00000000 00000000 00000001
[  242.313411@3] 26a0  00000001 00000031 00000078 00000031 00000032 c063cb1c 00000400 00400000
[  242.313547@3] 26c0  00000001 00000000 00000000 ef5726cc ef5726cc 00000000 6a814bb8 00000038
[  242.313680@3] 26e0  0a62d090 00000014 00000000 00000000 00000000 00000000 00000002 00000000
[  242.313706@3]
[  242.313706@3] R6: 0xc093ea18:
[  242.313836@3] ea18  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.313965@3] ea38  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.314101@3] ea58  00000064 00008000 0000000f c14c1100 00000004 00000008 0000002c c14a1000
[  242.314239@3] ea78  c14c10c0 00000000 00000003 c14c1080 00000001 c14c1000 c14c1040 00001000
[  242.314377@3] ea98  00b7f000 00b87000 00b8f000 00b97000 0000f600 00000001 0002f9f9 0003efff
[  242.314513@3] eab8  0000687f 00002000 00000000 00000032 0000fffa 00000001 ef201d80 ef2f4190
[  242.314652@3] ead8  00000001 00000001 ef201580 c14c5200 00000011 ef201500 0001ffff ef201b80
[  242.314792@3] eaf8  00000064 00000010 c15c5200 0000ffff ef201480 00100000 ef206000 ef201e00
[  242.314818@3]
[  242.314818@3] R7: 0xeead1f80:
[  242.314954@3] 1f80  00000001 00000000 ef3d5f80 eead1f40 ef3d7050 00000000 00000000 00000000
[  242.315090@3] 1fa0  00000000 001172dc eead1340 00000000 00000000 a1ff0008 00003881 00000000
[  242.315229@3] 1fc0  76697264 00007265 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.315366@3] 1fe0  00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.315500@3] 2000  00000000 00000003 00000000 00000000 ef572680 c0946c24 00000003 00000015
[  242.315639@3] 2020  edec8400 00000000 c093ea98 ef572680 c14baec0 eead2000 ef2bc780 eead3f1c
[  242.315771@3] 2040  eead3ed0 c06368b0 00000000 00000000 00000000 00000000 00000000 00000000
[  242.315899@3] 2060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.315925@3]
[  242.315925@3] R8: 0xeeb33a60:
[  242.316062@3] 3a60  00000000 12fabfc8 c094c57c 00000000 00000000 81240002 000034a7 00000000
[  242.316201@3] 3a80  31627375 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.316337@3] 3aa0  00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.316475@3] 3ac0  c008c364 ef417600 00000000 00000000 c02d0db4 ef572680 0000003f 00002080
[  242.316608@3] 3ae0  00000008 00000002 ef41765c eeb34280 00000000 00000000 00000000 00000000
[  242.316739@3] 3b00  eeb33b80 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.316867@3] 3b20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.317007@3] 3b40  20435744 2047544f 746e6f43 6c6c6f72 00007265 00000000 ffffffff ffffffff
[  242.317033@3]
[  242.317033@3] R9: 0xeead1f80:
[  242.317168@3] 1f80  00000001 00000000 ef3d5f80 eead1f40 ef3d7050 00000000 00000000 00000000
[  242.317304@3] 1fa0  00000000 001172dc eead1340 00000000 00000000 a1ff0008 00003881 00000000
[  242.317442@3] 1fc0  76697264 00007265 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.317579@3] 1fe0  00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff
[  242.317713@3] 2000  00000000 00000003 00000000 00000000 ef572680 c0946c24 00000003 00000015
[  242.317852@3] 2020  edec8400 00000000 c093ea98 ef572680 c14baec0 eead2000 ef2bc780 eead3f1c
[  242.317984@3] 2040  eead3ed0 c06368b0 00000000 00000000 00000000 00000000 00000000 00000000
[  242.318112@3] 2060  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.318174@3] CPU: 3 PID: 91 Comm: irq/63-dwc_otg_ Not tainted 3.10.67-rt71 #1
[  242.318351@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[  242.318478@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c00133e0>] (handle_IPI+0x170/0x208)
[  242.318586@3] [<c00133e0>] (handle_IPI+0x170/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[  242.318695@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[  242.318734@3] Exception stack(0xeead3e88 to 0xeead3ed0)
[  242.318817@3] 3e80:                   00000000 eeb33ae0 00000001 ef572680 ef572680 00000000
[  242.318908@3] 3ea0: c093ea98 eead2000 eeb33ae0 eead2000 00000001 eead3f1c 00000000 eead3ed0
[  242.318966@3] 3ec0: c005277c c0636418 60070113 ffffffff
[  242.319091@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c0636418>] (__schedule+0x44/0x5c0)
[  242.319204@3] [<c0636418>] (__schedule+0x44/0x5c0) from [<c0636a94>] (schedule+0x7c/0x94)
[  242.319335@3] [<c0636a94>] (schedule+0x7c/0x94) from [<c008c55c>] (irq_thread+0x100/0x14c)
[  242.319477@3] [<c008c55c>] (irq_thread+0x100/0x14c) from [<c0047df0>] (kthread+0x98/0xa0)
[  242.319608@3] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c)
[  242.319640@1] IPI backtrace for cpu 1
[  242.319652@1]
[  242.319713@1] CPU: 1 PID: 58 Comm: khungtaskd Not tainted 3.10.67-rt71 #1
[  242.319759@1] task: ef2fd280 ti: ef46e000 task.ti: ef46e000
[  242.319816@1] PC is at smp_send_all_cpu_backtrace+0x98/0xc4
[  242.319884@1] LR is at gic_raise_softirq+0x38/0x60
[  242.319947@1] pc : [<c0013244>]    lr : [<c02098fc>]    psr: 20030013
[  242.319947@1] sp : ef46ff18  ip : 00000004  fp : 00007fd4
[  242.319983@1] r10: c093e7f8  r9 : 00000000  r8 : 000003d4
[  242.320026@1] r7 : c0952fd8  r6 : 00002710  r5 : 00000002  r4 : c09a2560
[  242.320066@1] r3 : 0000000b  r2 : 00000008  r1 : 00000004  r0 : 066665b0
[  242.320114@1] Flags: nzCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment kernel
[  242.320161@1] Control: 10c5387d  Table: 2978004a  DAC: 00000015
[  242.320186@1]
[  242.320186@1] PC: 0xc00131c4:
[  242.320340@1] 31c4  eb076269 e3500000 18bd80f8 e59f3084 e1a01004 e1a00005 e59f707c e3026710
[  242.320482@1] 31e4  e5933000 e5933000 e5a1303c eb07584e e1a01005 e59f0064 eb1861e0 eb187266
[  242.320622@1] 3204  e59f005c eb1861dd e5943034 e284003c e3a01006 e12fff33 e594303c e30605b0
[  242.320763@1] 3224  e3400666 e313000f 1a000004 e3a00000 e59f1030 eb07583c f57ff05f e8bd80f8
[  242.320903@1] 3244  e5973004 e12fff33 e2566001 1afffff1 eafffff5 c09a2560 c063c4ec c0952fd8
[  242.321043@1] 3264  c07e92f4 c07e9318 c09a2598 e92d41f0 e1a05000 e59f61d8 e1a07001 eb07a7f0
[  242.321183@1] 3284  e3550006 e1a03006 ee1d2f90 e1a04000 e7928003 e7827003 ca000005 e59f31b4
[  242.321323@1] 32a4  e0852200 e0833102 e5932004 e2822001 e5832004 e3550006 979ff105 ea00005c
[  242.321349@1]
[  242.321349@1] LR: 0xc020987c:
[  242.321490@1] 987c  e3a03001 e58c3000 ea000008 e59f2030 e28c3c01 e0820100 e5b21004 e5831000
[  242.321631@1] 989c  e1520000 e2833004 1afffffa eafffff3 e3a00001 e8bd0010 e12fff1e c093ebe0
[  242.321771@1] 98bc  c093ed6c c093ebec e92d41f0 e1a06000 e1a08001 e3a04000 e3e00000 e59f503c
[  242.321911@1] 98dc  e59f703c ea000001 e7d53000 e1844003 e2802001 e3a01004 e1a00006 ebff8387
[  242.322052@1] 98fc  e5973000 e1530000 cafffff6 f57ff04f e1884804 e5953008 e5834f00 e8bd81f0
[  242.322192@1] 991c  c093ebe0 c093e7e8 e92d41f0 e1a07001 e1a04000 ebfa13de e3500000 15906018
[  242.322333@1] 993c  01a06000 e1a00004 ebfa13d9 e3500000 15905010 01a05000 e5953024 e3530000
[  242.322474@1] 995c  1a00000a e595301c e3530000 1a000005 e5953018 e1a00007 e12fff33 e5953014
[  242.322499@1]
[  242.322499@1] SP: 0xef46fe98:
[  242.322638@1] fe98  00000001 c00133e0 fe00110c c093ebe0 ef46fed0 fe001100 000003d4 c0008474
[  242.322778@1] feb8  c02098fc c0013244 20030013 ffffffff ef46ff04 c000dd40 066665b0 00000004
[  242.322913@1] fed8  00000008 0000000b c09a2560 00000002 00002710 c0952fd8 000003d4 00000000
[  242.323052@1] fef8  c093e7f8 00007fd4 00000004 ef46ff18 c02098fc c0013244 20030013 ffffffff
[  242.323191@1] ff18  00000001 ef2f8580 ef2f8aa8 00007fd4 ef2f8580 c008b274 ef2f8aa8 00000078
[  242.323327@1] ff38  00000000 ef2f8778 c008b064 ef279ef0 00000000 00000000 c008b064 00000000
[  242.323459@1] ff58  00000000 00000000 00000000 c0047df0 ffffffff 00000000 00000000 00000000
[  242.323594@1] ff78  00000000 00000000 ef46ff80 ef46ff80 00000000 00000000 ef46ff90 ef46ff90
[  242.323622@1]
[  242.323622@1] R4: 0xc09a24e0:
[  242.323753@1] 24e0  20070030 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.323885@1] 2500  410fc051 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.324014@1] 2520  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.324148@1] 2540  000006a0 00000000 00000000 00000000 00140014 ef7fe000 00000000 00000000
[  242.324289@1] 2560  00002710 00124f80 c0944380 c08f4b10 c08f4ad8 c06291f4 c0629228 c001f4bc
[  242.324426@1] 2580  c001f52c c001f584 00000000 00204000 00000000 c02098c4 00000001 00000003
[  242.324564@1] 25a0  00000000 000c000a c08e39c8 00180018 06000000 00000000 0d9fffff 00000000
[  242.324697@1] 25c0  00000000 00000000 00000000 00000000 ef203980 ffffffff 00000000 00000002
[  242.324723@1]
[  242.324723@1] R7: 0xc0952f58:
[  242.324865@1] 2f58  c080a37c 000001a4 c01e5f74 c01e5d88 c080a388 000001a4 c01e5f54 c01e5d48
[  242.325005@1] 2f78  c080a394 000001a4 c01e5f48 c01e5d1c c080a3a4 000001a4 c01e5f28 c01e5cec
[  242.325145@1] 2f98  c080a3b0 000001a4 c01e5f08 c01e5cbc c0876f84 000001a4 c01e5efc c01e5c90
[  242.325280@1] 2fb8  c080a3bc 000001a4 c01e5edc c01e5c50 00000000 00000000 00000000 00000000
[  242.325421@1] 2fd8  c01ea510 c01ea56c c01ea588 00002710 bf0ac9d0 bf0010ec c01ee340 c06559ec
[  242.325554@1] 2ff8  00000000 00000000 00000000 c01ee338 c06559ec 00000000 00000000 00000000
[  242.325691@1] 3018  c0998740 c0999678 c01ef1bc c01ef128 00000000 00000000 00000000 18a518a5
[  242.325829@1] 3038  c0953038 c0953038 00000000 00000000 ef344900 ef344900 00000001 c082c224
[  242.325856@1]
[  242.325856@1] R10: 0xc093e778:
[  242.325985@1] e778  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.326114@1] e798  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.326242@1] e7b8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.326372@1] e7d8  00000000 00000000 00000000 00000001 00000004 00000001 00000000 ef419380
[  242.326504@1] e7f8  00000000 00000078 00008000 00000009 00000001 00000000 00000001 0000000f
[  242.326635@1] e818  ee6b2800 00000000 0000000a 00000000 00000000 00000000 00000001 00000001
[  242.326769@1] e838  00000001 00000001 00000001 00000000 00000d23 00000d23 00035021 01ffffff
[  242.326898@1] e858  00000000 00000001 00000001 00000001 00000001 00000001 00000000 00000000
[  242.326958@1] CPU: 1 PID: 58 Comm: khungtaskd Not tainted 3.10.67-rt71 #1
[  242.327106@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[  242.327231@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c00133e0>] (handle_IPI+0x170/0x208)
[  242.327334@1] [<c00133e0>] (handle_IPI+0x170/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[  242.327442@1] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[  242.327479@1] Exception stack(0xef46fed0 to 0xef46ff18)
[  242.327546@1] fec0:                                     066665b0 00000004 00000008 0000000b
[  242.327638@1] fee0: c09a2560 00000002 00002710 c0952fd8 000003d4 00000000 c093e7f8 00007fd4
[  242.327712@1] ff00: 00000004 ef46ff18 c02098fc c0013244 20030013 ffffffff
[  242.327832@1] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c0013244>] (smp_send_all_cpu_backtrace+0x98/0xc4)
[  242.327955@1] [<c0013244>] (smp_send_all_cpu_backtrace+0x98/0xc4) from [<c008b274>] (watchdog+0x210/0x280)
[  242.328085@1] [<c008b274>] (watchdog+0x210/0x280) from [<c0047df0>] (kthread+0x98/0xa0)
[  242.328213@1] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c)
[  242.328276@0] IPI backtrace for cpu 0
[  242.328294@0]
[  242.328369@0] CPU: 0 PID: 512 Comm: in:imklog Not tainted 3.10.67-rt71 #1
[  242.328422@0] task: eed96e00 ti: ee1f4000 task.ti: ee1f4000
[  242.328505@0] PC is at ret_fast_syscall+0x0/0x30
[  242.328554@0] LR is at 0x182d9
[  242.328618@0] pc : [<c000e180>]    lr : [<000182d9>]    psr: 600e0013
[  242.328618@0] sp : ee1f5fa8  ip : 00000018  fp : 00078da9
[  242.328654@0] r10: 00000000  r9 : ee1f4000  r8 : c000e328
[  242.328697@0] r7 : 0000004e  r6 : b5208788  r5 : b5208784  r4 : b52086f0
[  242.328737@0] r3 : 00000000  r2 : 00000000  r1 : ee1f5f98  r0 : 00000000
[  242.328785@0] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[  242.328833@0] Control: 10c5387d  Table: 2e0ac04a  DAC: 00000015
[  242.328859@0]
[  242.328859@0] PC: 0xc000e100:
[  242.329017@0] e100  e51f0288 e5900000 ee010f10 e1a0200d eb002bb2 e1a096ad e1a09689 e3a08000
[  242.329163@0] e120  ea00002b e281c020 e5923064 e8ac6ff0 ee0d3f70 e3a04000 ee0d4f50 e1a05000
[  242.329307@0] e140  e2824020 e59f000c e3a01002 eb00fe6e e1a00005 e894aff0 c0943018 e320f000
[  242.329451@0] e160  e320f000 e320f000 e320f000 e320f000 e320f000 e320f000 e320f000 e320f000
[  242.329595@0] e180  f10c0080 e5991000 e3110007 1a000007 e59d1048 e5bde044 e16ff001 f57ff01f
[  242.329739@0] e1a0  e95d7ffe e1a00000 e28dd00c e1b0f00e e5ad0008 e1a0000d e1a02008 eb000d00
[  242.329884@0] e1c0  e3500000 0a000006 b3a07000 e89d007f ea00002f f10c0080 e5991000 e3110007
[  242.330028@0] e1e0  1afffff3 e59d1040 e5bde03c e16ff001 f57ff01f e95d7fff e1a00000 e28dd00c
[  242.330054@0]
[  242.330054@0] SP: 0xee1f5f28:
[  242.330198@0] 5f28  00000000 c00133e0 fe00110c c093ebe0 ee1f5f60 fe001100 c000e328 c0008474
[  242.330342@0] 5f48  000182d9 c000e180 600e0013 ffffffff ee1f5f94 c000dd40 00000000 ee1f5f98
[  242.330484@0] 5f68  00000000 00000000 b52086f0 b5208784 b5208788 0000004e c000e328 ee1f4000
[  242.330627@0] 5f88  00000000 00078da9 00000018 ee1f5fa8 000182d9 c000e180 600e0013 ffffffff
[  242.330770@0] 5fa8  b52086f0 b5208784 b6575460 00000000 ffffffff 00045f51 b52086f0 b5208784
[  242.330914@0] 5fc8  b5208788 0000004e b6577357 b6575d18 00001fa0 00078da9 00065170 b657545c
[  242.331054@0] 5fe8  00045f61 b6ea9a96 600e0030 b6575460 00000000 00000000 00000000 00000002
[  242.331195@0] 6008  00000000 bf000000 eed94200 c0946c24 00000002 00000015 edfd4600 00000000
[  242.331222@0]
[  242.331222@0] R1: 0xee1f5f18:
[  242.331365@0] 5f18  ee1f4000 00000000 00078da9 c000f950 00000000 c00133e0 fe00110c c093ebe0
[  242.331510@0] 5f38  ee1f5f60 fe001100 c000e328 c0008474 000182d9 c000e180 600e0013 ffffffff
[  242.331651@0] 5f58  ee1f5f94 c000dd40 00000000 ee1f5f98 00000000 00000000 b52086f0 b5208784
[  242.331793@0] 5f78  b5208788 0000004e c000e328 ee1f4000 00000000 00078da9 00000018 ee1f5fa8
[  242.331936@0] 5f98  000182d9 c000e180 600e0013 ffffffff b52086f0 b5208784 b6575460 00000000
[  242.332080@0] 5fb8  ffffffff 00045f51 b52086f0 b5208784 b5208788 0000004e b6577357 b6575d18
[  242.332224@0] 5fd8  00001fa0 00078da9 00065170 b657545c 00045f61 b6ea9a96 600e0030 b6575460
[  242.332362@0] 5ff8  00000000 00000000 00000000 00000002 00000000 bf000000 eed94200 c0946c24
[  242.332391@0]
[  242.332391@0] R8: 0xc000e2a8:
[  242.332535@0] e2a8  e24fee13 3798f107 e28d1008 e3a08000 e357080f e2270000 2a000fb6 ea00e953
[  242.332680@0] e2c8  e24ee004 e58de03c eaffffaa e1a01007 e28d0008 eb0008bb e28fe024 e1a07000
[  242.332824@0] e2e8  e28d1008 e3570f5f 3891007f 388d0030 3798f107 e3770001 1affffeb e28dd008
[  242.332968@0] e308  eaffffb1 e5ad0008 e1a0000d eb0008da eaffffad e320f000 c0942fac c000e9b0
[  242.333112@0] e328  c0039fac c002f400 c0028c7c c00d42e4 c00d4350 c00d32e0 c00d3308 c0048818
[  242.333256@0] e348  c00d32fc c00e1e94 c00e1bb4 c00da71c c00d2c20 c0048818 c00e1aa4 c00d2ea4
[  242.333400@0] e368  c0078580 c0048818 c0048818 c00d3904 c003cd54 c00f13d0 c0048818 c0078614
[  242.333545@0] e388  c0078a7c c0048818 c0035314 c0048818 c0048818 c003b314 c0048818 c0048818
[  242.333570@0]
[  242.333570@0] R9: 0xee1f3f80:
[  242.333714@0] 3f80  2f9fb841 c1111452 c1311452 c1411452 d0011452 d0111452 c8111452 00000000
[  242.333848@0] 3fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.333984@0] 3fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
[  242.334120@0] 3fe0  00000000 00000000 00000000 00000000 00000000 00000000 2f9fd821 2f9fdc21
[  242.334258@0] 4000  00000008 00000002 00000000 00000000 eed96e00 c0946c24 00000000 00000015
[  242.334404@0] 4020  edfd4600 00000000 c093ea98 eed96e00 c14a2ec0 ee1f4000 ef270b00 ee1f5f84
[  242.334542@0] 4040  ee1f5f38 c06368b0 00000000 00000000 00000000 00000000 00000000 01010000
[  242.334676@0] 4060  00000000 b65968f0 00000000 00000000 00000000 00000000 00000000 00000000
[  242.334736@0] CPU: 0 PID: 512 Comm: in:imklog Not tainted 3.10.67-rt71 #1
[  242.334910@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[  242.335037@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c00133e0>] (handle_IPI+0x170/0x208)
[  242.335141@0] [<c00133e0>] (handle_IPI+0x170/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[  242.335248@0] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[  242.335285@0] Exception stack(0xee1f5f60 to 0xee1f5fa8)
[  242.335382@0] 5f60: 00000000 ee1f5f98 00000000 00000000 b52086f0 b5208784 b5208788 0000004e
[  242.335475@0] 5f80: c000e328 ee1f4000 00000000 00078da9 00000018 ee1f5fa8 000182d9 c000e180
[  242.335517@0] 5fa0: 600e0013 ffffffff
[  242.335637@0] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000e180>] (ret_fast_syscall+0x0/0x30)
[  242.335978@1] Kernel panic - not syncing: hung_task: blocked tasks
[  242.336058@1] CPU: 1 PID: 58 Comm: khungtaskd Not tainted 3.10.67-rt71 #1
[  242.336238@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[  242.336399@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c062b824>] (panic+0x98/0x1f8)
[  242.336533@1] [<c062b824>] (panic+0x98/0x1f8) from [<c008b27c>] (watchdog+0x218/0x280)
[  242.336670@1] [<c008b27c>] (watchdog+0x218/0x280) from [<c0047df0>] (kthread+0x98/0xa0)
[  242.336797@1] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c)
[  242.336868@3] CPU3: stopping
[  242.336945@3] CPU: 3 PID: 34 Comm: ksoftirqd/3 Not tainted 3.10.67-rt71 #1
[  242.337122@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[  242.337245@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[  242.337349@3] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[  242.337454@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84)
[  242.337492@3] Exception stack(0xef2e5d58 to 0xef2e5da0)
[  242.337547@3] 5d40:                                                       40000001 ef572680
[  242.337640@3] 5d60: 00000003 edec8434 edec8400 ef572680 00000001 ef2bc780 c14baec0 ef2e4000
[  242.337732@3] 5d80: ef572c00 ef2e5db4 00000000 ef2e5da0 c0050df8 c0050e0c a0070113 ffffffff
[  242.337853@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c0050e0c>] (finish_task_switch+0x3c/0xb0)
[  242.337976@3] [<c0050e0c>] (finish_task_switch+0x3c/0xb0) from [<c06368cc>] (__schedule+0x4f8/0x5c0)
[  242.338096@3] [<c06368cc>] (__schedule+0x4f8/0x5c0) from [<c0637044>] (preempt_schedule_irq+0x40/0x68)
[  242.338215@3] [<c0637044>] (preempt_schedule_irq+0x40/0x68) from [<c000dd8c>] (svc_preempt+0x8/0x20)
[  242.338329@3] [<c000dd8c>] (svc_preempt+0x8/0x20) from [<c0036a28>] (add_timer+0x0/0x18)
[  242.338467@3] [<c0036a28>] (add_timer+0x0/0x18) from [<c05628a4>] (addrconf_verify+0x36c/0x394)
[  242.338592@3] [<c05628a4>] (addrconf_verify+0x36c/0x394) from [<c00363c8>] (call_timer_fn.isra.21+0x24/0x80)
[  242.338701@3] [<c00363c8>] (call_timer_fn.isra.21+0x24/0x80) from [<c0036654>] (run_timer_softirq+0x230/0x278)
[  242.338834@3] [<c0036654>] (run_timer_softirq+0x230/0x278) from [<c0030b8c>] (do_current_softirqs+0x128/0x224)
[  242.338962@3] [<c0030b8c>] (do_current_softirqs+0x128/0x224) from [<c0030cb8>] (run_ksoftirqd+0x30/0x50)
[  242.339075@3] [<c0030cb8>] (run_ksoftirqd+0x30/0x50) from [<c004fde8>] (smpboot_thread_fn+0x254/0x26c)
[  242.339200@3] [<c004fde8>] (smpboot_thread_fn+0x254/0x26c) from [<c0047df0>] (kthread+0x98/0xa0)
[  242.339327@3] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c)
[  242.339386@2] CPU2: stopping
[  242.339462@2] CPU: 2 PID: 513 Comm: rs:main Q:Reg Not tainted 3.10.67-rt71 #1
[  242.339650@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[  242.339775@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[  242.339879@2] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[  242.339985@2] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000df1c>] (__irq_usr+0x3c/0x60)
[  242.340023@2] Exception stack(0xee1f7fb0 to 0xee1f7ff8)
[  242.340092@2] 7fa0:                                     00079cc0 00077050 00000001 0007a338
[  242.340185@2] 7fc0: 00077050 b54021a8 00079cc0 00079cc0 b5d95770 b5d9576c 00000000 00077050
[  242.340260@2] 7fe0: 0006502c b5d95710 0003f9e7 0003e7fe 60040030 ffffffff
[  242.340320@0] CPU0: stopping
[  242.340391@0] CPU: 0 PID: 512 Comm: in:imklog Not tainted 3.10.67-rt71 #1
[  242.340566@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14)
[  242.340690@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208)
[  242.340793@0] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c)
[  242.340899@0] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000df1c>] (__irq_usr+0x3c/0x60)
[  242.340936@0] Exception stack(0xee1f5fb0 to 0xee1f5ff8)
[  242.341006@0] 5fa0:                                     0000004d b6577478 3165641f 63666261
[  242.341097@0] 5fc0: 00000000 ff000000 00000004 00000000 b657741b b6575d18 00001fa0 00078da9
[  242.341172@0] 5fe0: ffffffff b6575478 0002b9c5 b6e9b802 600f0030 ffffffff

[adbensi]

Now everytime when I try to save the changes of the powertop my system freezes ...
The problem may be wrong interrupts... if You disconnect the video and reconnect it fail
CLx, please, when possible attach the output of the command sysctl -a and cat /proc/interrupts
thanks a lot

[Clx]

Now everytime when I try to save the changes of the powertop my system freezes ...
The problem may be wrong interrupts... if You disconnect the video and reconnect it fail
CLx, please, when possible attach the output of the command sysctl -a and cat /proc/interrupts
thanks a lot

Here you go.

sysctl -a

Code: Select all

dev.cdrom.autoclose = 1
dev.cdrom.autoeject = 0
dev.cdrom.check_media = 0
dev.cdrom.debug = 0
dev.cdrom.info = CD-ROM information, Id: cdrom.c 3.20 2003/12/17
dev.cdrom.info = 
dev.cdrom.info = drive name:	
dev.cdrom.info = drive speed:	
dev.cdrom.info = drive # of slots:
dev.cdrom.info = Can close tray:	
dev.cdrom.info = Can open tray:	
dev.cdrom.info = Can lock tray:	
dev.cdrom.info = Can change speed:
dev.cdrom.info = Can select disk:
dev.cdrom.info = Can read multisession:
dev.cdrom.info = Can read MCN:	
dev.cdrom.info = Reports media changed:
dev.cdrom.info = Can play audio:	
dev.cdrom.info = Can write CD-R:	
dev.cdrom.info = Can write CD-RW:
dev.cdrom.info = Can read DVD:	
dev.cdrom.info = Can write DVD-R:
dev.cdrom.info = Can write DVD-RAM:
dev.cdrom.info = Can read MRW:	
dev.cdrom.info = Can write MRW:	
dev.cdrom.info = Can write RAM:	
dev.cdrom.info = 
dev.cdrom.info = 
dev.cdrom.lock = 1
dev.raid.speed_limit_max = 200000
dev.raid.speed_limit_min = 1000
dev.scsi.logging_level = 0
fs.aio-max-nr = 65536
fs.aio-nr = 0
fs.binfmt_misc.status = enabled
fs.dentry-state = 9870	5920	45	0	0	0
fs.dir-notify-enable = 1
fs.epoll.max_user_watches = 260625
fs.file-max = 99801
fs.file-nr = 864	0	99801
fs.inode-nr = 9326	205
fs.inode-state = 9326	205	0	0	0	0	0
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 128
fs.inotify.max_user_watches = 8192
fs.lease-break-time = 45
fs.leases-enable = 1
fs.nr_open = 1048576
fs.overflowgid = 65534
fs.overflowuid = 65534
fs.pipe-max-size = 1048576
fs.protected_hardlinks = 1
fs.protected_symlinks = 1
fs.quota.allocated_dquots = 0
fs.quota.cache_hits = 0
fs.quota.drops = 0
fs.quota.free_dquots = 0
fs.quota.lookups = 0
fs.quota.reads = 0
fs.quota.syncs = 0
fs.quota.warnings = 1
fs.quota.writes = 0
fs.suid_dumpable = 0
kernel.acct = 4	2	30
kernel.arc_firm_log = 0
kernel.auto_msgmni = 1
kernel.blk_iopoll = 1
kernel.cad_pid = 1
kernel.cap_last_cap = 36
kernel.core_pattern = core
kernel.core_pipe_limit = 0
kernel.core_uses_pid = 0
kernel.ctrl-alt-del = 0
kernel.dmesg_restrict = 0
kernel.domainname = (none)
kernel.hostname = odroid
kernel.hotplug = 
kernel.hung_task_check_count = 32768
kernel.hung_task_panic = 1
kernel.hung_task_timeout_secs = 120
kernel.hung_task_warnings = 10
kernel.keys.gc_delay = 300
kernel.keys.maxbytes = 20000
kernel.keys.maxkeys = 200
kernel.keys.root_maxbytes = 20000
kernel.keys.root_maxkeys = 200
kernel.kptr_restrict = 1
kernel.max_lock_depth = 1024
kernel.modprobe = /sbin/modprobe
kernel.modules_disabled = 0
kernel.msgmax = 8192
kernel.msgmnb = 16384
kernel.msgmni = 1445
kernel.ngroups_max = 65536
kernel.nmi_watchdog = 1
kernel.osrelease = 3.10.67-rt71
kernel.ostype = Linux
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.panic = 0
kernel.panic_on_oops = 0
kernel.pid_max = 32768
kernel.poweroff_cmd = /sbin/poweroff
kernel.print-fatal-signals = 0
kernel.printk = 4	4	1	7
kernel.printk_delay = 0
kernel.printk_ratelimit = 5
kernel.printk_ratelimit_burst = 10
kernel.pty.max = 4096
kernel.pty.nr = 5
kernel.pty.reserve = 1024
kernel.random.boot_id = 54195731-75b4-4928-9b46-ac1c44b29d62
kernel.random.entropy_avail = 3450
kernel.random.poolsize = 4096
kernel.random.read_wakeup_threshold = 64
kernel.random.uuid = fc0a1a47-f54d-4195-95f6-56a0284faf17
kernel.random.write_wakeup_threshold = 3072
kernel.randomize_va_space = 1
kernel.real-root-dev = 0
kernel.sched_child_runs_first = 0
kernel.sched_domain.cpu0.domain0.busy_factor = 64
kernel.sched_domain.cpu0.domain0.busy_idx = 2
kernel.sched_domain.cpu0.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu0.domain0.flags = 4143
kernel.sched_domain.cpu0.domain0.forkexec_idx = 0
kernel.sched_domain.cpu0.domain0.idle_idx = 1
kernel.sched_domain.cpu0.domain0.imbalance_pct = 125
kernel.sched_domain.cpu0.domain0.max_interval = 4
kernel.sched_domain.cpu0.domain0.min_interval = 1
kernel.sched_domain.cpu0.domain0.name = CPU
kernel.sched_domain.cpu0.domain0.newidle_idx = 0
kernel.sched_domain.cpu0.domain0.wake_idx = 0
kernel.sched_domain.cpu1.domain0.busy_factor = 64
kernel.sched_domain.cpu1.domain0.busy_idx = 2
kernel.sched_domain.cpu1.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu1.domain0.flags = 4143
kernel.sched_domain.cpu1.domain0.forkexec_idx = 0
kernel.sched_domain.cpu1.domain0.idle_idx = 1
kernel.sched_domain.cpu1.domain0.imbalance_pct = 125
kernel.sched_domain.cpu1.domain0.max_interval = 4
kernel.sched_domain.cpu1.domain0.min_interval = 1
kernel.sched_domain.cpu1.domain0.name = CPU
kernel.sched_domain.cpu1.domain0.newidle_idx = 0
kernel.sched_domain.cpu1.domain0.wake_idx = 0
kernel.sched_domain.cpu2.domain0.busy_factor = 64
kernel.sched_domain.cpu2.domain0.busy_idx = 2
kernel.sched_domain.cpu2.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu2.domain0.flags = 4143
kernel.sched_domain.cpu2.domain0.forkexec_idx = 0
kernel.sched_domain.cpu2.domain0.idle_idx = 1
kernel.sched_domain.cpu2.domain0.imbalance_pct = 125
kernel.sched_domain.cpu2.domain0.max_interval = 4
kernel.sched_domain.cpu2.domain0.min_interval = 1
kernel.sched_domain.cpu2.domain0.name = CPU
kernel.sched_domain.cpu2.domain0.newidle_idx = 0
kernel.sched_domain.cpu2.domain0.wake_idx = 0
kernel.sched_domain.cpu3.domain0.busy_factor = 64
kernel.sched_domain.cpu3.domain0.busy_idx = 2
kernel.sched_domain.cpu3.domain0.cache_nice_tries = 1
kernel.sched_domain.cpu3.domain0.flags = 4143
kernel.sched_domain.cpu3.domain0.forkexec_idx = 0
kernel.sched_domain.cpu3.domain0.idle_idx = 1
kernel.sched_domain.cpu3.domain0.imbalance_pct = 125
kernel.sched_domain.cpu3.domain0.max_interval = 4
kernel.sched_domain.cpu3.domain0.min_interval = 1
kernel.sched_domain.cpu3.domain0.name = CPU
kernel.sched_domain.cpu3.domain0.newidle_idx = 0
kernel.sched_domain.cpu3.domain0.wake_idx = 0
kernel.sched_latency_ns = 18000000
kernel.sched_migration_cost_ns = 500000
kernel.sched_min_granularity_ns = 2250000
kernel.sched_nr_migrate = 8
kernel.sched_rr_timeslice_ms = 10
kernel.sched_rt_period_us = 1000000
kernel.sched_rt_runtime_us = 950000
kernel.sched_shares_window_ns = 10000000
kernel.sched_time_avg_ms = 1000
kernel.sched_tunable_scaling = 1
kernel.sched_wakeup_granularity_ns = 3000000
kernel.sem = 250	32000	32	128
kernel.shm_rmid_forced = 0
kernel.shmall = 2097152
kernel.shmmax = 33554432
kernel.shmmni = 4096
kernel.softlockup_panic = 1
kernel.sysrq = 176
kernel.tainted = 0
kernel.threads-max = 15597
kernel.timer_migration = 1
kernel.usermodehelper.bset = 4294967295	31
kernel.usermodehelper.inheritable = 4294967295	31
kernel.version = #1 SMP PREEMPT RT Mon Feb 16 19:40:54 MSK 2015
kernel.watchdog = 1
kernel.watchdog_thresh = 10
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0
net.core.dev_weight = 64
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_budget = 300
net.core.netdev_max_backlog = 1000
net.core.netdev_tstamp_prequeue = 1
net.core.optmem_max = 65535
net.core.rmem_default = 514400
net.core.rmem_max = 26214400
net.core.rps_sock_flow_entries = 32768
net.core.somaxconn = 128
net.core.warnings = 1
net.core.wmem_default = 514400
net.core.wmem_max = 26214400
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_etime = 10
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_larval_drop = 1
net.ipv4.conf.all.accept_local = 0
net.ipv4.conf.all.accept_redirects = 1
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_notify = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.proxy_arp_pvlan = 0
net.ipv4.conf.all.route_localnet = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.src_valid_mark = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.default.accept_local = 0
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.accept_source_route = 1
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_notify = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.proxy_arp_pvlan = 0
net.ipv4.conf.default.route_localnet = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.src_valid_mark = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.eth0.accept_local = 0
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.accept_source_route = 1
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_notify = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.forwarding = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.promote_secondaries = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.proxy_arp_pvlan = 0
net.ipv4.conf.eth0.route_localnet = 0
net.ipv4.conf.eth0.rp_filter = 1
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.src_valid_mark = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.ip6tnl0.accept_local = 0
net.ipv4.conf.ip6tnl0.accept_redirects = 1
net.ipv4.conf.ip6tnl0.accept_source_route = 1
net.ipv4.conf.ip6tnl0.arp_accept = 0
net.ipv4.conf.ip6tnl0.arp_announce = 0
net.ipv4.conf.ip6tnl0.arp_filter = 0
net.ipv4.conf.ip6tnl0.arp_ignore = 0
net.ipv4.conf.ip6tnl0.arp_notify = 0
net.ipv4.conf.ip6tnl0.bootp_relay = 0
net.ipv4.conf.ip6tnl0.disable_policy = 0
net.ipv4.conf.ip6tnl0.disable_xfrm = 0
net.ipv4.conf.ip6tnl0.force_igmp_version = 0
net.ipv4.conf.ip6tnl0.forwarding = 0
net.ipv4.conf.ip6tnl0.log_martians = 0
net.ipv4.conf.ip6tnl0.mc_forwarding = 0
net.ipv4.conf.ip6tnl0.medium_id = 0
net.ipv4.conf.ip6tnl0.promote_secondaries = 0
net.ipv4.conf.ip6tnl0.proxy_arp = 0
net.ipv4.conf.ip6tnl0.proxy_arp_pvlan = 0
net.ipv4.conf.ip6tnl0.route_localnet = 0
net.ipv4.conf.ip6tnl0.rp_filter = 1
net.ipv4.conf.ip6tnl0.secure_redirects = 1
net.ipv4.conf.ip6tnl0.send_redirects = 1
net.ipv4.conf.ip6tnl0.shared_media = 1
net.ipv4.conf.ip6tnl0.src_valid_mark = 0
net.ipv4.conf.ip6tnl0.tag = 0
net.ipv4.conf.lo.accept_local = 0
net.ipv4.conf.lo.accept_redirects = 1
net.ipv4.conf.lo.accept_source_route = 1
net.ipv4.conf.lo.arp_accept = 0
net.ipv4.conf.lo.arp_announce = 0
net.ipv4.conf.lo.arp_filter = 0
net.ipv4.conf.lo.arp_ignore = 0
net.ipv4.conf.lo.arp_notify = 0
net.ipv4.conf.lo.bootp_relay = 0
net.ipv4.conf.lo.disable_policy = 1
net.ipv4.conf.lo.disable_xfrm = 1
net.ipv4.conf.lo.force_igmp_version = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.lo.log_martians = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.lo.medium_id = 0
net.ipv4.conf.lo.promote_secondaries = 0
net.ipv4.conf.lo.proxy_arp = 0
net.ipv4.conf.lo.proxy_arp_pvlan = 0
net.ipv4.conf.lo.route_localnet = 0
net.ipv4.conf.lo.rp_filter = 1
net.ipv4.conf.lo.secure_redirects = 1
net.ipv4.conf.lo.send_redirects = 1
net.ipv4.conf.lo.shared_media = 1
net.ipv4.conf.lo.src_valid_mark = 0
net.ipv4.conf.lo.tag = 0
net.ipv4.conf.sit0.accept_local = 0
net.ipv4.conf.sit0.accept_redirects = 1
net.ipv4.conf.sit0.accept_source_route = 1
net.ipv4.conf.sit0.arp_accept = 0
net.ipv4.conf.sit0.arp_announce = 0
net.ipv4.conf.sit0.arp_filter = 0
net.ipv4.conf.sit0.arp_ignore = 0
net.ipv4.conf.sit0.arp_notify = 0
net.ipv4.conf.sit0.bootp_relay = 0
net.ipv4.conf.sit0.disable_policy = 0
net.ipv4.conf.sit0.disable_xfrm = 0
net.ipv4.conf.sit0.force_igmp_version = 0
net.ipv4.conf.sit0.forwarding = 0
net.ipv4.conf.sit0.log_martians = 0
net.ipv4.conf.sit0.mc_forwarding = 0
net.ipv4.conf.sit0.medium_id = 0
net.ipv4.conf.sit0.promote_secondaries = 0
net.ipv4.conf.sit0.proxy_arp = 0
net.ipv4.conf.sit0.proxy_arp_pvlan = 0
net.ipv4.conf.sit0.route_localnet = 0
net.ipv4.conf.sit0.rp_filter = 1
net.ipv4.conf.sit0.secure_redirects = 1
net.ipv4.conf.sit0.send_redirects = 1
net.ipv4.conf.sit0.shared_media = 1
net.ipv4.conf.sit0.src_valid_mark = 0
net.ipv4.conf.sit0.tag = 0
net.ipv4.conf.wlan0.accept_local = 0
net.ipv4.conf.wlan0.accept_redirects = 1
net.ipv4.conf.wlan0.accept_source_route = 1
net.ipv4.conf.wlan0.arp_accept = 0
net.ipv4.conf.wlan0.arp_announce = 0
net.ipv4.conf.wlan0.arp_filter = 0
net.ipv4.conf.wlan0.arp_ignore = 0
net.ipv4.conf.wlan0.arp_notify = 0
net.ipv4.conf.wlan0.bootp_relay = 0
net.ipv4.conf.wlan0.disable_policy = 0
net.ipv4.conf.wlan0.disable_xfrm = 0
net.ipv4.conf.wlan0.force_igmp_version = 0
net.ipv4.conf.wlan0.forwarding = 0
net.ipv4.conf.wlan0.log_martians = 0
net.ipv4.conf.wlan0.mc_forwarding = 0
net.ipv4.conf.wlan0.medium_id = 0
net.ipv4.conf.wlan0.promote_secondaries = 0
net.ipv4.conf.wlan0.proxy_arp = 0
net.ipv4.conf.wlan0.proxy_arp_pvlan = 0
net.ipv4.conf.wlan0.route_localnet = 0
net.ipv4.conf.wlan0.rp_filter = 1
net.ipv4.conf.wlan0.secure_redirects = 1
net.ipv4.conf.wlan0.send_redirects = 1
net.ipv4.conf.wlan0.shared_media = 1
net.ipv4.conf.wlan0.src_valid_mark = 0
net.ipv4.conf.wlan0.tag = 0
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_echo_sysrq = 0
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_ratelimit = 1000
net.ipv4.icmp_ratemask = 6168
net.ipv4.igmp_max_memberships = 20
net.ipv4.igmp_max_msf = 10
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_dynaddr = 0
net.ipv4.ip_early_demux = 1
net.ipv4.ip_forward = 0
net.ipv4.ip_local_port_range = 32768	61000
net.ipv4.ip_local_reserved_ports = 
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ipfrag_high_thresh = 4194304
net.ipv4.ipfrag_low_thresh = 3145728
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 600
net.ipv4.ipfrag_time = 30
net.ipv4.neigh.default.anycast_delay = 100
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.locktime = 100
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.neigh.default.proxy_delay = 80
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.unres_qlen = 34
net.ipv4.neigh.default.unres_qlen_bytes = 65536
net.ipv4.neigh.eth0.anycast_delay = 100
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.locktime = 100
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.eth0.proxy_delay = 80
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.unres_qlen = 34
net.ipv4.neigh.eth0.unres_qlen_bytes = 65536
net.ipv4.neigh.ip6tnl0.anycast_delay = 100
net.ipv4.neigh.ip6tnl0.app_solicit = 0
net.ipv4.neigh.ip6tnl0.base_reachable_time_ms = 30000
net.ipv4.neigh.ip6tnl0.delay_first_probe_time = 5
net.ipv4.neigh.ip6tnl0.gc_stale_time = 60
net.ipv4.neigh.ip6tnl0.locktime = 100
net.ipv4.neigh.ip6tnl0.mcast_solicit = 3
net.ipv4.neigh.ip6tnl0.proxy_delay = 80
net.ipv4.neigh.ip6tnl0.proxy_qlen = 64
net.ipv4.neigh.ip6tnl0.retrans_time_ms = 1000
net.ipv4.neigh.ip6tnl0.ucast_solicit = 3
net.ipv4.neigh.ip6tnl0.unres_qlen = 34
net.ipv4.neigh.ip6tnl0.unres_qlen_bytes = 65536
net.ipv4.neigh.lo.anycast_delay = 100
net.ipv4.neigh.lo.app_solicit = 0
net.ipv4.neigh.lo.base_reachable_time_ms = 30000
net.ipv4.neigh.lo.delay_first_probe_time = 5
net.ipv4.neigh.lo.gc_stale_time = 60
net.ipv4.neigh.lo.locktime = 100
net.ipv4.neigh.lo.mcast_solicit = 3
net.ipv4.neigh.lo.proxy_delay = 80
net.ipv4.neigh.lo.proxy_qlen = 64
net.ipv4.neigh.lo.retrans_time_ms = 1000
net.ipv4.neigh.lo.ucast_solicit = 3
net.ipv4.neigh.lo.unres_qlen = 34
net.ipv4.neigh.lo.unres_qlen_bytes = 65536
net.ipv4.neigh.sit0.anycast_delay = 100
net.ipv4.neigh.sit0.app_solicit = 0
net.ipv4.neigh.sit0.base_reachable_time_ms = 30000
net.ipv4.neigh.sit0.delay_first_probe_time = 5
net.ipv4.neigh.sit0.gc_stale_time = 60
net.ipv4.neigh.sit0.locktime = 100
net.ipv4.neigh.sit0.mcast_solicit = 3
net.ipv4.neigh.sit0.proxy_delay = 80
net.ipv4.neigh.sit0.proxy_qlen = 64
net.ipv4.neigh.sit0.retrans_time_ms = 1000
net.ipv4.neigh.sit0.ucast_solicit = 3
net.ipv4.neigh.sit0.unres_qlen = 34
net.ipv4.neigh.sit0.unres_qlen_bytes = 65536
net.ipv4.neigh.wlan0.anycast_delay = 100
net.ipv4.neigh.wlan0.app_solicit = 0
net.ipv4.neigh.wlan0.base_reachable_time_ms = 30000
net.ipv4.neigh.wlan0.delay_first_probe_time = 5
net.ipv4.neigh.wlan0.gc_stale_time = 60
net.ipv4.neigh.wlan0.locktime = 100
net.ipv4.neigh.wlan0.mcast_solicit = 3
net.ipv4.neigh.wlan0.proxy_delay = 80
net.ipv4.neigh.wlan0.proxy_qlen = 64
net.ipv4.neigh.wlan0.retrans_time_ms = 1000
net.ipv4.neigh.wlan0.ucast_solicit = 3
net.ipv4.neigh.wlan0.unres_qlen = 34
net.ipv4.neigh.wlan0.unres_qlen_bytes = 65536
net.ipv4.ping_group_range = 1	0
net.ipv4.route.error_burst = 500
net.ipv4.route.error_cost = 100
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_thresh = -1
net.ipv4.route.gc_timeout = 300
net.ipv4.route.max_size = 2147483647
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.redirect_load = 2
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_silence = 2048
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_adv_win_scale = 1
net.ipv4.tcp_allowed_congestion_control = reno
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_available_congestion_control = reno
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_challenge_ack_limit = 100
net.ipv4.tcp_congestion_control = reno
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_early_retrans = 3
net.ipv4.tcp_ecn = 2
net.ipv4.tcp_fack = 1
net.ipv4.tcp_fastopen = 0
net.ipv4.tcp_fastopen_key = ebb31a82-5525d762-be1ae6bf-d1df5ee1
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_frto = 2
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_limit_output_bytes = 131072
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_max_orphans = 4096
net.ipv4.tcp_max_ssthresh = 0
net.ipv4.tcp_max_syn_backlog = 128
net.ipv4.tcp_max_tw_buckets = 4096
net.ipv4.tcp_mem = 17178	22905	34356
net.ipv4.tcp_min_tso_segs = 2
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.tcp_orphan_retries = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_rmem = 10240	87380	26214400
net.ipv4.tcp_sack = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_syn_retries = 6
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_thin_dupack = 0
net.ipv4.tcp_thin_linear_timeouts = 0
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_wmem = 10240	87380	26214400
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.udp_mem = 17298	23066	34596
net.ipv4.udp_rmem_min = 131072
net.ipv4.udp_wmem_min = 131072
net.ipv4.xfrm4_gc_thresh = 1024
net.ipv6.bindv6only = 0
net.ipv6.conf.all.accept_dad = 1
net.ipv6.conf.all.accept_ra = 1
net.ipv6.conf.all.accept_ra_defrtr = 1
net.ipv6.conf.all.accept_ra_pinfo = 1
net.ipv6.conf.all.accept_ra_rtr_pref = 1
net.ipv6.conf.all.accept_redirects = 1
net.ipv6.conf.all.accept_source_route = 0
net.ipv6.conf.all.autoconf = 1
net.ipv6.conf.all.dad_transmits = 1
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.conf.all.force_mld_version = 0
net.ipv6.conf.all.force_tllao = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.hop_limit = 64
net.ipv6.conf.all.max_addresses = 16
net.ipv6.conf.all.max_desync_factor = 600
net.ipv6.conf.all.mtu = 1280
net.ipv6.conf.all.ndisc_notify = 0
net.ipv6.conf.all.optimistic_dad = 0
net.ipv6.conf.all.proxy_ndp = 0
net.ipv6.conf.all.regen_max_retry = 3
net.ipv6.conf.all.router_probe_interval = 60
net.ipv6.conf.all.router_solicitation_delay = 1
net.ipv6.conf.all.router_solicitation_interval = 4
net.ipv6.conf.all.router_solicitations = 3
net.ipv6.conf.all.temp_prefered_lft = 86400
net.ipv6.conf.all.temp_valid_lft = 604800
net.ipv6.conf.all.use_tempaddr = 2
net.ipv6.conf.default.accept_dad = 1
net.ipv6.conf.default.accept_ra = 1
net.ipv6.conf.default.accept_ra_defrtr = 1
net.ipv6.conf.default.accept_ra_pinfo = 1
net.ipv6.conf.default.accept_ra_rtr_pref = 1
net.ipv6.conf.default.accept_redirects = 1
net.ipv6.conf.default.accept_source_route = 0
net.ipv6.conf.default.autoconf = 1
net.ipv6.conf.default.dad_transmits = 1
net.ipv6.conf.default.disable_ipv6 = 0
net.ipv6.conf.default.force_mld_version = 0
net.ipv6.conf.default.force_tllao = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.hop_limit = 64
net.ipv6.conf.default.max_addresses = 16
net.ipv6.conf.default.max_desync_factor = 600
net.ipv6.conf.default.mtu = 1280
net.ipv6.conf.default.ndisc_notify = 0
net.ipv6.conf.default.optimistic_dad = 0
net.ipv6.conf.default.proxy_ndp = 0
net.ipv6.conf.default.regen_max_retry = 3
net.ipv6.conf.default.router_probe_interval = 60
net.ipv6.conf.default.router_solicitation_delay = 1
net.ipv6.conf.default.router_solicitation_interval = 4
net.ipv6.conf.default.router_solicitations = 3
net.ipv6.conf.default.temp_prefered_lft = 86400
net.ipv6.conf.default.temp_valid_lft = 604800
net.ipv6.conf.default.use_tempaddr = 2
net.ipv6.conf.eth0.accept_dad = 1
net.ipv6.conf.eth0.accept_ra = 0
net.ipv6.conf.eth0.accept_ra_defrtr = 1
net.ipv6.conf.eth0.accept_ra_pinfo = 1
net.ipv6.conf.eth0.accept_ra_rtr_pref = 1
net.ipv6.conf.eth0.accept_redirects = 1
net.ipv6.conf.eth0.accept_source_route = 0
net.ipv6.conf.eth0.autoconf = 1
net.ipv6.conf.eth0.dad_transmits = 1
net.ipv6.conf.eth0.disable_ipv6 = 0
net.ipv6.conf.eth0.force_mld_version = 0
net.ipv6.conf.eth0.force_tllao = 0
net.ipv6.conf.eth0.forwarding = 0
net.ipv6.conf.eth0.hop_limit = 64
net.ipv6.conf.eth0.max_addresses = 16
net.ipv6.conf.eth0.max_desync_factor = 600
net.ipv6.conf.eth0.mtu = 1500
net.ipv6.conf.eth0.ndisc_notify = 0
net.ipv6.conf.eth0.optimistic_dad = 0
net.ipv6.conf.eth0.proxy_ndp = 0
net.ipv6.conf.eth0.regen_max_retry = 3
net.ipv6.conf.eth0.router_probe_interval = 60
net.ipv6.conf.eth0.router_solicitation_delay = 1
net.ipv6.conf.eth0.router_solicitation_interval = 4
net.ipv6.conf.eth0.router_solicitations = 3
net.ipv6.conf.eth0.temp_prefered_lft = 86400
net.ipv6.conf.eth0.temp_valid_lft = 604800
net.ipv6.conf.eth0.use_tempaddr = 0
net.ipv6.conf.ip6tnl0.accept_dad = -1
net.ipv6.conf.ip6tnl0.accept_ra = 1
net.ipv6.conf.ip6tnl0.accept_ra_defrtr = 1
net.ipv6.conf.ip6tnl0.accept_ra_pinfo = 1
net.ipv6.conf.ip6tnl0.accept_ra_rtr_pref = 1
net.ipv6.conf.ip6tnl0.accept_redirects = 1
net.ipv6.conf.ip6tnl0.accept_source_route = 0
net.ipv6.conf.ip6tnl0.autoconf = 1
net.ipv6.conf.ip6tnl0.dad_transmits = 1
net.ipv6.conf.ip6tnl0.disable_ipv6 = 0
net.ipv6.conf.ip6tnl0.force_mld_version = 0
net.ipv6.conf.ip6tnl0.force_tllao = 0
net.ipv6.conf.ip6tnl0.forwarding = 0
net.ipv6.conf.ip6tnl0.hop_limit = 64
net.ipv6.conf.ip6tnl0.max_addresses = 16
net.ipv6.conf.ip6tnl0.max_desync_factor = 600
net.ipv6.conf.ip6tnl0.mtu = 1452
net.ipv6.conf.ip6tnl0.ndisc_notify = 0
net.ipv6.conf.ip6tnl0.optimistic_dad = 0
net.ipv6.conf.ip6tnl0.proxy_ndp = 0
net.ipv6.conf.ip6tnl0.regen_max_retry = 3
net.ipv6.conf.ip6tnl0.router_probe_interval = 60
net.ipv6.conf.ip6tnl0.router_solicitation_delay = 1
net.ipv6.conf.ip6tnl0.router_solicitation_interval = 4
net.ipv6.conf.ip6tnl0.router_solicitations = 3
net.ipv6.conf.ip6tnl0.temp_prefered_lft = 86400
net.ipv6.conf.ip6tnl0.temp_valid_lft = 604800
net.ipv6.conf.ip6tnl0.use_tempaddr = -1
net.ipv6.conf.lo.accept_dad = -1
net.ipv6.conf.lo.accept_ra = 1
net.ipv6.conf.lo.accept_ra_defrtr = 1
net.ipv6.conf.lo.accept_ra_pinfo = 1
net.ipv6.conf.lo.accept_ra_rtr_pref = 1
net.ipv6.conf.lo.accept_redirects = 1
net.ipv6.conf.lo.accept_source_route = 0
net.ipv6.conf.lo.autoconf = 1
net.ipv6.conf.lo.dad_transmits = 1
net.ipv6.conf.lo.disable_ipv6 = 0
net.ipv6.conf.lo.force_mld_version = 0
net.ipv6.conf.lo.force_tllao = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.hop_limit = 64
net.ipv6.conf.lo.max_addresses = 16
net.ipv6.conf.lo.max_desync_factor = 600
net.ipv6.conf.lo.mtu = 65536
net.ipv6.conf.lo.ndisc_notify = 0
net.ipv6.conf.lo.optimistic_dad = 0
net.ipv6.conf.lo.proxy_ndp = 0
net.ipv6.conf.lo.regen_max_retry = 3
net.ipv6.conf.lo.router_probe_interval = 60
net.ipv6.conf.lo.router_solicitation_delay = 1
net.ipv6.conf.lo.router_solicitation_interval = 4
net.ipv6.conf.lo.router_solicitations = 3
net.ipv6.conf.lo.temp_prefered_lft = 86400
net.ipv6.conf.lo.temp_valid_lft = 604800
net.ipv6.conf.lo.use_tempaddr = -1
net.ipv6.conf.sit0.accept_dad = -1
net.ipv6.conf.sit0.accept_ra = 1
net.ipv6.conf.sit0.accept_ra_defrtr = 1
net.ipv6.conf.sit0.accept_ra_pinfo = 1
net.ipv6.conf.sit0.accept_ra_rtr_pref = 1
net.ipv6.conf.sit0.accept_redirects = 1
net.ipv6.conf.sit0.accept_source_route = 0
net.ipv6.conf.sit0.autoconf = 1
net.ipv6.conf.sit0.dad_transmits = 1
net.ipv6.conf.sit0.disable_ipv6 = 0
net.ipv6.conf.sit0.force_mld_version = 0
net.ipv6.conf.sit0.force_tllao = 0
net.ipv6.conf.sit0.forwarding = 0
net.ipv6.conf.sit0.hop_limit = 64
net.ipv6.conf.sit0.max_addresses = 16
net.ipv6.conf.sit0.max_desync_factor = 600
net.ipv6.conf.sit0.mtu = 1480
net.ipv6.conf.sit0.ndisc_notify = 0
net.ipv6.conf.sit0.optimistic_dad = 0
net.ipv6.conf.sit0.proxy_ndp = 0
net.ipv6.conf.sit0.regen_max_retry = 3
net.ipv6.conf.sit0.router_probe_interval = 60
net.ipv6.conf.sit0.router_solicitation_delay = 1
net.ipv6.conf.sit0.router_solicitation_interval = 4
net.ipv6.conf.sit0.router_solicitations = 3
net.ipv6.conf.sit0.temp_prefered_lft = 86400
net.ipv6.conf.sit0.temp_valid_lft = 604800
net.ipv6.conf.sit0.use_tempaddr = -1
net.ipv6.conf.wlan0.accept_dad = 1
net.ipv6.conf.wlan0.accept_ra = 1
net.ipv6.conf.wlan0.accept_ra_defrtr = 1
net.ipv6.conf.wlan0.accept_ra_pinfo = 1
net.ipv6.conf.wlan0.accept_ra_rtr_pref = 1
net.ipv6.conf.wlan0.accept_redirects = 1
net.ipv6.conf.wlan0.accept_source_route = 0
net.ipv6.conf.wlan0.autoconf = 1
net.ipv6.conf.wlan0.dad_transmits = 1
net.ipv6.conf.wlan0.disable_ipv6 = 0
net.ipv6.conf.wlan0.force_mld_version = 0
net.ipv6.conf.wlan0.force_tllao = 0
net.ipv6.conf.wlan0.forwarding = 0
net.ipv6.conf.wlan0.hop_limit = 64
net.ipv6.conf.wlan0.max_addresses = 16
net.ipv6.conf.wlan0.max_desync_factor = 600
net.ipv6.conf.wlan0.mtu = 1500
net.ipv6.conf.wlan0.ndisc_notify = 0
net.ipv6.conf.wlan0.optimistic_dad = 0
net.ipv6.conf.wlan0.proxy_ndp = 0
net.ipv6.conf.wlan0.regen_max_retry = 3
net.ipv6.conf.wlan0.router_probe_interval = 60
net.ipv6.conf.wlan0.router_solicitation_delay = 1
net.ipv6.conf.wlan0.router_solicitation_interval = 4
net.ipv6.conf.wlan0.router_solicitations = 3
net.ipv6.conf.wlan0.temp_prefered_lft = 86400
net.ipv6.conf.wlan0.temp_valid_lft = 604800
net.ipv6.conf.wlan0.use_tempaddr = 2
net.ipv6.icmp.ratelimit = 1000
net.ipv6.ip6frag_high_thresh = 4194304
net.ipv6.ip6frag_low_thresh = 3145728
net.ipv6.ip6frag_secret_interval = 600
net.ipv6.ip6frag_time = 60
net.ipv6.mld_max_msf = 64
net.ipv6.neigh.default.anycast_delay = 100
net.ipv6.neigh.default.app_solicit = 0
net.ipv6.neigh.default.base_reachable_time_ms = 30000
net.ipv6.neigh.default.delay_first_probe_time = 5
net.ipv6.neigh.default.gc_interval = 30
net.ipv6.neigh.default.gc_stale_time = 60
net.ipv6.neigh.default.gc_thresh1 = 128
net.ipv6.neigh.default.gc_thresh2 = 512
net.ipv6.neigh.default.gc_thresh3 = 1024
net.ipv6.neigh.default.locktime = 0
net.ipv6.neigh.default.mcast_solicit = 3
net.ipv6.neigh.default.proxy_delay = 80
net.ipv6.neigh.default.proxy_qlen = 64
net.ipv6.neigh.default.retrans_time_ms = 1000
net.ipv6.neigh.default.ucast_solicit = 3
net.ipv6.neigh.default.unres_qlen = 34
net.ipv6.neigh.default.unres_qlen_bytes = 65536
net.ipv6.neigh.eth0.anycast_delay = 100
net.ipv6.neigh.eth0.app_solicit = 0
net.ipv6.neigh.eth0.base_reachable_time_ms = 30000
net.ipv6.neigh.eth0.delay_first_probe_time = 5
net.ipv6.neigh.eth0.gc_stale_time = 60
net.ipv6.neigh.eth0.locktime = 0
net.ipv6.neigh.eth0.mcast_solicit = 3
net.ipv6.neigh.eth0.proxy_delay = 80
net.ipv6.neigh.eth0.proxy_qlen = 64
net.ipv6.neigh.eth0.retrans_time_ms = 1000
net.ipv6.neigh.eth0.ucast_solicit = 3
net.ipv6.neigh.eth0.unres_qlen = 34
net.ipv6.neigh.eth0.unres_qlen_bytes = 65536
net.ipv6.neigh.ip6tnl0.anycast_delay = 100
net.ipv6.neigh.ip6tnl0.app_solicit = 0
net.ipv6.neigh.ip6tnl0.base_reachable_time_ms = 30000
net.ipv6.neigh.ip6tnl0.delay_first_probe_time = 5
net.ipv6.neigh.ip6tnl0.gc_stale_time = 60
net.ipv6.neigh.ip6tnl0.locktime = 0
net.ipv6.neigh.ip6tnl0.mcast_solicit = 3
net.ipv6.neigh.ip6tnl0.proxy_delay = 80
net.ipv6.neigh.ip6tnl0.proxy_qlen = 64
net.ipv6.neigh.ip6tnl0.retrans_time_ms = 1000
net.ipv6.neigh.ip6tnl0.ucast_solicit = 3
net.ipv6.neigh.ip6tnl0.unres_qlen = 34
net.ipv6.neigh.ip6tnl0.unres_qlen_bytes = 65536
net.ipv6.neigh.lo.anycast_delay = 100
net.ipv6.neigh.lo.app_solicit = 0
net.ipv6.neigh.lo.base_reachable_time_ms = 30000
net.ipv6.neigh.lo.delay_first_probe_time = 5
net.ipv6.neigh.lo.gc_stale_time = 60
net.ipv6.neigh.lo.locktime = 0
net.ipv6.neigh.lo.mcast_solicit = 3
net.ipv6.neigh.lo.proxy_delay = 80
net.ipv6.neigh.lo.proxy_qlen = 64
net.ipv6.neigh.lo.retrans_time_ms = 1000
net.ipv6.neigh.lo.ucast_solicit = 3
net.ipv6.neigh.lo.unres_qlen = 34
net.ipv6.neigh.lo.unres_qlen_bytes = 65536
net.ipv6.neigh.sit0.anycast_delay = 100
net.ipv6.neigh.sit0.app_solicit = 0
net.ipv6.neigh.sit0.base_reachable_time_ms = 30000
net.ipv6.neigh.sit0.delay_first_probe_time = 5
net.ipv6.neigh.sit0.gc_stale_time = 60
net.ipv6.neigh.sit0.locktime = 0
net.ipv6.neigh.sit0.mcast_solicit = 3
net.ipv6.neigh.sit0.proxy_delay = 80
net.ipv6.neigh.sit0.proxy_qlen = 64
net.ipv6.neigh.sit0.retrans_time_ms = 1000
net.ipv6.neigh.sit0.ucast_solicit = 3
net.ipv6.neigh.sit0.unres_qlen = 34
net.ipv6.neigh.sit0.unres_qlen_bytes = 65536
net.ipv6.neigh.wlan0.anycast_delay = 100
net.ipv6.neigh.wlan0.app_solicit = 0
net.ipv6.neigh.wlan0.base_reachable_time_ms = 30000
net.ipv6.neigh.wlan0.delay_first_probe_time = 5
net.ipv6.neigh.wlan0.gc_stale_time = 60
net.ipv6.neigh.wlan0.locktime = 0
net.ipv6.neigh.wlan0.mcast_solicit = 3
net.ipv6.neigh.wlan0.proxy_delay = 80
net.ipv6.neigh.wlan0.proxy_qlen = 64
net.ipv6.neigh.wlan0.retrans_time_ms = 1000
net.ipv6.neigh.wlan0.ucast_solicit = 3
net.ipv6.neigh.wlan0.unres_qlen = 34
net.ipv6.neigh.wlan0.unres_qlen_bytes = 65536
net.ipv6.route.gc_elasticity = 9
net.ipv6.route.gc_interval = 30
net.ipv6.route.gc_min_interval = 0
net.ipv6.route.gc_min_interval_ms = 500
net.ipv6.route.gc_thresh = 1024
net.ipv6.route.gc_timeout = 60
net.ipv6.route.max_size = 4096
net.ipv6.route.min_adv_mss = 1220
net.ipv6.route.mtu_expires = 600
net.ipv6.xfrm6_gc_thresh = 1024
net.netfilter.nf_log.0 = NONE
net.netfilter.nf_log.1 = NONE
net.netfilter.nf_log.10 = NONE
net.netfilter.nf_log.11 = NONE
net.netfilter.nf_log.12 = NONE
net.netfilter.nf_log.2 = NONE
net.netfilter.nf_log.3 = NONE
net.netfilter.nf_log.4 = NONE
net.netfilter.nf_log.5 = NONE
net.netfilter.nf_log.6 = NONE
net.netfilter.nf_log.7 = NONE
net.netfilter.nf_log.8 = NONE
net.netfilter.nf_log.9 = NONE
net.phonet.local_port_range = 64	127
net.unix.max_dgram_qlen = 10
vm.admin_reserve_kbytes = 8192
vm.block_dump = 0
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.drop_caches = 0
vm.extfrag_threshold = 200
vm.highmem_is_dirtyable = 0
vm.laptop_mode = 0
vm.legacy_va_layout = 0
vm.lowmem_reserve_ratio = 32	32
vm.max_map_count = 65530
vm.mem_management_thresh = 16384
vm.min_free_kbytes = 3436
vm.min_free_order_shift = 1
vm.mmap_min_addr = 32768
vm.nr_pdflush_threads = 0
vm.oom_dump_tasks = 1
vm.oom_kill_allocating_task = 0
vm.overcommit_memory = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.panic_on_oom = 0
vm.percpu_pagelist_fraction = 0
vm.scan_unevictable_pages = 0
vm.stat_interval = 1
vm.swappiness = 60
vm.user_reserve_kbytes = 31132
vm.vfs_cache_pressure = 100

cat /proc/interrupts/

Code: Select all

           CPU0       CPU1       CPU2       CPU3       
 35:        105          0          0          0       GIC  vsync
 40:          0          1          0          0       GIC  eth0
 47:          0          0          0          0       GIC  keypad
 60:          2          0          0          0       GIC  sdio
 62:          0          0          0          0       GIC  dwc_otg, dwc_otg_hcd:usb2, dwc_otg_pcd
 63:          0          0          0     489148       GIC  dwc_otg, dwc_otg_hcd:usb1
 89:          0          0          0          0       GIC  amhdmitx
 92:      11005          0          0          0       GIC  MESON TIMER-F
 93:          0      10906          0          0       GIC  MESON TIMER-G
 94:          0          0      10922          0       GIC  MESON TIMER-H
 95:          0          0          0      10995       GIC  MESON TIMER-I
107:          0          0          0          0       GIC  uart_b_ttyS2:
110:       4202          0          0          0       GIC  sdhc
121:          0          0          0          0       GIC  rdma
122:       1382          0          0          0       GIC  uart_ao_ttyS0:
178:          0          0          0          0       GIC  audiodsp_mailbox
182:          0          0          0          0       GIC  ge2d irq
183:          0          0          0          0       GIC  amhdmitx-aocec
192:          0          0          0          0       GIC  Mali_GP
193:          0          0          0          0       GIC  Mali_GP_MMU
194:          0          0          0          0       GIC  Mali_PP_Broadcast
196:          0          0          0          0       GIC  Mali_PP0
197:          0          0          0          0       GIC  Mali_PP0_MMU
198:          0          0          0          0       GIC  Mali_PP1
199:          0          0          0          0       GIC  Mali_PP1_MMU
IPI0:          0          0          0          0  CPU wakeup interrupts
IPI1:          0          0          0          0  Timer broadcast interrupts
IPI2:      13470      14729      15604       6713  Rescheduling interrupts
IPI3:          4          6          3          5  Function call interrupts
IPI4:          0          0          0          0  Single function call interrupts
IPI5:          0          0          0          0  CPU stop interrupts
IPI6:          0          0          0          0  CPU backtrace
Err:          0

But what are you actually trying to achieve with powertop? Powersaving and realtime usually don't mix well...

[Clx]

Small update on rt-preempt kernel testing.
I've found that it mostly crashes when working with the file system.
Running the following crashes the system in a minute or two:

Code: Select all

stress --cpu 8 --io 4 --vm 2 --hdd 2 --vm-bytes 128M --timeout 2500s

I tried with: SD card, eMMC module, USB flash.

I repeated the stress test with default kernel and got the same crash!
It takes more time on default kernel to crash, like 5-15 minutes.
I created a topic about it here (so it doesn't get mixed with rt-preempt discussion) - http://forum.odroid.com/viewtopic.php?f=115&t=9631

[kinsa]

Hi,

I've ported some of your patches to the latest odroid c1 tree: https://github.com/kinsamanka/linux-odr ... .10.y-rt71

So far I haven't gotten any problems due to the file system; I'm using USB HDD instead of the solid state devices.

What I've noticed is that the latency of the wired lan is horrible, sometimes it disconnects. USB lan is OK.

I'm now running an overnight test, using both the stress test and the cyclictest running at the same time. (max 4 hrs now).

BTW, I've stripped the kernel to its bare minimum, no extra amlogic drivers (video, sound, etc...).

Cheers.

UPDATE:

Here's the link to the pre-compiled kernel: http://0ptr.link/files/odroidc1_3.10.70-rt71.tgz

Do note that it has a limited set of compiled modules, you need to have serial access to your board.

And here's the cyclictest result:

Code: Select all

root@Machinekit-C1:~# cyclictest -D1d -m -n -a0 -t1 -p99 -i1000
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 17.71 17.70 17.62 11/131 2409          

T: 0 ( 2109) P:99 I:1000 C:3610316 Min:      6 Act:   14 Avg:   12 Max:     572

[mlinuxguy]

Unless you go into the amlogic ethernet driver and fix all the spin_lock calls appropriately I'm not surprised.
As an example you might have to do this:

Function qla2x00_poll does local_irq_save() before calling qla24xx_intr_handler
which has a spinlock. Since spinlocks are sleepable on rt, it is not allowed
to call them with interrupts disabled. Therefore we use local_irq_save_nort()
instead which saves flags without disabling interrupts.

So you would need to check the poll function to see if it calls the intr handler, or check your spin_locks to see if any have disabled interrupts.

[kinsa]

If you start pinging to C1, the latency is very bad and doesn't improve over time.

If you ping from C1, the latency is a bit high initially but it improves over time (<2mins) to normal levels. The ping originating from the external host improves as well.

But once the ping from C1 is stopped, the latency from the external host deteriorates but not to the same level it had initially.

The amlogic ethernet driver doesn't have a poll function; I think it's using DMA to handle data transfer.

[kinsa]

I'm getting these errors:

Code: Select all

[  304.168888@0] BUG: scheduling while atomic: stress/2441/0x00010001
[  304.168906@0] Modules linked in: sg sd_mod usb_storage scsi_mod asix usbnet w1_gpio wire
[  304.168914@0] CPU: 0 PID: 2441 Comm: stress Not tainted 3.10.70-rt71-mmc #4
[  304.168939@0] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[  304.168954@0] [<c0011b04>] (show_stack+0x10/0x14) from [<c03f6844>] (__schedule_bug+0x44/0x5c)
[  304.168968@0] [<c03f6844>] (__schedule_bug+0x44/0x5c) from [<c0400ef4>] (__schedule+0x7c/0x4d4)
[  304.168977@0] [<c0400ef4>] (__schedule+0x7c/0x4d4) from [<c04015d0>] (schedule+0x7c/0x94)
[  304.168988@0] [<c04015d0>] (schedule+0x7c/0x94) from [<c04023a8>] (rt_spin_lock_slowlock+0x180/0x274)
[  304.169003@0] [<c04023a8>] (rt_spin_lock_slowlock+0x180/0x274) from [<c0328fec>] (aml_sdhc_irq+0x1c/0x290)
[  304.169016@0] [<c0328fec>] (aml_sdhc_irq+0x1c/0x290) from [<c0085820>] (handle_irq_event_percpu+0x68/0x194)
[  304.169027@0] [<c0085820>] (handle_irq_event_percpu+0x68/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8)
[  304.169037@0] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8)
[  304.169047@0] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40)
[  304.169061@0] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c)
[  304.169071@0] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c)
[  304.169080@0] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60)
[  304.169084@0] Exception stack(0xedbcdfb0 to 0xedbcdff8)
[  304.169090@0] dfa0:                                     b6e4e028 00000000 00000000 00000001
[  304.169097@0] dfc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000e 00000000
[  304.169103@0] dfe0: 000144b4 beeb9b70 b6d9b9e9 b6d9ba06 60000030 ffffffff
[  312.608864@3] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
[  312.608876@3] CPU: 3 PID: 76 Comm: irq/63-dwc_otg_ Tainted: G        W    3.10.70-rt71-mmc #4
[  312.608903@3] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[  312.608917@3] [<c0011b04>] (show_stack+0x10/0x14) from [<c03f5e08>] (panic+0x94/0x1ec)
[  312.608931@3] [<c03f5e08>] (panic+0x94/0x1ec) from [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108)
[  312.608942@3] [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108) from [<c0084db0>] (watchdog_timer_fn+0x38/0x174)
[  312.608956@3] [<c0084db0>] (watchdog_timer_fn+0x38/0x174) from [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c)
[  312.608967@3] [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c) from [<c004c0ec>] (hrtimer_interrupt+0x148/0x338)
[  312.608978@3] [<c004c0ec>] (hrtimer_interrupt+0x148/0x338) from [<c001c5c0>] (meson_timer_interrupt+0x70/0x88)
[  312.608988@3] [<c001c5c0>] (meson_timer_interrupt+0x70/0x88) from [<c0085820>] (handle_irq_event_percpu+0x68/0x194)
[  312.608998@3] [<c0085820>] (handle_irq_event_percpu+0x68/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8)
[  312.609009@3] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8)
[  312.609019@3] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40)
[  312.609032@3] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c)
[  312.609042@3] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c)
[  312.609052@3] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000db00>] (__irq_svc+0x40/0x84)
[  312.609055@3] Exception stack(0xeeaa1ee0 to 0xeeaa1f28)
[  312.609063@3] 1ee0: 00000000 000058f0 00000000 00000000 eeaa0000 eeae8e40 ef55b520 c05d59c0
[  312.609071@3] 1f00: ef55b500 00000000 c008605c eeaa1f3c eeaa1f14 eeaa1f28 c0085b84 c00520fc
[  312.609074@3] 1f20: 40000113 ffffffff
[  312.609087@3] [<c000db00>] (__irq_svc+0x40/0x84) from [<c00520fc>] (migrate_enable+0x24/0x15c)
[  312.609099@3] [<c00520fc>] (migrate_enable+0x24/0x15c) from [<c0085b84>] (irq_thread+0x9c/0x130)
[  312.609109@3] [<c0085b84>] (irq_thread+0x9c/0x130) from [<c0047f98>] (kthread+0x98/0xa0)
[  312.609119@3] [<c0047f98>] (kthread+0x98/0xa0) from [<c000dfd8>] (ret_from_fork+0x14/0x3c)
[  312.609127@1] CPU1: stopping
[  312.609134@1] CPU: 1 PID: 2429 Comm: stress Tainted: G        W    3.10.70-rt71-mmc #4
[  312.609149@1] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[  312.609158@1] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4)
[  312.609167@1] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8)
[  312.609176@1] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c)
[  312.609185@1] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000db00>] (__irq_svc+0x40/0x84)
[  312.609188@1] Exception stack(0xedc93e38 to 0xedc93e80)
[  312.609192@1] 3e20:                                                       00000001 00000000
[  312.609200@1] 3e40: 00000000 00000000 c3235d20 b4081000 eda33300 edff2e00 c0ac3b20 ec9d0204
[  312.609208@1] 3e60: 2035975f ec9d0000 00000000 edc93e80 c0402280 c00adbe8 60000113 ffffffff
[  312.609221@1] [<c000db00>] (__irq_svc+0x40/0x84) from [<c00adbe8>] (do_anonymous_page.isra.73+0x194/0x244)
[  312.609233@1] [<c00adbe8>] (do_anonymous_page.isra.73+0x194/0x244) from [<c00b1074>] (handle_mm_fault+0xd8/0xec)
[  312.609244@1] [<c00b1074>] (handle_mm_fault+0xd8/0xec) from [<c0016784>] (do_page_fault+0xbc/0x1e8)
[  312.609254@1] [<c0016784>] (do_page_fault+0xbc/0x1e8) from [<c0008320>] (do_DataAbort+0x34/0x98)
[  312.609263@1] [<c0008320>] (do_DataAbort+0x34/0x98) from [<c000dc94>] (__dabt_usr+0x34/0x40)
[  312.609265@1] Exception stack(0xedc93fb0 to 0xedc93ff8)
[  312.609271@1] 3fa0:                                     ffffffff b4081008 05310000 00000000
[  312.609279@1] 3fc0: b6d72008 00001000 aed72008 0000005a 08000000 00000000 00001000 00000000
[  312.609285@1] 3fe0: 00000002 beeb9b48 b6dc86cb 0000a186 80000030 ffffffff
[  312.609290@2] CPU2: stopping
[  312.609297@2] CPU: 2 PID: 2442 Comm: stress Tainted: G        W    3.10.70-rt71-mmc #4
[  312.609310@2] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[  312.609319@2] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4)
[  312.609327@2] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8)
[  312.609336@2] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c)
[  312.609345@2] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60)
[  312.609348@2] Exception stack(0xedbcffb0 to 0xedbcfff8)
[  312.609354@2] ffa0:                                     b6e4e028 00000000 b6e4d1b4 b6e4d460
[  312.609361@2] ffc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000f 00000000
[  312.609368@2] ffe0: 000144b4 beeb9b70 b6d9b9e9 b6d9b9ee 60000030 ffffffff
[  312.910969@3] SMP: failed to stop secondary CPUs

Where shall I start looking?

UPDATE:
Another one:

Code: Select all

[  715.793718@0] ------------[ cut here ]------------
[  715.793736@0] WARNING: at kernel/irq/handle.c:148 handle_irq_event_percpu+0xa0/0x194()
[  715.793746@0] irq 110 handler aml_sdhc_irq+0x0/0x290 enabled interrupts
[  715.793759@0] Modules linked in: sg sd_mod usb_storage scsi_mod asix w1_gpio usbnet wire
[  715.793765@0] CPU: 0 PID: 2650 Comm: stress Not tainted 3.10.70-rt71-mmc #4
[  715.793781@0] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[  715.793791@0] [<c0011b04>] (show_stack+0x10/0x14) from [<c0029594>] (warn_slowpath_common+0x4c/0x68)
[  715.793799@0] [<c0029594>] (warn_slowpath_common+0x4c/0x68) from [<c0029644>] (warn_slowpath_fmt+0x30/0x40)
[  715.793807@0] [<c0029644>] (warn_slowpath_fmt+0x30/0x40) from [<c0085858>] (handle_irq_event_percpu+0xa0/0x194)
[  715.793816@0] [<c0085858>] (handle_irq_event_percpu+0xa0/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8)
[  715.793824@0] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8)
[  715.793832@0] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40)
[  715.793843@0] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c)
[  715.793851@0] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c)
[  715.793859@0] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60)
[  715.793862@0] Exception stack(0xee0ddfb0 to 0xee0ddff8)
[  715.793867@0] dfa0:                                     000b1a7a 00000001 00000000 00000000
[  715.793873@0] dfc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000f 00000000
[  715.793878@0] dfe0: 000144b4 beceab70 0000a0e7 b6e2c9e0 60000030 ffffffff
[  715.793880@0] ---[ end trace 0000000000000002 ]---
[ 1020.608855@3] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0
[ 1020.608864@3] CPU: 3 PID: 2647 Comm: stress Tainted: G        W    3.10.70-rt71-mmc #4
[ 1020.608890@3] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[ 1020.608904@3] [<c0011b04>] (show_stack+0x10/0x14) from [<c03f5e08>] (panic+0x94/0x1ec)
[ 1020.608918@3] [<c03f5e08>] (panic+0x94/0x1ec) from [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108)
[ 1020.608929@3] [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108) from [<c0084db0>] (watchdog_timer_fn+0x38/0x174)
[ 1020.608943@3] [<c0084db0>] (watchdog_timer_fn+0x38/0x174) from [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c)
[ 1020.608954@3] [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c) from [<c004c0ec>] (hrtimer_interrupt+0x148/0x338)
[ 1020.608965@3] [<c004c0ec>] (hrtimer_interrupt+0x148/0x338) from [<c001c5c0>] (meson_timer_interrupt+0x70/0x88)
[ 1020.608975@3] [<c001c5c0>] (meson_timer_interrupt+0x70/0x88) from [<c0085820>] (handle_irq_event_percpu+0x68/0x194)
[ 1020.608985@3] [<c0085820>] (handle_irq_event_percpu+0x68/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8)
[ 1020.608995@3] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8)
[ 1020.609006@3] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40)
[ 1020.609018@3] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c)
[ 1020.609029@3] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c)
[ 1020.609038@3] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60)
[ 1020.609041@3] Exception stack(0xece9ffb0 to 0xece9fff8)
[ 1020.609048@3] ffa0:                                     00000000 b6ede1ac b6ede1b8 b6ede460
[ 1020.609055@3] ffc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000c 00000000
[ 1020.609061@3] ffe0: 000144b4 beceab70 b6e2c9e9 b6e2cb30 60000030 ffffffff
[ 1020.609069@2] CPU2: stopping
[ 1020.609076@2] CPU: 2 PID: 2649 Comm: stress Tainted: G        W    3.10.70-rt71-mmc #4
[ 1020.609090@2] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[ 1020.609099@2] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4)
[ 1020.609108@2] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8)
[ 1020.609117@2] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c)
[ 1020.609126@2] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60)
[ 1020.609129@2] Exception stack(0xedf85fb0 to 0xedf85ff8)
[ 1020.609135@2] 5fa0:                                     b6edf028 00000000 00000000 00000001
[ 1020.609142@2] 5fc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000e 00000000
[ 1020.609149@2] 5fe0: 000144b4 beceab70 b6e2c9e9 b6e2ca00 60000030 ffffffff
[ 1020.609153@1] CPU1: stopping
[ 1020.609160@1] CPU: 1 PID: 2639 Comm: stress Tainted: G        W    3.10.70-rt71-mmc #4
[ 1020.609173@1] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14)
[ 1020.609182@1] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4)
[ 1020.609190@1] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8)
[ 1020.609199@1] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c)
[ 1020.609208@1] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60)
[ 1020.609211@1] Exception stack(0xe4f13fb0 to 0xe4f13ff8)
[ 1020.609217@1] 3fa0:                                     b6ede460 beceab74 00000000 b6ede460
[ 1020.609224@1] 3fc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 00000004 00000000
[ 1020.609231@1] 3fe0: 000144b4 beceab60 b6e2c9e9 b6e2cb00 a0000030 ffffffff
[ 1020.910864@3] SMP: failed to stop secondary CPUs

UPDATE:

Managed to patch the sdhc driver, but not sure if it's correct though.

Currently undergoing stress testing.

[kinsa]

Just an update, passed the 4 hr mark and still no errors. Stress testing done on a UHS-1 card (disableuhs flag turned off).

I'm now currently testing on a class 10 sd card.

Here's my cyclictest results:

Code: Select all

root@machinekit:~# cyclictest -D1d -m -n -a0 -t1 -p99 -i1000 --smp# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 33.88 34.12 34.18 20/241 21457          

T: 0 ( 1990) P:99 I:1000 C:16180778 Min:      5 Act:   20 Avg:   12 Max:     517
T: 1 ( 1991) P:99 I:1500 C:10787158 Min:      5 Act:   10 Avg:   13 Max:     656
T: 2 ( 1992) P:99 I:2000 C:8090354 Min:      5 Act:   10 Avg:   13 Max:     332
T: 3 ( 1993) P:99 I:2500 C:6472261 Min:      6 Act:   11 Avg:   14 Max:     477

UPDATE:

Stress test on the class 10 sd card running for more than 2 hrs now without errors.

Interestingly, it has better results than the UHS-1 card:

Code: Select all

root@machinekit:~# cyclictest -D1d -m -n -a0 -t1 -p99 -i1000 --smp
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 33.78 33.96 33.98 22/241 2639          

T: 0 ( 2587) P:99 I:1000 C:7423122 Min:      6 Act:   15 Avg:   11 Max:      51
T: 1 ( 2588) P:99 I:1500 C:4948641 Min:      6 Act:   11 Avg:   12 Max:      42
T: 2 ( 2589) P:99 I:2000 C:3711448 Min:      6 Act:   15 Avg:   11 Max:      40
T: 3 ( 2590) P:99 I:2500 C:2969114 Min:      6 Act:   13 Avg:   12 Max:      43

[adbensi]

Me too, disableuhs turned off, but network slower yet, I would like to try this:
I have issues on the network too, but CLx says that with him work fine, may be another program cause the problem. I would like to try stop all process and test with static IP.

I had the impression that works with Sandisk SD UHS when I commented on setenv disableuhs boot.ini "disableuhs". My test stop because enough space left (works 7 minutes). Another test is change the filesystem.

[kinsa]

I had the impression that works with Sandisk SD UHS when I commented on setenv disableuhs boot.ini "disableuhs". My test stop because enough space left (works 7 minutes). Another test is change the filesystem.

You just need a bigger card or run on a different partition. The command

Code: Select all

stress --cpu 8 --io 4 --vm 2 --hdd 2 --vm-bytes 128M --timeout 2500s

needs at least 2G free space to work on.

I've been using a USB lan adapter with no issues.

All we need now is a working onboard LAN on rt-preempt and we're good to go

[Clx]

Hi Kinsa, thanks for joining me on this.
Actually, I've tried to do the same with the sdhc previously, but after that the kernel couldn't find the modules - after that I was distracted and switched to the other task.
Now I understand that the problem was in aufs - I ran "make modules -j5" and didn't notice the error, so after that I copied an almost empty modules dir:D

I've picked the new kernel version from Hardkernel, applied new rt74 patch and then applied your patches.
The code is here:
https://github.com/Clouded/linux-rt-odr ... 10.70-rt74

Could you please pull it so we would both be working on the same and fresh version?

UPDATE:

Compiled real time kernel 3.10.70-rt74 with modules can be downloaded here (in full configuration without disabling anything):
https://mega.co.nz/#!AM4j1AIb!p70ZU32GD ... ULRWBXc2NU

[kinsa]

I've diffed our tree and the only significant change is this file:

Code: Select all

/media/Data/work/linux-odroid$ git diff clouded/odroidc-3.10.70-rt74 -- kernel/sched/core.c
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index 42c24d0..1edc718 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -7489,7 +7489,7 @@ void __might_sleep(const char *file, int line, int preempt_offset)
 
        rcu_sleep_check(); /* WARN_ON_ONCE() by default, no rate limit reqd. */
        if ((preempt_count_equals(preempt_offset) && !irqs_disabled() &&
-        !is_idle_task(current)) || !is_idle_task(current)) || oops_in_progress)
+            !is_idle_task(current)) || oops_in_progress)
                return;
        if (system_state != SYSTEM_RUNNING &&
            (!__might_sleep_init_called || system_state != SYSTEM_BOOTING))

The other differences are just cosmetic.

I've packaged the RT kernel and the debs can be found here: http://0ptr.link/C1/pool/main/l/linux-upstream/

It contains the full configuration and can also be used under Arch linux.

Cheers!

PS: did you made a progress on RPi2?

[Clx]

PS: did you made a progress on RPi2?

RPi2 is 99% ready, except for the one thing - new RPi's USB driver uses FIQ and causes hangs when used on SMP system with RT PREEMPT.
And the standard USB driver seems to be broken for Raspberry Pi 2.

[adbensi]

Hello Kinsa,
Now your network work fine on the rt?

[kinsa]

only on usb lan

[adbensi]

sorry for not understanding, but what is the difference with the rt71 version to "finally working"? Here was fine, except for the EMMC and network. Best regards.

[kinsa]

The sdhc driver issue was fixed, so the stress test now runs OK. Although long term test is still needed.

The only remaining issue is the onboard network driver.

I've made an image for Machinekit, please have a lool at it

Cheers

[adbensi]

Very nice,
I am seen your posts on the machinekit, a lot of work, nice job, congratulations!

This is the machinekit image for ODROID?
http://0ptr.link/C1/pool/main/m/machine ... _armhf.deb

I compiled machinekit directly, needed to do the night rss. precompiled image is very great

I would suggest the cammill.org software

[kinsa]

Yup.

My next job is to port PICnc