Real time kernel 3.10.70-rt74 (PREEMPT_RT), finally working

by **Clx** » Tue Feb 03, 2015 11:43 pm

Hi all!

Together with mlinuxguy, adbensi and kinsa we've managed to build a working real time (fully preemptive) kernel.
It is overall stable (I couldn't crash it with stress for a few hours now), but it has few minor issues:
1) Low built-in Ethernet performance - USB-WiFi and USB-Ethernet work fine.
2) Poor RT performance on some SD cards - works great with eMMC module and some SD cards too.

The source code is available on Github - https://github.com/Clouded/linux-rt-odr ... 10.70-rt74
Basically, it's a fork of Hardkernel's Linux for Odroid-C1 with:
- rt-patch applied
- few minor fixes (manually applied patch rejects, made a few spinlocks static, fixed a timer typo)
- fixes by kinsa (sdhc and aufs)
- some configuration (enabled PREEMPT_RT, periodic timer ticks, performance governor default)

DOWNLOADS:

[UPDATE 23 MAR 2015]

Debs of kernel with modules and headers:
linux-image-3.10.72-rt77_20150319_armhf.deb
linux-headers-3.10.72-rt77_20150319_armhf.deb

Built with make-kpkg, installs smoothly on Ubuntu (dtbs included and installed automatically) and should work on other distros too.
To install download the package and run:

Code: Select all: sudo dpkg -i linux-image-3.10.72-rt77_20150319_armhf.deb

Patched source code is available here:
https://github.com/Clouded/linux-rt-odr ... 10.72-rt77

Cyclictest results:

Code: Select all: odroid@odroid:~$ sudo cyclictest --smp -p99 -m -n # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 0.15 0.37 0.47 1/223 1790 T: 0 ( 1778) P:99 I:1000 C:4501611 Min: 6 Act: 8 Avg: 9 Max: 42 T: 1 ( 1779) P:99 I:1500 C:3001053 Min: 6 Act: 8 Avg: 8 Max: 36 T: 2 ( 1780) P:99 I:2000 C:2250775 Min: 7 Act: 8 Avg: 8 Max: 38 T: 3 ( 1781) P:99 I:2500 C:1800608 Min: 6 Act: 8 Avg: 9 Max: 26

by **mlinuxguy** » Wed Feb 04, 2015 12:10 am

"Scheduling while atomic" indicates that you've tried to sleep somewhere that you shouldn't - like within a spinlock-protected critical section or an interrupt handler

When I did the RT patch port to the Odroid-U I also had to fix spin_lock calls that were missed by making them raw_spinlock()
-static DEFINE_SPINLOCK(unwind_lock);
+static DEFINE_RAW_SPINLOCK(unwind_lock);
(perhaps none of those got missed under recent RT patches)

I don't know that this was done correctly:

kernel/sched/core.c.rej looks like this:

Why did you leave out the: system_state != SYSTEM_RUNNING

I'll try applying the RT patches after my new Odroid-C1 comes in, I dedicated the test one to the basement geothermal monitoring...

by **Clx** » Wed Feb 04, 2015 12:42 am

mlinuxguy wrote:"Scheduling while atomic" indicates that you've tried to sleep somewhere that you shouldn't - like within a spinlock-protected critical section or an interrupt handler

When I did the RT patch port to the Odroid-U I also had to fix spin_lock calls that were missed by making them raw_spinlock()
-static DEFINE_SPINLOCK(unwind_lock);
+static DEFINE_RAW_SPINLOCK(unwind_lock);
(perhaps none of those got missed under recent RT patches)

I don't know that this was done correctly:
kernel/sched/core.c.rej looks like this:

Why did you leave out the: system_state != SYSTEM_RUNNING

I'll try applying the RT patches after my new Odroid-C1 comes in, I dedicated the test one to the basement geothermal monitoring...

@mlinuxguy, thanks for suggestions!

Is there a rule to determine which lock should be spin_lock and which raw_spin_lock?
Maybe spin_locks in tsdemux\etc should be raw_spin_locks?
https://github.com/Clouded/linux-rt-odroid-c1/commit/900ece653a351dbb7a50b4bc40bde7085dcc3a81

The original file used to create rt-patch did not have system_state != SYSTEM_RUNNING, so I thought it should be leaved, but I also tried with it and it's not the case of the problem.

by **Clx** » Wed Feb 04, 2015 12:57 am

I also tried with NO_HZ disabled, the error looks a little bit different now:

Code: Select all: [ 79.751468@0] ------------[ cut here ]------------ [ 79.751472@0] kernel BUG at kernel/rtmutex.c:875! [ 79.751476@0] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM [ 79.751492@0] Modules linked in: rtl8192cu rtlwifi rtl8192c_common joydev aml_i2c ads7846 spidev spicc nls_cp437 vfat fat [ 79.751498@0] CPU: 0 PID: 83 Comm: mmcqd/0 Tainted: G W 3.10.65-rt69 #1 [ 79.751502@0] task: ef2e5d80 ti: eeafe000 task.ti: eeafe000 [ 79.751510@0] PC is at rt_spin_lock_slowlock+0x70/0x270 [ 79.751513@0] LR is at rt_spin_lock_slowlock+0x58/0x270 [ 79.751518@0] pc : [<c063ebdc>] lr : [<c063ebc4>] psr: 600f0193 [ 79.751518@0] sp : eeaffbc8 ip : ef2e5d81 fp : 00000000 [ 79.751520@0] r10: c0935d40 r9 : 00000000 r8 : ef2e5d80 [ 79.751523@0] r7 : fe001100 r6 : eeae0e40 r5 : eeafe000 r4 : eeabbaf0 [ 79.751526@0] r3 : ef2e5d80 r2 : 00000000 r1 : ef2e5d80 r0 : 00000000 [ 79.751530@0] Flags: nZCv IRQs off FIQs on Mode SVC_32 ISA ARM Segment kernel [ 79.751534@0] Control: 10c5387d Table: 2929c04a DAC: 00000015 [ 79.751536@0] [ 79.751536@0] PC: 0xc063eb5c: [ 79.751545@0] eb5c e3a00000 e8bd81f0 e3e0006d e8bd81f0 e92d4ff0 e24dd03c e1a0300d e1a04000 [ 79.751554@0] eb7c e3c35d7f e3a02000 e3a03001 e3c5503f e5958010 e58d202c e5cd3034 eb00031d [ 79.751563@0] eb9c e5943008 e1a00004 e1a01008 e3a02000 e3530000 02843004 05843004 05843008 [ 79.751572@0] ebbc e3a03001 ebe8df39 e2509000 1a000069 e594300c e3c33001 e1580003 1a000000 [ 79.751580@0] ebdc e7f001f2 e2886e45 e3a0a002 e286600c e28d7004 e1a00006 eb000330 e5983000 [ 79.751589@0] ebfc e1a00006 e5883004 e5953010 e583a000 eb000291 e1a03009 e1a01007 e1a00004 [ 79.751598@0] ec1c e1a02008 ebe8e0df e3500000 0a000000 e7f001f2 e1a00004 e1a01008 e1a02007 [ 79.751607@0] ec3c e3a03001 ebe8df19 e3500000 1a00002e e5943004 e5932020 e1540002 1a00003b [ 79.751608@0] [ 79.751608@0] LR: 0xc063eb44: [ 79.751617@0] eb44 e1a00005 eb000331 e5941010 e5816000 f57ff05f eaffffd4 e3a00000 e8bd81f0 [ 79.751626@0] eb64 e3e0006d e8bd81f0 e92d4ff0 e24dd03c e1a0300d e1a04000 e3c35d7f e3a02000 [ 79.751635@0] eb84 e3a03001 e3c5503f e5958010 e58d202c e5cd3034 eb00031d e5943008 e1a00004 [ 79.751643@0] eba4 e1a01008 e3a02000 e3530000 02843004 05843004 05843008 e3a03001 ebe8df39 [ 79.751652@0] ebc4 e2509000 1a000069 e594300c e3c33001 e1580003 1a000000 e7f001f2 e2886e45 [ 79.751661@0] ebe4 e3a0a002 e286600c e28d7004 e1a00006 eb000330 e5983000 e1a00006 e5883004 [ 79.751670@0] ec04 e5953010 e583a000 eb000291 e1a03009 e1a01007 e1a00004 e1a02008 ebe8e0df [ 79.751679@0] ec24 e3500000 0a000000 e7f001f2 e1a00004 e1a01008 e1a02007 e3a03001 ebe8df19 [ 79.751680@0] [ 79.751680@0] SP: 0xeeaffb48: [ 79.751689@0] fb48 00000001 ef2e6060 00000000 ef2e5db8 c14aaef8 ef2e6060 00000000 ef2e6060 [ 79.751698@0] fb68 c063ebe0 00000000 c000e098 00000000 e7100000 c000de0c 00000000 ef2e5d80 [ 79.751706@0] fb88 00000000 ef2e5d80 eeabbaf0 eeafe000 eeae0e40 fe001100 ef2e5d80 00000000 [ 79.751715@0] fba8 c0935d40 00000000 ef2e5d81 eeaffbc8 c063ebc4 c063ebdc 600f0193 ffffffff [ 79.751724@0] fbc8 918e9948 630f630e ef4cd280 c0643b1c eeaffbf4 c005204c 00000001 ef4cd280 [ 79.751732@0] fbe8 c092cec0 c14aaec0 ef4cd280 00000000 00000001 c0946a01 00000000 eeabba00 [ 79.751741@0] fc08 eeabbaf0 eeae0e40 fe001100 0000006e c09aa6bc c0935d40 00000000 c04664d8 [ 79.751750@0] fc28 00000000 600f0193 c092e000 c063f828 c063f828 00000000 eeae0e40 fe001100 [ 79.751751@0] [ 79.751751@0] IP: 0xef2e5d01: [ 79.751760@0] 5d00 00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000 [ 79.751768@0] 5d20 0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751776@0] 5d40 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751784@0] 5d60 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751792@0] 5d80 00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001 [ 79.751801@0] 5da0 00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000 [ 79.751809@0] 5dc0 00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012 [ 79.751818@0] 5de0 4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000 [ 79.751826@0] 5e00 00000000 00000000 00691a40 00000000 00005f84 00000000 3abe19e8 00000000 [ 79.751828@0] [ 79.751828@0] R1: 0xef2e5d00: [ 79.751836@0] 5d00 00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000 [ 79.751844@0] 5d20 0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751852@0] 5d40 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751860@0] 5d60 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751868@0] 5d80 00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001 [ 79.751877@0] 5da0 00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000 [ 79.751885@0] 5dc0 00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012 [ 79.751894@0] 5de0 4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000 [ 79.751895@0] [ 79.751895@0] R3: 0xef2e5d00: [ 79.751903@0] 5d00 00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000 [ 79.751912@0] 5d20 0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751920@0] 5d40 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751927@0] 5d60 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.751936@0] 5d80 00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001 [ 79.751944@0] 5da0 00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000 [ 79.751953@0] 5dc0 00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012 [ 79.751961@0] 5de0 4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000 [ 79.751963@0] [ 79.751963@0] R4: 0xeeabba70: [ 79.751971@0] ba70 00000000 00000000 00000000 00000000 ef2520c0 00000004 eeae0f80 00000000 [ 79.751979@0] ba90 c0998d10 00000000 00000000 ef256080 00000000 00000000 00000000 00000000 [ 79.751987@0] bab0 00000000 00000000 00000000 00000000 00000000 eeabbe00 00000000 00000000 [ 79.751996@0] bad0 00000000 00000000 00000000 00000000 00000000 00000000 eeaffd80 00000000 [ 79.752004@0] baf0 042c042b eeabbaf4 eeabbaf4 ef2e5d81 00000000 00000000 00000000 00000002 [ 79.752012@0] bb10 00000000 00000000 00000000 00000000 00000000 00000000 00000001 eeabbc00 [ 79.752021@0] bb30 00000000 00000000 00000000 00000000 00000000 00000000 0000000d e6240000 [ 79.752029@0] bb50 00000000 0517341b eeae3280 63686473 5f64735f 5f6c6c61 736e6970 736e6900 [ 79.752031@0] [ 79.752031@0] R5: 0xeeafdf80: [ 79.752039@0] df80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752047@0] dfa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752055@0] dfc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752063@0] dfe0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752072@0] e000 00000002 00010003 00000001 00000000 ef2e5d80 c094ec24 00000000 00000015 [ 79.752080@0] e020 ee1c2200 00000000 c0946a98 ef2e5d80 c14aaec0 eeafe000 c0949ef8 eeaffe4c [ 79.752088@0] e040 eeaffe00 c063df28 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752096@0] e060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752098@0] [ 79.752098@0] R6: 0xeeae0dc0: [ 79.752107@0] 0dc0 76697264 ff007265 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff [ 79.752116@0] 0de0 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff [ 79.752124@0] 0e00 00000003 00000000 ef2ec780 eeae0dc0 ef2ec811 00000000 00000000 00000000 [ 79.752133@0] 0e20 00000000 3ccb37e1 eeae0d00 00000000 00000000 a1ff0008 00003962 00000000 [ 79.752141@0] 0e40 c04664bc eeabba00 00000000 00000000 c04683f8 ef2e6e00 0000006e 00002020 [ 79.752150@0] 0e60 00000000 00000001 c086d12c eeae1280 00000000 00000000 00000000 00000000 [ 79.752158@0] 0e80 00000001 00000001 00000000 00000000 00000000 00000000 ffffffff ffffffff [ 79.752167@0] 0ea0 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff [ 79.752168@0] [ 79.752168@0] R7: 0xfe001080: [ 79.752176@0] 1080 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752184@0] 10a0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752192@0] 10c0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752200@0] 10e0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752209@0] 1100 00000001 000000f8 00000002 000003ff 00000000 000000a0 000003ff 00000003 [ 79.752217@0] 1120 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752225@0] 1140 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752233@0] 1160 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752234@0] [ 79.752234@0] R8: 0xef2e5d00: [ 79.752243@0] 5d00 00000000 00000000 00000000 00000000 ef47a068 00000000 00000020 00000000 [ 79.752251@0] 5d20 0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752259@0] 5d40 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752267@0] 5d60 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752275@0] 5d80 00000000 00000000 eeafe000 00000002 00208840 00000000 00000000 00000001 [ 79.752283@0] 5da0 00000001 00000078 00000078 00000078 00000000 c0643ab8 00000400 00400000 [ 79.752292@0] 5dc0 00000001 00000000 00000000 c14ab374 c14ab374 00000001 918e56e0 00000012 [ 79.752300@0] 5de0 4093ebe0 00000000 f2900cb1 0000000b 4093ebe0 00000000 00000158 00000000 [ 79.752302@0] [ 79.752302@0] R10: 0xc0935cc0: [ 79.752310@0] 5cc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752318@0] 5ce0 00050005 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752326@0] 5d00 00000000 00000000 00000000 00000000 00000000 00000000 ef269680 00000000 [ 79.752334@0] 5d20 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752343@0] 5d40 0000006e 0000006e 00000000 00060000 c095b328 ef202780 00000000 c0946be8 [ 79.752352@0] 5d60 00000000 00000001 c092f908 c008f2cc eeae0e40 00000000 00000020 00000000 [ 79.752360@0] 5d80 00000000 00001a00 00000000 00000000 00001a01 80001a00 c000f550 00000000 [ 79.752368@0] 5da0 4e0c4e0c 00000000 00000000 00000000 00000000 00000000 34023402 c0935dbc [ 79.752371@0] Process mmcqd/0 (pid: 83, stack limit = 0xeeafe238) [ 79.752374@0] Stack: (0xeeaffbc8 to 0xeeb00000) [ 79.752380@0] fbc0: 918e9948 630f630e ef4cd280 c0643b1c eeaffbf4 c005204c [ 79.752385@0] fbe0: 00000001 ef4cd280 c092cec0 c14aaec0 ef4cd280 00000000 00000001 c0946a01 [ 79.752391@0] fc00: 00000000 eeabba00 eeabbaf0 eeae0e40 fe001100 0000006e c09aa6bc c0935d40 [ 79.752397@0] fc20: 00000000 c04664d8 00000000 600f0193 c092e000 c063f828 c063f828 00000000 [ 79.752403@0] fc40: eeae0e40 fe001100 0000006e c09aa6bc c0935d40 c008c14c eeabba4c 00000000 [ 79.752408@0] fc60: eeafe000 c0935d40 eeae0e40 00000000 fe001100 ef217c40 00000000 e8ab0510 [ 79.752414@0] fc80: 00000000 c008c2f8 00020000 c0935d40 0000006e c008f3c0 c092c1a4 c008ba5c [ 79.752420@0] fca0: 00000100 c000f0b0 fe00110c c0946be0 eeaffcd0 c000845c c0052750 c063f828 [ 79.752426@0] fcc0: 600f0013 ffffffff eeaffd04 c000dd40 00000000 a00f0013 00000001 042b042b [ 79.752432@0] fce0: eeabbaf0 eeaffd80 ef217800 eeabbaf0 ef217c40 00000000 e8ab0510 00000000 [ 79.752437@0] fd00: 00000001 eeaffd18 c0052750 c063f828 600f0013 ffffffff eeabbaf0 c063f1bc [ 79.752443@0] fd20: eeabba00 c0467f94 eeafe000 ef2e5d80 eeaffd4c c005360c ef217800 eeaffd80 [ 79.752449@0] fd40: 00000000 ef217800 00009456 c0376fe8 0000000d ef217800 eeaffd80 eeaffd90 [ 79.752455@0] fd60: c0930400 c03770e0 00000000 eeaffdb8 ef217800 00000005 c0930400 c03771d0 [ 79.752460@0] fd80: 00000000 eeaffdb8 00000000 00000000 00000000 00000000 eeaffd98 eeaffd98 [ 79.752466@0] fda0: c0376ea8 00000000 eeaffe04 eeaf4540 eeaffe04 c03830ac 0000000d e6240000 [ 79.752472@0] fdc0: 00000000 00000000 00000000 00000000 00000195 00000005 00000000 00000000 [ 79.752477@0] fde0: 00000000 eeaffd80 eec90800 c0383d88 eeafe000 00000000 eeaffed4 c063ed94 [ 79.752483@0] fe00: eeafe000 ef2e5d80 eeafe000 ef2e5d80 00000000 c002cfa4 eeafe000 ef2e5d80 [ 79.752489@0] fe20: 00000000 c002cfa4 eeafe000 ef2e5d80 eeafe000 c002d0c8 eeafe000 ef217800 [ 79.752495@0] fe40: ef217bd4 eeafe000 eeaf44a0 eeaf464c eeaf4540 ef217bf4 eeaffed4 c037793c [ 79.752501@0] fe60: e8ab0af8 eeaf4448 eeaf4554 00000000 ef2e5d80 c0048430 eeaffe78 eeaffe78 [ 79.752507@0] fe80: eeaf441c e8ab0af8 eea12f40 eeaf441c eec90800 eeaf4550 e8ab0af8 eeaf441c [ 79.752512@0] fea0: eeaf4424 c0384e40 e8ab03b0 eeaf4554 00000000 eeaf4400 00000000 00000000 [ 79.752518@0] fec0: ee332750 e8ab0af8 ee332720 c01e9794 e8ab03b1 ee332720 e8ab0af8 eec90800 [ 79.752524@0] fee0: e8ab0af8 122c8001 eeaf4400 eeaf4400 00000001 eeaf441c eeaf4424 c0385958 [ 79.752530@0] ff00: c01e9a4c eea12f40 eeafe000 ef217800 eea12f40 c002d0c8 eeafe000 eeaf441c [ 79.752536@0] ff20: e8ab0af8 eea12f40 eeafe000 00000000 00000001 122c8001 eeaf4424 c03865a4 [ 79.752541@0] ff40: 00000000 eea9fce8 00000000 eeaf441c c03864f0 00000000 00000000 00000000 [ 79.752547@0] ff60: 00000000 c0047dc4 ffffffff 00000000 ffffffff eeaf441c 00000000 00000000 [ 79.752553@0] ff80: eeafff80 eeafff80 00000000 00000000 eeafff90 eeafff90 eeafffac eea9fce8 [ 79.752558@0] ffa0: c0047d2c 00000000 00000000 c000e218 00000000 00000000 00000000 00000000 [ 79.752563@0] ffc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 79.752569@0] ffe0: 00000000 00000000 00000000 00000000 00000013 00000000 ffffffff ffffffff [ 79.752584@0] [<c063ebdc>] (rt_spin_lock_slowlock+0x70/0x270) from [<c04664d8>] (aml_sdhc_irq+0x1c/0x294) [ 79.752596@0] [<c04664d8>] (aml_sdhc_irq+0x1c/0x294) from [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180) [ 79.752605@0] [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180) from [<c008c2f8>] (handle_irq_event+0x78/0x98) [ 79.752615@0] [<c008c2f8>] (handle_irq_event+0x78/0x98) from [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178) [ 79.752623@0] [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178) from [<c008ba5c>] (generic_handle_irq+0x24/0x38) [ 79.752633@0] [<c008ba5c>] (generic_handle_irq+0x24/0x38) from [<c000f0b0>] (handle_IRQ+0x68/0x90) [ 79.752641@0] [<c000f0b0>] (handle_IRQ+0x68/0x90) from [<c000845c>] (gic_handle_irq+0x3c/0x5c) [ 79.752648@0] [<c000845c>] (gic_handle_irq+0x3c/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 79.752651@0] Exception stack(0xeeaffcd0 to 0xeeaffd18) [ 79.752655@0] fcc0: 00000000 a00f0013 00000001 042b042b [ 79.752661@0] fce0: eeabbaf0 eeaffd80 ef217800 eeabbaf0 ef217c40 00000000 e8ab0510 00000000 [ 79.752666@0] fd00: 00000001 eeaffd18 c0052750 c063f828 600f0013 ffffffff [ 79.752674@0] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f828>] (_raw_spin_lock+0x14/0x50) [ 79.752681@0] [<c063f828>] (_raw_spin_lock+0x14/0x50) from [<c063f1bc>] (rt_spin_lock_slowunlock+0xc/0x18) [ 79.752689@0] [<c063f1bc>] (rt_spin_lock_slowunlock+0xc/0x18) from [<c0467f94>] (aml_sdhc_request+0x2d0/0x304) [ 79.752698@0] [<c0467f94>] (aml_sdhc_request+0x2d0/0x304) from [<c0376fe8>] (mmc_start_request+0x138/0x140) [ 79.752707@0] [<c0376fe8>] (mmc_start_request+0x138/0x140) from [<c03770e0>] (mmc_wait_for_req+0x68/0xd4) [ 79.752714@0] [<c03770e0>] (mmc_wait_for_req+0x68/0xd4) from [<c03771d0>] (mmc_wait_for_cmd+0x84/0x94) [ 79.752722@0] [<c03771d0>] (mmc_wait_for_cmd+0x84/0x94) from [<c03830ac>] (get_card_status+0x70/0x84) [ 79.752730@0] [<c03830ac>] (get_card_status+0x70/0x84) from [<c0383d88>] (mmc_blk_err_check+0x328/0x520) [ 79.752738@0] [<c0383d88>] (mmc_blk_err_check+0x328/0x520) from [<c037793c>] (mmc_start_req+0x180/0x358) [ 79.752745@0] [<c037793c>] (mmc_start_req+0x180/0x358) from [<c0384e40>] (mmc_blk_issue_rw_rq+0x394/0xa08) [ 79.752753@0] [<c0384e40>] (mmc_blk_issue_rw_rq+0x394/0xa08) from [<c0385958>] (mmc_blk_issue_rq+0x4a4/0x520) [ 79.752760@0] [<c0385958>] (mmc_blk_issue_rq+0x4a4/0x520) from [<c03865a4>] (mmc_queue_thread+0xb4/0x138) [ 79.752770@0] [<c03865a4>] (mmc_queue_thread+0xb4/0x138) from [<c0047dc4>] (kthread+0x98/0xa0) [ 79.752779@0] [<c0047dc4>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c) [ 79.752784@0] Code: e594300c e3c33001 e1580003 1a000000 (e7f001f2) [ 81.239321@0] ---[ end trace 0000000000000002 ]--- [ 81.239325@0] Kernel panic - not syncing: Fatal exception in interrupt [ 81.239332@1] CPU1: stopping [ 81.239338@1] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G D W 3.10.65-rt69 #1 [ 81.239356@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 81.239365@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 81.239373@1] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 81.239380@1] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 81.239382@1] Exception stack(0xef2a9f98 to 0xef2a9fe0) [ 81.239386@1] 9f80: 00000001 00000000 [ 81.239392@1] 9fa0: 0000ec3e 00000000 ef2a8000 ef2a8000 c0945fd0 c06434ec c09aa6a4 c09aa6a4 [ 81.239397@1] 9fc0: 00000001 ef2a8000 00000000 ef2a9fe0 c000f54c c000f550 60010013 ffffffff [ 81.239406@1] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000f550>] (arch_cpu_idle+0x24/0x2c) [ 81.239416@1] [<c000f550>] (arch_cpu_idle+0x24/0x2c) from [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) [ 81.239427@1] [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) from [<0082fa64>] (0x82fa64) [ 81.239431@2] CPU2: stopping [ 81.239437@2] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G D W 3.10.65-rt69 #1 [ 81.239449@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 81.239457@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 81.239464@2] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 81.239471@2] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 81.239473@2] Exception stack(0xef2abf98 to 0xef2abfe0) [ 81.239476@2] bf80: 00000002 00000000 [ 81.239482@2] bfa0: 0000d498 00000000 ef2aa000 ef2aa000 c0945fd0 c06434ec c09aa6a4 c09aa6a4 [ 81.239488@2] bfc0: 00000001 ef2aa000 00000000 ef2abfe0 c000f54c c000f550 60000013 ffffffff [ 81.239496@2] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000f550>] (arch_cpu_idle+0x24/0x2c) [ 81.239505@2] [<c000f550>] (arch_cpu_idle+0x24/0x2c) from [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) [ 81.239512@2] [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) from [<0082fa64>] (0x82fa64) [ 81.239516@3] CPU3: stopping [ 81.239521@3] CPU: 3 PID: 0 Comm: swapper/3 Tainted: G D W 3.10.65-rt69 #1 [ 81.239533@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 81.239541@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 81.239548@3] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 81.239555@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 81.239557@3] Exception stack(0xef2adf98 to 0xef2adfe0) [ 81.239561@3] df80: 00000003 00000000 [ 81.239566@3] dfa0: 0000c68e 00000000 ef2ac000 ef2ac000 c0945fd0 c06434ec c09aa6a4 c09aa6a4 [ 81.239572@3] dfc0: 00000001 ef2ac000 00000000 ef2adfe0 c000f54c c000f550 60070013 ffffffff [ 81.239580@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000f550>] (arch_cpu_idle+0x24/0x2c) [ 81.239589@3] [<c000f550>] (arch_cpu_idle+0x24/0x2c) from [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) [ 81.239596@3] [<c0069ce0>] (cpu_startup_entry+0xa8/0x148) from [<0082fa64>] (0x82fa64)

by **mlinuxguy** » Wed Feb 04, 2015 1:16 am

I would go back to your previous and fix all the:

Code: Select all: - spin_lock(..whatever..); + raw_spin_lock(..whatever..);

An example patch:
http://comments.gmane.org/gmane.linux.rt.user/8310

Get that working before changing to noHz as the tickless change ripples out with lots of effects.

by **Clx** » Wed Feb 04, 2015 1:20 am

mlinuxguy wrote:I would go back to your previous and fix all the:
Code: Select all
- spin_lock(..whatever..); + raw_spin_lock(..whatever..);

An example patch:
http://comments.gmane.org/gmane.linux.rt.user/8310

Get that working before changing to noHz as the tickless change ripples out with lots of effects.

No, I changed from NO_HZ to the normal tick system. The NO_HZ was default in the kernel.
It seems like the problem is caused by aml_sdhc, will look into that...

by **mlinuxguy** » Wed Feb 04, 2015 1:24 am

You can look at my old 3.8.rt patch set:
https://github.com/hardkernel/linux/com ... beaf50e7e7
search down for something like: include/asm-generic/cmpxchg-local.h
and see my changes to spin_locks().

If that compare link above doesn't work start at the main GIT tree and hit compare:
https://github.com/hardkernel/linux/tre ... d-3.8.y-rt

by **Clx** » Wed Feb 04, 2015 1:33 am

mlinuxguy wrote:You can look at my old 3.8.rt patch set:
https://github.com/hardkernel/linux/com ... beaf50e7e7
search down for something like: include/asm-generic/cmpxchg-local.h
and see my changes to spin_locks().

If that compare link above doesn't work start at the main GIT tree and hit compare:
https://github.com/hardkernel/linux/tre ... d-3.8.y-rt

Thanks, the link works! I'm currently analyzing it.

Are you sure that all spin_locks should be changed to raw_spin_locks? That would work, but would also create a lot of non-preemptible sections, that may affect latencies. I thought that there should be some rule which sections should be spin_locked and which raw_spin_locked.

by **mlinuxguy** » Wed Feb 04, 2015 1:42 am

What I did was find all the missing hunks, fix those.
Then I looked at the crash stacks, found any spin_lock() and converted them to raw_spin_lock().
Once done I had a stable kernel, it would crash when I did things like UART, so I would go off to those routines and find spin_locks() there to convert.

It was a iterative process, I ended up leaving a few normal spin_locks behind....

by **Clx** » Wed Feb 04, 2015 1:52 am

mlinuxguy wrote:What I did was find all the missing hunks, fix those.
Then I looked at the crash stacks, found any spin_lock() and converted them to raw_spin_lock().
Once done I had a stable kernel, it would crash when I did things like UART, so I would go off to those routines and find spin_locks() there to convert.

It was a iterative process, I ended up leaving a few normal spin_locks behind....

Understood, thanks! I'll try to fix what I can. Maybe there will be a ready rt-kernel before your odroid-c1 arrives:)

by **Clx** » Thu Feb 05, 2015 11:47 pm

Converting spinlock_t mrq_lock to raw_spinlock_t mrq_lock in aml_sd (d7c9f83a8c6afc25a4196f265ad337438524c4ad) seems to fix the previous issue, but now it's complaining about something different...

Code: Select all: [ 82.641480@2] BUG: scheduling while atomic: swapper/2/0/0x00000002 [ 82.641483@2] Modules linked in: [ 82.641484@0] [ 82.641485@2] Modules linked in: [ 82.641487@2] Preemption disabled at: [ 82.641490@0] [< (null)>] (null) [ 82.641492@2] Preemption disabled at:[< (null)>] (null) [ 82.641492@0] [ 82.641493@2] [ 82.641498@2] CPU: 2 PID: 0 Comm: swapper/2 Not tainted 3.10.65-rt69 #1 [ 82.641501@0] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.10.65-rt69 #1 [ 82.641521@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 82.641531@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 82.641541@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0) [ 82.641549@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0) [ 82.641557@0] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0) [ 82.641565@2] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0) [ 82.641573@0] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94) [ 82.641581@2] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94) [ 82.641588@0] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) [ 82.641595@2] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) [ 82.641605@0] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 82.641613@2] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 82.641621@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 82.641629@2] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 82.641638@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 82.641646@2] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 82.641657@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 82.641666@2] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 82.641674@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 82.641682@2] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 82.641690@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 82.641698@2] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 82.641708@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 82.641716@2] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 82.641726@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314) [ 82.641732@2] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04) [ 82.641736@2] bad: scheduling from the idle thread! [ 82.641740@2] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W 3.10.65-rt69 #1 [ 82.641750@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 82.641761@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0057bb8>] (dequeue_task_idle+0x1c/0x28) [ 82.641770@2] [<c0057bb8>] (dequeue_task_idle+0x1c/0x28) from [<c0051afc>] (dequeue_task+0xb8/0xc4) [ 82.641779@2] [<c0051afc>] (dequeue_task+0xb8/0xc4) from [<c063df84>] (__schedule+0x598/0x5c0) [ 82.641788@2] [<c063df84>] (__schedule+0x598/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94) [ 82.641795@2] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) [ 82.641804@2] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 82.641813@2] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 82.641822@2] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 82.641832@2] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 82.641840@2] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 82.641849@2] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 82.641858@2] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 82.641865@2] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04) [ 82.761472@0] BUG: scheduling while atomic: swapper/0/0/0x00000002 [ 82.761476@0] Modules linked in: [ 82.761480@0] Preemption disabled at:[< (null)>] (null) [ 82.761481@0] [ 82.761487@0] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.10.65-rt69 #1 [ 82.761509@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 82.761521@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0) [ 82.761532@0] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0) [ 82.761541@0] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94) [ 82.761549@0] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) [ 82.761561@0] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 82.761570@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 82.761579@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 82.761591@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 82.761600@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 82.761609@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 82.761619@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 82.761631@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314) [ 82.781466@0] BUG: scheduling while atomic: swapper/0/0/0x00000002 [ 82.781471@1] BUG: scheduling while atomic: swapper/1/0/0x00000002 [ 82.781473@1] Modules linked in: [ 82.781474@0] [ 82.781475@1] Modules linked in: [ 82.781477@1] Preemption disabled at: [ 82.781479@0] [< (null)>] (null) [ 82.781481@1] Preemption disabled at:[< (null)>] (null) [ 82.781482@0] [ 82.781483@1] [ 82.781486@0] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.10.65-rt69 #1 [ 82.781490@1] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 3.10.65-rt69 #1 [ 82.781500@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 82.781510@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 82.781518@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0) [ 82.781526@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c063360c>] (__schedule_bug+0x7c/0xa0) [ 82.781534@0] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0) [ 82.781542@1] [<c063360c>] (__schedule_bug+0x7c/0xa0) from [<c063da70>] (__schedule+0x84/0x5c0) [ 82.781550@0] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94) [ 82.781557@1] [<c063da70>] (__schedule+0x84/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94) [ 82.781564@0] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) [ 82.781571@1] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) [ 82.781579@0] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 82.781587@1] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 82.781595@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 82.781603@1] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 82.781612@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 82.781620@1] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 82.781629@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 82.781638@1] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 82.781646@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 82.781654@1] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 82.781662@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 82.781670@1] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 82.781678@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 82.781687@1] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 82.781695@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314) [ 82.781701@1] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04) [ 82.781704@1] bad: scheduling from the idle thread! [ 82.781708@1] CPU: 1 PID: 0 Comm: swapper/1 Tainted: G W 3.10.65-rt69 #1 [ 82.781717@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 82.781729@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c0057bb8>] (dequeue_task_idle+0x1c/0x28) [ 82.781737@1] [<c0057bb8>] (dequeue_task_idle+0x1c/0x28) from [<c0051afc>] (dequeue_task+0xb8/0xc4) [ 82.781746@1] [<c0051afc>] (dequeue_task+0xb8/0xc4) from [<c063df84>] (__schedule+0x598/0x5c0) [ 82.781754@1] [<c063df84>] (__schedule+0x598/0x5c0) from [<c063e0ac>] (schedule+0x7c/0x94) [ 82.781762@1] [<c063e0ac>] (schedule+0x7c/0x94) from [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) [ 82.781771@1] [<c063ec84>] (rt_spin_lock_slowlock+0x178/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 82.781779@1] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 82.781789@1] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 82.781798@1] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 82.781806@1] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 82.781815@1] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 82.781824@1] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 82.781830@1] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04) [ 96.501355@0] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 1 [ 96.501362@0] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G W 3.10.65-rt69 #1 [ 96.501384@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 96.501396@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c0632e84>] (panic+0x98/0x1f8) [ 96.501407@0] [<c0632e84>] (panic+0x98/0x1f8) from [<c008b614>] (watchdog_timer_fn+0xd8/0x270) [ 96.501418@0] [<c008b614>] (watchdog_timer_fn+0xd8/0x270) from [<c004b628>] (__run_hrtimer.isra.15+0xc8/0x148) [ 96.501427@0] [<c004b628>] (__run_hrtimer.isra.15+0xc8/0x148) from [<c004c008>] (hrtimer_interrupt+0x148/0x31c) [ 96.501436@0] [<c004c008>] (hrtimer_interrupt+0x148/0x31c) from [<c001c6a0>] (meson_timer_interrupt+0x68/0x98) [ 96.501444@0] [<c001c6a0>] (meson_timer_interrupt+0x68/0x98) from [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180) [ 96.501452@0] [<c008c14c>] (handle_irq_event_percpu+0x4c/0x180) from [<c008c2f8>] (handle_irq_event+0x78/0x98) [ 96.501462@0] [<c008c2f8>] (handle_irq_event+0x78/0x98) from [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178) [ 96.501470@0] [<c008f3c0>] (handle_fasteoi_irq+0xf4/0x178) from [<c008ba5c>] (generic_handle_irq+0x24/0x38) [ 96.501479@0] [<c008ba5c>] (generic_handle_irq+0x24/0x38) from [<c000f0b0>] (handle_IRQ+0x68/0x90) [ 96.501486@0] [<c000f0b0>] (handle_IRQ+0x68/0x90) from [<c000845c>] (gic_handle_irq+0x3c/0x5c) [ 96.501494@0] [<c000845c>] (gic_handle_irq+0x3c/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 96.501497@0] Exception stack(0xc092fe08 to 0xc092fe50) [ 96.501502@0] fe00: 00000001 00000000 00000097 00000096 c0a601f4 c092e000 [ 96.501509@0] fe20: c092fecc c0979f80 c0949ef8 c0a601d0 00000000 00177000 00004e1f c092fe50 [ 96.501512@0] fe40: c063f7c4 c063f7f4 80000013 ffffffff [ 96.501522@0] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f7f4>] (_raw_spin_lock+0x40/0x50) [ 96.501529@0] [<c063f7f4>] (_raw_spin_lock+0x40/0x50) from [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270) [ 96.501540@0] [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 96.501550@0] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 96.501559@0] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 96.501569@0] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 96.501577@0] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 96.501586@0] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 96.501597@0] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 96.501609@0] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<c08f5a30>] (start_kernel+0x2c0/0x314) [ 96.501614@3] CPU3: stopping [ 96.501620@3] CPU: 3 PID: 34 Comm: ksoftirqd/3 Tainted: G W 3.10.65-rt69 #1 [ 96.501633@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 96.501641@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 96.501648@3] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 96.501654@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 96.501657@3] Exception stack(0xef2e5d48 to 0xef2e5d90) [ 96.501662@3] 5d40: 00000000 00000000 00000099 00000096 c0a601f4 ef2e4000 [ 96.501668@3] 5d60: ef2e5e0c c0979f80 ef2bc780 c0a601d0 0493e000 00177000 00000400 ef2e5d90 [ 96.501672@3] 5d80: c0052750 c063f7f4 80000013 ffffffff [ 96.501679@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f7f4>] (_raw_spin_lock+0x40/0x50) [ 96.501686@3] [<c063f7f4>] (_raw_spin_lock+0x40/0x50) from [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270) [ 96.501695@3] [<c063eb3c>] (rt_spin_lock_slowlock+0x30/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 96.501704@3] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 96.501713@3] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c003639c>] (call_timer_fn.isra.21+0x24/0x80) [ 96.501720@3] [<c003639c>] (call_timer_fn.isra.21+0x24/0x80) from [<c0036628>] (run_timer_softirq+0x230/0x278) [ 96.501730@3] [<c0036628>] (run_timer_softirq+0x230/0x278) from [<c0030b60>] (do_current_softirqs+0x128/0x224) [ 96.501738@3] [<c0030b60>] (do_current_softirqs+0x128/0x224) from [<c0030c8c>] (run_ksoftirqd+0x30/0x50) [ 96.501746@3] [<c0030c8c>] (run_ksoftirqd+0x30/0x50) from [<c004fdbc>] (smpboot_thread_fn+0x254/0x26c) [ 96.501756@3] [<c004fdbc>] (smpboot_thread_fn+0x254/0x26c) from [<c0047dc4>] (kthread+0x98/0xa0) [ 96.501764@3] [<c0047dc4>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c) [ 96.501768@2] CPU2: stopping [ 96.501773@2] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G W 3.10.65-rt69 #1 [ 96.501785@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 96.501793@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 96.501800@2] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 96.501807@2] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 96.501810@2] Exception stack(0xef2abe38 to 0xef2abe80) [ 96.501813@2] be20: 00000001 00000000 [ 96.501819@2] be40: 00000098 00000096 c0a601f4 ef2aa000 ef2761dc ef2abe8c ef275d80 ef2abe8c [ 96.501825@2] be60: 00000002 ef275801 00000000 ef2abe80 c063f7c4 c063f7f4 800f0013 ffffffff [ 96.501832@2] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c063f7f4>] (_raw_spin_lock+0x40/0x50) [ 96.501839@2] [<c063f7f4>] (_raw_spin_lock+0x40/0x50) from [<c063ec8c>] (rt_spin_lock_slowlock+0x180/0x270) [ 96.501848@2] [<c063ec8c>] (rt_spin_lock_slowlock+0x180/0x270) from [<c0372ff0>] (choose_freq+0x40/0x1dc) [ 96.501856@2] [<c0372ff0>] (choose_freq+0x40/0x1dc) from [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) [ 96.501866@2] [<c03732a4>] (cpufreq_interactive_timer+0x118/0x374) from [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) [ 96.501875@2] [<c03735ec>] (cpufreq_interactive_idle_notifier+0xec/0x108) from [<c004d460>] (notifier_call_chain+0x44/0x84) [ 96.501884@2] [<c004d460>] (notifier_call_chain+0x44/0x84) from [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) [ 96.501892@2] [<c004dacc>] (__atomic_notifier_call_chain+0x38/0x4c) from [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) [ 96.501902@2] [<c004daf8>] (atomic_notifier_call_chain+0x18/0x20) from [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) [ 96.501909@2] [<c0069d1c>] (cpu_startup_entry+0xe4/0x148) from [<0082fa04>] (0x82fa04) [ 96.765997@0] SMP: failed to stop secondary CPUs

by **Clx** » Fri Feb 06, 2015 1:36 am

Last issue was solved by disabling frequency scaling completely. Well, it's not really needed in RT kernel anyway, but I also had to disable thermal monitor due to the dependency. Will try to enable it partially. Also, /media/boot cannot be mounted, not sure why... Currently system works without it.

The kernel has been working for almost an hour now without an error.

by **mlinuxguy** » Fri Feb 06, 2015 2:19 am

Awesome work!
If what you have turns out to be stable, run some of the rt latency tests
https://rt.wiki.kernel.org/index.php/Cyclictest

You can preserve your kernel uImage for both standard kernel and RT, copy them back and forth to the boot partition to switch between
run 'cyclictest' under each kernel and report your latencies.

by **adbensi** » Sun Feb 08, 2015 5:34 am

Very nice job,
I would like to test preempt rt on C1 too...
Did you have the source on github?
Congratulations!

by **Clx** » Mon Feb 09, 2015 8:17 pm

adbensi wrote:Very nice job,
I would like to test preempt rt on C1 too...
Did you have the source on github?
Congratulations!

The source is here - https://github.com/Clouded/linux-rt-odroid-c1
Even though the kernel runs, it's broken and requires further research.

Disabling cpu frequency scaling results in kernel not being able to load it's own modules from /lib/modules/ (the ones that I compiled together with the kernel and copied there).
Depmod prints error on every module there, like they are all broken.
Didn't have time lately to work on that, but will try to do something in a couple of days.

by **adbensi** » Tue Feb 10, 2015 6:38 am

Thanks Mikhail,
I will try it too,
Best Regards

by **adbensi** » Thu Feb 12, 2015 2:26 am

root@odroid:~# uname -a
Linux odroid 3.10.65-rt69 #1 SMP PREEMPT RT Wed Feb 11 06:24:28 ACDT 2015 armv7l armv7l armv7l GNU/Linux

root@odroid:~# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
35: 38004 0 0 0 GIC am_osd_vsync, vsync
40: 0 1536 0 0 GIC eth0
47: 0 0 0 0 GIC keypad
60: 1033 0 0 0 GIC sdio
62: 0 0 0 0 GIC dwc_otg, dwc_otg_hcd:usb2, dwc_otg_pcd
63: 0 0 0 5055975 GIC dwc_otg, dwc_otg_hcd:usb1
89: 1 0 0 0 GIC amhdmitx
92: 119287 0 0 0 GIC MESON TIMER-F
93: 0 118608 0 0 GIC MESON TIMER-G
94: 0 0 118706 0 GIC MESON TIMER-H
95: 0 0 0 115550 GIC MESON TIMER-I
99: 1 0 0 0 GIC mmc_in
101: 1 0 0 0 GIC mmc_out
107: 0 0 0 0 GIC uart_b_ttyS2:
110: 15042 0 0 0 GIC sdhc
121: 38003 0 0 0 GIC rdma, osd_rdma
122: 0 0 0 0 GIC uart_ao_ttyS0:
178: 0 0 0 0 GIC audiodsp_mailbox
182: 0 0 0 0 GIC ge2d irq
183: 0 0 0 0 GIC amhdmitx-aocec
192: 0 0 0 0 GIC Mali_GP
193: 0 0 0 0 GIC Mali_GP_MMU
194: 0 0 0 0 GIC Mali_PP_Broadcast
196: 0 0 0 0 GIC Mali_PP0
197: 0 0 0 0 GIC Mali_PP0_MMU
198: 0 0 0 0 GIC Mali_PP1
199: 0 0 0 0 GIC Mali_PP1_MMU
IPI0: 0 0 0 0 CPU wakeup interrupts
IPI1: 0 0 0 0 Timer broadcast interrupts
IPI2: 28359 70613 68972 33179 Rescheduling interrupts
IPI3: 6 8 7 8 Function call interrupts
IPI4: 126 66 109 4 Single function call interrupts
IPI5: 0 0 0 0 CPU stop interrupts
IPI6: 0 0 0 0 CPU backtrace
Err: 0

Test case: clock_nanosleep(TIME_ABSTIME), Interval 10000 microseconds,. 10000 loops, no load.
cyclictest -t1 -p 80 -n -i 10000 -l 10000
policy: fifo: loadavg: 0.89 1.21 1.10 1/316 2025
T: 0 ( 2025) P:80 I:10000 C: 10000 Min: 11 Act: 13 Avg: 19 Max: 43

Test case: clock_nanosleep(TIME_ABSTIME), Interval 10000 micro seconds,. 10000 loops, 100% load.
cyclictest -t1 -p 80 -n -i 10000 -l 10000
policy: fifo: loadavg: 1.10 0.99 1.02 3/316 2028
T: 0 ( 2027) P:80 I:10000 C: 10000 Min: 10 Act: 23 Avg: 19 Max: 56

Test case: POSIX interval timer, Interval 10000 micro seconds,. 10000 loops, no load.
cyclictest -t1 -p 80 -i 10000 -l 10000
policy: fifo: loadavg: 0.81 0.81 0.93 1/315 2030
T: 0 ( 2030) P:80 I:10000 C: 10000 Min: 29 Act: 36 Avg: 45 Max: 103

Test case: POSIX interval timer, Interval 10000 micro seconds,. 10000 loops, 100% load.
cyclictest -t1 -p 80 -i 10000 -l 10000
policy: fifo: loadavg: 0.82 0.59 0.78 4/316 2033
T: 0 ( 2032) P:80 I:10000 C: 10000 Min: 29 Act: 35 Avg: 45 Max: 103

Test case: clock_nanosleep(TIME_ABSTIME), Interval 500 micro seconds,. 100000 loops, no load.
cyclictest -t1 -p 80 -i 500 -n -l 100000
policy: fifo: loadavg: 1.78 0.93 0.88 2/315 2035
T: 0 ( 2035) P:80 I:500 C: 100000 Min: 8 Act: 12 Avg: 14 Max: 74

Test case: clock_nanosleep(TIME_ABSTIME), Interval 500 micro seconds,. 100000 loops, 100% load.
cyclictest -t1 -p 80 -i 500 -n -l 100000
policy: fifo: loadavg: 1.22 0.93 0.88 2/315 2037
T: 0 ( 2037) P:80 I:500 C: 100000 Min: 8 Act: 20 Avg: 13 Max: 53

Test case: POSIX interval timer, Interval 500 micro seconds,. 100000 loops, no load.
cyclictest -t1 -p 80 -i 500 -l 100000
policy: fifo: loadavg: 1.39 1.06 0.93 1/315 2039
T: 0 ( 2039) P:80 I:500 C: 100000 Min: 25 Act: 44 Avg: 35 Max: 153

Test case: POSIX interval timer, Interval 500 micro seconds,. 100000 loops, 100% load.
cyclictest -t1 -p 80 -i 500 -l 100000
policy: fifo: loadavg: 1.08 1.03 0.93 2/315 2041
T: 0 ( 2041) P:80 I:500 C: 100000 Min: 25 Act: 28 Avg: 35 Max: 5448

root@odroid:~# cyclictest -p 80 -t5 -n
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 1.41 1.40 1.13 4/321 2020

T: 0 ( 2001) P:80 I:1000 C: 628676 Min: 9 Act: 27 Avg: 20 Max: 82
T: 1 ( 2002) P:80 I:1500 C: 419118 Min: 9 Act: 17 Avg: 19 Max: 81
T: 2 ( 2003) P:80 I:2000 C: 314338 Min: 9 Act: 30 Avg: 18 Max: 67
T: 3 ( 2004) P:80 I:2500 C: 251470 Min: 9 Act: 23 Avg: 19 Max: 72
T: 4 ( 2005) P:80 I:3000 C: 209558 Min: 9 Act: 22 Avg: 19 Max: 75

root@odroid:~# cyclictest -h 100 -q -i 1000 -l 1000000 -n -p 90 -t
# /dev/cpu_dma_latency set to 0us
# Histogram
000000 000000 000000 000000 000000
000001 000000 000000 000000 000000
000002 000000 000000 000000 000000
000003 000000 000000 000000 000000
000004 000000 000000 000000 000000
000005 000000 000000 000000 000000
000006 000000 000000 000000 000000
000007 000000 000000 000000 000000
000008 000351 000004 000225 000090
000009 297081 123482 109351 099271
000010 421718 383068 493099 576951
000011 152816 243205 239785 198354
000012 048019 112962 073045 053963
000013 012317 053596 028244 016136
000014 008699 017104 011050 009815
000015 006982 010354 006703 006332
000016 005526 007509 004723 004354
000017 004255 006108 003797 003547
000018 003916 005847 003807 003346
000019 004148 005450 004355 004007
000020 004184 004859 004220 003980
000021 003817 004113 003118 003212
000022 003446 003358 002618 002641
000023 003027 002913 002194 002266
000024 002579 002407 001841 001828
000025 002096 002020 001554 001557
000026 001808 001762 001299 001387
000027 001646 001617 001113 001227
000028 001439 001482 000913 001086
000029 001246 001294 000691 000957
000030 001196 001100 000522 000789
000031 001045 000961 000446 000654
000032 001002 000802 000332 000513
000033 000928 000664 000268 000416
000034 000817 000550 000194 000336
000035 000679 000409 000132 000255
000036 000625 000273 000099 000173
000037 000487 000188 000062 000120
000038 000433 000125 000041 000097
000039 000344 000119 000030 000077
000040 000276 000075 000024 000060
000041 000249 000057 000019 000036
000042 000188 000039 000016 000035
000043 000116 000037 000011 000021
000044 000097 000023 000009 000028
000045 000077 000014 000008 000015
000046 000055 000017 000007 000015
000047 000058 000008 000004 000008
000048 000041 000005 000005 000007
000049 000031 000008 000004 000008
000050 000028 000001 000006 000004
000051 000014 000000 000003 000003
000052 000014 000001 000003 000003
000053 000017 000001 000001 000005
000054 000015 000002 000001 000001
000055 000012 000001 000002 000004
000056 000006 000000 000000 000001
000057 000005 000000 000000 000001
000058 000008 000001 000000 000000
000059 000002 000001 000000 000002
000060 000004 000001 000003 000001
000061 000003 000001 000000 000000
000062 000000 000002 000002 000000
000063 000002 000000 000000 000000
000064 000000 000000 000000 000000
000065 000002 000000 000000 000001
000066 000002 000000 000001 000001
000067 000000 000000 000000 000001
000068 000001 000000 000000 000000
000069 000001 000000 000000 000000
000070 000001 000000 000000 000001
000071 000000 000000 000000 000000
000072 000001 000000 000000 000001
000073 000000 000000 000000 000000
000074 000000 000000 000000 000000
000075 000000 000000 000000 000000
000076 000001 000000 000000 000000
000077 000000 000000 000000 000000
000078 000000 000000 000000 000000
000079 000001 000000 000000 000000
000080 000000 000000 000000 000000
000081 000000 000000 000000 000000
000082 000000 000000 000000 000000
000083 000000 000000 000000 000000
000084 000000 000000 000000 000000
000085 000000 000000 000000 000000
000086 000000 000000 000000 000000
000087 000000 000000 000000 000000
000088 000000 000000 000000 000000
000089 000000 000000 000000 000000
000090 000000 000000 000000 000000
000091 000000 000000 000000 000000
000092 000000 000000 000000 000000
000093 000000 000000 000000 000000
000094 000000 000000 000000 000000
000095 000000 000000 000000 000000
000096 000000 000000 000000 000000
000097 000000 000000 000000 000000
000098 000000 000000 000000 000000
000099 000000 000000 000000 000000
# Total: 001000000 001000000 001000000 001000000
# Min Latencies: 00008 00008 00008 00008
# Avg Latencies: 00010 00011 00010 00010
# Max Latencies: 00079 00062 00066 00072
# Histogram Overflows: 00000 00000 00000 00000
# Histogram Overflow at cycle number:
# Thread 0:
# Thread 1:
# Thread 2:
# Thread 3:

by **mlinuxguy** » Thu Feb 12, 2015 8:00 am

Nice! I'll have to free up some time to patch my kernel with the RT patch set and do some testing
Been too busy porting python stuff to the C1....

by **adbensi** » Thu Feb 12, 2015 8:30 am

Nicely, python support is very very important

thanks a lot!

On the 3.10.67 network driver is very stable,
I will update this driver of the 3.10.65-rt and put it on the github,
but if Your port are newest kernel, please post here

I am very happy with this board.

by **adbensi** » Thu Feb 12, 2015 8:34 am

Huhuuuu patches-3.10.67-rt71.tar.xz available

by **Clx** » Fri Feb 13, 2015 7:02 am

adbensi wrote:Huhuuuu patches-3.10.67-rt71.tar.xz available

Applied over 3.10.67 from Hardkernel's Linux Github repo - https://github.com/Clouded/linux-rt-odr ... 10.67-rt71
Haven't seen errors as on the previous one even without sdhc and freq scaling fixes. Will be testing it tomorrow.

by **adbensi** » Fri Feb 13, 2015 7:40 am

Thanks,

I updated today with your tips was very easy

I notice different performances between FULL RT and DESKTOP kernel. Is the same with you?

With full RT Kernel:

root@odroid:~# ping www.ufscar.br
PING medusa.ufscar.br (200.9.84.70) 56(84) bytes of data.
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=1 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=2 ttl=56 time=5005 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=3 ttl=56 time=1001 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=4 ttl=56 time=17.6 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=5 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=6 ttl=56 time=16.1 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=7 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=8 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=9 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=10 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=11 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=12 ttl=56 time=1000 ms
--- medusa.ufscar.br ping statistics ---
21 packets transmitted, 20 received, 4% packet loss, time 24058ms
rtt min/avg/max/mdev = 16.132/1055.138/5005.960/971.807 ms, pipe 2

root@odroid:~# uname -a
Linux odroid 3.10.67-rt71 #3 SMP PREEMPT RT Fri Feb 13 04:00:02 ACDT 2015 armv7l armv7l armv7l GNU/Linux

==============================================================================
With Desktop Kernel:

root@odroid:~# ping www.google.com.br
PING www.google.com.br (216.58.222.35) 56(84) bytes of data.
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=1 ttl=53 time=22.4 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=2 ttl=53 time=23.3 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=3 ttl=53 time=23.1 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=4 ttl=53 time=24.6 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=5 ttl=53 time=23.4 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=6 ttl=53 time=23.3 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=7 ttl=53 time=31.0 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=8 ttl=53 time=30.0 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=9 ttl=53 time=23.8 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=10 ttl=53 time=21.6 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=11 ttl=53 time=21.7 ms
64 bytes from gru09s17-in-f3.1e100.net (216.58.222.35): icmp_seq=12 ttl=53 time=23.4 ms
--- www.google.com.br ping statistics ---
12 packets transmitted, 12 received, 0% packet loss, time 11016ms
rtt min/avg/max/mdev = 21.697/24.357/31.045/2.892 ms

root@odroid:~# uname -a
Linux odroid 3.10.67-55 #1 SMP PREEMPT Tue Feb 3 12:46:15 BRST 2015 armv7l armv7l armv7l GNU/Linux

by **robroy** » Fri Feb 13, 2015 7:53 am

Once you guys have everything working, could you zip up the kernel files for me and post them? Maybe add MIDI support too

That way, I can publish an updated version of my Ubuntu Studio image (Smooth Groove) for the benefit of artistic ODROIDians!

by **adbensi** » Fri Feb 13, 2015 11:07 am

My build works with network delay,
my be Clx have different results... I will try isolate the problem and then zip one image

by **Clx** » Fri Feb 13, 2015 8:33 pm

adbensi wrote:My build works with network delay,
my be Clx have different results... I will try isolate the problem and then zip one image

Maybe there was something wrong with www.ufscar.br?
I didn't test the network yet...

by **Clx** » Fri Feb 13, 2015 8:46 pm

Network testing:

Code: Select all: root@odroid:~# ping google.com -c 10 PING google.com (173.194.122.206) 56(84) bytes of data. 64 bytes from 173.194.122.206: icmp_seq=1 ttl=57 time=16.0 ms 64 bytes from 173.194.122.206: icmp_seq=2 ttl=57 time=13.7 ms 64 bytes from 173.194.122.206: icmp_seq=3 ttl=57 time=13.6 ms 64 bytes from 173.194.122.206: icmp_seq=4 ttl=57 time=15.7 ms 64 bytes from 173.194.122.206: icmp_seq=5 ttl=57 time=13.7 ms 64 bytes from 173.194.122.206: icmp_seq=6 ttl=57 time=20.1 ms 64 bytes from 173.194.122.206: icmp_seq=7 ttl=57 time=13.7 ms 64 bytes from 173.194.122.206: icmp_seq=8 ttl=57 time=13.8 ms 64 bytes from 173.194.122.206: icmp_seq=9 ttl=57 time=13.7 ms 64 bytes from 173.194.122.206: icmp_seq=10 ttl=57 time=14.1 ms --- google.com ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9013ms rtt min/avg/max/mdev = 13.673/14.862/20.179/1.963 ms root@odroid:~# Linux odroid 3.10.67-rt71 #1 SMP PREEMPT RT Thu Feb 12 19:33:00 MSK 2015 armv7l armv7l armv7l GNU/Linux

Results of a quick cyclictest

Code: Select all: odroid@odroid:~$ sudo cyclictest -l1000000 -m -n -a0 -t1 -p99 -i400 -q # /dev/cpu_dma_latency set to 0us # Total: 001000000 # Min Latencies: 00006 # Avg Latencies: 00008 # Max Latencies: 00046 # Histogram Overflows: 00000 # Histogram Overflow at cycle number: # Thread 0:

by **Clx** » Fri Feb 13, 2015 9:06 pm

I've updated the first post with a link on compiled kernel and modules. I'd appreciate any testing feedback.

by **adbensi** » Sat Feb 14, 2015 5:24 am

Very nice, I will try it,
thanks again

by **adbensi** » Sat Feb 14, 2015 7:24 am

The same issue

Please, did You can to do diff of the file attached with Your system?
and theses options :

root@odroid:~# cat /proc/irq/63/smp_affinity
2

PowerTOP 2.5 Overview Idle stats Frequency stats Device stats Tunables
>> Bad NMI watchdog should be turned off
Bad VM writeback timeout
Bad Autosuspend for USB device USB Receiver [Logitech]
Bad Autosuspend for USB device Basic Optical Mouse [Microsoft]
Bad Using 'ondemand' cpufreq governor
Good Autosuspend for USB device USB2.0 Hub [1-1]
Good Autosuspend for USB device DWC OTG Controller [usb1]
Good Autosuspend for USB device DWC OTG Controller [usb2]
Good Wake-on-lan status for device ip6tnl0
Good Wake-on-lan status for device eth0
Good Wake-on-lan status for device sit0

I get the same problem with Your pre-build kernel too. My kernel are correct, the problem are in my system..

root@odroid:~# uname -a
Linux odroid 3.10.67-rt71 #1 SMP PREEMPT RT Thu Feb 12 19:33:00 MSK 2015 armv7l armv7l armv7l GNU/Linux
root@odroid:~# ping http://www.ufscar.br
PING medusa.ufscar.br (200.9.84.70) 56(84) bytes of data.
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=1 ttl=56 time=462 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=2 ttl=56 time=898 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=3 ttl=56 time=999 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=4 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=5 ttl=56 time=19.2 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=6 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=7 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=8 ttl=56 time=17.0 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=9 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=10 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=11 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=12 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=13 ttl=56 time=18.5 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=14 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=15 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=16 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=17 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=18 ttl=56 time=880 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=19 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=20 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=21 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=22 ttl=56 time=1000 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=23 ttl=56 time=18.2 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=24 ttl=56 time=1008 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=25 ttl=56 time=1000 ms
^C
--- medusa.ufscar.br ping statistics ---
26 packets transmitted, 25 received, 3% packet loss, time 29508ms
rtt min/avg/max/mdev = 17.016/814.006/1008.979/363.517 ms, pipe 2
root@odroid:~#

by **Clx** » Tue Feb 17, 2015 1:17 am

adbensi wrote:The same issue
Please, did You can to do diff of the file attached with Your system?
and theses options :

root@odroid:~# cat /proc/irq/63/smp_affinity
2

PowerTOP 2.5 Overview Idle stats Frequency stats Device stats Tunables
>> Bad NMI watchdog should be turned off
Bad VM writeback timeout
Bad Autosuspend for USB device USB Receiver [Logitech]
Bad Autosuspend for USB device Basic Optical Mouse [Microsoft]
Bad Using 'ondemand' cpufreq governor
Good Autosuspend for USB device USB2.0 Hub [1-1]
Good Autosuspend for USB device DWC OTG Controller [usb1]
Good Autosuspend for USB device DWC OTG Controller [usb2]
Good Wake-on-lan status for device ip6tnl0
Good Wake-on-lan status for device eth0
Good Wake-on-lan status for device sit0

I get the same problem with Your pre-build kernel too. My kernel are correct, the problem are in my system..

root@odroid:~# uname -a
Linux odroid 3.10.67-rt71 #1 SMP PREEMPT RT Thu Feb 12 19:33:00 MSK 2015 armv7l armv7l armv7l GNU/Linux
root@odroid:~# ping http://www.ufscar.br
PING medusa.ufscar.br (200.9.84.70) 56(84) bytes of data.
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=1 ttl=56 time=462 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=2 ttl=56 time=898 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=3 ttl=56 time=999 ms
64 bytes from medusa3.ufscar.br (200.9.84.70): icmp_seq=4 ttl=56 time=1000 ms
--- medusa.ufscar.br ping statistics ---
26 packets transmitted, 25 received, 3% packet loss, time 29508ms

Sysctl is attached.

Code: Select all: odroid@odroid:~# cat /proc/irq/63/smp_affinity 8

Code: Select all: Bad VM writeback timeout Bad Autosuspend for USB device 802.11n WLAN Adapter [Realtek] Bad Using 'ondemand' cpufreq governor Good Wireless Power Saving for interface wlan0 Good Autosuspend for USB device USB2.0 Hub [1-1] Good Autosuspend for USB device DWC OTG Controller [usb1] Good Autosuspend for USB device DWC OTG Controller [usb2] Good Wake-on-lan status for device ip6tnl0 Good Wake-on-lan status for device eth0 Good Wake-on-lan status for device sit0 Good Wake-on-lan status for device wlan0

What are you trying to tune something with a PowerTop?

by **Clx** » Tue Feb 17, 2015 1:20 am

I was stressing the system with rt-kernel currently and got this:

Code: Select all: [2700.100921] Kernel panic - not syncing: Watchdog detected ha

I know that's a hard lockup, but the message is cut, so I don't know in what case it happened. Kernel log has no messages past that time.

[UPDATE]

Also, right after running powertop I've stumbled upon this:

Code: Select all: [ 106.842239@3] INFO: rcu_preempt self-detected stall on CPU { 3} (t=2100 jiffies g=619 c=618 q=343) [ 106.871293@0] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 0, t=2102 jiffies, g=619, c=618, q=343) [ 169.892252@3] INFO: rcu_preempt self-detected stall on CPU { 3} (t=8405 jiffies g=619 c=618 q=374) [ 169.921759@0] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 0, t=8407 jiffies, g=619, c=618, q=374) [ 232.942258@3] INFO: rcu_preempt self-detected stall on CPU { 3} (t=14710 jiffies g=619 c=618 q=387) [ 232.971178@0] INFO: rcu_preempt detected stalls on CPUs/tasks: { 3} (detected by 0, t=14712 jiffies, g=619, c=618, q=387) [ 242.202586@2] INFO: task kworker/2:1:44 blocked for more than 120 seconds. [ 242.282408@2] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 242.295041@2] BUG: using smp_processor_id() in preemptible [00000000 00000000] code: khungtaskd/58 [ 242.305269@3] IPI backtrace for cpu 3 [ 242.305286@3] [ 242.305360@3] CPU: 3 PID: 91 Comm: irq/63-dwc_otg_ Not tainted 3.10.67-rt71 #1 [ 242.305409@3] task: ef572680 ti: eead2000 task.ti: eead2000 [ 242.305488@3] PC is at __schedule+0x44/0x5c0 [ 242.305554@3] LR is at get_parent_ip+0x10/0x2c [ 242.305617@3] pc : [<c0636418>] lr : [<c005277c>] psr: 60070113 [ 242.305617@3] sp : eead3ed0 ip : 00000000 fp : eead3f1c [ 242.305654@3] r10: 00000001 r9 : eead2000 r8 : eeb33ae0 [ 242.305698@3] r7 : eead2000 r6 : c093ea98 r5 : 00000000 r4 : ef572680 [ 242.305739@3] r3 : ef572680 r2 : 00000001 r1 : eeb33ae0 r0 : 00000000 [ 242.305789@3] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel [ 242.305835@3] Control: 10c5387d Table: 2da4404a DAC: 00000015 [ 242.305861@3] [ 242.305861@3] PC: 0xc0636398: [ 242.306015@3] 6398 eaffff99 e92d4800 e3e01102 e28db004 e3a02082 ebffff94 e3700c02 13a00000 [ 242.306157@3] 63b8 e8bd8800 e52db004 e3a02082 e28db000 e28bd000 e8bd0800 eaffff8b e92d4ff0 [ 242.306297@3] 63d8 e28db020 e24dd02c e59f6598 e1a0200d e3c27d7f e59f3590 e3c7703f e1a09007 [ 242.306438@3] 63f8 e50b3034 e50b303c e50b3040 e50b3030 e50b3044 e50b3038 e3a00001 ebe870df [ 242.306578@3] 6418 ebef1b8a e51b4034 e1a05000 e7968100 ebe97a7e e5993004 e0848008 e5987450 [ 242.306719@3] 6438 e3c33101 e3530001 0a000004 e5973230 e3530000 1a000001 e1a00007 ebffd6b5 [ 242.306859@3] 6458 ebef1b7a e51b403c e30dcfa0 e7963100 e34cc093 e59c1004 e0843003 e593257c [ 242.306999@3] 6478 e3110080 e2822001 e583257c 0a000004 e5983528 e3530000 0a000001 e2880c05 [ 242.307025@3] [ 242.307025@3] LR: 0xc00526fc: [ 242.307165@3] 26fc 0a000014 e59f205c e3540000 e284101f a1a01004 e1a012c1 e204301f e5922000 [ 242.307305@3] 271c e7922101 e1a03332 e3130001 0a000009 e1a01007 e50b501c e50b4018 eb179613 [ 242.307445@3] 273c e5953008 e24b201c e59f101c e5930018 eb00d0dc ea000001 e1a01007 eb17960b [ 242.307585@3] 275c e24bd014 e8bd88f0 c063c4e4 c0052ffc e92d4818 e28db00c e1a04000 eb00976c [ 242.307725@3] 277c e3500000 0a000002 e3a04000 e1a00004 eb009767 e1a00004 e8bd8818 e59f30d4 [ 242.307865@3] 279c e92d4830 e1a05000 e28db00c e5933000 e3530000 e1a0300d e3c34d7f e3c4403f [ 242.308005@3] 27bc e5943004 1a000028 e3530000 aa000020 eb0695f8 e3500000 08bd8830 e59f3098 [ 242.308145@3] 27dc e5933000 e3530000 18bd8830 e59f008c e3001b3e e59f2088 e59f3088 e8bd4830 [ 242.308170@3] [ 242.308170@3] SP: 0xeead3e50: [ 242.308309@3] 3e50 00000003 c00133e0 fe00110c c093ebe0 eead3e88 fe001100 eeb33ae0 c0008474 [ 242.308448@3] 3e70 c005277c c0636418 60070113 ffffffff eead3ebc c000dd40 00000000 eeb33ae0 [ 242.308586@3] 3e90 00000001 ef572680 ef572680 00000000 c093ea98 eead2000 eeb33ae0 eead2000 [ 242.308724@3] 3eb0 00000001 eead3f1c 00000000 eead3ed0 c005277c c0636418 60070113 ffffffff [ 242.308863@3] 3ed0 eead2000 ef572680 c0923ec0 c0923ec0 c0923ec0 c0923ec0 c0923ec0 c0923ec0 [ 242.309001@3] 3ef0 00000000 0ec8b426 eead2000 ef572680 00000000 eead2000 00000000 eeb33ae0 [ 242.309139@3] 3f10 c092ba00 00000001 eead3f2c c0636a94 00000001 eeb33ac0 eeb33ac0 c008c55c [ 242.309277@3] 3f30 c092ba50 c008c9f4 00000000 c008c910 c008c45c ef279d60 00000000 eeb33ac0 [ 242.309302@3] [ 242.309302@3] FP: 0xeead3e9c: [ 242.309439@3] 3e9c 00000000 c093ea98 eead2000 eeb33ae0 eead2000 00000001 eead3f1c 00000000 [ 242.309579@3] 3ebc eead3ed0 c005277c c0636418 60070113 ffffffff eead2000 ef572680 c0923ec0 [ 242.309718@3] 3edc c0923ec0 c0923ec0 c0923ec0 c0923ec0 c0923ec0 00000000 0ec8b426 eead2000 [ 242.309855@3] 3efc ef572680 00000000 eead2000 00000000 eeb33ae0 c092ba00 00000001 eead3f2c [ 242.309993@3] 3f1c c0636a94 00000001 eeb33ac0 eeb33ac0 c008c55c c092ba50 c008c9f4 00000000 [ 242.310130@3] 3f3c c008c910 c008c45c ef279d60 00000000 eeb33ac0 c008c45c 00000000 00000000 [ 242.310265@3] 3f5c 00000000 00000000 c0047df0 ffffffff 00000000 00000000 eeb33ac0 00000000 [ 242.310403@3] 3f7c 00000000 eead3f80 eead3f80 00000000 00000000 eead3f90 eead3f90 eead3fac [ 242.310428@3] [ 242.310428@3] R1: 0xeeb33a60: [ 242.310565@3] 3a60 00000000 12fabfc8 c094c57c 00000000 00000000 81240002 000034a7 00000000 [ 242.310704@3] 3a80 31627375 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.310840@3] 3aa0 00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.310978@3] 3ac0 c008c364 ef417600 00000000 00000000 c02d0db4 ef572680 0000003f 00002080 [ 242.311113@3] 3ae0 00000008 00000002 ef41765c eeb34280 00000000 00000000 00000000 00000000 [ 242.311243@3] 3b00 eeb33b80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.311371@3] 3b20 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.311512@3] 3b40 20435744 2047544f 746e6f43 6c6c6f72 00007265 00000000 ffffffff ffffffff [ 242.311537@3] [ 242.311537@3] R3: 0xef572600: [ 242.311670@3] 2600 00000000 00000000 00000000 00000000 ef2f7888 00000000 00000020 00000000 [ 242.311802@3] 2620 0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.311931@3] 2640 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.312059@3] 2660 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.312192@3] 2680 00000001 00000000 eead2000 00000003 00208040 00000000 00000000 00000001 [ 242.312327@3] 26a0 00000001 00000031 00000078 00000031 00000032 c063cb1c 00000400 00400000 [ 242.312464@3] 26c0 00000001 00000000 00000000 ef5726cc ef5726cc 00000000 6a814bb8 00000038 [ 242.312596@3] 26e0 0a62d090 00000014 00000000 00000000 00000000 00000000 00000002 00000000 [ 242.312621@3] [ 242.312621@3] R4: 0xef572600: [ 242.312755@3] 2600 00000000 00000000 00000000 00000000 ef2f7888 00000000 00000020 00000000 [ 242.312887@3] 2620 0000c350 0000c350 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.313015@3] 2640 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.313144@3] 2660 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.313276@3] 2680 00000001 00000000 eead2000 00000003 00208040 00000000 00000000 00000001 [ 242.313411@3] 26a0 00000001 00000031 00000078 00000031 00000032 c063cb1c 00000400 00400000 [ 242.313547@3] 26c0 00000001 00000000 00000000 ef5726cc ef5726cc 00000000 6a814bb8 00000038 [ 242.313680@3] 26e0 0a62d090 00000014 00000000 00000000 00000000 00000000 00000002 00000000 [ 242.313706@3] [ 242.313706@3] R6: 0xc093ea18: [ 242.313836@3] ea18 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.313965@3] ea38 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.314101@3] ea58 00000064 00008000 0000000f c14c1100 00000004 00000008 0000002c c14a1000 [ 242.314239@3] ea78 c14c10c0 00000000 00000003 c14c1080 00000001 c14c1000 c14c1040 00001000 [ 242.314377@3] ea98 00b7f000 00b87000 00b8f000 00b97000 0000f600 00000001 0002f9f9 0003efff [ 242.314513@3] eab8 0000687f 00002000 00000000 00000032 0000fffa 00000001 ef201d80 ef2f4190 [ 242.314652@3] ead8 00000001 00000001 ef201580 c14c5200 00000011 ef201500 0001ffff ef201b80 [ 242.314792@3] eaf8 00000064 00000010 c15c5200 0000ffff ef201480 00100000 ef206000 ef201e00 [ 242.314818@3] [ 242.314818@3] R7: 0xeead1f80: [ 242.314954@3] 1f80 00000001 00000000 ef3d5f80 eead1f40 ef3d7050 00000000 00000000 00000000 [ 242.315090@3] 1fa0 00000000 001172dc eead1340 00000000 00000000 a1ff0008 00003881 00000000 [ 242.315229@3] 1fc0 76697264 00007265 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.315366@3] 1fe0 00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.315500@3] 2000 00000000 00000003 00000000 00000000 ef572680 c0946c24 00000003 00000015 [ 242.315639@3] 2020 edec8400 00000000 c093ea98 ef572680 c14baec0 eead2000 ef2bc780 eead3f1c [ 242.315771@3] 2040 eead3ed0 c06368b0 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.315899@3] 2060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.315925@3] [ 242.315925@3] R8: 0xeeb33a60: [ 242.316062@3] 3a60 00000000 12fabfc8 c094c57c 00000000 00000000 81240002 000034a7 00000000 [ 242.316201@3] 3a80 31627375 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.316337@3] 3aa0 00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.316475@3] 3ac0 c008c364 ef417600 00000000 00000000 c02d0db4 ef572680 0000003f 00002080 [ 242.316608@3] 3ae0 00000008 00000002 ef41765c eeb34280 00000000 00000000 00000000 00000000 [ 242.316739@3] 3b00 eeb33b80 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.316867@3] 3b20 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.317007@3] 3b40 20435744 2047544f 746e6f43 6c6c6f72 00007265 00000000 ffffffff ffffffff [ 242.317033@3] [ 242.317033@3] R9: 0xeead1f80: [ 242.317168@3] 1f80 00000001 00000000 ef3d5f80 eead1f40 ef3d7050 00000000 00000000 00000000 [ 242.317304@3] 1fa0 00000000 001172dc eead1340 00000000 00000000 a1ff0008 00003881 00000000 [ 242.317442@3] 1fc0 76697264 00007265 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.317579@3] 1fe0 00000000 00000000 ffffffff ffffffff 00000000 00000000 ffffffff ffffffff [ 242.317713@3] 2000 00000000 00000003 00000000 00000000 ef572680 c0946c24 00000003 00000015 [ 242.317852@3] 2020 edec8400 00000000 c093ea98 ef572680 c14baec0 eead2000 ef2bc780 eead3f1c [ 242.317984@3] 2040 eead3ed0 c06368b0 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.318112@3] 2060 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.318174@3] CPU: 3 PID: 91 Comm: irq/63-dwc_otg_ Not tainted 3.10.67-rt71 #1 [ 242.318351@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 242.318478@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c00133e0>] (handle_IPI+0x170/0x208) [ 242.318586@3] [<c00133e0>] (handle_IPI+0x170/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 242.318695@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 242.318734@3] Exception stack(0xeead3e88 to 0xeead3ed0) [ 242.318817@3] 3e80: 00000000 eeb33ae0 00000001 ef572680 ef572680 00000000 [ 242.318908@3] 3ea0: c093ea98 eead2000 eeb33ae0 eead2000 00000001 eead3f1c 00000000 eead3ed0 [ 242.318966@3] 3ec0: c005277c c0636418 60070113 ffffffff [ 242.319091@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c0636418>] (__schedule+0x44/0x5c0) [ 242.319204@3] [<c0636418>] (__schedule+0x44/0x5c0) from [<c0636a94>] (schedule+0x7c/0x94) [ 242.319335@3] [<c0636a94>] (schedule+0x7c/0x94) from [<c008c55c>] (irq_thread+0x100/0x14c) [ 242.319477@3] [<c008c55c>] (irq_thread+0x100/0x14c) from [<c0047df0>] (kthread+0x98/0xa0) [ 242.319608@3] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c) [ 242.319640@1] IPI backtrace for cpu 1 [ 242.319652@1] [ 242.319713@1] CPU: 1 PID: 58 Comm: khungtaskd Not tainted 3.10.67-rt71 #1 [ 242.319759@1] task: ef2fd280 ti: ef46e000 task.ti: ef46e000 [ 242.319816@1] PC is at smp_send_all_cpu_backtrace+0x98/0xc4 [ 242.319884@1] LR is at gic_raise_softirq+0x38/0x60 [ 242.319947@1] pc : [<c0013244>] lr : [<c02098fc>] psr: 20030013 [ 242.319947@1] sp : ef46ff18 ip : 00000004 fp : 00007fd4 [ 242.319983@1] r10: c093e7f8 r9 : 00000000 r8 : 000003d4 [ 242.320026@1] r7 : c0952fd8 r6 : 00002710 r5 : 00000002 r4 : c09a2560 [ 242.320066@1] r3 : 0000000b r2 : 00000008 r1 : 00000004 r0 : 066665b0 [ 242.320114@1] Flags: nzCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment kernel [ 242.320161@1] Control: 10c5387d Table: 2978004a DAC: 00000015 [ 242.320186@1] [ 242.320186@1] PC: 0xc00131c4: [ 242.320340@1] 31c4 eb076269 e3500000 18bd80f8 e59f3084 e1a01004 e1a00005 e59f707c e3026710 [ 242.320482@1] 31e4 e5933000 e5933000 e5a1303c eb07584e e1a01005 e59f0064 eb1861e0 eb187266 [ 242.320622@1] 3204 e59f005c eb1861dd e5943034 e284003c e3a01006 e12fff33 e594303c e30605b0 [ 242.320763@1] 3224 e3400666 e313000f 1a000004 e3a00000 e59f1030 eb07583c f57ff05f e8bd80f8 [ 242.320903@1] 3244 e5973004 e12fff33 e2566001 1afffff1 eafffff5 c09a2560 c063c4ec c0952fd8 [ 242.321043@1] 3264 c07e92f4 c07e9318 c09a2598 e92d41f0 e1a05000 e59f61d8 e1a07001 eb07a7f0 [ 242.321183@1] 3284 e3550006 e1a03006 ee1d2f90 e1a04000 e7928003 e7827003 ca000005 e59f31b4 [ 242.321323@1] 32a4 e0852200 e0833102 e5932004 e2822001 e5832004 e3550006 979ff105 ea00005c [ 242.321349@1] [ 242.321349@1] LR: 0xc020987c: [ 242.321490@1] 987c e3a03001 e58c3000 ea000008 e59f2030 e28c3c01 e0820100 e5b21004 e5831000 [ 242.321631@1] 989c e1520000 e2833004 1afffffa eafffff3 e3a00001 e8bd0010 e12fff1e c093ebe0 [ 242.321771@1] 98bc c093ed6c c093ebec e92d41f0 e1a06000 e1a08001 e3a04000 e3e00000 e59f503c [ 242.321911@1] 98dc e59f703c ea000001 e7d53000 e1844003 e2802001 e3a01004 e1a00006 ebff8387 [ 242.322052@1] 98fc e5973000 e1530000 cafffff6 f57ff04f e1884804 e5953008 e5834f00 e8bd81f0 [ 242.322192@1] 991c c093ebe0 c093e7e8 e92d41f0 e1a07001 e1a04000 ebfa13de e3500000 15906018 [ 242.322333@1] 993c 01a06000 e1a00004 ebfa13d9 e3500000 15905010 01a05000 e5953024 e3530000 [ 242.322474@1] 995c 1a00000a e595301c e3530000 1a000005 e5953018 e1a00007 e12fff33 e5953014 [ 242.322499@1] [ 242.322499@1] SP: 0xef46fe98: [ 242.322638@1] fe98 00000001 c00133e0 fe00110c c093ebe0 ef46fed0 fe001100 000003d4 c0008474 [ 242.322778@1] feb8 c02098fc c0013244 20030013 ffffffff ef46ff04 c000dd40 066665b0 00000004 [ 242.322913@1] fed8 00000008 0000000b c09a2560 00000002 00002710 c0952fd8 000003d4 00000000 [ 242.323052@1] fef8 c093e7f8 00007fd4 00000004 ef46ff18 c02098fc c0013244 20030013 ffffffff [ 242.323191@1] ff18 00000001 ef2f8580 ef2f8aa8 00007fd4 ef2f8580 c008b274 ef2f8aa8 00000078 [ 242.323327@1] ff38 00000000 ef2f8778 c008b064 ef279ef0 00000000 00000000 c008b064 00000000 [ 242.323459@1] ff58 00000000 00000000 00000000 c0047df0 ffffffff 00000000 00000000 00000000 [ 242.323594@1] ff78 00000000 00000000 ef46ff80 ef46ff80 00000000 00000000 ef46ff90 ef46ff90 [ 242.323622@1] [ 242.323622@1] R4: 0xc09a24e0: [ 242.323753@1] 24e0 20070030 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.323885@1] 2500 410fc051 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.324014@1] 2520 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.324148@1] 2540 000006a0 00000000 00000000 00000000 00140014 ef7fe000 00000000 00000000 [ 242.324289@1] 2560 00002710 00124f80 c0944380 c08f4b10 c08f4ad8 c06291f4 c0629228 c001f4bc [ 242.324426@1] 2580 c001f52c c001f584 00000000 00204000 00000000 c02098c4 00000001 00000003 [ 242.324564@1] 25a0 00000000 000c000a c08e39c8 00180018 06000000 00000000 0d9fffff 00000000 [ 242.324697@1] 25c0 00000000 00000000 00000000 00000000 ef203980 ffffffff 00000000 00000002 [ 242.324723@1] [ 242.324723@1] R7: 0xc0952f58: [ 242.324865@1] 2f58 c080a37c 000001a4 c01e5f74 c01e5d88 c080a388 000001a4 c01e5f54 c01e5d48 [ 242.325005@1] 2f78 c080a394 000001a4 c01e5f48 c01e5d1c c080a3a4 000001a4 c01e5f28 c01e5cec [ 242.325145@1] 2f98 c080a3b0 000001a4 c01e5f08 c01e5cbc c0876f84 000001a4 c01e5efc c01e5c90 [ 242.325280@1] 2fb8 c080a3bc 000001a4 c01e5edc c01e5c50 00000000 00000000 00000000 00000000 [ 242.325421@1] 2fd8 c01ea510 c01ea56c c01ea588 00002710 bf0ac9d0 bf0010ec c01ee340 c06559ec [ 242.325554@1] 2ff8 00000000 00000000 00000000 c01ee338 c06559ec 00000000 00000000 00000000 [ 242.325691@1] 3018 c0998740 c0999678 c01ef1bc c01ef128 00000000 00000000 00000000 18a518a5 [ 242.325829@1] 3038 c0953038 c0953038 00000000 00000000 ef344900 ef344900 00000001 c082c224 [ 242.325856@1] [ 242.325856@1] R10: 0xc093e778: [ 242.325985@1] e778 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.326114@1] e798 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.326242@1] e7b8 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.326372@1] e7d8 00000000 00000000 00000000 00000001 00000004 00000001 00000000 ef419380 [ 242.326504@1] e7f8 00000000 00000078 00008000 00000009 00000001 00000000 00000001 0000000f [ 242.326635@1] e818 ee6b2800 00000000 0000000a 00000000 00000000 00000000 00000001 00000001 [ 242.326769@1] e838 00000001 00000001 00000001 00000000 00000d23 00000d23 00035021 01ffffff [ 242.326898@1] e858 00000000 00000001 00000001 00000001 00000001 00000001 00000000 00000000 [ 242.326958@1] CPU: 1 PID: 58 Comm: khungtaskd Not tainted 3.10.67-rt71 #1 [ 242.327106@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 242.327231@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c00133e0>] (handle_IPI+0x170/0x208) [ 242.327334@1] [<c00133e0>] (handle_IPI+0x170/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 242.327442@1] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 242.327479@1] Exception stack(0xef46fed0 to 0xef46ff18) [ 242.327546@1] fec0: 066665b0 00000004 00000008 0000000b [ 242.327638@1] fee0: c09a2560 00000002 00002710 c0952fd8 000003d4 00000000 c093e7f8 00007fd4 [ 242.327712@1] ff00: 00000004 ef46ff18 c02098fc c0013244 20030013 ffffffff [ 242.327832@1] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c0013244>] (smp_send_all_cpu_backtrace+0x98/0xc4) [ 242.327955@1] [<c0013244>] (smp_send_all_cpu_backtrace+0x98/0xc4) from [<c008b274>] (watchdog+0x210/0x280) [ 242.328085@1] [<c008b274>] (watchdog+0x210/0x280) from [<c0047df0>] (kthread+0x98/0xa0) [ 242.328213@1] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c) [ 242.328276@0] IPI backtrace for cpu 0 [ 242.328294@0] [ 242.328369@0] CPU: 0 PID: 512 Comm: in:imklog Not tainted 3.10.67-rt71 #1 [ 242.328422@0] task: eed96e00 ti: ee1f4000 task.ti: ee1f4000 [ 242.328505@0] PC is at ret_fast_syscall+0x0/0x30 [ 242.328554@0] LR is at 0x182d9 [ 242.328618@0] pc : [<c000e180>] lr : [<000182d9>] psr: 600e0013 [ 242.328618@0] sp : ee1f5fa8 ip : 00000018 fp : 00078da9 [ 242.328654@0] r10: 00000000 r9 : ee1f4000 r8 : c000e328 [ 242.328697@0] r7 : 0000004e r6 : b5208788 r5 : b5208784 r4 : b52086f0 [ 242.328737@0] r3 : 00000000 r2 : 00000000 r1 : ee1f5f98 r0 : 00000000 [ 242.328785@0] Flags: nZCv IRQs on FIQs on Mode SVC_32 ISA ARM Segment user [ 242.328833@0] Control: 10c5387d Table: 2e0ac04a DAC: 00000015 [ 242.328859@0] [ 242.328859@0] PC: 0xc000e100: [ 242.329017@0] e100 e51f0288 e5900000 ee010f10 e1a0200d eb002bb2 e1a096ad e1a09689 e3a08000 [ 242.329163@0] e120 ea00002b e281c020 e5923064 e8ac6ff0 ee0d3f70 e3a04000 ee0d4f50 e1a05000 [ 242.329307@0] e140 e2824020 e59f000c e3a01002 eb00fe6e e1a00005 e894aff0 c0943018 e320f000 [ 242.329451@0] e160 e320f000 e320f000 e320f000 e320f000 e320f000 e320f000 e320f000 e320f000 [ 242.329595@0] e180 f10c0080 e5991000 e3110007 1a000007 e59d1048 e5bde044 e16ff001 f57ff01f [ 242.329739@0] e1a0 e95d7ffe e1a00000 e28dd00c e1b0f00e e5ad0008 e1a0000d e1a02008 eb000d00 [ 242.329884@0] e1c0 e3500000 0a000006 b3a07000 e89d007f ea00002f f10c0080 e5991000 e3110007 [ 242.330028@0] e1e0 1afffff3 e59d1040 e5bde03c e16ff001 f57ff01f e95d7fff e1a00000 e28dd00c [ 242.330054@0] [ 242.330054@0] SP: 0xee1f5f28: [ 242.330198@0] 5f28 00000000 c00133e0 fe00110c c093ebe0 ee1f5f60 fe001100 c000e328 c0008474 [ 242.330342@0] 5f48 000182d9 c000e180 600e0013 ffffffff ee1f5f94 c000dd40 00000000 ee1f5f98 [ 242.330484@0] 5f68 00000000 00000000 b52086f0 b5208784 b5208788 0000004e c000e328 ee1f4000 [ 242.330627@0] 5f88 00000000 00078da9 00000018 ee1f5fa8 000182d9 c000e180 600e0013 ffffffff [ 242.330770@0] 5fa8 b52086f0 b5208784 b6575460 00000000 ffffffff 00045f51 b52086f0 b5208784 [ 242.330914@0] 5fc8 b5208788 0000004e b6577357 b6575d18 00001fa0 00078da9 00065170 b657545c [ 242.331054@0] 5fe8 00045f61 b6ea9a96 600e0030 b6575460 00000000 00000000 00000000 00000002 [ 242.331195@0] 6008 00000000 bf000000 eed94200 c0946c24 00000002 00000015 edfd4600 00000000 [ 242.331222@0] [ 242.331222@0] R1: 0xee1f5f18: [ 242.331365@0] 5f18 ee1f4000 00000000 00078da9 c000f950 00000000 c00133e0 fe00110c c093ebe0 [ 242.331510@0] 5f38 ee1f5f60 fe001100 c000e328 c0008474 000182d9 c000e180 600e0013 ffffffff [ 242.331651@0] 5f58 ee1f5f94 c000dd40 00000000 ee1f5f98 00000000 00000000 b52086f0 b5208784 [ 242.331793@0] 5f78 b5208788 0000004e c000e328 ee1f4000 00000000 00078da9 00000018 ee1f5fa8 [ 242.331936@0] 5f98 000182d9 c000e180 600e0013 ffffffff b52086f0 b5208784 b6575460 00000000 [ 242.332080@0] 5fb8 ffffffff 00045f51 b52086f0 b5208784 b5208788 0000004e b6577357 b6575d18 [ 242.332224@0] 5fd8 00001fa0 00078da9 00065170 b657545c 00045f61 b6ea9a96 600e0030 b6575460 [ 242.332362@0] 5ff8 00000000 00000000 00000000 00000002 00000000 bf000000 eed94200 c0946c24 [ 242.332391@0] [ 242.332391@0] R8: 0xc000e2a8: [ 242.332535@0] e2a8 e24fee13 3798f107 e28d1008 e3a08000 e357080f e2270000 2a000fb6 ea00e953 [ 242.332680@0] e2c8 e24ee004 e58de03c eaffffaa e1a01007 e28d0008 eb0008bb e28fe024 e1a07000 [ 242.332824@0] e2e8 e28d1008 e3570f5f 3891007f 388d0030 3798f107 e3770001 1affffeb e28dd008 [ 242.332968@0] e308 eaffffb1 e5ad0008 e1a0000d eb0008da eaffffad e320f000 c0942fac c000e9b0 [ 242.333112@0] e328 c0039fac c002f400 c0028c7c c00d42e4 c00d4350 c00d32e0 c00d3308 c0048818 [ 242.333256@0] e348 c00d32fc c00e1e94 c00e1bb4 c00da71c c00d2c20 c0048818 c00e1aa4 c00d2ea4 [ 242.333400@0] e368 c0078580 c0048818 c0048818 c00d3904 c003cd54 c00f13d0 c0048818 c0078614 [ 242.333545@0] e388 c0078a7c c0048818 c0035314 c0048818 c0048818 c003b314 c0048818 c0048818 [ 242.333570@0] [ 242.333570@0] R9: 0xee1f3f80: [ 242.333714@0] 3f80 2f9fb841 c1111452 c1311452 c1411452 d0011452 d0111452 c8111452 00000000 [ 242.333848@0] 3fa0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.333984@0] 3fc0 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.334120@0] 3fe0 00000000 00000000 00000000 00000000 00000000 00000000 2f9fd821 2f9fdc21 [ 242.334258@0] 4000 00000008 00000002 00000000 00000000 eed96e00 c0946c24 00000000 00000015 [ 242.334404@0] 4020 edfd4600 00000000 c093ea98 eed96e00 c14a2ec0 ee1f4000 ef270b00 ee1f5f84 [ 242.334542@0] 4040 ee1f5f38 c06368b0 00000000 00000000 00000000 00000000 00000000 01010000 [ 242.334676@0] 4060 00000000 b65968f0 00000000 00000000 00000000 00000000 00000000 00000000 [ 242.334736@0] CPU: 0 PID: 512 Comm: in:imklog Not tainted 3.10.67-rt71 #1 [ 242.334910@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 242.335037@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c00133e0>] (handle_IPI+0x170/0x208) [ 242.335141@0] [<c00133e0>] (handle_IPI+0x170/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 242.335248@0] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 242.335285@0] Exception stack(0xee1f5f60 to 0xee1f5fa8) [ 242.335382@0] 5f60: 00000000 ee1f5f98 00000000 00000000 b52086f0 b5208784 b5208788 0000004e [ 242.335475@0] 5f80: c000e328 ee1f4000 00000000 00078da9 00000018 ee1f5fa8 000182d9 c000e180 [ 242.335517@0] 5fa0: 600e0013 ffffffff [ 242.335637@0] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c000e180>] (ret_fast_syscall+0x0/0x30) [ 242.335978@1] Kernel panic - not syncing: hung_task: blocked tasks [ 242.336058@1] CPU: 1 PID: 58 Comm: khungtaskd Not tainted 3.10.67-rt71 #1 [ 242.336238@1] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 242.336399@1] [<c0011b40>] (show_stack+0x10/0x14) from [<c062b824>] (panic+0x98/0x1f8) [ 242.336533@1] [<c062b824>] (panic+0x98/0x1f8) from [<c008b27c>] (watchdog+0x218/0x280) [ 242.336670@1] [<c008b27c>] (watchdog+0x218/0x280) from [<c0047df0>] (kthread+0x98/0xa0) [ 242.336797@1] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c) [ 242.336868@3] CPU3: stopping [ 242.336945@3] CPU: 3 PID: 34 Comm: ksoftirqd/3 Not tainted 3.10.67-rt71 #1 [ 242.337122@3] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 242.337245@3] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 242.337349@3] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 242.337454@3] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000dd40>] (__irq_svc+0x40/0x84) [ 242.337492@3] Exception stack(0xef2e5d58 to 0xef2e5da0) [ 242.337547@3] 5d40: 40000001 ef572680 [ 242.337640@3] 5d60: 00000003 edec8434 edec8400 ef572680 00000001 ef2bc780 c14baec0 ef2e4000 [ 242.337732@3] 5d80: ef572c00 ef2e5db4 00000000 ef2e5da0 c0050df8 c0050e0c a0070113 ffffffff [ 242.337853@3] [<c000dd40>] (__irq_svc+0x40/0x84) from [<c0050e0c>] (finish_task_switch+0x3c/0xb0) [ 242.337976@3] [<c0050e0c>] (finish_task_switch+0x3c/0xb0) from [<c06368cc>] (__schedule+0x4f8/0x5c0) [ 242.338096@3] [<c06368cc>] (__schedule+0x4f8/0x5c0) from [<c0637044>] (preempt_schedule_irq+0x40/0x68) [ 242.338215@3] [<c0637044>] (preempt_schedule_irq+0x40/0x68) from [<c000dd8c>] (svc_preempt+0x8/0x20) [ 242.338329@3] [<c000dd8c>] (svc_preempt+0x8/0x20) from [<c0036a28>] (add_timer+0x0/0x18) [ 242.338467@3] [<c0036a28>] (add_timer+0x0/0x18) from [<c05628a4>] (addrconf_verify+0x36c/0x394) [ 242.338592@3] [<c05628a4>] (addrconf_verify+0x36c/0x394) from [<c00363c8>] (call_timer_fn.isra.21+0x24/0x80) [ 242.338701@3] [<c00363c8>] (call_timer_fn.isra.21+0x24/0x80) from [<c0036654>] (run_timer_softirq+0x230/0x278) [ 242.338834@3] [<c0036654>] (run_timer_softirq+0x230/0x278) from [<c0030b8c>] (do_current_softirqs+0x128/0x224) [ 242.338962@3] [<c0030b8c>] (do_current_softirqs+0x128/0x224) from [<c0030cb8>] (run_ksoftirqd+0x30/0x50) [ 242.339075@3] [<c0030cb8>] (run_ksoftirqd+0x30/0x50) from [<c004fde8>] (smpboot_thread_fn+0x254/0x26c) [ 242.339200@3] [<c004fde8>] (smpboot_thread_fn+0x254/0x26c) from [<c0047df0>] (kthread+0x98/0xa0) [ 242.339327@3] [<c0047df0>] (kthread+0x98/0xa0) from [<c000e218>] (ret_from_fork+0x14/0x3c) [ 242.339386@2] CPU2: stopping [ 242.339462@2] CPU: 2 PID: 513 Comm: rs:main Q:Reg Not tainted 3.10.67-rt71 #1 [ 242.339650@2] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 242.339775@2] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 242.339879@2] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 242.339985@2] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000df1c>] (__irq_usr+0x3c/0x60) [ 242.340023@2] Exception stack(0xee1f7fb0 to 0xee1f7ff8) [ 242.340092@2] 7fa0: 00079cc0 00077050 00000001 0007a338 [ 242.340185@2] 7fc0: 00077050 b54021a8 00079cc0 00079cc0 b5d95770 b5d9576c 00000000 00077050 [ 242.340260@2] 7fe0: 0006502c b5d95710 0003f9e7 0003e7fe 60040030 ffffffff [ 242.340320@0] CPU0: stopping [ 242.340391@0] CPU: 0 PID: 512 Comm: in:imklog Not tainted 3.10.67-rt71 #1 [ 242.340566@0] [<c0014af0>] (unwind_backtrace+0x0/0xf8) from [<c0011b40>] (show_stack+0x10/0x14) [ 242.340690@0] [<c0011b40>] (show_stack+0x10/0x14) from [<c0013338>] (handle_IPI+0xc8/0x208) [ 242.340793@0] [<c0013338>] (handle_IPI+0xc8/0x208) from [<c0008474>] (gic_handle_irq+0x54/0x5c) [ 242.340899@0] [<c0008474>] (gic_handle_irq+0x54/0x5c) from [<c000df1c>] (__irq_usr+0x3c/0x60) [ 242.340936@0] Exception stack(0xee1f5fb0 to 0xee1f5ff8) [ 242.341006@0] 5fa0: 0000004d b6577478 3165641f 63666261 [ 242.341097@0] 5fc0: 00000000 ff000000 00000004 00000000 b657741b b6575d18 00001fa0 00078da9 [ 242.341172@0] 5fe0: ffffffff b6575478 0002b9c5 b6e9b802 600f0030 ffffffff

by **adbensi** » Wed Feb 18, 2015 9:08 am

Now everytime when I try to save the changes of the powertop my system freezes ...
The problem may be wrong interrupts... if You disconnect the video and reconnect it fail
CLx, please, when possible attach the output of the command sysctl -a and cat /proc/interrupts
thanks a lot

by **Clx** » Fri Feb 20, 2015 12:40 am

adbensi wrote:Now everytime when I try to save the changes of the powertop my system freezes ...
The problem may be wrong interrupts... if You disconnect the video and reconnect it fail
CLx, please, when possible attach the output of the command sysctl -a and cat /proc/interrupts
thanks a lot

Here you go.

sysctl -a

Code: Select all: dev.cdrom.autoclose = 1 dev.cdrom.autoeject = 0 dev.cdrom.check_media = 0 dev.cdrom.debug = 0 dev.cdrom.info = CD-ROM information, Id: cdrom.c 3.20 2003/12/17 dev.cdrom.info = dev.cdrom.info = drive name: dev.cdrom.info = drive speed: dev.cdrom.info = drive # of slots: dev.cdrom.info = Can close tray: dev.cdrom.info = Can open tray: dev.cdrom.info = Can lock tray: dev.cdrom.info = Can change speed: dev.cdrom.info = Can select disk: dev.cdrom.info = Can read multisession: dev.cdrom.info = Can read MCN: dev.cdrom.info = Reports media changed: dev.cdrom.info = Can play audio: dev.cdrom.info = Can write CD-R: dev.cdrom.info = Can write CD-RW: dev.cdrom.info = Can read DVD: dev.cdrom.info = Can write DVD-R: dev.cdrom.info = Can write DVD-RAM: dev.cdrom.info = Can read MRW: dev.cdrom.info = Can write MRW: dev.cdrom.info = Can write RAM: dev.cdrom.info = dev.cdrom.info = dev.cdrom.lock = 1 dev.raid.speed_limit_max = 200000 dev.raid.speed_limit_min = 1000 dev.scsi.logging_level = 0 fs.aio-max-nr = 65536 fs.aio-nr = 0 fs.binfmt_misc.status = enabled fs.dentry-state = 9870 5920 45 0 0 0 fs.dir-notify-enable = 1 fs.epoll.max_user_watches = 260625 fs.file-max = 99801 fs.file-nr = 864 0 99801 fs.inode-nr = 9326 205 fs.inode-state = 9326 205 0 0 0 0 0 fs.inotify.max_queued_events = 16384 fs.inotify.max_user_instances = 128 fs.inotify.max_user_watches = 8192 fs.lease-break-time = 45 fs.leases-enable = 1 fs.nr_open = 1048576 fs.overflowgid = 65534 fs.overflowuid = 65534 fs.pipe-max-size = 1048576 fs.protected_hardlinks = 1 fs.protected_symlinks = 1 fs.quota.allocated_dquots = 0 fs.quota.cache_hits = 0 fs.quota.drops = 0 fs.quota.free_dquots = 0 fs.quota.lookups = 0 fs.quota.reads = 0 fs.quota.syncs = 0 fs.quota.warnings = 1 fs.quota.writes = 0 fs.suid_dumpable = 0 kernel.acct = 4 2 30 kernel.arc_firm_log = 0 kernel.auto_msgmni = 1 kernel.blk_iopoll = 1 kernel.cad_pid = 1 kernel.cap_last_cap = 36 kernel.core_pattern = core kernel.core_pipe_limit = 0 kernel.core_uses_pid = 0 kernel.ctrl-alt-del = 0 kernel.dmesg_restrict = 0 kernel.domainname = (none) kernel.hostname = odroid kernel.hotplug = kernel.hung_task_check_count = 32768 kernel.hung_task_panic = 1 kernel.hung_task_timeout_secs = 120 kernel.hung_task_warnings = 10 kernel.keys.gc_delay = 300 kernel.keys.maxbytes = 20000 kernel.keys.maxkeys = 200 kernel.keys.root_maxbytes = 20000 kernel.keys.root_maxkeys = 200 kernel.kptr_restrict = 1 kernel.max_lock_depth = 1024 kernel.modprobe = /sbin/modprobe kernel.modules_disabled = 0 kernel.msgmax = 8192 kernel.msgmnb = 16384 kernel.msgmni = 1445 kernel.ngroups_max = 65536 kernel.nmi_watchdog = 1 kernel.osrelease = 3.10.67-rt71 kernel.ostype = Linux kernel.overflowgid = 65534 kernel.overflowuid = 65534 kernel.panic = 0 kernel.panic_on_oops = 0 kernel.pid_max = 32768 kernel.poweroff_cmd = /sbin/poweroff kernel.print-fatal-signals = 0 kernel.printk = 4 4 1 7 kernel.printk_delay = 0 kernel.printk_ratelimit = 5 kernel.printk_ratelimit_burst = 10 kernel.pty.max = 4096 kernel.pty.nr = 5 kernel.pty.reserve = 1024 kernel.random.boot_id = 54195731-75b4-4928-9b46-ac1c44b29d62 kernel.random.entropy_avail = 3450 kernel.random.poolsize = 4096 kernel.random.read_wakeup_threshold = 64 kernel.random.uuid = fc0a1a47-f54d-4195-95f6-56a0284faf17 kernel.random.write_wakeup_threshold = 3072 kernel.randomize_va_space = 1 kernel.real-root-dev = 0 kernel.sched_child_runs_first = 0 kernel.sched_domain.cpu0.domain0.busy_factor = 64 kernel.sched_domain.cpu0.domain0.busy_idx = 2 kernel.sched_domain.cpu0.domain0.cache_nice_tries = 1 kernel.sched_domain.cpu0.domain0.flags = 4143 kernel.sched_domain.cpu0.domain0.forkexec_idx = 0 kernel.sched_domain.cpu0.domain0.idle_idx = 1 kernel.sched_domain.cpu0.domain0.imbalance_pct = 125 kernel.sched_domain.cpu0.domain0.max_interval = 4 kernel.sched_domain.cpu0.domain0.min_interval = 1 kernel.sched_domain.cpu0.domain0.name = CPU kernel.sched_domain.cpu0.domain0.newidle_idx = 0 kernel.sched_domain.cpu0.domain0.wake_idx = 0 kernel.sched_domain.cpu1.domain0.busy_factor = 64 kernel.sched_domain.cpu1.domain0.busy_idx = 2 kernel.sched_domain.cpu1.domain0.cache_nice_tries = 1 kernel.sched_domain.cpu1.domain0.flags = 4143 kernel.sched_domain.cpu1.domain0.forkexec_idx = 0 kernel.sched_domain.cpu1.domain0.idle_idx = 1 kernel.sched_domain.cpu1.domain0.imbalance_pct = 125 kernel.sched_domain.cpu1.domain0.max_interval = 4 kernel.sched_domain.cpu1.domain0.min_interval = 1 kernel.sched_domain.cpu1.domain0.name = CPU kernel.sched_domain.cpu1.domain0.newidle_idx = 0 kernel.sched_domain.cpu1.domain0.wake_idx = 0 kernel.sched_domain.cpu2.domain0.busy_factor = 64 kernel.sched_domain.cpu2.domain0.busy_idx = 2 kernel.sched_domain.cpu2.domain0.cache_nice_tries = 1 kernel.sched_domain.cpu2.domain0.flags = 4143 kernel.sched_domain.cpu2.domain0.forkexec_idx = 0 kernel.sched_domain.cpu2.domain0.idle_idx = 1 kernel.sched_domain.cpu2.domain0.imbalance_pct = 125 kernel.sched_domain.cpu2.domain0.max_interval = 4 kernel.sched_domain.cpu2.domain0.min_interval = 1 kernel.sched_domain.cpu2.domain0.name = CPU kernel.sched_domain.cpu2.domain0.newidle_idx = 0 kernel.sched_domain.cpu2.domain0.wake_idx = 0 kernel.sched_domain.cpu3.domain0.busy_factor = 64 kernel.sched_domain.cpu3.domain0.busy_idx = 2 kernel.sched_domain.cpu3.domain0.cache_nice_tries = 1 kernel.sched_domain.cpu3.domain0.flags = 4143 kernel.sched_domain.cpu3.domain0.forkexec_idx = 0 kernel.sched_domain.cpu3.domain0.idle_idx = 1 kernel.sched_domain.cpu3.domain0.imbalance_pct = 125 kernel.sched_domain.cpu3.domain0.max_interval = 4 kernel.sched_domain.cpu3.domain0.min_interval = 1 kernel.sched_domain.cpu3.domain0.name = CPU kernel.sched_domain.cpu3.domain0.newidle_idx = 0 kernel.sched_domain.cpu3.domain0.wake_idx = 0 kernel.sched_latency_ns = 18000000 kernel.sched_migration_cost_ns = 500000 kernel.sched_min_granularity_ns = 2250000 kernel.sched_nr_migrate = 8 kernel.sched_rr_timeslice_ms = 10 kernel.sched_rt_period_us = 1000000 kernel.sched_rt_runtime_us = 950000 kernel.sched_shares_window_ns = 10000000 kernel.sched_time_avg_ms = 1000 kernel.sched_tunable_scaling = 1 kernel.sched_wakeup_granularity_ns = 3000000 kernel.sem = 250 32000 32 128 kernel.shm_rmid_forced = 0 kernel.shmall = 2097152 kernel.shmmax = 33554432 kernel.shmmni = 4096 kernel.softlockup_panic = 1 kernel.sysrq = 176 kernel.tainted = 0 kernel.threads-max = 15597 kernel.timer_migration = 1 kernel.usermodehelper.bset = 4294967295 31 kernel.usermodehelper.inheritable = 4294967295 31 kernel.version = #1 SMP PREEMPT RT Mon Feb 16 19:40:54 MSK 2015 kernel.watchdog = 1 kernel.watchdog_thresh = 10 net.bridge.bridge-nf-call-arptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-filter-pppoe-tagged = 0 net.bridge.bridge-nf-filter-vlan-tagged = 0 net.bridge.bridge-nf-pass-vlan-input-dev = 0 net.core.dev_weight = 64 net.core.message_burst = 10 net.core.message_cost = 5 net.core.netdev_budget = 300 net.core.netdev_max_backlog = 1000 net.core.netdev_tstamp_prequeue = 1 net.core.optmem_max = 65535 net.core.rmem_default = 514400 net.core.rmem_max = 26214400 net.core.rps_sock_flow_entries = 32768 net.core.somaxconn = 128 net.core.warnings = 1 net.core.wmem_default = 514400 net.core.wmem_max = 26214400 net.core.xfrm_acq_expires = 30 net.core.xfrm_aevent_etime = 10 net.core.xfrm_aevent_rseqth = 2 net.core.xfrm_larval_drop = 1 net.ipv4.conf.all.accept_local = 0 net.ipv4.conf.all.accept_redirects = 1 net.ipv4.conf.all.accept_source_route = 0 net.ipv4.conf.all.arp_accept = 0 net.ipv4.conf.all.arp_announce = 0 net.ipv4.conf.all.arp_filter = 0 net.ipv4.conf.all.arp_ignore = 0 net.ipv4.conf.all.arp_notify = 0 net.ipv4.conf.all.bootp_relay = 0 net.ipv4.conf.all.disable_policy = 0 net.ipv4.conf.all.disable_xfrm = 0 net.ipv4.conf.all.force_igmp_version = 0 net.ipv4.conf.all.forwarding = 0 net.ipv4.conf.all.log_martians = 0 net.ipv4.conf.all.mc_forwarding = 0 net.ipv4.conf.all.medium_id = 0 net.ipv4.conf.all.promote_secondaries = 0 net.ipv4.conf.all.proxy_arp = 0 net.ipv4.conf.all.proxy_arp_pvlan = 0 net.ipv4.conf.all.route_localnet = 0 net.ipv4.conf.all.rp_filter = 1 net.ipv4.conf.all.secure_redirects = 1 net.ipv4.conf.all.send_redirects = 1 net.ipv4.conf.all.shared_media = 1 net.ipv4.conf.all.src_valid_mark = 0 net.ipv4.conf.all.tag = 0 net.ipv4.conf.default.accept_local = 0 net.ipv4.conf.default.accept_redirects = 1 net.ipv4.conf.default.accept_source_route = 1 net.ipv4.conf.default.arp_accept = 0 net.ipv4.conf.default.arp_announce = 0 net.ipv4.conf.default.arp_filter = 0 net.ipv4.conf.default.arp_ignore = 0 net.ipv4.conf.default.arp_notify = 0 net.ipv4.conf.default.bootp_relay = 0 net.ipv4.conf.default.disable_policy = 0 net.ipv4.conf.default.disable_xfrm = 0 net.ipv4.conf.default.force_igmp_version = 0 net.ipv4.conf.default.forwarding = 0 net.ipv4.conf.default.log_martians = 0 net.ipv4.conf.default.mc_forwarding = 0 net.ipv4.conf.default.medium_id = 0 net.ipv4.conf.default.promote_secondaries = 0 net.ipv4.conf.default.proxy_arp = 0 net.ipv4.conf.default.proxy_arp_pvlan = 0 net.ipv4.conf.default.route_localnet = 0 net.ipv4.conf.default.rp_filter = 1 net.ipv4.conf.default.secure_redirects = 1 net.ipv4.conf.default.send_redirects = 1 net.ipv4.conf.default.shared_media = 1 net.ipv4.conf.default.src_valid_mark = 0 net.ipv4.conf.default.tag = 0 net.ipv4.conf.eth0.accept_local = 0 net.ipv4.conf.eth0.accept_redirects = 1 net.ipv4.conf.eth0.accept_source_route = 1 net.ipv4.conf.eth0.arp_accept = 0 net.ipv4.conf.eth0.arp_announce = 0 net.ipv4.conf.eth0.arp_filter = 0 net.ipv4.conf.eth0.arp_ignore = 0 net.ipv4.conf.eth0.arp_notify = 0 net.ipv4.conf.eth0.bootp_relay = 0 net.ipv4.conf.eth0.disable_policy = 0 net.ipv4.conf.eth0.disable_xfrm = 0 net.ipv4.conf.eth0.force_igmp_version = 0 net.ipv4.conf.eth0.forwarding = 0 net.ipv4.conf.eth0.log_martians = 0 net.ipv4.conf.eth0.mc_forwarding = 0 net.ipv4.conf.eth0.medium_id = 0 net.ipv4.conf.eth0.promote_secondaries = 0 net.ipv4.conf.eth0.proxy_arp = 0 net.ipv4.conf.eth0.proxy_arp_pvlan = 0 net.ipv4.conf.eth0.route_localnet = 0 net.ipv4.conf.eth0.rp_filter = 1 net.ipv4.conf.eth0.secure_redirects = 1 net.ipv4.conf.eth0.send_redirects = 1 net.ipv4.conf.eth0.shared_media = 1 net.ipv4.conf.eth0.src_valid_mark = 0 net.ipv4.conf.eth0.tag = 0 net.ipv4.conf.ip6tnl0.accept_local = 0 net.ipv4.conf.ip6tnl0.accept_redirects = 1 net.ipv4.conf.ip6tnl0.accept_source_route = 1 net.ipv4.conf.ip6tnl0.arp_accept = 0 net.ipv4.conf.ip6tnl0.arp_announce = 0 net.ipv4.conf.ip6tnl0.arp_filter = 0 net.ipv4.conf.ip6tnl0.arp_ignore = 0 net.ipv4.conf.ip6tnl0.arp_notify = 0 net.ipv4.conf.ip6tnl0.bootp_relay = 0 net.ipv4.conf.ip6tnl0.disable_policy = 0 net.ipv4.conf.ip6tnl0.disable_xfrm = 0 net.ipv4.conf.ip6tnl0.force_igmp_version = 0 net.ipv4.conf.ip6tnl0.forwarding = 0 net.ipv4.conf.ip6tnl0.log_martians = 0 net.ipv4.conf.ip6tnl0.mc_forwarding = 0 net.ipv4.conf.ip6tnl0.medium_id = 0 net.ipv4.conf.ip6tnl0.promote_secondaries = 0 net.ipv4.conf.ip6tnl0.proxy_arp = 0 net.ipv4.conf.ip6tnl0.proxy_arp_pvlan = 0 net.ipv4.conf.ip6tnl0.route_localnet = 0 net.ipv4.conf.ip6tnl0.rp_filter = 1 net.ipv4.conf.ip6tnl0.secure_redirects = 1 net.ipv4.conf.ip6tnl0.send_redirects = 1 net.ipv4.conf.ip6tnl0.shared_media = 1 net.ipv4.conf.ip6tnl0.src_valid_mark = 0 net.ipv4.conf.ip6tnl0.tag = 0 net.ipv4.conf.lo.accept_local = 0 net.ipv4.conf.lo.accept_redirects = 1 net.ipv4.conf.lo.accept_source_route = 1 net.ipv4.conf.lo.arp_accept = 0 net.ipv4.conf.lo.arp_announce = 0 net.ipv4.conf.lo.arp_filter = 0 net.ipv4.conf.lo.arp_ignore = 0 net.ipv4.conf.lo.arp_notify = 0 net.ipv4.conf.lo.bootp_relay = 0 net.ipv4.conf.lo.disable_policy = 1 net.ipv4.conf.lo.disable_xfrm = 1 net.ipv4.conf.lo.force_igmp_version = 0 net.ipv4.conf.lo.forwarding = 0 net.ipv4.conf.lo.log_martians = 0 net.ipv4.conf.lo.mc_forwarding = 0 net.ipv4.conf.lo.medium_id = 0 net.ipv4.conf.lo.promote_secondaries = 0 net.ipv4.conf.lo.proxy_arp = 0 net.ipv4.conf.lo.proxy_arp_pvlan = 0 net.ipv4.conf.lo.route_localnet = 0 net.ipv4.conf.lo.rp_filter = 1 net.ipv4.conf.lo.secure_redirects = 1 net.ipv4.conf.lo.send_redirects = 1 net.ipv4.conf.lo.shared_media = 1 net.ipv4.conf.lo.src_valid_mark = 0 net.ipv4.conf.lo.tag = 0 net.ipv4.conf.sit0.accept_local = 0 net.ipv4.conf.sit0.accept_redirects = 1 net.ipv4.conf.sit0.accept_source_route = 1 net.ipv4.conf.sit0.arp_accept = 0 net.ipv4.conf.sit0.arp_announce = 0 net.ipv4.conf.sit0.arp_filter = 0 net.ipv4.conf.sit0.arp_ignore = 0 net.ipv4.conf.sit0.arp_notify = 0 net.ipv4.conf.sit0.bootp_relay = 0 net.ipv4.conf.sit0.disable_policy = 0 net.ipv4.conf.sit0.disable_xfrm = 0 net.ipv4.conf.sit0.force_igmp_version = 0 net.ipv4.conf.sit0.forwarding = 0 net.ipv4.conf.sit0.log_martians = 0 net.ipv4.conf.sit0.mc_forwarding = 0 net.ipv4.conf.sit0.medium_id = 0 net.ipv4.conf.sit0.promote_secondaries = 0 net.ipv4.conf.sit0.proxy_arp = 0 net.ipv4.conf.sit0.proxy_arp_pvlan = 0 net.ipv4.conf.sit0.route_localnet = 0 net.ipv4.conf.sit0.rp_filter = 1 net.ipv4.conf.sit0.secure_redirects = 1 net.ipv4.conf.sit0.send_redirects = 1 net.ipv4.conf.sit0.shared_media = 1 net.ipv4.conf.sit0.src_valid_mark = 0 net.ipv4.conf.sit0.tag = 0 net.ipv4.conf.wlan0.accept_local = 0 net.ipv4.conf.wlan0.accept_redirects = 1 net.ipv4.conf.wlan0.accept_source_route = 1 net.ipv4.conf.wlan0.arp_accept = 0 net.ipv4.conf.wlan0.arp_announce = 0 net.ipv4.conf.wlan0.arp_filter = 0 net.ipv4.conf.wlan0.arp_ignore = 0 net.ipv4.conf.wlan0.arp_notify = 0 net.ipv4.conf.wlan0.bootp_relay = 0 net.ipv4.conf.wlan0.disable_policy = 0 net.ipv4.conf.wlan0.disable_xfrm = 0 net.ipv4.conf.wlan0.force_igmp_version = 0 net.ipv4.conf.wlan0.forwarding = 0 net.ipv4.conf.wlan0.log_martians = 0 net.ipv4.conf.wlan0.mc_forwarding = 0 net.ipv4.conf.wlan0.medium_id = 0 net.ipv4.conf.wlan0.promote_secondaries = 0 net.ipv4.conf.wlan0.proxy_arp = 0 net.ipv4.conf.wlan0.proxy_arp_pvlan = 0 net.ipv4.conf.wlan0.route_localnet = 0 net.ipv4.conf.wlan0.rp_filter = 1 net.ipv4.conf.wlan0.secure_redirects = 1 net.ipv4.conf.wlan0.send_redirects = 1 net.ipv4.conf.wlan0.shared_media = 1 net.ipv4.conf.wlan0.src_valid_mark = 0 net.ipv4.conf.wlan0.tag = 0 net.ipv4.icmp_echo_ignore_all = 0 net.ipv4.icmp_echo_ignore_broadcasts = 1 net.ipv4.icmp_echo_sysrq = 0 net.ipv4.icmp_errors_use_inbound_ifaddr = 0 net.ipv4.icmp_ignore_bogus_error_responses = 1 net.ipv4.icmp_ratelimit = 1000 net.ipv4.icmp_ratemask = 6168 net.ipv4.igmp_max_memberships = 20 net.ipv4.igmp_max_msf = 10 net.ipv4.inet_peer_maxttl = 600 net.ipv4.inet_peer_minttl = 120 net.ipv4.inet_peer_threshold = 65664 net.ipv4.ip_default_ttl = 64 net.ipv4.ip_dynaddr = 0 net.ipv4.ip_early_demux = 1 net.ipv4.ip_forward = 0 net.ipv4.ip_local_port_range = 32768 61000 net.ipv4.ip_local_reserved_ports = net.ipv4.ip_no_pmtu_disc = 0 net.ipv4.ip_nonlocal_bind = 0 net.ipv4.ipfrag_high_thresh = 4194304 net.ipv4.ipfrag_low_thresh = 3145728 net.ipv4.ipfrag_max_dist = 64 net.ipv4.ipfrag_secret_interval = 600 net.ipv4.ipfrag_time = 30 net.ipv4.neigh.default.anycast_delay = 100 net.ipv4.neigh.default.app_solicit = 0 net.ipv4.neigh.default.base_reachable_time_ms = 30000 net.ipv4.neigh.default.delay_first_probe_time = 5 net.ipv4.neigh.default.gc_interval = 30 net.ipv4.neigh.default.gc_stale_time = 60 net.ipv4.neigh.default.gc_thresh1 = 128 net.ipv4.neigh.default.gc_thresh2 = 512 net.ipv4.neigh.default.gc_thresh3 = 1024 net.ipv4.neigh.default.locktime = 100 net.ipv4.neigh.default.mcast_solicit = 3 net.ipv4.neigh.default.proxy_delay = 80 net.ipv4.neigh.default.proxy_qlen = 64 net.ipv4.neigh.default.retrans_time_ms = 1000 net.ipv4.neigh.default.ucast_solicit = 3 net.ipv4.neigh.default.unres_qlen = 34 net.ipv4.neigh.default.unres_qlen_bytes = 65536 net.ipv4.neigh.eth0.anycast_delay = 100 net.ipv4.neigh.eth0.app_solicit = 0 net.ipv4.neigh.eth0.base_reachable_time_ms = 30000 net.ipv4.neigh.eth0.delay_first_probe_time = 5 net.ipv4.neigh.eth0.gc_stale_time = 60 net.ipv4.neigh.eth0.locktime = 100 net.ipv4.neigh.eth0.mcast_solicit = 3 net.ipv4.neigh.eth0.proxy_delay = 80 net.ipv4.neigh.eth0.proxy_qlen = 64 net.ipv4.neigh.eth0.retrans_time_ms = 1000 net.ipv4.neigh.eth0.ucast_solicit = 3 net.ipv4.neigh.eth0.unres_qlen = 34 net.ipv4.neigh.eth0.unres_qlen_bytes = 65536 net.ipv4.neigh.ip6tnl0.anycast_delay = 100 net.ipv4.neigh.ip6tnl0.app_solicit = 0 net.ipv4.neigh.ip6tnl0.base_reachable_time_ms = 30000 net.ipv4.neigh.ip6tnl0.delay_first_probe_time = 5 net.ipv4.neigh.ip6tnl0.gc_stale_time = 60 net.ipv4.neigh.ip6tnl0.locktime = 100 net.ipv4.neigh.ip6tnl0.mcast_solicit = 3 net.ipv4.neigh.ip6tnl0.proxy_delay = 80 net.ipv4.neigh.ip6tnl0.proxy_qlen = 64 net.ipv4.neigh.ip6tnl0.retrans_time_ms = 1000 net.ipv4.neigh.ip6tnl0.ucast_solicit = 3 net.ipv4.neigh.ip6tnl0.unres_qlen = 34 net.ipv4.neigh.ip6tnl0.unres_qlen_bytes = 65536 net.ipv4.neigh.lo.anycast_delay = 100 net.ipv4.neigh.lo.app_solicit = 0 net.ipv4.neigh.lo.base_reachable_time_ms = 30000 net.ipv4.neigh.lo.delay_first_probe_time = 5 net.ipv4.neigh.lo.gc_stale_time = 60 net.ipv4.neigh.lo.locktime = 100 net.ipv4.neigh.lo.mcast_solicit = 3 net.ipv4.neigh.lo.proxy_delay = 80 net.ipv4.neigh.lo.proxy_qlen = 64 net.ipv4.neigh.lo.retrans_time_ms = 1000 net.ipv4.neigh.lo.ucast_solicit = 3 net.ipv4.neigh.lo.unres_qlen = 34 net.ipv4.neigh.lo.unres_qlen_bytes = 65536 net.ipv4.neigh.sit0.anycast_delay = 100 net.ipv4.neigh.sit0.app_solicit = 0 net.ipv4.neigh.sit0.base_reachable_time_ms = 30000 net.ipv4.neigh.sit0.delay_first_probe_time = 5 net.ipv4.neigh.sit0.gc_stale_time = 60 net.ipv4.neigh.sit0.locktime = 100 net.ipv4.neigh.sit0.mcast_solicit = 3 net.ipv4.neigh.sit0.proxy_delay = 80 net.ipv4.neigh.sit0.proxy_qlen = 64 net.ipv4.neigh.sit0.retrans_time_ms = 1000 net.ipv4.neigh.sit0.ucast_solicit = 3 net.ipv4.neigh.sit0.unres_qlen = 34 net.ipv4.neigh.sit0.unres_qlen_bytes = 65536 net.ipv4.neigh.wlan0.anycast_delay = 100 net.ipv4.neigh.wlan0.app_solicit = 0 net.ipv4.neigh.wlan0.base_reachable_time_ms = 30000 net.ipv4.neigh.wlan0.delay_first_probe_time = 5 net.ipv4.neigh.wlan0.gc_stale_time = 60 net.ipv4.neigh.wlan0.locktime = 100 net.ipv4.neigh.wlan0.mcast_solicit = 3 net.ipv4.neigh.wlan0.proxy_delay = 80 net.ipv4.neigh.wlan0.proxy_qlen = 64 net.ipv4.neigh.wlan0.retrans_time_ms = 1000 net.ipv4.neigh.wlan0.ucast_solicit = 3 net.ipv4.neigh.wlan0.unres_qlen = 34 net.ipv4.neigh.wlan0.unres_qlen_bytes = 65536 net.ipv4.ping_group_range = 1 0 net.ipv4.route.error_burst = 500 net.ipv4.route.error_cost = 100 net.ipv4.route.gc_elasticity = 8 net.ipv4.route.gc_interval = 60 net.ipv4.route.gc_min_interval = 0 net.ipv4.route.gc_min_interval_ms = 500 net.ipv4.route.gc_thresh = -1 net.ipv4.route.gc_timeout = 300 net.ipv4.route.max_size = 2147483647 net.ipv4.route.min_adv_mss = 256 net.ipv4.route.min_pmtu = 552 net.ipv4.route.mtu_expires = 600 net.ipv4.route.redirect_load = 2 net.ipv4.route.redirect_number = 9 net.ipv4.route.redirect_silence = 2048 net.ipv4.tcp_abort_on_overflow = 0 net.ipv4.tcp_adv_win_scale = 1 net.ipv4.tcp_allowed_congestion_control = reno net.ipv4.tcp_app_win = 31 net.ipv4.tcp_available_congestion_control = reno net.ipv4.tcp_base_mss = 512 net.ipv4.tcp_challenge_ack_limit = 100 net.ipv4.tcp_congestion_control = reno net.ipv4.tcp_dsack = 1 net.ipv4.tcp_early_retrans = 3 net.ipv4.tcp_ecn = 2 net.ipv4.tcp_fack = 1 net.ipv4.tcp_fastopen = 0 net.ipv4.tcp_fastopen_key = ebb31a82-5525d762-be1ae6bf-d1df5ee1 net.ipv4.tcp_fin_timeout = 60 net.ipv4.tcp_frto = 2 net.ipv4.tcp_keepalive_intvl = 75 net.ipv4.tcp_keepalive_probes = 9 net.ipv4.tcp_keepalive_time = 7200 net.ipv4.tcp_limit_output_bytes = 131072 net.ipv4.tcp_low_latency = 0 net.ipv4.tcp_max_orphans = 4096 net.ipv4.tcp_max_ssthresh = 0 net.ipv4.tcp_max_syn_backlog = 128 net.ipv4.tcp_max_tw_buckets = 4096 net.ipv4.tcp_mem = 17178 22905 34356 net.ipv4.tcp_min_tso_segs = 2 net.ipv4.tcp_moderate_rcvbuf = 1 net.ipv4.tcp_mtu_probing = 0 net.ipv4.tcp_no_metrics_save = 0 net.ipv4.tcp_orphan_retries = 0 net.ipv4.tcp_reordering = 3 net.ipv4.tcp_retrans_collapse = 1 net.ipv4.tcp_retries1 = 3 net.ipv4.tcp_retries2 = 15 net.ipv4.tcp_rfc1337 = 0 net.ipv4.tcp_rmem = 10240 87380 26214400 net.ipv4.tcp_sack = 1 net.ipv4.tcp_slow_start_after_idle = 1 net.ipv4.tcp_stdurg = 0 net.ipv4.tcp_syn_retries = 6 net.ipv4.tcp_synack_retries = 5 net.ipv4.tcp_thin_dupack = 0 net.ipv4.tcp_thin_linear_timeouts = 0 net.ipv4.tcp_timestamps = 1 net.ipv4.tcp_tso_win_divisor = 3 net.ipv4.tcp_tw_recycle = 0 net.ipv4.tcp_tw_reuse = 0 net.ipv4.tcp_window_scaling = 1 net.ipv4.tcp_wmem = 10240 87380 26214400 net.ipv4.tcp_workaround_signed_windows = 0 net.ipv4.udp_mem = 17298 23066 34596 net.ipv4.udp_rmem_min = 131072 net.ipv4.udp_wmem_min = 131072 net.ipv4.xfrm4_gc_thresh = 1024 net.ipv6.bindv6only = 0 net.ipv6.conf.all.accept_dad = 1 net.ipv6.conf.all.accept_ra = 1 net.ipv6.conf.all.accept_ra_defrtr = 1 net.ipv6.conf.all.accept_ra_pinfo = 1 net.ipv6.conf.all.accept_ra_rtr_pref = 1 net.ipv6.conf.all.accept_redirects = 1 net.ipv6.conf.all.accept_source_route = 0 net.ipv6.conf.all.autoconf = 1 net.ipv6.conf.all.dad_transmits = 1 net.ipv6.conf.all.disable_ipv6 = 0 net.ipv6.conf.all.force_mld_version = 0 net.ipv6.conf.all.force_tllao = 0 net.ipv6.conf.all.forwarding = 0 net.ipv6.conf.all.hop_limit = 64 net.ipv6.conf.all.max_addresses = 16 net.ipv6.conf.all.max_desync_factor = 600 net.ipv6.conf.all.mtu = 1280 net.ipv6.conf.all.ndisc_notify = 0 net.ipv6.conf.all.optimistic_dad = 0 net.ipv6.conf.all.proxy_ndp = 0 net.ipv6.conf.all.regen_max_retry = 3 net.ipv6.conf.all.router_probe_interval = 60 net.ipv6.conf.all.router_solicitation_delay = 1 net.ipv6.conf.all.router_solicitation_interval = 4 net.ipv6.conf.all.router_solicitations = 3 net.ipv6.conf.all.temp_prefered_lft = 86400 net.ipv6.conf.all.temp_valid_lft = 604800 net.ipv6.conf.all.use_tempaddr = 2 net.ipv6.conf.default.accept_dad = 1 net.ipv6.conf.default.accept_ra = 1 net.ipv6.conf.default.accept_ra_defrtr = 1 net.ipv6.conf.default.accept_ra_pinfo = 1 net.ipv6.conf.default.accept_ra_rtr_pref = 1 net.ipv6.conf.default.accept_redirects = 1 net.ipv6.conf.default.accept_source_route = 0 net.ipv6.conf.default.autoconf = 1 net.ipv6.conf.default.dad_transmits = 1 net.ipv6.conf.default.disable_ipv6 = 0 net.ipv6.conf.default.force_mld_version = 0 net.ipv6.conf.default.force_tllao = 0 net.ipv6.conf.default.forwarding = 0 net.ipv6.conf.default.hop_limit = 64 net.ipv6.conf.default.max_addresses = 16 net.ipv6.conf.default.max_desync_factor = 600 net.ipv6.conf.default.mtu = 1280 net.ipv6.conf.default.ndisc_notify = 0 net.ipv6.conf.default.optimistic_dad = 0 net.ipv6.conf.default.proxy_ndp = 0 net.ipv6.conf.default.regen_max_retry = 3 net.ipv6.conf.default.router_probe_interval = 60 net.ipv6.conf.default.router_solicitation_delay = 1 net.ipv6.conf.default.router_solicitation_interval = 4 net.ipv6.conf.default.router_solicitations = 3 net.ipv6.conf.default.temp_prefered_lft = 86400 net.ipv6.conf.default.temp_valid_lft = 604800 net.ipv6.conf.default.use_tempaddr = 2 net.ipv6.conf.eth0.accept_dad = 1 net.ipv6.conf.eth0.accept_ra = 0 net.ipv6.conf.eth0.accept_ra_defrtr = 1 net.ipv6.conf.eth0.accept_ra_pinfo = 1 net.ipv6.conf.eth0.accept_ra_rtr_pref = 1 net.ipv6.conf.eth0.accept_redirects = 1 net.ipv6.conf.eth0.accept_source_route = 0 net.ipv6.conf.eth0.autoconf = 1 net.ipv6.conf.eth0.dad_transmits = 1 net.ipv6.conf.eth0.disable_ipv6 = 0 net.ipv6.conf.eth0.force_mld_version = 0 net.ipv6.conf.eth0.force_tllao = 0 net.ipv6.conf.eth0.forwarding = 0 net.ipv6.conf.eth0.hop_limit = 64 net.ipv6.conf.eth0.max_addresses = 16 net.ipv6.conf.eth0.max_desync_factor = 600 net.ipv6.conf.eth0.mtu = 1500 net.ipv6.conf.eth0.ndisc_notify = 0 net.ipv6.conf.eth0.optimistic_dad = 0 net.ipv6.conf.eth0.proxy_ndp = 0 net.ipv6.conf.eth0.regen_max_retry = 3 net.ipv6.conf.eth0.router_probe_interval = 60 net.ipv6.conf.eth0.router_solicitation_delay = 1 net.ipv6.conf.eth0.router_solicitation_interval = 4 net.ipv6.conf.eth0.router_solicitations = 3 net.ipv6.conf.eth0.temp_prefered_lft = 86400 net.ipv6.conf.eth0.temp_valid_lft = 604800 net.ipv6.conf.eth0.use_tempaddr = 0 net.ipv6.conf.ip6tnl0.accept_dad = -1 net.ipv6.conf.ip6tnl0.accept_ra = 1 net.ipv6.conf.ip6tnl0.accept_ra_defrtr = 1 net.ipv6.conf.ip6tnl0.accept_ra_pinfo = 1 net.ipv6.conf.ip6tnl0.accept_ra_rtr_pref = 1 net.ipv6.conf.ip6tnl0.accept_redirects = 1 net.ipv6.conf.ip6tnl0.accept_source_route = 0 net.ipv6.conf.ip6tnl0.autoconf = 1 net.ipv6.conf.ip6tnl0.dad_transmits = 1 net.ipv6.conf.ip6tnl0.disable_ipv6 = 0 net.ipv6.conf.ip6tnl0.force_mld_version = 0 net.ipv6.conf.ip6tnl0.force_tllao = 0 net.ipv6.conf.ip6tnl0.forwarding = 0 net.ipv6.conf.ip6tnl0.hop_limit = 64 net.ipv6.conf.ip6tnl0.max_addresses = 16 net.ipv6.conf.ip6tnl0.max_desync_factor = 600 net.ipv6.conf.ip6tnl0.mtu = 1452 net.ipv6.conf.ip6tnl0.ndisc_notify = 0 net.ipv6.conf.ip6tnl0.optimistic_dad = 0 net.ipv6.conf.ip6tnl0.proxy_ndp = 0 net.ipv6.conf.ip6tnl0.regen_max_retry = 3 net.ipv6.conf.ip6tnl0.router_probe_interval = 60 net.ipv6.conf.ip6tnl0.router_solicitation_delay = 1 net.ipv6.conf.ip6tnl0.router_solicitation_interval = 4 net.ipv6.conf.ip6tnl0.router_solicitations = 3 net.ipv6.conf.ip6tnl0.temp_prefered_lft = 86400 net.ipv6.conf.ip6tnl0.temp_valid_lft = 604800 net.ipv6.conf.ip6tnl0.use_tempaddr = -1 net.ipv6.conf.lo.accept_dad = -1 net.ipv6.conf.lo.accept_ra = 1 net.ipv6.conf.lo.accept_ra_defrtr = 1 net.ipv6.conf.lo.accept_ra_pinfo = 1 net.ipv6.conf.lo.accept_ra_rtr_pref = 1 net.ipv6.conf.lo.accept_redirects = 1 net.ipv6.conf.lo.accept_source_route = 0 net.ipv6.conf.lo.autoconf = 1 net.ipv6.conf.lo.dad_transmits = 1 net.ipv6.conf.lo.disable_ipv6 = 0 net.ipv6.conf.lo.force_mld_version = 0 net.ipv6.conf.lo.force_tllao = 0 net.ipv6.conf.lo.forwarding = 0 net.ipv6.conf.lo.hop_limit = 64 net.ipv6.conf.lo.max_addresses = 16 net.ipv6.conf.lo.max_desync_factor = 600 net.ipv6.conf.lo.mtu = 65536 net.ipv6.conf.lo.ndisc_notify = 0 net.ipv6.conf.lo.optimistic_dad = 0 net.ipv6.conf.lo.proxy_ndp = 0 net.ipv6.conf.lo.regen_max_retry = 3 net.ipv6.conf.lo.router_probe_interval = 60 net.ipv6.conf.lo.router_solicitation_delay = 1 net.ipv6.conf.lo.router_solicitation_interval = 4 net.ipv6.conf.lo.router_solicitations = 3 net.ipv6.conf.lo.temp_prefered_lft = 86400 net.ipv6.conf.lo.temp_valid_lft = 604800 net.ipv6.conf.lo.use_tempaddr = -1 net.ipv6.conf.sit0.accept_dad = -1 net.ipv6.conf.sit0.accept_ra = 1 net.ipv6.conf.sit0.accept_ra_defrtr = 1 net.ipv6.conf.sit0.accept_ra_pinfo = 1 net.ipv6.conf.sit0.accept_ra_rtr_pref = 1 net.ipv6.conf.sit0.accept_redirects = 1 net.ipv6.conf.sit0.accept_source_route = 0 net.ipv6.conf.sit0.autoconf = 1 net.ipv6.conf.sit0.dad_transmits = 1 net.ipv6.conf.sit0.disable_ipv6 = 0 net.ipv6.conf.sit0.force_mld_version = 0 net.ipv6.conf.sit0.force_tllao = 0 net.ipv6.conf.sit0.forwarding = 0 net.ipv6.conf.sit0.hop_limit = 64 net.ipv6.conf.sit0.max_addresses = 16 net.ipv6.conf.sit0.max_desync_factor = 600 net.ipv6.conf.sit0.mtu = 1480 net.ipv6.conf.sit0.ndisc_notify = 0 net.ipv6.conf.sit0.optimistic_dad = 0 net.ipv6.conf.sit0.proxy_ndp = 0 net.ipv6.conf.sit0.regen_max_retry = 3 net.ipv6.conf.sit0.router_probe_interval = 60 net.ipv6.conf.sit0.router_solicitation_delay = 1 net.ipv6.conf.sit0.router_solicitation_interval = 4 net.ipv6.conf.sit0.router_solicitations = 3 net.ipv6.conf.sit0.temp_prefered_lft = 86400 net.ipv6.conf.sit0.temp_valid_lft = 604800 net.ipv6.conf.sit0.use_tempaddr = -1 net.ipv6.conf.wlan0.accept_dad = 1 net.ipv6.conf.wlan0.accept_ra = 1 net.ipv6.conf.wlan0.accept_ra_defrtr = 1 net.ipv6.conf.wlan0.accept_ra_pinfo = 1 net.ipv6.conf.wlan0.accept_ra_rtr_pref = 1 net.ipv6.conf.wlan0.accept_redirects = 1 net.ipv6.conf.wlan0.accept_source_route = 0 net.ipv6.conf.wlan0.autoconf = 1 net.ipv6.conf.wlan0.dad_transmits = 1 net.ipv6.conf.wlan0.disable_ipv6 = 0 net.ipv6.conf.wlan0.force_mld_version = 0 net.ipv6.conf.wlan0.force_tllao = 0 net.ipv6.conf.wlan0.forwarding = 0 net.ipv6.conf.wlan0.hop_limit = 64 net.ipv6.conf.wlan0.max_addresses = 16 net.ipv6.conf.wlan0.max_desync_factor = 600 net.ipv6.conf.wlan0.mtu = 1500 net.ipv6.conf.wlan0.ndisc_notify = 0 net.ipv6.conf.wlan0.optimistic_dad = 0 net.ipv6.conf.wlan0.proxy_ndp = 0 net.ipv6.conf.wlan0.regen_max_retry = 3 net.ipv6.conf.wlan0.router_probe_interval = 60 net.ipv6.conf.wlan0.router_solicitation_delay = 1 net.ipv6.conf.wlan0.router_solicitation_interval = 4 net.ipv6.conf.wlan0.router_solicitations = 3 net.ipv6.conf.wlan0.temp_prefered_lft = 86400 net.ipv6.conf.wlan0.temp_valid_lft = 604800 net.ipv6.conf.wlan0.use_tempaddr = 2 net.ipv6.icmp.ratelimit = 1000 net.ipv6.ip6frag_high_thresh = 4194304 net.ipv6.ip6frag_low_thresh = 3145728 net.ipv6.ip6frag_secret_interval = 600 net.ipv6.ip6frag_time = 60 net.ipv6.mld_max_msf = 64 net.ipv6.neigh.default.anycast_delay = 100 net.ipv6.neigh.default.app_solicit = 0 net.ipv6.neigh.default.base_reachable_time_ms = 30000 net.ipv6.neigh.default.delay_first_probe_time = 5 net.ipv6.neigh.default.gc_interval = 30 net.ipv6.neigh.default.gc_stale_time = 60 net.ipv6.neigh.default.gc_thresh1 = 128 net.ipv6.neigh.default.gc_thresh2 = 512 net.ipv6.neigh.default.gc_thresh3 = 1024 net.ipv6.neigh.default.locktime = 0 net.ipv6.neigh.default.mcast_solicit = 3 net.ipv6.neigh.default.proxy_delay = 80 net.ipv6.neigh.default.proxy_qlen = 64 net.ipv6.neigh.default.retrans_time_ms = 1000 net.ipv6.neigh.default.ucast_solicit = 3 net.ipv6.neigh.default.unres_qlen = 34 net.ipv6.neigh.default.unres_qlen_bytes = 65536 net.ipv6.neigh.eth0.anycast_delay = 100 net.ipv6.neigh.eth0.app_solicit = 0 net.ipv6.neigh.eth0.base_reachable_time_ms = 30000 net.ipv6.neigh.eth0.delay_first_probe_time = 5 net.ipv6.neigh.eth0.gc_stale_time = 60 net.ipv6.neigh.eth0.locktime = 0 net.ipv6.neigh.eth0.mcast_solicit = 3 net.ipv6.neigh.eth0.proxy_delay = 80 net.ipv6.neigh.eth0.proxy_qlen = 64 net.ipv6.neigh.eth0.retrans_time_ms = 1000 net.ipv6.neigh.eth0.ucast_solicit = 3 net.ipv6.neigh.eth0.unres_qlen = 34 net.ipv6.neigh.eth0.unres_qlen_bytes = 65536 net.ipv6.neigh.ip6tnl0.anycast_delay = 100 net.ipv6.neigh.ip6tnl0.app_solicit = 0 net.ipv6.neigh.ip6tnl0.base_reachable_time_ms = 30000 net.ipv6.neigh.ip6tnl0.delay_first_probe_time = 5 net.ipv6.neigh.ip6tnl0.gc_stale_time = 60 net.ipv6.neigh.ip6tnl0.locktime = 0 net.ipv6.neigh.ip6tnl0.mcast_solicit = 3 net.ipv6.neigh.ip6tnl0.proxy_delay = 80 net.ipv6.neigh.ip6tnl0.proxy_qlen = 64 net.ipv6.neigh.ip6tnl0.retrans_time_ms = 1000 net.ipv6.neigh.ip6tnl0.ucast_solicit = 3 net.ipv6.neigh.ip6tnl0.unres_qlen = 34 net.ipv6.neigh.ip6tnl0.unres_qlen_bytes = 65536 net.ipv6.neigh.lo.anycast_delay = 100 net.ipv6.neigh.lo.app_solicit = 0 net.ipv6.neigh.lo.base_reachable_time_ms = 30000 net.ipv6.neigh.lo.delay_first_probe_time = 5 net.ipv6.neigh.lo.gc_stale_time = 60 net.ipv6.neigh.lo.locktime = 0 net.ipv6.neigh.lo.mcast_solicit = 3 net.ipv6.neigh.lo.proxy_delay = 80 net.ipv6.neigh.lo.proxy_qlen = 64 net.ipv6.neigh.lo.retrans_time_ms = 1000 net.ipv6.neigh.lo.ucast_solicit = 3 net.ipv6.neigh.lo.unres_qlen = 34 net.ipv6.neigh.lo.unres_qlen_bytes = 65536 net.ipv6.neigh.sit0.anycast_delay = 100 net.ipv6.neigh.sit0.app_solicit = 0 net.ipv6.neigh.sit0.base_reachable_time_ms = 30000 net.ipv6.neigh.sit0.delay_first_probe_time = 5 net.ipv6.neigh.sit0.gc_stale_time = 60 net.ipv6.neigh.sit0.locktime = 0 net.ipv6.neigh.sit0.mcast_solicit = 3 net.ipv6.neigh.sit0.proxy_delay = 80 net.ipv6.neigh.sit0.proxy_qlen = 64 net.ipv6.neigh.sit0.retrans_time_ms = 1000 net.ipv6.neigh.sit0.ucast_solicit = 3 net.ipv6.neigh.sit0.unres_qlen = 34 net.ipv6.neigh.sit0.unres_qlen_bytes = 65536 net.ipv6.neigh.wlan0.anycast_delay = 100 net.ipv6.neigh.wlan0.app_solicit = 0 net.ipv6.neigh.wlan0.base_reachable_time_ms = 30000 net.ipv6.neigh.wlan0.delay_first_probe_time = 5 net.ipv6.neigh.wlan0.gc_stale_time = 60 net.ipv6.neigh.wlan0.locktime = 0 net.ipv6.neigh.wlan0.mcast_solicit = 3 net.ipv6.neigh.wlan0.proxy_delay = 80 net.ipv6.neigh.wlan0.proxy_qlen = 64 net.ipv6.neigh.wlan0.retrans_time_ms = 1000 net.ipv6.neigh.wlan0.ucast_solicit = 3 net.ipv6.neigh.wlan0.unres_qlen = 34 net.ipv6.neigh.wlan0.unres_qlen_bytes = 65536 net.ipv6.route.gc_elasticity = 9 net.ipv6.route.gc_interval = 30 net.ipv6.route.gc_min_interval = 0 net.ipv6.route.gc_min_interval_ms = 500 net.ipv6.route.gc_thresh = 1024 net.ipv6.route.gc_timeout = 60 net.ipv6.route.max_size = 4096 net.ipv6.route.min_adv_mss = 1220 net.ipv6.route.mtu_expires = 600 net.ipv6.xfrm6_gc_thresh = 1024 net.netfilter.nf_log.0 = NONE net.netfilter.nf_log.1 = NONE net.netfilter.nf_log.10 = NONE net.netfilter.nf_log.11 = NONE net.netfilter.nf_log.12 = NONE net.netfilter.nf_log.2 = NONE net.netfilter.nf_log.3 = NONE net.netfilter.nf_log.4 = NONE net.netfilter.nf_log.5 = NONE net.netfilter.nf_log.6 = NONE net.netfilter.nf_log.7 = NONE net.netfilter.nf_log.8 = NONE net.netfilter.nf_log.9 = NONE net.phonet.local_port_range = 64 127 net.unix.max_dgram_qlen = 10 vm.admin_reserve_kbytes = 8192 vm.block_dump = 0 vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.drop_caches = 0 vm.extfrag_threshold = 200 vm.highmem_is_dirtyable = 0 vm.laptop_mode = 0 vm.legacy_va_layout = 0 vm.lowmem_reserve_ratio = 32 32 vm.max_map_count = 65530 vm.mem_management_thresh = 16384 vm.min_free_kbytes = 3436 vm.min_free_order_shift = 1 vm.mmap_min_addr = 32768 vm.nr_pdflush_threads = 0 vm.oom_dump_tasks = 1 vm.oom_kill_allocating_task = 0 vm.overcommit_memory = 0 vm.overcommit_ratio = 50 vm.page-cluster = 3 vm.panic_on_oom = 0 vm.percpu_pagelist_fraction = 0 vm.scan_unevictable_pages = 0 vm.stat_interval = 1 vm.swappiness = 60 vm.user_reserve_kbytes = 31132 vm.vfs_cache_pressure = 100

cat /proc/interrupts/

Code: Select all: CPU0 CPU1 CPU2 CPU3 35: 105 0 0 0 GIC vsync 40: 0 1 0 0 GIC eth0 47: 0 0 0 0 GIC keypad 60: 2 0 0 0 GIC sdio 62: 0 0 0 0 GIC dwc_otg, dwc_otg_hcd:usb2, dwc_otg_pcd 63: 0 0 0 489148 GIC dwc_otg, dwc_otg_hcd:usb1 89: 0 0 0 0 GIC amhdmitx 92: 11005 0 0 0 GIC MESON TIMER-F 93: 0 10906 0 0 GIC MESON TIMER-G 94: 0 0 10922 0 GIC MESON TIMER-H 95: 0 0 0 10995 GIC MESON TIMER-I 107: 0 0 0 0 GIC uart_b_ttyS2: 110: 4202 0 0 0 GIC sdhc 121: 0 0 0 0 GIC rdma 122: 1382 0 0 0 GIC uart_ao_ttyS0: 178: 0 0 0 0 GIC audiodsp_mailbox 182: 0 0 0 0 GIC ge2d irq 183: 0 0 0 0 GIC amhdmitx-aocec 192: 0 0 0 0 GIC Mali_GP 193: 0 0 0 0 GIC Mali_GP_MMU 194: 0 0 0 0 GIC Mali_PP_Broadcast 196: 0 0 0 0 GIC Mali_PP0 197: 0 0 0 0 GIC Mali_PP0_MMU 198: 0 0 0 0 GIC Mali_PP1 199: 0 0 0 0 GIC Mali_PP1_MMU IPI0: 0 0 0 0 CPU wakeup interrupts IPI1: 0 0 0 0 Timer broadcast interrupts IPI2: 13470 14729 15604 6713 Rescheduling interrupts IPI3: 4 6 3 5 Function call interrupts IPI4: 0 0 0 0 Single function call interrupts IPI5: 0 0 0 0 CPU stop interrupts IPI6: 0 0 0 0 CPU backtrace Err: 0

But what are you actually trying to achieve with powertop? Powersaving and realtime usually don't mix well...

by **Clx** » Fri Feb 20, 2015 12:52 am

Small update on rt-preempt kernel testing.
I've found that it mostly crashes when working with the file system.
Running the following crashes the system in a minute or two:

Code: Select all: stress --cpu 8 --io 4 --vm 2 --hdd 2 --vm-bytes 128M --timeout 2500s

I tried with: SD card, eMMC module, USB flash.

I repeated the stress test with default kernel and got the same crash!
It takes more time on default kernel to crash, like 5-15 minutes.
I created a topic about it here (so it doesn't get mixed with rt-preempt discussion) - viewtopic.php?f=115&t=9631

by **kinsa** » Sat Mar 07, 2015 9:13 am

Hi,

I've ported some of your patches to the latest odroid c1 tree: https://github.com/kinsamanka/linux-odr ... .10.y-rt71

So far I haven't gotten any problems due to the file system; I'm using USB HDD instead of the solid state devices.

What I've noticed is that the latency of the wired lan is horrible, sometimes it disconnects. USB lan is OK.

I'm now running an overnight test, using both the stress test and the cyclictest running at the same time. (max 4 hrs now).

BTW, I've stripped the kernel to its bare minimum, no extra amlogic drivers (video, sound, etc...).

Cheers.

UPDATE:

Here's the link to the pre-compiled kernel: http://0ptr.link/files/odroidc1_3.10.70-rt71.tgz

Do note that it has a limited set of compiled modules, you need to have serial access to your board.

And here's the cyclictest result:

Code: Select all: root@Machinekit-C1:~# cyclictest -D1d -m -n -a0 -t1 -p99 -i1000 # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 17.71 17.70 17.62 11/131 2409 T: 0 ( 2109) P:99 I:1000 C:3610316 Min: 6 Act: 14 Avg: 12 Max: 572

by **mlinuxguy** » Sat Mar 07, 2015 10:38 am

Unless you go into the amlogic ethernet driver and fix all the spin_lock calls appropriately I'm not surprised.
As an example you might have to do this:

Function qla2x00_poll does local_irq_save() before calling qla24xx_intr_handler
which has a spinlock. Since spinlocks are sleepable on rt, it is not allowed
to call them with interrupts disabled. Therefore we use local_irq_save_nort()
instead which saves flags without disabling interrupts.

So you would need to check the poll function to see if it calls the intr handler, or check your spin_locks to see if any have disabled interrupts.

by **kinsa** » Sat Mar 07, 2015 3:32 pm

If you start pinging to C1, the latency is very bad and doesn't improve over time.

If you ping from C1, the latency is a bit high initially but it improves over time (<2mins) to normal levels. The ping originating from the external host improves as well.

But once the ping from C1 is stopped, the latency from the external host deteriorates but not to the same level it had initially.

The amlogic ethernet driver doesn't have a poll function; I think it's using DMA to handle data transfer.

by **kinsa** » Wed Mar 11, 2015 12:26 am

I'm getting these errors:

Code: Select all: [ 304.168888@0] BUG: scheduling while atomic: stress/2441/0x00010001 [ 304.168906@0] Modules linked in: sg sd_mod usb_storage scsi_mod asix usbnet w1_gpio wire [ 304.168914@0] CPU: 0 PID: 2441 Comm: stress Not tainted 3.10.70-rt71-mmc #4 [ 304.168939@0] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 304.168954@0] [<c0011b04>] (show_stack+0x10/0x14) from [<c03f6844>] (__schedule_bug+0x44/0x5c) [ 304.168968@0] [<c03f6844>] (__schedule_bug+0x44/0x5c) from [<c0400ef4>] (__schedule+0x7c/0x4d4) [ 304.168977@0] [<c0400ef4>] (__schedule+0x7c/0x4d4) from [<c04015d0>] (schedule+0x7c/0x94) [ 304.168988@0] [<c04015d0>] (schedule+0x7c/0x94) from [<c04023a8>] (rt_spin_lock_slowlock+0x180/0x274) [ 304.169003@0] [<c04023a8>] (rt_spin_lock_slowlock+0x180/0x274) from [<c0328fec>] (aml_sdhc_irq+0x1c/0x290) [ 304.169016@0] [<c0328fec>] (aml_sdhc_irq+0x1c/0x290) from [<c0085820>] (handle_irq_event_percpu+0x68/0x194) [ 304.169027@0] [<c0085820>] (handle_irq_event_percpu+0x68/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8) [ 304.169037@0] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) [ 304.169047@0] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40) [ 304.169061@0] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c) [ 304.169071@0] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c) [ 304.169080@0] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60) [ 304.169084@0] Exception stack(0xedbcdfb0 to 0xedbcdff8) [ 304.169090@0] dfa0: b6e4e028 00000000 00000000 00000001 [ 304.169097@0] dfc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000e 00000000 [ 304.169103@0] dfe0: 000144b4 beeb9b70 b6d9b9e9 b6d9ba06 60000030 ffffffff [ 312.608864@3] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0 [ 312.608876@3] CPU: 3 PID: 76 Comm: irq/63-dwc_otg_ Tainted: G W 3.10.70-rt71-mmc #4 [ 312.608903@3] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 312.608917@3] [<c0011b04>] (show_stack+0x10/0x14) from [<c03f5e08>] (panic+0x94/0x1ec) [ 312.608931@3] [<c03f5e08>] (panic+0x94/0x1ec) from [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108) [ 312.608942@3] [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108) from [<c0084db0>] (watchdog_timer_fn+0x38/0x174) [ 312.608956@3] [<c0084db0>] (watchdog_timer_fn+0x38/0x174) from [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c) [ 312.608967@3] [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c) from [<c004c0ec>] (hrtimer_interrupt+0x148/0x338) [ 312.608978@3] [<c004c0ec>] (hrtimer_interrupt+0x148/0x338) from [<c001c5c0>] (meson_timer_interrupt+0x70/0x88) [ 312.608988@3] [<c001c5c0>] (meson_timer_interrupt+0x70/0x88) from [<c0085820>] (handle_irq_event_percpu+0x68/0x194) [ 312.608998@3] [<c0085820>] (handle_irq_event_percpu+0x68/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8) [ 312.609009@3] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) [ 312.609019@3] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40) [ 312.609032@3] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c) [ 312.609042@3] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c) [ 312.609052@3] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000db00>] (__irq_svc+0x40/0x84) [ 312.609055@3] Exception stack(0xeeaa1ee0 to 0xeeaa1f28) [ 312.609063@3] 1ee0: 00000000 000058f0 00000000 00000000 eeaa0000 eeae8e40 ef55b520 c05d59c0 [ 312.609071@3] 1f00: ef55b500 00000000 c008605c eeaa1f3c eeaa1f14 eeaa1f28 c0085b84 c00520fc [ 312.609074@3] 1f20: 40000113 ffffffff [ 312.609087@3] [<c000db00>] (__irq_svc+0x40/0x84) from [<c00520fc>] (migrate_enable+0x24/0x15c) [ 312.609099@3] [<c00520fc>] (migrate_enable+0x24/0x15c) from [<c0085b84>] (irq_thread+0x9c/0x130) [ 312.609109@3] [<c0085b84>] (irq_thread+0x9c/0x130) from [<c0047f98>] (kthread+0x98/0xa0) [ 312.609119@3] [<c0047f98>] (kthread+0x98/0xa0) from [<c000dfd8>] (ret_from_fork+0x14/0x3c) [ 312.609127@1] CPU1: stopping [ 312.609134@1] CPU: 1 PID: 2429 Comm: stress Tainted: G W 3.10.70-rt71-mmc #4 [ 312.609149@1] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 312.609158@1] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) [ 312.609167@1] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8) [ 312.609176@1] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c) [ 312.609185@1] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000db00>] (__irq_svc+0x40/0x84) [ 312.609188@1] Exception stack(0xedc93e38 to 0xedc93e80) [ 312.609192@1] 3e20: 00000001 00000000 [ 312.609200@1] 3e40: 00000000 00000000 c3235d20 b4081000 eda33300 edff2e00 c0ac3b20 ec9d0204 [ 312.609208@1] 3e60: 2035975f ec9d0000 00000000 edc93e80 c0402280 c00adbe8 60000113 ffffffff [ 312.609221@1] [<c000db00>] (__irq_svc+0x40/0x84) from [<c00adbe8>] (do_anonymous_page.isra.73+0x194/0x244) [ 312.609233@1] [<c00adbe8>] (do_anonymous_page.isra.73+0x194/0x244) from [<c00b1074>] (handle_mm_fault+0xd8/0xec) [ 312.609244@1] [<c00b1074>] (handle_mm_fault+0xd8/0xec) from [<c0016784>] (do_page_fault+0xbc/0x1e8) [ 312.609254@1] [<c0016784>] (do_page_fault+0xbc/0x1e8) from [<c0008320>] (do_DataAbort+0x34/0x98) [ 312.609263@1] [<c0008320>] (do_DataAbort+0x34/0x98) from [<c000dc94>] (__dabt_usr+0x34/0x40) [ 312.609265@1] Exception stack(0xedc93fb0 to 0xedc93ff8) [ 312.609271@1] 3fa0: ffffffff b4081008 05310000 00000000 [ 312.609279@1] 3fc0: b6d72008 00001000 aed72008 0000005a 08000000 00000000 00001000 00000000 [ 312.609285@1] 3fe0: 00000002 beeb9b48 b6dc86cb 0000a186 80000030 ffffffff [ 312.609290@2] CPU2: stopping [ 312.609297@2] CPU: 2 PID: 2442 Comm: stress Tainted: G W 3.10.70-rt71-mmc #4 [ 312.609310@2] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 312.609319@2] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) [ 312.609327@2] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8) [ 312.609336@2] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c) [ 312.609345@2] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60) [ 312.609348@2] Exception stack(0xedbcffb0 to 0xedbcfff8) [ 312.609354@2] ffa0: b6e4e028 00000000 b6e4d1b4 b6e4d460 [ 312.609361@2] ffc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000f 00000000 [ 312.609368@2] ffe0: 000144b4 beeb9b70 b6d9b9e9 b6d9b9ee 60000030 ffffffff [ 312.910969@3] SMP: failed to stop secondary CPUs

Where shall I start looking?

UPDATE:
Another one:

Code: Select all: [ 715.793718@0] ------------[ cut here ]------------ [ 715.793736@0] WARNING: at kernel/irq/handle.c:148 handle_irq_event_percpu+0xa0/0x194() [ 715.793746@0] irq 110 handler aml_sdhc_irq+0x0/0x290 enabled interrupts [ 715.793759@0] Modules linked in: sg sd_mod usb_storage scsi_mod asix w1_gpio usbnet wire [ 715.793765@0] CPU: 0 PID: 2650 Comm: stress Not tainted 3.10.70-rt71-mmc #4 [ 715.793781@0] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 715.793791@0] [<c0011b04>] (show_stack+0x10/0x14) from [<c0029594>] (warn_slowpath_common+0x4c/0x68) [ 715.793799@0] [<c0029594>] (warn_slowpath_common+0x4c/0x68) from [<c0029644>] (warn_slowpath_fmt+0x30/0x40) [ 715.793807@0] [<c0029644>] (warn_slowpath_fmt+0x30/0x40) from [<c0085858>] (handle_irq_event_percpu+0xa0/0x194) [ 715.793816@0] [<c0085858>] (handle_irq_event_percpu+0xa0/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8) [ 715.793824@0] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) [ 715.793832@0] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40) [ 715.793843@0] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c) [ 715.793851@0] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c) [ 715.793859@0] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60) [ 715.793862@0] Exception stack(0xee0ddfb0 to 0xee0ddff8) [ 715.793867@0] dfa0: 000b1a7a 00000001 00000000 00000000 [ 715.793873@0] dfc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000f 00000000 [ 715.793878@0] dfe0: 000144b4 beceab70 0000a0e7 b6e2c9e0 60000030 ffffffff [ 715.793880@0] ---[ end trace 0000000000000002 ]--- [ 1020.608855@3] Kernel panic - not syncing: Watchdog detected hard LOCKUP on cpu 0 [ 1020.608864@3] CPU: 3 PID: 2647 Comm: stress Tainted: G W 3.10.70-rt71-mmc #4 [ 1020.608890@3] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 1020.608904@3] [<c0011b04>] (show_stack+0x10/0x14) from [<c03f5e08>] (panic+0x94/0x1ec) [ 1020.608918@3] [<c03f5e08>] (panic+0x94/0x1ec) from [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108) [ 1020.608929@3] [<c0084d3c>] (watchdog_check_hardlockup_other_cpu+0xcc/0x108) from [<c0084db0>] (watchdog_timer_fn+0x38/0x174) [ 1020.608943@3] [<c0084db0>] (watchdog_timer_fn+0x38/0x174) from [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c) [ 1020.608954@3] [<c004b7cc>] (__run_hrtimer.isra.18+0x94/0x10c) from [<c004c0ec>] (hrtimer_interrupt+0x148/0x338) [ 1020.608965@3] [<c004c0ec>] (hrtimer_interrupt+0x148/0x338) from [<c001c5c0>] (meson_timer_interrupt+0x70/0x88) [ 1020.608975@3] [<c001c5c0>] (meson_timer_interrupt+0x70/0x88) from [<c0085820>] (handle_irq_event_percpu+0x68/0x194) [ 1020.608985@3] [<c0085820>] (handle_irq_event_percpu+0x68/0x194) from [<c00859d0>] (handle_irq_event+0x84/0xa8) [ 1020.608995@3] [<c00859d0>] (handle_irq_event+0x84/0xa8) from [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) [ 1020.609006@3] [<c00888e8>] (handle_fasteoi_irq+0xec/0x1b8) from [<c0085110>] (generic_handle_irq+0x2c/0x40) [ 1020.609018@3] [<c0085110>] (generic_handle_irq+0x2c/0x40) from [<c000ef38>] (handle_IRQ+0x68/0x8c) [ 1020.609029@3] [<c000ef38>] (handle_IRQ+0x68/0x8c) from [<c0008458>] (gic_handle_irq+0x3c/0x5c) [ 1020.609038@3] [<c0008458>] (gic_handle_irq+0x3c/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60) [ 1020.609041@3] Exception stack(0xece9ffb0 to 0xece9fff8) [ 1020.609048@3] ffa0: 00000000 b6ede1ac b6ede1b8 b6ede460 [ 1020.609055@3] ffc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000c 00000000 [ 1020.609061@3] ffe0: 000144b4 beceab70 b6e2c9e9 b6e2cb30 60000030 ffffffff [ 1020.609069@2] CPU2: stopping [ 1020.609076@2] CPU: 2 PID: 2649 Comm: stress Tainted: G W 3.10.70-rt71-mmc #4 [ 1020.609090@2] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 1020.609099@2] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) [ 1020.609108@2] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8) [ 1020.609117@2] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c) [ 1020.609126@2] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60) [ 1020.609129@2] Exception stack(0xedf85fb0 to 0xedf85ff8) [ 1020.609135@2] 5fa0: b6edf028 00000000 00000000 00000001 [ 1020.609142@2] 5fc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 0000000e 00000000 [ 1020.609149@2] 5fe0: 000144b4 beceab70 b6e2c9e9 b6e2ca00 60000030 ffffffff [ 1020.609153@1] CPU1: stopping [ 1020.609160@1] CPU: 1 PID: 2639 Comm: stress Tainted: G W 3.10.70-rt71-mmc #4 [ 1020.609173@1] [<c001481c>] (unwind_backtrace+0x0/0xf8) from [<c0011b04>] (show_stack+0x10/0x14) [ 1020.609182@1] [<c0011b04>] (show_stack+0x10/0x14) from [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) [ 1020.609190@1] [<c0012cd4>] (ipi_cpu_stop+0x34/0xa4) from [<c0013160>] (handle_IPI+0xa8/0xd8) [ 1020.609199@1] [<c0013160>] (handle_IPI+0xa8/0xd8) from [<c0008470>] (gic_handle_irq+0x54/0x5c) [ 1020.609208@1] [<c0008470>] (gic_handle_irq+0x54/0x5c) from [<c000dcdc>] (__irq_usr+0x3c/0x60) [ 1020.609211@1] Exception stack(0xe4f13fb0 to 0xe4f13ff8) [ 1020.609217@1] 3fa0: b6ede460 beceab74 00000000 b6ede460 [ 1020.609224@1] 3fc0: 000144e4 000144d4 000144e0 000144d4 00000000 00000000 00000004 00000000 [ 1020.609231@1] 3fe0: 000144b4 beceab60 b6e2c9e9 b6e2cb00 a0000030 ffffffff [ 1020.910864@3] SMP: failed to stop secondary CPUs

UPDATE:

Managed to patch the sdhc driver, but not sure if it's correct though.

Currently undergoing stress testing.

by **kinsa** » Wed Mar 11, 2015 4:31 pm

Just an update, passed the 4 hr mark and still no errors. Stress testing done on a UHS-1 card (disableuhs flag turned off).

I'm now currently testing on a class 10 sd card.

Here's my cyclictest results:

Code: Select all: root@machinekit:~# cyclictest -D1d -m -n -a0 -t1 -p99 -i1000 --smp# /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 33.88 34.12 34.18 20/241 21457 T: 0 ( 1990) P:99 I:1000 C:16180778 Min: 5 Act: 20 Avg: 12 Max: 517 T: 1 ( 1991) P:99 I:1500 C:10787158 Min: 5 Act: 10 Avg: 13 Max: 656 T: 2 ( 1992) P:99 I:2000 C:8090354 Min: 5 Act: 10 Avg: 13 Max: 332 T: 3 ( 1993) P:99 I:2500 C:6472261 Min: 6 Act: 11 Avg: 14 Max: 477

UPDATE:

Stress test on the class 10 sd card running for more than 2 hrs now without errors.

Interestingly, it has better results than the UHS-1 card:

Code: Select all: root@machinekit:~# cyclictest -D1d -m -n -a0 -t1 -p99 -i1000 --smp # /dev/cpu_dma_latency set to 0us policy: fifo: loadavg: 33.78 33.96 33.98 22/241 2639 T: 0 ( 2587) P:99 I:1000 C:7423122 Min: 6 Act: 15 Avg: 11 Max: 51 T: 1 ( 2588) P:99 I:1500 C:4948641 Min: 6 Act: 11 Avg: 12 Max: 42 T: 2 ( 2589) P:99 I:2000 C:3711448 Min: 6 Act: 15 Avg: 11 Max: 40 T: 3 ( 2590) P:99 I:2500 C:2969114 Min: 6 Act: 13 Avg: 12 Max: 43

by **adbensi** » Wed Mar 11, 2015 6:32 pm

Me too, disableuhs turned off, but network slower yet, I would like to try this:
I have issues on the network too, but CLx says that with him work fine, may be another program cause the problem. I would like to try stop all process and test with static IP.

I had the impression that works with Sandisk SD UHS when I commented on setenv disableuhs boot.ini "disableuhs". My test stop because enough space left (works 7 minutes). Another test is change the filesystem.

by **kinsa** » Wed Mar 11, 2015 6:42 pm

adbensi wrote:I had the impression that works with Sandisk SD UHS when I commented on setenv disableuhs boot.ini "disableuhs". My test stop because enough space left (works 7 minutes). Another test is change the filesystem.

You just need a bigger card or run on a different partition. The command

Code: Select all: stress --cpu 8 --io 4 --vm 2 --hdd 2 --vm-bytes 128M --timeout 2500s

needs at least 2G free space to work on.

I've been using a USB lan adapter with no issues.

All we need now is a working onboard LAN on rt-preempt and we're good to go

by **Clx** » Thu Mar 12, 2015 10:15 pm

Hi Kinsa, thanks for joining me on this.
Actually, I've tried to do the same with the sdhc previously, but after that the kernel couldn't find the modules - after that I was distracted and switched to the other task.
Now I understand that the problem was in aufs - I ran "make modules -j5" and didn't notice the error, so after that I copied an almost empty modules dir:D

I've picked the new kernel version from Hardkernel, applied new rt74 patch and then applied your patches.
The code is here:
https://github.com/Clouded/linux-rt-odr ... 10.70-rt74

Could you please pull it so we would both be working on the same and fresh version?

UPDATE:

Compiled real time kernel 3.10.70-rt74 with modules can be downloaded here (in full configuration without disabling anything):
https://mega.co.nz/#!AM4j1AIb!p70ZU32GD ... ULRWBXc2NU

by **kinsa** » Fri Mar 13, 2015 7:15 pm

I've diffed our tree and the only significant change is this file:

Code: Select all: /media/Data/work/linux-odroid$ git diff clouded/odroidc-3.10.70-rt74 -- kernel/sched/core.c diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 42c24d0..1edc718 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -7489,7 +7489,7 @@ void __might_sleep(const char *file, int line, int preempt_offset) rcu_sleep_check(); /* WARN_ON_ONCE() by default, no rate limit reqd. */ if ((preempt_count_equals(preempt_offset) && !irqs_disabled() && - !is_idle_task(current)) || !is_idle_task(current)) || oops_in_progress) + !is_idle_task(current)) || oops_in_progress) return; if (system_state != SYSTEM_RUNNING && (!__might_sleep_init_called || system_state != SYSTEM_BOOTING))

The other differences are just cosmetic.

I've packaged the RT kernel and the debs can be found here: http://0ptr.link/C1/pool/main/l/linux-upstream/

It contains the full configuration and can also be used under Arch linux.

Cheers!

PS: did you made a progress on RPi2?

by **Clx** » Fri Mar 13, 2015 10:54 pm

kinsa wrote:PS: did you made a progress on RPi2?

RPi2 is 99% ready, except for the one thing - new RPi's USB driver uses FIQ and causes hangs when used on SMP system with RT PREEMPT.
And the standard USB driver seems to be broken for Raspberry Pi 2.

by **adbensi** » Sat Mar 14, 2015 10:13 am

Hello Kinsa,
Now your network work fine on the rt?

by **kinsa** » Sat Mar 14, 2015 11:00 am

only on usb lan

by **adbensi** » Sat Mar 14, 2015 5:57 pm

sorry for not understanding, but what is the difference with the rt71 version to "finally working"? Here was fine, except for the EMMC and network. Best regards.

by **kinsa** » Sat Mar 14, 2015 6:25 pm

The sdhc driver issue was fixed, so the stress test now runs OK. Although long term test is still needed.

The only remaining issue is the onboard network driver.

I've made an image for Machinekit, please have a lool at it

Cheers

by **adbensi** » Sat Mar 14, 2015 6:59 pm

Very nice,
I am seen your posts on the machinekit, a lot of work, nice job, congratulations!

This is the machinekit image for ODROID?
http://0ptr.link/C1/pool/main/m/machine ... _armhf.deb

I compiled machinekit directly, needed to do the night rss. precompiled image is very great

I would suggest the cammill.org software

by **kinsa** » Sat Mar 14, 2015 7:05 pm

Yup.

My next job is to port PICnc