Syncing two UAS devices causes Ubuntu to hang

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Sat Aug 10, 2019 12:56 am

Even if there is no speed fix the N2 is still ~3x faster than USB2 and ~2x faster than the XU4 for encrypted disks, but I hope there will be a speed improvement.

trex_daemon
Posts: 22
Joined: Wed Jul 17, 2019 4:47 pm
languages_spoken: english
ODROIDs: C2, N2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by trex_daemon » Sat Aug 10, 2019 6:31 pm

ok guys, but this capping won't fix the problem with usb tv tuners. they also have issues and this does not seems to fix it

User avatar
mad_ady
Posts: 6384
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 107 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by mad_ady » Sat Aug 10, 2019 6:56 pm

@trex_daemon: I agree about tv tuners not being fixed. However, as a workaround I think you can hook up a tv tuner stably to the otg port which uses a different bus.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Tue Aug 13, 2019 2:24 am

I have to report a step back. Regular rsync is failing again with the most recent kernel:

Code: Select all

root@odroid[~]$ dmesg | grep sg_tablesize
[    4.879248] usb: xhci: determined sg_tablesize: 2
[    4.950577] usb: xhci: determined sg_tablesize: 2
root@odroid[~]$ cat /sys/kernel/debug/aml_clkmsr/clkmsr | grep clk81
[ 7][ 222000000]clk81
root@odroid[~]$ uname -a
Linux droid 4.9.187-53 #1 SMP PREEMPT Fri Aug 9 06:12:18 -03 2019 aarch64 aarch64 aarch64 GNU/Linux
root@droid[~]$

Code: Select all

[67268.760244] xhci-hcd xhci-hcd.0.auto: xHCI host not responding to stop endpoint command.
[67268.760255] xhci-hcd xhci-hcd.0.auto: Assuming host is dying, halting host.
[67268.776459] xhci-hcd xhci-hcd.0.auto: Host not halted after 16000 microseconds.
[67268.776620] xhci-hcd xhci-hcd.0.auto: Non-responsive xHCI host is not halting.
[67268.776628] xhci-hcd xhci-hcd.0.auto: Completing active URBs anyway.
[67268.776817] hub 2-1.1:1.0: hub_ext_port_status failed (err = -108)
[67268.776883] xhci-hcd xhci-hcd.0.auto: HC died; cleaning up
[67268.777977] usb 1-1: USB disconnect, device number 2
[67268.777986] usb 1-1.1: USB disconnect, device number 3
[67268.777993] usb 1-1.1.1: USB disconnect, device number 4
[67268.781049] usb 1-1.1.3: USB disconnect, device number 5
[67268.783447] usb 2-1.1.1: USB disconnect, device number 4
[67268.783456] usb 2-1.1.1.4: USB disconnect, device number 7
[67268.783755] usb 2-1: USB disconnect, device number 2
[67268.783764] usb 2-1.1: USB disconnect, device number 3

trex_daemon
Posts: 22
Joined: Wed Jul 17, 2019 4:47 pm
languages_spoken: english
ODROIDs: C2, N2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by trex_daemon » Tue Aug 13, 2019 2:56 am

mad_ady wrote:
Sat Aug 10, 2019 6:56 pm
@trex_daemon: I agree about tv tuners not being fixed. However, as a workaround I think you can hook up a tv tuner stably to the otg port which uses a different bus.
Unless you are using 4 tv tuners:
https://www.instructables.com/id/Folow- ... d-N2-and-/

This is the second version of my tv tuner setup. As you can see, I've done a nice looking enclosure with a VFD display, and everything is designed to use the rear ports to connect the 4 tuners. The front sides sits close to a glass panel where the IR sits to be able to use it with a remote. The front port is not accessbile, as I didn't expect that I will need it.

The setup in my house is that there is a central TV server, served now by the N2 where my "main" tv is installed and in the other rooms I've installed just tvheadend clients (Basically raspberry Pis running Kodi).

This 'tv' server should serve all channels in the enttire house.

Now, there is indeed a workaround for using the front port. However, then I would have to destroy the enclosure I've build with quite a significant amount of effort, and also, on the other hand, I am not convinced that the front, OTG port can server 4 tuners (even if using a self powered hub).
So why I brought this board (instead of let's say a NUC) was that it has 4 high speed usb port, HDR and 4K capability, so it can be used a tv server and also as a Kodi client itself.

Now with this USB issue, it cannot serve it's purpose.
I do understand that it's not easy to develop a new board and that problems can arise, but now this issue is being going on since almost half a year, and the board is not quite usable.

Yes, throttling the disks solves partially the USB problem (however, these empiric tests that were made prove absolutely nothing) and guys, you must be really brave if you use this device to serve your data , as nobody can assure for now that it won't lead to corruption.

I am an Embedded Engineer in the Automotive industry, working for more than 10 years, and I highly doubt that you would really drive a car that has "such" a solution.
Why would be your data different ?

And besides that. this is just a workaround and not a solution.

As an Odroid customer (I have also a C2 as well) I really expect that a product's peripheral interfaces will work without isssues.
So Hardkernel/Amlogic should really come up with an official answer and a solution to this problem. Either a working solution that really solves the issues for all USB devices, not just hides them or to admit that there is a HW issue with the board, and then a recall is mandatory.

This is how it is done in the Automotive industry as well. Here, people won't die if this fails (As in car crashes) , but your data can die, and it could contain very precious data (let's say you keep 10 years of accounting data on it).

alvarow
Posts: 26
Joined: Wed Apr 10, 2019 10:44 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 10 times
Been thanked: 0
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by alvarow » Tue Aug 13, 2019 3:40 am

Losing data is definitely a big concern to me, especially since I was planning on using my N2 to backup my NAS to a USB drive attached to the N2. I went back to my Sheevaplug, and I am considering an Pi4, but I am waiting to see what happens here, I do like the board.

qcasey1
Posts: 23
Joined: Thu Apr 18, 2019 7:29 am
languages_spoken: english
ODROIDs: XU4, N2
Location: California
Has thanked: 5 times
Been thanked: 0
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by qcasey1 » Tue Aug 13, 2019 3:47 am

I agree, this is a huge bummer especially considering there hasn't been any official statement made by ODROID. In fairness to them though I think it was mentioned earlier in this thread that they're waiting on Amlogic to address some of these underlying issues. Hopefully we have something to work with sooner than later Image
trex_daemon wrote: This is how it is done in the Automotive industry as well. Here, people won't die if this fails (As in car crashes) , but your data can die, and it could contain very precious data (let's say you keep 10 years of accounting data on it).
This is silly. Keeping precious data without redundancy (on a 6 month old, two figure development board no less) is a recipe for disaster. I'd have no sympathy if I saw this actually happen.

Sent from my GM1917 using Tapatalk


elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Tue Aug 13, 2019 3:59 am

Nighti wrote:
Tue Aug 13, 2019 2:24 am
I have to report a step back. Regular rsync is failing again with the most recent kernel...
That's not good.

I wonder if there is any hardware fix that could be done (like improve isolation with a capacitor, inductor, etc) or if it's purely a software issue.

Are you willing to try the 5.3 kernel to help narrow down the issue?

User avatar
mad_ady
Posts: 6384
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 107 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by mad_ady » Tue Aug 13, 2019 4:07 am

@trex_daemon: great write-up - deffinetly worthy of Odroid Magazine. I do hope that usb gets fixed, but I fear the problem is with amlogic's soc/software blobs. They've had usb issues with C1 and C2 and they haven't been fixed 100%, but users learned to work around them. When I had webcams attached to C2s I had a watchdog script snap periodic pictures and if the pictures were less than 10kB in size it would trigger a usb bus reset. Sometimes it would happen a few times a day, other times a few times per hour.

I'm only suggesting you to have a plan B, because plan A may depend on Amlogic and will take a while.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Tue Aug 13, 2019 4:13 am

elatllat wrote:
Tue Aug 13, 2019 3:59 am
Nighti wrote:
Tue Aug 13, 2019 2:24 am
I have to report a step back. Regular rsync is failing again with the most recent kernel...
That's not good.

I wonder if there is any hardware fix that could be done (like improve isolation with a capacitor, inductor, etc) or if it's purely a software issue.

Are you willing to try the 5.3 kernel to help narrow down the issue?
In general yes, how stable is it?

Do you have a new build or is this the most recent one?

I also lost hope the 4.9 kernel getting stable in regards to USB any time soon. All patches are targeting to slow down USB access but no sign that at least the root cause is found.
Last edited by Nighti on Tue Aug 13, 2019 7:15 am, edited 2 times in total.

gregeeh
Posts: 5
Joined: Wed May 08, 2019 8:09 am
languages_spoken: english
Has thanked: 0
Been thanked: 2 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by gregeeh » Tue Aug 13, 2019 6:35 am

@trex_daemon: Completely agree with you. This has been going on for far too long now with no sight of a permanent proper solution. ie: No workarounds, just a fix that will enable us to use the device as intended. Presently I feel I have wasted my money purchasing this device as I cannot use it for the purpose I brought it to do. I have stopped several people buying this product and will continue to do so until the USB 3 ports are fixed, permanently.

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Tue Aug 13, 2019 9:28 am

Nighti wrote:
Tue Aug 13, 2019 4:13 am
...
In general yes, how stable is [5.3]?

Do you have a new build or is this the most recent one?

I also lost hope the 4.9 kernel getting stable in regards to USB any time soon. All patches are targeting to slow down USB access but no sign that at least the root cause is found.
I have not had any bugs with it,
yes I just uploaded RC4 to there,
I only ask you test for another data point as HK and AMLogic have not yet made clear what the cause of the issue is.
These users thanked the author elatllat for the post:
rooted (Tue Aug 13, 2019 12:28 pm)

trex_daemon
Posts: 22
Joined: Wed Jul 17, 2019 4:47 pm
languages_spoken: english
ODROIDs: C2, N2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by trex_daemon » Wed Aug 14, 2019 1:24 am

mad_ady wrote:
Tue Aug 13, 2019 4:07 am
@trex_daemon: great write-up - deffinetly worthy of Odroid Magazine

I did have my previous instructable regarding the media center with the C2 published in a magazine :)

User avatar
odroid
Site Admin
Posts: 31835
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 89 times
Been thanked: 252 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by odroid » Wed Aug 14, 2019 10:54 am

We've run two different test benches and there is no issue so far.
- N2 + four UAS SSD with multi-threaded DD test for 9 days.
- N2 + four UAS SSD RAID 5 mode with infinity f3write / f3read test for 5 days.
So we can say the USB 3.0 storage device stability seems to fine.

A quick list of what we did for the stability improvement.
- Lower the transaction packet size with sg_tablesize=2 parameter.
- Compensate the speed drop by a higher internal peripheral clock source settings (166Mhz -> 222Mhz)
- Assign the XHCI IRQ handler to a big A73 core.
- Change the minimum CPU clock freq from 100Mhz to 667Mhz.
- Performance governor (this was not effective probably)

Many users reported those changes could solve the stability issue.
But there are still a few users have the issue. (BTRFS must be isolated since the latest 4.9.197 still seems to have BTRFS related bugs).

So I want to hear sincerely next possible test conditions to reproduce the issue on our test bench.
Please give us any idea to find a right direction.

If possible, please also compare the USB-SSD bulk transfer speed between our latest kernel 4.9 and your trial kernel 5.3 too.
Run the following commands on the media mount point. The result will be very helpful to find a real root cause.
As far as I heard, Kernel 5.3 has some lower system clock frequency settings and it might affect the IO performance/stability.

Code: Select all

Write command
dd if=/dev/zero of=test oflag=direct bs=8M count=128

Read command 
dd if=test of=/dev/null iflag=direct bs=8M

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Wed Aug 14, 2019 11:20 am

Thanks for the update odroid.
As far as I know the new img fixed crashing for everyone except
- avatarNighti using rsync
- trex_daemon using 4 tv tuners
Nether have tried 5.3 or shared exact steps to reproduce so untill then we can just say good job on the fix.

5.3 maxes out a HDD just like 4.9, maybe you can test on the SSDs, but 5.4 may change that (hopefully not triggering this issue; you might want to ask Neal to test that)

... some may hope for a rev 0.4 (or hardware modification instructions) to make the faster speeds reliable.

User avatar
odroid
Site Admin
Posts: 31835
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 89 times
Been thanked: 252 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by odroid » Wed Aug 14, 2019 11:42 am

Thank you for the reply.
We will try to make a rsync test bed and try your pre-built Kernel 5.3 separately.

Since we have no DVB-Tx broadcasting system in Korea, it is really hard to test the USB tuner issue.
Instead of that, we connected four USB 2.0 webcams and ran four preview software(guvcview) on the Mate desktop for several days.
But there was no issue. As far as I know, all the USB tuners have USB 2.0 interface not 3.0.
Anyway, we have to find a way to emulate the TV tuners in our lab.

We think the USB 3.0 issue exists in the SoC inside because other S922X/S905X2/S922X-REVB/A311D Android TV boxes also have the same issue.
So I don't think any PCB changes can be effective.
These users thanked the author odroid for the post (total 2):
xabolcs (Wed Aug 14, 2019 5:16 pm) • elatllat (Wed Aug 14, 2019 9:10 pm)

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Wed Aug 14, 2019 1:30 pm

Thanks for the update elatllat and odroid. Once I have a little bit more spare time, I will try out 5.3 kernel and compare the results.

I would suggest to try slow (5400) HDD drives. I've a feeling it might have something to do with the way they behave. Like filling and emptying the buffers in a more chaotic way.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Wed Aug 14, 2019 1:33 pm

Here is something new happened:

Code: Select all

Aug 13 21:30:22 droid kernel: Bad mode in Error handler detected on CPU2, code 0xbf000002 -- SError
Aug 13 21:30:22 droid kernel: FAR:5559aa99c8
Aug 13 21:30:22 droid kernel: reg              value       pfn  reg              value       pfn
Aug 13 21:30:22 droid kernel: r0 :  0000000000000040  --------  r1 :  ffffffc033264600  --------
Aug 13 21:30:22 droid kernel: r2 :  0000000000000000  --------  r3 :  0000000000000000  --------
Aug 13 21:30:22 droid kernel: r4 :  ffffffbf03d5001c  --------  r5 :  0000000000000000  --------
Aug 13 21:30:22 droid kernel: r6 :  00000055b29288d0     ea96e  r7 :  0000000000000000  --------
Aug 13 21:30:22 droid kernel: r8 :  0000000000000000  --------  r9 :  0000000000000004  --------
Aug 13 21:30:22 droid kernel: r10:  0000000000000000  --------  r11:  000000b4000000a6  --------
Aug 13 21:30:22 droid kernel: r12:  000000000000000b  --------  r13:  000000100000000d  --------
Aug 13 21:30:22 droid kernel: r14:  000000b100000005  --------  r15:  00000000142e06db  --------
Aug 13 21:30:22 droid kernel: r16:  ffffff800922dcd8  --------  r17:  0000007f7e399a78     c4ae2
Aug 13 21:30:22 droid kernel: r18:  00000000fe7ef5e4  --------  r19:  0000000000000000  --------
Aug 13 21:30:22 droid kernel: r20:  ffffff8009f23e40  --------  r21:  00000000000006aa  --------
Aug 13 21:30:22 droid kernel: r22:  ffffffbf03d50000  --------  r23:  000000000000007d  --------
Aug 13 21:30:22 droid kernel: r24:  ffffffc0b0723df0  --------  r25:  ffffffc0bf3f6000  --------
Aug 13 21:30:22 droid kernel: r26:  ffffffc0740560e8  --------  r27:  ffffff8009f1b598  --------
Aug 13 21:30:22 droid kernel: r28:  ffffffc0bf3f60a0  --------  r29:  ffffffc0b0723cc0  --------
Aug 13 21:30:22 droid kernel: r30:  ffffff80091bf1c0      11bf  
Aug 13 21:30:22 droid kernel: pc :  ffffff80090d6860      10d6
Aug 13 21:30:22 droid kernel: sp :  ffffffc0b0723cc0  --------
Aug 13 21:30:22 droid kernel: unused :  0000005559aa99c8  --------
Aug 13 21:30:22 droid kernel: Internal error: Oops - bad mode: 0 [#1] PREEMPT SMP
Aug 13 21:30:22 droid kernel: Modules linked in: xt_conntrack xt_TCPMSS xt_tcpmss xt_policy bridge stp llc fuse ip6table_filter ip6_tables iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM xt_tcpudp iptable_mangle rtc_pcf8563 ir_lirc_codec lirc_dev meson_ir i2c_meson_master sch_fq_codel xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 firmware media_clock ip_tables x_tables ipv6
Aug 13 21:30:22 droid kernel: CPU: 2 PID: 3495 Comm: rsync Not tainted 4.9.187-53 #1
Aug 13 21:30:22 droid kernel: Hardware name: Hardkernel ODROID-N2 (DT)
Aug 13 21:30:22 droid kernel: task: ffffffc033264600 task.stack: ffffffc0b0720000
Aug 13 21:30:22 droid kernel: PC is at ___might_sleep+0x118/0x138
Aug 13 21:30:22 droid kernel: LR is at generic_file_read_iter+0xc0/0x808
Aug 13 21:30:22 droid kernel: pc : [<ffffff80090d6860>] lr : [<ffffff80091bf1c0>] pstate: 60000045
Aug 13 21:30:22 droid kernel: sp : ffffffc0b0723cc0
Aug 13 21:30:22 droid kernel: x29: ffffffc0b0723cc0 x28: ffffffc0bf3f60a0 
Aug 13 21:30:22 droid kernel: x27: ffffff8009f1b598 x26: ffffffc0740560e8 
Aug 13 21:30:22 droid kernel: x25: ffffffc0bf3f6000 x24: ffffffc0b0723df0 
Aug 13 21:30:22 droid kernel: x23: 000000000000007d x22: ffffffbf03d50000 
Aug 13 21:30:22 droid kernel: x21: 00000000000006aa x20: ffffff8009f23e40 
Aug 13 21:30:22 droid kernel: x19: 0000000000000000 x18: 00000000fe7ef5e4 
Aug 13 21:30:22 droid kernel: x17: 0000007f7e399a78 x16: ffffff800922dcd8 
Aug 13 21:30:22 droid kernel: x15: 00000000142e06db x14: 000000b100000005 
Aug 13 21:30:22 droid kernel: x13: 000000100000000d x12: 000000000000000b 
Aug 13 21:30:22 droid kernel: x11: 000000b4000000a6 x10: 0000000000000000 
Aug 13 21:30:22 droid kernel: x9 : 0000000000000004 x8 : 0000000000000000 
Aug 13 21:30:22 droid kernel: x7 : 0000000000000000 x6 : 00000055b29288d0 
Aug 13 21:30:22 droid kernel: x5 : 0000000000000000 x4 : ffffffbf03d5001c 
Aug 13 21:30:22 droid kernel: x3 : 0000000000000000 x2 : 0000000000000000 
Aug 13 21:30:22 droid kernel: x1 : ffffffc033264600 x0 : 0000000000000040 
Aug 13 21:30:22 droid kernel: 
                              SP: 0xffffffc0b0723c40:
Aug 13 21:30:22 droid kernel: 3c40  03d50000 ffffffbf 0000007d 00000000 b0723df0 ffffffc0 bf3f6000 ffffffc0
Aug 13 21:30:22 droid kernel: 3c60  740560e8 ffffffc0 09f1b598 ffffff80 bf3f60a0 ffffffc0 b0723cc0 ffffffc0
Aug 13 21:30:22 droid kernel: 3c80  091bf1c0 ffffff80 b0723cc0 ffffffc0 090d6860 ffffff80 60000045 00000000
Aug 13 21:30:22 droid kernel: 3ca0  b0723cf0 ffffffc0 091bf400 ffffff80 ffffffff 0000007f 59aa99c8 00000055
Aug 13 21:30:22 droid kernel: 3cc0  b0723cf0 ffffffc0 091bf1c0 ffffff80 74056250 ffffffc0 b0723e18 ffffffc0
Aug 13 21:30:22 droid kernel: 3ce0  0003d000 00000000 740560e8 ffffffc0 b0723db0 ffffffc0 0922b6f8 ffffff80
Aug 13 21:30:22 droid kernel: 3d00  bf3f6000 ffffffc0 00040000 00000000 b0723eb8 ffffffc0 b28eb8d0 00000055
Aug 13 21:30:22 droid kernel: 3d20  b0723eb8 ffffffc0 00000015 00000000 00000123 00000000 0000003f 00000000
Aug 13 21:30:22 droid kernel: 
                              X1: 0xffffffc033264580:
Aug 13 21:30:22 droid kernel: 4580  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 45a0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 45c0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 45e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 4600  00000802 00000000 ffffffff ffffffff 00000001 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 4620  b0720000 ffffffc0 00000002 00400000 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 4640  00000001 00000002 000000ad 00000000 0000a928 00000001 d2ae6200 ffffffc0
Aug 13 21:30:22 droid kernel: 4660  00000002 00000001 00000078 00000078 00000078 00000000 09c09458 ffffff80
Aug 13 21:30:22 droid kernel: 
                              X24: 0xffffffc0b0723d70:
Aug 13 21:30:22 droid kernel: 3d70  00000080 00000000 00000000 00000000 00020000 00000000 740560e8 ffffffc0
Aug 13 21:30:22 droid kernel: 3d90  bf3f6010 ffffffc0 00000004 00000000 b0723de0 ffffffc0 093b63e0 ffffff80
Aug 13 21:30:22 droid kernel: 3db0  b0723e40 ffffffc0 0922c60c ffffff80 00040000 00000000 bf3f6000 ffffffc0
Aug 13 21:30:22 droid kernel: 3dd0  00000000 00000000 b28eb8d0 00000055 b28eb8d0 00000055 00040000 00000000
Aug 13 21:30:22 droid kernel: 3df0  bf3f6000 ffffffc0 00040000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 3e10  00000000 00000000 00000000 ffffff80 0003d000 00000000 00003000 00000000
Aug 13 21:30:22 droid kernel: 3e30  b0723de0 ffffffc0 00000001 00000000 b0723e80 ffffffc0 0922dd34 ffffff80
Aug 13 21:30:22 droid kernel: 3e50  bf3f6000 ffffffc0 bf3f6000 ffffffc0 b28eb8d0 00000055 00040000 00000000
Aug 13 21:30:22 droid kernel: 
                              X25: 0xffffffc0bf3f5f80:
Aug 13 21:30:22 droid kernel: 5f80  00000000 00000000 00000000 00000000 00000000 00000000 baf7c900 ffffffc0
Aug 13 21:30:22 droid kernel: 5fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 5fc0  00000000 00000000 c5d2c550 ffffffc0 bcb34000 ffffffc0 bf3f5fd8 ffffffc0
Aug 13 21:30:22 droid kernel: 5fe0  bf3f5fd8 ffffffc0 bf3f5fe8 ffffffc0 bf3f5fe8 ffffffc0 bcb34198 ffffffc0
Aug 13 21:30:22 droid kernel: 6000  00000000 00000000 00000000 00000000 b69c7c20 ffffffc0 7418bc00 ffffffc0
Aug 13 21:30:22 droid kernel: 6020  740560e8 ffffffc0 09c29480 ffffff80 00000000 00000000 00000001 00000000
Aug 13 21:30:22 droid kernel: 6040  00020000 0002801d 00000001 00000000 bf3f6050 ffffffc0 bf3f6050 ffffffc0
Aug 13 21:30:22 droid kernel: 6060  00000000 00000000 00000000 00000000 00040000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 
                              X26: 0xffffffc074056068:
Aug 13 21:30:22 droid kernel: 6068  74056060 ffffffc0 00000000 00000000 00000001 00000000 74056080 ffffffc0
Aug 13 21:30:22 droid kernel: 6088  74056080 ffffffc0 005c0e03 00000000 00000000 00000000 740560a0 ffffffc0
Aug 13 21:30:22 droid kernel: 60a8  740560a0 ffffffc0 00000000 00000000 00000001 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 60c8  740560c8 ffffffc0 740560c8 ffffffc0 00000000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 60e8  000d81fd 000003ea 000003ec 00000000 00000000 00000000 ffffffff ffffffff
Aug 13 21:30:22 droid kernel: 6108  09c293c0 ffffff80 a22c8800 ffffffc0 74056250 ffffffc0 1c3d5000 ffffffc0
Aug 13 21:30:22 droid kernel: 6128  08fb0378 00000000 00000001 00000000 005c0e03 00000000 5d538e5d 00000000
Aug 13 21:30:22 droid kernel: 6148  035d93c3 00000000 5aeb43ae 00000000 00000000 00000000 5d3b2df4 00000000
Aug 13 21:30:22 droid kernel: 
                              X28: 0xffffffc0bf3f6020:
Aug 13 21:30:22 droid kernel: 6020  740560e8 ffffffc0 09c29480 ffffff80 00000000 00000000 00000001 00000000
Aug 13 21:30:22 droid kernel: 6040  00020000 0002801d 00000001 00000000 bf3f6050 ffffffc0 bf3f6050 ffffffc0
Aug 13 21:30:22 droid kernel: 6060  00000000 00000000 00000000 00000000 00040000 00000000 00000000 00000000
Aug 13 21:30:22 droid kernel: 6080  00000000 00000000 00000000 00000000 00000000 00000000 ca315240 ffffffc0
Aug 13 21:30:22 droid kernel: 60a0  00000080 00000000 00000020 00000020 00000020 00000000 0003f000 00000000
Aug 13 21:30:22 droid kernel: 60c0  00000000 00000000 b6b17f80 ffffffc0 00000000 00000000 bf3f60d8 ffffffc0
Aug 13 21:30:22 droid kernel: 60e0  bf3f60d8 ffffffc0 bf3f60e8 ffffffc0 bf3f60e8 ffffffc0 74056250 ffffffc0
Aug 13 21:30:22 droid kernel: 6100  00000000 00000000 00000000 00000000 d28a4e20 ffffffc0 2a9cc900 ffffffc0
Aug 13 21:30:22 droid kernel: 
                              X29: 0xffffffc0b0723c40:
Aug 13 21:30:22 droid kernel: 3c40  03d50000 ffffffbf 0000007d 00000000 b0723df0 ffffffc0 bf3f6000 ffffffc0
Aug 13 21:30:22 droid kernel: 3c60  740560e8 ffffffc0 09f1b598 ffffff80 bf3f60a0 ffffffc0 b0723cc0 ffffffc0
Aug 13 21:30:22 droid kernel: 3c80  091bf1c0 ffffff80 b0723cc0 ffffffc0 090d6860 ffffff80 60000045 00000000
Aug 13 21:30:22 droid kernel: 3ca0  b0723cf0 ffffffc0 091bf400 ffffff80 ffffffff 0000007f 59aa99c8 00000055
Aug 13 21:30:22 droid kernel: 3cc0  b0723cf0 ffffffc0 091bf1c0 ffffff80 74056250 ffffffc0 b0723e18 ffffffc0
Aug 13 21:30:22 droid kernel: 3ce0  0003d000 00000000 740560e8 ffffffc0 b0723db0 ffffffc0 0922b6f8 ffffff80
Aug 13 21:30:22 droid kernel: 3d00  bf3f6000 ffffffc0 00040000 00000000 b0723eb8 ffffffc0 b28eb8d0 00000055
Aug 13 21:30:22 droid kernel: 3d20  b0723eb8 ffffffc0 00000015 00000000 00000123 00000000 0000003f 00000000
Aug 13 21:30:22 droid kernel: 
Aug 13 21:30:22 droid kernel: Process rsync (pid: 3495, stack limit = 0xffffffc0b0720000)
Aug 13 21:30:22 droid kernel: Stack: (0xffffffc0b0723cc0 to 0xffffffc0b0724000)
Aug 13 21:30:22 droid kernel: 3cc0: ffffffc0b0723cf0 ffffff80091bf1c0 ffffffc074056250 ffffffc0b0723e18
Aug 13 21:30:22 droid kernel: 3ce0: 000000000003d000 ffffffc0740560e8 ffffffc0b0723db0 ffffff800922b6f8
Aug 13 21:30:22 droid kernel: 3d00: ffffffc0bf3f6000 0000000000040000 ffffffc0b0723eb8 00000055b28eb8d0
Aug 13 21:30:22 droid kernel: 3d20: ffffffc0b0723eb8 0000000000000015 0000000000000123 000000000000003f
Aug 13 21:30:22 droid kernel: 3d40: ffffff8009c02000 ffffffc033264600 0000000000020000 0000000000001000
Aug 13 21:30:22 droid kernel: 3d60: 00000000bf3f6010 000000000000007c 0000000000000080 0000000000000000
Aug 13 21:30:22 droid kernel: 3d80: 0000000000020000 ffffffc0740560e8 ffffffc0bf3f6010 0000000000000004
Aug 13 21:30:22 droid kernel: 3da0: ffffffc0b0723de0 ffffff80093b63e0 ffffffc0b0723e40 ffffff800922c60c
Aug 13 21:30:22 droid kernel: 3dc0: 0000000000040000 ffffffc0bf3f6000 0000000000000000 00000055b28eb8d0
Aug 13 21:30:22 droid kernel: 3de0: 00000055b28eb8d0 0000000000040000 ffffffc0bf3f6000 0000000000040000
Aug 13 21:30:22 droid kernel: 3e00: 0000000000000000 0000000000000000 0000000000000000 ffffff8000000000
Aug 13 21:30:22 droid kernel: 3e20: 000000000003d000 0000000000003000 ffffffc0b0723de0 0000000000000001
Aug 13 21:30:22 droid kernel: 3e40: ffffffc0b0723e80 ffffff800922dd34 ffffffc0bf3f6000 ffffffc0bf3f6000
Aug 13 21:30:22 droid kernel: 3e60: 00000055b28eb8d0 0000000000040000 0000000060000000 ffffff8009088adc
Aug 13 21:30:22 droid kernel: 3e80: 0000000000000000 ffffff8009083a1c 0000000000000800 00000040cd145000
Aug 13 21:30:22 droid kernel: 3ea0: ffffffffffffffff 0000007f7e399aa4 0000000000000215 0000000000040000
Aug 13 21:30:22 droid kernel: 3ec0: 0000000000000003 00000055b28eb8d0 0000000000040000 0000007f7e42f1a8
Aug 13 21:30:22 droid kernel: 3ee0: 00000000005c0e03 0000000000580e03 0000000000008000 0000000000008000
Aug 13 21:30:22 droid kernel: 3f00: 000000000000003f 0000000089f8cfa1 00000000dc487846 00000000206509a3
Aug 13 21:30:22 droid kernel: 3f20: 00000000bc1cc0ee 00000000a3d08ec2 0000000069e35df9 00000000142e06db
Aug 13 21:30:22 droid kernel: 3f40: 000000558837e798 0000007f7e399a78 00000000fe7ef5e4 0000000000000003
Aug 13 21:30:22 droid kernel: 3f60: 0000000000040000 0000000000000000 0000000000000000 0000000000000000
Aug 13 21:30:22 droid kernel: 3f80: 0000000000040000 0000000000040000 00000055b2e77970 00000055b2e75a78
Aug 13 21:30:22 droid kernel: 3fa0: 0000000000008000 0000007fc9abb0e0 0000005588343454 0000007fc9abb0e0
Aug 13 21:30:22 droid kernel: 3fc0: 0000007f7e399aa4 0000000060000000 0000000000000003 000000000000003f
Aug 13 21:30:22 droid kernel: 3fe0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
Aug 13 21:30:22 droid kernel: Call trace:
Aug 13 21:30:22 droid kernel: [<ffffff80090d6860>] ___might_sleep+0x118/0x138
Aug 13 21:30:22 droid kernel: [<ffffff80091bf1c0>] generic_file_read_iter+0xc0/0x808
Aug 13 21:30:22 droid kernel: [<ffffff800922b6f8>] __vfs_read+0xd0/0x110
Aug 13 21:30:22 droid kernel: [<ffffff800922c60c>] vfs_read+0x94/0x150
Aug 13 21:30:22 droid kernel: [<ffffff800922dd34>] SyS_read+0x5c/0xc8
Aug 13 21:30:22 droid kernel: [<ffffff8009083a1c>] __sys_trace_return+0x0/0x4
Aug 13 21:30:22 droid kernel: Code: 52800120 97ff3a17 17ffffd7 d53b4220 (373ff920) 
Aug 13 21:30:22 droid kernel: ---[ end trace a345c92a52ea53da ]---
Aug 13 21:30:22 droid kernel: BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:21
Aug 13 21:30:22 droid kernel: in_atomic(): 0, irqs_disabled(): 128, pid: 3495, name: rsync
Aug 13 21:30:22 droid kernel: CPU: 2 PID: 3495 Comm: rsync Tainted: G      D         4.9.187-53 #1
Aug 13 21:30:22 droid kernel: Hardware name: Hardkernel ODROID-N2 (DT)
Aug 13 21:30:22 droid kernel: Call trace:
Aug 13 21:30:22 droid kernel: [<ffffff800908b000>] dump_backtrace+0x0/0x268
Aug 13 21:30:22 droid kernel: [<ffffff800908b2ec>] show_stack+0x24/0x30
Aug 13 21:30:22 droid kernel: [<ffffff800943a738>] dump_stack+0xb4/0xe4
Aug 13 21:30:22 droid kernel: [<ffffff80090d684c>] ___might_sleep+0x104/0x138
Aug 13 21:30:22 droid kernel: [<ffffff80090d68d8>] __might_sleep+0x58/0x90
Aug 13 21:30:22 droid kernel: [<ffffff8009be8384>] down_read+0x2c/0x78
Aug 13 21:30:22 droid kernel: [<ffffff80090cd244>] blocking_notifier_call_chain+0x4c/0x88
Aug 13 21:30:22 droid kernel: [<ffffff8009121188>] profile_task_exit+0x30/0x40
Aug 13 21:30:22 droid kernel: [<ffffff80090aac1c>] do_exit+0x24/0xa58
Aug 13 21:30:22 droid kernel: [<ffffff800908b4b4>] die+0x1bc/0x1c0
Aug 13 21:30:22 droid kernel: [<ffffff800908bb18>] bad_mode+0xb0/0xc8
Aug 13 21:30:22 droid kernel: [<ffffff80091bf1c0>] generic_file_read_iter+0xc0/0x808
Aug 13 21:30:22 droid kernel: [<ffffff80091bf1c0>] generic_file_read_iter+0xc0/0x808
Aug 13 21:30:22 droid kernel: [<ffffff800922b6f8>] __vfs_read+0xd0/0x110
Aug 13 21:30:22 droid kernel: [<ffffff800922c60c>] vfs_read+0x94/0x150
Aug 13 21:30:22 droid kernel: [<ffffff800922dd34>] SyS_read+0x5c/0xc8
Aug 13 21:30:22 droid kernel: [<ffffff8009083a1c>] __sys_trace_return+0x0/0x4
Aug 13 21:30:26 droid kernel: Bad mode in Error handler detected on CPU5, code 0xbf000002 -- SError
Aug 13 21:30:26 droid kernel: FAR:7f899c2fe0
Aug 13 21:30:26 droid kernel: reg              value       pfn  reg              value       pfn
Aug 13 21:30:26 droid kernel: r0 :  0000007f899c2fe0     f5425  r1 :  0000000000000000  --------
Aug 13 21:30:26 droid kernel: r2 :  0000000000000028  --------  r3 :  0000000000000264  --------
Aug 13 21:30:26 droid kernel: r4 :  0000007f899c3008     d7a9d  r5 :  00000055bd8915d2     ea810
Aug 13 21:30:26 droid kernel: r6 :  00000000000081fd  --------  r7 :  00000000000081fd  --------
Aug 13 21:30:26 droid kernel: r8 :  00000000000000f0  --------  r9 :  03ffffffffffffff  --------
Aug 13 21:30:26 droid kernel: r10:  0200000000000000  --------  r11:  0000000100000000  --------
Aug 13 21:30:26 droid kernel: r12:  0000000000000000  --------  r13:  0000000000000010  --------
Aug 13 21:30:26 droid kernel: r14:  0000000000000000  --------  r15:  0000000000000000  --------
Aug 13 21:30:26 droid kernel: r16:  00000055930704e8     eaf88  r17:  0000007f89e8fa00     c542c
Aug 13 21:30:26 droid kernel: r18:  0000007f89f5fa70     eae49  r19:  0000000000000010  --------
Aug 13 21:30:26 droid kernel: r20:  0000007fd1aa0c28     ea0b9  r21:  000000000000203c  --------
Aug 13 21:30:26 droid kernel: r22:  0000000000000000  --------  r23:  0000005593074000     eaf2e
Aug 13 21:30:26 droid kernel: r24:  0000005593070000     eaf88  r25:  00000055bd8a9490     d79cc
Aug 13 21:30:26 droid kernel: r26:  000000002fb23a79  --------  r27:  0000007f899c2ff0     f5425
Aug 13 21:30:26 droid kernel: r28:  0000005593070000     eaf88  r29:  0000007fd1aa0b30     ea0b9
Aug 13 21:30:26 droid kernel: r30:  0000005592fff6dc     a1db9  
Aug 13 21:30:26 droid kernel: pc :  0000007f89e8fa54     c542c
Aug 13 21:30:26 droid kernel: sp :  0000007fd1aa0b30     ea0b9
Aug 13 21:30:26 droid kernel: unused :  0000007f899c2fe0     f5425
Aug 13 21:30:26 droid kernel: Internal error: Oops - bad mode: 0 [#2] PREEMPT SMP
Aug 13 21:30:26 droid kernel: Modules linked in: xt_conntrack xt_TCPMSS xt_tcpmss xt_policy bridge stp llc fuse ip6table_filter ip6_tables iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM xt_tcpudp iptable_mangle rtc_pcf8563 ir_lirc_codec lirc_dev meson_ir i2c_meson_master sch_fq_codel xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 firmware media_clock ip_tables x_tables ipv6
Aug 13 21:30:26 droid kernel: CPU: 5 PID: 3546 Comm: rsync Tainted: G      D W       4.9.187-53 #1
Aug 13 21:30:26 droid kernel: Hardware name: Hardkernel ODROID-N2 (DT)
Aug 13 21:30:26 droid kernel: task: ffffffc034c0b800 task.stack: ffffffc09e930000
Aug 13 21:30:26 droid kernel: PC is at 0x7f89e8fa54
Aug 13 21:30:26 droid kernel: LR is at 0x5592fff6dc
Aug 13 21:30:26 droid kernel: vma for 7f89e8fa54:
Aug 13 21:30:26 droid kernel: 7f89e0c000-7f89f4c000 r-xp 00000000 b3:02 1049929 
Aug 13 21:30:26 droid kernel: /lib/aarch64-linux-gnu/libc-2.27.so
Aug 13 21:30:26 droid kernel: 
Aug 13 21:30:26 droid kernel: vma for 5592fff6dc:
Aug 13 21:30:26 droid kernel: 5592ff1000-559305f000 r-xp 00000000 b3:02 1054064 
Aug 13 21:30:26 droid kernel: /usr/bin/rsync
Aug 13 21:30:26 droid kernel: 
Aug 13 21:30:26 droid kernel: pc : [<0000007f89e8fa54>] lr : [<0000005592fff6dc>] pstate: 20000000
Aug 13 21:30:26 droid kernel: sp : 0000007fd1aa0b30
Aug 13 21:30:26 droid kernel: x29: 0000007fd1aa0b30 x28: 0000005593070000 
Aug 13 21:30:26 droid kernel: x27: 0000007f899c2ff0 x26: 000000002fb23a79 
Aug 13 21:30:26 droid kernel: x25: 00000055bd8a9490 x24: 0000005593070000 
Aug 13 21:30:26 droid kernel: x23: 0000005593074000 x22: 0000000000000000 
Aug 13 21:30:26 droid kernel: x21: 000000000000203c x20: 0000007fd1aa0c28 
Aug 13 21:30:26 droid kernel: x19: 0000000000000010 x18: 0000007f89f5fa70 
Aug 13 21:30:26 droid kernel: x17: 0000007f89e8fa00 x16: 00000055930704e8 
Aug 13 21:30:26 droid kernel: x15: 0000000000000000 x14: 0000000000000000 
Aug 13 21:30:26 droid kernel: x13: 0000000000000010 x12: 0000000000000000 
Aug 13 21:30:26 droid kernel: x11: 0000000100000000 x10: 0200000000000000 
Aug 13 21:30:26 droid kernel: x9 : 03ffffffffffffff x8 : 00000000000000f0 
Aug 13 21:30:26 droid kernel: x7 : 00000000000081fd x6 : 00000000000081fd 
Aug 13 21:30:26 droid kernel: x5 : 00000055bd8915d2 x4 : 0000007f899c3008 
Aug 13 21:30:26 droid kernel: x3 : 0000000000000264 x2 : 0000000000000028 
Aug 13 21:30:26 droid kernel: x1 : 0000000000000000 x0 : 0000007f899c2fe0 
Aug 13 21:30:26 droid kernel: 
                              PC: 0x7f89e8f9d4:
Aug 13 21:30:26 droid kernel: f9d4  aa0903f7 d65f03c0 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: f9f4  00000000 00000000 00000000 4e010c20 8b020004 f101805f 540003c8 f100405f
Aug 13 21:30:26 droid kernel: fa14  54000202 4e083c01 361800a2 f9000001 f81f8081 d65f03c0 d503201f 36100082
Aug 13 21:30:26 droid kernel: fa34  b9000001 b81fc081 d65f03c0 b4000082 39000001 36080042 781fe081 d65f03c0
Aug 13 21:30:26 droid kernel: fa54  3d800000 373000c2 3c9f0080 36280062 3d800400 3c9e0080 d65f03c0 3d800400
Aug 13 21:30:26 droid kernel: fa74  ad010000 ad3f0080 d65f03c0 d503201f 12001c21 927cec03 3d800000 f104005f
Aug 13 21:30:26 droid kernel: fa94  7a402820 54000160 cb030082 91004063 d1014042 ac820060 ad3f0060 f1010042
Aug 13 21:30:26 droid kernel: fab4  54ffffa8 ad3e0080 ad3f0080 d65f03c0 d503201f d53b00e5 3727fe85 12000ca5
Aug 13 21:30:26 droid kernel: 
                              LR: 0x5592fff65c:
Aug 13 21:30:26 droid kernel: f65c  11001260 7100035f 1a931013 f94053a0 b7f87ae0 b0000380 b00003b7 f9471c00
Aug 13 21:30:26 droid kernel: f67c  b9400000 340000c0 9122e2e0 b9402800 12140c00 7140101f 54003d00 f9401320
Aug 13 21:30:26 droid kernel: f69c  36100073 321e0273 11001273 0b130061 b90093a3 0b1b0021 93407e73 f0000262
Aug 13 21:30:26 droid kernel: f6bc  911c8042 93407c21 94011709 91006262 8b13001b 52800001 f9003ba0 97fff56e
Aug 13 21:30:26 droid kernel: f6dc  b98097a4 91006368 f94057a1 aa0803e0 aa0403e2 f9003fa4 97fff45b aa0003e8
Aug 13 21:30:26 droid kernel: f6fc  121702a7 b94093a3 a94713a9 36480095 79402f60 321b0000 79002f60 9122e2e0
Aug 13 21:30:26 droid kernel: f71c  f9401000 f9000760 3400013a b0000380 79402f61 f947d000 32140021 79002f61
Aug 13 21:30:26 droid kernel: f73c  b9400000 cb20cb60 b81fc01a 9122e2e0 f94053a1 b9001361 b9402806 b2407fe0
Aug 13 21:30:26 droid kernel: 
                              SP: 0x7fd1aa0ab0:
Aug 13 21:30:26 droid kernel: 0ab0  d1aa0b00 0000007f 9302c6cc 00000055 89fd97a8 0000007f d1aa0c28 0000007f
Aug 13 21:30:26 droid kernel: 0ad0  0000203c 00000000 00000000 00000000 93074000 00000055 0000000c 00000000
Aug 13 21:30:26 droid kernel: 0af0  d1aa0b30 0000007f 92fff6c8 00000055 00000010 00000000 d1aa0c28 0000007f
Aug 13 21:30:26 droid kernel: 0b10  0000203c 00000000 00000000 00000000 93074000 00000055 b1668000 3276f253
Aug 13 21:30:26 droid kernel: 0b30  d1aa1c30 0000007f 93006b70 00000055 bd8a9490 00000055 00000000 00000000
Aug 13 21:30:26 droid kernel: 0b50  93070000 00000055 93070000 00000055 930711bc 00000055 bd8a9430 00000055
Aug 13 21:30:26 droid kernel: 0b70  93070000 00000055 0000003a 00000000 0000203c 00000000 0000003c 00000000
Aug 13 21:30:26 droid kernel: 0b90  93070000 00000054 93070000 00000055 899c2fe0 0000007f 00000019 00000000
Aug 13 21:30:26 droid kernel: 
                              r0: 0x7f899c2f60:
Aug 13 21:30:26 droid kernel: 2f60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2f80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fa0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fc0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fe0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 3000  00000000 00000000 00000000 00000000 00000000 00000000 00000000 000003ec
Aug 13 21:30:26 droid kernel: 3020  000003ea 00000007 bd8a9290 00000055 53c28fcc 00000000 002c55b1 000081fd
Aug 13 21:30:26 droid kernel: 3040  616c694b 5020616c 636e6972 2d737365 632e3631 00007262 00000000 000003ec
Aug 13 21:30:26 droid kernel: 
                              r3: 0x1e4:
Aug 13 21:30:26 droid kernel: 01e4  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0204  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0224  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0244  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0264  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0284  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 02a4  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 02c4  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r4: 0x7f899c2f88:
Aug 13 21:30:26 droid kernel: 2f88  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fa8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fc8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fe8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 3008  00000000 00000000 00000000 00000000 00000000 000003ec 000003ea 00000007
Aug 13 21:30:26 droid kernel: 3028  bd8a9290 00000055 53c28fcc 00000000 002c55b1 000081fd 616c694b 5020616c
Aug 13 21:30:26 droid kernel: 3048  636e6972 2d737365 632e3631 00007262 00000000 000003ec 000003ea 00000007
Aug 13 21:30:26 droid kernel: 3068  bd8a9290 00000055 53c28f2f 00000000 0023e837 000081fd 616c694b 5020616c
Aug 13 21:30:26 droid kernel: 
                              r5: 0x55bd891552:
Aug 13 21:30:26 droid kernel: 1550  706f4b20 00006569 8b675310 075a3a8c 6f4b2030 00656970 20531000 593a8c88
Aug 13 21:30:26 droid kernel: 1570  00003901 97e85310 02593a8c 00003031 88205310 07593a8c 6f4b2036 00656970
Aug 13 21:30:26 droid kernel: 1590  0e531000 593a8c94 00373102 40531000 5a3a8c8d 4b203107 6569706f 53100000
Aug 13 21:30:26 droid kernel: 15b0  3c8c88db 2e0b4920 53442e5f 6f74535f 00006572 ef835a10 3a79f042 81fd2fb2
Aug 13 21:30:26 droid kernel: 15d0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 15f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 1610  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 1630  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 1650  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 
                              r6: 0x817d:
Aug 13 21:30:26 droid kernel: 817c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 819c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 81bc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 81dc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 81fc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 821c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 823c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 825c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 827c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r7: 0x817d:
Aug 13 21:30:26 droid kernel: 817c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 819c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 81bc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 81dc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 81fc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 821c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 823c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 825c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 827c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r8: 0x70:
Aug 13 21:30:26 droid kernel: 0070  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0090  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 00b0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 00d0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 00f0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0110  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0130  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0150  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r9: 0x3ffffffffffff7f:
Aug 13 21:30:26 droid kernel: ff7c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ff9c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffbc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffdc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: fffc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 001c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 003c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 005c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 007c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r10: 0x1ffffffffffff80:
Aug 13 21:30:26 droid kernel: ff80  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffa0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffc0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffe0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0000  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0020  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0040  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0060  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r11: 0xffffff80:
Aug 13 21:30:26 droid kernel: ff80  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffa0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffc0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffe0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0000  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0020  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0040  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0060  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r16: 0x5593070468:
Aug 13 21:30:26 droid kernel: 0468  89e36f98 0000007f 89e6e458 0000007f 89eea9a0 0000007f 89e9dc50 0000007f
Aug 13 21:30:26 droid kernel: 0488  89e831d0 0000007f 89f836f8 0000007f 89edd7c0 0000007f 89ece460 0000007f
Aug 13 21:30:26 droid kernel: 04a8  89ece5e8 0000007f 89eed690 0000007f 89eea728 0000007f 89f82280 0000007f
Aug 13 21:30:26 droid kernel: 04c8  89e3ede0 0000007f 89e8771c 0000007f 89e2c600 0000007f 89f83340 0000007f
Aug 13 21:30:26 droid kernel: 04e8  89e8fa00 0000007f 89e87998 0000007f 89e9dcc0 0000007f 89f6a2b8 0000007f
Aug 13 21:30:26 droid kernel: 0508  89eacdb0 0000007f 89eaf600 0000007f 89ecec88 0000007f 89eae530 0000007f
Aug 13 21:30:26 droid kernel: 0528  89ece4a0 0000007f 89edd008 0000007f 89e76b70 0000007f 89e6a7e0 0000007f
Aug 13 21:30:26 droid kernel: 0548  89e83e58 0000007f 89f6b970 0000007f 89e88e30 0000007f 89ea9738 0000007f
Aug 13 21:30:26 droid kernel: 
                              r17: 0x7f89e8f980:
Aug 13 21:30:26 droid kernel: f980  92400c8e cb0e0084 a9ff5c96 a93f1ca6 cb0e0042 a9ff1c86 cb0e00a5 d1010042
Aug 13 21:30:26 droid kernel: f9a0  f1008042 a93f5cb6 a9ff5c96 a9be1ca6 a9ff1c86 54ffff68 a9412c2a a93f5cb6
Aug 13 21:30:26 droid kernel: f9c0  a9405c36 a93e1ca6 a9012c0a a9005c16 aa0803f6 aa0903f7 d65f03c0 00000000
Aug 13 21:30:26 droid kernel: f9e0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: fa00  4e010c20 8b020004 f101805f 540003c8 f100405f 54000202 4e083c01 361800a2
Aug 13 21:30:26 droid kernel: fa20  f9000001 f81f8081 d65f03c0 d503201f 36100082 b9000001 b81fc081 d65f03c0
Aug 13 21:30:26 droid kernel: fa40  b4000082 39000001 36080042 781fe081 d65f03c0 3d800000 373000c2 3c9f0080
Aug 13 21:30:26 droid kernel: fa60  36280062 3d800400 3c9e0080 d65f03c0 3d800400 ad010000 ad3f0080 d65f03c0
Aug 13 21:30:26 droid kernel: 
                              r18: 0x7f89f5f9f0:
Aug 13 21:30:26 droid kernel: f9f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: fa10  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: fa30  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: fa50  00000000 00000000 00000000 00000000 00000000 00000000 89f5c138 0000007f
Aug 13 21:30:26 droid kernel: fa70  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: fa90  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: fab0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: fad0  bd8a94e0 00000055 bd8a6000 00000055 89f5fad0 0000007f 89f5fad0 0000007f
Aug 13 21:30:26 droid kernel: 
                              r20: 0x7fd1aa0ba8:
Aug 13 21:30:26 droid kernel: 0ba8  00000019 00000000 93070000 00000055 93070000 00000055 00000000 0000000c
Aug 13 21:30:26 droid kernel: 0bc8  ffffffff 00000049 00001000 00000000 d1aa0c71 0000007f 93084ef0 00000055
Aug 13 21:30:26 droid kernel: 0be8  00000039 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 0c08  00000000 00000000 9304eb08 00000055 d1aa0c28 0000007f 00000000 00000000
Aug 13 21:30:26 droid kernel: 0c28  67616553 2f657461 696d6f43 302f7363 6d6f632d 73756c2d 30302f74 6d6f632d
Aug 13 21:30:26 droid kernel: 0c48  2f78782d 632d3330 6e2d6d6f 672d7565 64616c65 4b2f6e65 6c616c69 72502061
Aug 13 21:30:26 droid kernel: 0c68  65636e69 622d7373 2e5f2e2f 535f5344 65726f74 73736500 2031312d 69706f4b
Aug 13 21:30:26 droid kernel: 0c88  00720065 65636e69 762d7373 694b2f35 616c616c 69725020 7365636e 30762d73
Aug 13 21:30:26 droid kernel: 
                              r21: 0x1fbc:
Aug 13 21:30:26 droid kernel: 1fbc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 1fdc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 1ffc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 201c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 203c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 205c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 207c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 209c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r23: 0x5593073f80:
Aug 13 21:30:26 droid kernel: 3f80  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3fa0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3fc0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3fe0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 4000  00080006 00000000 93056f50 00000055 93056f58 00000055 00080104 00000000
Aug 13 21:30:26 droid kernel: 4020  93056c28 00000055 93056f90 00000055 00080203 00000000 93056d20 00000055
Aug 13 21:30:26 droid kernel: 4040  93056fb8 00000055 00010305 00000000 93056ff0 00000055 93056ff8 00000055
Aug 13 21:30:26 droid kernel: 4060  000c0404 00000000 93057028 00000055 93057030 00000055 000c0505 00000000
Aug 13 21:30:26 droid kernel: 
                              r24: 0x559306ff80:
Aug 13 21:30:26 droid kernel: ff80  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffa0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffc0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffe0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0000  93046428 00000055 00200008 01000080 93046428 00000055 00800020 04000102
Aug 13 21:30:26 droid kernel: 0020  93046428 00000055 01020020 10000102 93046428 00000055 9305ccf0 00000055
Aug 13 21:30:26 droid kernel: 0040  9305cd00 00000055 93059a28 00000055 9305cd10 00000055 9305cd20 00000055
Aug 13 21:30:26 droid kernel: 0060  9305cd30 00000055 9305cd40 00000055 9305cd58 00000055 9305cd68 00000055
Aug 13 21:30:26 droid kernel: 
                              r25: 0x55bd8a9410:
Aug 13 21:30:26 droid kernel: 9410  000002b2 00000024 00000039 00000000 00000000 00000000 00000061 00000000
Aug 13 21:30:26 droid kernel: 9430  67616553 2f657461 696d6f43 302f7363 6d6f632d 73756c2d 30302f74 6d6f632d
Aug 13 21:30:26 droid kernel: 9450  2f78782d 632d3330 6e2d6d6f 672d7565 64616c65 4b2f6e65 6c616c69 72502061
Aug 13 21:30:26 droid kernel: 9470  65636e69 622d7373 899c3300 0000007f 00000000 00000000 00000061 00000000
Aug 13 21:30:26 droid kernel: 9490  00000000 00000000 bd8a93d0 00000055 890ab010 0000007f 00000000 00000000
Aug 13 21:30:26 droid kernel: 94b0  bd8a4810 00000055 899c3018 0000007f 00000018 00008000 00000000 00000000
Aug 13 21:30:26 droid kernel: 94d0  000002c7 00000025 00000000 00000000 00000000 00000000 0001bb21 00000000
Aug 13 21:30:26 droid kernel: 94f0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 
                              r26: 0x2fb239f9:
Aug 13 21:30:26 droid kernel: 39f8  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3a18  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3a38  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3a58  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3a78  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3a98  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3ab8  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3ad8  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 3af8  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 
                              r27: 0x7f899c2f70:
Aug 13 21:30:26 droid kernel: 2f70  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2f90  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fb0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2fd0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 2ff0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:26 droid kernel: 3010  00000000 00000000 00000000 000003ec 000003ea 00000007 bd8a9290 00000055
Aug 13 21:30:26 droid kernel: 3030  53c28fcc 00000000 002c55b1 000081fd 616c694b 5020616c 636e6972 2d737365
Aug 13 21:30:26 droid kernel: 3050  632e3631 00007262 00000000 000003ec 000003ea 00000007 bd8a9290 00000055
Aug 13 21:30:26 droid kernel: 
                              r28: 0x559306ff80:
Aug 13 21:30:26 droid kernel: ff80  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffa0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffc0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: ffe0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:26 droid kernel: 0000  93046428 00000055 00200008 01000080 93046428 00000055 00800020 04000102
Aug 13 21:30:26 droid kernel: 0020  93046428 00000055 01020020 10000102 93046428 00000055 9305ccf0 00000055
Aug 13 21:30:26 droid kernel: 0040  9305cd00 00000055 93059a28 00000055 9305cd10 00000055 9305cd20 00000055
Aug 13 21:30:26 droid kernel: 0060  9305cd30 00000055 9305cd40 00000055 9305cd58 00000055 9305cd68 00000055
Aug 13 21:30:26 droid kernel: 
Aug 13 21:30:26 droid kernel: Process rsync (pid: 3546, stack limit = 0xffffffc09e930000)
Aug 13 21:30:26 droid kernel: ---[ end trace a345c92a52ea53db ]---
Aug 13 21:30:26 droid kernel: BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:21
Aug 13 21:30:26 droid kernel: in_atomic(): 0, irqs_disabled(): 128, pid: 3546, name: rsync
Aug 13 21:30:26 droid kernel: CPU: 5 PID: 3546 Comm: rsync Tainted: G      D W       4.9.187-53 #1
Aug 13 21:30:26 droid kernel: Hardware name: Hardkernel ODROID-N2 (DT)
Aug 13 21:30:26 droid kernel: Call trace:
Aug 13 21:30:26 droid kernel: [<ffffff800908b000>] dump_backtrace+0x0/0x268
Aug 13 21:30:26 droid kernel: [<ffffff800908b2ec>] show_stack+0x24/0x30
Aug 13 21:30:26 droid kernel: [<ffffff800943a738>] dump_stack+0xb4/0xe4
Aug 13 21:30:26 droid kernel: [<ffffff80090d684c>] ___might_sleep+0x104/0x138
Aug 13 21:30:26 droid kernel: [<ffffff80090d68d8>] __might_sleep+0x58/0x90
Aug 13 21:30:26 droid kernel: [<ffffff8009be8384>] down_read+0x2c/0x78
Aug 13 21:30:26 droid kernel: [<ffffff80090cd244>] blocking_notifier_call_chain+0x4c/0x88
Aug 13 21:30:26 droid kernel: [<ffffff8009121188>] profile_task_exit+0x30/0x40
Aug 13 21:30:26 droid kernel: [<ffffff80090aac1c>] do_exit+0x24/0xa58
Aug 13 21:30:26 droid kernel: [<ffffff800908b4b4>] die+0x1bc/0x1c0
Aug 13 21:30:26 droid kernel: [<ffffff800908bb18>] bad_mode+0xb0/0xc8
Aug 13 21:30:26 droid kernel: [<0000005592fff6dc>] 0x5592fff6dc
Aug 13 21:30:26 droid kernel: Bad mode in Error handler detected on CPU5, code 0xbf000002 -- SError
Aug 13 21:30:26 droid kernel: FAR:7fd1aa1f00
Aug 13 21:30:26 droid kernel: reg              value       pfn  reg              value       pfn
Aug 13 21:30:26 droid kernel: r0 :  0000000000000003  --------  r1 :  00000055930573f8     82c6a
Aug 13 21:30:26 droid kernel: r2 :  0000005593084d5c     f23c5  r3 :  000000559304ef38     95dd4
Aug 13 21:30:26 droid kernel: r4 :  0000000000000000  --------  r5 :  cccccccccccccccd  --------
Aug 13 21:30:26 droid kernel: r6 :  0000000000000000  --------  r7 :  0000000000000000  --------
Aug 13 21:30:26 droid kernel: r8 :  0000000000001560  --------  r9 :  0000000000001530  --------
Aug 13 21:30:26 droid kernel: r10:  0000000000001400  --------  r11:  0000000000000000  --------
Aug 13 21:30:28 droid kernel: r12:  0000000000000008  --------  r13:  0000000000001600  --------
Aug 13 21:30:28 droid kernel: r14:  0000000000000000  --------  r15:  0000000000000000  --------
Aug 13 21:30:28 droid kernel: r16:  0000000000001560  --------  r17:  0000007f89ecea78     c4ae2
Aug 13 21:30:28 droid kernel: r18:  0000000000000000  --------  r19:  0000005593084d5c     f23c5
Aug 13 21:30:28 droid kernel: r20:  0000000000000000  --------  r21:  00000055930843c8     f23c5
Aug 13 21:30:28 droid kernel: r22:  0000000000000001  --------  r23:  0000007fd1aa35c8     f2381
Aug 13 21:30:28 droid kernel: r24:  0000007fd1aa3548     f2381  r25:  0000000000000004  --------
Aug 13 21:30:28 droid kernel: r26:  00000000ffffffff  --------  r27:  0000007fd1aa3648     f2381
Aug 13 21:30:28 droid kernel: r28:  00000055930743d0      538f  r29:  0000007fd1aa3460     f2381
Aug 13 21:30:28 droid kernel: r30:  0000005593028a6c     82a4c  
Aug 13 21:30:28 droid kernel: pc :  00000055930209f8     95310
Aug 13 21:30:28 droid kernel: sp :  0000007fd1aa1f00     f5426
Aug 13 21:30:28 droid kernel: unused :  0000007fd1aa1f00     f5426
Aug 13 21:30:28 droid kernel: Internal error: Oops - bad mode: 0 [#3] PREEMPT SMP
Aug 13 21:30:28 droid kernel: Modules linked in: xt_conntrack xt_TCPMSS xt_tcpmss xt_policy bridge stp llc fuse ip6table_filter ip6_tables iptable_filter ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_nat iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack xt_CHECKSUM xt_tcpudp iptable_mangle rtc_pcf8563 ir_lirc_codec lirc_dev meson_ir i2c_meson_master sch_fq_codel xfrm4_tunnel tunnel4 ipcomp xfrm_ipcomp esp4 ah4 firmware media_clock ip_tables x_tables ipv6
Aug 13 21:30:28 droid kernel: CPU: 5 PID: 3520 Comm: rsync Tainted: G      D W       4.9.187-53 #1
Aug 13 21:30:28 droid kernel: Hardware name: Hardkernel ODROID-N2 (DT)
Aug 13 21:30:28 droid kernel: task: ffffffc034c0aa00 task.stack: ffffffc0afa4c000
Aug 13 21:30:28 droid kernel: PC is at 0x55930209f8
Aug 13 21:30:28 droid kernel: LR is at 0x5593028a6c
Aug 13 21:30:28 droid kernel: vma for 55930209f8:
Aug 13 21:30:28 droid kernel: 5592ff1000-559305f000 r-xp 00000000 b3:02 1054064 
Aug 13 21:30:28 droid kernel: /usr/bin/rsync
Aug 13 21:30:28 droid kernel: 
Aug 13 21:30:28 droid kernel: vma for 5593028a6c:
Aug 13 21:30:28 droid kernel: 5592ff1000-559305f000 r-xp 00000000 b3:02 1054064 
Aug 13 21:30:28 droid kernel: /usr/bin/rsync
Aug 13 21:30:28 droid kernel: 
Aug 13 21:30:28 droid kernel: pc : [<00000055930209f8>] lr : [<0000005593028a6c>] pstate: 20000000
Aug 13 21:30:28 droid kernel: sp : 0000007fd1aa1f00
Aug 13 21:30:28 droid kernel: x29: 0000007fd1aa3460 x28: 00000055930743d0 
Aug 13 21:30:28 droid kernel: x27: 0000007fd1aa3648 x26: 00000000ffffffff 
Aug 13 21:30:28 droid kernel: x25: 0000000000000004 x24: 0000007fd1aa3548 
Aug 13 21:30:28 droid kernel: x23: 0000007fd1aa35c8 x22: 0000000000000001 
Aug 13 21:30:28 droid kernel: x21: 00000055930843c8 x20: 0000000000000000 
Aug 13 21:30:28 droid kernel: x19: 0000005593084d5c x18: 0000000000000000 
Aug 13 21:30:28 droid kernel: x17: 0000007f89ecea78 x16: 0000000000001560 
Aug 13 21:30:28 droid kernel: x15: 0000000000000000 x14: 0000000000000000 
Aug 13 21:30:28 droid kernel: x13: 0000000000001600 x12: 0000000000000008 
Aug 13 21:30:28 droid kernel: x11: 0000000000000000 x10: 0000000000001400 
Aug 13 21:30:28 droid kernel: x9 : 0000000000001530 x8 : 0000000000001560 
Aug 13 21:30:28 droid kernel: x7 : 0000000000000000 x6 : 0000000000000000 
Aug 13 21:30:28 droid kernel: x5 : cccccccccccccccd x4 : 0000000000000000 
Aug 13 21:30:28 droid kernel: x3 : 000000559304ef38 x2 : 0000005593084d5c 
Aug 13 21:30:28 droid kernel: x1 : 00000055930573f8 x0 : 0000000000000003 
Aug 13 21:30:28 droid kernel: 
                              PC: 0x5593020978:
Aug 13 21:30:28 droid kernel: 0978  97ffc86e f9001fb8 f9002fbc 97ff7123 b0000183 b0000181 b0000180 911ee063
Aug 13 21:30:28 droid kernel: 0998  52802262 911f0021 9120e000 f9001fb8 f9002fbc 97ff7259 f9002fbc 9000029c
Aug 13 21:30:28 droid kernel: 09b8  f9475f80 b9400000 34ffe520 121d7ae0 528000c1 7100081f 52800080 1a800020
Aug 13 21:30:28 droid kernel: 09d8  b9007ba0 17fffed2 d282ac10 cb3063ff d282ac08 d282800a d282a609 d282c00d
Aug 13 21:30:28 droid kernel: 09f8  a9007bfd 910003fd 8b0803a8 8b0903a9 a9025bf5 2a0003f6 a906a3a8 8b0a03a0
Aug 13 21:30:28 droid kernel: 0a18  128005e8 a90153f3 f9003fa9 90000293 b90083a8 12800fe8 f9001bf7 9102a3b5
Aug 13 21:30:28 droid kernel: 0a38  b90087a8 d2827ff7 a946afaa aa0103f4 a947a7a8 3d802c00 3d803001 3d803402
Aug 13 21:30:28 droid kernel: 0a58  3d803803 3d803c04 8b0d03a0 f946366c 3c900005 3c910006 3c920007 f90a9ba2
Aug 13 21:30:28 droid kernel: 
                              LR: 0x55930289ec:
Aug 13 21:30:28 droid kernel: 89ec  17fffd64 b9429ac1 b0000160 913c4000 f900dec0 7100003f 1a9f17e0 b90372c0
Aug 13 21:30:28 droid kernel: 8a0c  17fffd80 a9be7bfd 900002e1 12000000 910003fd b943c821 f9000bf3 7100003f
Aug 13 21:30:28 droid kernel: 8a2c  1a9f1000 35000280 90000240 d2800002 52800001 f9452000 f9400c00 94006a5c
Aug 13 21:30:28 droid kernel: 8a4c  aa0003f3 97ff8326 aa0003e3 aa1303e2 52800060 f0000161 910fe021 97ffdfde
Aug 13 21:30:28 droid kernel: 8a6c  f0000161 52801d62 910fc021 52800180 97ffa82d 540000ec 52803e93 52800280
Aug 13 21:30:28 droid kernel: 8a8c  97ffc8c9 71000673 54ffffa1 17ffffe7 f0000161 52801c22 910fc021 52800000
Aug 13 21:30:28 droid kernel: 8aac  97ffa821 a9bd7bfd 90000242 910003fd f9477c42 a90153f3 aa0003f3 aa0103f4
Aug 13 21:30:28 droid kernel: 8acc  b9400040 340001c0 90000240 f946d800 79c04400 71000c1f 5400012d 90000260
Aug 13 21:30:28 droid kernel: 
                              SP: 0x7fd1aa1e80:
Aug 13 21:30:28 droid kernel: 1e80  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 1ea0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 1ec0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 1ee0  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 1f00  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 1f20  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 1f40  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 1f60  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 
                              r1: 0x5593057378:
Aug 13 21:30:28 droid kernel: 7378  64616572 6d5f615f 00006773 00000000 65666173 6972775f 00006574 00000000
Aug 13 21:30:28 droid kernel: 7398  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 73b8  01010101 01010101 01010101 01010101 02020202 02020202 03030303 06050404
Aug 13 21:30:28 droid kernel: 73d8  655f6f69 6d5f646e 69746c75 78656c70 006e695f 00000000 632e6f69 00000000
Aug 13 21:30:28 droid kernel: 73f8  6e797372 63203a63 656e6e6f 6f697463 6e75206e 65707865 64657463 6320796c
Aug 13 21:30:28 droid kernel: 7418  65736f6c 25282064 79622073 20736574 65636572 64657669 206f7320 29726166
Aug 13 21:30:28 droid kernel: 7438  73255b20 00000a5d 6b636f73 00007465 7373656d 20656761 00006466 00000000
Aug 13 21:30:28 droid kernel: 7458  63746162 69662068 0000656c 00000000 25206466 00000064 75626f69 756f2e66
Aug 13 21:30:28 droid kernel: 
                              r2: 0x5593084cdc:
Aug 13 21:30:28 droid kernel: 4cdc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4cfc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d1c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d3c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d5c  00343633 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d7c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d9c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4dbc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 
                              r3: 0x559304eeb8:
Aug 13 21:30:28 droid kernel: eeb8  69207325 70752073 61646f74 000a6574 696e6966 6e696873 73252067 0000000a
Aug 13 21:30:28 droid kernel: eed8  616e6572 676e696d 20732520 25206f74 00000a73 00000000 79706f63 00000000
Aug 13 21:30:28 droid kernel: eef8  25207325 3e2d2073 73252220 00000022 616e6572 2520656d 3e2d2073 73252220
Aug 13 21:30:28 droid kernel: ef18  00000022 00000000 616e6572 0000656d 65696c63 0000746e 646e6573 00007265
Aug 13 21:30:28 droid kernel: ef38  656e6567 6f746172 00000072 00000000 65636552 72657669 00000000 00000000
Aug 13 21:30:28 droid kernel: ef58  76726573 00007265 65636572 72657669 00000000 00000000 434f4c5b 5d454c41
Aug 13 21:30:28 droid kernel: ef78  00000000 00000000 2067736d 63656863 676e696b 61697620 70736920 746e6972
Aug 13 21:30:28 droid kernel: ef98  28202928 6e6f6369 706f5f76 22286e65 2c227325 73252220 65202922 6f6e7272
Aug 13 21:30:28 droid kernel: 
                              r5: 0xcccccccccccccc4d:
Aug 13 21:30:28 droid kernel: cc4c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: cc6c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: cc8c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: ccac  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: cccc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: ccec  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: cd0c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: cd2c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: cd4c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 
                              r8: 0x14e0:
Aug 13 21:30:28 droid kernel: 14e0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1500  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1520  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1540  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1560  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1580  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 15a0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 15c0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 
                              r9: 0x14b0:
Aug 13 21:30:28 droid kernel: 14b0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 14d0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 14f0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1510  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1530  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1550  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1570  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1590  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 
                              r10: 0x1380:
Aug 13 21:30:28 droid kernel: 1380  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 13a0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 13c0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 13e0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1400  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1420  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1440  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1460  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 
                              r13: 0x1580:
Aug 13 21:30:28 droid kernel: 1580  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 15a0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 15c0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 15e0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1600  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1620  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1640  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1660  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 
                              r16: 0x14e0:
Aug 13 21:30:28 droid kernel: 14e0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1500  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1520  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1540  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1560  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 1580  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 15a0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 15c0  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 
                              r17: 0x7f89ece9f8:
Aug 13 21:30:28 droid kernel: e9f8  128000e5 910143a7 b9404ba3 a9019fa7 93407c00 f90017a6 93407c42 29067fa5
Aug 13 21:30:28 droid kernel: ea18  d2800708 d4000001 b140041f 54fffda9 90000481 f9470c21 d53bd042 4b0003e3
Aug 13 21:30:28 droid kernel: ea38  12800000 b8216843 17ffffe6 94007b31 2a0203e3 373000c3 12122043 120a6063
Aug 13 21:30:28 droid kernel: ea58  7150107f 54000040 17ffff7a a9bf7bfd b0000340 91286000 910003fd 94007b47
Aug 13 21:30:28 droid kernel: ea78  a9bd7bfd 90000483 910003fd f946b063 f9000bf3 93407c13 b9400060 35000160
Aug 13 21:30:28 droid kernel: ea98  aa1303e0 d28007e8 d4000001 aa0003f3 b140041f 54000328 aa1303e0 f9400bf3
Aug 13 21:30:28 droid kernel: eab8  a8c37bfd d65f03c0 a901d7b4 aa0103f5 aa0203f4 94006a43 aa1503e1 2a0003e3
Aug 13 21:30:28 droid kernel: ead8  aa1403e2 aa1303e0 d28007e8 d4000001 aa0003f3 b140041f 540001e8 2a0303e0
Aug 13 21:30:28 droid kernel: 
                              r19: 0x5593084cdc:
Aug 13 21:30:28 droid kernel: 4cdc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4cfc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d1c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d3c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d5c  00343633 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d7c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4d9c  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4dbc  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 
                              r21: 0x5593084348:
Aug 13 21:30:28 droid kernel: 4348  00000000 00000001 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4368  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4388  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 43a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 43c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 43e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4408  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4428  00000000 00000000 00000000 00000000 00000001 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 
                              r23: 0x7fd1aa3548:
Aug 13 21:30:28 droid kernel: 3548  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3568  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3588  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 35a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 35c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 35e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3608  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3628  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 
                              r24: 0x7fd1aa34c8:
Aug 13 21:30:28 droid kernel: 34c8  9307c9d0 00000055 89a70e50 0000007f 0000000d 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 34e8  00000000 00000000 00000000 00000000 93070000 00000055 00000004 00000000
Aug 13 21:30:28 droid kernel: 3508  00000011 00000000 00000000 00000000 00000039 00000000 000244d9 00000000
Aug 13 21:30:28 droid kernel: 3528  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3548  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3568  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3588  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 35a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 
                              r26: 0xffffff7f:
Aug 13 21:30:28 droid kernel: ff7c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: ff9c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: ffbc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: ffdc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: fffc  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 001c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 003c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 005c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 007c  ******** ******** ******** ******** ******** ******** ******** ********
Aug 13 21:30:28 droid kernel: 
                              r27: 0x7fd1aa35c8:
Aug 13 21:30:28 droid kernel: 35c8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 35e8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3608  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3628  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3648  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3668  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 3688  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 36a8  00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 
                              r28: 0x5593074350:
Aug 13 21:30:28 droid kernel: 4350  00000000 00000000 93056f30 00000055 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4370  930571d0 00000055 930571f8 00000055 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4390  00000000 00000000 00000000 00000000 00000000 00000000 93057220 00000055
Aug 13 21:30:28 droid kernel: 43b0  93057258 00000055 930572b0 00000055 930572f0 00000055 9304eda0 00000055
Aug 13 21:30:28 droid kernel: 43d0  00000000 00000001 bd89c7a0 00000055 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 43f0  00008000 00000000 bd88c790 00000055 00000000 00000000 00000004 00000000
Aug 13 21:30:28 droid kernel: 4410  00010000 00000000 bd884780 00000055 00000000 00000000 00000000 00000000
Aug 13 21:30:28 droid kernel: 4430  00008000 00000000 fffffffe 00000001 ffffffff 00000004 00000000 00000000
Aug 13 21:30:28 droid kernel: 
Aug 13 21:30:28 droid kernel: Process rsync (pid: 3520, stack limit = 0xffffffc0afa4c000)
Aug 13 21:30:28 droid kernel: ---[ end trace a345c92a52ea53dc ]---
Aug 13 21:30:28 droid kernel: BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:21
Aug 13 21:30:28 droid kernel: in_atomic(): 0, irqs_disabled(): 128, pid: 3520, name: rsync
Aug 13 21:30:28 droid kernel: CPU: 5 PID: 3520 Comm: rsync Tainted: G      D W       4.9.187-53 #1
Aug 13 21:30:28 droid kernel: Hardware name: Hardkernel ODROID-N2 (DT)
Aug 13 21:30:28 droid kernel: Call trace:
Aug 13 21:30:28 droid kernel: [<ffffff800908b000>] dump_backtrace+0x0/0x268
Aug 13 21:30:28 droid kernel: [<ffffff800908b2ec>] show_stack+0x24/0x30
Aug 13 21:30:28 droid kernel: [<ffffff800943a738>] dump_stack+0xb4/0xe4
Aug 13 21:30:28 droid kernel: [<ffffff80090d684c>] ___might_sleep+0x104/0x138
Aug 13 21:30:28 droid kernel: [<ffffff80090d68d8>] __might_sleep+0x58/0x90
Aug 13 21:30:28 droid kernel: [<ffffff8009be8384>] down_read+0x2c/0x78
Aug 13 21:30:28 droid kernel: [<ffffff80090cd244>] blocking_notifier_call_chain+0x4c/0x88
Aug 13 21:30:28 droid kernel: [<ffffff8009121188>] profile_task_exit+0x30/0x40
Aug 13 21:30:28 droid kernel: [<ffffff80090aac1c>] do_exit+0x24/0xa58
Aug 13 21:30:28 droid kernel: [<ffffff800908b4b4>] die+0x1bc/0x1c0
Aug 13 21:30:28 droid kernel: [<ffffff800908bb18>] bad_mode+0xb0/0xc8
Aug 13 21:30:28 droid kernel: [<0000005593028a6c>] 0x5593028a6c
[/quote]

User avatar
mad_ady
Posts: 6384
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 107 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by mad_ady » Wed Aug 14, 2019 1:52 pm

@odroid: the webcams and gucview may use mjpeg to do the streaming. You need to force YUV mode to drastically increase throughput through the usb bus. Since the cameras are probably usb2, I doubt you can use more than a couple till you saturate the bus. I think you can access the yuv stream with ffmpeg/ffplay. I'd start with one camera and slowly add the others. You may experience image freezes/corrupt frames when bandwidth is tight.

trex_daemon
Posts: 22
Joined: Wed Jul 17, 2019 4:47 pm
languages_spoken: english
ODROIDs: C2, N2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by trex_daemon » Wed Aug 14, 2019 6:00 pm

I could test it with the tv tuners, but I am running coreelec on my Odroid N2.
I assume in the test builds it would be possible to include a 5.3 kernel, I don't know if they are doing major modifications.
Currently in the latest testbuild, Corelec is still using 4.9.182

User avatar
mad_ady
Posts: 6384
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 107 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by mad_ady » Wed Aug 14, 2019 6:06 pm

To narrow down the issues you should try it on the ubuntu image with HK's latest kernel. Coreelec may have custom changes/may be missing some features.

User avatar
tobetter
Posts: 3786
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: X, X2, U2, U3, XU3, C1
Location: Paju, South Korea
Has thanked: 29 times
Been thanked: 130 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by tobetter » Wed Aug 14, 2019 6:21 pm

trex_daemon wrote:
Wed Aug 14, 2019 6:00 pm
I could test it with the tv tuners, but I am running coreelec on my Odroid N2.
I assume in the test builds it would be possible to include a 5.3 kernel, I don't know if they are doing major modifications.
Currently in the latest testbuild, Corelec is still using 4.9.182
The kernel version is not the matter, the patches to set sg_tablesize=2 in the Linux kernel and U-boot to set CLK source to 222MHz are already merged. One thing missing change from HK Ubuntu is missing is to set IRQ to the big-core, which is not yet pushed since I have no proper change for CE. Please do below commands and share the output if you have the proper fixes.

Code: Select all

$ dmesg | grep sg_tablesize
$ cat /sys/kernel/debug/aml_clkmsr/clkmsr | grep clk81
If you like to test after assigning the IRQ to big-core, please run this command manually.

Code: Select all

# USB3_IRQ=`cat /proc/interrupts | grep "xhci-hcd:usb1" | awk -F : '{print $1}' | sed s/" "//g`
# echo 4 > /proc/irq/${USB3_IRQ}/smp_affinity_list

trex_daemon
Posts: 22
Joined: Wed Jul 17, 2019 4:47 pm
languages_spoken: english
ODROIDs: C2, N2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by trex_daemon » Wed Aug 14, 2019 9:54 pm

Code: Select all

CoreELEC:~/.config # dmesg | grep sg_tablesize
[    1.145607@5] usb: xhci: determined sg_tablesize: 2
[    1.146237@2] usb: xhci: determined sg_tablesize: 2
CoreELEC:~ #  cat /sys/kernel/debug/aml_clkmsr/clkmsr | grep clk81
[ 7][ 222000000]clk81
is already set.
I've added the commands to set the usb irq affinity to 4 and also verified that it's set, but still no change.
Last edited by trex_daemon on Wed Aug 14, 2019 11:36 pm, edited 1 time in total.

User avatar
mad_ady
Posts: 6384
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 107 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by mad_ady » Wed Aug 14, 2019 10:01 pm

clk81 is not set based on your output.

alvarow
Posts: 26
Joined: Wed Apr 10, 2019 10:44 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 10 times
Been thanked: 0
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by alvarow » Wed Aug 14, 2019 11:12 pm

I got clk81 set, sg_tablesize is 2 and the IRQ for USB3 is set to 4 ... I start and rsync (using rsyncd on the server side not SSH) over gigabit ethernet to a powered WD Elements USB3 drive (10tb is size matters) and I still freezes and reboots. The N2 is powered by the HK power supply, not USB.

I've been getting lots of ---[ cut here] --- as well, I am not sure if that is a result of added debugging to the latest kernels (I noticed it after the Aug 6th release) or if the panics are something else.

trex_daemon
Posts: 22
Joined: Wed Jul 17, 2019 4:47 pm
languages_spoken: english
ODROIDs: C2, N2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by trex_daemon » Wed Aug 14, 2019 11:36 pm

mad_ady wrote:
Wed Aug 14, 2019 10:01 pm
clk81 is not set based on your output.
It was left out of copy paste.
Is this the value it should have ?

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Thu Aug 15, 2019 3:58 am

elatllat wrote:
Wed Aug 14, 2019 11:20 am
...As far as I know the new img fixed crashing for everyone except
- avatarNighti [and alvarow] using rsync
- trex_daemon using 4 tv tuners
Nether have tried 5.3 or shared exact steps to reproduce...
avatarNighti and alvarow please let us know which of the following breaks USB3 on a fresh copy of the new img for you;

1)

Code: Select all

dd if=/dev/urandom of=test bs=1MB count=1000
echo 3 > /proc/sys/vm/drop_caches
sync
rsync test test2
dd if=/dev/urandom of=test bs=1MB count=1 seek=10
echo 3 > /proc/sys/vm/drop_caches
sync
rsync test test2
2)

Code: Select all

apt install mktorrent
dd if=/dev/urandom of=test bs=1M count=1000
echo 3 > /proc/sys/vm/drop_caches
sync
mktorrent -a 127.0.0.1 test
3)

Code: Select all

apt install iozone3
iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 test
4)

Code: Select all

apt install f3
f3write --end-at=10 ./
f3read --end-at=10 ./
and of course remember to double check

Code: Select all

grep clk81 /sys/kernel/debug/aml_clkmsr/clkmsr
cat /proc/cmdline
dmesg | grep sg_tablesize
lsusb -t
lsblk
free
and share whatever mkfs and mount commands were used.
Last edited by elatllat on Thu Aug 15, 2019 9:30 am, edited 1 time in total.
These users thanked the author elatllat for the post:
xabolcs (Thu Aug 15, 2019 8:55 am)

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Thu Aug 15, 2019 9:01 am

@elatllat: none of those. As long as you work on the same drive, it works. Looks like regular, sequential read/write on the same drive is not an issue.

But a real world scenario rsync from one drive to another drive with all the different file sizes is causing an issue.
These users thanked the author Nighti for the post:
Jay4692 (Thu Aug 15, 2019 9:03 am)

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Thu Aug 15, 2019 9:05 am

Nighti wrote:
Thu Aug 15, 2019 9:01 am
@elatllat: none of those. As long as you work on the same drive, it works....
Thanks for testing (yes those examples were previously effective on RAID), so how about these?

1.2)

Code: Select all

dd if=/dev/urandom of=/media/usb0/test bs=1MB count=1000
echo 3 > /proc/sys/vm/drop_caches
sync
rsync /media/usb0/test /media/usb1/test2
dd if=/dev/urandom of=/media/usb0/test bs=1MB count=1 seek=10
echo 3 > /proc/sys/vm/drop_caches
sync
rsync /media/usb0/test /media/usb1/test2
2.2)

Code: Select all

apt install mktorrent
dd if=/dev/urandom of=/media/usb0/test bs=1M count=1000
dd if=/dev/urandom of=/media/usb1/test2 bs=1M count=1000
echo 3 > /proc/sys/vm/drop_caches
sync
mktorrent -a 127.0.0.1 /media/usb0/test &
mktorrent -a 127.0.0.1 /media/usb1/test2
3.2)

Code: Select all

apt install iozone3
iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 /media/usb0/test &
iozone -e -I -a -s 100M -r 4k -r 16k -r 512k -r 1024k -r 16384k -i 0 -i 1 -i 2 /media/usb1/test2
4.2)

Code: Select all

apt install f3
f3write --end-at=10 /media/usb0/ &
f3write --end-at=10 /media/usb1/
f3read --end-at=10 /media/usb0/ &
f3read --end-at=10 /media/usb1/
Last edited by elatllat on Thu Aug 15, 2019 9:23 am, edited 2 times in total.

Jay4692
Posts: 13
Joined: Sat Jun 29, 2019 6:31 pm
languages_spoken: english, spanish
ODROIDs: N2
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Jay4692 » Thu Aug 15, 2019 9:18 am

For me what breaks USB 3 on a fresh image is rsync a large file or numerous files from one drive to another drive:

Code: Select all

rsync -av --progress /mnt/drive1/example_folder /mnt/drive2
rsync -v --progress /mnt/drive1/large_file /mnt/drive2
In my case I can easily break USB3 using the new image with a 32 Gb file which is an image of the whole SD Card of my system. Also transfering lots of archives and folders with -a.

My steps to reproduce the issue:

1- Create odroid user, change root password.
2- Create dirs on /mnt for both drives and chown them to user odroid.
3- Mount drives on etc/fstab with uuid and defaults,no-fail flags on the folders on /mnt.
4- rsync the big file from one drive to the other.

At least this is my experience, and none of my hardware is bad, drives and hub are running happily now on a RPi4.

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Thu Aug 15, 2019 9:22 am

Jay4692 wrote:
Thu Aug 15, 2019 9:18 am

Code: Select all

rsync -av --progress /mnt/drive1/example_folder /mnt/drive2
rsync -v --progress /mnt/drive1/large_file /mnt/drive2
Jay4692 that's not a reproducible test because we don't have your data.
Please make test files with with random data so others can run the same test on similar data.
(like try my #1.2 or if that fails to crash USB3 then try again by replacing count=1000 with count=32000)

also your "steps to reproduce the issue" are not unless they are commands we can copy paste, anything else is up for interpretation.
Last edited by elatllat on Thu Aug 15, 2019 9:46 am, edited 1 time in total.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Thu Aug 15, 2019 9:26 am

Playing with 1G files doesn't break it. Maybe swapping needs to kick in.

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Thu Aug 15, 2019 9:28 am

Nighti wrote:
Thu Aug 15, 2019 9:26 am
Playing with 1G files doesn't break it. Maybe swapping needs to kick in.
Please try #1.2 with 8GB or 32GB.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Thu Aug 15, 2019 9:35 am

elatllat wrote:
Thu Aug 15, 2019 9:28 am
Nighti wrote:
Thu Aug 15, 2019 9:26 am
Playing with 1G files doesn't break it. Maybe swapping needs to kick in.
Please try #1.2 with 8GB or 32GB.
Started four dd session on four drives (1T file size). The system is fighting hard. Keeps ~850MB always free.

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Thu Aug 15, 2019 9:56 am

Surely the next post by Jay4692, avatarNighti, or alvarow will be a concise script anyone can copy paste to reproduce the USB3 crashing from a new img with 2 empty spinning disks attached. This is starting to approach the ToS line where giving up on helping is the winning move.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Thu Aug 15, 2019 10:43 am

@elatllat: not sure how to interpret your post.

Code: Select all

cd /
rsync -av --exclude='mnt*' --exclude='proc*' --exclude='media*' --exclude='dev*' --exclude='var*' --exclude='sys*' / /media/usb0/tmp/ 
Very easy to reproduce.

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Thu Aug 15, 2019 11:17 am

Thanks Nighti

Code: Select all

rsync -a --exclude='mnt*' --exclude='proc*' --exclude='media*' --exclude='dev*' --exclude='var*' --exclude='sys*' / /media/usb0/tmp/ 
Did not break 4.9 or 5.3 on a single ext4 HDD for me.
Perhaps your hardware SNR tolerance is on the low end and odroid, tobetter, or someone at HK can advise how to further tweak, to test if that helps.
I read there was some power bleed on the boards that varied by unit, maybe connecting a scope to various test points under load would reveal why some units are more susceptible to this issue. Maybe HK will RMA a buggy unit to do that testing for us. I think odroid is away for a bit so an answer may take some time, until then if others can verify the above test; more than 2 data points would likely help HK.

Jay4692
Posts: 13
Joined: Sat Jun 29, 2019 6:31 pm
languages_spoken: english, spanish
ODROIDs: N2
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Jay4692 » Thu Aug 15, 2019 8:17 pm

elatllat wrote:
Thu Aug 15, 2019 9:56 am
Surely the next post by Jay4692, avatarNighti, or alvarow will be a concise script anyone can copy paste to reproduce the USB3 crashing from a new img with 2 empty spinning disks attached. This is starting to approach the ToS line where giving up on helping is the winning move.
I don't think answers like this, considering users trolls or stupid as your "meme" describes or turning the back on them because others don't have the issue that some users report, help in anyway. Even if the problem is related specifically to my board, manufacturing defects can occur, not reproducing the issue on your side doesn't mean I'm a troll, stupid or someone with a problem that has to be ignored, because the USB 3.0 speed on this board is slow compared to other boards, so even though my problem or someone else cannot be reproduced on your side, there is a common problem.

With that said I'm going to describe step by step how I mount my drives, previous steps that I do in a new image, that shouldn't be related, and a short script that triggers the issue, at least in my board.
I have connected two USB 3.0 spinning drives (drive1 is a 3TB and drive 2 is a 4TB in my case) formatted in EXT4, a USB hub with keyboard and mouse for both my desktop and the Odroid, HDMI ethernet and obviously the power supply. The latest image is flashed in a 32 GB SanDisk SD Card Class 10 A1.

1- Log in as user root and create new user with sudo privilegies

Code: Select all

adduser odroid
usermod -aG sudo odroid
2- Change password for root user, then logout

Code: Select all

passwd
logout
3- Log in as created user (odroid) and create directories where the drives will be mount (in my case I use /mnt) and set appropriate permissions

Code: Select all

mkdir /mnt/drive1
mkdir /mnt/drive2
chown -R odroid:odroid /mnt/drive1
chown -R odroid:odroid /mnt/drive2
chmod -R 775 /mnt/drive1
chmod -R 775 /mnt/drive2
4- Mount drives in /etc/fstab using UUID (ls -l /dev/disk/by-uuid/)

Code: Select all

nano /etc/fstab

UUID=uuid_of_drive1 /mnt/drive1 ext4 defaults,nofail 0 0
UUID=uuid_of_drive2 /mnt/drive2 ext4 defaults,nofail 0 0
5- Then simply reboot (I have experienced that even using mount -a doesn't properly mount the drives, a reboot is needed in my case)

These are steps that cannot be easily put on a script that someone can copy and paste, but mounting the drives with /etc/fstab with defaults flag (uses async instead of sync) is what easily breaks USB 3.0 for me. Maybe steps 1 and 2 can be omited, but I do them because I want my NAS to not use root user, change its password, and use another user with sudo privileges, then mounting the drives in folders I can set to share. Don't think these are steps that should break USB 3.0 transfers between drives, but I do them because I don't want to use the default /media/usb mount point, which in my experience limits USB 3.0 speed to 3 MB/s as it uses sync mount flag.

Then you can use a simple script that generates a large file in one drive, rsync to the other drive, then remove file from the first drive, rsync from the second drive to the first one. In my case it fails in the second rsync.

Code: Select all

dd if=/dev/urandom of=/mnt/drive1/test bs=1MB count=9000
rsync -v --progress /mnt/drive1/test /mnt/drive2
rm /mnt/drive1/test 
rsync -v --progress /mnt/drive2/test /mnt/drive1
Doing this same steps and script on a Raspberry Pi 4 running Raspbian on the USB 3.0 ports doesn't give any problems, also transfer speed is stable at around 70 MB/s while on the N2 bounces between 50 MB/s and 28 MB/s, staying most of the time at 30 MB/s. I say this to discard a hardware issue on my drives.

As I said, this steps breaks USB 3.0 on my N2, I don't want to troll anyone or write false posts with false problems on a forum to waste other people time, I report a problem that sadly happens to me and prevents me to use this board. I love the N2 compared to the RPi 4, is more powerful, efficient, generates less heat, uses standard HDMI output and the possibility to use eMMC storage is a great bonus, but in my case there is something wrong and doesn't work with my setup, while a RPi 4 with the same setup and configuration works like a charm.

User avatar
mad_ady
Posts: 6384
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 107 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by mad_ady » Thu Aug 15, 2019 8:47 pm

Hmm, interesting... Are you using keyboard/mouse when usb3 crashes occur, or are they only plugged in? Can you try again with low speed usb devices disconnected? The C2 had some issues when usb hi and low speed devices shared the same bus, but was stable when low speed devices were moved to usb-otg port. Could be related because of Amlogic heritage. Can you show a lsusb -t of your current setup?

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Thu Aug 15, 2019 9:18 pm

Jay4692 wrote:
Thu Aug 15, 2019 8:17 pm
...I don't think answers like this, ... help ...
Sorry for being blunt about vague bug reports, it resulted in immediate Improvement, but "starting to approach" was not temper enough.

remember to

Code: Select all

grep clk81 /sys/kernel/debug/aml_clkmsr/clkmsr
cat /proc/cmdline
dmesg | grep sg_tablesize
lsusb -t
lsblk
free
and please try Nighti's example to confirm it's reproducibility.
Also please try the 5.3 kernel on the chance that helps.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Fri Aug 16, 2019 2:15 am

I tried precompiled 5.3-rc4 kernel yesterday but without much luck. Looks like kernel is booting (blue light blinks) but no network/no screen output. Without serial console hard to tell why.

For me it's enough time spent on testing and crashing my drives. I feel like we are circling in the loop without really coming closer to a solution. I need a working machine and at the current state N2 can't be used as a file server. I attached my drives back to XU4 and everything is stable again. Thanks everybody for the time and effort trying to make it stable.

It's sad that HK still stays quiet to confirm if it's hardware or software related or both. I'm pretty sure we will see it in couple years. My feeling tells me it's a combination of both.

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Fri Aug 16, 2019 6:58 am

Nighti wrote:
Fri Aug 16, 2019 2:15 am
...enough time spent...
That's a fair choice.

HK fixed USB3 crashing for most others quickly,
and you only posted reproducible test yesterday (without answering some other questions)
so it may take a few weeks if the 2 others with this issue are persistent enough to burn time resolving it.

I'm just altruistically helping(or proding) others because, I have been helped in the same way.
(the pre built 5.3 is for emmc so edit boot.ini if using sdcard)

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Fri Aug 16, 2019 7:18 am

elatllat wrote:
Fri Aug 16, 2019 6:58 am
HK fixed USB3 crashing for most others quickly,
and you only posted reproducible test yesterday (without answering some other questions)
so it may take a few weeks if the 2 others with this issue are persistent enough to burn time resolving it.
Not sure if you've followed the whole thread. At some point it worked for me for couple days as-well but after the most recent updates it got worse. My USB2.0 drives never worked with N2 (stable).

Maybe you should read through all the posts in regards to N2-USB issues. Many people gave up already. I'm not the only one, like you're trying to picture it for whatever reason.

Me and many others provided more then enough examples where it's failing, with which devices and under which circumstances. This issue is not that easy to track down by creating "a script of death". I tried to eliminate as many factors as possible (usb cables, power supply) but the whole system still remains unstable and unpredictable in regards to USB. It was reported across not only hard drives but also input/streaming devices.

I'm glad you don't have this issue right now but please stop treating other members like they don't know what they are doing. It's not helpful!
(btw. I've modified boot.ini for sd-card and also tried to reroute tty output. But again without seeing kernel log it's more or less blind guessing).

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Fri Aug 16, 2019 7:41 am

@Nighti feel free to add to my (incomplete) summary;
elatllat wrote:
Wed Aug 14, 2019 11:20 am
...As far as I know the new img fixed USB3 crashing for everyone except
- Nighti, Jay4692, and alvarow using rsync
- trex_daemon using 4 tv tuners
Non have tested on the 5.3 kernel.
steps to reproduce;

Code: Select all

rsync -a --exclude='mnt*' --exclude='proc*' --exclude='media*' --exclude='dev*' --exclude='var*' --exclude='sys*' / /media/usb0/tmp/ 
example TV tuner that might reproduce USB3 crashing.

Jay4692
Posts: 13
Joined: Sat Jun 29, 2019 6:31 pm
languages_spoken: english, spanish
ODROIDs: N2
Has thanked: 5 times
Been thanked: 2 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Jay4692 » Fri Aug 16, 2019 7:59 am

mad_ady wrote:
Thu Aug 15, 2019 8:47 pm
Hmm, interesting... Are you using keyboard/mouse when usb3 crashes occur, or are they only plugged in? Can you try again with low speed usb devices disconnected? The C2 had some issues when usb hi and low speed devices shared the same bus, but was stable when low speed devices were moved to usb-otg port. Could be related because of Amlogic heritage. Can you show a lsusb -t of your current setup?
When the crashes occur they where only plugged in, because I end up doing the script on nano via ssh (for some reason minimal images don't have a cursor, so editing or write scripts is a bit of a pain for me, especially editing, without that cursor).

I'm not running my N2 anymore on my setup and it's a little bit late here, tomorrow I will give you the output of lsusb -t on the N2 with the same devices attached, except for the USB 2.0 hub, see if there is relation with the C2 issue.
elatllat wrote:
Thu Aug 15, 2019 9:18 pm
remember to

Code: Select all

grep clk81 /sys/kernel/debug/aml_clkmsr/clkmsr
cat /proc/cmdline
dmesg | grep sg_tablesize
lsusb -t
lsblk
free
and please try Nighti's example to confirm it's reproducibility.
Also please try the 5.3 kernel on the chance that helps.


Same to this, tomorrow I will set again my N2 with same devices and clean image and give you a response to all this.

Thanks to all.

User avatar
Nighti
Posts: 38
Joined: Tue Sep 13, 2016 11:59 am
languages_spoken: english
ODROIDs: XU4, N2
Has thanked: 1 time
Been thanked: 3 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by Nighti » Fri Aug 16, 2019 8:10 am

Sorry elatllat, I will not start looking through the posts of the last couple weeks/month just to show off or proof my point.
If you've an impression that N2 is perfect candidate as a file server right now for everybody, it's fine. I'm not! I tried to make it work for me and I'm thankful for all the help I received here in the forum but my result (for my device) after 3 weeks of tweaking and testing is just not usable.

I still have some services I can run on it, so it's not a complete waste of time/money. Once HK will release a 5.x kernel, I will check again but right now is the ball on the HK side to figure out what's wrong with it and why so many people report similar issues. I'm pretty sure there will be more as N2 is to tempting to be used as a small home/file server.

alvarow
Posts: 26
Joined: Wed Apr 10, 2019 10:44 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 10 times
Been thanked: 0
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by alvarow » Sat Aug 17, 2019 2:59 am

Hey, I am about to reflash my N2 to try the scenarios (I've been using the apt upgrade path) ... can someone confirm which version should I flash ? I see Aug 6th for minimal and Auth 12th with mate desktop, I don't use desktop, so is the minimal a valid test bed?

thanks

alvarow
Posts: 26
Joined: Wed Apr 10, 2019 10:44 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 10 times
Been thanked: 0
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by alvarow » Sat Aug 17, 2019 3:08 am

This exactly my scenario.. except I call the user something else other than odroid ... I am able to crash things with rsyncd over the gigabit network as well, not only usb3 to usb3 ...

Jay4692 wrote:
Thu Aug 15, 2019 9:18 am
For me what breaks USB 3 on a fresh image is rsync a large file or numerous files from one drive to another drive:

Code: Select all

rsync -av --progress /mnt/drive1/example_folder /mnt/drive2
rsync -v --progress /mnt/drive1/large_file /mnt/drive2
In my case I can easily break USB3 using the new image with a 32 Gb file which is an image of the whole SD Card of my system. Also transfering lots of archives and folders with -a.

My steps to reproduce the issue:

1- Create odroid user, change root password.
2- Create dirs on /mnt for both drives and chown them to user odroid.
3- Mount drives on etc/fstab with uuid and defaults,no-fail flags on the folders on /mnt.
4- rsync the big file from one drive to the other.

At least this is my experience, and none of my hardware is bad, drives and hub are running happily now on a RPi4.

elatllat
Posts: 1437
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2
Has thanked: 10 times
Been thanked: 28 times
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by elatllat » Sat Aug 17, 2019 4:12 am

alvarow wrote:
Sat Aug 17, 2019 2:59 am
... is the minimal a valid test bed?...
Yes.

alvarow
Posts: 26
Joined: Wed Apr 10, 2019 10:44 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 10 times
Been thanked: 0
Contact:

Re: Syncing two UAS devices causes Ubuntu to hang

Unread post by alvarow » Sat Aug 17, 2019 6:15 am

Here's my output of the requested commands:

Code: Select all

root@odroid:~# grep clk81 /sys/kernel/debug/aml_clkmsr/clkmsr
[ 7][ 222000000]clk81
root@odroid:~# cat /proc/cmdline
root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait rw console=ttyS0,115200n8  no_console_suspend fsck.repair=yes net.ifnames=0 elevator=noop hdmimode=1080p60hz cvbsmode=576cvbs max_freq_a53=1896 max_freq_a73=1800 maxcpus=6 voutmode=hdmi  disablehpd=false cvbscable=0 overscan=100  monitor_onoff=false usb-xhci.tablesize=2 logo=osd0,loaded
root@odroid:~# dmesg | grep sg_tablesize
[    4.807159] usb: xhci: determined sg_tablesize: 2
[    4.878482] usb: xhci: determined sg_tablesize: 2
root@odroid:~# lsusb -t
/:  Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/:  Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
    |__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
root@odroid:~# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda           8:0    0  9.1T  0 disk
`-sda1        8:1    0  9.1T  0 part
mmcblk1     179:0    0 29.9G  0 disk
|-mmcblk1p1 179:1    0  128M  0 part /media/boot
`-mmcblk1p2 179:2    0 29.8G  0 part /
root@odroid:~# free
              total        used        free      shared  buff/cache   available
Mem:        3930320      152428     3560976        3452      216916     3738632
Swap:             0           0           0
I did ran some of the rsync steps as suggested (test -> test2 with dd from /dev/random created file) and those didn't crash. When I started the rsync over gigabit ethernet from rsyncd running on my NAS into the USB3 drive... then it locked up and rebooted.

In other words, it still crashes same as before, just now on Aug 6th minimal image.
Last edited by alvarow on Sat Aug 17, 2019 6:21 am, edited 1 time in total.

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 1 guest