odroid n2 panic'ing under load

Post Reply
marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

odroid n2 panic'ing under load

Unread post by marmoset » Sat May 25, 2019 8:03 am

Hi,

I'm getting a panic while doing an rsync over ssh to a set of usb-3 drives set up as a raid 5 with mdadm. It's not easily repeatable but it happens pretty regularly, and is always:

May 24 08:28:53 odroid2 kernel: [208580.636795@5] Unable to handle kernel paging request at virtual address dead00000000047c
May 24 08:28:53 odroid2 kernel: [208580.639365@5] [dead00000000047c] address between user and kernel address ranges
May 24 08:28:54 odroid2 kernel: [208580.646795@5] Internal error: Oops: 96000004 [#1] PREEMPT SMP

I've attached the logs from the past few times it has happened.

Is there anything I can do to try and solve this, or any additional information that would help diagnose?

Thanks!
Attachments
od.txt
(51.69 KiB) Downloaded 20 times
Last edited by marmoset on Sat May 25, 2019 10:07 am, edited 1 time in total.

User avatar
rooted
Posts: 6588
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 88 times
Been thanked: 17 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by rooted » Sat May 25, 2019 12:11 pm

What power supply are you using? What USB drives and how many?

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Sat May 25, 2019 2:08 pm


Andrew Sayers
Posts: 32
Joined: Fri May 17, 2019 5:06 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 6 times
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by Andrew Sayers » Sat May 25, 2019 2:12 pm

This might be the same issue that a few of us are having. Or at least it's worth ruling out that possibility.

You might be interested in an issue with a pair of RAID-1 drives that I reported. odroid did a great job going through all the possibilities with me, and I expect they'll will want to know the same things from you (kernel version, `lsusb` output, power usage etc.). In fact, rooted is way ahead of me in asking those questions :)

You might also be interested in an issue with copying files over SSH, which seems to have the same root cause. That should give you a sense of what symptoms tend to be different (e.g. exact kernel output) and what tends to be the same (e.g. all the USB ports disappear).

If it turns out you are having the same issue as us, the good news is that they're already working on it. You might have some success with the USB quirks workaround in the mean time.

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Sat May 25, 2019 2:33 pm

Interesting, thanks. I'm already using usb-quirks (disabling uas), but I have not tried the g option yet. Before I disabled uas, I was getting regular drive timeouts (and the raid failing). I was happy when that all went away when I disabled UAS, but then ran into this seemingly new issue.

This does feel somewhat different, in that it's a kernel panic for a paging request vs a driver timing out, but I'll give it a shot.

User avatar
rooted
Posts: 6588
Joined: Fri Dec 19, 2014 9:12 am
languages_spoken: english
Location: Gulf of Mexico, US
Has thanked: 88 times
Been thanked: 17 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by rooted » Sat May 25, 2019 11:15 pm

Could you contact Ameridroid and verify that is the same exact power brick Hardkernel sells?

Also could you measure the incoming power with a DMM and look for a dip while transferring? If you don't have a multimeter I understand.

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Sun May 26, 2019 12:29 am

Sure, I will contact them.

I do have a multimeter, but I'm not sure how I'd measure the incoming power, can you give me more info on doing that? (IE where to put the probes)

Thanks!

Andrew Sayers
Posts: 32
Joined: Fri May 17, 2019 5:06 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 6 times
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by Andrew Sayers » Sun May 26, 2019 1:34 am

marmoset wrote:
Sat May 25, 2019 2:33 pm
Interesting, thanks. I'm already using usb-quirks (disabling uas), but I have not tried the g option yet. Before I disabled uas, I was getting regular drive timeouts (and the raid failing). I was happy when that all went away when I disabled UAS, but then ran into this seemingly new issue.

This does feel somewhat different, in that it's a kernel panic for a paging request vs a driver timing out, but I'll give it a shot.
This got me thinking, and it turns out that disabling UAS doesn't actually fix the problem for me either, just makes it take way longer to occur. I had been looking for a bug that happened in under a minute, not for one that takes 10+ minutes.

The kernel panic is an interesting point. I've now replicated the issue with :u, but still don't get a kernel panic. Having said that, it might just be that the bug just puts the kernel in some kind of weird state that causes it to crash in unpredictable ways.

I've now rewritten the top post in that thread with more detailed replication instructions. Let me know if anything could be better written. In particular, would you want the instructions to talk about disabling RAID before dding from the disks?

If your system behaves differently, I'd recommend detailing the differences and pasting the output of lsusb and lsusb -t in here for odroid to look at on Monday.

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Mon May 27, 2019 6:53 am

In theory, shouldn't need to remove them from the raid since it's just doing dd on the raw device, but who knows :)

I do think it's a different problem, I was able to get TB off them without the USB ports resetting/going away.

I have swapped disks around since I did actually need the array set up and content synced to it for something else, and one of the replacement testing ones did have a physical problem after a while, but these were going at the the same time:

8001506246656 bytes (8.0 TB, 7.3 TiB) copied, 65680 s, 122 MB/s

1394326474752 bytes (1.4 TB, 1.3 TiB) copied, 13831.3 s, 101 MB/s

(still responsive and normal after that)

root@odroid2:~# lsusb
Bus 002 Device 004: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 003: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@odroid2:~# lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 4: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
root@odroid2:~#

I can trigger the panic more if needed/if there's other debugging type stuff to set but they do all seem identical to the ones in the first attachment
These users thanked the author marmoset for the post:
Andrew Sayers (Mon May 27, 2019 8:56 am)

phaseshifter
Posts: 3458
Joined: Fri May 08, 2015 9:12 am
languages_spoken: english
ODROIDs: U-2,U3+,,XU-3,,XU3-LITE,,XU-4
C1+,,C-2,,,
N-1,,N-2,...other odroid acc`s as well
Has thanked: 37 times
Been thanked: 42 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by phaseshifter » Mon May 27, 2019 9:07 am

the standard issue supply by h-k is a 12 volt 2 amp you should have it written on your power brick somewhere
i think ameridroid do supply the 2 amp version..i have not checked their site lately but ..i would always use a 12 v 2A supply..minimal..specially if you have the n-2 loaded up with hdds..
Build It And They Will Come...Be Bold And Mighty Forces Will Come To Your Aid..!!!

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Wed May 29, 2019 3:53 am

ameridroid confirmed that the power brick does come from hardkernel FYI.

edit: derp, referred to the wrong company

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Sun Jun 09, 2019 1:22 am

Is there any further information I can provide or tests I can run? I set it up again to test and it still triggers (usb raid 5 with 3 drives, rsync to it while it's rebuilding either over nfs or ssh).

Same address:

Jun 8 03:11:22 odroid2 kernel: [43615.826406@3] Unable to handle kernel paging request at virtual address dead00000000047c
Jun 8 03:11:22 odroid2 kernel: [43615.828850@3] [dead00000000047c] address between user and kernel address ranges
Jun 8 03:11:23 odroid2 kernel: [43615.836639@3] Internal error: Oops: 96000004 [#1] PREEMPT SMP
Jun 8 03:11:23 odroid2 kernel: [43615.841850@3] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace xfs rtc_pcf
8563 mali_kbase i2c_meson_master sunrpc ip_tables x_tables ipv6 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid1 raid0 multipath linear md_mod
Jun 8 03:11:23 odroid2 kernel: [43615.866549@3] CPU: 3 PID: 1131 Comm: kcompactd0 Not tainted 4.9.177+ #1
Jun 8 03:11:23 odroid2 kernel: [43615.873132@3] Hardware name: Hardkernel ODROID-N2 (DT)

User avatar
odroid
Site Admin
Posts: 31799
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 86 times
Been thanked: 243 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by odroid » Mon Jun 10, 2019 9:11 am

Your issue seems to be very similar to Andrew's one. We will try to find a solution.
viewtopic.php?f=181&t=35031

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Mon Jun 10, 2019 11:15 am

OK, cool, thanks. To me it seemed different because I have disabled UAS, and USB never times out/disappears, it's just a panic. Before I disabled UAS, I had the same symptoms as Andrew.

But if you feel it's the same cause I'm fine with waiting.

User avatar
odroid
Site Admin
Posts: 31799
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 86 times
Been thanked: 243 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by odroid » Mon Jun 10, 2019 11:19 am

The Andrew's issue could be reproduced regardless of the UAS mode with two storage devices on our side.
So I could tell you the similarity.

User avatar
mad_ady
Posts: 6380
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 106 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by mad_ady » Tue Jun 11, 2019 1:58 am

Back when my XU4 was a nas I too had problems with usb stability on high throughput, but this workaround reduced throughput, but increased stability:

Code: Select all

device=sda
#if [ -f "/sys/class/block/$device/queue/max_sectors_kb" ]; then
#    logger -s -t $0 "Setting HDD $device speed limit"#   echo 100 > /sys/class/block/$device/queue/max_sectors_kb
#
#fi
Uncomment and try with 10, 20 or 100. Default is 1024.

User avatar
odroid
Site Admin
Posts: 31799
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English
ODROIDs: ODROID
Has thanked: 86 times
Been thanked: 243 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by odroid » Tue Jun 11, 2019 5:38 pm

@mad_ady,
Thank you for the idea.
We've run Andrew's test around 4 hours and there is no system error yet after changing the queue size to 32 from 1024. So far so good.
But overall transfer speed seems to have been reduced by 20 ~ 30%.
Anyway, we will update the test result again after 15 hours of more heavy test.

User avatar
mad_ady
Posts: 6380
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, N1, H2, N2
Location: Bucharest, Romania
Has thanked: 146 times
Been thanked: 106 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by mad_ady » Tue Jun 11, 2019 6:51 pm

Yes, it reduces throuhput. It should be used as a temporary workaround.
Sadly it works only for disks. USB3 Network, webcams will still trigger the problem.

joshua.yang
Posts: 240
Joined: Fri Sep 22, 2017 5:54 pm
languages_spoken: Korean, English
ODROIDs: XU4, XU4Q + Cloudshell2, H2
Has thanked: 2 times
Been thanked: 21 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by joshua.yang » Wed Jun 12, 2019 10:57 am

@odroid, @mad_ady

I have tested with two Sandisk USB stick, which is formatted as EXT4 file system.

To test them I entered

Code: Select all

echo 32 > /sys/class/block/sda/queue/max_sectors_kb 
echo 32 > /sys/class/block/sdb/queue/max_sectors_kb 
while true; do dd if=/dev/sda1 of=/dev/null; done &
while true; do dd if=/dev/sdb1 of=/dev/null; done &
commands.

It works for 20 hours and it looks reliable than defaults. No warning/critical messages on dmesg, it is just clean. :)

Andrew Sayers
Posts: 32
Joined: Fri May 17, 2019 5:06 pm
languages_spoken: english
ODROIDs: N2
Has thanked: 6 times
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by Andrew Sayers » Thu Jun 13, 2019 11:12 pm

@marmoset - does mad_ady's workaround change the behaviour you're seeing?

For the sake of people that got here from Google, the other thread now has detailed instructions to replicate the bug and implement the workaround.

marmoset
Posts: 14
Joined: Sat May 25, 2019 7:57 am
languages_spoken: english
ODROIDs: odroid-n2
Has thanked: 0
Been thanked: 3 times
Contact:

Re: odroid n2 panic'ing under load

Unread post by marmoset » Wed Jun 19, 2019 2:54 am

so far so good, it's been syncing for almost 24 hours and it didn't make it that long last time.
These users thanked the author marmoset for the post (total 2):
Andrew Sayers (Wed Jun 19, 2019 3:35 am) • odroid (Wed Jun 19, 2019 9:28 am)

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 3 guests