odroid n2 panic'ing under load

[marmoset]

Hi,

I'm getting a panic while doing an rsync over ssh to a set of usb-3 drives set up as a raid 5 with mdadm. It's not easily repeatable but it happens pretty regularly, and is always:

May 24 08:28:53 odroid2 kernel: [208580.636795@5] Unable to handle kernel paging request at virtual address dead00000000047c
May 24 08:28:53 odroid2 kernel: [208580.639365@5] [dead00000000047c] address between user and kernel address ranges
May 24 08:28:54 odroid2 kernel: [208580.646795@5] Internal error: Oops: 96000004 [#1] PREEMPT SMP

I've attached the logs from the past few times it has happened.

Is there anything I can do to try and solve this, or any additional information that would help diagnose?

Thanks!

[rooted]

What power supply are you using? What USB drives and how many?

[marmoset]

the power supply is one of these: https://ameridroid.com/products/12v-2a- ... odroid-hc2

3 drives, all are all self powered, one of each of these:

https://www.amazon.com/gp/product/B01HAPGEIE/
https://www.amazon.com/gp/product/B07D5V2ZXD/
https://www.amazon.com/gp/product/B07CQJBSQL/

[Andrew Sayers]

This might be the same issue that a few of us are having. Or at least it's worth ruling out that possibility.

You might be interested in an issue with a pair of RAID-1 drives that I reported. odroid did a great job going through all the possibilities with me, and I expect they'll will want to know the same things from you (kernel version, `lsusb` output, power usage etc.). In fact, rooted is way ahead of me in asking those questions

You might also be interested in an issue with copying files over SSH, which seems to have the same root cause. That should give you a sense of what symptoms tend to be different (e.g. exact kernel output) and what tends to be the same (e.g. all the USB ports disappear).

If it turns out you are having the same issue as us, the good news is that they're already working on it. You might have some success with the USB quirks workaround in the mean time.

[marmoset]

Interesting, thanks. I'm already using usb-quirks (disabling uas), but I have not tried the g option yet. Before I disabled uas, I was getting regular drive timeouts (and the raid failing). I was happy when that all went away when I disabled UAS, but then ran into this seemingly new issue.

This does feel somewhat different, in that it's a kernel panic for a paging request vs a driver timing out, but I'll give it a shot.

[rooted]

Could you contact Ameridroid and verify that is the same exact power brick Hardkernel sells?

Also could you measure the incoming power with a DMM and look for a dip while transferring? If you don't have a multimeter I understand.

[marmoset]

Sure, I will contact them.

I do have a multimeter, but I'm not sure how I'd measure the incoming power, can you give me more info on doing that? (IE where to put the probes)

Thanks!

[Andrew Sayers]

Interesting, thanks. I'm already using usb-quirks (disabling uas), but I have not tried the g option yet. Before I disabled uas, I was getting regular drive timeouts (and the raid failing). I was happy when that all went away when I disabled UAS, but then ran into this seemingly new issue.

This does feel somewhat different, in that it's a kernel panic for a paging request vs a driver timing out, but I'll give it a shot.

This got me thinking, and it turns out that disabling UAS doesn't actually fix the problem for me either, just makes it take way longer to occur. I had been looking for a bug that happened in under a minute, not for one that takes 10+ minutes.

The kernel panic is an interesting point. I've now replicated the issue with :u, but still don't get a kernel panic. Having said that, it might just be that the bug just puts the kernel in some kind of weird state that causes it to crash in unpredictable ways.

I've now rewritten the top post in that thread with more detailed replication instructions. Let me know if anything could be better written. In particular, would you want the instructions to talk about disabling RAID before dding from the disks?

If your system behaves differently, I'd recommend detailing the differences and pasting the output of lsusb and lsusb -t in here for odroid to look at on Monday.

[marmoset]

In theory, shouldn't need to remove them from the raid since it's just doing dd on the raw device, but who knows

I do think it's a different problem, I was able to get TB off them without the USB ports resetting/going away.

I have swapped disks around since I did actually need the array set up and content synced to it for something else, and one of the replacement testing ones did have a physical problem after a while, but these were going at the the same time:

8001506246656 bytes (8.0 TB, 7.3 TiB) copied, 65680 s, 122 MB/s

1394326474752 bytes (1.4 TB, 1.3 TiB) copied, 13831.3 s, 101 MB/s

(still responsive and normal after that)

root@odroid2:~# lsusb
Bus 002 Device 004: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 003: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@odroid2:~# lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 4: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
root@odroid2:~#

I can trigger the panic more if needed/if there's other debugging type stuff to set but they do all seem identical to the ones in the first attachment

[phaseshifter]

the standard issue supply by h-k is a 12 volt 2 amp you should have it written on your power brick somewhere
i think ameridroid do supply the 2 amp version..i have not checked their site lately but ..i would always use a 12 v 2A supply..minimal..specially if you have the n-2 loaded up with hdds..

[marmoset]

ameridroid confirmed that the power brick does come from hardkernel FYI.

edit: derp, referred to the wrong company

[marmoset]

Is there any further information I can provide or tests I can run? I set it up again to test and it still triggers (usb raid 5 with 3 drives, rsync to it while it's rebuilding either over nfs or ssh).

Same address:

Jun 8 03:11:22 odroid2 kernel: [43615.826406@3] Unable to handle kernel paging request at virtual address dead00000000047c
Jun 8 03:11:22 odroid2 kernel: [43615.828850@3] [dead00000000047c] address between user and kernel address ranges
Jun 8 03:11:23 odroid2 kernel: [43615.836639@3] Internal error: Oops: 96000004 [#1] PREEMPT SMP
Jun 8 03:11:23 odroid2 kernel: [43615.841850@3] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace xfs rtc_pcf
8563 mali_kbase i2c_meson_master sunrpc ip_tables x_tables ipv6 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid1 raid0 multipath linear md_mod
Jun 8 03:11:23 odroid2 kernel: [43615.866549@3] CPU: 3 PID: 1131 Comm: kcompactd0 Not tainted 4.9.177+ #1
Jun 8 03:11:23 odroid2 kernel: [43615.873132@3] Hardware name: Hardkernel ODROID-N2 (DT)

[odroid]

Your issue seems to be very similar to Andrew's one. We will try to find a solution.
viewtopic.php?f=181&t=35031

[marmoset]

OK, cool, thanks. To me it seemed different because I have disabled UAS, and USB never times out/disappears, it's just a panic. Before I disabled UAS, I had the same symptoms as Andrew.

But if you feel it's the same cause I'm fine with waiting.

[odroid]

The Andrew's issue could be reproduced regardless of the UAS mode with two storage devices on our side.
So I could tell you the similarity.

[mad_ady]

Back when my XU4 was a nas I too had problems with usb stability on high throughput, but this workaround reduced throughput, but increased stability:

Code: Select all

device=sda
#if [ -f "/sys/class/block/$device/queue/max_sectors_kb" ]; then
#    logger -s -t $0 "Setting HDD $device speed limit"#   echo 100 > /sys/class/block/$device/queue/max_sectors_kb
#
#fi

Uncomment and try with 10, 20 or 100. Default is 1024.

[odroid]

@mad_ady,
Thank you for the idea.
We've run Andrew's test around 4 hours and there is no system error yet after changing the queue size to 32 from 1024. So far so good.
But overall transfer speed seems to have been reduced by 20 ~ 30%.
Anyway, we will update the test result again after 15 hours of more heavy test.

[mad_ady]

Yes, it reduces throuhput. It should be used as a temporary workaround.
Sadly it works only for disks. USB3 Network, webcams will still trigger the problem.

[joshua.yang]

@odroid, @mad_ady

I have tested with two Sandisk USB stick, which is formatted as EXT4 file system.

To test them I entered

Code: Select all

echo 32 > /sys/class/block/sda/queue/max_sectors_kb 
echo 32 > /sys/class/block/sdb/queue/max_sectors_kb 
while true; do dd if=/dev/sda1 of=/dev/null; done &
while true; do dd if=/dev/sdb1 of=/dev/null; done &

commands.

It works for 20 hours and it looks reliable than defaults. No warning/critical messages on dmesg, it is just clean.

[Andrew Sayers]

@marmoset - does mad_ady's workaround change the behaviour you're seeing?

For the sake of people that got here from Google, the other thread now has detailed instructions to replicate the bug and implement the workaround.

[marmoset]

so far so good, it's been syncing for almost 24 hours and it didn't make it that long last time.