odroid n2 panic'ing under load
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
odroid n2 panic'ing under load
Hi,
I'm getting a panic while doing an rsync over ssh to a set of usb-3 drives set up as a raid 5 with mdadm. It's not easily repeatable but it happens pretty regularly, and is always:
May 24 08:28:53 odroid2 kernel: [208580.636795@5] Unable to handle kernel paging request at virtual address dead00000000047c
May 24 08:28:53 odroid2 kernel: [208580.639365@5] [dead00000000047c] address between user and kernel address ranges
May 24 08:28:54 odroid2 kernel: [208580.646795@5] Internal error: Oops: 96000004 [#1] PREEMPT SMP
I've attached the logs from the past few times it has happened.
Is there anything I can do to try and solve this, or any additional information that would help diagnose?
Thanks!
I'm getting a panic while doing an rsync over ssh to a set of usb-3 drives set up as a raid 5 with mdadm. It's not easily repeatable but it happens pretty regularly, and is always:
May 24 08:28:53 odroid2 kernel: [208580.636795@5] Unable to handle kernel paging request at virtual address dead00000000047c
May 24 08:28:53 odroid2 kernel: [208580.639365@5] [dead00000000047c] address between user and kernel address ranges
May 24 08:28:54 odroid2 kernel: [208580.646795@5] Internal error: Oops: 96000004 [#1] PREEMPT SMP
I've attached the logs from the past few times it has happened.
Is there anything I can do to try and solve this, or any additional information that would help diagnose?
Thanks!
- Attachments
-
od.txt
- (51.69 KiB) Downloaded 68 times
Last edited by marmoset on Sat May 25, 2019 10:07 am, edited 1 time in total.
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
the power supply is one of these: https://ameridroid.com/products/12v-2a- ... odroid-hc2
3 drives, all are all self powered, one of each of these:
https://www.amazon.com/gp/product/B01HAPGEIE/
https://www.amazon.com/gp/product/B07D5V2ZXD/
https://www.amazon.com/gp/product/B07CQJBSQL/
3 drives, all are all self powered, one of each of these:
https://www.amazon.com/gp/product/B01HAPGEIE/
https://www.amazon.com/gp/product/B07D5V2ZXD/
https://www.amazon.com/gp/product/B07CQJBSQL/
-
- Posts: 32
- Joined: Fri May 17, 2019 5:06 pm
- languages_spoken: english
- ODROIDs: N2
- Has thanked: 6 times
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
This might be the same issue that a few of us are having. Or at least it's worth ruling out that possibility.
You might be interested in an issue with a pair of RAID-1 drives that I reported. odroid did a great job going through all the possibilities with me, and I expect they'll will want to know the same things from you (kernel version, `lsusb` output, power usage etc.). In fact, rooted is way ahead of me in asking those questions
You might also be interested in an issue with copying files over SSH, which seems to have the same root cause. That should give you a sense of what symptoms tend to be different (e.g. exact kernel output) and what tends to be the same (e.g. all the USB ports disappear).
If it turns out you are having the same issue as us, the good news is that they're already working on it. You might have some success with the USB quirks workaround in the mean time.
You might be interested in an issue with a pair of RAID-1 drives that I reported. odroid did a great job going through all the possibilities with me, and I expect they'll will want to know the same things from you (kernel version, `lsusb` output, power usage etc.). In fact, rooted is way ahead of me in asking those questions

You might also be interested in an issue with copying files over SSH, which seems to have the same root cause. That should give you a sense of what symptoms tend to be different (e.g. exact kernel output) and what tends to be the same (e.g. all the USB ports disappear).
If it turns out you are having the same issue as us, the good news is that they're already working on it. You might have some success with the USB quirks workaround in the mean time.
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
Interesting, thanks. I'm already using usb-quirks (disabling uas), but I have not tried the g option yet. Before I disabled uas, I was getting regular drive timeouts (and the raid failing). I was happy when that all went away when I disabled UAS, but then ran into this seemingly new issue.
This does feel somewhat different, in that it's a kernel panic for a paging request vs a driver timing out, but I'll give it a shot.
This does feel somewhat different, in that it's a kernel panic for a paging request vs a driver timing out, but I'll give it a shot.
- rooted
- Posts: 8487
- Joined: Fri Dec 19, 2014 9:12 am
- languages_spoken: english
- Location: Gulf of Mexico, US
- Has thanked: 740 times
- Been thanked: 342 times
- Contact:
Re: odroid n2 panic'ing under load
Could you contact Ameridroid and verify that is the same exact power brick Hardkernel sells?
Also could you measure the incoming power with a DMM and look for a dip while transferring? If you don't have a multimeter I understand.
Also could you measure the incoming power with a DMM and look for a dip while transferring? If you don't have a multimeter I understand.
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
Sure, I will contact them.
I do have a multimeter, but I'm not sure how I'd measure the incoming power, can you give me more info on doing that? (IE where to put the probes)
Thanks!
I do have a multimeter, but I'm not sure how I'd measure the incoming power, can you give me more info on doing that? (IE where to put the probes)
Thanks!
-
- Posts: 32
- Joined: Fri May 17, 2019 5:06 pm
- languages_spoken: english
- ODROIDs: N2
- Has thanked: 6 times
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
This got me thinking, and it turns out that disabling UAS doesn't actually fix the problem for me either, just makes it take way longer to occur. I had been looking for a bug that happened in under a minute, not for one that takes 10+ minutes.marmoset wrote: ↑Sat May 25, 2019 2:33 pmInteresting, thanks. I'm already using usb-quirks (disabling uas), but I have not tried the g option yet. Before I disabled uas, I was getting regular drive timeouts (and the raid failing). I was happy when that all went away when I disabled UAS, but then ran into this seemingly new issue.
This does feel somewhat different, in that it's a kernel panic for a paging request vs a driver timing out, but I'll give it a shot.
The kernel panic is an interesting point. I've now replicated the issue with
:u
, but still don't get a kernel panic. Having said that, it might just be that the bug just puts the kernel in some kind of weird state that causes it to crash in unpredictable ways.I've now rewritten the top post in that thread with more detailed replication instructions. Let me know if anything could be better written. In particular, would you want the instructions to talk about disabling RAID before
dd
ing from the disks?If your system behaves differently, I'd recommend detailing the differences and pasting the output of
lsusb
and lsusb -t
in here for odroid to look at on Monday.-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
In theory, shouldn't need to remove them from the raid since it's just doing dd on the raw device, but who knows
I do think it's a different problem, I was able to get TB off them without the USB ports resetting/going away.
I have swapped disks around since I did actually need the array set up and content synced to it for something else, and one of the replacement testing ones did have a physical problem after a while, but these were going at the the same time:
8001506246656 bytes (8.0 TB, 7.3 TiB) copied, 65680 s, 122 MB/s
1394326474752 bytes (1.4 TB, 1.3 TiB) copied, 13831.3 s, 101 MB/s
(still responsive and normal after that)
root@odroid2:~# lsusb
Bus 002 Device 004: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 003: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@odroid2:~# lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 4: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
root@odroid2:~#
I can trigger the panic more if needed/if there's other debugging type stuff to set but they do all seem identical to the ones in the first attachment

I do think it's a different problem, I was able to get TB off them without the USB ports resetting/going away.
I have swapped disks around since I did actually need the array set up and content synced to it for something else, and one of the replacement testing ones did have a physical problem after a while, but these were going at the the same time:
8001506246656 bytes (8.0 TB, 7.3 TiB) copied, 65680 s, 122 MB/s
1394326474752 bytes (1.4 TB, 1.3 TiB) copied, 13831.3 s, 101 MB/s
(still responsive and normal after that)
root@odroid2:~# lsusb
Bus 002 Device 004: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 003: ID 0bc2:3322 Seagate RSS LLC
Bus 002 Device 002: ID 05e3:0620 Genesys Logic, Inc.
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
root@odroid2:~# lsusb -t
/: Bus 02.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/1p, 5000M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 5000M
|__ Port 2: Dev 3, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
|__ Port 4: Dev 4, If 0, Class=Mass Storage, Driver=usb-storage, 5000M
/: Bus 01.Port 1: Dev 1, Class=root_hub, Driver=xhci-hcd/2p, 480M
|__ Port 1: Dev 2, If 0, Class=Hub, Driver=hub/4p, 480M
root@odroid2:~#
I can trigger the panic more if needed/if there's other debugging type stuff to set but they do all seem identical to the ones in the first attachment
- These users thanked the author marmoset for the post:
- Andrew Sayers (Mon May 27, 2019 8:56 am)
-
- Posts: 3569
- Joined: Fri May 08, 2015 9:12 am
- languages_spoken: english
- ODROIDs: U-2,U3+,,XU-3,,XU3-LITE,,XU-4
C1+,,C-2,,,n2+2G and n2 4G
cloudshell I and shell II
N-1,,N-2,...other odroid acc`s as well..vu7 etc.. - Has thanked: 52 times
- Been thanked: 49 times
- Contact:
Re: odroid n2 panic'ing under load
the standard issue supply by h-k is a 12 volt 2 amp you should have it written on your power brick somewhere
i think ameridroid do supply the 2 amp version..i have not checked their site lately but ..i would always use a 12 v 2A supply..minimal..specially if you have the n-2 loaded up with hdds..
i think ameridroid do supply the 2 amp version..i have not checked their site lately but ..i would always use a 12 v 2A supply..minimal..specially if you have the n-2 loaded up with hdds..
Build It And They Will Come...Be Bold And Mighty Forces Will Come To Your Aid..!!!
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
ameridroid confirmed that the power brick does come from hardkernel FYI.
edit: derp, referred to the wrong company
edit: derp, referred to the wrong company
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
Is there any further information I can provide or tests I can run? I set it up again to test and it still triggers (usb raid 5 with 3 drives, rsync to it while it's rebuilding either over nfs or ssh).
Same address:
Jun 8 03:11:22 odroid2 kernel: [43615.826406@3] Unable to handle kernel paging request at virtual address dead00000000047c
Jun 8 03:11:22 odroid2 kernel: [43615.828850@3] [dead00000000047c] address between user and kernel address ranges
Jun 8 03:11:23 odroid2 kernel: [43615.836639@3] Internal error: Oops: 96000004 [#1] PREEMPT SMP
Jun 8 03:11:23 odroid2 kernel: [43615.841850@3] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace xfs rtc_pcf
8563 mali_kbase i2c_meson_master sunrpc ip_tables x_tables ipv6 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid1 raid0 multipath linear md_mod
Jun 8 03:11:23 odroid2 kernel: [43615.866549@3] CPU: 3 PID: 1131 Comm: kcompactd0 Not tainted 4.9.177+ #1
Jun 8 03:11:23 odroid2 kernel: [43615.873132@3] Hardware name: Hardkernel ODROID-N2 (DT)
Same address:
Jun 8 03:11:22 odroid2 kernel: [43615.826406@3] Unable to handle kernel paging request at virtual address dead00000000047c
Jun 8 03:11:22 odroid2 kernel: [43615.828850@3] [dead00000000047c] address between user and kernel address ranges
Jun 8 03:11:23 odroid2 kernel: [43615.836639@3] Internal error: Oops: 96000004 [#1] PREEMPT SMP
Jun 8 03:11:23 odroid2 kernel: [43615.841850@3] Modules linked in: rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace xfs rtc_pcf
8563 mali_kbase i2c_meson_master sunrpc ip_tables x_tables ipv6 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor
async_tx xor raid6_pq raid1 raid0 multipath linear md_mod
Jun 8 03:11:23 odroid2 kernel: [43615.866549@3] CPU: 3 PID: 1131 Comm: kcompactd0 Not tainted 4.9.177+ #1
Jun 8 03:11:23 odroid2 kernel: [43615.873132@3] Hardware name: Hardkernel ODROID-N2 (DT)
- odroid
- Site Admin
- Posts: 37229
- Joined: Fri Feb 22, 2013 11:14 pm
- languages_spoken: English, Korean
- ODROIDs: ODROID
- Has thanked: 1723 times
- Been thanked: 1120 times
- Contact:
Re: odroid n2 panic'ing under load
Your issue seems to be very similar to Andrew's one. We will try to find a solution.
viewtopic.php?f=181&t=35031
viewtopic.php?f=181&t=35031
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
OK, cool, thanks. To me it seemed different because I have disabled UAS, and USB never times out/disappears, it's just a panic. Before I disabled UAS, I had the same symptoms as Andrew.
But if you feel it's the same cause I'm fine with waiting.
But if you feel it's the same cause I'm fine with waiting.
- odroid
- Site Admin
- Posts: 37229
- Joined: Fri Feb 22, 2013 11:14 pm
- languages_spoken: English, Korean
- ODROIDs: ODROID
- Has thanked: 1723 times
- Been thanked: 1120 times
- Contact:
Re: odroid n2 panic'ing under load
The Andrew's issue could be reproduced regardless of the UAS mode with two storage devices on our side.
So I could tell you the similarity.
So I could tell you the similarity.
-
- Posts: 9367
- Joined: Wed Jul 15, 2015 5:00 pm
- languages_spoken: english
- ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
- Location: Bucharest, Romania
- Has thanked: 599 times
- Been thanked: 658 times
- Contact:
Re: odroid n2 panic'ing under load
Back when my XU4 was a nas I too had problems with usb stability on high throughput, but this workaround reduced throughput, but increased stability:
Uncomment and try with 10, 20 or 100. Default is 1024.
Code: Select all
device=sda
#if [ -f "/sys/class/block/$device/queue/max_sectors_kb" ]; then
# logger -s -t $0 "Setting HDD $device speed limit"# echo 100 > /sys/class/block/$device/queue/max_sectors_kb
#
#fi
- odroid
- Site Admin
- Posts: 37229
- Joined: Fri Feb 22, 2013 11:14 pm
- languages_spoken: English, Korean
- ODROIDs: ODROID
- Has thanked: 1723 times
- Been thanked: 1120 times
- Contact:
Re: odroid n2 panic'ing under load
@mad_ady,
Thank you for the idea.
We've run Andrew's test around 4 hours and there is no system error yet after changing the queue size to 32 from 1024. So far so good.
But overall transfer speed seems to have been reduced by 20 ~ 30%.
Anyway, we will update the test result again after 15 hours of more heavy test.
Thank you for the idea.
We've run Andrew's test around 4 hours and there is no system error yet after changing the queue size to 32 from 1024. So far so good.
But overall transfer speed seems to have been reduced by 20 ~ 30%.
Anyway, we will update the test result again after 15 hours of more heavy test.
-
- Posts: 9367
- Joined: Wed Jul 15, 2015 5:00 pm
- languages_spoken: english
- ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
- Location: Bucharest, Romania
- Has thanked: 599 times
- Been thanked: 658 times
- Contact:
Re: odroid n2 panic'ing under load
Yes, it reduces throuhput. It should be used as a temporary workaround.
Sadly it works only for disks. USB3 Network, webcams will still trigger the problem.
Sadly it works only for disks. USB3 Network, webcams will still trigger the problem.
-
- Posts: 503
- Joined: Fri Sep 22, 2017 5:54 pm
- languages_spoken: Korean, English
- ODROIDs: XU4, XU4Q + Cloudshell2, H2, N2
- Has thanked: 35 times
- Been thanked: 145 times
- Contact:
Re: odroid n2 panic'ing under load
@odroid, @mad_ady
I have tested with two Sandisk USB stick, which is formatted as EXT4 file system.
To test them I entered
commands.
It works for 20 hours and it looks reliable than defaults. No warning/critical messages on dmesg, it is just clean.
I have tested with two Sandisk USB stick, which is formatted as EXT4 file system.
To test them I entered
Code: Select all
echo 32 > /sys/class/block/sda/queue/max_sectors_kb
echo 32 > /sys/class/block/sdb/queue/max_sectors_kb
while true; do dd if=/dev/sda1 of=/dev/null; done &
while true; do dd if=/dev/sdb1 of=/dev/null; done &
It works for 20 hours and it looks reliable than defaults. No warning/critical messages on dmesg, it is just clean.

-
- Posts: 32
- Joined: Fri May 17, 2019 5:06 pm
- languages_spoken: english
- ODROIDs: N2
- Has thanked: 6 times
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
@marmoset - does mad_ady's workaround change the behaviour you're seeing?
For the sake of people that got here from Google, the other thread now has detailed instructions to replicate the bug and implement the workaround.
For the sake of people that got here from Google, the other thread now has detailed instructions to replicate the bug and implement the workaround.
- These users thanked the author Andrew Sayers for the post:
- MoonKid (Sun Dec 20, 2020 5:43 am)
-
- Posts: 14
- Joined: Sat May 25, 2019 7:57 am
- languages_spoken: english
- ODROIDs: odroid-n2
- Has thanked: 0
- Been thanked: 4 times
- Contact:
Re: odroid n2 panic'ing under load
so far so good, it's been syncing for almost 24 hours and it didn't make it that long last time.
- These users thanked the author marmoset for the post (total 2):
- Andrew Sayers (Wed Jun 19, 2019 3:35 am) • odroid (Wed Jun 19, 2019 9:28 am)
Who is online
Users browsing this forum: No registered users and 1 guest