ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post Reply
seppsyrj
Posts: 2
Joined: Mon Nov 16, 2020 9:10 pm
languages_spoken: english
ODROIDs: Odroid-C4
Has thanked: 3 times
Been thanked: 0
Contact:

ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by seppsyrj »

Hi

Setup:
ODROID-C4 running Ubuntu Minimal 20.04
BTRFS filesystem on two USB disks

Problem: kernel panics by itself or btrfs tool segfaults when modifying filesystem or umounting it.

NOTE: Btrfs works fine with Armbian https://www.armbian.com/odroid-c4/ (Ubuntu 20.04.1 LTS, focal), but GPIO interrupts (i.e. reading external buttons) does not work with it (handle /sys/class/gpio/gpioN/edge does not exist). Also gpiopower seems not be supported in Armbian.

I tried most of the listed third party OSes (https://wiki.odroid.com/odroid-c4/os_images/third_party) but only Armbian had a working btrfs.

I didn't try other filesystems on these USB disks, so I cannot rule out some USB driver related problem but I tried multiple different disks (USB 2 and 3).
Armbian has these in boot.ini: usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x331a:u

Any ideas to try? Am I the only one running btrfs?

-Seppo

PS. I've had btrfs running on Pi3 and Pi4 for years and want to use C4:s multiple USB3s to make a more flexible NAS server.

seppsyrj
Posts: 2
Joined: Mon Nov 16, 2020 9:10 pm
languages_spoken: english
ODROIDs: Odroid-C4
Has thanked: 3 times
Been thanked: 0
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by seppsyrj »

Adding usb-storage-quirks to boot.ini didn't help

cat /proc/cmdline
root=UUID=e139ce78-9841-40fe-8823-96a304a09859 rootwait rw console=ttyS0,115200n8 no_console_suspend fsck.repair=yes net.ifnames=0 elevator=noop hdmimode=1080p60hz cvbsmode=576cvbs max_freq_a55=1908 maxcpus=4 voutmode=hdmi disablehpd=false cvbscable=0 overscan=100 monitor_onoff=false logo=osd0,loaded hdmitx=cec3f sdrmode=auto enable_wol=0 gpiopower=479 usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u,0x0bc2:0x331a:u

Few seconds after mounting a btrfs system comes panic:

Message from syslogd@odroid at Nov 16 15:57:22 ...
kernel:[ 172.051787@1] Internal error: Oops: 96000005 [#1] PREEMPT SMP
Message from syslogd@odroid at Nov 16 15:57:22 ...
kernel:[ 172.651137@1] Process btrfs-transacti (pid: 2916, stack limit = 0xffffffc076ea0000)
Message from syslogd@odroid at Nov 16 15:57:22 ...
...
dmesg -T:

[Mon Nov 16 15:55:33 2020] fb: mem_free_work, free memory: addr:800000
[Mon Nov 16 15:56:44 2020] BTRFS info (device sda): disk space caching is enabled
[Mon Nov 16 15:56:44 2020] BTRFS info (device sda): has skinny extents
[Mon Nov 16 15:56:44 2020] BTRFS info (device sda): bdev /dev/sda errs: wr 79, rd 83, flush 0, corrupt 0, gen 0
[Mon Nov 16 15:56:44 2020] BTRFS info (device sda): bdev /dev/sdb errs: wr 12, rd 0, flush 4, corrupt 0, gen 0
[Mon Nov 16 15:57:21 2020] Unable to handle kernel NULL pointer dereference at virtual address 00000044
[Mon Nov 16 15:57:21 2020] pgd = ffffffc09a5e1000
[Mon Nov 16 15:57:21 2020] [0000000000000044] *pgd=0000000000000000, *pud=0000000000000000
[Mon Nov 16 15:57:21 2020] Internal error: Oops: 96000005 [#1] PREEMPT SMP
[Mon Nov 16 15:57:21 2020] Modules linked in: fuse squashfs cpufreq_ondemand cpufreq_powersave cpufreq_userspace cpufreq_conservative btrfs raid6_pq xor i2c_meson_master amvdec_ports sch_fq_codel amvdec_vp9 amvdec_vc1 amvdec_real amvdec_mmpeg4 amvdec_mpeg4 amvdec_mpeg12 amvdec_mmjpeg amvdec_mjpeg amvdec_h265 amvdec_h264mvc amvdec_mh264 amvdec_h264 amvdec_avs stream_input decoder_common firmware media_clock ip_tables x_tables ipv6 spidev spi_meson_spicc
[Mon Nov 16 15:57:21 2020] CPU: 1 PID: 2916 Comm: btrfs-transacti Not tainted 4.9.236-39 #1
[Mon Nov 16 15:57:21 2020] Hardware name: Hardkernel ODROID-C4 (DT)
[Mon Nov 16 15:57:21 2020] task: ffffffc0ab609c00 task.stack: ffffffc076ea0000
[Mon Nov 16 15:57:21 2020] PC is at __do_page_cache_readahead+0x4c/0x298
[Mon Nov 16 15:57:21 2020] LR is at ondemand_readahead+0x118/0x278
[Mon Nov 16 15:57:21 2020] pc : [<ffffff80091ce8cc>] lr : [<ffffff80091cec30>] pstate: 20c00145
...

[Mon Nov 16 15:57:22 2020] Call trace:
[Mon Nov 16 15:57:22 2020] Exception stack(0xffffffc076ea3320 to 0xffffffc076ea3450)
[Mon Nov 16 15:57:22 2020] 3320: 0000000000000000 0000007fffffffff ffffffc076ea34f0 ffffff80091ce8cc
[Mon Nov 16 15:57:22 2020] 3340: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[Mon Nov 16 15:57:22 2020] 3360: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[Mon Nov 16 15:57:22 2020] 3380: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[Mon Nov 16 15:57:22 2020] 33a0: 0000000000000000 0000000000000000 0000000000000000 0000000000000000
[Mon Nov 16 15:57:22 2020] 33c0: 0000000002420048 ffffffc0a7238a60 0000000000000000 0000000000001300
[Mon Nov 16 15:57:22 2020] 33e0: 0000000000000041 000000000000003f 000000000000003f 0000000000000000
[Mon Nov 16 15:57:22 2020] 3400: ffffffc0b5e800c0 0000000000000000 0000000000000a20 0000000000000000
[Mon Nov 16 15:57:22 2020] 3420: 000000003455d91d 00000040c50c4000 ffffffc0ca9e2a00 0000000000000000
[Mon Nov 16 15:57:22 2020] 3440: 0000000000000000 0000000000000000
[Mon Nov 16 15:57:22 2020] [<ffffff80091ce8cc>] __do_page_cache_readahead+0x4c/0x298
[Mon Nov 16 15:57:22 2020] [<ffffff80091cec30>] ondemand_readahead+0x118/0x278
[Mon Nov 16 15:57:22 2020] [<ffffff80091ceff0>] page_cache_sync_readahead+0x60/0xa0
[Mon Nov 16 15:57:22 2020] [<ffffff800209cf4c>] __load_free_space_cache+0x214/0x680 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff800209d4f4>] load_free_space_cache+0x13c/0x1f0 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff800202d208>] cache_block_group+0x178/0x3d8 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff800203cad0>] find_free_extent+0x418/0xf60 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff800203d6c0>] btrfs_reserve_extent+0xa8/0x1a0 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff800203d8e8>] btrfs_alloc_tree_block+0x130/0x3f0 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff8002022fcc>] __btrfs_cow_block+0x114/0x4f8 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff8002023518>] btrfs_cow_block+0xd0/0x250 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff8002026d08>] btrfs_search_slot+0x1c8/0x960 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff8002044b38>] btrfs_lookup_inode+0x50/0xe8 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff80020ad354>] __btrfs_update_delayed_inode+0x64/0x230 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff80020ad9fc>] __btrfs_run_delayed_items.isra.0+0x19c/0x200 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff80020adfb4>] btrfs_run_delayed_items+0x2c/0x38 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff800204fc08>] btrfs_commit_transaction.part.13+0x2a0/0x9e8 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff80020503b8>] btrfs_commit_transaction+0x68/0xc0 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff800204af78>] transaction_kthread+0x1a0/0x1d0 [btrfs]
[Mon Nov 16 15:57:22 2020] [<ffffff80090ca888>] kthread+0xf8/0x100
[Mon Nov 16 15:57:22 2020] [<ffffff80090836c0>] ret_from_fork+0x10/0x50
[Mon Nov 16 15:57:22 2020] Code: f9400381 52826003 b9407f80 52826015 (b9404662)
[Mon Nov 16 15:57:22 2020] ---[ end trace f12ac022966fbb50 ]---

Jschoeck
Posts: 17
Joined: Fri Nov 06, 2020 9:30 pm
languages_spoken: english, german
ODROIDs: HC2, HC4
Has thanked: 3 times
Been thanked: 0
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by Jschoeck »

I also have Internal error: Oops: 96000005 [#1] PREEMPT SMP on the HC4 using btrfs disks in the HC4.
Is there a possibility to circumvent / fix this or is it possible that btrfs doesn't work on the HC4 image or even hardware somehow?!
That would really be a shame, as that is my planned NAS setup and I got the HC4 just to finally be able to realize it...

ccdd33
Posts: 6
Joined: Sun Nov 17, 2019 10:39 am
languages_spoken: english
ODROIDs: C4
Has thanked: 0
Been thanked: 1 time
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by ccdd33 »

I am using this ubuntu image with kernel 5.x.
btrfs works well without kernel panic.
but i did not test gpiopower.

viewtopic.php?t=38557

Ubuntu 20.04 (Focal Fossa) Minimal Image with Linux kernel 5.x is ready to download...

Image version histories:

20201024
These users thanked the author ccdd33 for the post:
seppsyrj (Tue Nov 17, 2020 4:28 pm)

User avatar
tobetter
Posts: 6622
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: Many
Location: Paju, South Korea
Has thanked: 328 times
Been thanked: 921 times
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by tobetter »

I was about to reply the same with @ccdd33,

For BTRFS, you would need to use the more advanced kernel later than 5.4. the stock OS image from Hardkernel is shipped with Linux 4.9 kernel.

Please refer to the link for the image if you want to try 5.x kernel.
viewtopic.php?p=289476#p289476

For HC4, you can install the OS from the Petitboot using the Netboot Installer as well.
https://www.youtube.com/watch?v=ugDcoqfb4Xw&t=24s

@ccdd33, for GPIO power, the mainline kernel is a bit far from fully supported yet. So the suspend/resume would not properly work yet unfortunately.
These users thanked the author tobetter for the post:
seppsyrj (Tue Nov 17, 2020 4:28 pm)

Jschoeck
Posts: 17
Joined: Fri Nov 06, 2020 9:30 pm
languages_spoken: english, german
ODROIDs: HC2, HC4
Has thanked: 3 times
Been thanked: 0
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by Jschoeck »

I am also using the 20201024 Ubuntu Mate image.

When I do this for example:
mount -t btrfs /dev/sdb1 /mnt/ssd
btrfs subvolume create /mnt/ssd/ROOT

These errors freeze my system:

Code: Select all

kernel:[ 2411.320757@1] Internal error: Oops: 96000005 [#1] PREEMPT SMP
kernel:[ 2411.929413@1] Process btrfs (pid: 2996, stack limit = 0xffffffc0b4b5c000)
Complete error message text at https://pastebin.com/VDtuWquz

Jschoeck
Posts: 17
Joined: Fri Nov 06, 2020 9:30 pm
languages_spoken: english, german
ODROIDs: HC2, HC4
Has thanked: 3 times
Been thanked: 0
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by Jschoeck »

tobetter wrote:
Tue Nov 17, 2020 11:05 am
For BTRFS, you would need to use the more advanced kernel later than 5.4. the stock OS image from Hardkernel is shipped with Linux 4.9 kernel.

Please refer to the link for the image if you want to try 5.x kernel.
viewtopic.php?p=289476#p289476

For HC4, you can install the OS from the Petitboot using the Netboot Installer as well.
https://www.youtube.com/watch?v=ugDcoqfb4Xw&t=24s
I don't quite understand, it says in viewtopic.php?p=309319#p309319 that HC4 Ubuntu Mate image comes with kernel 5.8.16, so btrfs should be supported out of the box, no?
But you are right, my system says 4.9.236-51.

Is is possible to upgrade the kernel via SSH like on a normal linux machine or do we need to do it with petitboot for some reason?

User avatar
tobetter
Posts: 6622
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: Many
Location: Paju, South Korea
Has thanked: 328 times
Been thanked: 921 times
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by tobetter »

Jschoeck wrote:
Tue Nov 17, 2020 11:12 am
tobetter wrote:
Tue Nov 17, 2020 11:05 am
For BTRFS, you would need to use the more advanced kernel later than 5.4. the stock OS image from Hardkernel is shipped with Linux 4.9 kernel.

Please refer to the link for the image if you want to try 5.x kernel.
viewtopic.php?p=289476#p289476

For HC4, you can install the OS from the Petitboot using the Netboot Installer as well.
https://www.youtube.com/watch?v=ugDcoqfb4Xw&t=24s
I don't quite understand, it says in viewtopic.php?p=309319#p309319 that HC4 Ubuntu Mate image comes with kernel 5.8.16, so btrfs should be supported out of the box, no?
But you are right, my system says 4.9.236-51.
Sorry, I cannot find that Ubuntu Mate comes with 5.8.16 in the link.
Is is possible to upgrade the kernel via SSH like on a normal linux machine or do we need to do it with petitboot for some reason?
Sorry, the partition layout is different. Hardkernel stock Ubuntu image uses "FAT+EXT4" partitions while my custom Debian/Ubuntu images uses "EXT2+EXT4". Also both are based on different package repository server so it's not simply upgraded. The Petitboot is the boot manager, it does not required to be used all the time. But the Petitboot offer you the convenient way to install Debian or Ubuntu using the Netboot Installer without downloading and flashing any image to your SD card before power up.
https://www.youtube.com/watch?v=ugDcoqfb4Xw&t=29s

For HC4, you can ignore or remove the Petitboot if you like to boot from SD card and runs disk on the toast box as a storage. But you will need the Petitboot if you like to boot from SATA or USB storages other than SD card since the boot loader does not support load and boot Linux kernel from SATA storage.

Jschoeck
Posts: 17
Joined: Fri Nov 06, 2020 9:30 pm
languages_spoken: english, german
ODROIDs: HC2, HC4
Has thanked: 3 times
Been thanked: 0
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by Jschoeck »

Sorry I missed the fact that Ubuntu and Ubuntu Mate are different images and come with a different kernel.
So my question would be, is it possible to upgrade your Ubuntu Mate image to the 5.9 kernel?

User avatar
tobetter
Posts: 6622
Joined: Mon Feb 25, 2013 10:55 am
languages_spoken: Korean, English
ODROIDs: Many
Location: Paju, South Korea
Has thanked: 328 times
Been thanked: 921 times
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by tobetter »

Jschoeck wrote:
Tue Nov 17, 2020 11:25 am
Sorry I missed the fact that Ubuntu and Ubuntu Mate are different images and come with a different kernel.
So my question would be, is it possible to upgrade your Ubuntu Mate image to the 5.9 kernel?
Likely no, as I replied both images are referring to different package server. Even if it could be, the package dependency could be mess after update. So, I kindly recommend installing new after backing up your previous OS setup with 4.9 kernel.

elatllat
Posts: 1833
Joined: Tue Sep 01, 2015 8:54 am
languages_spoken: english
ODROIDs: XU4, N1, N2, C4, N2+, HC4
Has thanked: 57 times
Been thanked: 125 times
Contact:

Re: ODROID-C4: BTRFS on USB disks segfaults on Ubuntu

Post by elatllat »

btrfs works well for me with 5.4 from linux.org so as tobetter said updating your kernel should fix the issue (build it yourself or use a different distro).
These users thanked the author elatllat for the post:
seppsyrj (Tue Dec 01, 2020 4:20 pm)

Post Reply

Return to “Ubuntu”

Who is online

Users browsing this forum: No registered users and 0 guests