Compiling the android kernel for n2

[mad_ady]

I'm trying to add some extra configuration to voodik's ATV and I want to add one-wire temperature sensor support to the kernel. So, here I go on a new journey and I need some advice...

I'm guessing the kernel for N2 comes from here:
https://github.com/hardkernel/linux/tre ... .y-android. Though this branch doesn't have all the patches from the non-android one.
Do I need to use a cross compiler as stated here - https://wiki.odroid.com/odroid-n2/softw ... ng_android, or can I compile it on ubuntu 18.04 arm64 with gcc 7.5.0? I see the original one was built with gcc version 6.3.1 20170404 (Linaro GCC 6.3-2017.05).

I can get the default configuration from /proc/config.gz, so no problem there.
What commands do I need to build the kernel? I'm thinking of:

Code: Select all

make -j 4 Image dtbs modules

The modules should go to /vendor/lib/modules, I see.
Kernel and dtb seem to be flashed in two partitions:

Code: Select all

[    1.269566] meson-mmc: [mmc_read_partition_tbl] mmc read partition OK!
[    1.269568] meson-mmc: add_emmc_partition
[    1.269661] meson-mmc: [mmcblk0p01]           bootloader  offset 0x000000000001, size 0x00000000077f     
[    1.269738] meson-mmc: [mmcblk0p02]                  env  offset 0x000000000780, size 0x000000000080     
[    1.269817] meson-mmc: [mmcblk0p03]               ptable  offset 0x000000000800, size 0x000000000008     
[    1.269891] meson-mmc: [mmcblk0p04]                 misc  offset 0x000000000808, size 0x000000000008     
[    1.269974] meson-mmc: [mmcblk0p05]                 logo  offset 0x000000000810, size 0x000000001000     
[    1.270054] meson-mmc: [mmcblk0p06]                 dtbs  offset 0x000000001810, size 0x000000000100     
[    1.270138] meson-mmc: [mmcblk0p07]                 boot  offset 0x000000001910, size 0x000000008000     
[    1.270231] meson-mmc: [mmcblk0p08]             recovery  offset 0x000000009910, size 0x00000000c000     
[    1.270312] meson-mmc: [mmcblk0p09]                cache  offset 0x000000015910, size 0x000000200000     
[    1.270394] meson-mmc: [mmcblk0p10]                  odm  offset 0x000000215910, size 0x000000010000     
[    1.270475] meson-mmc: [mmcblk0p11]               system  offset 0x000000225910, size 0x000000380000
[    1.270552] meson-mmc: [mmcblk0p12]               vendor  offset 0x0000005a5910, size 0x000000100000     
[    1.270633] meson-mmc: [mmcblk0p13]              product  offset 0x0000006a5910, size 0x000000010000    

I'm not sure what size/offset are measured in. Bytes is too small, so it's probably blocks. I'm guessing 512 bytes long.

So my plan is - compile the new kernel (ideally on a c2, but I can do it on a pc if I must), dd the new kernel (is it in Image format?) and dtb to their partitions (e.g. with dd from android directly or offline with dd + offset) and copy over the modules.

Am I missing something?

Edit:
I extracted the kernel, dtb from their partitions and they look ok:

Code: Select all

root@localhost:~# file /kernel.img
/kernel.img: Android bootimg, kernel (0x1080000), ramdisk (0x1000000), page size: 2048, cmdline (otg_device=1 buildvariant=userdebug)
root@localhost:~# file /dtb.img
/dtb.img: Device Tree Blob version 17, size=74368, boot CPU=0, string block size=7632, DT structure block size=66680
root@localhost:~#

[secuflag]

Download toolchain from here: https://releases.linaro.org/components/ ... gnu.tar.xz

Code: Select all

$ git clone --depth 1 https://github.com/hardkernel/linux.git -b odroidn2-4.9.y-android
$ cd linux
$ export ARCH=arm64
$ export CROSS_COMPILE=aarch64-linux-gnu-
$ export PATH=/PATH/TO/gcc-linaro-6.3.1-2017.02-x86_64_aarch64-linux-gnu/bin/:$PATH
$ make odroidn2_android_defconfig
$ make -j$(nproc)

Replace /PATH/TO/ with the correct path.

BUT, to build a flashable boot image you also need the rootFS and ramdisk.img so I suggest you to download the whole android BSP and compile the boot image from there.

[mad_ady]

Thanks. If I replace only the kernel, wouldn't it boot an existing android image? If not, why not? What's the problem?
I don't want to build the whole android for a few modules. It doesn't sound normal...

[secuflag]

On Android the kernel it's packed inside boot.img file with rootFS, ramdisk and/or other tools so if you add or remove features you must recompile it and rebuild the boot.img.

[mad_ady]

Oh, ok, forgot about that... Can I cheat and extract the rootfs from the existing boot partition and glue it back to the new kernel with some dd trickery?

If I build the boot.img from the android bsp - does that involve building the android framework/apps as well, or is it a relatively fast build, with just the cost of sources/disk space?

Thanks for enlightning me.

[secuflag]

You can try with a tool like this: https://forum.xda-developers.com/showth ... ?t=2319018

If you download the BSP you can build only the boot image by using: make -j$(nproc) bootimage
It just take some minutes, depending on your hardware of course.

[mad_ady]

Thanks, I'm in your debt!

[mad_ady]

Ok, made a little time at work to play with this and I feel I'm close to success (but not there yet!).
So - I compiled the android kernel as instructed. All went well there. Next I'm trying to rebuild the bootimg:

Code: Select all

cd ~/development/bootimg_tools
chmod a+x *
cd ..
PATH=~/development/bootimg_tools/:$PATH
mkdir bootimg
cd bootimg

adb shell
odroidn2:/ # dd if=/dev/block/boot of=/dev/shm/boot.img
odroidn2:/ # dd if=/dev/block/dtbs of=/dev/shm/dtbs.img
odroidn2:/ # exit
adb pull /dev/shm/boot.img
adb pull /dev/shm/dtbs.img

Code: Select all

$ file boot.img 
boot.img: Android bootimg, kernel (0x1080000), ramdisk (0x1000000), page size: 2048, cmdline (otg_device=1 buildvariant=userdebug)
$ file dtbs.img
dtbs.img: Device Tree Blob version 17, size=74368, boot CPU=0, string block size=7632, DT structure block size=66680

$ boot_info 
PAGE SIZE: 2048
BASE ADDRESS: 0x01080000
RAMDISK ADDRESS: 0x01000000
CMDLINE: 'otg_device=1 buildvariant=userdebug'

$ unpack
Boot = boot.img
unmkbootimg version 1.2 - Mikael Q Kuisma <kuisma@ping.se>
Kernel size 9643391
Kernel address 0x1080000
Ramdisk size 1631542
Ramdisk address 0x1000000
Secondary size 0
Secondary address 0xf00000
Kernel tags address 0x100
Flash page size 2048
Board name is ""
Command line "otg_device=1 buildvariant=userdebug"

*** WARNING ****
This image is built using NON-standard mkbootimg!
OFF_KERNEL_ADDR is 0x01080000
Please modify mkbootimg.c using the above values to build your image.
****************

Extracting kernel to file zImage ...
Extracting root filesystem to file initramfs.cpio.gz ...
All done.
---------------
To recompile this image, use:
  mkbootimg --kernel zImage --ramdisk initramfs.cpio.gz --base 0x0 --cmdline 'otg_device=1 buildvariant=userdebug' -o new_boot.img
---------------
7472 blocks

To build the new boot img I tried this:

Code: Select all

mkbootimg --kernel ~/development/linux/arch/arm64/boot/Image.gz --ramdisk boot/ramdisk.cpio.gz --base 0x1000000 --ramdiskaddr 0x1000000 --cmdline 'otg_device=1 buildvariant=userdebug' -o new_boot.img

However, the new bootimg has a different kernel load address than the original one:

Code: Select all

$ file boot.img
boot.img: Android bootimg, kernel (0x1080000), ramdisk (0x1000000), page size: 2048, cmdline (otg_device=1 buildvariant=userdebug)
$ file new_boot.img 
new_boot.img: Android bootimg, kernel (0x1008000), ramdisk (0x1000000), page size: 2048, cmdline (otg_device=1 buildvariant=userdebug)

The message above suggests changing mkbootimg OFF_KERNEL_ADDR, but the one that came from xda_developers has only the binary file. Any suggestion on where I should look for a compatible mkbootimg (or source) that can build on N2?
Thanks!

[mad_ady]

Ok, I think I moved forward. I got a different mkbootimg:

Code: Select all

cd ~/development
git clone  https://github.com/osm0sis/mkbootimg.git
cd mkbootimg
make
cd ../bootimg_tools/
mv mkbootimg mkbootimg.ol
cp ../mkbootimg/mkbootimg .
cd ../android-development/
mkbootimg --kernel ~/development/linux/arch/arm64/boot/Image.gz --ramdisk boot/ramdisk.cpio.gz --base 0x1000000 --kernel_offset 0x80000 --ramdisk_offset 0x0 --cmdline 'otg_device=1 buildvariant=userdebug' -o new_boot.img

And now before and after:

Code: Select all

$ file boot.img 
boot.img: Android bootimg, kernel (0x1080000), ramdisk (0x1000000), page size: 2048, cmdline (otg_device=1 buildvariant=userdebug)
$ file new_boot.img 
new_boot.img: Android bootimg, kernel (0x1080000), ramdisk (0x1000000), page size: 2048, cmdline (otg_device=1 buildvariant=userdebug)

So, it looks good.

Old (original) modules have this vermagic:

Code: Select all

odroidn2:/ # modinfo /vendor/lib/modules/lcd.ko                                
filename:       /vendor/lib/modules/lcd.ko
description:    LCD Lowlevel Control Abstraction
author:         Jamey Hicks <jamey.hicks@hp.com>, Andrew Zabolotny <zap@homelink.ru>
license:        GPL
depends:        
intree:         Y
vermagic:       4.9.y SMP preempt mod_unload modversions aarch64

New drivers have the same vermagic, so I guess I don't need to overwrite them if I just compiled things into the kernel, right?:

Code: Select all

$ modinfo ./drivers/video/backlight/lcd.ko
filename:       /home/adrianp/development/linux/./drivers/video/backlight/lcd.ko
description:    LCD Lowlevel Control Abstraction
author:         Jamey Hicks <jamey.hicks@hp.com>, Andrew Zabolotny <zap@homelink.ru>
license:        GPL
depends:        
intree:         Y
vermagic:       4.9.y SMP preempt mod_unload modversions aarch64

If I were to compile things as modules, how would I copy them to one directory without keeping the directory structure (like make modules_install does)?
Also, when using modules, how do I load them on Android startup?

Now - for my use case I think I only need to do the following (when I have time to unbrick it):

Code: Select all

adb push new_boot.img /sdcard/
adb push ~/development/linux/arch/arm64/boot/dts/amlogic/meson64_odroidn2_android.dtb /sdcard/
adb shell
# dd if=/sdcard/new_boot.img /dev/block/boot
# dd if=/sdcard/meson64_odroidn2_android.dtb /dev/block/dtbs

Is my assumption correct and it should work?

Now I need to find 30 minutes to try it out... Maybe next week...
Thanks!

[tobetter]

@mad_ady, I hope I am not late to help you out.

I suggest using another tool abootimg instead of mkbootimg in case if you like to replace Kernel for the running Android system. For example, if you have Android Pie 64bit (v2020010), https://wiki.odroid.com/odroid-n2/os_im ... 4_20200110, you can download a file Fastboot update that has many files can be flashed to Android partitions in a memory card. From the file, take boot.img which is compress Linux kernel and ramdisk combo.

abootimg -x boot.img will extract the boot image.

Code: Select all

$ abootimg -x boot.img
writing boot image config in bootimg.cfg
extracting kernel in zImage
extracting ramdisk in initrd.img
extracting second stage image in stage2.img

After compiling the kernel for Android, you must be having arch/arm64/boot/zImage in your kernel tree. abootimg is your friend to merge your zImage to the original boot.img. Suppose that you run the command in the same directory where you extract the original boot.img, you will have new boot image boot_new.img that has your custom kernel.

Code: Select all

$ abootimg --create boot_new.img -f bootimg.cfg -k <path/of/your/kernel/arch/arm64/boot/zImage> -r initrd.img

Please do use fastboot instead of dd since fastboot is supporting by U-boot, as long as you do not break U-boot you can recover bricked Android due to invalid boot image.
https://wiki.odroid.com/getting_started/adb_fastboot

Android does not use the module directory like /lib/modules/4.9.y... like ordinary Android. But you can copy your .ko to anywhere and load them. Hardkernel's Android stores .ko in /vendor/lib/modules and a kernel module can be loaded by an init script.
https://github.com/codewalkerster/andro ... tem.rc#L96

Hope this could help you.

[mad_ady]

Thanks, I'll try your suggestions and see how it goes. Regarding dtb - is it read from the dtbs partition or from the meson64_odroidn2_android.dtb file in /odm?

If I manage to brick things, how can I get in fastboot mode? I know I can via serial (and interrupting uboot), but I should be able to run the fastboot command in boot.ini as well, right - if I don't have serial?

[tobetter]

Thanks, I'll try your suggestions and see how it goes. Regarding dtb - is it read from the dtbs partition or from the meson64_odroidn2_android.dtb file in /odm?

boot.ini in /odm has a higher priority than others flashed with fastboot.
https://github.com/hardkernel/u-boot/bl ... mmon.h#L97

If I manage to brick things, how can I get in fastboot mode? I know I can via serial (and interrupting uboot), but I should be able to run the fastboot command in boot.ini as well, right - if I don't have serial?

Absolutely, also you can use reboot fastboot in the Android command shell.

[mad_ady]

Ok, I managed to try out the kernel I compiled. I tried two things:
- one kernel with only 1-wire support
- one kernel with the necessary stuff built-in to support docker (I like to live dangerously!)

The one-wire kernel booted fine, though I'm not sure (yet) that one-wire works. I enabled it in the dtb before flashing:

Code: Select all

    onewire {
        compatible = "w1-gpio";
        gpios = <0x0000001a 0x0000003f 0x00000000>;
        status = "disabled";
        phandle = <0x000000e4>;
    };

However, under /sys/bus/w1/devices/ I don't get the w1_bus_master1 entry - so I may be missing something, not sure what...
The wire, w1_gpio and w1_therm modules are built-in, not modules - maybe that's an issue?

The second kernel with the docker stuff boots and reaches userspace for ~30s but causes a panic and triggers a reboot. I need to capture logs and see what may trigger it.

[tobetter]

Ok, I managed to try out the kernel I compiled. I tried two things:
- one kernel with only 1-wire support
- one kernel with the necessary stuff built-in to support docker (I like to live dangerously!)

The one-wire kernel booted fine, though I'm not sure (yet) that one-wire works. I enabled it in the dtb before flashing:

Code: Select all

    onewire {
        compatible = "w1-gpio";
        gpios = <0x0000001a 0x0000003f 0x00000000>;
        status = "disabled";
        phandle = <0x000000e4>;
    };

However, under /sys/bus/w1/devices/ I don't get the w1_bus_master1 entry - so I may be missing something, not sure what...
The wire, w1_gpio and w1_therm modules are built-in, not modules - maybe that's an issue?

The second kernel with the docker stuff boots and reaches userspace for ~30s but causes a panic and triggers a reboot. I need to capture logs and see what may trigger it.

For Docker, did you check your config with this config?
https://github.com/moby/moby/blob/maste ... -config.sh

[mad_ady]

Yes, and enabled critical docker modules for cgroups, namespaces, network

Edit: This is what I enabled for docker:

Code: Select all

 adrianp  BR odroidn2-4.9.y-android  ~ | development | linux  1  0  diff -u .config-with-temperature .config | grep '^\+' | grep -v '^\+#'
+++ .config	2020-01-17 12:00:35.347262575 +0200
+CONFIG_CGROUP_PIDS=y
+CONFIG_CGROUP_DEVICE=y
+CONFIG_PAGE_COUNTER=y
+CONFIG_MEMCG=y
+CONFIG_MEMCG_SWAP=y
+CONFIG_MEMCG_SWAP_ENABLED=y
+CONFIG_BLK_CGROUP=y
+CONFIG_CGROUP_WRITEBACK=y
+CONFIG_CFS_BANDWIDTH=y
+CONFIG_CGROUP_PERF=y
+CONFIG_NAMESPACES=y
+CONFIG_UTS_NS=y
+CONFIG_IPC_NS=y
+CONFIG_USER_NS=y
+CONFIG_PID_NS=y
+CONFIG_NET_NS=y
+CONFIG_BLK_DEV_THROTTLING=y
+CONFIG_CFQ_GROUP_IOSCHED=y
+CONFIG_NETFILTER_XT_MATCH_ADDRTYPE=y
+CONFIG_NETFILTER_XT_MATCH_IPCOMP=y
+CONFIG_NETFILTER_XT_MATCH_IPVS=y
+CONFIG_IP_VS=y
+CONFIG_IP_VS_TAB_BITS=12
+
+CONFIG_IP_VS_PROTO_TCP=y
+CONFIG_IP_VS_PROTO_UDP=y
+
+CONFIG_IP_VS_RR=y
+
+CONFIG_IP_VS_SH_TAB_BITS=8
+
+CONFIG_IP_VS_NFCT=y
+CONFIG_BRIDGE_VLAN_FILTERING=y
+CONFIG_VLAN_8021Q=y
+CONFIG_NET_CLS_CGROUP=y
+CONFIG_NET_L3_MASTER_DEV=y
+CONFIG_CGROUP_NET_PRIO=y
+CONFIG_CGROUP_NET_CLASSID=y
+CONFIG_BONDING=y
+CONFIG_DUMMY=y
+CONFIG_MACVLAN=y
+CONFIG_MACVTAP=y
+CONFIG_IPVLAN=y
+CONFIG_VXLAN=y
+CONFIG_NETCONSOLE=y
+CONFIG_NETPOLL=y
+CONFIG_NET_POLL_CONTROLLER=y
+CONFIG_VETH=y
+CONFIG_OVERLAY_FS=y

[ali2019]

Hello, @tobetter.

Can you please share link for download abootimg tool?

[tobetter]

Hello, @tobetter.

Can you please share link for download abootimg tool?

If you use Debian/Ubuntu, you can simply install it with sudo apt install abootimg. Done.

[mad_ady]

Hmm, I'm tinkering again with my custom android kernel - and I'm unable to get onewire working. Most likely I'm missing something (something not enabled) in the dtb...
This time I compiled W1 as kernel built-in, but w1-gpio and w1_therm as modules so I can insert them manually.

When the kernel boots, it says:

Code: Select all

[    0.864943] Driver for 1-wire Dallas network protocol.

When I insmod w1-gpio.ko, I get this in dmesg:

Code: Select all

[  233.351466] type=1400 audit(1579703801.146:172): avc: denied { open } for pid=5877 comm="getprop" path="/dev/__properties__/u:object_r:net_dns_prop:s0" dev="tmpfs" ino=3258 scontext=u:r:sudaemon:s0 tcontext=u:object_r:net_dns_prop:s0 tclass=file permissive=1
[  233.351518] type=1400 audit(1579703977.370:177): avc: denied { module_load } for pid=6053 comm="insmod" path="/vendor/lib/modules/w1-gpio.ko" dev="mmcblk0p12" ino=5516 scontext=u:r:sudaemon:s0 tcontext=u:object_r:vendor_file:s0 tclass=system permissive=1

When I insmod w1_therm.ko I don't get anything in the dmesg log.
The problem is - I never get a /sys/bus/w1/devices/w1_bus_master, let alone any slaves devices...
In the dtb I have enabled onewire, but not sure if I need to enable other things as well. The gpio pins are the same as in the linux dtb, but there is a different phandle (whatever that is):

Code: Select all

$ fdtput -t s meson64_odroidn2_android.dtb /onewire status enabled

Code: Select all

N2 linux dtb:
    onewire {
        compatible = "w1-gpio";
        gpios = <0x0000001a 0x0000003f 0x00000000>;
        status = "disabled";
        phandle = <0x000000e4>;
    };

N2 android dtb:
    onewire {
        compatible = "w1-gpio";
        gpios = <0x0000001a 0x0000003f 0x00000000>;
        status = "enabled";
        phandle = <0x000000f0>;
    };

I'm stuck. Any ideas on how to get onewire working?

[ali2019]

Hello, @tobetter.

I created new boot image with abootimg tool.
When try to flash to odroid device, I see waiting device issue.

$ fastboot flash boot boot_new.img
< waiting for any device>

$ adb devices
List of devices attached
001e064223c9 device

it seems that adb fount device connection.

I am trying on virtual box ubuntu with usb device connection.

Please advise me about waiting device issue.
Thanks

[joerg]

@mad_ady

I'm stuck. Any ideas on how to get onewire working?

Have you tried with status = "okay";?

[joerg]

@ali2019,
if the N2 answers at adb devices it is not in fastboot mode.
Try adb reboot fastboot. If this does not work, here on wiki page is described how to: https://wiki.odroid.com/odroid-n2/softw ... ndroid#faq

[mad_ady]

@mad_ady

I'm stuck. Any ideas on how to get onewire working?

Have you tried with status = "okay";?

No, the onewire treedidn't have a status entry. But adding one I guess can't hurt. I'll try tomorrow

[mad_ady]

Yay! Thank you @joerg! I played with the status and it seems the opposite of "disabled" is not "enabled". It could be "ok" or "okay". I played with it and it was "okay"!

Code: Select all

fdtput -t s meson64_odroidn2_android.dtb /onewire status okay

Rebooted, loaded the modules (I'll build them in the kernel next time) and voila!

Code: Select all

root@localhost:~# cat /sys/bus/w1/devices/28-0516866e14ff/w1_slave 
7b 01 4b 46 7f ff 0c 10 af : crc=af YES
7b 01 4b 46 7f ff 0c 10 af t=23687

Now I can repurpose my first C2 to live a new life
Thanks again!
I will also experiment with enabling docker support in the kernel. The N2 is wasted to run just Android IMHO.

[joerg]

Fine that I could help a good guy.

[ali2019]

Hello, @joerg

$ fastboot flash dtb <mylinux>/arch/arm64/boot/dts/amlogic/meson64_odroidn2_android.dtb
target reported max download size of 134217728 bytes
sending 'dtb' (72 KB)...
OKAY [ 0.031s]
writing 'dtb'...
FAILED (remote: 0x08000000cannot find partition)
finished. total time: 0.070s

When flash dtb file after build kernel, I am facing this FAILED issue.
I flashed boot.img file successfully.

I want a advise about this.

Thanks.

[tobetter]

Hello, @joerg

$ fastboot flash dtb <mylinux>/arch/arm64/boot/dts/amlogic/meson64_odroidn2_android.dtb
target reported max download size of 134217728 bytes
sending 'dtb' (72 KB)...
OKAY [ 0.031s]
writing 'dtb'...
FAILED (remote: 0x08000000cannot find partition)
finished. total time: 0.070s

When flash dtb file after build kernel, I am facing this FAILED issue.
I flashed boot.img file successfully.

I want a advise about this.

Thanks.

If you have a UART debug cable, connect to U-boot command line and run the command fastboot, it will show you what partitions are available. If you do not have partitions initialized or not sure, please do run command "fastboot oem fdisk" from your PC when N2 is running fastboot.

[mad_ady]

@ali2019: the partition name is dtbs, not dtb. Try:

Code: Select all

fastboot flash dtbs arch/arm64/boot/dts/amlogic/meson64_odroidn2_android.dtb 

By the way, what are you adding to your kernel? It may require changes to the dtb to enable.

[ali2019]

@mad_ady

Thanks for your reply.

I am building a usb driver on odroid-n2 device. I already tested usb driver code on x86 platform. But During migrate to odroid device, driver is not running.
So I am going to enable usb hid and rebuilt kernel from source code.

#
# USB Input Devices
#
CONFIG_USB_HID=y
# CONFIG_HID_PID is not set
# CONFIG_USB_HIDDEV is not set

So I configured kernel like above and try to build.

If you have any idea for usb driver, advise me.

Regards.

[mad_ady]

Ok, I've been testing some kernel parameters to enable docker support for android. I've been enabling options one by one, but now I've reached the option that causes a kernel panic.

http://paste.ubuntu.com/p/wwxFcSPHv9/

The items added to the config that cause the bootloop are:

Code: Select all

--- config	2020-02-26 10:11:04.002233768 +0200
+++ config.txt	2020-02-26 10:10:25.597804791 +0200
@@ -137,7 +137,9 @@
 CONFIG_MEMCG=y
 CONFIG_MEMCG_SWAP=y
 CONFIG_MEMCG_SWAP_ENABLED=y
-# CONFIG_BLK_CGROUP is not set
+CONFIG_BLK_CGROUP=y
+# CONFIG_DEBUG_BLK_CGROUP is not set
+CONFIG_CGROUP_WRITEBACK=y
 CONFIG_CGROUP_SCHED=y
 CONFIG_FAIR_GROUP_SCHED=y
 # CONFIG_CFS_BANDWIDTH is not set
@@ -292,6 +294,7 @@
 CONFIG_BLK_DEV_BSG=y
 # CONFIG_BLK_DEV_BSGLIB is not set
 # CONFIG_BLK_DEV_INTEGRITY is not set
+# CONFIG_BLK_DEV_THROTTLING is not set
 # CONFIG_BLK_CMDLINE_PARSER is not set
 
 #
@@ -308,6 +311,7 @@
 CONFIG_IOSCHED_NOOP=y
 CONFIG_IOSCHED_DEADLINE=y
 CONFIG_IOSCHED_CFQ=y
+# CONFIG_CFQ_GROUP_IOSCHED is not set
 # CONFIG_DEFAULT_DEADLINE is not set
 # CONFIG_DEFAULT_CFQ is not set
 CONFIG_DEFAULT_NOOP=y

So, basically enabling CONFIG_BLK_CGROUP causes the android kernel to reboot. Will try to add other things to the android kernel and skip this one and see how docker likes it...

[tobetter]

Ok, I've been testing some kernel parameters to enable docker support for android. I've been enabling options one by one, but now I've reached the option that causes a kernel panic.

http://paste.ubuntu.com/p/wwxFcSPHv9/

The items added to the config that cause the bootloop are:

Code: Select all

--- config	2020-02-26 10:11:04.002233768 +0200
+++ config.txt	2020-02-26 10:10:25.597804791 +0200
@@ -137,7 +137,9 @@
 CONFIG_MEMCG=y
 CONFIG_MEMCG_SWAP=y
 CONFIG_MEMCG_SWAP_ENABLED=y
-# CONFIG_BLK_CGROUP is not set
+CONFIG_BLK_CGROUP=y
+# CONFIG_DEBUG_BLK_CGROUP is not set
+CONFIG_CGROUP_WRITEBACK=y
 CONFIG_CGROUP_SCHED=y
 CONFIG_FAIR_GROUP_SCHED=y
 # CONFIG_CFS_BANDWIDTH is not set
@@ -292,6 +294,7 @@
 CONFIG_BLK_DEV_BSG=y
 # CONFIG_BLK_DEV_BSGLIB is not set
 # CONFIG_BLK_DEV_INTEGRITY is not set
+# CONFIG_BLK_DEV_THROTTLING is not set
 # CONFIG_BLK_CMDLINE_PARSER is not set
 
 #
@@ -308,6 +311,7 @@
 CONFIG_IOSCHED_NOOP=y
 CONFIG_IOSCHED_DEADLINE=y
 CONFIG_IOSCHED_CFQ=y
+# CONFIG_CFQ_GROUP_IOSCHED is not set
 # CONFIG_DEFAULT_DEADLINE is not set
 # CONFIG_DEFAULT_CFQ is not set
 CONFIG_DEFAULT_NOOP=y

So, basically enabling CONFIG_BLK_CGROUP causes the android kernel to reboot. Will try to add other things to the android kernel and skip this one and see how docker likes it...

Are you trying to build an Android kernel with Docker features to let Android start another OS?

[mad_ady]

I'm experimenting with starting docker containers either from within Android - it shouldn't be impossible: https://forums.docker.com/t/is-it-possi ... s/16135/12
Or, running it from a linux chroot from within Android (I'm using linux deploy).
I'm curious how far I can get. The truth is the N2 is way to powerful to waste it on Android alone. It can do lots of tasks in the background and docker seems to be the new linux "package manager"

[mad_ady]

Almost there...

Code: Select all

root@localhost:~# ./check-config.sh 
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled
- CONFIG_IP_NF_FILTER: enabled
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled
- CONFIG_IP_NF_NAT: enabled
- CONFIG_NF_NAT: enabled
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_BLK_CGROUP: missing
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: missing
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: missing
- CONFIG_CGROUP_NET_PRIO: missing
- CONFIG_CFS_BANDWIDTH: missing
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_NF_TARGET_REDIRECT: enabled
- CONFIG_IP_VS: enabled
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled
- CONFIG_EXT3_FS: enabled
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: enabled
- CONFIG_EXT3_FS_SECURITY: enabled
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled
    - CONFIG_NF_CONNTRACK_FTP: enabled
    - CONFIG_NF_NAT_TFTP: enabled
    - CONFIG_NF_CONNTRACK_TFTP: enabled
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: missing
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

root@localhost:~# docker info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 19.03.6
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.113
 Operating System: Ubuntu 18.04 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 3.624GiB
 Name: localhost
 ID: XYT2:K4WV:OILD:NSUO:GBGB:E4LK:EPIB:A2J3:NYCQ:4FIZ:EDY4:YYI3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
root@localhost:~# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
256ab8fe8778: Pull complete 
Digest: sha256:fc6a51919cfeb2e6763f62b6d9e8815acbf7cd2e476ea353743570610737b752
Status: Downloaded newer image for hello-world:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:281: applying cgroup configuration for process caused \"open /sys/fs/cgroup/cpuset/docker/cpuset.cpus: no such file or directory\"": unknown.
ERRO[0004] error waiting for container: context canceled 
root@localhost:~# ls -l /sys/fs/cgroup/cpuset/docker/           
total 0
-rw-r--r--. 1 root root 0 Feb 26 15:31 cgroup.clone_children
-rw-r--r--. 1 root root 0 Feb 26 15:31 cgroup.procs
-rw-r--r--. 1 root root 0 Feb 26 15:31 cpu_exclusive
-rw-r--r--. 1 root root 0 Feb 26 15:31 cpus
-r--r--r--. 1 root root 0 Feb 26 15:31 effective_cpus
-r--r--r--. 1 root root 0 Feb 26 15:31 effective_mems
-rw-r--r--. 1 root root 0 Feb 26 15:31 mem_exclusive
-rw-r--r--. 1 root root 0 Feb 26 15:31 mem_hardwall
-rw-r--r--. 1 root root 0 Feb 26 15:31 memory_migrate
-r--r--r--. 1 root root 0 Feb 26 15:31 memory_pressure
-rw-r--r--. 1 root root 0 Feb 26 15:31 memory_spread_page
-rw-r--r--. 1 root root 0 Feb 26 15:31 memory_spread_slab
-rw-r--r--. 1 root root 0 Feb 26 15:31 mems
-rw-r--r--. 1 root root 0 Feb 26 15:31 notify_on_release
-rw-r--r--. 1 root root 0 Feb 26 15:31 sched_load_balance
-rw-r--r--. 1 root root 0 Feb 26 15:31 sched_relax_domain_level
-rw-r--r--. 1 root root 0 Feb 26 15:31 tasks

Seems I'm hitting this problem: https://medium.com/nttlabs/cgroup-v2-596d035be4d7
Not sure if I can fix it, because I don't have systemd taking care of cgroup setup...