Kubernetes CGROUP PIDS

[magicroomy]

My basic goal is to teach my MC1 kubernetes.
I read countless tutorials and docs about the subject ( all are not current).
I always run against the following problem:
At some point in time the kubernetes guys at google decided that kubernetes requires a kernel feature called CGROUP PIDS.
In some forums I read that activating the CGROUP PIDS and recompiling the kernel leads to the system not booting anymore. I tried myself and can agree.
Does anybody know if there is progress made on this subject?
Perfect would be is somebody could tell me where to find a full sd card image containing a kernel with activated CGROUP PIDS.
I tried ubuntu with 4.14 kernel, I also tried armbian with 4.14 kernels or even 5.X kernels.. nothing worked.

[igorpec]

> with 4.14 kernel

We tried to enable this on 4.14.y but it doesn't wants to work ...

> even 5.X kernels.. nothing worked.

There it looks its enabled ... its in source config https://github.com/armbian/build/blob/m ... onfig#L149 while this doesn't mean it was actually build with this support. Check config in /boot directory to see if this was enabled or not. If not, this needs to be deeply investigated.

> Perfect would be

Its some work behind this and that solution.

[mad_ady]

How does it behave with cgroup pid active? Does the kernel boot?Does it gets stuck in userspace? Any last messages?

[magicroomy]

I can't tell if the kernel does anything while it should be booting, since I do not have a UART serial. (about to order one)
The effect is that I had a system up and running and could connect via ssh terminal.
After the kernel update, I cannot reach the odroid anymore. It also does not respond to ping.

[mad_ady]

How about the blue led state? Is it blinking?

[magicroomy]

Ok, here is what I did..
1. Download and flash image from here: https://wiki.odroid.com/odroid-xu4/os_i ... 10-minimal
2.Partially follow the instructions here : https://wiki.odroid.com/odroid-xu4/os_i ... 4/20190929
- I followed the instructions up to the point ---make odroidxu4_defconfig
- after I did this, I changed the .config and enabled CGROUP_PIDS by adding CGROUP_PIDS=y
- then I continued with make -j8 etc.
I copied the new kernel , installed the modules, all as described on the page mentioned under 2
When I finally say reboot. The odroid never comes back.
Blue led is constantly on.

[mad_ady]

It's hard to troubleshoot without a serial cable. Try enabling that option from within make menuconfig - maybe it has some dependencies.

[magicroomy]

I tried the menuconfig way... same result. LED in constant blue. Odroid does not respond anymore to ssh or can even ping. It also does not appear listed in my router.

[magicroomy]

Well I solved my actual problem: get Kubernetes running. I followed the odroid magazine article about installing kubernetes on a n2.
It seems the most important step here is to tell docker not to use CGROUPS but systemd as cgroup driver.

/etc/docker/daemon.json

Code: Select all

{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}

I assume this does not force kubernetes to struggle with CGroup PIDS.
The issue of this topic actually still exists. The kernel as described above seems to have a problem with CGROUP PIDS.
Nevertheless, from my point of view, this topic can be closed.
Thank you for your help.

[obscurerichard]

I too tried to get Kubernetes running on my ODROID HC1 running Ubuntu 20.04 and ran into the same issue with the stock vendor kernel.

I was trying to use https://k3s.io/ to get Kubernetes set up, using the HC1 as an agent.

Whenever the k3s-agent started, it would throw fatal errors related to PID limits (cannot set feature gate SupportPodPidsLimit to false), followed by a long go stack trace. It turns out that the CGROUP_PIDS feature is still turned off in the default kernel. This kernel is using v1 cgroups, but without the PIDS feature enabled Kubernetes fails to start up:

Code: Select all

Feb 12 20:20:00 host.example.com k3s[16364]: F0212 20:20:00.792512   16364 server.go:181] cannot set feature gate SupportPodPidsLimit to false, feature is locked to true
Feb 12 20:20:00 host.example.com k3s[16364]: goroutine 320 [running]:
Feb 12 20:20:00 host.example.com k3s[16364]: github.com/rancher/k3s/vendor/k8s.io/klog/v2.stacks(0x5914a01, 0x0, 0x7d, 0xcf)
Feb 12 20:20:00 host.example.com k3s[16364]:         /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/v2/klog.go:1026 +0x94

To work around this error I was able to follow the kernel rebuilding and installation instructions at https://wiki.odroid.com/odroid-xu4/os_i ... uild_guide and the kernel rebuild and reboot worked flawlessly.

The key change was to make sure that this was present in the kernel .config file:

Code: Select all

CGROUP_PIDS=y

The only 2 changes I made in the .config file were CONFIG_LOCALVERSION and CONFIG_CGROUP_PIDS.

Code: Select all

$ diff -u .config-old .config
--- .config-old 2021-02-13 22:02:50.324354658 +0000
+++ .config     2021-02-13 22:06:07.108667034 +0000
@@ -20,7 +20,7 @@
 #
 CONFIG_INIT_ENV_ARG_LIMIT=32
 # CONFIG_COMPILE_TEST is not set
-CONFIG_LOCALVERSION=""
+CONFIG_LOCALVERSION="cgroup_pids"
 # CONFIG_LOCALVERSION_AUTO is not set
 CONFIG_BUILD_SALT=""
 CONFIG_HAVE_KERNEL_GZIP=y
@@ -144,7 +144,7 @@
 CONFIG_FAIR_GROUP_SCHED=y
 CONFIG_CFS_BANDWIDTH=y
 CONFIG_RT_GROUP_SCHED=y
-# CONFIG_CGROUP_PIDS is not set
+CONFIG_CGROUP_PIDS=y
 CONFIG_CGROUP_RDMA=y
 CONFIG_CGROUP_FREEZER=y
 CONFIG_CPUSETS=y

As you can see, the custom kernel is running, with the pids cgroup enabled, and Kubernetes is running via k3s-agent:

Code: Select all

$ uname -a
Linux cthulhu 5.4.87cgroup_pids+ #1 SMP PREEMPT Sat Feb 13 21:59:51 UTC 2021 armv7l armv7l armv7l GNU/Linux
$ cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  11      17      1
cpu     2       92      1
cpuacct 2       92      1
blkio   6       91      1
memory  7       110     1
devices 10      91      1
freezer 4       17      1
net_cls 5       17      1
perf_event      8       17      1
net_prio        5       17      1
pids    9       102     1
rdma    3       1       1
$ sudo systemctl status k3s-agent | head -20
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-02-13 23:44:17 UTC; 15h ago
       Docs: https://k3s.io
    Process: 3141 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 3142 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 3143 (k3s-agent)
      Tasks: 99
     Memory: 657.0M
     CGroup: /system.slice/k3s-agent.service
             ├─ 3143 /usr/local/bin/k3s agent
             ├─ 3168 containerd
             ├─ 3438 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 30753d64c48c0451136cb4c2e9a188dfedd78a09cf3991fd20f0321dcadb9f18 -address /run/k3s/containerd/containerd.sock
             ├─ 3458 /pause
             ├─ 3512 /bin/sh /usr/bin/entry
             ├─ 3550 /bin/sh /usr/bin/entry
             ├─ 5903 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 2d66c74699861b344d33696f8939e232da894846e9aea4482635721c5a22750a -address /run/k3s/containerd/containerd.sock
             ├─ 5929 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7592977eac6ff880ed202875910d9f87976c42b87ea978324517bd5301eee4f6 -address /run/k3s/containerd/containerd.sock
             ├─ 5950 /pause
             ├─ 5958 /pause

While recompiling the kernel is an annoyance (Hardkernel, please enable this option in your stock kernels!) this did fix the problem.

[obscurerichard]

Changed.
https://github.com/hardkernel/linux/com ... 41e213e2d4

Today's Kernel 5.4.98-219 update package has it by default.

[obscurerichard]

I'd advise that folks who want to use Kubernetes and Ubuntu 20.04 on their XU4-compatible hardware use the latest stock kernel from ODROID now that this kernel change is present. I did this to update my system that had been running the custom kernel.

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo init 6

After that the kernel booted up fine and the k3s service worked fine too:

Code: Select all

$ uname -a
Linux cthulhu 5.4.98-219 #1 SMP PREEMPT Mon Feb 15 21:15:05 EST 2021 armv7l armv7l armv7l GNU/Linux
rbulling@cthulhu:~$ sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-02-20 17:14:38 UTC; 34min ago
       Docs: https://k3s.io
    Process: 873 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 894 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 911 (k3s-server)
      Tasks: 159
     Memory: 553.7M
     CGroup: /system.slice/k3s.service
             ├─ 911 /usr/local/bin/k3s server
             ├─1861 containerd
             ├─2696 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2697 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2698 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2699 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2782 /pause
             ├─2792 /pause
             ├─2798 /pause
             ├─2799 /pause
             ├─2885 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2905 /pause
             ├─2988 /metrics-server
             ├─2989 /traefik --configfile=/config/traefik.toml
             ├─3001 /coredns -conf /etc/coredns/Corefile
             ├─3009 local-path-provisioner start --config /etc/config/config.json
             ├─3070 /bin/sh /usr/bin/entry
             └─3128 /bin/sh /usr/bin/entry

It's so nice when a vendor listens to user feedback! Thank you, Hardkernel!

[obscurerichard]

I also wanted to see if I could also get the pairing of Ubuntu 18.04 and a 4.14.180+ locally-compiled kernel working on my HC1 with CGROUP_PIDS=y and k3s. I had an Ubuntu 18.04 SD card handy so I followed the steps https://wiki.odroid.com/odroid-xu4/os_i ... 4/20190929 on the ODROID wiki for Release Note of Ubuntu MATE 18.04.3 LTS (v4.2) to recompile that kernel, after setting CGROUP_PIDS=y but the kernel did not complete booting, I monitored it over the UART console and got:

Code: Select all

Kernel image @ 0x40008000 [ 0x000000 - 0x56f980 ]
## Loading init Ramdisk from Legacy Image at 42000000 ...
   Image Name:   uInitrd
   Image Type:   ARM Linux RAMDisk Image (uncompressed)
   Data Size:    11168113 Bytes = 10.7 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 44000000
   Booting using the fdt blob at 0x44000000
   Using Device Tree in place at 44000000, end 4401281d

Starting kernel ...

Then it hung. Given these experiences, going with Ubuntu 20.04 and the latest kernel is definitely the way to go to get Kubernetes and k3s running on XU-4 / HC1 hardware.

I'm not going to spend more effort getting this older kernel on Ubutntu 18.04 working, but I wanted to share my experience in case it helped other folks.