Kubernetes CGROUP PIDS

Post Reply
magicroomy
Posts: 15
Joined: Thu Aug 01, 2013 5:41 am
languages_spoken: english
ODROIDs: odroid-x, odroid-u3, odroid-c1, odroid-c2
Has thanked: 0
Been thanked: 1 time
Contact:

Kubernetes CGROUP PIDS

Post by magicroomy »

My basic goal is to teach my MC1 kubernetes.
I read countless tutorials and docs about the subject ( all are not current).
I always run against the following problem:
At some point in time the kubernetes guys at google decided that kubernetes requires a kernel feature called CGROUP PIDS.
In some forums I read that activating the CGROUP PIDS and recompiling the kernel leads to the system not booting anymore. I tried myself and can agree.
Does anybody know if there is progress made on this subject?
Perfect would be is somebody could tell me where to find a full sd card image containing a kernel with activated CGROUP PIDS.
I tried ubuntu with 4.14 kernel, I also tried armbian with 4.14 kernels or even 5.X kernels.. nothing worked.

igorpec
Posts: 992
Joined: Sat Dec 12, 2015 4:34 pm
languages_spoken: english,german,slovene
ODROIDs: XU4, HC1, C2, C1+
Has thanked: 59 times
Been thanked: 144 times
Contact:

Re: Kubernetes CGROUP PIDS

Post by igorpec »

> with 4.14 kernel

We tried to enable this on 4.14.y but it doesn't wants to work ...

> even 5.X kernels.. nothing worked.

There it looks its enabled ... its in source config https://github.com/armbian/build/blob/m ... onfig#L149 while this doesn't mean it was actually build with this support. Check config in /boot directory to see if this was enabled or not. If not, this needs to be deeply investigated.

> Perfect would be

Its some work behind this and that solution.
ARMBIAN
Linux for ARM development boards
What is Armbian?

User avatar
mad_ady
Posts: 9689
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 609 times
Been thanked: 721 times
Contact:

Re: Kubernetes CGROUP PIDS

Post by mad_ady »

How does it behave with cgroup pid active? Does the kernel boot?Does it gets stuck in userspace? Any last messages?

magicroomy
Posts: 15
Joined: Thu Aug 01, 2013 5:41 am
languages_spoken: english
ODROIDs: odroid-x, odroid-u3, odroid-c1, odroid-c2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by magicroomy »

I can't tell if the kernel does anything while it should be booting, since I do not have a UART serial. (about to order one)
The effect is that I had a system up and running and could connect via ssh terminal.
After the kernel update, I cannot reach the odroid anymore. It also does not respond to ping.

User avatar
mad_ady
Posts: 9689
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 609 times
Been thanked: 721 times
Contact:

Re: Kubernetes CGROUP PIDS

Post by mad_ady »

How about the blue led state? Is it blinking?

magicroomy
Posts: 15
Joined: Thu Aug 01, 2013 5:41 am
languages_spoken: english
ODROIDs: odroid-x, odroid-u3, odroid-c1, odroid-c2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by magicroomy »

Ok, here is what I did..
1. Download and flash image from here: https://wiki.odroid.com/odroid-xu4/os_i ... 10-minimal
2.Partially follow the instructions here : https://wiki.odroid.com/odroid-xu4/os_i ... 4/20190929
- I followed the instructions up to the point ---make odroidxu4_defconfig
- after I did this, I changed the .config and enabled CGROUP_PIDS by adding CGROUP_PIDS=y
- then I continued with make -j8 etc.
I copied the new kernel , installed the modules, all as described on the page mentioned under 2
When I finally say reboot. The odroid never comes back.
Blue led is constantly on.

User avatar
mad_ady
Posts: 9689
Joined: Wed Jul 15, 2015 5:00 pm
languages_spoken: english
ODROIDs: XU4, C1+, C2, C4, N1, N2, H2, Go, Go Advance
Location: Bucharest, Romania
Has thanked: 609 times
Been thanked: 721 times
Contact:

Re: Kubernetes CGROUP PIDS

Post by mad_ady »

It's hard to troubleshoot without a serial cable. Try enabling that option from within make menuconfig - maybe it has some dependencies.

magicroomy
Posts: 15
Joined: Thu Aug 01, 2013 5:41 am
languages_spoken: english
ODROIDs: odroid-x, odroid-u3, odroid-c1, odroid-c2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by magicroomy »

I tried the menuconfig way... same result. LED in constant blue. Odroid does not respond anymore to ssh or can even ping. It also does not appear listed in my router.

magicroomy
Posts: 15
Joined: Thu Aug 01, 2013 5:41 am
languages_spoken: english
ODROIDs: odroid-x, odroid-u3, odroid-c1, odroid-c2
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by magicroomy »

Well I solved my actual problem: get Kubernetes running. I followed the odroid magazine article about installing kubernetes on a n2.
It seems the most important step here is to tell docker not to use CGROUPS but systemd as cgroup driver.

/etc/docker/daemon.json

Code: Select all

{
"exec-opts": ["native.cgroupdriver=systemd"],
"log-driver": "json-file",
"log-opts": {
"max-size": "100m"
},
"storage-driver": "overlay2"
}
I assume this does not force kubernetes to struggle with CGroup PIDS.
The issue of this topic actually still exists. The kernel as described above seems to have a problem with CGROUP PIDS.
Nevertheless, from my point of view, this topic can be closed.
Thank you for your help.
These users thanked the author magicroomy for the post:
odroid (Thu Jan 30, 2020 10:17 am)

obscurerichard
Posts: 3
Joined: Sun Feb 14, 2021 7:16 am
languages_spoken: english
ODROIDs: HC-1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by obscurerichard »

I too tried to get Kubernetes running on my ODROID HC1 running Ubuntu 20.04 and ran into the same issue with the stock vendor kernel.

I was trying to use https://k3s.io/ to get Kubernetes set up, using the HC1 as an agent.

Whenever the k3s-agent started, it would throw fatal errors related to PID limits (cannot set feature gate SupportPodPidsLimit to false), followed by a long go stack trace. It turns out that the CGROUP_PIDS feature is still turned off in the default kernel. This kernel is using v1 cgroups, but without the PIDS feature enabled Kubernetes fails to start up:

Code: Select all

Feb 12 20:20:00 host.example.com k3s[16364]: F0212 20:20:00.792512   16364 server.go:181] cannot set feature gate SupportPodPidsLimit to false, feature is locked to true
Feb 12 20:20:00 host.example.com k3s[16364]: goroutine 320 [running]:
Feb 12 20:20:00 host.example.com k3s[16364]: github.com/rancher/k3s/vendor/k8s.io/klog/v2.stacks(0x5914a01, 0x0, 0x7d, 0xcf)
Feb 12 20:20:00 host.example.com k3s[16364]:         /go/src/github.com/rancher/k3s/vendor/k8s.io/klog/v2/klog.go:1026 +0x94
To work around this error I was able to follow the kernel rebuilding and installation instructions at https://wiki.odroid.com/odroid-xu4/os_i ... uild_guide and the kernel rebuild and reboot worked flawlessly.

The key change was to make sure that this was present in the kernel .config file:

Code: Select all

CGROUP_PIDS=y
The only 2 changes I made in the .config file were CONFIG_LOCALVERSION and CONFIG_CGROUP_PIDS.

Code: Select all

$ diff -u .config-old .config
--- .config-old 2021-02-13 22:02:50.324354658 +0000
+++ .config     2021-02-13 22:06:07.108667034 +0000
@@ -20,7 +20,7 @@
 #
 CONFIG_INIT_ENV_ARG_LIMIT=32
 # CONFIG_COMPILE_TEST is not set
-CONFIG_LOCALVERSION=""
+CONFIG_LOCALVERSION="cgroup_pids"
 # CONFIG_LOCALVERSION_AUTO is not set
 CONFIG_BUILD_SALT=""
 CONFIG_HAVE_KERNEL_GZIP=y
@@ -144,7 +144,7 @@
 CONFIG_FAIR_GROUP_SCHED=y
 CONFIG_CFS_BANDWIDTH=y
 CONFIG_RT_GROUP_SCHED=y
-# CONFIG_CGROUP_PIDS is not set
+CONFIG_CGROUP_PIDS=y
 CONFIG_CGROUP_RDMA=y
 CONFIG_CGROUP_FREEZER=y
 CONFIG_CPUSETS=y
As you can see, the custom kernel is running, with the pids cgroup enabled, and Kubernetes is running via k3s-agent:

Code: Select all

$ uname -a
Linux cthulhu 5.4.87cgroup_pids+ #1 SMP PREEMPT Sat Feb 13 21:59:51 UTC 2021 armv7l armv7l armv7l GNU/Linux
$ cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  11      17      1
cpu     2       92      1
cpuacct 2       92      1
blkio   6       91      1
memory  7       110     1
devices 10      91      1
freezer 4       17      1
net_cls 5       17      1
perf_event      8       17      1
net_prio        5       17      1
pids    9       102     1
rdma    3       1       1
$ sudo systemctl status k3s-agent | head -20
● k3s-agent.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s-agent.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-02-13 23:44:17 UTC; 15h ago
       Docs: https://k3s.io
    Process: 3141 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 3142 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 3143 (k3s-agent)
      Tasks: 99
     Memory: 657.0M
     CGroup: /system.slice/k3s-agent.service
             ├─ 3143 /usr/local/bin/k3s agent
             ├─ 3168 containerd
             ├─ 3438 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 30753d64c48c0451136cb4c2e9a188dfedd78a09cf3991fd20f0321dcadb9f18 -address /run/k3s/containerd/containerd.sock
             ├─ 3458 /pause
             ├─ 3512 /bin/sh /usr/bin/entry
             ├─ 3550 /bin/sh /usr/bin/entry
             ├─ 5903 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 2d66c74699861b344d33696f8939e232da894846e9aea4482635721c5a22750a -address /run/k3s/containerd/containerd.sock
             ├─ 5929 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/containerd-shim-runc-v2 -namespace k8s.io -id 7592977eac6ff880ed202875910d9f87976c42b87ea978324517bd5301eee4f6 -address /run/k3s/containerd/containerd.sock
             ├─ 5950 /pause
             ├─ 5958 /pause
While recompiling the kernel is an annoyance (Hardkernel, please enable this option in your stock kernels!) this did fix the problem.

User avatar
odroid
Site Admin
Posts: 38033
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1999 times
Been thanked: 1206 times
Contact:

Re: Kubernetes CGROUP PIDS

Post by odroid »

Changed.
https://github.com/hardkernel/linux/com ... 41e213e2d4

Today's Kernel 5.4.98-219 update package has it by default.
These users thanked the author odroid for the post:
obscurerichard (Sun Feb 21, 2021 3:03 am)

obscurerichard
Posts: 3
Joined: Sun Feb 14, 2021 7:16 am
languages_spoken: english
ODROIDs: HC-1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by obscurerichard »

I'd advise that folks who want to use Kubernetes and Ubuntu 20.04 on their XU4-compatible hardware use the latest stock kernel from ODROID now that this kernel change is present. I did this to update my system that had been running the custom kernel.

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo init 6
After that the kernel booted up fine and the k3s service worked fine too:

Code: Select all

$ uname -a
Linux cthulhu 5.4.98-219 #1 SMP PREEMPT Mon Feb 15 21:15:05 EST 2021 armv7l armv7l armv7l GNU/Linux
rbulling@cthulhu:~$ sudo systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2021-02-20 17:14:38 UTC; 34min ago
       Docs: https://k3s.io
    Process: 873 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 894 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 911 (k3s-server)
      Tasks: 159
     Memory: 553.7M
     CGroup: /system.slice/k3s.service
             ├─ 911 /usr/local/bin/k3s server
             ├─1861 containerd
             ├─2696 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2697 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2698 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2699 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2782 /pause
             ├─2792 /pause
             ├─2798 /pause
             ├─2799 /pause
             ├─2885 /var/lib/rancher/k3s/data/4064121d88670f3bcf447161d4ff1e055230940eef1e84e7c4661d529a755fbd/bin/cont>
             ├─2905 /pause
             ├─2988 /metrics-server
             ├─2989 /traefik --configfile=/config/traefik.toml
             ├─3001 /coredns -conf /etc/coredns/Corefile
             ├─3009 local-path-provisioner start --config /etc/config/config.json
             ├─3070 /bin/sh /usr/bin/entry
             └─3128 /bin/sh /usr/bin/entry

It's so nice when a vendor listens to user feedback! Thank you, Hardkernel!
These users thanked the author obscurerichard for the post:
odroid (Mon Feb 22, 2021 10:01 am)

obscurerichard
Posts: 3
Joined: Sun Feb 14, 2021 7:16 am
languages_spoken: english
ODROIDs: HC-1
Has thanked: 1 time
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by obscurerichard »

I also wanted to see if I could also get the pairing of Ubuntu 18.04 and a 4.14.180+ locally-compiled kernel working on my HC1 with CGROUP_PIDS=y and k3s. I had an Ubuntu 18.04 SD card handy so I followed the steps https://wiki.odroid.com/odroid-xu4/os_i ... 4/20190929 on the ODROID wiki for Release Note of Ubuntu MATE 18.04.3 LTS (v4.2) to recompile that kernel, after setting CGROUP_PIDS=y but the kernel did not complete booting, I monitored it over the UART console and got:

Code: Select all

Kernel image @ 0x40008000 [ 0x000000 - 0x56f980 ]
## Loading init Ramdisk from Legacy Image at 42000000 ...
   Image Name:   uInitrd
   Image Type:   ARM Linux RAMDisk Image (uncompressed)
   Data Size:    11168113 Bytes = 10.7 MiB
   Load Address: 00000000
   Entry Point:  00000000
   Verifying Checksum ... OK
## Flattened Device Tree blob at 44000000
   Booting using the fdt blob at 0x44000000
   Using Device Tree in place at 44000000, end 4401281d

Starting kernel ...


Then it hung. Given these experiences, going with Ubuntu 20.04 and the latest kernel is definitely the way to go to get Kubernetes and k3s running on XU-4 / HC1 hardware.

I'm not going to spend more effort getting this older kernel on Ubutntu 18.04 working, but I wanted to share my experience in case it helped other folks.

jason.witty
Posts: 6
Joined: Wed Sep 01, 2021 6:43 am
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by jason.witty »

Was this change rolled back? it seems like this option is not enabled for me. Do I need to install a specific kernel or rebuild the kernel again to enable? I am using HC2.
--

root@odroid1:~# uname -r
5.4.142-227

--

root@odroid1:~# k3s check-config

...
Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: missing
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
...

jason.witty
Posts: 6
Joined: Wed Sep 01, 2021 6:43 am
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by jason.witty »

yes, it seems this is rolled back.

i cloned:

git clone --depth 1 https://github.com/hardkernel/linux -b odroid-5.4.y

and when i check :

~/linux/arch/arm/configs/odroidxu4_defconfig

i find on line 147 is reverted back to: # CONFIG_CGROUP_PIDS is not set

jason.witty
Posts: 6
Joined: Wed Sep 01, 2021 6:43 am
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by jason.witty »

Additionally branch odroid-5.4.y will not build

https://wiki.odroid.com/odroid-xu4/os_i ... uild_guide

--


root@odroid1:~# git clone --depth 1 https://github.com/hardkernel/linux -b odroid-5.4.y
Cloning into 'linux'...
remote: Enumerating objects: 72470, done.
remote: Counting objects: 100% (72470/72470), done.
remote: Compressing objects: 100% (67189/67189), done.
remote: Total 72470 (delta 5905), reused 26791 (delta 4137), pack-reused 0
Receiving objects: 100% (72470/72470), 207.01 MiB | 8.14 MiB/s, done.
Resolving deltas: 100% (5905/5905), done.
Updating files: 100% (68982/68982), done.
root@odroid1:~# cd linux/
root@odroid1:~/linux# make odroidxu4_defconfig
HOSTCC scripts/basic/fixdep
HOSTCC scripts/kconfig/conf.o
HOSTCC scripts/kconfig/confdata.o
HOSTCC scripts/kconfig/expr.o
LEX scripts/kconfig/lexer.lex.c
/bin/sh: 1: flex: not found
make[1]: *** [scripts/Makefile.host:9: scripts/kconfig/lexer.lex.c] Error 127
make: *** [Makefile:590: odroidxu4_defconfig] Error 2

User avatar
odroid
Site Admin
Posts: 38033
Joined: Fri Feb 22, 2013 11:14 pm
languages_spoken: English, Korean
ODROIDs: ODROID
Has thanked: 1999 times
Been thanked: 1206 times
Contact:

Re: Kubernetes CGROUP PIDS

Post by odroid »

We will fix that issue soon.

jason.witty
Posts: 6
Joined: Wed Sep 01, 2021 6:43 am
languages_spoken: english
Has thanked: 0
Been thanked: 1 time
Contact:

Re: Kubernetes CGROUP PIDS

Post by jason.witty »

thank you !

\(^o^)/

AreaScout
Posts: 1718
Joined: Sun Jul 07, 2013 3:05 am
languages_spoken: german, english
ODROIDs: X2, U3, XU3, C2, HiFi Shield, XU4, XU4Q,
N1, Go, VU5A, Show2, CloudShell2,
H2, N2, VU7A, VuShell, Go2, C4
Has thanked: 103 times
Been thanked: 317 times
Contact:

Re: Kubernetes CGROUP PIDS

Post by AreaScout »

    jason.witty wrote:
    Wed Sep 01, 2021 8:22 am
    /bin/sh: 1: flex: not found
    For the build error sudo apt install flex should do the trick

    jason.witty
    Posts: 6
    Joined: Wed Sep 01, 2021 6:43 am
    languages_spoken: english
    Has thanked: 0
    Been thanked: 1 time
    Contact:

    Re: Kubernetes CGROUP PIDS

    Post by jason.witty »

    thank you for this comment as well, this is the first time I have recompiled a kernel and didn't realize this was a dependency issue. I had to install a few more if hardkernel wants to add to documentation.

    $ sudo apt install flex bc libssl-dev bison

    After re-enabling cgroups and recompiling kernel, restarting and trying to log in 4 times before it accepted my password this issue is now resolved. I can see cgroups enabled and has joined k3 cluster and visible in rancher. I have 2 more to go and then longhorn setup, hopefully its smooth sailing from here on out. :)

    It would be good if hardkernel could update this property again so i dont need to recompile kernel on each update, also for the other people above who had this issue earlier in the year i am assuming their stuff will break if they ran a kernel update. thank you. :)

    --

    $ k3s check-config
    ...
    Optional Features:
    - CONFIG_USER_NS: enabled
    - CONFIG_SECCOMP: enabled
    - CONFIG_CGROUP_PIDS: enabled
    - CONFIG_BLK_CGROUP: enabled
    ...

    pi@raspberrypi-master:~ $ kubectl get nodes
    NAME STATUS ROLES AGE VERSION
    raspberrypi-worker-2 Ready <none> 21d v1.21.3+k3s1
    raspberrypi-master Ready control-plane,master 21d v1.21.3+k3s1
    odroid1 Ready <none> 7m56s v1.21.4+k3s1
    raspberrypi-worker-3 Ready <none> 21d v1.21.3+k3s1
    raspberrypi-worker-1 Ready <none> 21d v1.21.3+k3s1

    jason.witty
    Posts: 6
    Joined: Wed Sep 01, 2021 6:43 am
    languages_spoken: english
    Has thanked: 0
    Been thanked: 1 time
    Contact:

    Re: Kubernetes CGROUP PIDS

    Post by jason.witty »

    i can confirm this is fixed in 5.4.142-228
    These users thanked the author jason.witty for the post:
    odroid (Thu Sep 02, 2021 9:19 am)

    User avatar
    odroid
    Site Admin
    Posts: 38033
    Joined: Fri Feb 22, 2013 11:14 pm
    languages_spoken: English, Korean
    ODROIDs: ODROID
    Has thanked: 1999 times
    Been thanked: 1206 times
    Contact:

    Re: Kubernetes CGROUP PIDS

    Post by odroid »

    Thank you for the confirmation.
    https://github.com/hardkernel/linux/com ... roid-5.4.y

    I also updated the wiki page about the missing dependent packages.

    Post Reply

    Return to “Ubuntu”

    Who is online

    Users browsing this forum: No registered users and 4 guests