Kernel memory leak in ubuntu-20.04-3.10-minimal-odroid-c1-20200519

Post Reply
User avatar
tosiara
Posts: 31
Joined: Wed Sep 17, 2014 11:14 pm
languages_spoken: english
ODROIDs: U3, C0, C1
Has thanked: 0
Been thanked: 0
Contact:

Kernel memory leak in ubuntu-20.04-3.10-minimal-odroid-c1-20200519

Post by tosiara »

Hi,

I recently upgraded my C1 from ubuntu 18 to ubuntu 20 (ubuntu-20.04-3.10-minimal-odroid-c1-20200519).

After this upgrade the board goes offline in about a week. Before going offline the OOM killer begins killing processes which makes the board even more unstable. I tried to monitor the memory and there is a clear pattern:
Screenshot 2024-03-18 at 17.03.27.png
Screenshot 2024-03-18 at 17.03.27.png (130 KiB) Viewed 102 times
On this graph you can see OOM instability on March 6, then reboot, then on March 17 again OOM.

Code: Select all

Mar 16 23:30:01 odp kernel: cron invoked oom-killer: gfp_mask=0xd0, order=2, oom_score_adj=0
Mar 16 23:30:01 odp kernel: cron cpuset=/ mems_allowed=0
Mar 16 23:30:01 odp kernel: CPU: 2 PID: 19344 Comm: cron Not tainted 3.10.107-25 #2
Mar 16 23:30:01 odp kernel: [<c0014a80>] (unwind_backtrace+0x0/0xf4) from [<c0011a7c>] (show_stack+0x10/0x14)
Mar 16 23:30:01 odp kernel: [<c0011a7c>] (show_stack+0x10/0x14) from [<c06801b0>] (dump_header+0x80/0x1c0)
Mar 16 23:30:01 odp kernel: [<c06801b0>] (dump_header+0x80/0x1c0) from [<c00ae884>] (oom_kill_process+0x70/0x3a0)
Mar 16 23:30:02 odp kernel: [<c00ae884>] (oom_kill_process+0x70/0x3a0) from [<c00af038>] (out_of_memory+0x2a8/0x2fc)
Mar 16 23:30:02 odp kernel: [<c00af038>] (out_of_memory+0x2a8/0x2fc) from [<c00b2fc8>] (__alloc_pages_nodemask+0x84c/0xa34)
Mar 16 23:30:02 odp kernel: [<c00b2fc8>] (__alloc_pages_nodemask+0x84c/0xa34) from [<c00b31c0>] (__get_free_pages+0x10/0x24)
Mar 16 23:30:02 odp kernel: [<c00b31c0>] (__get_free_pages+0x10/0x24) from [<c001aa9c>] (pgd_alloc+0x14/0xf4)
Mar 16 23:30:02 odp kernel: [<c001aa9c>] (pgd_alloc+0x14/0xf4) from [<c002a56c>] (mm_init+0xac/0xe8)
Mar 16 23:30:02 odp kernel: [<c002a56c>] (mm_init+0xac/0xe8) from [<c002aca8>] (dup_mm+0x68/0x440)
Mar 16 23:30:02 odp kernel: [<c002aca8>] (dup_mm+0x68/0x440) from [<c002b8a0>] (copy_process+0x7d8/0x106c)
Mar 16 23:30:02 odp kernel: [<c002b8a0>] (copy_process+0x7d8/0x106c) from [<c002c1e8>] (do_fork+0x88/0x2c4)
Mar 16 23:30:02 odp kernel: [<c002c1e8>] (do_fork+0x88/0x2c4) from [<c000dc20>] (ret_fast_syscall+0x0/0x38)
Mar 16 23:30:02 odp kernel: Mem-info:
Mar 16 23:30:02 odp kernel: Normal per-cpu:
Mar 16 23:30:02 odp kernel: CPU    0: hi:  186, btch:  31 usd:  40
Mar 16 23:30:02 odp kernel: CPU    1: hi:  186, btch:  31 usd:  52
Mar 16 23:30:02 odp kernel: CPU    2: hi:  186, btch:  31 usd:   0
Mar 16 23:30:02 odp kernel: CPU    3: hi:  186, btch:  31 usd:   0
Mar 16 23:30:02 odp kernel: HighMem per-cpu:
Mar 16 23:30:02 odp kernel: CPU    0: hi:   90, btch:  15 usd:  39
Mar 16 23:30:02 odp kernel: CPU    1: hi:   90, btch:  15 usd:   0
Mar 16 23:30:02 odp kernel: CPU    2: hi:   90, btch:  15 usd:   0
Mar 16 23:30:02 odp kernel: CPU    3: hi:   90, btch:  15 usd:   0
Mar 16 23:30:02 odp kernel: active_anon:55105 inactive_anon:2036 isolated_anon:0
                             active_file:3376 inactive_file:3462 isolated_file:0
                             unevictable:0 dirty:0 writeback:0 unstable:0
                             free:17046 slab_reclaimable:2608 slab_unreclaimable:101513
                             mapped:5405 shmem:2344 pagetables:924 bounce:30
                             free_cma:741
Mar 16 23:30:02 odp kernel: Normal free:18120kB min:2972kB low:3712kB high:4456kB active_anon:92956kB inactive_anon:88kB active_file:120kB inactive_file:56kB unevictable:0k
B isolated(anon):0kB isolated(file):0kB present:628736kB managed:552180kB mlocked:0kB dirty:0kB writeback:0kB mapped:92kB shmem:112kB slab_reclaimable:10432kB slab_unreclai
mable:406052kB kernel_stack:1744kB pagetables:3696kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:358 all_unreclaimable? yes
Mar 16 23:30:02 odp kernel: [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
                            [aml_sdhc_data_thread] SDHC_ESTA=0x0
Mar 16 23:30:02 odp kernel: lowmem_reserve[]:
Mar 16 23:30:02 odp kernel:  0 1704 1704
Mar 16 23:30:02 odp kernel: HighMem free:50064kB min:212kB low:504kB high:796kB active_anon:127464kB inactive_anon:8056kB active_file:13384kB inactive_file:13792kB unevicta
ble:0kB isolated(anon):0kB isolated(file):0kB present:218112kB managed:218112kB mlocked:0kB dirty:0kB writeback:0kB mapped:21528kB shmem:9264kB slab_reclaimable:0kB slab_un
reclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:73720988kB free_cma:2964kB writeback_tmp:0kB pages_scanned:53 all_unreclaimable? no
Mar 16 23:30:02 odp kernel: lowmem_reserve[]: 0 0 0
Mar 16 23:30:02 odp kernel: Normal: 1640*4kB (EM) 1412*8kB (EM) 12*16kB (MR) 3*32kB (R) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18144kB
Mar 16 23:30:02 odp kernel: HighMem: 61*4kB (C) 120*8kB (MC) 2288*16kB (MC) 359*32kB (MC) 10*64kB (UMC) 1*128kB (C) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 50068kB
Mar 16 23:30:02 odp kernel: 9186 total pagecache pages
Mar 16 23:30:02 odp kernel: 0 pages in swap cache
Mar 16 23:30:02 odp kernel: Swap cache stats: add 0, delete 0, find 0/0
Mar 16 23:30:02 odp kernel: Free swap  = 0kB
Mar 16 23:30:02 odp kernel: Total swap = 0kB
Mar 16 23:30:02 odp kernel: 211712 pages of RAM
Mar 16 23:30:02 odp kernel: 19232 free pages
Mar 16 23:30:02 odp kernel: 5853 reserved pages
Mar 16 23:30:02 odp kernel: 101568 slab pages
Mar 16 23:30:02 odp kernel: 604231 pages shared
Mar 16 23:30:02 odp kernel: 0 pages swap cached
Mar 16 23:30:02 odp kernel: [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
Mar 16 23:30:02 odp kernel: [  217]     0   217    98673      642     183        0          -250 systemd-journal
Mar 16 23:30:02 odp kernel: [  266]     0   266     4261      267       7        0         -1000 systemd-udevd
Mar 16 23:30:02 odp kernel: [  354]   102   354     4666      969      11        0             0 systemd-resolve
Mar 16 23:30:02 odp kernel: [  355]   100   355     5523      240      10        0             0 systemd-timesyn
Mar 16 23:30:02 odp kernel: [  359]     0   359     1220       88       4        0             0 cron
Mar 16 23:30:02 odp kernel: [  360]   107   360     1657      310       6        0          -900 dbus-daemon
Mar 16 23:30:02 odp kernel: [  361]     0   361    16475      762      22        0             0 NetworkManager
Mar 16 23:30:02 odp kernel: [  367]     0   367     4889     1288      13        0             0 networkd-dispat
Mar 16 23:30:02 odp kernel: [  372]   106   372     6381      326       9        0             0 rsyslogd
Mar 16 23:30:02 odp kernel: [  376]     0   376     1719      220       5        0             0 smartd
Mar 16 23:30:02 odp kernel: [  377]     0   377     3305      427       9        0             0 systemd-logind
Mar 16 23:30:02 odp kernel: [  378]     0   378     2188      116       6        0             0 wpa_supplicant
Mar 16 23:30:02 odp kernel: [  448]   109   448     3293      808       8        0             0 snmpd
Mar 16 23:30:02 odp kernel: [  454]     0   454     7873     1209      14        0             0 unattended-upgr
Mar 16 23:30:02 odp kernel: [  460]     0   460     2353      176       6        0         -1000 sshd
Mar 16 23:30:02 odp kernel: [  464]     0   464     2353      218       6        0         -1000 sshd
Mar 16 23:30:02 odp kernel: [  479]     0   479    52628     2019      29        0             0 apache2
Mar 16 23:30:02 odp kernel: [  496]   110   496   134640    31599     154        0             0 mysqld
Mar 16 23:30:02 odp kernel: [  583]     0   583      910       28       4        0             0 agetty
Mar 16 23:30:02 odp kernel: [  595]     0   595      529       25       3        0             0 agetty
Mar 16 23:30:02 odp kernel: [ 1203]     0  1203     8606      131       8        0             0 master
Mar 16 23:30:02 odp kernel: [ 1205]   108  1205     8727      192       8        0             0 qmgr
Mar 16 23:30:02 odp kernel: [ 1218]   108  1218     9307      223       8        0             0 tlsmgr
Mar 16 23:30:02 odp kernel: [ 1527]     0  1527     2098      116       5        0             0 sudo
Mar 16 23:30:02 odp kernel: [ 1532]  1101  1532     4468      471      10        0             0 tgq
Mar 16 23:30:02 odp kernel: [ 1535]     0  1535      424       72       2        0             0 door_daemon
Mar 16 23:30:02 odp kernel: [25370]     0 25370    11490      328      13        0             0 polkitd
Mar 16 23:30:02 odp kernel: [31298]    33 31298    11735      984      14        0             0 apache2
Mar 16 23:30:02 odp kernel: [31302]    33 31302    52730     3501      30        0             0 apache2
Mar 16 23:30:02 odp kernel: [31303]    33 31303    52730     2514      30        0             0 apache2
Mar 16 23:30:02 odp kernel: [31304]    33 31304    52730     2514      30        0             0 apache2
Mar 16 23:30:02 odp kernel: [31305]    33 31305    52730     2513      30        0             0 apache2
Mar 16 23:30:02 odp kernel: [31306]    33 31306    52736     2540      30        0             0 apache2
Mar 16 23:30:02 odp kernel: [ 3842]    33  3842    52730     2544      30        0             0 apache2
Mar 16 23:30:02 odp kernel: [14314]   108 14314     8679      325       7        0             0 pickup
Mar 16 23:30:02 odp kernel: [19318]    33 19318    20357     3356      29        0             0 php
Mar 16 23:30:02 odp kernel: [19324]    33 19324    20357     3327      28        0             0 php
Mar 16 23:30:02 odp kernel: [19326]    33 19326    20357     3229      30        0             0 php
Mar 16 23:30:02 odp kernel: [19327]    33 19327      373       61       3        0             0 sh
Mar 16 23:30:02 odp kernel: [19328]    33 19328    10509      468      10        0             0 rrdtool
Mar 16 23:30:02 odp kernel: [19329]    33 19329      373       61       2        0             0 sh
Mar 16 23:30:02 odp kernel: [19330]    33 19330    19845     2985      27        0             0 php
Mar 16 23:30:02 odp kernel: [19331]    33 19331      373       61       3        0             0 sh
Mar 16 23:30:02 odp kernel: [19338]    33 19338     1273      431       4        0             0 perl
Mar 16 23:30:02 odp kernel: [19339]    33 19339      373       61       2        0             0 sh
Mar 16 23:30:02 odp kernel: [19340]    33 19340      373       61       3        0             0 sh
Mar 16 23:30:02 odp kernel: [19343]    33 19343      373       15       2        0             0 sh
Mar 16 23:30:02 odp kernel: [19344]     0 19344     1822      349       5        0             0 cron
Mar 16 23:30:02 odp kernel: Out of memory: Kill process 496 (mysqld) score 154 or sacrifice child
Mar 16 23:30:02 odp kernel: Killed process 496 (mysqld) total-vm:538560kB, anon-rss:121280kB, file-rss:5116kB
When inspecting the processes I can't see any process taking the whole memory, so I suspect it is kernel memory leak.

Any ideas except downgrading to 18.04?

User avatar
EarBiteR
Posts: 33
Joined: Wed Mar 18, 2015 12:13 pm
languages_spoken: english
ODROIDs: N2,C4's,C1's running CoreElec and Bullseye
Has thanked: 3 times
Been thanked: 2 times

Re: Kernel memory leak in ubuntu-20.04-3.10-minimal-odroid-c1-20200519

Post by EarBiteR »

If you had made a copy of your SD and then upgraded that you could go back to by swapping sd's then you could save a package list from 18 and 20 and compare them to maybe spot the differences. Or be able to capture a list of what is running on each version. That might help to see the differences and give you a clue as to whats eating the memory. These are old but im sure they are still useful:
https://askubuntu.com/questions/152716/ ... emory-leak
https://superuser.com/questions/925250/ ... ing-memory

Hope this helps :-)

Post Reply

Return to “Issues”

Who is online

Users browsing this forum: No registered users and 1 guest