Oomkiller v0.2

Why oom-killer kills?!

OOM-killer functionality is controlled by function oom_kill.c[2] and if you check source code it grew from 700+ lines of code to 1100+ lines from 2.6 till 5.4 kernel version. Sadly I was able to find only on this resource[3] and this information dates 2009 year.

In 2 words if Linux free memory is depleting, oom-killer sends SIGKILL to the process with maximum badness score. Badness is defined by oom_kill.c and can be checked in /proc/$pid/oom_score file for each process ID. For example you can see OOM score for top 20 processes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19


$ while read -r pid comm; do [ -f /proc/$pid/oom_score ] && [ $(cat /proc/$pid/oom_score) != 0 ] && printf '%d\t%d\t\t%d\t%s\n' "$pid" "$(cat /proc/$pid/oom_score)" "$(cat /proc/$pid/oom_score_adj)" "$comm"; done < <(ps -e -o pid= -o comm=) | sort -k 2nr
2642	318		300	atom
3071	305		300	atom
3270	302		300	hugo
3281	302		300	hugo
3328	302		300	hugo
3345	302		300	hugo
25764	300		300	QtWebEngineProc
3269	300		300	sh
3280	300		300	sh
3327	300		300	sh
3344	300		300	sh
2087	67		0	Web Content
651	52		0	Plex Media Serv
1404	34		0	firefox
693	32		0	Plex Script Hos
5168	30		0	Web Content
2026	29		0	gnome-shell
1599	22		0	WebExtensions

Top process details:

1
2
3
4
5
6


$ cat /proc/2642/oom_score
318
$ cat /proc/2642/oom_score_adj
300
$ cat /proc/2642/oom_adj
5

What can we do if we don’t want oom-killer in Linux to kill our process? For example we are running programs A and B in userspace of asterisk user. Both programs consume almost identical amount of memory but if you can tolerate process B downtime, process A is business critical. And in case of oom-killer you can’t be sure which process is killed if memory consumption is grown rapidly.
We can raise priority of process A by setting negative oom_score_adj value:

1

echo -1000 > /proc/<pid>/oom_score_adj

Commands: Find pagesize of Linux OS

1

getconf PAGESIZE

Find memory size used by the process:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


$ cat /proc/2611/statm
204545 11860 4499 2111 0 9398 0

/proc/[pid]/statm Provides information about memory usage, measured in pages. The columns are:

                  size       (1) total program size
                             (same as VmSize in /proc/[pid]/status)
                  resident   (2) resident set size
                             (same as VmRSS in /proc/[pid]/status)
                  shared     (3) number of resident shared pages (i.e., backed by a file)
                             (same as RssFile+RssShmem in /proc/[pid]/status)
                  text       (4) text (code)
                  lib        (5) library (unused since Linux 2.6; always 0)
                  data       (6) data + stack
                  dt         (7) dirty pages (unused since Linux 2.6; always 0)

Links:

https://www.percona.com/blog/2019/08/02/out-of-memory-killer-or-savior/
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/mm/oom_kill.c?id=HEAD&h=master#n172 (fix unreclaimble typo)
old kernel http://catap.ru/blog/2009/05/03/about-memory-oom-killer/
about Linux memory management https://www.tldp.org/LDP/tlk/mm/memory.html
https://sysctl-explorer.net/vm/
https://sysctl-explorer.net/vm/oom_kill_allocating_task/
https://sysctl-explorer.net/vm/panic_on_oom/