OOM Killer – How To Create OOM Exclusions in Linux

Tags: , , , , , , , , , , , , , ,


When a linux machine runs low on memory the kernel will begin killing processes to free up ram. This is called the OOM Killer. OOM stands for out of memory. Unfortunately, the Linux kernel OOM killer often kills important processes. On numerous occasions my system has become completely hosed once OOM killer rears it’s ugly head. Luckily, you can tell the kernel to never OOM kill certain processes by supplying a list of pid numbers. If you’re running a system with high memory pressure, and want to ensure that important processes (sshd for instance) are never killed, these options may be of use to you.

Telling the OOM killer to ignore a process

Disabling OOM killer is done on a process by process basis, so you’ll need to know the PID of the running process that you want to protect. This is far from ideal, as process IDs can change frequently, but we can script around it.

As documented by http://linux-mm.org/OOM_Killer: “Any particular process leader may be immunized against the oom killer if the value of its /proc/$pid/oom_adj is set to the constant OOM_DISABLE (currently defined as -17).”

This means we can disable OOM killer on an individual process, if we know its PID, using the command below:

# OOM_DISABLE on $PID
echo -17 > /proc/$PID/oom_adj

Using pgrep we can run this knowing only the name of the process. For example, let’s ensure that the ssh listener doesn’t get OOM killed:

pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done

Here we used pgrep to search for the full command line (-f) matching “/usr/sbin/sshd” and then echo -17 into the procfs entry for each matching pid.

In order to automate this, you could run a cron regularly to update the oom_adj entry. This is a simple way to ensure that sshd is excluded from OOM killer after restarting the daemon or the server.

#/etc/cron.d/oom_disable
*/1 * * * * root pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done

The above job will run every minute, updating the oom_adj of the current process matching /usr/sbin/sshd. Of course this could be extended to include any other processes you wish to exclude from OOM killer.

I recommend disabling OOM killer at the individual processes level rather than turning it off system-wide. Disabling OOM killer altogether will cause your system to kernel panic under heavy memory pressure. By excluding critical administrative processes you should at least be able to log in to troubleshoot high memory use.

7 Responses to “OOM Killer – How To Create OOM Exclusions in Linux”

  1. T'Saavik Says:

    oom_adj is deprecated, please use oom_score_adj instead.

    */1 * * * * root pgrep -f “/usr/sbin/sshd” | while read PID; do echo -1000 > /proc/$PID/oom_score_adj; done

    [Reply]

    Keith Reply:

    Thanks. I believe this was changed in 2.6.36, so good to know for folks running newer kernels.

    [Reply]

    Brian Reply:

    Would you know if oom_score_adj works properly on kernels earlier than 2.6.36? Or, is oom_adj a more reliable setting to use on older kernels?

    [Reply]

  2. Vaitheeswaran Says:

    Hey!

    I just tried out this. It doesn’t seem to work. I have set both the files (oom_adj and oom_score_adj ) to -17 value.

    But my process gets killed.

    Is there any other way to do this ?

    [Reply]

  3. daryl Says:

    https://access.redhat.com/solutions/1171183

    notes that for RHEL6+7, oom_score_adj should be -1000

    [Reply]

  4. Franklin Anderson de Oliveira Souza Says:

    Insert into /etc/init.d/postgresql-9.x (Centos 6 – Init)

    for processo in $(pgrep postmaster)
    do
    echo “-1000” > /proc/$processo/oom_score_adj
    done

    [Reply]

  5. luke Says:

    Hi Guys,
    I have CentOS Linux release 7.2.1511 (Core).
    Im starting script which reading complete text files where the biggest is 1.3kb. After reading 5 files it is killed by OOM.
    Is there anything specific to perl what could prevent it from beeing killed?
    I used all possible echos to oom adj files and no joy.
    I also try to disable oom completely in /etc/sysctl.conf and still not joy.
    If you have any hints please share
    Best Regards
    luke

    [Reply]

Join the Conversation