How To Create OOM Killer Exceptions in Linux
When a linux machine runs extremely low on memory the kernel begins deciding which processes it thinks are least important and starts killing them off in order to keep the processes it thinks are more important running. Unfortunately, the kernel OOM (out of memory) killer rarely makes the right decision, and your system is usually hosed once OOM killer rears it’s ugly head. Luckily, you can tell the kernel to never OOM kill a given pid. So, If you’re running on a low memory system, and want to ensure that important processes (like sshd for instance) are never killed, these options may be of use to you.
Disabling OOM is done on a process by process basis, so you’ll need to know the PID of the running process that you want to protect. This is far from ideal, as process IDs can change frequently, but we can script around it.
As documented by http://linux-mm.org/OOM_Killer: “Any particular process leader may be immunized against the oom killer if the value of its /proc/$pid/oom_adj is set to the constant OOM_DISABLE (currently defined as -17).”
This means we can disable OOM on an individual process, if we know its PID, using the command below:
# OOM_DISABLE on $PID echo -17 > /proc/$PID/oom_adj
Using pgrep we can run this knowing only the name of the process. For example, let’s ensure that the ssh listener doesn’t get OOM killed:
pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done
Here we used pgrep to search for the full command line (-f) matching “/usr/sbin/sshd” and then echo -17 into the procfs entry for each matching pid.
In order to automate this, you could run a cron regularly to update the oom_adj entry. This is a simple way to ensure that sshd is excluded from OOM killer after restarting the daemon or the server.
#/etc/cron.d/oom_disable */1 * * * * root pgrep -f "/usr/sbin/sshd" | while read PID; do echo -17 > /proc/$PID/oom_adj; done
The above job will run every minute, updating the oom_adj of the current process matching /usr/sbin/sshd. Of course this could be extended to include any other processes you wish to exclude from OOM killer.