Exit code 137 is not an answer

A pod restarts. You run kubectl describe, and the last state says OOMKilled with exit code 137. Every guide agrees on what to do next: raise the memory limit and move on.

That works often enough that most people never ask what the two fields actually mean. Neither one tells you which process died, what it was holding when it died, or whether the memory went up over an hour or in a second. The kernel knew all three at the moment it acted. It wrote them to the node’s ring buffer and then dropped them.

What the API gives you

The whole record is two fields:

kubectl get pod api-7d9f -o jsonpath='{.status.containerStatuses[0].lastState.terminated}'

{ "exitCode": 137, "reason": "OOMKilled", "startedAt": "...", "finishedAt": "..." }

reason is real information. The container runtime sets it because it watched the container’s cgroup and saw an OOM event fire there, not because it inferred anything from the exit code. exitCode is 128 plus the signal number, and SIGKILL is 9.

That is the entire diagnosis. There is no field for the process, none for its resident set, and none for anything before the last instant.

137 is a signal number, not a diagnosis

Exit code 137 means the process was killed by SIGKILL. It says nothing about who sent it or why.

A liveness probe that fails past its threshold ends with SIGKILL after the termination grace period. So does kubectl delete --grace-period=0. So does a node draining under pressure, and so does any sidecar or supervisor that decides to kill its child. Every one of them produces 137.

The pairing of 137 with reason: OOMKilled is what makes it an OOM, and that reason comes from the cgroup, not the exit status. Reading 137 on its own as “out of memory” is a habit that works most of the time and is wrong the rest of it, usually on the incident where being wrong is expensive.

A container is a cgroup, not a process

Kubernetes accounts for memory per container. The kernel does not kill containers. It kills processes.

A container is a cgroup, and a cgroup holds however many processes the image starts: an init shim, a supervisor, a worker pool, a sidecar helper, whatever the entrypoint script forked. When the cgroup reaches memory.max and reclaim cannot bring it back under, the kernel runs the OOM killer scoped to that cgroup and picks one task out of it.

Which one is a decision with a specific algorithm behind it, and the answer is not “the one that allocated last” or “the one you were thinking of”.

How the kernel picks

out_of_memory() calls select_bad_process(), which scores every candidate task with oom_badness() and keeps the highest. The score is this, in full:

/* mm/oom_kill.c */
points = get_mm_rss_sum(p->mm) + get_mm_counter_sum(p->mm, MM_SWAPENTS) +
	mm_pgtables_bytes(p->mm) / PAGE_SIZE;

/* Normalize to oom_score_adj units */
adj *= totalpages / 1000;
points += adj;

Three terms, all in pages: resident set, swapped-out pages, and the memory the task’s own page tables occupy. Then a bias from oom_score_adj, scaled so that one unit is one thousandth of the memory available in that context.

Read what is absent. There is no term for how fast the task grew, none for how long it has been running, and none for whether it had just allocated or had been flat for a week. Page cache is not attributed to anyone. A task that mapped a large file and touched all of it looks the same as one that malloc’d the same amount, because both show up as resident pages.

The score is a photograph of size at one instant. It is a reasonable way to free the most memory with one kill, and it is not a claim about blame. The kernel is not trying to identify a leak. It is trying to survive.

Two tasks are exempt outright. oom_score_adj of -1000 returns LONG_MIN and makes a task unkillable, and so does a task already reaped or mid-vfork.

What Kubernetes writes into oom_score_adj

The kubelet sets oom_score_adj on every container from its QoS class. This is the part of the mechanism Kubernetes actually controls, and it is a heuristic the kubelet is explicit about:

// pkg/kubelet/qos/policy.go, reduced to the path most pods take
switch v1qos.GetPodQOS(pod) {
case v1.PodQOSGuaranteed:
	return guaranteedOOMScoreAdj // -997
case v1.PodQOSBestEffort:
	return besteffortOOMScoreAdj // 1000
}

oomScoreAdjust := 1000 - (1000*containerMemReq)/memoryCapacity
if int(oomScoreAdjust) < (1000 + guaranteedOOMScoreAdj) {
	return 1000 + guaranteedOOMScoreAdj // floor of 3
}
if int(oomScoreAdjust) == besteffortOOMScoreAdj {
	return int(oomScoreAdjust - 1) // 999, so Burstable still beats BestEffort
}
return int(oomScoreAdjust)

Guaranteed lands at -997. BestEffort lands at 1000. Burstable is scaled by how much of the node the container requested, so a pod requesting a tenth of a node gets 900, and one requesting almost nothing gets close to 999.

Note what that means during a node-level OOM: the bias is worth adj * totalpages / 1000, so on a 64 GiB node a Burstable container at 964 is carrying roughly 61 GiB of added score. It will be chosen ahead of a Guaranteed container that is genuinely much larger. That is the intent. It is also why “the biggest process died” is often false at the node level and usually true inside a single container, where every task shares the same oom_score_adj and the bias cancels out.

The report that has the answer

When the kernel kills, it dumps everything it used to decide. On a memcg OOM, this lands in the node’s kernel ring buffer:

oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-a3d2.scope,mems_allowed=0,oom_memcg=/kubepods/burstable/pod9f1c/a3d2,task_memcg=/kubepods/burstable/pod9f1c/a3d2,task=python3,pid=41827,uid=0
Memory cgroup out of memory: Killed process 41827 (python3) total-vm:2891244kB, anon-rss:1994104kB, file-rss:15872kB, shmem-rss:0kB, UID:0 pgtables:4188kB oom_score_adj:964

And before it, the table of every candidate it scored:

[  pid  ]   uid  tgid total_vm      rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[  41791]     0 41791    18942      842      117      725         0        73728        0           964 tini
[  41827]     0 41827   722811   502494   498526     3968         0      1071104        0           964 python3
[  41902]     0 41902    39217     3021     1544     1477         0       159744        0           964 gunicorn

That is the answer to the question you actually had. The process was python3, not the gunicorn parent you would have guessed from the deployment. It held 1.9 GiB anonymous, almost none of it file-backed, so this was heap and not a mapped file. Its page tables alone were a megabyte, which tells you the address space was fragmented across a lot of mappings. The other two tasks in the container were nowhere near the limit.

None of that reaches the API server. It is a line in dmesg on one node, in a fixed-size ring buffer, on a machine that may be gone by the time anyone asks. The kubelet does not read it, the runtime does not forward it, and the pod’s lastState has nowhere to put it.

The units are worth knowing if you ever read one of these directly. Columns in the task table are pages; fields in the kill message are kilobytes. The two agree: 498526 anonymous pages at 4 KiB is 1994104 kB.

The kill that leaves no trace

There is a worse case than a lossy diagnosis, and it is a kill you never hear about at all.

A memcg OOM kills the victim’s thread group. If the victim is not the container’s PID 1, PID 1 keeps running, the container never exits, and Kubernetes has nothing to report. No restart, no OOMKilled, no event. The application simply loses a worker and carries on with less capacity than it thinks it has.

Whether you are exposed to this depends on the cgroup version and one kubelet setting:

• cgroup v1: always exposed. One task dies, the container survives.
• cgroup v2, kubelet default: the kubelet sets memory.oom.group=1 on container cgroups, so the whole container is killed together and you do get the restart and the reason.
• cgroup v2 with singleProcessOOMKill: true: back to v1 behaviour, deliberately.

Even in the good case, group killing tells you the container died. It still does not tell you which task’s allocation crossed the limit, because by the time you look, everything in the cgroup is dead.

The cgroup counters are the one thing that survives, and they are worth watching precisely because they do not depend on the container dying:

cat /sys/fs/cgroup/kubepods/burstable/pod9f1c/a3d2/memory.events

low 0
high 0
max 3184
oom 2
oom_kill 2
oom_group_kill 1

oom_kill counts processes killed by any OOM killer in that cgroup. If it is climbing while the pod’s restart count is flat, you are in the silent case, and nothing in kubectl is going to say so. max counts how many times the cgroup hit its limit and had to reclaim, which is the pressure that precedes all of this and is usually visible long before anything dies.

Why the metrics do not close the gap

The obvious answer is to look at the memory graph. It does not work, for two reasons that compound.

The first is granularity. container_memory_working_set_bytes is a per-container figure: cAdvisor takes the cgroup’s usage and subtracts inactive file cache.

// cadvisor, lib/container/libcontainer/handler.go
workingSet := ret.Memory.Usage
if v, ok := s.MemoryStats.Stats[inactiveFileKeyName]; ok {
	ret.Memory.TotalInactiveFile = v
	if workingSet < v {
		workingSet = 0
	} else {
		workingSet -= v
	}
}

One number for the whole cgroup. It cannot separate three processes because the cgroup does not track them separately.

The second is resolution. cAdvisor’s housekeeping runs on the order of ten seconds and a typical Prometheus scrape is fifteen to sixty. A runaway allocation that crosses the limit can start and finish inside one interval, and the graph then shows a sawtooth: a normal value, a gap, a normal value after the restart. The peak that mattered was never sampled. The kill looks like it came out of nothing.

You end up reasoning about a spike from the two points on either side of it, which is how “raise the limit and see” became the standard advice. It is not that the advice is bad. It is that nothing available makes a better one possible.

Where that leaves you

The information exists. It is complete, it is accurate, and it is produced at exactly the right moment. It is thrown away because nothing on the path from the kernel to the API server was built to carry it.

Closing that gap means being on the node when it happens, which is a node agent doing two things:

1. Attribution at the moment of the kill. A kprobe on oom_kill_process fires with the victim task in hand, so the process, its RSS breakdown, its oom_score_adj and its cgroup are all readable before anything is torn down. That is the dmesg report, captured rather than logged.
2. Trajectory before it. Sampling cgroup memory continuously, at a period short enough to see the approach, gives you the shape of the climb. An hour of slow growth and a two second spike are different failures with different fixes, and they are indistinguishable in the artefacts Kubernetes keeps.

Neither is exotic. Both are things the node already knows and nothing currently asks it for.

I wrote OOM Oracle to do exactly that, because I got tired of answering “which process” with “probably the big one”. The kernel has never been vague about this. We just stopped listening at the container boundary.