OOM Oracle

A Kubernetes node agent that explains OOM kills at the level the control plane cannot. Kubernetes reports OOMKilled and exit code 137, then stops. This attaches an eBPF kprobe to oom_kill_process and samples cgroup memory continuously, so a report names which process died, what it held at the moment the kernel chose it, and the memory curve on the way there. Falls back to cgroup polling where BTF is absent, and ships a cosign-signed image and OCI Helm chart.

Origin: Open Source
Stack: GoeBPFCO-REcgroupsclient-goHelm
Release: v0.1.2
Source: github.com/ethan-kane-ops/k8s-pod-oom-oracle
Licence: Apache-2.0
Docs: ethan-kane-ops.github.io/k8s-pod-oom-oracle/

Recording 96×34 · 1:01

# OOM Oracle traced the same kill in the kernel.

$ k -n oom-oracle port-forward daemonset/oom-oracle 9090:9090 >/dev/null 2>&1 &

$ ./bin/oom-oracle inspect oom-gradual-leak
POD: oom-gradual-leak (namespace: default)
CONTAINER: leaker (image: docker.io/library/alpine:3.20)
QOS: Burstable

DIAGNOSIS: OOMKilled (2026-09-06 18:32:30 UTC)
  Limit:        128.0MiB
  Peak usage:   128.0MiB
  Kill count:   2
  Detected by:  ebpf
  Growth rate:  2.3MiB/s (fit R²=0.91 over 31s)

MEMORY TRAJECTORY (last 31s):
  18:31:58:    1.6MiB / 128.0MiB  [░░░░░░░░░░░░░░░░░░]   1%
  18:32:02:    5.8MiB / 128.0MiB  [░░░░░░░░░░░░░░░░░░]   5%
  18:32:06:   15.8MiB / 128.0MiB  [██░░░░░░░░░░░░░░░░]  12%
  18:32:11:   26.8MiB / 128.0MiB  [███░░░░░░░░░░░░░░░]  21%
  18:32:15:   36.1MiB / 128.0MiB  [█████░░░░░░░░░░░░░]  28%
  18:32:20:   45.9MiB / 128.0MiB  [██████░░░░░░░░░░░░]  36%
  18:32:23:   90.5MiB / 128.0MiB  [████████████░░░░░░]  71%
  18:32:29:   64.9MiB / 128.0MiB  [█████████░░░░░░░░░]  51%

VICTIM PROCESS:
  PID:             1528736 (in container: 1)
  Command:         sh
  Exit code:       137 (OOM)
  Memory at death: 127.6MiB
  Confidence:      traced in the kernel at the moment of the kill.

# Named the process, and showed the curve that got it there.

Recorded against a local kind cluster. The kernel does the kill; every figure was measured during the take.

The gap this fills

Kubernetes tells you OOMKilled and exit code 137, and then it stops. It cannot tell you which of the five processes in that container took the memory, what it held at the moment the kernel picked it, or whether usage climbed for an hour or spiked in a second.

On a pool of workers that all exec the same binary, “which one” is the entire question. The answer exists, but it is on the node, in a kernel ring buffer that most people debugging a production incident have no route to.

The usual workarounds do not close it. Metrics scraped every fifteen or thirty seconds miss a spike entirely, and the last sample before a kill is the one sample that does not matter. kubectl describe reports the outcome, never the cause. By the time anyone has shelled onto the node, the process is gone and so is its memory.

How it works

Three pieces, none of which are useful on their own.

A kprobe on oom_kill_process. The kernel makes the choice, so the kernel is where the answer is. The probe fires at the moment of selection and captures the victim’s PID, command and RSS before the process is torn down. It is compiled CO-RE, so one binary loads across kernel versions without a compiler on the node.

Continuous cgroup sampling. A trajectory has to already exist when the kill lands, because there is no way to reconstruct one afterwards. The agent samples memory usage per cgroup on a tight interval and keeps a rolling window, so the report can show the curve and fit a growth rate to it rather than reporting a single number with no shape.

Cgroup path to pod name. A cgroup path is not an answer anyone can act on. An informer over the node’s pods resolves it back to a namespace, pod and container name, which is what turns kernel data into something you can put in an incident channel.

What comes out

POD: payment-api-6d5f78 (namespace: default)
CONTAINER: web-server (image: payment:v1.2.0)
QOS: Burstable

DIAGNOSIS: OOMKilled (2026-08-13 08:15:22 UTC)
  Limit:        512.0MiB
  Peak usage:   512.0MiB
  Kill count:   1
  Detected by:  ebpf
  Growth rate:  1.0MiB/s (fit R2=0.97 over 1m0s)

MEMORY TRAJECTORY (last 1m0s):
  08:14:22:  412.0MiB / 512.0MiB  [##############....]  80%
  08:14:37:  460.0MiB / 512.0MiB  [################..]  90%  (stall 12%)
  08:14:52:  498.0MiB / 512.0MiB  [#################.]  97%  (stall 24%)
  08:15:07:  507.0MiB / 512.0MiB  [#################.]  99%  (stall 36%)
  08:15:22:  512.0MiB / 512.0MiB  [##################] 100%  (stall 48%)

VICTIM PROCESS:
  PID:             28145 (in container: 17)
  Command:         node ./dist/garbage-collector.js
  Exit code:       137 (OOM)
  Memory at death: 114.0MiB
  Confidence:      traced in the kernel at the moment of the kill.

PROCESSES IN CONTAINER AFTER THE KILL:
  memory.oom.group=0: the kernel killed only the process it
  selected, so these were still running after it died.
  1. node ./dist/server.js (PID 28102) - 390.0MiB
  2. node ./dist/worker.js (PID 28160) - 8.0MiB

The last block is the one that changes conclusions most often. The container was not over its limit because of the process that died. The process that died held 114MiB; the one still running held 390MiB. Raising the limit on this workload buys time and fixes nothing.

Two detectors, one of them a fallback

eBPF is the accurate path and it is not always available. BTF may be absent, the kernel may be too old, or the security posture may not allow the probe to attach. Rather than fail to start, the agent falls back to polling cgroup counters and memory.events.

The fallback is honestly worse and the report says so: every report names the detector that produced it, so nobody reads a polled approximation as a traced fact. Polling can tell you a kill happened and roughly when. It cannot tell you which process the kernel chose.

Running it

helm install oom-oracle oci://ghcr.io/ethan-kane-ops/charts/oom-oracle \
  --namespace oom-oracle --create-namespace
kubectl label namespace oom-oracle pod-security.kubernetes.io/enforce=privileged

The Pod Security label is not optional. hostPID, hostPath volumes and non-default capabilities are each outside baseline, so without it admission rejects the pods and the DaemonSet reports no event explaining why. That is a bad enough first-run experience that it is called out in the quickstart rather than left in the troubleshooting page.

Reports come out of a CLI against the agent’s HTTP API: oom-oracle watch for a live terminal dashboard, oom-oracle inspect for every recorded kill, and -o json for anything that needs to be piped.

Supply chain

This is a privileged node agent, which raises the bar on what it is reasonable to ask someone to install. The image and the OCI Helm chart are signed with cosign in keyless mode, so the signing identity is the GitHub Actions OIDC token bound to the release workflow and there are no keys to distribute. The chart is published to Artifact Hub, and the repository carries an OpenSSF Scorecard badge.

The eBPF source is GPL-2.0 while the rest of the project is Apache-2.0. That is not a licensing accident: the program calls GPL-only BPF helpers, and the kernel refuses to load one that declares anything else. It is compiled to a BPF object and loaded into the kernel, not linked into the Go binary.

Status

Released and installable, pre-1.0. Both detectors work, cgroup v1 and v2 are handled, and an end-to-end suite runs on kind in CI. The HTTP API and report JSON can still change shape, so breaking changes are written by hand in the changelog with the migration each one needs. A commit subject cannot tell you which JSON field was renamed.