Scale Sentry

# The operator now drives real traffic at the Service and watches
# the replica count climb. This is a live run, not a simulation.

$ kubectl get scalevalidation podinfo-default -w
NAME              PHASE     SLA   TRAFFIC   SCHEDULE   NEXT RUN   AGE
podinfo-default   Running                                         4s
podinfo-default   Succeeded   Pass   Pass                            95s

# The target scaled while the load was running.

$ kubectl get hpa podinfo && kubectl get pods -l app=podinfo
NAME      REFERENCE            TARGETS        MINPODS   MAXPODS   REPLICAS   AGE
podinfo   Deployment/podinfo   cpu: 50%/50%   1         5         2          11m
NAME                     READY   STATUS    RESTARTS   AGE
podinfo-66f9ddcd-dw7rh   1/1     Running   0          7m32s
podinfo-66f9ddcd-zht4f   1/1     Running   0          46s

# And the verdict, with the number it was judged on.

$ kubectl get scalevalidation podinfo-default -o custom-columns='SLA:.spec.sla,MEASURED:.status.scaleUpDuration,VERDICT:.status.slaStatus,REQUESTS:.status.totalRequests'
SLA     MEASURED          VERDICT   REQUESTS
1m30s   1m10.029137449s   Pass      135001

# Measured, not asserted. The HPA is now a tested claim.

# It also reports what it saw while it was in there.

$ kubectl get scalevalidation podinfo-default -o jsonpath='{range .status.diagnostics[*]}{.severity}{"  "}{.type}{"\n"}{end}'
Warning  DNSNdotsHigh
Warning  MissingPDB

The gap this fills

Nearly every service in a Kubernetes estate has an HPA, and almost none of them have ever been tested. The manifest exists, the metric is wired up, the replica range looks sensible, and the first time anyone finds out whether it reacts fast enough is during the traffic event it was written for.

The failure that actually costs you is subtler than “it did not scale”. It is cold-start traffic leakage: a new pod is declared Ready, the EndpointSlice updates, traffic arrives, and the pod is not yet able to serve it. The HPA looks correct in every dashboard. Replica count went up on schedule. Users got 502s anyway, and the errors are attributed to the wrong thing because the scaling graph looks fine.

Catching that needs load, cluster state, and request outcomes correlated on one timeline. Ad-hoc load tests give you two of the three.

How it works

A kubebuilder v4 controller built on controller-runtime, reconciling a ScaleValidation custom resource.

1. The controller resolves the CR’s targetRef and computes the load characteristics for the run.
2. It spawns two Jobs: a loadgen that drives traffic, and an observer that watches cluster state and scrapes cgroup metrics.
3. The loadgen drives HTTP/1.1, HTTP/2 or gRPC through the configured network path, whether that is ClusterIP, an Ingress, or a Gateway.
4. The observer correlates the loadgen’s request log against EndpointSlice updates and emits a structured verdict.
5. The verdict is written back to the CR’s status subresource: HPA latency, throttling, leakage diagnostics, and a pass, warn or fail band.

Every transition also emits a Kubernetes Event, so kubectl describe scalevalidation explains a failure without anyone going into controller logs.

Declaring a run

apiVersion: validation.scale-sentry.ek.co/v1beta1
kind: ScaleValidation
metadata:
  name: billing-service-validation
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: billing-service

  sla: 90s

  target:
    mode: AutoDiscoverProbe    # ServiceDefault | AutoDiscoverProbe | CustomPath
    port: 8080
    networkPath: Gateway       # ClusterIP | Ingress | Gateway
    host: billing.example.com  # optional Host override for edge routing
    protocol: HTTP2            # HTTP1 | HTTP2 | GRPC

  load:
    baseRps: 150
    warmupDuration: 15s
    profile:
      pattern: Ramp            # Constant | Poisson | Ramp | Step | Spike
      endRps: 600
      rampDuration: 2m

  disruption:
    injectPodDeletion: true
    minReplicasForChaos: 2
    triggerDelay: 30s

There is also an annotation bridge. Putting validation.scale-sentry.ek.co/enabled=true on an existing Deployment provisions a shadow ScaleValidation with no manifests to write, which is the difference between a tool teams adopt and a tool teams mean to adopt.

Decisions worth explaining

Three protocols, not one. Validating an HTTP/2 or gRPC service with an HTTP/1.1 client measures the wrong thing. Connection reuse, multiplexing and head-of-line behaviour all differ, and a single-protocol load generator quietly produces a number that does not describe the service under test.

Open-loop arrival models. Constant, Poisson, Ramp, Step and Spike. A closed-loop generator that waits for each response before sending the next one backs off exactly when the service degrades, which is the moment you most need it not to. A warmup phase keeps cold-start noise out of the latency histogram.

Optional chaos. Terminating a healthy replica at peak load tests terminationGracePeriodSeconds, preStop hooks and EndpointSlice propagation delay under the only conditions where they matter.

Finalizer teardown. Deleting a CR mid-run terminates its loadgen and observer Jobs rather than leaving them burning traffic at a production service. That is the kind of thing that only has to happen once.

Alongside the verdict there is a diagnostic pipeline: a readiness lag analyzer measuring the PodRunning to PodReady delta, a TCP and TLS handshake tester, a cgroup throttle watcher reading nr_throttled and nr_periods from cAdvisor to flag CFS quota throttling, and a DNS and PDB auditor for ndots:5 resolver pressure and missing disruption budgets.

Operating it

Installed from a signed OCI chart on GHCR:

helm install scale-sentry \
  oci://ghcr.io/ethan-kane-ops/charts/scale-sentry \
  --namespace scale-sentry --create-namespace

The controller exposes Prometheus metrics on :8080/metrics, with custom collectors for verdict distribution, run duration, HPA reaction latency and diagnostic alerts alongside the stock controller-runtime series. A starter Grafana dashboard ships in the repository. Both the metrics Service and the ServiceMonitor default to off, so the chart installs cleanly on clusters with no prometheus-operator.

Leader election is on by default via a coordination.k8s.io/Lease, so more than one replica is safe. Above one replica the chart also renders a PodDisruptionBudget and zone topology spread constraints.

Every released image and the chart are signed with cosign keyless, and carry a SLSA build provenance attestation and an SBOM published to the registry by the release workflow.