cellcast

# Enough to run the deploy and nothing else, expiring in fifteen minutes.

$ kubectl patch cluster euw1 --type=merge -p '{"spec":{"state":"DRAINING"}}'
cluster.cellcast.io/euw1 patched

$ ./bin/cellcast place --workload checkout-api --kubeconfig cellcast.kubeconfig
placed checkout-api on use1 (policy checkout-prod, LeastLoaded, confidence high)
credential valid for 15m0s
kubeconfig written to cellcast.kubeconfig (-rw-------, as apps/deployer)

# A cell that stops taking deploys stops being an answer.

# Now a different pipeline, carrying an identity no policy names.

$ ./bin/cellcast place --workload checkout-api --on-unavailable last-known --token-file $intern
NoPolicy: no placement policy permits this caller (--on-unavailable does not apply: the hub answered, and NoPolicy is not a failure to choose a cell)

# Refused, and the fallback did not apply. A refusal is not an outage, and a
# stance that answered one would be a route around the policy engine.

# Now the hub is gone. A pipeline declares up front what that means.

$ ./bin/cellcast place --workload checkout-api --on-unavailable last-known
the hub did not answer: calling the hub: Post "http://127.0.0.1:18080/api/v1/placement": dial tcp 127.0.0.1:18080: connect: connection refused
falling back to the last known cell for checkout-api: use1 (policy checkout-prod, LeastLoaded, decided 14s ago for system:serviceaccount:apps:deployer)
no credential was minted; cellcast replays a decision and never a credential, so this deploy needs one of its own

# A cell, from cache. No credential: minting runs through the hub, so a hub
# that cannot be reached cannot issue one. The stance says where, never how.

The gap this fills

A pipeline deploying into more than one cluster answers two questions before it can do anything: which cluster does this workload go to, and what credential gets it there.

The second question is usually answered with a kubeconfig in a CI secret, one per cluster, valid until somebody rotates it. It is the highest-value secret in the estate, it is readable by every job in the repository, and its blast radius is however long it takes for anyone to notice it leaked.

The first question is usually answered by adopting a multi-cluster control plane. Open Cluster Management, Karmada and Rancher Fleet all decide placement well, and all of them require handing over propagation: their CRDs, their manifest pipeline, their reconciliation loop. That is a large adoption cost for a team that already has a deploy pipeline it likes and only wants to know where to point it.

cellcast answers both questions and owns neither half of the deploy. It does not template, apply, roll back, or watch. It is queried.

How it works

A hub in one cluster, an agent in each of the others, and a client the pipeline invokes.

1. The pipeline runs cellcast place, presenting the workload identity token its CI platform already issues. No secret is configured, because none exists.
2. The hub verifies the token against the issuer’s published keys and maps its claims to an identity.
3. Policy filters the fleet down to the cells that identity is permitted to reach. The filter is the security control and it runs first.
4. The survivors are scored on capacity reported by the agents, and one is chosen.
5. The hub mints a credential for the chosen cell through that cluster’s TokenRequest API, scoped to a service account and bounded by the policy’s ceiling.
6. The client writes a kubeconfig at mode 600 and the pipeline uses it.

$ cellcast place --workload checkout-api --ttl 15m
placed checkout-api on prod-euw1 (policy app-prod, LeastLoaded, confidence high)
credential valid for 15m
kubeconfig written to cellcast.kubeconfig (0600, as apps/deployer)

$ kubectl --kubeconfig cellcast.kubeconfig -n apps apply -f deploy.yaml

Declaring a fleet

Three custom resources, all namespaced. Cluster registers a cell and names the agent permitted to report for it, TrustConfig says how the hub authenticates to that cell in order to mint, and PlacementPolicy says who may reach what.

apiVersion: cellcast.io/v1alpha1
kind: PlacementPolicy
metadata:
  name: checkout-prod
  namespace: cellcast-system
spec:
  subjects:
    - issuer: https://token.actions.githubusercontent.com
      subject: repo:acme/checkout:ref:refs/heads/main
      claims:
        repository: acme/checkout
  permittedCells:
    matchLabels:
      env: prod
    matchExpressions:
      - key: region
        operator: In
        values: [euw1, use1]
  strategy: LeastLoaded
  tokenTTL:
    default: 15m
    max: 30m
  allowDarkTargeting: false

A caller matching no policy is refused. There is no implicit “any cell” and no flag that turns one on, so the set of policy objects is the whole of what anybody can reach.

Decisions worth explaining

Filter, then score. Ordering these the other way produces a system that works in every test with a homogeneous fleet and hands out the wrong cluster the first time the emptiest cell is one the caller may not touch. The end-to-end test is built around exactly that fixture: the least loaded cell in the fleet is one the caller is not permitted to reach, so a scoring pass that ran first would return it.

Selectors are exact-match, with no globs. A pattern reads as narrower than it is, and the gap stays invisible until somebody notices an unexpected branch deploying to production.

A fallback returns a cell but never a credential. --on-unavailable is declared up front and defaults to fail. last-known reuses the cell chosen for this workload if it is still inside the cache TTL, and a cell name pins one in advance. Minting runs through the hub, so a hub that cannot be reached cannot issue one, and the kubeconfig at the target path is deleted on a fallback rather than left for the next step to pick up.

A refusal is not an outage. If the hub answers that the caller is not permitted, no stance applies and the command fails. Falling back there would turn the flag into a route around the policy engine.

Symmetric signing algorithms are refused, not merely unsupported. Accepting HS256 next to RS256 is the algorithm-confusion bug where an attacker signs a token using the issuer’s public key as an HMAC secret. The iss value is compared literally rather than by prefix, for the same reason.

The three binaries are a security boundary. The agent runs in every registered cell and does not link the minting code. A test asserts the import graph, because the reason for the split is invisible in a diff that quietly merges them.

Running it

The hub runs more than one replica. Every replica answers placements, because deciding reads and never writes; only the controllers take a lease. Capacity is in-memory per replica and arrives only from agent heartbeats, so a replica reports itself unready until it has heard from the fleet, with a bounded wait: agents heartbeat through a Service, a Service routes only to ready pods, and a hub whose replicas all restarted at once would otherwise wait for heartbeats nothing can deliver.

On shutdown a replica reports unready and keeps serving while Kubernetes takes it out of the Service, so a rolling update of cellcast does not fail the deploys running through it.

Everything the hub does is written to a single audit record, and the metrics and Kubernetes Events are implementations over that record rather than separate call sites. A test classifies every field on it, so adding one is a decision about whether it could carry token material.

Supply chain

Both charts and all three images are published from one tag by a workflow, and signed with cosign keyless: no long-lived key exists, the signer authenticates over OIDC and receives a certificate valid for minutes, and both signature and certificate land in a public transparency log.

A signature alone proves nothing, so verification names the expected signer, which is the project’s own release workflow:

cosign verify ghcr.io/ethan-kane-ops/cellcast-hub:v0.2.0 \
  --certificate-identity https://github.com/ethan-kane-ops/cellcast/.github/workflows/release.yml@refs/tags/v0.2.0 \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Every image carries an SBOM and SLSA build provenance recorded by the builder from what it compiled, rather than inferred afterwards by scanning a stripped binary. The release recipe runs those verify commands against what it has just published and fails if the identity is not the one the documentation tells adopters to expect.