Ephemeral and Preview Environments

Replacing the one shared staging environment with a disposable environment per pull request: isolation models from namespaces to vclusters, the data problem, dependencies you cannot clone, TTLs and cost, and parity measured rather than assumed.

Overview

Why shared staging decays

The shared staging environment is one of the most durable bad ideas in software delivery, and it survives because the failure is gradual. On day one it works; on day four hundred it is permanently half-broken, deployed to by three teams at once, different from production in ways nobody has enumerated, with test failures everyone ignores.

Nobody decided that. It is what a long-lived shared mutable environment converges to.

What ephemeral environments fix

They attack the root cause rather than the symptom. The environment stops being a shared resource and becomes a function of a commit.

Created fresh from declarative source, destroyed afterwards — Drift is impossible: no long-lived state to accumulate in
One per change — No contention; a failure is attributable to exactly one diff
Lives for hours — Nobody negotiates for a slot

Isolation is a spectrum

The isolation question is the first real design decision, and it is a spectrum, not a binary. Mature platforms offer more than one tier and let the change decide.

Namespace + quota + NetworkPolicy — Seconds to create, costs nothing — The large majority of application changes — Anything cluster-scoped: a new CRD, a validating webhook, a ClusterRole
vcluster — A control plane in pods inside a host namespace; a syncer projects workloads onto shared host nodes — Tenant gets cluster-admin and their own CRDs: what platform and operator development needs — The substrate itself
Full cluster — Minutes and money — The substrate itself — Nothing, but it is expensive

Everything outside the cluster

Namespaces do not create databases, object storage buckets, queues, or DNS records.

Crossplane — Represents them as Kubernetes objects; a per-PR Composition provisions the whole footprint and deleting the claim tears it down
Terraform — A workspace per PR does the equivalent in a pipeline

> [!IMPORTANT] Both leak money if the destroy path is less reliable than the create path. A TTL reaper that operates purely on labels, not on a database and not on a webhook, is non-negotiable.

Data is where these projects die

Nobody gets stuck creating a namespace; they get stuck on what goes in the database.

Seed fixtures — Fast, deterministic, versioned with the code — Miss anything that depends on realistic volume or shape
Anonymized production snapshots — Highest fidelity — Masking must happen inside production, in the export job, before data crosses the boundary
Synthetic generation — Splits the difference — A generator that must stay faithful as the schema evolves

> [!WARNING] A pipeline that copies raw production data into a preview environment and masks it there has already leaked it. That is a compliance incident regardless of how quickly the masking runs.

Parity as a measured thing

None of this makes the preview environment production. The existing topic "Staging Passes, Prod Fails" covers the failure symptom; this topic covers the preventative discipline.

Explicit list of accepted deltas — Parity becomes enumerated
Keep it short and review it — The verdict still transfers
Any difference not on the list is a bug in the environment — "Staging is not like prod" turns from a shrug into a backlog

Where interviewers push

Isolation model — Specifically what a namespace cannot isolate
Destroy path — How you guarantee it
Data strategy — Where masking happens
Uncloneable dependencies — What you do about them
Cost — Controls and TTLs
Parity — Your deltas, stated from memory, because someone who has run this can

Key concepts

Environment drift: The accumulated, undocumented divergence of a long-lived environment from its declarative source, caused by manual fixes, out-of-band config edits, and partial deploys. It is unavoidable in any mutable long-lived environment and structurally impossible in one created fresh per change, which is the core argument for ephemerality.
Namespace per PR: The cheapest isolation tier: a namespace named for the pull request, with a ResourceQuota, LimitRange, and NetworkPolicy. Seconds to create, near-zero marginal cost. Does not isolate anything cluster-scoped — CRDs, webhooks, ClusterRoles, PodSecurity, the ingress controller, and node-level configuration are all shared with every other environment.
Virtual cluster (vcluster): A full Kubernetes control plane running as pods inside a single host namespace: API server, controller manager, and a data store defaulting to embedded SQLite with etcd or an external database as options. A syncer translates workload resources into the host namespace so pods run on shared host nodes, while CRDs, RBAC, and webhooks exist only in the virtual API server. Gives a tenant cluster-admin without giving them the host cluster.
Per-environment cloud resources: The infrastructure a namespace cannot create — databases, buckets, queues, DNS records, certificates. Crossplane models these as Kubernetes objects so a Composition provisions and a deleted claim destroys them; Terraform with a per-PR workspace achieves the same in a pipeline. Either way, owner and expiry labels are what make automated cleanup possible.
TTL and reaper: Every environment carries an expiry timestamp. An independent scheduled job enumerates environments by label and deletes anything past its TTL, regardless of the state of the pull request. This is the only cleanup mechanism that survives a dropped webhook, a failed pipeline, or a deleted repository, and without it orphaned environments accumulate until someone reads a cloud bill.
Data seeding strategies: Three options with different tradeoffs. Seed fixtures: fast, deterministic, versioned with the code, low fidelity. Anonymized production snapshot: highest fidelity, slow, and only safe when masking runs inside production before export. Synthetic generation: realistic distributions with no real records, at the cost of maintaining a generator that tracks schema changes.
Service virtualization: Standing in for a dependency you cannot clone — a vendor sandbox with rate limits, licensed software, hardware. A recorded or scripted fake that responds to the same contract. Only trustworthy when backed by contract tests running against the real dependency on a schedule, otherwise the fake drifts and your environment starts passing tests production would fail.
Measured parity: An explicit, reviewed, and short list of accepted differences between a preview environment and production — replica counts, absent CDN, smaller database tier, mocked third parties. The operative rule is that any difference not on the list is a defect in the environment. It turns environment fidelity into something tracked rather than argued about.

When to use it

Any team where more than one change is in flight at a time and a shared staging environment has become a scheduling negotiation rather than a verification step
Changes that need a human to look at them — UI work, design review, product sign-off — where a reviewable URL attached to the pull request replaces a screenshot and a description
Platform and operator development, where the change is a CRD, a webhook, or a controller and a namespace cannot represent it but a full cluster per PR is too slow
Microservice estates where verifying one service change requires a coherent set of its dependencies rather than a single deployable
Migration and upgrade work — a schema change, a dependency major version, a Kubernetes upgrade — where you want to run the migration end to end on disposable infrastructure before touching anything permanent

Interview questions

Design a per-PR environment system end to end. What are the components and where does it break?
Compare namespace-per-PR, vcluster, and a full cluster per environment. When does each stop working?
Data is the hard part. Walk through the options and the compliance constraint.
What do you do about dependencies you cannot clone per pull request?
How do you control cost, and what does the cleanup path look like when everything else has failed?
How do you measure whether a preview environment is representative, and what do you do with the gap?