Release Manifests & Last-Known-Good

A release manifest pins every GPU-stack component to an exact hash so a validated Last-Known-Good build is reproducible on demand, not rediscovered from tip.

Key concepts

Release manifest
A versioned file pinning every component to an exact commit SHA or artifact digest. Analogous to repo manifest.xml, west.yml, or a conan.lock. It is the single source of truth for what a release contains; reading it reconstructs the stack exactly.
Pinning
Recording an immutable identity (a git revision SHA or a sha256: OCI digest) instead of a mutable reference (main, latest, a version range). Only pinned inputs make a build reproducible; a floating tag re-resolves and silently changes the stack.
Last-Known-Good (LKG)
The newest manifest that passed the entire validation suite, published under a stable LKG pointer. Engineers build against LKG rather than tip so they inherit a validated baseline instead of every in-flight regression.
Promote-on-green
The rule that a candidate manifest becomes LKG only when every required gate reports pass. Promotion moves a pointer or retags an artifact; it never triggers a rebuild, so the thing that passed is the thing that ships.
Artifact promotion vs rebuild
Immutable, content-addressed artifacts move between channels or registries unchanged (retag, copy by digest with oras cp or skopeo copy). Rebuilding to promote is forbidden because it discards the validation result tied to the original digest.
Provenance and attestation
Signed, verifiable metadata binding a manifest and its artifacts to how they were produced (cosign attest, SLSA provenance, in-toto). Consumers verify with cosign verify-attestation before trusting an LKG, so a promotion cannot be forged or tampered with.
Release recipe reproducibility
The property that the manifest plus its pinned inputs regenerate a specific release on demand months later. Achieved by pinning every input, archiving the manifest immutably, and never depending on mutable external state or latest toolchains.

When to use it

  • You ship a multi-component stack (firmware, kernel driver, ROCm) where the combination, not any single component, is what must be validated and reproduced.
  • Engineers report that tip is too unstable to develop against and need a trusted baseline that updates only after full validation passes.
  • A customer hits a bug on a release from months ago and you must reconstruct that exact stack to reproduce and bisect it.
  • You need to promote binaries between environments (dev to staging to release) without rebuilding, so the tested artifact is the shipped artifact.
  • You must roll back a regression introduced by a promotion and want a previous known-good pointer to revert to in one step.

Interview questions

  • Design an LKG promotion pipeline for a firmware plus kernel-driver plus ROCm stack. What are the stages and where does the LKG pointer live?
  • How do you pin a multi-component stack so a release is reproducible months later?
  • What are your promote-on-green criteria and gates for moving a manifest to LKG?
  • How do you give every engineer a trusted baseline to build against instead of tip?
  • A customer reports a bug on a release from months ago. How do you reproduce that exact stack?
  • LKG was promoted with a latent regression that CI missed. How do you roll back safely?

Sign up free to read the full topic

More DevOps topics

Start free on Cariara · Coding problems · System design studio