[Epic]: Software supply chain security: visibility, reproducibility, verification

[mchmarny]

Goal

Raise AICR's software supply chain security to a level customers and internal security review can sign off on. Three maturity stages, mapped to the SLSA / NIST SSDF outcomes:

1. Visibility — every container image AICR can deploy is enumerable, auditable, and consumable by standard tooling (CycloneDX, Trivy, Grype, Cosign).
2. Reproducibility — the same recipe rendered today and a year from now produces the same artifacts. Recipe → image set is deterministic.
3. Provenance & verification — every artifact AICR deploys can be cryptographically tied to a known publisher and a known build, with admission-time enforcement where chart internals make in-tree pinning impractical.

Why this matters

• Customer security review. Enterprise and regulated customers ask for an SBOM, a signing/provenance matrix, and a reproducibility statement before deploying. The SBOM and pinning story now exist; the signing/provenance matrix is in flight.
• Air-gap deployment. AICR pulls from 11+ public registries; customers behind a corporate firewall need a complete mirror list.
• Compromise blast radius. Tag-mutable references mean an upstream account compromise silently changes our deployments. Digest pinning + signature verification caps the blast radius and enables detection.
• Reproducible builds. "Same recipe → same bytes" is the foundation for diff-based audit, regression analysis, and incident response.
• SLSA / SSDF alignment. Visibility (level 1), Provenance (level 2), Hermetic builds (level 3) — each stage below corresponds to one of these.

Current state

From the published BOM (docs/user/container-images.md, generated by make bom):

Metric	Value
Components	22
Unique images	69
Distinct registries	11
Helm components without a chart-version pin	0 ✅
In-tree manifest images digest-pinned	4 / 11 (the 7 remaining are CRD-triplet refs whose schemas reject digests; tracked via 4 upstream signing requests below)
Components with documented signing/SBOM status	0

All chart versions are pinned (PR #777). Every Pod-spec image reference under recipes/components/*/manifests/ that AICR controls in-tree is now digest-pinned (PRs #778, #779); the seven CRD-triplet exemptions are documented in recipes/manifest_images_test.go::imageDigestExemptions with reasons and upstream tracking links. The digest-pin CI gate enforces sha256: specifically (PR #779). Untagged-reference cleanup landed via PR #761; CRD-style sibling triplets (NicClusterPolicy, Skyhook Package) and bare-scalar placeholders (vgpu-manager-style) are now correctly handled end-to-end (PRs #761, #776).

Child issues, by maturity stage

Stage 1 — Visibility

• Generate authoritative image BOM via helm template #742 — Generate authoritative image BOM via helm template (delivered via feat(bom): add CycloneDX 1.6 image BOM generator #747)
• Fix incomplete/untagged image references in component manifests #744 — Fix incomplete/untagged image references in component manifests (delivered via fix(recipes): fully-qualify image refs in component manifests #761)
• Document container image inventory in user docs #741 — Document container image inventory in user docs
  • 1a — Tooling, deterministic output, scheduled refresh (delivered via feat(bom): publish container image inventory as a doc artifact #763)
  • 1b — Prose around the auto-generated table (delivered via docs(bom): wrap auto-generated image inventory with hand-written prose #773)
• Air-gap and private-registry mirroring guide #743 — Air-gap and private-registry mirroring guide

Stage 2 — Reproducibility ✅

• Decide pinning policy for high-implicit-surface components #740 — ADR-006: Container image pinning policy (delivered via docs(adr): add ADR-006 container image pinning policy #775)
• Pin chart versions for all components in recipes/registry.yaml #748 — Pin chart versions for all components in recipes/registry.yaml
  • Phase A — external charts (nfd, k8s-ephemeral-storage-metrics) (delivered via feat(recipes): pin nfd and k8s-ephemeral-storage-metrics chart versions #758)
  • Phase B — NVIDIA-owned charts (gpu-operator, network-operator, nvidia-dra-driver-gpu, nodewright-operator) + bom-pinning-check CI gate (delivered via feat(recipes): pin chart versions for NVIDIA-owned components (#748 Phase B) #777)
• Digest-pin explicit image references in recipes #749 — Digest-pin explicit image references in recipes (delivered via feat(recipes): digest-pin explicit image references (#749) #778; sha256-only enforcement tightened via test(recipes): enforce sha256 specifically in digest-pin gate (CodeRabbit follow-up to #778) #779) — TestComponentManifestImagesAreDigestPinned enforces in CI; Renovate kubernetes manager rotates digests as upstream rebuilds the same tag

Stage 3 — Provenance & verification

• Supply-chain provenance audit per component #745 — Supply-chain provenance audit per component (cosign signatures, SBOMs, SLSA attestations; deploy-time digest verification policy for chart-default sub-images)
  • Upstream signing requests filed (track parity per upstream):
    • NVIDIA/gpu-operator#2432 — keyless cosign + SLSA + SBOM
    • Mellanox/network-operator#2555 — keyless cosign + SLSA + SBOM
    • kubernetes-sigs/dra-driver-nvidia-gpu#1105 — keyless cosign + SLSA + SBOM (project moved from NVIDIA/k8s-dra-driver-gpu)
    • NVIDIA/nodewright#224 — keyless cosign + SLSA + SBOM (formerly NVIDIA/skyhook)

Related issues (not children, but adjacent)

• fix(recipes): aws-efa hardcodes us-west-2 ECR; should template region (and partition) #764 — aws-efa hardcodes us-west-2 ECR, addressed via doc-pattern (delivered via fix(recipes): document aws-efa regional ECR override pattern #774).
• fix(bom): gpu-operator extracts vgpu-manager as a bare image ref #765 — gpu-operator BOM extracts vgpu-manager as a bare image ref; published BOM accuracy follow-up (delivered via fix(bom): reject bare scalars without tag, digest, or registry host #776).
• [Epic] Verifiable recipe test evidence for community contributions #750 — [Epic] Verifiable recipe test evidence for community contributions. Distinct epic; depends on this one (BOM is part of every evidence bundle).

Why this scope is finite

In-tree digest-pinning of every chart-default sub-image (gpu-operator's ~14, etc.) is not an end goal of this epic. Most charts don't expose a digest field for sub-images, and forking + maintaining that override matrix is intractable. The right answer for chart-default sub-images is admission-time digest verification (Kyverno / Cosign) plus signed image attestations from upstream — captured in #745 and the four upstream signing requests under it. We pin what we control in-tree (#748 ✅, #749 ✅) and verify what we don't (#745).

Definition of done

• BOM is generated automatically and consumable as CycloneDX 1.6 by Trivy/Grype/Cosign without conversion.
• BOM is published as a versioned doc artifact and refreshed weekly to surface upstream drift.
• Every chart in recipes/registry.yaml has a pinned version; CI fails if not.
• Every image reference AICR controls in-tree carries an @sha256: digest; CI fails if not. (CRD-triplet exemptions documented with reasons and upstream tracking; sha256-only enforcement via test(recipes): enforce sha256 specifically in digest-pin gate (CodeRabbit follow-up to #778) #779.)
• Every component's signing/SBOM/provenance status is documented; gaps have an owner or a deploy-time verification policy.
• A new component cannot land without satisfying all of the above (enforced in make qualify).
• Air-gap, security-review, and customer compliance docs reference the BOM and the provenance matrix as authoritative.

Out of scope

• AICR's own runtime images (aicr, aicrd, validators) — covered by goreleaser/SLSA tooling.
• Digest-pinning chart-default sub-images (gpu-operator's ~14 sub-images, kube-prometheus-stack's ~8, etc.) — most charts don't expose a digest field, so the right answer is admission-time digest verification under Supply-chain provenance audit per component #745, not in-tree pinning.

Sequencing

Stage 1 (visibility)                   Stage 2 (reproducibility) ✅          Stage 3 (verification)
─────────────────────                  ─────────────────────────              ──────────────────────
#742 ✅ ──┬─► #741 ✅ (a✅, b✅)         #740 ✅ ──┬─► #748 ✅ ──► #749 ✅       #745 (+ 4 upstream issues)
#744 ✅   └─► #743                                └─►

• Stage 1 is mostly delivered. Remaining: Air-gap and private-registry mirroring guide #743 air-gap mirroring guide.
• Stage 2 is fully delivered. Policy ADR (Decide pinning policy for high-implicit-surface components #740 ✅), chart pins (Pin chart versions for all components in recipes/registry.yaml #748 ✅), digest pins for explicit refs (Digest-pin explicit image references in recipes #749 ✅, sha256-tightened via test(recipes): enforce sha256 specifically in digest-pin gate (CodeRabbit follow-up to #778) #779). All CI-enforced.
• Stage 3 is in flight. Upstream signing requests filed against the four NVIDIA-owned upstream repos; Supply-chain provenance audit per component #745 (provenance audit + admission-time verification) is the next epic-internal step.