[Epic] Verifiable recipe test evidence for community contributions

[mchmarny]

Goal

Enable any community contributor to submit a new recipe PR with a signed evidence bundle that lets AICR maintainers approve the recipe without re-running validation themselves. AICR maintainers don't have access to every CSP, NCP, ISV platform, or accelerator — the recipe author does. This epic builds the framework that converts the author's local test run into a verifiable artifact a maintainer can trust.

Why this matters

• Maintainer reachability gap. AICR ships recipes for AWS / OCI / GCP / Azure / CoreWeave / Forge / on-prem combinations spanning H100 / GB200 / B200 / MI300 / various OS and intent variants. No single team has hands-on access to all of them. Today this is either a blocker for community contributions or a leap of faith for reviewers.
• The recipe already self-defines its tests. validation, componentRefs, criteria, and the existing aicr validate pipeline mean "did the recipe pass" is already a well-formed, machine-checkable question. We don't need to extend the recipe spec — we need to package the answer.
• Trust transfer, not test reproduction. The contributor runs aicr validate against real hardware they have access to. AICR's tooling cryptographically packages the recipe + snapshot + per-phase results + BOM into a signed bundle. The maintainer (or CI) verifies the bundle and reads a Markdown summary. The artifact is the trust handoff.

Architecture

┌─ Contributor side ─────────────┐    ┌─ Maintainer / CI side ──────────┐
│                                │    │                                 │
│  cluster matching recipe       │    │  PR landing in NVIDIA/aicr      │
│  criteria (real hardware)      │    │                                 │
│            │                   │    │   evidence-bundle.tar.gz        │
│            ▼                   │    │            │                    │
│  aicr snapshot ──► snapshot    │    │            ▼                    │
│  aicr validate                 │    │   aicr verify-evidence          │
│  aicr validate                 │    │     ├─ cosign verify            │
│    --emit-evidence             │    │     ├─ schema check             │
│            │                   │    │     ├─ recipe ↔ snapshot bind   │
│            ▼                   │    │     ├─ replay inline constraints│
│  evidence-bundle.tar.gz        │    │     └─ render Markdown summary  │
│  (in-toto attestation,         │    │            │                    │
│   cosign keyless signed)       │    │            ▼                    │
│            │                   │    │   GitHub Action posts review    │
│            └────► PR ──────────┼────┼─►  comment to the PR            │
│                                │    │                                 │
└────────────────────────────────┘    └─────────────────────────────────┘

Bundle anatomy

Every bundle is an in-toto attestation with a custom AICR predicate, signed with cosign keyless OIDC (KMS support is a future extension). The predicate carries:

• The canonical recipe YAML (post-overlay resolution)
• The cluster snapshot, including a fingerprint (detected service, accelerator, OS, k8s-version)
• AICR version, validator catalog version, validator image digests
• Per-phase results (CTRF + raw logs + structured Markdown evidence)
• The CycloneDX BOM (from [Epic]: Software supply chain security: visibility, reproducibility, verification #739) tying the recipe to the exact images deployed
• A manifest.json declaring schema version, file inventory, and content hashes

In-toto + cosign gives us sigstore-native verification today and keeps the door open for SLSA Verification Summary Attestation (VSA) interop later.

Child issues

Foundation

• Capture cluster fingerprint in snapshot for recipe-criteria binding #752 — Capture cluster fingerprint in snapshot for recipe-criteria binding
• Define evidence bundle format and emit via aicr validate --emit-evidence #754 — Define evidence bundle format and emit via aicr validate --emit-evidence

Verification surface

• aicr verify-evidence command for reviewers and CI #753 — aicr verify-evidence command for reviewers and CI

Community workflow

• Recipe contribution workflow with CI evidence gate #751 — Recipe contribution workflow with CI evidence gate

Related issues (not children, but contributing)

• Reuse Chainsaw health checks to improve readiness check in aicr validate --phase deployment #660 — Reuse Chainsaw health checks in aicr validate --phase deployment (deeper deployment evidence = higher-quality bundle content)
• Proposal: per-recipe stability signal (preview / experimental / stable / deprecated) #613 — Per-recipe stability signal (helps reviewers contextualize evidence — preview vs stable)
• Extract Validation as standalone portable type from pkg/recipe #732, Extract ValidatorCatalog as standalone portable type #733, Add WithCatalog() and WithValidation() recipe builder options #735 — Portable validation/catalog types, recipe builder options (enable external tooling to construct recipes and consume evidence programmatically)
• [Epic]: Support external validation controller integration #731 — External validation controller integration (alternative drivers for evidence generation)

Hard dependency

• [Epic]: Software supply chain security: visibility, reproducibility, verification #739 — The BOM is part of every evidence bundle, and the chart-version + digest pinning under Pin chart versions for all components in recipes/registry.yaml #748/Digest-pin explicit image references in recipes #749 make the recipe-to-images binding deterministic. Without [Epic]: Software supply chain security: visibility, reproducibility, verification #739 progressing, evidence bundles can vouch for "this recipe at this AICR version produced this image set on this date" but cannot vouch for byte-level reproducibility of the deployed images.

Definition of done

• A community contributor can submit a recipe PR including a signed evidence-bundle.tar.gz.
• aicr verify-evidence validates the bundle locally and via CI.
• A GitHub Action runs verify on every recipe PR and posts a structured Markdown summary as a review comment.
• The bundle format is publicly documented and versioned so third parties can verify without AICR.
• CONTRIBUTING.md describes the workflow end-to-end with a worked example.
• AICR maintainers can approve a recipe PR for hardware they don't physically have, on the strength of the bundle + verification report.

Out of scope

• Auto-merging recipe PRs based on evidence. Evidence is a verifiable input to maintainer judgment, not a replacement for it.
• Extending the recipe schema with new "acceptance criteria" fields. The recipe already self-defines its tests — we're packaging the result, not redefining what passing means.
• Hosting/storing third-party bundles long-term. The bundle is a PR artifact; cosign + Rekor provide the durable record.

Sequencing

Foundation                       Verification          Community workflow
───────────                      ──────────────         ──────────────────
  #752 fingerprint ──┐
                     ├──► #754 bundle + emit ──► #753 verify-evidence ──► #751 PR template + CI gate
  (snapshot work)

#754 (bundle format) depends on #752 (fingerprint) being trustworthy. #753 (verify-evidence) depends on #754. #751 (community workflow) wraps #753.