SafeOps Academy

Chapter 14: Supply Chain Security (Advanced)

Chapter 14: Supply Chain Security (Advanced)

Why This Chapter Exists

A successful CI build does not guarantee runtime trust. This chapter enforces a production rule: only verifiable artifacts may run.

The supply-chain baseline in this course is:

  • immutable artifact identity (digest or immutable tag)
  • SBOM generation
  • image signing and attestation
  • cluster-side verification before admission

Learning Objectives

By the end of this chapter, learners can:

  • explain why “build once, promote many” is required for provenance
  • generate and verify SBOM/signature evidence for one artifact
  • run policy rollout in Audit -> Enforce phases in non-production
  • document deny evidence and remediation path

The Incident Hook

An urgent fix is rebuilt from a developer workstation and pushed with a familiar tag. The deploy appears normal, but during incident triage the team cannot prove which workflow produced the binary. Dependency baseline, SBOM lineage, and signer identity are unclear. Rollback confidence drops because artifact trust is uncertain.

What AI Would Propose (Brave Junior)

  • “Rebuild locally and push now.”
  • “Skip signing for this release only.”
  • “Use mutable tags for faster retagging.”

Why this sounds reasonable:

  • shortest time-to-deploy
  • less CI friction under pressure

Why This Is Dangerous

  • No cryptographic provenance at runtime.
  • Rebuild variance breaks “tested artifact == deployed artifact”.
  • Incident response becomes trust investigation instead of recovery.

Guardrails That Stop It

  • Promote by artifact identity, not rebuild.
  • SBOM is generated per release artifact.
  • Signatures/attestations are verified before runtime admission.
  • Verification policy starts in Audit, then moves to Enforce.

Current Platform State

  • Kyverno engine is active: flux/infrastructure/policy/kyverno/
  • Chapter 14 policy pack is scaffolded but inactive by default: flux/infrastructure/policy/packs/chapter-14-supply-chain/
  • This allows safe engine rollout before strict enforcement.

Repository Mapping

  • Engine: flux/infrastructure/policy/kyverno/
  • Pack templates:
    • flux/infrastructure/policy/packs/chapter-14-supply-chain/verify-images.example.yaml
    • flux/infrastructure/policy/packs/chapter-14-supply-chain/verify-attestations.example.yaml
  • App references: flux/apps/**/deployment.yaml
  • Promotion model baseline: docs/course/chapter-04-gitops/

Safe Workflow (Step-by-Step)

  1. Pick immutable artifact reference.
  2. Generate SBOM and sign/attest in CI-compatible flow.
  3. Verify signature and attestation locally.
  4. Apply verify policy in Audit mode in develop.
  5. Observe reports/deny evidence and tune constraints.
  6. Move selected rules to Enforce only after stable audit results.

Lab Files

  • lab.md
  • runbook-supply-chain.md
  • quiz.md

Done When

  • learner proves artifact identity and provenance with command evidence
  • learner demonstrates policy behavior in Audit and Enforce
  • learner can handle unsigned/untrusted artifact deny without disabling guardrails

Handoff

Continue with Chapter 15 (Admission Policy Guardrails) for broader runtime policy enforcement beyond signatures.

Lab: Signed Artifacts and Verification Policy Rollout (Advanced)

Lab: Signed Artifacts and Verification Policy Rollout (Advanced)

Goal

Run a complete non-production supply-chain flow:

  • produce SBOM evidence for one artifact
  • sign and attest the artifact
  • validate evidence with cosign verify*
  • test verification policy in Audit, then Enforce

Prerequisites

  • cosign and syft installed
  • access to a test image in your registry
  • access to develop namespace
  • Kyverno engine running
  • starter policy templates available:
    • flux/infrastructure/policy/packs/chapter-14-supply-chain/verify-images.example.yaml
    • flux/infrastructure/policy/packs/chapter-14-supply-chain/verify-attestations.example.yaml

Quick checks:

command -v cosign
command -v syft
kubectl -n kyverno get pods
kubectl get ns develop

Step 1: Select Immutable Artifact

Use digest form whenever possible:

Quiz: Chapter 14 (Supply Chain Security)

Quiz: Chapter 14 (Supply Chain Security)

Questions

  1. Why is “build once, promote many” a core supply-chain guardrail?

  2. Why is digest reference stronger than mutable tag reference?

  3. What does image signing prove that SBOM alone does not?

  4. Why should verification policy start in Audit for non-production rollout?

  5. Which statement is correct?

  • A) If CI passes, runtime signature verification is redundant.
  • B) Admission verification can stop untrusted artifacts before execution.
  • C) Mutable tags improve incident forensics.

  1. What is the operational difference between Audit and Enforce in Kyverno?

    Runbook: Supply Chain Verification (Advanced)

    Runbook: Supply Chain Verification (Advanced)

    Purpose

    Operate artifact trust controls during deploy and incident response without bypassing safety.

    Scope

    This runbook covers:

    • signature and attestation verification checks
    • Kyverno verify policy behavior (Audit/Enforce)
    • incident triage for untrusted artifact events

    Pre-Deploy Checklist

    1. Artifact is immutable (digest or immutable env tag).
    2. SBOM/provenance evidence exists.
    3. Signature verification succeeds.
    4. Policy mode for target namespace is known (Audit or Enforce).

    Verification Commands

    cosign verify "$IMAGE_REF"
cosign verify-attestation --type spdx "$IMAGE_REF"
kubectl get cpol | rg "verify|attestation"
kubectl get policyreport -A

    Namespace-level incident evidence: