Chapter 16: Admission Policy Guardrails

Learning Objectives

By the end of this chapter, you will be able to:

• Deploy Kyverno policy packs in audit mode for safe rollout
• Graduate policies from audit to enforce using evidence from audit logs
• Configure break-glass exceptions with expiry and evidence requirements
• Evaluate policy coverage against the security baseline

Start with the video for the concept overview, then work through each lesson section.

Upstream guardrails (CI, PR reviews) are essential but can be bypassed. In this chapter, we implement a Cluster Gatekeeper using Kyverno. This is our final line of defense, ensuring that only workloads matching our production standards are admitted to the cluster.

1. The Problem: The “Policy Bypass”

A critical configuration error (like running as root or missing resource limits) slips through the CI/CD pipeline. Without a cluster-side check, the non-compliant workload runs in production, violating your security and stability invariants. The system’s safety depends entirely on every previous step being perfect.

2. The Concept: Cluster-Side Enforcement

We move from “hoping” the pipeline is correct to “enforcing” the state at the Kubernetes API boundary.

1. Admission Control: Every request to create or update a resource is intercepted by a policy engine.
2. Kyverno: A Kubernetes-native policy engine that uses standard YAML to define rules.
3. Fail-Closed: If a workload violates a critical policy (e.g., Security Context), the cluster rejects the request entirely.

3. The Code: The Policy Pack

Our sre/ repo includes standard policy packs for security, resources, and best practices. These policies are the definitive contract for what is allowed to run in our cluster.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-signed-images-example
spec:
  validationFailureAction: Audit
  background: false
  rules:
    - name: verify-ghcr-signatures
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "${image_registry}/*"
          mutateDigest: true
          attestors:
            - entries:
                - keyless:
                    # Replace with your CI issuer and subject constraints.
                    issuer: "https://token.actions.githubusercontent.com"
                    subject: "https://github.com/${git_owner}/*"

4. The Guardrail: Audit-First Rollout

We never block traffic blindly. Every new policy starts in Audit mode, where violations are recorded in a PolicyReport but not blocked. Only after the audit is clean do we move the policy to Enforce mode.

5. Verification: Did I Get It?

Check your current cluster compliance and test the boundary:

# View compliance reports for the develop namespace
kubectl get policyreports -n develop
# Try to deploy a non-compliant pod (e.g., missing resource limits)
kubectl run policy-test --image=nginx

Expected Output: If an enforcement policy is active, the command should fail with an error message from Kyverno explaining the violation.

Detailed Lessons

• The Incident: The Policy Bypass
• Investigation & Containment
• Workflow & Strategy
• Lab & Completion
• Quiz