Runbook: Admission Policy Operations (Advanced)

Purpose

Operate Kyverno guardrails safely while keeping deployment velocity and strong auditability.

Scope

This runbook covers:

policy engine health checks
Audit and Enforce rollout operations
deny triage and manifest remediation
break-glass exception discipline

Health and Inventory Checks

kubectl -n kyverno get pods
kubectl get cpol
kubectl get policyreport -A

If engine pods are not healthy, pause policy mode changes.

Operational Policy Modes

Audit: violations are reported but admission is allowed.
Enforce: violating resources are blocked at admission.

Check policy mode:

kubectl get cpol <policy-name> -o jsonpath='{.spec.validationFailureAction}'; echo

Patch policy mode:

kubectl patch cpol <policy-name> --type merge -p '{"spec":{"validationFailureAction":"Audit"}}'
kubectl patch cpol <policy-name> --type merge -p '{"spec":{"validationFailureAction":"Enforce"}}'

Deny Triage Workflow

Capture deny error from kubectl apply output.
Identify policy and rule from event/report.
Confirm whether violation is expected policy behavior.
Fix manifest to comply.
Re-apply and verify success.

Useful commands:

kubectl -n <ns> get events --sort-by=.lastTimestamp | tail -n 30
kubectl get policyreport -A | rg <ns>
kubectl describe cpol <policy-name>

Exception Workflow (Break-Glass)

Allowed only when all are true:

production incident pressure is confirmed
compliant fix is not immediately available
exception is minimal, scoped, and time-bound

Mandatory exception fields:

owner
reason
scope (namespace/resource/policy)
expiry timestamp
incident or approval reference

After incident:

remove exception
restore intended policy mode
capture preventive follow-up action

Failure Modes

False positives:

tighten match selectors and conditions
keep policy intent unchanged

Exception sprawl:

review active exceptions weekly
auto-expire or delete stale exceptions

Silent bypass:

audit RBAC and cluster-admin usage
verify no broad wildcard exclusions in policies