Chapter 05: CI/CD & Developer Guardrails
Learning Objectives
By the end of this chapter, you will be able to:
- Explain the Layered Defense model from workstation to cluster
- Install and trigger pre-commit hooks to catch issues before push
- Trace a change from workstation commit to cluster apply through all validation layers
- Verify and approve a Terraform plan artifact in GitHub Actions
Start with the video for the concept overview, then work through each lesson section.
Recall Check
Before continuing, quickly recall:
- What is the blast radius risk of bundling two changes in one PR? (Chapter 01)
- Why is a stale Terraform plan dangerous even when the lock is released? (Chapter 02)
- How does Flux decrypt SOPS-encrypted secrets inside the cluster? (Chapter 03)
- Why do we promote images across environments without rebuilding? (Chapter 04)
If you can’t answer these, revisit the corresponding chapter before proceeding.
CI/CD pipelines are where code becomes reality. Without guardrails at every stage, a single unvalidated change can bypass all cluster-side protections. In this chapter, we implement a Layered Defense model from workstation to production.
1. The Problem: The “Bypass” Incident
A developer pushes directly to main, skipping validation. An unreviewed application or infrastructure change reaches production. This bypass removes the last safe checkpoint, leading to untested mutations and a lack of peer-reviewed evidence.
2. The Concept: Layered Guardrails (Defense in Depth)
We don’t rely on a single big check. We use four layers of defense:
- Local (Pre-commit): Catches mistakes before code leaves your machine.
- CI (GitHub Actions): Enforces planning, scanning, and validation.
- Review (CodeRabbit AI): An automated second pair of eyes.
- Approval Gate: Final human sign-off on the reviewed plan.
3. The Code: Pre-commit & GitHub Workflows
Our sre/ repo uses .pre-commit-config.yaml to enforce workstation discipline and GitHub Actions to automate the Plan-Approve-Apply pattern.
Pre-commit baseline
default_install_hook_types:
- pre-commit
- pre-push
- pre-merge-commit
- prepare-commit-msg
repos:
- repo: local
hooks:
- id: master-branch-check
name: Protected branch guard
entry: scripts/pre-commit-master-check.sh
language: script
always_run: true
pass_filenames: false
stages: [pre-commit, pre-push, pre-merge-commit]
args:
- --protected=master
- --protected=main
- id: prevent-amend-after-push
name: Prevent amending pushed commits
entry: scripts/prevent-amend-after-push.sh
language: script
always_run: true
pass_filenames: false
stages: [prepare-commit-msg]
- repo: local
hooks:
- id: flux-kustomize-validate
name: Flux kustomize validate
entry: scripts/flux-kustomize-validate.sh
language: script
files: ^flux/.*\.ya?ml$
pass_filenames: true
require_serial: true
stages: [pre-commit]
- id: terraform-fmt
name: Terraform format check
entry: terraform fmt -recursive -diff -check
language: system
files: \.tf$
pass_filenames: false
stages: [pre-commit]
- id: terraform-validate
name: Terraform validate
entry: scripts/terraform-validate.sh
language: script
files: \.(tf|tfvars)$
pass_filenames: false
require_serial: true
stages: [pre-commit]
- id: terraform-security
name: Terraform security scan
entry: scripts/terraform-security.sh
language: script
files: \.(tf|tfvars)$
pass_filenames: false
require_serial: true
stages: [pre-commit]
- repo: local
hooks:
- id: no-secrets
name: Block sensitive files
entry: scripts/block-secrets.sh
language: script
files: (kubeconfig|\.key$|\.pem$|credentials|\.env$)
stages: [pre-commit]
- repo: https://github.com/koalaman/shellcheck-precommit
rev: v0.10.0
hooks:
- id: shellcheck
files: \.sh$
args: [--severity=warning]
stages: [pre-commit]
- repo: https://github.com/adrienverge/yamllint
rev: v1.35.1
hooks:
- id: yamllint
files: \.ya?ml$
args: [-d, relaxed]
stages: [pre-commit]
4. The Guardrail: Plan-Approve-Apply
We never allow a pipeline to automatically apply infrastructure changes. The Plan-Approve-Apply pattern ensures that every mutation is reviewed by a human and matches the uploaded plan artifact.
Plan-approve-apply workflow
Show the Terraform workflow
name: Terraform - Hetzner
on:
pull_request:
paths:
- "infra/terraform/hcloud_cluster/**"
- "flux/**"
- ".github/workflows/terraform-hcloud*.yml"
push:
branches: [main]
paths:
- "infra/terraform/hcloud_cluster/**"
- "flux/**"
- ".github/workflows/terraform-hcloud*.yml"
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
concurrency:
group: terraform-hcloud
cancel-in-progress: false
permissions:
contents: read
issues: write
jobs:
plan:
runs-on: ubuntu-latest
outputs:
has_changes: ${{ steps.plan.outputs.has_changes }}
defaults:
run:
working-directory: infra/terraform/hcloud_cluster
steps:
- name: Checkout
uses: actions/checkout@v5
- name: Setup kubectl
run: |
curl -sLO "https://dl.k8s.io/release/v1.34.1/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/
- name: Setup Terraform
uses: hashicorp/setup-terraform@v4
with:
terraform_version: "1.14.5"
terraform_wrapper: false
- name: Terraform fmt
run: terraform fmt -check -recursive
- name: Terraform init
env:
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
run: terraform init -input=false
- name: Terraform validate
run: terraform validate
- name: Terraform plan
id: plan
env:
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
TF_VAR_hcloud_token: ${{ secrets.HCLOUD_TOKEN }}
TF_VAR_ssh_public_key: ${{ secrets.HCLOUD_SSH_PUBLIC_KEY }}
TF_VAR_ssh_private_key: ${{ secrets.HCLOUD_SSH_PRIVATE_KEY }}
TF_VAR_flux_git_repository_url: https://github.com/${{ github.repository }}.git
TF_VAR_flux_git_repository_branch: main
TF_VAR_flux_kustomization_path: ./flux/bootstrap/flux-system
TF_VAR_flux_git_token: ${{ secrets.FLUX_GIT_TOKEN }}
TF_VAR_enable_ghcr: "true"
TF_VAR_ghcr_username: ${{ secrets.GHCR_USERNAME }}
TF_VAR_ghcr_token: ${{ secrets.GHCR_TOKEN }}
TF_VAR_sops_age_key: ${{ secrets.SOPS_AGE_KEY }}
TF_VAR_backup_s3_access_key_id: ${{ secrets.R2_ACCESS_KEY_ID }}
TF_VAR_backup_s3_secret_access_key: ${{ secrets.R2_SECRET_ACCESS_KEY }}
TF_VAR_backup_s3_bucket: ${{ secrets.R2_BUCKET }}
TF_VAR_backup_s3_endpoint: ${{ secrets.R2_ENDPOINT }}
TF_VAR_backup_s3_region: ${{ secrets.R2_REGION }}
run: |
set +e
set -o pipefail
terraform plan -input=false -lock-timeout=5m -no-color -detailed-exitcode -out=tfplan 2>&1 | tee plan.txt
exit_code=${PIPESTATUS[0]}
set -e
if [ "$exit_code" -eq 1 ]; then
echo "Terraform plan failed."
exit 1
fi
if [ "$exit_code" -eq 0 ]; then
echo "has_changes=false" >> "$GITHUB_OUTPUT"
else
echo "has_changes=true" >> "$GITHUB_OUTPUT"
fi
- name: Upload tfplan artifact
if: github.event_name == 'push' && steps.plan.outputs.has_changes == 'true'
uses: actions/upload-artifact@v4
with:
name: terraform-hcloud-tfplan
path: |
infra/terraform/hcloud_cluster/tfplan
infra/terraform/hcloud_cluster/plan.txt
retention-days: 1
approval:
runs-on: ubuntu-latest
needs: plan
if: github.event_name == 'push' && needs.plan.outputs.has_changes == 'true'
timeout-minutes: 60
steps:
- name: Manual approval gate
uses: pavlospt/manual-approval@v2
with:
secret: ${{ github.token }}
approvers: ldbl
minimum-approvals: 1
issue-title: "Terraform apply — ${{ github.sha }}"
issue-body: |
Terraform plan detected infrastructure changes on `main`.
**Commit:** ${{ github.sha }}
**Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
Approve or deny this apply.
exclude-workflow-initiator-as-approver: false
apply:
runs-on: ubuntu-latest
needs: [plan, approval]
if: github.event_name == 'push' && needs.plan.outputs.has_changes == 'true' && needs.approval.result == 'success'
defaults:
run:
working-directory: infra/terraform/hcloud_cluster
steps:
- name: Checkout
uses: actions/checkout@v5
- name: Setup kubectl
run: |
curl -sLO "https://dl.k8s.io/release/v1.34.1/bin/linux/amd64/kubectl"
chmod +x kubectl && sudo mv kubectl /usr/local/bin/
- name: Setup Terraform
uses: hashicorp/setup-terraform@v4
with:
terraform_version: "1.14.5"
- name: Terraform init
env:
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
run: terraform init -input=false
- name: Download tfplan artifact
uses: actions/download-artifact@v4
with:
name: terraform-hcloud-tfplan
path: infra/terraform/hcloud_cluster
- name: Terraform apply
env:
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
run: terraform apply -input=false -lock-timeout=5m tfplan
5. Verification: Did I Get It?
Verify your local guardrails and the CI pipeline status:
# Verify local hooks are active
pre-commit run --all-files
# Check GitHub Actions for 'Waiting for approval' status on recent runs