Core Track Guardrails-first chapter in core learning path.

Estimated Time

  • Reading: 20-25 min
  • Lab: 45-60 min
  • Quiz: 10-15 min

Prerequisites

Source Code References

  • .coderabbit.yml Members
  • .pre-commit-config.yaml Members
  • terraform-hcloud-destroy.yml Members
  • terraform-hcloud.yml Members

Sign in to view source code.

What You Will Produce

A reproducible lab result plus quiz verification and incident-safe operating evidence.

Chapter 05: CI/CD & Developer Guardrails

Learning Objectives

By the end of this chapter, you will be able to:

  • Explain the Layered Defense model from workstation to cluster
  • Install and trigger pre-commit hooks to catch issues before push
  • Trace a change from workstation commit to cluster apply through all validation layers
  • Verify and approve a Terraform plan artifact in GitHub Actions

Start with the video for the concept overview, then work through each lesson section.

Recall Check

Before continuing, quickly recall:

  • What is the blast radius risk of bundling two changes in one PR? (Chapter 01)
  • Why is a stale Terraform plan dangerous even when the lock is released? (Chapter 02)
  • How does Flux decrypt SOPS-encrypted secrets inside the cluster? (Chapter 03)
  • Why do we promote images across environments without rebuilding? (Chapter 04)

If you can’t answer these, revisit the corresponding chapter before proceeding.

CI/CD pipelines are where code becomes reality. Without guardrails at every stage, a single unvalidated change can bypass all cluster-side protections. In this chapter, we implement a Layered Defense model from workstation to production.


1. The Problem: The “Bypass” Incident

A developer pushes directly to main, skipping validation. An unreviewed application or infrastructure change reaches production. This bypass removes the last safe checkpoint, leading to untested mutations and a lack of peer-reviewed evidence.

2. The Concept: Layered Guardrails (Defense in Depth)

We don’t rely on a single big check. We use four layers of defense:

  1. Local (Pre-commit): Catches mistakes before code leaves your machine.
  2. CI (GitHub Actions): Enforces planning, scanning, and validation.
  3. Review (CodeRabbit AI): An automated second pair of eyes.
  4. Approval Gate: Final human sign-off on the reviewed plan.

3. The Code: Pre-commit & GitHub Workflows

Our sre/ repo uses .pre-commit-config.yaml to enforce workstation discipline and GitHub Actions to automate the Plan-Approve-Apply pattern.

Pre-commit baseline

default_install_hook_types:
  - pre-commit
  - pre-push
  - pre-merge-commit
  - prepare-commit-msg

repos:
  - repo: local
    hooks:
      - id: master-branch-check
        name: Protected branch guard
        entry: scripts/pre-commit-master-check.sh
        language: script
        always_run: true
        pass_filenames: false
        stages: [pre-commit, pre-push, pre-merge-commit]
        args:
          - --protected=master
          - --protected=main

      - id: prevent-amend-after-push
        name: Prevent amending pushed commits
        entry: scripts/prevent-amend-after-push.sh
        language: script
        always_run: true
        pass_filenames: false
        stages: [prepare-commit-msg]

  - repo: local
    hooks:
      - id: flux-kustomize-validate
        name: Flux kustomize validate
        entry: scripts/flux-kustomize-validate.sh
        language: script
        files: ^flux/.*\.ya?ml$
        pass_filenames: true
        require_serial: true
        stages: [pre-commit]

      - id: terraform-fmt
        name: Terraform format check
        entry: terraform fmt -recursive -diff -check
        language: system
        files: \.tf$
        pass_filenames: false
        stages: [pre-commit]

      - id: terraform-validate
        name: Terraform validate
        entry: scripts/terraform-validate.sh
        language: script
        files: \.(tf|tfvars)$
        pass_filenames: false
        require_serial: true
        stages: [pre-commit]

      - id: terraform-security
        name: Terraform security scan
        entry: scripts/terraform-security.sh
        language: script
        files: \.(tf|tfvars)$
        pass_filenames: false
        require_serial: true
        stages: [pre-commit]

  - repo: local
    hooks:
      - id: no-secrets
        name: Block sensitive files
        entry: scripts/block-secrets.sh
        language: script
        files: (kubeconfig|\.key$|\.pem$|credentials|\.env$)
        stages: [pre-commit]

  - repo: https://github.com/koalaman/shellcheck-precommit
    rev: v0.10.0
    hooks:
      - id: shellcheck
        files: \.sh$
        args: [--severity=warning]
        stages: [pre-commit]

  - repo: https://github.com/adrienverge/yamllint
    rev: v1.35.1
    hooks:
      - id: yamllint
        files: \.ya?ml$
        args: [-d, relaxed]
        stages: [pre-commit]

4. The Guardrail: Plan-Approve-Apply

We never allow a pipeline to automatically apply infrastructure changes. The Plan-Approve-Apply pattern ensures that every mutation is reviewed by a human and matches the uploaded plan artifact.

Plan-approve-apply workflow

Show the Terraform workflow
name: Terraform - Hetzner

on:
  pull_request:
    paths:
      - "infra/terraform/hcloud_cluster/**"
      - "flux/**"
      - ".github/workflows/terraform-hcloud*.yml"
  push:
    branches: [main]
    paths:
      - "infra/terraform/hcloud_cluster/**"
      - "flux/**"
      - ".github/workflows/terraform-hcloud*.yml"

env:
  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"

concurrency:
  group: terraform-hcloud
  cancel-in-progress: false

permissions:
  contents: read
  issues: write

jobs:
  plan:
    runs-on: ubuntu-latest
    outputs:
      has_changes: ${{ steps.plan.outputs.has_changes }}

    defaults:
      run:
        working-directory: infra/terraform/hcloud_cluster

    steps:
      - name: Checkout
        uses: actions/checkout@v5

      - name: Setup kubectl
        run: |
          curl -sLO "https://dl.k8s.io/release/v1.34.1/bin/linux/amd64/kubectl"
          chmod +x kubectl && sudo mv kubectl /usr/local/bin/

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: "1.14.5"
          terraform_wrapper: false

      - name: Terraform fmt
        run: terraform fmt -check -recursive

      - name: Terraform init
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
        run: terraform init -input=false

      - name: Terraform validate
        run: terraform validate

      - name: Terraform plan
        id: plan
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
          TF_VAR_hcloud_token: ${{ secrets.HCLOUD_TOKEN }}
          TF_VAR_ssh_public_key: ${{ secrets.HCLOUD_SSH_PUBLIC_KEY }}
          TF_VAR_ssh_private_key: ${{ secrets.HCLOUD_SSH_PRIVATE_KEY }}
          TF_VAR_flux_git_repository_url: https://github.com/${{ github.repository }}.git
          TF_VAR_flux_git_repository_branch: main
          TF_VAR_flux_kustomization_path: ./flux/bootstrap/flux-system
          TF_VAR_flux_git_token: ${{ secrets.FLUX_GIT_TOKEN }}
          TF_VAR_enable_ghcr: "true"
          TF_VAR_ghcr_username: ${{ secrets.GHCR_USERNAME }}
          TF_VAR_ghcr_token: ${{ secrets.GHCR_TOKEN }}
          TF_VAR_sops_age_key: ${{ secrets.SOPS_AGE_KEY }}
          TF_VAR_backup_s3_access_key_id: ${{ secrets.R2_ACCESS_KEY_ID }}
          TF_VAR_backup_s3_secret_access_key: ${{ secrets.R2_SECRET_ACCESS_KEY }}
          TF_VAR_backup_s3_bucket: ${{ secrets.R2_BUCKET }}
          TF_VAR_backup_s3_endpoint: ${{ secrets.R2_ENDPOINT }}
          TF_VAR_backup_s3_region: ${{ secrets.R2_REGION }}
        run: |
          set +e
          set -o pipefail
          terraform plan -input=false -lock-timeout=5m -no-color -detailed-exitcode -out=tfplan 2>&1 | tee plan.txt
          exit_code=${PIPESTATUS[0]}
          set -e
          if [ "$exit_code" -eq 1 ]; then
            echo "Terraform plan failed."
            exit 1
          fi
          if [ "$exit_code" -eq 0 ]; then
            echo "has_changes=false" >> "$GITHUB_OUTPUT"
          else
            echo "has_changes=true" >> "$GITHUB_OUTPUT"
          fi

      - name: Upload tfplan artifact
        if: github.event_name == 'push' && steps.plan.outputs.has_changes == 'true'
        uses: actions/upload-artifact@v4
        with:
          name: terraform-hcloud-tfplan
          path: |
            infra/terraform/hcloud_cluster/tfplan
            infra/terraform/hcloud_cluster/plan.txt
          retention-days: 1

  approval:
    runs-on: ubuntu-latest
    needs: plan
    if: github.event_name == 'push' && needs.plan.outputs.has_changes == 'true'
    timeout-minutes: 60
    steps:
      - name: Manual approval gate
        uses: pavlospt/manual-approval@v2
        with:
          secret: ${{ github.token }}
          approvers: ldbl
          minimum-approvals: 1
          issue-title: "Terraform apply — ${{ github.sha }}"
          issue-body: |
            Terraform plan detected infrastructure changes on `main`.

            **Commit:** ${{ github.sha }}
            **Run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}

            Approve or deny this apply.
          exclude-workflow-initiator-as-approver: false

  apply:
    runs-on: ubuntu-latest
    needs: [plan, approval]
    if: github.event_name == 'push' && needs.plan.outputs.has_changes == 'true' && needs.approval.result == 'success'

    defaults:
      run:
        working-directory: infra/terraform/hcloud_cluster

    steps:
      - name: Checkout
        uses: actions/checkout@v5

      - name: Setup kubectl
        run: |
          curl -sLO "https://dl.k8s.io/release/v1.34.1/bin/linux/amd64/kubectl"
          chmod +x kubectl && sudo mv kubectl /usr/local/bin/

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v4
        with:
          terraform_version: "1.14.5"

      - name: Terraform init
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
        run: terraform init -input=false

      - name: Download tfplan artifact
        uses: actions/download-artifact@v4
        with:
          name: terraform-hcloud-tfplan
          path: infra/terraform/hcloud_cluster

      - name: Terraform apply
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
        run: terraform apply -input=false -lock-timeout=5m tfplan

5. Verification: Did I Get It?

Verify your local guardrails and the CI pipeline status:

# Verify local hooks are active
pre-commit run --all-files
# Check GitHub Actions for 'Waiting for approval' status on recent runs

Detailed Lessons

Hands-On Materials

Labs, quizzes, and runbooks — available to course members.

  • Lab: CI/CD Guardrails in Practice Members
  • Quiz: CI/CD & Developer Guardrails Members

The Incident: The Bypass

Result: An untested change breaks production, and responders must reconstruct the intent after the damage has already landed. Observed Symptoms What the team sees first: There is no pull request discussion for the …

Investigation & Containment

Safe investigation sequence: Verify local hooks: Identify whether local pre-commit hooks ran or were bypassed (e.g., using --no-verify). Inspect CI path: Review the GitHub Actions logs for plan, approval, and apply …

Workflow & Automation

Key Hooks: Branch Protection: master-branch-check.sh blocks direct commits to main. History Safety: prevent-amend-after-push.sh prevents rewriting shared Git history. Secret Blocking: block-secrets.sh matches dangerous …

Lab & Completion

Done When You have completed this chapter when: You can explain the “Layered Defense” model and why each layer is necessary. You have successfully installed and triggered pre-commit hooks locally. You can …