SafeOps Academy

Chapter 04: GitOps & Version Promotion

Chapter 04: GitOps & Version Promotion

Why This Chapter Exists

Production safety depends on controlled promotion, not ad-hoc rebuilds. This chapter defines one deployment model:

  • develop deploys develop-* images
  • staging deploys staging-* images
  • production deploys production-* images from explicit promotion

The Incident Hook

A team rebuilds “the same” code for production during incident pressure. The binary differs from staging due to dependency drift and build-time variance. Rollback is confusing because the promoted artifact is not the one that was tested. Time is lost proving artifact lineage instead of restoring service.

What AI Would Propose (Brave Junior)

  • “Just rebuild from main and deploy to production now.”
  • “Use mutable latest tag for speed.”

Why this sounds reasonable:

  • fast and simple under pressure
  • fewer manual steps

Why This Is Dangerous

  • Rebuild breaks artifact immutability.
  • Mutable tags destroy auditability.
  • Incident response becomes guesswork across envs.

Guardrails That Stop It

  • Promotion without rebuild: staging-* is retagged to production-*.
  • Immutable env/version tags are required.
  • Flux image automation writes all image updates to Git.
  • GitOps-first rollback via commit revert.
  • Pre-commit branch/history hooks prevent risky Git operations before promotion PRs:
    • scripts/pre-commit-master-check.sh
    • scripts/prevent-amend-after-push.sh
  • Pre-commit manifest hook validates local Flux renders before promotion PRs:
    • scripts/flux-kustomize-validate.sh

Repo Mapping

  • docs/gitops-workflow.md
  • flux/bootstrap/infrastructure/image-automation/
  • flux/apps/backend/develop/, flux/apps/backend/staging/, flux/apps/backend/production/
  • flux/apps/frontend/overlays/develop/, flux/apps/frontend/overlays/staging/, flux/apps/frontend/overlays/production/

Current Model (As Implemented)

  1. Build on service develop branch pushes develop-* image tags.
  2. Build on service main branch pushes staging-* image tags.
  3. Manual promotion workflow retags selected staging-* image to:
  • production
  • production-v<major>.<minor>.<patch>-<short_sha>-<unix_ts>
  1. Flux ImagePolicy selects latest env-matching immutable tag.
  2. Flux ImageUpdateAutomation commits updated tags to Git and reconciles.

Lab Files

  • lab.md
  • quiz.md

Done When

  • learner can explain “promotion instead of rebuild”
  • learner can verify Flux image automation across all three environments
  • learner can perform and explain GitOps-first rollback

Lab: Version Promotion and Rollback with Flux GitOps

Lab: Version Promotion and Rollback with Flux GitOps

Goal

Validate the real promotion model:

  • develop and staging auto-update from env-specific tags
  • production updates only from explicit promotion tags
  • rollback is performed through Git (preferred path)

Prerequisites

  • Flux is installed and reconciling this repository
  • namespaces exist: develop, staging, production
  • access to service CI workflows (backend/frontend repos) for promotion trigger

Step 1: Baseline Verification

flux get kustomizations -n flux-system
flux get images all -A

kubectl -n develop get deploy backend -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n staging get deploy backend -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'
kubectl -n production get deploy backend -o jsonpath='{.spec.template.spec.containers[0].image}{"\n"}'

Expected:

Quiz: Chapter 04 (GitOps & Version Promotion)

Quiz: Chapter 04 (GitOps & Version Promotion)

Questions

  1. Why is “promotion instead of rebuild” a core production guardrail?

  2. Which tag families are expected in each namespace?

  • develop:
  • staging:
  • production:

  1. What is the risk of deploying mutable latest tags to production?

  2. In this model, what triggers production image creation?

  • A) push to develop
  • B) push to main
  • C) manual promotion workflow

  1. What does Flux ImageUpdateAutomation add from an audit perspective?

  2. Where should you look first if production does not pick a newly promoted tag?