Core Track Guardrails-first chapter in core learning path.

Estimated Time

  • Reading: 20-25 min
  • Lab: 45-60 min
  • Quiz: 10-15 min

Prerequisites

Source Code References

  • .sops.yaml Members
  • sops-encrypt-secret.sh Members

Sign in to view source code.

What You Will Produce

A reproducible lab result plus quiz verification and incident-safe operating evidence.

Chapter 03: Secrets Management (SOPS + Age)

Learning Objectives

By the end of this chapter, you will be able to:

  • Encrypt secrets with SOPS and Age before committing to Git
  • Trace the Flux decryption flow from encrypted YAML to cluster Secret
  • Execute secret rotation after a leak incident
  • Explain why git revert is not remediation for an exposed secret

Start with the video for the concept overview, then work through each lesson section.

Plaintext secrets in Git are a production incident waiting to happen. In this chapter, we implement a secure GitOps secret workflow using SOPS and Age to ensure that sensitive data remains encrypted throughout its lifecycle.


1. The Problem: The Durable Leak

A teammate commits a plaintext API key to fix a failing deploy. Reverting the commit is not enough; the secret is already exposed in the Git history and CI logs. This creates a high-pressure situation requiring immediate rotation and auditing across the entire fan-out surface.

2. The Concept: Encrypted-at-Rest GitOps

We use SOPS (Mozilla Secrets Operations) and Age to keep our secrets safe. Secrets are encrypted on the developer’s machine before they are committed, and Flux decrypts them inside the cluster using a private key stored in a sops-age secret.

3. The Code: Secret Structure & Policies

Our sre/ repo defines exactly how and which files should be encrypted. The .sops.yaml policy ensures that only the relevant data fields are encrypted, keeping the YAML metadata visible for auditing.

SOPS encryption policy

creation_rules:
  # Cloudflare API token (cert-manager + external-dns)
  - path_regex: flux/secrets/cloudflare/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Auth secrets (Dex OIDC)
  - path_regex: flux/secrets/auth/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Observability secrets (shared)
  - path_regex: flux/secrets/observability/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Development environment secrets
  - path_regex: flux/secrets/develop/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Staging environment secrets
  - path_regex: flux/secrets/staging/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

  # Production environment secrets
  - path_regex: flux/secrets/production/.*\.yaml
    encrypted_regex: ^(data|stringData)$
    age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl

4. The Guardrail: Secret Encryption Helper

To make the “Safe Path” the easiest path, we use a helper script that automates the encryption process, preventing accidental plaintext commits.

Secret encryption helper

Show the secret encryption helper
#!/bin/bash
set -euo pipefail

SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"

usage() {
    cat <<EOF
Usage: $0 ENVIRONMENT SECRET_NAME

Create and encrypt a Kubernetes Secret with SOPS

ARGUMENTS:
    ENVIRONMENT     Target environment (develop, staging, production)
    SECRET_NAME     Name of the secret (e.g., backend-secrets)

EXAMPLES:
    # Create encrypted secret for develop environment
    $0 develop backend-secrets

    # Create encrypted secret for production
    $0 production backend-secrets

WORKFLOW:
    1. Creates a plaintext secret template
    2. Opens it in \$EDITOR (or vi)
    3. After you save and quit, encrypts it with SOPS
    4. Saves encrypted version to flux/secrets/ENVIRONMENT/

NOTE:
    - Make sure .sops.yaml is configured with age public key
    - Make sure SOPS and age are installed
    - The plaintext file is automatically deleted after encryption

EOF
}

create_and_encrypt() {
    local env="$1"
    local secret_name="$2"
    local secrets_dir="${REPO_ROOT}/flux/secrets/${env}"
    local output_file="${secrets_dir}/${secret_name}.yaml"
    local temp_file="${secrets_dir}/${secret_name}.yaml.tmp"

    # Validate environment
    if [[ ! -d "${secrets_dir}" ]]; then
        echo "❌ Invalid environment: ${env}"
        echo "   Available: develop, staging, production"
        exit 1
    fi

    # Check if secret already exists
    if [[ -f "${output_file}" ]]; then
        echo "⚠️  Secret already exists: ${output_file}"
        read -p "   Edit existing secret with SOPS? (y/N): " -n 1 -r
        echo
        if [[ $REPLY =~ ^[Yy]$ ]]; then
            sops "${output_file}"
            echo "✅ Secret updated"
            exit 0
        else
            exit 1
        fi
    fi

    # Create template
    cat > "${temp_file}" <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ${secret_name}
  namespace: ${env}
type: Opaque
stringData:
  # Add your secret keys here
  # Example:
  # database-url: "postgresql://user:pass@host:5432/db"
  # api-key: "your-api-key"
  # jwt-secret: "your-jwt-secret"

  # TODO: Replace with actual secret values
  example-key: "example-value"
EOF

    echo "📝 Created template: ${temp_file}"
    echo "   Opening in editor..."
    echo

    # Open in editor
    ${EDITOR:-vi} "${temp_file}"

    echo
    read -p "Encrypt this secret? (y/N): " -n 1 -r
    echo
    if [[ ! $REPLY =~ ^[Yy]$ ]]; then
        echo "❌ Cancelled"
        rm "${temp_file}"
        exit 1
    fi

    # Encrypt with SOPS
    echo "🔐 Encrypting secret..."
    sops --encrypt "${temp_file}" > "${output_file}"

    # Remove temp file
    rm "${temp_file}"

    echo "✅ Encrypted secret created: ${output_file}"
    echo
    echo "Next steps:"
    echo "  1. Add to kustomization: edit ${secrets_dir}/kustomization.yaml"
    echo "  2. Uncomment: # - ${secret_name}.yaml"
    echo "  3. Commit: git add ${output_file} && git commit -m 'Add ${secret_name} for ${env}'"
    echo "  4. Push: git push"
}

ENV="$1"
SECRET_NAME="$2"

create_and_encrypt "${ENV}" "${SECRET_NAME}"

5. Verification: Did I Get It?

Create an encrypted secret and verify that Flux can successfully apply it:

scripts/sops-encrypt-secret.sh develop backend-secrets
# Commit the file and check cluster status
kubectl -n develop get secret backend-secrets

Detailed Lessons

Hands-On Materials

Labs, quizzes, and runbooks — available to course members.

  • Lab: Encrypted Secret -> Flux Decrypt -> Cluster Apply Members
  • Quiz: Chapter 03 (Secrets Management with SOPS) Members

The Incident: The Durable Leak

Result: Response now includes rotation, audit, and cross-team coordination under pressure. Observed Symptoms What the team sees first: The deploy is unblocked, but the secret is visible in the PR diff. The same value may …

Investigation & Containment

Safe investigation sequence: Revoke immediately: Rotate the exposed credential at the source (e.g., GitHub, AWS, etc.). Identify dependencies: Identify downstream sessions, tokens, or integrations that depend on it. Map …

Safe Workflow & Tools

Generate a new key: age-keygen -o age-new.agekey. Update .sops.yaml: Replace the old public key with the new one. Re-encrypt: Run sops rotate --in-place &lt;file&gt; for every secret. Update Cluster: Delete and recreate …

Lab & Completion

Verification Checklist backend-secrets.yaml in Git is encrypted (ENC[...] values). secrets-develop Kustomization is Ready in Flux. backend-secrets exists in the cluster namespace. No plaintext values appear in the Git …