Chapter 03: Secrets Management (SOPS + Age)
Learning Objectives
By the end of this chapter, you will be able to:
- Encrypt secrets with SOPS and Age before committing to Git
- Trace the Flux decryption flow from encrypted YAML to cluster Secret
- Execute secret rotation after a leak incident
- Explain why
git revertis not remediation for an exposed secret
Start with the video for the concept overview, then work through each lesson section.
Plaintext secrets in Git are a production incident waiting to happen. In this chapter, we implement a secure GitOps secret workflow using SOPS and Age to ensure that sensitive data remains encrypted throughout its lifecycle.
1. The Problem: The Durable Leak
A teammate commits a plaintext API key to fix a failing deploy. Reverting the commit is not enough; the secret is already exposed in the Git history and CI logs. This creates a high-pressure situation requiring immediate rotation and auditing across the entire fan-out surface.
2. The Concept: Encrypted-at-Rest GitOps
We use SOPS (Mozilla Secrets Operations) and Age to keep our secrets safe. Secrets are encrypted on the developer’s machine before they are committed, and Flux decrypts them inside the cluster using a private key stored in a sops-age secret.
3. The Code: Secret Structure & Policies
Our sre/ repo defines exactly how and which files should be encrypted. The .sops.yaml policy ensures that only the relevant data fields are encrypted, keeping the YAML metadata visible for auditing.
SOPS encryption policy
creation_rules:
# Cloudflare API token (cert-manager + external-dns)
- path_regex: flux/secrets/cloudflare/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Auth secrets (Dex OIDC)
- path_regex: flux/secrets/auth/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Observability secrets (shared)
- path_regex: flux/secrets/observability/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Development environment secrets
- path_regex: flux/secrets/develop/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Staging environment secrets
- path_regex: flux/secrets/staging/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
# Production environment secrets
- path_regex: flux/secrets/production/.*\.yaml
encrypted_regex: ^(data|stringData)$
age: age1730wxdzhzpwjmf04rdf9tdz5zndr7xjq2mqk5ydkme5s27vaha7s2xnjxl
4. The Guardrail: Secret Encryption Helper
To make the “Safe Path” the easiest path, we use a helper script that automates the encryption process, preventing accidental plaintext commits.
Secret encryption helper
Show the secret encryption helper
#!/bin/bash
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
usage() {
cat <<EOF
Usage: $0 ENVIRONMENT SECRET_NAME
Create and encrypt a Kubernetes Secret with SOPS
ARGUMENTS:
ENVIRONMENT Target environment (develop, staging, production)
SECRET_NAME Name of the secret (e.g., backend-secrets)
EXAMPLES:
# Create encrypted secret for develop environment
$0 develop backend-secrets
# Create encrypted secret for production
$0 production backend-secrets
WORKFLOW:
1. Creates a plaintext secret template
2. Opens it in \$EDITOR (or vi)
3. After you save and quit, encrypts it with SOPS
4. Saves encrypted version to flux/secrets/ENVIRONMENT/
NOTE:
- Make sure .sops.yaml is configured with age public key
- Make sure SOPS and age are installed
- The plaintext file is automatically deleted after encryption
EOF
}
create_and_encrypt() {
local env="$1"
local secret_name="$2"
local secrets_dir="${REPO_ROOT}/flux/secrets/${env}"
local output_file="${secrets_dir}/${secret_name}.yaml"
local temp_file="${secrets_dir}/${secret_name}.yaml.tmp"
# Validate environment
if [[ ! -d "${secrets_dir}" ]]; then
echo "❌ Invalid environment: ${env}"
echo " Available: develop, staging, production"
exit 1
fi
# Check if secret already exists
if [[ -f "${output_file}" ]]; then
echo "⚠️ Secret already exists: ${output_file}"
read -p " Edit existing secret with SOPS? (y/N): " -n 1 -r
echo
if [[ $REPLY =~ ^[Yy]$ ]]; then
sops "${output_file}"
echo "✅ Secret updated"
exit 0
else
exit 1
fi
fi
# Create template
cat > "${temp_file}" <<EOF
apiVersion: v1
kind: Secret
metadata:
name: ${secret_name}
namespace: ${env}
type: Opaque
stringData:
# Add your secret keys here
# Example:
# database-url: "postgresql://user:pass@host:5432/db"
# api-key: "your-api-key"
# jwt-secret: "your-jwt-secret"
# TODO: Replace with actual secret values
example-key: "example-value"
EOF
echo "📝 Created template: ${temp_file}"
echo " Opening in editor..."
echo
# Open in editor
${EDITOR:-vi} "${temp_file}"
echo
read -p "Encrypt this secret? (y/N): " -n 1 -r
echo
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
echo "❌ Cancelled"
rm "${temp_file}"
exit 1
fi
# Encrypt with SOPS
echo "🔐 Encrypting secret..."
sops --encrypt "${temp_file}" > "${output_file}"
# Remove temp file
rm "${temp_file}"
echo "✅ Encrypted secret created: ${output_file}"
echo
echo "Next steps:"
echo " 1. Add to kustomization: edit ${secrets_dir}/kustomization.yaml"
echo " 2. Uncomment: # - ${secret_name}.yaml"
echo " 3. Commit: git add ${output_file} && git commit -m 'Add ${secret_name} for ${env}'"
echo " 4. Push: git push"
}
ENV="$1"
SECRET_NAME="$2"
create_and_encrypt "${ENV}" "${SECRET_NAME}"
5. Verification: Did I Get It?
Create an encrypted secret and verify that Flux can successfully apply it:
scripts/sops-encrypt-secret.sh develop backend-secrets
# Commit the file and check cluster status
kubectl -n develop get secret backend-secrets