GitOps with Argo CD: Declarative Delivery, Drift Detection, Sync Waves, and Safe Rollbacks

GitOps makes a Git repository the source of truth for what runs in a cluster, and a controller inside the cluster keeps reality matched to it. Argo CD is the most widely used controller for this model. The hard parts are not the install. They are repository layout, deciding what the controller may delete, ordering changes that depend on each other, keeping secrets out of plain text, and getting back to a good state when a release fails.

Pull vs push delivery

In push delivery, a CI job holds cluster credentials and runs kubectl apply or helm upgrade. The pipeline knows what it applied, but the cluster state is only correct at the moment the job ends. Anything changed afterwards goes unnoticed, and the CI system becomes a high-value target because it can write to production.

In pull delivery, a controller running next to the workload compares the desired state in Git with the live state and reconciles the difference. CI's job ends when it has built an image and updated a manifest. The practical differences:

Two caveats. A single Argo CD instance managing many remote clusters is pull from the point of view of Git, but it is push from the point of view of those clusters: the hub stores credentials for every spoke and calls their API servers. That hub is your new high-value target. Running one instance per cluster removes this at the cost of more instances to operate.

Second, GitOps does not remove CI. Tests, image builds, signing, and scanning still happen there. Only the last step changes, from "apply to the cluster" to "commit a new desired state".

Repository layout: app repo vs config repo

Keep application source and deployment configuration in separate repositories unless the service is small and owned end to end by one team. The reasons are concrete:

• A config-only change, such as raising a replica count, should not rerun the application test suite.
• A CI bot that commits image tags back to the application repository creates noisy history and can trigger itself in a loop.
• Write access to production config can be narrower than write access to the code.

The cost is a two-step change when code and config must move together, such as a new required environment variable. Handle this by making the application tolerate both states during the transition.

Use directories for environments, not branches. Branch-per-environment looks tidy, but environments drift apart through cherry-picks, and merges between them carry changes you did not mean to promote. A directory diff shows exactly how staging and production differ.

example-repo-config/
  apps/
    checkout/
      base/
        deployment.yaml
        service.yaml
        kustomization.yaml
      overlays/
        staging/
          kustomization.yaml
        production/
          kustomization.yaml
          replicas-patch.yaml
  platform/
    ingress-nginx/
    external-secrets/
  projects/
    payments.yaml
  bootstrap/
    root-app.yaml
    applicationset-apps.yaml

Application and ApplicationSet

An Application binds a source (repository, revision, path) to a destination (cluster, namespace) under an AppProject. Here is a production application with an explicit, conservative sync policy:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-production
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: payments
  source:
    repoURL: https://git.example.com/example-org/example-repo-config.git
    targetRevision: main
    path: apps/checkout/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout-production
  revisionHistoryLimit: 10
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true
      - RespectIgnoreDifferences=true
      - FailOnSharedResource=true
    retry:
      limit: 3
      backoff:
        duration: 30s
        factor: 2
        maxDuration: 5m
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

The finalizer means deleting the Application also deletes its cluster resources. Without it, deleting the object leaves the workload running but unmanaged. Choose deliberately. For stateful platform components, omitting the finalizer can be the safer choice.

Helm charts work through the same object, with one thing to know: Argo CD renders charts with helm template and applies the output itself. There is no Helm release in the cluster, so helm list shows nothing, and Helm hooks are mapped onto Argo CD hooks where an equivalent exists.

Generating applications

Writing one Application per service per environment does not scale. An ApplicationSet generates them from a template and one or more generators. The Git directory generator below creates an application for every overlay, so adding a directory is how you onboard a service:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payments-apps
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: https://git.example.com/example-org/example-repo-config.git
        revision: main
        directories:
          - path: apps/*/overlays/*
  syncPolicy:
    preserveResourcesOnDeletion: true
    applicationsSync: create-update
  template:
    metadata:
      name: '{{index .path.segments 1}}-{{.path.basename}}'
      labels:
        env: '{{.path.basename}}'
    spec:
      project: payments
      source:
        repoURL: https://git.example.com/example-org/example-repo-config.git
        targetRevision: main
        path: '{{.path.path}}'
      destination:
        server: https://kubernetes.default.svc
        namespace: '{{index .path.segments 1}}-{{.path.basename}}'
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

The dangerous property of an ApplicationSet is that a generator returning fewer results means applications get deleted. Rename a directory, or merge a bad change to the generator, and the controller may remove applications and, through their finalizers, the workloads. That is why the example sets preserveResourcesOnDeletion: true and applicationsSync: create-update, which stops the controller deleting generated applications at all. You then clean up by hand, which is the right trade for production. A policy set on the controller itself takes precedence over this field unless override is enabled, so check how your instance is configured.

Preview what a change generates before merging it. The CLI can render an ApplicationSet from a file with argocd appset generate applicationset-apps.yaml, and the output should match what you expect item for item.

Sync policies, self-heal, and prune

Argo CD reports two independent states. Sync status (Synced, OutOfSync, Unknown) says whether the live objects match Git. Health status (Healthy, Progressing, Degraded, Suspended, Missing, Unknown) says whether they work. A Synced but Degraded application is common: the manifest applied perfectly and the pods are crash-looping.

Reasonable defaults: automated sync with prune and self-heal in development and staging. In production, decide per application. Many teams keep it on there too, because the review gate belongs on the pull request, and a manual sync button adds ceremony without adding review. Others keep manual sync for stateful or high-risk applications.

For individual resources that must survive pruning, such as a PersistentVolumeClaim, annotate them:

metadata:
  annotations:
    argocd.argoproj.io/sync-options: Prune=false

How drift detection works

The application controller renders manifests through the repo server, diffs them against the live state, and refreshes on a timer (timeout.reconciliation in argocd-cm, a few minutes by default). A webhook from the Git host to /api/webhook makes syncs near-immediate; without one, a merge can sit unnoticed until the next refresh.

Real clusters produce false drift. An autoscaler owns spec.replicas, a mutating webhook injects a sidecar, a controller fills in caBundle. Handle these with ignoreDifferences, as in the application above. Note the pairing with RespectIgnoreDifferences=true: without it, the field is ignored in the diff but a sync still writes the value from Git, so the autoscaler's replica count gets reset on each deploy. The cleanest fix for replicas is to remove the field from the manifest altogether. Then Git has no opinion.

Server-side diff (argocd.argoproj.io/compare-options: ServerSideDiff=true) reduces false positives by having the API server compute the expected result, including defaulting.

Self-heal also needs a break-glass procedure. If an on-call engineer must edit a live Deployment during an incident, automated sync will undo it. Agree the steps ahead of time:

# Stop reconciliation for one application
argocd app set checkout-production --sync-policy none

# ...apply the emergency change, commit the real fix to Git...

# Restore automation
argocd app set checkout-production --sync-policy automated --self-heal --auto-prune

If the Application itself is managed from Git by a parent or an ApplicationSet, the parent will restore the policy. Pause at the parent level, or use a sync window, covered below.

Sync waves and hooks

Argo CD orders a sync by phase, then wave, then kind, then name. The phases are PreSync, Sync, and PostSync, with SyncFail running on failure. Inside a phase, resources are grouped by the argocd.argoproj.io/sync-wave annotation, which defaults to 0 and may be negative. Lower waves go first, and the controller waits for a wave to be healthy before moving on.

That wait is what makes waves more than sorting. A typical plan:

Hooks are resources that run at a phase rather than being kept in sync. This one runs a migration before the rollout:

apiVersion: batch/v1
kind: Job
metadata:
  name: checkout-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation,HookSucceeded
spec:
  backoffLimit: 1
  activeDeadlineSeconds: 600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/myproject/checkout:1.42.0
          command: ["/srv/app/bin/migrate", "up"]
          envFrom:
            - secretRef:
                name: checkout-db

Details that bite here:

• PreSync runs before the Sync phase applies anything. If this release adds a Secret or ConfigMap the hook needs, it does not exist yet and the Job fails. Either move the migration into the Sync phase with a negative wave, as in the table, or give the config a PreSync hook with an earlier wave.
• Jobs are mostly immutable. BeforeHookCreation deletes the previous run so a new one can be created with the same name. Without a delete policy, a named hook fails on the second sync.
• A hook with no deadline can hang a sync. Set activeDeadlineSeconds. If a sync is stuck on a hook, argocd app terminate-op checkout-production stops the operation.
• Schema migrations do not roll back with Argo CD. Git history reverts manifests, not data. Use expand and contract: add columns in one release, remove old ones in a later release, so the previous image version still runs against the new schema.

Ordering across applications

Waves also work between child applications in an app-of-apps, but only if the parent can see child health. Argo CD does not assess Application health by default, so a parent would move on as soon as the child object exists. Add this health customization to argocd-cm:

data:
  resource.customizations.health.argoproj.io_Application: |
    hs = {}
    hs.status = "Progressing"
    hs.message = ""
    if obj.status ~= nil then
      if obj.status.health ~= nil then
        hs.status = obj.status.health.status
        if obj.status.health.message ~= nil then
          hs.message = obj.status.health.message
        end
      end
    end
    return hs

Two related problems: a custom resource and its CRD in the same sync fail dry-run because the kind does not exist yet. Put the CRD in an earlier wave and add SkipDryRunOnMissingResource=true to the custom resource. CRDs that exceed the annotation size limit of client-side apply need ServerSideApply=true.

Secrets: Sealed Secrets, External Secrets, and alternatives

Plain Kubernetes Secrets cannot go into Git, and base64 is encoding, not protection. Two patterns dominate.

Sealed Secrets stores ciphertext in Git. A controller holds a private key in the cluster and decrypts SealedSecret objects into normal Secrets.

# Fetch the public certificate once, then seal offline
kubeseal --fetch-cert \
  --controller-namespace kube-system \
  --controller-name sealed-secrets > pub-cert.pem

kubectl create secret generic checkout-db \
  --namespace checkout-production \
  --from-literal=password='change-me' \
  --dry-run=client -o yaml \
  | kubeseal --cert pub-cert.pem --format yaml \
  > apps/checkout/overlays/production/checkout-db-sealed.yaml

The default strict scope binds the ciphertext to the name and namespace, so it cannot be copied elsewhere and decrypted. The controller renews its sealing key periodically but keeps old keys, so old ciphertext still decrypts. That is not rotation of the secret itself: anyone who once read the value still knows it. Back up the key material, because a rebuilt cluster with a new key cannot decrypt anything in the repository.

External Secrets Operator stores only a reference in Git. The operator reads from a secret manager and writes a Kubernetes Secret.

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: https://vault.example.com
      path: secret
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: external-secrets
          serviceAccountRef:
            name: external-secrets
            namespace: external-secrets
---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-db
  namespace: checkout-production
  annotations:
    argocd.argoproj.io/sync-wave: "-2"
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: checkout-db
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: payments/checkout/db
        property: password

If a secret manager already exists, use External Secrets. Sealed Secrets suits small estates or air-gapped clusters. In both cases, remember that a value changing in a Secret does not restart pods reading it as environment variables. You need a rollout trigger, such as a reloader controller or a checksum annotation, or the rotation appears to work while the old password stays in use.

With External Secrets, an outage in the secret manager does not remove existing Secrets, but new namespaces and rotated values fail. Monitor the ExternalSecret status condition rather than assuming the sync result covers it.

Promotion across environments

Promotion in this model is a pull request that changes one value in one overlay. Build once, deploy that same artifact everywhere, and refer to it by digest so a retagged image cannot alter what is deployed.

# apps/checkout/overlays/staging/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: checkout-staging
resources:
  - ../../base
images:
  - name: registry.example.com/myproject/checkout
    digest: sha256:9f2c1e7ab3d04c5688e1f0a2b7c6d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0

A CI step that opens the promotion after staging is verified:

#!/usr/bin/env bash
set -euo pipefail

DIGEST="$1"
BRANCH="promote/checkout-${DIGEST:7:12}"

git switch -c "$BRANCH"
cd apps/checkout/overlays/production
kustomize edit set image \
  "registry.example.com/myproject/checkout@${DIGEST}"
cd -
git add apps/checkout/overlays/production/kustomization.yaml
git commit -m "promote checkout to production: ${DIGEST}"
git push origin "$BRANCH"
gh pr create --fill --base main --label promotion

The review then shows a one-line diff. Keep changes to the base small and separate from version promotions, because a base edit hits every environment at the same moment and bypasses the staged rollout. One way to avoid that is versioned bases, where each overlay refers to a tagged revision of the base.

For rollout within an environment across clusters, the ApplicationSet RollingSync strategy syncs labelled groups in steps. It is an optional feature that must be enabled on the controller, and it takes over sync timing from auto-sync on generated applications. Argo Rollouts addresses a different layer: canary and blue-green traffic shifting inside a single sync.

RBAC and projects

Treat the default project as unsafe: it allows any repository to deploy anywhere. Each team gets an AppProject that limits sources, destinations, and resource kinds.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: payments
  namespace: argocd
spec:
  description: Payments team workloads
  sourceRepos:
    - https://git.example.com/example-org/example-repo-config.git
  destinations:
    - server: https://kubernetes.default.svc
      namespace: checkout-*
  clusterResourceWhitelist: []
  namespaceResourceBlacklist:
    - group: ""
      kind: ResourceQuota
    - group: ""
      kind: LimitRange
  syncWindows:
    - kind: deny
      schedule: "0 22 * * 5"
      duration: 58h
      applications:
        - "*-production"
      manualSync: true
  roles:
    - name: ci-sync
      description: Allows CI to trigger and watch syncs
      policies:
        - p, proj:payments:ci-sync, applications, get, payments/*, allow
        - p, proj:payments:ci-sync, applications, sync, payments/*, allow

An empty clusterResourceWhitelist blocks cluster-scoped resources, which matters because a team that can create a ClusterRoleBinding through Argo CD effectively has cluster admin. The sync window denies automatic production syncs over a weekend while manualSync: true still lets a person push a fix through.

Instance-wide RBAC sits in argocd-rbac-cm and maps SSO groups to roles:

data:
  policy.default: role:readonly
  scopes: "[groups]"
  policy.csv: |
    p, role:payments-deployer, applications, get, payments/*, allow
    p, role:payments-deployer, applications, sync, payments/*, allow
    p, role:payments-deployer, applications, action/*, payments/*, allow
    p, role:payments-deployer, logs, get, payments/*, allow
    g, payments-engineers, role:payments-deployer

Actions to withhold unless needed: applications, update (can change source and destination, which sidesteps review), applications, override (can sync from a local or arbitrary revision), and exec, create (shell in pods). Also consider disabling the local admin account after SSO works. Since 3.0, update and delete on an application no longer implicitly cover its managed resources, and log access needs its own grant, so policies copied from older guides may behave differently.

Remember that the real authorization boundary in GitOps is Git. Whoever can merge to the config repository's main can deploy. Protect it with required reviews and code owners per directory, and tie Argo CD project permissions to the same teams.

Rollback

There are two ways back, and they are not equal.

Revert in Git. This is the default. The history stays honest, the review path is the same, and the fix survives the next reconcile.

git revert --no-edit 3f1a9c2
git push origin main
argocd app wait checkout-production --sync --health --timeout 300

Rollback in Argo CD. This redeploys an earlier entry from sync history. It is faster when Git or CI is unavailable, but the cluster now disagrees with Git. Argo CD refuses it while automated sync is on, because the next reconcile would reverse it, so disable automation first:

argocd app history checkout-production
argocd app set checkout-production --sync-policy none
argocd app rollback checkout-production 14

# Then land the matching revert in Git and re-enable automation
argocd app set checkout-production --sync-policy automated --self-heal --auto-prune

The history kept is bounded by revisionHistoryLimit. If the application is generated from an ApplicationSet or parent, restore any overridden policy through Git. Note that history rollback does not skip hooks: a PreSync migration in that revision runs again, which is another reason migrations must be idempotent.

Failure modes

If the controller is down, nothing reconciles but workloads keep running, which is a benign default. Alert on sync failures: argocd_app_info exposes sync_status and health_status labels, and notification triggers close the gap where a merge looks successful and the deploy failed.

Verifying it works

Run these against a non-production application. Each one tests a claim the setup makes.

1. Renders correctly. Run kustomize build apps/checkout/overlays/staging in CI on every pull request, plus argocd appset generate for ApplicationSets.
2. Diff is clean. argocd app diff checkout-staging exits 0 when there is no difference.
3. Self-heal works. Change a live field and watch it revert.
4. Prune works, and is bounded. Remove a test ConfigMap from Git and confirm deletion. Confirm that a resource marked Prune=false stays.
5. Failed sync is visible. Merge a bad image reference and check that the alert fires.
6. Rollback is rehearsed. Revert in Git, and run the history rollback once, on purpose.
7. RBAC holds. Test a denied action as well as an allowed one.
8. Restore works. Decrypt or fetch a secret in a fresh namespace or cluster using your backed-up keys.

# Overall state
argocd app get checkout-staging
argocd app wait checkout-staging --sync --health --timeout 300

# Drift test: expect the deployment to return to the Git-defined value
kubectl -n checkout-staging set env deployment/checkout DRIFT_TEST=1
sleep 30
kubectl -n checkout-staging get deployment checkout \
  -o jsonpath='{.spec.template.spec.containers[0].env}'

# RBAC: first should say yes, second should say no
argocd admin settings rbac can role:payments-deployer sync applications \
  'payments/checkout-staging' --policy-file policy.csv
argocd admin settings rbac can role:payments-deployer delete applications \
  'payments/checkout-staging' --policy-file policy.csv

# Metrics: applications not synced or not healthy
curl -s http://10.0.0.20:8082/metrics | grep '^argocd_app_info' \
  | grep -v 'health_status="Healthy"'

# Secrets synced
kubectl -n checkout-staging get externalsecret checkout-db \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'