DevOpsInterviews logo
← CI/CD, Release Engineering & GitOps / 16
hard★ EssentialRed HatRazorpaySalesforce

A teammate kubectl-applied a fix during an incident and Argo CD reverted it a minute later. What happened?

Self-heal did exactly what it was configured to do. The response needs an authorized emergency path, a controlled reconciliation pause when required, and evidence that automation can resume safely.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: The application runs with automated sync and self-heal enabled, so the controller diffed live state against Git, saw the manual patch as drift, and restored the declared state. Git never saw the fix. Use a fast authorized Git path or a bounded automation pause, then verify reconciliation and service recovery separately.

How to approach it

Name the mechanism in your first sentence: automated sync plus self-heal did this on purpose. Then pivot to design, because the real question is whether your incident process has a first-class Git lane.

A strong answer

Repository refresh timing and live-state self-heal are separate settings. Argo CD compares desired and live state and marks managed differences OutOfSync; a Git webhook speeds detection of a repository change, not every kind of drift. With automated.selfHeal enabled it acts on those differences: a field someone patched by hand gets restored to the manifest, and pruning can delete an application-tracked resource that is absent from the desired manifests. An unrelated, untracked resource is not automatically owned or pruned merely because it shares the namespace. Nothing mysterious happened to your teammate. The controller enforced exactly the invariant you asked for, against a fix that existed only in the cluster's memory.

Why the fix you applied during the incident disappeared WHAT AUTOMATED SYNC DOES again, on every refresh 1 Git holds desired state the source of truth now 2 Controller refreshes on a period, or a webhook 3 Desired against live for tracked resources only 4 Difference: OutOfSync including the one you made 5 selfHeal restores it back to the manifest value 6 Prune removes extras tracked, but absent from Git 7 Verify reconciliation matching state can still fail users Repository refresh timing and live-state self-heal are separate settings. A Git webhook speeds up detection of a change in the repository, not detection of every kind of drift. Nothing mysterious happened to your teammate. The fix existed only in the cluster's memory, and the controller enforced exactly the invariant somebody configured it to enforce. Prune only touches resources the Application tracks. An unrelated object is not owned or removed just for sharing the namespace. Git records intent; inspect live state too. Reconciliation can fail or remain blocked. A matching manifest does not prove that the service has recovered for its users.

So the design work is making sure going around Git never looks attractive:

An expedited lane for incidents. Use an incident-specific authorization path for a small config change, with the operator, approval and incident reference recorded. A Git commit preserves the diff; it does not by itself prove who authorized production access. Rehearse the path and measure its elapsed time.

Sync waves with health gates. Annotate resources with argocd.argoproj.io/sync-wave: negative for namespaces and config, then migration Jobs, then Deployments, then Ingress. Each wave pauses until the previous one reports Healthy, using built-in health checks for known kinds and Lua checks for CRDs. Without ordering, a rollout racing its ConfigMap is how self-healing turns into a flap loop that makes everyone distrust the controller.

Documented break-glass. Temporarily disable auto-sync on that application for the duration, where drift detection keeps reporting so the exception stays visible, then commit the emergency state or revert the patch before restoring automation. For generated Applications, change the controlling ApplicationSet policy as needed rather than fighting its reconciliation. IgnoreExtraneous changes sync-status calculation only: it is not a self-heal pause or prune exemption. Argo CD documents that distinction.

Where I would reverse the whole stance: if merging to the config repo takes fifteen minutes because every change queues behind human review of a shared monorepo, engineers will keep patching clusters, and self-heal becomes a fight you lose nightly. Fix the lane first, keep the enforcement second. Enforcement without a fast legitimate path produces shadow operations, which is worse than either alternative alone.

Git records desired state; the cluster records what is actually running. During an incident, inspect both and establish why they differ. A failed reconciliation can leave that difference unresolved indefinitely.

Incident drill: preserve the mitigation through reconciliation

In a fictional outage, an operator changes a Deployment's connection-pool limit from 100 to 30. Argo CD restores 100, and database saturation returns. First inspect the Application diff and controller operation history to establish which controller changed the field. Another operator or admission component could also be involved; the observed timing alone is insufficient.

If Git is available, commit the bounded change through the incident authorization path, sync the intended revision and verify both the live value and database recovery. If that path is unavailable, suspend the relevant automation deliberately, record who owns the exception and apply the mitigation. Confirm no parent controller immediately restores the previous automation policy.

Before re-enabling self-heal, reconcile the intended Git revision with the live change. Verify that generated manifests contain 30 and that the controller finishes successfully. Then check application behaviour again. A Synced label establishes agreement over compared resources; it does not prove that requests meet the service objective. Keep the incident open if the configuration matches but users still fail.

What interviewers probe next

"Manual versus automatic sync?" Choose from change risk and the required approval model. Automated reconciliation can suit a database with tested migrations; a stateless service can still require an approval window. Drift detection runs regardless, so even manual-sync apps surface divergence.

"Hooks versus waves?" Waves order resources within a sync phase, including hooks. Hooks run one-shot work: a PreSync migration Job before anything changes, PostSync smoke tests whose failure marks the sync failed. Sync phases do not supply an automatic rollback phase; restoration needs an explicit, rehearsed procedure or another rollout controller.

"Pruning worries you?" It should, it is how a renamed manifest deletes the old object. Scope it per application, enable it deliberately, rehearse it in staging first.

Common mistakes

Disabling self-heal without an owner, expiry and reconciliation plan for the exception.

Conflating self-heal with prune: one reverts modified fields, the other deletes extraneous objects, and they fail differently.

Claiming GitOps means nobody ever touches the cluster, which is false. It means cluster edits become deliberate, visible exceptions instead of silent ones.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.