Argo CD at Scale — Sharding, Performance Tuning, Failure Modes and Mistakes

Scaling one Argo CD instance to hundreds of clusters: controller sharding, repo-server and Redis tuning, and the metrics behind stale sync status. Also the failure modes and mistakes (prunes, cascading deletes, self-heal reverts) that take production down, with recovery and guardrails.

Overview

Argo CD works out of the box for a handful of clusters. At a few hundred clusters and thousands of Applications, three things start to hurt: the application controller cannot finish a reconciliation pass before the next one is due, the repo-server runs out of memory or time generating manifests, and the Kubernetes API client throttles itself. The user-facing symptom is the same for all three: sync status that is minutes old, syncs that queue, and dashboards that disagree with the cluster.

The second class of incidents is not about scale at all. Argo CD does exactly what Git and its configuration say, quickly and everywhere. A commit that removes manifests, an Application deleted with its finalizer still on, an ApplicationSet whose generator stops matching a cluster, or a hotfix made with kubectl while self-heal is on: each is the controller working as designed against a mistake.

> [!NOTE] > Defaults below are from the current Argo CD documentation (3.x). Where a feature arrived in a specific release, the version is named. Most controller and repo-server flags shown as --flag also exist as keys in the argocd-cmd-params-cm ConfigMap.

Symptom to first command

Sync status minutes old across many apps — argocd admin cluster shards — Scaling the application controller
One controller pod hot, others idle — argocd admin cluster stats --sharding-method round-robin — Scaling the application controller
Logs show Waited for ... due to client-side throttling — kubectl logs -n argocd argocd-application-controller-0 \ — grep throttling — Scaling the application controller
ComparisonError with Context deadline exceeded — kubectl top pod -n argocd -l app.kubernetes.io/name=argocd-repo-server — Repo-server, Redis and monorepos
Repo-server pods OOMKilled after a monorepo commit — kubectl describe pod -n argocd -l app.kubernetes.io/name=argocd-repo-server — Repo-server, Redis and monorepos
App OutOfSync right after a successful sync — argocd app diff APPNAME — OutOfSync loops and refresh storms
Application stuck deleting — kubectl get app APPNAME -n argocd -o jsonpath='{.metadata.finalizers}' — Stuck deletions, hooks and timeouts
Production resources gone after a merge — argocd app history APPNAME — Mistakes

What you will be able to do

Explain what each Argo CD component does and which one limits throughput at hundreds of clusters.
Shard the application controller, choose a sharding algorithm, and check the resulting distribution with argocd admin cluster shards.
Tune processors, Kubernetes client QPS and burst, reconciliation timeout, repo-server parallelism and caching, with the trade-off of each.
Read the controller and repo-server metrics and explain why sync status is stale.
Diagnose OutOfSync loops, stuck deletions, failing hooks and repo-server timeouts.
Recover from an auto-sync prune or a cascading delete, and put guardrails in place: AppProjects, sync windows, RBAC, Prune=false, Prune=confirm, Delete=false and preserveResourcesOnDeletion.

Key concepts

Controller shard: One application-controller replica responsible for a fixed set of clusters. Enabled by StatefulSet replicas plus the matching ARGOCD_CONTROLLER_REPLICAS value.
Sharding algorithm: legacy (default, uid based, non-uniform), round-robin (equal, experimental, documented from 2.8) or consistent-hashing (bounded loads, experimental, documented from 2.12).
Status and operation processors: Workers for the reconciliation queue (default 20) and the sync queue (default 10) in each controller replica.
Reconciliation timeout: How often every Application is refreshed from Git: timeout.reconciliation 120s plus up to 60s of jitter, so 3 minutes by default.
Manifest cache: Generated manifests cached by commit SHA in Redis for 24h by default; a new commit invalidates every Application in that repository unless manifest-generate-paths narrows it.
ignoreDifferences: Application or system setting that excludes fields (JSON pointer, jq expression or field manager) from the diff; RespectIgnoreDifferences=true also skips them during sync.
Resources finalizer: resources-finalizer.argocd.argoproj.io on an Application makes its deletion cascade to every managed resource (foreground by default, /background variant available).
Prune: Deleting live resources that are no longer in Git. Off by default for automated sync; controlled per resource with Prune=false or Prune=confirm.
Self-heal: Automated sync that also reverts live changes that drift from Git. The auto-sync docs say after a 5 second self-heal timeout; the controller reference shows timeout 0 with exponential backoff (2 s, factor 3, cap 300 s).
Sync window: A cron-scheduled allow or deny period on an AppProject that gates syncs for matching applications, namespaces or clusters; deny wins.

When to use it

One Argo CD instance manages tens to hundreds of clusters and sync status lags behind reality.
Repo-server pods are OOMKilled or apps show Context deadline exceeded after commits to a shared repository.
Applications flap between Synced and OutOfSync, or auto-sync runs over and over for the same app.
You are recovering from an auto-sync prune, a cascading delete or an ApplicationSet change that removed workloads.
You are designing guardrails (projects, RBAC, sync windows, sync options) before turning on auto-sync with prune in production.
You are preparing for a platform, SRE or DevOps interview that covers GitOps at scale.

Interview questions

You're running argocd centrally managing 200+ spoke clusters. The application controller is falling behind on reconciliation and applications are showing stale sync status. How do you actually scale this?
An engineer merged a commit that removed manifests and Argo CD auto-sync pruned production resources. How do you recover and what guardrails prevent it?
Why would sharding not help, even with 10 controller replicas?
Compare the legacy, round-robin and consistent-hashing sharding algorithms. Which would you choose?
Repo-server pods keep getting OOMKilled after commits to your monorepo. What do you do?
An Application is OutOfSync seconds after every successful sync and self-heal keeps re-syncing it. Walk through the fix.
An Application has been stuck deleting for an hour. What is going on and how do you clear it safely?
Someone deleted the parent Application in an app-of-apps setup. What happens, and how do you design so it cannot take production down?
During an incident an engineer ran kubectl rollout undo and the bad version came back a few seconds later. Explain and give the right procedure.
How do you stop an Application from deploying to the wrong cluster or namespace?