If you’ve ever shipped a bad release straight to 100% of your users and then spent the next hour frantically rolling back while Slack lights up with angry messages, you already understand why canary deployments exist. A plain Kubernetes Deployment gives you rolling updates, but it has no idea whether your new version is actually healthy — it just keeps replacing pods on a timer and hopes for the best. Argo Rollouts fixes that gap by giving you a real progressive delivery controller: gradual traffic shifting, automated metric analysis, and one-click rollback, all as native Kubernetes resources.

In this guide, we’re going to go far beyond a “hello world” canary. We’ll build a production-grade setup: a Rollout backed by real traffic-shaping through a service mesh, automated analysis using Prometheus metrics, manual and automatic promotion gates, and a CI/CD pipeline that ties it all together. By the end, you’ll have a canary pipeline you could actually trust with a real production service.

What You’ll Learn

By the end of this tutorial, you will know how to:

  • Install and configure Argo Rollouts on a Kubernetes cluster
  • Convert an existing Deployment into a Rollout resource
  • Define canary steps with weighted traffic shifting and pauses
  • Wire up an AnalysisTemplate that queries Prometheus and automatically aborts bad releases
  • Integrate Argo Rollouts with a traffic-management layer (Istio and NGINX Ingress examples)
  • Use the kubectl argo rollouts plugin and dashboard to watch a canary roll out live
  • Automate promotion and rollback inside a CI/CD pipeline
  • Apply production best practices around resource limits, RBAC, and failure handling

Prerequisites

Before we start, make sure you have:

  • A running Kubernetes cluster (1.24+), local (kind/minikube) or cloud-managed
  • kubectl configured to talk to your cluster
  • Helm 3 installed
  • Basic familiarity with Kubernetes Deployments and Services
  • (Optional but recommended) Prometheus already running in-cluster for metric-based analysis
  • (Optional) Istio or NGINX Ingress Controller if you want real traffic splitting rather than replica-based approximation

Why Not Just Use a Regular Deployment?

A standard rolling update shifts traffic by replacing pods, which only approximates a percentage split, and it has zero concept of “is this actually working.” If your new pods are crash-looping or leaking memory, Kubernetes will happily keep rolling forward. Argo Rollouts introduces a custom resource, the Rollout, that is a drop-in superset of Deployment but adds:

  • Weighted canary steps — send exactly 5%, then 20%, then 50% of traffic to the new version
  • Automated analysis — query Prometheus, Datadog, New Relic, or a custom webhook after each step, and abort automatically if error rates or latency spike
  • Native traffic routing integration — plug into Istio VirtualServices, NGINX canary annotations, AWS ALB, SMI, or Traefik for real percentage-based splitting instead of pod-count approximation
  • Instant rollback — abort a bad rollout and Argo Rollouts snaps traffic back to the stable ReplicaSet in seconds

Installing Argo Rollouts

Install the controller and CRDs into a dedicated namespace:

1
2
3
kubectl create namespace argo-rollouts

kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

Verify the controller is running:

1
kubectl get pods -n argo-rollouts

You should see the argo-rollouts deployment pod in Running state. Next, install the kubectl-argo-rollouts plugin, which gives you a live terminal dashboard for watching rollouts in progress:

1
2
3
4
5
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
sudo mv kubectl-argo-rollouts-linux-amd64 /usr/local/bin/kubectl-argo-rollouts

kubectl argo rollouts version

Converting a Deployment into a Rollout

Here’s a typical Deployment for a service called checkout-api:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 5
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: myregistry/checkout-api:1.0.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi

To turn this into a canary-capable Rollout, change kind to Rollout, and add a strategy block. The apiVersion changes to argoproj.io/v1alpha1:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: checkout-api
spec:
  replicas: 5
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: checkout-api
          image: myregistry/checkout-api:1.0.0
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 512Mi
  strategy:
    canary:
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

That’s the minimum viable canary: 5% of traffic for 2 minutes, then 20% for 5 minutes, then 50% for 5 minutes, then full rollout. Every pause without a duration waits indefinitely for a human to run kubectl argo rollouts promote.

Apply it, then trigger a new version by updating the image:

1
2
3
kubectl apply -f checkout-api-rollout.yaml

kubectl argo rollouts set image checkout-api checkout-api=myregistry/checkout-api:1.1.0

Watch it progress live:

1
kubectl argo rollouts get rollout checkout-api --watch

You’ll see a real-time ASCII visualization of the stable and canary ReplicaSets, their weights, and pod health.

Real Traffic Splitting with Istio

Replica-count-based canaries are a decent approximation but not exact — with 5 replicas, you can’t actually hit “5% of traffic.” For precise, production-grade traffic shifting, integrate with a service mesh. Here’s the Istio setup.

First, define the two Services Argo Rollouts will manage (stable and canary):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: v1
kind: Service
metadata:
  name: checkout-api-stable
spec:
  selector:
    app: checkout-api
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: checkout-api-canary
spec:
  selector:
    app: checkout-api
  ports:
    - port: 80
      targetPort: 8080

Then a VirtualService that Argo Rollouts will patch automatically as weights shift:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: checkout-api
spec:
  hosts:
    - checkout-api
  http:
    - route:
        - destination:
            host: checkout-api-stable
          weight: 100
        - destination:
            host: checkout-api-canary
          weight: 0

Now update the Rollout’s strategy to reference the Istio traffic routing:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
spec:
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        istio:
          virtualService:
            name: checkout-api
            routes:
              - primary
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

From here on, Argo Rollouts edits the VirtualService weights directly at every step — no approximation, exact percentage-based traffic splitting.

If you’re on NGINX Ingress instead of Istio, the equivalent block is trafficRouting.nginx.stableIngress: checkout-api-ingress, and Argo Rollouts manages the nginx.ingress.kubernetes.io/canary-weight annotation for you automatically.

Other Traffic Routing Providers

Istio and NGINX are the most common, but Argo Rollouts supports several other production traffic layers. Pick whichever matches your existing infrastructure — you don’t need to adopt a mesh just to get canary support.

AWS ALB Ingress Controller — Argo Rollouts manages weighted target groups behind an Application Load Balancer directly:

1
2
3
4
5
6
7
8
9
spec:
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        alb:
          ingress: checkout-api-ingress
          servicePort: 80

SMI (Service Mesh Interface) — useful for Linkerd or other SMI-compatible meshes:

1
2
3
4
5
6
7
8
9
spec:
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        smi:
          rootService: checkout-api
          trafficSplitName: checkout-api-split

Traefik — via the TraefikService CRD:

1
2
3
4
5
6
7
8
spec:
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        traefik:
          weightedTraefikServiceName: checkout-api-traefik

In every case, the pattern is identical: define stable/canary Services, point trafficRouting at your provider’s resource, and Argo Rollouts takes care of patching the weights at each step. This means you can swap the underlying networking layer later without touching your canary steps or analysis templates.

Locking Down RBAC

Never install Argo Rollouts with cluster-admin. The controller only needs to manage Rollouts, ReplicaSets, and the specific traffic-routing resources you use. Here’s a scoped ClusterRole that covers the Istio + Prometheus setup from above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: argo-rollouts-scoped
rules:
  - apiGroups: ["argoproj.io"]
    resources: ["rollouts", "rollouts/status", "rollouts/finalizers", "analysisruns", "analysistemplates", "experiments"]
    verbs: ["get", "list", "watch", "update", "patch", "create", "delete"]
  - apiGroups: ["apps"]
    resources: ["replicasets", "deployments"]
    verbs: ["get", "list", "watch", "update", "patch"]
  - apiGroups: [""]
    resources: ["services", "pods", "events"]
    verbs: ["get", "list", "watch", "update", "patch", "create"]
  - apiGroups: ["networking.istio.io"]
    resources: ["virtualservices", "destinationrules"]
    verbs: ["get", "list", "watch", "update", "patch"]

Bind it to the argo-rollouts ServiceAccount in the argo-rollouts namespace only, and if you’re running multi-tenant, prefer namespaced Role/RoleBinding pairs per team instead of a single cluster-wide grant.

Automated Analysis: Let Metrics Decide, Not the Clock

Timed pauses are better than nothing, but the real power of Argo Rollouts is automated analysis — querying real metrics after each step and aborting automatically if something looks wrong. This is what makes canary deployments genuinely production-safe instead of a fancy progress bar.

Define an AnalysisTemplate that queries Prometheus for the error rate of the canary pods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: checkout-api-error-rate
spec:
  args:
    - name: service-name
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      successCondition: result[0] <= 0.05
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(http_requests_total{service="{{args.service-name}}", status=~"5.."}[1m]))
            /
            sum(rate(http_requests_total{service="{{args.service-name}}"}[1m]))

This checks every minute, five times, that the 5xx error ratio stays under 5%. If it fails twice, the whole analysis run is marked as failed and Argo Rollouts automatically aborts and rolls back.

Now wire this template into the canary steps:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
spec:
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      trafficRouting:
        istio:
          virtualService:
            name: checkout-api
            routes:
              - primary
      steps:
        - setWeight: 5
        - pause: { duration: 1m }
        - analysis:
            templates:
              - templateName: checkout-api-error-rate
            args:
              - name: service-name
                value: checkout-api-canary
        - setWeight: 20
        - pause: { duration: 2m }
        - analysis:
            templates:
              - templateName: checkout-api-error-rate
            args:
              - name: service-name
                value: checkout-api-canary
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

Now every stage of the rollout has an actual health gate. If error rates spike at 20%, the rollout aborts before it ever touches half your traffic.

Background Analysis (Running Checks the Whole Time, Not Just Between Steps)

Step-scoped analysis blocks only run when the rollout passes through them. For metrics you want monitored continuously for the entire duration of the rollout — not just at discrete checkpoints — use analysis.templates at the top of the canary spec instead of inside steps:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
spec:
  strategy:
    canary:
      canaryService: checkout-api-canary
      stableService: checkout-api-stable
      analysis:
        templates:
          - templateName: checkout-api-error-rate
        args:
          - name: service-name
            value: checkout-api-canary
        startingStep: 1
      steps:
        - setWeight: 5
        - pause: { duration: 2m }
        - setWeight: 20
        - pause: { duration: 5m }
        - setWeight: 50
        - pause: { duration: 5m }
        - setWeight: 100

startingStep: 1 means background analysis kicks in as soon as the first setWeight step completes, and it keeps re-querying on its own interval for as long as the rollout is in progress — so a regression that only shows up 4 minutes into a 5-minute pause still gets caught.

Getting Paged: Argo Rollouts Notifications

Install the notifications controller (bundled with the standard manifest) and configure triggers so a failed or successful rollout reaches Slack, PagerDuty, or a webhook without anyone watching a terminal:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apiVersion: v1
kind: ConfigMap
metadata:
  name: argo-rollouts-notification-configmap
  namespace: argo-rollouts
data:
  service.slack: |
    token: $slack-token
  template.rollout-degraded: |
    message: |
      :rotating_light: Rollout *{{.rollout.name}}* in namespace *{{.rollout.namespace}}* has DEGRADED and was rolled back.
  trigger.on-rollout-step-completed: |
    - when: rollout.status.phase == 'Degraded'
      send: [rollout-degraded]

Then annotate the Rollout to subscribe it to that trigger:

1
2
3
metadata:
  annotations:
    notifications.argoproj.io/subscribe.on-rollout-step-completed.slack: platform-alerts

Now a Degraded rollout pages the platform-alerts Slack channel the moment the analysis gate fails — the same second traffic snaps back to stable.

GitOps: Driving Rollouts with Argo CD

Most production setups don’t kubectl apply manifests by hand — they run through Argo CD (or Flux) so the cluster state is always a reflection of Git. Argo Rollouts and Argo CD are built by the same project and integrate natively: Argo CD syncs the Rollout manifest like any other resource, and the Argo CD UI understands Rollout health status out of the box (you’ll see the canary progress bar directly in the Argo CD application view).

A typical GitOps flow looks like this:

  1. CI builds and pushes a new image tag, then updates the image reference in a Git repo (via a tool like Kustomize image overrides or a templated Helm values file).
  2. Argo CD detects the Git change and syncs the updated Rollout manifest to the cluster.
  3. The Argo Rollouts controller sees the new spec.template.spec.containers[].image and starts the canary steps automatically — Argo CD doesn’t need to know anything about canaries, it just syncs the spec.
  4. Argo Rollouts’ own analysis gates decide whether the rollout finishes or aborts; Argo CD reflects that as the Application’s health.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: checkout-api
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/myorg/checkout-api-manifests
    targetRevision: main
    path: k8s/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: checkout
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

Because selfHeal: true reconciles the live Rollout back to what’s in Git, don’t kubectl argo rollouts promote manual pauses this way in a GitOps setup — Argo CD will fight you. Instead, either keep manual pauses out of GitOps-managed Rollouts (use pure automated-analysis gating), or manage promotion through Argo CD’s own resource actions, which understand Rollout’s promote/abort as first-class Application actions in the UI and CLI (argocd app actions run checkout-api promote-full --kind Rollout).

Compatibility with Horizontal Pod Autoscaling

A Rollout works with a standard HorizontalPodAutoscaler exactly like a Deployment does — just target it as the scale subject:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: checkout-api
spec:
  scaleTargetRef:
    apiVersion: argoproj.io/v1alpha1
    kind: Rollout
    name: checkout-api
  minReplicas: 5
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

One subtlety: during a canary, both the stable and canary ReplicaSets share the total replica count that the HPA controls, split according to the current weight (rounded up so at least one canary pod exists once weight > 0). If your HPA is actively scaling while a canary is mid-rollout, set spec.strategy.canary.dynamicStableScale: true so the stable ReplicaSet scales down as the canary scales up, instead of both sitting at full size simultaneously and doubling your resource footprint.

Watching and Controlling a Rollout

The plugin gives you full manual control whenever you need to step in:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# Watch progress live
kubectl argo rollouts get rollout checkout-api --watch

# Manually promote past a pause
kubectl argo rollouts promote checkout-api

# Skip straight to 100% (use with care)
kubectl argo rollouts promote checkout-api --full

# Abort immediately and roll back to stable
kubectl argo rollouts abort checkout-api

# Undo entirely, back to the last stable revision
kubectl argo rollouts undo checkout-api

If you’d rather have a GUI, run the built-in dashboard:

1
kubectl argo rollouts dashboard

Then open http://localhost:3100 for a visual view of every Rollout in the cluster, including live traffic weight graphs.

Integrating with CI/CD

In production you don’t want humans typing promote at 2 AM. Here’s a minimal GitHub Actions job that deploys a new version and lets Argo Rollouts’ own analysis gates decide whether it succeeds, failing the pipeline if the rollout aborts:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
name: Deploy Canary

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure kubeconfig
        run: echo "${{ secrets.KUBECONFIG_DATA }}" | base64 -d > kubeconfig

      - name: Set new image
        run: |
          export KUBECONFIG=kubeconfig
          kubectl argo rollouts set image checkout-api \
            checkout-api=myregistry/checkout-api:${{ github.sha }}

      - name: Wait for rollout to finish or fail
        run: |
          export KUBECONFIG=kubeconfig
          kubectl argo rollouts status checkout-api --watch --timeout 900s

kubectl argo rollouts status --watch blocks until the Rollout is either Healthy (all steps and analyses passed, 100% traffic) or Degraded (an analysis failed and it aborted) — and it returns a non-zero exit code in the failure case, which fails your CI job automatically. No manual babysitting required.

Monitoring the Rollout Process Itself

Beyond analyzing your application’s metrics, it’s worth monitoring Argo Rollouts’ own health. The controller exposes Prometheus metrics on port 8090 by default — scrape them and build a dashboard around:

  • rollout_info_replicas_available — track available replicas per Rollout over time
  • rollout_phase — alert if any Rollout sits in Degraded or Paused (indefinite) for longer than expected
  • controller_clientset_k8s_request_total — catch API throttling if you’re running many Rollouts cluster-wide

A minimal ServiceMonitor (if you’re using the Prometheus Operator):

1
2
3
4
5
6
7
8
9
10
11
12
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: argo-rollouts-metrics
  namespace: argo-rollouts
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: argo-rollouts
  endpoints:
    - port: metrics
      interval: 30s

Import Argo Rollouts’ official Grafana dashboard (ID 19870 on grafana.com) for an out-of-the-box view of rollout phases, step progress, and analysis run pass/fail rates across every Rollout in the cluster — genuinely useful once you have more than two or three services on canary.

Incident Walkthrough: What Actually Happens When a Canary Fails

It helps to walk through a real failure end to end so the pieces above click together. Say checkout-api:1.1.0 ships with a bug that only appears under load — checkout starts throwing 500s once the database connection pool saturates.

  1. Rollout begins. setWeight: 5 fires, 5% of traffic (or exactly 5% via the Istio VirtualService) hits the canary pods.
  2. First analysis window opens. The checkout-api-error-rate AnalysisRun starts querying Prometheus every 1 minute.
  3. Error rate climbs. Under the small 5% slice, load might be light enough that nothing fails yet — this is exactly why later steps matter, not just the first one.
  4. setWeight: 20 fires, analysis restarts (or, with background analysis, keeps running continuously). Connection pool saturation starts kicking in at this higher volume; the error-rate query returns 0.09 (9%) against a successCondition of <= 0.05.
  5. First failed measurement recorded. failureLimit: 2 means one more bad reading and the AnalysisRun fails.
  6. Second measurement, one minute later, also fails. The AnalysisRun’s status flips to Failed.
  7. Argo Rollouts marks the Rollout Degraded and automatically sets the Istio VirtualService weight back to 100% stable / 0% canary — this happens in seconds, not minutes.
  8. The bad ReplicaSet is scaled down (or left at a minimal scaleDownDelaySeconds for debugging, if you’ve configured one).
  9. Notifications fire — the Slack trigger from the on-rollout-step-completed template above posts to platform-alerts the instant the phase becomes Degraded.
  10. CI/CD reflects the failure. The kubectl argo rollouts status --watch command in the GitHub Actions job exits non-zero, and the pipeline goes red — nobody had to notice the Slack message for the deploy to be blocked.

Total blast radius: roughly 20% of traffic for a couple of minutes, automatically contained, with the team paged before a human even opened a dashboard. That’s the entire value proposition of production canary deployments in one sequence.

If you want to inspect exactly what happened after the fact:

1
2
3
kubectl argo rollouts get rollout checkout-api
kubectl describe analysisrun -l rollouts-pod-template-hash=<canary-hash>
kubectl logs -n argo-rollouts deploy/argo-rollouts-controller-manager

Production Best Practices

A handful of things separate a “canary that works in a demo” from one you can trust in production:

  • Set progressDeadlineSeconds on the Rollout so a stuck rollout (pods that never become ready) fails fast instead of hanging forever.
  • Always pair setWeight steps with an analysis step, not just a timed pause — clocks don’t know if your service is broken, metrics do.
  • Scope RBAC tightly. The Argo Rollouts controller needs permissions to patch VirtualServices/Ingresses; don’t grant it cluster-admin. Use the namespaced install and a scoped ClusterRole.
  • Keep failureLimit low on AnalysisTemplates (1–2) — you want fast aborts, not five failed checks before anyone notices.
  • Version your AnalysisTemplates alongside your Rollouts in Git so a rollback of the manifest also rolls back the safety gates.
  • Test the abort path deliberately — deploy a known-bad image in staging and confirm the rollout actually aborts and traffic actually returns to stable. Don’t find out your metrics query has a typo during a real incident.
  • Alert on Degraded rollouts — pipe Argo Rollouts’ Kubernetes events into your alerting stack so a failed canary pages someone immediately.
  • Use restartAt or dynamicStableScale carefully on high-traffic services to avoid over-provisioning during long canary windows.
  • Prefer background analysis over step-only analysis for anything with load-dependent failure modes (connection pools, memory leaks, cache warm-up) — problems that only appear a few minutes into a step will be missed if analysis only samples right at the transition.
  • If you’re on GitOps (Argo CD/Flux), avoid indefinite manual pauses in Rollouts managed with selfHeal: true, or drive promotion through the GitOps tool’s own resource actions instead of raw kubectl argo rollouts promote.
  • Set dynamicStableScale: true whenever an HPA is attached to the Rollout, so you don’t silently double your pod count for the duration of every canary.
  • Wire up Notifications before you need them. A Degraded rollout that nobody sees until the next morning defeats most of the point of automated analysis.
  • Scrape the controller’s own metrics. If the controller itself is throttled or falling behind on many concurrent Rollouts, step timing and analysis intervals silently drift.

Common Pitfalls

  • Forgetting stableService/canaryService — without both, traffic routing integrations silently do nothing and you get replica-based approximation instead of real weighted splitting.
  • Prometheus query returns no data — if http_requests_total isn’t labeled the way your query expects, the AnalysisRun will error out (not fail gracefully), stalling the rollout. Test queries directly in Prometheus first.
  • Analysis interval shorter than metric scrape interval — you’ll just be re-querying stale data. Match interval to your actual scrape/aggregation window.
  • Mutating the Rollout mid-flight — changing spec.template while a rollout is in progress starts an entirely new rollout on top of the old one. Let one finish (or abort it) before pushing another change.
  • GitOps self-heal fighting manual promotion — if Argo CD has selfHeal: true and the live Rollout drifts from Git (e.g., you manually promoted past a pause), Argo CD will try to resync and can interfere. Handle promotion through Argo CD’s resource actions in GitOps setups, not raw kubectl commands.
  • HPA and canary both scaling replicas at once — without dynamicStableScale: true, a canary during an autoscaling event can temporarily run stable and canary ReplicaSets at combined full size, spiking resource usage well above what you’d expect.
  • AnalysisRun stuck in Inconclusive — this happens when a metric provider is unreachable (Prometheus down, network policy blocking the controller) rather than when the metric itself fails the condition. Check controller logs, not just the AnalysisRun status, to tell the difference.
  • Notifications never fire — the subscription annotation and the ConfigMap trigger name must match exactly; a typo in notifications.argoproj.io/subscribe.<trigger>.<service> silently does nothing with no error surfaced anywhere.

Wrapping Up

Argo Rollouts turns “deploy and hope” into “deploy and verify” — every step in the process now has a real, measurable gate instead of a fixed timer. Once you’ve got weighted traffic splitting through a mesh and automated Prometheus-backed analysis in place, a bad release gets caught and rolled back automatically before most of your users ever see it, and your CI pipeline doesn’t need a human standing by to make the call.

From here, natural next steps are exploring BlueGreen strategy for services that can’t tolerate mixed-version traffic at all, or Experiment resources for running two versions side-by-side purely for comparison without shifting production traffic. Both build on exactly the same Rollout foundation you just set up.

Happy Coding