Who is Ehsan Tatasadi?

Ehsan Tatasadi is a freelance Senior .NET & Azure Engineer based in Hamburg, Germany. He builds and modernizes .NET platforms on Azure, connects them with Dynamics 365, and runs them in production. He has 8+ years on the Microsoft stack, more than 7 of them for a DAX-listed energy utility and a regional grid operator.

What technologies does Ehsan Tatasadi specialize in?

C#, .NET 8 and ASP.NET Core with SQL Server; Azure (App Service, Functions, Key Vault, Private Endpoints, Application Insights); Dynamics 365 CE integration via the XRM SDK, C# plugins and JavaScript form scripts; Terraform and Azure DevOps YAML pipelines; React and TypeScript. He is certified in AZ-400, AZ-204, AI-103 and GH-300.

Where is Ehsan Tatasadi located?

Ehsan Tatasadi is based in Hamburg, Germany. He works remotely, on-site in Hamburg, and on-site anywhere in Germany on demand.

Explainer

Stuck Kubernetes rollout on AKS: ProgressDeadlineExceeded, no rollback

November 12, 2026 · Series: Containers on Azure

azureakskubernetesdeploymentsprobeskubectl

I deployed a broken version to Azure Kubernetes Service (AKS): its readiness check answered 404. For ten minutes, not one call failed. Then kubectl said the rollout had failed, and Kubernetes did nothing about it.

This explainer shows what a rolling update does with a version that never gets ready, where the error shows up, and why rolling back is your job. Everything below comes from the Microsoft Learn exercise, run on my own cluster.

Key points

The old pod keeps serving. With one replica, a rolling update may start one extra pod and take none away. The new pod starts next to the old one, and the old one stays until the new one is ready. Mine never was, and the client never saw an error.
kubectl get deployment looks healthy. READY 1/1 counted the old pod. The new pod was Running 0/1 with 0 restarts, and the reason was only in the events: Readiness probe failed: HTTP probe failed with statuscode: 404.
After ten minutes, Kubernetes only reports. After progressDeadlineSeconds (600 by default) the Deployment's Progressing condition gets the reason ProgressDeadlineExceeded, and kubectl rollout status exits with an error. Nothing is rolled back; both pods stay.
Undo is your job, and more replicas make a bigger hole. kubectl rollout undo put the old pod template back at once. With four replicas, the same broken version took one old pod away immediately: READY 3/4 for as long as the rollout stays stuck.

The handover: the new pod next to the old one

In the exercise, a small Python API runs on AKS. A Deployment keeps one replica of it running, behind a Service, the stable address that spreads calls over the pods. Then the exercise breaks it on purpose: the readiness probe, the check that says "send me traffic", asks for a path that doesn't exist.

readinessProbe:
  httpGet:
    path: /invalid-path
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 10

The liveness probe in the same file still checks /healthz. I applied the file and watched the pods with a small script of mine that prints every change:

> kubectl apply -f k8s/probe-failure-deployment.yaml
deployment.apps/api-deployment configured
> python pods.py --secs 40
   0.3s  api-deployment-695945c854-nx6qj  Running           1/1  restarts 0
   0.3s  api-deployment-6b4f47c8b7-6m9bb  ContainerCreating 0/1  restarts 0
   2.5s  api-deployment-6b4f47c8b7-6m9bb  Running           0/1  restarts 0

The new pod ran, but never got ready. The old pod stayed at 1/1. A client pod inside the cluster called the Service once a second, like a user would, and every answer came from the pod that started at 13:58, the old one:

> kubectl logs client --tail 4
14:07:06 200 pod started 13:58:45.05
14:07:07 200 pod started 13:58:45.05
14:07:08 200 pod started 13:58:45.05
14:07:09 200 pod started 13:58:45.05

Why? A rolling update follows two settings, both a quarter of the replicas by default:

Setting	Default	With 1 replica	With 4 replicas
`maxSurge`: extra pods during the rollout (rounded up)	25 %	1	1
`maxUnavailable`: pods that may be missing (rounded down)	25 %	0	1

With one replica, the rollout may start one extra pod but may take none away. So the old pod stays until the new one is ready, and a pod only gets traffic from the Service once it is ready.

Where's the error?

The usual first look is the Deployment itself: how many of its pods are ready, up to date and available. The new pod was 42 seconds old and had failed every check so far:

> kubectl get deployment api-deployment
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
api-deployment   1/1     1            1           8m30s

It looks perfectly healthy, because it counts ready pods, and the ready one is the old pod. The pod list shows the new pod at 0/1, with zero restarts: only the readiness probe fails, and the liveness probe, the one that restarts a container, still passes.

> kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
api-deployment-695945c854-nx6qj   1/1     Running   0          8m30s
api-deployment-6b4f47c8b7-6m9bb   0/1     Running   0          42s
client                            1/1     Running   0          8m11s

The reason is in the events, the cluster's notes about what happened to each object. A failed readiness probe is an event with the reason Unhealthy, every ten seconds again:

> kubectl get events --field-selector reason=Unhealthy
LAST SEEN   TYPE      REASON      OBJECT                                MESSAGE
2s          Warning   Unhealthy   pod/api-deployment-6b4f47c8b7-6m9bb   Readiness probe failed: HTTP probe failed with statuscode: 404

Then I asked kubectl to wait for the rollout, for thirty seconds. It timed out, with one old replica "pending termination": the rollout was waiting for the old pod to be allowed to go.

> kubectl rollout status deployment api-deployment --timeout 30s
Waiting for deployment "api-deployment" rollout to finish: 1 old replicas are pending termination...
error: timed out waiting for the condition

Ten minutes later: rolled back?

A Deployment has a progress deadline: progressDeadlineSeconds, 600 seconds by default, and you can make it shorter in the Deployment. If the rollout makes no progress for that long, the Deployment is marked as failed. I ran the same command without a timeout and let it wait:

> kubectl rollout status deployment api-deployment
Waiting for deployment "api-deployment" rollout to finish: 1 old replicas are pending termination...
error: deployment "api-deployment" exceeded its progress deadline

That came about ten minutes after I had applied the broken version. Did Kubernetes roll it back? No.

> kubectl get pods
NAME                              READY   STATUS    RESTARTS   AGE
api-deployment-695945c854-nx6qj   1/1     Running   0          17m
api-deployment-6b4f47c8b7-6m9bb   0/1     Running   0          10m
client                            1/1     Running   0          17m

Both pods were still there. The client's log for the last twelve minutes, the whole stuck rollout and a little before, held 0 failed calls and 718 answered ones. The Deployment only reports the failure, in its status condition Progressing:

> kubectl get deployment api-deployment -o jsonpath="{.status.conditions[1].reason}"
ProgressDeadlineExceeded

That's all. The Kubernetes docs say it plainly: Kubernetes takes no action on a stalled rollout other than reporting this condition. It keeps trying, and the old pod keeps serving, for as long as you let it. Release tools on top of Kubernetes can watch for a failed rollout and roll back for you. Kubernetes itself doesn't.

Undo, and three mistakes

kubectl rollout undo puts the previous pod template back, the description of the pods inside the Deployment:

> kubectl rollout undo deployment api-deployment
deployment.apps/api-deployment rolled back
> kubectl rollout status deployment api-deployment --timeout 120s
deployment "api-deployment" successfully rolled out

It finished at once (0.6 s), because the old pod had never left. kubectl also warned that an undo does not update the last-applied-configuration annotation of a Deployment managed with kubectl apply, so the next apply should come from a manifest that works.

Three mistakes make a stuck rollout worse:

Checking the Deployment at a glance. READY 1/1 was the old pod. Look at the pods and the events.
Waiting without a timeout. In a pipeline, kubectl rollout status is your gate: it exits with an error code when the rollout fails, so the pipeline can act on it. Give it a timeout that fits your app, two minutes for this one, and run the undo in the same line, so a broken version never sits there for ten minutes:

kubectl rollout status deployment api-deployment --timeout 120s ||
  kubectl rollout undo deployment api-deployment

The same wrong path for both probes. Here only the readiness probe was wrong. With the liveness probe on it too, the kubelet, the agent on the node, would restart the new pod again and again: a crash loop on top of the stuck rollout.

The same mistake with four replicas

A quarter of four is one: one pod extra, and one pod away. I scaled the Deployment to four, applied the same broken version again and watched the pods with the same script:

> python pods.py --secs 40
   0.3s  api-deployment-695945c854-56gl2  Terminating       1/1  restarts 0
   0.3s  api-deployment-695945c854-8h7sh  Running           1/1  restarts 0
   0.3s  api-deployment-695945c854-9xct9  Running           1/1  restarts 0
   0.3s  api-deployment-695945c854-nx6qj  Running           1/1  restarts 0
   0.3s  api-deployment-6b4f47c8b7-dfhtx  ContainerCreating 0/1  restarts 0
   0.3s  api-deployment-6b4f47c8b7-ljp88  ContainerCreating 0/1  restarts 0
   2.5s  api-deployment-6b4f47c8b7-dfhtx  Running           0/1  restarts 0
   2.5s  api-deployment-6b4f47c8b7-ljp88  Running           0/1  restarts 0
   2.5s  api-deployment-695945c854-56gl2  gone
> kubectl get deployment api-deployment
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
api-deployment   3/4     2            3           18m

One old pod was already terminating a third of a second in. The pod that left needs a replacement too, so two new pods started, and both got stuck at 0/1. Ready 3/4: a quarter of the capacity gone, for as long as the rollout stays stuck. The client still got its answers, from the three old pods. Nobody notices, until the traffic grows and three pods aren't enough. With a stuck rollout, more replicas don't mean more safety: they mean a bigger hole.

What happened when I ran it

Tested on November 2, 2026 in my own Azure subscription (AKS 1.36 in westeurope, Azure CLI 2.88.0, kubectl 1.36); everything was deleted afterwards. The cluster had one Standard_D2s_v5 node with the registry attached, as in the exercise. The broken version went in at 14:06:25 UTC; kubectl rollout status without a timeout waited 526.8 s and reported the missed deadline at 14:16:27. The 30-second wait took 30.4 s, and both waits exited with code 1. The undo took 0.6 s with one replica. With four replicas, the scale-up and the undo each took 11.8 s, and afterwards four pods of the old version were ready again, one of them new. The call counts came from the client's own log (kubectl logs client --since=12m, counted with PowerShell's Select-String).

What the video leaves out

minReadySeconds, and pausing a rollout with kubectl rollout pause / resume.
Release tools that roll back for you, such as Argo Rollouts or Flagger.
Liveness probe failures (a crash loop) and startup probes.
Fixing the probe with kubectl edit, as the exercise does: like the undo, it starts a new rollout with a template that works.

Links

Watch on YouTube

Video transcript

I deployed a broken version to Azure Kubernetes Service. Its readiness check answered 404. For ten minutes, not one call failed. Then kubectl said the rollout had failed. So what did Kubernetes do about it?

In this video, you'll see what a rollout does with a version that never gets ready, and what happens after ten minutes. At the end: the same mistake with four replicas, and what it costs.

What happens when the new version never gets ready? In the Microsoft Learn module's exercise, a small Python API runs on Azure Kubernetes Service. A deployment keeps one copy of it running, one replica in one pod, behind a Service, the stable address that spreads calls over the pods. The exercise breaks it on purpose. Its readiness probe, the check that says "send me traffic", asks for a path that doesn't exist. So I applied the broken version, the exercise's probe failure file. Configured. And I watched the pods with a small script that prints every change. The new pod ran, but never got ready: zero of one, zero of its one container ready. The old pod stayed at one of one. A client pod inside the cluster called the Service once a second, like a user would. What did the client get while the new pod failed its check? The old version. I read the client's last four lines. Every answer came from the pod that started at 13:58: the old one. Here's why. With one pod, a rollout may start one extra pod, but may take none away. So the new pod starts next to the old one, and the old one stays until the new one is ready. Those are two settings, maxSurge and maxUnavailable, a quarter of the replicas each by default: rounded up for the extra pods, rounded down for the ones taken away. And a pod only gets traffic once it's ready. I showed that in the first video of this series; the card is up there now.

So where does it say that something is wrong? The usual first look is the deployment itself: how many of its pods are ready, up to date and available. The new pod has failed its check for a minute now. Does the deployment look broken? No. One of one ready. I asked for it, the way you'd check a rollout at a glance. One ready, one up to date, one available. It looks perfectly healthy. It counts ready pods, and the ready one is the old pod. The pod list says zero of one, with zero restarts, because only the readiness check fails. The liveness check, the one that restarts a container, still passes. The reason is in the events, the cluster's notes about what happened to each object. I asked for the events with the reason Unhealthy. Readiness probe failed: HTTP probe failed with statuscode: 404. Every ten seconds, again. Then I asked kubectl to wait for the rollout, for thirty seconds. Timed out. Still one old replica pending termination: the rollout is waiting for the old pod to be allowed to go. This series runs one Azure service at a time, for real. Subscribe to follow the series.

How long does Kubernetes wait? For a pod that never gets ready, a deployment has a progress deadline, ten minutes by default. If the rollout makes no progress for that long, the deployment is marked as failed. The setting is called progressDeadlineSeconds, and you can make it shorter in the deployment. So I ran the same command again, this time without a timeout, and let it wait. Exceeded its progress deadline. About ten minutes after I had applied the broken version. After ten minutes, does Kubernetes roll it back? No. I listed the pods. Both still there. The old pod, seventeen minutes old, and the broken one, ten minutes old, still not ready, still zero restarts. And the users? I counted the client's failed calls in the last twelve minutes: the whole stuck rollout and a little before. Zero failed. And the answered ones: seven hundred eighteen. The deployment just reports it, in a status condition called Progressing. I asked for its reason. ProgressDeadlineExceeded. That's all. Kubernetes takes no action on a stalled rollout other than this report. It keeps trying, and the old pod keeps serving, for as long as you let it. Release tools on top of Kubernetes can watch for this condition and roll back for you. Kubernetes itself doesn't.

So how do you get out of it? I rolled back with kubectl rollout undo, which puts the previous pod template back: the description of the pods inside the deployment. Rolled back. And the rollout finished at once, because the old pod had never left. Three mistakes make this worse. One: checking the deployment at a glance. Ready one of one was the old pod. Two: waiting without a timeout. In a pipeline, kubectl rollout status is your gate. It exits with an error code when the rollout fails, so the pipeline can act on it. Give it a timeout that fits your app, two minutes for this one. And if it fails, run the undo in the same line, so a broken version never sits there for ten minutes. Three: the same wrong path for both checks. Here only the readiness probe was wrong. With the liveness probe on it too, the kubelet, the agent on the node, would restart the new pod again and again, and you'd be in a crash loop on top of the stuck rollout.

To sum it up: a rolling update starts the new pod first, and keeps the old one until the new one is ready. After ten minutes, the deployment is marked ProgressDeadlineExceeded, and nothing more. Undo is your job.

And with four replicas? A quarter of four is one: one pod extra, and one pod away. I scaled the deployment to four, and applied the same broken version again. Configured. And I watched the pods with the same script. One old pod was already terminating, a third of a second in. The pod that left needs a replacement too, so two new pods started, and both got stuck at zero of one. Then I asked for the deployment. Ready three of four. A quarter of the capacity gone, for as long as the rollout stays stuck. The client still got its answers, from the three old pods. Nobody notices, until the traffic grows and three pods aren't enough. With a stuck rollout, more replicas don't mean more safety: they mean a bigger hole.

To give a new version only a small share of real traffic first, the video on screen does exactly that, on Azure Container Apps. If this was useful, subscribe for more practical .NET and Azure.

Need this in your own Azure environment?

I build, migrate and run .NET platforms on Azure, with Azure DevOps and Terraform. If your team wants this set up properly, see what I offer.

See services