• English icon
  • German icon
Explainer

Kubernetes pod data lost after restart? AKS disks vs Azure Files

· Series: Containers on Azure

azureakskubernetesstoragepersistent-volumeskubectl

I wrote a note inside my pod on Azure Kubernetes Service (AKS), next to the log file the app keeps. Then I deleted the pod. Kubernetes started a new one in 22 seconds. One of the two files survived. The other was gone.

This explainer shows where a pod's data really lives, why ReadWriteOnce means one node and not one pod, and when the Azure disk behind a claim is deleted for good. Everything below comes from the Microsoft Learn exercise and a few extra steps, run on my own cluster.

Key points

  • A container's own files start clean. Every container starts from its image plus a fresh, empty writable layer. A replaced pod gets a new container, and even after a crash the kubelet restarts the container with a clean state. Anything you need later goes on a volume.
  • A claim is a request, a volume is the storage, the disk is what Azure bills. A PersistentVolumeClaim asks for size, access mode and a storage class; the class creates a PersistentVolume, and on AKS the default class managed-csi creates an Azure managed disk in the node resource group.
  • ReadWriteOnce means one node, not one pod. Two pods on the same node shared one disk. A pod on a second node waited in ContainerCreating because the disk was attached to the first node. For many nodes, use Azure Files (azurefile-csi, ReadWriteMany).
  • Deleting the claim deletes the disk. The built-in storage classes have reclaim policy Delete. A storage class with reclaimPolicy: Retain keeps the disk after the claim is gone, and you delete it yourself.

Gone with the container

In the exercise, a Deployment runs a small API. A Deployment keeps a set number of pods running from one template, and a pod is the unit that wraps your container. The app writes one line to a log file for every request it gets:

> python cfg.py get /logs/summary

GET /logs/summary -> 200

{… "total_requests":264, … "first_request":"2026-10-28T12:31:16.705746", …}

Then I wrote a note into the container's /tmp folder, deleted the pod and waited for the Deployment to replace it:

> kubectl exec deploy/aks-config-api -- sh -c "echo note > /tmp/note.txt; ls /tmp"

note.txt

> kubectl delete pod -l app=aks-config-api

pod "aks-config-api-876f7899f-jqpxj" deleted from default namespace

> kubectl rollout status deployment aks-config-api --timeout 300s

deployment "aks-config-api" successfully rolled out

In the new pod, /tmp was empty, while the log kept counting from the same first request:

> kubectl exec deploy/aks-config-api -- ls /tmp

> python cfg.py get /logs/summary

{… "total_requests":266, … "first_request":"2026-10-28T12:31:16.705746", …}

Why: every container starts from its image, plus a fresh, empty layer for whatever it writes. A new pod gets a new container, so it starts clean. Even a crash does that: the kubelet, the Kubernetes agent on the node (the virtual machine your pods run on), restarts a crashed container with a clean state. If you know Docker, that's the surprise: docker restart keeps a container's files; Kubernetes starts a fresh container instead.

The log survived because it was never in the container's own files. It was on a volume: storage from outside the pod, mounted at a folder path.

The request slip: claim, volume, disk

The exercise asks for storage with a PersistentVolumeClaim (a claim): a request that says how much storage, how it may be shared, and which storage class makes it. A storage class is the recipe that creates the storage. Rewritten short:

kind: PersistentVolumeClaim

metadata:

  name: api-logs-pvc

spec:

  resources: {requests: {storage: 1Gi}}

  accessModes: [ReadWriteOnce]

  storageClassName: managed-csi

The Deployment mounts the claim by its name, at the folder where the app writes its log:

# the pod template

volumes:

- name: log-storage

  persistentVolumeClaim: {claimName: api-logs-pvc}

# the container

volumeMounts:

- {name: log-storage, mountPath: /var/log/api}

The claim was Bound to a volume with a generated name. That storage didn't exist before: managed-csi (the default class on AKS) waits for the first pod that uses the claim, then creates it. Columns cut with "…", the long volume name shortened to its end:

> kubectl get pvc api-logs-pvc

NAME           STATUS   VOLUME      CAPACITY   ACCESS MODES   STORAGECLASS …

api-logs-pvc   Bound    pvc-…0001   1Gi        RWO            managed-csi …

> kubectl get pv

NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS …

pvc-…0001   1Gi        RWO            Delete           Bound …

Behind the volume is an Azure managed disk with the volume's name, in the cluster's node resource group, the resource group AKS keeps for the cluster's own resources:

> az disk list -g MC_rg-demo_aks-demo_westeurope --query "[].{name:name, gb:diskSizeGB}" -o table

Name                                      Gb

----------------------------------------  ----

pvc-00000000-0000-0000-0000-000000000001  1

So the claim is the request, the volume is the storage Kubernetes knows, and the disk is what Azure stores and bills. The data sits on that disk, outside every pod.

ReadWriteOnce: once per what?

Most people read ReadWriteOnce as one pod. I scaled the Deployment to two replicas while the cluster had a single node. Both started, both on the same node, both writing to the same disk:

> kubectl get pods -l app=aks-config-api -o wide

NAME                             READY   STATUS    … NODE …

aks-config-api-876f7899f-csl9m   1/1     Running   … node …vmss000000 …

aks-config-api-876f7899f-tmq9k   1/1     Running   … node …vmss000000 …

ReadWriteOnce means one node, not one pod. An Azure disk attaches to one virtual machine, the node, and every pod on that node can use it. (Kubernetes has a separate access mode, ReadWriteOncePod, for a single pod across the cluster.)

Then I added a second node and a second Deployment: two pods, one new ReadWriteOnce disk, and a pod anti-affinity rule that puts the two pods on different nodes. The first pod was Running after 15 seconds. The second stayed in ContainerCreating for the whole two and a half minutes I watched. The warning event says why (excerpt):

> kubectl events --types=Warning

FailedAttachVolume … Volume is already used by pod(s) disk-writer-…xqsgp

The disk was attached to the first node, and the second node can't attach it while the first pod uses it. No crash, no error in the app, just a pod that never starts. The AKS troubleshooting docs list this message as the Multi-Attach error. Without an anti-affinity rule the scheduler may still spread replicas across nodes, so a disk-backed Deployment with several replicas can hit it too.

Many nodes, one share

For pods on several nodes, use a different storage class: azurefile-csi. It creates an Azure Files share, a file share in an Azure storage account (AKS creates the account in the node resource group), and every node mounts it over the network. Its access mode is ReadWriteMany. In the claim, only two lines change:

kind: PersistentVolumeClaim

metadata: {name: shared-files}

spec:

  resources: {requests: {storage: 1Gi}}

  accessModes: [ReadWriteMany]

  storageClassName: azurefile-csi

With the same two-node Deployment, both pods waited about 25 seconds while the share was created, then ran, one on each node. One pod wrote a line into the share, and both read it:

> python cfg.py share files-writer

   0.8s  files-writer-6bc8ddf6b-85gqk writes /data/note.txt

   1.3s  files-writer-6bc8ddf6b-85gqk reads: written by the first pod

   1.8s  files-writer-6bc8ddf6b-zcmcc reads: written by the first pod

Why not always use a share? A database expects its files to itself, so the module puts databases on disks. Azure Files is for files that many pods share.

Azure Disk (managed-csi)Azure Files (azurefile-csi)
Access modeReadWriteOnceReadWriteMany
Who can use itthe pods on one node at a timepods on many nodes at once
Typical usea database's datafiles many pods share
On delete of the claimthe disk is deletedthe file share is deleted

The column that deletes the disk

The column in the volume list said it from the start: reclaim policy Delete. It decides what happens to the storage when its claim is deleted. I deleted the claim of the two-node test together with its Deployment (deleting only the Deployment would have kept the claim and its disk), and listed the disks again a minute and a half later:

> kubectl delete -f k8s/storage/disk-writer.yaml

persistentvolumeclaim "shared-disk" deleted from default namespace

deployment.apps "disk-writer" deleted from default namespace

> az disk list -g MC_rg-demo_aks-demo_westeurope --query "[].{name:name, gb:diskSizeGB}" -o table

Name                                      Gb

----------------------------------------  ----

pvc-00000000-0000-0000-0000-000000000001  1

One disk left, the exercise's. The test's disk was gone, and the data on it. The built-in storage classes all do that. To keep a disk, make your own storage class with reclaimPolicy: Retain and use it in the claim (adapted from the AKS docs example):

apiVersion: storage.k8s.io/v1

kind: StorageClass

metadata: {name: managed-csi-retain}

provisioner: disk.csi.azure.com

parameters: {skuName: StandardSSD_LRS}

reclaimPolicy: Retain

volumeBindingMode: WaitForFirstConsumer

Then the disk stays after the claim is gone. You delete it yourself, and Azure bills it until you do. AKS overwrites changes to its built-in classes, so add your own class instead of editing managed-csi.

What happened when I ran it

Tested on October 28, 2026 in my own Azure subscription (AKS 1.36 in westeurope, Azure CLI 2.88.0, kubectl 1.36); everything was deleted afterwards. The cluster started with one Standard_D2s_v5 node; adding the second node took 161.9 seconds. The replaced pod was ready 22.0 seconds after the delete; the first request to the app right after the rollout timed out once and answered on the retry. The second disk-writer pod sat in ContainerCreating from 4.0 seconds until the end of the 150-second watch. Deleting the disk claim with its Deployment took 31.2 seconds (Kubernetes keeps a claim while a pod still uses it); while the disk was still attached to the first node, Kubernetes recorded VolumeFailedDelete … is still attached to node warnings, and 90 seconds later the disk was gone. For Azure Files, the exercise gives the kubelet identity Storage Account Contributor on the node resource group; I did the same.

What the video leaves out

  • emptyDir: a scratch volume that survives a container crash but is deleted when the pod leaves the node.
  • StatefulSets with volumeClaimTemplates, one claim per replica, the usual answer for databases with several replicas.
  • Azure Container Storage, Azure Blob storage through the CSI driver, Premium SSD v2 and zone-redundant (ZRS) disks, which AKS uses in clusters across availability zones.
  • Volume expansion, snapshots and AKS backup.

Further reading

Links

Video transcript

I wrote a note inside my pod on Azure Kubernetes Service, next to the log file the app keeps. Then I deleted the pod. Kubernetes started a new one in twenty-two seconds. One of the two files survived. The other was gone.

In this video, you'll see where a pod's data really lives, and when it's gone for good. At the end: the one column in the volume list that said how my data would be deleted.

Which file survives a new pod? In the Microsoft Learn module's exercise, a deployment runs a small API. A deployment keeps a set number of pods running from one template, and a pod is the unit that wraps your container. The app writes one line to a log file for every request it gets. With a small script of mine that calls the app, I asked it what it had logged so far. Two hundred sixty-four requests, the first one at twelve thirty-one. Then I wrote a short note into the container's temp folder, with kubectl exec, which runs a command inside the container. There it is: note.txt. Then I deleted the pod, and asked kubectl to wait until the deployment had replaced it. Back after twenty-two seconds, with a new pod from the same template. One file was gone. Which one was gone: the note, or the log? The note. The temp folder was empty. I listed it again, in the new pod, with the same kubectl exec. Nothing. Not one file. And the log? I asked the app again, with the same script. Two hundred sixty-six requests, and the same first request at twelve thirty-one. Nothing lost. Here's why. Every container starts from its image, plus a fresh, empty layer for whatever it writes. A new pod gets a new container, so it starts clean. Even a crash does that. A node is the virtual machine your pods run on, and its agent, the kubelet, restarts a crashed container with a clean state. If you know Docker, that's the surprise: docker restart keeps a container's files. Kubernetes starts a fresh container instead. The log survived because it was never in the container's own files. It was on a volume: storage from outside the pod, mounted at a folder path. The container's own files live as long as the container. Anything you need later goes on a volume.

So where was the log, if not in the container? The exercise asks for storage with a PersistentVolumeClaim, or claim for short: a request that says how much storage, how it may be shared, and which storage class makes it. A storage class is the recipe that creates the storage. In its YAML: one gibibyte, ReadWriteOnce, and managed-csi, the default storage class on AKS, which creates an Azure managed disk. The deployment then mounts the claim by its name, at the folder where the app writes its log. I applied both, then asked kubectl for the claim, the way the module checks it. Bound, to a volume with a generated name. Bound means the claim has its storage. Did that storage exist before? No. The storage class created it, when the first pod needed the claim. So I listed the volumes, the PersistentVolumes, the cluster has. One gibibyte, ReadWriteOnce, bound to my claim. Keep an eye on one column here, reclaim policy. We'll come back to it. That's the piece of storage Kubernetes knows, and the volume the pod mounts. And behind it, in Azure? I listed the disks in the cluster's node resource group, the resource group AKS keeps for the cluster's own resources. One disk, one gigabyte, with the volume's name. So the claim is the request, the volume is the storage Kubernetes knows, and the disk is what Azure stores and bills. The data sits on that disk, outside every pod.

ReadWriteOnce: once per what? Most people read it as one pod. So I scaled the deployment to two replicas: two pods, one ReadWriteOnce claim, and the cluster still had a single node. Do both pods start? Both started. I asked kubectl where they run, with the wide output, which adds the node. Both Running, both on the same node, both writing to the same disk. ReadWriteOnce means one node, not one pod. An Azure disk attaches to one virtual machine, the node, and every pod on that node can use it. This series runs one Azure service at a time, for real. Subscribe to follow the series. So what happens with two nodes? I added a second node to the cluster. Then a second deployment: two pods, one new ReadWriteOnce disk, and a pod anti-affinity rule, which tells Kubernetes to put the two pods on different nodes. What happens to the pod on the second node? It waited. I watched both pods with a small script of mine, for two and a half minutes. The first one was Running after fifteen seconds. The second stayed in ContainerCreating the whole time. Kubernetes recorded why, in a warning event. I listed the warnings with kubectl events. Failed attach volume. Volume is already used by pod, and it names the first one. The disk was attached to the first node. The second node can't attach it while the first pod uses it. No crash, no error in the app, just a pod that never starts. When a pod hangs in ContainerCreating, the events say why. The video about pods that won't start shows how to read them. One Azure disk, one node at a time.

So how do pods on two nodes share data? With a different storage class: azurefile-csi. It creates an Azure Files share, a file share in an Azure storage account, and every node mounts it over the network. Its access mode is ReadWriteMany. In the claim, only two lines change: the access mode, and the storage class. I applied the same two-node deployment with that claim, and watched the pods with my script. Both pods waited twenty-five seconds while the share was created. Then both ran, one on each node. Then one pod wrote a line into the share, and both pods read it back. Written by the first pod. The second pod, on the other node, read the same line. So why not always use a share? A database expects its files to itself, so the module puts databases on disks. Azure Files is for files that many pods share. One node at a time: a disk. Many nodes and the same files: a share.

To sum it up: a container's own files start clean, every time a container starts. A claim keeps data on an Azure disk, outside the pod. ReadWriteOnce means one node, not one pod. For many nodes, use Azure Files.

And that column in the volume list? Reclaim policy: Delete. It says what happens to the disk when its claim is deleted. So I deleted the claim of the two-node test, together with its deployment. Deleting only the deployment would have kept the claim and its disk. Deleted, both. A minute and a half later, I listed the disks again, with the same command as before. One disk left, the exercise's. The test's disk was gone, and the data on it. The built-in storage classes all do that: the Azure disk goes with the claim. To keep it, make your own storage class with reclaimPolicy Retain, and use it in the claim. Then the disk stays after the claim is gone. You delete it yourself, and Azure bills it until you do.

Everything here started with one kubectl apply. The video on screen follows one, second by second, to the app's first answer. If this was useful, subscribe for more practical .NET and Azure.

More in this series

Previous in the series

Your Secret is readable: the Azure Kubernetes Service icon, a base64decode command and the value in plain text, no key needed.Explainer8:24

Kubernetes Secrets on AKS: base64, RBAC and Azure Key Vault

Next in the series

kubectl logs: NotFound: the Azure Kubernetes Service icon and the error from the server, because the pod is gone and its log with it.Explainer9:04

Kubernetes logs on AKS: kubectl logs vs Log Analytics

All 18 videos in Containers on Azure

Let’s talk

Need this in your own Azure environment?

I build, migrate and run .NET platforms on Azure, with Azure DevOps and Terraform. If your team wants this set up properly, see what I offer.

See services