My new version crashed on Azure Kubernetes Service (AKS): six crashes in under four minutes. I rolled it back with one command, and the app kept answering the whole time. Then I asked kubectl why it had crashed. Not found. The logs had gone with the pod.
This explainer shows where a pod's logs really live, why the events forget the crash after an hour, and how Container insights keeps a copy you can still query later. Everything below comes from the Microsoft Learn exercise, run on my own cluster.
Key points
kubectl logsreads files on the node, only for containers that still exist. Through the API server, it asks the kubelet on the pod's node to read the container's log file. A replaced, deleted or evicted pod takes its files with it.--previousonly reaches back inside the same pod. It reads the container before the current one. After a rollback the crash lived in a different pod, so there is nothing to read.- Events last one hour on AKS.
kubectl get eventsshowed the BackOff warning right after the rollback and nothing an hour later. The portal's live views forget too. - Container insights keeps a copy outside the cluster. An agent on every node sends the container output to a Log Analytics workspace (
ContainerLogV2) and the Warning events toKubeEvents. More than an hour later, a KQL query still found five of the six crashes.
Where kubectl logs reads from
In the exercise, a small Python API runs in the namespace aks-troubleshoot. A Deployment keeps one pod of it running, and the app refuses to start without its API key. The exercise's broken version is the same Deployment with the variable commented out (the value is the exercise's demo key):
# ERROR: API_KEY environment variable removed, causing application crash# env:# - name: API_KEY# value: "demo-api-key-12345"
I applied it, and a small script of mine printed every change of the pods with the restart count (trimmed):
> kubectl apply -f k8s/crashloop-deployment.yamldeployment.apps/api-deployment configured> python pods.py --secs 1500.4s api-deployment-64598b4ccd-mjkwf ContainerCreating 0/1 restarts 00.4s api-deployment-695945c854-nx6qj Running 1/1 restarts 04.9s api-deployment-64598b4ccd-mjkwf Running 0/1 restarts 1 last exit 120.7s api-deployment-64598b4ccd-mjkwf Running 0/1 restarts 2 last exit 132.1s api-deployment-64598b4ccd-mjkwf CrashLoopBackOff 0/1 restarts 2 last exit 193.4s api-deployment-64598b4ccd-mjkwf Running 0/1 restarts 4 last exit 1120.7s api-deployment-64598b4ccd-mjkwf CrashLoopBackOff 0/1 restarts 4 last exit 1
The old pod …nx6qj kept running with zero restarts: a Deployment keeps the old pod until the new one is ready. A client pod calling the Service every second logged no failed call. The exercise's own log command, by label, found the reason:
> kubectl logs -l app=api2026-11-02 14:01:45,119 - ERROR - FATAL: API_KEY environment variable is not set. Exiting.2026-11-02 14:02:39,658 - INFO - GET /healthz - Health check requested…
Where that line lives. The container writes to standard out, and the node stores it in a file. kubectl logs asks the cluster's API server, which asks the kubelet on that node, the agent that runs the pods. The kubelet reads the file and streams it back. A helper pod that can see the node's disk (a node debug pod) listed the log folders of the two pods:
> python nodelogs.py$ kubectl debug node/aks-nodepool1-21626672-vmss000000 --attach --image=mcr.microsoft.com/cbl-mariner/busybox:2.0 -- ls /var/log/pods (filtered)/var/log/pods on the node:aks-troubleshoot_api-deployment-64598b4ccd-mjkwf_<uid> 5.logaks-troubleshoot_api-deployment-695945c854-nx6qj_<uid> 0.log
One folder per pod, one file per container start, numbered from zero. The crashed pod was at 5.log, its sixth start; the files of the earlier starts were already gone. Then I rolled back and asked for the crashed pod's log by name:
> kubectl rollout undo deployment api-deploymentdeployment.apps/api-deployment rolled back> kubectl logs api-deployment-64598b4ccd-mjkwferror: error from server (NotFound): pods "api-deployment-64598b4ccd-mjkwf" not found in namespace "aks-troubleshoot"
The rollback went back to the working version and deleted the crashed pod. Did the node keep its files? The folder, yes. The files, no:
> python nodelogs.py/var/log/pods on the node:aks-troubleshoot_api-deployment-64598b4ccd-mjkwf_<uid>aks-troubleshoot_api-deployment-695945c854-nx6qj_<uid> 0.log
More than an hour later, the crashed pod's folder was gone as well. When a pod goes, its log files go with it. Even while it lives, the kubelet keeps only the current container's log and the one before it. kubectl logs can only read what's still there.
Events: gone after one hour
Events are Kubernetes' own notes about what happened to an object: scheduled, pulled, started, back off. Right after the rollback, the warning a crash loop leaves behind was still there, for a pod that no longer existed:
> kubectl get events --field-selector reason=BackOffLAST SEEN TYPE REASON OBJECT MESSAGE19s Warning BackOff pod/api-deployment-64598b4ccd-mjkwf Back-off restarting failed container api in pod api-deployment-64598b4ccd-mjkwf_aks-troubleshoot(<uid>)
An hour later, the same command:
> kubectl get events --field-selector reason=BackOffNo resources found in aks-troubleshoot namespace.
On AKS, events are available for one hour after they are generated. There is no history behind them. The Azure portal doesn't help here either: its Live Logs and Live Events read the same Kubernetes API as kubectl, so they only see what still exists.
Container insights: a copy outside the cluster
Logs that outlive the pod have to leave the cluster while the pod is still alive; the app doesn't change. On AKS that's Container insights, part of Azure Monitor. I turned it on before the crash, with my own Log Analytics workspace (without --workspace-resource-id, the CLI creates a default workspace outside your resource group):
$ws = az monitor log-analytics workspace show -g rg-demo -n law-demo --query id -o tsvaz aks enable-addons -a monitoring -g rg-demo -n aks-demo --workspace-resource-id $ws
The add-on took 142 seconds. It runs an agent on every node as a DaemonSet, one copy of a pod per node:
> kubectl get daemonset ama-logs -n kube-systemNAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGEama-logs 1 1 1 1 1 <none> 9m6s
The agent reads the same log files as kubectl and sends every line to the workspace, into the table ContainerLogV2. Warning events go to a second table, KubeEvents; Normal events are left out by default. More than an hour after the crash, this query (KQL, the query language of Log Analytics) asked for every fatal line in the namespace:
ContainerLogV2| where PodNamespace == "aks-troubleshoot"| where LogMessage has "FATAL"| project TimeGenerated, PodName, LogMessage| order by TimeGenerated asc
14:00:15 api-deployment-64598b4ccd-mjkwf 2026-11-02 14:00:15,612 - ERROR - FATAL: API_KEY environment variable is not set. Exiting.14:00:31 api-deployment-64598b4ccd-mjkwf 2026-11-02 14:00:31,011 - ERROR - FATAL: API_KEY environment variable is not set. Exiting.14:00:58 api-deployment-64598b4ccd-mjkwf 2026-11-02 14:00:58,043 - ERROR - FATAL: API_KEY environment variable is not set. Exiting.14:01:45 api-deployment-64598b4ccd-mjkwf 2026-11-02 14:01:45,119 - ERROR - FATAL: API_KEY environment variable is not set. Exiting.14:03:12 api-deployment-64598b4ccd-mjkwf 2026-11-02 14:03:12,051 - ERROR - FATAL: API_KEY environment variable is not set. Exiting.5 rows
Five lines, all from the pod kubectl couldn't find any more. Five of the six crashes: the very first one, seconds after the pod started, never arrived. The BackOff warnings that kubectl get events had forgotten were still in KubeEvents, with their counts:
KubeEvents| where Namespace == "aks-troubleshoot" and Reason == "BackOff"| project TimeGenerated, Name, Reason, Message, Count| order by TimeGenerated asc
14:00:16 api-deployment-64598b4ccd-mjkwf Back-off restarting failed container api in pod api-deployment-64598b4ccd-mjkwf_aks-troubleshoot(<uid>) x1… (the same event with the counts x5, x7, x8, x9)5 rows
The copy isn't instant. Right after the rollback, the workspace had only one of the crashes; the rest arrived later. Asking for the newest line from the namespace showed the gap (118 seconds on a first try, 99 on this one):
newest line in the workspace: 15:05:46 UTC, now: 15:07:25 UTC, 99 s behind
A copy can lag or miss a line. It also costs money: the workspace bills by the gigabyte it takes in, and the first 5 GB per billing account each month are free. Analytics tables keep the data for 30 days by default. The cluster forgets; the workspace keeps what the agent copied.
Three ways to lose the evidence
Three habits lose the crash:
| What you do | What happens | In my run |
|---|---|---|
kubectl logs deployment/<name> | it picks one pod, not every pod | "Found 2 pods, using" the healthy one: 517 lines, not one fatal |
kubectl logs <pod> --previous | it reads the container before the current one, only inside the same pod | after the rollback: BadRequest, the pod left never had a previous container |
| rolling back first | the rollout deletes the crashed pod and its log files | NotFound |
> kubectl logs deployment/api-deploymentFound 2 pods, using pod/api-deployment-695945c854-nx6qj2026-11-02 13:58:45,055 - INFO - Application started successfully…> kubectl logs deployment/api-deployment --previousError from server (BadRequest): previous terminated container "api" in pod "api-deployment-695945c854-nx6qj" not found
Save the crash log first. Roll back, but before that, write what the crashed pod said into a file: kubectl logs <pod> > crash.log, with its name, while it still exists. Or turn on Container insights before you need it.
The log command that hid 720 lines
While everything was healthy again, I counted lines with PowerShell: the whole log of the pod that was left, the exercise's command, and the same with --tail=-1:
> (kubectl logs deployment/api-deployment).Count730> (kubectl logs -l app=api).Count10> (kubectl logs -l app=api --tail=-1).Count732
With a label selector, kubectl logs shows only the last 10 lines of each pod. Without one, it shows everything; --tail=-1 means every line (two more had arrived in the meantime). Every health check and every call writes a line, so the log grows fast. In the first chapter, the fatal line showed up only because the crashed container had printed just that one line. A fatal line eleven lines from the end would never have been on my screen.
What happened when I ran it
Tested on November 2, 2026 in my own Azure subscription (AKS 1.36 in westeurope, Azure CLI 2.88.0); everything was deleted afterwards. The cluster had one Standard_D2s_v5 node (free tier, Azure CNI, the registry attached, as in the exercise) and a workspace in the same region. Creating it took 294 seconds, the monitoring add-on 142. Right after the add-on returned, kubectl rollout status daemonset ama-logs still answered NotFound; the DaemonSet appeared seconds later. kubectl rollout undo warned that the Deployment's last-applied annotation from kubectl apply would not be updated. The NotFound and BadRequest answers above exited with code 1. Each KQL query took 3.5 to 4.2 seconds, the node listings about 2.3; the kubectl commands above answered in under a second each.
What the video leaves out
- The portal views (Workloads, Live Logs, Live Events, Diagnose and solve problems), which read the same API as
kubectl. - Metrics:
kubectl top, Prometheus and Grafana. - Collecting Normal events too (
collect_all_kube_eventsin the agent's ConfigMap) and the other collection settings. - Structured logs and correlation IDs, which make the copy easier to search.
- The module's other two breaks (a readiness probe on the wrong path, a Service selector that matches nothing): separate explainers.
Further reading
- Microsoft Learn module: Monitor and troubleshoot applications on Azure Kubernetes Service
- kubectl logs (Kubernetes reference)
- Use Kubernetes events for troubleshooting in AKS
- Enable monitoring for AKS
