• English icon
  • German icon
Explainer

I changed the ConfigMap. Why does my app still show the old value?

· Series: Containers on Azure

azureakskubernetesconfigmapkubectldevops

I changed one setting in a Kubernetes ConfigMap on Azure Kubernetes Service (AKS): the API version, from 1.0.0 to 2.0.0. kubectl said configured, and the ConfigMap said 2.0.0. My app, still running, answered 1.0.0. No error, no restart.

This explainer shows why a running pod keeps old settings, when it picks up new ones, and a naming trick that turns every settings change into a rollout you can undo. Everything below comes from the Microsoft Learn exercise, run on my own cluster.

Key points

  • An environment variable is a copy taken at the start. Kubernetes copies ConfigMap values into a container's environment once, when the container starts, and never updates them in a running process.
  • Changing a ConfigMap starts nothing. A Deployment rolls out new pods only when its own pod template changes. The template holds the ConfigMap's name, not its values, so the rollout history stayed at one revision.
  • Mounted files follow, a minute or so later. The kubelet refreshes a mounted ConfigMap on its own schedule. In my run the file switched after 81 seconds. A file mounted with subPath never changed, and an app that read its settings at startup never noticed either way.
  • A restart is the update. kubectl rollout restart writes a timestamp into the pod template, so the Deployment starts a new pod, and the new pod copies the current settings. A new, immutable ConfigMap per version does the same with history and undo.

Copied at the start

A ConfigMap is a Kubernetes object that holds settings as keys and values, outside your container image. The exercise puts three settings into one: a name, the API version and a log path. A pod wraps your container; a Deployment keeps your pods running, and its pod template is the recipe for every new pod. In the exercise, that recipe loads the whole ConfigMap with envFrom, a single reference by name, so every key becomes an environment variable:

envFrom:

- configMapRef:

    name: api-config

I applied the change and asked Kubernetes for the value stored in the ConfigMap:

> kubectl get configmap api-config -o jsonpath="{.data.API_VERSION}"

2.0.0

Then I asked the app, over its public address, which version it runs, and the container itself:

> python cfg.py get /healthz

GET /healthz -> 200

{"status":"alive",… "version":"1.0.0","student":"Ehsan"}



> kubectl exec deploy/aks-config-api -- printenv API_VERSION

1.0.0

Environment variables are set once, when the container starts. Kubernetes never rewrites them inside a running process. kubectl describe pod shows both halves of that: where the settings come from, and that the container started before the change, with no restart since.

Started:  Wed, 28 Oct 2026 13:30:55 +0100

Restart Count:  0

Environment Variables from:

api-config  ConfigMap  Optional: false

Kubernetes did keep the new value. Did it roll out a new pod for it? No. The rollout history, the list of every version of the pod template, still had one entry:

> kubectl rollout history deployment aks-config-api

REVISION  CHANGE-CAUSE

1  <none>

A Deployment starts new pods only when its pod template changes. The template holds the ConfigMap's name, not its values, so a changed ConfigMap is not a change to the Deployment.

Mounted as files

A ConfigMap can also be mounted as a volume: every key becomes a file in a folder. I added that to the deployment, and mounted the API version a second time as a single file with subPath:

volumeMounts:

- name: config-volume

  mountPath: /etc/config

- name: config-volume

  mountPath: /etc/api-version

  subPath: API_VERSION

Changing the template started a rollout: a new pod, ready after 22 seconds. The new pod copied the ConfigMap as it was when it started, so it answered 2.0.0. Inside it, /etc/config held one file per key: API_VERSION, LOG_PATH, STUDENT_NAME.

Then I changed the ConfigMap to 3.0.0. A small script of mine applied it and read four places every second for three minutes: the mounted file, the subPath file, the environment variable and the app's answer.

> python cfg.py change k8s/configmap-v3.yaml --secs 180

1.1s  file 2.0.0 | subPath 2.0.0 | env 2.0.0 | app 2.0.0

1.7s  applied: configmap/api-config configured

81.1s  file 3.0.0  (was 2.0.0)

180.2s  file 3.0.0 | subPath 2.0.0 | env 2.0.0 | app 2.0.0
What the pod readsAfter the change
Environment variable (envFrom)never, until the container starts again
Mounted folder (/etc/config)after the kubelet's next sync: 81 s here
Single file with subPathnever, until the container starts again
An app that read its settings at startupnever, until it reads again or restarts

Mounted files follow the ConfigMap, but not right away. The kubelet, the Kubernetes agent on every node, checks mounted ConfigMaps on its periodic sync (once a minute by default) and reads them from its own cache, so expect a minute or two, not a second.

How does a file change without breaking a read halfway? The files are links into a hidden folder:

> kubectl exec deploy/aks-config-api -- ls -la /etc/config

… ..data -> ..2026_10_28_12_33_19.3747003581

… API_VERSION -> ..data/API_VERSION

The kubelet writes a complete new folder, then switches the one ..data link. That's also why the subPath file never changed: subPath mounts the file itself, as it was at the start, not the link, so the switch never reaches it.

And the app? The exercise's API reads its version from the environment variable, once, at startup. The file changed under it, and the app never looked. A mounted file only helps if your code reads the file, and reads it again.

Make it roll out

The fix is one command, and the exercise gives it as a tip:

> kubectl rollout restart deployment aks-config-api

deployment.apps/aks-config-api restarted

Twenty-two seconds later the new pod was ready, the app answered 3.0.0, and the subPath file said 3.0.0 as well. rollout restart doesn't touch the ConfigMap. It writes one annotation into the pod template:

> kubectl get deployment aks-config-api -o jsonpath="{.spec.template.metadata.annotations}"

{"kubectl.kubernetes.io/restartedAt":"2026-10-28T13:35:01+01:00"}

To the Deployment, that is a change: it starts a new pod, and the new pod copies the current settings when it starts. Change the ConfigMap, then restart the deployment.

A rollout you can undo

The naming trick: give the next settings a new ConfigMap with a new name, mark it immutable, and point the deployment at that name.

apiVersion: v1

kind: ConfigMap

metadata:

  name: api-config-v4

data:

  API_VERSION: "4.0.0"

immutable: true

The name is part of the pod template, so applying the deployment rolled out on its own, and the app answered 4.0.0. Every settings change is now a revision: kubectl rollout undo took the deployment back to the template that named api-config, and the app answered 3.0.0 again. And the immutable ConfigMap can't be changed by mistake:

> kubectl apply -f k8s/versioned/configmap-v4-edit.yaml

The ConfigMap "api-config-v4" is invalid: data: Forbidden: field is immutable when `immutable` is set

You don't rename by hand: Kustomize, built into kubectl (kubectl apply -k), can generate the ConfigMap with a hash of its contents in the name and update the references to match. New settings, new name.

What happened when I ran it

Tested on October 28, 2026 in my own Azure subscription (AKS 1.36 in westeurope, Azure CLI 2.88.0, kubectl 1.36); everything was deleted afterwards. The cluster had one Standard_D2s_v5 node (created in 271 s), and the exercise's API answered 51 s after kubectl apply. Each rollout (the files deployment, the restart, the new ConfigMap, the undo) took 22–23 s. The first request right after a rollout twice timed out once before the next one answered; the run's script retried and printed both.

What the video leaves out

  • Azure App Configuration: its Kubernetes provider runs in the cluster and generates ConfigMaps from settings kept in App Configuration, so several clusters share one place for settings.
  • binaryData, optional: true references and the 1 MiB limit per ConfigMap.
  • Helm's checksum annotation on the pod template, the same idea as the naming trick done by a chart.
  • Secrets and persistent storage, the other two parts of the module (separate explainers).

Further reading

Links

Video transcript

I changed one setting in a Kubernetes ConfigMap on Azure Kubernetes Service: the API version, from version one to version two. kubectl, the Kubernetes command line, said configured. The ConfigMap said version two. My app, still running, answered version one. No error, no restart. Nothing.

In this video, you'll see why a running pod keeps old settings, and when it picks up new ones. At the end: a naming trick that makes every settings change a rollout you can undo.

So where did version two go? A ConfigMap is a Kubernetes object that holds settings as keys and values, outside your container image. The Microsoft Learn module's exercise puts three settings in one: a name, the API version, and a log path. A pod wraps your container. A Deployment keeps your pods running, and its pod template is the recipe for every new pod. In the exercise, that recipe loads the ConfigMap with envFrom, a single line that points at it by name. So every key becomes an environment variable. I applied my change and asked Kubernetes for the value stored in the ConfigMap now. Two point zero point zero. The change was there. Then I asked the app itself, over its public address, which version it runs. One point zero point zero. The old value. So I asked the container directly, with kubectl exec and printenv. The same, inside the container: one point zero point zero. Environment variables are set once, when the container starts. Kubernetes never rewrites them inside a running process. I described the pod, to see where its settings come from, and when it started. Environment variables from api-config, the ConfigMap. Started before my change. Restart count: zero. Kubernetes did keep the new value. Did it roll out a new pod for it? No. I asked the Deployment for its rollout history: the list of every version of its pod template. One revision. Nothing rolled out. A Deployment starts new pods only when its pod template changes. The template holds the ConfigMap's name, not its values. So a changed ConfigMap is not a change to the Deployment. An environment variable is a copy, taken at the start. Change the ConfigMap, and nothing that already runs will know.

So what if the settings were files instead? A ConfigMap can also be mounted as a volume: every key becomes a file in a folder. I added that to the deployment, as a folder called config. And I mounted the API version a second time, as a single file, with subPath. Changing the template started a rollout: a new pod, ready after twenty-two seconds. The deploy video showed what happens in those seconds. The new pod copied the ConfigMap as it was when it started, so it answered version two. Inside the new pod, I listed the mounted folder, to see what's really there. API version, log path, student name: one file for each key. Then I changed the ConfigMap again, to version three. A small script of mine applied it, then read four places every second, for three minutes: the file, the subPath file, the environment variable, and the app's answer. How long until the mounted file shows version three? About eighty seconds. Here is the run, three minutes squeezed into a few seconds. Applied at two seconds. The file switched to three at eighty-one. And after three minutes, the subPath file, the environment variable and the app still said version two. Mounted files do follow the ConfigMap, but not right away. The kubelet, the Kubernetes agent on every node, refreshes them on its own schedule. By default it checks once a minute, and it reads the ConfigMap from its own cache. So expect a minute or two, not a second. This series runs one Azure service at a time, for real. Subscribe to follow the series. And how does a file change without breaking a read halfway? I listed the folder again, with the details. The files are links into a hidden folder, called dot dot data. The kubelet writes a complete new folder, then switches that one link. That's why the subPath file never changed. subPath mounts the file itself, as it was at the start, not the link. So the switch never reaches it. And the app? It reads its version from the environment variable, once, at startup. The file changed under it, and the app never looked. A mounted file only helps if your code reads the file, and reads it again. Files follow the ConfigMap, a minute or so later. Your app follows only if it reads them again.

So how does version three reach the app? The fix is one command: kubectl rollout restart. I ran it on the deployment. Restarted. Twenty-two seconds later, the new pod was ready. Then I asked the app again, the same way as before. Version three. And the subPath file said three as well. rollout restart doesn't touch the ConfigMap. I asked the deployment what the restart wrote into its pod template. One annotation: restartedAt, with the time of the restart. To the Deployment, that is a change. So it starts a new pod, and the new pod copies the current settings when it starts. A restart is the update. Change the ConfigMap, then restart the deployment.

To sum it up, before the naming trick: environment variables are copied when the container starts, and never again. Mounted files follow the ConfigMap, a minute or so later, except with subPath. And your app sees a change only when it reads again, or when a restart starts a new pod.

And the naming trick? I created a new ConfigMap for the next settings, called api-config-v4, marked it immutable, and pointed the deployment at that name. The name is part of the pod template. So the deployment rolled out on its own, and the app answered version four. Every settings change is now a revision. So I ran rollout undo. Then I asked the app again. Version three. The old settings came back with the old template. And the immutable one can't be changed by mistake. I tried to apply a new value to it anyway. Forbidden: field is immutable. And you don't rename by hand. Kustomize, built into kubectl, can add a hash of the contents to the name, and update the deployment to match. New settings, new name.

The video on screen deploys to Azure Container Apps, which has the same trap with secrets, and its own fix. If this was useful, subscribe for more practical .NET and Azure.

More in this series

Previous in the series

2 CPUs. 0 pods.: the Azure Kubernetes Service icon, a node with 1900m for pods, two pods asking 1 CPU each in Pending, and Insufficient cpu.Explainer9:17

What happens when a pod uses more than its limit?

All 13 videos in Containers on Azure

Let’s talk

Need this in your own Azure environment?

I build, migrate and run .NET platforms on Azure, with Azure DevOps and Terraform. If your team wants this set up properly, see what I offer.

See services