• English icon
  • German icon
Explainer

What happens when a pod uses more than its limit?

· Series: Containers on Azure

azureakskubernetesresourceskubectldevops

Every Kubernetes deployment can carry four numbers: a request and a limit, for CPU and for memory. I ran the Microsoft Learn module's example on its own one-node cluster, then pushed pods past their limits on purpose. Kubernetes did three different things, and only one of them showed up as an error.

This explainer shows what requests and limits really do, and what happens past each limit, with the real output from my own Azure Kubernetes Service (AKS) cluster.

Key points

  • A request is a booking. The scheduler adds up the requests of the pods on a node and places a new pod only where its request still fits. It never looks at how busy the node really is, and it never adds up limits. A pod that doesn't fit stays Pending.
  • A node gives pods less than it has. On a two-CPU node, AKS keeps 100 millicores for itself (1,900 are allocatable), and the system pods had already booked 927. The module's example asks for 1,000 per pod: neither of its two pods started.
  • Past the CPU limit, a container waits. The kernel gives it its share of every 100 ms slice. The same work took 3.09 s instead of 1.64 s, and the pod stayed Running with zero restarts.
  • Past the memory limit, the container is killed. OOMKilled, exit code 137, then Kubernetes starts it again. The reason is in the container's last state, not in the events, and its own log just stops. A limit without a request becomes the request.

The request is a booking

A request is what Kubernetes books for your container on a node: so much CPU, so much memory. The scheduler, the part of Kubernetes that picks a node for every new pod, adds up those bookings before it places anything. CPU is counted in millicores: 1,000m is one full CPU.

The module's example asks for two pods, each with 1 CPU and 2 GiB:

resources:

  requests:            # booked by the scheduler

    cpu: "1000m"

    memory: "2Gi"

  limits:              # enforced by the kernel

    cpu: "2000m"

    memory: "4Gi"

The exercise's cluster has one node; mine was a Standard_D2s_v5 with two CPUs and 8 GB. I applied the example and watched both pods for a minute with a small script of mine (like kubectl get pods --watch, with the seconds):

> python watch.py pods api-big --secs 60

0.4s  api-big-6dd546c879-ftqhh  Pending 0/1

0.4s  api-big-6dd546c879-q52gv  Pending 0/1

60.3s  time is up

Both Pending from the first second to the last: no node assigned, no restart, no crash. kubectl describe nodes shows why:

> kubectl describe nodes

Capacity:

cpu:  2

Allocatable:

cpu:  1900m

Resource  Requests  Limits

cpu  927m (48%)  9642m (507%)

Allocatable is 1,900 millicores, not 2,000: AKS keeps 100 for itself on a node this size. The system pods that AKS runs on every cluster (DNS, networking, metrics) had already booked 927. That leaves 973, and each pod asks for 1,000. Not even the first pod gets a place. Memory had room for both; CPU had room for neither.

The same output shows the other half of the rule: the limits on that node added up to 507 % of its CPU. The scheduler only adds up requests. Limits can promise far more than the node has, and real usage doesn't count either.

A booking also stays as long as its pod exists, even if the pod never runs. While I recorded the previous video, my earlier test pods were still on the node, one of them stuck at its image and never started. One more pod asked for only 50 millicores and stayed Pending too:

> kubectl describe pod -l app=api-crash

Requests:

cpu:  50m

FailedScheduling … 0/1 nodes are available: 1 Insufficient cpu. …

When a pod is Pending, describe the pod, then the node. Ask for less, delete pods you don't need, or add a node.

Past the CPU limit

A limit is the ceiling: the most your container may use. The module's hands-on exercise sets a CPU limit of 500m, half a CPU. My demo app has a /work endpoint, a loop that keeps one CPU busy for about a second and a half. I deployed it twice: with the exercise's limits (api-mem) and with a CPU limit of 2 (api-fast), and ran the same request inside each pod:

> kubectl exec deploy/api-fast -- python get.py http://localhost:8000/work

200 {"pod": "api-fast-…", "wall_s": 1.64, "cpu_s": 1.63}



> kubectl exec deploy/api-mem -- python get.py http://localhost:8000/work

200 {"pod": "api-mem-…", "wall_s": 3.09, "cpu_s": 1.54}

With a limit of two, the work had a whole CPU the whole time: 1.64 seconds on the clock. With half a CPU, almost twice as long, for about the same CPU time. A second try gave 1.62 and 3.11.

That's throttling. Every scheduling period (100 ms by default) the kernel gives the container its share, 50 ms for half a CPU, and then makes it wait for the next slice. The work still gets done, in twice as many slices. And Kubernetes doesn't notice:

> kubectl get pods -l app=api-mem

NAME  READY  STATUS  RESTARTS  AGE

api-mem-666bbc8499-kdp82  1/1  Running  0  17s

Over its CPU limit, a container is never killed. It waits. Nothing in the status tells you; you see it as slower answers.

Past the memory limit

Same pod, same exercise settings: it requests 256Mi of memory and its limit is 512Mi (Mi = mebibytes, the binary kind of megabytes). A second endpoint takes memory in steps of 50 MB, holds on to it and prints a line after each step. I asked it for 700.

The module says a request set too low causes these kills. So where does it die, at the request or at the limit? At the limit, and it doesn't slow down first:

> kubectl exec deploy/api-mem -- python get.py "http://localhost:8000/alloc?mb=700"

command terminated with exit code 137

137 is 128 + 9: signal 9, SIGKILL, a kill no program can catch or clean up after. The kernel can make a process wait for the CPU, but it can't take back memory the app is already holding, so when the container goes over its memory limit, the kernel's out-of-memory killer stops it. Where Kubernetes says so:

> kubectl describe pod -l app=api-mem

State:  Running

Last State:  Terminated

Reason:  OOMKilled

Exit Code:  137

Restart Count:  1

Limits:

memory:  512Mi

Requests:

memory:  256Mi

Normal  Started  46s (x2 over 63s) … Container started

The events at the bottom said nothing about it. The reason sits in the container's last state, and Kubernetes had already started the container again, in the same second. What the app said before it died:

> kubectl logs deploy/api-mem --previous

aks-api v1 listening on :8000

holding 50 MB

holding 100 MB

…

holding 400 MB

holding 450 MB

Then nothing: no error, no goodbye. The next 50, plus the app's own memory, crossed 512. Here --previous worked because the new container was already running, so the previous one was the container that got killed (the previous explainer in this series shows when it fails).

And the request: 256Mi. The app held 450 far above it with no problem. A request is not a ceiling; the limit is.

Past the…What happensRestartsWhere you see it
request (memory or CPU)nothing, if the node has room0—
CPU limitthrottled: waits for its share of each 100 ms slice0slower answers
memory limitOOMKilled, exit code 137, restarted+1describe pod → Last State

A limit alone

How can a limit alone keep a pod from ever starting? I gave a pod a CPU limit of 3 and no request at all, then asked Kubernetes which resources it had stored:

> kubectl get pod -l app=api-limits -o jsonpath='{.items[0].spec.containers[0].resources}'

{"limits":{"cpu":"3"},"requests":{"cpu":"3"}}

I never wrote that request. Without a request, Kubernetes copies the limit and books it: three CPUs on a node that gives pods 1.9.

> kubectl describe pod -l app=api-limits

FailedScheduling … 0/1 nodes are available: 1 Insufficient cpu. …

Pending for the whole 30 seconds I watched. So set both: a request for what your pod needs to run, and a limit for what it must never pass.

What happened when I ran it

Tested on October 26, 2026 in my own Azure subscription (AKS 1.36 in westeurope, Azure CLI 2.88.0, kubectl 1.36); everything was deleted afterwards. One AKS cluster served four videos of this series; this page uses its resource steps. The cluster had one Standard_D2s_v5 node (2 vCPU, 8 GB) and the free tier; the app is a small Python API shaped like the module exercise's (port 8000), without the exercise's model in Microsoft Foundry, plus two endpoints for this test: /work (a CPU loop) and /alloc (holds memory in 50 MB steps; its MB are binary, like the Mi of the limit).

  • The node: capacity 2 CPUs and 8,126,896 KiB of memory, allocatable 1,900m and 5,927,344 KiB; 14 system pods had requested 927m CPU (48 %) and 994Mi memory (17 %).
  • The module's example: both pods Pending for 60 seconds, Node: <none>, PodScheduled: False.
  • The OOM kill: the container started at 15:01:19 and was killed at 15:01:36; the new one started in the same second.

What the video leaves out

  • QoS classes (both pods here were Burstable: requests below limits) and node-pressure eviction, where pods using more than their requests are evicted first.
  • How to measure throttling directly (the container's CPU statistics), kubectl top, LimitRange and ResourceQuota (defaults set at admission), the Vertical Pod Autoscaler.

Further reading

Links

Video transcript

Every Kubernetes deployment can carry four numbers: a request and a limit, for CPU and for memory. I ran the Microsoft Learn module's example on its own one-node cluster, then pushed pods past their limits, on purpose. Kubernetes did three different things. Only one of them showed up as an error.

In this video, you'll see what requests and limits really do, and what happens past each limit. And at the end: how a limit alone can keep a pod from ever starting.

What does a request actually do? A request is what Kubernetes books for your container on a node: so much CPU, so much memory. The scheduler, the part of Kubernetes that picks a node for every new pod, adds up those bookings before it places anything. CPU is counted in millicores. A thousand millicores are one full CPU. The module's example asks for two pods, one CPU and two gigabytes each. The exercise's cluster has one node, and mine had two CPUs and eight gigabytes of memory. So how many of the two pods start? Neither of them. I applied the example and watched both pods for a minute, with a small script of mine: like kubectl get pods in watch mode, with the seconds. Both Pending, from the first second to the last. No node assigned, no restart, no crash. So where did two CPUs go? I asked Kubernetes to describe the node, to see what it can give, and what is already booked. Allocatable: nineteen hundred millicores, not two thousand. AKS keeps a hundred for itself on a node this size. And the system pods that AKS runs on every cluster, for DNS, networking and metrics, had already booked nine hundred twenty-seven. That leaves nine hundred seventy-three. Each pod asks for a thousand. It doesn't fit, so not even the first pod gets a place. Memory had room for both. CPU had room for neither. The scheduler only adds up these bookings. It doesn't care how busy the node really is. And a booking stays as long as its pod exists, even if the pod never runs. While I recorded the previous video, about pods that never start, my earlier test pods were still on the node. One more pod asked for only fifty millicores, a twentieth of a CPU, and stayed Pending too. I described it. Insufficient CPU. The earlier pods had booked the rest, even one that never got past pulling its image. The request is a booking. If it doesn't fit, the pod waits in Pending, however idle the node is. Ask for less, delete pods you don't need, or add a node.

And what happens at the CPU limit? A limit is the ceiling: the most your container may use. Limits aren't booked at all: on my node, the limits of the system pods added up to five hundred and seven percent of its CPU. The module's hands-on exercise sets a CPU limit of five hundred millicores, half a CPU. My demo app has a work endpoint: a loop that keeps one CPU busy for about a second and a half. I deployed it twice: once with the exercise's limits, as api-mem, the pod I'll fill with memory later, and once with a CPU limit of two, as api-fast. The half-CPU pod gets work that wants a whole CPU. Is that pod killed, or slowed down? Slowed down. Nothing was killed. First, the pod with the limit of two. I ran the work from inside the pod and asked for the time on the clock, and the CPU time it used. One point six four seconds on the clock, and one point six three of CPU. It had a whole CPU, the whole time. Then exactly the same request, to the pod with the exercise's limit of half a CPU. Three point zero nine seconds. Almost twice as long, for about the same CPU time, one and a half seconds. A second try gave almost the same numbers. That's throttling. Every hundred milliseconds, the kernel gives the container its share: fifty milliseconds for half a CPU. Then it has to wait for the next slice. The work still gets done, in twice as many slices. This series runs one Azure service at a time, for real. Subscribe to follow the series. And did Kubernetes even notice? I listed the pod after both runs. Running, zero restarts. Nothing in the status tells you the pod is throttled. You only see it as slower answers. Over its CPU limit, a container is never killed. It waits.

And when a container needs more memory than its limit? Same pod, same exercise settings: it requests two hundred fifty-six Mi of memory, and its limit is five hundred twelve. Mi means mebibytes, the binary kind of megabytes. My app has a second endpoint that takes memory in steps of fifty megabytes, holds on to it, and prints a line after each step. I asked it for seven hundred. The module says a request set too low causes these kills. So where does it die: at the request, or at the limit? At the limit. And it doesn't slow down first: it's killed. I sent the request from inside the pod, the same way as the work, only to the memory endpoint. Command terminated, exit code one three seven, after one and a half seconds. One three seven is a hundred twenty-eight plus nine: signal nine, a kill that no program can catch or clean up after. Memory isn't like CPU time. The kernel can make a process wait for the CPU, but it can't take back memory the app is already holding. So when the container goes over its limit, the kernel's out-of-memory killer stops it. So where does Kubernetes say what happened? I described the pod and went to its container. Last state: terminated. Reason: OOMKilled, out of memory. Exit code one three seven. Restart count: one. The events at the bottom of the output said nothing about it. The reason sits in the container's last state, and Kubernetes had already started the container again, in the same second. And what did the app say before it died? I asked for the logs of the previous container. Holding fifty megabytes, a hundred, up to four hundred fifty. Then nothing. No error, no goodbye. The next fifty, plus the app's own memory, crossed five hundred twelve. Previous means the container before the current one, and here that was the one that got killed. The previous video shows when that fails. And notice the request: two hundred fifty-six. The app held four hundred fifty, far above it, with no problem. A request is not a ceiling. The limit is. Memory can't wait. Over its limit, the container is killed, and Kubernetes starts it again.

To sum it up: a request is a booking. The scheduler adds them up, and a pod that doesn't fit waits in Pending. Over the CPU limit, the container waits for its share, and gets slower. Over the memory limit, it's killed and restarted: OOMKilled, exit code one three seven.

And how can a limit alone keep a pod from ever starting? I gave a pod a CPU limit of three, and no request at all. Then I asked Kubernetes which resources it had actually stored for that pod, straight from the pod's definition. Limits: CPU three. Requests: CPU three. I never wrote that request. Without a request, Kubernetes copies the limit and books it. Three CPUs, on a node that gives pods one point nine. I described the pod, to see what the scheduler said. Insufficient CPU, and Pending for the whole thirty seconds I watched. So set both: a request for what your pod needs to run, and a limit for what it must never pass.

Pending is only the first stop. The video on screen follows a pod from kubectl apply to its first answer, second by second. If this was useful, subscribe for more practical .NET and Azure.

More in this series

Previous in the series

Pod won't start: the Azure Kubernetes Service icon next to the errors ImagePullBackOff, CrashLoopBackOff and CreateContainerConfigError, and the command kubectl describe pod.Explainer9:33

My pod won't start. Which command shows why?

Next in the series

ConfigMap, old value: the Azure Kubernetes Service icon, the command kubectl rollout restart and the result new pod, new value.Explainer7:48

I changed the ConfigMap. Why does my app still show the old value?

All 13 videos in Containers on Azure

Let’s talk

Need this in your own Azure environment?

I build, migrate and run .NET platforms on Azure, with Azure DevOps and Terraform. If your team wants this set up properly, see what I offer.

See services