• English icon
  • German icon
Explainer

Why didn't my Container App scale on a queue?

· Series: Containers on Azure

azureazure-container-appskedamanaged-identitydevops

I put 200 messages into an Azure Storage queue for a worker on Azure Container Apps. For 22 minutes, not one of them was processed. Two fixes later, ten replicas cleared the same queue in about two minutes.

This explainer shows how a Container App finds out that messages are waiting (it doesn't: KEDA counts them for it), the two things my worker was missing, and where the real error was hiding. Everything below comes from a real run in my own Azure subscription.

Key points

  • Without a scale rule, nothing counts your queue. A worker without ingress gets no HTTP requests, so the default HTTP rule has nothing to count. My worker kept the one replica it started with for fifteen minutes; the platform's log said "no triggers defined".
  • A queue rule hands the counting to KEDA. Every 30 seconds KEDA asks the queue how many messages are waiting and divides by the queue length setting: a target per replica, not a threshold. 200 messages ÷ 10 = 20 replicas, capped at the maximum of 10.
  • The rule's identity needs a role. The scale rule signs in with the app's managed identity, but an identity is only a badge. Without a data role on the storage account the scaler can't count, and the app shows no error. With Storage Queue Data Contributor it scaled from 1 to 10.
  • Any message wakes the app; an empty queue lets it go. Three messages at zero replicas woke one replica in 41 seconds. After the queue was empty, all ten replicas stayed five more minutes, then went within 40 seconds.

Nothing is counting

A queue worker has no ingress: no public HTTP endpoint, nobody calls it, it pulls its work from the queue. I created mine like that first: no ingress, a minimum of zero, a maximum of ten, and no scale rule.

Container Apps scales on rules. The default rule counts HTTP requests, and a worker without ingress gets none. So nothing watched the queue: the app kept the one replica it started with, for fifteen minutes, while 200 messages waited. It went neither up to ten nor down to zero. The docs warn that an app like this can scale to zero and never start again; mine simply never changed. Either way, the queue doesn't move the count.

That one replica didn't process a single message either, which leads to the second problem below.

The queue rule

A scale rule is how you tell Container Apps what to watch. Underneath, it runs KEDA, an open-source autoscaler. For a queue rule KEDA polls every 30 seconds, asks the queue how many messages are waiting and computes desiredReplicas = ceil(messages ÷ queueLength), between your minimum and maximum. Picture a post room at night: the desks are the replicas, the mail tray is the queue, and KEDA is the inspector who counts the letters.

The rule I added, with the app's own system-assigned managed identity (no connection string):

az containerapp update -n queue-worker -g rg-demo \
--scale-rule-name queue-scaling \
--scale-rule-type azure-queue \
--scale-rule-metadata accountName=mystorage \
queueName=orders queueLength=10 \
--scale-rule-identity system

The same in the template, as Container Apps stored it:

scale:
minReplicas: 0
maxReplicas: 10
rules:
- name: queue-scaling
azureQueue:
accountName: mystorage
queueName: orders
queueLength: 10
identity: system

The command from the docs failed on my machine. Azure CLI 2.88.0 answered unrecognized arguments: --scale-rule-identity system: that flag comes from the containerapp CLI extension (az extension add -n containerapp), which is in preview. With the extension (1.3.0b5) the rule went in.

The badge and the key

With the rule in place and the identity on, the next four minutes looked exactly like the first fifteen: one replica, 200 messages, no error on the app.

The scale rule authenticates with the app's identity. The identity is the badge; a role is the key. Without a role on the storage account, the door stays locked: KEDA can't read the queue length, and it scales nothing. The worker had the same problem. Its own log said, every ten seconds:

error: HttpResponseError: This request is not authorized to perform this operation using this permission.

The fix was one role assignment for the app's identity on the storage account:

principal=$(az containerapp show -n queue-worker -g rg-demo --query identity.principalId -o tsv)
storageId=$(az storage account show -n mystorage -g rg-demo --query id -o tsv)
az role assignment create --assignee-object-id $principal \
--assignee-principal-type ServicePrincipal \
--role "Storage Queue Data Contributor" --scope $storageId

Storage Queue Data Contributor lets the worker read and delete messages, and it let the scaler count them. 93 seconds after the role assignment, the first message was gone; 13 seconds later, ten replicas were running.

Ten replicas, then none

One replicaTen replicas (my run)
200 messages × 5 s of workabout 17 minutesabout 2 minutes after the first message
Messages per replica20019 to 26

The queue was empty 223 seconds after the role assignment. All ten replicas stayed for five more minutes: KEDA waits five minutes after the queue is empty before it scales down to your minimum, here zero. Then they went within 40 seconds: 10 → 9 → 6 → 3 → 0.

The last test: at zero replicas, three messages, with the queue length still at ten per replica. One replica started 41 seconds later and the three messages were done 25 seconds after that. The queue length sets how many messages each replica should take, not when the first one starts. KEDA's default activation threshold for the queue is zero: any message wakes the app.

Where the real error was hiding

The app itself showed nothing: running, no error. The live system log stream (az containerapp logs show --type system) only said:

unable to get external metric azure-queue-orders for queue-worker: … encountered error

No word about permissions. The workspace behind the Container Apps environment had the full line, in the table ContainerAppSystemLogs_CL:

KEDAScalerFailed: GET https://mystorage.queue.core.windows.net/orders
RESPONSE 403: 403 This request is not authorized to perform this operation using this permission.
ERROR CODE: AuthorizationPermissionMismatch

When a queue worker doesn't scale, query the system logs in Log Analytics, and check the role before anything else.

What happened when I ran it

Tested on October 20, 2026 in my own Azure subscription (westeurope, Azure CLI 2.88.0, containerapp extension 1.3.0b5); everything was deleted afterwards. The worker is my own Python app (azure-storage-queue with ManagedIdentityCredential): it takes one message, works on it for 5 seconds, deletes it. qwatch.py, also my own, sent the messages and polled the replica count and the queue length. The environment took 17.6 minutes to create, the cloud build 68 seconds, the app 41 seconds.

The first try, no ingress and no rule:

> az containerapp show -n queue-worker -g rg-demo --query "{ingress:properties.configuration.ingress, scale:properties.template.scale}" -o json
{
"ingress": null,
"scale": {
"cooldownPeriod": 300,
"maxReplicas": 10,
"minReplicas": 0,
"pollingInterval": 30,
"rules": null
}
}

Ten more minutes of it (after the first three):

3 s replicas 1 queue 200
302 s replicas 1 queue 200
602 s replicas 1 queue 200
summary: first replica: 3 s | peak: 1 replicas at 3 s | queue empty: no | 0 replicas: not reached

After the role, the scale-up, the drain and the scale-in:

80 s replicas 1 queue 200
93 s replicas 1 queue 199
106 s replicas 10 queue 196
223 s replicas 10 queue 0
482 s replicas 10 queue 0
494 s replicas 9 queue 0
507 s replicas 6 queue 0
520 s replicas 3 queue 0
533 s replicas 0 queue 0

Three messages at zero:

28 s replicas 0 queue 3
41 s replicas 1 queue 3
66 s replicas 1 queue 0

My watcher polled about every 13 seconds, so the step from 1 to 10 replicas happened somewhere between 93 and 106 seconds; the docs describe scale-up steps of 1, 4, 8, 16 per evaluation, which my polls couldn't resolve. And one more detail: a CPU rule with a minimum of zero was accepted by the API (az containerapp update printed the rule with empty metadata in its response, the same display quirk as the HTTP rule in the previous video); the docs say CPU and memory rules can't scale to zero, which this run didn't test further.

Other scalers in the module

Event sourceScale rule typeMetric per replicaNote
Azure Storage queueazure-queuequeueLengthmanaged identity or connection string
Azure Service Busazure-servicebusmessageCount50 messages ÷ 5 = 10 replicas; topics need subscriptionName
Azure Event Hubsazure-eventhubunprocessedEventThresholdmore replicas than partitions don't help
Apache KafkakafkalagThresholdthe partition count caps useful replicas
CroncrondesiredReplicas in a time windowwith other rules, the highest count wins

Further reading

Video transcript

I put two hundred messages into a queue for a worker on Azure Container Apps. For twenty-two minutes, not one of them was processed. One replica, the running copy of the app, sat there doing nothing. Two fixes later, ten replicas cleared the same queue in about two minutes.

In this video, you'll see how a Container App finds out that messages are waiting, and the two things mine was missing. And at the end, where the real error was hiding.

Why didn't the worker scale on its own? A queue worker has no ingress, no public HTTP endpoint. Nobody calls it. It pulls its work from the queue. So I created it like that. Then I asked Container Apps for the app's ingress and its scale settings. No ingress, a minimum of zero, a maximum of ten, and no scale rule. Then I sent two hundred messages. Fifteen minutes later, how many replicas were running? Still one. The replica it started with. Not ten, and not zero either. My watcher checked the replicas and the queue every thirty seconds, for ten minutes. One replica, two hundred messages, the whole time. Container Apps scales on rules. The default rule counts HTTP requests, and a worker without ingress gets none. Then I asked for the platform's own system log, the log Container Apps writes about scaling, not my app's output. It said it plainly: no triggers defined. Nothing was watching, not the requests and not the queue. It started one replica when I created it, and with nothing counting, nothing ever changed that. Not up to ten, and not down to zero. And that one replica? It didn't process a single message. We'll get to why.

So who counts the messages? A scale rule. Container Apps runs KEDA, an open-source autoscaler, under the hood. A queue rule tells it which queue to watch. Think of a post room at night. The desks are the replicas, the mail tray is the queue, and KEDA is the inspector who counts the letters. Every thirty seconds, KEDA asks the queue how many messages are waiting, and divides by the queue length setting: the messages each replica should take. That's a target per replica, not a threshold. With a queue length of ten, two hundred messages ask for twenty replicas. My maximum was ten, so ten. Here's the rule I added, with one update command. A rule with a name, of the type azure queue. Then the storage account, the queue name, and ten messages per replica. And the app's own managed identity to sign in. No connection string. I ran it, exactly as the docs show it. It failed: unrecognized arguments. That flag needs the containerapp extension for the Azure CLI, still in preview. With the extension installed, the rule went in. This series runs one Azure service at a time, for real. Subscribe to follow the series.

Did the rule fix it? The rule was in, the identity was on, and two hundred messages were waiting. What happened over the next four minutes? Nothing. My watcher counted the replicas and the queue every fifteen seconds, for four minutes. One replica, two hundred messages, and no error on the app. Back to the post room. The inspector walks to the tray every thirty seconds, counts the letters, and switches on desks. The managed identity is the inspector's badge. But a badge isn't a key. Without a role on the storage account, the door stays locked, and the inspector can't count. The worker had the same problem. I asked Container Apps for the worker's own console log, its last six lines. It said: this request is not authorized to perform this operation using this permission. So I gave the app's identity one role on the storage account: Storage Queue Data Contributor. It lets the worker read and delete messages, and it let the scaler count them. Then I started the watcher again, counting every ten seconds until the app was back at zero. Ninety-three seconds later, the first message was gone. Thirteen seconds after that, ten replicas were running.

How fast did ten replicas clear the queue? Each message took five seconds of work. One replica would need about seventeen minutes for all two hundred. Ten replicas emptied the queue about two minutes after the first message. They shared the work evenly. To check, I counted the processed messages per replica in the app's console logs, across the whole run. Nineteen to twenty-six messages each. Every replica just takes the next message. Then the queue was empty, and all ten stayed for five more minutes. KEDA waits five minutes after the queue is empty before it scales down to your minimum, here zero. My watcher was still counting, every ten seconds. Ten replicas for five minutes. Then they went in forty seconds: nine, six, three, zero. One last test. At zero replicas, I sent three messages. The queue length was still ten per replica. Does a replica start? Yes. Here's that run: the watcher sent the three messages and counted every ten seconds. Forty-one seconds later, one replica was running, and twenty-five seconds after that, the three messages were done. The queue length sets how many messages each replica should take, not when the first one starts. Any message at all wakes the app.

To sum it up. Without a rule, nothing counts your queue, so the replica count doesn't move. A queue rule lets KEDA count the messages every thirty seconds and divide by your target per replica. Any message wakes the app. The rule's identity needs a role on the storage account, or it counts nothing. And when the queue is empty, the replicas stay five more minutes, then go.

And the error from the start. The app itself showed nothing. I asked Container Apps for the live system log stream, the last thirty lines. It only said: unable to get external metric, encountered error. It never said permission. Log Analytics did: the workspace behind the Container Apps environment. I queried its table, Container App System Logs, for this app: the time, the reason and the message, oldest first. There was the full line: KEDA scaler failed. Response four-oh-three, authorization permission mismatch. So when a queue worker doesn't scale, query that table, and check the role before anything else.

My worker was running and doing nothing. The video on screen now shows an app that says healthy while it's broken. If this was useful, subscribe for more practical .NET and Azure.

Need this in your own Azure environment?

I build, migrate and run .NET platforms on Azure, with Azure DevOps and Terraform. If your team wants this set up properly, see what I offer.

See services