Monitor Azure OpenAI in .NET — Token Usage, Cost & Latency with OpenTelemetry and Application Insights

Every chat completion costs money and takes time — and by default neither shows up anywhere. No token counts, no latency numbers, no trace of what your app actually sent downstream. The first time anyone asks "what does our AI feature cost per day?" or "why was that request slow?", the honest answer is: nobody knows.

This walkthrough fixes that with OpenTelemetry and Application Insights: distributed traces across the whole request, token usage as queryable custom metrics, and an optional cost estimate per request. It builds directly on the keyless authentication setup from the previous tutorial — a .NET minimal API calling a Microsoft Foundry resource with no API keys anywhere. That part is the baseline here; you can diff the two repos to see exactly what adding observability takes.

The three signals

Every /chat request produces three kinds of telemetry in Application Insights:

• A distributed trace — the inbound HTTP request, a custom chat completion span carrying the deployment name and token counts as tags, and the outbound Azure OpenAI HTTP call as its child. One click answers "where did the time go?".
• Custom metrics — chat.tokens.prompt, chat.tokens.completion and chat.tokens.total as counters, chat.request.duration as a latency histogram. All tagged with the deployment name, so every chart can be split by model.
• A cost estimate — chat.request.cost in USD per request, recorded only when pricing is configured.

Wiring up OpenTelemetry

The Azure Monitor distro does the heavy lifting — one call wires up traces, metrics and logs, including ASP.NET Core and HttpClient instrumentation, so the outgoing Azure OpenAI calls show up in traces automatically. Only the custom instruments need registering on top:

builder.Services.AddOpenTelemetry()
    .UseAzureMonitor()
    // Our own instruments (token counts, latency) from Telemetry.cs.
    .WithMetrics(metrics => metrics.AddMeter(Telemetry.SourceName))
    // Our own spans around each model call.
    .WithTracing(tracing => tracing.AddSource(Telemetry.SourceName));

UseAzureMonitor() reads APPLICATIONINSIGHTS_CONNECTION_STRING by convention — locally from a gitignored appsettings.local.json, on App Service from an app setting that Terraform provisions. No connection string, and the app simply runs unobserved.

Instrumenting the model call

The interesting part is what happens around the actual model call. A custom activity wraps it, and the token usage that every non-streaming completion already carries gets read and recorded:

using var activity = Telemetry.ActivitySource.StartActivity("chat completion");
activity?.SetTag("gen_ai.request.model", deployment);

var stopwatch = Stopwatch.StartNew();
ChatCompletion completion = await chatClient.CompleteChatAsync(request.Message);
stopwatch.Stop();

// Every non-streaming completion carries its token usage —
// it only has to be read and recorded.
var tags = new TagList { { "gen_ai.request.model", deployment } };
Telemetry.PromptTokens.Add(completion.Usage.InputTokenCount, tags);
Telemetry.CompletionTokens.Add(completion.Usage.OutputTokenCount, tags);
Telemetry.TotalTokens.Add(completion.Usage.TotalTokenCount, tags);
Telemetry.RequestDuration.Record(stopwatch.Elapsed.TotalMilliseconds, tags);

The cost estimate is deliberately simple: input and output prices per one million tokens in appsettings.json (Pricing:InputPer1MTokens / Pricing:OutputPer1MTokens), multiplied by the token counts of each request. Prices change, so both default to 0 — which keeps the metric off until you look up the current pricing for your model and region. When enabled, each request records chat.request.cost and tags its span with chat.estimated_cost_usd.

Querying it in Application Insights

The custom metrics land in the customMetrics table, the custom spans in dependencies — both queryable with KQL. Tokens per day:

customMetrics
| where name == "chat.tokens.total"
| summarize tokens = sum(valueSum) by bin(timestamp, 1d)
| render columnchart

And the top-10 most expensive requests, straight from the span tags:

dependencies
| where name == "chat completion"
| extend cost_usd = todouble(customDimensions["chat.estimated_cost_usd"])
| top 10 by cost_usd
| project timestamp, duration, cost_usd

You don't have to build the dashboards yourself: the Terraform also deploys an Azure Workbook ("Azure OpenAI Observability", under Monitor > Workbooks) with tokens over time, latency p50/p95, estimated cost, requests per model and the most-expensive-requests table.

Run it

az login

cd infra
cp terraform.tfvars.example terraform.tfvars   # fill in subscription + your object ID
terraform init
terraform apply

This provisions everything from the keyless setup plus a Log Analytics workspace, Application Insights and the workbook. Point the app at the openai_v1_endpoint and application_insights_connection_string outputs via appsettings.local.json, then dotnet run and send a request.

A single request makes for empty charts, though. The repo ships a small load generator that sends a batch of varied prompts so the token, latency and cost charts have something to show:

cd tools/load-generator
dotnet run    # 40 requests to http://localhost:5002

Deployment is one dotnet publish and one az webapp deploy; the exact commands are in the repo's README. On App Service, the same code reports through the connection string Terraform already set — no code change, no config edit.

What you'll need

• An Azure subscription
• Azure CLI, logged in with az login
• .NET 10 SDK
• Terraform >= 1.9

In a real enterprise setup there's more on top — alerting on cost and latency thresholds, sampling strategies for high traffic, and dashboards per team — but traces, token metrics and cost per request are the foundation every one of those builds on.