How to test a Kubernetes Service on AKS (not with port-forward)

I tested a Service on Azure Kubernetes Service (AKS) with kubectl port-forward, the way the Microsoft Learn module does: 200, OK. Then a pod inside the cluster called the same Service: connection refused. Twice.

This explainer shows what port-forward really connects to, why a pod that never gets ready still answers it, and how to test a Service for real. Everything below comes from the Microsoft Learn exercise and a few small additions of mine, run on my own cluster.

Key points

• port-forward uses the Service only to find a pod. kubectl reads the Service's selector, picks one matching pod and opens a tunnel to it through the API server and the kubelet. Six requests through port-forward reached one pod; the same six calls through the Service, from inside the cluster, reached three.
• port-forward ignores readiness. A pod that failed its readiness probe (Running, 0/1) answered port-forward with 200, while both calls through its Service were refused. The Service sends calls only to ready pods.
• The endpoint slice and the Endpoints line answer different questions. The slice lists every pod that matches the labels, ready or not. The Endpoints: line of kubectl describe service lists only the ready ones, the ones that get traffic.
• To test a Service, call it from inside the cluster, by its name. Check the endpoint slices and the Endpoints line first. A port-forward tests one pod, and it dies with that pod.

Where port-forward really goes

In the exercise, a small Python API runs in the namespace aks-troubleshoot, behind a Service: a stable address that spreads calls over the app's pods. Its selector says which pods belong to it, and it maps its port 80 to the container's port 8000:

apiVersion: v1
kind: Service
metadata:
  name: api-service
spec:
  type: ClusterIP
  selector:
    app: api
  ports:
  - name: http
    port: 80
    targetPort: 8000
    protocol: TCP

The exercise checks it with port-forward: kubectl opens local port 8080 and forwards it to the Service's port 80, and a request to localhost answers. pf.py is my small wrapper around that step: it starts kubectl port-forward, prints kubectl's own lines with the seconds since the start, sends the requests to localhost:8080 and stops the forward again.

> python pf.py service/api-service /healthz
  0.0s  $ kubectl port-forward service/api-service 8080:80
  0.5s  kubectl: Forwarding from 127.0.0.1:8080 -> 8000
  0.5s  kubectl: Forwarding from [::1]:8080 -> 8000
  1.2s  kubectl: Handling connection for 8080
  1.3s  GET /healthz -> 200  {"status":"healthy","timestamp":"2026-11-02T13:58:56.934290"}
  2.3s  forward stopped

Look at the arrow: kubectl forwards to 8000, the container's port, not to the Service's port 80. Then I scaled the app to three replicas, three pods behind the same Service. The app's /api/info answers with the start time of its pod, so every answer says which pod replied.

> kubectl scale deployment api-deployment --replicas 3
deployment.apps/api-deployment scaled
> python pf.py service/api-service /api/info --n 6
  0.0s  $ kubectl port-forward service/api-service 8080:80
  0.6s  kubectl: Forwarding from 127.0.0.1:8080 -> 8000
  0.6s  kubectl: Forwarding from [::1]:8080 -> 8000
  1.1s  kubectl: Handling connection for 8080
  1.3s  GET /api/info -> 200  pod started 13:58:45.05
  1.3s  kubectl: Handling connection for 8080
  1.4s  GET /api/info -> 200  pod started 13:58:45.05
  1.4s  kubectl: Handling connection for 8080
  1.5s  GET /api/info -> 200  pod started 13:58:45.05
  1.5s  kubectl: Handling connection for 8080
  1.6s  GET /api/info -> 200  pod started 13:58:45.05
  1.6s  kubectl: Handling connection for 8080
  1.7s  GET /api/info -> 200  pod started 13:58:45.05
  1.7s  kubectl: Handling connection for 8080
  1.8s  GET /api/info -> 200  pod started 13:58:45.05
  2.9s  forward stopped

Six answers, one pod. Then the same six calls from inside the cluster: a client pod I had started there runs a tiny Python script that calls the Service by its name.

> kubectl exec client -- python /c/call.py http://api-service/api/info 6
1 200 pod started 14:16:35.84
2 200 pod started 14:16:35.66
3 200 pod started 14:16:35.84
4 200 pod started 14:16:35.66
5 200 pod started 13:58:45.05
6 200 pod started 14:16:35.66

Three different pods: the Service spread them out. The reason: kubectl doesn't send your request to the Service. It opens a tunnel through the cluster's API server, the front door for every kubectl command, to the kubelet, the agent on the node, and from there into one pod. It reads the Service's selector, the label rule that says which pods belong to it, and picks one matching pod. That's all it uses the Service for.

port-forward tests one pod. The Service is everything in front of the pods.

A pod that never gets ready

A readiness probe is the check that decides whether a pod gets traffic. I deployed a second app, api-unready, with its own Service and with the exercise's broken probe: a check that asks for a path that doesn't exist.

readinessProbe:
  httpGet:
    path: /invalid-path
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 10

pods.py is another small script of mine: it prints the app's pods whenever one changes.

> kubectl apply -f k8s/unready.yaml
deployment.apps/api-unready created
service/api-unready created
> python pods.py --app api-unready --secs 30
   0.3s  api-unready-64cc88bb49-w6vpd  ContainerCreating 0/1  restarts 0
   2.5s  api-unready-64cc88bb49-w6vpd  Running           0/1  restarts 0

Running, 0/1: it runs, but it never gets ready. Does it show up in the Service's endpoint slice, the list of addresses behind the Service? Yes:

> kubectl get endpointslices -l kubernetes.io/service-name=api-unready
NAME                ADDRESSTYPE   PORTS   ENDPOINTS    AGE
api-unready-8srvl   IPv4          8000    10.224.0.6   32s

The Service itself says something else (the output shortened to the lines that matter):

> kubectl describe service api-unready
Selector:                 app=api-unready
Type:                     ClusterIP
Port:                     http  80/TCP
TargetPort:               8000/TCP
Endpoints:

Then the same two paths as before. port-forward to the new Service, with the same command, and a call through the Service's name from the client pod:

> python pf.py service/api-unready /api/info
  0.0s  $ kubectl port-forward service/api-unready 8080:80
  0.7s  kubectl: Forwarding from 127.0.0.1:8080 -> 8000
  0.7s  kubectl: Forwarding from [::1]:8080 -> 8000
  1.3s  kubectl: Handling connection for 8080
  1.5s  GET /api/info -> 200  pod started 14:19:54.66
  2.5s  forward stopped
> kubectl exec client -- python /c/call.py http://api-unready/api/info 2
1 FAIL URLError <urlopen error [Errno 111] Connection refused>
2 FAIL URLError <urlopen error [Errno 111] Connection refused>

port-forward doesn't care whether a pod is ready. It only needs a running pod. The readiness probe exists to keep traffic away from a pod like this, and port-forward walks right past it.

When no pod matches the selector

So does port-forward ignore the Service completely? Not quite. The exercise has a second break: it changes the Service's selector to a label that no pod has. The pods are healthy and ready; the Service just stops looking for them.

  selector:
    app: api-v2

> kubectl get endpointslices -l kubernetes.io/service-name=api-service
NAME                ADDRESSTYPE   PORTS     ENDPOINTS   AGE
api-service-mvjnn   IPv4          <unset>   <unset>     20m
> kubectl exec client -- python /c/call.py http://api-service/api/info 2
1 FAIL URLError <urlopen error [Errno 111] Connection refused>
2 FAIL URLError <urlopen error [Errno 111] Connection refused>
> python pf.py service/api-service /healthz
  0.0s  $ kubectl port-forward service/api-service 8080:80
 60.3s  kubectl: error: timed out waiting for the condition
 61.0s  kubectl exited with code 1

<unset>: no pod matches, so there is no address at all. The client was refused, and port-forward waited a minute for a pod to pick, then gave up. Both failed, for different reasons: the Service had no endpoint, and port-forward had no pod to pick. Only the endpoint slice showed why.

How to test a Service for real

Three checks, in this order:

# 1. Which pods match the selector? <unset> = none
kubectl get endpointslices -l kubernetes.io/service-name=api-service

# 2. Which of them are ready? The Endpoints line
kubectl describe service api-service

# 3. Call it the way your other apps do: from a pod, by its name
kubectl exec client -- python /c/call.py http://api-service/api/info 6

1. The endpoint slices, by the Service's name label. <unset> means no pod matches: check the labels on both sides, the Service's selector and the pod template. An address there means a pod matches, not that it's ready.
2. Describe the Service. Its Endpoints: line lists only the ready pods. Empty while pods are running means a readiness problem: look at the pod's events.
3. Call it from inside the cluster, from a pod, by the Service's name. Mine was the small Python client; any pod with curl does the same. That's the path your other apps use: the cluster's DNS turns the name into the Service's address, and the Service spreads the call over every ready pod.

If those three look right and the call still fails, look at the ports: the Service's port, and the targetPort, the port the container listens on.

Keep port-forward for what it is: a quick, private line to one pod, for TCP only (no UDP). Great for looking at one pod from your laptop; not proof that the Service works. And because the tunnel goes through the API server, it can also bypass the cluster's network-level controls, so who may use it is worth restricting.

When the pod behind the forward goes away

What happens to an open port-forward when its one pod goes away? I opened it again, got two answers from the pod that started at 13:58, and then deleted exactly that pod (kubectl's "Handling connection" lines left out; its full error is below):

> python pf.py service/api-service /api/info --kill
  0.0s  $ kubectl port-forward service/api-service 8080:80
  0.6s  kubectl: Forwarding from 127.0.0.1:8080 -> 8000
  1.2s  GET /api/info -> 200  pod started 13:58:45.05
  1.4s  GET /api/info -> 200  pod started 13:58:45.05
  1.6s  deleting the pod behind the forward: api-deployment-695945c854-nx6qj
  8.0s  kubectl: error: lost connection to pod
  8.0s  GET /api/info -> FAIL  ConnectionResetError [WinError 10054] An existing connection was forcibly closed
 13.1s  GET /api/info -> FAIL  URLError <urlopen error [WinError 10061] No connection could be made
 18.2s  GET /api/info -> FAIL  URLError <urlopen error [WinError 10061] No connection could be made
 20.2s  kubectl had already exited with code 1

The forward died with its pod. It doesn't move on to another one, even though two more pods were still behind the same Service. Meanwhile, the client inside the cluster kept calling the Service, once a second:

> kubectl logs client --tail 8
14:20:43 200 pod started 14:16:35.66
14:20:44 200 pod started 14:16:35.84
14:20:45 200 pod started 14:20:34.35
14:20:46 200 pod started 14:16:35.66
14:20:47 200 pod started 14:20:34.35
14:20:48 200 pod started 14:16:35.66
14:20:49 200 pod started 14:20:34.35
14:20:50 200 pod started 14:16:35.84

200 every second, from the other two pods and from the new one (started 14:20:34) that replaced the deleted pod.

What happened when I ran it

Tested on November 2, 2026 in my own Azure subscription (AKS 1.36 in westeurope, Azure CLI 2.88.0); everything was deleted afterwards. The cluster had one node (Standard_D2s_v5, Kubernetes v1.36.4) with the container registry attached, as in the exercise, and kubectl 1.36.1; the exercise's app and manifests ran unchanged in its namespace. My additions: the client pod, the second app api-unready with the exercise's broken probe, and the two helper scripts. The apply, scale, get and describe commands each took under a second, the calls from the client pod 1.5 to 1.7 s per run, and the port-forward with no matching pod gave up after 60.3 s with exit code 1. The pod that never got ready was Running after 2.5 s and stayed 0/1 for the 30 s I watched it. In the last test, the pod was deleted 1.6 s after the start; the next request, at 8.0 s, found the tunnel gone. kubectl's full error at that moment:

an error occurred forwarding 8080 -> 8000: error forwarding port 8000 to pod 76384ad233c3f04a440c8448770a1e70b5f0486b6470954bec93a24e07438900, uid : network namespace for sandbox "76384ad233c3f04a440c8448770a1e70b5f0486b6470954bec93a24e07438900" is closed

What the video leaves out

• The path from outside: Ingress and a LoadBalancer Service's external IP (a separate explainer follows one kubectl apply through the Azure load balancer to the first answer).
• NodePort Services and kubectl proxy.
• UDP: kubectl port-forward is implemented for TCP ports only.

Further reading

• Microsoft Learn module: Monitor and troubleshoot applications on Azure Kubernetes Service
• Use port forwarding to access applications in a cluster (Kubernetes docs)
• EndpointSlices and Service (Kubernetes docs)