DevOpsInterviews logo
← Containers & Kubernetes / 08
easyNewGoogleMicrosoftAccenture

How does a Service know which pods belong to it? Explain labels and selectors.

Selectors match object labels; owner references record controller relationships. Keeping those mechanisms separate explains how a Service can select the wrong pods without owning them.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: Labels are key-value pairs on objects; a selector is a query over them. A Service holds a selector and continuously includes every pod that matches it and passes readiness. Nothing is registered or attached, so a label typo produces a Service with no endpoints and no error anywhere.

How to approach it

The point to land is that the relationship is a live query rather than a link. Once that is clear, the debugging answer follows immediately: if a Service is not working, ask it which pods it currently matches, and the answer is usually none. Show the command that does that.

A strong answer

A label is arbitrary metadata you attach to an object:

metadata:
  labels:
    app: checkout
    tier: backend

A selector is a query. A Service says which labels it wants:

spec:
  selector:
    app: checkout

Kubernetes evaluates that continuously. Any pod in the same namespace carrying app: checkout is a candidate, and if it is passing its readiness probe it goes into the Service's endpoint list and starts receiving traffic. Delete the pod and it drops out. Add a new one with that label and it joins. Nothing announced itself.

Selectors are used by many resources, but controller ownership also uses owner references. A Deployment manages its ReplicaSets, which in turn manage pods. A NetworkPolicy uses selectors for both the pods it protects and the pods allowed to reach them. A ServiceMonitor uses one to find what to scrape.

Which makes the debugging move obvious. When a Service returns nothing, check what it actually matches:

kubectl get endpoints -n prod checkout
kubectl get pods -n prod -l app=checkout

endpoints empty and the labelled pod list empty means the selector and the pod labels disagree, which is nearly always app: checkout against app: checkout-service, or a label on the Deployment that never reached the pods. An empty ready endpoint list with matching pods calls for checking readiness, pod addresses and endpoint reconciliation. A failed readiness probe is one possible cause.

One detail that catches people out: labels on a Deployment are not automatically on its pods. The Deployment has its own metadata.labels, and separately a spec.template.metadata.labels for the pods it creates. The Service selects the pods, so it is the template labels that matter. Setting the first and not the second is a common and confusing mistake.

Another: a Deployment's spec.selector is immutable after creation. Change it and the apply is rejected, so you delete and recreate. That is deliberate, because changing it would orphan the existing ReplicaSet and its pods, which would keep running with nothing managing them.

Two conventions worth knowing. The app.kubernetes.io/name, app.kubernetes.io/instance and app.kubernetes.io/version labels are the standard set, and tooling expects them. And annotations are the other metadata field: same shape, but not queryable, meant for data tools read rather than data used for selection. Put a git commit in an annotation and an application name in a label.

Separate selection from ownership during a rollout

Suppose a Service selects app: checkout while two Deployment revisions carry that label plus different version labels. Both revisions are candidates for Service traffic. The Deployment's controller relationships are also recorded in metadata.ownerReferences; labels alone do not establish exclusive ownership of arbitrary objects.

Use EndpointSlices to inspect the current backend addresses and conditions:

kubectl -n prod get endpointslices -l kubernetes.io/service-name=checkout -o yaml
kubectl -n prod get pods -l app=checkout --show-labels

Compare each selected pod's readiness, address and Service target port. If the selector matches an unintended debugging pod, fix the label convention or narrow the Service selector through review. Do not delete every matching pod: some may belong to another controller.

A selector switch also takes time to propagate, and established connections can continue using the old backend. For a blue-green release, verify new connection routing and drain the previous version according to the application's connection lifetime. An API patch completing does not mean every connection moved.

What interviewers probe next

"How do you do a blue-green switch with labels?" Run both versions with a version label, and have the Service selector name one. Changing the selector switches the desired backend set. Endpoint propagation and existing connections prevent a literally instantaneous move of all traffic.

"What selector operators exist?" Equality, and set-based (in, notin, exists). Services only support equality; Deployments and NetworkPolicies support both through matchExpressions.

"Why does a pod keep receiving traffic after you delete it?" Removal from endpoints and removal from the node's routing rules happen concurrently, so there is a short window where traffic still arrives. That is why a preStop hook with a short sleep, plus graceful shutdown, is the standard pattern.

Common mistakes

Labelling the Deployment but not the pod template, then wondering why the Service matches nothing.

Debugging a Service without ever checking kubectl get endpoints, which answers the question directly.

Assuming an empty endpoint list means a networking problem, when a failing readiness probe produces the same symptom.

Exporting an unbounded label such as a request ID into metrics. A commit identifier can be useful object metadata, but decide whether it needs label selection and whether monitoring will ingest it.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.