DevOpsInterviews logo
← Containers & Kubernetes / 18
hardCloudflareGoogleRed Hat

How does a pod get an IP address, and what happens to traffic between two pods on different nodes?

Trace sandbox setup, address allocation and a cross-node packet, then distinguish pod reachability from Service translation.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: The kubelet asks its CRI runtime to create a pod sandbox. The runtime normally creates the network namespace and invokes CNI to configure it; the plugin’s IPAM may use node CIDRs, a cluster pool or cloud addresses. The Kubernetes network model then requires that every pod can reach every other pod without NAT, which the plugin delivers either by routing the pod CIDRs natively or by encapsulating in an overlay.

How to approach it

Give the three rules of the network model first, because every CNI decision is downstream of them. Then walk one packet across nodes, which is what the question is really asking.

A strong answer

The model imposes three constraints: ordinary pod-network workloads receive pod addresses and can communicate across nodes without NAT when policy permits, and the IP a pod sees itself as is the IP others see it as. That last one rules out the port-mapping approach Docker used by default, and it is why Kubernetes needs a plugin at all.

When the kubelet is told to run a pod, it asks the CRI runtime to create the sandbox. The runtime normally creates its network namespace and calls the CNI plugin, which allocates an address through its configured IPAM and, in a common Linux design, creates a veth pair with one end inside the namespace as eth0 and the other on the host, and installs routes. The pod is now addressable.

Crossing nodes is where implementations diverge. In native routing mode, each node's pod CIDR is a route the underlying network knows about, either because the plugin programs the cloud provider's route table or because it speaks BGP to the fabric. A packet leaves pod A, hits the node's routing table, goes out the physical interface unencapsulated, and the fabric delivers it to node B, which routes it into pod B. Fast, fully visible to network tooling, and it requires the underlying network to cooperate.

In overlay mode, the packet is encapsulated in VXLAN or Geneve and tunnelled node-to-node. The underlying network only sees node-to-node traffic and never learns about pod IPs. Works anywhere, costs you an encapsulation header (so a lower effective MTU, which is a classic source of mysterious failures for large packets) and some CPU.

How a pod gets an IP, and how a packet reaches another node SANDBOX SETUP, ON ONE NODE CROSSING NODES 1 Kubelet asks for a sandbox RunPodSandbox over CRI 2 Runtime makes a netns one per pod, not per container 3 CNI plugin invoked handed the namespace path 4 IPAM allocates node CIDR, pool or cloud address 5 veth pair created eth0 inside, peer on the host 6 Routes installed the pod is now addressable 7 Packet leaves pod A into the node routing table 8 Native route or overlay the one real fork 9 Node B routes it in to pod B's namespace NetworkPolicy is enforced by the plugin, not by Kubernetes. A cluster whose CNI does not implement it silently ignores every policy you write. The model, and everything above is downstream of it: every pod gets its own IP, pods reach each other without NAT, and the IP a pod sees itself as is the IP others see. The last rule is why port mapping was never an option here. Native routing: the fabric knows each node's pod CIDR and the packet goes out unencapsulated, fast and visible to network tooling. Overlay: VXLAN or Geneve, so the fabric sees only nodes. Overlay costs MTU: small requests work, large ones hang Services are a separate layer: kube-proxy DNATs the virtual Service IP to a ready endpoint.

Services are a separate layer on top. A Service IP is virtual; implementation details vary, and IPVS mode binds Service IPs to a dummy interface. kube-proxy programs the node so that traffic to that IP is DNAT'd to one of the ready endpoints: historically iptables rules, now increasingly nftables, and in eBPF-based plugins the translation happens in the kernel datapath without iptables at all. That eBPF path is the significant change of recent years, because the iptables approach degrades with rule count and the eBPF one uses map lookups instead.

Two things worth naming unprompted. Network policy is enforced by the plugin, not by Kubernetes, so a cluster whose CNI does not implement NetworkPolicy silently ignores every policy you write. And MTU mismatches from overlay encapsulation produce the signature failure where small requests succeed and large ones hang.

Packet exercise: direct pod access works but the Service fails

In a fictional cluster, a client reaches a destination pod IP across nodes but cannot use its Service address. That observation narrows the problem toward Service selection or translation, rather than proving the entire CNI is broken. Inspect the Service selector and EndpointSlices, destination ports and the node's configured service datapath. Readiness filtering or a wrong target port can produce the same user symptom as missing translation.

Now small cross-node requests succeed while larger transfers stall. Compare the path MTU and overlay overhead, and inspect whether fragmentation-needed messages can return. The pattern suggests an MTU problem but does not prove it; application timeouts and downstream limits also deserve comparison.

Document the actual plugin mode. A host-network pod, cloud-addressed pod and overlay-connected Linux pod do not follow identical namespace and routing setup. Trace the packet through the observed interfaces and policies instead of forcing every implementation into a veth-and-VXLAN drawing.

What interviewers probe next

"Where does the pod IP come from on a cloud provider?" Often from the VPC itself, with the plugin attaching secondary IPs to the node's interface, which makes pods first-class VPC citizens and caps pod density by instance type.

"What happens to the IP when a pod restarts?" A container restart within an existing pod sandbox normally retains the pod IP. Recreating the pod or its sandbox can change it; do not rely on the address surviving replacement, which is why Services and DNS exist.

"Why is eBPF a big deal here?" It removes the iptables rule-chain traversal from the packet path and uses efficient kernel maps for supported datapath operations. Cilium layer-7 parsing and policy use an Envoy proxy; eBPF alone does not provide every L7 feature. Cilium Envoy integration.

Common mistakes

Saying "the CNI assigns it" with no account of what the plugin actually does.

Confusing pod networking with Service networking. One is real addressing, the other is virtual IPs and DNAT.

Forgetting that NetworkPolicy needs plugin support, which is how teams end up believing they are segmented when they are not.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.