TL;DR: Start by benchmarking two TP=4 replicas against one TP=8 replica. A 70B FP16 model needs about 140GB for weights before KV cache and runtime overhead. TP=2 leaves little headroom on 80GB cards, so four TP=2 replicas are a candidate only after measured memory and latency checks. Choose the layout that meets latency and concurrency targets, and place replicas across failure domains where capacity permits.
How to approach it
Do the memory arithmetic out loud first, because the choice is constrained before it is optimised. Then ask for the traffic shape: a chat product with strict time-to-first-token wants a different answer from a batch summarisation pipeline. Say explicitly which of latency and throughput you are optimising, since the two configurations trade directly against each other.
A strong answer
Start with what has to fit. A 70B model in FP16 is roughly 140GB of weights, so it does not fit on one 80GB card and needs at least two. TP=2 puts about 70GB of weights per GPU and leaves under 10GB for KV cache, activations and fragmentation, which is tight. TP=4 puts about 35GB per GPU, leaving roughly 45GB of nominal capacity before runtime allocations, memory reservation and cache. Use the engine's reported usable cache, not this subtraction, to set concurrency. These are approximate decimal-GB calculations; reconcile them with the device's reported byte capacity. The cache is not a detail: it is what sets how many concurrent sequences you can serve, and a configuration that fits the weights but starves the cache produces a replica that hits its concurrency ceiling almost immediately.
What tensor parallelism costs. Tensor parallel layers exchange partial results through collectives; the exact pattern depends on the model and implementation. NVLink within a node usually makes these exchanges cheaper. Cross-node TP can work with a suitable high-bandwidth interconnect, including InfiniBand or RoCE, but needs measurement. Benchmark TP=4 and TP=8 at representative prompt lengths, batch sizes and concurrency; more GPUs do not guarantee lower latency. See vLLM parallelism guidance.
What replicas cost. Each replica duplicates weights and maintains its own KV cache. Two TP=4 replicas use about 280GB of aggregate weight memory; one TP=8 replica uses about 140GB. Replicas can process requests independently, but routing and cache locality affect throughput. Losing one of two replicas leaves roughly half the serving capacity only if the other remains healthy. Eight GPUs on one host still share a host failure domain.
The decision rule. Use the smallest TP degree that fits weights and leaves the KV cache you need for target concurrency, then spend the rest of the fleet on replicas. Deviate toward higher TP when the product's differentiator is time-to-first-token on long prompts, or when the model simply cannot fit at lower degrees. For this question, start with two TP=4 replicas and compare against TP=8. TP=8 may win on per-request latency or cache capacity because it stores only one copy of the weights. TP=2 needs a separate capacity check; it is not the default recommendation for long contexts or high concurrency.
Quantisation changes the calculation. FP8 weights are roughly 70GB before scales and runtime overhead. That may fit on one 80GB device, but it leaves less than 10GB of nominal space for everything else. Recompute usable cache and validate model quality and hardware support before selecting one GPU per replica. KV-cache size depends on layers, KV heads, head dimension, cache dtype and active tokens; grouped-query attention can reduce it substantially.
What interviewers probe next
"Where does pipeline parallelism fit?" When the model does not fit within one node's tensor-parallel group at all. It partitions layers and can avoid expensive cross-node tensor collectives. Stage balance and pipeline bubbles affect performance, so compare it when topology or memory makes pure TP unattractive.
"How would you validate the choice?" Load-test both at the real prompt and output length distribution, and compare TTFT, inter-token latency and tokens per second per GPU. Aggregate throughput per dollar is the number that decides it, and single-request latency is the constraint it operates under.
"What if traffic doubles?" Add replicas. Changing TP degree is a redeploy with different memory characteristics and a fresh cache; adding replicas is routine.
Common mistakes
Maximising tensor parallelism because more GPUs sounds faster. It can raise collective overhead and concentrate all serving capacity in one replica.
Sizing on weights alone and discovering the concurrency ceiling in production, when the KV cache turned out to be 8GB per card.
Spanning a slow network with a tensor-parallel group without measuring collective time and per-token latency.