DevOps Scenario-Based & Troubleshooting Interview Questions

Status codes describe an HTTP failure category. Locate the component that emitted the response, then use its logs to distinguish connection, capacity and application failures.

Three places, and which one depends on how the thing was started. Knowing which logs the runtime collects and which the application writes itself, and that journald may not persist across reboots, saves the first twenty minutes.

A common scenario question, and the one where a list of commands most often stands in for an argument about what each one eliminates. Empty logs is the clue, not the obstacle.

The gentlest scheduling question there is, and it still separates people who read the events from people who guess. Check whether the pod has a node before choosing between scheduling and startup diagnosis.

The pods are Running, the dashboard is green, and users get errors. The gap between Running and receiving traffic is where this question lives, and it has a precise answer.

A senior favourite, because the naive answer is to look at averages and the average is exactly what is hiding the problem. Tail latency has a short list of causes and they are all structural.

Tests whether you know that Kubernetes waits, and for how long. The default timings surprise most candidates, and they explain why a node failure hurts longer than people expect.

Intermittent DNS is the most-reported and least-understood Kubernetes failure. There are three well-known causes and each leaves a different fingerprint.

A deceptively deep question, because there are at least six places a version can get stuck and they span the pipeline, the cluster and the edge. Work the path in order.

Tests the mitigate-before-diagnose reflex under time pressure, plus whether you know the two ways a disk can be full and the trap that makes freed space not come back.

Both can be right, which is the whole point. Retransmissions mean a segment was not acknowledged, and the network is only one of the places that can happen.

The classic gap between what the backend measures and what the user experiences. A 504 is emitted by something in front of the backend, which is exactly the part your service dashboards do not cover.

Automation can renew a certificate without deploying it correctly. Check the served leaf and chain, the client trust store and the exact verification failure.

Low utilisation with terrible tail latency is not a contradiction. It is the CFS quota doing exactly what it was configured to do. Most candidates have never read nr_throttled.

An idle database with a starved application is the tell. The connections are held, not used. More database capacity does not release leaked connections.

Established traffic keeps working while new connections die at random, which is exactly why this one confuses people. The table is full of flows that will not be needed again for days.

A two-second dependency should cost you latency, not three services. The mechanics are arithmetic: Little's law plus retry amplification. The fix is a budget, not a ban.

Repeated action items are a measurable defect with a mechanical cause. Answering 'culture' without a mechanism describes the symptom back rather than fixing it.

Emergency mode after a storage change is usually one line in fstab. The order of operations is what decides the outcome here, along with whether the recovery preserves data-mount dependencies rather than booting past them.

One disk metric explaining a whole cluster's weirdness is the point of this question. Restarting etcd first turns a slow cluster into a down one, because a restarted member has to catch up before it can vote.

One saturated core on a proxy is rarely about raw traffic volume. It is a parallelism failure with three usual suspects, and the per-thread view names the culprit in minutes.

A hang suggests missing progress; a refusal suggests an explicit error. Packet evidence distinguishes firewall behavior from routing failures and closed listeners. Then rule counters, not eyeballs, finger the guilty line.

Everyone reaches for reinstall. The disciplined path is one journal file, four failure signatures, and an explicit decision about whether this host deserves surgery at all.

Same-node good, cross-node bad is not a random clue, it is nearly a diagnosis. One short decision tree separates overlay reachability, encapsulation blocks, MTU and policy.

The reflex is rollback and it is usually right. Shared state, changed credentials and unavailable old artifacts can make an undo incomplete or unsafe. Verify those dependencies before choosing the recovery.

An unchanged pipeline can fail because of environment state, external inputs or a latent artifact defect. Failures that start on an expiry boundary narrow the field to two suspects fast.

This one separates people who read journals from people who restart harder. The exit code and signal systemd already recorded usually name the disease.

Nothing errors because dropping unmatched lines is documented behaviour. Counting the tagged failures converts an invisible leak into a number, and numbers have owners.

NFS waits can stall shutdown and inflate load average while CPUs stay idle. Check the blocked kernel path before deciding whether signals or mount recovery can release it.

Load balancing makes a per-host fault look like a flaky library. The exact rejection string tells you which way the offending clock leans before you log into anything.

The box looks CPU-starved but the CPUs are starving themselves. Reading the reclaim loop correctly separates people who have sat on the console from people who have read about it.

The kill was not random and it was not fair. Learn the scoring function the kernel actually uses, then bend it with oom_score_adj before your monitoring dies during the next leak.

The leaf is valid and the root is trusted, yet verification fails. The gap lives in the middle of the chain, and the exact verify code names it before you open a single config file.

The backup is doing its job and the database is paying for it. Attribution first, then enforcement that actually binds on modern kernels, because priority support depends on the active kernel and IO scheduler.

Self-healing unavailability is a timeout arithmetic story. Pull the balancer's removal timestamps, lay them against the GC log, and the pause that ejects the instance will be sitting there in plain sight.

Spikes that march in step with BGSAVE are a page-size story, not a network story. Check actual huge-page usage, fork duration and copy-on-write pressure on the deployed kernel.

More partitions should mean more throughput, so why did the backlog deepen? Because rebalance behavior depends on the group protocol, and the way consumers divide work decides who pays for it.

A public registry throttles by source IP and your whole fleet shares one. Why fifty nodes count as a single anonymous user, and authentication and caching changes that reduce upstream demand.

The outage window closed, but every process born since then still tries to fetch secrets at boot with nothing to fall back on. Why restarts keep reopening an incident that already ended.

Gradual drift produces no errors on the host that has it, only weirdness everywhere else. Sweep first, histogram second, repair third: the distribution shape names the culprit before you log into anything.

Duplicate-IP claims need evidence, and one command provides it. Then the real skill: telling a genuine two-owner clash apart from ARP flux, because the fixes could not be more different.

Bandwidth graphs look healthy and the partner swears they are fine. The wall is ports, not bits: the available source-port budget depends on the gateway and its address allocation.

Every client fails at once, which rules out pods and points at shared PKI. Know which certificates share one expiry clock, and a controlled recovery that preserves cluster state.

Exit codes said green while the warehouse went hollow. Where detection should have sat, why retries could not help, and the backfill discipline finance will actually accept.

Mixed resolution after a transfer is not propagation voodoo. Compare the parent delegation, authoritative answers and validation results. NXDOMAIN and SERVFAIL are clues; neither names a unique cause.

Purges act on exact cache keys, and the edge is full of variants the purge never named. Verify per variant at real POPs, then fix the key model instead of chasing URLs forever.

Split-brain writes are a fencing failure first and a data problem second. Freeze writes, read the divergence boundary from replication positions, and let the payment ledger arbitrate reality.

Slow leaks hide inside restart cycles and percentage thresholds. Plot weeks of absolute memory against deploys, prove it with profiler deltas, and alert on predicted time-to-limit instead.

Stateful firewalls keep established flows alive after their rule vanishes, so cleanup day passes every check and lag appears days later. Flow logs name the hour; inventories prevent the repeat.

Wall-clock schedules meet daylight-saving transitions: spring forward deletes an hour of cron targets, fall back repeats one. Schedule in UTC, add idempotency keys, rehearse in staging.

Fork failures can reflect PID limits or memory allocation failure even when host RAM looks available. Find which ceiling broke, cap PIDs per cgroup so one team's leak costs one pod, and fix the unbounded spawn.

Stop the source before tuning anything: kill the runner, revoke its credentials, shed test-shaped traffic at the edge, protect checkout. Forensics and guardrails only after customers stop feeling it.

Queue time, prefill and decode fail for different reasons and the aggregate hides which. Split the metric first and the candidate list drops from twenty to three.

A reported successful rebuild still leaves several places to investigate: a cache, a routing pointer, a chunk that survived deletion, or a document that never reached the pipeline at all.

The automation stopped because it could not decide, which is the correct behaviour and an incomplete design. Somebody has to decide, and the interesting question is what evidence that decision needs.

The pods are Pending, the nodes are Ready, and the GPUs have vanished from allocatable. Something changed on the node under a fleet that nobody considered part of the deploy surface.

A loop with no step cap has no natural ceiling, and runaway spending keeps accruing while you investigate. Stop the spend first, then find out what it was doing.

A drain that hangs without erroring is almost always eviction being refused, and the two refusals that look identical from the outside have opposite fixes.

Raising the limit buys hours and proves nothing. The six-hour periodicity is the diagnosis: something is opening descriptors on a code path that never closes them.

Two commands disagreeing is the diagnosis, not the mystery. A deleted file with an open handle is invisible to du and fully charged by df, and rm made it worse rather than better.

The disk is not full, the keyspace quota is. Defragmenting without disarming the alarm leaves the cluster read-only, and disarming without compacting puts it back within minutes.

The volume is in one availability zone and the scheduler cannot move it. Spare capacity in the other two zones is not spare capacity for this pod, and no amount of scaling helps.

A health endpoint that returns 200 because the process is running tells you the process is running. Interviewers want to hear where you draw the line between a check that pages and a check that removes traffic.

The pod's resolver is configured to try five search domains before the real one. An uncached external hostname can trigger several search-expanded queries before its absolute lookup, and the failures are what you are paying for.

Two controllers with different opinions and no shared hysteresis. Flat traffic with an oscillating node count means the loop is driven by something other than demand, and the candidates are few.

The heap is the part you configured and the smaller part of what the process uses. The kernel counts everything the cgroup touches, including the page cache the JVM never asked for.

Code rolled back and the database did not. The old version is now running against a schema it has never seen, and that is a worse incident than the one you were trying to end.

Running pods already have their images, so a credential expiry is invisible until something needs to start. That gap is why this always surfaces during an unrelated incident, at the worst moment.

One partition has one consumer, by design. If the key distribution sends a third of the traffic to one partition, the parallelism you bought is unavailable to the work that needs it.

Every SDK fetches its credentials from a link-local metadata endpoint with a modest request budget. Enough containers refreshing at once and the endpoint starts refusing, which the SDK reports as an auth problem.

Somebody silenced it during an incident nine days ago to stop the noise, and nothing ever expired the silence. The monitoring was working perfectly and nobody was listening.