TL;DR: Metrics measure aggregate behaviour, traces describe instrumented request paths, and logs record individual events. Choose the first signal from the symptom and available evidence. Correlating them is more useful than imposing one order on every incident.
How to approach it
Explain what each signal can establish and what it can miss. Use the initial symptom to choose a starting point, then narrow the investigation by service, time window and affected requests.
A strong answer
Metrics describe quantities such as request rate, errors, latency distributions and queue depth. Bounded dimensions make them efficient for population-level alerts and trends; excessive cardinality can make them expensive. They do not usually contain individual request context, although exemplars can link a measurement to an example trace.
Traces connect instrumented spans through propagated context. A trace can show where a particular request spent time across services, subject to missing instrumentation and sampling. Head sampling selects requests early and can miss rare outcomes; tail sampling can select by observed outcomes but requires buffering and operational capacity. Correlated logs and profiling can also explain parts of the request path.
Logs record individual events, including error details and relevant execution context. Their cost depends on volume, indexing, retention and queries. Use structured fields, redact sensitive values, and collect enough context to investigate without recording credentials or unnecessary personal data.
The order in an incident:
For a service-latency incident, begin with the alerting metric to establish impact and affected cohorts. Other incidents may start with a user report, a failed job or a specific log event.
Open a trace for an affected request when available. Identify long spans and unexplained time, then test whether the slow operation is the cause or a downstream consequence. Missing instrumentation limits the conclusion.
Read the logs for that service, in that time window, ideally filtered by the trace ID. Now you are reading twenty relevant lines instead of searching a million.
That last point is the practical glue: inject the trace ID into every log line. Then a trace links directly to its logs and the three stop being three separate tools. OpenTelemetry exists largely to make that correlation standard across languages and vendors rather than something each team wires up differently.
What to instrument on day one: the four golden signals as metrics (traffic, errors, latency, saturation), tracing on every inbound and outbound call, and structured logs as JSON with a trace ID and a level, at info for lifecycle events and error for things a human should see.
Follow one slow checkout without overclaiming
A checkout latency graph shows an 800 ms tail while the request success rate stays steady. Compare affected routes, versions and zones before assuming the whole service is slow. Open a representative sampled trace: suppose 600 ms is spent waiting for a connection and the database span lasts 30 ms. That directs the next check toward pool occupancy, acquisition timeouts and unreleased connections; a database CPU graph alone would miss the wait.
Now use the trace ID to find relevant application logs. A cancellation error after the pool wait may be a consequence of the request deadline, rather than the initiating fault. Check ordering and reproduce the acquisition delay before calling that error the cause. Missing spans may mean incomplete instrumentation or propagation, so account for unexplained time explicitly.
Finally, compare several affected and unaffected requests. Sampling can hide rare failures, and one trace cannot establish population frequency. Use metrics to measure impact, traces to test where time went, and logs to explain a specific execution. If the initial symptom is a systemd execution failure, the unit journal may already be the shortest route to decisive evidence.
What interviewers probe next
"Where do events and profiles fit?" Events are discrete records of something notable, a deploy or a config change, and overlaying them on a metric graph answers "what changed" faster than anything else. Continuous profiling adds a fourth signal for CPU and memory inside a process, which traces cannot see.
"How do you cut the logging bill without losing the ability to debug?" Measure which records support diagnosis or retention obligations. Sample repetitive successful events and bound repeated errors during a storm while retaining counts and representative detail. Verify that audit records and rare failures remain available.
"Can you get metrics from logs?" Yes, by counting matching lines, and it is expensive and slow compared with a counter. Useful for something you failed to instrument, not a design.
Common mistakes
Searching an unrestricted log corpus without a hypothesis, service or time window, even when a narrower signal is already available.
Depending on brittle free-text message patterns when a stable structured event or metric can express the same condition.
Logging without a trace ID or a request ID, so correlating anything means guessing by timestamp.
Treating traces as a debugging luxury, then having no way to localise latency across six services.