DevOpsInterviews logo
← Incident Response & Production Debugging / 02
easyNewRed HatIBMTCS

You are on a Linux box you have never seen and something is failing. Where are the logs?

Three places, and which one depends on how the thing was started. Knowing which logs the runtime collects and which the application writes itself, and that journald may not persist across reboots, saves the first twenty minutes.

Grounded in researched DevOps, SRE and platform engineering interview loops, written to a senior-engineer editorial bar, and never padded to hit a word count.

TL;DR: journalctl -u <unit> for anything systemd started, /var/log/ for services that write their own files, and the container runtime for anything containerised. Check whether journald is persistent before you go looking for yesterday, because the default on some distributions keeps nothing across a reboot.

How to approach it

Answer by how the process was started, because that determines where its output went. Then give the flags that actually get used rather than a list of everything journalctl accepts. Mentioning the persistence question unprompted is the detail that signals you have lost logs to it before.

A strong answer

systemd units. Systemd commonly routes stdout and stderr to the journal, but unit settings can redirect them.

journalctl -u nginx -n 100 --no-pager   # last 100 lines for one unit
journalctl -u nginx -f                  # follow
journalctl -u nginx --since "10 min ago"
journalctl -p err -b                    # errors only, this boot
journalctl -b -1                        # the previous boot
journalctl -k                           # kernel messages

-b for the current boot and -b -1 for the previous one are what you use after an unexplained reboot. -p err cuts a wall of informational noise to the lines someone should read.

The thing to check early: journald's storage. With Storage=auto, the presence of /var/log/journal/ affects persistent storage selection. Check the effective configuration and retained boots rather than inferring all behavior from that directory. That is the default on some minimal images and it is the reason a machine that just rebooted has nothing to say about why.

journalctl --list-boots
# Inspect journald.conf and its drop-ins for the effective Storage policy.

Configure persistent storage and its retention limits through the distribution's supported procedure, then verify retained history.

Files under /var/log. Applications that manage their own logging write here, and the layout depends on the distribution. /var/log/syslog or /var/log/messages for general system messages, /var/log/auth.log or /var/log/secure for authentication, and per-application directories like /var/log/nginx/ and /var/log/postgresql/. When you do not know what a service writes, ask the package:

ls -lt /var/log | head -20        # what changed most recently
lsof -p <pid> | grep -i log       # what this process actually has open

lsof on the process is the reliable answer when documentation and convention disagree, which they often do for anything installed from a vendor tarball.

Containers. Logs are not in /var/log inside the container in any well-behaved setup, because the convention is to write to stdout and let the runtime collect it.

docker logs -f --tail=100 <container>
kubectl logs -n prod deploy/checkout --tail=100
kubectl logs -n prod checkout-7d9f --previous   # the container that just died
kubectl logs -n prod deploy/checkout -c sidecar # a specific container in the pod

--previous is the one people forget, and it can retrieve the retained previous container attempt, because the current attempt may have written nothing yet.

On the host, the runtime writes those streams to files under /var/log/pods/ and /var/log/containers/ on a Kubernetes node, which is useful when the API server is unreachable and kubectl is not an option.

Two more habits. Check dmesg -T early for anything hardware, kernel or OOM related, since a process killed for exceeding its cgroup memory limit leaves nothing in its own logs and a clear line in the kernel ring buffer. And check the clock and timezone before comparing timestamps across machines, because an hour's offset sends people down the wrong path faster than almost anything else.

Confirm the destination configured for the actual process

For a systemd service, inspect StandardOutput, StandardError, the effective unit and application logging configuration. Journal capture is common, but a unit can redirect output elsewhere. A process can also write its own file while its startup messages go to the journal.

Check available boots with journalctl --list-boots and inspect the effective journald storage policy, including drop-ins. The existence of /var/log/journal is a clue, not proof of retained history: explicit volatile storage, retention limits or forwarding can change what remains locally. Configure persistence through the supported policy and verify a controlled reboot before relying on it.

For containers, select the exact pod UID and container name. Runtime-collected stdout, application files and a central logging service can have different retention. If the local previous attempt is gone, use retained central logs and events. Preserve timestamps and clock context while redacting credentials before attaching evidence to an incident.

What interviewers probe next

"The disk filled with logs. What do you do?" Rotate rather than delete, because deleting a file a process still holds open frees nothing. logrotate with copytruncate or a postrotate signal so the daemon reopens its file.

"How do you follow one request across services?" Correlate request or trace IDs across relevant log records; a central store makes that practical across many services, which is the argument for shipping logs somewhere rather than reading them per host.

"What is the difference between journald and rsyslog?" journald is systemd's structured binary store queried with journalctl. rsyslog is the traditional daemon writing plain text under /var/log, and many systems run both, with journald forwarding to rsyslog.

Common mistakes

Looking in /var/log inside a container, where a correctly configured application writes nothing.

Reading kubectl logs without --previous on a crash loop and concluding there is no output.

Assuming journald kept yesterday, when volatile storage discarded it at the reboot you are investigating.

Ignoring dmesg, which is where the OOM killer records the process it terminated.

That one was free, and so are 18 answers per topic without an account. Signing in doubles that to 28, keeps your bookmarks, and tracks which topics you keep getting wrong.one Google click · no card · nothing to cancel
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
NOTES & DISCUSSION · 0

Nothing here yet. Say how you would answer it.