systemd service lifecycle: dependencies, readiness and restart behavior
Trace systemd service states, readiness and restart limits. Use systemctl and journalctl to distinguish an enabled unit, a running process and a healthy application.
TL;DR: A running process may still be initializing. Choose a service type that reports the startup boundary you need, express dependency ordering separately from activation, and give repeated failures a bounded restart policy.
Read service state before restarting
A service can pass through activating, active and deactivating to inactive, or enter failed after an unsuccessful operation. SubState adds detail such as running, exited or auto-restart. These are unit-manager states; an active unit does not prove that application requests succeed.
Enabled and active answer different questions. Enabling a unit configures installation links, often so a target pulls it in at boot. It does not start the service unless you also request that action, for example with enable --now. A disabled unit can already be running because it was started manually or pulled in another way.
Use this read-only sequence on a host with the example service, substituting your real unit name:
systemctl status report-worker.service --no-pager
systemctl cat report-worker.service
systemctl show report-worker.service -p ActiveState -p SubState -p Result -p ExecMainStatus -p MainPID -p NRestarts
journalctl -u report-worker.service --since '-10 min' --no-pager
If Result=start-limit-hit, inspect earlier journal entries for the failure that caused the repeated starts. Clearing the failed state before recording that evidence makes the investigation harder. ExecMainStatus also needs its context: an exit status and a terminating signal are different outcomes. Use the status output and journal to interpret it.
What “started” tells another service
systemd manages units, with service units describing supervised processes. The unit's startup result is one input to dependency scheduling. It is not a continuous application health check. A daemon can remain alive while its database connections fail or its request queue stops draining.
With Type=simple, systemd considers startup successful very early, before the new process has successfully executed the configured program. Type=exec waits for execution to succeed, which catches failures such as a missing executable. Neither waits for an HTTP listener to become useful. A notification-aware daemon can use Type=notify and send READY=1 after initialization. Its implementation must actually send that notification. These distinctions are documented in the systemd service manual.
For a worker that needs 20 seconds to load its index, “the executable launched” and “the index accepts queries” are different promises. Select the promise deliberately. Do not set Type=notify on an arbitrary executable and expect systemd to infer readiness from its logs.
Ordering and dependency strength answer separate questions
After=database.service orders the units when both participate in a start transaction. It does not, by itself, pull the database into that transaction. Wants= requests another unit with a weaker relationship; Requires= creates a stronger requirement. Combine the relationship with ordering when startup failure must prevent the dependent unit from starting. The exact propagation behavior matters, so consult the unit dependency reference.
Even a correct startup relationship cannot guarantee the dependency stays healthy forever. Applications need bounded retries and reconnection behavior after startup. network-online.target is similarly a boot-time synchronization point whose meaning depends on the network manager; it cannot promise that a remote database remains reachable.
Review a worker's failure policy
This illustrative fragment assumes /opt/reports/worker supports systemd notifications, runs in the foreground and owns its graceful shutdown behavior. It is not an installation command.
[Unit]
Description=Report queue worker
Wants=network-online.target
After=network-online.target
StartLimitIntervalSec=120
StartLimitBurst=4
[Service]
Type=notify
User=report-worker
ExecStart=/opt/reports/worker
Restart=on-failure
RestartSec=5
TimeoutStartSec=45
TimeoutStopSec=30
Suppose a malformed configuration makes the process exit after one second. A five-second delay avoids a tight restart loop, but without a start limit the service can still fail indefinitely. The configured burst caps repeated starts within the interval. Hitting the limit is a diagnostic event; increasing it does not repair the configuration.
A controlled stop is different from an unexpected failure. The worker should stop accepting work, finish or return in-flight jobs, and exit within the stop timeout. If it is killed after that timeout, delivery semantics determine whether the unfinished job is retried. Link the unit's behavior to the queue's delivery guarantees, especially for jobs that charge customers or send notifications.
| Observation | Useful evidence | Likely next investigation |
|---|---|---|
| Start immediately fails | Exec result and journal | Path, permissions, user or configuration |
| Unit active, endpoint unavailable | Listener and readiness timeline | Initialization versus reported startup |
| Repeated short-lived processes | Restart count and exit reason | Crash loop or watchdog failure |
| Stop exceeds its timeout | In-flight work and signal handling | Shutdown deadline and job recovery |
Inspect the effective unit
Drop-ins can override the file someone pasted into a ticket. Compare systemctl cat with the properties reported by systemctl show, and check the running host’s systemd version before relying on a newer directive. systemd-analyze verify can catch unit-file errors, but it does not prove that a daemon sends readiness notifications or shuts down cleanly.
Self-check: the dependent API starts before the worker's index loads, although After=report-worker.service is present. Adding a sleep hides the race only for one observed startup time. First determine the worker's service type and readiness contract. A real readiness notification, or an application-level retry with a deadline, handles variable initialization time without assuming it always finishes in a fixed number of seconds.