Linux Fundamentals, September 2024

How systemd Starts and Supervises Everything

The older approach ran numbered scripts in order, and the ordering was the only expression of dependency. It was easy to read and slow, because nothing could start until everything before it had finished.

The replacement declares what each service needs and lets the manager work out the order, starting anything with satisfied dependencies immediately. That change explains both the speed and most of the confusion.

Units are the single abstraction

TypeDescribes
serviceA process to start and supervise
socketA listening socket that can start a service on demand
timerA schedule that activates another unit
mountA filesystem to be mounted
targetA grouping used as a synchronisation point
pathA file or directory to watch for changes

Everything is a unit, which is why one command inspects services, mounts, timers, and boot targets. A target holds no processes of its own and exists to give other units something to depend on.

Wants and requires are not the same

A weak dependency starts the other unit and does not fail if it cannot start. A strong dependency causes this unit to fail if the other one does.

The distinction is the single most useful thing to know when a service refuses to start, because a strong dependency on something unavailable produces a failure that appears to be about the wrong service entirely.

Ordering is expressed separately again. Declaring that one unit must start after another says nothing about whether it needs it, and units frequently declare both.

Starting on demand

A socket unit lets the manager hold a listening port and start the service only when a connection arrives, handing over the already open socket.

Two useful properties follow. Boot completes without waiting for services nobody has used yet, and connections arriving during a restart are held rather than refused, because the socket stayed open throughout.

Supervision is the other half

Starting a process is straightforward. Knowing whether it is still working is not, and this is where the manager does most of its work.

It tracks every process a service creates using kernel groups rather than process identifiers, so a service that forks repeatedly is still accounted for correctly. Restart policies are declarative, with limits so a service failing immediately does not restart in a tight loop forever.

Resource limits attach at the same level, which means memory and processor limits apply to the whole service rather than to one process within it.

Logs became structured

Output written to the standard channels is captured automatically, so a service needs no logging configuration to have its output recorded.

Entries carry metadata including the originating unit, the process, and the priority, which makes filtering precise rather than a text search. The trade is that the store is a binary format requiring its own tool, and that the default configuration on many systems does not persist it across reboots.

Diagnosing a failure

The status of a unit reports whether it is running, what it last exited with, and the recent log lines, which resolves most cases immediately.

When a service does not start at all, the question is usually dependency rather than the service. Listing what a unit requires, and what failed before it, converts a confusing failure into an obvious one, and blaming the visible unit is the common wrong turn.

Note: a unit that reports as active while the service is not working usually declares itself started when the process launches rather than when it is ready. Declaring readiness explicitly is what makes dependent services wait for the right moment.