systemd

#What it is

systemd is the first process Linux starts, and it manages everything after that. You describe a service in a small text file called a unit, and systemd takes care of starting it at boot, restarting it if it dies, and collecting its output in the journal.

It also has timers, which do the same job as cron but with better logging and the ability to catch up on runs missed while the machine was off.

#Why I use it

Every long-running piece of these builds is a systemd unit:

• the Hermes Telegram gateway (a user service),
• the OpenCanary honeypot,
• Loki, Alloy and Grafana (their packages ship units),
• the nightly digest script (a oneshot service plus a timer).

#Benefits

• Restart=always means a crash is a blip, not an outage.
• One place for logs: journalctl -u <service> shows why something failed, which is usually the fastest route to a fix.
• Least privilege is built in. User=nobody and StateDirectory= let a script run unprivileged and still get a writable folder.
• Timers beat cron for anything you want to debug: systemctl list-timers tells you when a job last ran and when it runs next.

#Commands worth knowing

sudo systemctl enable --now <svc>    # start now AND at every boot
sudo systemctl status <svc> --no-pager
sudo journalctl -u <svc> -n 40 --no-pager
sudo systemctl daemon-reload         # after editing a unit file
systemctl list-timers

#Gotchas I hit

• PATH inside a unit is not your shell's PATH. OpenCanary's launcher calls python3 and twistd by name, so the unit needs Environment=PATH=/opt/opencanary/bin:... or it silently picks up the system Python and crash-loops.
• User services stop when you log out unless you run sudo loginctl enable-linger "$USER". Without it, the Hermes gateway dies the moment your SSH session ends.
• Group changes need a restart. Adding a service user to a group (usermod -aG) does nothing until that service restarts.