Set Up a systemd Alert on Failure for Every Service

Set it up yourself with OnFailure=

systemd can start another unit whenever a unit enters the failed state, using OnFailure=. The trick is a top-level drop-in, which adds that setting to every service at once. This takes about ten minutes.

1. Write the alert script

Save this as /usr/local/bin/notify-failure. Put your own address in the last line.

#!/bin/bash
# Usage: notify-failure <unit>
unit="$1"
host="$(hostname)"
{
  echo "$unit failed on $host"
  echo
  systemctl status --full --no-pager "$unit"
  echo
  journalctl -u "$unit" -n 40 --no-pager
} | mail -s "[$host] $unit failed" [email protected]

Make it executable with sudo chmod 755 /usr/local/bin/notify-failure.

This needs the mail command and a server that can send email. Prefer Slack? Swap the mail line for a webhook call:

curl -fsS -m 10 -H 'Content-Type: application/json' \
  -d "{\"text\": \"$unit failed on $host\"}" \
  https://hooks.slack.com/services/YOUR/WEBHOOK/URL

2. Add a template service that runs it

Create /etc/systemd/system/[email protected]:

[Unit]
Description=Send a failure alert for %i

[Service]
Type=oneshot
ExecStart=/usr/local/bin/notify-failure %i

3. Hook it into every service

Create /etc/systemd/system/service.d/10-notify-failure.conf:

[Unit]
OnFailure=notify-failure@%N.service

%N is the failed unit's name, so a failed backup.service starts [email protected]. Top-level drop-ins like service.d/ arrived in systemd 244. Check yours with systemctl --version.

4. Stop the alert service from alerting on itself

The drop-in also lands on notify-failure@ itself. Mask it there, as the systemd.unit manual shows, then reload:

sudo mkdir -p /etc/systemd/system/[email protected]
sudo ln -s /dev/null /etc/systemd/system/[email protected]/10-notify-failure.conf
sudo systemctl daemon-reload

5. Test it with a service that always fails

printf '[Service]\nExecStart=/bin/false\n' | sudo tee /etc/systemd/system/alert-test.service
sudo systemctl daemon-reload
sudo systemctl start alert-test.service
systemctl status [email protected]

You should get an alert within a few seconds. If not, the status output shows what the script hit. Clean up with sudo rm /etc/systemd/system/alert-test.service, then sudo systemctl daemon-reload and sudo systemctl reset-failed alert-test.service.

Where the do-it-yourself setup falls short

• Email often can't leave the server. AWS limits port 25 and Google Cloud blocks it, so you also need a mail relay.
• A dead server sends nothing. If the machine hangs or loses its network, the script never runs.
• Timers that never fire don't fail. A disabled or broken timer just stops, so OnFailure= never triggers.
• Restart loops are noisy or silent. With Restart=, you may get an alert on every attempt or none, depending on RestartMode= and your systemd version.
• Older servers need per-unit edits. Below systemd 244, you add OnFailure= to each unit by hand.
• The alert says what, not why. You still read raw logs at 2 a.m. to find the cause.

How FailNote closes those gaps

FailNote is a small alert agent for Linux servers. You run one install command, and every service and timer on the machine is covered. It sends alerts through our servers, so blocked mail ports don't matter.

curl -fsSL https://app.itadminutils.com/install.sh | sudo sh -s -- YOUR_KEY

Your dashboard shows the command with your key filled in. The agent is one shell script you can read before you run it.

• Every failure explained. Alerts use the same checks as our systemd failure analyzer, so you see the likely cause and the next command to run.
• Silence counts too. If a server or a timer stops checking in, you hear about that as well.
• One alert per incident. A crash loop sends one message and a recovery note, not fifty pings.
• Your channels. Email, Slack, Discord, Microsoft Teams or any webhook.
• Old and new systemd. It works the same on servers that predate top-level drop-ins.

Here's what a failure alert looks like:

FAILED: backup.service on web-01
Result: exit-code (exited, status=203/EXEC)
Cause:  systemd could not execute the configured command (Permission denied)
Check:  file /opt/backup/run.sh && namei -l /opt/backup/run.sh
Log:    backup.service: Failed at step EXEC spawning /opt/backup/run.sh: Permission denied

That one comes up a lot. Our guide to status 203/EXEC errors walks through every cause.

What the agent sends

Each minute it sends the hostname, OS name, systemd version and the names of any failed units. When a unit fails, it also sends that unit's systemctl status and its last 40 log lines. Set FAILNOTE_LOG_LINES=0 to keep logs on the server.

Pricing

Free for one server, with every feature. Pro covers up to 10 servers for $9 a month, and you can cancel anytime.