At boot
Systemd Generators: Early-Boot Linux Persistence
How systemd generators run as root at every boot and daemon-reload, how attackers use them to create units that exist only in /run, and how to audit them.
- MITRE ATT&CK
- T1543.002
- Tactics
- Persistence, Privilege escalation, Defense evasion
- Runs
- At boot
- Set up by
- root
- Locations
- /etc/systemd/system-generators/*
- /usr/local/lib/systemd/system-generators/*
- /usr/lib/systemd/system-generators/*
- /run/systemd/system-generators/*
- /etc/systemd/user-generators/*
- /usr/lib/systemd/user-generators/*
How it works
Generators are small executables that systemd runs very early at boot, and again at every systemctl daemon-reload and daemon-reexec, before it loads any unit. Their job is to translate other configuration into units. systemd-fstab-generator turns /etc/fstab into mount units, systemd-getty-generator adds consoles, systemd-sysv-generator wraps init scripts in services, and systemd-rc-local-generator hooks in rc.local.
Each generator receives three output folders as arguments, which for the system manager are:
| Argument | Folder | Priority of what it writes |
|---|---|---|
$1 (normal) | /run/systemd/generator | After /etc and /run/systemd/system, before /usr/local/lib and /usr/lib |
$2 (early) | /run/systemd/generator.early | Before /etc/systemd/system: overrides admin units |
$3 (late) | /run/systemd/generator.late | After everything, including vendor units |
A generator writes unit files, drop-ins and .wants/ links there, and systemd loads them like any other unit. /run is a tmpfs: the output vanishes at shutdown and is rebuilt at every boot.
System generators run as root. User generators run in each user's manager as that user, but their folders are system-wide only, so planting one takes root. A file in /etc/systemd/system-generators overrides a vendor generator of the same name, and a link to /dev/null disables it. systemd expects generators to be fast, to write only to their output folders and to avoid side effects, yet each one is just an executable run as root on every boot.
A stock system has a dozen or two generators, mostly from systemd itself plus a few from other packages (netplan on Ubuntu, nfs-utils, and some distribution tooling, sometimes as shell scripts). Admins almost never add one, so a generator no package installed is rare and worth reading.
Environment generators (system-environment-generators, user-environment-generators) are a related mechanism: they print VAR=value lines that become the service manager's environment, and so the environment of every service it starts.
Where it lives
| Path | What | Who can write it |
|---|---|---|
/etc/systemd/system-generators/ | Admin generators; override vendor names | root |
/run/systemd/system-generators/ | Runtime generators | root |
/usr/local/lib/systemd/system-generators/ | Locally installed generators | root |
/usr/lib/systemd/system-generators/, /lib/systemd/system-generators/ | Package generators | root (package manager) |
/etc/systemd/user-generators/, /usr/lib/systemd/user-generators/ (also under /run and /usr/local/lib) | Generators run by every user manager | root |
/usr/lib/systemd/system-environment-generators/, /etc/systemd/system-environment-generators/ | Environment generators | root |
/run/systemd/generator, generator.early, generator.late | Generator output, rebuilt at boot | generators |
How attackers use it
A generator that writes a service
The persistent artifact is the generator; the service it creates exists only in /run. Searching /etc/systemd/system and /usr/lib/systemd/system finds nothing, and deleting the generated unit from /run lasts until the next boot or daemon-reload.
#!/bin/sh
# /etc/systemd/system-generators/systemd-netcfg-generator
cat > "$1/netcfg-sync.service" <<'EOF'
[Unit]
Description=Network configuration sync
[Service]
ExecStart=/usr/lib/.netcfg/syncd
Restart=always
EOF
mkdir -p "$1/multi-user.target.wants"
ln -sf "$1/netcfg-sync.service" "$1/multi-user.target.wants/netcfg-sync.service"
systemctl status netcfg-sync shows the unit loaded from /run/systemd/generator/, which an analyst may take for normal system plumbing.
A drop-in written to the early folder
Output in generator.early outranks /etc, so a generator can add a command to a legitimate service for this boot without touching any file under /etc or /usr/lib. This is the generator form of a systemd service drop-in:
mkdir -p "$2/cron.service.d"
printf '[Service]\nExecStartPost=/usr/lib/.netcfg/syncd\n' > "$2/cron.service.d/50-sync.conf"
Replacing or editing a vendor generator
A script with a vendor generator's name in /etc/systemd/system-generators replaces it. An edited shell-script generator in /usr/lib keeps its name and its package; only a checksum comparison shows the change.
How to check a host
On a live host
Do not run systemctl daemon-reload before you have read every generator: it executes them.
ls -la --time-style=full-iso /etc/systemd/system-generators /run/systemd/system-generators \
/usr/local/lib/systemd/system-generators /usr/lib/systemd/system-generators \
/etc/systemd/user-generators /usr/lib/systemd/user-generators 2>/dev/null
file /usr/lib/systemd/system-generators/* # ELF binary or script
dpkg -S /usr/lib/systemd/system-generators/* 2>&1 | grep 'no path found' # rpm -qf on RHEL
sudo dpkg -V systemd # then each package that owns a generator
ls -laR /run/systemd/generator /run/systemd/generator.early /run/systemd/generator.late
systemctl show -p FragmentPath,SourcePath netcfg-sync.service
systemd-analyze unit-paths
Every file in /etc/systemd/system-generators, /usr/local/lib/systemd/system-generators and /run/systemd/system-generators needs an explanation, since no package installs there. In the output folders, generated units normally carry a SourcePath= naming the configuration they came from (/etc/fstab, /etc/init.d/<name>); a service with no plausible source is suspect. Check environment generators the same way.
Offline or from a collection
Collections rarely include /run, so the generated units are usually not there; capture /run/systemd/generator* before a suspect host is rebooted. Read each generator file, run file and strings on binaries, and compare against /var/lib/dpkg/info/*.list and *.md5sums (rpm --root /mnt/image -Va on RPM systems). The journal shows the units the generator produced starting at each boot: journalctl -D /mnt/image/var/log/journal -u <unit>.
Linux Persistence checks generator folders offline from the collection command.
What Linux Persistence flags
Every file in a system-generators or user-generators folder under /etc, /run, /usr/lib, /lib and /usr/local/lib is one entry that fires at boot. The shared command heuristics run on the whole file read as text:
- Downloads and runs code, Reverse or bind shell and Decodes and runs a payload when a script generator fetches, connects or decodes.
- Runs from a hidden folder when it references a path such as
/usr/lib/.netcfg/syncd, and Runs from a temporary folder for/tmp,/var/tmpor/dev/shm. - Detaches from the session, Interpreter with inline code, Sets LD_PRELOAD and Embedded encoded blob where they appear.
Compiled generators are binaries, and the text heuristics rarely match them. For those the package layer is the check: Not installed by any package (medium) for a file in /usr/lib or /lib that dpkg does not own, and Differs from package package (high) for a vendor generator whose checksum changed. An unowned generator in /etc/systemd/system-generators or /etc/systemd/user-generators is flagged too (medium); generators in /usr/local/lib and /run get no package flag, so read every file listed there. Changed date marks generators modified in the 7 days before the newest file in the collection. program also runs from mechanisms marks a generator path that another mechanism also runs.
Limits: the tool never runs generators and does not read their output in /run/systemd/generator*. It does not read environment generators or check file modes. The collection command includes /usr/local/lib/systemd; /run only exists on a live host.
Cleaning up
- Preserve first: copy the generator with
cp -aand hash it, and copy/run/systemd/generator*now, because a reboot erases them. Savesystemctl showfor each unit it produced. - Stop what it produced, for example
systemctl stop netcfg-sync.service, and kill leftover processes. - Delete the generator, or for an edited vendor generator reinstall its package (
apt-get install --reinstall <pkg>,dnf reinstall <pkg>). Then runsystemctl daemon-reload, which rebuilds the output folders without it, and confirm the generated unit is gone. - Search for the payload in other units, cron and init scripts, and check the initramfs, which carries its own copy of systemd and its generators on many distributions:
lsinitramfs(Debian, Ubuntu) orlsinitrd(RHEL, Fedora) lists it. - A generator is root code at every boot: treat the host as root-compromised and rotate its credentials and keys.
References
See also
Related techniques
On Linux Forensics
The artifacts behind this technique, in the Linux Forensics reference: