At boot
SysV Init Scripts: Linux Persistence via /etc/init.d
How attackers persist with /etc/init.d scripts and rc?.d start links on SysV and systemd hosts, and how to find, verify and remove a malicious init script.
- MITRE ATT&CK
- T1037.004
- Tactics
- Persistence, Privilege escalation
- Runs
- At boot
- Set up by
- root
- Locations
- /etc/init.d/*
- /etc/rc[0-6S].d/*
- /etc/rc.d/init.d/*
- /etc/rc.d/rc[0-6].d/*
- /etc/inittab
How it works
Under SysV init, /sbin/init reads /etc/inittab, enters the default runlevel (2 on Debian, 3 or 5 on RHEL), and the rc script runs each link in /etc/rcN.d/: first every K* link with stop, then every S* link with start, in lexical order, so the two digits after the letter set the order. The links point to shell scripts in /etc/init.d (/etc/rc.d/init.d on RHEL, where /etc/init.d is itself a link). Scripts run as root with init's minimal environment and / as working directory, and handle start, stop, restart and status arguments.
Links are managed by update-rc.d on Debian (from the LSB header's Default-Start and Default-Stop) and by chkconfig on RHEL (from a # chkconfig: 2345 90 10 line or an LSB header). Many scripts also source a settings file, /etc/default/<name> on Debian or /etc/sysconfig/<name> on RHEL, as shell code.
Most current distributions boot with systemd, which still honours init scripts through systemd-sysv-generator. At every boot and daemon-reload, the generator wraps each executable script in /etc/init.d that has no native unit of the same name in a <name>.service under /run/systemd/generator.late/. If S links exist in rc2.d to rc5.d, it makes the service wanted by multi-user.target or graphical.target. Two consequences:
- A script that shares its name with a native unit (
ssh,cronand others on Debian and Ubuntu) is ignored at boot. On Debian, even/etc/init.d/<name> startis redirected tosystemctl. Edits to such a script rarely run. - A new script with start links does run at boot, as root, with no unit file on disk.
Upstream systemd deprecated this compatibility layer in version 256 and is removing it. On the newest releases, check that systemd-sysv-generator still exists before assuming an init script runs. Alpine and Gentoo use OpenRC instead: scripts also live in /etc/init.d, but runlevel links live in /etc/runlevels/<level>/.
Where it lives
| Path | What | Who can write it |
|---|---|---|
/etc/init.d/<name> | Scripts on Debian and Ubuntu (a link to rc.d/init.d on RHEL) | root |
/etc/rc.d/init.d/<name> | Scripts on RHEL, CentOS, older Fedora | root |
/etc/rc0.d/ … /etc/rc6.d/, /etc/rcS.d/ | S/K links per runlevel on Debian | root |
/etc/rc.d/rc0.d/ … rc6.d/ | Same on RHEL (/etc/rcN.d are links) | root |
/etc/default/<name>, /etc/sysconfig/<name> | Settings sourced by scripts | root |
/etc/inittab | sysvinit runlevel and respawn table | root |
/etc/runlevels/<level>/ | OpenRC enablement links | root |
/run/systemd/generator.late/<name>.service | Units systemd builds from scripts | systemd-sysv-generator |
How attackers use it
A new script with start links
The classic form needs root and nothing else. The LSB header makes update-rc.d or chkconfig --add accept it, and the script can be linked by hand just as well (ln -s ../init.d/netconsole-agent /etc/rc2.d/S99netconsole-agent).
#!/bin/sh
### BEGIN INIT INFO
# Provides: netconsole-agent
# Required-Start: $network $remote_fs
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop:
# Short-Description: Network console agent
### END INIT INFO
case "$1" in
start) nohup /usr/lib/.ncon/agent >/dev/null 2>&1 & ;;
esac
exit 0
XorDDoS is publicly documented installing randomly named scripts in /etc/init.d with start links, alongside cron entries that restore them.
Editing a script or its settings file
A line added to a packaged script, or to the settings file it sources, runs whenever the script does. The settings file is the quieter choice, since few people read it:
# appended to /etc/default/<name>, sourced by /etc/init.d/<name>
/usr/lib/.ncon/agent >/dev/null 2>&1 &
On systemd hosts this only works for scripts without a native unit; on hosts still booting SysV init, it always works.
A respawn line in inittab
On real sysvinit hosts (Devuan, old CentOS 6 servers, appliances), init itself restarts a process listed with respawn whenever it exits:
# /etc/inittab
ka:2345:respawn:/usr/lib/.ncon/agent
For the single-file variant of boot persistence, see rc.local; for the modern equivalent, systemd services.
How to check a host
On a live host
Do not run service --status-all or /etc/init.d/<name> status on a suspect host: both execute the scripts.
ls -la --time-style=full-iso /etc/init.d/
ls -l /etc/rc?.d/ | grep ' S[0-9]' # Debian start links (/etc/rc.d/rc?.d on RHEL)
chkconfig --list 2>/dev/null # RHEL: SysV services and their runlevels
ls -la /run/systemd/generator.late/ # services systemd built from scripts
systemctl list-units --all | grep -E 'LSB:|SYSV:' # their descriptions start with LSB: or SYSV:
systemctl show -p SourcePath,ExecStart netconsole-agent.service
dpkg -S /etc/init.d/* 2>&1 | grep 'no path found' # RHEL: rpm -qf /etc/rc.d/init.d/*
sudo dpkg -V | grep -E '/etc/(init\.d|default)/'
grep -v '^#' /etc/inittab 2>/dev/null
For every script no package owns, read the whole body, not just the header: the Short-Description is whatever the author typed. Check whether a native unit with the same name exists (systemctl cat <name>.service) to know whether the script runs at boot at all.
Offline or from a collection
List the scripts and the links with their targets (find /mnt/image/etc -path '*rc*.d/*' -type l -printf '%p -> %l\n'). A link's mtime is roughly when the script was enabled. Debian tracks packaged init scripts and /etc/default files as conffiles: compare their checksums with Conffiles: in /var/lib/dpkg/status, and ownership with /var/lib/dpkg/info/*.list. On RPM systems, use rpm --root /mnt/image -Va. Read /etc/inittab if present. In the journal, the generated service appears under the script's name: journalctl -D /mnt/image/var/log/journal -u <name>.service.
Linux Persistence does this offline from the collection command.
What Linux Persistence flags
Each script in /etc/init.d or /etc/rc.d/init.d is one entry, except README, skeleton and insserv's .depend.* files. Its description comes from the Short-Description: or chkconfig description: line. It is shown as enabled, with its runlevels, when an S link in /etc/rc0.d to /etc/rc6.d (or /etc/rc.d/rcN.d) points to it. It is disabled when the collection has rc links but none start it, and "enablement unknown" when the collection has no rc links at all.
The shared command heuristics run on the script body, comment lines skipped:
- Downloads and runs code, Reverse or bind shell and Decodes and runs a payload for fetch-and-run, socket and decode-and-run patterns.
- Runs from a hidden folder and Runs from a temporary folder for paths like
/usr/lib/.ncon/agentor/dev/shm/x. - Detaches from the session for
nohup,setsidorscreen -dm, common in hand-written start sections. - Interpreter with inline code, Sets LD_PRELOAD, Turns shell history off, Makes a file immutable and Embedded encoded blob where they appear.
The package layer uses dpkg: Not installed by any package (medium) for a script in /etc/init.d that no package owns, and Differs from package package (low) for a packaged script that was edited. Heuristic hits in an unmodified packaged script are shown as info. Changed date marks scripts modified in the 7 days before the newest file in the collection. program also runs from mechanisms fires when another mechanism, such as a cron job or rc.local, calls the same /etc/init.d/<name> path.
Limits: /etc/inittab, /etc/default, /etc/sysconfig and OpenRC's /etc/runlevels are not read. On Alpine and Gentoo, enablement is therefore unknown. The tool does not check the executable bit, whether a native unit shadows the script, or whether the host still has systemd-sysv-generator. Without a dpkg database (RHEL collections), there is no package verdict.
Cleaning up
- Preserve first:
cp -athe script, its rc links, its settings file and the program it starts, and hash them. - Stop it without running it.
/etc/init.d/<name> stopandsystemctl stopboth execute the script's code;systemctl kill -s KILL <name>.servicesignals its processes directly, or kill the PIDs you found withps. - Remove the links (
update-rc.d -f <name> removeon Debian,chkconfig --del <name>on RHEL), delete the script, and runsystemctl daemon-reloadon systemd hosts. For an edited packaged script, restore the original: on Debian, delete it and runapt-get install --reinstall -o Dpkg::Options::=--force-confmiss <package>. - Check the other places the payload may be planted: cron, rc.local, systemd units and generators,
/etc/default,/etc/inittab. - Init scripts run as root: treat the host as root-compromised and rotate its credentials and keys.
References
See also
Related techniques
On Linux Forensics
The artifacts behind this technique, in the Linux Forensics reference: