Skip to content

At boot

SysV Init Scripts: Linux Persistence via /etc/init.d

How attackers persist with /etc/init.d scripts and rc?.d start links on SysV and systemd hosts, and how to find, verify and remove a malicious init script.

MITRE ATT&CK
T1037.004
Tactics
Persistence, Privilege escalation
Runs
At boot
Set up by
root
Locations
  • /etc/init.d/*
  • /etc/rc[0-6S].d/*
  • /etc/rc.d/init.d/*
  • /etc/rc.d/rc[0-6].d/*
  • /etc/inittab

How it works

Under SysV init, /sbin/init reads /etc/inittab, enters the default runlevel (2 on Debian, 3 or 5 on RHEL), and the rc script runs each link in /etc/rcN.d/: first every K* link with stop, then every S* link with start, in lexical order, so the two digits after the letter set the order. The links point to shell scripts in /etc/init.d (/etc/rc.d/init.d on RHEL, where /etc/init.d is itself a link). Scripts run as root with init's minimal environment and / as working directory, and handle start, stop, restart and status arguments.

Links are managed by update-rc.d on Debian (from the LSB header's Default-Start and Default-Stop) and by chkconfig on RHEL (from a # chkconfig: 2345 90 10 line or an LSB header). Many scripts also source a settings file, /etc/default/<name> on Debian or /etc/sysconfig/<name> on RHEL, as shell code.

Most current distributions boot with systemd, which still honours init scripts through systemd-sysv-generator. At every boot and daemon-reload, the generator wraps each executable script in /etc/init.d that has no native unit of the same name in a <name>.service under /run/systemd/generator.late/. If S links exist in rc2.d to rc5.d, it makes the service wanted by multi-user.target or graphical.target. Two consequences:

  • A script that shares its name with a native unit (ssh, cron and others on Debian and Ubuntu) is ignored at boot. On Debian, even /etc/init.d/<name> start is redirected to systemctl. Edits to such a script rarely run.
  • A new script with start links does run at boot, as root, with no unit file on disk.

Upstream systemd deprecated this compatibility layer in version 256 and is removing it. On the newest releases, check that systemd-sysv-generator still exists before assuming an init script runs. Alpine and Gentoo use OpenRC instead: scripts also live in /etc/init.d, but runlevel links live in /etc/runlevels/<level>/.

Where it lives

PathWhatWho can write it
/etc/init.d/<name>Scripts on Debian and Ubuntu (a link to rc.d/init.d on RHEL)root
/etc/rc.d/init.d/<name>Scripts on RHEL, CentOS, older Fedoraroot
/etc/rc0.d/ … /etc/rc6.d/, /etc/rcS.d/S/K links per runlevel on Debianroot
/etc/rc.d/rc0.d/ … rc6.d/Same on RHEL (/etc/rcN.d are links)root
/etc/default/<name>, /etc/sysconfig/<name>Settings sourced by scriptsroot
/etc/inittabsysvinit runlevel and respawn tableroot
/etc/runlevels/<level>/OpenRC enablement linksroot
/run/systemd/generator.late/<name>.serviceUnits systemd builds from scriptssystemd-sysv-generator

How attackers use it

The classic form needs root and nothing else. The LSB header makes update-rc.d or chkconfig --add accept it, and the script can be linked by hand just as well (ln -s ../init.d/netconsole-agent /etc/rc2.d/S99netconsole-agent).

#!/bin/sh
### BEGIN INIT INFO
# Provides:          netconsole-agent
# Required-Start:    $network $remote_fs
# Required-Stop:
# Default-Start:     2 3 4 5
# Default-Stop:
# Short-Description: Network console agent
### END INIT INFO
case "$1" in
  start) nohup /usr/lib/.ncon/agent >/dev/null 2>&1 & ;;
esac
exit 0

XorDDoS is publicly documented installing randomly named scripts in /etc/init.d with start links, alongside cron entries that restore them.

Editing a script or its settings file

A line added to a packaged script, or to the settings file it sources, runs whenever the script does. The settings file is the quieter choice, since few people read it:

# appended to /etc/default/<name>, sourced by /etc/init.d/<name>
/usr/lib/.ncon/agent >/dev/null 2>&1 &

On systemd hosts this only works for scripts without a native unit; on hosts still booting SysV init, it always works.

A respawn line in inittab

On real sysvinit hosts (Devuan, old CentOS 6 servers, appliances), init itself restarts a process listed with respawn whenever it exits:

# /etc/inittab
ka:2345:respawn:/usr/lib/.ncon/agent

For the single-file variant of boot persistence, see rc.local; for the modern equivalent, systemd services.

How to check a host

On a live host

Do not run service --status-all or /etc/init.d/<name> status on a suspect host: both execute the scripts.

ls -la --time-style=full-iso /etc/init.d/
ls -l /etc/rc?.d/ | grep ' S[0-9]'                   # Debian start links (/etc/rc.d/rc?.d on RHEL)
chkconfig --list 2>/dev/null                         # RHEL: SysV services and their runlevels
ls -la /run/systemd/generator.late/                  # services systemd built from scripts
systemctl list-units --all | grep -E 'LSB:|SYSV:'    # their descriptions start with LSB: or SYSV:
systemctl show -p SourcePath,ExecStart netconsole-agent.service
dpkg -S /etc/init.d/* 2>&1 | grep 'no path found'    # RHEL: rpm -qf /etc/rc.d/init.d/*
sudo dpkg -V | grep -E '/etc/(init\.d|default)/'
grep -v '^#' /etc/inittab 2>/dev/null

For every script no package owns, read the whole body, not just the header: the Short-Description is whatever the author typed. Check whether a native unit with the same name exists (systemctl cat <name>.service) to know whether the script runs at boot at all.

Offline or from a collection

List the scripts and the links with their targets (find /mnt/image/etc -path '*rc*.d/*' -type l -printf '%p -> %l\n'). A link's mtime is roughly when the script was enabled. Debian tracks packaged init scripts and /etc/default files as conffiles: compare their checksums with Conffiles: in /var/lib/dpkg/status, and ownership with /var/lib/dpkg/info/*.list. On RPM systems, use rpm --root /mnt/image -Va. Read /etc/inittab if present. In the journal, the generated service appears under the script's name: journalctl -D /mnt/image/var/log/journal -u <name>.service.

Linux Persistence does this offline from the collection command.

What Linux Persistence flags

Each script in /etc/init.d or /etc/rc.d/init.d is one entry, except README, skeleton and insserv's .depend.* files. Its description comes from the Short-Description: or chkconfig description: line. It is shown as enabled, with its runlevels, when an S link in /etc/rc0.d to /etc/rc6.d (or /etc/rc.d/rcN.d) points to it. It is disabled when the collection has rc links but none start it, and "enablement unknown" when the collection has no rc links at all.

The shared command heuristics run on the script body, comment lines skipped:

  • Downloads and runs code, Reverse or bind shell and Decodes and runs a payload for fetch-and-run, socket and decode-and-run patterns.
  • Runs from a hidden folder and Runs from a temporary folder for paths like /usr/lib/.ncon/agent or /dev/shm/x.
  • Detaches from the session for nohup, setsid or screen -dm, common in hand-written start sections.
  • Interpreter with inline code, Sets LD_PRELOAD, Turns shell history off, Makes a file immutable and Embedded encoded blob where they appear.

The package layer uses dpkg: Not installed by any package (medium) for a script in /etc/init.d that no package owns, and Differs from package package (low) for a packaged script that was edited. Heuristic hits in an unmodified packaged script are shown as info. Changed date marks scripts modified in the 7 days before the newest file in the collection. program also runs from mechanisms fires when another mechanism, such as a cron job or rc.local, calls the same /etc/init.d/<name> path.

Limits: /etc/inittab, /etc/default, /etc/sysconfig and OpenRC's /etc/runlevels are not read. On Alpine and Gentoo, enablement is therefore unknown. The tool does not check the executable bit, whether a native unit shadows the script, or whether the host still has systemd-sysv-generator. Without a dpkg database (RHEL collections), there is no package verdict.

Cleaning up

  1. Preserve first: cp -a the script, its rc links, its settings file and the program it starts, and hash them.
  2. Stop it without running it. /etc/init.d/<name> stop and systemctl stop both execute the script's code; systemctl kill -s KILL <name>.service signals its processes directly, or kill the PIDs you found with ps.
  3. Remove the links (update-rc.d -f <name> remove on Debian, chkconfig --del <name> on RHEL), delete the script, and run systemctl daemon-reload on systemd hosts. For an edited packaged script, restore the original: on Debian, delete it and run apt-get install --reinstall -o Dpkg::Options::=--force-confmiss <package>.
  4. Check the other places the payload may be planted: cron, rc.local, systemd units and generators, /etc/default, /etc/inittab.
  5. Init scripts run as root: treat the host as root-compromised and rotate its credentials and keys.

References

See also

On Linux Forensics

The artifacts behind this technique, in the Linux Forensics reference: