At boot
Kernel Modules: Linux Persistence via modules-load.d
How attackers load rootkit kernel modules at boot and abuse modprobe install lines, what the configuration reveals, and how to check and clean a Linux host.
- MITRE ATT&CK
- T1547.006
- Tactics
- Persistence, Privilege escalation, Defense evasion
- Runs
- At boot
- Set up by
- root
- Locations
- /etc/modules
- /etc/modules-load.d/*.conf
- /etc/modprobe.d/*.conf
- /usr/lib/modules-load.d/*.conf
- /usr/lib/modprobe.d/*.conf
How it works
Two sets of configuration files decide what happens around module loading.
Loading at boot
systemd-modules-load.service reads the *.conf files of /etc/modules-load.d, /run/modules-load.d, /usr/local/lib/modules-load.d and /usr/lib/modules-load.d (a same-name file in /etc wins), one module name per line, # and ; starting comments, and loads each one early in boot. Debian and Ubuntu keep the older /etc/modules, read through the link /etc/modules-load.d/modules.conf. The kernel command line can add modules_load=. Modules are found by name in /lib/modules/$(uname -r)/ through the modules.dep index that depmod builds, so a module must be copied into that tree and indexed before a name in the configuration loads it. Separately, udev loads the drivers that match the hardware present.
What modprobe runs instead
The *.conf files of /etc/modprobe.d, /run/modprobe.d, /usr/local/lib/modprobe.d and /usr/lib/modprobe.d (/lib/modprobe.d on older layouts) hold alias, options, blacklist, softdep, install and remove lines. install <module> <command> tells modprobe to run a shell command instead of loading the module; remove does the same for unloading. The command runs as root, for whatever asked for the module: boot-time loading, udev's hardware matching, or the kernel's on-demand loading, which runs the helper named in /proc/sys/kernel/modprobe and can be set off by an unprivileged process, for example by opening a socket of a protocol family whose module is not loaded yet.
install <module> /bin/false (or /bin/true) is the standard way to block a module and appears in most hardening guides.
What stops a hostile module is the kernel, not the configuration: with Secure Boot and lockdown, or module.sig_enforce=1, unsigned modules are refused; otherwise an unsigned or out-of-tree module only taints the kernel. kernel.modules_disabled=1 blocks all loading until the next reboot.
Where it lives
| Path | What | Who can write it |
|---|---|---|
/etc/modules | Debian/Ubuntu list of modules loaded at boot | root |
/etc/modules-load.d/*.conf, /run/…, /usr/lib/modules-load.d/*.conf | Modules loaded at boot, local and vendor | root |
/etc/modprobe.d/*.conf, /run/…, /usr/lib/modprobe.d/*.conf | modprobe configuration, including install and remove | root |
/lib/modules/<kernel>/ (kernel/, extra/, updates/) | Module files and the modules.dep index | root |
/usr/src/<name>-<version>/dkms.conf, /var/lib/dkms/ | DKMS sources rebuilt for every new kernel | root |
/etc/sysctl.conf, /etc/sysctl.d/*.conf | Can set kernel.modprobe at boot | root |
initramfs (/boot/initrd.img-*, /boot/initramfs-*.img) | Modules and configuration loaded before the root file system | root |
How attackers use it
A rootkit module loaded at every boot
The payload is a loadable kernel module that hides files, processes, ports and itself. Diamorphine and Reptile are open-source examples seen in intrusions; Skidmap loaded malicious modules to hide its cryptominer. Persistence is three steps:
cp nf_flow_ext.ko /lib/modules/$(uname -r)/kernel/net/netfilter/
depmod -a
echo nf_flow_ext > /etc/modules-load.d/nf-flow.conf
Once loaded, a rootkit typically unlinks itself from the module list, so lsmod stops showing it; the configuration line and the .ko stay on disk. A kernel update breaks the module unless it is rebuilt, which is why some implants ship as DKMS packages or reload through another mechanism, such as a unit running insmod.
A command hidden in an install line
# /etc/modprobe.d/usb-storage.conf
install usb_storage /usr/local/lib/.cache/usbmon; /sbin/modprobe --ignore-install usb_storage
Chaining modprobe --ignore-install loads the real module afterwards, so the device keeps working and nobody notices. On a module loaded at every boot this is boot persistence; on one any user can make the kernel request, it is a root trigger for the attacker's own unprivileged account.
Redirecting the helper
# /etc/sysctl.d/60-perf.conf
kernel.modprobe = /var/lib/.kmod/modprobe
Every on-demand module request then runs the attacker's program as root.
How to check a host
On a live host
A loaded rootkit can falsify anything the kernel reports, so treat live output as a hint and confirm offline or from a memory image.
cat /etc/modules /etc/modules-load.d/*.conf /run/modules-load.d/*.conf /usr/lib/modules-load.d/*.conf 2>/dev/null
modprobe -c | grep -E '^(install|remove) ' # effective modprobe configuration
modprobe -n -v usb_storage # dry run: prints what would run, runs nothing
sysctl kernel.modprobe kernel.tainted kernel.modules_disabled
dmesg | grep -iE 'taint|module verification'
lsmod | awk 'NR>1 {print $1}' | xargs -n1 modinfo -F filename | xargs -r dpkg -S 2>&1 | grep 'no path found'
# use: … | xargs -r rpm -qf | grep 'not owned' on RHEL/Fedora
for m in /sys/module/*; do [ -e "$m/initstate" ] && ! grep -q "^${m##*/} " /proc/modules && echo "$m"; done
dkms status
journalctl -b -u systemd-modules-load
A non-zero kernel.tainted with bit 12 (out-of-tree) or bit 13 (unsigned) set means such a module was loaded since boot; NVIDIA, VirtualBox or ZFS explain it on many hosts. The loop lists modules present in sysfs but missing from /proc/modules, which catches rootkits that unlink themselves from only one of the two.
Offline or from a collection
Read the configuration from the image, then the module tree: .ko files that no package owns (dpkg -S, rpm -qf; DKMS builds in updates/dkms are expected exceptions), recent mtimes under /lib/modules, modules.dep entries for paths you cannot explain, and modinfo <file>.ko for the author, description, signer and vermagic. List the initramfs too (lsinitramfs on Debian, lsinitrd on RHEL). Loaded but hidden modules only show in a memory image.
Linux Persistence reads the configuration side from a collection command archive.
What Linux Persistence flags
The tool reads configuration, not the kernel: /etc/modules and the modules-load.d and modprobe.d folders under /etc, /run, /usr/lib and /lib. It does not see which modules are loaded, does not read .ko files, signatures or taint, and cannot spot a module hidden in memory.
Every module name in /etc/modules or modules-load.d is listed as a boot entry, with no content flag: review the list. In modprobe.d, only install lines become entries, and install <module> /bin/true or /bin/false (also under /usr/bin) is skipped as a block.
- modprobe runs a command: any other
installline (high). - Downloads and runs code, Reverse or bind shell, Decodes and runs a payload (high): the install command fetches code, opens a shell on a socket or decodes a payload.
- Runs from a temporary folder (high) and Runs from a hidden folder (medium): the command calls something under
/tmp,/var/tmp,/dev/shmor a dot-named path, as in the example above. - Not installed by any package: an unowned configuration file, low in
/etc/modprobe.dand/etc/modules-load.d, medium under/usr/libor/lib./etc/modulesis exempt, since installers write it. - Differs from package package: a packaged file whose checksum changed, low in
/etc, high under/usr/libor/lib. - Changed date: modified within 7 days before the newest file in the collection (low).
- program also runs from mechanisms: the install command's program is also started by another mechanism, such as a systemd unit (medium).
Flags in unmodified packaged files are shown as info only. The package checks need /var/lib/dpkg in the collection; RPM databases are not read. remove lines, kernel.modprobe in sysctl.d, the kernel command line, DKMS trees and the initramfs are not read.
Cleaning up
- Capture memory before anything else: a loaded rootkit exists only there, and a reboot destroys it. Then copy the configuration files and the
.kowith their times, hash them, and savelsmod,dmesgand/proc/sys/kernel/tainted. - Remove the configuration line or file, delete the module file, run
depmod -a, and remove any DKMS module (dkms remove <name>/<version> --all). Rebuild the initramfs (update-initramfs -uordracut -f). - Do not trust
rmmodon a module that hides itself. Reboot into the cleaned system, and prefer a rebuild: code that ran in the kernel could change anything. - Look for the reloader: units, cron, rc.local or udev rules running
insmodormodprobe, and preloaded libraries such as ld.so.preload that rootkits often pair with a module. - Rotate every credential and key the host held. For the future, enforce module signatures with Secure Boot, or set
kernel.modules_disabled=1once boot has finished on servers that never load modules later.
References
See also
Related techniques
On Linux Forensics
The artifacts behind this technique, in the Linux Forensics reference: