Skip to content

At boot

Kernel Modules: Linux Persistence via modules-load.d

How attackers load rootkit kernel modules at boot and abuse modprobe install lines, what the configuration reveals, and how to check and clean a Linux host.

MITRE ATT&CK
T1547.006
Tactics
Persistence, Privilege escalation, Defense evasion
Runs
At boot
Set up by
root
Locations
  • /etc/modules
  • /etc/modules-load.d/*.conf
  • /etc/modprobe.d/*.conf
  • /usr/lib/modules-load.d/*.conf
  • /usr/lib/modprobe.d/*.conf

How it works

Two sets of configuration files decide what happens around module loading.

Loading at boot

systemd-modules-load.service reads the *.conf files of /etc/modules-load.d, /run/modules-load.d, /usr/local/lib/modules-load.d and /usr/lib/modules-load.d (a same-name file in /etc wins), one module name per line, # and ; starting comments, and loads each one early in boot. Debian and Ubuntu keep the older /etc/modules, read through the link /etc/modules-load.d/modules.conf. The kernel command line can add modules_load=. Modules are found by name in /lib/modules/$(uname -r)/ through the modules.dep index that depmod builds, so a module must be copied into that tree and indexed before a name in the configuration loads it. Separately, udev loads the drivers that match the hardware present.

What modprobe runs instead

The *.conf files of /etc/modprobe.d, /run/modprobe.d, /usr/local/lib/modprobe.d and /usr/lib/modprobe.d (/lib/modprobe.d on older layouts) hold alias, options, blacklist, softdep, install and remove lines. install <module> <command> tells modprobe to run a shell command instead of loading the module; remove does the same for unloading. The command runs as root, for whatever asked for the module: boot-time loading, udev's hardware matching, or the kernel's on-demand loading, which runs the helper named in /proc/sys/kernel/modprobe and can be set off by an unprivileged process, for example by opening a socket of a protocol family whose module is not loaded yet.

install <module> /bin/false (or /bin/true) is the standard way to block a module and appears in most hardening guides.

What stops a hostile module is the kernel, not the configuration: with Secure Boot and lockdown, or module.sig_enforce=1, unsigned modules are refused; otherwise an unsigned or out-of-tree module only taints the kernel. kernel.modules_disabled=1 blocks all loading until the next reboot.

Where it lives

PathWhatWho can write it
/etc/modulesDebian/Ubuntu list of modules loaded at bootroot
/etc/modules-load.d/*.conf, /run/…, /usr/lib/modules-load.d/*.confModules loaded at boot, local and vendorroot
/etc/modprobe.d/*.conf, /run/…, /usr/lib/modprobe.d/*.confmodprobe configuration, including install and removeroot
/lib/modules/<kernel>/ (kernel/, extra/, updates/)Module files and the modules.dep indexroot
/usr/src/<name>-<version>/dkms.conf, /var/lib/dkms/DKMS sources rebuilt for every new kernelroot
/etc/sysctl.conf, /etc/sysctl.d/*.confCan set kernel.modprobe at bootroot
initramfs (/boot/initrd.img-*, /boot/initramfs-*.img)Modules and configuration loaded before the root file systemroot

How attackers use it

A rootkit module loaded at every boot

The payload is a loadable kernel module that hides files, processes, ports and itself. Diamorphine and Reptile are open-source examples seen in intrusions; Skidmap loaded malicious modules to hide its cryptominer. Persistence is three steps:

cp nf_flow_ext.ko /lib/modules/$(uname -r)/kernel/net/netfilter/
depmod -a
echo nf_flow_ext > /etc/modules-load.d/nf-flow.conf

Once loaded, a rootkit typically unlinks itself from the module list, so lsmod stops showing it; the configuration line and the .ko stay on disk. A kernel update breaks the module unless it is rebuilt, which is why some implants ship as DKMS packages or reload through another mechanism, such as a unit running insmod.

A command hidden in an install line

# /etc/modprobe.d/usb-storage.conf
install usb_storage /usr/local/lib/.cache/usbmon; /sbin/modprobe --ignore-install usb_storage

Chaining modprobe --ignore-install loads the real module afterwards, so the device keeps working and nobody notices. On a module loaded at every boot this is boot persistence; on one any user can make the kernel request, it is a root trigger for the attacker's own unprivileged account.

Redirecting the helper

# /etc/sysctl.d/60-perf.conf
kernel.modprobe = /var/lib/.kmod/modprobe

Every on-demand module request then runs the attacker's program as root.

How to check a host

On a live host

A loaded rootkit can falsify anything the kernel reports, so treat live output as a hint and confirm offline or from a memory image.

cat /etc/modules /etc/modules-load.d/*.conf /run/modules-load.d/*.conf /usr/lib/modules-load.d/*.conf 2>/dev/null
modprobe -c | grep -E '^(install|remove) '      # effective modprobe configuration
modprobe -n -v usb_storage                       # dry run: prints what would run, runs nothing
sysctl kernel.modprobe kernel.tainted kernel.modules_disabled
dmesg | grep -iE 'taint|module verification'
lsmod | awk 'NR>1 {print $1}' | xargs -n1 modinfo -F filename | xargs -r dpkg -S 2>&1 | grep 'no path found'
# use: … | xargs -r rpm -qf | grep 'not owned'  on RHEL/Fedora
for m in /sys/module/*; do [ -e "$m/initstate" ] && ! grep -q "^${m##*/} " /proc/modules && echo "$m"; done
dkms status
journalctl -b -u systemd-modules-load

A non-zero kernel.tainted with bit 12 (out-of-tree) or bit 13 (unsigned) set means such a module was loaded since boot; NVIDIA, VirtualBox or ZFS explain it on many hosts. The loop lists modules present in sysfs but missing from /proc/modules, which catches rootkits that unlink themselves from only one of the two.

Offline or from a collection

Read the configuration from the image, then the module tree: .ko files that no package owns (dpkg -S, rpm -qf; DKMS builds in updates/dkms are expected exceptions), recent mtimes under /lib/modules, modules.dep entries for paths you cannot explain, and modinfo <file>.ko for the author, description, signer and vermagic. List the initramfs too (lsinitramfs on Debian, lsinitrd on RHEL). Loaded but hidden modules only show in a memory image.

Linux Persistence reads the configuration side from a collection command archive.

What Linux Persistence flags

The tool reads configuration, not the kernel: /etc/modules and the modules-load.d and modprobe.d folders under /etc, /run, /usr/lib and /lib. It does not see which modules are loaded, does not read .ko files, signatures or taint, and cannot spot a module hidden in memory.

Every module name in /etc/modules or modules-load.d is listed as a boot entry, with no content flag: review the list. In modprobe.d, only install lines become entries, and install <module> /bin/true or /bin/false (also under /usr/bin) is skipped as a block.

  • modprobe runs a command: any other install line (high).
  • Downloads and runs code, Reverse or bind shell, Decodes and runs a payload (high): the install command fetches code, opens a shell on a socket or decodes a payload.
  • Runs from a temporary folder (high) and Runs from a hidden folder (medium): the command calls something under /tmp, /var/tmp, /dev/shm or a dot-named path, as in the example above.
  • Not installed by any package: an unowned configuration file, low in /etc/modprobe.d and /etc/modules-load.d, medium under /usr/lib or /lib. /etc/modules is exempt, since installers write it.
  • Differs from package package: a packaged file whose checksum changed, low in /etc, high under /usr/lib or /lib.
  • Changed date: modified within 7 days before the newest file in the collection (low).
  • program also runs from mechanisms: the install command's program is also started by another mechanism, such as a systemd unit (medium).

Flags in unmodified packaged files are shown as info only. The package checks need /var/lib/dpkg in the collection; RPM databases are not read. remove lines, kernel.modprobe in sysctl.d, the kernel command line, DKMS trees and the initramfs are not read.

Cleaning up

  1. Capture memory before anything else: a loaded rootkit exists only there, and a reboot destroys it. Then copy the configuration files and the .ko with their times, hash them, and save lsmod, dmesg and /proc/sys/kernel/tainted.
  2. Remove the configuration line or file, delete the module file, run depmod -a, and remove any DKMS module (dkms remove <name>/<version> --all). Rebuild the initramfs (update-initramfs -u or dracut -f).
  3. Do not trust rmmod on a module that hides itself. Reboot into the cleaned system, and prefer a rebuild: code that ran in the kernel could change anything.
  4. Look for the reloader: units, cron, rc.local or udev rules running insmod or modprobe, and preloaded libraries such as ld.so.preload that rootkits often pair with a module.
  5. Rotate every credential and key the host held. For the future, enforce module signatures with Secure Boot, or set kernel.modules_disabled=1 once boot has finished on servers that never load modules later.

References

See also

On Linux Forensics

The artifacts behind this technique, in the Linux Forensics reference: