At SSH login
sshd_config Backdoors: Linux Persistence via OpenSSH
How attackers change sshd_config, its drop-ins and Match blocks to keep SSH access to Linux hosts, and how to audit the effective OpenSSH server settings.
- MITRE ATT&CK
- T1098.004
- Tactics
- Persistence, Defense evasion
- Runs
- At SSH login
- Set up by
- root
- Locations
- /etc/ssh/sshd_config
- /etc/ssh/sshd_config.d/*.conf
- /etc/ssh/sshrc
- /etc/default/ssh
- /etc/sysconfig/sshd
How it works
sshd reads /etc/ssh/sshd_config when it starts and again on SIGHUP (systemctl reload ssh on Debian and Ubuntu, systemctl reload sshd on RHEL and Fedora). Sessions already open are not affected. For most keywords the first value obtained wins, so position matters. Debian 11+, Ubuntu 20.04+, RHEL 9+ and Fedora put Include /etc/ssh/sshd_config.d/*.conf at the top of the main file, which lets a drop-in override anything below it. RHEL ships 50-redhat.conf there; cloud images add files such as 50-cloud-init.conf.
Match starts a conditional block (User, Group, Host, Address, LocalPort and others). Its settings override the global ones for matching connections, until the next Match line or the end of the file. A block at the end of a long file is easy to overlook, and a plain sshd -T does not show it.
The keywords that decide who gets in:
| Keyword | Default | Why it matters |
|---|---|---|
AuthorizedKeysFile | .ssh/authorized_keys .ssh/authorized_keys2 | Where public keys are read |
AuthorizedKeysCommand | none | A program whose output counts as authorized keys |
TrustedUserCAKeys, AuthorizedPrincipalsFile | none | Certificate logins signed by a CA |
PermitRootLogin | prohibit-password (since OpenSSH 7.0) | yes adds password logins as root |
PermitEmptyPasswords | no | yes lets accounts without a password in |
PermitUserEnvironment | no | Enables ~/.ssh/environment and environment= key options |
PermitUserRC | yes | Runs ~/.ssh/rc at login |
ForceCommand | none | Replaces every session's command |
sshd also takes -o keyword=value and -f file on its command line. Debian's ssh.service adds $SSHD_OPTS from /etc/default/ssh, and RHEL's sshd.service adds $OPTIONS from /etc/sysconfig/sshd, so a setting can live outside the configuration files.
Where it lives
| Path | What | Who can write it |
|---|---|---|
/etc/ssh/sshd_config | Main configuration | root |
/etc/ssh/sshd_config.d/*.conf | Drop-ins pulled in by Include | root, packages, cloud-init |
/usr/share/openssh/sshd_config | Debian/Ubuntu template the main file is generated from; sshd does not read it | packages |
/usr/etc/ssh/sshd_config | Vendor defaults on openSUSE, used when /etc has none | packages |
/etc/default/ssh, /etc/sysconfig/sshd | Extra command-line options for the service | root |
/etc/systemd/system/ssh.service.d/, sshd.service.d/ | Unit drop-ins that can change ExecStart | root |
/etc/ssh/sshrc, ~/.ssh/rc, ~/.ssh/environment | Login script and per-user variables | root; the account for ~ |
/usr/sbin/sshd, sshd-session (OpenSSH 9.8+, in /usr/lib/openssh/ or /usr/libexec/openssh/) | The server binaries | packages |
How attackers use it
Read keys from a place nobody checks
AuthorizedKeysFile .ssh/authorized_keys /usr/share/misc/.cache/%u
Every account now also accepts keys from a hidden file, while the users' own files stay clean.
Hand authentication to a program
AuthorizedKeysCommand /usr/local/bin/ssh-key-sync %u
AuthorizedKeysCommandUser nobody
The script prints the attacker's key whatever the user name. sshd requires the program to be owned by root and not writable by group or others. Legitimate setups look the same: SSSD (sss_ssh_authorizedkeys), Google OS Login, EC2 Instance Connect.
Trust a certificate authority
TrustedUserCAKeys /etc/ssh/.ca.pub
Any certificate signed by the attacker's CA, with the account name among its principals, logs in. No authorized_keys file changes.
Widen the door for one source
Match Address 198.51.100.0/24
PermitRootLogin yes
PasswordAuthentication yes
PermitEmptyPasswords yes
Combined with an account whose password is empty (see local accounts), this opens a password-less login for one network only.
Let users set LD_PRELOAD
PermitUserEnvironment yes
# /home/alice/.ssh/environment
LD_PRELOAD=/home/alice/.cache/libhook.so
The variable reaches the shell and every program it starts. Key-level variants are covered in SSH authorized keys.
Force every session through a wrapper
ForceCommand /usr/local/sbin/session-audit
A wrapper that records keystrokes or starts a payload, then runs $SSH_ORIGINAL_COMMAND or the login shell, is invisible to users.
Replace or patch sshd
A trojanised sshd, or a library it loads, is beyond any configuration parser. Ebury, documented by ESET for over a decade, steals credentials and accepts a backdoor password from inside OpenSSH processes; the 2024 xz-utils backdoor (CVE-2024-3094) reached sshd through liblzma. Package verification, below, is the check for that.
How to check a host
On a live host
sudo sshd -T | grep -Ei '^(authorizedkeys|authorizedprincipals|trustedusercakeys|permit|passwordauthentication|forcecommand|port|listenaddress)'
sudo sshd -T -C user=root,host=example.com,addr=198.51.100.7 | grep -Ei '^(permitrootlogin|passwordauthentication|permitemptypasswords)'
grep -rniE '^\s*(include|match|authorizedkeys|trustedusercakeys|permit|forcecommand)' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/
ls -la --time-style=full-iso /etc/ssh/ /etc/ssh/sshd_config.d/
systemctl cat ssh.service sshd.service 2>/dev/null | grep -E '^(ExecStart|EnvironmentFile)'
cat /etc/default/ssh /etc/sysconfig/sshd 2>/dev/null
ps -eo pid,args | grep '[s]shd'
dpkg -V openssh-server # rpm -V openssh-server on RHEL and Fedora
sshd -T prints the effective configuration with drop-ins and first-value-wins applied; -C evaluates the Match blocks for one connection. ps shows any -f or -o the running daemon got. dpkg -V prints ??5?????? for a binary whose checksum changed and nothing when all match, but the database it compares with sits on the same host.
On Debian and Ubuntu, sshd_config is not a dpkg conffile: ucf manages it and records its checksum in /var/lib/ucf/hashfile. A mismatch with md5sum /etc/ssh/sshd_config means the file changed after installation, by an administrator, an image build or an intruder. On RHEL and Fedora, rpm -V openssh-server covers it (marked c).
Offline or from a collection
- Read
sshd_configand the drop-ins in the orderIncludeloads them, apply first-value-wins by hand, and note everyMatchblock. - Read the service unit, its drop-ins, and
/etc/default/sshor/etc/sysconfig/sshdfor-fand-o. - Hash
/usr/sbin/sshdandsshd-sessionagainstvar/lib/dpkg/info/openssh-server.md5sums, or runrpm --root <mount> -V openssh-server; keep copies of the binaries for analysis. - Compare file times with
/var/log/dpkg.logor/var/log/dnf.rpm.log: a config that changed when no package was installed was edited by hand.
Linux Persistence reads sshd_config, its drop-ins, /etc/ssh/sshrc and users' ~/.ssh/rc and ~/.ssh/environment from a collection.
What Linux Persistence flags
Each flagged setting becomes an entry with its line; inside a Match block, the condition is shown with it.
- Custom AuthorizedKeysFile (medium): a listed file other than
.ssh/authorized_keysor.ssh/authorized_keys2(with or without%h/) ornone. - AuthorizedKeysCommand set (medium): any value except
none. - Root login allowed (medium):
PermitRootLogin yes;prohibit-passwordandforced-commands-onlyare not flagged. - PermitUserEnvironment on (medium): any value other than
no, a list of variable names included. - Empty passwords allowed (high):
PermitEmptyPasswords yes. - ForceCommand set (info): any
ForceCommand, for review. - SSH rc script (medium) for
/etc/ssh/sshrcand~/.ssh/rc; SSH environment file (low) for~/.ssh/environment. - The value of each flagged setting and the content of rc and environment files go through the command heuristics: an
AuthorizedKeysCommandunder/tmpadds Runs from a temporary folder, anLD_PRELOAD=line adds Sets LD_PRELOAD. - With the dpkg database in the collection, an entry from a drop-in no package installed gets Not installed by any package (low), except cloud-init's
50-cloud-init.confand60-cloudimg-settings.conf;sshd_configitself is not reported, since Debian generates it outside dpkg. Changed date marks files modified within 7 days of the collection, and program also runs from mechanisms links anAuthorizedKeysCommandprogram that another mechanism also starts.
Limits: each file is read on its own. The tool does not apply Include order or first-value-wins, so a flagged line may be overridden elsewhere (confirm with sshd -T), and it reads every file in sshd_config.d, including names the *.conf glob skips. A drop-in that sets none of the keywords above produces no entry. TrustedUserCAKeys, AuthorizedPrincipalsFile, PasswordAuthentication, Port, ListenAddress, LogLevel, PermitUserRC and command-line options are not checked, and the sshd binaries are not verified: the only binaries the tool hashes are PAM modules.
Cleaning up
- Preserve
sshd_config, the drop-ins, the unit files and/etc/default/sshor/etc/sysconfig/sshdwithcp -a, hash them, and save thesshd -Toutput. - Keep a root session open, fix the lines, run
sshd -t(syntax check), then reload. A reload does not end open sessions: find and end the intruder's separately. - Remove what the settings pointed to: the extra key file, the
AuthorizedKeysCommandprogram, the CA key, rogue~/.ssh/environmentand rc files. - If the binaries fail verification, rebuild rather than clean. Reinstalling
openssh-server(apt install --reinstall,dnf reinstall) replaces the binaries but not modified configuration files, and not the root access used to plant them. - Rotate: a patched sshd sees every password typed at login, a logging
ForceCommandsees everything typed in sessions (sudo passwords included), and root on the host could use any forwarded SSH agent. Check PAM for the same intruder's other doors.
References
See also
Related techniques
On Linux Forensics
The artifacts behind this technique, in the Linux Forensics reference: