Skip to content

At SSH login

sshd_config Backdoors: Linux Persistence via OpenSSH

How attackers change sshd_config, its drop-ins and Match blocks to keep SSH access to Linux hosts, and how to audit the effective OpenSSH server settings.

MITRE ATT&CK
T1098.004
Tactics
Persistence, Defense evasion
Runs
At SSH login
Set up by
root
Locations
  • /etc/ssh/sshd_config
  • /etc/ssh/sshd_config.d/*.conf
  • /etc/ssh/sshrc
  • /etc/default/ssh
  • /etc/sysconfig/sshd

How it works

sshd reads /etc/ssh/sshd_config when it starts and again on SIGHUP (systemctl reload ssh on Debian and Ubuntu, systemctl reload sshd on RHEL and Fedora). Sessions already open are not affected. For most keywords the first value obtained wins, so position matters. Debian 11+, Ubuntu 20.04+, RHEL 9+ and Fedora put Include /etc/ssh/sshd_config.d/*.conf at the top of the main file, which lets a drop-in override anything below it. RHEL ships 50-redhat.conf there; cloud images add files such as 50-cloud-init.conf.

Match starts a conditional block (User, Group, Host, Address, LocalPort and others). Its settings override the global ones for matching connections, until the next Match line or the end of the file. A block at the end of a long file is easy to overlook, and a plain sshd -T does not show it.

The keywords that decide who gets in:

KeywordDefaultWhy it matters
AuthorizedKeysFile.ssh/authorized_keys .ssh/authorized_keys2Where public keys are read
AuthorizedKeysCommandnoneA program whose output counts as authorized keys
TrustedUserCAKeys, AuthorizedPrincipalsFilenoneCertificate logins signed by a CA
PermitRootLoginprohibit-password (since OpenSSH 7.0)yes adds password logins as root
PermitEmptyPasswordsnoyes lets accounts without a password in
PermitUserEnvironmentnoEnables ~/.ssh/environment and environment= key options
PermitUserRCyesRuns ~/.ssh/rc at login
ForceCommandnoneReplaces every session's command

sshd also takes -o keyword=value and -f file on its command line. Debian's ssh.service adds $SSHD_OPTS from /etc/default/ssh, and RHEL's sshd.service adds $OPTIONS from /etc/sysconfig/sshd, so a setting can live outside the configuration files.

Where it lives

PathWhatWho can write it
/etc/ssh/sshd_configMain configurationroot
/etc/ssh/sshd_config.d/*.confDrop-ins pulled in by Includeroot, packages, cloud-init
/usr/share/openssh/sshd_configDebian/Ubuntu template the main file is generated from; sshd does not read itpackages
/usr/etc/ssh/sshd_configVendor defaults on openSUSE, used when /etc has nonepackages
/etc/default/ssh, /etc/sysconfig/sshdExtra command-line options for the serviceroot
/etc/systemd/system/ssh.service.d/, sshd.service.d/Unit drop-ins that can change ExecStartroot
/etc/ssh/sshrc, ~/.ssh/rc, ~/.ssh/environmentLogin script and per-user variablesroot; the account for ~
/usr/sbin/sshd, sshd-session (OpenSSH 9.8+, in /usr/lib/openssh/ or /usr/libexec/openssh/)The server binariespackages

How attackers use it

Read keys from a place nobody checks

AuthorizedKeysFile .ssh/authorized_keys /usr/share/misc/.cache/%u

Every account now also accepts keys from a hidden file, while the users' own files stay clean.

Hand authentication to a program

AuthorizedKeysCommand /usr/local/bin/ssh-key-sync %u
AuthorizedKeysCommandUser nobody

The script prints the attacker's key whatever the user name. sshd requires the program to be owned by root and not writable by group or others. Legitimate setups look the same: SSSD (sss_ssh_authorizedkeys), Google OS Login, EC2 Instance Connect.

Trust a certificate authority

TrustedUserCAKeys /etc/ssh/.ca.pub

Any certificate signed by the attacker's CA, with the account name among its principals, logs in. No authorized_keys file changes.

Widen the door for one source

Match Address 198.51.100.0/24
    PermitRootLogin yes
    PasswordAuthentication yes
    PermitEmptyPasswords yes

Combined with an account whose password is empty (see local accounts), this opens a password-less login for one network only.

Let users set LD_PRELOAD

PermitUserEnvironment yes
# /home/alice/.ssh/environment
LD_PRELOAD=/home/alice/.cache/libhook.so

The variable reaches the shell and every program it starts. Key-level variants are covered in SSH authorized keys.

Force every session through a wrapper

ForceCommand /usr/local/sbin/session-audit

A wrapper that records keystrokes or starts a payload, then runs $SSH_ORIGINAL_COMMAND or the login shell, is invisible to users.

Replace or patch sshd

A trojanised sshd, or a library it loads, is beyond any configuration parser. Ebury, documented by ESET for over a decade, steals credentials and accepts a backdoor password from inside OpenSSH processes; the 2024 xz-utils backdoor (CVE-2024-3094) reached sshd through liblzma. Package verification, below, is the check for that.

How to check a host

On a live host

sudo sshd -T | grep -Ei '^(authorizedkeys|authorizedprincipals|trustedusercakeys|permit|passwordauthentication|forcecommand|port|listenaddress)'
sudo sshd -T -C user=root,host=example.com,addr=198.51.100.7 | grep -Ei '^(permitrootlogin|passwordauthentication|permitemptypasswords)'
grep -rniE '^\s*(include|match|authorizedkeys|trustedusercakeys|permit|forcecommand)' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/
ls -la --time-style=full-iso /etc/ssh/ /etc/ssh/sshd_config.d/
systemctl cat ssh.service sshd.service 2>/dev/null | grep -E '^(ExecStart|EnvironmentFile)'
cat /etc/default/ssh /etc/sysconfig/sshd 2>/dev/null
ps -eo pid,args | grep '[s]shd'
dpkg -V openssh-server        # rpm -V openssh-server on RHEL and Fedora

sshd -T prints the effective configuration with drop-ins and first-value-wins applied; -C evaluates the Match blocks for one connection. ps shows any -f or -o the running daemon got. dpkg -V prints ??5?????? for a binary whose checksum changed and nothing when all match, but the database it compares with sits on the same host.

On Debian and Ubuntu, sshd_config is not a dpkg conffile: ucf manages it and records its checksum in /var/lib/ucf/hashfile. A mismatch with md5sum /etc/ssh/sshd_config means the file changed after installation, by an administrator, an image build or an intruder. On RHEL and Fedora, rpm -V openssh-server covers it (marked c).

Offline or from a collection

  • Read sshd_config and the drop-ins in the order Include loads them, apply first-value-wins by hand, and note every Match block.
  • Read the service unit, its drop-ins, and /etc/default/ssh or /etc/sysconfig/sshd for -f and -o.
  • Hash /usr/sbin/sshd and sshd-session against var/lib/dpkg/info/openssh-server.md5sums, or run rpm --root <mount> -V openssh-server; keep copies of the binaries for analysis.
  • Compare file times with /var/log/dpkg.log or /var/log/dnf.rpm.log: a config that changed when no package was installed was edited by hand.

Linux Persistence reads sshd_config, its drop-ins, /etc/ssh/sshrc and users' ~/.ssh/rc and ~/.ssh/environment from a collection.

What Linux Persistence flags

Each flagged setting becomes an entry with its line; inside a Match block, the condition is shown with it.

  • Custom AuthorizedKeysFile (medium): a listed file other than .ssh/authorized_keys or .ssh/authorized_keys2 (with or without %h/) or none.
  • AuthorizedKeysCommand set (medium): any value except none.
  • Root login allowed (medium): PermitRootLogin yes; prohibit-password and forced-commands-only are not flagged.
  • PermitUserEnvironment on (medium): any value other than no, a list of variable names included.
  • Empty passwords allowed (high): PermitEmptyPasswords yes.
  • ForceCommand set (info): any ForceCommand, for review.
  • SSH rc script (medium) for /etc/ssh/sshrc and ~/.ssh/rc; SSH environment file (low) for ~/.ssh/environment.
  • The value of each flagged setting and the content of rc and environment files go through the command heuristics: an AuthorizedKeysCommand under /tmp adds Runs from a temporary folder, an LD_PRELOAD= line adds Sets LD_PRELOAD.
  • With the dpkg database in the collection, an entry from a drop-in no package installed gets Not installed by any package (low), except cloud-init's 50-cloud-init.conf and 60-cloudimg-settings.conf; sshd_config itself is not reported, since Debian generates it outside dpkg. Changed date marks files modified within 7 days of the collection, and program also runs from mechanisms links an AuthorizedKeysCommand program that another mechanism also starts.

Limits: each file is read on its own. The tool does not apply Include order or first-value-wins, so a flagged line may be overridden elsewhere (confirm with sshd -T), and it reads every file in sshd_config.d, including names the *.conf glob skips. A drop-in that sets none of the keywords above produces no entry. TrustedUserCAKeys, AuthorizedPrincipalsFile, PasswordAuthentication, Port, ListenAddress, LogLevel, PermitUserRC and command-line options are not checked, and the sshd binaries are not verified: the only binaries the tool hashes are PAM modules.

Cleaning up

  1. Preserve sshd_config, the drop-ins, the unit files and /etc/default/ssh or /etc/sysconfig/sshd with cp -a, hash them, and save the sshd -T output.
  2. Keep a root session open, fix the lines, run sshd -t (syntax check), then reload. A reload does not end open sessions: find and end the intruder's separately.
  3. Remove what the settings pointed to: the extra key file, the AuthorizedKeysCommand program, the CA key, rogue ~/.ssh/environment and rc files.
  4. If the binaries fail verification, rebuild rather than clean. Reinstalling openssh-server (apt install --reinstall, dnf reinstall) replaces the binaries but not modified configuration files, and not the root access used to plant them.
  5. Rotate: a patched sshd sees every password typed at login, a logging ForceCommand sees everything typed in sessions (sudo passwords included), and root on the host could use any forwarded SSH agent. Check PAM for the same intruder's other doors.

References

See also

On Linux Forensics

The artifacts behind this technique, in the Linux Forensics reference: