Skip to content

At SSH login

SSH Authorized Keys: Linux Persistence via authorized_keys

How attackers plant SSH keys, forced commands and ~/.ssh/rc scripts to keep access to Linux hosts, and how to find rogue authorized_keys entries.

MITRE ATT&CK
T1098.004
Tactics
Persistence
Runs
At SSH login
Set up by
Users and root
Locations
  • ~/.ssh/authorized_keys
  • ~/.ssh/authorized_keys2
  • /root/.ssh/authorized_keys
  • ~/.ssh/rc
  • ~/.ssh/environment
  • /etc/ssh/sshrc

How it works

When a client offers a public key, sshd looks up the target account and reads the files named by AuthorizedKeysFile, relative to that account's home. The OpenSSH default is .ssh/authorized_keys .ssh/authorized_keys2. Debian and Ubuntu leave the line commented out, so both files count; RHEL and Fedora set AuthorizedKeysFile .ssh/authorized_keys, so authorized_keys2 is ignored there. If the client proves it holds the private half of any listed key, the login succeeds without a password.

Each line is [options] type base64-key [comment]. The comment is free text that sshd never checks. Options restrict or extend what the key may do:

OptionEffect
command="…"Runs this instead of the shell or the requested command; the client's command is in SSH_ORIGINAL_COMMAND
environment="NAME=value"Sets a variable for the session, only if PermitUserEnvironment allows it
from="pattern-list"Accepts the key only from matching addresses or host names
no-pty, no-port-forwarding, restrictNarrow what the session may do
cert-authorityTrusts certificates signed by this key rather than the key itself
no-user-rcSkips ~/.ssh/rc for this key

With StrictModes yes (the default), sshd ignores the file when it, ~/.ssh or the home folder is writable by group or others. A planted key that "does not work" often fails this check.

After authentication and before the shell or command starts, sshd runs ~/.ssh/rc with sh if it exists, otherwise /etc/ssh/sshrc. PermitUserRC no or the no-user-rc option disables the per-user script. ~/.ssh/environment is read only when PermitUserEnvironment is not no, and no is the default.

Keys survive password changes, and with UsePAM yes (the default on Debian, Ubuntu, RHEL and Fedora) locking the password with passwd -l does not stop a key login: sshd checks the ! lock itself only when PAM is off. On cloud images, cloud-init writes the metadata keys into the default user's file at first boot.

Where it lives

PathWhatWho can write it
~/.ssh/authorized_keysKeys for that accountThe account, root
~/.ssh/authorized_keys2Legacy second file, read by default except where AuthorizedKeysFile drops itThe account, root
/root/.ssh/authorized_keysRoot logins, allowed with keys under the default PermitRootLogin prohibit-passwordroot
/var/lib/<svc>/.ssh/, /var/www/.ssh/, /srv/<svc>/.ssh/Keys for service accounts whose home is thereThe service account, root
Path set by AuthorizedKeysFile, e.g. /etc/ssh/authorized_keys/%uCentral key files on some hardened hostsroot
~/.ssh/rcScript run at every SSH login of that accountThe account, root
/etc/ssh/sshrcSystem script, run when the user has no ~/.ssh/rcroot
~/.ssh/environmentVariables set at every SSH loginThe account, root

No package ships an authorized_keys file: every key was put there by a person, cloud-init or configuration management.

How attackers use it

Append a key

Cryptomining groups such as TeamTNT add their own key to root's file, and unauthenticated Redis servers have long been abused by pointing CONFIG SET dir at /root/.ssh and saving a key as the database file, which leaves binary junk around the key line.

mkdir -p /root/.ssh && chmod 700 /root/.ssh
echo 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK3x...Qz root@backup01' >> /root/.ssh/authorized_keys
chattr +i /root/.ssh/authorized_keys

The immutable attribute makes a later cleanup fail with "Operation not permitted".

Copy a legitimate comment

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGb1...kP alice@laptop
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIM9q...7w alice@laptop

A reviewer reading comments sees the same owner twice.

Hide where nobody looks

Keys in authorized_keys2, or under /var/www/.ssh/ after giving www-data a shell (see local accounts), escape scripts that only read /home/*/.ssh/authorized_keys.

Forced command on someone else's key

Prepending command= to an administrator's existing key runs a payload at each of their logins and still hands them a shell:

command="/var/tmp/.x/upd >/dev/null 2>&1 & exec $SHELL -l" ssh-ed25519 AAAAC3Nza...Lq admin@jump01

The giveaway: scp, sftp and git over SSH to that account stop working, because the forced command ignores SSH_ORIGINAL_COMMAND.

Session environment

Once PermitUserEnvironment yes is set in sshd_config, a key option or ~/.ssh/environment can preload a library into the user's shell:

environment="LD_PRELOAD=/home/alice/.cache/libnss.so" ssh-ed25519 AAAAC3Nza...Lq alice@laptop

A login script

# ~/.ssh/rc: runs with sh before the shell at every SSH login; must not write to stdout
nohup /dev/shm/.k >/dev/null 2>&1 &

How to check a host

On a live host

List every key file sshd would read, for every account, with owner, mode, times, attributes and fingerprints:

getent passwd | while IFS=: read -r user _ uid _ _ home shell; do
  for f in "$home/.ssh/authorized_keys" "$home/.ssh/authorized_keys2"; do
    [ -f "$f" ] || continue
    echo "== $user (uid $uid, $shell) $f"
    stat -c 'mtime %y  ctime %z  %U:%G %a' "$f"
    lsattr "$f" 2>/dev/null
    ssh-keygen -lf "$f"
  done
done

Then check what sshd uses, find key files outside homes, and match fingerprints with logins:

sudo sshd -T | grep -Ei '^(authorizedkeys|trustedusercakeys|permituserenvironment|permituserrc)'
sudo find / /home -xdev \( -name 'authorized_keys*' -o -path '*/.ssh/rc' -o -path '*/.ssh/environment' \) 2>/dev/null
ls -l /etc/ssh/sshrc 2>/dev/null
grep 'Accepted publickey' /var/log/auth.log    # /var/log/secure on RHEL; or journalctl -u ssh -u sshd

Each Accepted publickey for root from 203.0.113.7 port 50122 ssh2: ED25519 SHA256:… line names the fingerprint used: match it with ssh-keygen -lf to learn which key opened which session, from where.

Offline or from a collection

  • Read every .ssh/authorized_keys* under /root, /home/* and the homes /etc/passwd lists for service accounts (/var/lib, /srv, /var/www, /opt), plus whatever AuthorizedKeysFile points to.
  • No package owns these files, so dpkg -V and rpm -V cannot help. Compare with your inventory: configuration management, cloud metadata, the image's defaults.
  • Compare file mtime and ctime with the first Accepted publickey line of each fingerprint in rotated auth logs and the journal.
  • Binary junk around a key line (a Redis dump), or a file inside /root/.ssh owned by a service account, points to an exploit writing the file.

Linux Persistence does this offline from the collection command, which archives /root/.ssh, /home/*/.ssh and /var/lib/*/.ssh.

What Linux Persistence flags

Each key line becomes one entry named after its comment (or its fingerprint when there is none), with the key type, the SHA256 fingerprint in the format sshd logs, the options, the from= restriction (any when absent) and the forced command.

  • Key opens a root login (low): the key sits in root's authorized_keys or authorized_keys2.
  • Key sets environment variables (medium): the line has an environment= option.
  • Weak or obsolete key type (info): the key is ssh-dss (DSA), or an ssh-rsa key under 2048 bits (the size is read from the key itself).
  • Same key for accounts (low): one fingerprint opens several accounts on the host.
  • Another key uses the comment “comment” (medium): two different keys on the host share a comment. Both lines are flagged, the legitimate one included, and a user who generated a new key without changing the comment triggers it too.
  • Changed date (low): the key file was modified within 7 days before the newest file in the collection.
  • Forced commands go through the command heuristics, so a command= can raise Reverse or bind shell, Downloads and runs code, Decodes and runs a payload, Runs from a temporary folder, Runs from a hidden folder or Detaches from the session, and program also runs from mechanisms when the same program is started by another mechanism.
  • ~/.ssh/rc and /etc/ssh/sshrc raise SSH rc script (medium) and ~/.ssh/environment raises SSH environment file (low); their content goes through the same heuristics, so an LD_PRELOAD= line adds Sets LD_PRELOAD. These entries are listed under the sshd mechanism.

Limits: the tool reads files, not sshd's effective settings. It does not check permissions or immutable attributes, does not follow a custom AuthorizedKeysFile (the setting is flagged on the sshd side), cannot see what an AuthorizedKeysCommand returns, and only places key files under /root, /home/*, /var/lib/*, /srv/*, /var/www and /var/spool/*.

Cleaning up

  1. Preserve first: copy each file with cp -a (keeps times), sha256sum it, save the stat output and fingerprints, and export the auth log lines for each rogue fingerprint.
  2. If lsattr shows i, record it, then clear it with chattr -i.
  3. Remove only the rogue lines; do not delete a file that also holds legitimate keys. Remove a rogue ~/.ssh/rc, ~/.ssh/environment or /etc/ssh/sshrc.
  4. End the sessions that used the key: who, ss -tnp state established '( sport = :ssh )' and the auth log give the processes to stop.
  5. Look for the same key or payload elsewhere: every account's two key files, AuthorizedKeysFile, AuthorizedKeysCommand and TrustedUserCAKeys in sshd_config, cloud instance metadata (cloud-init can put a key back), configuration management, and other hosts.
  6. To block an account while you investigate, expire it with usermod -e 1 <user>. A nologin shell alone still allows port forwarding with ssh -N.
  7. Rotate: private keys stored on the host (~/.ssh/id_*) may have been copied; known_hosts and shell history show where they lead. Replace them everywhere they are trusted.

References

See also

On Linux Forensics

The artifacts behind this technique, in the Linux Forensics reference: