How to Check if a Linux Server Is Compromised: the Order Matters More Than the Commands

There is a checklist for this on every sysadmin blog, and they all list roughly the same commands: ps, netstat, last, find / -mtime. The commands are fine. The problem is that they are usually presented in the wrong order, and the order is the part that decides whether the answer means anything.

If a machine has been rooted properly, the tools you are about to run are the tools the attacker had the most reason to tamper with. Running ps on a box with a kernel module hiding processes gives you a clean, reassuring, entirely false answer.

Step zero, and it is not a command

Decide first whether you are investigating or recovering. They pull in opposite directions.

Investigating means the machine stays exactly as it is: no reboot, no package updates, no cleanup. A reboot destroys the process table, the network state, and anything living only in memory, which on a modern intrusion can be most of it.

Recovering means the machine goes offline now and gets rebuilt from a known source. That is the right call more often than people admit, but it ends the investigation.

You cannot do both on the same box. Every hour spent poking at a live compromised host is an hour the attacker may still be using it.

If the machine matters, the honest sequence is: snapshot the disk, capture memory if you have the tooling, then take it off the network.

Look from outside the machine first

This is the step the checklists skip, and it is the only one an attacker on that host cannot manipulate.

Network flows from your router, hypervisor, or cloud provider. If the box is talking to an address it has no business talking to, or moving a volume of data it has never moved before, you learn that without asking the box anything. A compromised host lies about itself. Your switch does not.

The same applies to a disk snapshot. Mounting the filesystem read only from a different, trusted machine turns every subsequent check into a real one. A rootkit that hides files from ls on the running system hides nothing from another kernel reading the same blocks.

What to actually look at, in descending order of usefulness

Accounts and authentication. Read /etc/passwd and /etc/shadow for accounts you did not create, especially any with UID 0 other than root. Check ~/.ssh/authorized_keys for every user that has a home directory, root included. An added public key is the most common persistence mechanism there is, because it survives password changes and it looks like configuration.

Scheduled work. crontab -l for each user, plus /etc/cron.*, plus systemd timers via systemctl list-timers --all. Timers are checked far less often than cron and they do the same job.

Services that start themselves. systemctl list-unit-files --state=enabled and then look for unit files whose names you do not recognise, or whose ExecStart points somewhere writable like /tmp, /dev/shm, or a user home directory.

Listening sockets. ss -tulpn rather than the deprecated netstat. You are looking for a listener you cannot account for, and for a process name that does not match the port.

Package integrity. On Debian and Ubuntu, debsums -c lists files that no longer match what the package manager installed. On RHEL family systems, rpm -Va. This is genuinely useful and almost nobody runs it. It is also worth knowing its limit: it validates files that came from packages, so anything dropped outside a package is invisible to it.

Recently modified files. find / -xdev -mtime -7 -type f gives you the last week of changes. Expect noise from logs and caches. Look for changes in /bin, /sbin, /usr/bin, /lib, and for anything executable in /tmp or /dev/shm.

The log question, answered honestly

Everyone says read the logs. The useful version is narrower: read the logs you shipped somewhere else.

Logs on the compromised machine tell you what the attacker chose to leave. /var/log/auth.log or /var/log/secure may show the successful login that started this, or it may show a clean file with a suspicious gap. A gap is itself a finding. So is a wtmp that last reads without complaint but whose size does not match its history.

If you are running fail2ban or shipping logs to a separate host, that external copy is worth more than everything on the local disk.

The one conclusion you are allowed to draw

If you find a rootkit, you know the machine is compromised. If you find nothing, you have not learned that the machine is clean. You have learned that the checks you ran did not detect anything, which is a much smaller statement.

Tools like rkhunter and chkrootkit are worth running for the same reason and with the same caveat: they match known signatures, on a system that may be lying to them.

This is why the rebuild option keeps coming back. A machine you rebuild from source, with keys rotated and credentials replaced, is in a state you can reason about. A machine that passed a checklist is in a state you hope about.

What to do before you need this

Three things make the difference between a two hour answer and a two day one, and all three have to exist before the incident:

• Logs shipped off the machine, so there is a copy the attacker never touched.
• A file integrity baseline, from AIDE, Tripwire, or even a stored hash list. Comparing against a known good state is a different activity from guessing.
• Snapshots you can actually mount, which is what turns the trusted-machine method above from a good idea into a fifteen minute job.

If you are hardening a server rather than investigating one, the groundwork is covered in our Linux hardening guide, and process isolation is covered in Linux sandboxing.