Lessons from a Real Server Break-In: 8 Habits That Keep Hosting Infrastructure Safe
We recently investigated a break-in on a hosting company's management infrastructure. Nothing about it was exotic: no zero-day exploit, no nation-state tooling. It was a chain of ordinary habits, each one harmless on its own, that together gave an attacker quiet access for more than a week. Here is how it unfolded, what eventually exposed it, and the eight habits that would have stopped it at almost every step.
How the Break-In Unfolded
- A shared root password. Several servers, including a management server and a jump host, used the same root password, and password login over SSH was allowed.
- A quiet first login. The attacker logged in once with that password and left an SSH tunnel open. A tunnel opens no terminal, so it never appeared in
who. It stayed open for nine days. - A noisy second phase. More than a week later, several IPs logged in with the same password, added their own SSH keys, created systemd services and cron jobs, and dropped a web shell into the website.
- Hidden tracks. The attacker set file modification times back by a week so new files looked old, used a one-shot web shell that deleted itself, and left a hidden PHP file in
/tmp. - Discovery. It was found because the owner asked for a routine security check and the login log showed root password logins from IPs nobody recognised.
8 Habits That Break the Chain
1. Keys only, and no password reused anywhere
Turn off password login for SSH and give every person their own key. A leaked password is then useless, and one stolen credential never opens every machine. See setting up SSH key login.
2. Treat jump hosts and management servers as the crown jewels
A jump host or management server can reach everything else, which makes it the most valuable target. Give it its own credentials, the strictest firewall and the closest monitoring. If one is ever compromised, rebuild it rather than cleaning it.
3. Get an alert for every login
The tunnel ran for nine days because nobody was told about the login. A Telegram message for each SSH and panel login makes an unknown IP stand out at once. Setting this up takes about ten minutes: auditing logins with Telegram alerts.
4. Look for connections, not just sessions
Check ss -tnp for long-lived SSH connections from unknown IPs, not only who. Tunnels and SFTP sessions are invisible to the usual commands.
5. Two-factor authentication on every panel
Hosting panels, billing systems and server control panels hold the keys to everything. Turn on two-step verification and limit admin pages to known IPs where you can. For aaPanel, see the aaPanel security hardening guide.
6. Don't trust file dates
Modification times are easy to fake. When you hunt for planted files, use the change time and birth time, which a normal user-space process cannot set back:
find /var/www /tmp /dev/shm -newerct "-14 days" -type f 2>/dev/null
stat /path/to/suspicious.php # compare Modify with Change and Birth7. Close what does not need to be open
Database ports, panel ports and admin paths should not face the whole internet. Allow your own IPs, put websites behind a proxy that hides the origin, and keep the firewall's default policy as deny. See hiding your origin IP with Cloudflare.
8. After an incident, rotate everything and keep the evidence
Stop the active sessions, but copy logs and suspicious files somewhere safe before you delete anything. Then rotate every password, key and API token the compromised machines could see, including database and payment gateway credentials. A full cleanup checklist is in checking whether a Linux server is hacked.
What Good Infrastructure Gives You When It Goes Wrong
Containment depends on control you have before anything goes wrong. On IMIDC dedicated servers you can reach the server's screen through the VNC console even after you firewall everything else off, boot a rescue system to inspect the disks offline, and reinstall a server you no longer trust, all yourself from the client area (see the dedicated server control panel guide). Our 24/7 support team helps with log review and cleanup when you need a second pair of eyes.
A 10-Minute Self-Check
grep "Accepted password" /var/log/auth.log* | tail # any password logins?
ssh-keygen -lf /root/.ssh/authorized_keys # do you know every key?
ss -tnp | grep sshd # unknown long-lived connections?
crontab -l; ls /etc/cron.d /etc/systemd/system # anything you did not add?
find /tmp /var/tmp /dev/shm -name ".*" -type f # hidden files in temp dirs?If anything looks unfamiliar, stop and investigate before you change more. Open a ticket and the IMIDC team will help you work through it.