Command Center

Howto up: Help, my server got hacked !!! PDF Print E-mail
Written by wunk
Sunday, 27 June 2004

It tends to happen on the most inconvenient moments.., friday afternoon at 4:30pm, saturdayevening before going out, sundaymorning when you have a hangover..., your server starts acting up..

Apache crashes for no reason, your mrtg/traffic graphs fly off the scale, server becomes dreadfully slow.., and slowly you start to realise.., dammit, I may have been hacked...., SOMEONE HELP ME...

Well..., let's try and do just that..., click below for the article..



First of all.., relax a bit, and don't panic, you'll make errors on your server when you're not thinking clearly and acting in a panicking way, which can result in even more damage then already has been done..

Basically hacked webservers, which tend to be hacked the most because they can generally have a lot of possible exploits in shitty programmed PHP scripts (In dutch sysadmin terms I personally call this PHPruts, which translates to PHPshit..), can be devided in two major categories:

* Roothacked (hacker got root privileges on the server)
* Not root hacked

Now if you're a victim of the first, you have all reason to panic.., although this can be blamed on your own account in 99,9% of the cases (always stay current with kernel and application updates, ESPECIALLY kernel updates if a major rootexploit got discovered), it's already too late and blaming yourself doesn't help.., if you want to be secure, the only remedy is to back up your data, format and reinstall the system and put the backup back.., you can try and clean it up as good as you can, but you can never really trust anything on the server anymore, there's no way in telling what the hacker did AFTER he god root.. (unless he was REALLY sloppy and left a .bash_history behind somewhere)

The most common hack is nevertheless the non-root hack.., these are mostly done through sloppy PHP scripts and exploits in PHP, though PHP is a beauty of a language to program in, it's still as leaky as a straw basket filled with water..

But how to detect which hack is on YOUR server ?

Detecting a root hacked server is actually the hardest of the two, when a hacker has gotten himself rootaccess, he'll do everything to cover his tracks and make his precense undetectable, it takes a keen eye to see something is wrong then, however if you have the 'feeling' that your server isn't acting completely at sense and a bit off, there's a good chance you may be right..

A good toolkit to start is chkrootkit, it can be found at:
http://www.chkrootkit.org/
Download the kit, untar it, and run it.., don't be alarmed if something pops up with a suspected rootkit, in my experience, chkrootkit will give false positives at times, especially bindshell or lkm processes (these tend to pop up when a hosting panel package is running, or due to MySQL or apache daemons being a bit off in ps compared to /proc )

If it doesn't give a specific INFECTED message, there's a good chance it's not that bad.., if it DOES give an INFECTED message, start googeling about what the rootkit is about, and start planning a reinstall, best part now would be to pull the network cable from the NIC so the hacker can't touch your server and do anything destructive if he's still on your server to try and cover his tracks.. (like an rm -rf / ), do NOT turn off the server yet, back up your data first, there's a good chance that if you shut down, it'll fail to boot due to incompatible hacked versions of libraries and/or binaries.. (if you did, it's time to get out a recover CD which lets you mount linux partitions)

Now, to see what's going on on the non roothacked server..
First do a ps aux > /root/pslist
This will give you a current processlist saved in a textfile so you can see what was running, since hackers generally get themselves in through apache and PHP, the processes will be running as the same user apache is running under (most likely the apache or nobody user), this gives an advantage in identifying the processes..

First, stop apache: /etc/init.d/httpd stop
Now let's see what's still running as the apache user: ps aux | grep apache | grep -v grep

Nothing SHOULD show up, if stuff DOES show up, it's fishy, since nothing should be running anymore as the apache user..

Sloppy hackers leave easy marks here, processes like ./httpd, ./ps, ./whatevername shouldn't be running, and are easily identifyable as shady processes.., note them down, and kill -9 the processnumbers

Now the smarter hackers manage to leave out the ./ part, which makes it a bit harder to identify them, especially when they're running as named, httpd, procps or some other normal daemon name.., once again, it should NOT be running as the same user as apache.., if it does, it's shady, and kill it..

do a find / -name processnameyoukilled to see where the applications reside that you just killed, 9 out of 10 chance that it's in /tmp or /var/tmp (exactly why you should have /tmp on a seperate and secured partition (see my Securing a redhat / Fedora install article), remove them where they pop up, and lock down that directory (chmod it to 000, and change ownership to root.root if possible), DO NOT TRY AND RUN THE APPLICATIONS, ESPECIALLY WHEN YOU'RE ROOT, you may be curious what they do, but if you love your data and server, you better not do it ;-)

Right, now that you nailed down those processes and removed the hazardous files, it's time for the most boring part of the job.., time to start digging logs.. (yech), you should be able to find in the apache logs which site and webpage caused it.., look in the /root/pslist file you made earlier to the time the processes started, this will narrow down the search a lot, since you can search a few minutes before and after that time in the apache access logs..

'What the HECK am I looking for then ?'

Good question.., big chance you are looking for a bad php script, count out .html files, those are too static to be exploitable.., a generic exploit used is through a command given to a php script, like: index.php?bad=stuff-goes.here, the ?cmd= variable is often used to inject bad code into PHP and open a loophole.., being handy with grep, awk, less and tools like that really helps here, since you can easily filter out exact times, extentions and columns, nice example:

cat access_log | grep "26/Jun/2004:16:23" | grep php | awk '{print $7}' | less

What this does, is cat the access_log, only show that exact date and time, with lines containing php, only column 7 from that, in a scrollable list..

When you find the flakey PHP script, I suggest either suspending the site or removing that script..

I hope these guidelines helped you somewhat :-)
Last Updated ( Tuesday, 07 September 2004 )






Reference: http://www.fedora-linux.org/content/view/26/33/