Re: [PLUG] Hacked server - recovery

[Russ Thompson <vigornj@gmail.com>]

Sorry to hear about your server Eric, there has been a lot of this going around.

Couple steps to help:

1 - Install rkhunter (as mentioned) and chkroot kit, both are comparable but signatures differer slightly.

2 - Ensure your tmp directories (mainly /tmp) are mounted securely with (noexec,nosuid,nodev).  Many of these exploits take advantage of insecure tmp directories.  Mounting securely prevents execution.

3 - Install CFG/LFD (we are mainly after LFD functions), LFD (login failure daemon) does a lot more than just that, it monitors files, resources, users etc and if they go over certain defined thresholds you get alerted.  It's very very verbose in the beginning, so be prepared to adjust settings accordingly.

4 - I'm not familiar with your application at all, so I apologize if this question isn't helpful.  Is your application built in PHP? If it's i.e RoR, perl, python etc.  You can disable php, in cases where an application is RoR and that's all the server is doing, I often link other languages binaries not in usage to custom script (which prevents access unless a key is provided, which unlocks).

5 - If you're using PHP, suphp or suhosin also take a look at mod_security.  I personally use suphp, it maintains user permissions restricting it to users permissions.  Mod_Security, also very popular, is an application layer firewall, it can be rather confusing at first because there are thousands of different ways to configure it, but is a very useful tool.

6 - If you're sending transactional e-mail (not spam etc), you may not want to deal with the hassle of dealing with mail.  Since transactional e-mail providers are very cheap (i.e Postmarkapp.com is $1.50 per 1000 e-mails) and they tend to do e-mail a whole lot better with good sender scores, statistics etc.

7 - Running Apache? Consider Litespeedtech.com, it's a drop in free replacement for apache with slightly better performance and more security features built in.  There is also nginx which I'm a huge fan of.

8 - Permissions, really get a tight hold of your permissions.  Create a user specifically for mail and only allow that user to relay/send mail.  Run your run application/apache etc as a whole other user, will take just a hair of scripting.

9 - I could probably go on all day, there are so many different things to consider :)

I'm going to say this was not a full hack or root exploit, most likely an injection or cross side scripting attack, very common among spammers and PHP.  Their intentions are what you saw, send spam.

-Russ

On Thu, Feb 23, 2012 at 6:59 PM, Eric at Lucii.org <eric@lucii.org> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

It's a Linode and, yes, we're making a snapshot as soon as we've confirmed the spam is stopped.
Then, application-level off site backups will allow us to recover in a snap.
That is, until we the the approval to replace the [expletive deleted] software!

Thanks!

On 02/23/2012 06:52 PM, Brett.Yeagley@bentley.com wrote:
> Is this a physical box or a VM? If a VM I would get a snapshot or create a template after you are "clean" again. This way you can get back to a clean system quickly in the future. Of course the question then is figuring out they got in originally because the weakness would obviously still be there.
> If it's a physical box, I'd highly recommend thinking about virtualizing so have the increased flexibility!
>
>

> *From*: Tom Haines [mailto:hainest@gmail.com]
> *Sent*: Thursday, February 23, 2012 11:42 PM
> *To*: Philadelphia Linux User's Group Discussion List <plug@lists.phillylinux.org>
> *Subject*: [PLUG] Hacked server - recovery

>
> You definitely want to run something like rkhunter to see what happened. My policy has always been to wipe the box and rebuild completely from backups. It's not worth risking that rkhunter missed something.
>
> On Thursday, February 23, 2012, Eric at Lucii.org wrote:
>

> I'm trying to recover an Ubuntu-based web server that was hacked.
> It was/is running a 2.x version of OpenRealty that contains more
> vulnerabilities than I could imagine.
>
> The hacker used it to send spam (no surprise.)  I was in a hurry
> so to stop it I just did apt-get remove postfix.  That worked in
> the same way that decapitation cures a headache.
>
> Now that I *believe* I've cleaned up the malicious code and I'd
> like to re-install and turn on postfix again.  Before I do, is
> there a way to either throttle the email output (our expected
> output is a couple of emails a day from the contact form) OR fire
> off an alarm if there are more than <arbitrary low number> emails
> being sent in a single hour?  Perhaps there is yet another
> alternative that I've not thought of?  (So far, I've thought of:
> not re-installing Postfix, replacing the web site code, and moving
> to Tibet.)  I don't have authorization to replace this code yet
> and my wife won't move to Tibet so that's out too... for now.
>
> Eric

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

> ___________________________________________________________________________
> Philadelphia Linux Users Group         --        http://www.phillylinux.org
> Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
> General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

- --
#  Eric Lucas
#
#                "Oh, I have slipped the surly bond of earth
#                 And danced the skies on laughter-silvered wings...
#                                        -- John Gillespie Magee Jr
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk9G0toACgkQ2sGpvXQrZ/7bPwCg7Qsrz8RcLJ4LTSMVgybT0HUp
1TwAoMop1PcVStKtPFodQAM2fc6viPz3
=+H46

-----END PGP SIGNATURE-----
___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug

___________________________________________________________________________
Philadelphia Linux Users Group         --        http://www.phillylinux.org
Announcements - http://lists.phillylinux.org/mailman/listinfo/plug-announce
General Discussion  --   http://lists.phillylinux.org/mailman/listinfo/plug