Questions[hacking](server)

I run a small internet based business from home and make a living at it to feed my family, but I'm still a one man show and internet security is far from my area of expertise.

Yesterday I received two emails from a guy who calls himself an "ethical hacker" and has identified two vulnerabilities in my system which he says could be exploited by hackers. I believe him.

The problem is, at the bottom of each email he says he "expects a bounty to be paid". Is this black mail? Is this his way of saying you'd better pay me or I'm going to wreak havoc? Or is this a typical and legitimate method for people to make a living without any nefarious intentions?

EDIT: For more clarification: He gave me two examples of vulnerabilities with screenshots and clear instructions on how to fix those vulnerabilities. One was to change the "?all" part of my SPF record to "-all" to block all other domains from sending emails for my domain. In the other email he explained how my site was able to be shown inside an iframe (enabling a technique called "clickjacking") and he also included an example of the code and instructions on how to prevent it.

This is a Canonical Question about Server Security - Responding to Breach Events (Hacking)
See Also:

• Tips for Securing a LAMP Server
• Reinstall after a Root Compromise?

Canonical Version
I suspect that one or more of my servers is compromised by a hacker, virus, or other mechanism:

• What are my first steps? When I arrive on site should I disconnect the server, preserve "evidence", are there other initial considerations?
• How do I go about getting services back online?
• How do I prevent the same thing from happening immediately again?
• Are there best practices or methodologies for learning from this incident?
• If I wanted to put a Incident Response Plan together, where would I start? Should this be part of my Disaster Recovery or Business Continuity Planning?

Original Version

2011.01.02 - I'm on my way into work at 9.30 p.m. on a Sunday because our server has been compromised somehow and was resulting in a DOS attack on our provider. The servers access to the Internet has been shut down which means over 5-600 of our clients sites are now down. Now this could be an FTP hack, or some weakness in code somewhere. I'm not sure till I get there.

How can I track this down quickly? We're in for a whole lot of litigation if I don't get the server back up ASAP. Any help is appreciated. We are running Open SUSE 11.0.

---

2011.01.03 - Thanks to everyone for your help. Luckily I WASN'T the only person responsible for this server, just the nearest. We managed to resolve this problem, although it may not apply to many others in a different situation. I'll detail what we did.

We unplugged the server from the net. It was performing (attempting to perform) a Denial Of Service attack on another server in Indonesia, and the guilty party was also based there.

We firstly tried to identify where on the server this was coming from, considering we have over 500 sites on the server, we expected to be moonlighting for some time. However, with SSH access still, we ran a command to find all files edited or created in the time the attacks started. Luckily, the offending file was created over the winter holidays which meant that not many other files were created on the server at that time.

We were then able to identify the offending file which was inside the uploaded images folder within a ZenCart website.

After a short cigarette break we concluded that, due to the files location, it must have been uploaded via a file upload facility that was inadequetly secured. After some googling, we found that there was a security vulnerability that allowed files to be uploaded, within the ZenCart admin panel, for a picture for a record company. (The section that it never really even used), posting this form just uploaded any file, it did not check the extension of the file, and didn't even check to see if the user was logged in.

This meant that any files could be uploaded, including a PHP file for the attack. We secured the vulnerability with ZenCart on the infected site, and removed the offending files.

The job was done, and I was home for 2 a.m.

---

The Moral - Always apply security patches for ZenCart, or any other CMS system for that matter. As when security updates are released, the whole world is made aware of the vulnerability. - Always do backups, and backup your backups. - Employ or arrange for someone that will be there in times like these. To prevent anyone from relying on a panicy post on Server Fault.

Someone has, for the second time, appended a chunk of javascript to a site I help run. This javascript hijacks Google adsense, inserting their own account number, and sticking ads all over.

The code is always appended, always in one specific directory (one used by a third party ad program), affects a number of files in a number of directories inside this one ad dir (20 or so) and is inserted at roughly the same overnight time. The adsense account belongs to a Chinese website (located in a town not an hour from where I will be in China next month. Maybe I should go bust heads... kidding, sort of), btw... here is the info on the site: http://serversiders.com/fhr.com.cn

So, how could they append text to these files? Is it related to the permissions set on the files (ranging from 755 to 644)? To the webserver user (it's on MediaTemple so it should be secure, yes?)? I mean, if you have a file that has permissions set to 777 I still can't just add code to it at will... how might they be doing this?

Here is a sample of the actual code for your viewing pleasure (and as you can see... not much to it. The real trick is how they got it in there):

<script type="text/javascript"><!--
google_ad_client = "pub-5465156513898836";
/* 728x90_as */
google_ad_slot = "4840387765";
google_ad_width = 728;
google_ad_height = 90;
//-->
</script>
<script type="text/javascript"
src="http://pagead2.googlesyndication.com/pagead/show_ads.js">
</script>

Since a number of folks have mentioned it, here is what I have checked (and by checked I mean I looked around the time the files were modified for any weirdness and I grepped the files for POST statements and directory traversals:

• access_log (nothing around the time except normal (i.e. excessive) msn bot traffic)
• error_log (nothing but the usual file does not exist errors for innocuous looking files)
• ssl_log (nothing but the usual)
• messages_log (no FTP access in here except for me)

*UPDATE:** OK, solved it. Hackers from China had physically placed a file in our site that allows them to do all manner of administrative things (database access, delete and create files and dirs, you name it, they had access). We were lucky they didn't do something more destructive. There was nothing in the normal apache log files but I found a different set of log files in a web server log analyzer and the evidence was in there. They were accessing this file with their own admin username and password and then editing whatever they needed right there on the server. Their file has "apache" set as the user while all other files on our site have a different user name. Now I need to figure out how they physically got this file onto our system. I suspect blame for this will eventually rest with our web host (Media Temple), unless they actually had our FTP login... not sure how I will determine that, however, as this file has probably been there for a while.

After reading this question on a server compromise, I started to wonder why people continue to seem to believe that they can recover a compromised system using detection/cleanup tools, or by just fixing the hole that was used to compromise the system.

Given all the various root kit technologies and other things a hacker can do most experts suggest you should reinstall the operating system.

I am hoping to get a better idea why more people don't just take off and nuke the system from orbit.

Here are a couple points, that I would like to see addressed.

• Are there conditions where a format/reinstall would not clean the system?
• Under what types conditions do you think a system can be cleaned, and when must you do a full reinstall?
• What reasoning do you have against doing a full reinstall?
• If you choose not to reinstall, then what method do you use to be reasonably confident you have cleaned and prevented any further damage from happening again.

What are the tell-tale signs that a Linux server has been hacked? Are there any tools that can generate and email an audit report on a scheduled basis?