This is a Canonical Question about understanding and remediating the Heartbleed security issue.

What exactly is CVE-2014-0160 AKA "Heartbleed"? What is the cause, what OSs and versions of OpenSSL are vulnerable, what are the symptoms, are there any methods to detect a successful exploit?

How can I check to see if my system is affected? How can this vulnerability be mitigated? Should I be concerned that my keys or other private data have been compromised? What other side effects should I be concerned about?

This is a Canonical Question about File Permissions on a Linux web server.

I have a Linux web server running Apache2 that hosts several websites. Each website has its own folder in /var/www/.

/var/www/contoso.com/
/var/www/contoso.net/
/var/www/fabrikam.com/

The base directory /var/www/ is owned by root:root. Apache is running as www-data:www-data. The Fabrikam website is maintained by two developers, Alice and Bob. Both Contoso websites are maintained by one developer, Eve. All websites allow users to upload images. If a website is compromised, the impact should be as limited as possible.

I want to know the best way to set up permissions so that Apache can serve the content, the website is secure from attacks, and the developers can still make changes. One of the websites is structured like this:

/var/www/fabrikam.com
    /cache
    /modules
    /styles
    /uploads
    /index.php

How should the permissions be set on these directories and files? I read somewhere that you should never use 777 permissions on a website, but I don't understand what problems that could cause. During busy periods, the website automatically caches some pages and stores the results in the cache folder. All of the content submitted by website visitors is saved to the uploads folder.

A security auditor for our servers has demanded the following within two weeks:

• A list of current usernames and plain-text passwords for all user accounts on all servers
• A list of all password changes for the past six months, again in plain-text
• A list of "every file added to the server from remote devices" in the past six months
• The public and private keys of any SSH keys
• An email sent to him every time a user changes their password, containing the plain text password

We're running Red Hat Linux 5/6 and CentOS 5 boxes with LDAP authentication.

As far as I'm aware, everything on that list is either impossible or incredibly difficult to get, but if I don't provide this information we face losing access to our payments platform and losing income during a transition period as we move to a new service. Any suggestions for how I can solve or fake this information?

The only way I can think to get all the plain text passwords, is to get everyone to reset their password and make a note of what they set it to. That doesn't solve the problem of the past six months of password changes, because I can't retroactively log that sort of stuff, the same goes for logging all the remote files.

Getting all of the public and private SSH keys is possible (though annoying), since we have just a few users and computers. Unless I've missed an easier way to do this?

I have explained to him many times that the things he's asking for are impossible. In response to my concerns, he responded with the following email:

I have over 10 years experience in security auditing and a full understanding of the redhat security methods, so I suggest you check your facts about what is and isn't possible. You say no company could possibly have this information but I have performed hundreds of audits where this information has been readily available. All [generic credit card processing provider] clients are required to conform with our new security policies and this audit is intended to ensure those policies have been implemented* correctly.

*The "new security policies" were introduced two weeks before our audit, and the six months historical logging was not required before the policy changes.

In short, I need;

• A way to "fake" six months worth of password changes and make it look valid
• A way to "fake" six months of inbound file transfers
• An easy way to collect all the SSH public and private keys being used

If we fail the security audit we lose access to our card processing platform (a critical part of our system) and it would take a good two weeks to move somewhere else. How screwed am I?

Update 1 (Sat 23rd)

Thanks for all your responses, It gives me great relief to know this isn't standard practice.

I'm currently planning out my email response to him explaining the situation. As many of you pointed out, we have to comply with PCI which explicitly states we shouldn't have any way to access plain-text passwords. I'll post the email when I've finished writing it. Unfortunately I don't think he's just testing us; these things are in the company's official security policy now. I have, however, set the wheels in motion to move away from them and onto PayPal for the time being.

Update 2 (Sat 23rd)

This is the email I've drafted out, any suggestions for stuff to add/remove/change?

Hi [name],

Unfortunately there is no way for us to provide you with some of the information requested, mainly plain-text passwords, password history, SSH keys and remote file logs. Not only are these things technically impossible, but also being able to provide this information would be both a against PCI Standards, and a breach of the data protection act.
To quote the PCI requirements,

8.4 Render all passwords unreadable during transmission and storage on all system components using strong cryptography.

I can provide you with a list of usernames and hashed passwords used on our system, copies of the SSH public keys and authorized hosts file (This will give you enough information to determine the number of unique users can connect to our servers, and the encryption methods used), information about our password security requirements and our LDAP server but this information may not be taken off site. I strongly suggest you review your audit requirements as there is currently no way for us to pass this audit while remaining in compliance of PCI and the Data Protection act.

Regards,
[me]

I will be CC'ing in the company's CTO and our account manager, and I'm hoping the CTO can confirm this information is not available. I will also be contacting the PCI Security Standards Council to explain what he's requiring from us.

Update 3 (26th)

Here are some emails we exchanged;

RE: my first email;

As explained, this information should be easily available on any well maintained system to any competent administrator. Your failure to be able to provide this information leads me to believe you are aware of security flaws in your system and are not prepared to reveal them. Our requests line up with the PCI guidelines and both can be met. Strong cryptography only means the passwords must be encrypted while the user is inputting them but then they should be moved to a recoverable format for later use.

I see no data protection issues for these requests, data protection only applies to consumers not businesses so there should be no issues with this information.

Just, what, I, can't, even...

"Strong cryptography only means the passwords must be encrypted while the user is inputting them but then they should be moved to a recoverable format for later use."

I'm going to frame that and put it on my wall.

I got fed up being diplomatic and directed him to this thread to show him the response I got:

Providing this information DIRECTLY contradicts several requirements of the PCI guidelines. The section I quoted even says storage (Implying to where we store the data on the disk). I started a discussion on ServerFault.com (An on-line community for sys-admin professionals) which has created a huge response, all suggesting this information cannot be provided. Feel free to read through yourself

https://serverfault.com/questions/293217/

We have finished moving over our system to a new platform and will be cancelling our account with you within the next day or so but I want you to realize how ridiculous these requests are, and no company correctly implementing the PCI guidelines will, or should, be able to provide this information. I strongly suggest you re-think your security requirements as none of your customers should be able to conform to this.

(I'd actually forgotten I'd called him an idiot in the title, but as mentioned we'd already moved away from their platform so no real loss.)

And in his response, he states that apparently none of you know what you're talking about:

I read in detail through those responses and your original post, the responders all need to get their facts right. I have been in this industry longer than anyone on that site, getting a list of user account passwords is incredibly basic, it should be one of the first things you do when learning how to secure your system and is essential to the operation of any secure server. If you genuinely lack the skills to do something this simple I'm going to assume you do not have PCI installed on your servers as being able to recover this information is a basic requirement of the software. When dealing with something such as security you should not be asking these questions on a public forum if you have no basic knowledge of how it works.

I would also like to suggest that any attempt to reveal me, or [company name] will be considered libel and appropriate legal action will be taken

Key idiotic points if you missed them:

• He's been a security auditor longer than anyone else on here has (He's either guessing, or stalking you)
• Being able to get a list of passwords on a UNIX system is 'basic'
• PCI is now software
• People shouldn't use forums when they're not sure of security
• Posing factual information (to which I have email proof) online is libel

Excellent.

PCI SSC have responded and are investigating him and the company. Our software has now moved onto PayPal so we know it's safe. I'm going to wait for PCI to get back to me first but I'm getting a little worried that they might have been using these security practices internally. If so, I think it is a major concern for us as all our card processing ran through them. If they were doing this internally I think the only responsible thing to do would be to inform our customers.

I'm hoping when PCI realize how bad it is they will investigate the entire company and system but I'm not sure.

So now we've moved away from their platform, and assuming it will be at least a few days before PCI get back to me, any inventive suggestions for how to troll him a bit? =)

Once I've got clearance from my legal guy (I highly doubt any of this is actually libel but I wanted to double check) I'll publish the company name, his name and email, and if you wish you can contact him and explain why you don't understand the basics of Linux security like how to get a list of all the LDAP users passwords.

Little update:

My "legal guy" has suggested revealing the company would probably cause more problems than needed. I can say though, this is not a major provider, they have less 100 clients using this service. We originally started using them when the site was tiny and running on a little VPS, and we didn't want to go through all the effort of getting PCI (We used to redirect to their frontend, like PayPal Standard). But when we moved to directly processing cards (including getting PCI, and common sense), the devs decided to keep using the same company just a different API. The company is based in the Birmingham, UK area so I'd highly doubt anyone here will be affected.

This is a Canonical Question about Server Security - Responding to Breach Events (Hacking)
See Also:

• Tips for Securing a LAMP Server
• Reinstall after a Root Compromise?

Canonical Version
I suspect that one or more of my servers is compromised by a hacker, virus, or other mechanism:

• What are my first steps? When I arrive on site should I disconnect the server, preserve "evidence", are there other initial considerations?
• How do I go about getting services back online?
• How do I prevent the same thing from happening immediately again?
• Are there best practices or methodologies for learning from this incident?
• If I wanted to put a Incident Response Plan together, where would I start? Should this be part of my Disaster Recovery or Business Continuity Planning?

Original Version

2011.01.02 - I'm on my way into work at 9.30 p.m. on a Sunday because our server has been compromised somehow and was resulting in a DOS attack on our provider. The servers access to the Internet has been shut down which means over 5-600 of our clients sites are now down. Now this could be an FTP hack, or some weakness in code somewhere. I'm not sure till I get there.

How can I track this down quickly? We're in for a whole lot of litigation if I don't get the server back up ASAP. Any help is appreciated. We are running Open SUSE 11.0.

---

2011.01.03 - Thanks to everyone for your help. Luckily I WASN'T the only person responsible for this server, just the nearest. We managed to resolve this problem, although it may not apply to many others in a different situation. I'll detail what we did.

We unplugged the server from the net. It was performing (attempting to perform) a Denial Of Service attack on another server in Indonesia, and the guilty party was also based there.

We firstly tried to identify where on the server this was coming from, considering we have over 500 sites on the server, we expected to be moonlighting for some time. However, with SSH access still, we ran a command to find all files edited or created in the time the attacks started. Luckily, the offending file was created over the winter holidays which meant that not many other files were created on the server at that time.

We were then able to identify the offending file which was inside the uploaded images folder within a ZenCart website.

After a short cigarette break we concluded that, due to the files location, it must have been uploaded via a file upload facility that was inadequetly secured. After some googling, we found that there was a security vulnerability that allowed files to be uploaded, within the ZenCart admin panel, for a picture for a record company. (The section that it never really even used), posting this form just uploaded any file, it did not check the extension of the file, and didn't even check to see if the user was logged in.

This meant that any files could be uploaded, including a PHP file for the attack. We secured the vulnerability with ZenCart on the infected site, and removed the offending files.

The job was done, and I was home for 2 a.m.

---

The Moral - Always apply security patches for ZenCart, or any other CMS system for that matter. As when security updates are released, the whole world is made aware of the vulnerability. - Always do backups, and backup your backups. - Employ or arrange for someone that will be there in times like these. To prevent anyone from relying on a panicy post on Server Fault.

We all know it happens. A bitter old IT guy leaves a backdoor into the system and network in order to have fun with the new guys and show the company how bad things are without him.

I've never personally experienced this. The most I've experienced is somebody who broke and stole stuff right before leaving. I'm sure this happens, though.

So, when taking over a network that can't quite be trusted, what steps should be taken to ensure everything is safe and secure?