This is more of a theoretical question and request for advice and pointers than a question about specific software.
I have a distro-agnostic Linux machine (meaning I don't really care what to put in there atm), which I intend to place in a DC somewhere, where the location is not very secure (long story why), and where there is a risk of the server being physically removed and opened up for data extraction.
Is there a way to make life hard for the hackers even if they have the option of pulling the hard drives out, and basically doing whatever they want to them? The software in question is some locked down websites, user files, mysql db etc, all pretty standard stuff for a LAMP machine.
All that comes to mind is to encrypt the entire FS (no problem to add better spindles and more CPU muscle for that), but is there anything besides that?
As long as somebody might remove you disks, the only non-hazardous way to prevent data extraction is to do full disk encryption of the root file system (and any other file systems).
You can set it up so that if the server reboots, it will stop booting and wait for you to ssh into it to decrypt the root file system. There is just a small initrd image
There are a number of manual guides out there for this, but it has been included in debian (and others) by cryptsetup the last few years. Check the /usr/share/doc/cryptsetup/README.remote.gz file from the cryptsetup package for how to set it up on your server.
Nobody will be able to extract any useful data from your server even if they remove the disks physically to steal or copy them, and if you suffer from a power loss or any other reboot, this root fs encryption won't keep your remote server unavailable.
Keep in mind that the theory goes "if someone has access to the physical machine, they can own you. " but...
Have you looked at Truecrypt for whole disk encryption? I would also look at disabling as many port as possible on the physical machine (USB, etc)
You could make a bomb and put it inside the server, with a hidden switch. If that is not switched off and the lid is removed.. boom!
No, really, all jokes aside. Disk encryption would be the best idea that I can think of just like that. The built in luks encryption you can make shoud be sufficient, especially if you use a long and complex password.
Bear ind mind though, that this kind of encryption requires a physical access to the server, to boot it. The password needs to be entered in order to boot anything from the disk.
Now, that we're in the theoretical corner. Even though you have full disk encryption, it is possible to freeze down the memory and keep the keys required to access your data. (See this http://www.youtube.com/watch?v=JDaicPIgn9U)
So, in reality the best thing you can do is have good physical security.