I'm thinking of developing a kind of programming testing site where users could write PHP code and then it would be executed on the server to test the output. I know you want to always prevent users from executing code but is there a way that it could be setup so that the system isn't compromised?
Or does this just open up way too many security holes?
This is just a thought. I have not considered this any more than a few minutes. But one possible way of doing this would be to have your web application take in the code as a string and send it off to a Virtual Machine for processing.
That way you can keep your web application that provides the other functionality of your site from the arbitrary PHP execution of users.
The virtual machine could be set up so that the PHP scripts that are run have no access outside of the folder it is run in, no access to any part of the network except to receive requests from and return data to the host machine. You could even freeze the VM in a state so that any changes made to it would be undone on a restart.
Again, this is just a thought. There may be downsides that I am not seeing. One key thing to remember though is that you need to make sure that, at no point, is the PHP server on the host machine ever to execute the code being sent to or returned from the VM.
some things you must do to increase security:
you should use iptables to restrict your testing-user from performing any unwanted network actions:
iptables -t mangle -A OUTPUT -p tcp -m owner --uid-owner 1001 -j REJECT
you must update your system software regurarly! :-)
at the end, consider security like an onion. do the best on any layer to secure your system. in case of a security breach on one layer, underlying layers should block attackers...
I think it's doable, but you've certainly got your work cut out for you if you want to do it in any kind of secure fashion. This is essentially what any web host who offers PHP is doing for anyone willing to throw a bit of money at them.
What your difference would be is that it'd be open to just anyone in general, unless you do plan on restricting access to authenticated users.
You'll want to read up on hardening your system all the way from PHP / apache down to the OS. There are a number of things that a hardened system will prevent, though, which will limit the usefulness of the testing.
If you do pursue this I would highly recommend a human validation / anti-bot mechanism like Captcha / reCaptcha on your code page.
If you're not well versed on system security then I wouldn't touch it, personally.