Should server's run their own firewalls (Windows Firewall, Linux IPTables), or is it simply not worth it? Is the answer different for 5 servers vs 50 vs 500? (Assuming the prior has servers on the same network as workstations, the larger have seperate server lans).
Does the OS matter?
When do you use/not use them.
"Should servers run their own firewalls?"
I love questions like that!
There are, of course, arguments for and against, but the only answer is what squillman alludes to...another question.
The needs of any one computer's firewall are directly related to the surrounding network topology and intended usage.
If you have a servers in a network otherwise composed of other servers, who's primary job it is to interact with the other servers on that network, the entirety of which is firewalled appropriately from the internet-at-large (and any other appropriate internal networks), then I say no. There's little need to impose the additional management overhead of implementing, documenting, and maintaining per-host firewalls. I would recommend a Network Intrusion Detection System (NIDS) so that you can monitor host behavior, and as always, make regular backups and take them offline (ie eject the tapes), but otherwise, it's just more stuff for you to remember to change as the needs of your servers are adjusted.
If you have a server that is isolated on the "external" side of the router, otherwise unprotected from the internet, or which requires placement that can't make use of a hardware-based firewall-type device, then yes. If, for whatever reason, your webserver is all by its lonesome outside of the cozy confines of the internal network, then yes, firewall that puppy, and do yourself a favor and make it deny-by-default. If, for whatever reason, that server has some sort of privileged access to internal servers via VPN, static route, or dual-homed network cards, make sure to firewall the outgoing connections too. There's always the chance that whatever-non-root-running daemon you've got can be compromised and initiate network connections attacking vulnerable, trusting internal resources. Stop it before it starts.
If you have a user desktop, firewall it. Make it part of your image, use a firewall software that supports Active Directory (or whatever centralized control you've got) administration, and specifically allow the services that you need. Your users will get malware. It's inevitable, and if sufficiently "mal", it will try to attack your other resources. Make sure you see the alarms when this happens.
I'm sure I missed some situations, but generally speaking, the security of a firewall is a trade-off in manageability and usability. Sometimes it's worth it, and sometimes it's not.
IMO, if you have more than a handful of servers, running firewalls on hosts is probably a bad idea, because you're setting yourself up to fail. Unless you have very robust change and configuration management, you're bound to run into misconfigurations and stale configurations.
You're also creating an internal controls issue. Do you want your sysadmins or application admins with root controlling network access? Sounds like trouble to me -- that role belongs with network admins or dedicated firewall/security admins.
I could see it as appropriate if you are running servers in a "hostile" environment. A server at a customer site, on a colo network, etc.
The answer, really, is that it depends on the needs of your environment. I personally don't use them on servers, but then I also have a lot of other things in place to protect things. If I didn't then I would use them. There simply is no hard and fast rule such as you're asking for.
If you give us some more information about your environment we could help with some advice on what might make sense for you.
Yes they should regardless of the number of servers. Plus if you use Windows servers - enable IPSec communications between servers.
Ideally you would also want to separate different server roles to different VLANs as well.
As others have effectively said, it depends upon the environment. In my opinion, it depends upon whether you can absolutely guarantee the security of the network the servers are attached to. If the network if appropriately firewalled from any other network, and you can guarantee that the network is secure (for example from having additional devices outside your control attached to it), then not having a software firewall is OK. If you do have a software firewall on the servers, you should make sure you document what you're doing, for your own sake.