Why would I need a firewall if my server is well configured?

I note that you've done a great job tying down several different daemons, and from what you've said I think it unlikely that you'll expose yourself to trouble through those services you have already secured. This still leaves you in a "everything is permitted except that which I have forbidden" state, and you can't get out of that state by hunting down daemon after daemon and securing them one by one.

A firewall configured to DENY ANY ANY by default moves you to a "everything is forbidden except that which is permitted" mode of operation, and I have found over many years that they're better.

Right now, given a legitimate user with a legitimate shell on your system, she could decide to run some local unprivileged daemon for proxying web requests for the internet, or start file sharing on port 4662, or accidentally open up a listener by using -g with ssh port tunneling, not understanding what it does; or a sendmail install could leave you running an MUA on port 587 which was improperly configured despite all the work you'd done on securing the MTA sendail on port 25; or a hundred and one things could happen that bypass your careful and thoughtful security simply because they weren't around when you were thinking carefully about what to forbid.

Do you see my point? At the moment, you've put a lot of effort into securing all the things you know about, and it sounds like they won't bite you. What may bite you is the things you don't know about, or that aren't even there, right now.

A firewall which defaults to DENY ANY ANY is the sysadmin way of saying that if something new comes along and opens up a network listener on this server, noone will be able to talk to it until I have given explicit permission.

Principle of Least Privilege. A firewall helps you get there. Principle of Defense in Depth. A firewall helps you get there, too. Any well-designed configuration explicitly relies on these two in one way or another.

Another thing is that your servers will most likely be commodity hardware, or hardware specific for handling server software running on top of a standard server OS (Unix, NT, Linux). That is, they do not have specialized hardware to handle and filter incoming traffic efficiently. Do you want your server to handle every single possible multicast, ICMP packet or port scan coming its way?

Most likely what you want is for your servers to physically handle requests to only some ports (80, 443, your ssl port, your typical oracle 1521 port, your rsync port, etc.) Yes, of course you set up software firewalls on your servers to listen to those ports only. But your NICs will still bear the brunt of unwanted traffic (be it malignant or normal in your organization.) If your NICs are getting hammered, so are the network paths going through your servers (and possibly between your servers and internals clients and connections to other internal servers and services.)

Not only do your NICs get hammered, your software firewall is also going to be engaged as it has to inspect every single packet or datagram it gets.

Firewalls on the other hand, specially those on the edges of subnets (or separating your subnets from the outside world) tend to be specialized hardware specifically built for handling that type of volume.

You can surround N number of servers with M number of firewalls (with N >> M). And you set your firewall hardware to dump anything that is not directed towards specific ports. Port scans, ICMPs and other crap are out. Then you fine-tune the software firewall in your servers according to their specific function.

Now you have just reduced (but not eliminated) the probability of a total blackout, reducing it to a partitioning of the network or partial failure at worst. And thus, you have increased your systems' ability to survive an attack or misconfiguration.

Not having a firewall because your servers have one is like feeling safe in having your seat belt on while driving at 120mph under zero visibility due to fog. It doesn't work that way.

There are many attacks you could be succeptible to if you do not have a firewall that does some kind of packet level inspection:

Example is the Christmas Tree Packet

http://en.wikipedia.org/wiki/Christmas_tree_packet

DDOS attacks could be run against your system, a firewall (external maybe, before any of your servers) would stop / slow / kill traffic before it crippled your servers.

Just because you do not have financial, or personal data on the servers doesn't mean you won't get 'hurt'. I'm sure you pay for bandwidth, or CPU usage, or you have a metered rate. Imagine over the course of a night (while you are sleeping) someone runs up your meter (I have seen this happen with VOIP Switch providers, hit in the night for MILLIONS OF MINUTES of traffic, that they have to foot the bill for).

So be smart, use the protection if it is there, you are NOT PERFECT, neither is software. It is only secure until the next exploit is found. ;)

You or someone else may make an error on your server setup one day, a firewall then gives you a 2nd chance of stopping someone getting in. We are not perfect, we make errors, and therefore a bit of "unneeded" insurance can be worthwhile.

(Try not to run your firewall on the same OS as your servers, as otherwise a single bug in the OS.... I consider all versions of Unix to be the same OS, as they have so much in common)

If you can enforce a principle of least privilege w/o using a firewall you probably do not need it. From my point of view building a secure system without using a firewall requires more effort, and I am quite lazy. Why should I bother restricting TCP connections using other tools and probably many config files when I can separate privileges on a transport level using a single config.

A firewall also can intercept unwanted packets from reaching your servers. Instead of dealing with them at the individual server level you can deal with them at the firewall. You can keep all of this configuration activity on the single firewall instead of multiple servers.

For example, if an attacker has gained control of an external IP and is deluging your servers with unwanted packets and you wish to mitigate the effects it has on your servers ... you could either configure each of your affected servers to drop the malicious packets or simply make the change at your firewall and all of your servers are protected. Having the firewall has decreased your reaction time.

Firewalls are spicialized in traffic manipulation. They do it quick and have resources. And you dont waste server resources to filter traffic (disk io / proc time / etc). You shuld configure some security in server environment but all trafic inspection and virus scanning and so on should do specialized servers.

I would be concerned that if you ever do get hacked and don't have a firewall in place. The hackers could open other ports on your servers. Also, if a consultant is brought in to do some cleanup and auditing, the first thing they will say is, "WHAT?!?! You don't have a firewall!" Then you could be burned.