Advantages of firewall:

1. You can filter outbound traffic.
2. Layer 7 firewalls (IPS) can protect against known application vulnerabilities.
3. You can block a certain IP address range and/or port centrally rather than trying to ensure that there is no service listening on that port on each individual machine or denying access using TCP Wrappers.
4. Firewalls can help if you have to deal with less security aware users/administrators as they would provide second line of defence. Without them one has to be absolutely sure that hosts are secure, which requires good security understanding from all administrators.
5. Firewall logs would provide central logs and help in detecting vertical scans. Firewall logs can help in determining whether some user/client is trying to connect to same port of all your servers periodically. To do this without a firewall one would have to combine logs from various servers/hosts to get a centralized view.
6. Firewalls also come with anti-spam / anti-virus modules which also add to protection.
7. OS independent security. Based on host OS, different techniques / methods are required to make the host secure. For example, TCP Wrappers may not be available on Windows machines.

Above all this if you do not have firewall and system is compromised then how would you detect it? Trying to run some command 'ps', 'netstat', etc. on local system can't be trusted as those binaries can be replaced. 'nmap' from a remote system is not guaranteed protection as an attacker can ensure that root-kit accepts connections only from selected source IP address(es) at selected times.

Hardware firewalls help in such scenarios as it is extremely difficult to change firewall OS/files as compared to host OS/files.

Disadvantages of firewall:

1. People feel that firewall will take care of security and do not update systems regularly and stop unwanted services.
2. They cost. Sometimes yearly license fee needs to be paid. Especially if the firewall has anti-virus and anti-spam modules.
3. Additional single point of failure. If all traffic passes through a firewall and the firewall fails then network would stop. We can have redundant firewalls, but then previous point on cost gets further amplified.
4. Stateful tracking provides no value on public-facing systems that accept all incoming connections.
5. Stateful firewalls are a massive bottleneck during a DDoS attack and are often the first thing to fail, because they attempt to hold state and inspect all incoming connections.
6. Firewalls cannot see inside encrypted traffic. Since all traffic should be encrypted end-to-end, most firewalls add little value in front of public servers. Some next-generation firewalls can be given private keys to terminate TLS and see inside the traffic, however this increases the firewall's susceptibility to DDoS even more, and breaks the end-to-end security model of TLS.
7. Operating systems and applications are patched against vulnerabilities much more quickly than firewalls. Firewall vendors often sit on known issues for years without patching, and patching a firewall cluster typically requires downtime for many services and outbound connections.
8. Firewalls are far from perfect, and many are notoriously buggy. Firewalls are just software running on some form of operating system, perhaps with an extra ASIC or FPGA in addition to a (usually slow) CPU. Firewalls have bugs, but they seem to provide few tools to address them. Therefore firewalls add complexity and an additional source of hard-to-diagnose errors to an application stack.

TCP Wrappers could be arguably called a host-based firewall implementation; you're filtering network traffic.

For the point on an attacker making outbound connections on an arbitrary port, a firewall would provide a means of controlling outgoing traffic as well; a properly configured firewall manages ingress and egress in a way which is appropriate to the risk related to the system.

On the point about how any TCP vulnerability isn't mitigated by a firewall, you're nt familiar with how firewalls work. Cisco has a whole bunch of rules available for download which identify packets constructed in a way that would cause particular operating systems problems. If you grab Snort and start running it with the right rule set, you will also get alerted on this kind of thing. And of course, Linux iptables can filter out malicious packets.

Basically, a firewall is proactive protection. The farther you get away from being proactive, the most likely that you'll find yourself in a situation where you're reacting to a problem rather than preventing the problem. Concentrating your protection at the border, as with a dedicated firewall, makes things easier to manage because you have a central choke point rather than duplicating rules everywhere.

But no single thing is necessarily a final solution. A good security solution generally is multi-layer, where you have a firewall at the border, TCP wrappers at the device, and probably some rules on internal routers as well. You should usually protect the network from the Internet, and protect the nodes from each other. This multi-layer approach isn't like drilling a hole through multiple sheets of plywood, it's more like putting up a pair of doors so an intruder has two locks to break instead of just one; this is called a man trap in physical security, and most every building has one for a reason. :)

(You may want to read "Life without Firewalls")

Now: What about having a legacy system for which no patches get published anymore? What about not being able to apply the patches to N-machines at the time you need to do so, while at the same time you can apply them in fewer nodes in the network (firewalls)?

There's no point in debating the firewall's existence or need. What really matters is that you have to implement a security policy. To do so you will use whatever tools will implement it and help you manage, expand and evolve it. If firewalls are needed to do so, that's fine. If they are not needed that's fine too. What really matters is having a working and verifiable implementation of your security policy.

Most of your explanations seem to refute a need for a firewall, but I don't see a con to having one, other than the small amount of time to set one up.

Few things are a "necessity" in a strict meaning of the word. Security is more about setting up all the blockades you can. The more work needed to break into your server means less chance of successful attack. You want to make it more work to break into your machines than somewhere else. Adding a firewall makes more work.

I think a key use is redundancy in security. Another plus of firewalls is you can simply drop attempts to connect to any port rather than responding to rejected requests - this will make nmapping a little more inconvenient for an attacker.

Most important to me on the practical note of your question is you can lock SSH, ICMP, and other internal services down to local subnets as well as rate limit incoming connections to help alleviate DOS attacks.

"The point of security isn't to defend yourself after a successful attack - that's already proven to be impossible - it's to keep the attackers out in the first place."

I disagree. Limiting damages can be just as important. (under this ideal why hash passwords? or stick your database software on a different server than your web applications?) I think the old saying "Don't stick all of your eggs in one basket" is applicable here.

Should I firewall my server? Good question. It would seem that there is little point to slapping a firewall on top of a network stack that already rejects connection attempts to all but the handful of ports that are legitimately open. If there is a vulnerability in the OS that allows maliciously crafted packets to disrupt/exploit a host, would a firewall running on that same host prevent the exploit? Well, maybe ...

And that is probably the strongest reason to run a firewall on every host: A firewall might prevent a network stack vulnerability from being exploited. Is that a strong enough reason? I don't know, but I suppose one could say, "No one ever got fired for installing a firewall."

Another reason to run a firewall on a server is to decouple these two otherwise strongly correlated concerns:

1. From where, and to what ports, do I accept connections on?
2. Which services are running and listening for connections?

Without a firewall, the set of services running (along with the configurations for tcpwrappers and such) completely determines the set of ports the server will have open, and from whom connections will be accepted. A host-based firewall gives the admin additional flexibility to install and test new services in a controlled way before making them more widely available. If such flexibility is not required, then there is less reason to install a firewall on a server.

On a final note, there is one item not mentioned in your security checklist that I always add, and that is a host-based intrusion detection system (HIDS), such as AIDE or samhain. A good HIDS makes it extremely difficult for an intruder to make unwanted changes to the system and remain undetected. I believe all servers should be running some kind of HIDS.

A firewall is a tool. It doesn't make things secure in and of itself, but it can make a contribution as a layer in a secure network. That doesn't mean you need one, and I certainly worry about people who blindly say "I've got to get a firewall" without understanding why they think that way and who don't understand the strengths and weaknesses of firewalls.

There are a lot of tools we can say we don't need... Is it possible to run a Windows computer without Antivirus? Yes it is... but it's a nice layer of insurance to have one.

I'd say the same about firewalls - whatever else you can say about them, they are a nice level of insurance. They are not a substitute for patching, for locking machines down, for disabling services you don't use, for logging, etc... but they can be a useful supplement.

I'd also suggest that the equation changes somewhat depending on whether or not you're talking about placing a firewall in front of a group of carefully tended servers, as you seem to be, or a typical LAN with a mix of workstations and servers, some of which might be running some pretty hairy stuff despite the best efforts and wishes of the IT team.

One more thing to consider is the the benefit of creating an obviously hardened target. Visible security, whether it's bright lights, heavy locks and an obvious alarm box on a building; or an obvious firewall on a business' range of IP addresses can deter the casual intruder - they'll go looking for easier prey. This won't deter the determined intruder who knows you have information they want and is determined to get it, but deterring the casual intruders is still worthwhile - especially if you know that any intruder whose probes persists past the deterrent needs to be taken especially seriously.

All great questions. BUT - I'm very surprised PERFORMANCE is not been brought to the table.

For highly (CPU-wise) used Web front ends, local firewall really degrades performance, period. Try a load test and see. I saw this tons of times. Turning off firewall increased performance (request per-sec) by 70% or more.

This trade-off must be considered.

A firewall is additional protection. Three particular scenarios it protects against are network stack attacks (i.e. your server OS has a vulnerability to on specially crafted packets that never reach the level of ports), a successful intrusion making a connection to "phone home" (or send spam, or whatever), or a successful intrusion opening a server port or, less detectable, watch for a port knocking sequence before opening a port. Granted, the last two have to do with mitigating the damage of an intrusion rather than preventing it, but that doesn't mean it's useless. Remember that security is not an all-or-nothing proposition; one takes a layered approach, belt and suspenders, to achieve a level of security that is sufficient for your needs.

I'm no security expert by any means, but it sounds as though you are firewall'ed. It seems as though you've taken some of the core functionality of a firewall and made it part of your policies and procedures. No, you don't need a firewall if you are going to do the same job as a firewall yourself. As for myself, I'd rather do the best I can in keeping up security, but have a firewall looking over my shoulder, but at some point when you can do everything the firewall is doing, it starts to become irrelevant.

In general, security is an onion thing, as already mentioned. There are reasons firewalls exist, and it's not just all the other lemmings being stupid morons.

This answer comes, as searching for 'fail2ban' on this page did not give me any results. So if I double other content, bear with me. And excuse me, if I rant a little, I provide plain experience as this might come in handy for others. :)

Networking considerations, local vs. external

This is rather Linux specific and concentrates on host based firewalling, which is usually the use case. External firewalling goes hand in hand with a proper network structure and other security considerations usually go into that. Either you know what is implied here, then you will likely not need this posting. Or you don't, then just read on.

Running firewalls externally and locally may seem counter-intuitive and double work. But this also gives the possibility turn of the rules on the external one, without compromising security on all other hosts being behind it. The need could arise from either debugging reasons or because somebody has just fucked up. Another use case will come down there in the 'adaptive global firewalling' section, for which you will also need both global and local firewalls.

Cost and availability and the same story all the time:

Firewalling is just one aspect of a proper secured system. Not installing a firewall as it 'costs' money, introduces a SPOF or whatever is just bullshit, pardon my French here. Just setup a cluster. Oh, but what if the fire cell has an outage? Then setup your cluster on spanning two or more fire compartments.

But what if the whole datacenter is unreachable, as both of the external carriers are out of business (excavator killed your fiber)? Just make your cluster spanning several datacenters, then.

That's expensive? Clusters are too complex? Well, paranoia has to be paid for.

Just whining about SPOFs, but not wanting to pay more money or creating little more complex setups is a clear case of double standards or just a small wallet on company or customer side.

This pattern applies to ALL these discussions, no matter which service is the current matter of the day. No matter if it's a VPN gateway, a Cisco ASA used just for firewalling, a MySQL or PostgreSQL database, a virtual system or server hardware, a storage backend, switches/routers, ...

By now you should get the idea.

Why bother with firewalling?

In theory your reasoning is sound. (Only running services can be exploited.)

But this is only half the truth. Firewalling, especially stateful firewalling can do much more for you. Stateless firewalls are only important if you experience performance issues like others already mentioned.

Easy, central, discrete access control

You mentioned TCP wrappers which implement basically the same functionality for securing your. For the sake of the argument, let's assume someone doesn't know of tcpd and likes using a mouse? fwbuilder might come into mind.

Having full access from your management network is something you should have enabled, which is something of the first use cases of your host based firewall.

How about a multi-server setup, where the database runs somewhere else and you cannot put both/all the machines within a shared (private) subnet for whatever reason? Use the firewall to allow MySQL access on port 3306 only for the single given IP address of the other server, done, simple.

And that also works flawlessly for UDP. Or whatever protocol. Firewalls can be so damn flexible. ;)

Portscan mitigation

Further, with firewalling, general portscans can be detected and mitigated as the amount of connections per timespan can be monitored via the kernel and its networking stack, and the firewall can act upon that.

Also invalid or obscure packets can be handled prior to them ever reaching your application.

Outbound traffic limiting

Filtering outbound traffic is usually a pain in the ass. But it can be a must, depending on contract.

Statistics

Another thing that a firewall can give you, is statistics. (Think watch -n1 -d iptables -vnxL INPUT with having added some rules for special IP addresses right at the top to see if packets are coming through.)

You can see in plain daylight if things work or they do not. Which is VERY useful when troubleshooting connections and being able to tell the other person on the telephone you do not get packets, without having to resort to chatty tcpdump's. Networking is fun, most people just do now know what they are doing, and all to often it's just simple routing errors. Hell, even I don't always know what I am doing. Although I have worked with literally dozens of complex systems and appliances, often with tunneling, too, by now.

IDS/IPS

Layer7 firewalling is usually snake-oil (IPS/IDS), if not attended properly and updated regularly. Plus the licenses are damn expensive, so I'd spare getting one if you have no real need for getting everything money can buy you.

Masqerading

Easy, just try this with your wrappers. :D

Local port forwarding

See masquerading.

Securing password access channels with dynamic IP addresses

What about if customers have dynamic IP addresses and there is not a VPN setup deployed? Or other dynamic approaches to firewalling? This is already hinted at in the question, and here will come a use case for the Unfortunately, I can't find any firewalls that do that. part.

Having disabled the root account access via a password is a must. Even if access is limited to certain IP addresses.

Also, still having another blanko account ready with a password login if ssh keys get lost or deployment fails is very handy if something goes really wrong (a user has administrative access to the machine and 'things happened'?). It is the same idea for network access as it is having single-user mode on Linux or using init=/bin/bash via grub for local access really really bad and cannot use a live disk for whatever reason. Don't laugh, there are virtualization products prohibiting that. Even if the functionality exists, what if an outdated software version is run lacking the functionality?

Anyway, even if you run your ssh daemon on some esoteric port and not on 22, if not having implemented things like port knocking (to open even another port and thus mitigating portscans, slowly bordering on being too impractical), port scans will detect your service eventually.

Usually you also set up all servers with the same configuration, with the same ports and services for efficiency reasons. You cannot set up ssh to a different port on every machine. Also you cannot change it on all machines every time you consider it being 'public' information, because it already is after a scan. The question of nmap being legal or not is not an issue when having a hacked Wi-Fi connection at your disposal.

If this account is not named 'root', people may not be able to guess the user account name of your 'backdoor'. But they will know, if they get another server from your company, or just buy some webspace, and have an unchrooted/unjailed/uncontainered look at /etc/passwd.

For purely theoretical illustration now, they could use a hackable website there to gain access to your server and look up how things are usually run at your place. Your hack search tools might not run 24/7 (usually they do at night for disk performance reasons for the filesystem scans?) and your virus scanners are not updated the second a new zero-day sees the light of the day, so it will not detect these happenings at once, and without other protection measures you may never even know what happened. To get back to reality, if someone has access to zero-day exploits, it's very likely he will not target your servers anyway as these are just expensive. This is just for illustrating that there is always a way into a system if the 'need' arises.

But on the topic again, just don't use an extra passworded account, and don't bother? Please read on.

Even if attackers to get the name and port of this extra account, a fail2ban + iptables combination will stop them short, even if you used only an eight-letter password. Plus fail2ban can be implemented for other services, too, widening the monitoring horizon!

For your own services, if the need ever arose: Basically every service logging failures to a file can get fail2ban support via providing a file, what regex to match and how many failures are allowed, and the firewall will just happily ban every IP address it is told to.

I am not telling to use 8-digit passwords! But if they get banned for 24 hours for five wrong password tries, you can guess how long they will have to try if they do not have a botnet at their disposal even with such lousy security. And you'd be astonished what passwords customers tend to use, not just for ssh. Having a look at peoples' mail passwords via Plesk tells you everything you'd rather not want to know, if you'd ever do, but what I am not trying to imply here of course. :)

Adaptive global firewalling

fail2ban is just one application which uses something along the lines of iptables -I <chain_name> 1 -s <IP> -j DROP, but you can easily build such stuff yourself with some Bash magic quite fast.

To further expand something like this, aggregate all fail2ban IP addresses from servers within your network on an extra server, that curates all the lists and passes it in turn to your core firewalls blocking all the traffic already on the edge of your network.

Such functionality cannot be sold (of course it can, but it will just be a brittle system and suck), but has to be interweaved into your infrastructure.

While at it, you can also use blacklist IP addresses or lists from other sources, be it aggregated by yourself or external ones.