We have a debate in our office going on whether it's necessary to get a hardware firewall or set up a virtual one on our VMWare cluster.
Our environment consists of 3 server nodes (16 cores w/ 64 GB RAM each) over 2x 1 GB switches w/ an iSCSI shared storage array.
Assuming that we would be dedicating resources to the VMWare appliances, would we have any benefit of choosing a hardware firewall over a virtual one?
If we choose to use a hardware firewall, how would a dedicated server firewall w/ something like ClearOS compare to a Cisco firewall?
I've always been reluctant to host a firewall in a virtual machine, for a couple of reasons:
With a hypervisor, the attack surface is wider. Hardware firewalls usually have a hardened OS (read-only fs, no build tools) which will reduce the impact of a potential system compromise. Firewalls should protect the hosts, not the other way around.
We've seen in details what bad NICs can do (or can't), and that's something you want to avoid. While the same bugs can affect appliances, hardware has been selected and is known to work with the installed software. It goes without saying that the software vendor support may not help you if you have issues with drivers, or with any hardware configuration that they don't recommend.
Edit:
I wanted to add, like @Luke said, that plenty of hardware firewall vendors have high availability solutions, with stateful connection state passed from active unit to standby. I've been personally satisfied w/ Checkpoint (on old nokia IP710 platforms). Cisco has ASA and PIX failover/redundancy, pfsense has CARP and IPCop has a plugin. Vyatta can do more (pdf), but it's more than a firewall.
Assuming the software is the same (usually isn't), virtual firewalls can be better than a physical firewall because you have better redundancy. A firewall is just a server with CPU, RAM, and uplink adapters. It's the same argument as a physical web server verses a virtual one. If the hardware fails a virtual server can be migrated to another host automatically. The only downtime is the amount of time it takes for the virtual firewall be migrated to another host, and perhaps the time it takes for the OS to boot.
A physical firewall is bound to the resources it has. A virtual firewall is limited to the resources inside a host. Typically x86 hardware is far cheaper than that of an physical enterprise firewall. What you have to consider is the cost the hardware, plus cost of the software (if not using open source), plus the cost of your time (which will depend on the software vendor you go with). After you compare the cost, what features are you getting on either side?
When comparing firewalls, virtual or physical, it really depends on the feature set. Cisco firewalls have a feature called HSRP which allows you to run two firewalls as one (master and slave) for failover. Non-Cisco firewalls have a similar technology called VRRP. There's also CARP.
When comparing a physical firewall to a virtual one make sure you're doing an apples to apples comparison. What features are important to you? What is the configuration like? Is this software used by other enterprises?
If you need powerful routing, Vyatta is a good bet. It has firewall capabilities. It has a very Ciso-like configuration console. They have a free community edition at vyatta.org and a supported version (with some extra featutes) at vyatta.com. The documentation is very clean and straightforward.
If you need a powerful firewall, take a look at pfSense. It can also do routing.
We decided to run two Vyatta instances with VRRP on our ESXi hosts. To get the redundancy we needed with Cisco (two power supplies per firewall, two firewalls) it would have cost $15-30k. For us Vyatta community edition was a good option. It has a command line only interface, but with the documentation it was easy to configure.
I go with dedicated hardware because it's purpose-built. Having an appliance is handy in that respect, especially if it's a VPN endpoint or some other gateway. It frees your VMWare cluster up from that responsibility. In terms of hardware/RAM/CPU resources, running a software solution is definitely fine. But that's not really a concern.
Of course it's not necessary, and for most people, it will get the job done. Just make some considerations that your traffic may trombone across your virtual switch uplinks unless you dedicate NICs to the firewall VM. (You'll have to do this on each box you want to be able to vMotion to).
Personally? I prefer dedicated hardware because it's really not that expensive. You can get performance numbers on the dedicated hardware from the manufacturer, but your VM firewall performance is completely subjective to how busy your hosts are.
I say try out the software one, see how it goes. If down the road you need to install a hardware one, then do so.