Interestingly I have not found any good search results when searching for "OpenVPN vs IPsec". So here's my question:
I need to set up a private LAN over an untrusted network. And as far as I know, both approaches seem to be valid. But I do not know which one is better.
I would be very thankful if you can list the pros and cons of both approaches and maybe your suggestions and experiences regarding what to use.
Update (Regarding the comment/question):
In my concrete case, the goal is to have any number of servers (with static IPs) connected transparently to each other. But a small portion of dynamic clients like "road warriors" (with dynamic IPs) should also be able to connect. The main goal is however having a "transparent secure network" run on top of the untrusted network. I am quite a newbie so I do not know how to correctly interpret "1:1 Point to Point Connections" => The solution should support broadcasts and all that stuff so it is a fully functional network.
I have all of the scenarios setup in my environment. (openvpn site-site, road warriors; cisco ipsec site-site, remote users)
By far the openvpn is faster. The openvpn software is less overhead on the remote users. The openvpn is/can be setup on port 80 with tcp so that it passes at places that have limited free internet. The openvpn is more stable.
Openvpn in my environment does not force policy to the end user. Openvpn key distribution is a little harder to do securely. Openvpn key passwords are up to the end users (they can have blank passwords). Openvpn is not approved by certain auditors (the ones that only read bad trade rags). Openvpn takes a little bit of brains to setup (unlike cisco).
This is my experience with openvpn: I know that most of my negatives can be alleviated through either configuration changes or process changes. So take all my negatives with a bit of skepticism.
One key advantage of OpenVPN over IPSec is that some firewalls don't let IPSec traffic through but do let OpenVPN's UDP packets or TCP streams travel without hindrance.
For IPSec to function your firewall either needs to be aware of (or needs to ignore and route without knowing what it is) packets of the IP protocol types ESP and AH as well as the more ubiquitous trio (TCP, UDP and ICMP.
Of course you might find some corporate environments the other way around: allowing IPSec through but not OpenVPN, unless you do something crazy like tunneling it via HTTP, so it depends on your intended environments.
OpenVPN can do Ethernet-layer tunnels, which IPsec cannot do. This is important for me because I want to tunnel IPv6 from anywhere that has only IPv4 access. Maybe there is a way to do this with IPsec, but I haven't seen it. Also, in a newer version of OpenVPN you will be able to make Internet-layer tunnels which can tunnel IPv6, but the version in Debian squeeze can't do that, so an Ethernet-layer tunnel works nicely.
So if you want to tunnel non-IPv4 traffic, OpenVPN wins over IPsec.
OpenVPN is
much easier to administer set-up and use in my opinion.. Its fully transparent VPN, which i love...
IPsec is more a "professional" approach with many more options regarding classical routing within vpns..
If you want just a point - to - point vpn (1-to-1), i would suggest using OpenVPN
Hope this Helps :D
I had some experience with managing dozens of sites around the country (NZ) each connecting to the Internet via ADSL. They had been operating with IPSec VPN going to a single site.
The customers requirement changed and they needed to have two VPNs, one going to the main site the other going to a failover site. The customer wanted both VPNs to be active at the same time.
We found that the ADSL routers in use were not coping with this. With one IPSec VPN they were fine but as soon as two VPNs were brought up the ADSL router rebooted. Note that the VPN was initiated from a server inside the office, behind the router. We got technicians from the supplier to check the routers and they sent many diagnostics back to the vendor but no fix was found.
We tested OpenVPN and there were no problems. On consideration of the costs involved (replace dozens of ADSL routers or change VPN technology) it was decided to change to OpenVPN.
We also found diagnostics easier (OpenVPN is much clearer) and many other aspects of management overhead for such a large and widespread network was a lot easier. We never looked back.
I use OpenVPN for a site-to-site VPN and it works great. I really love how customizable OpenVPN is for each situation. The only issue I've had is that OpenVPN isn't multithreaded, therefore you can only get as much bandwidth as 1 CPU can handle. The testing I've done, we've been able to push ~375 MBits/sec across the tunnel with no problems, which is more than enough for most people.
Open VPN site-to-site is much better over IPSEC.We have a client for whom we installed Open-VPN in an MPLS network which worked fine and supported faster and more secure encryption such as Blow-fish 128 bit CBC. At another site which is connected via public IP we used this connection as well in in low bandwith such as 256kbps/128kbps.
However let me point out that IPSec VTI interfaces are now supported in Linux/Unix. This allows you to create routable and secure tunnels much in the same way as OpenVPN site to site or GRE over IPSec.