fission's questions

In Windows there appear to be two ways to set up IPsec:

1. The IP Security Policy Management MMC snap-in (part of secpol.msc, introduced in Windows 2000).
2. The Windows Firewall with Advanced Security MMC snap-in (wf.msc, introduced in Windows 2008/Vista).

My question concerns #2 – I already figured out what I need to know for #1. (But I want to use the ‘new’ snap-in for its improved encryption capabilities.)

I have two Windows Server 2008 R2 computers in the same domain (domain members), on the same subnet:

server2  172.16.11.20
server3  172.16.11.30

My goal is to encrypt all communication between these two machines using IPsec in tunnel mode, so that the protocol stack is:

• IP
• ESP
• IP
• …etc.

First, on each computer, I created a Connection Security Rule:

• Endpoint 1: (local IP address), eg 172.16.11.20 for server2
• Endpoint 2: (remote IP address), eg 172.16.11.30
• Protocol: Any
• Authentication: Require inbound and outbound, Computer (Kerberos V5)
• IPsec tunnel:
  • Exempt IPsec protected connections
  • Local tunnel endpoint: Any
  • Remote tunnel endpoint: (remote IP address), eg 172.16.11.30

At this point, I can ping each machine, and Wireshark shows me the protocol stack; however, nothing is encrypted (which is expected at this point). I know that it's unencrypted because Wireshark can decode it (using the setting Attempt to detect/decode NULL encrypted ESP payloads) and the Monitor > Security Associations > Quick Mode display shows ESP Encryption: None.

Then on each server, I created Inbound and Outbound Rules:

• Protocol: Any
• Local IP addresses: (local IP address), eg 172.16.11.20
• Remote IP addresses: (remote IP address), eg 172.16.11.30
• Action: Allow the connection if it is secure
  • Require the connections to be encrypted

The problem: Though I create the Inbound and Outbound Rules on each server to enable encryption, the data is still going over the wire (wrapped in ESP) with NULL encryption. (You can see this in Wireshark.)

When the arrives at the receiving end, it's rejected (presumably because it's unencrypted). [And, disabling the Inbound rule on the receiving end causes it to lock up and/or bluescreen – fun!] The Windows Firewall log says, eg:

2014-05-30 22:26:28 DROP ICMP 172.16.11.20 172.16.11.30 - - 60 - - - - 8 0 - RECEIVE

I've tried varying a few things:

• In the Rules, setting the local IP address to Any
• Toggling the Exempt IPsec protected connections setting
• Disabling rules (eg disabling one or both sets of Inbound or Outbound rules)
• Changing the protocol (eg to just TCP)

But realistically there aren't that many knobs to turn.

Does anyone have any ideas? Has anyone tried to set up tunnel mode between two hosts using Windows Firewall?

I've successfully got it set up in transport mode (ie no tunnel) using exactly the same set of rules, so I'm a bit surprised that it didn't Just Work™ with the tunnel added.

I have the following scenario:

Computer A: 198.51.100.8, netmask 255.255.255.0
Computer B: 203.0.113.9, netmask 255.255.255.0
Both computers are on the same LAN segment; no default gateway is specified in either case.

To get these two computers to communicate with each other, I've added two static routes, like so:

route add 203.0.113.9 mask 255.255.255.255 198.51.100.8

However, I would prefer to add the static routes by specifying a network interface, instead of by specifying a gateway IP address.

This is possible with Linux by using a command such as:

ip route add 203.0.113.9 dev eth0

and similarly in FreeBSD:

route add 203.0.113.9/32 -iface fxp0 -cloning

However, I'm at a loss of how to do this with Windows. Ideally I want to do something like:

route add 203.0.113.9 mask 255.255.255.255 if 2

but that merely prints out the usage for the route command, which tells me that I'm doing it wrong. I've also tried to use netsh, which tells me:

> netsh routing ip add persistentroute 203.0.113.9 255.255.255.255 "Local Area Connection"
Specify the next-hop for non point-to-point interfaces.

Any thoughts or suggestions?

Update: When I originally posted this question, I was using Windows XP. But I neglected to mention that.

I'll leave Grizly's original answer, as it is correct for my original question. But if you're using a newer version of Windows than XP/2003, give one of the other answers a try.