Shane Madden's questions

When setting up an L2TP VPN service for remote access on Windows Routing and Remote Access, we ran into an odd problem with setting options on the connecting clients. The way to provide these settings to L2TP clients in RRAS in through DHCP (MS doc) - an L2TP client sends a DHCPInform packet, which then applies the settings from DHCP options (option 15 for the DNS search suffix, and option 121 for split tunnel routes).

In most L2TP client VPN endpoints like Cisco ASAs, the L2TP service crafts the DHCP response with the DNS and routing settings configured - but in RRAS, you use the DHCP relay built into RRAS, or a DHCP server on the local subnet - but these options must be set in DHCP, not via RADIUS attribute or RRAS setting or anything else. (for too much detail on how the DHCPInform works and why it's so necessary, follow this crazy thread between a couple of our Server Fault regulars)

We found that no Mac (or iOS, for that matter) clients would successfully retrieve these options on our newly configured Windows Server 2016 VPN server, instead timing out on their attempts to get DHCP from the VPN server (from /var/log/ppp.log after turning on verbose logging):

...
Fri Dec  1 12:09:18 2017 : sent [IP data <src addr 192.0.2.8> <dst addr 255.255.255.255> <BOOTP Request> <type INFORM> <client id 0x08000000010000> <parameters = 0x6 0x2c 0x2b 0x1 0xf9 0xf>]
Fri Dec  1 12:09:21 2017 : sent [IP data <src addr 192.0.2.8> <dst addr 255.255.255.255> <BOOTP Request> <type INFORM> <client id 0x08000000010000> <parameters = 0x6 0x2c 0x2b 0x1 0xf9 0xf>]
Fri Dec  1 12:09:24 2017 : No DHCP server replied

Watching the RRAS server as this happens, the DHCP Relay seems to see these packets, but oddly, reports them as discarded (the numbers in the circled columns both increment immediately when the client logs the DHCP requests being sent):

After turning up the RRAS logging knobs to the max, I expected the Windows event log to shed some light on the reason for the discards (since the documentation exists for all of these events), but none of these events began logging. Finally, after turning on the RRAS tracing logs, C:\Windows\system32\tracing\IPBOOTP.LOG finally provided a hint as to why the Mac's DHCP packets were unacceptable:

[9712] 12:09:10: dropping REQUEST with secs-since-boot 0 on interface 42 (192.0.2.8)

Why won't the RRAS DHCP Relay do its job and send these VPN clients' DHCP packets to the DHCP server?

We have a tool being developed that will keep specific attributes of Active Directory user objects up to date with an authoritative source of employee information truth elsewhere, so that when someone's phone number or manager or location changes, Active Directory is automatically updated.

For normal users, delegation of manipulating to those properties is simple to handle using the delegation tools, but protected users, who have the adminSDHolder ACL applied, it's more difficult.

When adding an ACE to the adminSDHolder ACL using the UI, you're only able to grant access to all properties (which we don't want for security reasons), or properties that exist on the adminSDHolder object itself - not user properties like department.

How do you grant access to specific properties of user objects under the protection of adminSDHolder?

This vCenter server was just upgraded to 5.1 update 1. I'm going through hosts and bringing firmware up to date, then upgrading them from various versions of 5.0 to 5.1u1.

vCenter 5.1u1 seems to have an interesting new behavior: it's removing hosts from maintenance mode when they reconnect after being disconnected -- but very inconsistently, I've seen it maybe 4 or 5 times on ~25-30 host reboots. I've only seen it happen on 5.0 hosts that have not yet been upgraded to 5.1.

In the image, I placed the host in maint mode and rebooted it into the HP SPP DVD's automatic update mode. After its usual ~40 minute update process, the host came back online.. and 7 seconds before even logging that the host had reconnected, vCenter had sent the host a task to exit maintenance mode.

In my understanding, the only time vCenter should drop a host out of maintenance mode is when vCenter put it into maintenance mode itself (such as a VUM upgrade task).

Why would this vCenter be unilaterally exiting a host from user-initiated maintenance mode?

Edit, additional info:

I ran the firmware upgrades on 5 more hosts, all at the same time. Two of them exited maint mode after reconnecting, three did not. The common factor of those exiting maint mode seems to be how long they were offline; the two that took a few tries to boot to the virtual media are the two that got knocked out of maint mode.

• esx31 (image above): 45 minutes unresponsive
• esx19 (exited maint): 87 minutes unresponsive
• esx24 (stayed in maint): 32 minutes unresponsive
• esx29 (stayed in maint): 39 minutes unresponsive
• esx32 (stayed in maint): 30 minutes unresponsive
• esx34 (exited maint): 70 minutes unresponsive

Edit: The disconnect time idea seems to have been a red herring, as it's not happening consistently.

Additionally, in the vpxd.log the exit maint mode task initiation seems to always immediately follow this vim.EnvironmentBrowser.queryProvisioningPolicy SOAP call. Here's the lines, slightly trimmed for clarity:

15:27:49.535 [info 'vpxdvpxdVmomi'] [ClientAdapterBase::InvokeOnSoap] Invoke done (esx31, vim.EnvironmentBrowser.queryProvisioningPolicy)
15:27:49.560 [info 'commonvpxLro'] [VpxLRO] -- BEGIN task -- esx31 -- HostSystem.exitMaintenanceMode --

Note that on the nodes that don't get the exit task, the vim.EnvironmentBrowser.queryProvisioningPolicy event still occurs. I'm not seeing any other differences in events before or after this in the reconnect process, aside from the extra events caused by exiting maintenance mode.

Given the log's mention of provisioning policies, looking for autodeploy-related maintenance mode issues turns up complaints about similar behavior (though I'm not using autodeploy at all).

Reverse ARP is.. well, pretty much dead, as far as I'm aware? One of the great Internet success stories in killing off a protocol? It's been deprecated in favor of BOOTP (and later, DHCP) for almost three decades.

So, I was a little surprised to notice a VM mercilessly asking for an IP address via RARP during a PXE boot - even after getting a perfectly good IP address via DHCP.

At the start of the boot, the first broadcast packet that gets sent is a Reverse ARP packet, followed immediately by a DHCP broadcast.

20:31:19.408086 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46
20:31:19.441857 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0c:29:20:fd:ce, length 548
20:31:19.443536 IP 192.168.100.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300

Apparently it doesn't like that first DHCP response, as it waits for another (note that I'm only capturing broadcast packets, so tcpdump doesn't see the rest of the DHCP conversation), but not before sending another couple of RARP requests:

20:31:19.935341 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46
20:31:20.935426 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46
20:31:21.500371 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:0c:29:20:fd:ce, length 548
20:31:21.501288 IP 192.168.100.1.67 > 255.255.255.255.68: BOOTP/DHCP, Reply, length 300
20:31:21.504278 ARP, Request who-has 192.168.100.40 tell 192.168.100.6, length 46
20:31:22.935467 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46

Ok, great; it got an IP, then ARP'd for the PXE server so that it could fire up the TFTP. It snuck another RARP in there at the end, fine. Now it's all the way into the pxelinux environment:

And then? Another RARP request, exactly 1 second after the last one.

20:31:23.935340 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46

And another, exactly 2 seconds later.

20:31:25.935384 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46

And then some. 3 seconds later, 5 seconds after that, 8 seconds, 13 seconds, 21 seconds.. and at that point it finally subsides.

20:31:28.935548 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46
20:31:33.935518 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46
20:31:41.935633 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46
20:31:54.935970 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46
20:32:15.936232 ARP, Reverse Request who-is 00:0c:29:20:fd:ce tell 00:0c:29:20:fd:ce, length 46

All while in the pxelinux environment, with a working IP address already bound to that NIC.

So, does anyone have any idea what this VM (or rather, every ESX(i) VM, at least on 4.1 and 5.0) is thinking?

I've verified that it occurs on both the emulated E1000 as well as the vmxnet3 device; is this a bit of "special" behavior of the VMware PXE code, or is this a typical behavior of any ol' PXE code?

Does it make any kind of sense for it to be looking for RARP at all since as a protocol it's not able to provide any PXE server information (as far as I know)?

Do I need to bite the bullet and set up rarpd to see how the PXE device will react to it?

Dell offers an add-on hardware card for some of their servers that provides a poor man's RAID1 between two SD cards (one card gets all the reads until it dies, writes are mirrored) for booting a simple OS that has very little write load to the OS storage (ie ESXi) - see here (warning: PDF link).

It's great for running a diskless VM host without needing to boot from SAN or worry about a single little cheap flash failure from taking out a host - but it's very much proprietary, to the point that it's integrated to the BIOS of the systems that support it.

Are there any other solutions similar to this out there, offered by server hardware vendors or otherwise (I'm imagining a custom USB flash reader with integrated RAID1?), that allow redundancy between two cheap little chunks of flash storage, hopefully with better compatibility than "a couple specific Dell servers"?

In an ideal world, configuring puppet to install the open-vm-tools should be as simple as this:

class vm-tools {
    package { 'open-vm-tools':
        ensure => installed
    }
    package { 'open-vm-dkms':
        ensure => installed
    }
}

But, that opens up an ugly can of dependency creep; it installs X, which obviously doesn't belong on servers. As of Ubuntu 10.04, these packages both end up recommending the open-vm-toolbox package of GUI tools:

# apt-cache depends open-vm-dkms
open-vm-dkms
  Depends: dkms
  Depends: make
  Suggests: open-vm-toolbox
  Recommends: open-vm-tools

# apt-cache depends open-vm-tools
open-vm-tools
  Depends: libc6
  Depends: libfuse2
  Depends: libgcc1
  Depends: libglib2.0-0
  Depends: libicu44
  Depends: libstdc++6
  Recommends: open-vm-source
  Recommends: open-vm-toolbox
  Recommends: ethtool
  Recommends: zerofree

Recommended packages are always installed by default. It's clearly not desirable to install X dependencies by default when installing a package that is described as "CLI Tools".

The feature request against Debian was immediately rejected for this reason, but cooler heads did not prevail in Ubuntu. It seems that within the last week, there's some recognition that this was an ill-advised change, but that's of no help until the next LTS release rolls around.

The behavior to install recommended packages is easily enough disabled on the command line with the --no-install-recommends option, but through puppet there's no support for doing this, and a tangled mess of tickets requesting that support haven't gone far in 3 years.

The other option is to just disable recommended packages throughout the whole system via apt.conf, which is a massive change to package behavior with impacts reaching further than I'd like.

I've resigned myself to doing it the lazy way;

exec { 'open-vm-tools install':
    command => '/usr/bin/apt-get install -y --no-install-recommends open-vm-dkms open-vm-tools',
    creates => '/usr/lib/open-vm-tools',
}

But this is clearly "doing it wrong". Am I missing something that would make this all work the way it's supposed to, or is this the best hackish workaround to this issue?

Recently ran into an issue where adding a dynamic NAT to an interface broke all translated traffic going through the interface with RPF failures. We found that the addition of the command caused NAT reverse path filtering to start dropping most traffic on the interface, previously RPF checking was not happening (or at least, was not previously showing up in packet-tracer results).

The interface already had hundreds of NAT exemptions, statics, and dynamic policy NATs, but adding the first dynamic NAT (with a little /29 source restriction, and a destination interface unrelated to 99% of the translations that broke) triggered the failures.

Here's what broke it:

nat (Public) 33 172.16.14.0 255.255.255.248 outside tcp 0 0 udp 0
global (Voice) 33 10.0.8.180 netmask 255.255.255.255

The referenced rule in the RPF fails was the main PAT rule for the inside; nothing about the added config should have interfered with successful reverse of that rule:

Phase: 9
Type: NAT
Subtype: rpf-check
Result: DROP
Config:
nat (public) 0 0.0.0.0 0.0.0.0 outside
nat-control
  match ip Public any inside any
    no translation group, implicit deny
    policy_hits = 7274
Additional Information:

Full packet-tracer results before NAT added: http://pastey.net/149630 and after: http://pastey.net/149631

So, the question is: is this really how NAT RPF behaves - only filtering for an interface once that interface has a dynamic NAT attached? Why did a dynamic NAT trigger it, but the dynamic policy NATs (behaving exactly the same, but with specified source and dest instead of just a source) did not? Or was it always happening, just with an implicit allow on the interface until there was a dynamic NAT, which then turned into an implicit deny if no rules matched?

5520 running 8.2.2, if it matters.

A Cisco ASA can have sub-interfaces defined on an interface, vlan tags through that physical interface which are considered by the software as a separate logical interface.

This page includes information on the maximum number of sub-interfaces that can be defined; what I'm interested in confirming whether this limit is a global limit including every defined vlan on any interface, or a per-interface limit.

The page linked seems to imply that it's global, but the configuration "feels" more like a per-interface limit:

interface GigabitEthernet0/2.28
 vlan 28
 nameif foo
 security-level 5
 ip address 10.2.3.1 255.255.255.0 

My hunch is that it's global, but my Google-fu is failing to find anything to back that up (and we'll need to get working on getting a new firewall budgeted if I'm right; proof is good before going to the bean counters). Can anyone confirm?