Dan Carley's questions

Prologue:

On a number of machines, which happen to act as NFS clients, netstat reports two ports that are open with no PID listed for an associated daemon. Ordinarily this might be a bit concerning.

# netstat -lnp | egrep -- '- +$'
tcp        0      0 0.0.0.0:57448           0.0.0.0:*               LISTEN     -
udp        0      0 0.0.0.0:48933           0.0.0.0:*                          -

Additionally netcat confirms that the TCP port really is open.

# nc -v localhost 57448
localhost [127.0.0.1] 57448 (?) open
^C

Yet lsof reports nothing for these two ports. Intrigue grows.

# lsof -i TCP:57448 -i UDP:48933

However rpcinfo finally points us in the right direction. It's being held open by nlockmgr, aka lockd for NFS. Call off the search.

# rpcinfo -p | egrep '57448|48933'
    100021    1   udp  48933  nlockmgr
    100021    3   udp  48933  nlockmgr
    100021    4   udp  48933  nlockmgr
    100021    1   tcp  57448  nlockmgr
    100021    3   tcp  57448  nlockmgr
    100021    4   tcp  57448  nlockmgr

It becomes clear that lockd/rpc.lockd is called when mounting an NFS export. This is a kernel thread (is it always?) which binds itself to one TCP and one UDP port in the ephemeral range. The ports are typically reconfigurable with the fs.nfs.nlm_tcpport and fs.nfs.nlm_udpport sysctls.

Questions:

I am intrigued though. Would love some kernel-internals insight as to..

1. Why isn't the kernel thread's PID visible from netstat?

2. Why aren't the bound ports visible from lsof?

Preface:

We have caching resolvers at each of our geographic network locations. These are clustered for resiliency and their locality reduces the latency of internal requests generated by our servers.

This works well. Except that a vast quantity of the requests seen over the wire are lookups for the same records, generated by applications which don't perform any DNS caching of their own.

Questions:

1. Is there a significant benefit to running lightweight caching daemons on the individual servers in order to reduce repeated requests from hitting the network?

2. Does anyone have experience of using [u]nscd, lwresd or dnscache to do such a thing? Are there any other packages worth looking at?

3. Any caveats to beware of? Besides the obvious, caching and negative caching stale results.

So ONTAP 7.3.2 just became GD.

• Has anyone taken the plunge -- GA or GD alike?
• What platform are you running?
• What are your licensed features -- NFS, CIFS, iSCSI, FC, Multistore, API, etc?
• Any gotchas to report from in-the-field?

Background:

I am finally setting aside some time to join the 21st Century and look at Puppet.

As it stands today we version control all server configurations in a repository that is held internally at the office. When an update needs making, the changes are checked back into the repos and manually pushed out to the machine in question. This usually means SFTP'ing to the remote machine and then moving files into place, with the relevant permissions, from a shell.

So I am hopeful that Puppet is going to be an simple yet amazing extension to what we already have.

Now I consider the process that we currently have to be reasonably secure. On the assumption that our internal network will always be relatively more secure than the public networks in our datacentres.

• The process is always one way. Changes traverse from a secure environment to insecure and never the other way round.

• The master store is in the safest possible place. The risk of compromise, either by stealing configurations or sending out malicious modifications, is greatly reduced.

Question:

From what I understand of the Puppet server/client model is that the clients poll and pull updates down directly from the server. The traffic is SSL wrapped so cannot be intercepted or spoofed. But it differs from what we currently do because the Puppet server[s] would need to be hosted in a public location. Either centrally, or one for each datacentre site that we maintain.

So I am wondering:

• Am I being unnecessarily paranoid about the change from push to pull?

• Am I being unnecessarily paranoid about centrally storing all of that information on a public network?

• How are others maintaining multiple networks - separate server for each site?

Update 30/07/09:

I guess that one of my other big concerns is placing so must trust in a single machine. The puppetmaster(s) would be firewalled, secured and such. But even so any public machine with listening services has an attack surface of a certain size.

Presumably if the master has permission to update any file on any one of the puppet clients, then it's compromise would ultimately result in the compromise of all it's clients. The "kings to the kingdom" so to speak.

• Is that hypothesis correct?

• Is there any way that it can be mitigated?