Magellan's questions

When attempting to use modules copied into the Puppet modules directory, my puppet client returns ' Could not retrieve catalog from remote server: Error 400 on SERVER: can't convert nil in String' errors when connecting to the Puppet master server.

[root@puppetmaster modules]# rpm -qa *puppet*
puppet-2.7.18-1.el6.noarch
puppet-server-2.7.18-1.el6.noarch

[root@puppetmaster modules]# uname -sr
Linux 2.6.32-279.el6.x86_64

Code all checks out and is valid. SELinux is turned on.

Due to a series of poor network design decisions (mostly) made many years ago in order to save a few bucks here and there, I have a network that is decidedly sub-optimally architected. I'm looking for suggestions to improve this less-than-pleasant situation.

We're a non-profit with a Linux-based IT department and a limited budget. (Note: None of the Windows equipment we have runs does anything that talks to the Internet nor do we have any Windows admins on staff.)

Key points:

• We have a main office and about 12 remote sites that essentially double NAT their subnets with physically-segregated switches. (No VLANing and limited ability to do so with current switches)
• These locations have a "DMZ" subnet that are NAT'd on an identically assigned 10.0.0/24 subnet at each site. These subnets cannot talk to DMZs at any other location because we don't route them anywhere except between server and adjacent "firewall".
• Some of these locations have multiple ISP connections (T1, Cable, and/or DSLs) that we manually route using IP Tools in Linux. These firewalls all run on the (10.0.0/24) network and are mostly "pro-sumer" grade firewalls (Linksys, Netgear, etc.) or ISP-provided DSL modems.
• Connecting these firewalls (via simple unmanaged switches) is one or more servers that must be publically-accessible.
• Connected to the main office's 10.0.0/24 subnet are servers for email, tele-commuter VPN, remote office VPN server, primary router to the internal 192.168/24 subnets. These have to be access from specific ISP connections based on traffic type and connection source.
• All our routing is done manually or with OpenVPN route statements
• Inter-office traffic goes through the OpenVPN service in the main 'Router' server which has it's own NAT'ing involved.
• Remote sites only have one server installed at each site and cannot afford multiple servers due to budget constraints. These servers are all LTSP servers several 5-20 terminals.
• The 192.168.2/24 and 192.168.3/24 subnets are mostly but NOT entirely on Cisco 2960 switches that can do VLAN. The remainder are DLink DGS-1248 switches that I am not sure I trust well enough to use with VLANs. There is also some remaining internal concern about VLANs since only the senior networking staff person understands how it works.

All regular internet traffic goes through the CentOS 5 router server which in turns NATs the 192.168/24 subnets to the 10.0.0.0/24 subnets according to the manually-configured routing rules that we use to point outbound traffic to the proper internet connection based on '-host' routing statements.

I want to simplify this and ready All Of The Things for ESXi virtualization, including these public-facing services. Is there a no- or low-cost solution that would get rid of the Double-NAT and restore a little sanity to this mess so that my future replacement doesn't hunt me down?

Basic Diagram for the main office:

These are my goals:

• Public-facing Servers with interfaces on that middle 10.0.0/24 network to be moved in to 192.168.2/24 subnet on ESXi servers.
• Get rid of the double NAT and get our entire network on one single subnet. My understanding is that this is something we'll need to do under IPv6 anyway, but I think this mess is standing in the way.

I have a problem with Guests not booting under VMWare ESXi 5.0 on my IBM x3550M3 server.

Note: Investigation eventually determined that problem was with the VMware client on a Lenovo Edge laptop, the only Windows box available in a Linux IT shop.

vSphere Client v4 and v5 duplicated behavior on the Lenovo Edge. As indicated in the comment to the accepted answer, replacing the workstation with one using different video was the "fix" for this particular issue.

The ESXi host boots just fine. The Client connects just fine. Guests can be configured but do not successfully boot. The initial guest memory consumption jumps up to 560MB and drops down to 40MB after a few seconds. Initial CPU usage is 1 full CPU (3000Ghz per the chart) and immediately drops downm to 29Mhz. Guests do not display any output in the Console tab but show a state of 'Powered On'. No errors in the Events tab. Switching Guest from BIOS to EFI makes no difference.

VMs are listed as Version 7 and the behavior is duplicated across all availabled Guest OS flavors. Problem also duplicated when server is booted up in Legacy Only mode.

Logs do not contain anything particularly suspicious.

Edit: No firewalls, routers, or VLANs in between the client and server.

Edit 2: We have tried to Boot Guest into BIOS screen at Next Boot checkbox in the Guest Setting. Was not successful.

Edit 3: 500GB datastore with 1 40GB VM on it. Plenty of space.

Edit 4: Guests copied from my old ESXi 4 server DO NOT boot on the ESXi 5 system. Initially it complains about too little Video RAM being configured for the default 2500x1600, but it still doesn't work properly even after I bump the Video RAM settings or switch it to Auto-Detect.

We are unable to mount NFS shares to our Debian Lenny Database/Web-App NFS client from our Fedora 8 NFS server. Manual mount command with options and Assisted mount using fstab options returns the same behavior. Machine crashed unexpectedly 6 days ago, but this problem appears to have arisen 3 days ago. (And Yes, just reported to me this morning by the staff responsible for it)

This same server is working properly for all other NFS clients. The NFS client also servers some of its shares back to other clients and the NFS server, which is also working properly.

Processes depending upon these mounts hang, and have been since the 26th. Cron has been turned off to keep the load average to an appropriate level.

Mounts are authenticating properly on the NFS server based upon 'authenticated mount request' messages on the server but the client is

# mount -vvv -t nfs server.example.org:/shared/foo /shared/foo/
mount: fstab path: "/etc/fstab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: spec:  "server.example.org:/shared/foo"
mount: node:  "/shared/foo/"
mount: types: "nfs"
mount: opts:  "(null)"
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = "server.example.org:/shared/foo"
mount: external mount: argv[2] = "/shared/foo/"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw"
mount.nfs: trying 192.168.xxx.xxx prog 100003 vers 3 prot TCP port 2049
mount.nfs: trying 192.168.xxx.xxx prog 100005 vers 3 prot UDP port 51852

And there it sits indefinitely with not further on-screen output. Most likely because of the problem indicated by the following:

Mar 28 10:17:14 db kernel: [1299206.229436] mount.nfs     D e250c5d5     0 20597  20596
Mar 28 10:17:14 db kernel: [1299206.229439]        c0a3cde0 00000086 f7555b00 e250c5d5 0001ca16 c0a3cf6c ce0a9020 0000000d 
Mar 28 10:17:14 db kernel: [1299206.229444]        0013bc68 077ffe57 00000003 00000000 00000000 00000000 00000000 00000246 
Mar 28 10:17:14 db kernel: [1299206.229447]        c0a77c90 00000000 c0a77c98 ce000a7c f8e047c1 c02c93a4 f8e0479c f4518588 
Mar 28 10:17:14 db kernel: [1299206.229451] Call Trace:
Mar 28 10:17:14 db kernel: [1299206.229465]  [<f8e047c1>] rpc_wait_bit_killable+0x25/0x2a [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229485]  [<c02c93a4>] __wait_on_bit+0x33/0x58
Mar 28 10:17:14 db kernel: [1299206.229490]  [<f8e0479c>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229505]  [<f8e0479c>] rpc_wait_bit_killable+0x0/0x2a [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229519]  [<c02c9428>] out_of_line_wait_on_bit+0x5f/0x67
Mar 28 10:17:14 db kernel: [1299206.229523]  [<c0138859>] wake_bit_function+0x0/0x3c
Mar 28 10:17:14 db kernel: [1299206.229528]  [<f8e04c06>] __rpc_execute+0xbe/0x1d9 [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229543]  [<f8dffa72>] rpc_run_task+0x40/0x45 [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229557]  [<f8dffb00>] rpc_call_sync+0x38/0x52 [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229573]  [<f8e80351>] nfs3_rpc_wrapper+0x14/0x49 [nfs]
Mar 28 10:17:14 db kernel: [1299206.229591]  [<f8e8044f>] do_proc_fsinfo+0x54/0x75 [nfs]
Mar 28 10:17:14 db kernel: [1299206.229607]  [<f8e80481>] nfs3_proc_fsinfo+0x11/0x36 [nfs]
Mar 28 10:17:14 db kernel: [1299206.229621]  [<f8e70514>] nfs_probe_fsinfo+0x78/0x47f [nfs]
Mar 28 10:17:14 db kernel: [1299206.229634]  [<f8dffd1f>] rpc_shutdown_client+0x9d/0xa5 [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229647]  [<f8dffb58>] rpc_ping+0x3e/0x47 [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229662]  [<f8e00845>] rpc_bind_new_program+0x69/0x6f [sunrpc]
Mar 28 10:17:14 db kernel: [1299206.229677]  [<f8e71584>] nfs_create_server+0x37b/0x4fa [nfs]
Mar 28 10:17:14 db kernel: [1299206.229693]  [<c01621c1>] __alloc_pages_internal+0xb5/0x34e
Mar 28 10:17:14 db kernel: [1299206.229700]  [<c013882c>] autoremove_wake_function+0x0/0x2d
Mar 28 10:17:14 db kernel: [1299206.229703]  [<c01e7e3c>] idr_get_empty_slot+0x11c/0x1ed
Mar 28 10:17:14 db kernel: [1299206.229711]  [<f8e78fbd>] nfs_get_sb+0x528/0x810 [nfs]
Mar 28 10:17:14 db kernel: [1299206.229724]  [<c01e8125>] idr_pre_get+0x21/0x2f
Mar 28 10:17:14 db kernel: [1299206.229729]  [<c0180159>] vfs_kern_mount+0x7b/0xed
Mar 28 10:17:14 db kernel: [1299206.229734]  [<c0180209>] do_kern_mount+0x2f/0xb8
Mar 28 10:17:14 db kernel: [1299206.229738]  [<c019264a>] do_new_mount+0x55/0x89
Mar 28 10:17:14 db kernel: [1299206.229743]  [<c0192825>] do_mount+0x1a7/0x1c6
Mar 28 10:17:14 db kernel: [1299206.229747]  [<c02ca52a>] error_code+0x72/0x78
Mar 28 10:17:14 db kernel: [1299206.229752]  [<c0190895>] copy_mount_options+0x90/0x109
Mar 28 10:17:14 db kernel: [1299206.229756]  [<c01928b1>] sys_mount+0x6d/0xa8
Mar 28 10:17:14 db kernel: [1299206.229760]  [<c0108857>] sysenter_past_esp+0x78/0xb1
Mar 28 10:17:14 db kernel: [1299206.229766]  =======================

Networking is operating properly as the production users on the Database Web-App Front End are seeing no interruptions in service nor any performance issues.

Memory is fine:

db:/var/log# free -m
             total       used       free     shared    buffers     cached
Mem:         24352      19426       4926          0        281      18283
-/+ buffers/cache:        860      23492
Swap:         7632          0       7632

/etc/fstab:

server.example.org:/shared/foo  /foo        nfs defaults    0 0

Relevant line from server's /etc/exports: /shared/foo 192.168.xxx.xxx(rw,no_root_squash)

TCPDump looks normal. I can post it if someone wants me to, but it's rather large and doesn't appear to have anything distinctly nasty in the output.

I have a group of offices that are all connected to the main office via DSL links on the far end to save costs. (We're a non-profit, don't ask)

We've historically had noticeable problems with the link-up between the ISP that handles our remote sites and the ISP that handles the T1 lines our OpenVPN runs on, so those links frequently go down.

Our mail server's public interface is on the 1st provider's network so it worked just fine, but it's much slower because it's also DSL.

To address the upstream network unreliability issues, I'd written a script that simply modifies the DNS records on the remote sites to point to the internal IP if the tunnel is up or the public IP if the VPN tunnel to the main site is down.

How can I do this in a more elegant fashion that will be instantaneous (instead of my cron-driven scripts) and transparent to the users?

Edit: Remote Offices: Ubuntu 9.10 LTSP servers running various vendor-provided Actiontecs & Motorola and a few with Netgears and Linksys firewall. Main office: Almost 100% Linux (CentOS, in this case) with multiple Netgear FVS318/338 series firewalls with individual firewalls for each IP on our /27. (another don't ask, it was before I got here)

We're primarily an Ubuntu shop, but have recently configured an OpenVPN network to allow senior management to connect to our servers from their home PCs.

One of the problems we're seeing is a large increase in the number of DNS lookups coming down through the OpenVPN pipes into our DNS server for home web traffic that should be going to their ISP's DNS server and not forwarding through our private DNS server.

That traffic IS successfully stopped at the DNS server, but are there options in XP/Vista/Win7 that will allow us to restrict DNS lookup requests to certain domain names and not send those requests down the OpenVPN pipes?

We're running Postfix 2.3.3 on CentOS 5.4 for our SMTP service. Fairly standard setup with about 450 users. We're using a standard /etc/aliases file with includes to create various dynamically-generate aliases, including an "all staff" email address.

Management would like use of certain of these aliases to be limited to authorized senders and other aliases to be open to all. Is there an option to set up ACLs for individual aliases such that only certain senders are successfully delivered to those aliases?

I have no problem with it being accepted and a rejection message being sent. The folks doing the CC to the entire universe don't understand how 'Reply All' works anyway and as much as 10 years of reminders hasn't gotten through to them.