growse's questions

I've looking after some Debian boxes and occasionally I see a big spike in the network traffic. I'm graphing metrics with graphite (being fed by a sensu check which gathers per-interface metrics every minute) and occasionally see this sort of thing:

I have no idea what's causing this, as I never manage to catch it while it's in progress. It'd be nice to figure out what's causing this, so what's the best approach to try and figure out what this could be?

I guess what I'm really after is this: Is there a way to audit a network connection (and process id/name) if the amount of data it has sent/received trips over a certain amount or rate?

I'm trying to put together a fabric script that creates a virtual instance via an API and then runs puppet on that instance. I've got a task that creates the VM, and a task that can 'bootstrap' the VM. However, I'm having difficulty linking these together, because I'm not sure how to pass the some data generated in the first task into the second task as the hostname. E.g.

def createVM():
    newhostname = local('/usr/bin/createVM')
    bootstrap(newhostname)

def bootstrap(hostname):
    env.hosts = [hostname]
    run('puppet agent -t')

This doesn't appear to work, and I get prompted for the hostname to run the fabric script on, if I just execute fab createVM.

What's the best way of doing this?

I'm trying to administer a Windows 7 machine remotely. I've enabled WinRM and can use Enter-PsSession to connect to the remote machine.

However, I'm noticing a difference between running a particular command locally, vs running it remotely, even though I'm connecting with the same user account (which is a Domain Admin).

The output from the remote session is:

> enter-pssession -computername  REMOTEHOST
[REMOTEHOST} > Get-WURebootStatus
New-Object : Creating an instance of the COM component with CLSID {C01B9BA0-BEA7-41BA-B604-D0A36F469133} from the IClassFactory failed due to the following error: 80070005.
At C:\Windows\system32\WindowsPowerShell\v1.0\Modules\pswindowsupdate\Get-WURebootStatus.ps1:52 char:33
+             $objSystemInfo= New-Object <<<<  -ComObject "Microsoft.Update.SystemInfo"
+ CategoryInfo          : NotSpecified: (:) [New-Object], UnauthorizedAccessException
+ FullyQualifiedErrorId : System.UnauthorizedAccessException,Microsoft.PowerShell.Commands.NewObjectCommand

ExecutionPolicy is set to 'Unrestricted', and this command works great when I'm using a local powershell session on the remote machine.

Is there a different security context for remote powershell sessions?

edit: the specific line it's failing on is this one:

$objSystemInfo= New-Object -ComObject "Microsoft.Update.SystemInfo"

I'm trying to build an unattended Windows 7 image script that will do a simple domain join. The catch is that the DHCP-provided DNS server knows absolutely nothing about my domain, so I need to set the DNS server to something that is aware of the domain before the UnattendedJoin task runs. I do this with a Microsoft-Windows-DNS-Client in the unattend.xml.

I'm at the point where I know the DNS settings are being set, because the Local Area Connection NIC IP settings are correct once the Win7 machine comes up. However, it never makes it to the domain. The specific error reported is:

[DJOIN.EXE] Unattended Join: Begin
[DJOIN.EXE] Unattended Join: Loading input parameters...
[DJOIN.EXE] Unattended Join: AccountData = [NULL]
[DJOIN.EXE] Unattended Join: UnsecureJoin = [NULL]
[DJOIN.EXE] Unattended Join: MachinePassword = [secret not logged]
[DJOIN.EXE] Unattended Join: JoinDomain = [domain.example.com]
[DJOIN.EXE] Unattended Join: JoinWorkgroup = [NULL]
[DJOIN.EXE] Unattended Join: Domain = [DOMAIN]
[DJOIN.EXE] Unattended Join: Username = [domainuser]
[DJOIN.EXE] Unattended Join: Password = [secret not logged]
[DJOIN.EXE] Unattended Join: MachineObjectOU = [OU=Clients,OU=Bucket,DC=example,DC=domain,DC=com]
[DJOIN.EXE] Unattended Join: DebugJoin = [true]
[DJOIN.EXE] Unattended Join: DebugJoinOnlyOnThisError = [NULL]
[DJOIN.EXE] Unattended Join: Enabled DC Locator ETW tracing. Log file: C:\Windows\Panther\UnattendGC\UnattendedJoinDCLocator.etl
[DJOIN.EXE] Unattended Join: Checking that auto start services have started.
[DJOIN.EXE] Unattended Join: Joining domain [domain.example.com]...
[DJOIN.EXE] Unattended Join: Calling DsGetDcName for domain.example.com...
Unattended Join: DsGetDcName failed: 0x54b, last error is 0x0, will retry in 5 seconds...

This error appears to mean 'I can't find the domain'. What's odd is that if I try and do a manual domain join once the install has finished, it works fine. So for some reason, even though the DNS settings are being correctly set before (I can see it further up the log file), DJOIN.EXE isn't using them.

What's going on here?

I've got two webservers which each have a disk attached. This disk is synced between them using drbd (2:8.3.13-1.1ubuntu1) in 'dual-primary' mode, and over the top of this I run ocfs2 (1.6.4-1ubuntu1) as a cluster filesystem. The nodes communicate on a private network 192.168.3.0/24. For the most part, this is stable, and works well.

Last night, there appeared to have been a network outage. This resulted in a split-brain scenario where node01 was left in Standalone and Primary, while node02 was left in WFConnection and primary. Recovery was a manual process this morning of diffing the two filesystems, deciding that node01 should be authoritative, putting node02 into secondary and then issuing drbdadm connect commands on each node. Remounting the filesystem after this and we're back up and running.

My question is: Is this type of outage always going to require a manual resolution? Or are there ways in which this process can be automated? My understanding was that drbd should try to be intelligent in the event of a split brain about working out which node should become primary and secondary. It seems that in this case, a simple network outage left both in primary, which my config just says 'disconnect'. Looking at the logs, what I find interesting is the fact that they both seemed to agree that node02 should be the SyncSource, and yet when looking at the rsync log, it's actually node01 that has the most recent changes. Also interesting is the line on node01 stating 'I shall become SyncTarget, but I am primary!'. To me, it looks like drbd tried to resolve this, but failed for some reason.

Is there a better way of doing this?

The config for r0 is this:

resource r0 {
    meta-disk internal;
    device /dev/drbd0;
    disk /dev/xvda2;

    syncer { rate 1000M; }
    net {
        #We're running ocfs2, so two primaries desirable.
        allow-two-primaries;

        after-sb-0pri discard-zero-changes;
        after-sb-1pri discard-secondary;
        after-sb-2pri disconnect;

    }
    handlers{
        before-resync-target "/sbin/drbdsetup $DRBD_MINOR secondary";

        split-brain "/usr/lib/drbd/notify-split-brain.sh root";
    }
    startup { become-primary-on both; }

    on node02 { address 192.168.3.8:7789; }
    on node01 { address 192.168.3.1:7789; }
}

I've also put the kern.log files on pastebin:

Node01: http://pastebin.com/gi1HPtut

Node02: http://pastebin.com/4XSCDQdC

A Windows service has 4 different startup types that can be configured: Automatic, Automatic (delayed), Manual and Disabled.

I have a service running on a combination of Windows 7 and Server 2008 R2 Desktops that I need to set as 'Automatic (delayed)' using a GPO setting, but from what I can see in the GPO editor, the delayed option is missing:

Have I missed something obvious, or is this a rather basic omission from Microsoft?

I've noticed that there's functionality enabled in nginx by default, whereby a url request without a trailing slash for a directory which exists in the filesystem automatically has a slash added through a 301 redirect.

E.g. if the directory css exists within my root, then requesting http://example.com/css will result in a 301 to http://example.com/css/.

However, I have another site where the SSL is offloaded by a load-balancer. In this case, when I request https://example.com/css, nginx issues a 301 redirect to http://example.com/css/, despite the fact that the HTTP_X_FORWARDED_PROTO header is set to https by the load balancer.

Is this an nginx bug? Or a config setting I've missed somewhere?

I've got some kiosk computers that are on a domain and permanently logged in, and receive updates for FEP through WSUS. This works great, except that occasionally, I get this dialog appearing in the bottom right hand corner:

I thought that by setting the No auto-restart with logged on users for scheduled automatic updates installations GPO to 'enabled', these wouldn't appear, but it seems like they still show up. Here's a screenshot of the Windows Update GPO I'm applying:

What am I missing? How do I turn these notifications off?

I've got a bit of a basic networking question here regarding ping.

When pinging a particular host on a completely different subnet, I get a response like this:

PING myhost.example.com (1.2.3.4) 56(84) bytes of data.
64 bytes from 1.2.3.4: icmp_req=1 ttl=115 time=1.88 ms
64 bytes from 1.2.3.4: icmp_req=2 ttl=115 time=1.66 ms
64 bytes from 1.2.3.4: icmp_req=3 ttl=115 time=1.96 ms
64 bytes from 1.2.3.4: icmp_req=4 ttl=115 time=1.95 ms

--- myhost.example.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 48191ms
rtt min/avg/max/mdev = 1.669/1.868/1.969/0.127 ms

This initially looks normal, but then you see that the total time is 48 seconds to get these 4 responses. When the ping is running, there is a noticable gap of around 10 seconds before each line is printed. However, the rtt is pretty much what I'd expect.

This is causing a bit of an issue, as I'm trying to monitor this host with nagios, and the host check is currently getting a 'Network unreachable' error. The network is reachable, the rtt is sane, but I can't help wonder if this slow total time might have something to do with it.

The host is 14 hops away, and the traceroute looks like this (I've anonymised the intermediate steps, they are all different IPs!):

traceroute to myhost (1.2.3.4), 30 hops max, 60 byte packets
 1  10.A.B.C (10.A.B.C)  0.680 ms  0.729 ms  0.833 ms
 2  10.A.B.C (10.A.B.C)  0.636 ms  0.678 ms  0.759 ms
 3  10.A.B.C (10.A.B.C)  0.803 ms  0.867 ms  0.923 ms
 4  10.A.B.C (10.A.B.C)  0.702 ms  0.738 ms  0.782 ms
 5  10.A.B.C (10.A.B.C)  1.027 ms  1.251 ms  1.342 ms
 6  10.A.B.C (10.A.B.C)  2.688 ms  1.436 ms  1.484 ms
 7  10.A.B.C (10.A.B.C)  2.960 ms  3.475 ms  3.527 ms
 8  10.A.B.C (10.A.B.C)  1.284 ms  1.310 ms  1.393 ms
 9  10.A.B.C (10.A.B.C)  1.990 ms  1.865 ms  1.964 ms
10  10.A.B.C (10.A.B.C)  1.750 ms  1.841 ms  1.748 ms
11  10.A.B.C (10.A.B.C)  1.849 ms  1.614 ms  1.628 ms
12  10.A.B.C (10.A.B.C)  1.997 ms  2.150 ms  2.119 ms
13  10.A.B.C (10.A.B.C)  2.442 ms  2.454 ms  2.560 ms
14  1.2.3.4 (1.2.3.4)  1.978 ms * *

What would cause this?

I've got about 30 Windows 2008 R2 servers as members of a domain, and am attempting to configure the certificates part correctly for remote desktop access to those servers.

The catch is that the clients that need to connect to these servers are not on the domain. The clients are on the same internal network as all the domain computers.

So far, I have done the following:

1. Created the CA
2. Configured a certificate template for Remote Desktop Authentication
3. Configured the Default GPO to enable auto-enrollment and to get the remote desktop servers to enroll a certificate from the RDP cert template
4. Installed the CA root cert into my local computer trusted cert store on the non-domain client

This seems to work, in that each server has gone through the auto-enrollment process.

The problem is that when I connect with an RDP client, I receive a certificate warning stating:

A revocation check could not be performed for the certificate

Looking at the certificate details, I can see it's the correct certificate for the machine, and it has been signed by the CA root, which I have installed and trusted. The CRL Distribution Points entry on the certificate states:

URL=ldap:///CN=domain-ad-CA,CN=host,CN=CDP,CN=Public Key Services,CN=Services,CN=Configuration,DC=domain,DC=example,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint (ldap:///CN=domain-ad-CA,CN=ad,CN=CDP,CN=Public%20Key%20Services,CN=Services,CN=Configuration,DC=domain,DC=thomsonreuters,DC=com?certificateRevocationList?base?objectClass=cRLDistributionPoint)

The root CA cert has no CRL location listed.

At a guess, the client is attempting to contact the LDAP url and failing, but it's not clear why this should be. How do I get the client to perform revocation checks?

I'm running an Oracle 11 box as a ZFS storage appliance, and I'm taking regular snapshots of the ZFS filesystems, via cron.

In the past, I know that if I wanted to grab a particular file from a snapshot, a read-only copy was kept in .zfs/snapshot/{name}/ and I could just navigate there and pull the file out. This is documented on Oracle's website.

However, I went to do this the other day, and noticed that the ZFS directories within the snapshot directories are all empty. zfs list -t snapshot correctly shows the list of snapshots that should be present, and .zfs/snapshots correctly contains a directory for each snapshot, and in each snapshot there is a directory present for each ZFS filesystem. However, these directories appear to be empty.

I just tested a restore by touching a file in a little-used share and rolling back to the latest hourly snapshot, and this appears to have worked fine. So the rollback functionality is there.

Did Oracle change how snapshots are done? Or is something seriously wrong here?

I've got an HP D2700 enclosure that I'm looking to shove some 2.5" SSD drives in. Looking at the prices of HP's SSD drives vs something like an Intel 710 and even something less 'enterprisey', there's quite a difference in price.

I know the HP SSD's will obviously work, but I've heard rumours that buying an Intel/Crucial/whatever SATA SSD, bunging it in an HP 2.5" caddy and putting it in a D2700 won't work.

Is there an enclosure / disk compatibility issue I should watch out for here?

On the one hand, they're all just SATA devices, so the enclosure should treat them all the same. On the other, I'm not particularly well-versed in the various different SSD flavours to know whether there's a good technical reason why one type of drive would work, yet another one wouldn't. I can also imagine that HP are annoying enough to do firmware checks on any disks and have the controller reject those it doesn't like.

For background, the D2700 already has 12x 300GB 10k SAS drives in it, and I was planning on getting 8x 500GB (or thereabouts) SSDs to create another zpool. Whole thing is connected to an HP X1600 running Solaris 11.

I ran a NAS/SAN box on Solaris 11 Express before Solaris 11 was released. The box is an HP X1600 with an attached D2700. In all, 12x 1TB 7200 SATA disks, 12x 300GB 10k SAS disks in separate zpools. Total RAM is 30GB. Services provided are CIFS, NFS and iSCSI.

All was well, and I had a ZFS memory usage graph looking like this:

A fairly healthy Arc size of around 23GB - making use of the available memory for caching.

However, I then upgraded to Solaris 11 when that came out. Now, my graph looks like this:

Partial output of arc_summary.pl is:

System Memory:
     Physical RAM:  30701 MB
     Free Memory :  26719 MB
     LotsFree:      479 MB

ZFS Tunables (/etc/system):

ARC Size:
     Current Size:             915 MB (arcsize)
     Target Size (Adaptive):   119 MB (c)
     Min Size (Hard Limit):    64 MB (zfs_arc_min)
     Max Size (Hard Limit):    29677 MB (zfs_arc_max)

It's targetting 119MB while sitting at 915MB. It's got 30GB to play with. Why? Did they change something?

Edit

To clarify, arc_summary.pl is Ben Rockwood's, and the relevent lines generating the above stats are:

my $mru_size = ${Kstat}->{zfs}->{0}->{arcstats}->{p};
my $target_size = ${Kstat}->{zfs}->{0}->{arcstats}->{c};
my $arc_min_size = ${Kstat}->{zfs}->{0}->{arcstats}->{c_min};
my $arc_max_size = ${Kstat}->{zfs}->{0}->{arcstats}->{c_max};
my $arc_size = ${Kstat}->{zfs}->{0}->{arcstats}->{size};

The Kstat entries are there, I'm just getting odd values out of them.

Edit 2

I've just re-measured the arc size with arc_summary.pl - I've verified these numbers with kstat:

System Memory:
     Physical RAM:  30701 MB
     Free Memory :  26697 MB
     LotsFree:      479 MB

ZFS Tunables (/etc/system):

ARC Size:
     Current Size:             744 MB (arcsize)
     Target Size (Adaptive):   119 MB (c)
     Min Size (Hard Limit):    64 MB (zfs_arc_min)
     Max Size (Hard Limit):    29677 MB (zfs_arc_max)

The thing that strikes me is that the Target Size is 119MB. Looking at the graph, it's targeted the exact same value (124.91M according to cacti, 119M according to arc_summary.pl - think the difference is just 1024/1000 rounding issues) ever since Solaris 11 was installed. It looks like the kernel's making zero effort to shift the target size to anything different. The current size is fluctuating as the needs of the system (large) fight with the target size, and it appears equilibrium is between 700 and 1000MB.

So the question is now a little more pointed - why is Solaris 11 hard setting my ARC target size to 119MB, and how do I change it? Should I raise the min size to see what happens?

I've stuck the output of kstat -n arcstats over at http://pastebin.com/WHPimhfg

Edit 3

Ok, weirdness now. I know flibflob mentioned that there was a patch to fix this. I haven't applied this patch yet (still sorting out internal support issues) and I've not applied any other software updates.

Last thursday, the box crashed. As in, completely stopped responding to everything. When I rebooted it, it came back up fine, but here's what my graph now looks like.

It seems to have fixed the problem.

This is proper la la land stuff now. I've literally no idea what's going on. :(

I'm setting up a continuous integration solution involving .NET solutions and MSSQL 2008 Databases.

All my build servers sit in an AD security group called Build Servers. I've created the DOMAIN\Build Servers object as an MSSQL login and granted the group the dbcreator and securityadmin server roles. I've not granted any specific database roles, because my understanding is that these server roles should let any AD account in that group create, alter, drop and database or user.

I'm running a sample job that should apply some simple amends to a particular database. However, I'm getting the following error:

Detailed message Cannot open database "awesome-database" requested by the login. The login failed. 
Database.dbschema : Deploy error TSD01234: Login failed for user 'DOMAIN\BUILDSERVER01$'.

DOMAIN\BUILDSERVER01$ is definitely in the Build Servers group, so it should be able to log in and alter any database.

Have I missed something obvious here, or do I have to go and map this login to every single database that exists?

Edit

If I grant sysadmin, it works. Hmmm.

Is there a way in which ZFS can be prompted to redistribute a given filesystem over the all of the disks in its zpool?

I'm thinking of a scenario where I have a fixed size ZFS volume that's exported as a LUN over FC. The current zpool is small, just two 1TB mirrored disks, and the zvol is 750GB in total. If I were to suddenly expand the size of the zpool to, say, 12 1TB disks, I believe the zvol would still effectively be 'housed' on the first two spindles only.

Given that more spindles = more IOPS, what method could I use to 'redistribute' the zvol over all 12 spindles to take advantage of them?

I'm currently snapshotting my ZFS-based NAS nightly and weekly, a process that has saved my ass a few times. However, while the creation of the snapshot is automatic (from cron), the deletion of old snapshots is still a manual task. Obviously there's a risk that if I get hit by a bus, or the manual task isn't carried out, the NAS will run out of disk space.

Does anyone have any good ways / scripts they use to manage the number of snapshots stored on their ZFS systems? Ideally, I'd like a script that iterates through all the snapshots for a given ZFS filesystem and deletes all but the last n snapshots for that filesystem.

E.g. I've got two filesystems, one called tank and another called sastank. Snapshots are named with the date on which they were created: sastank@AutoD-2011-12-13 so a simple sort command should list them in order. I'm looking to keep the last 2 week's worth of daily snapshots on tank, but only the last two days worth of snapshots on sastank.

What would be the best way to implement an LTO tape-based backup system on a ZFS fileserver? I've got about 6TB that need to be backed up on a daily basis, along with an existing HP 1840 LTO4 tape drive and a bunch of tapes.

I've already got the ZFS doing automatic snapshots every day, but now want to add a layer of offline storage to this. Ideally, the resulting system would implement some sort of father/son rotation system, so I have daily diff tapes for the last two weeks along with a full backup every week.

Reading around, I've seen that some people have used a combination of zfs send / dd / tar to achieve this, but it's not clear to me exactly how this should be implemented.

Update

I've just read about zfs send -i which supposedly sends the increment between two different snapshots. While I feared this whole exercise might make me write some scripts, that should make the diff backup task a little easier. Still no idea how to handle tape changes via a script though.

I've got a REG_BINARY registry key that I'd like to set via GPO. However, the GPO interface to add registry settings simply has a 'Value Data' text field.

I can set the type to REG_BINARY, but what format is the text field expecting the data in so that the key is correctly set?

Should I be converting my data to Hex before entering it here, or should I be doing this in a completely different way?

I'm logging into a Windows 2003 system from a Windows 7 workstation using Remote Desktop. I'm aiming to try and copy some files to the server from the client. In the 'local resources' section of the RDC, I've got all the drives ticked so it should be sharing those drives with the remote server.

However, when I log in, none of these shares get mapped, and if I navigate to \\tsclient, I just get a blank listing.

What's wrong here? What do I need to enable to allow this functionality to work?

What would be the best approach for load-balancing at least 2-3 Windows 2008 R2 IIS webservers running a multitude of .NET applications? My choices appear to be:

1) Hardware-based network device load balancer, like a Cisco CSS
2) Windows NLB
3) Some sort of linux based proxy, either haproxy or other

The three servers sit as VMs on a vSphere farm, so I have the ability to clone to up the instance count in times of high load. I control the switch that the vSphere hosts are plugged into (Cisco 3750), but don't control the switching/routing infrastructure beyond that to the clients.

(1) Is too expensive, and probably overkill for my needs. I've included this in case someone figures out a cunning way to do it on my existing network kit, which I doubt.

(2) would seem to be the obvious "built-in" option, but seems to be quite fiddly messing around with network interfaces, multicast, and generally other things that seem to be needlessly complex. It's also fairly stupid, in that it can't remove hosts from the pool if they start throwing 500 errors or otherwise go wrong

(3) is the most interesting option, as it would appear to offer the most flexibility and customizability, but without having to mess around with the network. However, while I'm familiar with the reverse-proxy capabilities of lighttpd etc, I'm not that well read on other options like HAProxy, which might be able to offer a lot more.

Which would you go for, and is there anything I've not thought of?