sysadmin1138's questions

We are using the mod_auth_openid plugin to provide OpenID support for part of our site. It works pretty well, but we're running into one of those "Do this, except when" conditions, and I'm not sure where I'm missing.

There are a few URIs that we don't want this applied. On the surface, this is pretty straight forward.

<Directory "/opt/homeapp/web">
  AuthType openid-connect
  SetEnvIf Request_URI "^/(callbacks.php.*)$" allow

  require claim hd:example.com
  require env allow
  require valid-user
</Directory>

This works fine for things that hit the callbacks.php file, and any other actual-file.

Where my problem comes in, is attempting to match Symfony routes. If the incoming request is to /combobulator/newForm, it seems like Request_URI should be equal to /combobulator/newForm. However, this is definitely not the case.

SetEnvIf Request_URI "^/combobulator/(.*)" allow
SetEnvIf Request_URI "combobulator/(.*)$" allow
SetEnvIf Request_URI "combobulator" allow
SetEnvIf Request_URI "(combobulator)" allow

All of those don't do the thing.

How do you match routes with SetEnvIf, or is that even possible?

One of the projects I'm working on is moving certain puppet-applied ulimit settings away from "that sounds about right" to dynamically allocated based on the environment. This is for single-application environments, so I'm mostly worried about preventing the application from resource starvation while keeping the kernel and utility-spaces in enough handles and whatnot to do what they should.

We get persistent requests from app-teams for moar file handles! so I'm attempting to find a way to deal with that. So I made a puppet-fact:

Facter.add('app2_nofile') do
  confine :kernel => 'Linux'
  setcode do
    kernel_nofile = `/bin/cat /proc/sys/fs/file-max`.chomp
    app2_limit = (kernel_nofile.to_i * 0.85).round
    app2_limit
  end
end

Which does what it says on the tin. It takes the kernel value defined in /proc/sys/fs/file-max and take 85% of it, leaving 15% for system usage. Set a soft and hard nofile ulimit using this ::app2_nofile fact in another puppet resource so /etc/security/limits.conf is updated, and tada! Simple! If they want more file-handles, they'll have to be smarter about writing the app.

Except, it didn't work. When attempting to open a user session (su app2_user -) with the user with that nofile ulimit, we get the error message:

Could not open session

Which is bad.

Clearly, there is an upper-bound somewhere independent of simple ulimits. Or maybe I'm understanding how they fundamentally work. How does nofile limits interact with each other, and what would cause the session to not be able to be created?

Further testing suggests that the upper-bound may be a static boundary, or more complex than simple percentages. A small-RAM system with a file-max of 797,567 can have this ulimit set very high and I'll get no reproduction. On a larger system with 1,619,938 I can have that ulimit set to about 63% before I get "could not open session." I don't have anything larger right now to test with to see if that percentage moves with bigger RAM.

I do get an audit.log entry:

type=USER_START msg=audit(1416420909.479:511331): user pid=5022 uid=0 auid=1194876420 ses=44826 
subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_open 
acct="app2" exe="/bin/su" hostname=? addr=? terminal=pts/0 res=failed'

The op was a PAM operation.

I have a requirement to provide a highly available MySQL database with encryption-at-rest on a 2.6.32 linux kernel. The 'highly available' part isn't that hard, but the 'with encryption-at-rest' is proving to be a challenge when used in combination with HA.

The key problem comes with mounting the encrypted storage. On all of our other encrypted-at-rest systems there is a command that needs to be run by a human, who is then prompted for the encryption key. This model has a fairly obvious flaw when it comes to a cluster arrangement where services will have to start automatically.

I am currently at a loss over how to provide encryption-at-rest in an HA environment and not also store key passphrases on the same system.

I can see two possible scenarios, either of which will work with my environment but I'm not sure of the details to make them work. Or even if it is possible.

Scenario 1: CLVM & Cluster

• A volume is shared between my cluster members.
• This volume is set up roughly:
  • cryptsetup stuff on the physical device
  • LVM stuff on the new crypt-device
• Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
• Cluster Services are started via command run by a human, which supplies the decryption key, which in turn activates the CLVM stuff.

This way running nodes have access to the CLVM volume so they can start services when told to by the cluster manager. Reboots of nodes will still require a human, and the crypt passphrase is never kept on disk anywhere.

Scenario 2: DRBD & Cluster

• A volume is created on each cluster member
• cryptsetup stuff is run on the physical device
• drbd is configured on top of the crypted device, to replicate it between each node
• LVM or a filesystem is placed on top of the drbd volume
• Cluster Services are set to not join the cluster automatically, relying on a manual intervention.
• Cluster services are started by a human who provides the decryption key, which in turn makes the LVM (or filesystem) visible-but-not-mounted.

As with the CLVM setup, nodes don't join the cluster until they have visibility into the presumably-shared storage.

The thing is, I'm not sure if either of the above work that way. Both assume it's possible to layer an LVM PV on top of an encrypted volume (e.g. pvcreate /dev/mapper/cryptmysql). This may not be possible.

This is very system dependent, but chances are near certain we'll scale past some arbitrary cliff and get into Real Trouble. I'm curious what kind of rules-of-thumb exist for a good RAM to Disk-space ratio. We're planning our next round of systems, and need to make some choices regarding RAM, SSDs, and how much of each the new nodes will get.

But now for some performance details!

During normal workflow of a single project-run, MongoDB is hit with a very high percentage of writes (70-80%). Once the second stage of the processing pipeline hits, it's extremely high read as it needs to deduplicate records identified in the first half of processing. This is the workflow for which "keep your working set in RAM" is made for, and we're designing around that assumption.

The entire dataset is continually hit with random queries from end-user derived sources; though the frequency is irregular, the size is usually pretty small (groups of 10 documents). Since this is user-facing, the replies need to be under the "bored-now" threshold of 3 seconds. This access pattern is much less likely to be in cache, so will be very likely to incur disk hits.

A secondary processing workflow is high read of previous processing runs that may be days, weeks, or even months old, and is run infrequently but still needs to be zippy. Up to 100% of the documents in the previous processing run will be accessed. No amount of cache-warming can help with this, I suspect.

Finished document sizes vary widely, but the median size is about 8K.

The high-read portion of the normal project processing strongly suggests the use of Replicas to help distribute the Read traffic. I have read elsewhere that a 1:10 RAM-GB to HD-GB is a good rule-of-thumb for slow disks, As we are seriously considering using much faster SSDs, I'd like to know if there is a similar rule of thumb for fast disks.

I know we're using Mongo in a way where cache-everything really isn't going to fly, which is why I'm looking at ways to engineer a system that can survive such usage. The entire dataset will likely be most of a TB within half a year and keep growing.

We are attempting to set up a guest WLAN network that is isolated from the rest of our network. This is proving difficult due to a couple of technical reasons. My first choice was to use a separate VLAN, on which our Firewall's handy WLAN port would handle DHCP, DNS and the network isolation we need. Unfortunately, due to the fact that our main office and our Internet connection itself are in different locations connected by way of a Metro Ethernet connection, I'm at the mercy of our ISP for VLAN transit.

They won't pass a second VLAN between our two sites. And my hardware doesn't support 802.1ad "Q-in-Q", which would also solve this problem. So I can't use the VLAN method for isolation. At least not without spending money.

As our Firewall can handle IPSec site-to-site VPN connections, I hope it is possible to connect a Server 2008R2 (standard) server I have in the office location to the WLAN and provide gateway services to the firewall. Thusly:

Unfortunately, I don't know if it is possible to connect the two this way. The firewall has a pretty flexible IPSec/L2TP implementation (I've used it to connect iPads in the wild), but is neither Kerberized or supports NTLM. The Connection Security Rules view on the Windows server seems to get close to what I think needs to be done, but I'm failing on figuring out how to get it to do what I need it to do.

Is this even possible, or do I need to pursue alternate solution?

We're building a product that is likely to generate very large XFS volumes, and I'm trying to discover the scaling bottlenecks we're likely to run into given the architecture.

As we manipulate files they get placed into directories on the XFS volumes. Due to the number of files we handle, the file-count is definitely in the tens of millions and will likely get into the hundreds of millions before too long after release. We know this because our current product behaves this way, so it is reasonable to expect our next one to do similarly.

Therefore, correct early engineering is in order.

This week the files are based on the following rough layout:

$ProjectID/$SubProjectID/[md5sum chunked into groups of 4]/file

Which gives directories that look kind of like:

0123456/001/0e15/a644/8972/19ac/b4b5/97f6/51d6/9a4d/file

The reason for chunking the md5sum is to avoid the "big pile of files/directories in one directory" problem. Due to the md5sum chunking it means that 1 file causes 8 directories to be created. This has pretty clear inode impacts, but I'm unclear on what those impacts will be for XFS once we get up to scale.

What are the impacts?

This is with kernel 2.6.32 by the way, CentOS 6.2 at the moment (this can change if needed).

In testing I've created the xfs volume with defaults, and am not using any mount-options. This is to smoke out problems early. noatime is something of a no-brainer as we don't need it. Overall XFS tuning is another problem I need to tackle, but right now I'm concerned about the metadata multiplier effect we've engineered in right now.

I already know what a better solution will be, I just don't know if I have a case to push for the change.

Since md5sums are significantly unique in the first digits, and individual sub-projects rarely exceed 5 million files, it seems to me that we only need the first two chunks. Which would yield layouts like:

0123456/001/0e15/a644/897219acb4b597f651d69a4d/file

A completely full first and second levels would have 2¹⁶ first level directories and 2¹⁶ second level directories in each first level directory, for a total of 2³² directories on the volume.

The hypothetical 5 million file sub-project would therefore have 2¹⁶ first level directories, roughly 76 (+/- 2) second tier directories in each, and one or two third tier directories in each second tier directory.

This layout is a lot more metadata efficient. I just don't know if is worth the effort to change how things are going right now.

We have a GlusterFS cluster we use for our processing function. We want to get Windows integrated into it, but are having some trouble figuring out how to avoid the single-point-of-failure that is a Samba server serving a GlusterFS volume.

Our file-flow works like this:

1. Files are read by a Linux processing node.
2. The files are processed.
3. Results (can be small, can be quite large) are written back to the GlusterFS volume as they're done.
   • Results can be written to a database instead, or may include several files of various sizes.
4. The processing node picks up another job off of the queue and GOTO 1.

Gluster is great since it provides a distributed volume, as well as instant replication. Disaster resilience is nice! We like it.

However, as Windows doesn't have a native GlusterFS client we need some way for our Windows-based processing nodes to interact with the file store in a similarly resilient way. The GlusterFS documentation states that the way to provide Windows access is to set up a Samba server on top of a mounted GlusterFS volume. That would lead to a file flow like this:

That looks like a single-point-of-failure to me.

One option is to cluster Samba, but that appears to be based on unstable code right now and thus out of the running.

So I'm looking for another method.

Some key details about the kinds of data we throw around:

• Original file-sizes can be anywhere from a few KB to tens of GB.
• Processed file-sizes can be anywhere from a few KB to a GB or two.
• Certain processes, such as digging in an archive file like .zip or .tar can cause a LOT of further writes as the contained files are imported into the file-store.
• File-counts can get into the 10's of millions.

This workload does not work with a "static workunit size" Hadoop setup. Similarly, we've evaluated S3-style object-stores, but found them lacking.

Our application is custom written in Ruby, and we do have a Cygwin environment on the Windows nodes. This may help us.

One option I'm considering is a simple HTTP service on a cluster of servers that have the GlusterFS volume mounted. Since all we're doing with Gluster is essentially GET/PUT operations, that seems easily transferable to an HTTP-based file-transfer method. Put them behind a loadbalancer pair and the Windows nodes can HTTP PUT to their little blue heart's content.

What I don't know is how GlusterFS coherency would be maintained. The HTTP-proxy layer introduces enough latency between when the processing node reports that it is done with the write and when it is actually visible on the GlusterFS volume, that I'm worried about later processing stages attempting to pick up the file won't find it. I'm pretty sure that using the direct-io-mode=enable mount-option will help, but I'm not sure if that is enough. What else should I be doing to improve coherency?

Or should I be pursuing another method entirely?

As Tom pointed out below, NFS is another option. So I ran a test. Since the above mentioned files have client-supplied names that we need to keep, and can come in any language, we do need to preserve the file-names. So I built a directory with these files:

When I mount it from a Server 2008 R2 system with the NFS Client installed, I get a directory listing like this:

Clearly, Unicode is not being preserved. So NFS isn't going to work for me.

When doing a puppet agent call from a new image, I'm getting a err: Could not find class custommod error. The module itself is in /etc/puppet/modules/custommod same as all of the other modules we're calling, but this one is obstinante.

[site.pp]

node /clunod-wk\d+\.sub\.example\.local/ {
      include base
      include curl
      include custommod
      class{ "custommod::apps": frontend => "false}
      [...]
}

When the puppetmaster is run with debug output, it clearly finding the information for base and curl:

debug: importing '/etc/puppet/modules/base/manifests/init.pp' in environment production
debug: Automatically imported base from base into production
debug: importing '/etc/puppet/modules/curl/manifests/init.pp' in environment production
debug: Automatically imported curl from curl into production
err: Could not find class custommod for clunod-wk0130.sub.example.local at /etc/puppet/manifests/site.pp:84 on node clunod-wk0130.sub.example.local

Line 84 is include custommod

An abbreviated directory and file structure:

/etc/puppet
   |- manifests
   |     |- site.pp
   |
   |- modules
         |- base
         |    |- manifests
         |          |- init.pp
         |
         |- curl
         |    |- manifests
         |          |- init.pp
         |   
         |- custommod
              |- files 
              |     |- apps
              |         |- [...]
              |
              |- manifests
                    |- init.pp
                    |- apps.pp

I did check spelling :}

The content of init.pp in the custommod directory is completely unremarkable:

class custommod {
}

The intent is to create an empty class for the apps.pp file, which is where the meat is.

class custommod::apps {

    [lots of stuff]
}

Only, it's never getting to the apps file. If I comment out the include custommod, the above error is generated on the class{ "custommod::apps": frontend => "false} line instead.

What am I missing in my hunt to find out how this error is being generated? I need to note that this repo works just fine if it is run locally via puppet apply.

We're in the process of migrating away from a monolithic puppet repository that contains all of the config. This one repo contains things that really shouldn't be on every node ever, so a puppetmaster-based system seems the best way to appropriately segregate things.

The problem I'm having is that the same repo works just fine when copied to the local nodes and puppet apply /etc/puppet/manifests/site.pp run on it. Our installs run very cleanly.

When I put the puppet repo on the puppetmaster and get the agent signed, it doesn't appear to do anything other than report it's local catalog to the puppetmaster.

For a while there, and I haven't been able to reproduce the config that did this, one of our self-built modules was throwing an error that suggested it was attempting to copy a file from the local machine's modules directory, rather than the puppetmaster like it should.

Which suggests that there may be module and manifest syntax differences when building a repo for local use and via puppetmaster.

If there are any differences, what are they, or what are the main tripping points for conversion efforts?

The /etc/puppet/puppet.conf file on the master:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
pluginsync=true

# Set up Environments
## Each environment has a dedicated 'modules' directory under /environments/ from
## which puppet will preferentially pull. Otherwise, it'll use the top level  
## 'modules' directory. That is where common code goes.
[master]
  manifest=$confdir/manifests/site.pp
  modulepath=$confdir/environments/$environment/modules:$confdir/modules
  ssl_client_header= SSL_CLIENT_S_DN
  ssl_client_verify_header = SSL_CLIENT_VERIFY
[production]
  manifest=$confdir/manifests/site_prod.pp
[integration]
  manifest=$confdir/manifests/site_integ.pp
[development]
  manifest=$confdir/manifests/site_dev.pp
[operations]
  manifest=$confdir/manifests/site_ops.pp

At the moment, the site_prod.pp file and ilk are symlinks to site.pp.

The classes defined in the site.pp file work off of the hostname. As I mentioned, when run via puppet apply things work just fine. If worse comes to worst I can do what I need by way of a temporary NFS mount but I'd rather not do that.

site.pp For your enjoyment:

filebucket { "main": server => $server; }
File { backup => "main" }

Exec { path => "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" }

import "stages.pp"
import "int_setup.pp"
import "nodes.pp"

The imported .pp files are in the same directory as site.pp. nodes.pp is where the meat is, but also contains secret-sauce. Some excerpts:

node /clunod\-wk\d+\.sub\.example\.local/ {

  class { "int_setup": stage => "setup"; }
  include base
  include curl
  include custommodule
  [...]
}

Which can and does match nodes named clunod-wk01234.sub.example.local when run via puppet apply.

[int-setup.pp]

class int_setup {
  file { "/var/cluebat":
    ensure => directory,
    mode => "755",
    owner => "root",
    group => "root";
  }

  group{"puppet":
    ensure => present
  }

}

I'm having a hard time figuring out how to rename a Windows 7 computer remotely. This is for automating Win 7 builds in a vSphere 5 environment, and I'm trying to get it as hands-off as I can. So far I've managed to get everything but the machine rename automated (or automatable).

WinRM is working, so remote powershell methods do work.

invoke-command -computername "W7-Img3-RPT49VA" -scriptblock {commands go here; have another one}

The above does work. I can do things like get directory listings and run commands. However, getting the domain changed hasn't worked. Per this SF question, I tried the following in the scriptblock:

$comp=get-wmiobject -class computersystem ; $comp.rename("W7-clone-42")

That returns ReturnValue : 5 which after much searching translates to "Access Denied". The other suggestion on that page:

wmic computersystem rename "W7-Clone-42"

Returns the unhelpful "Invalid Verb Switch". Another internet source suggests the following formulation:

wmic computersystem where Name="W7-Img3-RPT49VA" call rename name="W7-clone-42"

Which gives alternately Invalid Verb Switch, or invalid parameter. As a test, I ran the above command directly on my management station rather than via invoke-command and also got access-denied.

Going old-school, I copied netdom to the target machine.

netdom renamecomputer W7-Img3-RPT49VA /newname:W7-clone-42

Which gives me 'access denied'.

Throughout this all, the credentials I've been testing with have Domain Admin. The intent is to crank the exact priv down once I've identified the workable methods. The Security event-log on the target machine definitely shows the successful logins throughout all of this.

The alternate method, handle the rename through the vSphere System Customization process, is still available. I even have an answer file for it, but I don't know how to prepare the template-machine to allow it to be used. Either way will get me what I need.

What am I missing? The wmic syntax is clearly wrong, but the other two methods return 'access denied' so I have low hopes of it working once the correct syntax is worked out. Is this a UAC interaction problem?

I'm in the process of staging up some new virtualization servers, and part of that is to get some higher-bandwidth pipes into them. The ultimate goal is to bind 4 GigE ports into a single trunk carrying 802.1q tagged traffic. I can get that far, however I've run into a strange problem. But first, a diagram.

----------       ----------  1GbE trunks 
|        | 10GbE |        | ------------- --------
|  SW1   |-------|   SW2  | ------------- | VM1  |
|        |       |        | ------------- --------
----------       ----------
     |                |  1GbE  -----------
     | 1GbE           |--------| client2 |
     |                         -----------
----------
|        | 1GbE -----------
|  SW3   |------| client1 |
|        |      -----------
----------

All the switches are HP ProCurve 2910al switches and are not stacked. Client2 in the above diagram is in the same VLAN as VM1 is. Client1 is in a different VLAN. For the VM machine (CentOS 6) both iptables and SELinux have been disabled.

My problem is that when trunking is involved, two-way network traffic is impossible when talking to either Client machine. TCPDUMP shows that pings are received by them and ECHO REPLY packets are sent, but the VM host never sees them. At the same time, if I try to ping the VM from a client machine, it also doesn't work. The fact I can't ping client2, which is on the same subnet, suggests something is screwy in the network layer somewhere.

Strangely, from the VM host I can ping the gateway IPs on any of the switches. If I use a single interface everything works fine both with and without VLAN tagging. If I just bind a single interface and turn VLAN tagging on that interface, I can go anywhere. Build a trunk, and I'm limited to the switch-fabric.

The type of trunk doesn't seem to matter. Right now they're configured with mode 0 trunks (balance-rr), though using LACP/802.1qa behaves the same way.

vlan 70 
   name "Virtualization Subnet" 
   untagged 35,36,38,40 
   tagged Trk1-Trk2,Trk5,Trk8 
   no ip address 
   jumbo 
   exit 

That's the VLAN config on SW2 up there. SW1's VLAN 70 definition has the "ip address" defined on it. The above snippet is in the fully-untrunked mode. When I'm trunked:

trunk 35-36,38,40 Trk16 trunk
vlan 70 
   name "Virtualization Subnet" 
   tagged Trk1-Trk2,Trk5,Trk8,Trk16
   no ip address 
   jumbo 
   exit 

The 802.1qa/LACP version trades out the trunk definition for trunk 35-36,38,40 Trk16 lacp but as I said, doesn't change the problem presentation.

Client2 is actually connected to SW1, but putting it there in the chart would have made formatting trickier. In any case, the only thing in the Interface stanza is a name directive; it is listed as an untagged port in the vlan 70 stanza for SW1.

What am I missing?

While doing some unrelated troubleshooting (at least I think so, shared-printer issues) I came across a set of Event Log entries that have me concerned.

Machine Name:  labcomputer82
Source: Security-Kerberos
Event ID: 4
Event Description:

The Kerberos client received a KRB_AP_ERR_MODIFIED error from the server labcomputer143$. The target name used was RPCSS/imagemaster4.ad.domain.edu. This indicates that the target server failed to decrypt the ticket provided by the client. This can occur when the target server principal name (SPN) is registered on an account other than the account the target service is using. Please ensure that the target SPN is registered on, and only registered on, the account used by the server. This error can also happen when the target service is using a different password for the target service account than what the Kerberos Key Distribution Center (KDC) has for the target service account. Please ensure that the service on the server and the KDC are both updated to use the current password. If the server name is not fully qualified, and the target domain (AD.DOMAIN.EDU) is different from the client domain (AD.DOMAIN.EDU), check if there are identically named server accounts in these two domains, or use the fully-qualified name to identify the server.

There are three machine names used in this message. It's generated on labcomputer82, it's attempting to talk to another lab workstation called labcomputer143, and the service in question (RPCSS) refers to the name of the machine that this machine was imaged from (and possibly also that of labcomputer143, I'm not sure). The thing that has me raising both eyebrows is that the machine named labcomputer82 is attempting to use an SPN of RPCSS/imagemaster4.ad.domain.edu.

The SPN attribute on the computer object in AD looks just fine. It has all the names it should have.

These machines are imaged using Ghost and (at least in this specific case) sysprep was not used. Of the over 3,000 computer objects in our AD domain, somewhere around 1,700 of them are computer-lab seats that are frequently imaged and as of September the majority were imaged using the Ghost/Profile-Copy method instead of the Ghost/sysprep method Microsoft recommends. If this error report is something major that Windows is quietly working around, perhaps kerberos is broken for these machines and it's failing back to NTLMv2, I'd like to know so I can add pressure in my drive for sysprep adoption.

We're trying to get Windows Media Services running, and so far it works well. However, we'd also like the ability to allow people to administer WMS without being in the Administrators group for the server in question. When we add a group to the local Administrators group things work as they should. Unfortunately, how to make this happen is not terribly well described in the documentation.

The closest I've come is a Microsoft MSDN article:

http://msdn.microsoft.com/en-us/library/dd892105(VS.85).aspx

Which doesn't seem to do what I need it to do, membership in 'Administrators' is still required. Is this the only way to accomplish this (meaning, I missed a step) or is there another way to do it?

This is running on Server 2008R2.

There are several things that are keeping IPv6 deployment from being a topic of active discussion here at my work. There are the usual technical issues, but one non-technical one appears to be a major stumbling block on the path to actually getting a deployment project going.

Addresses, memorizing of.

Specifically, IPv4 addresses are comprehensible, and IPv6 addresses just look like a big long string of hex. The human mind has real trouble memorizing lists of more than 7-8 items, and an IPv4 address (192.168.231.148) has four items in it which makes it easy for us to memorize. A fully populated IPv6 address has not only 8 sections, but each section has 4 hex digits in it. IPv6 addresses were not designed for memorization. To the technician who knows that the DNS server is at 192.168.42.42 (or more likely "42.42", since the company prefix is likely memorized), the idea of memorizing an IPv6 address fills them with dread. Which in turn makes them much less enthusiastic about participating in an IPv6 deployment project.

Because of how our network works we're not fully dynamic in terms of v4 addressing. We have several to many subnets that are entirely statically assigned for a variety of reasons, chief among them being that the overhead of static DHCP assignments is perceived as being too great. Also, some devices still aren't smart enough to pull DNS addresses out of DHCP while also having a static assignment, and therefore require manually configured DNS settings. Therefore, some v6 address memorization will have to be done.

We're not under any mandate to get v6 out the door, so we don't have pressure from the top. However, it is time to start prepping our infrastructure to handle IPv6 even if we don't convert wholesale.

For those of you who have been in IPv6-land for a while, what short-cut methods do you use to discuss or keep track of subnets and specific/critical IP addresses? If I can help reduce some of the dread surrounding IPv6 we might get the project going.

We have a user who has found an interesting misbehavior on our file-serving cluster. He was able to create a directory on his home directory using WinXP and not have it be visible by a Win7 machine. This goes both ways, he was able to create one with Win7 that XP can't see.

WinXP:

Win7:

The server is Server 2008 SP2 in a Failover Cluster. The server only sees the 'testing-7' directory in Explorer and command-line. I checked our backup software, which uses shadow-copy, and it only sees the 'testing-7' directory as well. However, 'Previous Versions' on an XP station happily shows previous versions of the 'testing-xp' directory that are quite visible and obviously usable.

I ran chkdsk against the volume in the off chance that there was some corruption lurking, but it didn't find anything.

Additionally, he was able to create a file called "Arrow.docx" in the same directory on both computers. In this document he saved different data. After rebooting everything, the win-7 machine saw the arrow.docx file with the win-7 data in it, and the WinXP machine saw the arrow.docx file with the WinXP data in it.

The machines used were fairly standard images in our Computer Labs (being in the middle of the XP to Win7 transition, we have both flavors at the moment). I can see all of the files he created on my own stations and they behave exactly like he said.

I've been able to replicate the problem, but it seems to only affect the one volume that we recently migrated to a new storage array. The migration method was pretty simple:

1. Create the new LUN, format it, give it a drive letter, and add it to the target resource.
2. Use robocopy to mirror the data to the new LUN
3. Use it again to capture the changed data
4. In Failover Manager I performed some 3-card-monty to get the drive-letters swapped
5. I ran a powershell script I wrote to copy the directory-quota data

That all seemed to work. But now this has come up. Something has gone deeply weird here, and I'm looking for suggestions.

Edit RE: Offline files
Changing the offline files to disabled on my station causes everything to show up normally (where in this case normal is "as Win7 sees the world"), with the formerly XP-only files just gone. However, on another admins station, he still can't make the Win7 files appear. This is closer! But it does raise the question of lost-data.

We just added a new Server 2008 (sp2) Domain Controller in a new Site, our first such config. It's over a VPN gateway WAN (10Mbit). Unfortunately it is displaying a strange network symptom. Connections to the SMB ports (TCP/139 and TCP/445) are being actively refused... if the connection is coming in on pure IPv4. If the incoming connection is coming by way of the 6to4 tunnel those connections establish and work just fine.

It isn't the Firewall, since this behavior can be replicated with the firewall turned off. Also, it's actually issuing RST packets to connection attempts; something that only happens with a Windows Firewall if there is a service behind a port and the service itself denies access. I doubt it's some firewall device on the wire, since the server this one replaced was running Samba and access to it from our main network functioned just fine.

I'm thinking it might have something to do with the Subnet lists in AD Sites & Services, but I'm not sure. We haven't put any IPv6 addresses in there, just v4, and it's the v4 connections that are being denied. Unfortunately, I can't figure this out. We need to be able to talk to this DC from the main campus. Is there some kind of site-based SMB-level filtering going on? I can talk to the DC's on campus just fine, but that's over that v6 tunnel. I don't have access to a regular machine on that remote subnet, which limits my ability to test.

I have an XP workstation on a remote subnet that is not able to download the group policies. The Event Log entries are pretty clear. Upon troubleshooting it is clear that the workstation isn't able to translate the domains DNS into something it can connect to.

\\our.ad.dns.domain\SysVol\our.ad.dns.domain\Policies{5310CCFF-43F3-4424-A7AC-96942D065331}\

Won't resolve. "net view \\our.ad.dns.domain" returns "network path not found".

And yet, nslookup on our.ad.dns.domain returns the IPs of all of the domain controllers. What's more, these work:

• \\addc1\SysVol\our.ad.dns.domain\Policies{5310CCFF-43F3-4424-A7AC-96942D065331}\
• \\addc2\SysVol\our.ad.dns.domain\Policies{5310CCFF-43F3-4424-A7AC-96942D065331}\
• \\addc3\SysVol\our.ad.dns.domain\Policies{5310CCFF-43F3-4424-A7AC-96942D065331}\

So it can talk to the DC's just fine once it figures out how to get there.

The problem initially presented as an inability to log in, which was traced down not being able to resolve one of the domain controller's name, "addc3". A "net view \\addc3" failed with "network path not found". Once we added an ADDC3 entry into WINS (yes we still have it) resolution started working.

So WINS resolution is working fine, but for some reason DNS lookups aren't being performed when resolving Windows addresses.

Unfortunately for all involved, this one subnet is served by an ISC DHCP server that I don't have access to. All but two of the rest of our subnets are served by Microsoft DHCP servers I DO have access to. So it could be a case of a bad DHCP option, but I can't get in to look, nor do I know the right questions to ask of the network guys.

We're in the process of moving some directories from NetWare to Windows, and have run into a difference of permissioning. Because NetWare makes this easy, we have whole volumes where no users have any rights at the top of the volume, and the first, second, and third tiers of directories are where the rights are granted. Due to how the NetWare trustee system worked, if you had access to a directory deep in the tree, you'd be able to browse to it from root with no problem. This had the handy side-effect of only showing the directories you have access to when you enumerate a directory you otherwise have no permissions to be in.

The 'only showing the directories you have access to' thing is resolved through Microsoft's Access Based Enumeration (ABE), and yea this is a good thing.

The problem we're having is figuring out what rights and security policies need to be set to in order to allow users to browse from a root share down to a directory they have access to. Examples make this easier to explain.

\\server\share\finance\audit\auditreports\HR-Q4-2007

The audit team grants rights for the HR Managers to the audit report directory above ("HR-Q4-2007"). Under NetWare, this would allow the HR managers to start at \\server\share\ and then browse through finance, audit, and auditreports, to get to the directory. One permission, and it just worked.

The "Bypass Traverse Checking" security policy means that the HR managers can map a drive directly to \\server\share\finance\audit\auditreports\HR-Q4-2007\ and it'd just work. That's not what we want, we want the user to be able to start at the top and browse down.

Does this require the use of the 'Traverse Folder' NTFS right to enable this? If so, it means a much more complex permission environment, but we can work it out. How is this problem solved when ABE is also in use?

I've been looking around for a hosting provider. In the process I ran into one that seemed to have a clear preference for apache 1.3. Considering that 1.3 was released something like 11 years ago, I am unclear why someone would have such a preference.

At work we still use 1.3 for our main web-site. But that's because 1.3 is still the default Apache server for Solaris 8, which is what's running said web-site. Internal aps are hosted on Apache 2.2 more and more often as they're moved to Linux (SLES10 and soon 11).

Heck, NetWare 6.5 ships with 2.0 as default, and you can download 2.2 if you really want to run it. 1.3 was the default on NW6.0.

But, I am not a web-master, so I don't know the compatibility details for why Apache 1.3 could be a preferred platform. I know Apache 'just works', which may be part of it. What is it about 1.3 that is attractive over its newer brethren?

Certain journaled file-systems do suffer from performance penalties for hosting the journal on the same drive as the filesystem itself. Using an external journal can improve write speeds. With the advent of SSD's, it strikes me as entirely possible that a single SSD could support multiple external journals for filesystems hosted on traditional rotational magnetic media. While the Anandtech "SSD Anthology" did cover some of this, they didn't cover the "multiple journal" test-case.

Journals tend to be pretty small, so even a small, fast 32GB SSD could provide a lot of speed for multiple large file-systems. It would probably wear much faster than general I/O patterns out suggest, which is why having a larger device than you'll end up using is a good idea.

Have any of you done something like this, or even just used an external journal? I'm curious about real-world cases.