wfaulk's questions

The SPF specification says:

The published SPF record for a given domain name SHOULD remain small enough that the results of a query for it will fit within 512 octets. Otherwise, there is a possibility of exceeding a DNS protocol limit.

…

Note that when computing the sizes for replies to queries of the TXT format, one has to take into account any other TXT records published at the domain name.

It also points out that more recent DNS specifications allow for larger UDP responses (the reason for the limitation, as the SPF spec implies you should not rely on DNS over TCP working), but that doesn't really seem to override the "SHOULD".

The problem is that so many organizations require TXT records on the same domain for verification purposes (things like facebook-domain-verification, google-site-verification, atlassian-domain-verification, adobe-sign-verification, etc.) and can quickly pump the size of the total TXT RRset well over 512 bytes.

It looks like the majority of big organizations are complying with this, but there are a few that go over:

% dig +noall +stats netflix.com TXT | grep 'MSG SIZE'
;; MSG SIZE  rcvd: 593

% dig +noall +stats linkedin.com TXT | grep 'MSG SIZE'
;; MSG SIZE  rcvd: 632

% dig +noall +stats twitter.com TXT | grep 'MSG SIZE'
;; MSG SIZE  rcvd: 642

% dig +noall +stats microsoft.com TXT | grep 'MSG SIZE'
;; MSG SIZE  rcvd: 1459

(You can see the potential truncation happening by running something like dig +notcp +noedns +ignore microsoft.com TXT.)

I've been right up against the edge for six months, and now I need to add another verification record for a new vendor that will push me well past 512 bytes. I've done as much as I can to consolidate my SPF record, and I've made sure that I can't remove existing verification records.

What should I do here? I can't not have the verification records, but I don't want to ignore the SPF spec, either. That said, Microsoft seems to be ignoring it, and I don't think they're getting their mail rejected.

I am attempting to do conditional forwarding of a particular zone to a set of Amazon Route 53 nameservers that are authoritative for that zone. When I try to add the nameservers, it fails validation, with the not-so-useful error message "A timeout occurred during validation".

If I attempt to query records in the zone from that nameserver from the command line (via something like nslookup host.example.com ns-000.awsdns-00.com), it works properly.

The error prevents me from being able to click "OK" and create the conditional forwarding zone at all.

If I have multiple VMware ESXi servers controlled by vCenter that share a datastore over FibreChannel, what is the best practice for zoning those connections?

I see two obvious ways to do this:

• Create one zone per server, each zone containing the server and the datastore
• Create one big zone that contains all of the servers and the datastore

Is one better than the other? Or is there another way that makes more sense?

How does any of that change if there are multiple datastores?

Yesterday I babysat a Windows 2008 R2 server for three hours applying multiple rounds of Windows Updates. This is not a good use of my time.

This is a server that cannot apply updates automatically; I have to schedule maintenance windows in order to reboot the server. I want to bring Windows completely up to date during that maintenance window, which often means multiple rounds of a check-for-updates/download-updates/apply-updates/reboot cycle.

I know that I can download updates and apply them manually, but that doesn't really help when Windows determines after the first round of updates that it needs to apply even more updates.

The biggest problem is that it can easily take ten minutes for Windows to determine what updates it needs to apply. It can then take easily another ten minutes to download these updates. Then it can take ten minutes to reboot and finish the update installation. If it decides that you need three rounds of updates, that's an hour and a half, and that assumes that Windows doesn't freak out and make one of those steps take forever, or fail, making you start over.

Is there any way to determine all of the updates that would be needed to bring a Windows system up to date beforehand, so that I can just apply them all by hand and avoid having to wait for Windows to perform its interminably long checks and ridiculously slow downloads multiple times during my maintenance window.

(I feel like this should be a FAQ, but I can't find it.)

I have a FreeBSD 10.0 system on which I have MySQL running with its InnoDB database files stored on ZFS. It seemed to have been going fine for months, but lately, and seemingly suddenly, performance has bottomed out. After debugging a while surrounding MySQL, I finally decided to just see if just reading the database files from the filesystem was slow.

I picked one table whose database file was about 16GB, ran:

time cat table.ibd > /dev/null

and got:

cat table.ibd > /dev/null 1.24s user 64.35s system 0% cpu 1:00:34.65 total

In comparison, a copy of the file (with some changes, I'm sure: the database is live) living on UFS on the same system gives me:

cat table.ibd > /dev/null 0.20s user 9.34s system 5% cpu 9.550 total

Here are non-default ZFS options on this system:

/boot/loader.conf:
  vfs.zfs.arc_max=17179869184

-

/etc/sysctl.conf:
  vfs.zfs.prefetch_disable=1

-

zfs get:
  recordsize 16K
  compression on
  atime off
  primarycache metadata
  zfs:zfs_nocacheflush 1

There are 12 snapshots on this filesystem right now. (Which seems excessive; I'm going to see if any of them can be deleted.)

There's no L2ARC for this pool (or any other pool on the system).

I've tried all three values for primarycache, and I've tried reenabling the prefetch, none of which seem to have had any significant effect.

The zpool is four 2-disk mirrors:

% zpool status mysqlrot
  pool: mysqlrot
state: ONLINE
  scan: scrub repaired 0 in 9h45m with 0 errors on Fri Jun 26 12:46:33 2015
config:

    NAME        STATE     READ WRITE CKSUM
    mysqlrot    ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        mfid9   ONLINE       0     0     0
        mfid10  ONLINE       0     0     0
      mirror-1  ONLINE       0     0     0
        mfid11  ONLINE       0     0     0
        mfid12  ONLINE       0     0     0
      mirror-2  ONLINE       0     0     0
        mfid13  ONLINE       0     0     0
        mfid14  ONLINE       0     0     0
      mirror-3  ONLINE       0     0     0
        mfid15  ONLINE       0     0     0
        mfid16  ONLINE       0     0     0
    spares
      mfid19    AVAIL

errors: No known data errors

One oddity is how the raw devices are set up. The storage controller is an MFI controller, and each raw disk is actually configured in the MFI controller as a 1-disk RAID0 volume:

% sudo mfiutil show volumes
mfi0 Volumes:
  Id     Size    Level   Stripe  State   Cache   Name
 mfid0 (  185G) RAID-1      64K OPTIMAL Disabled <OS>
 mfid1 (  558G) RAID-0      64K OPTIMAL Disabled <DB0A>
 mfid2 (  558G) RAID-0      64K OPTIMAL Disabled <DB0B>
 mfid3 (  558G) RAID-0      64K OPTIMAL Disabled <DB1A>
 mfid4 (  558G) RAID-0      64K OPTIMAL Disabled <DB1B>
 mfid5 (  558G) RAID-0      64K OPTIMAL Disabled <DB2A>
 mfid6 (  558G) RAID-0      64K OPTIMAL Disabled <DB2B>
 mfid7 (  558G) RAID-0      64K OPTIMAL Disabled <DB3A>
 mfid8 (  558G) RAID-0      64K OPTIMAL Disabled <DB3B>
 mfid9 (  558G) RAID-0      64K OPTIMAL Disabled <DB4A>
mfid10 (  558G) RAID-0      64K OPTIMAL Disabled <DB4B>
mfid11 (  558G) RAID-0      64K OPTIMAL Disabled <DB5A>
mfid12 (  558G) RAID-0      64K OPTIMAL Disabled <DB5B>
mfid13 (  558G) RAID-0      64K OPTIMAL Disabled <DB6A>
mfid14 (  558G) RAID-0      64K OPTIMAL Disabled <DB6B>
mfid15 (  558G) RAID-0      64K OPTIMAL Disabled <DB7A>
mfid16 (  558G) RAID-0      64K OPTIMAL Disabled <DB7B>
mfid17 (  558G) RAID-0      64K OPTIMAL Disabled <DB8A>
mfid18 (  558G) RAID-0      64K OPTIMAL Disabled <DB8B>
mfid19 (  558G) RAID-0      64K OPTIMAL Disabled <SPARE0>

No errors that I've been able to find in any log file or utility.

Anyone have any ideas where to look?

Various data:

% zpool list mysqlrot
NAME       SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
mysqlrot  2.17T  1.49T   701G    68%  1.00x  ONLINE  -

Copying 15.3GB file to the affected filesystem (from a UFS filesystem):

% time sudo cp test.file /var/lib/mysql/mysqlrot/test.file
sudo cp test.file /var/lib/mysql/mysqlrot/test.file  0.02s user 44.23s system 1% cpu 1:06.93 total

(That's 66.93 seconds.)

Reading same file from the affected filesystem:

# time cat test.file > /dev/null
cat test.file > /dev/null  4.23s user 268.50s system 0% cpu 25:29.27 total

(That's 1529.27 seconds: almost 23 times longer than the write.)

Interestingly, cp performs far better than cat:

% time sudo cp /var/lib/mysql/v4netrot/test.file /dev/null
sudo cp /var/lib/mysql/v4netrot/test.file /dev/null  0.03s user 33.63s system 0% cpu 3:05.99 total

(185.99 seconds)

iostat data during copy to:

                        extended device statistics
device     r/s    w/s    kr/s    kw/s qlen svc_t  %b
mfid0      0.0    0.6     0.0     8.8    0   0.6   0
mfid1      0.4   12.6     2.7    70.5    0   0.5   0
mfid2      1.8   12.8    10.8    70.5    0   1.6   2
mfid3      0.2   10.0     1.5    92.4    0   0.7   0
mfid4      0.0   10.4     0.0    92.4    0   0.5   0
mfid5      0.2    9.4     1.2    39.7    0   0.5   0
mfid6      0.6    9.8     3.9    39.7    0   0.6   0
mfid7      0.6    4.8     0.3    46.1    0   0.9   0
mfid8      1.8    4.8    11.4    46.1    0   0.8   0
mfid9      0.4 1327.2     2.9 26686.5    0   0.5  23
mfid10     0.8 1328.2     1.8 26686.5    0   0.5  20
mfid11     1.4 1304.8     8.4 26357.6    0   0.5  23
mfid12     1.4 1304.6     2.6 26357.6    0   0.6  31
mfid13     1.6 1120.6     3.5 26194.2    0   0.6  25
mfid14     0.4 1122.6     2.7 26194.2    0   0.5  22
mfid15     0.8 1406.6     5.5 26188.5    0   0.5  22
mfid16     1.0 1174.6     2.0 21534.3   10   4.9  74
mfid17     5.8   24.2   152.9   300.6    0   0.3   0
mfid18     4.0   23.6    76.7   300.6    0   0.3   0
mfid19     0.0    0.0     0.0     0.0    0   0.0   0

The kw/s data ranged from about 17k to about 25k, and was pretty consistent between drives.

iostat when catting from:

                        extended device statistics
device      r/s   w/s    kr/s    kw/s qlen svc_t  %b
mfid0       0.0   0.0     0.0     0.0    0   0.0   0
mfid1       0.4  17.2    11.4    63.4    0   0.5   0
mfid2       0.0  17.0     0.0    63.4    0   0.4   0
mfid3       0.0  14.0     0.0    56.4    0   0.4   0
mfid4       0.4  13.6     0.2    56.4    0   0.4   0
mfid5       0.8   9.6     4.8    37.3    0   0.8   0
mfid6       0.0   9.8     0.0    37.3    0   0.4   0
mfid7       0.2   3.8    17.2    11.9    0   0.6   0
mfid8       0.2   3.8     1.4    11.9    0   0.5   0
mfid9    1208.8   0.0  6831.4     0.0    0   0.1  11
mfid10    129.4   0.0   780.7     0.0    0   0.2   2
mfid11    906.4   0.0  5858.5     0.0    0   0.1  10
mfid12    600.5   0.0  2673.0     0.0    0   0.1   5
mfid13    136.2   0.0   803.9     0.0    0   0.2   3
mfid14    316.1   0.0  1895.3     0.0    0   0.1   4
mfid15    243.6   0.0  1414.5     0.0    0   0.1   2
mfid16    129.0   0.0   768.8     0.0    0   0.2   2
mfid17      3.8  25.8    29.8   274.1    0   0.2   0
mfid18      6.0  25.6    96.6   274.1    0   0.2   0
mfid19      0.0   0.0     0.0     0.0    0   0.0   0

the kr/s numbers were wildly inconsistent, but those are representative numbers.

iostat while cping from (to /dev/null):

                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
mfid0      0.0   0.0     0.0     0.0    0   0.0   0
mfid1     21.0  66.6   107.6  2351.7    0   0.9   6
mfid2     17.6  66.8   106.8  2351.7    0   1.0   6
mfid3     17.6  39.0   116.9  2111.3    0   1.1   6
mfid4     18.8  39.6    99.8  2111.3    0   1.3   7
mfid5     23.2  62.4   172.2  2076.1    0   1.1   7
mfid6     23.0  62.0   130.0  2076.1    0   1.4   9
mfid7     16.2  62.6   112.6  2125.3    0   1.0   6
mfid8     17.4  63.0   107.6  2125.3    0   0.7   4
mfid9    237.5  44.6  5140.6   807.0    0   3.1  22
mfid10   263.7  43.6  5530.5   807.0    0   1.5  14
mfid11   252.7  55.8  5297.6   802.4    0   2.6  20
mfid12   298.1  55.6  5361.9   802.4    0   2.5  21
mfid13   275.3  46.2  5116.4   801.4    0   2.8  22
mfid14   252.9  42.4  5107.7   801.4    2   3.1  21
mfid15   270.9  43.8  4546.5   943.7    0   1.2  12
mfid16   257.7  44.0  5642.5   943.7    0   2.5  19
mfid17     7.8  23.0    73.1   244.9    0   0.3   0
mfid18     0.8  24.2    44.4   244.9    0   0.2   0
mfid19     0.0   0.0     0.0     0.0    0   0.0   0

It does not seem CPU-bound. top shows that a cp from the affected filesystem to /dev/null consumes about 18% of one core (out of 48) and the rest of the cores are showing over 95% idle.

last pid: 12474;  load averages:  1.65,  1.26,  1.14                                                                                                                                        up 39+05:42:19  14:29:08
147 processes: 1 running, 146 sleeping
CPU 0:   0.0% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.4% idle
CPU 1:   0.0% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.8% idle
CPU 2:   0.4% user,  0.0% nice,  0.8% system,  0.0% interrupt, 98.8% idle
CPU 3:   0.0% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.8% idle
CPU 4:   0.0% user,  0.0% nice,  2.3% system,  0.0% interrupt, 97.7% idle
CPU 5:   0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 6:   0.0% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.8% idle
CPU 7:   0.0% user,  0.0% nice,  2.7% system,  0.0% interrupt, 97.3% idle
CPU 8:   0.4% user,  0.0% nice,  1.9% system,  0.0% interrupt, 97.7% idle
CPU 9:   0.0% user,  0.0% nice,  2.3% system,  0.0% interrupt, 97.7% idle
CPU 10:  0.4% user,  0.0% nice,  1.9% system,  1.2% interrupt, 96.5% idle
CPU 11:  0.0% user,  0.0% nice,  3.9% system,  0.0% interrupt, 96.1% idle
CPU 12:  0.0% user,  0.0% nice,  1.6% system,  0.8% interrupt, 97.7% idle
CPU 13:  0.0% user,  0.0% nice,  2.7% system,  0.0% interrupt, 97.3% idle
CPU 14:  0.0% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.4% idle
CPU 15:  0.4% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.4% idle
CPU 16:  0.0% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.4% idle
CPU 17:  0.4% user,  0.0% nice,  3.1% system,  0.0% interrupt, 96.5% idle
CPU 18:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 19:  0.8% user,  0.0% nice,  0.8% system,  0.0% interrupt, 98.4% idle
CPU 20:  0.0% user,  0.0% nice,  2.7% system,  0.0% interrupt, 97.3% idle
CPU 21:  0.4% user,  0.0% nice,  1.9% system,  0.0% interrupt, 97.7% idle
CPU 22:  0.0% user,  0.0% nice,  2.3% system,  0.0% interrupt, 97.7% idle
CPU 23:  0.4% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.4% idle
CPU 24:  0.0% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.8% idle
CPU 25:  0.8% user,  0.0% nice,  0.8% system,  0.0% interrupt, 98.4% idle
CPU 26:  0.4% user,  0.0% nice,  0.8% system,  0.0% interrupt, 98.8% idle
CPU 27:  0.0% user,  0.0% nice,  4.7% system,  0.0% interrupt, 95.3% idle
CPU 28:  0.0% user,  0.0% nice,  0.8% system,  0.0% interrupt, 99.2% idle
CPU 29:  0.4% user,  0.0% nice,  0.8% system,  0.0% interrupt, 98.8% idle
CPU 30:  0.0% user,  0.0% nice,  2.7% system,  0.0% interrupt, 97.3% idle
CPU 31:  0.0% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.8% idle
CPU 32:  0.0% user,  0.0% nice,  0.8% system,  0.0% interrupt, 99.2% idle
CPU 33:  0.0% user,  0.0% nice,  3.5% system,  0.0% interrupt, 96.5% idle
CPU 34:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 35:  0.0% user,  0.0% nice,  0.4% system,  0.0% interrupt, 99.6% idle
CPU 36:  1.2% user,  0.0% nice,  1.6% system,  0.0% interrupt, 97.3% idle
CPU 37:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 38:  0.4% user,  0.0% nice,  1.2% system,  0.4% interrupt, 98.1% idle
CPU 39:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
CPU 40:  0.0% user,  0.0% nice,  1.9% system,  0.0% interrupt, 98.1% idle
CPU 41:  0.0% user,  0.0% nice,  1.2% system,  0.0% interrupt, 98.8% idle
CPU 42:  0.4% user,  0.0% nice,  1.9% system,  0.0% interrupt, 97.7% idle
CPU 43:  0.0% user,  0.0% nice,  4.7% system,  0.0% interrupt, 95.3% idle
CPU 44:  0.0% user,  0.0% nice,  2.3% system,  0.0% interrupt, 97.7% idle
CPU 45:  0.4% user,  0.0% nice,  2.7% system,  0.0% interrupt, 96.9% idle
CPU 46:  0.4% user,  0.0% nice,  3.5% system,  0.0% interrupt, 96.1% idle
CPU 47:  0.4% user,  0.0% nice,  1.6% system,  0.0% interrupt, 98.1% idle
Mem: 82G Active, 23G Inact, 15G Wired, 3340K Cache, 1655M Buf, 4858M Free
ARC: 12G Total, 527M MFU, 11G MRU, 4375K Anon, 377M Header, 89M Other
Swap: 4096M Total, 12M Used, 4084M Free

  PID USERNAME     PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
12461 root          31    0 12268K  2552K zio->i 30   0:07  18.16% cp
 3151 mysql         24    0 92309M 92073M uwait   3  22:06   7.47% mysqld{mysqld}
 3151 mysql         22    0 92309M 92073M select 33  15:30   4.79% mysqld{mysqld}
 3151 mysql         20    0 92309M 92073M uwait  46 895:41   0.49% mysqld{mysqld}
12175 wfaulk        20    0 23864K  6404K CPU25  25   0:03   0.29% top
 6074 root          20    0 84348K 40372K kqread 25   0:11   0.20% vc-aggregator{vc-aggregator}

(The rest of the processes show 0.00% CPU utilization.)

I'm trying to set up a RAID10 set on an LSI MegaRAID controller. It is unclear to me how the RAID controller will actually lay out the RAID set on the physical drives.

Part of the problem is that MegaRAID seems to use terms very inconsistently; the same idea often has multiple terms, and it seems as if sometimes they use the same term to refer to multiple ideas. I'm going to try to use the terms that the MegaCli command seems to use most frequently.

MegaRAID requires that there be two to eight arrays within a RAID10 set. Each array must contain physical drives in multiples of two. Each array in the RAID10 set must have the same number of physical drives.

Is each array a RAID10 set, and then the arrays are joined together? If so, does the fact that the arrays have to be the same size imply that the arrays are being striped? If so, given that each array is striped, and then there's an additional layer of striping on top of that, should I be concerned about that redundancy in regards to performance? (Or would it be a good thing?)

If each array is a RAID10 set, though, why does MegaRAID require that you have at least two of them?

If each array is not a RAID10 set, why does it require that arrays have physical drives in multiples of two?

In the documentation, it refers to arrays as spans, which it elsewhere defines like this:

Disk spanning allows multiple drives to function like one big drive. Spanning overcomes lack of disk space and simplifies storage management by combining existing resources or adding relatively inexpensive resources. For example, four 20 GB drives can be combined to appear to the operating system as a single 80 GB drive. Spanning alone does not provide reliability or performance enhancements. Spanned virtual drives must have the same stripe size and must be contiguous.

Which, to me, implies concatenation, or, at best, striping. Let's be generous and call it RAID0.

So if I have to define two arrays, and arrays are spans, and spans are RAID0, then I'm defining two RAID0 sets. And if I have to define two of them, it'd make sense that it's mirroring those. But that would be a RAID1 of two RAID0s, or RAID0+1, which is bad.

I'd expect, then, that if I defined three arrays, that would be creating a three-way mirror of RAID0 sets, but if I actually do that, the logical drive still has half the capacity of all the drives in total, not a third. So that conclusion doesn't make sense, either.

The problem is that I have 18 drives I want in a RAID10 set. In every other RAID system I've used, I'd just create 9 two-drive mirrors and then stripe them, but I can't do that with MegaRAID. Because there have to be between two and eight arrays, and each array must have an even number of drives, the only config I can come up with that works is three arrays of six drives each, but I just feel strange doing that without having a better notion of how those disks are actually going to be laid out.

Am I overthinking this? Should I just let MegaRAID do its thing and just hope that the drives are laid out optimally?

I have some HP switches that I log into via ssh. The switches send a terminal command to disable line wrap ("rmam" in terminfo parlance), but then fail to reenable it, which screws up my terminal after I exit the ssh session. I can fix the terminal by running tput smam.

Is there any way to get ssh to automatically run that command after my ssh session ends?

It wouldn't kill me to run it as a shell autocommand, or alias ssh to always run that command afterwards, but I'd prefer to solve the problem via ssh so that I can limit the command to just be run after I connect to known-bad hosts.

My ssh client is OpenSSH_6.2p2, but I can change or update if there's a new feature somewhere.

I have servers named like server.prod.example.com, and I regularly log into them as server.prod. Recently, these hostnames started resolving to 127.0.53.53.

It turns out that ICANN recently enabled the .prod TLD. In addition, every request that goes to the .prod nameservers get resolved to 127.0.53.53 instead of coming back as NXDOMAIN, which would allow resolution to continue to work properly. (I'm guessing the point behind this is to let people know that their stuff will break worse before those start resolving to something real.)

How can I avoid having to type in my domain name for every host like this?

Is this still biting you occasionally? I couldn't find a list of new TLDs and when they were added, so I set one up myself: https://twitter.com/newgtldannounce

In my experience, files that have the file descriptor of txt in lsof output are the executable file itself and shared objects. The lsof man page says that it means "program text (code and data)".

While debugging a problem, I found a large number of data files (specifically, ElasticSearch database index files) that lsof reported as txt. These are definitely not executable files. The process was ElasticSearch itself, which is a java process, if that helps point someone in the right direction.

I want to understand how this process is opening and using these files that gets it to be reported in this way. I'm trying to understand some memory utilization, and I suspect that these open files are related to some metrics I'm seeing in some way.

The system is Solaris 10 x86.

I am running a JVM to support ElasticSearch. I am still working on sizing and tuning, so I left the JVM's max heap size at ElasticSearch's default of 1GB. After putting data in the database, I find that the JVM's process is showing 50GB in SIZE in top output. It appears that this is actually causing performance problems on the system; other processes are having trouble allocating memory.

In asking the ElasticSearch community, they suggested that it's "just" filesystem caching. In my experience, filesystem caching doesn't show up as memory used by a particular process. Of course, they may have been talking about something other than the OS's filesystem cache, maybe something that the JVM or ElasticSearch itself is doing on top of the OS. But they also said that it would be released if needed, and that didn't seem to be happening.

So can anyone help me figure out how to tune the JVM, or maybe ElasticSearch itself, to not use so much RAM.

System is Solaris 10 x86 with 72GB RAM. JVM is "Java(TM) SE Runtime Environment (build 1.7.0_45-b18)".

I'd like to record data about system activity under FreeBSD for future analysis. If I were running a SysV system, I'd just sar and its related utilities, but that doesn't exist in the BSDs. (And bsdsar has gone missing.) I don't really care about specific formats or utilities, but I want to keep information about CPU, RAM, and VM utilization, as well as I/O information. At the same time, full acct process accounting both seems like overkill and doesn't really provide data I want: notably, data about preexisting processes.

I could just log the output from long-running vmstat, iostat, etc. commands, but it seems like there ought to be something tidier.

Ultimately, the question is:

What is a good way to find out recent past system utilization under FreeBSD?

I have a Solaris 10 x86_64 system running MySQL 5.5. Under heavy usage times, we're getting very slow responses from the database: slow queries running into minutes that normally return in sub-second times. CPU utilization is in the 60-70% range. Load average regularly gets well into the 20s, infrequently into the 40s, and I've seen it up to the 50s. (Two four-core CPUs with HyperThreading enabled.) It acts like an I/O problem, as if it's waiting on disks to write, but I'm not seeing any indications that there is any actual I/O problem. Average disk wait times are consistently 0, average wait queues are in the 0.2-0.3 range, and disk busy percentages occasionally creep into the 15% area. (All of this as according to sar.)

The storage is a zfs zpool of 5 zdev mirrors of two SAS drives. I do not have an intent log device, but I don't see that as being an issue with this workload.

What am I missing?

I have a set of eight HP ProCurve 2910al-48G Ethernet switches at my datacenter that are set up in a star topology with no physical loops. I want to partially mesh the switches for redundancy and manage the loops with a spanning-tree protocol.

However, our connection to the datacenter is provided by two uplinks, each to a Cisco 3750. The datacenter's switches are handling the redundant connection using PVST spanning-tree, which is a Cisco-proprietary spanning-tree implementation that my HP switches do not support.

It appears that my switches are not participating in the datacenter's spanning-tree domain, but are blindly passing the BPDUs between the two switchports on my side, which enables the datacenter's switches to recognize the loop and put one of the uplinks into the Blocking state. This is somewhat supposition, but I can confirm that, while my switches say that both of the uplink ports are forwarding, only one is passing any real quantity of data. (I am assuming that I cannot get the datacenter to move away from PVST. I don't know that I'd want them to make that significant of a change anyway.)

The datacenter has also sent me this output from their switches (which I have expurgated of any identifiable info):

3750G-1#sh spanning-tree vlan nnn

VLAN0nnn
  Spanning tree enabled protocol ieee
  Root ID    Priority    10
             Address     00d0.0114.xxxx
             Cost        4
             Port        5 (GigabitEthernet1/0/5)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32mmm  (priority 32768 sys-id-ext nnn)
             Address     0018.73d3.yyyy
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/5             Root FWD 4         128.5    P2p
Gi1/0/6             Altn BLK 4         128.6    P2p
Gi1/0/8             Altn BLK 4         128.8    P2p

and:

3750G-2#sh spanning-tree vlan nnn

VLAN0nnn
  Spanning tree enabled protocol ieee
  Root ID    Priority    10
             Address     00d0.0114.xxxx
             Cost        4
             Port        6 (GigabitEthernet1/0/6)
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32mmm  (priority 32768 sys-id-ext nnn)
             Address     000f.f71e.zzzz
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Gi1/0/1             Desg FWD 4         128.1    P2p
Gi1/0/5             Altn BLK 4         128.5    P2p
Gi1/0/6             Root FWD 4         128.6    P2p
Gi1/0/8             Desg FWD 4         128.8    P2p

The uplinks to my switches are on Gi1/0/8 on both of their switches. The uplink ports are configured with a single tagged VLAN. I am also using a number of other tagged VLANs in my switch infrastructure. And, to be clear, I am passing the tagged VLAN I'm receiving from the datacenter to other ports on other switches in my infrastructure.

My question is: how do I configure my switches so that I can use a spanning tree protocol inside my switch infrastructure without breaking the datacenter's spanning tree that I cannot participate in?

I occasionally find myself in a situation where an undermaintained system has an account that's been locked out. The problem is that there are a variety of ways in which an account can be locked out, each with their own method of being unlocked.

It's not that the account is being locked improperly, just unexpectedly, but finding the correct lock to reset is difficult.

My most recent attack of this problem was on a SUSE system, and it turned out that the password had expired (which wasn't initially known because the login attempts were not through a system that provided that sort of feedback), and then also locked due to failed login attempts.

Is there a list somewhere of all of the different possible account locks and how to disable them? I'm intending for actual brokenness, such as home directory access problems, corrupt PAM libraries, etc., to be out of scope for this question.

There is a lot of community feeling about what Linux distributions are appropriate for production server environments and which aren't, however, a lot of this feeling seems religiously based, and seldom presented with supporting evidence.

Assuming that we were trying to select a Linux distribution to standardize on (because we have an interest in keeping our environments as homogeneous as possible), what criteria are important, and how do you make determinations about how well different distributions meet those criteria?

Windows Server provides a certificate authority service. However, it's not clear from its documentation how (or if) the root certificate gets distributed to clients.

• Do domain member computers automatically trust the root certificate?
  • If so, how and when do they get the certificate?
• Is there any user interaction required for the root certificate to be installed or trusted?
• Does the client poll Active Directory? Is it in AD DNS?
• Will it only get it during login?
• What if a domain member remotely VPNs into the LAN?
• Are there any caveats for different versions of Windows clients?

I am trying to connect a Windows 7 Ultimate machine to a Windows 2k8 domain and it's not working. I get this error:

Note: This information is intended for a network administrator. If you are not your network's administrator, notify the administrator that you received this information, which has been recorded in the file C:\Windows\debug\dcdiag.txt.

DNS was successfully queried for the service location (SRV) resource record used to locate a domain controller for domain "example.local":

The query was for the SRV record for _ldap._tcp.dc._msdcs.example.local

The following domain controllers were identified by the query:
dc1.example.local
dc2.example.local

However no domain controllers could be contacted.

Common causes of this error include:

• Host (A) or (AAAA) records that map the names of the domain controllers to their IP addresses are missing or contain incorrect addresses.

• Domain controllers registered in DNS are not connected to the network or are not running.

The client is in an office connected remotely via MPLS to the data center where our domain controllers exist. I don't seem to have anything blocking connectivity to the DCs, but I don't have total control over the MPLS circuit, so it's possible that there's something blocking connectivity.

I have tried multiple clients (Win7 Ultimate and WinXP SP3) in the one office and get the same symptoms on all of them.

I have no trouble connecting to either of the domain controllers, though I have, admittedly, not tried every possible port. ICMP, LDAP, DNS, and SMB connections all work fine.

Client DNS is pointing to the DCs, and "example.local" resolves to the two IP addresses of the DCs.

I get this output from the NetLogon Test command line utility:

C:\Windows\System32>nltest /dsgetdc:example.local
Getting DC name failed: Status = 1355 0x54b ERROR_NO_SUCH_DOMAIN

I have also created a separate network to emulate that office's configuration that's connected to the DC network via LAN-to-LAN VPN instead of MPLS. Joining Windows 7 computers from that remote network works fine.

The only difference I can find between the two environments is the intermediate connectivity, but I'm out of ideas as to what to test or how to do it. What further steps should I take?

(Note that this isn't actually my client workstation and I have no direct access to it; I'm forced to do remote hands access to it, which makes some of the obvious troubleshooting methods, like packet sniffing, more difficult. If I could just set up a system there that I could remote into, I would, but requests to that effect have gone unanswered.)

2011-08-25 update:
I had DCDIAG.EXE run on a client attempting to join the domain:

C:\Windows\System32>dcdiag /u:example\adminuser /p:********* /s:dc2.example.local

Directory Server Diagnosis

Performing initial setup:
   Ldap search capabality attribute search failed on server
   dc2.example.local, return value = 81

This sounds like it was able to connect via LDAP, but the thing that it was trying to do failed. But I don't quite follow what it was trying to do, much less how to reproduce it or resolve it.

2011-08-26 update:
Using LDP.EXE to try and make an LDAP connection directly to the DCs results in these errors:

ld = ldap_open("10.0.0.1", 389);
Error <0x51>: Fail to connect to 10.0.0.1.
ld = ldap_open("10.0.0.2", 389);
Error <0x51>: Fail to connect to 10.0.0.2.
ld = ldap_open("10.0.0.1", 3268);
Error <0x51>: Fail to connect to 10.0.0.1.
ld = ldap_open("10.0.0.2", 3268);
Error <0x51>: Fail to connect to 10.0.0.2.

This would seem to point fingers at LDAP connections being blocked somewhere. (And 0x51 == 81, which was the error from DCDIAG.EXE from yesterday's update.) I could swear I tested this using TELNET.EXE weeks ago, but now I'm thinking that I may have assumed that its clearing of the screen was telling me that it was waiting and not that it had connected.

I'm tracking down LDAP connectivity problems now. This update may become an answer.

I have a RHEL5 workstation that has recently started to "hiccup". About every thirty seconds, it apparently completely stops execution for about 4 seconds. Seemingly nothing runs during that period. Long term processes seem to catch up to their input, but new processes simply don't get started.

Concrete examples:

• I have this loop running in a shell:

  while date; do
     sleep 0.2
  done
  

  Output merely skips over the missing seconds:

  Fri Aug 13 15:20:29 EDT 2010
  Fri Aug 13 15:20:29 EDT 2010
  Fri Aug 13 15:20:29 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:30 EDT 2010
  Fri Aug 13 15:20:34 EDT 2010
  Fri Aug 13 15:20:34 EDT 2010
  Fri Aug 13 15:20:35 EDT 2010
  Fri Aug 13 15:20:35 EDT 2010
  Fri Aug 13 15:20:35 EDT 2010
  

• If typing in a terminal, either local console or remote via ssh or telnet, echoback pauses during the unresponsive time, but catches back up when it starts responding again, with apparently no loss of input, just lag.

• pings go unresponded-to during the unresponsive time, but are responded to when it comes back:

  64 bytes from xxx: icmp_seq=1911 ttl=64 time=0.203 ms  
  64 bytes from xxx: icmp_seq=1912 ttl=64 time=0.199 ms  
  64 bytes from xxx: icmp_seq=1913 ttl=64 time=3202 ms  
  64 bytes from xxx: icmp_seq=1914 ttl=64 time=2196 ms  
  64 bytes from xxx: icmp_seq=1915 ttl=64 time=1197 ms  
  64 bytes from xxx: icmp_seq=1916 ttl=64 time=195 ms  
  64 bytes from xxx: icmp_seq=1917 ttl=64 time=0.201 ms  
  64 bytes from xxx: icmp_seq=1918 ttl=64 time=0.206 ms
  

  This would seem to imply that it is actually receiving input during the unresponsive period, as those ICMP packets are not being retransmitted.

• vmstat 1 output also delays, but does not catch up. It's almost as if those few seconds didn't happen. It also shows an uptick in waiting processes, and a downtick in interrupts and context switches:

  procs -----------memory----------  ---swap-- -----io---- --system-- -----cpu------
   r  b   swpd   free   buff  cache    si   so    bi    bo    in   cs us sy  id wa st
   0  0    132 3111220 305540 588012    0    0     0     0  1035  151  1  1  99  0  0
   0  0    132 3111096 305540 588012    0    0     0     0  1019  125  0  0  99  0  0
   0  0    132 3111220 305540 588012    0    0     0    44  1034  154  0  1  99  0  0
   1  0    132 3111096 305540 588012    0    0     0     0  1016  131  0  0  99  0  0
   6  0    132 3111096 305540 588012    0    0     0     0   417   82  0  0 100  0  0
   0  0    132 3111220 305540 588012    0    0     0     0  1041  155  0  1  99  0  0
   0  0    132 3111096 305540 588012    0    0     0     0  1019  123  1  1  99  0  0
   0  0    132 3111220 305540 588012    0    0     0     0  1032  142  0  1  99  0  0
   0  0    132 3111096 305544 588008    0    0     0    44  1019  134  0  0  99  0  0
  

Rebooting makes the problem go away for a while. This most recent time it took six days to come back. I'm not sure if that's consistent or not.

I had initially suspected that the problem might be related to the nVidia video driver module, but I shut down X Windows and removed the module, without change in the symptoms.

There is nothing in dmesg or /var/log/messages that seems remotely relevant or in any way coincides with the hiccups. It does not appear to be an issue with a hard drive, as I would expect iowait to be prominent during the unresponsive period if that were the case, but it's not. It feels unlikely to be a hardware problem, as the hiccups are pretty regular. I've been unable to time them down to milliseconds, but it's a pretty consistent 30/4/30/4/30/4.

Any ideas?

My organization has a leased color printer. We pay a per-page cost to the lessor, and the color page cost is far greater than the black-and-white page cost. Our users are pretty good about selecting the correct mode. But the problem comes when they want to print a big job that has only a few color pages. They don't want to manually search through the job to find the color pages and separate them out, and management would like them to not print hundreds of B&W pages at color costs.

For example, imagine a printer where B&W pages cost 1¢ and color pages cost 8¢. A user wants to print a 200 page document. Most of the pages are just black text, but there are ten pages of color diagrams. The user wants to get the diagrams printed in color, but the only way for him to do that is to print the whole job in "color", costing $16, or manually find the ten color pages, print them separately for 80¢, then print the rest of the document for $1.80, for a total of $2.60, or forego the color pages altogether and just print the whole thing in B&W for $2.

Is there a piece of software than can automatically find the color pages in a print job, send those pages to the color printer, and then print the rest of the job to a B&W printer? What would be ideal is some sort of print filter so that the user could just print the whole job as a color job, and the software would intercept it, chop it up based on which pages were in color, and send each segment to the appropriate printer. I've found PaperCutNG, which does exactly what I want, but, honestly, it's not worth the money for that one feature. I was hoping to find a free solution.

My print server is a Windows 2003 machine, and a solution that runs there would be preferable, but I can transition to a different OS if needed. A client-side solution would also be acceptable.

When accessing an Exchange 2003 server via IMAP, emails that were sent as text/plain (and ones that had no MIME encoding specified at all) get automatically converted to multipart/alternative with the original text/plain body and a text/html body. This is … stupid. It doesn't even bother to specify a monospaced font.

The new MIME part starts like this:

Content-Type: text/html;
    charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; =
charset=3Diso-8859-1">
<META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version =
6.5.7654.12">
<TITLE>{{subject}}</TITLE>

</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<BR>

<P><FONT SIZE=3D2>{{body}}

(All the "3D" stuff is quoted-printable encoding for an equals sign; there's nothing wrong on that front, surprisingly.)

How can I make this stop?