Tom O'Connor's questions

I'm a huge fan of automation and so on, so having the ability to do this is kinda important.

In this instance, we're naming the file Autounattend.xml, putting it in the root of the boot disk, and recreating the ISO for booting from DVD. I'd like to use WDS, but that's not an option right now.

I used System Image Manager to create the Answer file, a section of which is shown below.

<component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DiskConfiguration>
                <Disk wcm:action="modify">
                    <CreatePartitions>
                        <CreatePartition wcm:action="modify">
                            <Order>1</Order>
                            <Size>71024</Size>
                            <Type>Primary</Type>
                        </CreatePartition>
                    </CreatePartitions>
                    <ModifyPartitions>
                        <ModifyPartition wcm:action="modify">
                            <Active>true</Active>
                            <Extend>false</Extend>
                            <Format>NTFS</Format>
                            <Order>1</Order>
                            <PartitionID>1</PartitionID>
                        </ModifyPartition>
                    </ModifyPartitions>
                    <DiskID>0</DiskID>
                    <WillWipeDisk>true</WillWipeDisk>
                </Disk>
                <WillShowUI>OnError</WillShowUI>
            </DiskConfiguration>
            <ImageInstall>
                <OSImage>
                    <InstallTo>
                        <DiskID>0</DiskID>
                        <PartitionID>1</PartitionID>
                    </InstallTo>
                    <InstallFrom>
                        <MetaData wcm:action="add">
                            <Key>/IMAGE/NAME</Key>
                            <Value>Windows Server 2012 SERVERSTANDARD</Value>
                        </MetaData>
                    </InstallFrom>
                    <WillShowUI>OnError</WillShowUI>
                </OSImage>
            </ImageInstall>

Still shows this:

I've tried all sorts of stuff, including wcm:action="add" throughout. I've no idea why this might not work on windows 2012. I've got identical config on the Windows 7 Autounattend.xml file, and that works fine.

Any ideas?

EDIT: Oddly enough, this works. Spoke too soon. This crashes with the error below.

<component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DiskConfiguration>
                <Disk wcm:action="add">
                    <CreatePartitions>
                        <CreatePartition wcm:action="add">
                            <Order>1</Order>
                            <Type>Primary</Type>
                            <Extend>true</Extend>
                        </CreatePartition>
                    </CreatePartitions>
                    <ModifyPartitions>
                        <ModifyPartition wcm:action="add">
                            <Active>true</Active>
                            <Extend>true</Extend>
                            <Format>NTFS</Format>
                            <Order>1</Order>
                            <PartitionID>1</PartitionID>
                        </ModifyPartition>
                    </ModifyPartitions>
                    <DiskID>0</DiskID>
                </Disk>
                <WillShowUI>OnError</WillShowUI>
            </DiskConfiguration>
            <ImageInstall>
                <OSImage>
                    <InstallTo>
                        <DiskID>0</DiskID>
                        <PartitionID>1</PartitionID>
                    </InstallTo>
                    <InstallFrom>
                        <MetaData wcm:action="add">
                            <Key>/IMAGE/NAME</Key>
                            <Value>Windows Server 2012 SERVERSTANDARD</Value>
                        </MetaData>
                    </InstallFrom>
                    <WillShowUI>OnError</WillShowUI>
                    <InstallToAvailablePartition>false</InstallToAvailablePartition>
                </OSImage>
            </ImageInstall>

Over a cluster of over 12 centos 5.8 servers, I deployed logstash using the native logstash shipper, which sends /var/log/*/*.log back to a central logstash server.

We tried using rsyslogd as the shipper, but due to a bug in rsyslogd's ImFile module, if the remote end didn't reply, the logs would pile up in memory.

We're currently using Redis as the transport mechanism, so logstash01 has redis running locally, bound to the IP for the VLAN for these logs.

So logstash-shipper sends to redis on logstash01. logstash01 sends to Elasticsearch running in a separate process.

Here's what we're seeing. Elasticsearch has 141 blocked threads. Stracing the elasticsearch parent shows:

futex(0x7f4ccd1939d0, FUTEX_WAIT, 26374, NULL

Here's the jstack from elasticsearch

Here's the jstack from logstash

So.. Last night, some of the webservers (whose logs are tailed by logstash) went nuts, with load averages over 500.

On logstash01, there's this

Dec 19 00:44:45 logstash01 kernel: [736965.925863] Killed process 23429 (redis-server) total-vm:5493112kB, anon-rss:4248840kB, file-rss:108kB

So OOM-killer killed redis-server, which then meant logs piled up in memory on the servers which were shipping stuff.. Which somehow means that apache gets its knickers in a twist. (Frankly, I'm not sure how, I just assume it's tailing the log)..

This is my theory of how events unfolded:

1. We had a traffic spike.
2. An immense amount of logs were generated.
3. These piled up in Redis, as logstash/elasticsearch only seems to be able to handle 300-400 new events / second.
4. Redis had filled up entirely to the point where OOM-killer slaughtered it senselessly.
5. Redis stops accepting new items.
6. Items now start to pile up on the remote hosts side.
7. Everything goes nuts. Apache stops accepting requests. (Why?).

Questions are these:

8. Why does apache go nuts if there's just something tailing its log. Is it that the thing tailing it blocks apache from writing?

9. Is there a sane way to make elasticsearch faster/better/resilient?

10. Is there a sane way to make redis resilient and not die because of being OOM'd

11. Is there a fundamental flaw in the way I've set it all up, or does everyone have this problem?

-- EDIT --

Some specs for @lusis.

admin@log01:/etc/init$ free -m
             total       used       free     shared    buffers     cached
Mem:          7986       6041       1944          0        743       1157
-/+ buffers/cache:       4140       3845
Swap:         3813       3628        185

Filesystem             Size  Used Avail Use% Mounted on
/dev/sda2               19G  5.3G   13G  31% /
udev                   3.9G  4.0K  3.9G   1% /dev
tmpfs                  1.6G  240K  1.6G   1% /run
none                   5.0M     0  5.0M   0% /run/lock
none                   3.9G     0  3.9G   0% /run/shm
/dev/sda1               90M   72M   14M  85% /boot
/dev/mapper/data-disk  471G  1.2G  469G   1% /data

/dev/sda2 on / type ext3 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
/dev/sda1 on /boot type ext2 (rw)
/dev/mapper/data-disk on /data type ext3 (rw)
/data/elasticsearch on /var/lib/elasticsearch type none (rw,bind)

log01:/etc/init$ top 
top - 14:12:20 up 18 days, 21:59,  2 users,  load average: 0.20, 0.35, 0.40
Tasks: 103 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
Cpu0  :  3.0%us,  1.0%sy,  0.0%ni, 95.7%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu1  : 12.0%us,  1.0%sy,  0.0%ni, 86.6%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Cpu2  :  4.7%us,  0.3%sy,  0.0%ni, 94.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu3  :  5.6%us,  1.3%sy,  0.0%ni, 93.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  :  5.3%us,  1.3%sy,  0.0%ni, 93.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu5  :  6.4%us,  1.0%sy,  0.0%ni, 92.3%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   8178120k total,  6159036k used,  2019084k free,   761780k buffers

So.. I've installed Logstash, and instead of using the logstash shipper (because it needs the JVM and is generally massive), I'm using rsyslogd with the following configuration.

# Use traditional timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
$IncludeConfig /etc/rsyslog.d/*.conf


# Provides kernel logging support (previously done by rklogd)
$ModLoad imklog
# Provides support for local system logging (e.g. via logger command)
$ModLoad imuxsock

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none;local6.none            /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 *

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log

In /etc/rsyslog.d/logstash.conf there are 28 file monitor blocks using imfile

$ModLoad imfile   # Load the imfile input module
$ModLoad imklog   # for reading kernel log messages
$ModLoad imuxsock # for reading local syslog messages

$InputFileName /var/log/rabbitmq/startup_err
$InputFileTag rmq-err:
$InputFileStateFile state-rmq-err
$InputFileFacility local6
$InputRunFileMonitor
....
$InputFileName /var/log/some.other.custom.log
$InputFileTag cust-log:
$InputFileStateFile state-cust-log
$InputFileFacility local6
$InputRunFileMonitor
....
*.* @@10.90.0.110:5514

There are 28 InputFileMonitor blocks, each monitoring a different custom application logfile..

If I run

[root@secret-gm02 ~]# lsof|grep rsyslog
rsyslogd   5380        root  cwd       DIR              253,0       4096          2 /
rsyslogd   5380        root  rtd       DIR              253,0       4096          2 /
rsyslogd   5380        root  txt       REG              253,0     278976    1015955 /sbin/rsyslogd
rsyslogd   5380        root  mem       REG              253,0      58400    1868123 /lib64/libgcc_s-4.1.2-20080825.so.1
rsyslogd   5380        root  mem       REG              253,0     144776    1867778 /lib64/ld-2.5.so
rsyslogd   5380        root  mem       REG              253,0    1718232    1867780 /lib64/libc-2.5.so
rsyslogd   5380        root  mem       REG              253,0      23360    1867787 /lib64/libdl-2.5.so
rsyslogd   5380        root  mem       REG              253,0     145872    1867797 /lib64/libpthread-2.5.so
rsyslogd   5380        root  mem       REG              253,0      85544    1867815 /lib64/libz.so.1.2.3
rsyslogd   5380        root  mem       REG              253,0      53448    1867801 /lib64/librt-2.5.so
rsyslogd   5380        root  mem       REG              253,0      92816    1868016 /lib64/libresolv-2.5.so
rsyslogd   5380        root  mem       REG              253,0      20384    1867990 /lib64/rsyslog/lmnsd_ptcp.so
rsyslogd   5380        root  mem       REG              253,0      53880    1867802 /lib64/libnss_files-2.5.so
rsyslogd   5380        root  mem       REG              253,0      23736    1867800 /lib64/libnss_dns-2.5.so
rsyslogd   5380        root  mem       REG              253,0      20768    1867988 /lib64/rsyslog/lmnet.so
rsyslogd   5380        root  mem       REG              253,0      11488    1867982 /lib64/rsyslog/imfile.so
rsyslogd   5380        root  mem       REG              253,0      24040    1867983 /lib64/rsyslog/imklog.so
rsyslogd   5380        root  mem       REG              253,0      11536    1867987 /lib64/rsyslog/imuxsock.so
rsyslogd   5380        root  mem       REG              253,0      13152    1867989 /lib64/rsyslog/lmnetstrms.so
rsyslogd   5380        root  mem       REG              253,0       8400    1867992 /lib64/rsyslog/lmtcpclt.so
rsyslogd   5380        root    0r      REG                0,3          0 4026531848 /proc/kmsg
rsyslogd   5380        root    1u     IPv4         1200589517        0t0        TCP 10.10.10.90 t:40629->10.10.10.90:5514 (ESTABLISHED)
rsyslogd   5380        root    2u     IPv4         1200589527        0t0        UDP *:45801 
rsyslogd   5380        root    3w      REG              253,3   17999744    2621483 /var/log/messages
rsyslogd   5380        root    4w      REG              253,3      13383    2621484 /var/log/secure
rsyslogd   5380        root    5w      REG              253,3       7180    2621493 /var/log/maillog
rsyslogd   5380        root    6w      REG              253,3      43321    2621529 /var/log/cron
rsyslogd   5380        root    7w      REG              253,3          0    2621494 /var/log/spooler
rsyslogd   5380        root    8w      REG              253,3          0    2621495 /var/log/boot.log
rsyslogd   5380        root    9r      REG              253,3 1064271998    2621464 /var/log/custom-application.monolog.log
rsyslogd   5380        root   10u     unix 0xffff81081fad2e40        0t0 1200589511 /dev/log  

You can see that there are nowhere near 28 logfiles actually being read.

I really had to get one file monitored, so I moved it to the top, and it picked it up, but I'd like to be able to monitor all 28+ files, and not have to worry.

OS is

Centos 5.5 

Kernel 2.6.18-308.el5

rsyslogd 3.22.1, compiled with:
    FEATURE_REGEXP:             Yes
    FEATURE_LARGEFILE:          Yes
    FEATURE_NETZIP (message compression):   Yes
    GSSAPI Kerberos 5 support:      Yes
    FEATURE_DEBUG (debug build, slow code): No
    Atomic operations supported:        Yes
    Runtime Instrumentation (slow code):    No

Questions:

Why is rsyslogd only monitoring a very small subset of the files? How can I fix this so that all the files are monitored?

We've been advised by our tape library vendor that one of the reasons we might be seeing lots of errors is if our server room is particularly dusty.

It doesn't look dusty, but that's not to say it's not there.

We've got an environment sensor cluster which measures Temperature, Airflow and Relative Humidity.

I should probably point out that the low-hanging fruit solution I came up with is to use Sellotape (scotch tape) in a loop, one side stuck to the server cabinet, the other side free-hanging.

I've also put a couple of other tape loops by the exit and intake fans of the hardware (not blocking airflow, naturally).

How can we (electronically, ideally) measure dust levels?

I've recently been thinking again about a product that Viprinet provide, basically they've got a pair of routers, one that lives in a datacentre, Their VPN Multichannel Hub and the on-site hardware, their VPN multichannel routers

They've also got a bunch of interface cards (like HWICs) for 3G, UMTS, Ethernet, ADSL and ISDN adapters.

Their main spiel seems to be bonding across different media. It's something that I'd really like to use for a couple of projects, but their pricing is really quite extreme, the hub is about 1-2k, the routers are 2-6k, and the interface modules are 200-600 each.

So, what I'd like to know is, is it possible with a couple of stock Cisco routers, 28xx or 18xx series, to do something similar, and basically connect a bunch of different WAN ports, but have it all presented neatly as one channel back to the internet, with seamless (or nearly) failover if one of the WAN interfaces should fail.

Basically, If i got 3x 3G to ethernet modems, and each on a different network, I'd like to be able to loadbalance/bond across all of them, without having to pay Viprinet for the privilege.

Does anyone know how I'd go about configuring something for myself, based around standard protocols (or vendor specific ones), but without actually having to buy the Viprinet hardware?

I've got a server running Postgresql 9.1 replication. I wrote a script which prints out the current replication lag (master inserts a unix timestamp every 60s, and the slave compares it to the current timestamp).

I've added

extend replag /usr/local/bin/check_lag_quietly.sh

to snmpd.conf, and restarted the snmpd service, but when I snmpwalk the host, either from localhost, or from another machine on the network, I can see all of the other OIDs, but not this one.
If I do SNMP-Get as

tom.oconnor@charcoal-black:~$ snmpget -v2c -cpublic dns-2 UCD-SNMP-MIB::extTable
UCD-SNMP-MIB::extTable = No Such Object available on this agent at this OID

Then it can't find the OID there either.

Net-SNMP version 5.4.2.1, Ubuntu 10.04.

Bacula won't make use of 2 tape devices simultaneously. (Search for #-#-# for the TL;DR)

A little background, perhaps.

In the process of trying to get a decent working backup solution (backing up >20TB ain't cheap, or easy) at $dayjob, we bought a bunch of things to make it work.

Firstly, there's a Spectra Logic T50e autochanger, 40 slots of LTO5 goodness, and that robot's got a pair of IBM HH5 Ultrium LTO5 drives, connected via FibreChannel Arbitrated Loop to our backup server.

There's the backup server.. A Dell R715 with 2x 16 core AMD 62xx CPUs, and 32GB of RAM. Yummy. That server's got 2 Emulex FCe-12000E cards, and an Intel X520-SR dual port 10GE NIC.

We were also sold Commvault Backup (non-NDMP).

Here's where it gets really complicated.

Spectra Logic and Commvault both sent respective engineers, who set up the library and the software. Commvault was running fine, in so far as the controller was working fine. The Dell server has Ubuntu 12.04 server, and runs the MediaAgent for CommVault, and mounts our BlueArc NAS as NFS to a few mountpoints, like /home, and some stuff in /mnt.

When backing up from the NFS mountpoints, we were seeing ~= 290GB/hr throughput. That's CRAP, considering we've got 20-odd TB to get through, in a <48 hour backup window. The rated maximum on the BlueArc is 700MB/s (2460GB/hr), the rated maximum write speed on the tape devices is 140MB/s, per drive, so that's 492GB/hr (or double it, for the total throughput).

So, the next step was to benchmark NFS performance with IOzone, and it turns out that we get epic write performance (across >20 threads), and it's like 1.5-2.5TB/hr write, but read performance is fecking hopeless. I couldn't ever get higher than 343GB/hr maximum. So let's assume that the 343GB/hr is a theoretical maximum for read performance on the NAS, then we should in theory be able to get that performance out of a) CommVault, and b) any other backup agent.

Not the case. Commvault seems to only ever give me 200-250GB/hr throughput, and out of experimentation, I installed Bacula to see what the state of play there is. If, for example, Bacula gave consistently better performance and speeds than Commvault, then we'd be able to say "**$.$ Refunds Plz $.$**"

#-#-#

Alas, I found a different problem with Bacula. Commvault seems pretty happy to read from one part of the mountpoint with one thread, and stream that to a Tape device, whilst reading from some other directory with the other thread, and writing to the 2nd drive in the autochanger.

I can't for the life of me get Bacula to mount and write to two tape drives simultaneously.

Things I've tried:

• Setting Maximum Concurrent Jobs = 20 in the Director, File and Storage Daemons
• Setting Prefer Mounted Volumes = no in the Job Definition
• Setting multiple devices in the Autochanger resource.

Documentation seems to be very single-drive centric, and we feel a little like we've strapped a rocket to a hamster, with this one. The majority of example Bacula configurations are for DDS4 drives, manual tape swapping, and FreeBSD or IRIX systems.

I should probably add that I'm not too bothered if this isn't possible, but I'd be surprised. I basically want to use Bacula as proof to stick it to the software vendors that they're overpriced ;)

I read somewhere that @KyleBrandt has done something similar with a modern Tape solution..

Configuration Files: bacula-dir.conf

#
# Default Bacula Director Configuration file

Director {                            # define myself
  Name = backuphost-1-dir
  DIRport = 9101                # where we listen for UA connections
  QueryFile = "/etc/bacula/scripts/query.sql"
  WorkingDirectory = "/var/lib/bacula"
  PidDirectory = "/var/run/bacula"
  Maximum Concurrent Jobs = 20
  Password = "yourekiddingright"         # Console password
  Messages = Daemon
  DirAddress = 0.0.0.0
  #DirAddress = 127.0.0.1
}

JobDefs {
  Name = "DefaultFileJob"
  Type = Backup
  Level = Incremental
  Client = backuphost-1-fd 
  FileSet = "Full Set"
  Schedule = "WeeklyCycle"
  Storage = File
  Messages = Standard
  Pool = File
  Priority = 10
  Write Bootstrap = "/var/lib/bacula/%c.bsr"
}

JobDefs {
  Name = "DefaultTapeJob"
  Type = Backup
  Level = Incremental
  Client = backuphost-1-fd
  FileSet = "Full Set"
  Schedule = "WeeklyCycle"
  Storage = "SpectraLogic"
  Messages = Standard
  Pool = AllTapes
  Priority = 10
  Write Bootstrap = "/var/lib/bacula/%c.bsr"
  Prefer Mounted Volumes = no

}

#
# Define the main nightly save backup job
#   By default, this job will back up to disk in /nonexistant/path/to/file/archive/dir
Job {
  Name = "BackupClient1"
  JobDefs = "DefaultFileJob"
}

Job {
  Name = "BackupThisVolume"
  JobDefs = "DefaultTapeJob"
  FileSet = "SpecialVolume"
}
#Job {
#  Name = "BackupClient2"
#  Client = backuphost-12-fd
#  JobDefs = "DefaultJob"
#}

# Backup the catalog database (after the nightly save)
Job {
  Name = "BackupCatalog"
  JobDefs = "DefaultFileJob"
  Level = Full
  FileSet="Catalog"
  Schedule = "WeeklyCycleAfterBackup"
  # This creates an ASCII copy of the catalog
  # Arguments to make_catalog_backup.pl are:
  #  make_catalog_backup.pl <catalog-name>
  RunBeforeJob = "/etc/bacula/scripts/make_catalog_backup.pl MyCatalog"
  # This deletes the copy of the catalog
  RunAfterJob  = "/etc/bacula/scripts/delete_catalog_backup"
  Write Bootstrap = "/var/lib/bacula/%n.bsr"
  Priority = 11                   # run after main backup
}

#
# Standard Restore template, to be changed by Console program
#  Only one such job is needed for all Jobs/Clients/Storage ...
#
Job {
  Name = "RestoreFiles"
  Type = Restore
  Client=backuphost-1-fd                 
  FileSet="Full Set"                  
  Storage = File                      
  Pool = Default
  Messages = Standard
  Where = /srv/bacula/restore
}

FileSet {
  Name = "SpecialVolume"
  Include {
    Options {
      signature = MD5
    }
  File = /mnt/SpecialVolume
  }
  Exclude {
    File = /var/lib/bacula
    File = /nonexistant/path/to/file/archive/dir
    File = /proc
    File = /tmp
    File = /.journal
    File = /.fsck
  }
}


# List of files to be backed up
FileSet {
  Name = "Full Set"
  Include {
    Options {
      signature = MD5
    }
    File = /usr/sbin
  }

  Exclude {
    File = /var/lib/bacula
    File = /nonexistant/path/to/file/archive/dir
    File = /proc
    File = /tmp
    File = /.journal
    File = /.fsck
  }
}

Schedule {
  Name = "WeeklyCycle"
  Run = Full 1st sun at 23:05
  Run = Differential 2nd-5th sun at 23:05
  Run = Incremental mon-sat at 23:05
}

# This schedule does the catalog. It starts after the WeeklyCycle
Schedule {
  Name = "WeeklyCycleAfterBackup"
  Run = Full sun-sat at 23:10
}

# This is the backup of the catalog
FileSet {
  Name = "Catalog"
  Include {
    Options {
      signature = MD5
    }
    File = "/var/lib/bacula/bacula.sql"
  }
}

# Client (File Services) to backup
Client {
  Name = backuphost-1-fd
  Address = localhost
  FDPort = 9102
  Catalog = MyCatalog
  Password = "surelyyourejoking"          # password for FileDaemon
  File Retention = 30 days            # 30 days
  Job Retention = 6 months            # six months
  AutoPrune = yes                     # Prune expired Jobs/Files
}

#
# Second Client (File Services) to backup
#  You should change Name, Address, and Password before using
#
#Client {
#  Name = backuphost-12-fd                
#  Address = localhost2
#  FDPort = 9102
#  Catalog = MyCatalog
#  Password = "i'mnotjokinganddontcallmeshirley"         # password for FileDaemon 2
#  File Retention = 30 days            # 30 days
#  Job Retention = 6 months            # six months
#  AutoPrune = yes                     # Prune expired Jobs/Files
#}


# Definition of file storage device
Storage {
  Name = File
# Do not use "localhost" here    
  Address = localhost                # N.B. Use a fully qualified name here
  SDPort = 9103
  Password = "lalalalala"
  Device = FileStorage
  Media Type = File
}

Storage {
  Name = "SpectraLogic"
  Address = localhost
  SDPort = 9103
  Password = "linkedinmakethebestpasswords"
  Device = Drive-1
  Device = Drive-2
  Media Type = LTO5
  Autochanger = yes
}



# Generic catalog service
Catalog {
  Name = MyCatalog
# Uncomment the following line if you want the dbi driver
# dbdriver = "dbi:sqlite3"; dbaddress = 127.0.0.1; dbport =  
  dbname = "bacula"; DB Address = ""; dbuser = "bacula"; dbpassword = ""
}

# Reasonable message delivery -- send most everything to email address
#  and to the console
Messages {
  Name = Standard

  mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: %t %e of %c %l\" %r"
  operatorcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula: Intervention needed for %j\" %r"
  mail = root@localhost = all, !skipped            
  operator = root@localhost = mount
  console = all, !skipped, !saved
#
# WARNING! the following will create a file that you must cycle from
#          time to time as it will grow indefinitely. However, it will
#          also keep all your messages if they scroll off the console.
#
  append = "/var/lib/bacula/log" = all, !skipped
  catalog = all
}


#
# Message delivery for daemon messages (no job).
Messages {
  Name = Daemon
  mailcommand = "/usr/lib/bacula/bsmtp -h localhost -f \"\(Bacula\) \<%r\>\" -s \"Bacula daemon message\" %r"
  mail = root@localhost = all, !skipped            
  console = all, !skipped, !saved
  append = "/var/lib/bacula/log" = all, !skipped
}

# Default pool definition
Pool {
  Name = Default
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 365 days         # one year
}

# File Pool definition
Pool {
  Name = File
  Pool Type = Backup
  Recycle = yes                       # Bacula can automatically recycle Volumes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 365 days         # one year
  Maximum Volume Bytes = 50G          # Limit Volume size to something reasonable
  Maximum Volumes = 100               # Limit number of Volumes in Pool
}

Pool {
  Name = AllTapes
  Pool Type = Backup
  Recycle = yes
  AutoPrune = yes                     # Prune expired volumes
  Volume Retention = 31 days         # one Moth
}

# Scratch pool definition
Pool {
  Name = Scratch
  Pool Type = Backup
}

#
# Restricted console used by tray-monitor to get the status of the director
#
Console {
  Name = backuphost-1-mon
  Password = "LastFMalsostorePasswordsLikeThis"
  CommandACL = status, .status
}

bacula-sd.conf

#
# Default Bacula Storage Daemon Configuration file
#

Storage {                             # definition of myself
  Name = backuphost-1-sd
  SDPort = 9103                  # Director's port      
  WorkingDirectory = "/var/lib/bacula"
  Pid Directory = "/var/run/bacula"
  Maximum Concurrent Jobs = 20
  SDAddress = 0.0.0.0
#  SDAddress = 127.0.0.1
}

#
# List Directors who are permitted to contact Storage daemon
#
Director {
  Name = backuphost-1-dir
  Password = "passwordslinplaintext"
}

#
# Restricted Director, used by tray-monitor to get the
#   status of the storage daemon
#
Director {
  Name = backuphost-1-mon
  Password = "totalinsecurityabound"
  Monitor = yes
}


Device {
  Name = FileStorage
  Media Type = File
  Archive Device = /srv/bacula/archive
  LabelMedia = yes;                   # lets Bacula label unlabeled media
  Random Access = Yes;
  AutomaticMount = yes;               # when device opened, read it
  RemovableMedia = no;
  AlwaysOpen = no;
}


Autochanger {
   Name = SpectraLogic
   Device = Drive-1
   Device = Drive-2
   Changer Command = "/etc/bacula/scripts/mtx-changer %c %o %S %a %d"
   Changer Device = /dev/sg4
}

Device {
   Name = Drive-1
   Drive Index = 0
   Archive Device = /dev/nst0
   Changer Device = /dev/sg4
   Media Type = LTO5
   AutoChanger = yes
   RemovableMedia = yes;
   AutomaticMount = yes;
   AlwaysOpen = yes;
   RandomAccess = no;
   LabelMedia = yes

}

Device {
   Name = Drive-2
   Drive Index = 1
   Archive Device = /dev/nst1
   Changer Device = /dev/sg4
   Media Type = LTO5
   AutoChanger = yes
   RemovableMedia = yes;
   AutomaticMount = yes;
   AlwaysOpen = yes;
   RandomAccess = no;
   LabelMedia = yes
}

# 
# Send all messages to the Director, 
# mount messages also are sent to the email address
#
Messages {
  Name = Standard
  director = backuphost-1-dir = all
}

bacula-fd.conf

#
# Default  Bacula File Daemon Configuration file
#

#
# List Directors who are permitted to contact this File daemon
#
Director {
  Name = backuphost-1-dir
  Password = "hahahahahaha"
}

#
# Restricted Director, used by tray-monitor to get the
#   status of the file daemon
#
Director {
  Name = backuphost-1-mon
  Password = "hohohohohho"
  Monitor = yes
}

#
# "Global" File daemon configuration specifications
#
FileDaemon {                          # this is me
  Name = backuphost-1-fd
  FDport = 9102                  # where we listen for the director
  WorkingDirectory = /var/lib/bacula
  Pid Directory = /var/run/bacula
  Maximum Concurrent Jobs = 20
  #FDAddress = 127.0.0.1
  FDAddress = 0.0.0.0
}

# Send all messages except skipped files back to Director
Messages {
  Name = Standard
  director = backuphost-1-dir = all, !skipped, !restored
}

The vast majority of questions and so on regarding the interoperability of Active and Open directories involves getting Mac clients to see an AD and auth against it.

What we'd like to do is get a Windows 7 workstation to auth completely against Open Directory. We tried setting it up as an NT4 type PDC, and that doesn't work satisfactorily.

We tried using pGina and the LDAP backend, which allows Authentication, but has no support for Authorization, and as a result, if we mount an NFS Share, the user has the rights to do anything they damn well please. Not ideal for security (Totally bloody unacceptable, actually).

We tried using a Samba server (newer version than on the Open Directory Server) as an intermediate, so that it knows about the LDAP server on the OD Server, but uses Samba 4 instead of v3. That didn't work either. We could login, but couldn't mount, and if we did, we had the same rights as with pGina. If we right-click the mounted drive in Windows, and have a look at NFS UID, it returns -2, not the correct (mapped) UID.

So the final plan I've got is to use an Active Directory, inside a Windows 2008R2 Virtual Machine. What I want to achieve is to have the Active Directory sync it's user data from OpenDirectory (read-only would be fine). That way, we'd have the ability to connect Windows 7 clients to a "virtual domain" which would actually just grab information from OD's LDAP.

All the information I've found is about how to go the other way.

Does anyone know how we can do this?

We've got a Macbook Air (Actually, we've got some Airs, Some Pros and some iMacs). We've got a consistently annoying problem, which is that when they connect to our DHCP server, they've actually got 2 different IP addresses, and that causes 2 different IP addresses to get set in our DHCP lease file, as well as 2 in our DNS (automagically updated).

On linux workstations with multiple NICs, we bond/team them into bond0 and set the MAC address by hand to be equal to one of the NICs in the bond. This pretty much fixes all the associated ballache.

I just tried to bond the NICs on the Air, using Manage Virtual Interfaces (Advanced settings in Networking, under the little gearwheel in the Adaptors list). I could add eth0 (the USB Ethernet dongle) to the bond just fine, but the wireless Airport adapter didn't show up.

Is it not possible to bridge across the interfaces in this way?

Is there another way to do it?

Are we crazy for wanting to do it this way? Is there a better way?

If I put a new script into /etc/cron.d/ how does the system know to insert it into the global crontab?

Do I need to restart a process to pick it up?

Does init run crontab every minute, and it'll get re-scanned then?

Does it use inotify and watch /etc/cron.d?

Couldn't find anything terribly obvious, and I've always wondered

Scenario: Brand new Dell M610 blades.

Connected to the network with a 10GE uplink into a Dell PowerConnect M6220 blade switch.

DHCP server evidently working, as everything else boots / gets addresses fine.

TFTP server working fine.

Can't seem to get a DHCP address from the server.

Is there something on a blade chassis level interfering? Will I need to set up something special on the chassis? On the switch? On the blade?

• Portfast = Enabled.
• IP Helper = Points at our DHCP server (BUT! Shouldn't be needed, everything is somewhere on 192.168.0.0/16)
• DHCP Relay = Enabled
• DHCP Snooping = Disabled

Now What?

Here's a weird problem.

We've got a number of devices with dual-NIC mainboards. Some are Realtek NICs, which suck. Some are Intel e1000s, which don't.

I've just noticed on 2 machines, one is an Intel NIC, one is a Realtek, that when I put the MAC address of one machine into the dhcpd.conf file on our DHCP server to get it to PXE boot the machine into a rebuild environment, initially everything is fine.

The server gets a DHCP allocation, and PXE boots into the Ubuntu preseed enviroment.

On one or two machines, it gets as far as Ubuntu's DHCP network configuration, and fails. If i pull up a busybox shell (on tty2 on the installing machine), and run ip link, I can see that the UP flag is set on the other NIC.

Here's some stuff.

  host xeon16-ghz240-gb48-node1 {
        hardware ethernet BC:AE:C5:07:1F:18;
        filename "pxelinux.0";
        next-server 192.168.123.80;
  }

That's what's in dhcpd.conf

This is what ip link on the evil machine looks like.

Only one NIC is actually connected (deliberately).

As you can see, the NIC that's in the dhcpd config, is not marked as UP, and the link that is UP, isn't the one in DHCP.

So far I've seen this on two brands of dual-NIC configuration.

Does anyone know 1) what's causing it, and b) What we can do about it?

We've got 2 UPSes in our server room. One's 3 phase, and has a 3phase supply, yet the APC network stats thing only says

Incoming Voltage: 230.1 Volts

The other is a single phase one with no network-type monitoring.

What I'd really like to know, and can't seem to find out anywhere.. What does a UPS do with the 3 separate phases? Does it use both of them to charge the batteries, then provide a single sinewave output? This would be ideal, because then we'd be able to effectively use the UPS to convert 3 separate phases (which must remain so, else you get bangy power supplies and the magic smoke escapes).

For reference.. the 3-phase one is a "Smart-UPS RT 8000 XL".

Question the First: Why does the voltage on the 3-phase UPS' web interface say 230, not 415?

Question the Second: What on earth does the UPS do with 3 phases of power?

Yes, I've looked on APC's website, but couldn't find a definitive answer.

We have a number of hand-built (with fpm and jenkins) .deb files in a local Apt repository (reprepro). These .debs contain a .desktop file that will be picked up by xdg-desktop in a post-inst script.

If we install the deb file by hand, on a new system, everything is fine.

If we install a new version with apt-get install, we get this error

xdg-desktop-menu: file '/usr/local/share/applications/customthingy.desktop' does not exist

If I download the deb file with apt-get install -d customthingy, and run

dpkg -i /var/cache/apt/archives/customthingy_2-r3_all.deb

I get the same, xdg-desktop error as before. So that rules out a problem with apt.

If I list the contents of the downloaded deb,

tom.oconnor@charcoal-black:~$ dpkg --contents /var/cache/apt/archives/customthingy_2-r3_all.deb |grep ".desktop"
-rw-r--r-- root/root       201 2011-07-28 20:02 ./usr/local/share/applications/customthingy.desktop

You can see the file exists.

However.. If we purge before reinstalling,

tom.oconnor@charcoal-black:~$ sudo apt-get purge customthingy
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED
  customthingy*
0 upgraded, 0 newly installed, 1 to remove and 84 not upgraded.
After this operation, 0B of additional disk space will be used.
Do you want to continue [Y/n]? y
(Reading database ... 219342 files and directories currently installed.)
Removing customthingy ...
Purging configuration files for customthingy ...

And then

tom.oconnor@charcoal-black:~$ sudo apt-get install customthingy
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed
  customthingy
0 upgraded, 1 newly installed, 0 to remove and 84 not upgraded.
Need to get 0B/4,030B of archives.
After this operation, 0B of additional disk space will be used.
Selecting previously deselected package customthingy.
(Reading database ... 219319 files and directories currently installed.)
Unpacking customthingy (from .../customthingy_2-r3_all.deb) ...
Setting up customthingy (2-r3) ...

EDIT: Contents of Postinst script

#!/bin/sh

# Add an entry to the system menu

XDG_DESKTOP_MENU="`which xdg-desktop-menu 2> /dev/null`"

if [ ! -x "$XDG_DESKTOP_MENU" ]; then
  echo "WARNING: Could not find xdg-desktop-menu" >&2
else
  "$XDG_DESKTOP_MENU" install --mode system /usr/local/share/applications/customthingy.desktop
  "$XDG_DESKTOP_MENU" forceupdate --mode system
fi

There's no error. So.. The questions are these:

1. Is this expected behavior, or a bug in apt/dpkg?
2. Do we have a malformed package with customthingy.deb that is preventing a future reinstall run from working?
3. Is it safe to assume that post-inst will always happen at the very end of installation, and we can safely assume that all files will have been extracted before this point in time?
4. Are we doing something massively weird?

I'm currently in the middle of a process to specify and design a new server environment for graphics and video rendering.

We know we want to use blades, and we're pretty sure of our vendor decisions.

We've got one massive problem, and that's power.
At the moment, we're looking at getting more 3 phase power into the building. One idea that I've heard of, is using 48v DC to power the servers and networking equipment.

Questions are these:

1. What are the advantages of using 48V DC over 230VAC or 415VAC 3phase. (We're in the UK.)
2. Do all of the decent (Dell, HP, IBM, Cisco)/most servers and network vendors support 48V DC power supplies?
3. Can we expect this to be more expensive in hardware costs?
4. Will this actually be more power efficient than using single/threephase, if all a PSU is gonna do is downcovert to +/- 12v/5v/3.3v.
5. Can we expect lower cooling demands because of lower amount of powerloss in downconversion from 230v.

I'm changing the way that our DHCP/DNS stuff works at work. Currently we've got 3 DNS servers, and a DHCP box. All of them are VMs.

There's a circular dependency where stuff booting requires NFS, which requires DNS. So when we reboot stuff, things might come back subtly broken until the DNS is up, and we restart some services.

What I want to do is have a few low power servers, probably dual core Atoms or similar, running from SSDs, so that they boot damn fast. I want to make the whole thing boot as near to instantaneously as possible.

Ideally I'd like to use Ubuntu 11.10, or Debian 6 as the OS. I'm not interested in Gentoo or compiling my own kernel. This needs to be reasonably supportable by myself.

Other than SSD drives, what other optimization steps can I take to improve boot speed?

How do you handle the departure process when privileged or technical staff resign / get fired? Do you have a checklist of things to do to ensure the continuing operation / security of the company's infrastructure?

I'm trying to come up with a nice canonical list of things that my colleagues should do when I leave (I resigned a week ago, so I've got a month to tidy up and GTFO).

So far I've got:

1. Escort them off the premises

2. Delete their email Inbox (set all mail to forward to a catch-all)

3. Delete their SSH keys on server(s)

4. Delete their mysql user account(s)

   ...

So, what's next. What have I forgotten to mention, or might be similarly useful?

(endnote: Why is this off-topic? I'm a systems administrator, and this concerns continuing business security, this is definitely on-topic.)

I've got another interesting one.

I'm about to backup and reinstall the HR Administrator's PC. I suspect that the fastest way to do this is to use the Windows 7 Transfer tool, and create a backup of the entire Users and Settings profiles on the NAS.

I don't see a problem with this.
She claims that nobody else should be able to see the information on her computer. Fair enough. I think that the systems administrator (me), should be in a significant enough level of trust to be able to make a backup, no questions asked, and delete the backup once the task is complete.

Her view is, that nobody (not even the other directors) should be able to view the HR documentation on her PC.

We already have a semi-backup (files, not user-state) on box.net, which does allow granular access to various users.

Questions:

1) Which one of us is nuts, her or me?

2) Do you trust your sysadmins to take backups of company policy / HR files?

3) Does anyone have a LART?

This is a really tricky one, and to some extent it's not a technical problem, and might not belong here, but

Server Fault is for system administrators ... who manage or maintain computers in a professional capacity

And I do.. and this is one of my tasks.. Anyway.

Imagine you had 5000 + double-sided pages of A4. Company data, all business critical.
You need to back it up, somehow. Proposed solutions so far are:

1. PDF -> Online storage
2. PDF -> DVD / BluRay / Tape
3. PDF -> Portable HDD / SSD / Flash drive.
4. Buy/Lease/Hire/'Steal' a big photocopier, and make copies.
5. ???

Immediate problems with the above:

1. What if the storage partner goes bust?
2. DVDs do rot over time. Tapes similarly.
3. These too, break over time.
4. Expensive. Slow. Heavy. Not Tree Friendly.

The Question(s):

What is the gold-standard for long-to-medium term data preservation and archiving? Have you solved a similar problem in the workplace?

After the initial loading, there is some requirement to add to the collection roughly 100 pages a month. Retrieval should be possible, easily, but probably is infrequent.
Ideally I'd like to guarantee that the solution will be workable long after I have left the company, and that it won't require a massive amount to keep it maintainable, so storing many many DVDs is not only not ideal, but also not a good long-term solution.

While just making paper copies is certainly the easiest, it's not the most environmentally friendly, not by a long way. It's also not very manageable, difficult to search, index, and so on. Combined with heavy, and difficult to physically store.

I quite like the idea in principle of having everything stored electronically, but the actual mechanism of doing this needs to be transparent and easy. I really don't want to be responsible for this forever and a day, supporting office users as they cock it up, and lose documents. I also don't want to be reliant on a single storage vendor, what if Dropbox (We have an online backup solution ATM, but it isn't Dropbox.) were to go bust, or otherwise experience a catastrophic event, how many businesses who are using their services would be up the creek, sans paddle?

There's some budget flexibility here, but I suspect anything that costs more than our current online backup (which is like 2500USD/year), would be viewed less than favourably, compared to just putting it in a shoebox under a bed. Which is no-doubt what would happen if I did nothing, and resigned tomorrow.

Any ideas?

-Edit-

The reason for doing this is twofold.

1) provide a sensible secure backup of business critical paperwork in the event the office burns down.

2) to satisfy data archiving laws WRT uk tax law for businesses and so on.

Edit 2:

Having some mechanism for indexing the documents would be bloody useful too..

Today, one of our developers had his laptop stolen from his house. Apparently, he had a full svn checkout of the company's source code, as well as a full copy of the SQL database.

This is one massive reason why I'm personally against allowing company work on personal laptops.
However, even if this had been a company owned laptop, we'd still have the same problem, although we would be in a slightly stronger position to enforce encryption (WDE) on the whole disk.

Questions are these:

1. What does your company do about company data on non company owned hardware?
2. Is WDE a sensible solution? Does it produce a lot of overhead on reads/writes?
3. Other than changing passwords for things that were stored/accessed from there, is there anything else you can suggest?