Kamil Kisiel's questions

I'm looking for ways to manage the installation of non-distro software, with multiple versions of the same software installed simultaneously. For example we'd have ProgramA 1.6 and ProgramB 1.7 installed simultaneously, and users would be able to switch between them.

A common solution to this seems to be environment modules which I know is used with success at many academic sites. Users can then just run module load ProgramA/1.6 and then running ProgramA would get them the right version. So conceptually I can maintain a repository of all the modulefiles which users would use to activate the right environment.

My problem lies mostly with creating a reproducible way to actually build the software used by the modules. I'd need a reproducible way to build each software package. What would be a good way to do this? Is it possible to achieve with RPMs in an alternate root location and/or alternate RPM db? Is there another tool I could use to manage the software builds? Should I just cobble it together with shell scripts?

The platform we're running on is Scientific Linux 6, which is basically equivalent to Red Hat 6.

We recently bought some new hardware for a database server which we were intending to dedicate to the operation of PostgreSQL. However now we have the requirement to also run MySQL as some software we want to use only supports that database. Since the storage on this machine is the most suitable for hosting a DB, and we don't currently have the budget for more hardware,we're thinking of running both of them on the same server.

Are there any caveats or best practices we should be aware of?

What's the difference between restarting a machine using the ipmitool subcommands chassis power cycle versus chassis power reset?

Every now and then when users SSH from their OS X (Snow Leopard) workstation to one of our Linux hosts they receive the message:

/usr/bin/xauth: ~/.Xauthority not writable, changes will be ignored

Of course, their X forwarded applications will not work at this point.

However, if they log out and log right back in again they do not get the message and everything works as expected.

On their Mac they get their home directory via AFP. The Linux machines get it via NFS.

Any ideas on what could be going on here?

Is there a way to see from the server which clients have an NFS (v3) export mounted?

I thought that showmount -a would show this information but it doesn't seem to be the case. It does appear to show a list of connected clients but it doesn't show which exports they have mounted.

The server is running SLES 10.

The output of showmount -a I get is (some similar lines removed for privacy purposes):

All mount points on oneose:
10.0.0.0/8,10.2.128.0/22,netboot.cluster.example.com:/mnt/foo
10.0.0.0/8,10.2.128.0/22,netboot.cluster.example.com:/mnt/project/foo1
10.0.0.0/8,10.2.128.0/22,netboot.cluster.example.com:/mnt/project/foo2
10.0.0.0/8:/mnt/home
10.128.1.2:10.0.0.0/8
10.2.0.65:10.0.0.0/8
10.2.128.0/22:/mnt/project
10.2.128.10:10.0.0.0/8,10.2.128.0/22
10.2.128.30:10.0.0.0/8
10.2.128.30:10.0.0.0/8,10.2.128.0/22
10.2.128.40:10.0.0.0/8
10.2.128.40:10.0.0.0/8,10.2.128.0/22,netboot.cluster.example.com
10.2.128.50:10.0.0.0/8
10.2.129.10:10.0.0.0/8
10.2.129.10:10.0.0.0/8,10.2.128.0/22
10.2.129.11:10.0.0.0/8
10.2.129.11:10.0.0.0/8,10.2.128.0/22
10.2.129.12:10.0.0.0/8
10.2.129.12:10.0.0.0/8,10.2.128.0/22
10.2.129.13:10.0.0.0/8
10.2.129.13:10.0.0.0/8,10.2.128.0/22
10.2.129.14:10.0.0.0/8
10.2.129.14:10.0.0.0/8,10.2.128.0/22
10.2.129.15:10.0.0.0/8
10.2.129.15:10.0.0.0/8,10.2.128.0/22
10.2.129.16:10.0.0.0/8
10.2.129.16:10.0.0.0/8,10.2.128.0/22
10.2.129.17:10.0.0.0/8
10.2.129.17:10.0.0.0/8,10.2.128.0/22
10.2.129.18:10.0.0.0/8
10.2.129.18:10.0.0.0/8,10.2.128.0/22
10.2.129.19:10.0.0.0/8
10.2.129.19:10.0.0.0/8,10.2.128.0/22
10.2.129.20:10.0.0.0/8
10.2.129.20:10.0.0.0/8,10.2.128.0/22
10.2.129.21:10.0.0.0/8
10.2.129.21:10.0.0.0/8,10.2.128.0/22
10.2.129.22:10.0.0.0/8
10.2.129.22:10.0.0.0/8,10.2.128.0/22
10.2.129.23:10.0.0.0/8
10.2.129.23:10.0.0.0/8,10.2.128.0/22
10.2.129.24:10.0.0.0/8
10.2.129.24:10.0.0.0/8,10.2.128.0/22
10.2.129.25:10.0.0.0/8
10.2.129.25:10.0.0.0/8,10.2.128.0/22
10.2.129.26:10.0.0.0/8
10.2.129.26:10.0.0.0/8,10.2.128.0/22
10.2.129.27:10.0.0.0/8
10.2.129.27:10.0.0.0/8,10.2.128.0/22
10.2.129.28:10.0.0.0/8
10.2.129.28:10.0.0.0/8,10.2.128.0/22
10.2.129.29:10.0.0.0/8
10.2.129.29:10.0.0.0/8,10.2.128.0/22
10.2.129.30:10.0.0.0/8
10.2.129.30:10.0.0.0/8,10.2.128.0/22
10.2.129.31:10.0.0.0/8
10.2.129.31:10.0.0.0/8,10.2.128.0/22
10.2.129.32:10.0.0.0/8
10.2.129.32:10.0.0.0/8,10.2.128.0/22
10.2.129.33:10.0.0.0/8
10.2.129.33:10.0.0.0/8,10.2.128.0/22
10.2.129.34:10.0.0.0/8
10.2.129.34:10.0.0.0/8,10.2.128.0/22
10.2.129.35:10.0.0.0/8
10.2.129.35:10.0.0.0/8,10.2.128.0/22
10.2.129.36:10.0.0.0/8
10.2.129.36:10.0.0.0/8,10.2.128.0/22
10.2.129.37:10.0.0.0/8
10.2.129.37:10.0.0.0/8,10.2.128.0/22
10.2.129.38:10.0.0.0/8
10.2.129.38:10.0.0.0/8,10.2.128.0/22
10.2.129.39:10.0.0.0/8
10.2.129.39:10.0.0.0/8,10.2.128.0/22
10.2.129.40:10.0.0.0/8
10.2.129.40:10.0.0.0/8,10.2.128.0/22
10.2.129.41:10.0.0.0/8
10.2.129.41:10.0.0.0/8,10.2.128.0/22
10.2.129.42:10.0.0.0/8
10.2.129.42:10.0.0.0/8,10.2.128.0/22
10.2.129.43:10.0.0.0/8
10.2.129.43:10.0.0.0/8,10.2.128.0/22
10.2.129.44:10.0.0.0/8
10.2.129.44:10.0.0.0/8,10.2.128.0/22
10.2.129.45:10.0.0.0/8
10.2.129.45:10.0.0.0/8,10.2.128.0/22
10.2.129.46:10.0.0.0/8
10.2.129.46:10.0.0.0/8,10.2.128.0/22
10.2.129.47:10.0.0.0/8
10.2.129.47:10.0.0.0/8,10.2.128.0/22
10.2.129.48:10.0.0.0/8
10.2.129.48:10.0.0.0/8,10.2.128.0/22
10.2.129.49:10.0.0.0/8
10.2.129.49:10.0.0.0/8,10.2.128.0/22
10.2.129.50:10.0.0.0/8
10.2.129.50:10.0.0.0/8,10.2.128.0/22
10.2.129.51:10.0.0.0/8
10.2.129.51:10.0.0.0/8,10.2.128.0/22
10.2.129.52:10.0.0.0/8
10.2.129.52:10.0.0.0/8,10.2.128.0/22
10.2.129.53:10.0.0.0/8
10.2.129.53:10.0.0.0/8,10.2.128.0/22
10.2.129.54:10.0.0.0/8
10.2.129.54:10.0.0.0/8,10.2.128.0/22
10.2.129.55:10.0.0.0/8
10.2.129.55:10.0.0.0/8,10.2.128.0/22
10.2.129.56:10.0.0.0/8
10.2.129.56:10.0.0.0/8,10.2.128.0/22
10.2.129.57:10.0.0.0/8
10.2.129.57:10.0.0.0/8,10.2.128.0/22
10.2.129.58:10.0.0.0/8
10.2.129.58:10.0.0.0/8,10.2.128.0/22
10.2.129.59:10.0.0.0/8
10.2.129.59:10.0.0.0/8,10.2.128.0/22
10.2.129.60:10.0.0.0/8
10.2.129.60:10.0.0.0/8,10.2.128.0/22
10.2.129.60:10.2.128.0/22
10.2.129.61:10.0.0.0/8
10.2.129.61:10.0.0.0/8,10.2.128.0/22
10.2.129.62:10.0.0.0/8
10.2.129.62:10.0.0.0/8,10.2.128.0/22
10.2.129.63:10.0.0.0/8
10.2.129.63:10.0.0.0/8,10.2.128.0/22
10.2.129.64:10.0.0.0/8
10.2.129.64:10.0.0.0/8,10.2.128.0/22
10.2.129.65:10.0.0.0/8
10.2.129.65:10.0.0.0/8,10.2.128.0/22
10.2.129.65:10.2.128.0/22
10.2.129.66:10.0.0.0/8
10.2.129.66:10.0.0.0/8,10.2.128.0/22
10.2.129.66:10.2.128.0/22
10.2.129.67:10.0.0.0/8
10.2.129.67:10.0.0.0/8,10.2.128.0/22
10.2.129.68:10.0.0.0/8
10.2.129.68:10.0.0.0/8,10.2.128.0/22
10.2.129.69:10.0.0.0/8
10.2.129.69:10.0.0.0/8,10.2.128.0/22
10.2.129.70:10.0.0.0/8
10.2.129.70:10.0.0.0/8,10.2.128.0/22
10.2.129.71:10.0.0.0/8
10.2.129.71:10.0.0.0/8,10.2.128.0/22
10.2.129.72:10.0.0.0/8
10.2.129.72:10.0.0.0/8,10.2.128.0/22
10.2.129.73:10.0.0.0/8
10.2.129.73:10.0.0.0/8,10.2.128.0/22
10.2.129.74:10.0.0.0/8
10.2.129.74:10.0.0.0/8,10.2.128.0/22
10.2.129.75:10.0.0.0/8
10.2.129.75:10.0.0.0/8,10.2.128.0/22
10.2.129.76:10.0.0.0/8
10.2.129.76:10.0.0.0/8,10.2.128.0/22
10.2.129.77:10.0.0.0/8,10.2.128.0/22
10.2.129.78:10.0.0.0/8,10.2.128.0/22
10.2.129.79:10.0.0.0/8,10.2.128.0/22
10.2.129.80:10.0.0.0/8,10.2.128.0/22
10.2.129.81:10.0.0.0/8,10.2.128.0/22
10.2.129.82:10.0.0.0/8,10.2.128.0/22
10.2.129.83:10.0.0.0/8,10.2.128.0/22
10.2.129.84:10.0.0.0/8,10.2.128.0/22
10.2.129.85:10.0.0.0/8,10.2.128.0/22
10.2.129.86:10.0.0.0/8,10.2.128.0/22
10.2.129.87:10.0.0.0/8,10.2.128.0/22
10.2.129.88:10.0.0.0/8,10.2.128.0/22
10.2.129.89:10.0.0.0/8,10.2.128.0/22
10.2.129.90:10.0.0.0/8,10.2.128.0/22
10.2.16.208:10.0.0.0/8
10.2.16.222:10.0.0.0/8
10.3.0.249:10.0.0.0/8
10.3.0.67:10.0.0.0/8
10.3.1.6:10.0.0.0/8
10.3.128.10:10.0.0.0/8
10.3.128.20:10.0.0.0/8
10.3.129.1:10.0.0.0/8
10.3.129.2:10.0.0.0/8
10.3.129.3:10.0.0.0/8
10.3.129.4:10.0.0.0/8
10.3.16.216:10.0.0.0/8
10.3.2.11:10.0.0.0/8
10.3.2.12:10.0.0.0/8
10.3.2.1:10.0.0.0/8
10.3.2.3:10.0.0.0/8
10.3.2.4:10.0.0.0/8
10.3.2.5:10.0.0.0/8
10.3.2.6:10.0.0.0/8
10.3.2.7:10.0.0.0/8
10.3.2.9:10.0.0.0/8
10.3.3.11:10.0.0.0/8
10.3.3.12:10.0.0.0/8
10.3.3.13:10.0.0.0/8
10.3.3.14:10.0.0.0/8
10.3.3.15:10.0.0.0/8
10.3.3.16:10.0.0.0/8
10.3.3.17:10.0.0.0/8
10.3.3.18:10.0.0.0/8
10.3.3.19:10.0.0.0/8
10.3.3.1:10.0.0.0/8
10.3.3.20:10.0.0.0/8
10.3.3.21:10.0.0.0/8
10.3.3.24:10.0.0.0/8
10.3.3.26:10.0.0.0/8
10.3.3.27:10.0.0.0/8
10.3.3.28:10.0.0.0/8
10.3.3.29:10.0.0.0/8
10.3.3.2:10.0.0.0/8
10.3.3.30:10.0.0.0/8
10.3.3.32:10.0.0.0/8
10.3.3.33:10.0.0.0/8
10.3.3.34:10.0.0.0/8
10.3.3.35:10.0.0.0/8
10.3.3.36:10.0.0.0/8
10.3.3.37:10.0.0.0/8
10.3.3.38:10.0.0.0/8
10.3.3.39:10.0.0.0/8
10.3.3.3:10.0.0.0/8
10.3.3.40:10.0.0.0/8
10.3.3.41:10.0.0.0/8
10.3.3.4:10.0.0.0/8
10.3.3.5:10.0.0.0/8
10.3.3.6:10.0.0.0/8
10.3.3.7:10.0.0.0/8
10.3.3.8:10.0.0.0/8
10.4.0.10:10.0.0.0/8
10.4.0.14:10.0.0.0/8
10.4.0.6:10.0.0.0/8

I have some servers with RAID arrays based on LSI Logic MPT chips (specifically the SAS1068E). Currently the only way I'm aware of to get the disk status is to use the lsiutil program and view the status there interactively.

I'd like to be able to monitor this through a cron job or other script but there doesn't seem to be a non-interactive mode to lsiutil. What do people use?

I have two identically configured OpenBSD machines that use sendmail for mail delivery. I basically made no changes to the sendmail install other than setting up an alias root: [email protected] on each one. They're both on the same subnet(s) and both can access the mail server mail.example.com. However, one of them cannot deliver mail. I'm not terribly familiar with sendmail, so hopefully someone can explain this behaviour. I've posted the /var/log/maillog/ from each machine below:

The machine which delivers successfully:

Dec  7 11:57:35 gw003 sendmail[9640]: nB7JvZmS009640: from=kamil, size=60, class=0, nrcpts=1, msgid=<200912071957.nB7JvZmS009640@gw00
3.backbone.example.com>, relay=root@localhost
Dec  7 11:57:35 gw003 sm-mta[30258]: nB7JvZlR030258: from=<[email protected]>, size=401, class=0, nrcpts=1, msgid=<2
[email protected]>, proto=ESMTP, daemon=MTA, [email protected] [127.
0.0.1]
Dec  7 11:57:35 gw003 sendmail[9640]: nB7JvZmS009640: [email protected], ctladdr=kamil (1031/1031), delay=00:00:00, xdelay=00:
00:00, mailer=relay, pri=30060, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (nB7JvZlR030258 Message accepted for delivery)
Dec  7 11:57:35 gw003 sm-mta[29848]: STARTTLS=client, relay=mail.example.com., version=TLSv1/SSLv3, verify=FAIL, cipher=AES256-SHA,
 bits=256/256
Dec  7 11:57:35 gw003 sm-mta[29848]: nB7JvZlR030258: to=<[email protected]>, ctladdr=<[email protected]> (1031/1
031), delay=00:00:00, xdelay=00:00:00, mailer=esmtp, pri=30401, relay=mail.example.com. [10.2.32.11], dsn=2.0.0, stat=Sent (4b1d5e2
f-0000528c Message accepted for delivery)

The machine which is failing to deliver:

Dec  7 12:00:35 gw004 sendmail[12957]: nB7K0ZdA012957: from=kamil, size=60, class=0, nrcpts=1, msgid=<200912072000.nB7K0ZdA012957@gw0
04.backbone.example.com>, relay=root@localhost
Dec  7 12:00:35 gw004 sm-mta[25655]: nB7K0Z7U025655: <[email protected]>... User unknown
Dec  7 12:00:35 gw004 sendmail[12957]: nB7K0ZdA012957: [email protected], ctladdr=kamil (1031/1031), delay=00:00:00, xdelay=00
:00:00, mailer=relay, pri=30060, relay=[127.0.0.1] [127.0.0.1], dsn=5.1.1, stat=User unknown
Dec  7 12:00:35 gw004 sm-mta[25655]: nB7K0Z7U025655: from=<[email protected]>, size=60, class=0, nrcpts=0, proto=ESM
TP, daemon=MTA, [email protected] [127.0.0.1]
Dec  7 12:00:35 gw004 sendmail[12957]: nB7K0ZdA012957: nB7K0ZdB012957: DSN: User unknown
Dec  7 12:00:36 gw004 sm-mta[25655]: nB7K0Z7W025655: from=<>, size=2203, class=0, nrcpts=1, msgid=<200912072000.nB7K0ZdB012957@gw004.
backbone.example.com>, proto=ESMTP, daemon=MTA, [email protected] [127.0.0.1]
Dec  7 12:00:36 gw004 sendmail[12957]: nB7K0ZdB012957: to=kamil, delay=00:00:01, xdelay=00:00:01, mailer=relay, pri=31084, relay=[127
.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (nB7K0Z7W025655 Message accepted for delivery)
Dec  7 12:00:36 gw004 sm-mta[9947]: nB7K0Z7W025655: to=<[email protected]>, delay=00:00:01, xdelay=00:00:00, mailer=
local, pri=32465, dsn=2.0.0, stat=Sent

I have a bunch of VM servers running Xen VMs from iSCSI LUNs mounted in the Dom0 host via Open-iSCSI. It's just a simple single-path setup (redundancy is on the target side via clustered storage and a virtual IP)

I'm considering using the settings from the Open-iSCSI README for iSCSI root, since that's basically what it is:

8.2 iSCSI settings for iSCSI root
---------------------------------

When accessing the root parition directly through a iSCSI disk, the
iSCSI timers should be set so that iSCSI layer has several chances to try
to re-establish a session and so that commands are not quickly requeued to
the SCSI layer. Basically you want theopposite of when using dm-multipath.

For this setup, you can turn off iSCSI pings by setting:

node.conn[0].timeo.noop_out_interval = 0 
node.conn[0].timeo.noop_out_timeout= 0

And you can turn the replacement_timer to a very long value:

node.session.timeo.replacement_timeout = 86400

Are there any other recommended settings for running VMs with a root on an iSCSI LUN I should think about?

This machine has a ton of swap, yet processes still occasionally get killed by the oom-killer. Can anyone explain this behavior, and more importantly how to keep it from occurring?

Dmesg output:

python invoked oom-killer: gfp_mask=0x1200d2, order=0, oomkilladj=4
Pid: 13996, comm: python Not tainted 2.6.27-gentoo-r8cluster-e1000 #9

Call Trace:
 [<ffffffff8025ab6b>] oom_kill_process+0x57/0x1dc
 [<ffffffff802460c7>] getnstimeofday+0x53/0xb3
 [<ffffffff8025ae78>] badness+0x16a/0x1a9
 [<ffffffff8025b0a9>] out_of_memory+0x1f2/0x25c
 [<ffffffff8025e181>] __alloc_pages_internal+0x30f/0x3b2
 [<ffffffff8026fea0>] read_swap_cache_async+0x48/0xc0
 [<ffffffff8026ff6f>] swapin_readahead+0x57/0x98
 [<ffffffff80266d0e>] handle_mm_fault+0x408/0x706
 [<ffffffff8057da33>] do_page_fault+0x42c/0x7e7
 [<ffffffff8057baf9>] error_exit+0x0/0x51

Mem-Info:
Node 0 DMA per-cpu:
CPU    0: hi:    0, btch:   1 usd:   0
CPU    1: hi:    0, btch:   1 usd:   0
CPU    2: hi:    0, btch:   1 usd:   0
CPU    3: hi:    0, btch:   1 usd:   0
Node 0 DMA32 per-cpu:
CPU    0: hi:  186, btch:  31 usd: 103
CPU    1: hi:  186, btch:  31 usd:  48
CPU    2: hi:  186, btch:  31 usd: 136
CPU    3: hi:  186, btch:  31 usd: 183
Active:480346 inactive:483 dirty:0 writeback:10 unstable:0
 free:3408 slab:5146 mapped:1408 pagetables:2687 bounce:0
Node 0 DMA free:8024kB min:20kB low:24kB high:28kB active:1156kB inactive:0kB present:8364kB pages_scanned:3246 all_unreclaimable? yes
lowmem_reserve[]: 0 2003 2003 2003
Node 0 DMA32 free:5608kB min:5716kB low:7144kB high:8572kB active:1920228kB inactive:1932kB present:2051308kB pages_scanned:2941301 all_unreclaimable? yes
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 8*4kB 3*8kB 4*16kB 3*32kB 4*64kB 3*128kB 2*256kB 3*512kB 3*1024kB 1*2048kB 0*4096kB = 8024kB
Node 0 DMA32: 42*4kB 6*8kB 1*16kB 0*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 0*2048kB 1*4096kB = 5608kB
325424 total pagecache pages
323900 pages in swap cache
Swap cache stats: add 20776604, delete 20452704, find 7856195/10744535
Free swap  = 151691424kB
Total swap = 156290896kB
524032 pages RAM
9003 pages reserved
331431 pages shared
186210 pages non-shared
Out of memory: kill process 12965 (bash) score 2236480 or a child
Killed process 13996 (python)

VM-related sysctls:

vm.overcommit_memory = 0
vm.panic_on_oom = 0
vm.oom_kill_allocating_task = 0
vm.oom_dump_tasks = 0
vm.overcommit_ratio = 50
vm.page-cluster = 3
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
vm.dirty_writeback_centisecs = 500
vm.dirty_expire_centisecs = 3000
vm.nr_pdflush_threads = 2
vm.swappiness = 60
vm.nr_hugepages = 0
vm.hugetlb_shm_group = 0
vm.hugepages_treat_as_movable = 0
vm.nr_overcommit_hugepages = 0
vm.lowmem_reserve_ratio = 256   256 32
vm.drop_caches = 0
vm.min_free_kbytes = 5740
vm.percpu_pagelist_fraction = 0
vm.max_map_count = 65536
vm.laptop_mode = 0
vm.block_dump = 0
vm.vfs_cache_pressure = 100
vm.legacy_va_layout = 0
vm.zone_reclaim_mode = 0
vm.min_unmapped_ratio = 1
vm.min_slab_ratio = 5
vm.stat_interval = 1
vm.numa_zonelist_order = default

Question for anyone out there managing a network of openSUSE machines. How do you keep track of and apply updates? I know about YaST Online Update (YOU) but it seems more geared towards keeping a single machine up to date. It doesn't seem to scale well to a larger number of machines. How do you keep your machines updated? Our network is fairly heterogenous in terms of package installation as the servers are mostly infrastructure machines with varying roles.

I know that SUSE Linux Enterprise has tools to manage updates network-wide, but updating to that is currently not an option for budget reasons.

In Linux if you go digging in /proc/<pid>/fd often you'll see output like:

lrwx------ 1 root root 64 Jul 30 15:14 0 -> /dev/null
lrwx------ 1 root root 64 Jul 30 15:14 1 -> /dev/null
l-wx------ 1 root root 64 Jul 30 15:14 10 -> pipe:[90222668]
lr-x------ 1 root root 64 Jul 30 15:14 11 -> pipe:[90222669]
l-wx------ 1 root root 64 Jul 30 15:14 13 -> pipe:[90225058]
lr-x------ 1 root root 64 Jul 30 15:14 14 -> pipe:[90225059]

How do I get more info about the open pipes, such as which process is on the other end?

This is kind of related to Password Best Practices but more specific.

Do you use the same root password for all the servers in your organization? For within a class of devices?

I'd like to clone all the ZFS ACLs from one file to another.

With POSIX ACLs this can be done by piping the output of getfacl to setfacl.

Is there an easy and quick way to do this with the NFSv4 style ACLs in ZFS? I know I can read off the output of ls -lV and then enter it as part of a chmod, but I can't seem to find a functional equivalent to the POSIX way of copying ACLs.

I'd like to do some analysis of our NFS server to help track down potential bottlenecks in our applications. The server is running SUSE Enterprise Linux 10.

The kind of things I'm looking to know are:

• Which files are being accessed by which clients
• Read/write throughput on a per-client basis
• Overhead imposed by other RPC calls
• Time spent waiting on other NFS requests, or disk I/O, to service a client

I already know about the statistics available in /proc/net/rpc/nfsd and in fact I wrote a blog post describing them in depth. What I'm looking for is a way to dig deeper and help understand what factors are contributing to the performance seen by a particular client. I want to analyze the role the NFS server plays in the performance of an application on our cluster so that I can think of ways to best optimize it.

I've been considering deploying mod_auth_kerb on our internal web servers to enable SSO. The one obvious problem I can see is that it's an all-or-nothing approach, either all your domain users can access a site or not.

Is it possible to combine mod_auth_kerb with something like mod_authnz_ldap to check for group membership in a particular group in LDAP? I'm guessing the KrbAuthoritative option would have something to do with this?

Also, as I understand it, the module sets the username to be username@REALM after authentication, but of course in the directory the users are stored as the username only. Furthermore, some internal sites we run such as trac already have a user profile linked to each username. Is there a way to resolve this, perhaps by stripping off the realm bit after authentication somehow?

I'm wondering if this is possible. I'd like users to be able to launch screen on some of our hosts for the purpose of multiplexing their terminal login sessions. However, I don't want them to be able to detach the screen and leave it running in the background.

The reason for this being that the only way to log in to these machines is through a cluster job submission system, so if a screen becomes detached and the user logs off the only way to get back would be if they happened to land back on the same node on their next job submissions.

My Xen servers are openSUSE 11.1 with open-iscsi to our iSCSI SAN cluster. The SAN modules are in an IP failover group behind a virtual IP that the initiators connect to.

In the event that the primary SAN server goes down, the secondary picks up the role of serving as the target. This is all handled by the LeftHand SAN/iQ software and works well in most situations.

The problem I have is that occasionally some of my Xen DomUs will have their root filesystem go read-only after an IP failover. It's not consistent, and happens to a different subset each time a failover occurs. They're all running the same openSUSE 11.1 software image.

The root filesystems for each DomU are mounted by open-iscsi in the Dom0 and then Xen uses the standard block device driver to expose it to the DomU.

The exact symptom is that as a root as running touch /test returns the error "read-only filesystem". However, the output of mount shows it as being mounted read-write. Of course, all other I/O on the domU is also failing at this time so the machine comes down hard. Simply restarting it with xm from the Dom0 without even reconnecting the iSCSI session makes everything work again.

On the Dom0 side the syslog messages during the fail-over are something like the following:

kernel: connection1:0: iscsi: detected conn error (1011)
iscsid: Kernel reported iSCSI connection 1:0 error (1011) state (3)
iscsid: connection1:0 is operational after recovery (1 attempts) 

I'm having a hard time figuring out at what layer to debug this problem, is it something in the DomU kernel? or at the Dom0 or Xen level? I think there's likely some parameter somewhere that needs tweaking to increase some kind of timeout, but I'm not sure where to look.

I don't really think it is an issue with open-iscsi simply because the connected block device is still readable and writeable from the Dom0.

Here's a problem which I am at a loss to diagnose:

Our user home directories are served via NFS from an Apple XServe running Mac OS X 10.5.7. Normally they are exported to our default office subnet, "lan". Recently I have been building a new subnet, "farm". The computers on "farm" run the same OS (openSUSE 11.1, and Gentoo) as the ones on "lan", and the software versions are the same.

The problem is that when my users have been using a machine on "farm" for some time (5 minutes, sometimes 30, sometimes a full hour) the NFS mount seems to just hang. Attempting to do an ls on the directory or anything else (such as a login, etc) that tries to access the user home directory just gets stuck. Mounts to other NFS servers from the "hung" machine seem to work as expected.

There is nothing in the logs of either the client or the server that indicates any problem. The same types of clients work just fine from the default "lan" subnet.

I've tried all sorts of different configurations of the NFS server and client (disabling/enabling kerberos, different mount options) but nothing appears to make any difference.

I'm strongly suspecting some network-level problems between these two subnets, perhaps some mangling by firewall/router (OpenBSD with pf as the packet filter). The connection between the two sets of machines is fairly simple: x serve --> switch --> router --> switch --> clients

I'm pretty much at a loss as to debugging methods to try next, or what the possible solution may be. Any ideas as to how to approach this problem from this point?

Update:

Still haven't been able to resolve this. I thought I had nipped it in the bud when I disabled scrub on the internal interfaces, but the problem has manifested itself again. What's strange is that pf seems to still be modifying some packets.

An example conversation, on the farm vlan side:

09:17:39.165860 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472382:2887472382(0) win 5840 <mss 1460,sackOK,timestamp 236992843 0,nop,wscale 6> (DF)
09:17:39.166124 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: . ack 43 win 65535 <nop,nop,timestamp 316702204 236992843> (DF)
09:17:54.164490 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472385:2887472385(0) win 5840 <mss 1460,sackOK,timestamp 236996593 0,nop,wscale 6> (DF)
09:17:54.164760 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: R 1441270809:1441270809(0) ack 43 win 65535 (DF)
09:17:54.164776 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: R 4243886205:4243886205(0) ack 46 win 0 (DF)
09:17:54.164989 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472388:2887472388(0) win 5840 <mss 1460,sackOK,timestamp 236996593 0,nop,wscale 6> (DF)
09:17:57.164066 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472388:2887472388(0) win 5840 <mss 1460,sackOK,timestamp 236997343 0,nop,wscale 6> (DF)
09:17:57.164330 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: . ack 49 win 65535 <nop,nop,timestamp 316702384 236997343> (DF)
09:18:03.163468 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472388:2887472388(0) win 5840 <mss 1460,sackOK,timestamp 236998843 0,nop,wscale 6> (DF)
09:18:03.163732 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: . ack 49 win 65535 <nop,nop,timestamp 316702444 236998843> (DF)

and the same on the lan vlan:

09:17:39.165876 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472382:2887472382(0) win 5840 <mss 1460,sackOK,timestamp 236992843 0,nop,wscale 6> (DF)
09:17:39.166110 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: . ack 1 win 65535 <nop,nop,timestamp 316702204 236992843> (DF)
09:17:54.164505 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472385:2887472385(0) win 5840 <mss 1460,sackOK,timestamp 236996593 0,nop,wscale 6> (DF)
09:17:54.164740 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: R 1:1(0) ack 1 win 65535 (DF)
09:17:54.164745 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: R 2802615397:2802615397(0) ack 4 win 0 (DF)
09:17:54.165003 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472388:2887472388(0) win 5840 <mss 1460,sackOK,timestamp 236996593 0,nop,wscale 6> (DF)
09:17:54.165239 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: S 449458819:449458819(0) ack 2887472389 win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 316702354 236996593,sackOK,eol> (DF)
09:17:55.123665 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: S 449458819:449458819(0) ack 2887472389 win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 316702363 236996593,sackOK,eol> (DF)
09:17:57.124839 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: S 449458819:449458819(0) ack 2887472389 win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 316702383 236996593,sackOK,eol> (DF)
09:17:57.164082 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472388:2887472388(0) win 5840 <mss 1460,sackOK,timestamp 236997343 0,nop,wscale 6> (DF)
09:17:57.164316 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: . ack 1 win 65535 <nop,nop,timestamp 316702384 236997343> (DF)
09:18:01.126690 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: S 449458819:449458819(0) ack 2887472389 win 65535 <mss 1460,nop,wscale 3,nop,nop,timestamp 316702423 236997343,sackOK,eol> (DF)
09:18:03.163483 node001.farm.foo.com.769 > barstar.lan.foo.com.nfsd: S 2887472388:2887472388(0) win 5840 <mss 1460,sackOK,timestamp 236998843 0,nop,wscale 6> (DF)
09:18:03.163717 barstar.lan.foo.com.nfsd > node001.farm.foo.com.769: . ack 1 win 65535 <nop,nop,timestamp 316702444 236998843> (DF)

I should also mention that we have other NFS traffic going through this same machine, but from a different NFS server. We've been using that for years and have not had any problems there. Similarly, these XServes have been serving NFS to Linux clients on their own subnet for a long while as well and continue to do so.

We have a small cluster of Xen servers in the office. Each virtual machine system image is on its own block device on our iSCSI SAN. The servers all connect their iSCSI initiator to every VM LUN simultaneously.

To prevent the same VM from being started on multiple hosts we just ensure that each virtual machine is only added to one server through the use of xm new. Judicious monitoring of all the servers through virt-manager helps as well. However, there really there are no safeguards that would prevent someone from accidentally launching the same VM on multiple hosts.

My question is, what would be a good way to implement a better management system for a setup like this? Are there any existing tools that will manage a Xen setup like this with a shared storage back-end?

We're using openSUSE 11.1 as the host OS.

Currently we manually document the connections to the ports of our switches. Of course, maintenance is a chore, and the documentation is out of date as soon as you save it.

Are there any tools for querying switches, preferably via SNMP, that can tell you what is connected on the other end?

For the record, we use primarily HP ProCurve switches.