Steve Madsen's questions

What is the cleanest way to set a custom response code within Apache without resorting to CGI?

Twice in recent days I've wanted to do this. The first time I retired a web app. Status code 410 (gone) seemed the most appropriate. I came up with this snippet using mod_rewrite:

ErrorDocument 410 /retired.html

RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !retired.html
RewriteRule . - [G]

retired.html is a message displayed to anyone following links from elsewhere, so they understand what happened.

Now I have a web service API that should only work over HTTPS. For the HTML side, I have rewrite rules to redirect from HTTP to HTTPS, but for API access, I'd rather that unsecured requests get a hard error. (Some client HTTP libraries follow the redirect on GET, so it appears to work, then fail in weird ways on POST, PUT or DELETE.)

410 (gone) isn't right for this, 403 (forbidden) is closer, but still doesn't feel right. Obviously 3xx codes are completely wrong.

Is there no way, purely within Apache, to set the response status code for a request?

Yesterday I upgraded a Debian Lenny server to Squeeze. This server hosts two KVM guests. One has run Debian Squeeze itself all along, the other was originally Debian Lenny, upgraded to Squeeze the day before.

Everything appears to be working fine, however performance of the KVM guests is now quite poor. Everything seems to take much longer than it used to: logging in via SSH, checking for and installing updates with aptitude, requests to web applications running in a guest, etc.

Googling has left me the impression that much has changed in KVM from Lenny to Squeeze, and I also found some vague reference to there being a good way and a bad way to launch guests, but nothing concrete.

I am happy to provide configuration files, but rather than fill this question with noise, I'd like a little direction about where to look.

Pertinent packages are installed:

i   kvm                  - dummy transitional package from kvm to qemu-kvm
i   libvirt-bin          - the programs for the libvirt library
i A libvirt0             - library for interfacing with different virtualization
i A qemu-kvm             - Full virtualization on x86 hardware

kvm is Debian version 1:0.12.5+dfsg-5+squeeze6, qemu-kvm is 0.12.5+dfsg-5+squeeze6.

Here is the command for launching one of the guests, from ps:

/usr/bin/kvm -S -M pc -enable-kvm -m 768 -smp 1,sockets=1,cores=1,threads=1 -name apps -uuid 636b6620-0949-bc88-3197-37153b88772e -nodefaults -chardev socket,id=monitor,path=/var/lib/libvirt/qemu/apps.monitor,server,nowait -mon chardev=monitor,mode=readline -rtc base=utc -boot c -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/raid/kvm-images/apps.qcow2,if=none,id=drive-virtio-disk0,boot=on,format=raw -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -device virtio-net-pci,vlan=0,id=net0,mac=54:52:00:27:5e:02,bus=pci.0,addr=0x3 -net tap,fd=35,vlan=0,name=hostnet0 -device virtio-net-pci,vlan=1,id=net1,mac=54:52:00:40:cc:7f,bus=pci.0,addr=0x4 -net tap,fd=36,vlan=1,name=hostnet1 -chardev pty,id=serial0 -device isa-serial,chardev=serial0 -usb -vnc 127.0.0.1:0 -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6

I am running a Debian Linux server on Lenny. Within it, I am running another Lenny instance using KVM. Both servers are externally available, with public IPs, as well as a second interface with private IPs for the LAN. Everything works fine, except the VM sees all network traffic as originating from the host server. I suspect this might have something to do with the iptables-based firewall I'm running on the host.

What I'd like to figure out is: how to I properly configure the host's networking such that all of these requirements are met?

1. Both host and VMs have 2 network interfaces (public and private).
2. Both host and VMs can be independently firewalled.
3. Ideally, VM traffic does not have to traverse the host firewall.
4. VMs see real remote IP addresses, not the host's.

Currently, the host's network interfaces are configured as bridges. eth0 and eth1 do not have IP addresses assigned to them, but br0 and br1 do.

/etc/network/interfaces on the host:

# The primary network interface
auto br1
iface br1 inet static
    address 24.123.138.34
    netmask 255.255.255.248
    network 24.123.138.32
    broadcast 24.123.138.39
    gateway 24.123.138.33
    bridge_ports eth1
    bridge_stp off

auto br1:0
iface br1:0 inet static
    address 24.123.138.36
    netmask 255.255.255.248
    network 24.123.138.32
    broadcast 24.123.138.39

# Internal network
auto br0
iface br0 inet static
    address 192.168.1.1
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
    bridge_ports eth0
    bridge_stp off

This is the libvirt/qemu configuration file for the VM:

<domain type='kvm'>
  <name>apps</name>
  <uuid>636b6620-0949-bc88-3197-37153b88772e</uuid>
  <memory>393216</memory>
  <currentMemory>393216</currentMemory>
  <vcpu>1</vcpu>
  <os>
    <type arch='i686' machine='pc'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='cdrom'>
      <target dev='hdc' bus='ide'/>
      <readonly/>
    </disk>
    <disk type='file' device='disk'>
      <source file='/raid/kvm-images/apps.qcow2'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <interface type='bridge'>
      <mac address='54:52:00:27:5e:02'/>
      <source bridge='br0'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <mac address='54:52:00:40:cc:7f'/>
      <source bridge='br1'/>
      <model type='virtio'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
  </devices>
</domain>

Along with the rest of my firewall rules, the firewalling script includes this command to pass packets destined for a KVM guest:

# Allow bridged packets to pass (for KVM guests).
iptables -A FORWARD -m physdev --physdev-is-bridged -j ACCEPT

(Not applicable to this question, but a side-effect of my bridging configuration appears to be that I can't ever shut down cleanly. The kernel eventually tells me "unregister_netdevice: waiting for br1 to become free" and I have to hard reset the system. Maybe a sign I've done something dumb?)

For a few weeks now, smartd has been reporting that it is skipping some of its scheduled self-tests on the weekends:

Apr 24 18:29:32 calvin smartd[4758]: Device: /dev/sda, skip scheduled Offline Immediate Test; 40% remaining of current Self-Test.
Apr 24 18:29:33 calvin smartd[4758]: Device: /dev/sdb, skip scheduled Offline Immediate Test; 50% remaining of current Self-Test.

The drives in this RAID-1 array are set to run an offline test four times a day, a short self-test at 2am every day, and a long self-test on Saturdays at 2am. For some reason, it looks like the long self-test is taking longer, causing the other scheduled tests to be skipped.

First question: is this a sign of likely drive failure?

Then today, smartd reported that a self-test failed. Here is the output of smartctl -a /dev/sdb:

smartctl version 5.38 [i686-pc-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.8 family
Device Model:     ST3250823AS
Serial Number:    3ND1GNBC
Firmware Version: 3.03
User Capacity:    250,059,350,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   7
ATA Standard is:  Exact ATA specification draft version not indicated
Local Time is:    Sun Apr 25 13:15:34 2010 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
     was completed without error.
     Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
     without error or no self-test has ever 
     been run.
Total time to complete Offline 
data collection:    ( 430) seconds.
Offline data collection
capabilities:     (0x5b) SMART execute Offline immediate.
     Auto Offline data collection on/off support.
     Suspend Offline collection upon new
     command.
     Offline surface scan supported.
     Self-test supported.
     No Conveyance Self-test supported.
     Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
     power-saving mode.
     Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
     General Purpose Logging supported.
Short self-test routine 
recommended polling time:   (   1) minutes.
Extended self-test routine
recommended polling time:   (  84) minutes.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   047   039   006    Pre-fail  Always       -       168450357
  3 Spin_Up_Time            0x0003   098   098   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       33
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       9
  7 Seek_Error_Rate         0x000f   087   060   030    Pre-fail  Always       -       654745480
  9 Power_On_Hours          0x0032   055   055   000    Old_age   Always       -       40141
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       51
194 Temperature_Celsius     0x0022   037   062   000    Old_age   Always       -       37 (0 17 0 0)
195 Hardware_ECC_Recovered  0x001a   047   039   000    Old_age   Always       -       168450357
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0000   100   253   000    Old_age   Offline      -       0
202 TA_Increase_Count       0x0032   100   253   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     40131         -
# 2  Extended offline    Completed: read failure       30%     40129         379795511
# 3  Short offline       Completed without error       00%     40084         -
# 4  Short offline       Completed without error       00%     40060         -
# 5  Short offline       Completed without error       00%     40036         -
# 6  Short offline       Completed without error       00%     40013         -
# 7  Short offline       Completed without error       00%     39990         -
# 8  Extended offline    Completed without error       00%     39977         -
# 9  Short offline       Completed without error       00%     39919         -
#10  Short offline       Completed without error       00%     39895         -
#11  Short offline       Completed without error       00%     39872         -
#12  Short offline       Completed without error       00%     39848         -
#13  Short offline       Completed without error       00%     39824         -
#14  Short offline       Completed without error       00%     39801         -
#15  Extended offline    Completed without error       00%     39789         -
#16  Short offline       Completed without error       00%     39754         -
#17  Short offline       Completed without error       00%     39732         -
#18  Short offline       Completed without error       00%     39707         -
#19  Short offline       Completed without error       00%     39683         -
#20  Short offline       Completed without error       00%     39660         -
#21  Short offline       Completed without error       00%     39636         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Given that this drive is about 4.5 years old, I am probably tempting fate by keeping it in service.

SMART doesn't seem to get much respect as a reliable way to predict drive failure. What else can I use to get an early indication of drive failure?

I have a problem that crops up when using Mac OS X's Terminal (TERM=xterm): sometimes it gets itself into a state where lines that scroll off the top are not added to the scrollback buffer. I'm not using screen or similar; this is a plain bash shell inside a Terminal tab.

It doesn't do this immediately after opening a new tab. I believe it is a side-effect of something I've run in the problem tab. My guess is that it has something to do with the xterm emulation, possibly the scroll region.

What hasn't worked:

• Soft and hard reset via the Shell menu
• Running reset in the tab

Is there a reliable way to reset Terminal and/or the xterm state?

Can anyone provide a explanation for what's happening, even if the only fix is to close the tab and open a new one?

I am trying to install Xen 3.2 on Debian Lenny on a server with 2 NICs, one connected to the private LAN and the other to the Internet. This server has been in operation in a non-Xen state for a while and everything works fine. With Xen, however, networking isn't coming up properly. I'm having a hard time figuring out the proper way to configure this, as there is a lot of conflicting advice on the Xen site/wiki, mailing lists and even here.

Here are the basics: eth0 is the private network, eth1 is the public Internet. Ultimately, I want dom0 (and all domU's) to have access to both interfaces. There is currently an iptables script in place (from pre-Xen use) that sets up a firewall and port forwarding for other client machines on the private network. So even before I worry about installing a domU, I'm trying to get things working for bridging, knowing that's where I need to be.

The Xen Networking wiki and the Debian Xen wiki both point me toward using Xen's bridging scripts, but when I do this, eth1 does not come up automatically at boot time, so dom0 does not have access to the public network. It comes up fine when I sudo ifup eth1 manually. Note that I'm not even worrying about domU's yet, and I'm well aware that the out-of-the-box Xen bridging won't do anything for eth1 <-> domU.

This ServerFault question advises against using the Xen bridging scripts. I tried this configuration and dom0 can't access either private or public networks.

This mailing list thread says that the Debian bridge-utils package should do everything I need, but it seems to imply that if I want my dom0 to be on the public network, I have to assign an address to both the real eth1 as well as the bridge. So I need to burn two public addresses in this case?

/etc/network/interfaces

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth1
allow-hotplug eth1
iface eth1 inet static
    address 24.123.138.34
    netmask 255.255.255.248
    network 24.123.138.32
    broadcast 24.123.138.39
    gateway 24.123.138.33
    # dns-* options are implemented by the resolvconf package, if installed
    dns-nameservers 127.0.0.1
    dns-search lightyearsoftware.com

auto eth1:0
iface eth1:0 inet static
    address 24.123.138.36
    netmask 255.255.255.248
    network 24.123.138.32
    broadcast 24.123.138.39
    gateway 24.123.138.33

# Internal network
auto eth0
allow-hotplug eth0
iface eth0 inet static
    address 192.168.1.1
    netmask 255.255.255.0
    network 192.168.1.0
    broadcast 192.168.1.255
#   mtu 9000    # memory alloc failures with 2.6.26-2 e1000 driver

#auto br-eth0 br-eth1
#iface br-eth0 inet manual
#   bridge_ports eth0
#iface br-eth1 inet manual
#   bridge_ports eth1

/etc/xen/xend-config.sxp uses (network-script network-dummy) when the last 4 lines of the interfaces file are uncommented, (network-script network-bridge) otherwise.

I'm more than happy to provide any other diagnostic information that can help.

I have a server hosting a web site and other services that needs to be reinstalled. I would like to relocate these services to another server temporarily, with as little downtime as possible. Both servers are in the same data center, and can be on the same network switch.

What is the best technique for moving these services with minimal downtime? The site is database-driven, so ideally I want a "railroad switch" event, where I can ensure all traffic is moved to the new server at once. I don't want to have a situation where the old database gets updates after I've migrated the data to the new one.

Two things I have considered:

Change the DNS to point to the temporary service. The major issue here is that I don't control the propagation time for DNS, and other servers can hold on to the cached results for a while, leaving the site "down" for users that get the old address.

Is there a way to fix that problem with Apache + redirects? I suspect not, since name-based virtual hosting breaks without the domain name, which I can't use because it's stale.

Bind the old IP address to the new server and (temporarily) assign the old server a different IP during reinstallation. I can leave DNS alone in this case.

Are there any other simple solutions I am overlooking?