Karolis T.'s questions

I have a server with software RAID 1, two hot-swap sata disks. One hard drive started showing errors, I'm thinking about removing and replacing it, only problem is that I have no idea which of the two correspond to which devices. And I can't shut the server down to find out.

I have /dev/sda and /dev/sdb, /dev/sda is the failing one. Thought about doing something along the lines

# mdadm --manage /dev/md0 --remove /dev/sda1

then somehow stop/suspend the drive using tuning software and try to listen which of the two stopped, but that's not gonna work in a noisy server environment. Drive panels have no LEDs.

Thanks for any ideas!

I know there are two physical network interfaces in our server (haven't seen it live, but person from datacenter confirmed two ethernet sockets). Problem is, I can only see one via ifconfig -a.

At first, here's what lspci reported:

# lspci -vv|grep -i ethernet
00:19.0 Ethernet controller: Intel Corporation Unknown device 10bd (rev 02)
03:02.0 Ethernet controller: Intel Corporation 82541GI/PI Gigabit Ethernet

After running

# update-pciids

it nows looks like

# lspci -vv|grep -i ethernet
00:19.0 Ethernet controller: Intel Corporation 82566DM-2 Gigabit Network Connection (rev 02)
03:02.0 Ethernet controller: Intel Corporation 82541GI Gigabit Ethernet Controller (rev 05)

dmidecode output:

# dmidecode|grep -i ethernet
    Type: Ethernet
    Description: Intel 82541PI Ethernet Device
    Type: Ethernet
    Description: Intel 82566DM Ethernet Devic

I have downloaded latest e1000 driver from intel and compiled it against current kernel headers. Used driver version is e1000-8.0.13.

# uname -r
2.6.18-6-686

After connecting via serial cable and doing rmmod/modprobe for e1000, result is still the same. I have also tried same steps for e1000e-1.0.2.5 with same result.

After modprobe, dmesg doesn't show anything about eth1 being found.

There is a file /etc/udev/rules.d/z25_persistent-net.rules with following content:

...
# PCI device 0x8086:0x1076 (e1000)
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="00:15:17:28:44:db", NAME="eth0"

and no entry for eth1. Maybe this file controls which eth devices are created after loading the driver? Problem is, I don't know the MAC of my missing interface, any way to find the MAC?

Thanks!

EDIT1:

Here's the contents of /sys/class/net directory

/sys/class/net# ls
eth0  lo  sit0

I had a look at /lib/udev/write_net_rules which is a tool generating /etc/udev/rules.d/z25_persistent-net.rules file. This tool uses entries in /sys/class/net, so it now seems unlikely that changing z25_persistent-net.rules would be of any help.

EDIT2:

from e1000e README

....
This driver supports kernel versions 2.4.x and 2.6.x.  This driver includes
support for Itanium(R)2-based systems.
....
- The following adapters do not support Jumbo Frames:
     Intel(R) 82562V 10/100 Network Connection
     Intel(R) 82566DM Gigabit Network Connection
     Intel(R) 82566DC Gigabit Network Connection
     Intel(R) 82566MM Gigabit Network Connection
     Intel(R) 82566MC Gigabit Network Connection
     Intel(R) 82562GT 10/100 Network Connection
     Intel(R) 82562G 10/100 Network Connection
     Intel(r) 82566DC-2 Gigabit Network Connection
     Intel(R) 82562V-2 10/100 Network Connection
     Intel(R) 82562G-2 10/100 Network Connection
     Intel(R) 82562GT-2 10/100 Network Connection
     Intel(R) 82583V Gigabit Network Connection
     Intel(R) 82578DC Gigabit Network Connection

can someone confirm (with sources) that I really need to upgrade my kernel in order for this to work?

There's an USB/Serial converter connecting two boxes.

Box A has USB end, box B has serial port end.

On box A:

screen /dev/ttyUSB0 9600

On box B (the one I'm trying to connect to from box A):

/etc/inittab:
...
T0:23:respawn:/sbin/getty -L ttyS0 9600 vt100
...

What I write to /dev/ttyS0 on box B is seen in the terminal on box A (via screen),

but data written to dev/ttyUSB0 on box A is not received on box B, for example, I can't type anything, though I see the login promt.

Any ideas? Thanks

EDIT1:

I've tried minicom and picocom with same results as above.

Here's the output of lsusb on box A:

Bus 005 Device 002: ID 14dd:0002  
Bus 005 Device 001: ID 0000:0000  
Bus 004 Device 001: ID 0000:0000  
Bus 002 Device 001: ID 0000:0000  
Bus 003 Device 002: ID 067b:2303 Prolific Technology, Inc. PL2303 Serial Port
Bus 003 Device 001: ID 0000:0000  
Bus 001 Device 001: ID 0000:0000

EDIT2:

Output of DMESG on box A:

usb 3-2: new full speed USB device using uhci_hcd and address 2
usb 3-2: configuration #1 chosen from 1 choice
usbcore: registered new driver usbserial
drivers/usb/serial/usb-serial.c: USB Serial support registered for generic
usbcore: registered new driver usbserial_generic
drivers/usb/serial/usb-serial.c: USB Serial Driver core
drivers/usb/serial/usb-serial.c: USB Serial support registered for pl2303
pl2303 3-2:1.0: pl2303 converter detected
usb 3-2: pl2303 converter now attached to ttyUSB0
usbcore: registered new driver pl2303
drivers/usb/serial/pl2303.c: Prolific PL2303 USB to serial adaptor driver

EDIT3:

box B (I've increased baud rate since):

stty -a -F /dev/ttyS0
speed 115200 baud; rows 0; columns 0; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^H; eof = ^D; eol = ; eol2 = ; swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R;
werase = ^W; lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke

Running screen with such params on box A ends the same way:

screen /dev/ttyUSB0 115200,-parenb,-parodd,cs8,hupcl,-cstopb,cread,clocal,-crtscts,-ignbrk,-brkint,-ignpar,-parmrk,-inpck,-istrip,-inlcr,-igncr,-icrnl,-ixon,-ixoff,-iuclc,-ixany,imaxbel,-iutf8,-opost,-olcuc,-ocrnl,-onlcr,-onocr,-onlret,-ofill,-ofdel,nl0,cr0,tab0,bs0,vt0,ff0,-isig,-icanon,-iexten,-echo,-echoe,-echok,-echonl,-noflsh,-xcase,-tostop,-echoprt,-echoctl,-echoke

I'm looking for a simple tool to report server bandwidth usage based on IP/Process.

The reason for that is one of our servers is "bleeding" bandwidth and there is a need to find out where it's going.

I'm currently using vnstat, but it only reports usage as far as I know.

This happens only on large commit (resulting in a failed commit):

Revelant section from virtual host config in Apache

   <LimitExcept GET PROPFIND OPTIONS REPORT>
      Require valid-user
   </LimitExcept>
   Dav svn
   SVNPath /home/svn/

Commit result:

Transmitting file data ..............................svn: Commit failed
(details follow):
svn: PUT of
'/!svn/wrk/48583f7d-0e01-410d-8941-33d2ba3574b4/WAP/.../htdocs/images/rt.gif':
SSL negotiation failed: SSL error: parse tlsext (https://...)

I found references to it here: http://code.google.com/p/support/issues/detail?id=1395

stating that OpenSSL should be compiled with TLS extension, but in my case, it doesn't error out at the start, just on large commits.

Any ideas? Thanks

EDIT: solved

it appears that switching static MAC definitions combined with mapping vif/0 and vif/0 statically to eth0 and eth1 (in domU, as per http://tomclegg.net/xen-eth0-renamed) done it.

I'm troubleshooting this strange issue (well, for me at least) on my xen domU.

Here's the basic layout:

-------- -------------
|      | | XEN  d0   |
|  A   | | --------- |
|      | | | xen dU| |
-------- -------------

Host A is unrelated machine with IP 213.226.13x.yyy/24

XEN d0 is the hypervisor with IP 213.226.13x.zzz/24

XEN dU is the guest OS (debian etch) I'm trying to configure to see the gateway, it has IP 213.226.13x.jjj/24

The gateway is 213.226.13x.1.

All machines are on the same subnet. The problem is that gues OS can't ping the gateway, but can ping dom0 and host A no problem.

dom0 and host A can ping each other, the gateway and guest.

Things I've tried on host A:

ARP ping

$ arping -c 1 -i eth0 213.226.13x.1

ARPING 213.226.13x.1
60 bytes from 00:02:b3:e8:30:7e (213.226.13x.1): index=0 time=224.113 usec
...

(strangely, arping -c 1 -i eth0 00:02:b3:e8:30:7e does not work).

Broadcast ping

$ ping -b 213.226.13x.255

WARNING: pinging broadcast address
PING 213.226.13x.255 (213.226.13x.255) 56(84) bytes of data.
64 bytes from 213.226.13x.18: icmp_seq=1 ttl=64 time=0.463 ms
64 bytes from 213.226.13x.28: icmp_seq=1 ttl=64 time=0.478 ms (DUP!)
...

Same stuff on xen guest doesn't return any results (both, arping and ping -b).

$ iptables -L output on both, host A and xen guest is the same (zero rules):

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Things I've ruled out as of this time:

damaged physical link - XEN dom0 wouldn't work also, which is not the case.

firewall on gateway - my ARP pings would still get through?

firewall on xen guest - iptables -L has no rules

routing issue - not sure about this one, but I wouldn't be able to ping host A and dom0 with messed up routing table, would I?

Any help is very much appreciated

EDIT1: additional info

I'm not using xen's bridging scripts, network-dummy instead with manually created br-eth1 for eth1 which is the interface for the local network. eth0 is the bridge created by XEN installation for eth0 interface (renamed to peth0 AFAIK).

Here's how /etc/network/interfaces look like on dom0:

# The loopback network interface
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static
  address 213.226.13x.zzz
  netmask 255.255.255.0
  gateway 213.226.13x.1

auto eth1
iface eth1 inet manual

auto br-eth1
iface br-eth1 inet static
  address 192.168.1.11
  netmask 255.255.255.0

And "brctl show" (with domU being off)..

bridge name bridge id       STP enabled interfaces
br-eth1     8000.000423d0d059   no      eth1
eth0        8000.000423d0d058   no      peth0

EDIT2: additional info

XEN config for domU:

#
#  Bootloader + kernel + memory size
#
bootloader  = '/usr/lib/xen-3.2-1/bin/pygrub'
kernel      = '/boot/vmlinuz-2.6.18-5-xen-686'
ramdisk     = '/boot/initrd.img-2.6.18-5-xen-686'
memory      = '3072'
vcpus       = '4'

#
#  Disk device(s).
#
root        = '/dev/sda1 ro'
disk        = [
                  'phy:/dev/vg_main/domain-disk,sda1,w',
                  'phy:/dev/vg_main/domain-swap,sda2,w',
              ]


#
#  Hostname
#
name        = '...'

#
#  Networking
#
vif = [
        'ip=213.226.13x.jjj,mac=00:16:3E:83:0B:BC,bridge=eth0',
        'ip=192.168.1.12,mac=00:16:3E:83:0B:AC,bridge=br-eth1',
]

#
#  Behaviour
#
on_poweroff = 'destroy'
on_reboot   = 'restart'
on_crash    = 'restart'

Having to manage a couple of remote dedicated servers, working with network related stuff can be quite stressful.

Is there any possible way I could connect to those boxes (possibly via third box) and not have to depend on their networking configuration at all?

Something like "xm console", but not xen related? Hope my question is clear enough to answer.

For example:

$ ifconfig dummy0 up
$ ifconfig dummy0 "192.168.1.190 netmask 255.255.255.0"

Calling ifconfig with no parameters shows the interface

dummy0    Link encap:Ethernet  HWaddr b6:1f:f3:92:6d:20  
      inet addr:192.168.1.190  Bcast:192.168.1.255  Mask:255.255.255.0
      inet6 addr: fe80::b41f:f3ff:fe92:6d20/64 Scope:Link
      UP BROADCAST RUNNING NOARP  MTU:1500  Metric:1
      RX packets:0 errors:0 dropped:0 overruns:0 frame:0
      TX packets:15 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:0 
      RX bytes:0 (0.0 B)  TX bytes:1050 (1.0 KiB)

How can I bring the interface down so that it doesn't show up in

• ifconfig
• ifconfig -a
• ifconfig dummy0

without rmmod dummy

because dummy is used just for example purposes.

If there is no way to do that, what "throw-away" IP could I set to it and be safe from any trouble?

like

$ ifconfig dummy0 down
$ ifconfig dummy0 0.0.0.0

Is bonding the preferred way of implementing ISP redundancy?

In the texts I've seen, bond device has a netmask, gateway of it's own. How can this be obtained if there are two different gateways from two uplinks, which one to choose?

Do I need any special routing rules to go with it or does simply configuring separate interfaces (using Debian, /etc/network/interfaces), i.e eth1, eth2 for their corresponding uplinks and bonding them to bond0 handle routing automatically?

If I want to NAT client machines, do they use bond device's IP as a gateway? Does the bond0 device is the device that goes into iptables nat rules?

Thanks

What are my options to forward two USB connected phones to xen guest?

I've read about PCI-passthrough http://www.wlug.org.nz/XenPciPassthrough, but I'm sure usb controller in the server isn't a pci card.

There's device level forwarding, but I need to forward two devices, this here doesn't say how to do it: http://www.olivetalks.com/2008/02/03/usb-forwarding-on-xen-it-just-does-not-work/

Would something as simple as:

usbdevice = [ 'host:xxx', 'host:yyy', ]

work?

EDIT: I'm now starting a bounty. This is really important for me and for other people also, hoping someone who have this resolved will be able to help.

How should I go about installing debian to a remote server, that I have no physical access to?

Background about the server: debian installation, no xen or lvm, ext3 fs takes all disk with 11G free space.

Here's my plan of action, please comment and suggest improvements.

• attempt to shrink mounted partition to current_data_size + 2G.
• use fdisk/mkfs.ext3 to create a new partition in space that was freed up
• install a temporary FS in the new partition (could be debian) using some un-attended/remote installation technique (any suggestions?)
• Modify grub's menu.lst to boot kernel from the new partition (is that enough to get the new OS running?)
• reboot..
• After getting into the new OS (how do I make sure it has an active sshd server?) use FS tools to wipe to old OS, use resize2fs to grow the new partition to overtake the space that was freed up from clearing old one.

Some random questions: can I install new OS using LVM and then, when additional space is available, extend VGs/LVs to take it up?

EDIT:

Am I correct having realized that shrinking a mounted filesystem has pitfalls, but shrinking a mounted partition is impossible?

The system has an unsed partition for swap, 2.5G size, maybe I'll be able to swapoff it and format for new OS installation, what do you think?

What are my options to mirror a running system (without taking it down) to have an as close as possible copy for use as a XEN guest?

Now I'm planning to install the same version of OS as domU and simply rsync everything down to it from the live system. Not sure about /boot | kernel stuff though, any problems syncing these?

Also, can I set a remote "nice" level for rsync, cap bandwidth - to avoid putting stress on the live server?

In gentoo, this is as simple as:

dhcp_ethX="nodns nogateway"

How should I go about this in the interfaces file?

I have 3 NICs that I want to be seen by my domU, even though they're not configured in dom0.

Here's my sample lines from interfaces file

auto lo
iface lo inet loopback

# Local network, cable labeled M3
auto eth0
allow-hotplug eth0
iface eth0 inet static
        address 192.168.1.184
        netmask 255.255.255.0
        gateway 192.168.1.1

# cable labeled M1
auto eth1
allow-hotplug eth1
iface eth1 inet manual
        hwaddress ether 00:19:5B:33:86:D5
        up ifconfig eth1 up

# cable labeled M2
auto eth2
allow-hotplug eth2
iface eth2 inet manual
        hwaddress ether 00:19:5B:33:86:D3
        up ifconfig eth2 up

I'm trying to use multiple bridge configuration with xend, but it complains about the "Link not in running state" and interface being down, although I can see them in ifconfigs' output.

What are my options to make those NICs visible in domU without configuring them in dom0?

I'm currently trying to simulate the environment via XEN. I have installed two debian systems with such FS layout:

cltest1:/etc# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/xvda2            6.0G  417M  5.2G   8% /
tmpfs                 257M     0  257M   0% /lib/init/rw
udev                   10M   16K   10M   1% /dev
tmpfs                 257M  4.0K  257M   1% /dev/shm

Host cltest2 is identical.

Here's my drbd.conf

global {
  minor-count 1;
}

resource mysql {
  protocol C;

  syncer {
    rate 10M; # 10 Megabytes
  }

  on cltest1 {
    device    /dev/drbd0;
    disk      /dev/xvda2;
    address   192.168.1.186:7789;
    meta-disk internal;
  }

  on cltest2 {
    device    /dev/drbd0;
    disk      /dev/xvda2;
    address   192.168.1.187:7789;
    meta-disk internal;
  }
}

I have not created filesystem on drbd0

Starting DRBD via init.d script errors out with:

Starting DRBD resources:    [ d(mysql) /dev/drbd0: Failure: (114) Lower device is already claimed. This usually means it is mounted.

[mysql] cmd /sbin/drbdsetup /dev/drbd0 disk /dev/xvda2 /dev/xvda2 internal --set-defaults --create-device  failed - continuing!

Running: drbdadm create-md mysql gives:

cltest1:/etc# drbdadm create-md mysql
md_offset 6442446848
al_offset 6442414080
bm_offset 6442217472

Found ext3 filesystem which uses 6291456 kB
current configuration leaves usable 6291228 kB

Device size would be truncated, which
would corrupt data and result in
'access beyond end of device' errors.
You need to either
   * use external meta data (recommended)
   * shrink that filesystem first
   * zero out the device (destroy the filesystem)
Operation refused.

Command 'drbdmeta /dev/drbd0 v08 /dev/xvda2 internal create-md' terminated with exit code 40
drbdadm aborting

As I understand, all of my problems are because I don't have unallocated disk space on xvda2.

What are my options besides shrinking FS and connecting a separate physical disk? Can't the meta-data be stored on a file in the local filesystem?

Here's the situation, there's a small server room in our office (~4 active servers), but only one monitor/keyboard. Currently monitor and keyboard are connected to the most used server, but if the need to diagnose a different box would arise, I'd have to reconnect them manually.

Is there any (preferably cheap) solution that would allow me to switch to what server monitor/keyboard is connected without physically reconnecting them?

Monitor + Keyboard | | periferial switch | | server1... server2...

Can't solve this one, here's my .htaccess:

AuthPAM_Enabled Off

AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthName "MESSAGE"
Require ldap-group cn=CHANGED, cn=CHANGED

AuthLDAPURL "ldap://localhost/dc=CHANGED,dc=CHANGED?uid?sub?(objectClass=posixAccount)"
AuthLDAPBindDN CHANGED
AuthLDAPBindPassword CHANGED
AuthLDAPGroupAttribute memberUid

AuthLDAPURL is correct, BindDN and BindPassword are correct also (verified with ldapvi -D ..).

Apache version: Apache/2.2.9 (Debian)

The error message seems cryptic to me, I have AuthzLDAPAuthoritative on so where's the problem.

EDIT:

LDAP modules are loaded, the problem is not with them being missing.

# ls /etc/apache2/mods-enabled/*ldap*
/etc/apache2/mods-enabled/authnz_ldap.load  /etc/apache2/mods-enabled/ldap.load

EDIT2:

Solved it by changing funky

Require ldap-group cn=CHANGED, cn=CHANGED

line with

Require valid-user

Since AuthzLDAPAuthoritative is on, no other auth methods will be used and valid-user requirement will auth via LDAP. (right? :/)

What is the fastest way of computing real directory size? I somehow catch myself needing that a lot.

Simply doing:

# du -hs /dir

Is too slow. Is there any service I could run which would periodically compute directory sizes and cache them for later reference? (something like a locate database)

I wonder what is the general preference and where does it come from. Having used FreeBSD actively for a couple of years I'm leaning towards Gentoo, but I've had an unpleasant experience of wasting valuable time because Gentoo was installed on a really old machine with unfunny build times.

Had it been something prebuilt package based, less time would be wasted. (I know FreeBSD has binary packages, but problem with these are that they're seemingly unmaintained past a version release and the only way to get fresh stuff is to compile it yourself. Don't know about Gentoo really).

What's your take on this, have you had any experiences that made you switch from source based distro to something else?

EDIT: think I should clarify this recent "experience of mine"

The story goes like this. Having changed jobs, I've got to maintain a rather old machine which hosts LDAP (OpenLDAP) with office user login information. It came to me having to reboot the beast (it wasn't rebooted for 8 months). After the reboot, OpenLDAP didn't came online. It seems like slapd and some other binaries got removed somehow while the system was running. After building ldap package for the first time, I wondered why I got no slapd binary (took 15mins). Some time later I've tracked the issue down to "minimal" flag being on by default, which builds just the libs, no server binaries. End result - ~1hour of crippled office productivity and colleagues using excuses as "I didn't do that because our main server was down".

--

Really appreciate your answers, but so far no compelling arguments to use source based vs prebuilt package based. Storage space is not an issue these days.

How viable as a backup strategy would be periodical LVM snapshots of xen domU's? Pros, cons, any gotchas?

To me it seems like the perfect solution for a fast, brainless restore. Any investigation could take place on the broken logical volume with domU successfuly running without interruption.

EDIT:

Here's where I'm at now, when doing full system backups.

• lvm snapshot of domU disk
• a new logical volume which size equals the snapshot size.
• dd if=/dev/snapshot of=/dev/new_lv
• disposing of snapshot with lvremove
• optional verification with kpartx/mount/ls

Now I need to automate this.