Home / user-30525

pehrs's questions

Martin Hope
pehrs
Asked: 2015-03-15 06:38:58 +0800 CST
  • 2

I have run into a problem with NFS which has me baffled. I can't even come up with a plausible explanation for it. I have ten machines:

slave1 (10.0.0.10) - slave10 (10.0.0.20)

Each one runs NFS server and exports a directory.

I mount the directories as usual:

sudo mount 10.0.0.11:/var/export /mnt/slave/1/ -o soft
sudo mount 10.0.0.12:/var/export /mnt/slave/2/ -o soft
...
sudo mount 10.0.0.20:/var/export /mnt/slave/10/ -o soft

At this point the mounts look like this:

10.0.0.11:/var/export on /mnt/slaves/1 type nfs (rw,soft,vers=4,addr=10.0.0.11,clientaddr=10.3.3.212)
10.0.0.12:/var/export on /mnt/slaves/2 type nfs (rw,soft,vers=4,addr=10.0.0.12,clientaddr=10.3.3.212)
10.0.0.13:/var/export on /mnt/slaves/3 type nfs (rw,soft,vers=4,addr=10.0.0.13,clientaddr=10.3.3.212)
10.0.0.14:/var/export on /mnt/slaves/4 type nfs (rw,soft,vers=4,addr=10.0.0.14,clientaddr=10.3.3.212)
10.0.0.15:/var/export on /mnt/slaves/5 type nfs (rw,soft,vers=4,addr=10.0.0.15,clientaddr=10.3.3.212)
10.0.0.16:/var/export on /mnt/slaves/6 type nfs (rw,soft,vers=4,addr=10.0.0.16,clientaddr=10.3.3.212)
10.0.0.17:/var/export on /mnt/slaves/7 type nfs (rw,soft,vers=4,addr=10.0.0.17,clientaddr=10.3.3.212)
10.0.0.18:/var/export on /mnt/slaves/8 type nfs (rw,soft,vers=4,addr=10.0.0.18,clientaddr=10.3.3.212)
10.0.0.19:/var/export on /mnt/slaves/9 type nfs (rw,soft,vers=4,addr=10.0.0.19,clientaddr=10.3.3.212)
10.0.0.20:/var/export on /mnt/slaves/10 type nfs (rw,soft,vers=4,addr=10.0.0.20,clientaddr=10.3.3.212)

Now, for what has me baffled. Some of these mounts are, randomly, pointing at the wrong server!

For example, the files /mnt/slaves/2 may be from slave5 (10.0.0.15). Or perhaps from slave9. Or if I am lucky from slave2. Unmounting and mounting again, using exactly the same line causes the mount to randomly point at another slave. By remounting enough times (and after each check if I got the right slave) I can get the mounts right, but it is a big annoyance.

This problem is showing up using ubuntu 14.04.1 LTS.

Some info:

  • It's not DNS. We are not using DNS.
  • It's not an IP/Mac address conflict. All other prototocols (SSH, HTTP etc) goes to the right slave.
  • I can't see anything in the logs, as far as I can tell the master thinks it communicates with the right slave.
  • Using wireshark I see some traffic to the right slave. The master sends the SETCLIENTID command, so it initiates communication. After that communication just stops. No SETCLIENTID_CONFIRM is sent.

So at this point I wonder:

  • Is this some kind of known bug?
  • Is there a workaround for it?

I have managed to reproduce the same behavior on two different Ubuntu systems setup at different times, so it is not isolated to a single server.

ubuntu
Martin Hope
pehrs
Asked: 2014-12-03 02:35:53 +0800 CST
  • 5

I have a number of systems on which I need to modify the kernel cmdline, adding a few options.

At the moment I do it using the following procedure:

  1. Open /etc/default/grub
  2. Modify GRUB_CMDLINE_LINUX_DEFAULT, adding the options
  3. Run update-grub
  4. Reboot

However, I would prefer to automate this process as a part of a packet installation, and avoid modifying the default file (as this is generally brittle).

What I would want to do is something like:

  1. Drop a file (in /etc/grub.d/?), overriding GRUB_CMDLINE_LINUX_DEFAULT or similar
  2. Run update-grub and reboot

There are a pile of scripts in /etc/grub.d/* which are used to build the actual menu config, however, there is no obvious way to interact with them. The script which generates the config only seems to read /etc/default/grub :(

Can somebody enlighten me if there is a way to drop a file to modify the default kernel commandline?

ubuntu
Martin Hope
pehrs
Asked: 2014-01-15 06:39:49 +0800 CST
  • 1

The setup is as follows. Ubuntu-11.10 VM running on an ESXi host. The VM has 3 vCPU assigned.

When running dump (to backup the harddisk) it seems all the processes are assigned to the first vCPU. As we are using compression this loads the vCPU at 100%, slowing down the dump to a crawl.

I have (by using taskset) spread the dump processes to other vCPUs and thereby gained a factor 3 speedup of the backup. Their original affinity was set to 0-2, and if I remove the affinity I have set they all move back to the first vCPU.

I can not fathom what may cause the scheduler to allocate all processes to the same CPU. Can somebody please explain why the task-scheduler may act like this?

ubuntu
Martin Hope
pehrs
Asked: 2012-09-27 02:01:07 +0800 CST
  • 0

I have run into an interesting problem with a Juniper SRX 240 box. The box is running BGP as a border router, and is announcing a /24. To ensure that the vlan with the network stayed "up" I set the gigether-options loopback option on one of the interfaces which is part of an ethernet-switching bridge. This forced the vlan up, just as I wanted, and ensured the box did not withdraw the announcement in BGP when all servers behind the box were removed.

Sadly, the same option seems to wreck havoc on the ARP tables in the SRX. It causes 5-30 seconds long bursts of packet loss (around 5% over long periods of time) and I get duplicate answers to ARP queries.

  1. Is this behavior documented somewhere?
  2. Is there a better way to keep the vlan up?
juniper
Martin Hope
pehrs
Asked: 2011-05-06 08:35:39 +0800 CST
  • 2

As you are probably aware World IPv6 day is coming on 8 June. Our network is IPv6 enabled, so the basics are all there, but our users currently defaults to IPv4 for almost everything (IPv6 is <1% of the external traffic). I have verified that our DNS servers handles IPv6 properly, but I am not certain what to expect when the users start to get AAAA records for services like Google and Akamai.

  • What should I expect to break on World IPv6 day?
ipv6
Martin Hope
pehrs
Asked: 2011-01-25 10:49:42 +0800 CST
  • 0

I have a rather complicated issue with an old Cisco 5509 Switch and a few kvm/qemu based virtual machines. First, the setup of the system looks like this:

|--------------------------------------------     -------------|
||----------|             VMHOST            |     |5509        |
||VM1       |                               |     |            |
||    ------|                               |     |         101|-------Juniper
||    |vmnic|---vnet0<->br0<->eth3.101--eth3|-----|Trunk       |
||----------|                               |     |            |
|                                           |     |            |
|--------------------------------------------     --------------

So, the virtual machine is not vlan aware, but connects over a bridge that tags up the traffic with vlan 101, this vlan is then sent in on a trunk port on the 5509, which in sends out the traffic on a port (101) removing the tagging delivering it juniper.

The problem is that this setup works well for broadcast traffic. I can arping between vm1 and Juniper. However, unicast is lost somewhere between juniper and eth3, but only traffic from vm1 to juniper!

Some logs: Arping and ping from vm1 to host

sudo arping 192.168.0.2
ARPING 192.168.0.2
60 bytes from 00:05:85:cc:f2:10 (192.168.0.2): index=0 time=3.354 msec
60 bytes from 00:05:85:cc:f2:10 (192.168.0.2): index=1 time=3.739 msec
60 bytes from 00:05:85:cc:f2:10 (192.168.0.2): index=2 time=1.511 msec
^C
--- 192.168.0.2 statistics ---
3 packets transmitted, 3 packets received,   0% unanswered (0 extra)
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
^C
--- 192.168.0.2 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2004ms

And the dump from eth3 when running the two commands

sudo tcpdump -ei eth3
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
19:25:06.871102 00:16:3e:3e:02:11 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Request who-has 192.168.0.2 tell 192.168.0.1, length 28
19:25:06.872563 00:05:85:cc:f2:10 (oui Unknown) > 00:16:3e:3e:02:11 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 101, p 0, ethertype ARP, Reply 192.168.0.2 is-at 00:05:85:cc:f2:10 (oui Unknown), length 46
19:25:07.871848 00:16:3e:3e:02:11 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Request who-has 192.168.0.2 tell 192.168.0.1, length 28
19:25:07.874369 00:05:85:cc:f2:10 (oui Unknown) > 00:16:3e:3e:02:11 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 101, p 0, ethertype ARP, Reply 192.168.0.2 is-at 00:05:85:cc:f2:10 (oui Unknown), length 46
19:25:08.872454 00:16:3e:3e:02:11 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Request who-has 192.168.0.2 tell 192.168.0.1, length 28
19:25:09.028734 00:05:85:cc:f2:10 (oui Unknown) > 00:16:3e:3e:02:11 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 101, p 0, ethertype ARP, Reply 192.168.0.2 is-at 00:05:85:cc:f2:10 (oui Unknown), length 46
19:25:13.686148 00:16:3e:3e:02:11 (oui Unknown) > 00:05:85:cc:f2:10 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.1 > 192.168.0.2: ICMP echo request, id 1002, seq 1, length 64
19:25:14.690923 00:16:3e:3e:02:11 (oui Unknown) > 00:05:85:cc:f2:10 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.1 > 192.168.0.2: ICMP echo request, id 1002, seq 2, length 64
19:25:15.690788 00:16:3e:3e:02:11 (oui Unknown) > 00:05:85:cc:f2:10 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.1 > 192.168.0.2: ICMP echo request, id 1002, seq 3, length 64

Ping in the other direction: 

run ping 192.168.0.1    
PING 192.168.0.1 (192.168.0.1): 56 data bytes
^C
--- 192.168.0.1 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

And the associated packet dump from eth3, showing that the packets get from Juniper to vm1 and all the way back to eth3 before they disapear.

sudo tcpdump -ei eth3
tcpdump: WARNING: eth3: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth3, link-type EN10MB (Ethernet), capture size 65535 bytes
19:27:46.960138 00:05:85:cc:f2:10 (oui Unknown) > 00:16:3e:3e:02:11 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.2 > 192.168.0.1: ICMP echo request, id 61736, seq 0, length 64
19:27:46.970773 00:16:3e:3e:02:11 (oui Unknown) > Broadcast, ethertype 802.1Q (0x8100), length 46: vlan 101, p 0, ethertype ARP, Request who-has 192.168.0.2 tell 192.168.0.1, length 28
19:27:46.972689 00:05:85:cc:f2:10 (oui Unknown) > 00:16:3e:3e:02:11 (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 101, p 0, ethertype ARP, Reply 192.168.0.2 is-at 00:05:85:cc:f2:10 (oui Unknown), length 46
19:27:46.973052 00:16:3e:3e:02:11 (oui Unknown) > 00:05:85:cc:f2:10 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.1 > 192.168.0.2: ICMP echo reply, id 61736, seq 0, length 64
19:27:47.959952 00:05:85:cc:f2:10 (oui Unknown) > 00:16:3e:3e:02:11 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.2 > 192.168.0.1: ICMP echo request, id 61736, seq 1, length 64
19:27:47.960300 00:16:3e:3e:02:11 (oui Unknown) > 00:05:85:cc:f2:10 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.1 > 192.168.0.2: ICMP echo reply, id 61736, seq 1, length 64
19:27:49.048280 00:05:85:cc:f2:10 (oui Unknown) > 00:16:3e:3e:02:11 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.2 > 192.168.0.1: ICMP echo request, id 61736, seq 2, length 64
19:27:49.048618 00:16:3e:3e:02:11 (oui Unknown) > 00:05:85:cc:f2:10 (oui Unknown), ethertype 802.1Q (0x8100), length 102: vlan 101, p 0, ethertype IPv4, 192.168.0.1 > 192.168.0.2: ICMP echo reply, id 61736, seq 2, length 64
8 packets captured
8 packets received by filter
0 packets dropped by kernel

Some relevant configuration. First vm1:

eth1      Link encap:Ethernet  HWaddr 00:16:3e:3e:02:11  
          inet addr:192.168.0.1  Bcast:192.168.0.255  Mask:255.255.255.0
          inet6 addr: fe80::216:3eff:fe3e:211/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:1953 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3933 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:153032 (153.0 KB)  TX bytes:315162 (315.1 KB)
          Interrupt:10 Base address:0x6000

Then vmhost:

brctl show
bridge name bridge id       STP enabled interfaces
br0     8000.001e68a9b341   no      eth3.101
                            vnet0

And the interfaces and bridges (non relevant stuff removed)

eth3      Link encap:Ethernet  HWaddr 00:1e:68:a9:b3:41  
          inet6 addr: fe80::21e:68ff:fea9:b341/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:4306 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4870 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:329486 (329.4 KB)  TX bytes:419680 (419.6 KB)
          Interrupt:47 Base address:0xc000 

eth3.101  Link encap:Ethernet  HWaddr 00:1e:68:a9:b3:41  
          inet6 addr: fe80::21e:68ff:fea9:b341/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:2082 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3697 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:159118 (159.1 KB)  TX bytes:306482 (306.4 KB)

br0       Link encap:Ethernet  HWaddr 00:1e:68:a9:b3:41  
          inet6 addr: fe80::490:41ff:fea8:25bd/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:4006 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:296858 (296.8 KB)  TX bytes:468 (468.0 B)

vnet0     Link encap:Ethernet  HWaddr fe:16:3e:3e:02:11  
          inet6 addr: fe80::fc16:3eff:fe3e:211/64 Scope:Link
          UP BROADCAST RUNNING PROMISC MULTICAST  MTU:1500  Metric:1
          RX packets:3940 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2004 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:500 
          RX bytes:315680 (315.6 KB)  TX bytes:184138 (184.1 KB)

The relevant part of the 5509 configuration

set vlan 100-104
set spantree disable all
set trunk 3/8  on dot1q 101-104,201-204,301-304,401-404,501-504,1002-1005
set vlan 101  4/1

And finally the juniper box

fe-0/0/0 {
    unit 0 {
        family inet {
            address 192.168.0.2/24;
        }
    }
}

At this point I am starting to strongly suspect a configuration issue somewhere in the 5509 or bridge setup, but I can't even start to imaging what might cause this issue. Anybody with networking experience that can suggest a way to attack this problem? If you need any more information just ask.

Edit:

Some further debugging hints that this is related to the filters in the Juniper box. I still don't know what's going on, but the "Input DA rejects" counter increases with each lost packet. 

run show interfaces fe-0/0/0 extensive    
Physical interface: fe-0/0/0, Enabled, Physical link is Up
  Interface index: 129, SNMP ifIndex: 118, Generation: 130
  Link-level type: Ethernet, MTU: 1514, Link-mode: Half-duplex, Speed: 100mbps,
  MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Enabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  CoS queues     : 8 supported, 8 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 00:05:85:cc:f2:10, Hardware address: 00:05:85:cc:f2:10
  Last flapped   : 2011-01-24 19:03:05 CET (16:10:25 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :               109620                    0 bps
   Output bytes  :               331366                    0 bps
   Input  packets:                 2035                    0 pps
   Output packets:                 5611                    0 pps
  Input errors:
    Errors: 1, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0,
    L3 incompletes: 1, L2 channel errors: 0, L2 mismatch timeouts: 0,
    FIFO errors: 0, Resource errors: 0
  Output errors:
    Carrier transitions: 7, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0,
    FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Egress queues: 8 supported, 4 in use
  Queue counters:       Queued packets  Transmitted packets      Dropped packets
    0 best-effort                 5611                 5611                    0
    1 expedited-fo                   0                    0                    0
    2 assured-forw                   0                    0                    0
    3 network-cont                   0                    0                    0
  Active alarms  : None
  Active defects : None
  MAC statistics:                      Receive         Transmit
    Total octets                             0           331926
    Total packets                            0             5611
    Unicast packets                          0             3234
    Broadcast packets                        0             2377
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
  Filter statistics:
    Input packet count                   64407
    Input packet rejects                 62371
    Input DA rejects                     62371
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 1, CAM source filters: 0
  Autonegotiation information:
    Negotiation status: Complete
    Link partner:
        Link mode: Full-duplex, Flow control: None, Remote fault: OK,
        Link partner Speed: 100 Mbps
  Packet Forwarding Engine configuration:
    Destination slot: 0
    Direction : Output 
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95       95000000    95              0      low    none
    3 network-control         5        5000000     5              0      low    none

  Logical interface fe-0/0/0.0 (Index 68) (SNMP ifIndex 136) (Generation 133)
    Flags: SNMP-Traps Encapsulation: ENET2
    Traffic statistics:
     Input  bytes  :               222600
     Output bytes  :               331366
     Input  packets:                 2035
     Output packets:                 5611
    Local statistics:
     Input  bytes  :               112980
     Output bytes  :               328006
     Input  packets:                 1995
     Output packets:                 5571
    Transit statistics:
     Input  bytes  :               109620                    0 bps
     Output bytes  :                 3360                    0 bps
     Input  packets:                   40                    0 pps
     Output packets:                   40                    0 pps
    Protocol inet, MTU: 1500, Generation: 139, Route table: 0
      Flags: None
      Addresses, Flags: Is-Preferred Is-Primary
        Destination: 192.168.0/24, Local: 192.168.0.2, Broadcast: 192.168.0.255,
        Generation: 140
vlan kvm-virtualization cisco bridge juniper
Martin Hope
pehrs
Asked: 2010-12-21 05:31:44 +0800 CST
  • 13

I have a virtualization server to which I connect over ssh. If I now change the size of the window it will automatically propagate to the server. It's most easily seen using stty -a, checking the columns and row values.

I then use virsh console to connect to the serial interface on a KVM based virtual machine. When I now change the size of the window it does not propagate to the virtual server. This is most easily seen by checking stty -a, which is not updated on the virtual machine when I change window size. This means that line breaks does not work correctly in the terminal and any application that relies on window size for formatting (emacs, man, etc) gets messed up unless the window size on the client matches the default size on the server.

A workaround is to manually set the window size to match the client window using stty, but I wonder if there is any way to get this information to propagate and set the window size in the virtual machine automatically.

linux kvm-virtualization terminal stty
Martin Hope
pehrs
Asked: 2010-11-06 08:34:20 +0800 CST
  • 1

I am converting some scripts from an ancient SuSe install, after a swap to ubuntu. These scripts uses a tool called mkpasswd, which sets user passwords. I wonder if there is something similar for Ubuntu so I don't have to code it myself. The tool is very simple. It reads a file like

username1:password2
username2:password2
....
usernameN:passwordN

And sets the passwords on the users.

ubuntu bulk-action password-management
Martin Hope
pehrs
Asked: 2010-10-29 05:08:21 +0800 CST
  • 0

On one of our OpenSUSE (11.3) virtualization servers we have had some stability problems recently. The server is pretty heavily loaded, with up to 100 virtual domains, all running on LVM snapshots. We have had the server suddenly hang (still responding on ping, unable to login).

The server writes a lot of messages from LVM into the log, but I am not sure if these are normal or not. This is a sample of the log right before the server hang: 

Oct 28 13:26:35 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:26:35 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:27:15 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:27:15 xen lvm[3384]: dm_task_run failed, errno = 22, Invalid argument
Oct 28 13:28:25 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:30:45 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:30:55 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:30:55 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:05 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:05 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:05 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:15 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:21 xen kernel: [76866.784142] ata3.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Oct 28 13:31:21 xen kernel: [76866.784158] sr 2:0:0:0: [sr0] CDB: Get event status notification: 4a 01 00 00 10 00 00 00 08 00
Oct 28 13:31:21 xen kernel: [76866.784183] ata3.00: cmd a0/00:00:00:08:00/00:00:00:00:00/a0 tag 0 pio 16392 in
Oct 28 13:31:21 xen kernel: [76866.784186]          res 40/00:02:00:08:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
Oct 28 13:31:21 xen kernel: [76866.784192] ata3.00: status: { DRDY }
Oct 28 13:31:25 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:25 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:26 xen kernel: [76871.824061] ata3: link is slow to respond, please be patient (ready=0)
Oct 28 13:31:31 xen kernel: [76876.808057] ata3: device not ready (errno=-16), forcing hardreset
Oct 28 13:31:31 xen kernel: [76876.808080] ata3: soft resetting link
Oct 28 13:31:31 xen kernel: [76876.972455] ata3: nv_mode_filter: 0x739f&0x739f->0x739f, BIOS=0x7000 (0xc0000000) ACPI=0x701f (60:900:0x11)
Oct 28 13:31:31 xen kernel: [76876.988343] ata3.00: configured for UDMA/33
Oct 28 13:31:31 xen kernel: [76876.988971] ata3: EH complete
Oct 28 13:31:35 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:31:45 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:32:05 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:32:35 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:35:55 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:36:05 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 28 13:36:25 xen lvm[3384]: Another thread is handling an event. Waiting...

At these point I have four questions:

  1. Are these log messages normal for LVM?
  2. Why does dm_task_run print errors about invalid arguments in the log?
  3. What might cause the SATA link reset?
  4. Does any of this hint what might cause the server to hang?

Edit: On the next hang I got some more information:

Oct 30 01:59:25 xen lvm[3384]: Another thread is handling an event. Waiting...
Oct 30 02:01:15 xen kernel: [208260.160037] BUG: soft lockup - CPU#3 stuck for 67s! [qemu-dm:12987]
Oct 30 02:01:15 xen kernel: [208260.160053] BUG: soft lockup - CPU#4 stuck for 67s! [qemu-dm:26429]
Oct 30 02:01:15 xen kernel: [208260.160070] Modules linked in:
Oct 30 02:01:15 xen kernel: [208260.160074] Modules linked in: bridge stp llc bridge usbbk stp llc gntdev usbbk gntdev netbk netbk blkbk blkbk blkback hcd usbcore sd_mod dm_snapshot xenblk ehci_hcd cdrom xennet usbcore edd dm_mod dm_snapshot fan pata_amd xenblk sata_nv libata cdrom scsi_mod thermal xennet processor thermal_sys edd hwmon
Oct 30 02:01:15 xen kernel: [208260.160276]  dm_modCPU 4
Oct 30 02:01:15 xen kernel: [208260.160283] Modules linked in: fan bridge stp pata_amd llc usbbk gntdev netbk sata_nv blkbk blkback_pagemap libata blktap domctl xenbus_be scsi_mod evtchn nfs lockd thermal fscache nfs_acl auth_rpcgss processor sunrpc fuse loop thermal_sys tg3 pcspkr i2c_nforce2 hwmon i2c_core ghes 8250_pnp
Oct 30 02:01:15 xen kernel: [208260.160342] CPU 3  8250
Oct 30 02:01:15 xen kernel: [208260.160346] Modules linked in: shpchp bridge stp llc k10temp usbbk gntdev netbk hed blkbk blkback_pagemap blktap domctl sr_mod xenbus_be evtchn nfs forcedeth pci_hotplug lockd serial_core serio_raw sg fscache nfs_acl button auth_rpcgss sunrpc fuse usbhid hid linear ohci_hcd loop sd_mod ehci_hcd tg3 pcspkr i2c_nforce2 usbcore i2c_core ghes 8250_pnp dm_snapshot 8250 shpchp k10temp xenblk hed sr_mod cdrom forcedeth xennet edd dm_mod pci_hotplug serial_core fan serio_raw sg pata_amd button usbhid hid sata_nv libata linear scsi_mod ohci_hcd sd_mod thermal ehci_hcd processor thermal_sys hwmon usbcore dm_snapshot
Oct 30 02:01:15 xen kernel: [208260.160471]  xenblk
Oct 30 02:01:15 xen kernel: [208260.160477]  cdrom
xen lvm snapshot
Martin Hope
pehrs
Asked: 2010-06-01 00:59:08 +0800 CST
  • 2

I have an Ubuntu server running apache2 which i expect to be hit by around 500-1000 (concurent) users for a limited amount of time. The server serves a mixture of custom (rather light) php pages connected to a postgresql db (around 20 Mb in size) and static content. The hardware is stable and pretty beefy:

  • Intel Xeon E5420 @ 2.5 GHz
  • 12 GB RAM

During previous rushes on this server I have increased ServerLimit, the MaxClients for the MPM modules and decreased Timeout and KeepAliveTimeout. It has worked, but been sluggish and I have a feeling more can be done. How would you suggest configuring the Apache server to handle this kind of load?

apache-2.2 configuration scaling
Martin Hope
pehrs
Asked: 2010-04-18 02:44:42 +0800 CST
  • 2

I have a Linux log server where multiple applications write data. Data is written in bursts, and in a lot of different files. I need to make a backup of this mess, preferably preserving as much coherence between the file versions as possible and avoiding getting truncated files. Total amount of data on the server is about 100Gb. What I really would want (but can't) is to shut-down, backup the system cold and then start it up again.

What kind of guarantees against concurrent modification does the various backup tools give? When do they "freeze" the file versions? I am looking at rsync, dump and tar at the moment, but I am open for other (open source) alternatives.

Changing the application or blocking writing for backups is sadly not an option. System is not running LVM (yet), but I have considered that for rebuilding the system and then snapshots.

backup rsync dump tar
Martin Hope
pehrs
Asked: 2010-03-16 05:05:26 +0800 CST
  • 4

I have worked a few different places, each one with a different solution, so I would like to hear your suggestions and ideas.

I am looking at a situation with multiple server-rooms. We have a mixture of copper (both ethernet and telephone), multi-mode fibre and single-mode fibre. We have all types of connectors: RJ-11, RJ-45, LC, SC, FC, SMA and several I probably forgot about. We have a lot of people working in the area, and keeping track of cables is turning into a full time job.

So, here are a the questions:

  • What types and lengths of cable do you keep in stock on site?
  • Do you make your own or buy pre-made?
  • Adapters? Dampeners?
  • How do you manage cable inventory?
  • How do you label the cables?
  • Any other tricks to stop this from driving me crazy?
networking cable server-room
Martin Hope
pehrs
Asked: 2010-02-18 05:43:35 +0800 CST
  • 1

I have noticed that my syslog reports incorrect time for some authentication entries. The typical behaviour looks like this: 

Feb 16 13:32:20 dev sshd[29537]: Invalid user oracle from 110.188.0.123
Feb 16 12:32:20 dev sshd[29538]: input_userauth_request: invalid user oracle
Feb 16 13:32:20 dev sshd[29537]: pam_unix(sshd:auth): check pass; user unknown
Feb 16 13:32:20 dev sshd[29537]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=110.188.0.123
Feb 16 13:32:20 dev sshd[29537]: pam_succeed_if(sshd:auth): error retrieving information about user oracle
Feb 16 13:32:22 dev sshd[29537]: Failed password for invalid user oracle from 110.188.0.123 port 64368 ssh2
Feb 16 12:32:23 dev sshd[29538]: Received disconnect from 110.188.0.123: 11: Bye Bye

Notice how the hour swaps back and forth. Has anybody seen a similar behaviour? What might cause this? It seems systematic and it's the same entries for each connection attempt that are reported an hour earlier than they should be. System clock is GMT. Timezone GMT+1.

time ssh timezone syslog
Martin Hope
pehrs
Asked: 2010-02-16 01:40:01 +0800 CST
  • 0

When running SVN UP against a server some of my ubuntu systems doesn't want to store a permanent exception. SVN output looks like this: 

Error validating server certificate for 'https://aaa.bbb.ccc.se:443':
 - The certificate is not issued by a trusted authority. Use the
fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: aaa.bbb.ccc.se
 - Valid: from Tue, 29 Sep 2009 18:58:06 GMT until Thu, 29 Sep 2011 18:58:06 GMT
 - Issuer: CA, yyy, sss, Somewhere, SE
 - Fingerprint: 21:f3:62:28:d2:61:48:d1:40:47:20:6a:f4:a8:c7:75:05:c9:11:c6 
(R)eject or accept (t)emporarily?

Normally I would expect the last line to be

(R)eject, accept (t)emporarily or accept (p)ermanently?

I assume this is an access right problem somewhere, but I can't seem to figure out where.

The last few lines from strace are: 

fstat64(5, {st_mode=S_IFREG|0644, st_size=1892, ...}) = 0
fstat64(5, {st_mode=S_IFREG|0644, st_size=1892, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb6c41000
read(5, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 1892
_llseek(5, -28, [1864], SEEK_CUR)       = 0
read(5, "\nCET-1CEST,M3.5.0,M10.5.0/3\n", 4096) = 28
close(5)                                = 0
munmap(0xb6c41000, 4096)                = 0
lstat64("/home/pehrs/.subversion/auth/svn.ssl.server/d3b23fecd33cba74aff2290b03a87862", 0xbfe6e84c) = -1 ENOENT (No such file or directory)

Before this there is about 200 calls to gettimeofday()

Can somebody help me figure out how to store the exception permanently, so I don't have to check the fingerprint everytime i pull from the repo?

svn permissions ssl ssl-certificate