wazoox's questions

I have a Debian server running KVM+libvirt. The VMs disks are LVs on the host. The trouble is that if I create LVs inside the guests, they are recursively detected on the host and that creates all sort of problems, like the inability to resize or delete properly LVs.

So I added a filter rule to /etc/lvm/lvm.conf like this, to exclude scanning LVs inside other LVs:

filter = [ "r|/dev/dm-*|", "a|/dev/sd.*|" ]

Now my problem is : how do I get LVM to rescan only the "real" VGs and LVs? I now have errors like this:

# lvs
  WARNING: Device for PV 73C0Ma-sAYV-Aa10-pAHB-ZAZA-3QjH-Rk1eTS not found or rejected by a filter.
  WARNING: Device for PV fcEG5k-BLzO-dbAr-Yw8o-dPw2-1hAk-VyluvW not found or rejected by a filter.
  LV              VG              Attr       LSize    Pool Origin    Data%  Meta%  Move Log Cpy%Sync Convert
  HaProxy         data            -wi-ao----   20,00g                                                       
  backup          data            -wi-ao----    4,00t                                                       
  database        data            -wi-ao----   16,00g                                                       
  lvol0           data            swi-a-s---   16,07g      stone-web 70,43                                  
  stone-vtls      data            -wi-ao----    2,00t                                                       
  stone-web       data            owi-aos---   16,00g                                                       
  lv0             ftpstorage      -wi-----p-   10,00t                                                       

The last lv in this case is actually a LV from a VM, not from the host... How do I get rid of it without rebooting the host?

To redirect several subdomains on the same IP to different hosts, I'm using haproxy (1.7.5, Debian stable) and it works fine. However I also want to restrict access to some hosts by IP and display a clean message (proper 403 error page) to rejected hosts, and that's where I can't find a solution.

The best I have so far is a configuration that use a "reject" backend; however I don't know how to configure this one to get anything but SSL Errors from the browser end.

The configuration looks like that:

frontend http_redirect
    bind *:80
    redirect scheme https if !{ ssl_fc }

frontend tls_router
    bind *:443
    mode tcp
    option tcplog
    option tcpka

    acl demo_acl req_ssl_sni -i demo.myhost.org
    acl www_acl req_ssl_sni -i www.myhost.org
    acl demo_network_allowed src 10.1.1.0/24

    use_backend demo_tls if demo_acl
    use_backend wwww_tls if www_acl
    use_backend reject_access if demo_acl !demo_network_allowed

backend www_tls
    mode tcp
    option tcpka
    server www_srv 192.168.1.2:443

backend demo_tls
    mode tcp
    option tcpka
    server demo_srv 192.168.1.3:443

backend reject_access
    mode http
    # errorfile 403 /etc/haproxy/errors/403.http
    # server demo  192.168.1.2:443
    http-request set-path www.myhost.org/403.html
    http-request redirect scheme https if ! { ssl_fc }

As is clear from the "reject_access" backend, I tried several things with the same result:

$ LANG=C wget  --no-check-certificate -S https://demo.myhost.org
--2019-07-01 18:48:31--  https://demo.host.org/
Résolution de demo.myhost.org? 10.12.24.1
Connexion à demo.myhost.org|10.12.24.1|:443? connecté.
OpenSSL: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol
Incapable d'établir une connexion SSL.

Any help in this configuration welcome.

I've got a Debian/Jessie Samba 4.2.14 running as an AD member. ADC is a Windows2008R2 server. Join worked without problem.

# net ads testjoin
Join is OK

wbinfo -u and wbinfo -g work perfectly and provides a list of users and groups from the AD as expected. wbinfo -i <user> works too:

# wbinfo -i TESTAD\\testuser
TESTAD\testuser:*:4294967295:4294967295:testuser:/home/TESTAD/testuser:/bin/false

Edit: something's wrong here, because wbinfo -i maps all users and groups to the id 4294967295 which is, as @TheSkunk remarked, 2^32 -1.

Edit 2: wbinfo --sid-to-uid TESTAD\\testuser fails. Surely I must set up explicitely some idmap parameters (the defaults apparently don't work at all) but how?

Edit 3: I've added these 2 lines to smb.conf:

idmap config * : backend = tdb
idmap config * : range = 10000-30000

And now ẁbinfo -i TESTDOMAIN\testuserreports a valid id, and a different one for each and every user. However I still have the same problems (all users mapping to nobody,idandgetent` not knowing of AD users, etc).

However getent passwd TESTAD\\testuser fails:

# getent passwd TESTAD\\testuser
# echo $? 
2

I can connect to the server with any AD account using smbclient:

# smbclient //srv1/data -U TESTAD\\testuser
Enter TESTAD\testuser's password: 
Domain=[TESTAD] OS=[Windows 6.1] Server=[Samba 4.2.14-Debian]
smb: \> ls
  .                                   D        0  Fri Feb 17 16:23:04 2017
  ..                                  D        0  Wed Feb  1 16:47:02 2017
  test.txt                            N        5  Fri Feb 17 14:38:21 2017
  popo                                D        0  Fri Feb 17 16:23:04 2017

                117125466112 blocks of size 1024. 117052392484 blocks available
smb: \> 

However the connection is mapped to nobody/nogroup, and created files are owned by nobody too. Windows machines fail to connect using any AD account. However if I create a local account with smbpasswd -a <user>, they can connect using it. However, their connection parameters, files, etc. are all mapped to nobody though the account exists locally too.

Here's the current smb.conf (as close to default as possible):

[global]
        workgroup = TESTAD
        realm = TESTAD.lan
        server role = member server
        security = ADS
        map to guest = Bad User
        obey pam restrictions = Yes
        pam password change = Yes
        passwd program = /usr/bin/passwd %u
        passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
        unix password sync = Yes
        syslog = 0
        log file = /var/log/samba/log.%m
        max log size = 1000
        dns proxy = No
        usershare allow guests = Yes
        panic action = /usr/share/samba/panic-action %d
        idmap config * : backend = tdb
        inherit permissions = Yes
        inherit acls = Yes


[DATA]
        path = /mnt/raid/
        read only = No
        guest ok = Yes

here is /etc/nsswitch.conf (I've tried adding and removing 'winbindd from shadow, no change at all):

# cat /etc/nsswitch.conf 
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         compat winbind
group:          compat winbind
shadow:         compat winbind
gshadow:        files

hosts:          files mdns4_minimal [NOTFOUND=return] dns wins
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

I don't understand why authentication never seems to go through winbind. I'm getting desperate, any ideas?

I need to migrate an existing storage server running Debian 6, with many TB of data, to a HA setup. I can't simply shut down this server for any prolonged time (HA is required after all).

I plan to migrate this server to a DRBD cluster. I can't buy two new servers for this cluster; I need to reuse the existing server as a member of the cluster for cost reasons. Remember, I can't shut down anything for more than a few minutes at a time.

I have a new server ready, running Debian 8 to be the "master". I'd need to set up a "split brain" cluster using this "master" server; copy all data from the existing old server to the cluster; switch the existing services to the cluster; reconfigure the old server with Debian 8, then add it to the cluster as a "slave", resynchronize data, and at last have a redundant setup.

I've set up the cluster on the new machine, however, how to proceed? I can't even start the drbd service as the other "node" doesn't even exist yet...

I'm starting up from a previous drbd configuration that worked fine... with two available nodes.

Any help appreciated.

I've set up a pair of identical servers with RAID arrays (8 cores, 16GB RAM, 12x2 TB RAID6), 3 10GigE interfaces, to host some highly available services.

The systems are currently running Debian 7.9 Wheezy oldstable (because corosync/pacemaker are not available on 8.x stable nor testing).

• Local disk performance is about 900 MB/s write, 1600 MB/s read.
• network throughput between the machines is over 700MB/s.
• through iSCSI, each machine can write to the other's storage at more than 700 MB/s.

However, no matter the way I configure DRBD, the throughput is limited to 100MB/s. It really looks like some hardcoded limit. I can reliably lower performance by tweaking the settings, but it never goes over 1Gbit (122MB/s are reached for a couple of seconds at a time). I'm really pulling my hair on this one.

• plain vanilla kernel 3.18.24 amd64
• drbd 8.9.2~rc1-1~bpo70+1

The configuration is split in two files: global-common.conf:

global {
        usage-count no;
}

common {
        handlers {
        }

        startup {
        }

        disk {
                on-io-error             detach;
         #       no-disk-flushes ;
        }
        net {
                max-epoch-size          8192;
                max-buffers             8192;
                sndbuf-size             2097152;
        }
        syncer {
                rate                    4194304k;
                al-extents              6433;
        }
}

and cluster.res:

resource rd0 {
        protocol C;
        on cl1 {
                device /dev/drbd0;
                disk /dev/sda4;
                address 192.168.42.1:7788;
                meta-disk internal;
        }

        on cl2 {
                device /dev/drbd0;
                disk /dev/sda4;
                address 192.168.42.2:7788;
                meta-disk internal;
        }
}

Output from cat /proc/drbdon slave :

version: 8.4.5 (api:1/proto:86-101)
srcversion: EDE19BAA3D4D4A0BEFD8CDE 
 0: cs:SyncTarget ro:Secondary/Secondary ds:Inconsistent/UpToDate C r-----
    ns:0 nr:4462592 dw:4462592 dr:0 al:0 bm:0 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:16489499884
        [>....................] sync'ed:  0.1% (16103024/16107384)M
        finish: 49:20:03 speed: 92,828 (92,968) want: 102,400 K/sec

Output from vmstat 2 on master (both machines are almost completely idle):

procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0      0 14952768 108712 446108    0    0   213   254   16    9  0  0 100  0
 0  0      0 14952484 108712 446136    0    0     0     4 10063 1361  0  0 99  0
 0  0      0 14952608 108712 446136    0    0     0     4 10057 1356  0  0 99  0
 0  0      0 14952608 108720 446128    0    0     0    10 10063 1352  0  1 99  0
 0  0      0 14951616 108720 446136    0    0     0     6 10175 1417  0  1 99  0
 0  0      0 14951748 108720 446136    0    0     0     4 10172 1426  0  1 99  0

Output from iperf between the two servers:

------------------------------------------------------------
Client connecting to cl2, TCP port 5001
TCP window size:  325 KByte (default)
------------------------------------------------------------
[  3] local 192.168.42.1 port 47900 connected with 192.168.42.2 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  6.87 GBytes  5.90 Gbits/sec

Apparently initial synchronisation is supposed to be somewhat slow, but not this slow... Furthermore it doesn't really react to any attempt to throttle sync rate like drbdadm disk-options --resync-rate=800M all.

I've got a setup where a windows 2008R2 VM runs under KVM on a bridged network and captures a lot of multicast UDP data (from videosurveillance cameras). Strangely, the VM network drops packets after a number of concurrent connections ( more than 128 apparently).

kernel is 2.6.35.11 amd64, kvm 0.76 (Debian Lenny).

However if I setup two different tap0 and tap1 interfaces, bridged through the same br0 to eth0 and eth1, the second interface never drops a single packet. The first interface tap0, however, drops lots of packets even if most connections go to the other one.

But if I keep using eth0 for the host, and dedicate eth1 to the VM, eth1 (the VM) will still drops tons of packets (about 20 to 25%). Apparently only the second interface behaves correctly.

Does anyone have any idea about what may be happening here? I can't make sense of it. I don't understand why the fist interface always drop packets.

Here is the kvp-ifup script :

#!/bin/sh
INTERFACE_LAN=bond0
INTERFACE_BRIDGE=br0
# set address to "DHCP" for DHCP
BRIDGE_ADDRESS=192.168.1.1
BRIDGE_BROADCAST=192.168.1.255
BRIDGE_NETMASK=255.255.255.0

echo "Configure $1 interface..."
/usr/sbin/tunctl -u 'root' -t $1
echo "Activating link for $1..."
/sbin/ip link set $1 up
sleep 1
echo "Configure $INTERFACE_BRIDGE bridge..."
/usr/sbin/brctl addbr $INTERFACE_BRIDGE
/usr/sbin/brctl addif $INTERFACE_BRIDGE $1
/usr/sbin/brctl addif $INTERFACE_BRIDGE $INTERFACE_LAN
echo "Activating promiscuous mode on $1 and $INTERFACE_LAN..."
/sbin/ifconfig $1 0.0.0.0 promisc up
/sbin/ifconfig $INTERFACE_LAN 0.0.0.0 promisc up
sleep 2
echo "IP address on $INTERFACE_BRIDGE..."

if [ "DHCP" == "$BRIDGE_ADDRESS" ]; then
        /sbin/pump -i br0
else
        # static
        /sbin/ifconfig $INTERFACE_BRIDGE $BRIDGE_ADDRESS broadcast $BRIDGE_BROADCAST netmask $BRIDGE_NETMASK
fi

echo "Done."

I tried both with a bond (different modes) and no bonding.

The physical NICS are Ethernet controller: Intel Corporation 82574L Gigabit Network Connection (e1000e linux driver)

the VM are using the e1000 driver (better results than virtio)

iptables is perfectly empty.

As for offloading, I cannot check but it should be the defaults :

• rx-checksumming: on
• tx-checksumming: on
• scatter-gather: on
• tcp segmentation offload: on
• udp fragmentation offload: off
• generic segmentation offload: on
• large receive offload: off

The setup is as follow : 3 Linux servers with Intel CX4 10 GigE controllers and an X-Serve with a Myricom 10 GigE CX4 controller are connected to a ProCurve 4208vl switch, with a myriad of other machines connected through good ol' 1000 base-T.

The interfaces are actually set up as 10 Gig, according to both the switch monitoring interface and the servers (ethtool, etc). However a single connection between two 10 GigE equipped machines through the switch is limited to exactly 1Gb.

If I connect two of the 10 GigE machines directly with a CX4 cable, netperf reports the link bandwidth as 9000 Mb/s. NFS achieves about 550 MB/s transfers. But when I'm using the switch, the connection tops at 950 Mb/s through netperf and 110 MB/s with NFS.

When I open several connections from 3 of the machines to the 4th, I get 350 MB/s of NFS transfer speed. So each individual 10 GigE ports actually can reach much more than 1 Gb, but individual connections are strictly limited to 1 Gb.

Conclusion : the 10 GigE connection through the switch behaves exactly like a trunk of 10 1 Gb connections. That doesn't make any sense to me, unless HP planned these ports only for cascading switches or strictly for many-clients-to-single-server connection. Unfortunately this is NOT the envisioned setup, we need big throughput from machine to machine.

Is this a not-so-known (or carefully hidden...) limitation of this type of switch? Should I suggest seppuku to the HP representative? Does anyone have any idea on how to enable a proper behaviour ? I upgraded for an hefty price from bonded 1Gb links to 10 GigE and see exactly ZERO gain! That's absolutely unacceptable.

I have trouble getting the max throughput out of my setup. The hardware is as follow :

• dual Quad-Core AMD Opteron(tm) Processor 2376
• 16 GB DDR2 ECC RAM
• dual Adaptec 52245 RAID controllers
• 48 1 TB SATA drives set up as 2 RAID-6 arrays (256KB stripe) + spares.

Software :

• Plain vanilla 2.6.32.25 kernel, compiled for AMD-64, optimized for NUMA; Debian Lenny userland.
• benchmarks run : disktest, bonnie++, dd, etc. All give the same results. No discrepancy here.
• io scheduler used : noop. Yeah, no trick here.

Up until now I basically assumed that striping (RAID 0) several physical devices should augment performance roughly linearly. However this is not the case here :

• each RAID array achieves about 780 MB/s write, sustained, and 1 GB/s read, sustained.
• writing to both RAID arrays simultaneously with two different processes gives 750 + 750 MB/s, and reading from both gives 1 + 1 GB/s.
• however when I stripe both arrays together, using either mdadm or lvm, the performance is about 850 MB/s writing and 1.4 GB/s reading. at least 30% less than expected!
• running two parallel writer or reader processes against the striped arrays doesn't enhance the figures, in fact it degrades performance even further.

So what's happening here? Basically I ruled out bus or memory contention, because when I run dd on both drives simultaneously, aggregate write speed actually reach 1.5 GB/s and reading speed tops 2 GB/s.

So it's not the PCIe bus. I suppose it's not the RAM. It's not the filesystem, because I get exactly the same numbers benchmarking against the raw device or using XFS. And I also get exactly the same performance using either LVM striping and md striping.

What's wrong? What's preventing a process from going up to the max possible throughput? Is Linux striping defective? What other tests could I run?

Here's my problem : I need to archive to tar files a lot ( up to 60 TB) of big files (usually 30 to 40 GB each). I would like to make checksums ( md5, sha1, whatever) of these files before archiving; however not reading every file twice (once for checksumming, twice for tar'ing) is more or less a necessity to achieve a very high archiving performance (LTO-4 wants 120 MB/s sustained, and the backup window is limited).

So I'd need some way to read a file, feeding a checksumming tool on one side, and building a tar to tape on the other side, something along :

tar cf - files | tee tarfile.tar | md5sum -

Except that I don't want the checksum of the whole archive (this sample shell code does just this) but a checksum for each individual file in the archive.

I've studied GNU tar, Pax, Star options. I've looked at the source from Archive::Tar. I see no obvious way to achieve this. It looks like I'll have to hand-build something in C or similar to achieve what I need. Perl/Python/etc simply won't cut it performance-wise, and the various tar programs miss the necessary "plugin architecture". Does anyone know of any existing solution to this before I start code-churning ?

I have a serious problem with a SAN storage array connected to a Linux box through Fibre Channel. Here is the configuration :

• Debian with plain vanilla linux 2.6.27.25
• Fibre controller QLogic 4Gb dual port ( ISP2432 based)

Basically the problem is: how to get this #?@!! FC controller/driver to recognize properly configuration changes (new or removed LUNs) of the storage array?

1. when I create a new LUN on my array (typically a snapshot of some existing LUN) and map it to my HBA, I can't get it recognized properly : rescan-scsi-bus -l -w -r actually detects something ( a generic /dev/sgXX device) however no block device is created ( /dev/sdXX).

2. same thing when issuing a LIP and rescan manually:

   echo 1 > /sys/class/fc_host/host6/issue_lip

   echo "- - -" > /sys/class/scsi_host/host6/scan

3. if I remove an existing LUN, neither issuing LIPs and rescans or rescan-scsi-bus have any effect. The previous devices remain there and of course don't work ( "file -s /dev/sdXX -> I/O error").

4. reloading the qla2xxx driver works. However it's completely unworkable in a production environment.

Apparently this is a very common problem with QLogic. Some sort of solution exists that works only when using the QLogic issued driver available only for RedHat and Suse enterprise distros : see this explanation.

Additional info :

Here is the scsi devices before LIP and rescan:

# sg_map -x
/dev/sg0  0 0 0 0  0  /dev/sda
/dev/sg1  0 0 1 0  5  /dev/scd0
/dev/sg2  1 0 0 0  0  /dev/sdb
/dev/sg3  6 0 0 0  0  /dev/sdc
/dev/sg4  6 0 0 1  0  /dev/sdd
/dev/sg5  6 0 0 2  3

After a LIP and rescan, I have a new sg device, but no matching drive. If I reload the driver, a drive appear:

# sg_map -x
/dev/sg0  0 0 0 0  0  /dev/sda
/dev/sg1  0 0 1 0  5  /dev/scd0
/dev/sg2  1 0 0 0  0  /dev/sdb
/dev/sg3  6 0 0 0  0  /dev/sdc
/dev/sg4  6 0 0 1  0  /dev/sdd
/dev/sg5  6 0 0 2  3
/dev/sg6  6 0 0 3  3

~# sg_map -x
/dev/sg0  0 0 0 0  0  /dev/sda
/dev/sg1  0 0 1 0  5  /dev/scd0
/dev/sg2  1 0 0 0  0  /dev/sdb
/dev/sg3  8 0 0 0  0  /dev/sdc
/dev/sg4  8 0 0 1  0  /dev/sdd
/dev/sg5  8 0 0 2  0  /dev/sde
/dev/sg6  8 0 0 3  3

Edit: OK, obviously this is a tough nut to crack. I'll ask the LKML and report here.

I'm considering the idea of using Bacula for some backups, however to save tapes and limit manual operations, I'd like to make a two-stage backup :

1. backup machines to disk (easy enough in bacula)
2. When done, consolidate disk backups on tape. That's the tricky point...

Does anyone know how to "migrate" a given backup from a disk support to tape from within Bacula?

Apparently LTO-4 drives can both read and write LTO-3 tapes. Are these tapes fully compatible with LTO-3 drives? I'm interested in real life experience more than theoretical vendor-touted compatibility :)