mr.simonski's questions

We run haproxy as load balancer and to provide backup functionality for our API service.

Our API is a Java Application using the Jersey framework running in Apache Tomcat. Some time ago we ran into a bad performance problem on the side of our API and we just couldn't figure out what was going on. After digging deep we found that the option httpchk was causing the problem:

    option httpchk GET /resource/healthcheck
    http-check expect rstatus (2|3)[0-9][0-9]
    server server1_8585 x.x.x.x:8585 check inter 1s fastinter 2s fall 2 rise 1 pool-purge-delay 30s

The packets send by haproxy did not include a Host header and because of that Tomcat tries to resolve the host which leads to terrible performance issues. Because of that we tuned the backend configuration to this:

The backend configuration looks like:

backend api_example_com
    balance roundrobin
    option httpchk GET /resource/healthcheck HTTP/1.1\r\nHost:\ server1.example.com
    http-check expect rstatus (2|3)[0-9][0-9]

    server server1_8585 x.x.x.x:8585 check inter 10s fastinter 5s fall 2 rise 1 pool-purge-delay 30s
    server server2_8585 y.y.y.y:8585 check inter 10s fastinter 5s fall 2 rise 1 pool-purge-delay 30s backup
    server server3_8585 z.z.z.z:8585 check inter 10s fastinter 5s fall 2 rise 1 pool-purge-delay 30s backup

But now we have the problem that the backup APIs get a wrong Host name as well if health-checked.

Any idea how to set up a separate health check per server?

I have a docker swarm running our business stack defined in a docker-compose.yml on two servers (nodes). The docker-compose has defined cAdvisor starting on each of the two nodes like that:

  cadvisor:
    image: gcr.io/google-containers/cadvisor:latest
    command: "--logtostderr --housekeeping_interval=30s"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /:/rootfs:ro
      - /var/run:/var/run
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk:/dev/disk/:ro
    ports:
      - "9338:8080"
    deploy:
      mode: global
      resources:
        limits:
          memory: 128M
        reservations:
          memory: 64M

On a third server I run a docker separately from the docker swarm on node 1 and 2 and this server is used to run Prometheus and Grafana. Prometheus is configured to scrape only the node1:9338 resource to get the cAdvisor information.

I occasionally get the problem that when scraping node1:9338 not all containers running on both nodes 1 and 2 are shown in the cAdvisor statistics.

I was assuming that cAdvisor is synching its information in the swarm so that I'm able to configure Prometheus to only use node1:9338 as entrypoint into the docker swarm and scraping the information.

Or do I have to also put node2:9338 into my Prometheus configuration to always get all information of all nodes? If yes, how should this scale then because I would need to add each new node to the Prometheus config.

Running Prometheus together with the business stack in one swarm is no option.

edit: I experienced today when opening the cAdvisor metrics URL http://node1:9338/metrics as well as http://node2:9338/metrics a strange behaviour as I see the same information of all containers running on node1 on both URLs. The information of the containers running on node2 are missing when requesting http://node2:9338/metrics.

Could it be that the docker-internal load balancing is routing the request from http://node2:9338/metrics to the node1:9338 cAdvisor so the metrics of node1 are shown despite node2 is requested?

We have a Docker Swarm deployment running on two nodes (node1 and node2) for our business application.

The application needs a volume to store persistent data. As it is unclear where a container is deployed (node1 or node2) as well as maybe two containers of our application should run on both nodes we needed a solution to provide a shared volume for all nodes.

For sharing a volume we set up a NFS server on a third node3 with the following /etc/exports file:

/srv          *(rw,sync,anonuid=1000,anongid=1000,all_squash,subtree_check,crossmnt,fsid=root)

(I use anonuid/gid to explicitly set for each files in the export the user information for a known user in the node3 system. all_squash is used to make sure all file rights from all accessing users are rewritten to this local user)

In our docker-compose.yml we use the following setup to include the volume:

volumes:
  nfs-data:
    driver: local
    driver_opts:
      type: nfs
      o: nfsvers=4,addr=node3.example.com,rw,nolock,soft

We now ran into a problem where the the container just didn't want to start up, the error message was:

failed to copy file info for /var/lib/docker/volumes/MY_CONTAINER_nfs-data/_data: failed to chown /var/lib/docker/volumes/MY_CONTAINER_nfs-data/_data: lchown /var/lib/docker/volumes/MY_CONTAINER_nfs-data/_data: operation not permitted

After some digging around I found out that the problem relies in the initial empty folder on the exported NFS directory on the node3 server. As soon as I put in an empty file the startup of the container in node1 and node2 is working totally fine.

Does anyone has an explanation for that?

I have set up a haproxy configuration with a backend with two servers looking like that:

...
default
        option log-health-checks
...
mailers mta
        mailer smtp1 127.0.0.1:25
...
backend s_api
        balance roundrobin
        option httpchk GET /sites?site=q&limit=1

        http-check expect rstatus (2|3)[0-9][0-9]

        server s1_a 1.2.3.4:3600 check inter 5s fall 4 rise 1
        server s2_b 1.2.3.5:3600 check backup

        timeout queue 60s
        timeout server 60s
        timeout connect 60s

        email-alert mailers mta
        email-alert level notice
        email-alert from [email protected]
        email-alert to [email protected]

I now get noticed via email when a server gets down and up again. What I would like to archive is to start a local bash script when a server is going up and another one when the server is going down.

How could that be done?

I have a problem with my Debian Servers. We run 4 different server which all have Intel CPUs and 128GB of RAM. Two of them run Wheezy, two of them run Jessie. We run a Java software on those systems which is heavily using memory and could eat up all memory.

For those cases I installed a swap partition on every server which is held on a RAID 1 running on 2 SSDs.

Problem with the Jessie systems: when the system nearly runs out of memory it starts swapping. This is tuned by the vm.swappiness = 10 parameter and looks ok to me. But the swapping itself is done so heavily, that the system totally hangs/freezes. There is so much disk io done that the system is not responding anymore.

I did some tests on all systems an artificially filled up the RAM to 120% by using:

stress --vm-bytes $(awk '/MemFree/{printf "%d\n", $2 * 1.2;}' < /proc/meminfo)k --vm-keep -m 1

The system start swapping and freezes while the swapping of the 20% is running. After ~20s the system is back and usable again but during the freeze nothing works anymore.

Of course this behaviour is not acceptable for a productive system. What I would expect is that swapping has a high priority but should never use more than 90% of all system resource so that the system still can be handled somehow.

Tuning the swappiness to different values didn't help..

We're using the following kernels:

Wheezy: Linux A 3.2.0-4-amd64 #1 SMP Debian 3.2.68-1+deb7u1 x86_64 GNU/Linux

Jessie: Linux B 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt20-1+deb8u4 (2016-02-29) x86_64 GNU/Linux

Did anyone run into the same problem and found a solution?

Edit: Thank you all for the comments and explanations. Of course I don't want to use swap as spare memory. The 120% usage was just a test. In production, the systems uses maybe 100,0001% of the memory and already stops being responsive. In the production mode with our software running there is also a high frequency of changing data so that the system might be busy by just swapping a very small amount of data back and forth the whole time.

we have a 4 root (dedicated) servers (no virtual servers/vservers, full hardware servers with asus mainboards and 6 real Intel cores) running Debian hosted by another company and standing in their location. Each server has its own IPv4 and multiple services are running on those machines.

My aim is to connect all 4 server to a private network with private IP addresses (something 192.168.x.x) on a virtual network interface. Later I would like to bind special services like mysql database only to the virtual interface and block most traffic on the public interface (eth0) with iptables. Also there should be a directory somewhere in the filesystem of each server which holds synched content for all 4 servers.

What kind of software solution should I use to configure this setup? Is this done with OpenVPN? Or some other solution?

We don't need hard encryption on the wire with big keys as the speed matters most.

Thanks for any hint, Simon

I have a Apache Webserver with mod_proxy and a Tomcat server with my Grails web app runnning. The Apache is redirecting with proxy the request to example.com:80 to my Tomcat running on example.com:8008.

I need to redirect requests the following way:

http://subdomain1.example.com:80/some-nice-seo-url

from apache should get per proxy to

http://example.com:8008/some-nice-seo-url-subdomain1

How can I archieve this with the following apache2 config:

NameVirtualHost *:80

<VirtualHost *:80>
        ServerName example.com
        ServerAlias www.example.com
        <Proxy *>
                Order deny,allow
                Allow from all
        </Proxy>

        # compress text, html, javascript, css, xml:
        AddOutputFilterByType DEFLATE text/plain
        AddOutputFilterByType DEFLATE text/html
        AddOutputFilterByType DEFLATE text/xml
        AddOutputFilterByType DEFLATE text/css
        AddOutputFilterByType DEFLATE application/xml
        AddOutputFilterByType DEFLATE application/xhtml+xml
        AddOutputFilterByType DEFLATE application/rss+xml
        AddOutputFilterByType DEFLATE application/javascript
        AddOutputFilterByType DEFLATE application/x-javascript

        ExpiresActive On
        ExpiresByType text/html "access plus 7 days"
        ExpiresByType image/gif "access plus 1 months"
        ExpiresByType image/jpeg "access plus 1 months"
        ExpiresByType image/png "access plus 1 months"
        ExpiresByType text/css "access plus 14 days"
        ExpiresByType text/javascript "access plus 14 days"
        ExpiresByType application/x-javascript "access plus 14 days"
        ExpiresByType image/ico "access plus 1 months"

        RewriteEngine on
        # www to non-www using search-engine friendly 301
        RewriteCond %{HTTP_HOST} ^www.example.com [NC]
        RewriteRule ^(.*)$ http://example.com$1 [L,R=301]

        # IP canonicalization
        RewriteCond %{HTTP_HOST} ^1\.2\.3\.4
        RewriteRule ^(.*)$ http://example.com/$1 [L,R=301]


        <Directory />
                Options FollowSymLinks
                 AllowOverride None
        </Directory>
        <Directory /opt/example/web-app>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        # redirect everything else to TOMCAT
        ProxyPass               / http://1.2.3.4:8008/
        ProxyPassReverse        / http://1.2.3.4:8008/
        ProxyPassReverseCookieDomain localhost example.com
        ProxyPreserveHost On
</VirtualHost>

Thank you for any help!

We have a debian linux server which has a single 120GB SSD and 2x2TB HDDs as RAID 1. We now have to move to another server which has 2x240GB SSD and a single 2TB HDD. The aim is to move the current 120GB system drive SSD to a 240GB RAID1 SSD and move the data from the current 2TB RAID1 HDD to the single 2TB HDD.

Moving the 2TB wont be a problem, so I'm focusing my question to the system drive SSD. Our current setup is somehow complicated. The 120GB SSD has the following partitions:

fdisk -l /dev/sda

Disk /dev/sda: 120.0 GB, 120034123776 bytes
255 heads, 63 sectors/track, 14593 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          66      523264   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              66       14594   116696064   83  Linux

and fstab tells us:

cat /etc/fstab
# /etc/fstab: static file system information.
#
# <file system>                         <mount point>   <type>  <options>       <dump>  <pass>
        proc                            /proc           proc    defaults          0       0
        /dev/mapper/vgdebian-root       /               ext4    errors=remount-ro 0       1
        /dev/mapper/vgdebian-swap       none            swap    sw                0       0
        /dev/sda1                       /boot           ext3    defaults          0       2

So the boot stuff with kernel and a busybox linux with dropbear is all on /dev/sda1. Then dropbear helps to decrypt the /dev/sda2 partition, which is enrypted by cryptsetup and managed by LVM.

I'm not sure which is the best way to move all of /dev/sda to a newly created SDD-RAID1? Should I first make a dd-copy to one of the new disks, enlarge the /dev/sda2 partition (each of the new SSDs is now 240GB instead of 120GB) and make dropbear aware of the new 240GB SSD? Should I then start copying all of the first 240GB SDD to the second one and init a mdadm create array command?

Or should I create a clean /dev/md0 array on the new 240GB SSDs and then copying the whole old drive to this maybe /dev/md0 named device?

How would dropbear/busybox react on the RAID? It might be that the new /dev/sda1 has to be copied to /dev/sdb1 so that dropbear/busybox can be bootet from both new SSDs. The RAID1 would exists first when booting the decrypted LVM debian - am I right?

Maybe someone can give me some hints if it is possible to move such a enrypted system anyway. Thanks for any help.

Edit: I transfered all 120GB of our old SSD to one of the new 240GB SSD of the new server via dd command (tut here: https://library.linode.com/migration/ssh-copy). Then I changed some config to dropbear and reassembled initramfs and rebootet - the system works as usual on new server.

Now I needed to resize the image of the old SSD so I enlarged /dev/sda2 to maximum, then I enlarged physical volume, logical volume and later the file system. I rebooted and everthing works fine (tut here: http://ubuntuforums.org/showthread.php?p=4530641).

Last thing: moving the whole stuff from single SSD to RAID1. Any hints someone?

Edit2: Currently I'm trying to get the RAID1 thing running. After the old system works on one of the 240GB SSDs there are two tutorials I found for migrating from a non-RAID system to RAID1: pug.org/mediawiki/index.php/Vorhandenes_MD-RAID1_im_laufenden_Betrieb_verschl%C3%BCsseln and howtoforge.com/software-raid1-grub-boot-debian-etch-p2. Both work from a running system. I use the first tut for the LUKs stuff and the second one for the rest - hope it will work out.

I have a dedicated Webserver running Debian 6 and some Apache, Tomcat, Asterisk and Mail-stuff. Now we needed to add VPN support for a special program. We installed OpenVPN and registered with a VPN provider. The connection works well and we have a virtual tun0 interface for tunneling.

To archive the goal for only tunneling a single program through VPN, we start the program with

sudo -u username -g groupname command

and added a iptables rule to mark all traffic coming from groupname

iptables -t mangle -A OUTPUT -m owner --gid-owner groupname -j MARK --set-mark 42

Afterwards we tell iptables to to some SNAT and tell ip route to use special routing table for marked traffic packets.

Problem: if the VPN failes, there is a chance that the special to-be-tunneled program communicates over the normal eth0 interface.

Desired solution: All marked traffic should not be allowed to go directly through eth0, it has to go through tun0 first.

I tried the following commands which didn't work:

iptables -A OUTPUT -m owner --gid-owner groupname ! -o tun0 -j REJECT

iptables -A OUTPUT -m owner --gid-owner groupname -o eth0 -j REJECT

It might be the problem, that the above iptable-rules didn't work due to the fact, that the packets are first marked, then put into tun0 and then transmitted by eth0 while they are still marked.. I don't know how to de-mark them after in tun0 or to tell iptables, that all marked packet may pass eth0, if they where in tun0 before or if they going to the gateway of my VPN provider.

Does someone has any idea to a solution?

Some config infos:

iptables -nL -v --line-numbers -t mangle

Chain OUTPUT (policy ACCEPT 11M packets, 9798M bytes) 
num   pkts  bytes target     prot opt in     out     source       destination 
1     591K   50M   MARK       all  --  *      *       0.0.0.0/0    0.0.0.0/0   owner GID match 1005 MARK set 0x2a 
2    82812 6938K CONNMARK   all  --  *      *         0.0.0.0/0    0.0.0.0/0   owner GID match 1005 CONNMARK save

iptables -nL -v --line-numbers -t nat

Chain POSTROUTING (policy ACCEPT 393 packets, 23908 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1       15  1052 SNAT       all  --  *      tun0    0.0.0.0/0            0.0.0.0/0           mark match 0x2a to:VPN_IP

ip rule add from all fwmark 42 lookup 42

ip route show table 42

default via VPN_IP dev tun0