Hank's questions

We're renting a number of hosts in a public datacenter. The datacenter does not offer private VLANs; all hosts receive one (or more) public IPv4/IPv6 addresses. The hosts come with very modern CPUs (Haswell quad-core, 3.4GHz) and have Gbit uplinks. The different areas (rooms? floors? buildings?) of the datacenter are interconnected with - from what I can tell - Gbit or 500Mbit links. Our hosts are running debian wheezy. Currently we're running just above 10 hosts, with the expectation of growth in the near future.

I am looking for a way to get all hosts to communicate with each other, securely and confidentially. Layer 3 is fine, layer 2 ok (but not necessary). Since I don't have access to VLANs, it'll have to be a VPN of some sort.

What is important to me:

1. high throughput, ideally close to wirespeed
2. decentralized, meshed architecture - this is to make sure that throughput is not slowed down by a central element (e.g. VPN concentrator)
3. CPU footprint is not excessive (given AESNI and GCM-cipher suites, I'm hoping this is not a ridiculous requirement)
4. operational ease of use; not too complicated to setup; network can grow without loosing established connections

We're currently using tinc. It ticks [2] and [4], but I only reach about 600Mbit/s (simplex) of a 960Mbit/s wirespeed, and I loose one core completely. Also, tinc 1.1 - currently in development - is not yet multithreaded, so I'm stuck with singlecore performance.

Traditional IPSec is out of the question, since it requires a central element, or a sh*tload of tunnels to be configured (to achieve [2]). IPsec with opportunistic encryption would be a solution, but I'm not sure it ever made it into stable production code.

I've stumbled across tcpcrypt today. Except for the missing authentication, it looks like what I want. Userspace implementation smells slow, but so are all the other VPNs as well. And they speak of a kernel implementation. I have not tried it yet, and am interested how it behaves re [1] and [3].

What other options are there? What are people doing, who are not on AWS?

Additional Info

I'm interested in GCM, hoping that it will reduce the CPU footprint. See Intel's paper on the topic. When talking to one of the tinc developers, he explained that even using AESNI for the encryption, the HMAC (e.g. SHA-1) is still very expensive at Gbit speed.

Final Update

IPsec in transport mode works perfectly and does exactly what I want. After much evaluation I have chosen Openswan over ipsec-tools, simply because it supports AES-GCM. On the Haswell CPUs I measure about 910-920Mbit/sec simplex throughput with about 8-9% CPU load of one kworkerd.

I'm trying to restore a full backup taken from one node of a three-node percona cluster (percona cluster 5.5, galera 2.1, wsrep sst method is rsync, innodb tables only).

Backup was taken like this:

rm -rf /tmp/backup/mysqldb
innobackupex --user=bkpuser --password=xxxx --galera-info --no-timestamp /tmp/backup/mysqldb/
innobackupex --apply-log --use-memory=2G /tmp/backup/mysqldb/

The restore procedure I'm attempting is like this:

1. shut down mysql on all three nodes
2. on first node
   1. delete contents of mysql data dir
   2. delete mysql redologs/binlogs, doublewrite file etc (they are located in a separate folder)
   3. copy back database tablespace files for my database instance
   4. copy back database tablespace files for mysql database
   5. redologs/binlogs, doublewrite file, etc
   6. start mysql with wsrep_urls = gcomm:// to initialize cluster
3. on second and third node
   1. delete mysql redologs/binlogs, doublewrite file etc
   2. remove galera.cache and grastate.dat
   3. start mysql (wsrep_urls = gcomm://firstnode:port,gcomm://secondnode:port,gcomm://thirdnode:port)
4. once the cluster is all synced, restart first node with full wsrep_urls configuration

(I know wsrep_urls is deprecated, but I haven't gotten galera to work with wsrep_cluster_address yet.)

My problem is that the second and third nodes report errors after SST. For each table, I see this error:

130225 15:44:43 [ERROR] Cannot find or open table myTestDb/settings from the internal data dictionary of InnoDB though the .frm file for the table exists. Maybe you have deleted and recreated InnoDB data files but have forgotten to delete the corresponding .frm files of InnoDB tables, or you have moved .frm files to another database? or, the table contains indexes that this version of the engine doesn't support. See http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html how you can resolve the problem.

Mysql show tables shows the tables as existing, but when attempting to select from them, it reports error Table 'myTestDb.settings' does not exist ...

I have tried to delete the local tablespace files before starting up mysql and requesting sst, same result.

How should I perform the restore? Should I copy back the backup files to all nodes?

I am planning a large deployment of a Glassfish-powered web application:

• A number of nodes running multiple Glassfish 3.1.1 instances are running the web application.
• The web application consists of a GoogleWebToolkit frontend and a REST gateway.
• The instances are combined behind an Apache 2.2 reverse proxy / loadbalancer.
• All client communication (mobile app, browser and other web apps) is over HTTPS, SSL is terminated at the Apache loadbalancer.

What is the benefit of running communication between Apache and Glassfish over mod-jk / AJP13 compared to HTTP in terms of performance and availability?

I had a perfectly fine running installation process for virtual machines based on an instalinux CD, creating Debian 5 lenny VMs within minutes. The preseed file ensures installation of puppet, which takes care of "personalization" of the VMs later, so really I need only one installimage (which is why I'm not interested in FAI or similar heavyweights).

Since Debian 6 has come out, the preseed doesn't work anymore - it seems to be installing squeeze, but then the grub-installation fails, and I don't want squeeze anyway (yet).

This is the section of the preseed file that specifies the source:

####################################################################
# Installation Sources
####################################################################

# Where are we pulling bits from?
d-i     mirror/http/hostname    string ftp.de.debian.org
d-i     mirror/http/directory   string /debian/
d-i     mirror/suite            string lenny

# Post install APT setup
d-i     apt-setup/uri_type      select d-i
d-i     apt-setup/hostname      string ftp.de.debian.org
d-i     apt-setup/directory     string /debian/
d-i     apt-setup/another       boolean false
d-i     apt-setup/security-updates      boolean false
d-i     finish-install/reboot_in_progress note
d-i     prebaseconfig/reboot_in_progress        note

d-i     apt-setup/non-free      boolean true
d-i     apt-setup/contrib      boolean true

Any idea how to get it to really install lenny (oldstable)? Thanks!

The frontend of my web application is formed by currently two Apache reverse proxies, using mod_proxy_balancer to distribute traffic over a number of backend application servers. Both frontend reverse proxies, running on separate hosts, are accessible from the internet. DNS round robin distributes traffic over both. In the future, the number of reverse proxies is likely to grow, since the webapplication is very bandwidth-heavy.

My question is: how do I keep the state of both reverse balancers / proxies in sync?

For example, for maintenance purposes, I might want to reduce the load on one of the backend appservers. Currently I can do that by accessing the Balancer-Manager web form on each proxy, and change the distribution rules. But I have to do that on each proxy manually and make sure I enter the same stuff.

Is it possible to "link" multiple instances of mod_proxy_balancer? Or is there a tool out there that connects to a number of instances, and updates all with the same information?

Update: The tool should retrieve the runtime status and make runtime changes, just like the existing Balancer-Manager, only for a number of proxies - not just for one. Modification of configuration files is not what I'm interested in (as there are plenty tools for that).