Bradford's questions

I am running some command in cron that does an rsync over two data centers. The connection rate between the data centers is around 1Mbps. I noticed that rsync keeps failing around 2.7GB to 2.9GB of transferred data, which could equal around an hour or more on said connection. I know in the past that our firewall is configured to close idle connections that have opened for more than an hour.

But would an rsync transfer really be idle? What can I do to fix this?

Error:

rsync: connection unexpectedly closed (2987596424 bytes received so far) [receiver]

rsync error: error in rsync protocol data stream (code 12) at io.c(605) [receiver=3.0.9]

rsync: connection unexpectedly closed (523 bytes received so far) [generator]

rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9]

I already have this in my ~/.ssh/config (but maybe these values need to be increased?):

Host *
  ServerAliveInterval 60
  ServerAliveCountMax 15

Internally, we have several VM hosts. We are constantly provisioning new guests on these hosts. What tool to you guys use to keep track of what hosts hosts you have out there, the host system stats, what guests you have on those hosts, notes about what applications are running on the guests, and it'd be nice to see what virtual IPs are listed on each guest, and be able to track it back to the host?

We use VMWare

I'm having to deploy my application to three servers and soon to be four. This is becoming a pain and I'm looking to automate it. I was going to script this, but this is such a common problem and there are so many tools out there (Puppet, Chef, Pallet, MCollective) that I thought someone may have an existing strategy in place that they are willing to share.

We have 3 web servers (and will soon have 4). Here's the steps I currently use to deploy:

log on to production web server 1

> service httpd stop
> service tomcat5 stop

log on to QA server

> scp ROOT.war [email protected]:/usr/share/tomcat5/webapps/

Sometimes it is more complex, though, and I may need to copy any of the following: /etc/httpd/conf.d/mod_jk.conf /etc/tomcat5/context.xml

But I can't just copy over context.xml from QA, since the connection strings are different in QA and production.

restart everything (after ROOT.war has transferred)

> service tomcat5 start
> service httpd start

Usually, I deploy to server1, test, and then deploy to server2 and server3. I like to wait until server 2 is completed ready before I start deploying to server 3 -- can Puppet do this (check for the HTTP status of /)?

So, can I have Puppet deploy server1 and then have it do the deploy and wait to server2, server3, and soon to be server4?

Phased rollouts and perhaps easy rollback would be features I'm looking for.

Is it possible to setup some DNS server where it will default to an IP address if the the host name it tries to resolve fails to resolve? I'm having a problem where a host name is intermittently failing to resolve.

I have 2-3 node clusters for our load balancers and various web applications. I'm having to make the change in QA first, then in staging (on 2-3 servers), then in production (on 2-3) servers. Is a configuration management tool like chef or puppet appropriate here? Or is it overkill? If it's overkill are there any tips to making this easier.