skmbr's questions

For the last few years we have been running Varnish as a cache and load balancer in front of several apache servers serving several thousand websites.

We also use monit to ensure that if varnish ever dies it gets restarted. The varnish section in monitrc looks like this:

  # Check varnish on port 80
  check process varnish with pidfile /var/run/varnishd.pid
  start program = "/etc/init.d/varnish start"
  stop program = "/etc/init.d/varnish stop"
  if failed host 127.0.0.1 port 80 protocol http
    and request "/monit-check-url"
    then restart

This has worked fine at least 3 years. We get occasional failures of the port 80 check, but monit restarts varnish accordingly and it's generally unnoticeable to users.

However, over the last few weeks we are seeing flurries of these failures, usually over a period of a couple of hours, and users are noticing connection failures. Today has been particularly bad.

There are no clues in syslog (it's a debian box btw) as suggested by the "Varnish crashing" section at: https://www.varnish-cache.org/docs/3.0/tutorial/troubleshooting.html and all we see in there is monit failing it's check on port 80 then stopping and starting varnish.

Additionally we are not seeing any spike in bandwidth or number of hits to the backend webservers that would suggest it's failing under higher than normal load.

We were running Varnish 3.0.3 which I upgraded to 3.0.7 but the problem has continued. No other changes have been made to this box that coincide with the problems starting, and the varnish configuration hasn't been changed in quite a long time.

Has anyone had any similar experiences with varnish or have any suggestions on troubleshooting this further? Could it be some sort of attack?

Any help or advice greatly appreciated!

I have the following setup:

M1 = Master 1
S1 = Slave 1
S2 = Slave 2
S3 = Slave 3

Where all slaves are replicating directly from M1. I would like to move S2 and S3 to be slaves of S1.

I'm not sure about how to do it without any downtime and with replication picking up where it left off. How to I ensure that the slaves continue from the correct point in the logs from their new master without missing any writes or trying to repeat writes that they had already done before the change.

One of the slaves is at an offsite location on a relatively slow connection, and it's quite a large DB, so rebuilding the slaves from a dump or copying data files would take too long, and seems to me as though it should be unnecessary!

Any help or advice greatly appreciated!

Simon