Home / server / Questions / 478031
In Process
TheCloudlessSky
Asked: 2013-02-13 06:59:21 +0800 CST

Idea for taking down half of the servers vs rolling restart?

  • 772

We're setting up HAProxy to balance our application (ASP.NET MVC 3 running on IIS). We want our process to be without scheduled maintenance for deployment. I'm trying to figure out the correct way to do a "see-saw" approach to only serve one version of the application to new requests at a single time. Here's what I've come up with so far:

  1. Remove the first half of the servers from HAProxy by reloading the config:

    $ sed -i 's/web01.*/& disabled/' /etc/haproxy/haproxy.cfg
$ sed -i 's/web02.*/& disabled/' /etc/haproxy/haproxy.cfg
$ /etc/init.d/haproxy reload

  2. Update the application on each down instance and poke them with curl to warm them.

  3. Bring back the first half and remove the second half:

    $ sed -i 's/\(web01.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
$ sed -i 's/\(web02.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
$ sed -i 's/web03.*/& disabled/' /etc/haproxy/haproxy.cfg
$ sed -i 's/web04.*/& disabled/' /etc/haproxy/haproxy.cfg
$ /etc/init.d/haproxy reload

  4. Repeat step 2 for the second half.

  5. Bring back the second half:

    $ sed -i 's/\(web03.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
$ sed -i 's/\(web04.*\) disabled$/\1/' /etc/haproxy/haproxy.cfg
$ /etc/init.d/haproxy reload

With the approach described above, the existing requests would gracefully continue using V1 of the app and then V2 for any subsequent requests. I'm OK with this because it'd significantly reduce the chance of serving the two different versions.

However, using sed to replace the config seems like a hack. Here are my other alternatives I've thought of using:

  1. Use the UNIX socket to do this same "see-saw" approach. However, because the UNIX socket can only work with one server at a time and the deployment could take a few seconds per-server, it could lead to HTML at V2 but JavaScript for V1. If anything, it's much easier to write backwards compatible code than forwards compatible. So using this approach seems like it might not work.

  2. Use rolling deployments across the servers with either UNIX sockets or the health check. During my research, I've come across this approach as being most popular. However, this leads to the same problems as the UNIX sockets - having to write forward compatible code.

Maybe I'm completely over complicating this or missing something obvious...

load-balancing

1 Answers

  1. So, you're worried about some assets not lining up version-wise with each other. This isn't really a load-balancer problem but a code problem. A common practice is to 'version' the assets using a parameter. E.g., my.html might contain:

    <link rel="stylesheet" href="http://foo.com/css/blah.css?v123">

    You should have a global setting somewhere in your code which you increment on release. In any case, you need to force a reload of the assets client-side as they'll be cached there.

    • 1