Home / server / Questions / 345807
Accepted
David542
Asked: 2012-01-02 20:26:56 +0800 CST

Keeping a server live while pushing changes to production

  • 772

I am using EC2 as the live production server, and I am using git for source control. Currently, when I push changes to the site, I have a bash script that runs & pulls the most recent changes from git, and restarts the server. This causes the site to be down for about ten seconds while this is happening.

I would like to be able to push changes and keep the site live. (My initial thought would be to fire up an additional EC2 instance with the existing code while these changes are being pushed to the server that is being updated.) What are some ways to accomplish keeping the site live while pushing changes?

backup linux apache-2.2 amazon-ec2

2 Answers

  1. Best Answer

    I like to do a transition from a server running old code to a server running new code using a load balancer.

    If you have enough live servers that you can handle twice the current traffic, you can:

    1. Take half of your servers out of service
    2. Upgrade those servers.
    3. Switch the servers running old code out of service an the servers running new code into service (atomically).
    4. Upgrade the second half.
    5. Bring all servers into service.

    If you don't have enough running servers to spare, you can follow a procedure like this:

    1. Bring up new servers with the new code.
    2. Add the new servers to the load balancer and bring them into service while taking the old servers out of service.
    3. Throw away the old servers or leave them on standby for a fast rollback if necessary.

    This works best for stateless instances like web servers. A database requires a different approach, but often your web servers are using a single database that does not need to change that often, and your database changes can be made to work with both the old and new code for the transition.

    We have found one issue when using Amazon's Elastic Load Balancer (ELB) to do the above type of live switchover to new servers. When you take the old instances out of service, the ELB rudely drops any active connnections. Depending on your application you may be able to live with it, especially if your previous behavior was a complete outage with a server restart.

    Here's the thread at Amazon where we reported this bug. You can follow it for updates:

    Existing connections dropped rudely when instance taken out of ELB
    https://forums.aws.amazon.com/thread.jspa?threadID=61278

    Jump in with your +1 if this is an issue for you as Amazon has asked for further feedback there.

    • 4

  2. You probably want the "graceful restart" rather than a normal "restart". In Apache, you can probably do /etc/init.d/apache2 graceful (Ubuntu) or /etc/init.d/apache graceful (RHEL/CentOS) to cause new connections to use the newly updated instance of the application. Another benefit is that existing requests won't be interrupted while the restart is done.

    From your description, this should solve your problem. See the "graceful" section of man apachectl for more details about what "graceful" means.

    • 1