Using Upstart to manage Unicorn w/ rbenv + bundler binstubs w/ ruby-local-exec shebang

I picked little a different solution from SpamapS's.. I'm also running an app with preload_app = true, managed by Upstart.

When I was looking to solve this problem myself, I'd been using Upstart's "exec" to start my app ("exec bundle exec unicorn_rails blah blah"). Then I found your question, and it made me realize that instead of using Upstart's "exec" to specify my executable, I could use a script stanza, which would be run in its own process, the process that Upstart would watch.

So, my Upstart config file includes this:

respawn

script
  while true; do
    if [ ! -f /var/www/my_app/shared/pids/unicorn.pid ]; then
      # Run the unicorn master process (this won't return until it exits).
      bundle exec unicorn_rails -E production -c /etc/unicorn/my_app.rb >>/var/www/my_app/shared/log/unicorn.log
    else
      # Someone restarted the master; wait for the new master to exit.
      PID=`cat /var/www/my_app/shared/pids/unicorn.pid`
      while [ -d /proc/$PID ]; do
        sleep 2
      done
      # If we get here, the master has exited, either because someone restarted
      # it again (in which case there's already a new master running), or
      # it died for real (in which case we'll need to start a new process).
      # The sleep above is a tradeoff between polling load and mimizing the
      # restart delay when the master dies for real (which should hopefully be
      # rare).
    fi
  done
end script

The before_fork in my Unicorn config file is just as suggested in the example from the unicorn site, http://unicorn.bogomips.org/examples/unicorn.conf.rb:

before_fork do |server, worker|
  ActiveRecord::Base.connection.disconnect! if defined?(ActiveRecord::Base)

  old_pid = '/var/www/my_app/shared/pids/unicorn.pid.oldbin'
  if server.pid != old_pid
    begin
      sig = (worker.nr + 1) >= server.worker_processes ? :QUIT : :TTOU
      Process.kill(sig, File.read(old_pid).to_i)
    rescue Errno::ENOENT, Errno::ESRCH
      # someone else did our job for us
    end
  end
  sleep 0.5
end

So: at startup, the Upstart script doesn't find the pidfile, so it runs unicorn_rails, which keeps running.

Later, we redeploy our app, and a Capistrano task triggers app restart via:

kill -USR2 `cat /var/www/my_app/shared/pids/unicorn.pid`

This tells the old Unicorn master to start a new Unicorn master process, and as the new master starts workers, the Unicorn before_fork block sends TTOU signals to the old master to off the old workers (gracefully), then QUIT once there's only one worker left.

That QUIT causes the old master to exit (but only once there are new workers already handling the load), so the "bundle exec unicorn_rails" returns in the unicorn script. That script then loops around, sees the existing pidfile, and waits for the process to exit. It won't exit until the next deployment, but we'll loop around again if it does; we'd also loop around again anytime the master dies.

If the bash script itself dies, Upstart will restart it, because that's the process it's watching (as you see if you ever do status my_app -- Upstart reports the bash script's PID. You can still stop my_app, or restart my_app, which don't do any of the graceful stuff.

Indeed, a limitation of upstart is that it cannot track daemons that do what unicorn is doing.. that being fork/exec and exit their main process. Believe it or not, sshd does the same thing on SIGHUP, and if you look, /etc/init/ssh.conf makes sure sshd runs in the foreground. This is also one reason apache2 still uses an init.d script.

It sounds like gunicorn actually sort of daemonizes itself when receiving SIGUSR1 by forking and then exitting. This would be confusing for any of the process managers out there that try and keep a process alive.

I think you have two options. 1 is just to not use the SIGUSR1 and stop/start gunicorn when you need it.

The other option is to not use upstart's pid tracking, and just do this:

start on ..
stop on ..

pre-start exec gunicorn -D --pid-file=/run/gunicorn.pid
post-stop exec kill `cat /run/gunicorn.pid`

Not as sexy as pid tracking, but at least you won't have to write a whole init.d script.

(incidentally, this has nothing to do with the shebangs/execs. Both of those things work just like running a regular executable, so they wouldn't cause any extra forks).

According to the upstart documentation, you can tell upstart to not send a TERM and KILL signals to a service by saying so in your pre-stop block. All you have to do is say start in your pre-stop and it will abort the signal sending.

Together with the trick above by Bryan who figured out that a script can also include an infinite loop and not be the actual unicorn process -- I created an example that works using this technique.

The example is fairly simple, it handles sending USR2 when running upstart stop unicorn. And it will also respawn a new unicorn if for some reason all unicorns died on their own.

Code and output of tests is here - https://gist.github.com/kesor/6255584