Home / server / Questions / 613513
In Process
msanford
Asked: 2014-07-18 17:56:24 +0800 CST

Upstart can't stop foreverjs services on shutdown?

  • 772

Scenario

We use a Vagrant-based virtual machine running Ubuntu 12.04 LTS for development of three Node.JS servers that are started using foreverjs. We run the services under the vagrant user and mount the projects from the host to the virtual machine as shared folders.

When starting a script with forever, it will fork to the background after starting, maintaining a central list of all scripts running (per user). For scripts that use different names, you can reference them by name as well as a numeric index.

Scripts are then managed (started, restarted, stopped, etc) using the forever command.

Upstart configuration

I have created similar upstart conf files for each service thusly:

description "Control server.js"

chdir /vagrant/server

start on vagrant-mounted

stop on runlevel [016]

expect fork

pre-start script
  test -d /vagrant/server
end script

exec sudo su vagrant -c "/usr/local/bin/forever start server.js"

Problem — hangs on shutdown

The upstart successfully starts the service once the vagrant shared folder has mounted. However, shutting down the virtual machine (i.e., vagrant halt) hangs until it times out and force-shuts-down the VM.

I assume this is because it doesn't know how to stop the forever service, since forever is merely an interface to the services I actually want to stop.

My only attempt didn't work (it does stop the service, but the Vagrant hangs to timeout):

pre-stop script
  forever stop server.js
end script

The node services do not need to terminate gracefully: I would be happy with any scenario that kills the scripts but does not cause the Vagrant to hang on shutdown.

Also, removing the stop on runlevel line seems to have no effect.

vagrant

1 Answers

  1. First, your problem seems to be improper fork tracking. To solve this issue, I am going to make a few tweaks to your upstart config:

    • First, scratch the usage of su to change the user. su should not be used for daemons, only regular users (it can cause problems with pam being triggered), and (more relevantly) it forks the process to the background (harder for Upstart to track and kill your daemon). We will use setuid vagrant in the upstart config instead.

    • The next problem is that forever forks the process to the background and forks itself to the background. This leaves Upstart tracking some random pid that probably is not even alive anymore. The solution is to use Upstart's automatic respawning feature. I will demonstrate by example below.

      description "server.js"

start on vagrant-mounted
stop on runlevel [016]

chdir /vagrant/server
setuid vagrant

respawn
respawn limit unlimited

exec server.js

    Note that you do not need to check if /vagrant/server is a directory because upstart automatically creates it because of the chdir stanza.

    • 1