When I shutdown Ubuntu running mongodb (installed via apt), occasionally mongodb doesn't appear to finish shutting down completely - the log file matches a healthy shutdown, closing files, cleaning up the journal etc.. but the filnal [interruptThread] shutdown: removing fs lock... isn't present, and (unsurprisingly) mongodb refuses to start at the next bootup, complaining of a stale lock file.
It appears to me that there is a timeout for services to quit by themselves that is expiring, and thus mongodb is killed forcibly.
The offending mongodb is part of a replica set that sync's over ssh tunnels. It may thus be possible that a socket is hanging and holding up the final cleanup (including removing the lock file)??
What can I do about this? Can I set a longer timeout for mongodb to shutdown? Can I insist that mongodb be allowed to die cleanly before ssh tunnels are killed? Is there another explanation that I'm missing??
Indeed, there is a limit on the time that services are allowed to take to shut down. Ubuntu uses upstart to start and stop services, and upstart's default limit for that is 5 seconds (see the upstart documentation on kill timeout). Possibly it will solve your problem if you increase that timeout. You have to place the
kill timeoutdirective mentioned in the link in the upstart configuration file for mongodb. On Ubuntu 12.04, this file is found at/etc/init/mongodb.conf. You can find an upstart file that contains thekill timeoutdirective in the current mongodb source. (However, this file may not work on your Ubuntu system due to different paths.)I had the same problem on Ubuntu 12.04 even without replication, and for me
kill timeout 300fixed the problem.