Home / server / Questions / 156370
Accepted
mbac32768
Asked: 2010-07-01 09:16:38 +0800 CST

Daemons die with bus error when their binaries live on NFS

  • 772

We have some daemons executing on a number of hosts.

The daemon executable images are these very large binaries that are hosted on NFS.

When the binaries are updated on the NFS server, the previously running daemons sometimes drop dead with a Bus error. I'm assuming what's happening is the NFS server is replacing the binaries in a way that's invisible to the VFS layer on the NFS clients so they end up loading pages from the updated binary, which of course leads to madness.

We tried moving the new binaries into place instead of cp, but that doesn't seem to fix it.

I'm considering simply mlock()'ing the binary in the daemon startup script, but surely there's magic NFS options or semantics that we should be abusing. Is there a better way to fix this?

nfs bus

2 Answers

  1. Best Answer

    The best solution we've found is to always install the binary with a version string at the end of its name, and maintain a symbolic link always pointing to the latest version.

    /mnt/foo/bar -> bar-20111201000000
/mnt/foo/bar-20111201000000
/mnt/foo/bar-20111115000000

    When you install the new version you atomically move a new symbolic link over the old one.

    When you run the binary off of NFS, your process maps the versioned binary name, which new installations won't disturb. It also has this neat bonus feature where you can run ps and immediately see which version of the binary is running.

    • 2

  2. This is a common issue with NFS. When you remove the file, the existing NFS connection believes that the stat table it has is correct, goes to reload and gets a bus error.

    What you want to do is move the existing binary, put the new binary in place, after each of the machines have started using the new binary, remove the old one. Apache does this when it tries to mmap served files from NFS that change as well.

    • 0