How do you manage and deploy FreeBSD's ports in a large environment?

I have never been fully satisfied with the ports system in a large environment -- It always seems like you need to apply some external management to it in order to make it work well.

My best tips (in order of ascending preference, "worst" solution to "best" solution):

If you're building on each host, don't.
If you must, don't do it over NFS with read-write mounts like you describe: You can usually trust the ports to Do The Right Thing and not stomp on the ports tree if you provide alternate work directories, but it's always better to be safe than sorry: Run a local CVS/csup mirror and csup all your hosts from that box, then build locally as you would if they were individual machines.
Yes, I know this means having more disk space on the hosts and an extra step. It's also almost guaranteed to be problem-free.
Caveat: You probably want to sync the package configuration files (rsync or similar) from a designated "configuration host" to ensure consistency on each machine (you can even rsync the whole ports tree if you want, rather than using csup on each node).

Use a Build Host, create packages, and install those.
A much better solution than building on each individual machine: Use a build host to create packages, and point your tools at those packages.
This means keeping a build host around for every architecture you run (or cross-compiling), but it's ultimately nicer for your target machines (no large compile jobs, a guarantee of consistency)

Use a configuration/system management tool.
This is the solution I wound up with -- I build a standard server image and deploy it around my environment using radmind. You can do similar things with Puppet or Chef. This has all the advantages of using a build host (consistency, less load on the individual servers), and adds the benefit of configuration management.

Caveat: This only works really well if your machines are "identical" -- That is you can install the same set of ports on all of them. It can work if you have varying sets of ports, but that substantially increases the administrative overhead.

Disclaimer: I'm the port maintainer for sysutils/radmind. Yeah, I like it that much that I adopted it.

All of this is based on my experience managing various-sized FreeBSD environments (ranging from 1-2 machines to over 100). Configuration/System Management tools that push and maintain a standardized image are really the best way to handle this in my experience.

Strange that nobody mentioned ports-mgmt/tinderbox:

Tinderbox is a package building system for FreeBSD ports, based on official Portbuild scripts used on pointyhat building cluster. Tinderbox was written by Joe Marcus Clarke.

You can define multiple jails (base system versions) and multiple portstrees. The combination of jail and portstree is called a build. A Tinderbox jail is not what is understood as a jail in FreeBSD, it is in fact a given world in a chroot. Tinderbox supports automatic tracking of dependencies and only rebuilds packages that changed since last run. Tinderbox has support for email notification of failed builds. Tinderbox also integrates well with ccache.

Tinderbox is designed to easily provide package sets of ports you need, for platforms and architectures you need. Tinderbox is also excellent tool for testing new ports and port upgrades, especially for testing dependencies and packing lists. It's also useful for testing ports on various releases of FreeBSD, since you can run FreeBSD 6.X world as a jail on FreeBSD 7.X/8.X host.

Also switching to pkgng greatly simplifies package deployments.
Check it out on github: https://github.com/pkgng/pkgng

I have managed 100+ FreeBSD servers by simply sharing /usr read-only over well-tuned NFS, moving the package databases from /var to /usr and symlinking to them (not strictly necessary but enables pkg_info and such). There may have been one or two other files that needed to be moved in one direction or the other and symlinked but the whole setup took me about an hour to figure out. It worked very, very well. Had I run into scaling issues I would have added additional NFS servers and split the workload but it never came up. Performance was never an issue for me (in fact it was great) but I suppose you could put the NFS server's /usr (or a copy thereof) on an md.

It appears as nobody got a good solution to this unfortunately. Most likely this is due to limitations in the underlaying tools.

Here's what I came up with: I scrapped the idea of having the entire ports-tree exported. Instead, I gave in and put a full ports-tree on each node. I then mounted 'packages' over NFS (to enable distribution of packages).

I'm also intending to utilize a caching proxy (probably Squid) to speed up the portnap process. I wrote a short post about how to setup this on my blog.

References:

• Idea for proxy
• Implementation of http proxy in portsnap (or rather globally)