Currently I have a few FreeBSD servers with one of them acting as build box sharing its /usr/ports and /usr/ports/distfiles (as separate file systems) through NFS. Recently I've added OpenBSD and Gentoo boxes to this network. Now I want to use existing FreeBSD build box to share OpenBSD and Gentoo distfiles through NFS. Is it safe to place all distfiles for different systems into the same directory?
Sounds like a pretty good way to handle your distfiles. Most of the time *BSD and Gentoo will download vanilla sources for the packages and add their own patch sets (if any) over them.
Best way to see if this works would be to make a test setup with only a few servers or VMs, and see if there will be filename/checksum mismatches when you install your own typical set of packages on your systems.
Be careful with your /usr/ports/packages across your *BSD systems, you might want to keep the different flavors on separated NFS shares.
I'm not familiar with *BSD but under Gentoo, you can share portage over NFS fine.