How to delete directories without symlinks in one folder

As Bron as said, there is no official way of doing it (more strongly: there cannot be one due to the filesystem design, you can only roll your own). This would be how I would script it in bash (assuming the current structure, the fgrep is too lenient to be used generally):

# Current list of symlinks (will return build.3 and build.4 in this case)
symed=`ls -1 | xargs -l readlink`;
# Loop over all build directories
for dir in build.*;
do
    # Either it's a known symlink, or we remove it
    fgrep -q $dir <<< $symed || rm -rf $dir;
done

Not so much a "proper" way, but how often are you doing it? My usual approach would be to run up a small piece of Perl (choose your own poison) which read the directory contents, examined the symlinks, and built a list of directories to delete.

There's no metadata on the destination of a symlink that will tell you it's being linked to.

You'll want to take care about race conditions too - if a separate task is updating the symlinks, you don't want to delete the new directory before it gets linked!