Before I begin, I'm aware of all the issues that can arise from installing VMWare server on a DC. I did it anyway for a learning experience.
I'm trying to run a linux distro on my DC so I need to virtualise rather than run another box. My domain is pretty small so I'm not worried about load issues but the virtual network adapters VMware creates are affecting my DNS and generating all sorts of errors.
I'm not 100% sure but it looks like VMware requires the virtual adapters to work properly so there's no way of avoiding that issue. However, I might be able to disable the connections from registering with DNS.
edit:
VS creates a series of virtual network adapters which register as actual NICs in Windows. Since the machine is a DC, it will register these into DNS and may cause confusion when trying to resolve the server's name. I'm aware of the problem but am desperately trying to avoid having to migrate the DC/exchange to a virtual machine to get everything working in the one box. Therefore, I'm after some possible solutions to this problem.
The correct thing to do here would have been to have virtualised the entire server with something like esxi and then installed the DC and the linux partition side-by-side.
However, if you insist on stepping over to the dark side, you need to go to each one of those virtual NICs in the Network Connections screen, go to properties, then properties for the connection, then properties for TCI/IP v4, then Advanced, then DNS, and un-check "Register this connection's address in DNS"
Repeat and rinse for all the virtual network cards. I doubt this is the only issue that you'll encounter, but it should get you past this hurdle.
Also, go into DNS manager, and set the DNS server to listen on only the real (non-vmware) interfaces. Make sure you uncheck all the vmware IPv4 and IPv6 addresses. That should stop A record lookup for the server name or domain name returning the vmware interface addresses, among other problems.
Depending on the VM network settings, this may break their DNS resolution. I think you will be better off finding another way to fix that (manual settings; altering the vmware dhcp settings, if poosible; etc.).