Until recently, we had at work a small cluster of about 20 small Windows servers (which have now all been virtualized). They were all configured to synchronize with the local time server. It was on an 1Gb sub-network in our own DC. I never got them to be less than about 100ms away from each other, which I consider to be an incredibly big difference.
Is that a normal value? What is a realistic expectation of time difference between machines running on a 1Gb network, and all connected to the same time server, and updating frequently, say every 5 minutes?
I would like to know this as setting timeouts and other parameters in a distributed application requires to take that difference into consideration.
NTP usually hovers in the <10-20ms range on a "reasonable" network. If you're using Windows built-in NTP client however, that's a different story. The Windows built-in client is only meant to keep the time synchronized enough for Kerberos, 5 minutes. They had no intention of keeping very accurate time; so 100ms is doing pretty good.
If you need more accurate time than 100ms, install an actual NTP client, like the Meinberg binaries of ISC's Reference NTPd.
If you need ultra accurate time, you should be looking at Precision Time Protocol (PTP). Be warned that PTP is much more complicated than NTP, so unless you need ultra accurate timestamping or something similar, NTP is highly advised.
NTP ought to be able to do a lot better than 100ms
Virtualized machines are notorious for having odd time sync issues.
You don't mention your virtualization environment, but check and make sure all the hypervisors are NTP sync'd as well. (Generally it is best for a guest VM machine to take its clock from its host hypervisor.)
See this 'Known Issue' in the NTP documentation -- it gives some more detail.
http://support.ntp.org/bin/view/Support/KnownOsIssues#Section_9.2.2.
See this paper from VMware for excruciating detail about problems with timekeeping on Windows hardware in general and additional problems with timekeeping in virtual machines:
http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf