Connecting your ntpd to NTP servers outside your LAN to time sync can lead to the inconsistencies you are seeing, because every connection will have to go thru several routers, each one with unpredictable latencies depending on traffic. If each server connects by itself, the time between all the servers will drift a little.
To avoid the inconsistency, the best approach is to create a local NTP server, syncing time with an external NTP Server Pool, and using this local NTP server as "master clock" to your local servers:
Local NTP Server
You configure this server to sync with a NTP server on a pool geographically close to you. On this site you can get a list of servers and get one close to you. It will keep the time in sync with them.
NTP Daemon on your servers
Configure the ntpd on every server to connect to your local NTP server. As you mantain a local NTP server on a very fast, low latency LAN, the skew will be minimal.
Using this implementation the time on your servers will be in sync with each other.
As pointed out by others, a local NTP server which syncs from a public one, and provides a low latency service for your other servers will probably be the best you can do.
You can get a very accurate external sync from GPS if you can see the sky, but that's often not practical.
I suspect if you are having problems, you have one of two issues: Either you are polling NTP (e.g. ntpdate from cron) instead of running a local daemon, or your problem is actually to do with testing and the latency in your network, and your servers might be better synchronized than you think.
Either way one of the obvious things to get right is good quality time source. Either get your own GPS receiver and/or atomic clock to your data center, or connect to multiple external stratum 0 clocks. Later strategy is a little uncertain. You never know what sort of clock in internet is providing it as stratum 0, they can be anything from really good clocks to grandpa's tick tocks.
p.s. Use ntpq -c peers to see stratum level of the clocks you are connecting to. More information about debugging, see http://www.clock.org/ntp/debug.html
Connecting your ntpd to NTP servers outside your LAN to time sync can lead to the inconsistencies you are seeing, because every connection will have to go thru several routers, each one with unpredictable latencies depending on traffic. If each server connects by itself, the time between all the servers will drift a little.
To avoid the inconsistency, the best approach is to create a local NTP server, syncing time with an external NTP Server Pool, and using this local NTP server as "master clock" to your local servers:
Local NTP Server
You configure this server to sync with a NTP server on a pool geographically close to you. On this site you can get a list of servers and get one close to you. It will keep the time in sync with them.
NTP Daemon on your servers
Configure the ntpd on every server to connect to your local NTP server. As you mantain a local NTP server on a very fast, low latency LAN, the skew will be minimal.
Using this implementation the time on your servers will be in sync with each other.
As pointed out by others, a local NTP server which syncs from a public one, and provides a low latency service for your other servers will probably be the best you can do.
You can get a very accurate external sync from GPS if you can see the sky, but that's often not practical.
I suspect if you are having problems, you have one of two issues: Either you are polling NTP (e.g. ntpdate from cron) instead of running a local daemon, or your problem is actually to do with testing and the latency in your network, and your servers might be better synchronized than you think.
Either way one of the obvious things to get right is good quality time source. Either get your own GPS receiver and/or atomic clock to your data center, or connect to multiple external stratum 0 clocks. Later strategy is a little uncertain. You never know what sort of clock in internet is providing it as stratum 0, they can be anything from really good clocks to grandpa's tick tocks.
p.s. Use
ntpq -c peersto see stratum level of the clocks you are connecting to. More information about debugging, see http://www.clock.org/ntp/debug.html