NTP Servers Not Syncing At Boot

Backstory: I have a couple of internal startum 1 NTP clocks with GPS receivers, and 2 public NTP servers that are virtualized on top of VMware ESXi which take time from the S1 clocks and distribute it. Otherwise this setup works rather fine and provides good time when compared to other public servers.

Problem: When I reboot the virtual machines, they do not start syncing properly, and get stuck in an unsynchronised state. Below is the ntpq -p output after a reboot.

root@server:~$ ntpq -p
 remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .GPS.            1 u   27   64    3    1.533  -258.43 5948.73
 192.168.2.40    .GPS.            1 u   24   64    3    1.118  -258.47 6138.19
 192.168.3.42    .GPS.            1 u   24   64    3    0.709  -258.42 5655.02
 194.100.49.151  194.100.49.134   2 u   22   64    3    8.124  -258.74 7131.65
 gbg1.ntp.se     .PPS.            1 u   26   64    3   21.856  -258.43 4876.90
 ntp2.sptime.se  .PPS.            1 u   23   64    3   19.991  -258.42 7764.97
 ntp1.sptime.se  .PPS.            1 u   27   64    3   20.489  -258.41 8574.46

If I then run ntp service restart I get this:

root@server:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .GPS.            1 u    2   64    1    1.517  -258.45   0.065
 192.168.2.40    .GPS.            1 u    1   64    1    1.126  -258.46   0.025
 192.168.3.42    .GPS.            1 u    2   64    1    0.719  -258.42   0.020
 194.100.49.151  194.100.49.134   2 u    5   64    1    8.041  -258.72   0.000
 gbg1.ntp.se     .PPS.            1 u    6   64    1   21.839  -258.41   0.000
 ntp2.sptime.se  .PPS.            1 u    4   64    1   19.968  -258.41   0.000
 ntp1.sptime.se  .PPS.            1 u    3   64    1   20.418  -258.43   0.000

A second later it steps:

root@server:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .STEP.          16 u    2   64    0    0.000    0.000   0.000
 192.168.2.40    .STEP.          16 u    2   64    0    0.000    0.000   0.000
 192.168.3.42    .STEP.          16 u    8   64    0    0.000    0.000   0.000
 194.100.49.151  194.100.49.134   2 u    -   64    1    7.976   -0.261   0.000
 gbg1.ntp.se     .PPS.            1 u    -   64    1   21.840    0.060   0.000
 ntp2.sptime.se  .STEP.          16 u    6   64    0    0.000    0.000   0.000
 ntp1.sptime.se  .STEP.          16 u    6   64    0    0.000    0.000   0.000

And after that we're back to normal operation:

root@server:~$ ntpq -p
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
 192.168.1.40    .GPS.            1 u    1   64    1    1.474    0.044   0.017
*192.168.2.40    .GPS.            1 u    1   64    1    1.102    0.030   0.005
 192.168.3.42    .GPS.            1 u    1   64    1    0.674    0.049   0.009
 194.100.49.151  194.100.49.134   2 u    8   64    1    7.976   -0.261   0.000
 gbg1.ntp.se     .PPS.            1 u    8   64    1   21.840    0.060   0.000
 ntp2.sptime.se  .PPS.            1 u    6   64    1   19.979    0.059   0.000
 ntp1.sptime.se  .PPS.            1 u    5   64    1   20.440    0.048   0.000

So it seems that after reboot the system clock is off by quite a bit, which is to be expected, but why ntpd doesn't panic and just steps the clock is a bit hard for me to understand.

Here's my ntp.conf

tinker panic 0
# /etc/ntp.conf, configuration for ntpd; see ntp.conf(5) for help

driftfile /var/lib/ntp/ntp.drift


# Enable this if you want statistics to be logged.
statsdir /var/log/ntpstats/

statistics loopstats peerstats clockstats
filegen loopstats file loopstats type day enable
filegen peerstats file peerstats type day enable
filegen clockstats file clockstats type day enable


# You do need to talk to an NTP server or two (or three).
#server ntp.your-provider.example

# pool.ntp.org maps to about 1000 low-stratum NTP servers.  Your server will
# pick a different set every time it starts up.  Please consider joining the
# pool: <http://www.pool.ntp.org/join.html>
server 192.168.1.40  iburst
server 192.168.2.40 iburst
server 192.168.3.42 iburst
server time1.mikes.fi
server ntp1.gbg.netnod.se
server ntp2.sptime.se
server ntp1.sptime.se

# Access control configuration; see /usr/share/doc/ntp-doc/html/accopt.html for
# details.  The web page <http://support.ntp.org/bin/view/Support/AccessRestrictions>
# might also be helpful.
#
# Note that "restrict" applies to both servers and clients, so a configuration
# that might be intended to block requests from certain clients could also end
# up blocking replies from your own upstream servers.

# By default, exchange time with everybody, but don't allow configuration.
restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

# Local users may interrogate the ntp server more closely.
restrict 127.0.0.1
restrict ::1

# Clients from this (example!) subnet have unlimited access, but only if
# cryptographically authenticated.
#restrict 192.168.123.0 mask 255.255.255.0 notrust


# If you want to provide time to your local subnet, change the next line.
# (Again, the address is an example only.)
#broadcast 192.168.123.255

# If you want to listen to time broadcasts on your local subnet, de-comment the
# next lines.  Please do this only if you trust everybody on the network!
#disable auth
#broadcastclient