Ping a Specific Port

Question

smccloud

Asked: 2012-09-15 09:33:16 +0800 CST2012-09-15 09:33:16 +0800 CST 2012-09-15 09:33:16 +0800 CST

Weird routing issue

772

I am working on getting networking working correctly on a cluster of 14 virtual servers at a customer site. 11 of them work fine for routing and 3 don't work correctly for their administrative network (172.28.56.0). All are running Windows Web Server 2008R2.

Default gateway is set on the production network (172.28.58.0) and not on the administrative network (handled with persistent static routes).

On a working server, route print gives me the following (MACs redacted)

===========================================================================
Interface List
 11...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection
 13...XX XX XX XX XX XX00 0c 29 85 b2 98 ......Intel(R) PRO/1000 MT Network Connection #2
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 15...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
===========================================================================

IPv4 Route Table

===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
           0.0.0.0         0.0.0.0      172.28.58.1    172.28.58.11     266
        10.18.1.22 255.255.255.255      172.28.58.1    172.28.58.11      11
         10.32.0.0     255.255.0.0      172.28.56.1   172.28.56.201      11
         127.0.0.0       255.0.0.0          On-link       127.0.0.1     306
         127.0.0.1 255.255.255.255          On-link       127.0.0.1     306
   127.255.255.255 255.255.255.255          On-link       127.0.0.1     306
      172.28.34.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.42.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.56.0    255.255.255.0          On-link   172.28.56.201     266
      172.28.56.0    255.255.255.0      172.28.56.1   172.28.56.201      11
    172.28.56.201  255.255.255.255          On-link   172.28.56.201     266
    172.28.56.255  255.255.255.255          On-link   172.28.56.201     266
      172.28.58.0  255.255.255.224          On-link    172.28.58.11     266
      172.28.58.0  255.255.255.224      172.28.58.1    172.28.58.11      11
      172.28.58.1  255.255.255.255      172.28.58.1    172.28.58.11      11
     172.28.58.11  255.255.255.255          On-link    172.28.58.11     266
     172.28.58.31  255.255.255.255          On-link    172.28.58.11     266
      172.28.60.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      172.28.63.0    255.255.255.0      172.28.56.1   172.28.56.201      11
      192.168.0.0      255.255.0.0      172.28.56.1   172.28.56.201      11
        224.0.0.0        240.0.0.0          On-link       127.0.0.1     306
        224.0.0.0        240.0.0.0          On-link   172.28.56.201     266
        224.0.0.0        240.0.0.0          On-link    172.28.58.11     266
  255.255.255.255  255.255.255.255          On-link       127.0.0.1     306
  255.255.255.255  255.255.255.255          On-link   172.28.56.201     266
  255.255.255.255  255.255.255.255          On-link    172.28.58.11     266
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
      172.28.56.0    255.255.255.0      172.28.56.1       1
      172.28.63.0    255.255.255.0      172.28.56.1       1
      192.168.0.0      255.255.0.0      172.28.56.1       1
      172.28.60.0    255.255.255.0      172.28.56.1       1
        10.32.0.0      255.255.0.0      172.28.56.1       1
      172.28.34.0    255.255.255.0      172.28.56.1       1
      172.28.42.0    255.255.255.0      172.28.56.1       1
          0.0.0.0          0.0.0.0      172.28.58.1  Default
===========================================================================

IPv6 Route Table

===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

On one of the non-working server, route print gives me the following (MACs redacted)

===========================================================================
Interface List
 11...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection
 13...XX XX XX XX XX XX ......Intel(R) PRO/1000 MT Network Connection #2
  1...........................Software Loopback Interface 1
 12...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter
 14...00 00 00 00 00 00 00 e0 Microsoft ISATAP Adapter #2
 16...00 00 00 00 00 00 00 e0 Teredo Tunneling Pseudo-Interface
===========================================================================

IPv4 Route Table

===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
           0.0.0.0         0.0.0.0      172.28.58.1    172.28.58.21     266
         10.32.0.0     255.255.0.0      172.28.56.1   172.28.56.211      11
         127.0.0.0       255.0.0.0          On-link       127.0.0.1     306
         127.0.0.1 255.255.255.255          On-link       127.0.0.1     306
   127.255.255.255 255.255.255.255          On-link       127.0.0.1     306
       172.28.34.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.42.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.56.0   255.255.255.0      172.28.56.1   172.28.56.211      11
     172.28.56.211 255.255.255.255          On-link   172.28.56.211     266
       172.28.58.0   255.255.255.0      172.28.58.1    172.28.58.21      11
       172.28.58.0 255.255.255.224          On-link    172.28.58.21     266
      172.28.58.21 255.255.255.255          On-link    172.28.58.21     266
      172.28.58.31 255.255.255.255          On-link    172.28.58.21     266
       172.28.60.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       172.28.63.0   255.255.255.0      172.28.56.1   172.28.56.211      11
       192.168.0.0     255.255.0.0      172.28.56.1   172.28.56.211      11
         224.0.0.0       240.0.0.0          On-link       127.0.0.1     306
         224.0.0.0       240.0.0.0          On-link   172.28.56.211     266
         224.0.0.0       240.0.0.0          On-link    172.28.58.21     266
   255.255.255.255 255.255.255.255          On-link       127.0.0.1      06
   255.255.255.255 255.255.255.255          On-link   172.28.56.211     266
   255.255.255.255 255.255.255.255          On-link    172.28.58.21     266
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
      172.28.56.0    255.255.255.0      172.28.56.1       1
      172.28.60.0    255.255.255.0      172.28.56.1       1
      172.28.63.0    255.255.255.0      172.28.56.1       1
      172.28.34.0    255.255.255.0      172.28.56.1       1
      172.28.42.0    255.255.255.0      172.28.56.1       1
      192.168.0.0      255.255.0.0      172.28.56.1       1
        10.32.0.0      255.255.0.0      172.28.56.1       1
          0.0.0.0          0.0.0.0      172.28.58.1  Default
      172.28.58.0    255.255.255.0      172.28.58.1       1
===========================================================================

IPv6 Route Table

===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    306 ::1/128                  On-link
  1    306 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

I am at a complete loss why the non-working servers have no On-link route for 172.28.56.0. Does anyone have any suggestions on what I should be looking at to figure this out? Also, I do have "physical" access to the console if needed through vSphere Client.

ipconfig /all from non-working machine (sensitive info redacted) Windows IP Configuration

   Host Name . . . . . . . . . . . . : XXX-XXXXXXX4
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : XXX.XXXXXXX.com
                                       XXXXXXX.com
                                       XXXXX.XXXXXX.org
                                       XXXXXX.org

Ethernet adapter Admin:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.56.211(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 172.28.56.209
                                       172.28.60.209
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Prod:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.58.21(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.224
   Default Gateway . . . . . . . . . : 172.28.58.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{4E9F1EC4-67C6-4B48-B3D1-4FDAF38452DB}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{7D31E116-0136-4E0F-B120-BF6C06FB9131}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Teredo Tunneling Pseudo-Interface:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

ipconfig /all from working machine (sensitive info redacted) Windows IP Configuration

   Host Name . . . . . . . . . . . . : XXX-XXXXXXX1
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Mixed
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : XXX.XXXXXXX.com
                                       XXXXXXX.com
                                       XXXXX.XXXXXX.org
                                       XXXXXX.org

Ethernet adapter Admin:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.56.201(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Default Gateway . . . . . . . . . :
   DNS Servers . . . . . . . . . . . : 172.28.56.209
                                       172.28.60.209
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Prod:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Network Connection #2
   Physical Address. . . . . . . . . : XX-XX-XX-XX-XX-XX
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   IPv4 Address. . . . . . . . . . . : 172.28.58.11(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.224
   Default Gateway . . . . . . . . . : 172.28.58.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{713322C9-3AEA-4517-B8C6-4AAD3BFA4E62}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{2387819F-D4FA-4F52-8EEE-C8BF25A0846F}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #2
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter Local Area Connection* 12:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Teredo Tunneling Pseudo-Interface
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

1 Answers

Voted

August · Answer 1 · 2012-09-15T10:20:06+08:00

The non-working server has a persistent route for:

172.28.58.0    255.255.255.0      172.28.58.1       1

...that the working server does not have, and the 172.28.58.x network is (apparently) a 255.255.255.224 network according to the IP config so I'd remove that route on the non-working server. I'm not sure what type of traffic is flowing on each NIC/subnets or what specifically "doesn't work correctly for their administrative network (172.28.56.0)"

If all non-working servers are on the ESXi host (and you've confirmed that that is the common thread by vMotioning one of them to another ESXi host and it works), then your issue might be in:

the ESXi host's physical network connections (switch ports not configured correctly for the ESXi host connections)
you have swapped your vSwitch mappings on the VM (NIC1 mapped to Prod vswitch & NIC2 mapped to Admin vswitch or vice versa)
your vSwitch uplinks are swapped between physical NICs on the ESXi host (Prod vswitch uplinks to physical NIC1 & Admin vswitch uplinks to physical NIC2 or vice versa)

If you know the problem is limited to the one ESXi hosts, double-check the vswitch settings and compare to a working host to make sure they are the same and that the physical connections to the network for the host are the same as a working host.

Weird routing issue

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?