Configuring Azure Internal Load Balancer

I am attempting to configure a VNET using the new Internal Load Balancing feature in Azure. I plan to have 2 frontend servers communicating with 2 backend servers over an ILB (Internal Load Balancer).

I believe I have followed all of the instructions correctly, but the machines in my network still can't seem to communicate using the ILB.

What I have done:

I have setup a regional virtual network with 2 subnets, Subnet-1 and Subnet-2 by uploading the following configuration file since Regional VNETs are not available through the UI yet.

Network configuration:

<NetworkConfiguration xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/ServiceHosting/2011/07/NetworkConfiguration">
  <VirtualNetworkConfiguration>
    <VirtualNetworkSites>
      <VirtualNetworkSite name="my-net" Location="West US">
        <AddressSpace>
          <AddressPrefix>192.168.10.0/24</AddressPrefix>
        </AddressSpace>
        <Subnets>
          <Subnet name="BackendSubnet">
            <AddressPrefix>192.168.10.0/27</AddressPrefix>
          </Subnet>
          <Subnet name="FrontendSubnet">
            <AddressPrefix>192.168.10.32/27</AddressPrefix>
          </Subnet>
          <Subnet name="GatewaySubnet">
            <AddressPrefix>192.168.10.64/29</AddressPrefix>
          </Subnet>
        </Subnets>
      </VirtualNetworkSite>
    </VirtualNetworkSites>
  </VirtualNetworkConfiguration>
</NetworkConfiguration>

I created 2 VMs in 2 Cloud Services and added them to my VNet with the following powershell.

$imgname = "ad072bd3082149369c449ba5832401ae__Windows-Server-Remote-Desktop-Session-Host-on-Windows-Server-2012-R2-20140403-2126"
$vmname1 = "Backend1"
$vmname2 = "Backend2"
$vmname3 = "Frontend1"
$vmname4 = "Frontend2"
$backendSubnet = "BackendSubnet"
$frontendSubnet = "FrontendSubnet"
$adminname = "SomeAdmin"
$adminpw = "SomePassword"
$backend1 = New-AzureVMConfig -Name $vmname1 -InstanceSize Small -ImageName $imgname | Add-AzureProvisioningConfig -Windows -AdminUsername $adminname -Password $adminpw | Set-AzureSubnet $backendSubnet
$backend2 = New-AzureVMConfig -Name $vmname2 -InstanceSize Small -ImageName $imgname | Add-AzureProvisioningConfig -Windows -AdminUsername $adminname -Password $adminpw | Set-AzureSubnet $backendSubnet
$frontend1 = New-AzureVMConfig -Name $vmname3 -InstanceSize Small -ImageName $imgname | Add-AzureProvisioningConfig -Windows -AdminUsername $adminname -Password $adminpw | Set-AzureSubnet $frontendSubnet
$frontend2 = New-AzureVMConfig -Name $vmname4 -InstanceSize Small -ImageName $imgname | Add-AzureProvisioningConfig -Windows -AdminUsername $adminname -Password $adminpw | Set-AzureSubnet $frontendSubnet
New-AzureVM -ServiceName "my-net-backend" -VMs $backend1 -Location "West US" -VNetName "my-net" 
New-AzureVM -ServiceName "my-net-backend" -VMs $backend2 -Location "West US"
New-AzureVM -ServiceName "my-net-frontend" -VMs $frontend1 -Location "West US" -VNetName "my-net" 
New-AzureVM -ServiceName "my-net-frontend" -VMs $frontend2 -Location "West US"

I remoted to each VM and configured IIS on port 80 with a simple HTML page to test with and verified it works locally, anonymous access, etc.

Then I created a DNS server in the VNET with the following powershell:

$dnsname = "MyDNS"
$dnsconfig = New-AzureVMConfig -Name $dnsname -InstanceSize Small -ImageName $imgname | Add-AzureProvisioningConfig -Windows -AdminUsername $adminname -Password $adminpw | Set-AzureSubnet $frontendSubnet

After the VM was ready, I configured my server as a DNS server in Windows 2012 R2 and I added a forward lookup zone with a simple suffix of "mytest.net"

Then in the Azure Management UI, I added this DNS server to my VNET. After the VNET reconfigured, I rebooted all of my VMs.

Once my VM's were back online, I remoted to my DNS server and created an A-Record for "test.mytest.net"

After which I remoted to 1 server in each subnet and ensured the NSLOOKUP for "test.mytest.net" resolved correctly.

Everything seemed to be going smoothly at this point.

Finally, I started to configure the new Azure Internal Load Balancer using Powershell as follows:

Add-AzureInternalLoadBalancer -InternalLoadBalancerName "MyILB" -SubnetName "BackendSubnet" -ServiceName "my-net-backend" -StaticVNetIPAddress "192.168.10.30"
Get-AzureVM -ServiceName "my-net-backend" -Name $backend1 | Add-AzureEndpoint -Name "backend-ep" -LBSetName "backend-ep-set" -Protocol tcp -LocalPort 80 -PublicPort 80 -ProbePort 80 -ProbeProtocol http -ProbeIntervalInSeconds 10 -InternalLoadBalancerName "MyILB" | Update-AzureVM
Get-AzureVM -ServiceName "my-net-backend" -Name $backend2 | Add-AzureEndpoint -Name "backend-ep" -LBSetName "backend-ep-set" -Protocol tcp -LocalPort 80 -PublicPort 80 -ProbePort 80 -ProbeProtocol http -ProbeIntervalInSeconds 10 -InternalLoadBalancerName "MyILB" | Update-AzureVM

From my understanding, I just created an ILB named MyILB with a static IP of 192.168.10.30, then on each VM in my backend, I created an endpoint for HTTP port 80 and added that machine to the ILB. At this point, I would assume that HTTP port 80 traffic addressed to 192.168.10.30 would be load balanced between my Backend1 and Backend2 VMs. However, this does not appear to work as I expected.

I connected to the DNS Server and created an A-Record for 192.168.10.30 with a name of backend.mytest.net and verified this name could be resolved on all servers.

When I remote to a frontend server and use either the DNS name backend.mytest.net or 192.168.10.30 in the browser I get failure.

I have also verified that on both backend servers HTTP 80 inbound is allowed in the firewall for all regions (domain, public and private).

The management UI doesn't show these endpoints on the VMs, but I suspect that is because they are created via the powershell commands and the UI doesn't support this yet.

Anyone have any insight or thoughts here? I know I am on the edge of getting this to work.

Instructions used:
Internal Load Balancing Announcement
Regional Virtual Networks
Bad Request Troubleshooting
Setting up DNS in Azure