Ugh. We built a greenfield AD and Exchange 2010 environment a few months back for a large client. Outlook Anywhere isn't working. Outlook clients just never connect, doesn't seem to be an error or timeout.
This was a migration. Originally we had a cert mismatch for Outlook Anywhere so I'm fairly certain it never worked. We recently changed everyone's SMTP domain suffix to match the webmail URL's domain, so I expected that OA would start working. Now we have a snow day, lots of people at home, and it turns out it doesn't work. Looks like there's some sort of problem with proxied RPC connecting to OAB but I'm quite stumped as to how to fix this.
Autodiscover is good, OWA, ActiveSync, EWS all have been working for months.
We have 2 mailbox servers in a DAG, 1 mailbox server for public folders and journals, a 2-server CAS array. On the inside, we have NLB for the array. On the outside, we have a simply Netscaler LB VIP for the array that does nothing special, just a TCP 443 passthrough.
RCA shows this - it's always about 33 seconds so I assume there's a 30-second timeout here.
Attempting to ping RPC proxy webmail.5ssl.com. RPC Proxy was pinged successfully. Additional Details Elapsed Time: 201 ms.
Attempting to ping the MAPI Mail Store endpoint with identity: PFEXCAS.internal.domain.name.com:6001. The attempt to ping the endpoint failed. Additional Details An RPC error was thrown by the RPC Runtime process. Error 1818 CallCancelled
Elapsed Time: 33536 ms.
Now, both CAS servers are indeed listening on TCP 6001. PFEXCAS is the name of the NLB VIP.
This works for ports 6001, 6002, 6004, so that's good
rpcping -t ncacn_http -s pfexcas.internal.domain.name.com -o RpcProxy=pfexcas.internal.domain.name.com -P " , , " -I " , , " -H 1 -u 10 -a connect -F 3 -v 3 -e 6004
This does not, it gives a 1722
rpcping -t ncacn_http -s pfexcas.internal.domain.name.com -o RpcProxy=webmail.5ssl.com -P " , , " -I " , , " -H 1 -u 10 -a connect -F 3 -v 3 -e 6004
Test-OutlookConnectivity -Protocol:Http -verbose works just fine on both servers, including the GetNewDSA and RFRI::GetReferral for the Address Book, those fail in our next test.
This is Ex2010, so HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Rpc\RpcProxy\ValidPorts_AutoConfig_Exchange has been checked and looks correct on both CAS servers
PFEXCAS:6001-6004;PFEXCAS.internal.domain.name:6001-6004;PFEXCAS01:6001-6004;PFEXCAS01.internal.domain.name:6001-6004;PFEXCAS02:6001-6004;PFEXCAS02.internal.domain.name:6001-6004;PFEXMB01:6001-6004;PFEXMB01.internal.domain.name:6001-6004;PFEXMB02:6001-6004;PFEXMB02.internal.domain.name:6001-6004;PFEXMB03:6001-6004;PFEXMB03.internal.domain.name:6001-6004
Running Test-OutlookConnectivity -RpcProxyTestType:Internal -RpcTestType:Server -verbose gives us this (results same/similar when run on either CAS server. So there's something wrong with RPC proxying to the address book but hell if I can figure out what. Everything else in the environment seems to be working just fine, and this setup was configured quite closely to the original environment which had working OA.
VERBOSE: [19:08:13.652 GMT] Test-OutlookConnectivity : Target Client Access server = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.652 GMT] Test-OutlookConnectivity : Using connection parameters :
'internal.domain.name.com\extest_23e99fa922cb4: RpcProxy/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic]'
VERBOSE: [19:08:13.652 GMT] Test-OutlookConnectivity : Pinging RpcProxy at the following URL:
https://PFEXCAS01.internal.domain.name.com/rpc/RpcProxy.dll.
ClientAccessServer ServiceEndpoint Scenario Result Latency
(MS)
------------------ --------------- -------- ------ -------
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com RpcProxy::VerifyRpcProxy. Success 46.80
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:13.698 GMT] Test-OutlookConnectivity : Using connection parameters :
'internal.domain.name.com\extest_23e99fa922cb4: Rfri/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic],
[PFEXCAS01.internal.domain.name.com/Negotiate]'
VERBOSE: [19:08:13.714 GMT] Test-OutlookConnectivity : Mailbox = 'internal.domain.name.com\extest_23e99fa922cb4', Owner =
'/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb'
VERBOSE: [19:08:13.714 GMT] Test-OutlookConnectivity : Address Book Operation = 'GetNewDSA', Input = 'userLegacyDN',
Input Value = '/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb'
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : Microsoft.Exchange.Rpc.ServerUnavailableException: Error 0x6ba
(The RPC server is unavailable) from cli_RfrGetNewDSA
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:21.342
EEInfo: Generating component: 2
EEInfo: Status: 0x000006BA
EEInfo: Detection location: 1710
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 1
EEInfo: prm[0]: Long val: 0 (0x00000000)
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:21.342
EEInfo: Generating component: 13
EEInfo: Status: 0x000006BA
EEInfo: Detection location: 1352
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 1
EEInfo: prm[0]: Long val: -1073606646 (0xC002100A)
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:21.342
EEInfo: Generating component: 14
EEInfo: Status: 0xC002100A
EEInfo: Detection location: 1380
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 2
EEInfo: prm[0]: Long val: 12175 (0x00002F8F)
EEInfo: prm[1]: Unicode string: /rpc/rpcproxy.dll?PFEXCAS01.internal.domain.name.com:6002
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:21.342
EEInfo: Generating component: 14
EEInfo: Status: 0x00010000
EEInfo: Detection location: 1385
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 2
EEInfo: prm[0]: Long val: 16 (0x00000010)
EEInfo: prm[1]: Binary Buffer:
<junk>
at ThrowRpcException(Int32 rpcStatus, String message)
at Microsoft.Exchange.Rpc.RpcClientBase.ThrowRpcExceptionWithEEInfo(Int32 rpcStatus, String routineName)
at Microsoft.Exchange.Rpc.Rfri.RfriRpcClient.GetNewDSA(String userDN, String& server)
at Microsoft.Exchange.Monitoring.RfriClientWrapper.<>c__DisplayClass7.<GetNewDSA>b__6()
at Microsoft.Exchange.Monitoring.OutlookConnectivityContext.RpcExceptionWrapper(Func`1 protectedCall)
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : AddressBook operation failed. Operation = 'GetReferral', Error
= 'The Address Book Referral operation failed. Operation = '<GetNewDSA>b__6', returnValue = '1722', server =
'PFEXCAS01.internal.domain.name.com', user = '/o=FIVESTAR/ou=Exchange Administrative Group
(FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb', .'
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com RFRI::GetReferral. Failure -1.00
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : Using connection parameters :
'internal.domain.name.com\extest_23e99fa922cb4: Nspi/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic],
[PFEXCAS01.internal.domain.name.com/Negotiate]'
VERBOSE: [19:08:21.342 GMT] Test-OutlookConnectivity : Address Book Operation = 'Bind'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Microsoft.Exchange.Rpc.ServerUnavailableException: Error 0x6ba
(The RPC server is unavailable) from cli_NspiBind
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:28.955
EEInfo: Generating component: 2
EEInfo: Status: 0x000006BA
EEInfo: Detection location: 1710
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 1
EEInfo: prm[0]: Long val: 0 (0x00000000)
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:28.955
EEInfo: Generating component: 13
EEInfo: Status: 0x000006BA
EEInfo: Detection location: 1352
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 1
EEInfo: prm[0]: Long val: -1073606646 (0xC002100A)
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:28.955
EEInfo: Generating component: 14
EEInfo: Status: 0xC002100A
EEInfo: Detection location: 1380
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 2
EEInfo: prm[0]: Long val: 12175 (0x00002F8F)
EEInfo: prm[1]: Unicode string: /rpc/rpcproxy.dll?PFEXCAS01.internal.domain.name.com:6004
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:28.955
EEInfo: Generating component: 14
EEInfo: Status: 0x00010000
EEInfo: Detection location: 1385
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 2
EEInfo: prm[0]: Long val: 16 (0x00000010)
EEInfo: prm[1]: Binary Buffer:
<junk>
at ThrowRpcException(Int32 rpcStatus, String message)
at Microsoft.Exchange.Rpc.RpcClientBase.ThrowRpcExceptionWithEEInfo(Int32 rpcStatus, String routineName)
at Microsoft.Exchange.Rpc.Nspi.NspiRpcClient.Bind(UInt32 flags, IntPtr stat, IntPtr guid)
at Microsoft.Exchange.Monitoring.NspiClientWrapper.<Bind>b__4()
at Microsoft.Exchange.Monitoring.OutlookConnectivityContext.RpcExceptionWrapper(Func`1 protectedCall)
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Address Book operation failed. Details: The NSPI operation
failed. Operation = '<Bind>b__4', ReturnValue = '1722', Server = 'PFEXCAS01.internal.domain.name.com', User =
'/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb', ..
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com NSPI::GetProfileDetails. Failure -1.00
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Using connection parameters :
'internal.domain.name.com\extest_23e99fa922cb4: Mailbox/RPC-over-HTTP, [PFEXCAS01.internal.domain.name.com/Basic],
[PFEXCAS01.internal.domain.name.com/Negotiate]'
VERBOSE: [19:08:28.971 GMT] Test-OutlookConnectivity : Mailbox Operation: 'EcDoConnectEx', Target mailbox:
'/o=FIVESTAR/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=extest_23e99fa922cb46cb'
VERBOSE: [19:08:28.986 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:28.986 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:36.599 GMT] Test-OutlookConnectivity : Microsoft.Exchange.Rpc.ServerUnavailableException: Error 0x6ba
(The RPC server is unavailable) from EcDoConnectEx
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:36.599
EEInfo: Generating component: 13
EEInfo: Status: 0x000006BA
EEInfo: Detection location: 1352
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 1
EEInfo: prm[0]: Long val: -1073606646 (0xC002100A)
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:36.599
EEInfo: Generating component: 14
EEInfo: Status: 0xC002100A
EEInfo: Detection location: 1380
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 2
EEInfo: prm[0]: Long val: 12175 (0x00002F8F)
EEInfo: prm[1]: Unicode string: /rpc/rpcproxy.dll?PFEXCAS01.internal.domain.name.com:6001
EEInfo: ComputerName: n/a
EEInfo: ProcessID: 4520
EEInfo: Generation Time: 2015-01-27 19:08:36.599
EEInfo: Generating component: 14
EEInfo: Status: 0x00010000
EEInfo: Detection location: 1385
EEInfo: Flags: 0
EEInfo: NumberOfParameters: 2
EEInfo: prm[0]: Long val: 16 (0x00000010)
EEInfo: prm[1]: Binary Buffer:
<junk>
at ThrowRpcException(Int32 rpcStatus, String message)
at Microsoft.Exchange.Rpc.RpcClientBase.ThrowRpcExceptionWithEEInfo(Int32 rpcStatus, String routineName)
at Microsoft.Exchange.Rpc.ExchangeClient.ExchangeRpcClient.EcDoConnect_Wrapped(Boolean isEx, Void** pcxh, Byte*
szUserDN, UInt32 ulFlags, UInt32 ulConMod, UInt32 cbLimit, UInt32 ulCpid, UInt32 ulLcidString, UInt32 ulLcidSort,
UInt32 ulIcxrLink, UInt16 usFCanConvertCodePages, UInt32* pcmsPollsMax, UInt32* pcRetry, UInt32* pcmsRetryDelay,
UInt16* picxr, Byte** pszDNPrefix, Byte** pszDisplayName, UInt16* rgwClientVersion, UInt16* rgwServerVersion, UInt16*
rgwBestVersion, UInt32* pulTimeStamp, Byte* rgbAuxIn, UInt32 cbAuxIn, Byte* rgbAuxOut, UInt32* pcbAuxOut)
at Microsoft.Exchange.Rpc.ExchangeClient.ExchangeRpcClient.EcDoConnect_Internal(Boolean isEx, IntPtr& contextHandle,
String userDn, Int32 flags, Int32 sizeLimit, Int32 conMod, Int32 cpid, Int32 lcidString, Int32 lcidSort, Int32
sessionIdLink, Boolean canConvertCodePages, TimeSpan& pollsMax, Int32& retryCount, TimeSpan& retryDelay, String&
dnPrefix, String& displayName, Int16[] clientVersion, Int16[]& serverVersion, Int16[]& bestVersion, Int32& timeStamp,
Byte[] auxIn, Boolean compressAuxIn, Boolean xorMagicAuxIn, Byte[]& auxOut, Int32 maxSizeAuxOut, Boolean&
wasCompressedAuxOut, Boolean& wasXorMagicAuxOut)
at Microsoft.Exchange.Rpc.ExchangeClient.ExchangeRpcClient.EcDoConnectEx(IntPtr& contextHandle, String userDn, Int32
flags, Int32 sizeLimit, Int32 conMod, Int32 cpid, Int32 lcidString, Int32 lcidSort, Int32 sessionIdLink, Boolean
canConvertCodePages, TimeSpan& pollsMax, Int32& retryCount, TimeSpan& retryDelay, String& dnPrefix, String&
displayName, Int16[] clientVersion, Int16[]& serverVersion, Int16[]& bestVersion, Int32& timeStamp, Byte[] auxIn,
Boolean compressAuxIn, Boolean xorMagicAuxIn, Byte[]& auxOut, Int32 maxSizeAuxOut, Boolean& wasCompressedAuxOut,
Boolean& wasXorMagicAuxOut)
at Microsoft.Exchange.Monitoring.MailboxTask.<>c__DisplayClassb.<Connect>b__a()
at Microsoft.Exchange.Monitoring.OutlookConnectivityContext.RpcExceptionWrapper(Func`1 protectedCall)
VERBOSE: [19:08:36.599 GMT] Test-OutlookConnectivity : EcDoConnectEx Result: '1722'
VERBOSE: [19:08:36.599 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com Mailbox::Connect. Failure -1.00
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : RPC Endpoint = 'PFEXCAS01.internal.domain.name.com'
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : This step was skipped. This may have been caused by the failure
in a previous scenario. Operation = 'MailboxTask::Logon'.
PFEXCAS01.FVE.AD.... PFEXCAS01.internal.domain.name.com Mailbox::Logon. Skipped -1.00
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : Admin Audit Log: Entered Handler:OnComplete.
VERBOSE: [19:08:36.615 GMT] Test-OutlookConnectivity : Ending processing &
This was a stupid configuration Heisenbug of some kind. The registry was configured properly for RPC, and the problem was not specific to the OAB, that's just the first thing that's checked. It took a reboot of the CAS servers to get it to work.
Further, the real root cause was that I had recently installed a third CAS server as a test, not in the array. I fed it dummy values for public URLs that weren't reachable, and for some reason (wish I knew) those got written into Autodiscover. I uninstalled the CAS role from that test machine and it cleaned up autodiscover, but still had trouble with RPCPing until the reboot of the remaining CAS servers.