Okay so...this all started during our Office 365 setup. According to Microsoft, you have to delete your on-premises federation trust from Exchange, verify the domain, then add it back...otherwise you get an obscure error message when validating the domain name.
So I did this...except now the federation trust is broken. I get the following message from "Test-FederationTrust -Verbose":
VERBOSE: [19:43:14.005 GMT] Test-FederationTrust : Retrieved Token Issuer Uri from Federation Metadata:
urn:federation:MicrosoftOnline.
VERBOSE: [19:43:14.005 GMT] Test-FederationTrust : Retrieved Token Issuer Certificate from Federation Metadata:
<snip>.
VERBOSE: [19:43:14.005 GMT] Test-FederationTrust : Retrieved Token Issuer Previous Certificate from Federation
Metadata: <snip>.
VERBOSE: [19:43:14.005 GMT] Test-FederationTrust : Retrieved Token Issuer End Point from Federation Metadata:
https://login.microsoftonline.com/extSTS.srf.
VERBOSE: [19:43:14.005 GMT] Test-FederationTrust : Retrieved Web Requestor Redirect End Point from Federation Metadata:
https://login.microsoftonline.com/login.srf.
VERBOSE: [19:43:14.912 GMT] Test-FederationTrust : Failed to request delegation token. Reason: <S:Fault
xmlns:S="http://www.w3.org/2003/05/soap-envelope"><S:Code><S:Value>S:Sender</S:Value><S:Subcode><S:Value>wst:FailedAuth
entication</S:Value></S:Subcode></S:Code><S:Reason><S:Text xml:lang="en-US">Authentication
Failure</S:Text></S:Reason><S:Detail><psf:error
xmlns:psf="http://schemas.microsoft.com/Passport/SoapServices/SOAPFault"><psf:value>0x80048821</psf:value><psf:internal
error><psf:code>0x80041012</psf:code><psf:text>The entered and stored passwords do not match.
</psf:text></psf:internalerror></psf:error></S:Detail></S:Fault>
Microsoft.Exchange.Net.WSTrust.SoapFaultException: Soap fault exception received.
at Microsoft.Exchange.Net.WSTrust.SoapClient.Invoke(IEnumerable`1 headers, XmlElement bodyContent)
at Microsoft.Exchange.Net.WSTrust.SecurityTokenService.IssueToken(DelegationTokenRequest request)
at Microsoft.Exchange.Management.SystemConfigurationTasks.TestFederationTrust.GetDelegationToken(ADUser user, Uri
target, SecurityTokenService securityTokenService)
What the heck does this mean? There's no passwords in a federated trust! I've attempted to recreate the trust multiple times to no avail. I also attempted to re-use the certificate the trust was working with before but that wasn't working either.
This also breaks the organizational relationships with the same message. I've asked our MSP and they have no idea what's wrong. Before I drop the money on a support ticket to Microsoft themselves ...has anyone seen this error message before?
I have also posted my Get-FederationTrust output below (scrubbed for security purposes, obviously):
RunspaceId : 5de750d3-a3c9-4502-a108-8b1f12d77fda
ApplicationIdentifier : 000000004804FA68
ApplicationUri : mydomain.com
OrgCertificate : [Subject]
CN=Federation
[Issuer]
CN=Federation
[Serial Number]
<snip>
[Not Before]
10/27/2017 11:58:27 AM
[Not After]
10/27/2022 11:58:27 AM
[Thumbprint]
<snip>
OrgNextCertificate :
OrgPrevCertificate :
OrgPrivCertificate : <snip>
OrgNextPrivCertificate :
OrgPrevPrivCertificate :
TokenIssuerCertificate : [Subject]
CN=Live ID STS Signing Public Key
[Issuer]
CN=Live ID STS Signing Public Key
[Serial Number]
<snip>
[Not Before]
12/6/2016 5:06:29 PM
[Not After]
12/5/2021 5:06:29 PM
[Thumbprint]
<snip>
TokenIssuerPrevCertificate : [Subject]
CN=Live ID STS Signing Public Key
[Issuer]
CN=Live ID STS Signing Public Key
[Serial Number]
<snip>
[Not Before]
7/18/2014 3:53:40 PM
[Not After]
7/17/2019 3:53:40 PM
[Thumbprint]
<snip>
PolicyReferenceUri : EX_MBI_FED_SSL
TokenIssuerMetadataEpr : https://nexus.microsoftonline-p.com/FederationMetadata/2006-12/FederationMetadata.xml
MetadataPollInterval : 1.00:00:00
TokenIssuerType : LiveId
TokenIssuerUri : urn:federation:MicrosoftOnline
TokenIssuerEpr : https://login.microsoftonline.com/extSTS.srf
WebRequestorRedirectEpr : https://login.microsoftonline.com/login.srf
MetadataEpr :
MetadataPutEpr :
TokenIssuerCertReference : stscer
TokenIssuerPrevCertReference : stsbcer
NamespaceProvisioner : LiveDomainServices2
AdminDisplayName :
ExchangeVersion : 0.10 (14.0.100.0)
Name : Microsoft Federation Gateway
DistinguishedName : CN=Microsoft Federation Gateway,CN=Federation Trusts,CN=<my CN>,CN=Mi
crosoft Exchange,CN=Services,CN=Configuration,DC=mydomain,DC=com
Identity : Microsoft Federation Gateway
Guid : fa98ab67-228f-4b8a-9f94-69b1d1609ec9
ObjectCategory : Divcom.com/Configuration/Schema/ms-Exch-Fed-Trust
ObjectClass : {top, msExchFedTrust}
WhenChanged : 10/27/2017 12:13:31 PM
WhenCreated : 10/27/2017 11:58:29 AM
WhenChangedUTC : 10/27/2017 4:13:31 PM
WhenCreatedUTC : 10/27/2017 3:58:29 PM
OrganizationId :
OriginatingServer : dc.mydomain.com
IsValid : True
This ended up resolving on its own since the other side of the trust also switched to O365. Not the answer I was hoping to get, but this is no longer relevant.