I have a weird situation. My Win 2016 servers were ruined by AD group polices.
IIS 10 have started to throw 403 error after automatic AD group policy update at the moment X. Beforehand remote party called us with no problems via https with client certificates.
I have a backup before the moment X and after the moment X, but I cannot pinpoint the setting that was changed by AD GP.
I have a setup paper with all settings needed to setup our web service from scratch. I have checked them all against victim servers. Nothing changed. At every level Anonymous Auth is enabled and SSL Settings->Client certificates->Accept.
Our admins swear they have changed nothing. IIS logs show nothing after moment X.
Any ideas where to look at? Registry? Metabase?
Exact error returned to the caller as per SOAP e2e trace logs:
The HTTP request was forbidden with client authentication scheme 'Anonymous'.[The remote server returned an error: (403) Forbidden.]
A common issue with Group Policy and IIS servers is when a GPO overrides user rights assigned by the local security policy and the IIS service account(s) cannot log on. The Windows Security log should have EventId 4625 logon failure events for your IIS server service accounts if this is happening.
You can quickly validate any GPO interference by running RSOP.MSC from your IIS server and navigating to Computer Configuration > Policies > Windows Settings > Security Settings > Local Policies > User Rights Assignment
Look at the Source GPO column to identify which GPO is applying a setting. Use gpedit.msc to view/edit settings applied by local group policy.
You can use this article to validate your IIS server permissions. Start with the 'Windows user rights that are assigned by local security policy' table at the bottom of the page.
Remember: GPOs are not the enemy as long as they are tested and applied in a controlled manner.
Short answer
Check for rogue certificates. In my case there was one non–self-signed certificate in the Trusted Root Certification Authorities certificate store. Our admins have had added this government-issued certificate to the entire forest (via Group Policy) without proper testing beforehand.
Details
I have put here some general troubleshooting guidelines in order to help people to solve such problems faster than 3 weeks as it was in my case.
AD Group Policies
If you are not familiar with Group Policies use the command below to get human-readable report (best viewed by IE):
To view more details use
RSOP.MSC
as per @twconnell recommendation. RSOP.MSC shows effective snapshot of both local and group polices applied. Verify security atComputer Configuration > Windows Settings > Security Settings > Local Policies > User Rights Assignment
. In my case I have thoroughly read this excellent SO answer and followed every link there. Then I have checked access rights on the each file, folder and app pool. There was nothing wrong there.IIS logs
Look at the IIS logs. There should be more details than just
403
code provided to the client. In my case there was403 16
code there.Look at this troubleshooting guide. In my case it was that short:
Client certificate is untrusted or invalid
Certificate Revocation List (CRL)
My certificate CA was an AD-provided company CA, so I have initially thought that there might be some problem with certificate revocation list (CRL). Best practical guide is this one. To understand further details read this, this and finally this monster. In my case there were no problems with CRL as I have followed this, this and this guides beforehand. Result:
SSL Debug
To debug SSL use
openssl
tool from client side (given myserver uses common 443 port for https):In my case
acceptable CAs for client certificates
was initially empty so I have setHKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\SendTrustedIssuerList
DWORD value to 1 and rebooted server as per this guide (near bottom). Note that the empty list of acceptable CAs for client certificates IIS problem may also occur due to huge list of CA as per this SF question.When you have a list of
acceptable CAs for client certificates
in theopenssl
output you can compare it with contents of configured certificate stores at your server. In my case there were only one certificate there belonging toTrusted Root Certification Authorities
. So after a few days of investigation that lead me to this ms-support article #2802568. Effectively one improperly placed non-self-signed certificate doomed entire auth chain.Epilogue
Quoting @tylerl