Looks like I've found myself in a bit of trouble here...
Background information
I am trying to clean up a domain that had 5 domain controllers, bringing it down to a two-DC environment. It is a very small environment with less than 20 machines. We ended up with 5 domain controllers simply as a result of unfinished work that we're finally getting around to in an effort to raise the domain functional level from 2012 to 2019.
The setup is as follows:
bmdc9
is Windows Server 2016 the PDC with all of the FSMO roles.bmdc8
is Windows Server 2016.bmdc10
is Windows Server 2016 Core.
The goal is to create a new domain controller bmdc13
and bmdc14
which will both be Windows Server 2019. Then, to raise the domain functional level to 2019.
Current issues:
#1: Whenever we run the DC demote wizard on bmdc8
we're getting an error saying:
No other domain controller could be contacted, but other domain controller objects are in the directory. If you are certain that this is the last domain controller for the domain and want to proceed, confirm that this is the last domain controller in the domain.
The problem is, this is NOT the last DC. VMs bmdc9
and bmdc10
are still up and active.
#2: Whenever we shut down (Answered this myself, see Nov 5th comment below)bmdc10
we completely lose all DNS for external addresses. Meaning, if we try to ping www.google.com we get an error message saying:
Ping request could not find host www.google.com. Please check the name and try again.
#3: When I run repadmin /replsum
I am getting the following output:
What really strange here is that bmdc6
no longer exists. It was demoted earlier today (without errors) and removed from the domain gracefully.
#4: Whenever I go into Active Directory Sites and Services I am seeing more DCs than I should do:
For some reference:
bmdc4
was demoted earlier today.bmdc6
was demoted earlier today.bmdc8
up and running, but I would like to demote.bmdc9
up and running. The current PDC.bmdc10
up and running, but I would like to demote.
Other information:
In case it is helpful, below is some additional information:
Question(s)
Basically, I'm at a loss as to how unhealthy my environment is at the moment.
- Why am I getting the replication errors? And, how do I clean that up?
- Why are the old demoted DCs still showing up under Active Directory Sites and Domains? (i.e.
bmdc4
andbmdc6
). How do I remove them? - Why is DNS not working unless
bmdc10
is up even though all NICs in our network havebmdc9
as the primary DNS server andbmdc10
as the secondary? - Is it safe to proceed with demoting
bmdc8
even with the aforementioned 'last domain controller' warning? - What's the quickest non-intrusive way to fix all of this?
My ultimate goal is to migrate over to Server 2019 and raise the domain functional level to 2019. I don't care if that involves creating more DCs and retiring everything I have now, just as long as I don't lose the objects/users/computers/passwords in my current domain.
If anyone can help me out, that would be amazing. Thanks in advance.
Update(s)
2021-11-05 @ 14:06: This is what I get when I run repadmin /showreps
Default-First-Site-Name\BMDC9
DSA Options: IS_GC
Site Options: (none)
DSA object GUID: **REMOVED**
DSA invocationID: **REMOVED**
==== INBOUND NEIGHBORS ======================================
DC=xxxx,DC=local
Default-First-Site-Name\BMDC8 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:37 failed, result 8456 (0x2108):
The source server is currently rejecting replication requests.
5700 consecutive failure(s).
Last success @ 2021-06-15 11:19:34.
Default-First-Site-Name\BMDC10 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:56:14 was successful.
CN=Configuration,DC=xxxx,DC=local
Default-First-Site-Name\BMDC8 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:37 failed, result 8456 (0x2108):
The source server is currently rejecting replication requests.
3444 consecutive failure(s).
Last success @ 2021-06-15 10:51:35.
Default-First-Site-Name\BMDC10 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:37 was successful.
CN=Schema,CN=Configuration,DC=xxxx,DC=local
Default-First-Site-Name\BMDC8 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:38 failed, result 8456 (0x2108):
The source server is currently rejecting replication requests.
3427 consecutive failure(s).
Last success @ 2021-06-15 10:51:35.
Default-First-Site-Name\BMDC10 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:38 was successful.
DC=DomainDnsZones,DC=xxxx,DC=local
Default-First-Site-Name\BMDC8 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:38 failed, result 8456 (0x2108):
The source server is currently rejecting replication requests.
3465 consecutive failure(s).
Last success @ 2021-06-16 17:20:40.
Default-First-Site-Name\BMDC10 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:38 was successful.
DC=ForestDnsZones,DC=xxxx,DC=local
Default-First-Site-Name\BMDC8 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:38 failed, result 8456 (0x2108):
The source server is currently rejecting replication requests.
3431 consecutive failure(s).
Last success @ 2021-06-15 10:51:35.
Default-First-Site-Name\BMDC10 via RPC
DSA object GUID: **REMOVED**
Last attempt @ 2021-11-05 13:49:38 was successful.
Source: Default-First-Site-Name\BMDC8
******* 5700 CONSECUTIVE FAILURES since 2021-06-16 17:20:40
Last error: 8456 (0x2108):
The source server is currently rejecting replication requests.
OK, firstly I want to attribute some credit to @GregAskew as some of his responses helped me troubleshoot the issue.
Should anyone else run into this problem, below is how I resolve it:
Per Greg's suggestion, running
repadmin /showreps
highlighted that the problem wasbmdc8
. I was able to fix this by simply removing that machine from the network. Unfortunately, demoting the domain controller did not work . (i.e. the wizard errored out). As such, I had to follow the Microsoft documentation to manually remove a domain controller from the domain. Once that was done, everything worked just fine from a replication standpoint.Manually deleted the old decommissioned DCs from Active Directory Sites and Services.
DNS was a mess. Each DC had another DC as
Forwarders
. What I did was cleared out all Forwarders, then added a public DNS provider as our forwarder. Once I did that, the internet would work fine even whenbmdc10
was down.Demoting the DC would not work. And so, I had to manually remove
bmdc8
from the domain.None of this involved downtime. Nor did I lose any domain objects. Everything has been fine and could have been done during business hours.