I used this sequence of netdom commands to rename one of our AD controllers from "A" to "B". The rationale was that the historical name "A" would be much better suited for a new server that we want to install.
This renaming "A" to "B" went smoothly and in particular, the AD functionality is up and running.
However, we managed to install the new server only under a new name "C". It can also readily be renamed to "D" or "E" or ... , but not to "A". I thought it might be the old DNS name still being around. Instead of waiting weeks until it would age out automatically, I deleted it from DNS, but ths did not help.
What I get when attempting to rename "C" to "A" is an "internal error", without any notable messages in the event log. What could cause this?
Somewhere there must still be a record on the A->B server containing a record of A. You removed the entry from DNS which is good. I would also check Active Directory (make sure there isn't a disabled account for that old 'A' name) as well as the registry - this link shows how to check the registry for AD objects, though a global search wouldn't be bad either!
Check the servicePrincipalName attribute on the A->B server. It should only have records for the new name if everything went smoothly. But it may still have lingering entries from the old name particularly if you have a large environment and replication takes a while.
I had something close to this happen when I had two servers.
Server A was a 2012R2 new server that was replacing a DC (Server B 2008 Server).
-I renamed server B renamed server to Server C
-Then renamed server A to server B but it wouldn't take.
-I had to actually delete DNS records that pointed to Server B and actually had to delete Server C (old server B) out of Active Directory.
Try that if you can.
Delete your old server out of the domain if possible then try the rename.