RobbieCrash's questions

tl;dr: TLS 1.2 between Server 2012 R2 and Chromium based browsers fails when using AD CS issued certs. Works fine on Server 2016+, and on 2012 R2 with Firefox/IE/Cygwin-curl.

We have several internal Server 2012 R2 web servers which we are trying to move off of publicly issued certificates, onto ones issued by our AD integrated CA, and eliminate less secure crypto settings, including CBC MAC. Server 2012 R2 does not support ECDHE_RSA with GCM, which means that we're trying to use an ECDH based cert. We have experienced a similar issue, however, when we allow cipher suites with CBC-MAC, such as TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 with an RSA cert issued from the same CA. Using our public wildcard cert issued by GlobalSign, we're able to connect with all browsers.

The enterprise CA, and the offline root CA are both trusted, and we've verified that this is working properly. Certificates using multiple different templates issued to 2016 and 2019 servers work without issue across all browsers. The ECDH and RSA based templates work equally well on 2016+.

ECDH certificate template crypto:

RSA certificate template crypto:

Both RSA and ECDH certificates on the 2012 R2 servers are accepted by Firefox (once it's been told to trust them both via policy, and manually setting security.enterprise_roots.enabled), pre-Chromium Edge, IE, and Cygwin's curl and wget. I've confirmed we're using modern ciphers in the registry, re-setting them with IISCrypto, and verified there are compatible available ciphers being offered by the server with OpenSSL, and nmap. Likewise, I've confirmed that clients are actually able to connect using those ciphers.

Firefox shows it's connecting with TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, which according to Qualys is supported in the version of Chrome which we're using.

With the ECDH

PORT    STATE SERVICE
443/tcp open  https
| ciphers:
|   TLSv1.1:
|     ciphers:
|       TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 (secp256r1) - A
|       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
|       TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: A

Each time we try to connect with Chrome a pair of event 36874/36888 are logged stating there were no supported cipher suites on the client.

A list of the cipher suites we experience the issue with when using an Enterprise CA issued cert, most of which were enabled just for testing (warnings snipped):

PORT    STATE SERVICE
443/tcp open  https
| ciphers:
|   SSLv3:
|     ciphers:
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_MD5 (rsa 2048) - C
|     compressors:
|       NULL
|     cipher preference: server
|     warnings:
|   TLSv1.0:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 1024) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 1024) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_MD5 (rsa 2048) - C
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.1:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 1024) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 1024) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_MD5 (rsa 2048) - C
|     compressors:
|       NULL
|     cipher preference: server
|   TLSv1.2:
|     ciphers:
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA (secp256r1) - A
|       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA (secp256r1) - A
|       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 (dh 1024) - A
|       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 1024) - A
|       TLS_DHE_RSA_WITH_AES_256_CBC_SHA (dh 1024) - A
|       TLS_DHE_RSA_WITH_AES_128_CBC_SHA (dh 1024) - A
|       TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA256 (rsa 2048) - A
|       TLS_RSA_WITH_AES_256_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_AES_128_CBC_SHA (rsa 2048) - A
|       TLS_RSA_WITH_3DES_EDE_CBC_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_SHA (rsa 2048) - C
|       TLS_RSA_WITH_RC4_128_MD5 (rsa 2048) - C
|     compressors:
|       NULL
|     cipher preference: server
|_  least strength: C

My questions are:

1. Why do Chromium based browsers fail to negotiate a cipher suite when there are compatible cipher suites offered by the server?
2. What do I need to set in order to allow Server 2012 R2 to use internally issued certs?
3. Is there a better way to handle what I'm trying to do, aside from upgrading the application servers to 2016/2019? At this point, it seems like disabling TLS completely and putting everything behind a LB/reverse proxy may be a better idea, even if these are single-server solutions.

EDIT: I've opened a bug with Chromium and they have confirmed that the cipher suites being offered by the server should be accepted by Chrome. They are now investigating after I've provided network logging captured via CHROME://Net-Export. This may be related to an old Chrome/2012 bug. Once Google has reported back about what the issue is, I'll update this post again. However, at this point it looks as though there's nothing wrong with the configuration on our side.

Thanks for looking!

In a Hybrid Exchange 2016/EOL environment we have our webserver configured to send mail directly out via our on-premise Exchange server. We had an issue, now resolved, that was blocking the server from sending some, but not all messages. The From address on these messages was set to [email protected], which is a real email address, so while we don't have a copy of the sent message, we do have NDRs for the messages that failed. We can't just blanket re-send everything as there are an unknown number of emails that were successfully sent.

All told, there are around 12,000 messages that need to be identified and referenced to the orders they are in regards to. There are ~15 different subjects which we need to sort through.

If possible, I'd rather not have to go through Outlook to do isolate the failures, but NDRs don't seem to show up when I try to run an eDiscovery or Content search on the mailbox.

Once I have the messages exported, I can go through them relatively easily and automate the correlation process, but I can't figure out how to isolate, or export just the failure messages, without having to drop into Outlook and do it manually. The manual process on a subset of these messages has yielded different results for each person that's done it, and is incredibly tedious, so I'd like to avoid it.

Is there a PowerShell method to isolate all the NDRs, or something that I'm missing from within the 365 portal?

Is there a way that this can be done so that I can export the NDRs including the subject line to different folders for easier matching with the orders they represent?

I've got a Server 2016 Standard guest running on a Server 2016 Datacenter Hyper-V 4 node cluster which cannot start the Hyper-V NIC due to the following error:

"Windows cannot verify the digital signature for the drivers required for this device. A recent hardware or software change might have installed a file that is signed incorrectly or damaged, or that might be malicious software from an unknown source. (Code 52)"

I'm facing all the same problems and have tried all the same solutions, that were tried in THIS thread, all to no avail. This includes:

• Removing recently installed updates
• Removing and re-adding the vNIC at the host level
• Removing all phantom NICs from the guest
• Attempting to update the driver in the guest, but it's the most recent one
• Attempting to update the driver for the host NIC, also the most recent one
• Restoring from backup, but the problem comes back after installing any updates released after October 2017
• Screaming into the void

I can get it working by disabling driver signature verification, but I really don't want to do that as this is a security related server. I have multiple other VMs running extremely similar configurations, at the guest and OS level, in the cluster, all of which are fully patched, none of which have any issues.

I really don't want to have to reinstall and reconfigure, especially if this is caused by some particular configuration on the VM which will make this occur again. Reinstalling means I need to reconfigure not only the server, but a bunch of the clients that depend on it as its certificate will change.

There's nothing that seems pertinent in the eventlogs.

We are looking to add custom attributes to the User class in our AD schema and I cannot find anything regarding a length limit for the actual attribute name, only general limits to their values. Are there specific concerns with adding longer attribute names to classes in AD?

For example, is adding the attribute Organization_DateTime_NewOfficeMoveDate ok, or should it just be Org_OfficeMoveDate? Do the limits apply universally across domain/forest functional levels?

I have a remote DC that has been unable to completely replicate AD for at least 90 days so has been tombstoned, had their site-to-site link severed, and has been purged from the organization's main AD infrastructure. This is fine, we do not wish to fix this or bring their now-local AD back into the fold; the office will move forward as its own separate AD office. We were in the middle of preparing for an AD migration when their link to the rest of AD failed, however no changes were made yet.

Currently, none of the domain admin credentials that we have work on that DC. I have 5 different domain admins who swear they know their password, and know what their password was the last time AD replicated successfully. None of us are able to log in, nor can we use the master Enterprise Admin account, which is a password that has definitely not changed between when everything was working correctly and now.

It is not possible that all our domain admins do not remember their password. The passwords do not have an expiry on them, and it is not that we're being forced to change our password, the local DC just thinks they're wrong. The local admin says he hasn't done anything to their DC while we were working on preparing for the AD migration, and his account only had administrative rights over their users and their OU so he shouldn't have been able to change any of our passwords or anything similar.

So how could this have happened? Does booting into DSRM on a DC do something to all domain admin accounts if you bung it up? AKA, did our local guy try to take matters into his own hands and break something? Is it possible that the server was compromised in some way that would cause this?

The only other server on-site is a domain joined file server which local administrator credentials still work to access, so it isn't a huge problem.

The existing DC is Server 2008 R2 as is the existing file server, and all computers in the office are on fully updated Windows 7 machines.

Further relevant details: We have one global domain in our forest that has remote domain controllers in multiple offices, each office is an AD site. One of those offices is separating from our global infrastructure. Before we could properly migrate them out of our domain, their ISP collapsed and their internet connection was lost. Since that site is not coming back to our control, it was removed from the domain. To assist with them migrating to their own domain we were going to run ADMT, but since their local DC cannot be authenticated against with admin rights, we cannot run ADMT.

User profiles are workstation local with a home drive stored on the file server which we have administrator access to, so manually moving data is an option.

I have several thoughts/questions about how to proceed:

1. I can restore their site and DC and go through the whole un-tombstoning process, but this is a significant amount of work just to get them to a point where we can remove them from AD again.
2. We can scrap their entire local AD and get them to deploy a new one, and migrate all the computers to a new environment, but this is also a significant amount of work.
3. If we decide to go with option 2 our biggest concern will be getting user profiles migrated from the current AD to the new one. Is there a tool that can assist with this given that administrative access to the current domain is non-existent? Or is this basically going to just be, create the new AD, join the PC, initialize the new user profile and copy data from the old profile to the new one and reconfigure accounts?
4. Is there a better way of doing this that I'm not thinking of? All the tools I know of to assist with something like this require administrative access to the source AD.

We're setting up a bunch of new meeting rooms as a result of an office move and I'm unsure of how, or if, I can configure something the way it needs to be. One of the meeting rooms is split into three different rooms which can be expanded as needed. We need to be able to book each of the three rooms individually, but also need to be able to book either the left and centre, right and centre, or all three.

Setting up the six different rooms in Exchange is no problem, (1,2,3,1+2,2+3,1+2+3), but is there a way for me to prevent people from booking room 1+2+3 when room 2 is already booked? Or do I just need to tell users that they need to book each room?

We are running a hybrid Exchange Online solution, with an on-prem Exchange 2013 server running the latest CU package. All rooms are setup to automatically accept, and there are no delegates for any of the room scheduling.

Our setup is as follows: Site0: Exch00 - Original On-Prem Exchange server. CAS, MBX, and all transport roles. Was not configured to perform any hybrid/management work with 365. DC0 - Site local DC to site where Exch00 is

Site1: Exch01 - Hybrid server, all 365 management etc. MBX, CAS, contains resource mailboxes and mailboxes to be exported to PST or migrated to 365. DC1 - Site local DC for Exch01 Client computers

Sites2-5: client computers, local DCs.

External sites: lots of offices that are not AD integrated and are using Outlook Anywhere.

365 - Setup with dirsync, not ADFS

Site 0 has failed, and does not need to be recovered. All user and resource mailboxes and other user data that was needed has already been migrated from those servers, and the site was set to be decommissioned properly at the end of the month. However, a hardware failure has taken care of that for us. We can restore from backup if needed, but since we're just going to be restoring to decommission I'd rather avoid that if possible.

We are currently having issues with Equipment calendars (but not Rooms) not being available. They are all stored in the same database, on the same server. All other calendars are functioning properly, it's just Equipment mailboxes which aren't working.

In Site1 Outlook is prompting for a password constantly but is functioning regardless of if the password is entered or not. Users in the external sites are not experiencing this issue.

Prior to the hardware failure last night, everything was working properly even though all Exchange services were disabled on Exch00. The DC has already been removed from AD Topology via ADSIedit, and there are no traces of it in our KCC. The site still exists in ADS&S I have the following questions:

1. Can I remove all traces of Exch00 via ADSIedit, as I would any other failed Exchange server, or are there additional concerns because of the 365 tie in?

2. Is there a separate arbitration mailbox that's used for Equipment mailboxes than there is for Rooms? If so, is there a proper process for recreating this arbitration mailbox? If not, is it likely that the problems with the Equipment mailboxes will be resolved once Exchange has updated topology information?

3. What could be causing the Outlook password prompting? The DC in Site1 is fully functional as is Exch01. Sites2-5 are also having the same password prompting issue.

We have 6 different AD sites all connected by IPSec tunnels in a full mesh aside from one office which cannot connect to our data centre for stupid ISP related reasons that are not fixable currently. We have AD integrated DNS and there are two web servers, with public IPs, which replicate the internal address across our AD sites and are accessed via the IPSec tunnels. Since both of these servers are in our DC the site which cannot connect to the DC is unable to access both of these sites.

We need to replicate the rest of our AD DNS to this site, so removing replication is not an option. These sites are also in our primary domain's DNS, and updating them to either use an alternate name, or alternate domain suffix is not a viable option as it will break links for this office.

Is there a way to have this one site use alternate DNS records and not replicate those across AD?

Is my best option to GPO all client/server computers in that site to add the public IP for these two hosts to the local hosts file?

This doesn't need to be a permanent, elegant fix as we will be changing ISPs soon(ish). I just need something that will be stable and reliable for this office for the mid-term future.

To clarify: Toronto and New York can talk to Tokyo, each other and the data centre. Tokyo can talk to Toronto and New York, but not the data centre. We have resources in each site, as well as AD replication, that need to be accessed from all the other sites.

Two of these resources are in our data centre, and have public IPs as well as internal IPs, and use the same hostname internally and externally. That hostname is integrated into our Active Directory DNS. All access from New York and Toronto to these sites is done via the internal IP addresses.

We need Tokyo to access these two hosts via their public IP, not their internal one. We cannot use different hostnames as these sites are heavily integrated into our other applications, communications, etc.

We're preparing to rid ourselves of Exchange on-premise and are migrating from an outsourced hosting provider to go to Office 365. We're running a fully patched version of Exchange 2013, atop Server 2012, also fully patched.

During the migration of test mailboxes we found that many mailboxes across multiple databases are corrupted. More information about the cause can be found HERE. Essentially: the SAN which stores our Exchange VM is oversubscribed and routinely has I/O wait in excess of 5 seconds, and sustained read speeds rarely pass 500KBps.

The slow speeds would be enough to cause significant wasted time during the migration, but when mailboxes with corruption are encountered, migrating 1GB of data goes from a 2-3 hour affair to a 10-20 hour one. Each of the mailboxes that have issues (that I've found so far) give messages similar to the below when checked against get-mailboxstatistics:

WARNING: The object <GUID> has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Cannot extract the property value of 'DeletedItemCount'. Source: PropTag(DeletedMsgCount), PropType(Int), RawValue(-2), RawValueType(System.Int32). Target:
Type(System.Nullable`1[System.UInt32]), IsMultiValued(False). Error Details: <n/a>

Running New-MailboxRepairRequest against all databases identified some corruption and repaired it, but not all of it. I can't seem to find a way to get Get-MailboxStatistics to log the fact that there is something broken in each of these mailboxes, though I'm sure there is one. Moving mailboxes from one database to another seems to fix the problem. We've got ~50 DBs, and about 50 users per DB, so going through this manually is out.

What I want to do is, via PowerShell (excuse lazy pseudo code please):

foreach ($mailbox in $database){
    if get-mailboxstatisics -eq $corrupted {
        move $mailbox to $otherdb
        wait
        move $mailbox back to $database}
    }

However, I can't figure out how to catch the "Warning: this is broken" text from Get-MailboxStatistics, and the resultant object returned doesn't have anything in it that shows it's broken.

Do I just need to catch the warning and assume everything that complains about inconsistencies can be fixed this way, then go back and check the list of mailboxes that actually have an issue after they've been moved out and back to see if things are still broken? Is there a better way to do what I need to do?

Replacing the SAN is out of the realm of possibilities, so is fixing any other underlying causes.

Over the few weeks our Exchange server has had problems with its backup jobs failing to complete which has caused our normally ~empty log drives to fill up to the point where Exchange has dismounted databases and logged various errors regarding replaying log files. Lamentably, nobody in the backup team did their job properly, so over the weekend we had a failure situation which dismounted ~40 databases due to there being ~100GB of logs, which generally sit at ~3GB. This caused everyone working the weekend to not look at the history of the issue, and rather than reach out to contact anyone else, enable circular logging after everyone on the team was instructed not to, remount all the databases and call it a day.

We've not heard of any data loss from any users yet, but the fact that all databases encountered this, and that there are complaints about logging failures, replay failures, and unexpected dismounts I'm concerned that there may be some.

Aside from firing the backup team, the weekend monitors and the admin that decided to enable circular logging after a significant amount of time between backups without saving the logs anywhere in case of a need to restore from backup and get whatever we could back, what is my best course of action to determine if we lost anything?

Are there particular events which may be buried in the 3,000,000 log entries that span the six hour long section while this was going on? Is performing an integrity check recommended? Defrag, on or offline?

On the Exchange server the following is typically what happened, I've stripped event source and ID because everything seems to be generic and of little assistance in determining if things actually went super south, or just ruined my Monday:

1. At 'TIME' the Microsoft Exchange Information Store Database 'DATABASE' copy on this server experienced a serious error which caused it to terminate its functional activity. The error returned by the remount attempt was "There is only one copy of this mailbox database (DATABASE). Automatic recovery is not available.". Consult the event log on the server for other storage and "ExchangeStoreDb" events for more specific information about the failures.

2. Information Store - DATABASE (9564) DATABASE: An attempt to write to the file "F:\Logs\DATABASE\E0Etmp.log" at offset 1048576 (0x0000000000100000) for 0 (0x00000000) bytes failed after 0.000 seconds with system error 112 (0x00000070): "There is not enough space on the disk. ". The write operation will fail with error -1808 (0xfffff8f0). If this error persists then the file may be damaged and may need to be restored from a previous backup.

3. Information Store - DATABASE (9564) DATABASE: Unable to create a new logfile because the database cannot write to the log drive. The drive may be read-only, out of disk space, misconfigured, or corrupted. Error -529.

4. Information Store - DATABASE (9564) DATABASE: The logfile sequence in "F:\Logs\DATABASE\" has been halted due to a fatal error. No further updates are possible for the databases that use this logfile sequence. Please correct the problem and restart or restore from backup.

5. Information Store - DATABASE (9564) DATABASE: Database recovery/restore failed with unexpected error -510.

6. The Microsoft Exchange Mailbox Replication service was unable to process jobs in a mailbox database. Database: DATABASE Error: MapiExceptionMdbOffline: Unable to open message store. (hr=0x80004005, ec=1142) Diagnostic context:

7. At 'TIME', the copy of database 'DATABASE' on this server encountered an error during the mount operation. For more information, consult the Event log on the server for "ExchangeStoreDb" or "MSExchangeRepl" events. The mount operation will be tried again automatically.

This is a standalone server, so the only one copy errors seem to be expected. There are also numerous client access errors logged during the time that this was happening, which I've omitted.