jcrossbeam's questions

I have an Active Directory with 6 DCs, where the newly added DC02 is not replicating. It's a mixed environment from Windows 2003 It is also the owner of PDC, RID and Infrastructure roles. Now I'm wondering what the best way to move forward is.

The new DC is replacing a demoted DC with a different name, and initially everything looked okay with the new one after it was added. I let it sit for about a week after promoting it, to let it finish all replication before moving the FSMO-roles over. It didn't give me any fuss about transferring the roles over, so I assumed it was all working as intended. After a while we started noticing that new DNS-records added to DC02 wasn't replicated to the other DCs.

From what I can tell, the situation now is as follows:

• DC02 does not have SYSVOL and NETLOGON shared
• The other DCs are still seemingly replicating amongst eachother, but none have a connection to DC02. DC02receives updates from the other DCs
• Other DCs report an ERROR for the role holder of PDC, RID, and Infrastructure

From my reseach, I'm thinking I have two options, and wondering which would be the best to use:

1. A D2/D4 Burflags authoritative restore, setting D4 on a working DC and D2 on the DC02 (and all the other DCs?). I've not done this before, and feel tentative as I'm not really sure what it will do.
2. Taking the faulty DC02 offline permanently and Seize the FSMO-roles back to the original DC. Not really sure what this would do either.

I'd appreciate any tips on how to move forward, and which precautions might be necessary.

Update 1:

The old FSMO-owner DC01 gives this when running a dcdiag Is it safe to assume these errors are caused by the faulty DC, and should still be okay to tranfer back the roles to this server?

dcdiag /q:

Dcdiag could not locate (null) in the dcdiag's cache of servers. Try running this dcdiag test against this server, to avoid any problems caused by replication latency. ......................... DC01 failed test RidManager An Warning Event occured. EventID: 0x8000072D Time Generated: 09/14/2016 14:11:57 (Event String could not be retrieved) ......................... DC01 failed test kccevent An Error Event occured. EventID: 0x40000004 Time Generated: 09/14/2016 13:32:33 Event String: The kerberos client received a An Error Event occured. EventID: 0xC000001B Time Generated: 09/14/2016 14:01:07 Event String: While processing a TGS request for the target An Error Event occured. EventID: 0x40000004 Time Generated: 09/14/2016 14:01:15 Event String: The kerberos client received a ......................... DC01 failed test systemlog Warning: DcGetDcName(PDC_REQUIRED) call failed, error 1355 A Primary Domain Controller could not be located. The server holding the PDC role is down. ......................... domain.com failed test FsmoCheck

Netdom query fsmo:

Schema owner DC01.domain.com Domain role owner DC01.domain.com The parameter is incorrect. Try "NETDOM HELP" for more information

Setup looks as follows:

• HP Proliant DL380 G7
  
• 6 x 3TB Sata drives (surveillance level) configured with hardware RAID 1+0 with the SATA controller on board. Model is Seagate SV35
  
• 192GB RAM

VMware ESXi 6.0

• One VM guest running Centos 6.7 (Kernel 2.6.32-573)

Datastore is made up of all the remaining disk space after the ESXi-installation (little less than 8tb)

• 1 VMDK file for the system partition at 100GB
  
• 1 VMDK file for the data partition at around 7.7TB

On the guest CentOS, the system partition is LVM ext4
The data partition is a LVM with a single PV, LV and VG ext4

Now the problem I have is that data transfer speeds on the disk is extremely slow. Trying to copy a semi-large file (10-30 GB) from one place on the LVM to another on the LVM starts out with a transfer rate of around 240MB/s which is the speed I'd expect it to have, but just after a few seconds (30ish usually) it drops down to 1-4 MB/s, and viewing iotop tells me a process starts running called flush-253:2 which seem to slow everything down.

I've been using rsync --progress to get a better picture of the transfer speeds in real time, but I'm seeing the same result with a cp operation.

When it finally finishes, I have tried performing the same procedure again, with the same file to the same location. The second time the indicated transfer speed of rsync keeps steady at around 240MB/s throughout the whole transfer, but when rsync indicated the file transfer is complete it hangs at that state for about as long as it took to complete the first copy procedure. I can see the flush-253:2 process working just as hard for both procedures.

Now I know the setup isn't optimal, and I would have preferred to have a separate disk for the ESXi system, but I don't feel like that should be the cause of this extreme slow transfer rates.

I've searched for information regarding the flush-process, and as far as I can tell, it basically writes data from memory on to the actual disks, but I haven't found anyone saying they've experienced this level of slow transfer rates. The system is not in production yet, and CPU is hardly even running at all, and it has around 100GB of free memory to use when the copy procedures run.

Does anyone have any idea on what to try? I've seen similar results on a different system which is basically setup the same way, except on completely different (somewhat lesser) hardware. I have also a third system running CentOS 5 and ext3 on LVM, which does not have any issues like this.

EDIT 1: I realize now had remembered incorrectly, and the system partition is also lvm, but still a separate volume from the data partition

[root@server /]# mount
/dev/mapper/vg1-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw,rootcontext="system_u:object_r:tmpfs_t:s0")
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/vg1-lv_home on /home type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/mapper/vg_8tb-lv_8tb on /datavolume type ext4 (rw,nobarrier)


[root@server /]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_1-lv_root<br>
                       50G  9.7G   37G  21% /
tmpfs                  91G     0   91G   0% /dev/shm
/dev/sda1             477M   52M  400M  12% /boot
/dev/mapper/vg_1-lv_home
                       45G   52M   43G   1% /home
/dev/mapper/vg_8tb-lv_8tb
                      7.9T  439G  7.1T   6% /datavolume

Update 1: I have tried increasing the dirty_ratio all the way up to 90, and still saw no improvements. I also tried mounting it with -o nobarriers, and still the same result

Update 2: Sorry to everyone who are trying to help me about the confusion, now that I've had a look myself, the hardware is actually a HP Proliant 380 G7, I don't know if that makes any difference.

I have also had a look myself at the raid configuration, and it seems we're using a P410 raid controller, and when I'm booting into the raid management, it says

HP Smart array (I think) P410 "SOMETHING", with 0MB in parenthesis

I'm guessing that might mean we have 0MB in write cache?

I'm a little out of my depth here when it comes to hardware, can you add a write cache module(?) to this raid controller if one doesn't already exist? Or do you need a new controller/move to a SAN? How can I tell if it has a write cache, but perhaps the battery is dead?

Update 3: Thanks to your suggestions and some futher research, I'm now going to try and install the HP smart array driver vib file in the ESXi, and hopefully get a clearer picture of what I have. I also found the option in the system BIOS to enable drive cache, so I might have a last resort in case it turns out we don't have write cache on the controller.

Update 4 (solved): Thanks to all who suggested solutions, and yes it turned out there was no cache module present on the disk controller.

To anyone having similar problems, I installed the hpssacli utility VIB for ESXi, and could with the following output confirm what had been suggested in the replies.

Cache Board Present: False

Smart Array P410i in Slot 0 (Embedded)
Bus Interface: PCI
Slot: 0
Serial Number: 
Controller Status: OK
Hardware Revision: C
Firmware Version: 6.62
Rebuild Priority: Medium
Surface Scan Delay: 15 secs
Surface Scan Mode: Idle
Parallel Surface Scan Supported: No
Wait for Cache Room: Disabled
Surface Analysis Inconsistency Notification: Disabled
Post Prompt Timeout: 0 secs
Cache Board Present: False
Drive Write Cache: Disabled
Total Cache Size: 0 MB
SATA NCQ Supported: True
Number of Ports: 2 Internal only
Driver Name: HP HPSA
Driver Version: 5.5.0
PCI Address (Domain:Bus:Device.Function): 0000:05:00.0
Host Serial Number: 
Sanitize Erase Supported: False
Primary Boot Volume: logicaldrive 1
Secondary Boot Volume: None

I have just tried to replace one of my domain controllers in a single domain forest keeping the old IP (But a new computername). As this is an environment I have "inherited" from a previous sysadmin I don't have all the background facts for how it was setup. I think this might have been the first DC in the AD, but it's an old Windows 2003 which has been acting up in all manners of ways lately.

Before the change I had:

• 2 x Windows 2003 DC
• 2 x Windows 2008 R2 DC
• All but the remaining Win 2003 has GC

After the change I have a feeling the new DC's DNS service isn't working as it should. If I have that set as my primary DNS, I get a strange latency in my dns-queries for public sites. Queries for internal records seem to be working. I tried setting the timeout for forwarders down to 1 sec on the new DC, which got me a shorter latency for the public queries, which has me thinking all public site queries time-out on that server, and forwards it.

Before I demoted the old machine out of the AD, I made sure all FSMO-roles were on a different DC, and as stated before, there are several GCs.

Is there some DNS setting for the domain that I've missed, or do you have any ideas on where to look next?

Is the new computername using the same IP the culprit? Would it be wise to demote the new DC, leave the AD, change names, then join again?

In the server manager DNS-logs, I get a warning each restart saying:

The DNS server is waiting for Active Directory Domain Services (AD DS) to signal that the initial synchronization of the directory has been completed

But right after that, there's an information entry saying:

"The DNS server has started.

Running NSLOOKUP, and choosing server "new DC" works fine, and is instant, for internal, but times out for public queries. Running PING from the new DC to a public site resolves the public IP fine.

The firewall is turned off on the server, and the DNS Server-service is running.