Emmaly's questions

A single user on Windows 7 on a Windows Domain can not access/use/manipulate network printers shared from a server running Windows 2008 R2. The error that occurs is "0x00000005 Access is denied". Any other user I create in the system can access and print to said printers, even with identical group memberships. I expect that if I wipe out the user's local profile (it's not a roaming profile), it will remove the problem.

Is there anything specific that can be checked in the user's profile or Registry HKCU to discover the cause of the "access is denied" error?

Details:

• All printers shared from the Windows Server are failing with "0x00000005 Access is denied" when the user attempts to view the printer properties, view the print queue, or print to the printer.
• The problem is specific to the local user profile. This user on other computers does not fail and other users on this computer also do not fail.
• Killing the local user profile and starting it from scratch is completely off the table.
• Testing the Effective Permissions for this user shows that the user does have the "Print" and "Read Permissions" privileges on the affected printers.

Edit: This user's computer ended up with a virus afterward and so we wiped and reinstalled the system. The problem didn't occur after the wipe.

Is there a way to monitor the DFSR backlog in a manner more efficient than using dfsrdiag.exe backlog?

I wrote a program that just slurps in the backlog count via dfsrdiag.exe backlog /smem:alpha /rmem:beta /rgname:domain\namespace\foldername /rfname:foldername with five minute intervals. Each time it runs, it takes quite a while (between 2 to 5 minutes) to get the resulting value. That means that in the end, it runs for a few minutes to collect the info and then delays for five minutes. It feels like it is probably expensive in some fashion in order to get this info. It also returns the top 100 files in the backlog. I really only want the backlog count alone and don't care about the files themselves. This is being used to create historical graphs.

Info for these DFSR peers: Windows 2008 R2 on four servers, three distant offices connected via 50-100Mb Internet connections, 30 replication groups, several replication groups are very large in file total size (1-2TB each) though most are small (500MB-10GB).

How do Windows DFS referrals work for laptops away from the office?

[Edit 2014-11-26] I already understand how sites are defined by subnets, etc. But what I want to know is what happens to users that are not fitting within any sites at all? Home users via VPN, for example. The VPN doesn't supply an IP that is within any of the sites either due to how the VPN system works, so that's not going to help define a site. Does it treat all DFS namespace targets as exactly equal since the client is in zero sites at the moment? Is there any way to encourage DFSN to treat the DFSN targets in one site as preferred based on proximity or the user's usual site? (I know how to say "treat X server as preferred when all is equal" but that's not what I'm asking for here.)

The rest of this text is just background for your interest or entertainment.

I was under the impression that the laptop would retain whatever last site it was at or the site of the last DC it connected to. And then that site would determine which site that laptop is presently at for the purposes of referral ranking.

The reason I ask is because we're getting some laptop users getting some strange referrals to an office that they've never actually been to. And those laptops are having a great connection to the servers in the other sites that they should be connecting to. The strange referral target in this case is a server that isn't known for being reliable. If it were basing it on some speed or reliability metric, it's got it completely wrong. The question is just that first line up there, the rest of this text is just background.

The hardware/software involved is: Windows 2008 R2 servers for all things; Windows 7 64-bit and Windows XP 32-bit laptops.

And this is a question about DFS Namespaces, not about DFS Replication.

How DFS Works: DFS Processes and Interactions states the following:

When least expensive target selection is enabled, DFS places targets in the referral in the following order:

Targets in the same site as the client are listed in random order at the top of the referral.

Targets outside of the client’s site are listed in order of lowest cost to highest cost. Referrals with the same cost are grouped together and within each group the targets are listed in random order.

If the laptop is at a client office, hotel, home, etc, is it still considered to be within a site? If so, I assume that site would be used in the first group of targets above. If it isn't within a site, I'd assume it skips that first group altogether and just randomly assigns the targets. But the behavior we're seeing is that it is more like the latter than the former.

Edit: (5/21/2012 3am PT)
It seems that the folders with "Override referral ordering" set to "First among targets of equal cost" get goofy. The goofy part is that the target with that enable seems to be the last in order, not first. I'm unsetting these to see what happens. Will report later.

Is there a way to break a very large replication group into smaller replication groups while avoiding a huge backlog while it re-syncs or verifies the files in each smaller replication group?

I have a very large replication group that covers 450,000 files and is 1.5TB and it has a lot of activity coming from three separate offices in three different cities, two of which are 1000+ miles apart. I have already structured a new DFS Namespace to handle the addressing of the new smaller replication groups and that is working well. But the question here is how to go about breaking the actual replication group itself into smaller chunks.

This is needed because some offices don't need big chunks of that existing replication group and sometimes we want a local representation of part of that directory structure to show what is local only and not replicate that portion (like a temp directory of sorts).

Question: Is there a way to make this 350,000 file backlog complete faster? For nearly every file the only change was a change to the ACL for each affected file. Some files have changed content, but that is not the common case in this situation.

This might be fixed. I'll edit this text to confirm success/fail after a period of time and verification. Toward the end of this question text I have detailed the changes made recently that might have fixed it.

We have a DFSR replication group with about 450,000 files and takes up 1.5TB of space. In this situation, there are two Windows Server 2008 R2 servers that are about 500 miles apart. There are other servers, but they aren't involved in this replication group. Server ALPHA is the main server and is the one used by most of the staff. Server BETA is the server in the remote office and is less busy.

Here is a graph of backlog for this replication group (PNG hosted on Google Drive) showing the slow sync progress.

I needed to remove a permission entry that was in the root directory of that replication group, which of course was inherited across most of the subfolders. I made this change on server ALPHA. Right away after that, DFSR had a 350,000 file backlog. It has been more than a week and now it is at 267,000. The only thing that changed (initially) was the single permission change.

This is what happened (this is not the solution, just another explanation of what happened to cause this issue): http://blogs.technet.com/b/askds/archive/2012/04/14/saturday-mail-sack-because-it-turns-out-friday-night-was-alright-for-fighting.aspx#dfsr

Any changes that occur on server BETA are replicated to server ALPHA very quickly since there is no backlog in that direction. Any files changed on BETA do make it to ALPHA without trouble.

It's replicating 24/7 at full speed across a 50Mbps connection one end to a fiber 100Mbps on the other end. The staging area is 100GB on each server. There is nothing interesting in the event logs at all. There is an unrelated high watermark event that shows up for an unrelated replication group that is neither for this particular replication nor for this ALPHA/BETA server pair. In particular there are no event log entries for high watermark nor for connection errors.

ALPHA's view of the replication group:

Bandwidth Savings: 99.83% reduction (30.85 MB replicated instead of 18.1 GB)

I believe that the 30.85MB/18.1GB happened since I last restarted the DFSR service on ALPHA and BETA. If so, this shows that even though it is taking a very long time (longer than I believe it should take) it isn't actually transferring the file contents across the wire.

Replicated folder: 1.46TB (actual size), 439,387 (files), 52,886 (folders)

Conflict and Deleted folder: 100.00GB (configured size), 34.01GB (actual size), 19,620 (files), 2,393 (folders)

Staging folder: 200.00GB (configured size), 92.54GB (actual size)

I got one high watermark error in the logs (May 14, 7pm) and so have upped the staging quota to 200GB from 100GB. I know that the Microsoft-approved route is to increase by 20%, but I'm not playing around on this. We have plenty of disk space to spare on the staging disk arrays.

Disabling anti-virus on all servers did not help, though I thought it would have helped a little bit. For now I have re-enabled anti-virus but set the replication group's path to be excluded from scanning in order to remove that variable from the equation.

Is there a way to get this to go faster? I would just make this change on server BETA as well, but there are files that have changed on ALPHA but haven't replicated to BETA and by making the inherited permission change on BETA would push OLD files from BETA to ALPHA (because DFSR seems to ignore file timestamps when comparing which file is the winner in a collision). And having that happen would be rather bad.

The backlog is reducing slowly. Very, very slowly. It is going forward, though. But at this rate, it will be weeks before it finishes. I'm contemplating just shoving a copy of the data set onto a 3TB drive and shipping it to the remote office. Is there a better way?

May 16, 4am US PT: What might have fixed the problem (assuming it's honestly fixed, anyway):

I made multiple changes to the DCs that should have been made a long time ago. The problem is that this network was inherited from someone else who probably inherited it from someone else, etc. I can't promise which change fixed the problem. Here they are in no particular order:

• All DCs were not in the "Domain Controllers" OU. I've never seen a Windows Domain that had their DCs elsewhere. I moved them back to where they belonged. They were previously in OUs that were segregated by the name of the city each office is in. (I have a feeling I've got some plumbing work to deal with now that I moved those, but all seems okay at present...)
• AVG Anti-Virus is running on all DCs and DFSR-participating servers. I excluded the replicated folders and the staging folders from active/on-access scanning. I don't think this fixed the problem and I'm likely to test this issue later on to see if undoing that change will interfere with the replication speed of DFSR. That's a challenge for another day.
• dcdiag.exe complained of a DNS issue with regard to RODCs. I remedied that problem even though we have no RODCs on the domain at all. I doubt this fixed anything.
• One of the _ldap._tcp.domain.GUID._msdcs.DOMAIN.NET SRV records was missing for one of the DCs (not one of the DFSR servers) and I remedied that. I don't think this helped either.
• One of the times I rebooted server BETA it complained of a bad shutdown of the DFSR database (event 2212) and it then proceeded to take hours to rebuild the database. When finished it reported event 2214 to let me know it finished. After that, replication was still running extremely slowly, but it might have helped unstick whatever was stuck.
• One of the DCs didn't have 127.0.0.1 as a secondary DNS server in its interface configuration. I added it. This wasn't one of the DFSR servers, so that probably had nothing to do with it.
• I followed the TechNet Blog: Tuning replication performance in DFSR recommended Registry settings for DFSR servers. I used all of the "tested high performance value" values except for AsyncIoMaxBufferSizeBytes was set to 4194304, which is one notch lower than the high value. This could have helped with the problem... or maybe not. It's difficult to tell when one changes too many variables.
• dcdiag.exe complained about a problem with communicating with the RPC service on BETA, but only after already making the above changes. This seemed to be the most likely issue going on, but there was nothing I did to correct it. The VPN was running properly and the firewall wasn't blocking it. It's possible that one of the above items is what caused and then remedied the RPC issue or it could have been simple coincidence. I am not getting that error now and replication is running smoothly at present.

The moral of the story is: change one thing at a time or you'll never really know what fixed it. But I was desperate and was running out of time to fix it, so I just fired a bunch of bullets at the problem. If I ever pinpoint the fix, I'll report that here. Don't bank on me narrowing it down, though.

EDIT 5/21/2012: I solved this by driving for about seven hours with a spare server (GAMMA) to the remote office yesterday. GAMMA is now acting as their primary local server while their usual server (BETA) catches up on the replication. Since I put it into place, the servers have been going about double the replication speed. While this tells me it could be a VPN-related issue, I'm less inclined to believe that it is since all new updates seem to replicate to GAMMA from ALPHA have been very quick and going well.

EDIT 5/22/2012: It's at 12000 right now and should be finished in a few hours. I'll post a nice graph of the progress from slow start to fast finish. The problem is that the only thing that really actually "fixed" it is the local server connection. I'm presently thinking that maybe the VPN is part of the problem. And if that's the case, I feel that this question isn't quite answered yet. After I've had some more time to check out how things are replicating via the VPN and seeing any failures, I'll debug and report the progress.

If something changes I'll update here.