Home / server / Questions / 920018
Accepted
Dale Mahalko
Asked: 2018-07-09 10:03:44 +0800 CST

Apparently Robocopy does not understand Windows Server 2016 deduplication. Is there a way to make it work without corrupting the volume?

  • 772

Scenario:

  • Doing a server migration from old Server 2008 R2 to new Server 2016, following this Server Fault guide: File server migration using Robocopy

  • After Robocopy completes, enable deduplication on Server 2016 for the copied volume, and then use PowerShell to start dedpulicating manually. After many hours it completes and recovers about 25% of disk space.

  • Run Robocopy again to copy anything that may have been missed in the initial copy, as a final check of the new server.

....but Robocopy (run from Server 2016) doesn't understand deduplication and so instead proceeds to trash the deduplication chunkstore..

-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows                              
-------------------------------------------------------------------------------

  Started : Sunday, July 8, 2018 12:10:02 PM
   Source : \\SERVER-2008\e$\
     Dest : \\SERVER-2016\e$\

    Files : *.*

  Options : *.* /TEE /S /E /COPYALL /PURGE /MIR /ZB /NP /MT:32 /R:1 /W:10 

------------------------------------------------------------------------------

    *EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\
    *EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\
    *EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\
      *EXTRA File         253504    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\DedupFileList.1
      *EXTRA File         253504    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\DedupFileList.2
      *EXTRA File             28    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\stamp.dat
    *EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\COW\
    *EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\COW\00010000\
      *EXTRA File         196608    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\COW\00010000\00000046.00.RB
      *EXTRA File         106496    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\COW\00010000\00000048.00.RB

[.......]

*EXTRA File           30.3 m    \\SERVER-2016\e$\System Volume Information\Dedup\ChunkStore\{B7E1F3A4-AAD9-4449-9DF7-5489421C9EC5}.ddp\Stream\000f0000.00000002.ccc
*EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\Logs\
  *EXTRA File         29.7 m    \\SERVER-2016\e$\System Volume Information\Dedup\Logs\00000001.kmchange.log
  *EXTRA File        999.8 m    \\SERVER-2016\e$\System Volume Information\Dedup\Logs\changes.optimization.1.10.archive.etl
  *EXTRA File       1000.0 m    \\SERVER-2016\e$\System Volume Information\Dedup\Logs\changes.optimization.1.11.archive.etl
  *EXTRA File        735.5 m    \\SERVER-2016\e$\System Volume Information\Dedup\Logs\changes.optimization.1.12.archive.etl
  *EXTRA File        999.8 m    \\SERVER-2016\e$\System Volume Information\Dedup\Logs\changes.optimization.1.9.archive.etl
  *EXTRA File          1.3 m    \\SERVER-2016\e$\System Volume Information\Dedup\Logs\changes.optimization.2.1.archive.etl
*EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\Settings\
  *EXTRA File             76    \\SERVER-2016\e$\System Volume Information\Dedup\Settings\Dedup.00.cfg
  *EXTRA File             76    \\SERVER-2016\e$\System Volume Information\Dedup\Settings\Dedup.01.cfg
  *EXTRA File           2228    \\SERVER-2016\e$\System Volume Information\Dedup\Settings\dedupConfig.01.xml
  *EXTRA File           2228    \\SERVER-2016\e$\System Volume Information\Dedup\Settings\dedupConfig.02.xml
  *EXTRA File              0    \\SERVER-2016\e$\System Volume Information\Dedup\Settings\VolumeJobLock.bin
*EXTRA Dir        -1    \\SERVER-2016\e$\System Volume Information\Dedup\State\
  *EXTRA File           2982    \\SERVER-2016\e$\System Volume Information\Dedup\State\chunkStoreStatistics.xml
  *EXTRA File           2592    \\SERVER-2016\e$\System Volume Information\Dedup\State\dedupStatistics.xml
  *EXTRA File         11.5 m    \\SERVER-2016\e$\System Volume Information\Dedup\State\GCReservedSpaceBitmap.tmp
  *EXTRA File          1.0 g    \\SERVER-2016\e$\System Volume Information\Dedup\State\GCReservedSpaceContainer.ccc
  *EXTRA File         46.0 m    \\SERVER-2016\e$\System Volume Information\Dedup\State\GCReservedSpaceDeleteLogs.tmp
  *EXTRA File          1.0 m    \\SERVER-2016\e$\System Volume Information\Dedup\State\GCReservedSpaceFileList.tmp
  *EXTRA File           4096    \\SERVER-2016\e$\System Volume Information\Dedup\State\GroupCommitFlushControl0.bin
  *EXTRA File           2066    \\SERVER-2016\e$\System Volume Information\Dedup\State\optimizationState.xml

[......]

I aborted it moments after seeing this fly by in the log and recognizing what was happening. But the damage is already done, the data on the deduplicated new server was instantly corrupted by Robocopy as it stormed through \System Volume Information. The new server drive partition has be formatted and recopied all over again from Server 2008.

Is there a safe way to use Robocopy so that it doesn't touch the deduplication volume data?

Also, I have a new concern.. if Robocopy can destroy a deduplicated volume, what else is unsafe to use with a deduplicated volume, that sees right through it and can destroy the underlying data that should only be accessible by the server? (probably should be a separate question..)

windows-server-2016

5 Answers

  1. Best Answer

    The System Volume Information directory should be excluded using the /XD switch. Probably a good idea to exclude other hidden/system directories such as $RECYCLE.BIN.

    • 17

  2. Two command line switches that were used lead to this: /MIR and /ZB. As the documentation ( robocopy /??? ) describes:

    /MIR :: MIRror a directory tree (equivalent to /E plus /PURGE).
    /ZB :: use restartable mode; if access denied use Backup mode.

    It's the combination that did you in: /MIR will delete (as pointed out when you run robocopy without arguments) and "Backup mode" defeats most permissions in order to be able to read files "normally" unreadable in order to make complete backups.

    "Backup mode" is notably undefined in the "help" description. You've got to know that the Windows CreateFile API supports a flag called FILE_FLAG_BACKUP_SEMANTICS, which in combination with a certain access right SE_BACKUP_NAME (which is given to the Administrator group by default - also the Backup Operators group, duh) bypasses normal file security.

    You didn't know that? Then you may also not know that robocopy wasn't originally part of Windows at all - it was part of a supplement called the "Windows Resource Kit" which was used mainly by programmers and hard-core sysadmins back in the day, and although it was grandfathered into the Windows distribution back in Windows Server 2008 it has never ever received any attention - except for additional performance options, woot! Particularly, no attention from program managers dedicated to UI or usability. So it's a raw bit of power that can be used - or misued! - at your own risk.

    (A good rule of thumb: Don't use command line options you don't really understand.)

    Information you might like to know about "Backup mode" file access:

    https://isc.sans.edu/forums/diary/Use+The+Privilege/20483/

    https://docs.microsoft.com/en-us/windows/desktop/api/FileAPI/nf-fileapi-createfilea

    https://docs.microsoft.com/en-us/windows/desktop/FileIO/file-security-and-access-rights

    • 7

  3. Here are the followup results using the other answers provided, and testing with a deduplicated destination. (Meta: I don't know if I should be including this as an edit at the bottom of my original question.)

    The Robocopy command line evolved to finally look like this:

    robocopy \\OLD-SERVER\e$\ \\NEW-SERVER\e$\ /MIR /COPYALL /DCOPY:DAT /NP /Z /B /J /SL /MT:128 /R:1 /W:10 /LOG+:robocopy-log.txt /TEE /XD "Recycler" "Recycled" "$Recycle.bin" "System Volume Information" /XF "pagefile.sys" "swapfile.sys" "hiberfil.sys"

    Options and purpose:

    • /MIR - Mirror source to destination, and delete files and directories on the destination, if they are no longer present on the source
    • /COPYALL - Copy all file info: data, attributes, and timestamps, NTFS Security ACLs, Owner info, Auditing info (not all included by default)
    • /DCOPY:DAT - Copy all directory info - data, attributes, timestamps (original creation timestamp is not copied by default; normally this changes to the date that it was copied by Robocopy)
    • /NP - Don't display progress
    • /Z - Use restartable mode
    • /B - Copy files in Backup mode (I don't know if this is needed for user directories where they are the exclusive owner, excluding the administrator. This option will destroy a deduplicated destination volume without excluding "System Volume Information")
    • /J - Copy using unbuffered I/O (faster copy of large multi-gig files)
    • /SL - Copy symbolic links rather than the target
    • /MT:128 - Use maximum CPU threads (better use of 10 gigabit Ethernet and many CPU cores)
    • /R:1 - If file access error, retry 1 time
    • /W:10 - If file access error, wait 10 seconds before retry
    • /LOG+ - Log the output to text file, append if log file already exists
    • /TEE - Print results to screen and to log file
    • /XD - Exclude directories, and everything within them. Names with spaces in them need be enclosed in quotes: "Recycler" "Recycled" "$Recycle.bin" "System Volume Information"
    • /XF - Exclude files: virtual memory and hibernation files if they happen to be present on the source: "pagefile.sys" "swapfile.sys" "hiberfil.sys"

    Final re-run:

                Total    Copied   Skipped  Mismatch    FAILED    Extras 
 Dirs :    158189    153466    158186         0         0         0
Files :   1116292         0   1116296         0         0         0
Bytes :   1.350 t         0   1.350 t         0         0         0
Times :   0:01:04   0:00:00                       0:00:00   0:01:04

    Deuplication report

    ,

    Also, I do not know the proper channels to report bugs to Microsoft, but I have linked to this discussion at the bottom of Microsoft's deduplication documentation, on their Windows IT Pro Center website:

    https://docs.microsoft.com/en-us/windows-server/storage/data-deduplication/overview

    • 1

  4. The problem is that you are not copying the folders you need but the entire volume, this contains the hidden system folder "System Volume Information" which is used for anything related to the file system. Deduplication and File Server Resource Manager are storing their data in there too. By copying the volume to another one and using the /MIR*, also by using the /B** you're using the backup mode which can copy folders your admin account can't, you're actually replacing the System Volume Information on the Target and this destroys the Dedup's chunk store. I would advise against this type of copying, it is preferable to do it by folder OR exclude the "System Volume Information" folder at all (it will save a lot of hair pulling and swearing in the short/long-run).

    "*" Mirrors the contents from the source to the target, removes files not existing in source from the target. ** Backup mode, copies files that can't be accessed by an admin's account (uses the SeBackupPrivilege for the source read and SeRestorePrivilege to copy to the target folder).

    • 1

  5. Maybe it's just me, but my first thought was - don't ever try to copy the drive itself "e$". I would only ever Robocopy the specific folders that were created for user content, not any system folders created by Windows itself.

    • 0