Home / server / Questions / 109269
Accepted
Beau
Asked: 2010-02-04 10:22:50 +0800 CST

How to move or delete files from a folder containing 2 million files on an NTFS drive?

  • 772

The issue is that any modification to the directory locks up Explorer indefinitely, though Samba access to other directories still works. I've tried moving files locally and over Samba.

Even enumerating the directory to get the list of files locks up the computer indefinitely.

I tried using Python's win32file.FindFilesIterator to iterate the files but that also hangs.

My idea was to move each file to a different directory (in a directory above the directory we're dealing with) based on its timestamp, so that we'd have at most a thousand or so files in each directory... But since I can't even enumerate the files, that's been a non-starter.

If I have to give up and just nuke the directory I'm willing to do that, but a standard delete also hangs indefinitely.

I have set these two parameters to increase speed and they also did not help the issue:

R:\>fsutil behavior query disablelastaccess
disablelastaccess = 1

R:\>fsutil behavior query disable8dot3
disable8dot3 = 1

These are all sequential images that would have run into the 'bug' with 8.3 filenames whereby many similarly named files in one directory can take a long time to compute 8.3 filenames. From what I understand this data is stored in the file system even after disable8dot3 is enabled, so it may still be contributing to the problem.

Any ideas?

windows-server-2003 ntfs

8 Answers

  1. I don't know if it will work, neither if it is practical approach for you, but what about putting the harddisk into a Linux Computer and try it with Linux?

    (There are bootable Live CDs for download - you don't even have to physically move the harddisk).

    • 4

  2. Two things you should not do:

    1. Place so many files into one folder
    2. Use Windows Explorer to view such a folder, let alone try to do anything with it.

    What's happening is that Explorer is accessing each and every one of those files for the information that will, or could, be displayed in the Explorer interface. Additionally, if you have on access antivirus scanning taking place (and if not, why not), each file will be scanned, even if it's only to determine whether or not it is one of the types that should be scanned. All this takes a long time when dealing with so many files.

    Short term solution - Use the command line or some form of scripting instead.

    Long term solution - Develop a sensible scheme for ensuring you never have more than a few thousand files in any one folder.

    • 1
  3. Best Answer

    Depending on how the files were added and how badly the folder is fragmented the quickest way might even be to move off the files you want to keep and nuke the partition!

    • 1

  4. You could learn a little ahk or php. ahk would be easier to learn and someone on http://webchat.freenode.net/?channels=#ahk will probably be willing to help you get something together.

    quick ex. from when I needed to do this...

    Loop, C:\Users\PC\Web Sites\Folderwithtomanyfiles\*.htm, , 1  ; Recurse into subfolders.
{
Match :=
Match1 :=
name :=
name1 :=
M :=
File :=
total++
;the names of my files were each 7digits with some extra info I didn't want to share here so... 
RegExMatch(A_LoopFileFullPath, "Folderwithtomanyfiles\\(\d{7}.*?htm)", name)
RegExMatch(A_LoopFileFullPath, "Folderwithtomanyfiles\\(\d{7})", Match)
StringTrimLeft, Ma, Match1, 6
M .= Ma . "\"
StringTrimLeft, Ma, Match1, 5
M .= Ma . "\"
StringTrimLeft, Ma, Match1, 4
M .= Ma . "\"
StringTrimLeft, Ma, Match1, 3
M .= Ma . "\"
StringTrimLeft, Ma, Match1, 2
M .= Ma . "\"
StringTrimLeft, Ma, Match1, 1
M .= Ma
FileCreateDir, C:\Users\PC\Web Sites\fewer file names\%M%
FileAppend, %File%, C:\Users\PC\Web Sites\fewer file names\%M%\%name1%
created++
}
MsgBox Created %created% out of %total% 
}
;example of path ... 
;"C:\Users\PC\Web Sites\fewer file names\3\33\233\5233\35233\435233\0435233.htm
;turns out I should have put a few thousand files in each folder but oh well, hope the example helps.
    • 1

  5. Try using PowerShell. Using the Move-Item Cmdlet

    HTH

    • 0

  6. There's always cmd.exe and its del command.

    • 0

  7. You will need to reboot for the FSUTIL changes to take effect, just in case you haven't.

    The general order of complexity for NTFS functions (Add, Delete, Search) with 8.3 names disabled is O(Log N) so pretty much anything you do is going to take the same length of time once things have gotten to this state.

    If you have any AV active on the system then turn it off while you fix this.

    Out of curiosity do you get better performance if you copy files out in chunks using something like Robocopy? I suspect that it won't make much of a difference but it should take half the time a delete takes (there's only a search of the directory involved not a search then a delete of the entry). You'll still have the problem of deleting the originals eventually though.

    • 0

  8. use robocopy with the /mov option

    http://technet.microsoft.com/en-us/library/cc733145.aspx

    For example I had a folder with 1.2 m email files in it, which would hang explorer. The files were named starting with the date so I created folders for each month of the year and then used robocopy filtering on the date in the filenames.

    robocopy e:\sent-old\ e:\sentmail\2013-4 201304*.* /mov

    • 0