What is the difference between DEP always on ("/NoExecute=AlwaysOn" in boot.ini) and DEP opt-out ( "/NoExecute=OptOut" in boot.ini) with no exceptions?
"no exceptions" = empty list of programs for which DEP does not apply. DEP = Data Execution Prevention (hardware).
One would expect it to work the same way, but it makes a difference for some applications:
E.g. for all versions of UltraEdit 14 (14.2). It crashes at startup for DEP always on, at least on Microsoft Windows XP Professional Edition x64 edition. (Update 2010-03-11: this problem has been fixed with UltraEdit 15.2 and later.)
Update 2010-07-12: Mozilla Thunderbird 2.0.0.19 also crashes at startup for DEP set to the highest level (as UltraEdit used to do).
Is there a difference between Windows XP, Windows Vista and Windows 7?
Is there a difference between 32 bit and 64 bit versions of Windows?
Configuration Description
OptIn This setting is the default configuration. On systems with processors that can implement hardware-enforced DEP, DEP is enabled by default for limited system binaries and programs that "opt-in." With this option, only Windows system binaries are covered by DEP by default.
OptOut DEP is enabled by default for all processes. You can manually create a list of specific programs that do not have DEP applied by using the System dialog box in Control Panel. Information technology (IT) professionals can use the Application Compatibility Toolkit to "opt-out" one or more programs from DEP protection. System compatibility fixes, or shims, for DEP do take effect.
AlwaysOn This setting provides full DEP coverage for the whole system. All processes always run with DEP applied. The exceptions list to exempt specific programs from DEP protection is not available. System compatibility fixes for DEP do not take effect. Programs that have been opted-out by using the Application Compatibility Toolkit run with DEP applied.
AlwaysOff This setting does not provide any DEP coverage for any part of the system, regardless of hardware DEP support. The processor does not run in PAE mode unless the /PAE option is present in the Boot.ini file.
For question 1: I think this difference is caused by the backdoors that Microsoft has put into hardware DEP for OptOut, according to Fabrice Roux (see below). In the case of IrfanView, for which Steve Gibson observed the same difference as I did for UltraEdit (see the question), the difference is caused by a non-DEP aware EXE packer (ASPack) that Microsoft coded a backdoor for.
Sources:
From http://blog.fabriceroux.com/index.php/2007/02/26/hardware_dep_has_a_backdoor?blog=1, "Hardware DEP has a backdoor" by Fabrice Roux. 2007-02-26. "IrfanView was not using any trick to evade DEP ... Microsoft just coded a backdoor used only in OPTOUT. Bascially Microsoft checks the executable header for a section matching one of the 3 strings. If one these strings is found, DEP will be turned OFF for this application by windows. ... 'aspack', 'pcle', 'sforce'"
From http://www.grc.com/sn/sn-078.htm, by Steve Gibson. "I can’t find any documentation on Microsoft’s site anywhere, because we’re seeing a difference between always-on and opt-out. That is, you would imagine that always-on mode would be the same as opting out if you weren’t having any opt-out programs. It turns out it’s not the case. For example ... the IrfanView file viewer ... runs fine in opt-out mode, even if it has not been opted out. But it won’t launch, Windows blocks it from launching ... in always-on mode."
From http://www.grc.com/sn/sn-083.htm, by Steve Gibson. "... IrfanView ... won’t run with DEP turned on. It’s because it uses an EXE packer, an executable compression program called ASPack. And it makes sense that it wouldn’t because naturally an executable compressor has got to decompress the executable, so it allocates a bunch of data memory into which it decompresses the compressed executable, and then it runs it. Well, it’s running a data allocation, which is exactly what DEP is designed to stop. On the other hand, UPX, which is actually the leading and most popular EXE compressor, it’s DEP- compatible because those guys realized, hey, when we allocate this memory, we should mark the pages as executable."
Setting the System DEP Policy level to OptOut allows for any 32bit thread to call SetSystemDEPPolicy(0) which will disable DEP protection for the memory space allocated to that process. If the system's DEP Policy is set to AlwaysOn, invocations of SetSystemDEPPolicy will throw an error.
KB 875352 is actually the one: OptOut lets you specify a list of programs for which DEP does not apply, AlwaysOn will apply DEP to everything always.
A detailed description of the Data Execution Prevention (DEP) feature in Windows XP Service Pack 2, Windows XP Tablet PC Edition 2005, and Windows Server 2003
The shorter, snappier title is MS KB 875352 ;-)
More than your ever needed to know...
There's a fairly big discussion about this on Steve Gibson's securable homepage. Also, they've discussed it quite often in the security now podcast. You can find the transcripts also on grc.com
I know this is an old thread, but I got asked this again today and want to put this to rest. AlwaysOn is going to enable DEP on all programs. OptOut means simply that User can specify applications that would be OptOut of DEP. If application is not listed in Exceptions then DEP is enabled for that application.