/3GB switch on win2k3 server with 6gb RAM and PAE

I thought I would post a followup for others to try to consolidate the information into a single post (based on what I have learnt and the other information already posted).

PAE:

PAE will allow a 32bit Windows server to make use or more than 4GB RAM, with the maximum being dependant on the version of windows you are running (Wikipedia has a nice reference here)

One thing to note, if you have Data Execution Prevention (DEP) or NoExecution (NX) turned on, then this will effectivly enable PAE without having to explicitly enable it in the boot.ini.

Bottom line, PAE has no effect of the amount of memory a single 32bit process can access. It only affects the total amount of memory windows can 'see' and make use of (so you can have 2 processes each using 2GB, with Windows using 2GB on a 6GB system)

3GB:

Firstly, when I am speaking about the memory available to single 32bit process, I am refering to the processes virtual address space. For a 32bit process on a 32bit Windows OS this is limited to 4GB.

On a system without the /3GB switch, the 4GB of virtual address space is split 2GB / 2GB between the running process and the Windows kernel.

When you enable the 3GB switch, the split in virtual address space is changed to 3GB / 1GB. This allows the process to use more memory, at the expense of the kernel memory. Please note: Windows will only allow a process to use more the 2GB or memory if the executable has been compiled with the IMAGEFILELARGEADDRESSAWARE flag set.

Now, as has been mentioned in other posts, the penalty of using the 3GB switch is that the kernel has less memory to work with. And one of the main casualties of the reduced memory is the number of Page Table Entries (PTEs) available. A page table is the data structure used by the Windows Virtual Memory Manager to store the mapping between virtual addresses and physical addresses in memory. If you have insufficient free PTEs, then windows may fail to allocate memory to a process when requested, even if the process has not yet exhausted its address space.

The free PTE count can be measured using perfmon (\Memory\Free System Page Table Entries). Anything under 5000 is considered critical by Microsoft. As an example, on the servers mentioned in the original post, without the 3GB switch and with the process running, the free PTE count was around 160k. After enabling 3GB but before the process had started, windows was reporting 3.5k free PTEs (a dramatic reduction). This number would have dropped quickly if we had started the process.

The way to compensate for this dramatic change is to enable the USERVA switch in the boot.ini. By setting USERVA=2800, this moves the 3GB / 1GB split in memory and 'gives back' approximately 250MB back to the kernel for its use. By way of example, after setting USERVA=2800 in the boot.ini on our system, the free PTE count now sits around 60k with the process running (a lot better than the 3.5k we were seeing).

More information on the USERVA switch can be found in the Microsoft KB article.

It is also worth mentioning that enabling PAE also has an impact on the free PTE count. The PAE switch causes each PTE entry to use twice the normal allotted virtual address space.

Hopefully this provides a nice concise summary of the information for anyone looking at a later date.

Cheers Sam

Listen to this episode of RunAs Radio.

The main focus of this episode covers the obvious “why 64bit is better” – however the guest goes into “why /3GB should be avoided.” Basically it significantly reduces the number of page table entries available to the OS – so much so that the OS can become unstable.

In a nut shell - It may be appropriate for a server that is servicing a single function (like an AD controller or SQL Server), however it should be avoided on a system that is providing multiple functions. At the end of the day “your mileage may vary” – remember that /3GB can easily make the OS unstable.

You might want to consider a 64 bit OS even if the app is only 32 bit. On Windows x64, 32 bit processes get 4GB of RAM, not 2GB.

See the Virtual Memory section of KB294418 for details.

From what I know, the PAE switch allows access to memory above 4GB to the applications on the server. According to this technote the applications don't really know about this memory swapping, it's all handled within Windows. I think using the /3GB switch in the 6GB server would limit the kernel to 1GB. Another limitation introduced by the simultaneous use of /3GB and /PAE is the server will not address more than 16GB.

Unless you're trying to eke out every MB of memory for the application, you may want to just use /PAE without /3GB. That way if someday down the line you pop in a total of 24 or 32GB of RAM, you won't have to try and figure out why Windows uses only 16GB.

On my previous system, I used the /3GB switch because one of my applications needed lots of memory. Recently, I upgraded to a 64-bits Vista system and upgraded the application to a 64-bits version, which grants it the full 12 GB that I have now. However, other 32-bits applications are still limited to the same 3 GB of memory (or 2 GB without the switch) simply because they can't see beyond the size of an address pointer. (And in 32-bits systems, address pointers are 32 bits...)

However, the /3GB switch combined with more than 3 GB of RAM is still useful as long as Windows itself is able to handle larger address pointers. It allows Windows to keep more applications in-memory, thus it swaps a lot less to disk, which increases the performance.

PAE is a hardware trick where the processor has a few more pins to send an address over. They have increased from 32 pins (bits) to 36 pins. This allows up to 64 GB of RAM for any application that is aware of these additional pins. Microsoft makes good use of this in their server software, thus Windows can handle up to 64 GB. If the process that eats up that much memory is also aware of these additional pins, it too could allocate up to 64 GB of RAM. This technique for applications is called Address Windowing Extensions. Of course, this also requires special code within the application to process this memory and reminds me of the old MS-DOS era where application pointers were limited to just 16 bits at first (64 KB) but because the processor had 20 pins (later 24), you could use special 32-bits pointers to point at an address or just stick to the older 16-bits pointers and stay limited to just 64 KB. (20 bits is 1024 KB and DOS used everything above the bottom 640 KB, or 1 GB. 24 bits is 4 GB which was popular for the first 80386 processors as upper limit.)

As I read Mark Russinovich's excellent post Pushing the Limits of Windows: Physical Memory, yes... that 1GB of memory available to the kernel on a 32-bit install of Windows with the /3GB switch persists even on systems with up to 128GB of RAM (supported on 32-bit installs of W2K3 Datacenter).

The maximum 32-bit limit of 128GB, supported by Windows Server 2003 Datacenter Edition, comes from the fact that structures the Memory Manager uses to track physical memory would consume too much of the system's virtual address space on larger systems. The Memory Manager keeps track of each page of memory in an array called the PFN database and, for performance, it maps the entire PFN database into virtual memory. Because it represents each page of memory with a 28-byte data structure, the PFN database on a 128GB system requires about 930MB. 32-bit Windows has a 4GB virtual address space defined by hardware that it splits by default between the currently executing user-mode process (e.g. Notepad) and the system. 980MB therefore consumes almost half the available 2GB of system virtual address space, leaving only 1GB for mapping the kernel, device drivers, system cache and other system data structures, making that a reasonable cut off:*

Raymond Chen, one of Microsoft's Windows shell programmers, has a great series of articles on his blog about the /3GB switch, PAE, AWE and NX. They should be required reading for anyone trying to understand how it all works, how it interacts, and why in many circumstances it's probably not what you want:

http://blogs.msdn.com/oldnewthing/archive/2004/08/22/218527.aspx