I have MySQL running on a 32-bit RHEL box. The server itself has 4GB total memory with 2GB allocated to MySQL.
I would like to know the max amount of memory I can put in the box and how much of that I can allocate to MySQL.
I have heard both 2GB and 4GB as the per-process-limit on a 32-bit OS...
Ultimately I'd like to know if I can increase the memory for MySQL without upgrading to a 64-bit OS.
It depends on your kernel and configuration. It is usually either 2Gb or 3Gb.
The application itself may have a self-imposed lower limit though. Some 32bti apps/services won't use more than 2G even if the OS would let them because their own internal memory management uses (or allows for) signed 32 bit integers for everything.
If your CPU, memory controller, OS and application all support PAE then that would allow more RAM to be used by the single process. PAE is supported by the Linux kernel and there are no doubt pre-build kernels in the RHEL repositories that have it enabled (it is not enabled in most default kernels as there are performance implications) - I don't know if MySQL will have support for it though.
If you are using an older kernel it might be worth looking into the memory map configuration - while most new kernels default to a 1G/3G split between kernel and userland mapped address space within each process, in older times (when machines with RAM close to the limit of a 32bit address space were practically unheard of) a 2G/2G split was the default but could be changed if you needed 1G/3G.
Without digging deep into the subject, my guess would be that (unless you are using an older kernel and that is what is imposing the 2G limit rather than a 3G one) MySQL will not use more then 2G without the upgrade to the 64-bit compile. The extra RAM is unlikely to be wasted though - the kernel will use it for cache/buffers so while MySQL using it directly would be much more efficient (as there would be many less cases of MySQL's cache and the OS cache both holding the same blocks of data) it won't be standing idle so some benefit will be seen.
Most modern distributions and processors support PAE, which is Intel's instruction set allowing memory addressing beyond the 32-bit limitation. PAE allows a maximum of 64GB of RAM to be accessed by the processor. PAE has additional CPU overhead.
32-bit Linux has a per process limit of 3GB addressable even with PAE. Typically, the kernel reserves approximately 1GB.
MySQL is a single process daemon. As such, it will be unable to address outside of the 32-bit limitation even with PAE.
Some useful references:
I think each process can see (address) 4 GB, but 2 GB of that is kernel space and cannot be filled by application data.
I am therefor assuming that MySQL could potentially use up 2 GB times number of MySQL instances GB of total memory (physical memory and swap space).