I'm trying to start 64-bit guest (MacOSX and Windows7) on 32-bit host (Hardened Gentoo Linux, kernel 2.6.28-hardened-r9) using VMware Workstation (6.5.3.185404 and 7.0.1.227600).
If VT-X disabled in BIOS, VMware refuse to start 64-bit guest (as expected). If VT-X enabled in BIOS, VMware start guest without complaining, but then, in about a second (I suppose as soon as guest try to switch on 64-bit) my host reboots (actually, it's more like reset - normal reboot procedure skipped and BIOS POST start immediately).
My hardware is Core 2 Duo 6600 on ASUS P5B-Deluxe with latest stable BIOS 1101. I've power-cycled system, then enabled Vanderpool in BIOS. My CPU doesn't support Trusted Execution Technology, and there no way to disable it in BIOS. I've rebooted several times after that, sometimes with power-cycled, and ensure Vandertool is enabled in BIOS.
I've also run VMware-guest64check-5.5.0-18463 tool, and it report "This host is capable of running a 64-bit guest operating system under this VMware product.".
About a year ago I tried to disable hardened in kernel to ensure this isn't because of PaX/GrSecurity, but that doesn't help. I have not checked 32-bit guests with VT-X enabled yet, but without VT-X they works ok. ASUS provide "beta" BIOS updates, but according to their descriptions these updates doesn't fix this issue, so I'm not sure is it good idea to try it.
My best guess now it's motherboard/BIOS bug. Any ideas?
Update 1: I've tried to boot vt.iso provided at http://communities.vmware.com/docs/DOC-8978 and here is it report:
CPU 0: VT is enabled on this core
CPU 1: VT is enabled on this core
Update 2: I've just tried to boot 32-bit guests (Windows7, Ubuntu9.04 and Gentoo) using all possible virtualization modes. In Automatic, Automatic with Replay, Binary translation everything works, in Intel VT-x/EPT or AMD-V/RVI I got message "This host does not support EPT. Using software virtualization with a software MMU." and everything works. BUT in Intel VT-x or AMD-V mode all 32-bit guests reset host just like 64-bit guests! So, this issue is not specific to 64-bit guests.
One more thing. Using Intel VT-x or AMD-V mode for both 32/64-bit guests my host reset right after starting VM, i.e. before VM BIOS POST and before guest even start booting. But using Intel VT-x/EPT or AMD-V/RVI VM BIOS runs ok, then 64-bit guests start booting (Windows7 completed "Loading files" progressbar), and only after that host reset.
edit: I no longer believe that my response, below, is useful. I would encourage others to provide alternative answers.
Running 64-bit guests on 32-bit hosts is a dodgy affair to begin with. Surprisingly, VMware Workstation apparently can accomplish this under limited circumstances, but I think you would be much better off with a 64-bit host OS.
Here's what I would recommend:
P2V your existing 32-bit Gentoo host and make sure that it boots in VMware Workstation. Depending on whether you use fixed IP addressing or can foresee other potential conflicts, you may want to be careful about virtual network adapter settings for the purposes of this test.
Make a backup everything on the box, including your new Gentoo virtual machine.
Slick it and install any fully supported 64-bit host OS.
Launch your existing 32-bit box in a VM. Of course, you can run it in full screen mode, give it a bridged network adapter, and use it as your "primary" operating system. If you have enough RAM and give it a physical disk of its own, you shouldn't notice any significant change in performance.
Run 32-bit and 64-bit guest operating systems to your heart's content.
There may be reasons why this wouldn't be possible for you, but if you have no blocking issues, it could give you a fully supported path forward.
Finally, in current kernels (tested 2.6.32-hardened-r22 and 2.6.35-hardened-r5) this issue can be fixed by switching off CONFIG_PAX_MEMORY_UDEREF!