How to run VMWare ESX or ESXi in a virtual machine?

VMWare ESX or ESXi CAN run inside a virtual machine, provided certain prerequisites are satisfied.
This kind of setup is of course completely useless (and totally unsupported) in a production environment, but can be very useful for two purposes:

• Testing or studying ESX or ESXi if you don't have a physical server available.
• Testing or studying the whole Virtual Infrastructure if you don't have at least two servers and a SAN.

Prerequisites:

• You need some physical resources. In order to run ESX or ESXi in VM, the VM needs at least 1.5 GB of memory, two VCPUs and enough disk space for the server itself and for the VMs you will run inside it.
• You absolutely need a physical CPU with native virtualization support (Intel VT or AMD-V).
• You need to run VMWare Workstation 6.5, VMWare Server 2 or VMWare Fusion 5 on the physical host. Previous versions can't succesfully run ESX or ESXi in a VM.
• A 64-bit OS on the physical host is useful but not required.

Setup:

• Enable native virtualization support for your CPU in the motherboard BIOS (it's often not enabled by default).
• Install your preferred virtualization software. I've tested everything succesfully using VMWare Workstation 6.5.2 on a Windows XP x64 host, but it should work with VMWare Server 2.0 and/or Linux hosts, too.
• Create a custom VM using those setting:
  • Hardware compatibility level: latest
  • Guest operating system: other 64-bit
  • Virtual CPUs: at least 2
  • Memory: at least 1.5 GB
  • Networking: Host-only or NAT
  • SCSI adapter: LSI Logic
  • Virtual disk type: SCSI
  • Virtual disks: as you wish; I suggest using at least two virtual disks, a 10-GB one for installing the system and another one where to create a datastore. The space should be pre-allocated.
  • Remove floppy, sound card, USB controller, etc. Leave only networking and storage.
  • CPU Execution mode: Intel VT-x or AMD-V (very important).

• Manually edit the VMX file of the virtual machine you created, setting the following parameters:

  guestOS = "vmkernel"
monitor_control.vt32 = "TRUE"
monitor_control.restrict_backdoor = "TRUE"

• Start the VM and install ESX or ESXi from the installation ISO image.

• Configure the networking to allow the ESX or ESXi virtual server to talk with the host.

Usage:

• Use your web browser to connect to you virtual server's IP address and download the VI Client.
• Install the VI Client on the host.
• Connect to the virtual ESX/ESXi server.
• Create a VM as you wish.
• Power up the VM.

• If everything is done correctly, the VM will start. If it complains about not being able to power on a VM inside a VM, then there is an error with the `monitor_control.restrict_backdoor' parameter (or you're using an old version of VMWare Workstation/Server).

• Enjoy :-)

Installing a 64 bit OS is pre required for installing the VM ware ESXi version.I dont think u can carry on with ths installation if the CPU dosent support.

I was able to successfully install ESXi 4.1 into a Virtualbox VM. I had the same issue with the keyboard not being recognized. Out of a whim, I tried installing it on Virtualbox 3.2.6 making a few adjustments and found no problems.

When you create the VM, set memory to 2gb, the hard drive to 10gb, I changed the network card to Bridged so I could use the client from my host desktop, then click the advanced option and select the Intel 1000 Server MT card. I also changed the video to 128mb, but I doubt that would hinder anything if it were lower. Walla, no keyboard problems and the install completed fine. you can then do all your configuration changes with no problems.