Home / server / Questions / 604899
In Process
RSW
Asked: 2014-06-14 03:36:43 +0800 CST

Where to install ESXi so that all local drives are available to VSAN

  • 772

I am currently setting up a new 3-host VSAN cluster that will be placed into production soon. (Note this is the newer "Virtual SAN" (VSAN) technology, not the older vSphere Storage Appliance (VSA) technology). This is the first time I have worked with VSAN.

Each of the three hosts in the cluster has four 1TB local HDD's and one 200GB local SSD (which VSAN needs for read/write caching) to contribute to the cluster. I have installed ESXi 5.5 directly on the first local HDD of each host.

I added the three hosts to vCenter and launched the vSphere Web Client to configure VSAN. But instead of seeing all four local HDD's on each host as being available to VSAN, only three are available.

From what I have read in the VSAN documentation, disks used by VSAN must solely be used by VSAN. That is, once VSAN "takes over" a local drive, that drive can not be used for any other purposes (such as reserving partitions for use by other OS's). However, it wasn't clear to me if that also meant not being able to store the ESXi bootable partition on one of the VSAN local drives.

I've heard a lot about installing ESXi to a bootable USB stick as an option, but have never tried it since we've always just placed the ESXi boot partition on our traditional SAN. (For background, the new VSAN cluster is going to replace our blades and traditional SAN due to downsizing and cost-cutting measures).

So to summarize, here are my questions:

  1. Is it possible to install ESXi to one of the local HDD's and still have the rest of the space on that HDD be available to VSAN?
  2. If the answer to question #1 is "no", then is the next best option to install ESXi to a bootable USB stick? (and if so, are there any notable tradeoffs to doing this that I should be aware of?)
virtualization

3 Answers

  1. The VSAN design guide recommends the following:

    • Use SD, USB, or hard disk devices as the installation media whenever ESXi hosts are configured with as much as 512GB memory.

    • Use a separate magnetic disk or solid-state disk as the installation device whenever ESXi hosts are configured with more than 512GB memory.

    My recommendation for your specific Supermicro configuration is to use the internal USB with an enterprise purpose-built USB card (there should be a slot on the motherboard). If not, and your hosts are using a SAS HBA or motherboard SATA ports, and you have spare drive slots, using a small single SSD is an okay alternative.

    In terms of the caveats associated with USB/SDHC card boot in vSphere, my experience is here:
    What happens when the USB key or SD card I've installed VMware ESXi on fails?

    VSAN accounts for this situation reasonably well.

    • 5

    1. You can, but you don't really want to, because then you have one disk smaller than the rest, and you can't use passthrough if you wanted to.

    2. Most servers have an internal SD-card slot (or sometimes mirrored slots), or an internal USB port directly on the motherboard. You can install it on one of these. I have a bunch running off the SD card, and two that use USB stick.

    ESXi is smart enough to know that it's installed on flash media. Have a look at this KB from VMWare.

    If you do end up on USB instead of SD, make sure you buy a good, proper flash drive that has a nice fast write speed. I just spent my entire day copying 60GB of files onto a 64GB flash drive that could only write at 3MB/sec. It was horrific. The USB sticks that cost twice the price typically get way, way more than twice the performance.

    • 4

  3. Several people have already advised you to not use the HDD but to use an internal SD-card/USB-stick so I will not repeat this. Anyway, I think it's possible to boot ESXi hosts with auto-deploy so maybe that's a possible solution for you, too:

    "With Auto Deploy, vSphere hosts are network-booted from a central Auto Deploy server where the ESXi software is installed directly into the server’s memory."

    Auto Deploy

    edit: I havn't used Auto Deploy myself so I can't tell you how good it really works

    edit2: As Mark Henderson points out in the comments below, you shouldn't host the boot server on the cluster. If all nodes are down (power outage or something) you wouldn't be able to get the cluster running. Nodes down -> boot server down -> you can't boot any host -> you can't start the boot server.

    • 0