I'm building VMs in my lab and I want to replicate the situation that my fileservers will be encountering in production. Here's a brief overview of what I want to do.
I have multiple ESXi 4 servers. They are accessing a SAN, and each virtual machine will exist in its own LUN on the array. I also have LUNs which house data that needs accessed by the VMs.
My goal is to have a VM (lets call it fs, for fileserver) started on VMhostA. I want fs to be able to access a data LUN in raw mode, meaning that if fs goes away, I want to be able to mount it as ext3 (or whatever) from a physical (non-virtual) machine. This means that the data on the LUN must not be housed in a vmdk file. In addition, I want to be able to use VMotion to move that VM to VMhostB (and obviously, maintain access to the data LUN)
It is my understanding that I need to present the LUN as a Raw Disk Mapping (RDM)? From what I have read, the RDM file created can be stored with the fs virtual machine or on another datastore. Would it be correct to assume that it should be stored with fs?
Also, there exists two types of RDM, it seems. There are physical and virtual modes. I have encountered conflicting documentation from various sources, so I'm not sure what to think. Does it matter which I select in this case? What are the differences, as they apply to my situation?
Thanks very much for reading all the way through ;-)
I don't have direct experience using RDMs in vSphere 4 ESXi (only in ESX 3.5), but everything you're saying re: what you want to accomplish with presenting your data LUN via RDM jibes with my knowledge. The mapping file is typically stored in the datastore with the virtual machine using it, as you surmise.
re: virtual RDM mode versus physical RDM mode - The background I'm finding seems to indicate that in a virtual/physical cluster you'll want to use physical mode. Having said that, I'm also finding documentation that indicates that nothing is written to the LUN other than the guest filesystem information even in virtual mode (see http://www.virtuallifestyle.nl/2010/01/recommended-detailed-material-on-rdms/). I would suspect that, since you're looking for the capability to mount the RDM LUN "cold" on another machine (i.e. not a true cluster application) that virtual mode will work fine for you.
Since virtual mode seems the most flexible, and gives you the most potential to use VMFS features on the LUN, I'd suspect that you'll want to try using virtual mode first.
My recommendation would be "test it out and see how you like it". Obviously, you being Matt Simmons, you're already doing that... >smile<
RDM's are definitely what you want to use and what you have outlined will work the way you intend.
The choice between Virtual and Physical mode depends on use case, and as you indicated Virtual Mode is the most flexible. Physical compatibility mode is only required when you need to use software within the Guest that requires direct access to the disk hardware - SAN management software for example. It's also required for mixed physical\virtual clustering like MSCS, probably for the same reason - I suspect (but I'm not certain) that all nodes in the cluster need to be able to talk directly to the same shared hardware in the same way. Physical compatibility mode has no effect on standard vMotion but it does prevent Storage vMotion, Snapshotting, cloning or Conversion of a VM into a template. All of those restrictions stem from the fact that a physical mode RDM can't support VMware snapshotting techniques.
An RDM in Virtual mode behaves exactly like a disk presented via a VMDK - Storage VMotion \ cloning \ Snapshots etc all work.
As to your other question about the files involved, an RDM is made up of two parts; a stub\mapping file (ending in xxx-rdm.vmdk) that contains meta data for the volume, and the actual raw LUN that the stub points to. The mapping file can be stored on any VMDK (or NFS share) visible to the ESXi host(s) but the most sensible place to put it is with the main VM files because it is really just a small text file (a few kB). Snapshot delta files will end up in the same place if you choose to use those and have selected Virtual Compatibility Mode so you should factor in capacity for those if you are going to do that.