Home / server / Questions / 974643
In Process
Hagen von Eitzen
Asked: 2019-07-11 00:40:48 +0800 CST

VMware volume has fluctuating capacity

  • 772

We are running four ESXi 6.5.0 VMware hosts, managed vie vSphere web client.

Recently, I added a new VMFS 6 volume in our storage, originally created with 2TB capacity, but quickly extended to 3TB before actually taking it to production. Ever since, I do not get consistent volume cacpacity readings. For example: A guest system had a disk of 1700GB originally on a different volume; I migrated that to the new volume, and after that try to enlarge it. So in change settings, the disk showed as 1700GB and maximum size 2.77TB - but when I attempted to set it to 2000GB, I got an alarm "insufficient disk capacity", and - lo and behold - the maximum size had gone down to 1.86TB! With this, I managed to increase my 1700 to 1900GB only ...

When checking the volume info under "configure" - "general" - "capacity", it sometimes shows 1.86TB as capacity (and allocated), but sometimes without anything special happening in-between it shows 2.77TB (with 1.86TB allocated). At times when the lower capacity is shown, clicking "update capacity" changes nothing.

Under "device backing", it shows 2.77TB (as far as I can tell without fluctuation?).

Under "monitor" - "performance" - "overview", I sometimes see 2.77 and sometime 1.86TB as well.

And so on ...

Each of the four hosts shows the volume correctly as LUN 10 = 2.77TB.

Strangely, the event log for the volume shows several entries "Capacity of ... enlarged from 2047894093824 bytes to 3047816167424 bytes". More specifically, I have the "volume created" event at 2019-07-05 11:44:16 and those "capacity extended" events at 2019-07-05 13:33:44, 2019-07-05 13:37:04, 2019-07-05 15:23:28, 2019-07-05 15:53:40, 2019-07-05 16:23:41, 2019-07-05 17:19:30, 2019-07-10 07:58:05, 2019-07-10 09:58:01. That's eight times up to now, way more than e.g. once per host, and mostly (but not completely) correlated with times when I have vsphere open, all in all not really explicable to me.

Questions:

  • What can be the cause of these observed fluctuations?
  • How can it be mended?
  • Once mended, how can I then be sure that it has been mended for good (I do not want to extend a disk an then be informed at a random moment that pat of the disk does not "exist")?
vmware-esxi

3 Answers

  1. I have a few ideas, I can't confirm whether they're actually what you're seeing though:

    1. What version of VMFS are you running on each system? Older versions appear to handle large files (near 2tb) differently than newer ones.
    2. Are there any volume healing operations you can try? It could be that the metadata on one or more of the disks is out of date and reports the wrong values when VMware tries to read it.
    3. How big are the individual storages used as VMFS backend? Are they large enough to hold a single 2TB file? I'm not sure if VMFS can distribute single VM disks across multiple nodes.
    4. I actually don't know VMFS that well, but I do know that ZFS and BTRFS (which are Copy on Write filesystems) are sometimes unable to precisely tell you the size of individual files on disk due to cluster/inode slack, snapshots, metadata, and other things that may not be counted when just querying the file size itself. With ZFS you can do zfs list -o space to show exactly what's used for what, perhaps VMFS has some similar option?

    Here's the wikipedia page of VMFS for reference, note that it has a bunch of constraints especially in older versions that may be relevant to your use case. https://en.wikipedia.org/wiki/VMware_VMFS

    Hope this helps!

    • 1

  2. Could your server be swapping memory into disk? Vmware has an option to disable this: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.resmgmt.doc/GUID-C0DBB2A3-4B44-45BE-927E-10EEB2EB8CBE.html

    • 0

  3. There is no variation.

    It depends how space is interpreted: as fake space in base 10 or as true space in base 2. HDD manufactures list the space in fake-space, in base 10 and in a similar manner some programs read it.

    What that means for you: your fake-3TB volume actually has 3 000 000 000 000 B, which in true binary space means 2.72TB. If you want real 3TB, you will have to make a volume in size of 3 221 225 472 000 B.

    In a similar manner, a 2TB base 10 capacity will actually have 1.81TB.

    Your volume of 2047894093824 B is under 2TB. That would best translate as 2 000 000 000 KB which is 1.86TB. If you want 2TB, make it have 2199023255552 B (or 2200000000000 if it's easier for you).

    Your allocation of 1700GB in TB has the size of 1.66TB. 1900G is 1.8555TB so that is why it worked.

    To eliminate the confusion it was officially proposed a while ago that the binary true values to be stated in KiB/MiB/GiB/TiB and the 10-base ones with the old names, but many did not agree with this implementation of naming.

    Therefore, if you create volumes make sure you take into account the binary values, not the 10-base translation of them.

    If you create a 2048GB volume it will actually have exactly 2TB if the GB value you state is in binary format and will still have less if it's not.

    • -1