Questions[vmware-vsphere](server)

This seems basic, but I'm confused about the patching strategy involved with manually updating standalone VMware ESXi hosts. The VMware vSphere blog attempts to explain this, but the actual process is still not clear to me.

From the blog:
Say Patch01 includes updates for the following VIBs: "esxi-base", "driver10" and "driver 44". And then later Patch02 comes out with updates to "esxi-base", "driver20" and "driver 44". P2 is cumulative in that the "esxi-base" and "driver44" VIBs will include the updates in Patch01. However, it's important to note that Patch02 not include the "driver 10" VIB as that module was not updated.

This VMware Communities post gives a different answer. This one contradicts the other.

Many of the ESXi installations I encounter are standalone and do not utilize Update Manager. It is possible to update an individual host using the patches make available through the VMWare patch download portal. The process is quite simple, so that part makes sense.

The bigger issue is determining what exactly to actually download and install. In my case, I have a good number of HP-specific ESXi builds that incorporate sensors and management for HP ProLiant hardware.

• Let's say that those servers start with an ESXi build #474610 from 9/2011.
• Looking at the patch portal screenshot below, there is a patch for ESXi update01, build #623860. There are also patches for builds #653509 and #702118.
• Coming an old version of ESXi (e.g. vendor-specific build), what is the proper approach to bring the system fully up-to-date? Which patches are cumulative and which need to be applied sequentially? Is installing the newest build the right approach, or do I need to step back and patch incrementally?
• Another consideration is the large size of the patch downloads. At sites with limited bandwidth, downloading of multiple ~300mb patches is difficult.

I am using VMware ESXi. In our team we use to provide snapshots for long term backup.

Then we faced issues like memory spillover and the server got hang up.

I started reading in VMware knowledgebase articles and everywhere. Everywhere it was recommended not to have snapshots for a long time.

Even VMware advised to keep snapshots for maximum of three days.

But our team kept asking us to have at least two permanent snapshots (till deleting the VM). Sometimes we may use the VM for a year).

1. one snapshot is for fresh machine state. (So when we complete testing an application, we will revert back to fresh state and install another application) (If I did not allow that, I may often need to host the VM.)

2. Next snapshot for keeping the VM in some state (maybe they would have found an issue and keep that state for some time. Or they may install prerequisites for the application and keep the machine ready for testing.)

Logically, their needs seems to be fair. But if I allow that, I am to permit them to hold the snapshots for long time. We are not using our VM as a mail server or database server.

Why is keeping snapshots for long time having an adverse effect?

Why are snapshots considered as temporary backups, not real backups?

The Problem

We have an issue with performance on an existing platform, so I'm turning to the hive mind for a second opinion on this. The performance issue so far relates to IOPS rather than throughput.

The Scenario

A blade centre of 16 hosts, each with 64GB of RAM. (It's a Dell M1000e w/ M610s, but that's probably not relevant) 500 VMs, all web servers (or associated web technologies such as MySQL, load balancers, etc), around 90% are Linux and the rest Windows. Hypervisor is VMWare vSphere. We need to provide host HA, so local storage is out. As such the hosts just have an SD card to boot.

A bit of background thinking

At the moment we are up to 6 hosts (the blade centre will be at full capacity in a years time at current growth) and we are running iSCSI to a Dell MD3220i w/ MD1220 for expansion.

Possible options we have considered, and immediate thoughts along with them:

• Spreading the VMs across NFS datastores, and running NFS storage that meets performance requirement for up to a given number of VMs. NFS seems cheaper to scale, as well as been abstracted a bit more than block level storage so we can move it around as needed.
• Adding more MD3220i controllers/targets. We are concerned though that doing this could have a negative effect somehow in how VMWare handles having lots of targets.
• Swapping all disks from Nearline SAS to SSD. This ought to entirely solve the IOPS issue, but has the obvious side effect of slashing our storage capacity. Also it's still very expensive.
• vSphere 5 has a storage appliance. We haven't researched this much, but it must work well?

The Question

What sort of storage would you run underneath all of that? It wouldn't need to scale to another blade centre, it would just need to provide relatively good performance for all of those VMs.

I'm not looking for "Buy SAN x because it's the best" answers. I'm looking for thoughts on the various SAN technologies (iSCSI, FC, FCoE, InfiniBand, NFS, etc), different types of storage (SATA, SAS, SSD), and methodologies for handling storage for 100s of VMs (Consolidation, Separation, Sharding, etc).

Absolutely any thoughts, links, guides, pointers etc are welcome on this. I'd also love to hear thoughts on the above options we'd already considered.

Many thanks in advance for any input!

Update 5th March '12

Some fantastic responses so far, thank you very much everyone!

Going by the responses to this question so far, I'm beginning to think the following route is the way:

• Tier the available storage to the VMWare cluster and place VM disks on suitable storage for their workloads.
• Potentially make use of a SAN that is able to manage the placement of data on to suitable storage automagically.
• Infiniband looks to be the most cost effective to get the required bandwidth with the hosts at full capacity.

It definitely sounds like it would be worth making use of the pre-sales services of a major SAN vendor to get their take on the scenario.

I'm going to continue to consider this problem for a while. In the mean time any more advise gratefully received!

UPDATE: I updated the title of the message, because I've recently seen more of these problems with this exact time amount of 17163091968s. This should help people investigating the symptoms to find this page. See my (self-)accepted answer below.

I have a bunch of 64-bit Ubuntu 10.04 LTS VM's in a VMware vSphere datacenter. VMware tools is installed (vSphere Client says "OK").

I've seen some of the VM's hang a few times with the following error in syslog. When checking the situation from vSphere, the console was black, and the "Reboot guest" command didn't do anything, so I had to power cycle the VM.

Dec  1 11:44:15 s0 kernel: [18446744060.007150] BUG: soft lockup - CPU#0 stuck for 17163091988s! [jed:26674]
Dec  1 11:44:15 s0 kernel: [18446744060.026854] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs xt_tcpudp iptable_filter ip_tables x_tables acpiphp fbcon tileblit font bitblit softcursor ppdev vga16fb psmouse parport_pc shpchp vgastate i2c_piix4 lp parport serio_raw intel_agp floppy mptspi mptscsih vmw_pvscsi e1000 mptbase
Dec  1 11:44:15 s0 kernel: [18446744060.026899] CPU 0:
Dec  1 11:44:15 s0 kernel: [18446744060.026900] Modules linked in: btrfs zlib_deflate crc32c libcrc32c ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs exportfs reiserfs xt_tcpudp iptable_filter ip_tables x_tables acpiphp fbcon tileblit font bitblit softcursor ppdev vga16fb psmouse parport_pc shpchp vgastate i2c_piix4 lp parport serio_raw intel_agp floppy mptspi mptscsih vmw_pvscsi e1000 mptbase
Dec  1 11:44:15 s0 kernel: [18446744060.026920] Pid: 26674, comm: jed Not tainted 2.6.32-30-server #59-Ubuntu VMware Virtual Platform
Dec  1 11:44:15 s0 kernel: [18446744060.026922] RIP: 0033:[<00007f92e03d2ce6>]  [<00007f92e03d2ce6>] 0x7f92e03d2ce6
Dec  1 11:44:15 s0 kernel: [18446744060.026930] RSP: 002b:00007fff6069b770  EFLAGS: 00000202
Dec  1 11:44:15 s0 kernel: [18446744060.026932] RAX: 00007f92e27e7e10 RBX: 00007f92e06d5e40 RCX: 0000000000020000
Dec  1 11:44:15 s0 kernel: [18446744060.026933] RDX: 00007f92e27e7e10 RSI: 0000000000020209 RDI: 0000000000000002
Dec  1 11:44:15 s0 kernel: [18446744060.026934] RBP: ffffffff81013cae R08: 0000000000000001 R09: 0000000000000000
Dec  1 11:44:15 s0 kernel: [18446744060.026935] R10: 00007f92e06d6398 R11: 0000000000000870 R12: 00000000000000c0
Dec  1 11:44:15 s0 kernel: [18446744060.026937] R13: 00007f92e299dca0 R14: 0000000000000020 R15: 00007f92e06d5e40
Dec  1 11:44:15 s0 kernel: [18446744060.026939] FS:  00007f92e105b700(0000) GS:ffff880009c00000(0000) knlGS:0000000000000000
Dec  1 11:44:15 s0 kernel: [18446744060.026940] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec  1 11:44:15 s0 kernel: [18446744060.026941] CR2: 00007ff12ea15000 CR3: 0000000267067000 CR4: 00000000000006f0
Dec  1 11:44:15 s0 kernel: [18446744060.026968] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec  1 11:44:15 s0 kernel: [18446744060.026989] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Dec  1 11:44:15 s0 kernel: [18446744060.026991] Call Trace:

(There is no trace - that's the last line.)

I don't seem to have the other errors anymore, but I'm quite sure the process mentioned above (jed) was different in the other dumps.

• What could cause this problem?

• How to prevent this from happening?

Some extra info:

• The value 17163091988 is a bit (pun intended) suspicious - it's 1111111111000000000000000000010100 in binary. Maybe the error was trying to say 20 seconds (10100 in binary)?

• I'm not sure whether the problem persists with the latest 10.04 kernel (2.6.32-35).

• I've also seen task ... blocked for more than 120 seconds problems - maybe they could be related?

• the vSphere client shows no alerts or migration tasks for the VM.

We are planning a slow migration from VMware (and third party apps) to open source alternatives (free would be great).

Basically, we want to start with some little cluster lab, then migrate the production environment (35+ ESX, 1500 VMs) in the future (X years, there is no hurry... yet)

Our bet is CentOS/Scientific Linux as the operating system of choice and KVM as the hypervisor.

The vCenter alternative we are thinking about, is Convirt, but we don't know if all the features we use in VMware will be supplied by Convirt (HA, DRS, clustering,...), or we should try some other alternatives (any ideas?)

The monitoring is being replaced by Nagios and the backup/replication will be replaced by some scripting magic.

So, is there anyone who can give us some advices, or in a similar situation?

PS.- This is my first question in serverfault, and my english level is not so good, but I hope the question is understandable.

PS2.- I forgot to mention that we provide also VDIs. And the alternative we've been thinking is Spice.