What are some pitfalls or lessons learned after converting existing hardware to a virtualized environment? Is there anything you tried to virtualize but will never do again?
ALWAYS eject any virtual media (CD/DVD/Floppy) once you've finished as failure to do so will often stop a vMotion in its tracks.
Get your NTP and DNS setup correctly, this will save you from contemplating suicide :)
You can never have enough memory, or storage.
Make sure you have remote, OS-less, access to your machines such as HP's iLO system.
Keep a repository of OS/App .ISO files.
Not a direct answer to your question but in the hope that someone saves themselves from tearing their hair out in the future by finding this response - HP blade servers don't ship with their 'VT'-bit enabled by default, you have to enable it in the BIOS (F9). Without this ESX 3.5U4 doesn't throw a useful error, no it just hangs prior to the code install :(
To answer the question as asked - pitfalls related to P2V migrations.
First off - P2V migrations work very well for the most part. The cleaner and newer the systems the better but even with migrating old (NT4 systems) my success rate after more than a hundred migrations in a range of environments has been around 90%. That's systems that have migrated and been handed over to production on the day (well mostly on the night) planned. I've only ever had one system that we had to reverse out of after an apparently successful migration - a SQL box that needed more CPU horsepower than the platform could ever deliver. VMware Converter is good and free (for the non enterprise version), Platespin is very good (but costly).
That said - there are things to avoid.
MSCS Clusters. You can make them work but it's never a great idea and Microsoft will almost certainly not help you in any way if you have problems later. Build new stand alone systems instead.
Large SQL servers - emphasis on large. These should have been red flagged from a CPU requirements POV in advance but don't be tempted to move one if you aren't certain that the target VM will have ample CPU headroom.
If you are planning to change system names or ip-addresses (or both) during the migration then first consider don't doing that and if you absolutely have no choice then make sure you have people on hand who understand how those changes might affect the systems in question. My worst migration ever was an RSA ACE server used for authenticating a DMZ located VPN where the client refused to listen to my objections and insisted on changing both name and ip-address during the migration.
Related to the above - if you have anything other than a completely flat network then build some test VM's and make 100% sure that your VM networks perfectly replicate the physical ones you are migrating from.
In Windows AD environments always make sure you have a local admin account on the box being migrated. And test it before you migrate.
Make sure you have a good idea of how long things will take. P2V copy times will vary depending on available network bandwidth (obviously) but also can be dramatically affected by the number of files in each volume being migrated. This is particularly a problem with Platespin migrating NT4* systems but will affect any P2V software copying at the file level (which generally applies if you opt to resize volumes). Copy rates of 70-80Megabyte a second are possible with GigE networks, relatively fast source and a good target setup but 20-30Megabyte/sec is more typical and for the aforementioned NT systems with 100Meg networks and lots of files I've seen copy rates drop down into the 50kilobyte/sec range.
Ideally you would get rid of these but some people don't have that luxury and getting such an OS of the completely unrepairable hardware its probably running on is almost always a good idea.
Have a solid backup strategy in place beforehand. Decide if you are going
to back up the VM as if it were on bare metal, or if you are going to
back up the virtual hard disks on the data stores (or both). Generally
speaking, I found that my required backup footprint increased
significantly at first, so be prepared for an initial spike
where you might be backing up both an old physical machine and a new VM
before you get all of the kinks worked out.
VM sprawl is also something to watch out for. Once virtualization starts taking off, the urge to move everything to VM becomes great. While this can work, you probably didn't order exactly enough hardware right off the bat.
I think there are machines that can't be converted, and other machines that
probably shouldn't be converted. While it is nice to be able to take a
10 year old physical machine and clone it to a VM, with warts and all,
there are certainly scenarios where you would be better off building a
clean OS and migrating objects from the physical machine over. Sometimes
you are better off not converting over the cobwebs.
Be prepared to use a lot of network ports. If you have systems that run on different VLANS, while single ports could be trunked, you probably want to have individual ports for your VLANs feeding into your vSwitch. If you want redundancy, and you are using iSCSI, you could be looking at a lot of NICs.
From my experience, be VERY careful about your storage medium. We went with an iSCSI SAN that turned out to only support 100Mbit connections. Running one VM on the system wasn't bad, two was less the adequate... and by the time we reached our target of 8 VMs they were horrible.
My personal lesson learned: Check the rated IOPS and read more reviews about a product that relate to the way you intend to use the storage device
Another handy thing I've learned... Making a 'backup' disk image after a base install and hardening will quicken the build of any other system and is a very handy thing to keep around.
Try not to run production database servers in a Virtual Environment. The overheads for I/O is unacceptable. We had huuuge problems when our DBA allowed our primary MSSQL server to become virtualised. Queries were taking thousands of milliseconds to run. When we convinced them to move it back to a dedicated box, there was a 10,000% increase in throughput and speed.
VMWare Converter creates virtual machines that boot from scsi. MS virtual machines cannot boot from scsi. [edit - apparently version 4 of the converter now lets you specify SCSI or IDE, I love those guys]
If you're going to virtualize a non-ACPI physical machine, buy some software for the purpose. (unless you have a couple of weeks for an exciting journey in discovery!)
Also, VMWare Converter will tackle jobs at which MS SCVMM will throw up its hands in dispair.
Bring a lot of RAM.
Don't do anything until the virtualization tools (whether VMWare or MS) have installed.
If you're going to move it to another platform/version, uninstall the aforementioned tools.
Mind your CPU limits. P2V of a 2 CPU windows 2000 taught me that only 1 is supported.
In case you don't have one already - Have a complete backup of the physical machine prior to the migration. An image is probably best or and an ASR/system restore or whatever that gives you a complete system snapshot, instead of the usual content backup that most machines have.
P2V tools can backfire on you unexpectedly, ruining the physical machine (i had VMWare converter kill a machine i was trying to P2V once, luckily it was just a test migration). Be prepared to have to restore the system from scratch. Yes, this is maybe a 1000 to one chance, but do YOU want to be that one?
If you're going to use SAN to store your VM images, make sure that you label your devices and hosts VERY CLEARLY. Removing host-to-disk mappings on the SAN does terrible things if they are still in use by VMs.
Microsoft won't support Exchange 2003 running in VMware (at least that was the officially response). With a lot of arm twisting we were able to get some unofficial support from them, but it caused extra headaches in an already stressful crisis.
ALWAYS eject any virtual media (CD/DVD/Floppy) once you've finished as failure to do so will often stop a vMotion in its tracks.
Get your NTP and DNS setup correctly, this will save you from contemplating suicide :)
You can never have enough memory, or storage.
Make sure you have remote, OS-less, access to your machines such as HP's iLO system.
Keep a repository of OS/App .ISO files.
Not a direct answer to your question but in the hope that someone saves themselves from tearing their hair out in the future by finding this response - HP blade servers don't ship with their 'VT'-bit enabled by default, you have to enable it in the BIOS (F9). Without this ESX 3.5U4 doesn't throw a useful error, no it just hangs prior to the code install :(
To answer the question as asked - pitfalls related to P2V migrations.
First off - P2V migrations work very well for the most part. The cleaner and newer the systems the better but even with migrating old (NT4 systems) my success rate after more than a hundred migrations in a range of environments has been around 90%. That's systems that have migrated and been handed over to production on the day (well mostly on the night) planned. I've only ever had one system that we had to reverse out of after an apparently successful migration - a SQL box that needed more CPU horsepower than the platform could ever deliver. VMware Converter is good and free (for the non enterprise version), Platespin is very good (but costly).
That said - there are things to avoid.
MSCS Clusters. You can make them work but it's never a great idea and Microsoft will almost certainly not help you in any way if you have problems later. Build new stand alone systems instead.
Large SQL servers - emphasis on large. These should have been red flagged from a CPU requirements POV in advance but don't be tempted to move one if you aren't certain that the target VM will have ample CPU headroom.
If you are planning to change system names or ip-addresses (or both) during the migration then first consider don't doing that and if you absolutely have no choice then make sure you have people on hand who understand how those changes might affect the systems in question. My worst migration ever was an RSA ACE server used for authenticating a DMZ located VPN where the client refused to listen to my objections and insisted on changing both name and ip-address during the migration.
Related to the above - if you have anything other than a completely flat network then build some test VM's and make 100% sure that your VM networks perfectly replicate the physical ones you are migrating from.
In Windows AD environments always make sure you have a local admin account on the box being migrated. And test it before you migrate.
Make sure you have a good idea of how long things will take. P2V copy times will vary depending on available network bandwidth (obviously) but also can be dramatically affected by the number of files in each volume being migrated. This is particularly a problem with Platespin migrating NT4* systems but will affect any P2V software copying at the file level (which generally applies if you opt to resize volumes). Copy rates of 70-80Megabyte a second are possible with GigE networks, relatively fast source and a good target setup but 20-30Megabyte/sec is more typical and for the aforementioned NT systems with 100Meg networks and lots of files I've seen copy rates drop down into the 50kilobyte/sec range.
clean OS and migrating objects from the physical machine over. Sometimes you are better off not converting over the cobwebs.
From my experience, be VERY careful about your storage medium. We went with an iSCSI SAN that turned out to only support 100Mbit connections. Running one VM on the system wasn't bad, two was less the adequate... and by the time we reached our target of 8 VMs they were horrible.
My personal lesson learned: Check the rated IOPS and read more reviews about a product that relate to the way you intend to use the storage device
Another handy thing I've learned... Making a 'backup' disk image after a base install and hardening will quicken the build of any other system and is a very handy thing to keep around.
Use redunant network for vmotion/vmkernel traffic. You don't want virtual machines shutting down just because a switch rebooted.
Oh, and leave one DC/DNS/DHCP-server out of the virtualization. Your users will hate you less if you get a major SAN crash.
Try not to run production database servers in a Virtual Environment. The overheads for I/O is unacceptable. We had huuuge problems when our DBA allowed our primary MSSQL server to become virtualised. Queries were taking thousands of milliseconds to run. When we convinced them to move it back to a dedicated box, there was a 10,000% increase in throughput and speed.
VMWare Converter creates virtual machines that boot from scsi. MS virtual machines cannot boot from scsi. [edit - apparently version 4 of the converter now lets you specify SCSI or IDE, I love those guys]
If you're going to virtualize a non-ACPI physical machine, buy some software for the purpose. (unless you have a couple of weeks for an exciting journey in discovery!)
Also, VMWare Converter will tackle jobs at which MS SCVMM will throw up its hands in dispair.
Bring a lot of RAM.
Don't do anything until the virtualization tools (whether VMWare or MS) have installed.
If you're going to move it to another platform/version, uninstall the aforementioned tools.
Mind your CPU limits. P2V of a 2 CPU windows 2000 taught me that only 1 is supported.
In case you don't have one already - Have a complete backup of the physical machine prior to the migration. An image is probably best or and an ASR/system restore or whatever that gives you a complete system snapshot, instead of the usual content backup that most machines have.
P2V tools can backfire on you unexpectedly, ruining the physical machine (i had VMWare converter kill a machine i was trying to P2V once, luckily it was just a test migration). Be prepared to have to restore the system from scratch. Yes, this is maybe a 1000 to one chance, but do YOU want to be that one?
If you're going to use SAN to store your VM images, make sure that you label your devices and hosts VERY CLEARLY. Removing host-to-disk mappings on the SAN does terrible things if they are still in use by VMs.
Microsoft won't support Exchange 2003 running in VMware (at least that was the officially response). With a lot of arm twisting we were able to get some unofficial support from them, but it caused extra headaches in an already stressful crisis.