We have just moved our infrastructure from physical servers to virtual machines. Everything is running great and we are happy with the result of the move. We have identified one problem, and that is reading/writing performance. We have an application that compiles files and writes to disk. This is considerably slower on the new virtual machines compared to the physical machines.
Is there a performance bottleneck when writing to a virtual hard drive compared to a physical hard drive?
This depends on how your storage is accessed by the virtual hosts. Is it local storage in the hosts drive bays, or remote SAN storage over iSCSI/Fibre Channel?
What are the IO speeds you are seeing when benchmarking the physical and virtual instances of these servers?
What are the hardware specs and configuration (drives, controllers, cache, RAID levels, SAS, SCSI) of your previous physical servers, and your new vmware host servers?
There are some overheads to disk throughput on VMs, but not inherently big. It's much more likely to be a shift in how the storage is being made available, contention from multiple VMs accessing the same storage, or a change/problem in the underlying disk storage hardware (e.g. lack of write cache on the vmware hosts).
There's a whole bunch of different ways to improve your IO performance on vmware, but it'll depend on having more info about your current setup.
You'll see a performance loss, simply because there are now 4 machines worth of writes going to a single machines worth of disk performance. Good Virtualisation design allows for this issue, at the simplest level with multiple local drives, at a more sophisticated level with a SAN of some sort.
If you've implemented VMware server on a Windows host, there's also the potential issue of the virtual disks being scanned by the real time virus scanner, which destroys performance.
If you're after best performance, it'd be worth looking at implementing the free version of ESXi which avoids the overhead of a Linux or Windows OS intermediary between the guest OS and the hardware