We're currently running a couple of continuous build/test agents in VMWare Player. They regularly fail due to timing/speed problems. That is: because there are two VMs on the same box, they contend for (e.g.) disk access and are massively slowed down.
Would upgrading to VMWare Workstation (or Server or ESXi) improve things, or should I request a physical box for one of the agents instead?
Unless you REALLY need the host client functionality you get with Player/Workstation/Server I'd always suggest going with ESXi - it's that much more functional, stable and performant. Obviously it's dedicated at that point but that's the price you pay.
ESX (i) is what is called a "type 1 hypervisor": a dedicated virtualization operating system running directly on the host hardware, with full access and control to hardware resources; Player/Workstation/Server are "type 2 hypervisors": they are applications running on an underlying operating system (Windows or Linux); they have to go through it to access hardware, and this is the main reason ESX (i) is definitely better; the difference in performances is simply astounding.
That said, if performance is critical, virtualization is clearly not the best option here. The two VMs are going to compete anyway for hardware resources, regardless of the virtualization system you're using (although ESX (i) makes a really better job at managing them); and unless more than two times the resources consumed by each VM are available on the host, they will not be able to run smoothly; this means, if each VM needs 2 cores and 2 GB of memory, the host will need at least 4 cores and 5 GB of memory (there is of course some virtualization overhead) in order to fulfill their requirements. About disk I/O, the best thing you can do is placing the two VMs on different physical disks; not partitions or volumes: you have to actually use different physical disks in order to have them not compete for disk I/O.
As Chopper says, the difference between ESX and VMWare player is stunning (as is, to be fair, the difference between HyperV server and Virtual PC). It's a fundamental change the in way virtualisation is delivered which greatly benefits the guest OSes.
However the only way to know if its fast enough for your needs is to test test test... and as much as I love virtualisation's ability to solve problems, it isn't always the answer to everything. We have a few "performance critical" tasks here that we don't see ourselves migrating to VMs any time soon.
I would say not only go with ESXi, but match it to what you are going to be running in production as close as possible. You'll have less surprises in the end.