I am building a small setup of ESX5 servers. There will be different apps running for different companies on them. Each company has a VM that has one Internet facing vNIC and one vNIC facing to this company's private virtual network (backend). The backend networks are for separate web server and database VMs.
The question is: would it be performance and/or security wise better to create a separate virtual switch for each company or just use VLAN tags? I mean if all the private networks are load-wise fully utilized, would it make a difference if they used separate virtual switches?
Separate vSwitches will require separate physical uplinks. If you can afford the NICs to do this it might be a good way to go. If there is a need to QoS an individual company's data at the physical switch as this is not something VMware's vswitch is really capable of, then this also might give you some flexibility as you'll know which company's data is coming from which physical cables more predictable. If you expect traffic to stay low, this may not be an issue.
To answer your question more directly: Yes, if you put the companies on to separate vSwitches then it will not really be possible for one to talk to the other, but VLANS will give you almost the same isolation. I'd have to see a better diagram, specifically regarding the "Internet connection" vNic and how you plan to handle that as far as subnetting goes to better answer your question.
I'd use VLAN's because you can actually separate traffic even when it is crossing multiple switches. Plus VLAN's will be lighter than another virtual-switch. VLANs allow sharing a switch among more than one LAN by filtering and limiting broadcast traffic.
I think using VLAN's is the way go.
In general you want seperate vswitches. Each switch would have at least 1 uplink. In high traffic networks you can run into nic contention on the physical nic.