I have a pool of XenServer hosts running the Free version of XenServer 5.6 FP1. I was wondering if I change the network backend to use Open vSwitch if I can specify ACLs on individual network VIFs without needing to use the DVS appliance (distributed virtual switch) which requires an Advanced License or higher.
Basically I'm looking for a way to isolate VMs on my network so that if a user had root access on the command line they couldn't access other servers they should not be able to (without using a VLAN).
Peter is right, you can use the "ovs-ofctl" to add ACLs. The problem you'll have is that every time you restart or migrate a VM, the VIFs will be connected to fresh switch ports and will lose their configuration. You could try customising /etc/xensource/scripts/vif -- this script is responsible for adding a VIF to a vswitch port... this would be the ideal port to re-add ACLs.
Open vSwitch supports sFlow traffic monitoring that you can use to detect suspicious activity and manage XenServer network and system performance. The ovs-* commands are used to configure the vSwitch, it looks like you can use ovs-ofctl to add ACLs.