Is there a setting on a ESXi 5 vSwitch that will make it so DHCP broadcasts aren't forwarded to the rest of the network? I have a small micro server I'm testing some things out and I'd like a DHCP server to run for my VMs but don't want to change any other settings for the rest of the subnet not on the ESXi host.
No; vSwitches are switches, not firewalls (leaving out the Cisco 1000v, since that's a different ball of wax entirely).
If you need to isolate this network from your live network, then do so - make a new port group that's not attached to your production network.