I have been wanting to experiment with Windows clustering systems in my spare time, so I need a free, preferably open source, iSCSI Target that can support 2k3 and 2k8 fail-over and possibly High availability clustering. I have tried the ubuntu iscsi target package in a vmware environment, but it fails at the 2k8 tests.
In simple terms I want to build a "Poor man's San" for windows and have it be able to do more than just be drives.
Take a look at Starwind Software's solution. they have a free version which I use in my lab all the time. I think they also have a trial of there bigger version which I know for sure supports clustering.
As a general rule, I'd be leery of running an iSCSI target inside a VM. We're not quite there yet with x86-VMs to be certain of timing. Storage drivers tend to be more sensitive to the sort of network latencies NIC drivers have been dealing with for decades, and VMs tend to increase variability of that sort.
This isn't to say it can't be done. Rather, it probably shouldn't be done unless you have a deep understanding of your environment.
You're looking for SCSI-3 persistent reservations. There was some discussion here that an add-in for OpenFiler would be made available as a commerical add-in.
Open-E DSS Lite will do what you want. The no-cost version will work with up to 2TB of storage. Give that a shot.
See the answer to this question for an exhaustive list of options. The summary: there are several free to low cost solutions, but no open source.
Another option if you are already running VMware is to let it provide the same disk to the virtual machines as per VMware Virtual Machine Clustered Filesystem Setup
The VMs will just see it as a common storage disk over SCSI (not iSCSI).