My hostd.log file on vSphere 5 says that partedUtil is timing out while trying to write the partition table on an iSCSI target being served from OpenIndiana/napp-it.
I read that sized over 2TB give problems, so I sized the volume to 500GB.
I tried mounting the target in Ubuntu 11.04 with open-iscsi and it worked, so this seems to be an issue with the vSphere implementation of iSCSI or not?
iSCSI is a standard protocol, and ESXi 5 is standard a iSCSI initiator, so it should be compatible with any iSCSI target; there really is not such a thing as a "compatible iSCSI target".
If we are instead talking about supported iSCSI targets, then you should have a look at the VMware HCL: http://www.vmware.com/resources/compatibility/search.php?deviceCategory=san. Anything that's not there should work, but it's not guaranteed to, and may require some tweaking or even not work at all.
there are considerations with VMware iSCSI. Take openfiler/IET for example.