I have a fairly common setup - vSphere and ESXi hosts using FreeNAS as the VM store. The servers can see each other (obviously) so I want to segregate system admin traffic and user traffic onto different VLANs, and restrict the management IPs on both boxes.
Configuring management access on ESXi is easy, but I can't figure how to do it on FreeNAS. At the moment the relevant FreeNAS config is that it has one active NIC (10G Chelsio) with IP of say 192.168.1.2, and no VLANs have been set up on the network yet. What I'd like is to do one or more of the following:
Create two VLANs, say 1 and 2, with any VLAN able to access sharing services on the sharing ports, but only VLAN 2 able to reach the admin IP/port
Create two IPs on the one NIC, say 192.168.1.2 and 192.168.1.3, with only 192.168.1.3 able to reach the management login.
Blocking the management access ports (80,443 etc) for VLAN != 2 and/or IP != 192.168.1.3.
As FreeNAS isn't a router or firewall it doesn't have much built in to do this, so I'm not sure how to go about doing these things. It can't be uncommon to have it directly connected to the general LAN, so I'm hoping there's a straightforward helpful answer to the above 3 approaches, so I can choose which works best for me and figure out how to combine them if useful.
One answer that would be most of what you are looking for would be to keep storage on one VLAN. It is a best practice to keep storage on a dedicated VLAN. In large enviornments, it should have dedicated switches, mainly due to the high throughput of storage hosting. This would leave you with at least three VLANs, user/general, management, and storage. You could allow the admins into storage and management as desired.
As far as the firewall rules go, I was assuming that you would do this through your router, firewall, or switch ACLs. Using my example names above, you would configure your User VLAN to be restricted from accessing the Storage or Management VLANs. The management VLAN could be allowed to communicate with the storage VLAN, for the purposes of managing the FreeNAS server.
I was assuming that you had another device to do the access control, since the question you asked was directed towards the configuration of the interfaces.
To give more specific help, it would be great if you added what hardware you are working with for switches, routers, or firewalls.
VMware's recommended best practices are to have separate switches and ports for the iSCSI SAN.
https://pubs.vmware.com/vsphere-65/topic/com.vmware.vcli.examples.doc/GUID-D9F3A3EC-4599-42B4-935D-4AD48017F0D4.html
In the above situation you would have a dedicated iSCSI vSwitch with it's own uplink. For best security the uplink would connect to a dedicated managed Layer 2 switch which also connects to the FreeNAS array. If you wanted to run the iSCSI connection on a dedicated VLAN you can then set the VLANs for the FreeNAS and the ESXi port group on the switch.
The FreeNAS should not need to be VLAN aware as the switch will be doing all the work.
If the FreeNAS is being run as a VM that autostarts on the ESXi device the above scenario would be the same with the additional step of adding the VM's iSCSI NIC to the same port group as the VMKernel NIC.