Sizing Switches for Storage and Production

1) Should you always completely separate the storage network switches from production switches or are VLANs fine to segment this traffic? Is there a golden rule here?

There is no golden rule here. Physical switch segregation happens when the storage network demands more features/reliability than the data-network environment can provide. But if the data network stack is robust enough, a VLAN is all that's needed to segregate.

If you're dropping in a new storage network, you go with physical separation for two major reasons:

1. Your Security posture demands that such traffic be segregated. You'll probably do this if your DMZ networks are on physically separate hardware.
2. The demands of the storage network (jumbo packets, backplane bandwidth, etc) can't be satisfied with what you already have.

2) How do you properly size a switch for your environment based on the specifications the manufacturer provide (Throughput, Forwarding Throughput, Stacking Throughput, Max Mac)?

You determine the requirements of your network, and match those to the specifications. Kinda broad, but that's what you do. Things to pay attention to in your modeling:

1. Port density in physical space. Datacenters will require higher port densities than office spaces.
2. Port-interconnectivity in network access. An office switch will probably have very little peer-to-peer communication and be almost exclusively upstream to other networks. Where datacenter networks can be strongly peer-to-peer. For networks with a lot of peer communication you want higher backplane bandwidths, and for L2/3 switches faster switching methods.
3. Switch special features. If you're going to need add-on features like WACP, 802.11x, port access rules, or other such things, get a switch that can do it. The more features you need, the beefier the CPU you'll need.
4. Size of L2 networks. Switches need to track MAC addresses, and for each L2 network they're on (VLAN) they'll need to keep abreast of where clients are located on each network. Big central switches will need big MAC tables.

3) If you have two switch options and one has a maximum Mac address of 8,000 vs. another with 16,0000. What does this really mean to me? How do make sure one vs. another is sized properly for me?

How big are your L2 networks? You need one MAC address for each network interface (physical or virtual) on a network, and the switch needs to track all of those so it knows how to forward packets through its switching infrastructure. An 8K MAC table means that it can accommodate L2 networks with up to about 8K clients on them. That's all.

4) Besides VLAN and Jumbo Frame support, is there any other "Must" haves for a virtual environments production or storage networks?

Storage networking protocols are less tolerant of out-of-order arrival than other TCP protocols, so engineering to prevent such is much more important than it is for general data networks. There is no bullet-point on the feature list to look for, though.

Storage stacks are very sensitive to latency, so minimize that. If you're in a large datacenter with a high port-density, you'll probably want to go with a chassis+blade style switch over stackables as they tend to be lower latency between peers (sub-milisecond latencies do add up).

Large virtualization hosts generally can take advantage of trunking (LACP, 802.11ad, EtherChannel), which allows higher per-host bandwidth utilization and multiple VLAN support inside the VM environment.