I work as a Network Admin in a big company. Lately we noticed an issue with our network infrastructure. Basically our network backend lies on a Catalyst as main L3 backend switch, and few Cisco Nexus switches as edge L2 switches, connected to that Catalyst.
The issue appears as we try to sniff traffic on one of our hosts - we then (always) see unicast traffic between other hosts.
I'll try to be more elaborate: Assuming I'm on the host 10.0.0.1, with mac MAC, I run the command -
tcpdump -i eth0 ether host not MAC and host not 10.0.0.1 and not broadcast and not multicast
I will always see traffic between other hosts.
I read a Cisco article about Unicast Flooding, however - the "phenomenon" occurs not only when passing between VLANs in our network, but also on the on the very same VLAN. Is it possible that it happens when passing between switches in the same VLAN (our VLANs span on many switches)? All switches are connected by a trunk to the Catalyst...
Any ideas?
Thanks.
Edit:
It seems that we found the source of our problems.
Basically, each time one of the switches gets a frame with a MAC address it doesn't recognize - it floods it to all ports. This is normal - and the way things should go. However, in our current settings, a MAC entry in the switch should "live" for 30 minutes. If a MAC was not seen for 30 minutes, it will be deleted from the switch until seen again. If a packet is sent to that MAC and it's not in the table - all ports will be flooded in order to find the destination MAC port (we expect to get an answer from one of the ports).
We found one of the destination MACs and looked for it in the switch MAC table. The table didn't contain the MAC while the network was flooded. We tried ARPing the address related to that MAC - and the flood stopped (as the MAC re-appeared in the MAC table).
However, after a few seconds, the MAC disappeared from the MAC table again and the flood started again.
It seems that the flood issue derives from an issue with the MAC tables on our switches. It seems as if they "forget" MAC addresses quickly than they should (MACs should stay for 30 minutes) and flood all packets with that MAC.