Here's the issue...we have a network with a lot of Cisco switches.
Someone plugged in a hub on the network, and then we started seeing "weird" behavior; errors in communication between clients and servers, or network timeouts, dropping network connections, etc. It seemed that somehow that hub (or SOHO switch) was particularly freaking out our Cisco 3700 series switches.
Disconnect that hub or netgear-type SOHO switch and things settled down again.
We're in the process of trying to get a centralized logging server for SNMP and management, etc., to see if we can trap errors or narrow down when someone does this sort of thing without our knowledge because things seem to work, for the most part, without issue, we just get freaky oddball incidents on particular switches that don't seem to have any explanation until we find out someone decided to take matters into their own hands to expand available ports in their room.
Without getting into procedure changes or locking down ports or "in our organization they'd be fired" answers, can someone explain why adding a small switch or hub, not necessarily a SOHO router (even a dumb hub apparently caused the 3700's to freak out) sending DHCP request out, will cause issues? The boss said it's because the Cisco's are getting confused because that rogue hub/switch is bridging multiple MAC's/IP's into one port on the Cisco switches and they just choke on that, but I thought their routing tables should be able to handle multiple machines coming into the port. Anyone see that behavior before and have a clearer explanation of what's happening?
I'd like to know for future troubleshooting and better understanding that just waving my hand and saying "you just can't".
Here's a show run
Current configuration : 25591 bytes
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname ###########
!
boot-start-marker
boot-end-marker
!
enable secret 5 ############
!
!
!
no aaa new-model
switch 1 provision ws-c3750g-24ps
switch 2 provision ws-c3750-48ts
switch 3 provision ws-c3750-48ts
switch 4 provision ws-c3750-48ts
switch 5 provision ws-c3750-48ts
system mtu routing 1500
authentication mac-move permit
ip subnet-zero
ip routing
!
!
!
mls qos map policed-dscp 24 26 46 to 0
mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos srr-queue input bandwidth 90 10
mls qos srr-queue input threshold 1 8 16
mls qos srr-queue input threshold 2 34 66
mls qos srr-queue input buffers 67 33
mls qos srr-queue input cos-map queue 1 threshold 2 1
mls qos srr-queue input cos-map queue 1 threshold 3 0
mls qos srr-queue input cos-map queue 2 threshold 1 2
mls qos srr-queue input cos-map queue 2 threshold 2 4 6 7
mls qos srr-queue input cos-map queue 2 threshold 3 3 5
mls qos srr-queue input dscp-map queue 1 threshold 2 9 10 11 12 13 14 15
mls qos srr-queue input dscp-map queue 1 threshold 3 0 1 2 3 4 5 6 7
mls qos srr-queue input dscp-map queue 1 threshold 3 32
mls qos srr-queue input dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
mls qos srr-queue input dscp-map queue 2 threshold 2 33 34 35 36 37 38 39 48
mls qos srr-queue input dscp-map queue 2 threshold 2 49 50 51 52 53 54 55 56
mls qos srr-queue input dscp-map queue 2 threshold 2 57 58 59 60 61 62 63
mls qos srr-queue input dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31
mls qos srr-queue input dscp-map queue 2 threshold 3 40 41 42 43 44 45 46 47
mls qos srr-queue output cos-map queue 1 threshold 3 5
mls qos srr-queue output cos-map queue 2 threshold 3 3 6 7
mls qos srr-queue output cos-map queue 3 threshold 3 2 4
mls qos srr-queue output cos-map queue 4 threshold 2 1
mls qos srr-queue output cos-map queue 4 threshold 3 0
mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47
mls qos srr-queue output dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
mls qos srr-queue output dscp-map queue 3 threshold 3 16 17 18 19 20 21 22 23
mls qos srr-queue output dscp-map queue 3 threshold 3 32 33 34 35 36 37 38 39
mls qos srr-queue output dscp-map queue 4 threshold 1 8
mls qos srr-queue output dscp-map queue 4 threshold 2 9 10 11 12 13 14 15
mls qos srr-queue output dscp-map queue 4 threshold 3 0 1 2 3 4 5 6 7
mls qos queue-set output 1 threshold 1 138 138 92 138
mls qos queue-set output 1 threshold 2 138 138 92 400
mls qos queue-set output 1 threshold 3 36 77 100 318
mls qos queue-set output 1 threshold 4 20 50 67 400
mls qos queue-set output 2 threshold 1 149 149 100 149
mls qos queue-set output 2 threshold 2 118 118 100 235
mls qos queue-set output 2 threshold 3 41 68 100 272
mls qos queue-set output 2 threshold 4 42 72 100 242
mls qos queue-set output 1 buffers 10 10 26 54
mls qos queue-set output 2 buffers 16 6 17 61
mls qos
!
!
!
!
!
spanning-tree mode pvst
spanning-tree etherchannel guard misconfig
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
!
class-map match-all AutoQoS-VoIP-RTP-Trust
match ip dscp ef
class-map match-all AutoQoS-VoIP-Control-Trust
match ip dscp cs3 af31
!
!
policy-map AutoQoS-Police-CiscoPhone
class AutoQoS-VoIP-RTP-Trust
set dscp ef
police 320000 8000 exceed-action policed-dscp-transmit
class AutoQoS-VoIP-Control-Trust
set dscp cs3
police 32000 8000 exceed-action policed-dscp-transmit
!
!
!
!
interface GigabitEthernet1/0/1
switchport trunk encapsulation dot1q
switchport trunk native vlan 11
switchport mode trunk
spanning-tree portfast
!
!
!
interface GigabitEthernet5/0/1
!
interface GigabitEthernet5/0/2
!
interface GigabitEthernet5/0/3
!
interface GigabitEthernet5/0/4
!
interface Vlan1
ip address ############## 255.255.0.0
!
ip classless
ip route 0.0.0.0 0.0.0.0 ##############
no ip http server
no ip http secure-server
!
!
ip sla enable reaction-alerts
!
!
!
line con 0
line vty 0 4
password 7 ############
login
line vty 5 15
password 7 ############
login
!
end
We operate a couple network implementations where third-party connections are linked up to a centralized Cisco backbone (i.e. multi-tenant setup). I can say I've seen a bunch of diverse (okay, ghetto) devices connected up to the Catalyst platform, and if there's one thing I've learned, it's that the Cisco platform is remarkably resilient to these kinds of things.
There is one achilles heel, though - A cheap hub in the right configuration can easily bring down an entire network with a broadcast storm, and it's not even the Cisco platform's fault. I discovered this in a production configuration, and the only real research I did was finding the closest trash can for that hub, but here's how it happened:
Everything runs smoothly until that workstation sends out a broadcast announcement. While the hub and the Cisco are smart enough to prevent a broadcast storm on other broadcast packets, the hub isn't smart enough to detect that two of its ports are connected to each other, and will use up almost 100% of its processing power to broadcast that packet in an infinite loop back and forth, as well as out all the other ports (i.e. the uplink to your Cisco).
If this is the case in your configuration, you will notice that across your network, all of the ports on that broadcast VLAN will go nuts, up until the hub can't sustain the capacity and drops the magical looping packet (could be a couple minutes depending on the competing traffic), and then all is back to normal.
In this situation SNMP won't help you since all the ports on that VLAN go crazy with traffic. However, Wireshark is your friend here, since it's easy to capture which IP (and sometimes machine name if it's a broadcast packet) caused the loop, and quickly locate the offending device.
May not be the exact case you're experiencing, but this one bit us hard and might give you some ideas to research with your situation.
Trivial points, but I haven't seen them mentioned yet:
Make sure your switch ports aren't forced to full duplex because they won't work with hubs if they are. Come to think of it - if their duplex or their speeds are forced they likely won't connect correctly to a Netgear type switch which wants to auto-negotiate
Make sure your users don't connect the switch to two network outlets "to double their bandwidth".
Cisco switches used to (and maybe still do - I'm out of that business now) have a feature called "Faststart". For any port with Faststart enabled the switch would not do a full spanning tree analysis and by enabling the feature the admin "promised" not to connect a switch or hub to that port. The reason for doing this was to avoid the client PC's DHCP request timing out before the Cisco decided it was safe to enable the port. You may want to look at that also. (If I have totally mis-remembered all this I apologise in advance and hope someone with more up-to-date knowledge will correct me)
If you're using port security on the ports, you'd end up with problems whenever there's more than the expected number of MACs on a given port. The main problems with sticking hubs into a switched network is that you end up with larger collision domains (instead of "switch + end unit", you end up with "switch + everything downstream on the hub"), you can cause switching loops (if a hub is ever connected to two switches) and you may end up with a network that's large enough that the broadcast domain is "too wide" (there's an absolute requirement for a LAN to be small enough that packets from one extreme to the other will be seen within the Ethernet pre-amble; as hubs tend towards being worse at letting packets through at pace (being cheaper and all), that can end up being violated).
If you have a larger collision domain, anything that tries to talk to/from that collision domain will end up having lowqer throughput.
If you have switching loops, you risk broadcast storms and ports randomly not forwarding traffic as Spanning Tree shuts down forwarding ports.
If you have a too-wide broadcast domain, you'll end up having spurious re-transmits as you get late collisions.
this is a classic switching problem.
if you do the cisco ccnp or just study spanning tree protocol the answer becomes obvious.
spanning tree prevents loops-> it does this by initially determining a "root bridge"->
this is determined by the lowest switch priority-> if there is a tie then mac addresses of the switches are compared-> often old switch equates to lower mac address-> all traffic on the network goes through the old switch!!!! = crash fixed by setting your fastest/and most centrally connected switch with a hard-coded priority that is lower than the default.
by default cisco use per-vlan-spanning-tree (pvst+) command: spanning-tree vlan 1-666 priority 24576
i recommend using rapid-spanning-tree "spanning-tree mode rapid-pvst" which basically speeds things up a lot
from wikipedia (http://en.wikipedia.org/wiki/Spanning_tree_protocol)
The root bridge of the spanning tree is the bridge with the smallest (lowest) bridge ID. Each bridge has a unique identifier (ID) and a configurable priority number; the bridge ID contains both numbers. To compare two bridge IDs, the priority is compared first. If two bridges have equal priority, then the MAC addresses are compared. For example, if switches A (MAC=0200.0000.1111) and B (MAC=0200.0000.2222) both have a priority of 10, then switch A will be selected as the root bridge. If the network administrators would like switch B to become the root bridge, they must set its priority to be less than 10.
I had this happening in a non Cisco Environment. A hub was plugged into a network port which then had three computers connected to it. One user, no issues. When everyone else came in to play it started killing RDP connections to the Terminal Server. Pull the hub out and problem goes away.