I'm not our normal network guy... I've just been drafted to help with this issue, so please bear with me.
We have a fairly large (~4,000 devices?) network mostly comprised of HP Procurve gear. From time to time over the last two weeks, we've been getting some wicked broadcast storms that pretty much take down everything. I set up Wireshark to do 3MB dumps, and I caught some of this in the act this morning.
There are thousands of ping requests. They appear to come from the MAC addresses of our switches and APs, and are sent to an IPv4 multicast MAC. The source IP addresses do not match those of the switches and APs... they are IP addresses assigned to a few clients on the network. The destination IP address is always that of the gateway.
My read on this is that the firmware on these Procurve devices is either compromised or gone crazy... or someone is spoofing the addresses and causing this mess. Neither of which seem likely here, so I am asking if you have any ideas of other things to consider.
On another note, our network is not subnetted. (Yes, I know, I know... not my call, unfortunately.) Everything is flat.
Thank you for your time.
Edit: Below are two sequential packets from my capture. I posted the full Wireshark summary. I apologize that it is a bit messy, but it better explains what I am seeing.
No. Time Source Destination Protocol Info
16885 2.094869 1.2.41.194 1.2.32.250 ICMP Echo (ping) request
Frame 16885 (98 bytes on wire, 98 bytes captured)
Arrival Time: Aug 31, 2010 07:59:11.185552000
[Time delta from previous captured frame: 0.000123000 seconds]
[Time delta from previous displayed frame: 0.000123000 seconds]
[Time since reference or first frame: 2.094869000 seconds]
Frame Number: 16885
Frame Length: 98 bytes
Capture Length: 98 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:icmp:data]
[Coloring Rule Name: ICMP]
[Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: Procurve_44:0f:26 (00:1f:fe:44:0f:26), Dst: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Destination: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Address: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: Procurve_44:0f:26 (00:1f:fe:44:0f:26)
Address: Procurve_44:0f:26 (00:1f:fe:44:0f:26)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol, Src: 1.2.41.194 (1.2.41.194), Dst: 1.2.32.250 (1.2.32.250)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 84
Identification: 0x7718 (30488)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 2
[Expert Info (Note/Sequence): "Time To Live" only 2]
[Message: "Time To Live" only 2]
[Severity level: Note]
[Group: Sequence]
Protocol: ICMP (0x01)
Header checksum: 0xd710 [correct]
[Good: True]
[Bad : False]
Source: 1.2.41.194 (1.2.41.194)
Destination: 1.2.32.250 (1.2.32.250)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0 ()
Checksum: 0xf112 [correct]
Identifier: 0xdffd
Sequence number: 0 (0x0000)
Data (56 bytes)
0000 f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66 ..D.b.....B...'f
0010 8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00 ..N......:-.....
0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0030 00 00 00 00 00 00 00 00 ........
Data: F81744FA628D0000801142DA8FE227668FE24E0D00890089...
[Length: 56]
No. Time Source Destination Protocol Info
16886 2.094991 1.2.44.101 1.2.32.250 ICMP Echo (ping) request
Frame 16886 (98 bytes on wire, 98 bytes captured)
Arrival Time: Aug 31, 2010 07:59:11.185674000
[Time delta from previous captured frame: 0.000122000 seconds]
[Time delta from previous displayed frame: 0.000122000 seconds]
[Time since reference or first frame: 2.094991000 seconds]
Frame Number: 16886
Frame Length: 98 bytes
Capture Length: 98 bytes
[Frame is marked: True]
[Protocols in frame: eth:ip:icmp:data]
[Coloring Rule Name: ICMP]
[Coloring Rule String: icmp || icmpv6]
Ethernet II, Src: HewlettP_05:5e:70 (00:17:a4:05:5e:70), Dst: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Destination: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
Address: IPv4mcast_62:20:fa (01:00:5e:62:20:fa)
.... ...1 .... .... .... .... = IG bit: Group address (multicast/broadcast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Source: HewlettP_05:5e:70 (00:17:a4:05:5e:70)
Address: HewlettP_05:5e:70 (00:17:a4:05:5e:70)
.... ...0 .... .... .... .... = IG bit: Individual address (unicast)
.... ..0. .... .... .... .... = LG bit: Globally unique address (factory default)
Type: IP (0x0800)
Internet Protocol, Src: 1.2.44.101 (1.2.44.101), Dst: 1.2.32.250 (1.2.32.250)
Version: 4
Header length: 20 bytes
Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
0000 00.. = Differentiated Services Codepoint: Default (0x00)
.... ..0. = ECN-Capable Transport (ECT): 0
.... ...0 = ECN-CE: 0
Total Length: 84
Identification: 0x7718 (30488)
Flags: 0x00
0.. = Reserved bit: Not Set
.0. = Don't fragment: Not Set
..0 = More fragments: Not Set
Fragment offset: 0
Time to live: 2
[Expert Info (Note/Sequence): "Time To Live" only 2]
[Message: "Time To Live" only 2]
[Severity level: Note]
[Group: Sequence]
Protocol: ICMP (0x01)
Header checksum: 0xd46d [correct]
[Good: True]
[Bad : False]
Source: 1.2.44.101 (1.2.44.101)
Destination: 1.2.32.250 (1.2.32.250)
Internet Control Message Protocol
Type: 8 (Echo (ping) request)
Code: 0 ()
Checksum: 0xf112 [correct]
Identifier: 0xdffd
Sequence number: 0 (0x0000)
Data (56 bytes)
0000 f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66 ..D.b.....B...'f
0010 8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00 ..N......:-.....
0020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0030 00 00 00 00 00 00 00 00 ........
Data: F81744FA628D0000801142DA8FE227668FE24E0D00890089...
[Length: 56]
9 times in 10 when I see something like that, it is a loop somewhere. If it is infrequent, someone is plugging something in, then getting baffled why it isn't working, then unplugging it again. This is murder to find in a lab or development environment, which is why I always try to insist on separating the two physically and with a firewall.
The next most frequent issue is some kind of viral activity. Less frequently it is a spanning-tree storm, and less frequently than that is an actually broken network device.
Loops have always been the low-hanging fruit for me. If spanning-tree isn't enabled you might look into that; some HP switches even have loop-protect features that will shut down a port if they detect a loop on that port. Very cool to see in action.
check for the default advertisement timings for your routing protocols.
and what spanning tree setup are you using on these hp switches?
Please can you provide more information about the destination IP address? From your capture, it appears to be a unicast address. Can you confirm that it comes from the same /16 (Class B) as the rest of your gear?
Edit based on comment:
Could you provide a topology diagram? If I understand you correctly, you have something like the following physical topology, with your switches plugged in to the firewall:
{ Router --- Firewall ---{ [multiple switches] {Some further followup questions:
If you have a valid support contract, I'd consider opening a case with HP and asking them what would case the switches to use a multicast MAC address when sending traffic to a known unicast destination. This seems like a bug to me.
I'm going to dump a lot of disjoint thoughts and observations on you, in case any of them are able to stimulate your thinking or someone else's.
The multicast destination MAC address corresponds to unicast IPv4 address of your firewall.
It's as if some software took your firewall's IP address, pretended it was a multicast address, then followed the formula for generating a corresponding Ethernet multicast MAC address. That is, it took the last 23 bits of the IP address, and appended them to the end of the 01:00:5e OUI.
I have a vague recollection that there may be protocols that do this (use a multicast address based on a unicast address), but my vague memories tell me this is something more likely to be done in IPv6 than IPv4. I'll have to research it some more later.
Update: I was thinking of IPv6 Neighbor Discovery's (IPv6's ARP equivalent's) use of a "solicited-node multicast address". I'm not sure if this is much of a lead, though, because I can't see why anyone would want to do that for IPv4 pings.
The payloads of those multicasted ping packets you captured don't look like typical ping payloads; they have meaningful data in them that might be a clue.
Normal ping payloads usually contain every byte value from 0x00 to 0xff in order, so in the ASCII version of the dump, you'll see every character in order. These pings you captured seem to contain meaningful data. I see some definite IP addresses, and some possible port numbers:
0000 f8 17 44 fa 62 8d 00 00 80 11 42 da 8f e2 27 66 ..D.b.....B...'f0010 8f e2 4e 0d 00 89 00 89 00 3a 2d df 00 00 00 00 ..N......:-.....At offset
0x000c, I see8f e2 27 66, which is IP address[your class B].39.102. Reverse-DNS lookup on that revealsjustin.[your school].edu. Is that hostname meaningful to you? Tracking down what that host is, and what software and services it's running (and possibly whether it's infected or pwned) might yield clues as to what this traffic is.Directly following that, at offset
0x0010, I see8f e2 4e 0d, which is IP address[your class B].78.13, which I can't get a reverse-DNS result on, but maybe you can figure out what it is from where you are.Directly after those IP addresses, I see what I would guess to be two port numbers,
00 89( == 137, NetBIOS Name Service), and00 89again.Could this be a netbios-ns message that somehow got written out as ICMP instead of UDP? Seems unlikely. Too many fields in the wrong places, it seems to me.
Was this supposed to be an ICMP destination unreachable message in response to a netbios-ns message, but the ICMP header got written wrong (as "echo request", instead of "destination unreachable")? Seems unlikely too. Too many fields in the wrong places again.
Is this some sort of malware message, coordinating which hosts are infected/pwned and which ones to attack next? Hanlon's razor would seem to discount that, but I think it's plausible.
Update: Come to think of it, perhaps the most likely possibility is that something is just re-using a buffer to fill the body of this ping request. Thus the contents of it might be a red herring.
Are the TTLs of these frames always just 2, or do you see bursts of descending TTLs?
Same TTLs always would indicate that the packet storm is something of a pure layer-2 bridging loop. Decrementing TTLs would indicate that layer 3, the IP layer, is getting involved; one of those student's personal wireless router's NAT gateway code could be buggy and forwarding certain frames it has no business forwarding, potentially creating a loop.
That is, I'm suggesting that there might be two separate problems interacting here. One is whatever's sending these weird pings with meaningful payloads to a multicast MAC address but unicast IPv4 address. The second problem could be a separate buggy device that sees those frames it wouldn't normally see, and forwards them back onto the network an extra time, creating a loop.
I'm not making sense of your question. You say that you have ping requests going to multicast addresses? I've never seen that and I'm wondering how exactly you determined that, are they actually ICMP echo request packets going to multicast addresses?. You also stated that the ping requests appear to be coming from your switches, but then you go on to say that the source MAC addresses are not from your switches so I'm confused as to what you're actually seeing. Can you give us more detail on what exactly you're seeing in the capture? Maybe even post a line or two of the capture showing the packets you're referring to. Thanks.