So I am experiencing some weirdness on one of my servers. It has two on board gigabit ethernet ports on it's motherboard both of which seem to have been detected correctly (Broadcom NetXtreme II Gigabit Ethernet). Both ports are plugged into two separate managed switches on the same subnet.
I have given both eth0 and eth1 independent IP addresses on the same subnet (10.0.10.20 and 10.0.10.30 respectively). I have verified that I can ping both of them.
On the managed switches if I turn off the port that eth1 is connected too I can no longer ping 10.0.10.30 as expected. Turning the port back on resume the service. When I turn off the port that eth0 is connected to I can no longer ping either address. Turning the port back on resumes the service for both addresses.
"ifdown eth0" also takes down both interfaces (though ifconfig shows eth1 as still up)
If I put eth0 on a separate network while still being physically up I encounter the same problem of both interfaces not responding (this is my biggest issue with this problem).
There are no errors generated in the boot log or messages log, and the messages log reflects that both interfaces go completely down whenever I take eth0 off the same network as eth1 for any reason.
The only errors that I have seen what-so-ever are on ifconfig where at the very bottom of eth0 and eth1 it will show one of three messages (they are not consistent and not always there). The messages are:
Interrupt:185 Memory:ea000000-ea012800
Interrupt:17
Interrupt:18
Googling those has resulted in some vague questions, but never a solution.
Looking at this it strikes me as an arp flux problem, not a firmware issue. This can arise when you have two nics in the same subnet, and in your case on the same L2 topology. http://linux-ip.net/html/ether-arp.html
You can validate this by putting them on different ip/subnets and see if you still have the same issues. If you need to get this to work take a look at arp_filter and your routing very carefully.
You know, Linux doesn't really like two NICs on the same subnet. At least, not with the basic ifconfig tools. Only one of them gets used to transmit, they seem to share the same MAC, etc. This causes all sorts of weird things to happen. Just like MattyB just mentioned.
The best solution is to use the iproute2 tools (slightly easier to read blog post) to configure your NICs. Try this:
Initialize interfaces:
Configure IP addresses
Then setting up the routing table so that one of those NICs is the 'default'. ip route add 10.0.0.0/16 via 10.0.10.20
--Christopher Karel