Questions[bonding](server)

I have 2 linux boxes running centos 6.5 each with 2 interfaces bonded together, linked to a Cisco 2960-S switch with lacp configured ports.

The configuration on the switch

port-channel load-balance src-dst-mac
!
interface Port-channel1
 switchport access vlan 100
 switchport mode access
!
interface Port-channel2
 switchport access vlan 100
 switchport mode access
!
interface FastEthernet0
 no ip address
!
interface GigabitEthernet0/1
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 1 mode active
!
interface GigabitEthernet0/2
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 1 mode active
!
interface GigabitEthernet0/3
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 2 mode active
!
interface GigabitEthernet0/4
 switchport access vlan 100
 switchport mode access
 speed 1000
 duplex full
 spanning-tree portfast
 channel-protocol lacp
 channel-group 2 mode active
!

and on the both linux sides I've loaded the kernel bonding module with the configuration

alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1

Now the problem is that I transfer many files from one server to another monitoring the traffic graphs showing that the speed doesn't exceed the 1Gb/s speed for the bonding interface bond0.

is there any problem with the configuration ? shouldn't the speed be doubled to 2Gb/s ?

I've read up a little bit on bonding nics with ifenslave; what I'm having trouble understanding is whether there is special configuration needed in order to split the bonds across two switches. For example, if I have several servers that all have two nics each, and two separate switches, do I just configure the bonds and plug 1 nic from each into switch #1 and the other from each into switch #2? or is there more to it than that?

If the bonds are active-backup, will a nic failure on single machine mean that server may become disconnected since the rest of the machines are using the primary nic and it's using the secondary? Or do you link the switches with one cable as well?

I have set up two interfaces as bonding slaves (eth0 and eth1 bonded as bond0). How are people monitoring the status of the slaves?

How are you notified if the link fails on one interface? Do you poll something in /sys/class/net/bond0 or /proc/net/bonding/bond0 ? Is there a daemon I can configure to get notifications?

Some context: I'm running SUSE 11 server SP1 and I'm not interested in any graphical tool as this server is not running X11.

I directly connected two PowerEdge 6950 crossover (using straight lines) on two different PCIe-adapters.

I get a gigabit link on each of these lines (1000 MBit, full duplex, flow contol in both directions).

Now I am trying to bond these interfaces into bond0 using the rr-algorithm on both sides (I want to get 2000 MBit for a single IP session).

When I tested the throughput by transferring /dev/zero to /dev/null using dd bs=1M and netcat in tcp mode I get a throughput of 70 MB/s - not - as expected more than 150MB/s.

When I use the single lines I get about 98 MB/s on each line, if I used a different direction for each line. When I use the single lines I get 70 MB/s and 90 MB/s on the line, if traffic goes into the "same" direction.

After reading through the bonding-readme (/usr/src/linux/Documentation/networking/bonding.txt) I found the following section to be useful: (13.1.1 MT Bonding Mode Selection for Single Switch Topology)

balance-rr: This mode is the only mode that will permit a single TCP/IP connection to stripe traffic across multiple interfaces. It is therefore the only mode that will allow a single TCP/IP stream to utilize more than one interface's worth of throughput. This comes at a cost, however: the striping often results in peer systems receiving packets out of order, causing TCP/IP's congestion control system to kick in, often by retransmitting segments.

    It is possible to adjust TCP/IP's congestion limits by
    altering the net.ipv4.tcp_reordering sysctl parameter. The
    usual default value is 3, and the maximum useful value is 127.
    For a four interface balance-rr bond, expect that a single
    TCP/IP stream will utilize no more than approximately 2.3
    interface's worth of throughput, even after adjusting
    tcp_reordering.

    Note that this out of order delivery occurs when both the
    sending and receiving systems are utilizing a multiple
    interface bond.  Consider a configuration in which a
    balance-rr bond feeds into a single higher capacity network
    channel (e.g., multiple 100Mb/sec ethernets feeding a single
    gigabit ethernet via an etherchannel capable switch).  In this
    configuration, traffic sent from the multiple 100Mb devices to
    a destination connected to the gigabit device will not see
    packets out of order.  However, traffic sent from the gigabit
    device to the multiple 100Mb devices may or may not see
    traffic out of order, depending upon the balance policy of the
    switch.  Many switches do not support any modes that stripe
    traffic (instead choosing a port based upon IP or MAC level
    addresses); for those devices, traffic flowing from the
    gigabit device to the many 100Mb devices will only utilize one
    interface.

Now I changed that parameter on both connected servers on all lines (4) from 3 to 127.

After bonding again I get about 100 MB/s but still not more than that.

Any ideas why?

Update: Hardware details from lspci -v:

24:00.0 Ethernet controller: Intel Corporation 82571EB Gigabit Ethernet Controller (rev 06)
        Subsystem: Intel Corporation PRO/1000 PT Dual Port Server Adapter
        Flags: bus master, fast devsel, latency 0, IRQ 24
        Memory at dfe80000 (32-bit, non-prefetchable) [size=128K]
        Memory at dfea0000 (32-bit, non-prefetchable) [size=128K]
        I/O ports at dcc0 [size=32]
        Capabilities: [c8] Power Management version 2
        Capabilities: [d0] MSI: Mask- 64bit+ Count=1/1 Enable-
        Capabilities: [e0] Express Endpoint, MSI 00
        Kernel driver in use: e1000
        Kernel modules: e1000

Update final results:

8589934592 bytes (8.6 GB) copied, 35.8489 seconds, 240 MB/s

I changed a lot of tcp/ip and low-level-driver options. This includes enlargement of the network buffers. This is why dd now shows numbers greater than 200 MB/s: dd terminates while there is still output waiting to be transferred (in send buffers).

Update 2011-08-05: Settings that were changed to achive the goal (/etc/sysctl.conf):

# See http://www-didc.lbl.gov/TCP-tuning/linux.html
# raise TCP max buffer size to 16 MB. default: 131071
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
# raise autotuninmg TCP buffer limits
# min, default and max number of bytes to use
# Defaults:
#net.ipv4.tcp_rmem = 4096 87380 174760
#net.ipv4.tcp_wmem = 4096 16384 131072
# Tuning:
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
# Default: Backlog 300
net.core.netdev_max_backlog = 2500
#
# Oracle-DB settings:
fs.file-max = 6815744
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
kernel.shmmax = 2147659776
kernel.sem = 1250 256000 100 1024
net.core.rmem_default = 262144
net.core.wmem_default = 262144
#
# Tuning for network-bonding according to bonding.txt:
net.ipv4.tcp_reordering=127

Special settings for the bond-device (SLES: /etc/sysconfig/network/ifcfg-bond0):

MTU='9216'
LINK_OPTIONS='txqueuelen 10000'

Note that setting the biggest possible MTU was the key to the solution.

Tuning of the rx/tx buffers of the involved network cards:

/usr/sbin/ethtool -G eth2 rx 2048 tx 2048
/usr/sbin/ethtool -G eth4 rx 2048 tx 2048

I have Verizon DSL in my office, I put DSL in about a year ago for $29.95 month...I added a new phone line recently and it was cheaper to actually get it bundled with DSL so now I have two DSL lines...my plan was to shut the first one off when my 1 year contract comes up (in September).

A couple of times DSL has gone out on one line so I just used the other, which is a nice redundancy to have - but it doesn't happen often. (I unplugged one line and plugged in the other)

Question is, is there any way to use both DSL lines together so that

1. Increase bandwidth and effective speed might increase (is that possible?)
2. have them both on and connected someway so that traffic on my network would just use either one, and if one went down the traffic would route automatically?

If I can either increase speed by having two, or at a minimum get some automatic redundancy, I see no reason to keep both on....