Background: I have a RHEL6 storage server (lots of HDs) and a RHEL6 Application server (little storage, lots of CPU/memory). They are currently connected to a 1GB Catalyst 2960-X switch. There is also a windows server and 15 client windows computers that pretty much act as terminals also connected. The storage server contains all the data and is linked to the application server via a single NFS4 mount.
Usage: Users SSH (with X11) to the application server to run jobs. Users have the option to run jobs on the storage server as well (because in the past, all we had was the storage server). The storage server will backup all of its data to the windows server nightly.
Question: Since there will be alot of network traffic between the application and storage server, is there a better way to connect the two? For example, I could connect a direct line between the two and mount the NFS4 mount on that dedicated interface. Would it better to use Channel Bonding (I have 4 NICS on each server)? Would that actually help since I'm only using a single NFS mount? Improved fault tolerance is not a concern on this system considering we are only using one line right now. Alternatively, I can used the 2nd NIC on each server for a separate VLAN. That would at least separate the SSH/X11 client traffic from the NFS4/CIFS traffic.
Avoid direct connections between servers, it'll be something of a nightmare to set up and it's horrible for maintenance. Setting up an Etherchannel between the switch and the server is the preferred option, and you can load balance connections over all 4 NICs with a couple different strategies (source MAC, destination MAC, etc.). However, between two hosts, the maximum throughput will still be only 1 Gbps, but you can have 1 Gbps between app and storage, while the app server is communicating with clients on the other NICs.
If the storage on the storage server is only accessed by the application server and the windows server, another option would be to install 10 GE NICs in all the servers, get a cheap 10GE switch, and set up a small storage network.
EDIT: Another option is to separate out the traffic on separate VLAN, that way the storage traffic gets it's own interfaces which better guarantees throughput.