(Update: Feb 5, 2010 – I even more recently obtained a Cisco IOS switch and have included the configuration bits for IOS below.)
I recently obtained a Dell PowerConnect 5224 Gigabit switch which has the ability to combine multiple twisted-pair or fiber Ethernet links into one fault-tolerant and load balanced logical link. It also appears that its configuration syntax is very similar to that of a Cisco switch. In Linux this is called bonding, in switches its commonly referred to as a port channel. Either way, Its using the LACP (802.1ad) Protocol behind the scenes.
Configuring the switch for LACP bonding
Cisco IOS switch LACP configuration
Enabling LACP across two ports in IOS is pretty straightforward. The first thing to do is associate the ports with the channel-group. This is good to do early so that when you apply switchport parameters to the Port-channel interface it automagically applies them to the GigabigEthernet interfaces.
Here are the relevant portions of my running configuration.
interface Port-channel2
description LACP Channel for mk2
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 1,2
switchport mode trunk
spanning-tree portfast trunk
!
interface GigabitEthernet1/0/23
description mk2 eth0
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode active
!
interface GigabitEthernet1/0/24
description mk2 eth1
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 2 mode active
! |
Dell PowerConnect switch LACP configuration
The Dell switch configuration is surprisingly easy. A port-channel is automatically created when the Linux host brings up it’s bond interface(s). Just figure out which ports you want to use for your bond and enable LACP on them. I used ports 1/23 and 1/24 (ports 23 & 24 on switch 1).
Vty-0#config
Vty-0(config)#interface ethernet 1/23
Vty-0(config-if)#lacp
Vty-0(config-if)#exit
Vty-0(config)#interface ethernet 1/24
Vty-0(config-if)#lacp
Vty-0(config-if)#exit |
‘show run’ now indicates that the selected ports are LACP enabled.
Vty-0#show run
building running-config, please wait.....
...
!
interface ethernet 1/23
switchport allowed vlan add 1 untagged
switchport native vlan 1
lacp
!
interface ethernet 1/24
switchport allowed vlan add 1 untagged
switchport native vlan 1
lacp
!
... |
At this point your port-channel will be down. Don’t worry, it will automagically come up when the Linux host brings up the bond interface. You can verify that its down by issuing the following:
Vty-0#show interfaces status port-channel 1
% Trunk 1 does not exist. |
Note: This assumes you have no pre-existing port-channels, if you do have other port-channels configured you should iterate the port-channel number to be one more than the number of already defined port-channels.
Configuring the Linux host for LACP bonding:
There are a few places where you define the parameters of the bond. The kernel module defies the protocol, frequency and other attributes of the low-level bond channel configuration. The command ifenslave will create a bond device and allow you to manage the Ethernet devices within it (add/remove,etc.). Finally the network address configuration is handled by ifconfig, consistent with most other network interfaces in Linux. Luckily most of this is taken care of automatically by the networking init scripts.
Linux Kernel Module Configuration
LACP is referred to in linux as bonding mode 4, so we need to inform the kernel module to use this bonding mode. We’ll also pass it a few other parameters like the frequency of which to scan for changes in status.
Add the following to your module config file, in gentoo this is /etc/modules.autoload.d/kernel-2.6. This will pass the following options to the kernel module the next time it is inserted.
Red Hat and CentOS Kernel Module Configuration
#/etc/modprobe.conf
alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1 |
RHEL 6 and CentOS 6 Kernel Module Configuration
#/etc/modprobe.d/bonding.conf
#Deprecated syntax
#alias bond0 bonding
#Updated syntax
alias netdev-bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1 |
Debian Kernel Module Configuration
# /etc/modules: kernel modules to load at boot time.
bonding mode=4 miimon=100 lacp_rate=1 |
Ubuntu Kernel Module Configuration
# /etc/modprobe.d/bonding.conf
bonding mode=4 miimon=100 lacp_rate=1 |
Gentoo Kernel Module Setup
#/etc/modules.autoload.d/kernel-2.6
bonding miimon=100 mode=4 lacp_rate=1 |
Linux Network Configuration
Red Hat and CentOS Network Setup
#/etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes |
#/etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes |
#/etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
IPADDR=10.0.0.80
NETMASK=255.255.255.0
BROADCAST=10.0.0.255
GATEWAY=10.0.0.1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no |
Debian / Ubuntu Network Setup
#/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
auto bond0
iface bond0 inet static
address 10.0.0.80
gateway 10.0.0.1
broadcast 10.0.0.255
netmask 255.255.255.0
up /sbin/ifenslave bond0 eth1 eth2
down /sbin/ifenslave -d bond0 eth0 eth1 |
*Note* This is dependant upon the ifenslave package, to install run the following:
apt-get install ifenslave |
Ubuntu 10.04 or newer support an updated interfaces(5) syntax
#/etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
auto eth0
iface eth0 inet manual
bond-master bond0
auto eth1
iface eth1 inet manual
bond-master bond0
auto bond0
iface bond0 inet static
address 10.0.0.80
gateway 10.0.0.1
netmask 255.255.255.0
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 4
bond-slaves none |
Gentoo Network Setup
#/etc/conf.d/net
config_eth0=( "null" )
config_eth1=( "null" )
slaves_bond0="eth0 eth1"
config_bond0=( "10.0.0.80/24" ) |
We also need to create a symlink in /etc/init.d for the new bond0 interface and turn off eth0 as it is controlled by the bond now. The following will disable eth0 and enable bond0 on boot.
cd /etc/init.d
ln -s net.lo net.bond0
rc-update del eth0 default
rc-update add bond0 default |
Now you can bring up the bond interface.
/etc/init.d/net.bond0 start |
Checking the Status of the bonded interface
You can check the status of your bond now from within Linux by using the /proc and /sys interfaces into the Linux bond driver.
$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.1.1 (September 26, 2006)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: fast
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 17
Partner Key: 1
Partner Mac Address: 00:77:54:71:a8:6f
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:99:97:60:9d:48
Aggregator ID: 1
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:00:85:60:9d:49
Aggregator ID: 1 |
You can check the bond from the switch.
Cisco IOS
Switch#show interfaces Port-channel 2
Port-channel2 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 001b.0dbf.ba17 (bia 001b.0dbf.ba17)
Description: LACP Channel for mk2
MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Full-duplex, 1000Mb/s, link type is auto, media type is unknown
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/0/23 Gi1/0/24
ARP type: ARPA, ARP Timeout 04:00:00
Last input 1d23h, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 5000 bits/sec, 7 packets/sec
1060041 packets input, 193406916 bytes, 0 no buffer
Received 18241 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 11873 multicast, 0 pause input
0 input packets with dribble condition detected
3181997 packets output, 2735804051 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out |
Dell PowerConnect:
Vty-0#show interfaces status port-channel 1
Information of Trunk 1
Basic information:
Port type: 1000t
Mac address: 00-30-F1-71-A8-82
Configuration:
Name:
Port admin: Up
Speed-duplex: Auto
Capabilities: 10half, 10full, 100half, 100full, 1000full,
Flow control: Disabled
Current status:
Created by: Lacp
Link status: Up
Port operation status: Up
Operation speed-duplex: 1000full
Flow control type: None
Member Ports: Eth1/23, Eth1/24, |
That’s is all the configuration work that I needed to perform, I hope it saves you time (I spent a while digging through dell’s site and the linux kernel docs to find the right combination of options). Please, let me know if you had troubles with these directions or if you have questions. keith (at) backdrift.org
July 2nd, 2010 at 7:27 am
Hello,
there is a problem in the file modprobe.conf, you write “options bond0..” but it is “options bonding…”
thanks for this howto
[Reply]
August 20th, 2010 at 1:28 pm
Thank you. This was exactly what I was looking for. Very clear and easy to follow. Bonding/aggregating links to my storage node has solved a mysterious dying port problem on my powerconnect switch. Cheers.
[Reply]
January 29th, 2011 at 7:51 am
Before I try this – I have a question. Will it allow a single process to transfer data over both ports to a single destination? That is, if I have 2 gigabit ports and want to copy a file will the copy use both ports and possibly go at more than 1 gigabit per second? Or is the selection of a port deterministic and the transfer will use only one port?
Thanks for any guidance.
Dan Feenberg
NBER
[Reply]
February 24th, 2011 at 4:04 pm
Great how-to.
Daniel, as I understand it, you’re not going to get a 2gig session, however if someone else connects to the same endpoint as you at or near the same time, I believe that their session/connection will go over the unused 1gig line. so instead of both of you clogging up one port, you’ll be split and each able to have your own port. Any successive connections will probably be determined by the OS.
[Reply]
April 24th, 2011 at 6:35 pm
for scaling above 1gbit between two hosts:
hash by src-ip – dst-ip-port – some switches can do that.
also note that the bonding driver in centos5 / rhel5 is said to be not smp-capable, bringing in its’ own performance issues (see the presentation about mysql by facebook)
[Reply]
June 13th, 2011 at 8:51 am
[...] http://backdrift.org/howtonetworkbonding [...]
February 8th, 2012 at 5:50 am
Hi,
i am using centos 5.5 as a firewall and have three ethernet ports on it as:
eth0 – 192.168.2.1 (LAN port)
eth1 – 202.xx.xx.94 (WAN1)
eth2 – 202.xx.xx.93 (WAN2)
i want to bond the two WAN ports for outgoing traffic, can i do that?
and if yes, what should i do to their assigned public IPs?… configure or not? and what IP should i assign to bond0 interface in that case?
Please help!!!
regards
[Reply]
jorge robles Reply:
May 28th, 2012 at 9:11 pm
let me know if you have a solution, i’m having the same issue as you.
[Reply]
Keith Reply:
May 28th, 2012 at 9:23 pm
As long as these are ethernet interfaces there is no distinction between LAN and WAN. You would simply assign the appropriate IP addresses to the bond interface. You can assign multiple IP addresses using interface aliases which would look something like this:
bond0:0 202.xx.xx.93
bond0:1 202.xx.xx.94
Hope that helps
[Reply]
February 28th, 2012 at 7:51 am
Thanks for the quick infos about port channel / lacp. peace.
[Reply]
March 10th, 2012 at 2:47 am
If you use LACP for bonding (mode=4) then you will in fact double link speed (for example 2x 100 Mbps = 200 Mbps or 2x 1 Gbps = 2 Gbps).
As far as I know if one port goes down your link will still be up but at 1 Gbps.
There are some redundancy modes as well which will not increase speed but just increase redundancy. LACP more or less does both (though speed is halved if one link is down), but the mode where all data is sent twice will deliver the best redundancy; it’s like RAID1 in network throughput
[Reply]
April 13th, 2012 at 3:59 pm
[...] The linux kernel module bonding implements LACP when loaded with the parameter mode=4. More about LACP configuration on IOS switches, Dell switches and Linux machines on http://backdrift.org/howtonetworkbonding. [...]
April 27th, 2012 at 10:05 am
[...] Network refs: http://support.dell.com/support/edocs/network/m8024k/en/ucg/html/linkagg.htm http://backdrift.org/howtonetworkbonding [...]
May 17th, 2012 at 5:23 am
Hello all. Thank you for your post, its very interesting.
I have a question. I have a machine with 6 ethernet nics. So, I would like to set up a bond interface using mode=4 for improve band with speed and fault tolerance. But, I would like to connect three of those nics to one chasis and the rest to other chasis, Its possible to configure one lacp port in one switch and other one in the other chasis?
I dont know if this question its so stupid…
Thank you.
[Reply]
Keith Reply:
May 17th, 2012 at 10:38 am
Hey @Antonio, typically you would accomplish that by stacking your two switches together and creating a lacp group with the ports split between the two switches. If your switches aren’t stackable, you could accomplish a similar level of redundancy and performance by utilizing a different bonding mode.
Hope that helps
[Reply]
June 13th, 2012 at 3:49 pm
Just saw a bit of a typo that I wanted to help clear up. With the modprobe.conf file, if you have a more current version of redhat/centos this is not the file you want to use. Try instead /etc/modprobe.d/bonding.conf . You’ll have to create the file. Write the same information
studyhat.blogspot.com/2011/11/redhat-centos-6-bonding.html
[Reply]
June 23rd, 2012 at 9:17 am
@Antonio: your switches needs a feature that is called Multi-Chassis Link Aggregation to accomplish that (other switch vendors may have a different name for that)
cheers
[Reply]
September 10th, 2012 at 11:57 pm
[...] How to configure network bonding in Linux | Backdrift. Share this:TwitterFacebookLike this:LikeBe the first to like this. [...]
January 16th, 2013 at 6:44 am
can i change Full-duplex, 1000Mb/s to Full-duplex, 100Mb/s in 2960s.please if have solution lz reply me first……..
nasherul
[Reply]
January 16th, 2013 at 6:47 am
can i change 0/23 gigabit Ethernet port Full-duplex, 1000Mb/s to Full-duplex, 100Mb/s in 2960s.please if have solution lz reply me first……..
[Reply]
March 18th, 2013 at 1:21 am
@nasherul, of course you can. On cisco it would be something like this:
| SW#(config-if)# speed 100
| SW#(config-if)# duplex full
Sorry for the late reply
Regards
[Reply]
March 27th, 2013 at 9:44 pm
[...] http://backdrift.org/howtonetworkbonding [...]
April 19th, 2013 at 10:05 am
Excellent doc!
Have you experimented with mtu or jumbo frames?
[Reply]
Keith Reply:
April 19th, 2013 at 10:09 am
Hey David,
Yeah, in order to set jumbo frames you’d just need to set MTU=9000 on each of the slave eth interfaces, and the bond interface. Of course the upstream network switching and routing gear would need to support jumbo frames as well.
Hope that helps!
[Reply]
April 24th, 2013 at 7:13 am
On CentOS 6.2 and more, the ‘alias bond0 bonding’ is depreciate and write the warning message if you use.If you want to fix this problem use, ‘alias netdev-bond0 bonding’ in /etc/modprobe.d/bonding.conf
[Reply]
Keith Reply:
April 24th, 2013 at 9:20 am
Thanks for the heads up! I just added a section to the doc that shows the deprecated syntax and the new syntax.
[Reply]
Barrow Reply:
May 8th, 2013 at 1:15 pm
the guide from RedHat didn’t say netdev-bond0…
https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sec-Using_Channel_Bonding.html
[Reply]
April 30th, 2013 at 5:55 pm
LACP is 802.3ad, not 802.1ad.
[Reply]
May 3rd, 2013 at 4:51 am
[...] Sounds like your switch isn't doing the LACP bonding correctly and is getting confused seeing the same IP address mapped to NICs on different ports. This might be helpful if you've not come across it already – http://backdrift.org/howtonetworkbonding [...]
May 12th, 2013 at 11:32 pm
You need to take part in a contest for one of the finest blogs on the internet. I am going to recommend this web site!
[Reply]