LACP – How to Configure Network Bonding in Linux

Tags: , , , , , , , , ,


(Update: Feb 5, 2010 – I even more recently obtained a Cisco IOS switch and have included the configuration bits for IOS below.)

I recently obtained a Dell PowerConnect 5224 Gigabit switch which has the ability to combine multiple twisted-pair or fiber Ethernet links into one fault-tolerant and load balanced logical link. It also appears that its configuration syntax is very similar to that of a Cisco switch. In Linux this is called bonding, in switches its commonly referred to as a port channel. Either way, Its using the LACP (802.3ad) Protocol behind the scenes.

Configuring the switch for LACP bonding


Cisco IOS switch LACP configuration

Enabling LACP across two ports in IOS is pretty straightforward. The first thing to do is associate the ports with the channel-group. This is good to do early so that when you apply switchport parameters to the Port-channel interface it automagically applies them to the GigabigEthernet interfaces.

Here are the relevant portions of my running configuration.

interface Port-channel2
 description LACP Channel for mk2
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,2
 switchport mode trunk
 spanning-tree portfast trunk
!
interface GigabitEthernet1/0/23
 description mk2 eth0
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 2 mode active
!
interface GigabitEthernet1/0/24
 description mk2 eth1
 switchport trunk encapsulation dot1q
 switchport mode trunk
 channel-group 2 mode active
!

Dell PowerConnect switch LACP configuration

The Dell switch configuration is surprisingly easy. A port-channel is automatically created when the Linux host brings up it’s bond interface(s). Just figure out which ports you want to use for your bond and enable LACP on them. I used ports 1/23 and 1/24 (ports 23 & 24 on switch 1).

Vty-0#config
Vty-0(config)#interface ethernet 1/23
Vty-0(config-if)#lacp
Vty-0(config-if)#exit
Vty-0(config)#interface ethernet 1/24
Vty-0(config-if)#lacp
Vty-0(config-if)#exit

‘show run’ now indicates that the selected ports are LACP enabled.

Vty-0#show run
building running-config, please wait.....
...
!
interface ethernet 1/23
 switchport allowed vlan add 1 untagged
 switchport native vlan 1
 lacp
!
interface ethernet 1/24
 switchport allowed vlan add 1 untagged
 switchport native vlan 1
 lacp
!
...

At this point your port-channel will be down. Don’t worry, it will automagically come up when the Linux host brings up the bond interface. You can verify that its down by issuing the following:

Vty-0#show interfaces status port-channel 1
% Trunk 1 does not exist.

Note: This assumes you have no pre-existing port-channels, if you do have other port-channels configured you should iterate the port-channel number to be one more than the number of already defined port-channels.

Configuring the Linux host for LACP bonding:


There are a few places where you define the parameters of the bond. The kernel module defies the protocol, frequency and other attributes of the low-level bond channel configuration. The command ifenslave will create a bond device and allow you to manage the Ethernet devices within it (add/remove,etc.). Finally the network address configuration is handled by ifconfig, consistent with most other network interfaces in Linux. Luckily most of this is taken care of automatically by the networking init scripts.

Linux Kernel Module Configuration


LACP is referred to in linux as bonding mode 4, so we need to inform the kernel module to use this bonding mode. We’ll also pass it a few other parameters like the frequency of which to scan for changes in status.

Add the following to your module config file, in gentoo this is /etc/modules.autoload.d/kernel-2.6. This will pass the following options to the kernel module the next time it is inserted.

Red Hat and CentOS Kernel Module Configuration

#/etc/modprobe.conf
 
alias bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1

RHEL 6 and CentOS 6 Kernel Module Configuration

#/etc/modprobe.d/bonding.conf
 
#Deprecated syntax
#alias bond0 bonding  
 
#Updated syntax
alias netdev-bond0 bonding
options bond0 miimon=100 mode=4 lacp_rate=1

Debian Kernel Module Configuration

# /etc/modules: kernel modules to load at boot time.
 
bonding mode=4 miimon=100 lacp_rate=1

Ubuntu Kernel Module Configuration

# /etc/modprobe.d/bonding.conf
 
bonding mode=4 miimon=100 lacp_rate=1

Gentoo Kernel Module Setup

#/etc/modules.autoload.d/kernel-2.6
 
bonding miimon=100 mode=4 lacp_rate=1

Linux Network Configuration


Red Hat and CentOS Network Setup

#/etc/sysconfig/network-scripts/ifcfg-eth0
 
DEVICE=eth0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
#/etc/sysconfig/network-scripts/ifcfg-eth1
 
DEVICE=eth1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
MASTER=bond0
SLAVE=yes
#/etc/sysconfig/network-scripts/ifcfg-bond0
 
DEVICE=bond0
IPADDR=10.0.0.80
NETMASK=255.255.255.0
BROADCAST=10.0.0.255
GATEWAY=10.0.0.1
ONBOOT=yes
BOOTPROTO=none
USERCTL=no

Debian / Ubuntu Network Setup

#/etc/network/interfaces 
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
auto bond0
iface bond0 inet static
	address 10.0.0.80
	gateway 10.0.0.1
	broadcast 10.0.0.255
	netmask 255.255.255.0
	up /sbin/ifenslave bond0 eth1 eth2
	down /sbin/ifenslave -d bond0 eth0 eth1

*Note* This is dependant upon the ifenslave package, to install run the following:

apt-get install ifenslave

Ubuntu 10.04 or newer support an updated interfaces(5) syntax

#/etc/network/interfaces 
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).
 
auto eth0
    iface eth0 inet manual
    bond-master bond0
 
auto eth1
     iface eth1 inet manual
     bond-master bond0
 
auto bond0
     iface bond0 inet static
     address 10.0.0.80
     gateway 10.0.0.1
     netmask 255.255.255.0
 
 
bond-mode 802.3ad
bond-miimon 100
bond-lacp-rate 4
bond-slaves none

Gentoo LACP bonding Setup

#/etc/conf.d/net
 
config_eth0=( "null" )
config_eth1=( "null" )
 
slaves_bond0="eth0 eth1"
 
config_bond0=( "10.0.0.80/24" )

We also need to create a symlink in /etc/init.d for the new bond0 interface and turn off eth0 as it is controlled by the bond now. The following will disable eth0 and enable bond0 on boot.

  cd /etc/init.d
  ln -s net.lo net.bond0
  rc-update del eth0 default
  rc-update add bond0 default

Now you can bring up the bond interface.

  /etc/init.d/net.bond0 start

Checking the Status of the bonded LACP interface


You can check the status of your bond now from within Linux by using the /proc and /sys interfaces into the Linux bond driver.

$ cat /proc/net/bonding/bond0  
 
Ethernet Channel Bonding Driver: v3.1.1 (September 26, 2006)
 
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
 
802.3ad info
LACP rate: fast
Active Aggregator Info:
	Aggregator ID: 1
	Number of ports: 2
	Actor Key: 17
	Partner Key: 1
	Partner Mac Address: 00:77:54:71:a8:6f
 
Slave Interface: eth0
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:99:97:60:9d:48
Aggregator ID: 1
 
Slave Interface: eth1
MII Status: up
Link Failure Count: 0
Permanent HW addr: 00:00:85:60:9d:49
Aggregator ID: 1

You can check the bond from the switch.

Cisco IOS

Switch#show interfaces Port-channel 2
Port-channel2 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 001b.0dbf.ba17 (bia 001b.0dbf.ba17)
  Description: LACP Channel for mk2
  MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Full-duplex, 1000Mb/s, link type is auto, media type is unknown
  input flow-control is off, output flow-control is unsupported 
  Members in this channel: Gi1/0/23 Gi1/0/24 
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 1d23h, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 5000 bits/sec, 7 packets/sec
     1060041 packets input, 193406916 bytes, 0 no buffer
     Received 18241 broadcasts (0 multicast)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 11873 multicast, 0 pause input
     0 input packets with dribble condition detected
     3181997 packets output, 2735804051 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

Dell PowerConnect:

Vty-0#show interfaces status port-channel 1
Information of Trunk 1
 Basic information: 
  Port type: 1000t
  Mac address: 00-30-F1-71-A8-82
 Configuration: 
  Name: 
  Port admin: Up
  Speed-duplex: Auto
  Capabilities: 10half, 10full, 100half, 100full, 1000full, 
  Flow control: Disabled
 Current status: 
  Created by: Lacp
  Link status: Up
  Port operation status: Up
  Operation speed-duplex: 1000full
  Flow control type: None
  Member Ports: Eth1/23, Eth1/24,

That’s is all the configuration work that I needed to perform. I hope it saves you time. I spent a while digging through dell’s site and the linux kernel docs to find the right combination of options. Please let me know if you had troubles with these directions or if you have questions. keith (at) backdrift.org

Additional docs on linux bonding driver https://www.kernel.org/doc/Documentation/networking/bonding.txt

48 Responses to “LACP – How to Configure Network Bonding in Linux”

  1. AntoineM Says:

    Hello,

    there is a problem in the file modprobe.conf, you write “options bond0..” but it is “options bonding…”

    thanks for this howto

    [Reply]

    MikeR Reply:

    Actually, the “alias bond0 bonding” makes “options bond0″ correct, since it’s really options bonding … ” I think it’s just there for clarities sake.

    [Reply]

  2. Gavin Says:

    Thank you. This was exactly what I was looking for. Very clear and easy to follow. Bonding/aggregating links to my storage node has solved a mysterious dying port problem on my powerconnect switch. Cheers.

    [Reply]

  3. Daniel Feenberg Says:

    Before I try this – I have a question. Will it allow a single process to transfer data over both ports to a single destination? That is, if I have 2 gigabit ports and want to copy a file will the copy use both ports and possibly go at more than 1 gigabit per second? Or is the selection of a port deterministic and the transfer will use only one port?

    Thanks for any guidance.
    Dan Feenberg
    NBER

    [Reply]

  4. Damon Says:

    Great how-to.
    Daniel, as I understand it, you’re not going to get a 2gig session, however if someone else connects to the same endpoint as you at or near the same time, I believe that their session/connection will go over the unused 1gig line. so instead of both of you clogging up one port, you’ll be split and each able to have your own port. Any successive connections will probably be determined by the OS.

    [Reply]

  5. darkfader Says:

    for scaling above 1gbit between two hosts:

    hash by src-ip – dst-ip-port – some switches can do that.

    also note that the bonding driver in centos5 / rhel5 is said to be not smp-capable, bringing in its’ own performance issues (see the presentation about mysql by facebook)

    [Reply]

  6. Linux Ethernet Bonding – WTF?! « Bill Sigmund Says:

    […] http://backdrift.org/howtonetworkbonding […]

  7. musa Says:

    Hi,

    i am using centos 5.5 as a firewall and have three ethernet ports on it as:
    eth0 – 192.168.2.1 (LAN port)
    eth1 – 202.xx.xx.94 (WAN1)
    eth2 – 202.xx.xx.93 (WAN2)

    i want to bond the two WAN ports for outgoing traffic, can i do that?
    and if yes, what should i do to their assigned public IPs?… configure or not? and what IP should i assign to bond0 interface in that case?

    Please help!!!

    regards

    [Reply]

    jorge robles Reply:

    let me know if you have a solution, i’m having the same issue as you.

    [Reply]

    Keith Reply:

    As long as these are ethernet interfaces there is no distinction between LAN and WAN. You would simply assign the appropriate IP addresses to the bond interface. You can assign multiple IP addresses using interface aliases which would look something like this:

    bond0:0 202.xx.xx.93
    bond0:1 202.xx.xx.94

    Hope that helps

    [Reply]

  8. Thomas Says:

    Thanks for the quick infos about port channel / lacp. peace.

    [Reply]

  9. Scorcy Says:

    If you use LACP for bonding (mode=4) then you will in fact double link speed (for example 2x 100 Mbps = 200 Mbps or 2x 1 Gbps = 2 Gbps).

    As far as I know if one port goes down your link will still be up but at 1 Gbps.

    There are some redundancy modes as well which will not increase speed but just increase redundancy. LACP more or less does both (though speed is halved if one link is down), but the mode where all data is sent twice will deliver the best redundancy; it’s like RAID1 in network throughput 🙂

    [Reply]

  10. TP-Link TL-SG3210, 8-Port, Managed Switch » Philipp Klaus's Computing Blog Says:

    […] The linux kernel module bonding implements LACP when loaded with the parameter mode=4. More about LACP configuration on IOS switches, Dell switches and Linux machines on http://backdrift.org/howtonetworkbonding. […]

  11. Setting up Openfiler with 3x Dell MD1200 enclosures « IT Notepad Says:

    […] Network refs: http://support.dell.com/support/edocs/network/m8024k/en/ucg/html/linkagg.htm http://backdrift.org/howtonetworkbonding […]

  12. Antonio Says:

    Hello all. Thank you for your post, its very interesting.

    I have a question. I have a machine with 6 ethernet nics. So, I would like to set up a bond interface using mode=4 for improve band with speed and fault tolerance. But, I would like to connect three of those nics to one chasis and the rest to other chasis, Its possible to configure one lacp port in one switch and other one in the other chasis?

    I dont know if this question its so stupid…

    Thank you.

    [Reply]

    Keith Reply:

    Hey @Antonio, typically you would accomplish that by stacking your two switches together and creating a lacp group with the ports split between the two switches. If your switches aren’t stackable, you could accomplish a similar level of redundancy and performance by utilizing a different bonding mode.

    Hope that helps

    [Reply]

  13. Tyler Says:

    Just saw a bit of a typo that I wanted to help clear up. With the modprobe.conf file, if you have a more current version of redhat/centos this is not the file you want to use. Try instead /etc/modprobe.d/bonding.conf . You’ll have to create the file. Write the same information
    studyhat.blogspot.com/2011/11/redhat-centos-6-bonding.html

    [Reply]

  14. Sebastian Says:

    @Antonio: your switches needs a feature that is called Multi-Chassis Link Aggregation to accomplish that (other switch vendors may have a different name for that)

    cheers

    [Reply]

  15. How to configure network bonding in Linux | Backdrift « My Blog Says:

    […] How to configure network bonding in Linux | Backdrift. Share this:TwitterFacebookLike this:LikeBe the first to like this. […]

  16. nasherul Says:

    can i change Full-duplex, 1000Mb/s to Full-duplex, 100Mb/s in 2960s.please if have solution lz reply me first……..

    nasherul

    [Reply]

  17. nasherul Says:

    can i change 0/23 gigabit Ethernet port Full-duplex, 1000Mb/s to Full-duplex, 100Mb/s in 2960s.please if have solution lz reply me first……..

    [Reply]

  18. Alex Says:

    @nasherul, of course you can. On cisco it would be something like this:
    | SW#(config-if)# speed 100
    | SW#(config-if)# duplex full

    Sorry for the late reply
    Regards

    [Reply]

  19. CentOS 6 KVM, Bonded and Bridged Networking … | Ian Works Here Says:

    […] http://backdrift.org/howtonetworkbonding […]

  20. David Dionne Says:

    Excellent doc!

    Have you experimented with mtu or jumbo frames?

    [Reply]

    Keith Reply:

    Hey David,

    Yeah, in order to set jumbo frames you’d just need to set MTU=9000 on each of the slave eth interfaces, and the bond interface. Of course the upstream network switching and routing gear would need to support jumbo frames as well.

    Hope that helps!

    [Reply]

  21. Gab Says:

    On CentOS 6.2 and more, the ‘alias bond0 bonding’ is depreciate and write the warning message if you use.If you want to fix this problem use, ‘alias netdev-bond0 bonding’ in /etc/modprobe.d/bonding.conf

    [Reply]

    Keith Reply:

    Thanks for the heads up! I just added a section to the doc that shows the deprecated syntax and the new syntax.

    [Reply]

    Barrow Reply:

    the guide from RedHat didn’t say netdev-bond0…

    https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sec-Using_Channel_Bonding.html

    [Reply]

  22. Hans Says:

    LACP is 802.3ad, not 802.1ad.

    [Reply]

  23. NIC Bonding speed vs LAG Says:

    […] Sounds like your switch isn't doing the LACP bonding correctly and is getting confused seeing the same IP address mapped to NICs on different ports. This might be helpful if you've not come across it already – http://backdrift.org/howtonetworkbonding […]

  24. Samantha Says:

    I’ve been browsing online greater than three hours today, but I never discovered any interesting article like yours. It is lovely value enough for me. In my opinion, if all site owners and bloggers made just right content material as you probably did, the net might be much more useful than ever before.

    [Reply]

  25. Bonding a Bond with LACP - The UNIX and Linux Forums Says:

    […] Wiki Configure Bonding / Link-Aggregation using LACP under RHEL 5.4 (or CentOS) | itground How to configure network bonding in Linux | Backdrift I've done this successfully many times with Solaris by creating separate aggregate devices (using […]

  26. neodyme Says:

    sh int port-channel 2 is meaningless, when looking for lacp info on cisco ios.
    use “sh etherchannel detail” for getting lacp information.

    [Reply]

  27. James Says:

    Any help, I can’t seem to get the thoughput.

    Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

    Bonding Mode: IEEE 802.3ad Dynamic link aggregation
    Transmit Hash Policy: layer2 (0)
    MII Status: up
    MII Polling Interval (ms): 100
    Up Delay (ms): 0
    Down Delay (ms): 0

    802.3ad info
    LACP rate: fast
    Aggregator selection policy (ad_select): stable
    Active Aggregator Info:
    Aggregator ID: 59
    Number of ports: 4
    Actor Key: 17
    Partner Key: 1
    Partner Mac Address: c4:71:fe:ed:69:00

    Slave Interface: p1p1
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 0
    Permanent HW addr: 00:e0:4c:68:1e:fd
    Aggregator ID: 59
    Slave queue ID: 0

    Slave Interface: p1p2
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 0
    Permanent HW addr: 00:e0:4c:68:1e:fe
    Aggregator ID: 59
    Slave queue ID: 0

    Slave Interface: p1p3
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 0
    Permanent HW addr: 00:e0:4c:68:1e:ff
    Aggregator ID: 59
    Slave queue ID: 0

    Slave Interface: p1p4
    MII Status: up
    Speed: 1000 Mbps
    Duplex: full
    Link Failure Count: 0
    Permanent HW addr: 00:e0:4c:68:1f:00
    Aggregator ID: 59
    Slave queue ID: 0

    —————- CISCO —————-

    interface Port-channel1
    switchport
    switchport trunk encapsulation dot1q
    switchport mode trunk
    mtu 9000
    spanning-tree portfast trunk
    !

    interface GigabitEthernet1/9 – 10, 11,12
    switchport trunk encapsulation dot1q
    switchport mode trunk
    mtu 9000
    channel-protocol lacp
    channel-group 1 mode active
    spanning-tree portfast
    !

    The server I have connected on the other side, is connected via 10 gig fiber. I need this 4gig link to access the server. Any help would be great..

    [Reply]

  28. LACP - bonding cisco switch | blog Says:

    […] Pro linux se používá termín Bonding  viz.: http://backdrift.org/howtonetworkbonding […]

  29. Just wondering Says:

    You have a typo there, 802.1ad is a different technology altogether (QinQ), I believe you meant 802.3ad for LACP port channels…

    [Reply]

    Keith Reply:

    Good catch thank you

    [Reply]

  30. Intel Gigabit ET Dual Port Server Adapter Says:

    […] http://backdrift.org/howtonetworkbonding […]

  31. Sam McLeod Says:

    FYI – bond-lacp-rate = 4 is a typo – it cannot be set to 4.

    It just happens to work as it falls back to 1 if you set it to anything greater than 0.

    [Reply]

  32. Aaron Says:

    The module options weren’t working for me. I found adding them in /etc/sysconfig/network-scripts/ifcfg-bond0 with the line

    BONDING_OPTS=”miimon=100 mode=4″

    worked great. Also, you may want to add something about the xmit_hash_policy module option. It’s imperative if you’re doing balancing for stuff that’s not local, so that you use the recipient IP/port instead of the next hop mac address.

    [Reply]

  33. Florida Says:

    Everything is very open with a vey clear explanation of the issues.
    It was really informative. Your site is useful.
    Thanks for sharing!

    [Reply]

  34. umakant Says:

    Hi All,
    i have a question can anyone help me to resolve the issue or clarify .

    I have 2 dell servers with double 10 ports in bonding connected to nexus 6k in lacp and on nexus port channel now when i am taking a iperf connection from server to server it is showing nearly 906 Gb bandwidth.
    witch is nearly hafl as expected .

    now i want to confirm that how lacp bonding on linux server take decision to forward a file to server to server connected in redundant mode with 2 nexus will it taking forwarding on basis of MAC address with only one link or on IP basis.
    kindly clarify..

    @Umakant

    [Reply]

  35. Manish Says:

    Can I use LACP on ports of two separate switches and which bond I have to use for same and why.

    [Reply]

    Keith Reply:

    Generally yes you can bond across two physical switches. Typically top of rack switches are stacked into one logical switch, and LACP bonding mode is used. Other linux bonding modes are available if LACP is not supported by the switch. Refer back to the documentation of your specific switches to understand the best method for switch level redundancy.

    [Reply]

  36. Mister Says:

    Hi,

    I have two or more Linux servers configured as firewalls and I would bridge the portchannel between two Cisco switch linking one channel per firewall. So I can distribute traffic and cpu load between more firewall, and also realize HA.
    Is it possibile?

    Tnx in advance.

    [Reply]

  37. forex forum Says:

    At this time it looks like Expression Engine is the top blogging platform
    out there right now. (from what I’ve read) Is that what you are using on your blog?

    [Reply]

  38. ridhoyp Says:

    hi,

    nice tutorial, wanna asks

    i have 4 port 10gbps , all i want is make bonding with concurrent transfer,, so the transfer rate is balance all ethernet port, is linux can? with bonding mode6 and layer3+4 policy.

    Regards,
    RidhoYP

    [Reply]

  39. zobelloindia Says:

    Wow, Again i am here and your awesome blog. Keep going. We are the Indian Men’s Clothing Manufacturer. Our Online Store name is Zobello. Please visit once. Thanks

    [Reply]

Join the Conversation