Link Aggregation vs ECMP: Combining Network Links

You bond four 10 Gbps uplinks between two switches and expect a 40 Gbps pipe. Then the nightly backup that used to finish at 2 a.m. still finishes at 2 a.m., and a single file transfer still crawls along at roughly 10 Gbps. Nothing is broken. Link aggregation gives the network more aggregate capacity and a survivable path, but it does not hand any single conversation more bandwidth than one member, because the hardware picks a member per flow with a hash very much like a router picks a next hop. The same principle appears when you add a second routed uplink and the routing protocol quietly installs both paths while the traffic still does not split evenly. In both cases the mechanism is doing exactly what it was designed to do, and understanding why removes a great deal of misplaced troubleshooting.

Link aggregation and ECMP are the two ways a network combines connections, and they get confused because they share the same instinct: use more than one link so the network is faster and no single cable can take everything down. The difference is where they operate and what the rest of the network can see. Link aggregation fuses physical links into one logical interface at Layer 2, so the routing protocol sees a single neighbor. ECMP leaves the links as independent routed paths at Layer 3 and lets the routing protocol install several equal cost next hops for the same prefix. That distinction drives everything else, including how failures are detected, how traffic is distributed, and how much path level visibility you keep. This guide walks through both mechanisms, explains what the hash actually decides, and shows how to combine them without creating the imbalance people fear.

At a Glance: Link Aggregation vs ECMP

What Link Aggregation Actually Does

Link aggregation, standardized as IEEE 802.1AX after starting life as 802.3ad, combines multiple physical Ethernet links between two endpoints into a single logical link. Every major vendor has its own name for it. Cisco calls it EtherChannel or a port channel, Juniper calls it a bundle, Linux calls it bonding, and Windows calls it NIC teaming, but the idea is identical: several cables become one interface for the purpose of configuration, Spanning Tree, and routing. The logical link carries the combined capacity of its members and stays up as long as enough members remain healthy, which is why it shows up everywhere from the access switch to the server NIC to the uplink pair between two closets.

The protocol that makes this safe is LACP, the Link Aggregation Control Protocol, which exchanges LACPDUs between the two ends so each side knows which links the other side considers part of the bundle. LACP negotiates membership rather than assuming it, which prevents the classic failure where one end bundles four links while the other bundles three and the mismatched link becomes a loop. It runs in active or passive mode, and by default it uses slow timers. Manual, or static, bundles skip negotiation entirely and are best avoided unless a device genuinely cannot speak LACP, because they offer no protection against a one sided misconfiguration. Members must also agree on speed, duplex, MTU, and VLAN settings, and most platforms suspend a member that does not match the rest.

Multichassis LAG extends the same idea across two physical switches so that a server or access switch can bundle links to two different chassis and survive the loss of either one. Cisco sells this as vPC or VSS, Arista as MLAG, and Juniper as MC-LAG. It is genuinely useful, but it adds a peer link between the two chassis that becomes a critical dependency, plus a set of failure modes around orphan ports and split brain that a plain routed design does not have. Reach for it when a device truly can only bundle to one logical neighbor, not as a default.

What ECMP Actually Does

ECMP is a routing concept rather than a link protocol. When a routing protocol calculates two or more paths to the same destination with an identical metric, ECMP lets the router install all of them instead of choosing one and discarding the rest. In OSPF and IS-IS this often happens automatically whenever the topology is symmetric, because equal bandwidth links produce equal cost by default. BGP needs explicit multipath configuration, and it will only use paths that are equal in the attributes it considers, which is why operators tune the maximum path count and relax the as path check for internal sessions. The forwarding table then holds several next hops for that prefix, and the hardware chooses between them per flow.

The structural difference from a LAG is that each ECMP path remains a real routed link. It has its own subnet or point to point addressing, its own routing adjacency, and its own place in the topology, which is exactly why ECMP scales so well in a leaf and spine fabric. A leaf with four uplinks to four spines running ECMP has 400 Gbps of aggregate uplink bandwidth if each link is 100 Gbps, and it got there without bonding anything. The routing protocol sees four independent paths, so it can react to each one individually. That independence is also why ECMP can use links that terminate on different devices, something a plain LAG cannot do without multichassis tricks.

The Hash Decides Which Link Carries a Flow

Both mechanisms distribute traffic with a hash, and understanding that hash is the key to setting expectations. A forwarding device reads a set of header fields, runs them through a hash function, and uses the result to select a member of the bundle or an ECMP next hop. The fields typically include source and destination MAC addresses and VLAN for Layer 2, source and destination IP and protocol for Layer 3, and source and destination TCP or UDP ports for Layer 4. Tunneled traffic is the complication, because a router often has to hash on the outer header unless it is configured to look inside, which is why encapsulation protocols carry an entropy field such as the VXLAN source port.

Because the selection is per flow, every packet in a given conversation follows the same member, which preserves packet order and keeps TCP happy. It also means two 10 Gbps links do not add up to a 20 Gbps transfer. The aggregate capacity is real, and many independent flows will use it, but any single flow is capped at the speed of the one link the hash chose. A small number of very large flows, sometimes called elephant flows, can therefore leave one member near saturation while the others run lightly, and that is normal behavior rather than a fault. When you see a 60/40 or 70/30 split in member counters, the question is not whether the split is uneven but whether the traffic mix and hash configuration explain it.

Two refinements matter in production. The first is that a naive hash mod N remaps almost every flow when the number of members changes, so losing one link out of four can disturb far more traffic than the link itself carried. Platforms address this with resilient or consistent hashing, which pre allocates buckets and remaps only the flows that belonged to the changed member. The second is hash polarization, where several stages in a path make the same choice and effectively shrink the paths available. It is why some platforms use a different seed or offset for the bundle hash than for the ECMP hash. Fragmentation is a related trap, since later IPv4 fragments do not carry Layer 4 ports, so a router that hashes on ports can send them down a different link than the first fragment and cause reordering.

Failure Detection and Path Visibility

The two mechanisms fail in different ways, and that difference often matters more than raw bandwidth. A LAG detects member loss through LACP and physical link state, and because the bundle is one logical interface, the routing adjacency normally survives as long as at least one member is healthy. LACP fast timers cut detection to a few seconds, and a minimum link threshold lets you fail the whole bundle on purpose rather than limp along with a single member and no real redundancy. The tradeoff is that the member links are invisible to routing. The protocol sees one interface, so per member policy, metrics, and telemetry are limited.

ECMP has the opposite profile. Each path is independent, so each one gets its own failure domain, its own counters, and its own place in the routing policy. The cost is that a failure is not free: the path has to be withdrawn by the routing protocol or detected by BFD, and until it is, traffic can be black holed into a link that is no longer forwarding. Interior gateway protocol timers are often measured in seconds, which is why BFD, with sub second detection, is standard in fabrics that run ECMP. ECMP also multiplies control plane state. Four uplinks mean four adjacencies to maintain, not one, and platform limits on the number of equal cost paths become a real constraint as the fabric grows.

Combining Link Aggregation and ECMP

The two are not rivals in a real network, and most large designs use both at once. A router can run ECMP across several next hops where each next hop is itself a LAG, so traffic is hashed first to a path and then to a member inside that path. A leaf might have two uplinks to each of two spines, with the pair to each spine bundled and ECMP choosing between the two bundles. This gives you the redundancy of independent paths and the simple cabling of bundles, at the cost of a second hash stage that has to be tuned so it does not simply repeat the first decision.

The way to combine them well is to let each mechanism do the job it is best at. Use ECMP where paths are routed and independent, and reserve bundles for places where a single logical interface genuinely simplifies the design, such as a server dual homing to a switch pair or a Layer 2 trunk that must cross a link. The trend in modern data center fabrics pushes even further: instead of bundling uplinks between a leaf and a spine, many operators run a separate Layer 3 link to each spine, one BGP session per link. That keeps every path visible, removes LACP from the uplink entirely, and lets the fabric converge on individual link state rather than a bundle threshold.

Choosing the Right Approach for Each Link

Reach for link aggregation when two devices sit next to each other, when you need one logical interface and one routing adjacency, or when Layer 2 extension is the requirement. It is also the natural choice for server connectivity, where a dual port NIC or a pair of adapters bonds into one interface and the switch sees a single logical neighbor. Multichassis LAG is justified when a device cannot run routing or when operational simplicity outweighs the extra failure modes of the peer link. In each of these cases the bundle is doing real work: it hides member churn from the network and turns several cables into one manageable thing.

Reach for ECMP when the paths are routed, when they terminate on different devices, or when you want per path visibility, policy, and telemetry. It fits leaf and spine fabrics, WAN designs with multiple transports of equal cost, and cloud or transit connections where you want the routing protocol to react to each link. Prefer independent routed links over a routed bundle when you value fault isolation and horizontal scale, and accept the extra adjacencies and addressing that come with them. The honest answer is that neither is automatically faster. Both are per flow, both are limited by hashing, and the choice is really about failure domains, path visibility, and how much control plane state you are willing to run.

Designing for Real Throughput

The effectiveness of either mechanism depends far more on the traffic than on the configuration. If your workload is a handful of large flows, no amount of bundling will spread them evenly, because the hash assigns each flow to one member. If it is thousands of short connections, even a modest bundle will look well balanced. Before you promise a team that two links will double their throughput, look at the flow profile. That conversation prevents the classic disappointment where a storage replication stream, which is one long lived flow, refuses to exceed the speed of a single member no matter how many links you add.

Tune the hash to match the traffic. Including Layer 4 ports gives the algorithm more entropy and better spread for the many flow case, but it can split fragments of the same datagram across members, so avoid it on paths that carry fragmented UDP or tunneled traffic unless you also carry entropy in the outer header. Verify what you configured rather than assuming the default, because many platforms ship with a conservative hash that ignores ports. Then measure properly: check per member or per path counters over time, and test with a realistic mix of parallel and single streams instead of a single speed test that will always ride one link.

On the operational side, enable fast detection and decide what should happen when capacity drops. A LAG with a minimum link threshold fails predictably instead of silently running at reduced capacity, and BFD gives ECMP the sub second reaction that routing timers alone cannot. Watch for MTU and fragmentation issues at the boundaries, keep the path MTU consistent, and clamp TCP MSS where tunnels are involved. Finally, record the design itself, because the whole arrangement depends on assumptions about which links are bundled and which paths are equal that are easy to break accidentally.

Documenting Bundles and Paths

The design only holds if you know what it is. A replaced cable, a firmware change, or a well meaning engineer who moves a port can quietly remove a member from a bundle or change a path’s metric so that it is no longer equal cost, and nothing in the daily traffic will announce it. The link that was supposed to be the redundant path may still be plugged in and still be down, or may be forwarding with a higher metric that takes it out of the ECMP group entirely. Without a record of the intended design, these changes surface only during an incident, which is the worst time to discover that a bundle has been running on one member for months.

Record the things that make the design work: which physical ports belong to each bundle, its LACP mode and minimum link threshold, which routed paths form each ECMP group, and the metric that makes them equal. Keep that beside your port and cable records so a maintenance window can be checked against the intended topology before anyone touches a cable. Teams that keep cable and port documentation current can see at a glance whether a bundle still has all of its members and whether a change preserved the diversity the design assumed, which is far cheaper than reconstructing it from counters after a failure.

What to Take Away

Link aggregation and ECMP both multiply connections, and the reason they sometimes disappoint is that both distribute per flow rather than per packet. A bundle gives you aggregate capacity and a survivable logical link between two neighbors, while ECMP gives you independent routed paths with real visibility and fault isolation. Neither makes a single conversation faster than the link it lands on. Once you internalize that the hash, not the cable count, decides how traffic moves, the uneven splits and the capped single stream stop looking like bugs and start looking like a design you can tune.

The practical next step is to audit one bundle or one ECMP group this week against what you believe it should be. Count the members, confirm the hash fields, check whether every path still has an equal metric, and verify that failure detection is fast enough for what the link carries. You will occasionally find a surprise, and finding it on a quiet afternoon is a very different experience from finding it during an outage.