Spine Leaf vs Three Tier: Oversubscription and Capacity Math

A data center design review rarely argues about whether the network should work. It argues about the number that decides how much it costs: the oversubscription ratio. One engineer wants 1:1 everywhere because storage replication and AI training are saturating links, another points at the bill for optics and asks why every server needs a dedicated path to every spine, and the room settles on a compromise that nobody can defend six months later when the fabric is full.

The argument is hard because the two architectures hide their costs in different places. A three tier design hides oversubscription in blocked links and in ratios that compound at every layer. A spine leaf design hides it in port count, fiber, and optics, because every leaf needs a link to every spine whether that link carries traffic today or not. This guide puts both designs through the same arithmetic, then shows how to verify the numbers on the equipment you already have.

At a Glance: Three Tier vs Spine Leaf

What Each Design Is Actually Doing

A three tier network separates three jobs. Access switches connect endpoints and provide port density. Distribution switches aggregate access switches, apply policy, and form the boundary between Layer 2 and Layer 3. Core switches move traffic between distribution blocks and toward the WAN or internet. The separation is useful because each layer can be sized and replaced on its own, and because policy has a natural home at distribution.

A spine leaf network collapses access and distribution into the leaf, then connects every leaf to every spine. The spine layer only switches between leaves, so it carries no endpoint ports and no policy that depends on where a host lives. Every leaf to spine link is a routed point to point link, and ECMP spreads flows across all of them. The result is a fabric where every server is the same number of hops from every other server, which is why storage, virtualization, and AI workloads tend to fit it well.

The drawing makes the cost difference obvious. The three tier design can be built with two uplinks from each access switch and a pair of distribution switches, and spanning tree will keep one of those paths idle unless you use LAG or a routed design. The spine leaf design leaves no cabled uplink idle, but it needs a link from every leaf to every spine, and that link count grows with both dimensions. Neither property is free, and the rest of this article turns both into numbers.

The Oversubscription Formula

Oversubscription compares the bandwidth offered to endpoints with the bandwidth available to carry it upstream. For a leaf or access switch, the formula is:

oversubscription = access bandwidth / uplink bandwidth

Access bandwidth is the sum of the connected port speeds facing servers. A 48 port 25G leaf with all 48 ports connected has 1,200 Gbps of access. If only 24 ports are connected, the access side is 600 Gbps even though the switch has 48 ports, and that distinction matters when the design is compared with a switch datasheet. With four 100G uplinks, the uplink side is 400 Gbps, so the ratio at full fill is 1,200 / 400, or 3:1, and at half fill it is 600 / 400, or 1.5:1.

The ratio is directional. A full duplex 100G link carries 100G in each direction, so count it once per direction, not twice. If you are calculating the ability to absorb traffic from servers into the fabric, use the access side as the numerator and the uplink side as the denominator. If you are calculating how much traffic can reach a server, the same numbers apply in reverse, which is why a symmetric design is easier to reason about.

Do not size the ratio at 100 percent utilization. A link that is 100 percent full has no room for a rerouted flow, a backup job, or a microburst, and it will drop or pause traffic before the average graph looks alarming. Plan the uplink to run at about 70 percent at peak, which means a nominal 3:1 design behaves like 4.3:1 when you need headroom. That gap is the difference between a design that survives a spine failure and one that only survives a quiet Tuesday.

The right ratio depends on the workload, not on a vendor best practice slide. A rack of web servers that mostly talk to a load balancer tolerates 6:1 better than a rack of database nodes replicating to each other at line rate. The useful question is not “what ratio is best” but “which flows cross the uplink, how large are they, and how much contention can the application tolerate before latency becomes visible”. A storage cluster with synchronous replication will tell you quickly, and a batch analytics job will hide the problem until the month end run.

Why Three Tier Ratios Compound

In a three tier design, a flow from a server in one access block to a server in another crosses at least two oversubscription points: access to distribution, and distribution to core. Each tier has its own ratio, and the ratios multiply when both tiers are saturated. If access is 3:1 and distribution is 3:1, the planning bound end to end is 9:1. That does not mean every flow sees 9:1, because not every flow is active at once, but it does mean the fabric has three places to queue instead of one.

The compounding is easy to miss because each tier looks reasonable in isolation. A 3:1 access layer is normal. A 3:1 distribution layer is normal. Put them together, add a core that was sized for north south traffic, and the first east west workload that crosses blocks discovers a ceiling that nobody wrote down. The table below is the arithmetic a design review should be arguing about.

The table is deliberately small. The point is the multiplication, and the fact that the second tier can quietly change the first. If the same network uses 4 x 100G from distribution to core, the end to end bound becomes 6:1, and 2 x 100G would make it 12:1. That is fine for north south web traffic and painful for a storage cluster that expects to move a full backup window across blocks.

Three tier also pays a redundancy tax that does not show up in the ratio. Spanning tree blocks redundant links to prevent loops, so a pair of uplinks may provide one active path and one standby path. LAG bundles both links into one logical path, but the bundle is still a pair, and a distribution switch failure can take out a larger block than a single leaf failure would. The design is not wrong, it is simply optimized for a different traffic pattern and a different failure domain.

Sizing a Spine Leaf Fabric

Spine leaf looks simpler until you count ports. Start with the leaf. A 48 port 25G leaf with 8 x 100G uplinks gives you 48 server ports, 1,200 Gbps of access, and 800 Gbps of uplink capacity if all eight uplinks are used. That is 1.5:1. If you only use four of the eight uplinks, the same leaf is 3:1, and the other four ports are spare capacity or a future spine connection.

Now choose the spine. The rule that catches people is that every spine needs a port for every leaf link, not every leaf. With 48 leaves and one link per leaf per spine, each spine needs 48 ports. A 64 port spine can do that with 16 ports left. Four such spines give every leaf four 100G uplinks, or 400 Gbps, which is 3:1 against 1,200 Gbps of access when all 48 leaf ports are populated. Eight spines give 800 Gbps, or 1.5:1 at full population, and each spine still needs only 48 ports.

The table exposes the trade. You can reach 800 Gbps per leaf with eight 64 port spines and one link per spine, or with four 128 port spines and two links per spine. The first option uses more spine chassis with fewer ports each, the second uses fewer, larger chassis. Both use the same number of leaf uplink ports, 384, because the total is leaves times uplinks. The difference is where those ports live and how much bandwidth one spine failure removes.

For the worked example, use 24 racks with two leaves per rack, 24 dual homed servers per rack, 48 leaves in total, four 64 port spines, and one 100G link from every leaf to every spine. Each leaf has 24 server ports in use at 25G, so 600 Gbps of connected access, and four 100G uplinks, so the ratio at current fill is 1.5:1. The same leaf has 48 ports and can reach 1,200 Gbps when fully populated, at which point the ratio becomes 3:1. The fabric has 192 leaf to spine links, 384 optics, and 48 ports used on each spine. That is a coherent design, not a compromise assembled from spare parts.

Port, Cable, and Optic Counts

The bills that surprise people are not the switch line items. They are the optics, the patch panels, the fiber trays, and the time spent labeling and testing every link. Count them before the purchase order, not after.

The formulas are short:

fabric links   = leaves x spines x links per leaf per spine
fabric optics  = fabric links x 2
spine ports    = leaves x links per leaf per spine
leaf uplinks   = spines x links per leaf per spine

Using the 48 leaf, 4 spine, one link per spine example: 192 fabric links, 384 optics, 48 spine ports per spine, and 4 uplinks per leaf. Add the server side. If 24 servers per rack are dual homed to the two leaves, that is 24 x 2 x 24 racks, or 1,152 server links and 2,304 server optics. The fabric optics are a small fraction of the total, which is why a spine leaf design is often limited by server connectivity cost rather than by the spine layer itself.

Distances decide the optic type. A DAC cable is normally the cheapest option within a rack or between adjacent racks, an active optical cable or a short range optic covers longer runs inside a row, and single mode optics cover the building or campus distances. The exact reach and speed combinations are vendor specific, so verify the optic part numbers against the switch datasheet rather than assuming a cable that worked on one platform will work on another.

What Fails, and What It Costs

A ratio is a steady state number. The number that matters during an incident is the ratio after a component is gone. In a spine leaf fabric, a leaf failure affects the servers in that rack, while a spine failure affects every leaf at once, but only by one spine’s worth of uplink capacity.

With four spines and one link per spine, each leaf has 400 Gbps of uplink. Lose one spine and the leaf has 300 Gbps, so the ratio moves from 1.5:1 to 2:1 at current fill, or from 3:1 to 4:1 when the leaf is fully populated. With eight spines at one link each, losing one moves 800 Gbps to 700 Gbps, which is still above the 600 Gbps currently connected and about 1.7:1 at full population. With four spines and two links per spine, losing one moves 800 Gbps to 600 Gbps, or 1:1 at current fill and 2:1 at full population. The arithmetic is the same in every case: remaining uplinks divided by remaining spines.

The leaf failure math is different because it depends on how servers are connected. A server with two NICs, one to each leaf in a pair, keeps one path and half its bandwidth when a leaf dies. A server with one NIC loses connectivity entirely. MLAG or EVPN multihoming can make the pair look like one logical switch and speed up convergence, but it cannot create a path that the server does not have. If the application cannot tolerate a single leaf failure, the server needs two links to two leaves, and the rack needs a plan for what happens when one of those leaves is upgraded.

The three tier failure story is broader. A distribution switch failure can remove a block of access switches, and a core failure can affect every distribution block that uses it. Spanning tree or LAG convergence adds time on top of the capacity loss. That is not an argument against three tier, it is an argument for measuring the failure domain in racks rather than in links.

Verifying the Fabric From the Devices

The design document is a hypothesis. LLDP, BGP, and interface counters are the evidence. Start at one leaf and confirm that it sees every spine, then compare the count with the design. If a leaf sees three spines when the drawing says four, you have found a cabling or configuration error before it becomes a capacity surprise.

On Cisco NX-OS, the neighbor table is the quickest check. The exact columns vary by release, but the device ID, local interface, and remote port are the fields that matter.

leaf01# show lldp neighbors
Device ID       Local Intf    Hold-time  Capability  Port ID
spine01         Eth1/49       120        BR          Eth1/49
spine02         Eth1/50       120        BR          Eth1/50
spine03         Eth1/51       120        BR          Eth1/51
spine04         Eth1/52       120        BR          Eth1/52

Four rows, four spines, four uplinks. If you run Arista EOS, the same check is show lldp neighbors with Port, Neighbor Device ID, and Neighbor Port ID columns. On Junos, use show lldp neighbors and read Local Interface, Chassis Id, and System Name. The command is the same idea on every platform; the column order is not.

Next, confirm that the routing protocol agrees with the cabling. A physical link without a session is still a broken path, and a session without a route is a different kind of broken path.

leaf01# show ip bgp summary
Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.0.1        4 65000    1842    1840       42    0    0 12d04h   128
10.0.0.5        4 65000    1841    1839       42    0    0 12d04h   128
10.0.0.9        4 65000    1840    1838       42    0    0 12d04h   128
10.0.0.13       4 65000    1839    1837       42    0    0 12d04h   128

The state column is the signal. A number means the session is established and that many prefixes were received. Active, Idle, or Connect means the link may be up while the control plane is not, and that is exactly the failure a cable test will not find.

Then check that the fabric is actually using all of the paths. A route with four next hops is ECMP working. A route with one next hop is a design that will not deliver the ratio you planned, even if every cable is plugged in.

leaf01# show ip route 10.20.0.0/24
  *via 10.0.0.1,  [20/0], 12d04h, bgp-65000, external
  *via 10.0.0.5,  [20/0], 12d04h, bgp-65000, external
  *via 10.0.0.9,  [20/0], 12d04h, bgp-65000, external
  *via 10.0.0.13, [20/0], 12d04h, bgp-65000, external

Finally, measure the uplink instead of trusting the label. On Arista EOS, show interfaces counters rates gives a per port rate view. On NX-OS, use show interface Ethernet1/49 and read the 30 second rate lines, or the equivalent on your platform. A port that negotiated 100G but is passing 38 Gbps is not the same as a port that is idle, and a port that is passing 95 Gbps is already in the danger zone regardless of the design ratio.

leaf01# show interfaces counters rates
Port      InMbps  OutMbps  InPct  OutPct
Et49      38210   36140    38.2   36.1
Et50      37420   35880    37.4   35.9
Et51      36910   35220    36.9   35.2
Et52      38120   36010    38.1   36.0

The column names and the percentage base vary by platform, so confirm them against your vendor’s documentation. What matters is the peak, not the sample you happened to take at 10 a.m. Collect the same output every five minutes for a week before you declare the fabric healthy.

To make the checks repeatable, write the output into a dated directory and parse the same shape every time. One row per leaf, with the number of spine neighbors and the number of ECMP next hops, is enough to spot a cabling or routing change.

STAMP=2026-10-29
mkdir -p "audit/${STAMP}"

# NX-OS and EOS: suppress paging so the capture does not hang
for sw in leaf01 leaf02 leaf03 leaf04; do
  ssh "$sw" "show lldp neighbors | no-more" > "audit/${STAMP}/${sw}-lldp.txt"
done

for f in "audit/${STAMP}"/leaf*-lldp.txt; do
  leaf=$(basename "$f" -lldp.txt)
  printf '%s\t%s\n' "$leaf" "$(grep -c 'spine' "$f")"
done

The expected output is one line per leaf with the spine count. If the design says four spines and a leaf reports three, do not adjust the spreadsheet. Fix the link or the session, then rerun the check. The same pattern works for BGP sessions: count the established neighbors per leaf and compare with the spine count.

Recording the Fabric in Obelinf

The arithmetic above is only useful if the inventory agrees with it. A fabric diagram that was correct at cutover and has been edited by hand ever since is worse than no diagram, because it gives the team confidence in a number that nobody verified. The cable records and rack positions are the inputs to the same calculation, so keep them in one place and query them rather than rebuilding the picture from memory.

Start with the cables. The cable tracking list in Obelinf returns both endpoints and the device names, which is exactly the shape needed to count leaf to spine links. The pipeline below writes one row per cable and then counts the links that start at a leaf and end at a spine.

export STAMP=2026-10-29
export OBELINF_ORG="acme-netops"
export OBELINF_API_KEY="your_read_only_key"
export API="https://api.obelinf.com/${OBELINF_ORG}"

curl --fail --silent --show-error \
  -H "Authorization: Bearer ${OBELINF_API_KEY}" \
  "${API}/cables?all=true" \
  | jq -r '.cables[]
      | select(.term_side_a_device_name != null and .term_side_b_device_name != null)
      | [.term_side_a_device_name, .term_side_b_device_name, (.type // ""), (.status // "")]
      | @tsv' > "audit/${STAMP}/fabric-links.tsv"

awk -F'\t' '$1 ~ /^leaf/ && $2 ~ /^spine/ { count[$1]++ }
  END { for (leaf in count) print leaf "\t" count[leaf] }' \
  "audit/${STAMP}/fabric-links.tsv" | sort

Each line is a leaf and its documented spine links. Compare that count with the LLDP output from the same day. A mismatch is either a missing cable record or a link that was never installed, and both are worth fixing before the next capacity review. The same data feeds the network topology view, so the diagram a manager sees and the count an engineer audits come from the same rows rather than two separate truths.

Rack records add the physical constraint. The rack management list returns the rack height and the number of front and rear units already used, which is the fastest way to see whether a leaf pair will fit in the rack you planned.

curl --fail --silent --show-error \
  -H "Authorization: Bearer ${OBELINF_API_KEY}" \
  "${API}/racks?all=true" \
  | jq -r '.racks[] | [.name, .height, .front_used, .rear_used] | @tsv'

A 48U rack with 44U used has room for a pair of 1U leaves but not for the cable management and power upgrades that usually arrive with them. That is a planning fact, not a monitoring alert, and it belongs next to the fabric design.

The First Step

Pick one leaf and run the LLDP check today. Count the spines it sees, compare that count with the design, then read the uplink rates and compute the real ratio from the actual port speeds rather than the planned ones. If the number is worse than the drawing, you have found the next capacity conversation. If it matches, you have a baseline that will still be useful when the first spine fails.

The design choice between three tier and spine leaf is less interesting than the arithmetic that follows it. Three tier compounds its ratios across tiers and pays for redundancy in blocked links. Spine leaf pays for redundancy in ports and optics, but every link carries traffic and every server is the same distance from every other server. Pick the one whose costs match the traffic pattern, write the ratio down, and then verify it on the equipment. A fabric that matches its documentation is worth more than a fabric that matches a diagram.