ASN Planning for Multi-Site Networks

A network that grows from one data center to four stops being a network and becomes a routing problem. Each new site brings its own transit, its own peering options, and its own reason to announce prefixes, and if the ASN and prefix plan was never decided, every site invents its own. The symptoms are familiar: a site that cannot be summarized because its prefix sits in the wrong spot, an internal route that leaks to a transit provider, and a peer that filters your announcement because nobody registered the origin.

This is a planning guide for that moment. It covers the ASN decision, the prefix allocations that make summarization possible, the internal control plane that connects the sites, and the verification that proves a newly built site is announcing exactly what it should before it announces anything else.

At a Glance: Multi-Site ASN Approaches

There is no universally right answer, but there is a wrong order. Decide the ASN model and the prefix plan before you order anything, because the prefix layout is the part that is expensive to change once it is live.

The ASN Decision Comes Before the Design

The ASN question resolves to one test: does each site need to be a distinct autonomous system on the public internet. A site needs its own public ASN when a peer, a customer, or a regulator must see it as a separate policy domain, such as when the site peers at its own exchange under its own identity, or when local ownership means the site negotiates its own transit independently.

Most networks land on one public ASN for locally owned sites and a private ASN for any part that never faces the internet. That is the lowest cost model and the reason is operational: a peer filters on one origin, a registry record is maintained once, and a nighttime change at one site follows a policy that already exists elsewhere.

If you do split into multiple public ASNs, know what the registry expects. ARIN issues one ASN per discrete network under its Multiple Discrete Networks policy and wants proof of a unique routing policy for anything beyond that. RIPE NCC expects each ASN to be multihomed. Plan the justification before the sites are built, not after.

Allocate Prefixes So the Sites Summarize

Prefix planning is the part that quietly decides how much of the global table you consume. A site prefix that sits next to another site prefix summarizes into one announcement. A site prefix scattered across space does not, and the more specifics you announce, the more state every peer has to carry on your behalf.

The rules that keep this clean are simple. Reserve a larger block per region and carve site prefixes from the inside, keeping each site aligned to its prefix boundary. Keep each site’s IPv4 at a /24 or larger, because many networks filter anything longer. For IPv6, give each site a /48 out of your allocation, which is the smallest unit that propagates reliably almost everywhere.

Plan the aggregates as if the number of sites will double. It costs nothing to reserve a /22 for a region and announce one /23 from it, and it saves a renumbering exercise when the third site in that region appears.

A site only needs to announce its specific prefix when a peer requires a more specific route, which is rare. Aggregate first, add specifics only when a real reachability problem forces it, and record why if you do.

router bgp 393216
 address-family ipv4 unicast
  aggregate-address 198.51.100.0/23 summary-only
  aggregate-address 203.0.113.0/23 summary-only

With summary-only, the aggregate is advertised and the covered specifics are suppressed, which is what you want while all four sites are up. If a site goes down, the aggregate keeps working because the other site in the pair still carries it.

Build the Internal Control Plane

Internal BGP connects the sites, and the classic mistake is a full mesh that nobody wants to maintain once the fourth site appears. A full mesh requires every pair of routers to be configured, so the session count grows with the square of the site count. Two route reflectors remove that, because each site peers with the reflectors and the reflectors redistribute routes between them.

On a route reflector, each site is a client. The session carries your own ASN on both ends, and next-hop-self makes sure a client always sees the reflector as the way to reach a prefix rather than an unreachable next hop inside another site.

# FRR
router bgp 393216
 bgp router-id 10.0.0.1
 neighbor 10.0.0.2 remote-as 393216
 neighbor 10.0.0.2 update-source lo
 address-family ipv4 unicast
  neighbor 10.0.0.2 activate
  neighbor 10.0.0.2 route-reflector-client
  neighbor 10.0.0.2 next-hop-self

Verify the control plane from the reflector before you trust the edges. You want every site session established. You also want your aggregates to appear in the empty path list, because a prefix you expect to originate but cannot find there is being learned from somewhere else.

# FRR
vtysh -c "show bgp ipv4 unicast summary"
vtysh -c "show bgp ipv4 unicast regexp ^$"

BGP router identifier 10.0.0.1, local AS number 393216
Neighbor        V         AS   MsgRcvd   MsgSent  TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.0.2        4      393216     18244     18250       0    0    0 3d02h          42
10.1.4.1        4      393216      9042      9050       0    0    0 3d02h          18

The State/PfxRcd column is the one to read. A number means the session is established and that many prefixes were accepted. Active or Idle means a transport or reachability problem, not a policy one.

If a site also needs its own policy domain, a confederation avoids forcing one flat ASN on everyone. The public identity stays a single number, while each part runs its own internal number.

router bgp 65100
 bgp confederation identifier 393216
 bgp confederation peers 65101
 neighbor 10.0.0.9 remote-as 65101

Confederations add real complexity, so use one only when the policy difference is genuine. A flat domain with route reflectors is easier to reason about for most organizations.

Give Each Site a Defined External Role

An internal control plane means nothing without a clear external role at each site, because that role decides what the site announces and what it accepts. Write it down before the site is built, and keep the list short enough that an engineer can read it at 3 in the morning.

The pattern that scales is to announce the same aggregate from every site that has transit. That way a transit failure moves traffic to another site instead of blackholing the prefix, and every site carries the same origin, which is what your ROA authorizes anyway. The exception is a site with a single upstream and no diversity, where announcing locally still works but the failure mode is a full outage at that site.

Do not accept a default route from a peer at an exchange. Peers announce specific prefixes, and a default route accepted from a peer turns the exchange into an unintended transit path that the other side did not agree to.

Keep Policy Identical at Every Site

The failure that hurts most in a multi site network is policy drift: one site filters correctly, another does not, and the difference only shows up in an incident. Define the policy once and push the same prefix lists and community plan to every edge.

ip prefix-list PL-CUSTOMER-ROUTES permit 198.51.100.0/23
ip prefix-list PL-CUSTOMER-ROUTES permit 203.0.113.0/23

route-map RM-PEERS-IN permit 10
 match ip address prefix-list PL-CUSTOMER-ROUTES
 set local-preference 120

ip prefix-list PL-BOGONS deny 0.0.0.0/8 le 32
ip prefix-list PL-BOGONS deny 10.0.0.0/8 le 32
ip prefix-list PL-BOGONS deny 172.16.0.0/12 le 32
ip prefix-list PL-BOGONS deny 192.168.0.0/16 le 32
ip prefix-list PL-BOGONS permit 0.0.0.0/0 le 32

Apply the same policy at every site and verify it the same way. Compare what a router actually advertises against the plan, because a route map that exists in the configuration and a route map that is attached to a neighbor are not the same thing.

# Cisco IOS
show ip bgp neighbors 192.0.2.1 advertised-routes

     Network          Next Hop            Metric LocPrf Weight Path
*>   198.51.100.0/23  0.0.0.0                  0         32768 i
*>   203.0.113.0/23   0.0.0.0                  0         32768 i

Two aggregates, empty path, weight 32768: those are locally originated and correctly summarized. A longer list here, or a more specific route appearing next to the aggregate, is the signal that a site is announcing more than the plan allows.

The same discipline covers origin validation. If you have one ROA per aggregate with the exact origin and length, every site must originate that aggregate with that ASN and nothing longer, or the announcement becomes invalid. A useful check before changing anything at a site is to look at your own prefix from the outside.

vtysh -c "show bgp ipv4 unicast 198.51.100.0/23"

If the best path to your own aggregate is learned from a peer rather than originated locally, routing is looping or the local announcement is being suppressed, and neither is something to discover during a failover.

Model Each Site Against Its ASN

The plan lives in configuration, but the relationships do not, and that is what a provider migration or an audit will ask about. Which sites announce this ASN. Which providers carry it. Which prefixes belong to which site. Keeping that in one place turns a painful reconstruction into a lookup, and an inventory that links ASNs to the sites and circuit providers they touch keeps a multi site network coherent as sites are added.

If your inventory is in Obelinf, the site relationships are explicit records rather than a diagram. List the sites to get their IDs, then attach each one to the ASN.

export OBELINF_API_KEY="your_read_write_key"
export OBELINF_ORG="acme-netops"

curl --fail --silent --show-error -H "Authorization: Bearer ${OBELINF_API_KEY}" \
  "https://api.obelinf.com/${OBELINF_ORG}/sites?all=true" \
  | jq -r '.sites[] | [.id, .name] | @tsv'

# Attach a site to the ASN that announces it
curl --fail --silent --show-error -X POST \
  -H "Authorization: Bearer ${OBELINF_API_KEY}" \
  -H "Content-Type: application/json" \
  "https://api.obelinf.com/${OBELINF_ORG}/asns/${ASN_ID}/sites" \
  -d '{"site_id":"'"${SITE_ID}"'"}'

The detail endpoint returns the sites and providers attached to the ASN, which is the quickest way to answer “which parts of the network does this number belong to” during a change review.

curl --fail --silent --show-error -H "Authorization: Bearer ${OBELINF_API_KEY}" \
  "https://api.obelinf.com/${OBELINF_ORG}/asns/${ASN_ID}" \
  | jq -r '.asn.sites[].name, "providers:", .asn.providers[].name'

Bring Each Site Up Without Leaking

A new site joins the network in a fixed order, and skipping a step is how a private route reaches the public internet.

1. Establish the internal session first. Bring up iBGP to both reflectors and confirm the session shows a prefix count. The site now has the internal table but announces nothing externally.
2. Confirm the filters before the external session. Attach the inbound and outbound prefix lists and route maps, then check that the outbound policy permits only your aggregates.
3. Open the external session and watch what you send. Check advertised-routes immediately, before the peer has accepted anything, to confirm only the aggregates leave.
4. Verify the announcement from outside. Query a public looking glass or an RPKI validator to confirm your aggregate is visible and valid from a vantage point you do not control.
5. Only then add the site to the failover plan. A site that is not yet verified should not be a target for traffic during another site’s maintenance.

# BIRD, checking what a site is actually advertising
birdc show protocols all edge_transit
birdc show route table master4

The last check is the one teams forget, and it is the reason a private range occasionally appears on the internet. Query your own prefix from outside, confirm the origin is your ASN, and confirm the aggregate is the only thing visible.

The first concrete step is the prefix plan rather than the config. Open your address plan today and mark which blocks are contiguous enough to summarize, which sites sit in the wrong place, and what a third site in each region would need. Fix the plan on paper before the next router is ordered, because a prefix that summarizes cleanly saves the same work at every site that follows.