Disaster Recovery Site Planning: Distance, Replication and Failover

Your primary site is healthy today, which is exactly why disaster recovery planning is easy to postpone until the week you need it. A good recovery plan is a set of deliberate tradeoffs made while you have time to measure them: how far apart your sites must be to survive the same event, how much data you can afford to lose in the seconds between writes, and how quickly a second location can take over when the first one cannot. Those decisions interact with each other too. Distance constrains replication, replication constrains recovery time, and recovery time constrains where a failover site is actually useful.

This guide walks through the three decisions that decide whether a recovery site works in practice. You will see the geo distance rules that insurers and auditors actually check, how synchronous and asynchronous replication differ in real world behavior, and how to choose a failover site that is far enough to be safe, close enough to be reachable, and connected enough to take production load without a surprise.

Why Distance Is a Design Variable

A disaster recovery site that shares the same risk footprint as your primary is not a recovery site, it is a second building that fails for the same reason. Flood plains, wildfire corridors, seismic zones, hurricanes and ice storms, and the power grid that feeds them all create correlated risk that distance is meant to break. The question is not how far you can afford to place a site, it is how far you must place it to make the failures you care about uncorrelated.

Distance also sets the physical cost of your plan. Every kilometer adds propagation delay to any replication that must be acknowledged before a transaction completes, it adds latency to user traffic after failover, and it adds travel time for the people who may need to physically touch equipment. A plan built around a site on the other side of the continent survives a regional event beautifully and fails the test of getting a spare part to a failed array at two in the morning. The best plans make those tradeoffs explicitly instead of discovering them during an incident.

Geo Distance Rules That Shape Site Placement

You will see three distance bands in practice because they map to different failure modes and different replication options. Metro sites under roughly 100 kilometers give you synchronous replication with single digit millisecond round trips and fast failover, which is ideal for the building fire, power substation failure, or fiber cut that takes down one facility without affecting the city. The catch is that a metro site still sits inside the same hurricane track, the same river basin, and often the same power transmission corridor, so it does not protect against the regional events that make headlines.

Regional sites at roughly 100 to 400 kilometers break most of those correlations while keeping travel and latency manageable. This is where asynchronous replication becomes the default, where you can place the recovery site on a separate grid interconnection and in a different flood and wildfire zone, and where an engineer can still drive to the site within a few hours. Distant sites beyond 400 kilometers, often cross country or cross region, survive even the largest weather and grid events and satisfy the strictest regulatory expectations for geographic isolation, but they only work with asynchronous replication and they demand that your applications tolerate the replication lag and the higher latency that users see after a failover.

Regulators and insurers rarely dictate an exact kilometer number, they audit for independence. You should expect to show that your primary and recovery sites do not share a flood plain, a wildfire hazard zone, a seismic fault, a power grid interconnection, or a single network corridor. A common audit finding is diverse addresses that still rely on the same carrier hotel or the same river crossing for fiber, which means the sites are logically close even when the map says they are far apart. Documenting sites with their real world flood, grid, and carrier context is how you prove independence instead of asserting it.

Synchronous vs Asynchronous Replication

Synchronous replication confirms the write at the recovery site before it tells the application the write succeeded. Your recovery point objective is effectively zero because there is no acknowledged write that exists only at the primary. The price is latency. Light needs about one millisecond for every 200 kilometers of fiber, plus switching and storage acknowledgement, so a metro pair at 50 kilometers may add two to five milliseconds per write and remain workable for many databases, while the same pattern at 500 kilometers adds latency that your application and your users will feel on every transaction. Throughput also becomes coupled between sites, so a slowdown or congestion event at the recovery site slows your primary even when the primary is healthy.

Asynchronous replication confirms the write locally and ships the change shortly after, usually in order, often with batching and compression. Your primary stays fast and the distance between sites stops mattering for application latency, but your recovery point objective becomes the replication lag: the seconds or minutes of data still in flight that you will lose if the primary disappears without a graceful shutdown. For many workloads that window is seconds under normal conditions and longer during peak load or network degradation, which means your RPO is not a single number but a distribution you must measure. Asynchronous also tolerates bursty bandwidth instead of demanding sustained peak bandwidth at all times, which makes long distance links far more practical to operate.

The choice is not always binary. Semi synchronous and quorum based replication wait for at least one remote acknowledgement without waiting for a full remote commit, which gives you a bounded lag with less latency than full synchronous. Storage arrays and databases often offer those middle modes under different names, and they pair well with a metro synchronous pair plus a distant asynchronous replica, which is the three site pattern many regulated teams run: metro for zero loss failover, distant for regional survival.

How to Choose Your Failover Site

A failover site is a promise that a second place can run your workloads at the moment your first place cannot, and that promise breaks on details that never appear on a real estate brochure. Start with hazard separation. Pull the flood maps, wildfire risk maps, and seismic hazard maps for both addresses and confirm they sit in different zones, then add power. Two sites that sit on the same grid interconnection or that depend on the same transmission corridor still share fate during a major outage. Ask the utility or the colocation provider which interconnection and substation feeds each site, and treat a vague answer as a risk to resolve before you sign.

Then verify network diversity with the same rigor. Diverse last mile is not diverse if both paths enter the same carrier hotel, ride the same river crossing, or terminate on the same metro ring. Request the actual fiber route maps and the list of upstream carriers and peering points for each site, and compare them side by side. A circuit inventory that records provider, path, and hand off for every WAN and interconnect link makes that comparison a query instead of a scavenger hunt. The same inventory is how you avoid the common surprise where a recovery test works from the office but fails from production because the replication path uses a circuit nobody remembered to document.

Capacity and readiness decide whether the cutover completes. Confirm rack space, power, and cooling headroom for the load you will move, and confirm that the network devices at the recovery site have the port count and optics to carry both normal traffic and the surge that follows a failover. Walk through the promotion steps out loud with the team that will execute them: how DNS or anycast or BGP cutover happens, how storage or database promotion is fenced to avoid split brain, and how application tier dependencies are restarted in order. A site that is perfect on hazard and fiber maps still fails if promotion requires a manual step that only one person knows and that person is unreachable when the alert fires.

Putting Distance and Replication Together

The cleanest way to choose is to work backwards from the recovery objectives your business already agreed to. If your recovery point objective is zero for a transactional database, you are choosing synchronous or at least semi synchronous, which chooses a metro site inside that latency budget, and you accept that regional events need a second asynchronous replica farther away. If your recovery point objective is minutes and your priority is geographic isolation over zero loss, you are choosing asynchronous and you are free to place the site where hazard separation is strongest. Either way, measure what you promised. Track replication lag as a time series, not as a best case snapshot, and alert on percentiles that match your RPO.

You should also design for the network after failover. Users and downstream systems will reach the recovery site from wherever they are, so model the latency and bandwidth they will see once traffic shifts. A metro site keeps that shift small. A distant site may need content delivery, read replicas, or DNS steering that you have already tested, not that you will invent during the event. Test the full path twice a year at full load, including replication reestablishment when the primary returns, because the failback is where split brain and resync surprises live. Teams that only test failover in one direction learn the second half of the lesson during a real incident.

Keeping the Plan Documented and Current

A plan that lives in a presentation ages quickly. Sites change power providers, carriers reroute fiber, racks fill up, and runbooks acquire stale steps the moment someone forgets to update them. What keeps a recovery site real is a single inventory where sites, racks, power capacity, circuits, and promotion steps reference each other so that a change in one place surfaces in the others. When your site record notes the grid interconnection and the fiber route, your replication choice stays honest. When your rack management shows true headroom at the recovery site, your failover does not discover a full rack. When your runbook references the circuits and endpoints that replication actually uses, your test fails for a fixable reason before the incident instead of after it.

If you already document infrastructure in one place, tie your disaster recovery decisions to those records and review them on the same cadence as your replication lag tests. Confirm hazard separation after any site move, revalidate carrier diversity after any circuit change, and rehearse promotion after any database or storage upgrade. Your future team will not remember the conversation where you chose async over sync. They will remember whether the documentation let them run the cutover they inherited.

A well chosen recovery site is not the farthest you can place it and it is not the closest you can get away with. It is the location, at the distance your replication choice allows, that breaks the hazards your primary cares about, that you can still operate and reach, and that you have proven with a cutover test instead of a map. Get that combination right and your disaster recovery plan becomes something your team trusts, not something they hope they will never need to use.