Technology recovery: data centre, DR and near-site architectures
In this lesson you will:
- Compare DR patterns from backup-and-restore to active-active on RTO, RPO, cost and cyber resilience
- Explain three-site designs and the synchronous-asynchronous replication trade-off
- Design cloud resilience across zones, regions and providers
- Plan failover decisions, shared dependencies and fail-back
Technology recovery is where most continuity budgets go and where the regulatory requirements are most specific. The goal is not the most sophisticated architecture. It is an architecture that demonstrably meets the RTO and RPO from the BIA, survives the severe scenarios and can be operated by your team under pressure.
1. DR patterns
| Pattern | How it works | Fits |
|---|---|---|
| Backup and restore | Infrastructure is rebuilt and data restored from backups after the event | Tier 3–4 services |
| Pilot light | Core data is replicated continuously; minimal infrastructure runs; the rest is built on failover from code or images | Tier 2–3 |
| Warm standby | A scaled-down but working copy runs in the DR location with replicated data; scale up on failover | Tier 1–2 |
| Hot standby (active-passive) | A full-capacity copy is ready; failover in minutes | Tier 1, strict RTO |
| Active-active | Both sites serve traffic simultaneously; loss of one shifts load | Tier 1 with near-zero RTO; MIIs, payment switches |
Two cautions apply to every pattern:
- Replication copies corruption. Logical corruption, a bad deployment or ransomware encryption propagates to the standby within seconds. Every pattern needs point-in-time, isolated, immutable backups (Lesson 4.4).
- Faster patterns are harder to operate. Active-active needs careful data consistency design (conflict handling, split-brain prevention). An organisation that cannot run it well may recover slower than with a simpler design.
2. Three-site designs and replication
There is a physics trade-off:
- Synchronous replication confirms a transaction only after it is written at both sites, so data loss is zero. It adds latency for every write, so the sites must be close (typically tens of kilometres).
- Asynchronous replication confirms locally and ships changes afterwards. Distance is not limited, but the most recent transactions may be lost (RPO of seconds to minutes).
- A DR site must be far enough not to share hazards: the same flood plain, seismic zone, power grid or telecom route.
The three-site answer combines both: a near site with synchronous replication (zero data loss) and a far DR site with asynchronous replication (geographic separation). SEBI's 2024 framework for MIIs requires exactly this: a near site with near-zero RPO and a DR site staffed and able to operate independently. Large banks use similar designs for core banking and payment systems.
3. Cloud resilience
In the cloud, the same patterns apply at different layers:
- Multiple availability zones in one region protect against hardware, data-hall and power failures. Multi-zone synchronous replication is usually feasible because zones are close. They do not protect against a region-wide or control-plane failure.
- Multiple regions protect against regional events. India now has several hyperscaler regions, which lets regulated entities keep data in India while separating DR geographically.
- Multiple providers protect against provider-wide failure and support exit, at high cost and complexity. Most institutions use them selectively, for example multi-cloud backups or a second provider for a few critical services.
- Isolated copies in a separate account, with separate credentials and immutability, protect against ransomware and administrator compromise, which none of the other layers do.
Shared responsibility. The cloud provider is responsible for the resilience of the cloud: facilities, hardware and its services. You are responsible for resilience in the cloud: architecture, configuration, data protection, failover and testing. Provider SLAs compensate for downtime; they do not recover your service.
4. Shared and hidden dependencies
Recovery often fails not on the main application but on something everything depends on:
| Dependency | Failure mode | Mitigation |
|---|---|---|
| Identity provider (SSO, MFA) | Unavailable or compromised; no one can log in to fix anything | Break-glass accounts outside SSO; DR identity path; offline MFA tokens |
| DNS | Cannot redirect traffic to DR | Low TTLs for failover records; secondary DNS provider; tested runbook |
| Certificates and keys (KMS, HSM) | Keys unavailable in DR region; certificates bound to primary | Replicate keys appropriately; DR certificates issued in advance |
| Secrets management | Applications cannot start in DR | Replicated secrets store with access tested |
| Monitoring and logging | Blind during failover | Monitoring outside the failed environment |
| CI/CD and infrastructure-as-code | Cannot rebuild | Pipelines and artefact repositories available in DR |
| Third-party endpoints | Partners whitelist only primary IPs | Pre-register DR IPs with aggregators, banks and regulators' systems |
5. Failover decisions and fail-back
Technology can fail over in minutes, but the decision to fail over often takes hours. That was the lesson of the 2021 exchange halt. Define in advance:
- criteria for invoking DR, such as an outage expected to exceed X minutes or data corruption suspected;
- who decides, with named deputies, and the maximum time allowed to decide (SEBI: declare a disaster within 30 minutes for MIIs);
- what to check before failing over in a cyber event: is the DR environment clean? Is the attacker still present?
Fail-back to the primary site is a planned change, often harder than failover, because data written in DR must return safely. Plan and test it, and do not rush it.
Regulator lens
RBI and SEBI expect DR drills that actually run critical systems from DR, not just component tests, with recovery times measured and reported. SEBI requires MIIs to run live trading from the DR site at least twice a year. Inspectors compare drill times with RTOs and look for unresolved drill observations.
Case study — Tarang Ledger: target DR architecture
Design decisions:
- Primary: Mumbai region, three zones, with synchronous database replication across zones (unchanged).
- Warm standby in a second Indian region of the same provider. The database is replicated asynchronously with a target lag of ≤ 5 minutes. The application tier runs at 20% capacity and scales up on failover. Target RTO for Tier 1 is 2 hours.
- Zero RPO for payment confirmations without cross-region synchronous replication: every confirmation is also held by the aggregator that processed it. After failover, an automated job reconciles the last 15 minutes against aggregator APIs and restores any missing confirmations. This was the first time the team had met an RPO from a recoverable external source rather than replication.
- Dependencies: break-glass admin accounts outside SSO; DNS failover records at 60-second TTL; DR IP addresses pre-registered with PA-1, PA-2 and the SMS gateways; secrets replicated.
- Decision: the CTO or, in their absence, the head of cloud operations may invoke DR. Criteria: Tier 1 unavailable for 30 minutes with no fix in sight, or the cloud provider confirms a regional incident.
Key takeaways
- Patterns range from backup-and-restore to active-active; choose by RTO, RPO, cost and operability.
- Replication copies corruption, so every pattern needs isolated, immutable backups.
- Sync replication means zero loss but short distance; async gives distance with some loss. Three-site designs (SEBI MIIs) combine both.
- In the cloud, zones, regions and providers protect against different failures. Shared responsibility means you own resilience in the cloud.
- Plan for identity, DNS, keys, secrets, monitoring, pipelines and partner whitelists, and define failover decision criteria and fail-back.
Self-check
- Why does synchronous replication limit the distance between sites?
- Which cloud resilience layer protects against an administrator account compromise?
- Give two hidden dependencies that commonly break a DR failover.