DR drills and live switchover: measuring RTO and RPO
In this lesson you will:
- Plan a DR drill that produces credible evidence of recovery capability
- Run a live switchover safely with change control, rollback and communication
- Measure RTO and RPO at step level and compare with objectives
- Recognise and avoid drills that look successful but prove little
DR drills are the most scrutinised exercises in BFSI. Regulators specify their frequency, customers ask for the reports and boards want assurance that critical systems can be recovered. A good drill proves the capability honestly. A staged drill that avoids the hard parts gives false comfort that is worse than none.
1. Planning a credible drill
A drill plan states:
- Scope: which services and components, and which scenario is assumed (for example, loss of the primary region).
- Objectives: recovery within the RTO from the decision; data loss within the RPO; business validation by users; fail-back within a set window.
- Type: component test, full failover with business running from DR, or a switchover with fail-back the same night.
- Prerequisites: change approval, backups verified, monitoring in DR, partners informed (aggregators, banks, exchanges, NPCI where relevant).
- Roles: drill lead, technical teams, business validators, observers (internal audit, customers, regulator if invited), safety officer.
- Go / no-go criteria before starting and stop / rollback criteria during.
- Timing: for live drills, a lower-risk window, but for MIIs and some services a live trading day is required.
- Communication: to staff, customers (if any impact is possible) and supervisors.
2. Running a live switchover safely
Live switchovers carry real risk, since the drill itself can cause an outage. Controls:
- treat the drill as a planned change with CAB approval;
- verify backups and replication health before starting;
- brief all participants with the runbook and roles;
- use the real runbook, executed by the people who would do it in a real event, including alternates;
- record timestamps for every step;
- keep a rollback path ready and a safety officer empowered to use it;
- validate with real users and transactions: business users confirm they can do their work; for payments, test transactions run end to end;
- for SEBI MIIs, live trading or operations from the DR site at least twice a year, and near-site switchover tests, as specified.
3. Measuring RTO and RPO properly
What to measure:
| Measure | How |
|---|---|
| Time to decision | From simulated incident start to logged DR declaration |
| Technical recovery time | From declaration to systems available in DR |
| Business validation time | From systems available to business confirming service |
| Total recovery time | From incident start (or declaration, stating which) to service restored at MBCO; compare with RTO and tolerance |
| Data loss (RPO achieved) | Compare the last transaction in primary with the first in DR; reconcile with independent sources |
| Fail-back time and data integrity | Measured the same way |
Report both "from declaration" and "from incident start". Regulators and customers care about the latter, because it is what customers experience. Measure each step, since overruns in individual steps are where improvements come from.
4. Drills that prove little
Watch for these signs of a drill that looks successful but is not credible:
| Warning sign | Why it matters |
|---|---|
| Only infrastructure brought up; no business validation | Applications may not work end to end |
| DR environment "pre-warmed" specially for the drill | Real failover would not have that advantage |
| Run by the one expert who built it, not the runbook | Proves one person's knowledge, not organisational capability |
| Clock starts at "begin failover", not at incident | Hides decision delays, often the longest step |
| Interfaces to partners not tested | Aggregators, banks and exchanges may reject DR IPs or certificates |
| Same scenario every time | Other failure modes untested |
| No fail-back | Fail-back failures are common and costly |
| Findings closed by "noted" | Nothing improves |
5. Evidence
Keep a drill pack: plan, approvals, participant list, timestamped log, screenshots or system evidence, business validation sign-offs, data reconciliation, issues found, report and actions. Regulators, auditors and customers will ask for it. It also makes the next drill easier to plan.
Regulator lens
Inspectors compare drill reports with RTOs and with previous drills: are times improving? Are the same issues recurring? They check that critical systems were actually run from DR, for the required duration, with business transactions, and that observations were closed. For MIIs they verify live trading from DR at the required frequency.
Case study — Tarang Ledger: first regional failover drill (month 9)
Scope: collections service (Tier 1); scenario "Mumbai region unavailable"; live failover at 22:00 on a non-peak day with fail-back at 02:00. Sahyadri observed, and validated payments using test borrower accounts.
Results (Figure 7.6):
| Measure | Target | Actual |
|---|---|---|
| Decision time | 30 min | 25 min |
| Declaration → service at MBCO | ~1 h 32 min | 1 h 39 min |
| Total from incident start | 2 h (RTO 3 h) | 2 h 04 min |
| RPO for confirmations | 0 | 0 (aggregator reconciliation found and restored 3 confirmations) |
| RPO other data | 5 min | 2 min replication lag |
| Fail-back | 1 h | 1 h 40 min (one rollback and retry) |
Findings: smoke tests took longer because test accounts were not ready; reconciliation was slower because the script ran single-threaded; fail-back needed a retry due to replication direction misconfiguration. All three went to the corrective action log; the month-12 drill re-tested them.
Tarang shares the drill report with all NBFC customers as contract evidence. Sahyadri files it as evidence for its outsourcing oversight.
Key takeaways
- Plan drills with scope, scenario, objectives, prerequisites, roles, go/no-go, stop/rollback, timing and communication.
- Run live switchovers as planned changes, with the real runbook, real operators (including alternates), timestamps, business validation and rollback.
- Measure decision, technical, validation and total times, RPO achieved and fail-back; report from incident start as well as declaration.
- Avoid drills that prove little: no business validation, pre-warming, one expert, untested interfaces, no fail-back, findings "noted".
- Keep a full drill pack as evidence.
Self-check
- Why should recovery time be reported from incident start as well as from DR declaration?
- Name three signs that a DR drill may not be credible.
- How can RPO achieved be verified after a failover?