Continuity Strategies, DR Architecture & Data Resilience, 30 min read

Data resilience: backup, immutability, isolated recovery and clean rooms

In this lesson you will:

  • Apply the 3-2-1-1-0 rule to critical data
  • Design backups that survive ransomware and administrator compromise
  • Plan recovery from a cyber attack using an isolated recovery environment
  • Prove recoverability through restore testing and integrity checks

For financial institutions, data is the business: ledgers, balances, holdings, positions, policies, claims. Systems can be rebuilt from code and images, but lost or corrupted data cannot be recreated by redeploying. Modern attackers know this. Ransomware groups routinely look for backups and delete or encrypt them before triggering the main attack. Data resilience is the discipline of making sure a clean, recent, usable copy always survives.

1. The 3-2-1-1-0 rule

The 3-2-1-1-0 backup rule: three copies, two media, one off-site, one immutable or air-gapped, and zero errors on restore tests.
Figure 4.9 The 3-2-1-1-0 rule, extended for ransomware.
  • 3 copies of data, including production.
  • 2 different media or storage services, so one failure mode cannot destroy all copies.
  • 1 copy off-site or in another region.
  • 1 copy immutable (cannot be changed or deleted until a retention period ends) or air-gapped (logically or physically disconnected).
  • 0 errors when restore tests are run.

RBI's operational resilience guidance specifically expects institutions to protect data through resilient backups, including immutable or offline copies. NIST's data integrity practice guides (SP 1800-11, -25 and -26) describe how to identify, protect, detect and recover from attacks on data.

2. Designing backups that survive an attacker

An attacker with domain or cloud administrator rights can usually delete anything that administrator can reach. Defences:

Control Purpose
Separate backup account or domain Backup storage sits in a different cloud account or AD forest, with no trust from production
Separate credentials and MFA Production administrators cannot log in to the backup environment; backup admin uses hardware MFA
Immutability Object lock or WORM in compliance mode, so even the backup administrator cannot shorten retention
Delete protection with multi-person approval Changes to retention or deletion need two people
Network isolation Backup systems are reachable only through tightly controlled paths; vault copies may be air-gapped
Monitoring Alerts on unusual deletion, retention changes, failed jobs or sudden change in backup size (a sign of encryption)
Encryption with protected keys Keys stored so that losing production does not lose the ability to decrypt backups

Retention depth matters for cyber recovery. Attackers often sit in a network for weeks before triggering ransomware, so the last clean backup may be weeks old. Keep enough history, for example daily backups for 35 days and weekly for 3 months, to go back past the likely intrusion date. Balance this against data protection rules on retention.

3. Recovering from a cyber attack: the clean room

Clean-room recovery: from the isolated vault, choose a known-good recovery point, restore into an isolated environment, scan and validate, rebuild and harden, then promote to production.
Figure 4.10 Recovery through an isolated recovery environment (clean room).

Traditional DR restores into the same environment as quickly as possible. After a cyber attack, that may restore the attacker's foothold too. An isolated recovery environment (IRE), or clean room, avoids reinfection:

  1. Vault. Immutable copies held in a separate, isolated account.
  2. Choose the recovery point. Forensic findings identify when the attacker got in and which data or systems were affected. Choose the latest point known to be clean.
  3. Restore into the clean room. A separate network segment or account, with new credentials, no trust to production and controlled internet access.
  4. Scan and validate. Malware scanning, checks for indicators of compromise, application integrity and data integrity checks (reconciling balances, totals and control figures).
  5. Rebuild and harden. Rebuild systems from trusted, patched images, rotate all secrets and keys, and close the vulnerability exploited.
  6. Promote. Reconnect in stages under heightened monitoring.

This takes longer than failover, which is why the BIA's MBCO and workarounds matter. You may need a degraded service, or a partner's service, to run while the clean room does its work.

4. Data integrity, not just availability

Backups protect against loss; they do not detect corruption. Financial data needs integrity controls:

  • reconciliation of balances and positions with external sources (banks, clearing, depositories, aggregators);
  • control totals and checksums for critical datasets;
  • detection of unexpected mass changes;
  • the ability to identify which transactions occurred after the recovery point, so they can be re-entered or reconciled.

Lesson 4.3 showed Tarang meeting zero RPO by reconciling against aggregator records. This is the same principle: know an independent source of truth for your most critical data.

5. Proving recoverability

An untested backup is a hope. Testing should include:

Test Frequency (typical, Tier 1) Proves
Backup job monitoring Daily Jobs ran and completed
File or record-level restore Monthly Backups are readable
Full system restore to an isolated environment Quarterly RTO achievable from backup; procedures work
Clean-room recovery exercise Yearly Cyber recovery end to end, including forensics and validation
Immutability test Yearly Retention cannot be shortened or copies deleted, even by admins

Record actual restore times and compare them with RTOs. Regulators and auditors ask for this evidence.

Auditor lens

Auditors ask: "Show me that a production administrator cannot delete your backups", "When did you last restore your core database in full, and how long did it take?" and "How would you know which backup is clean after ransomware?" Backups in the same account with the same credentials, or restore tests that only recover a few files, are high-risk findings.

Case study — Tarang Ledger: building the vault

Tarang's gap: backups were in the same cloud account as production, reachable by the same administrator roles. In a ransomware scenario they could be deleted (S-01).

New design:

  • Backup vault account in the second region, under a separate organisational unit, with no production roles able to assume into it.
  • Object lock in compliance mode: daily database backups retained 35 days, weekly for 13 weeks, monthly for 12 months, consistent with customer contracts and DPDP retention guidance.
  • Vault admin uses hardware security keys; retention changes need approval from two of three named people (CTO, Meera, head of cloud operations).
  • Alerts to the security operations team on any vault policy change or backup size anomaly.
  • Clean room: an isolated account in the second region with its own network, a restore pipeline and a scanning toolkit. First full exercise planned for month 9.

First quarterly restore test: full production database restored into the clean room in 4 h 50 min (down from 6 h 20 min, thanks to parallel restore). Integrity validation against aggregator totals took 1 h 30 min more, which was not in the original estimate, so the runbook was updated.

Key takeaways

  • Apply 3-2-1-1-0: three copies, two media, one off-site, one immutable or air-gapped, zero restore errors. RBI expects immutable or offline copies.
  • Make backups survive an attacker: separate account and credentials, immutability, multi-person approval, isolation, monitoring, protected keys and enough retention depth.
  • Recover from cyber attacks through a clean room: choose a known-good point, restore in isolation, scan, validate integrity, rebuild, then promote.
  • Protect integrity with reconciliation against independent sources of truth.
  • Test restores in full and time them against RTO.

Self-check

  1. What does the second "1" in 3-2-1-1-0 add, and why?
  2. Why might the last clean backup after ransomware be weeks old?
  3. What steps happen in a clean room before systems are promoted to production?