One Friday Night, 27 Days of Orders Disappeared
An anonymised case: a pet supplies retailer had an engineer altering a database schema on a Friday evening. One mistyped command emptied the order detail table. Looking for a backup, the team discovered the host's snapshot schedule was monthly and the most recent copy was 27 days old. After the restore, 27 days of orders, payment reconciliation and shipping records were gone. Admin staff rebuilt them by hand from bank deposits and courier manifests over three weeks, and ate the cost of 41 duplicate shipments. The real loss was not the data. It was that nobody in the company knew how old their backups were.
Who Should Do This Now, and Who Can Wait
Do it now
- Your system holds orders, payment reconciliation or member personal data that changes daily.
- A day of downtime costs revenue directly: e-commerce, bookings, ordering, B2B quoting.
- You are subject to Taiwan's Personal Data Protection Act and must adopt appropriate security measures under Article 27.
- Your data sits across three or more locations: server, object storage, third-party APIs.
- You have already had an accidental deletion, or received a ransomware notice.
Wait for now
- A static brochure site with no database. Code in Git and assets in version control is enough.
- Everything lives in a SaaS platform and you already export CSVs to your own storage on a schedule.
- Internal tools where three days of downtime costs nothing. A weekly manual export is fine.
- Nobody owns the monthly job of checking backup status. Assign a person before you buy architecture.
- You cannot list which machines hold your data. Do the inventory first.
Four Routes and What They Cost
| Route | Typical RPO | Typical RTO | Annual cost | Limits |
|---|---|---|---|---|
| Host built-in snapshots | 24 hours | 1–4 hours | ~20% of hosting fee | Same account and region as production; account suspension or full-instance deletion takes the backup too |
| Scheduled backup to third-party object storage | 1–24 hours | 2–8 hours | NT$3,000–30,000 | You build the scheduling and alerting; restore is manual |
| Object storage plus binary-log point-in-time recovery | 1–5 minutes | 2–6 hours | NT$20,000–80,000 | More complex database configuration; log integrity needs periodic verification |
| Cross-region hot standby with failover | Near zero | 5–30 minutes | From NT$150,000 | Roughly double the infrastructure cost, and accidental deletions replicate to the standby, so you still need real backups |
For most small and mid-sized companies the sweet spot is route two or three. Route four only pays when an hour of downtime costs six figures.
Four-Phase Rollout
- Phase 0: Inventory and RPO/RTO definition (3–5 days). Split data into three classes: database, user uploads, configuration and secrets. The owner answers two questions — how many hours of data can you afford to lose (RPO) and how long can you afford to be down (RTO). Deliverables: a data asset register and an RPO/RTO table in Notion or Google Sheets.
- Phase 1: Build the backup architecture (1–2 weeks). Implement 3-2-1: three copies, two media types, one offsite. Use restic for client-side encryption and deduplication, rclone to push to Cloudflare R2 or Backblaze B2; Laravel projects can use spatie/laravel-backup directly. Deliverables: backup scripts, key custody arrangement, retention policy.
- Phase 2: Monitoring and alerting (3–5 days). The dangerous failure mode is the silent one. Use Healthchecks.io as a dead man's switch: if a job does not check in on schedule, an alert fires to Slack. Deliverables: alert rules and a monthly backup status report.
- Phase 3: Restore drill and runbook (3–5 days). Actually restore onto a clean machine, with a stopwatch running. Deliverables: a measured RTO figure, a DR runbook (who calls whom, how DNS is switched, where the keys live) and a handover record covering at least two people.
Real Cost Breakdown
Setup cost, estimated against 2026 Taiwan market rates: basic tier NT$40,000–80,000, standard tier with point-in-time recovery NT$120,000–250,000, cross-region hot standby from NT$350,000. ScriptWalker's Backup and Disaster Recovery build package starts at NT$60,000, with maintenance from NT$3,000 per month.
The recurring costs people forget:
- Object storage: R2 standard is US$0.015 per GB per month with no egress charge; B2 is US$6 per TB per month with free downloads up to three times your stored volume.
- The cold storage trap: S3 Glacier Deep Archive stores at roughly US$1 per TB per month, but carries a 180-day minimum billing period and retrieval around US$0.02 per GB — pulling back 500GB in an emergency is a sudden unbudgeted expense.
- Egress: retrieving from AWS S3 costs bandwidth; R2 and B2 do not charge it. That gap is the real selection criterion.
- Extra disk charges for host snapshots, usually billed as a percentage of the instance fee.
- Quarterly restore drills: 4–8 hours each, roughly NT$20,000–40,000 a year in labour.
- Temporary instances spun up during a restore, plus the annual fee for a team password manager holding your keys.
What Clients Imagine vs What Happens
- Imagined: the host has backups, so we are safe. Reality: snapshots sit in the same account and region as production. Suspend the account or delete the instance and the backup goes with it.
- Imagined: the job runs, so we have backups. Reality: a backup you have never restored is not a backup. The classics are 0-byte dump files and dumps taken without transaction consistency, which restore into data that does not reconcile.
- Imagined: restoring takes half an hour. Reality: pulling 100GB from cold storage, decrypting, importing and waiting out DNS TTL usually measures at 4–9 hours.
- Imagined: more copies means more safety. Reality: with no retention policy, storage cost doubles within a year, and ransomware encrypts every writable copy at once.
- Imagined: this is the engineer's job. Reality: RPO is an owner's decision. How many hours of orders you can afford to lose determines how much this costs.
Six Traps and How to Avoid Them
- Backups in the same account as production. Fix: separate account, separate region, and enable immutability such as Object Lock.
- Backing up the database but not user uploads or environment config. Fix: the asset register mandates all three classes; no sign-off if one is missing.
- Uploading backups unencrypted. Fix: encrypt client-side before upload and store keys in a password manager, never alongside the backup.
- Discovering the breakage three months late. Fix: a dead man's switch that alerts on missing check-ins, not just on reported failures.
- Never running a drill. Fix: quarterly restore drills with the measured RTO written into the report; if the number fails, the budget moves.
- Keys and credentials known to one person. Fix: at least two holders, written into the offboarding checklist.
Success Metrics and the 90-Day Roadmap
- Day 30: asset register complete, 100% backup success rate, alerts landing in Slack, first restore drill done with a measured RTO on record.
- Day 60: point-in-time recovery to any minute, retention policy live (7 daily / 4 weekly / 12 monthly), offsite copy enabled, monthly storage spend inside budget.
- Day 90: full site failover drill completed, RTO inside target, runbook understood by at least two people, first annual cost and risk review done.
Decision Checklist
- ☐ I can say when the last successful backup ran
- ☐ I know the RPO we can live with (hours of data lost)
- ☐ I know the RTO we can live with (hours of downtime)
- ☐ Backups live in a different account from production
- ☐ Backups live in a different geographic region
- ☐ User-uploaded files are backed up too
- ☐ Environment config and secrets are backed up too
- ☐ Backups are encrypted before upload
- ☐ A failed backup actively notifies a human
- ☐ We ran at least one restore drill in the past six months
- ☐ There is a written runbook more than one person understands
- ☐ There is an explicit retention and deletion policy
- ☐ Someone reviews monthly storage and egress spend
Fewer than 8 ticks means your current backups probably will not save you.
FAQ
My host says it takes daily backups. Do I still need my own?
Yes. Host snapshots solve hardware failure. They do not solve account suspension, full-instance deletion, or ransomware encrypting every writable copy at once. The cheapest fix is a daily encrypted copy pushed to a different provider's object storage, which costs a few hundred NT dollars a month.
How do I actually set RPO and RTO?
Work backwards from money. Calculate what an hour of downtime costs in revenue and how many person-days it takes to rebuild a day of data, then pick the tier whose annual cost sits below your expected loss. A sensible starting point for most SMBs is RPO of one hour and RTO of four hours. Pushing to a five-minute RPO and a thirty-minute RTO moves you up a full cost bracket.
Is storing backups overseas a personal data problem?
Taiwan's Personal Data Protection Act does not ban cross-border transfer outright, though the competent authority may restrict it in specific circumstances under Article 21. The practical approach is to encrypt client-side before upload, keep the keys in Taiwan, and document storage location and encryption method in your personal data file security plan. For large volumes or sensitive categories, get compliance advice first.
How long does a restore drill take?
The first one usually runs 4–8 hours, because that is when you discover the missing files, missing keys and missing documentation. Later drills typically fit inside two hours. Run one per quarter, and deliberately have someone who does not normally own this task perform it — that is the only way to test whether the runbook is clear enough.
Next Step
If you cannot say what date your last successful backup ran, do not start with architecture. Start with a 60-minute backup health check: we map your data assets, current backup state and the RPO/RTO you can actually live with, and you walk away with a risk list and tiered pricing before deciding how far to go.
- Email: [email protected]
- Phone: 0916-224-047
- LINE: @ufv9089p