Behind the Scenes

How Fast Can a Broken Website Update Be Rolled Back? Our Deployment Rollback Process: Keep 5 Releases, Switch Back Within 90 Seconds, and Make Database Migrations Add-Only

2026.10.18 · 29 views
How Fast Can a Broken Website Update Be Rolled Back? Our Deployment Rollback Process: Keep 5 Releases, Switch Back Within 90 Seconds, and Make Database Migrations Add-Only
“

The worst part of discovering a broken checkout page after launch is that undoing it can take half a day. ScriptWalker shares its internal rollback process: release directories switched by symlink, automatic health checks after each deploy, and two-phase expand-then-contract database changes, along with an honest look at the costs and where it doesn't fit. ”

Share:

Short answer: Our target is rolling back within 90 seconds. Every deploy goes into a new release directory, the server keeps the last 5 releases, and a rollback just points the "current" link back. Database changes are add-only, so the previous code still reads data correctly after a rollback.

A few years ago we took over an e-commerce site whose previous team updated it by overwriting files on the live server. One change to shipping calculation turned the checkout page blank, and the engineer had to rebuild the old files from memory. It took over 4 hours, and about 30 orders never went through. The problem wasn't the bad change; it was that there was no undo button.

What we do

Bottom line: we split "deploying" from "switching," and the switch can always be reversed.

The flow: push to main → GitHub Actions runs tests → deploy to a new release directory → run add-only migrations → switch the current link → automatic health checks → on failure, switch back to the previous release and alert the on-call engineer.

SituationOld way (overwrite files)Our way
Code bugManually recover old files, 1–4 hoursSwitch the link back, about 90 seconds
Renamed database columnOld code can't read it; no way backOld and new columns coexist; safe to roll back
Deploy fails midwaySite is half old, half newFails before the switch; users notice nothing

Why we do it

Bottom line: most small-business sites have no standby environment, so fast rollback is more realistic than zero errors.

We considered blue-green deployment, two full environments taking turns, but it needs a second server, which doesn't make sense for clients on tight monthly budgets. Release directories with a link switch need only a little extra disk on the same server and get close to the same result. The database is the hardest part to undo, so we accept the hassle of shipping column changes in two steps in exchange for being able to roll back any release. Automatic health checks mean a rollback doesn't wait for someone to notice.

How we run it: tools and files

Bottom line: 5 steps, all in the deploy script, none relying on an engineer's memory.

  • Step 1, test: GitHub Actions runs PHPUnit and static analysis; a failure stops the deploy.
  • Step 2, release directory: Deployer places the code in a new timestamped directory, with the shared .env and uploads kept in shared.
  • Step 3, migrations: Run with Laravel migrations, adding columns and tables only; dropping columns waits two releases.
  • Step 4, switch and check: After switching the current link, the script hits 3 health-check URLs (home, login, checkout) within 60 seconds.
  • Step 5, roll back and alert: If any check fails, it switches back, Sentry logs the error, and the team's LINE group is notified.

Server directory layout:

/var/www/app
├── current -> releases/20261018-1030
├── releases/
│   ├── 20261018-1030
│   ├── 20261015-1610
│   └── ... (last 5 kept)
└── shared/
    ├── .env
    └── storage/

What it costs us

Bottom line: fast rollback costs more steps and slower column changes.

  • Renames take two deploys: Add the new column and write to both, then drop the old one after confirmation, about 1–2 weeks end to end.
  • Disk space: Keeping 5 releases uses about 1–3 GB more.
  • Initial setup: The first rollout takes 1–2 working days.
  • Data isn't rolled back: Code goes back; orders and member data already written stay.

Where it doesn't fit

Bottom line: not every project needs this process.

  • Purely static brochure sites: hosts like Cloudflare Pages already offer version rollback.
  • Shared hosting that can't run commands or create symlinks.

Why clients should care

Bottom line: rollback speed decides how many orders you lose when something breaks.

For a site taking 8 orders an hour, cutting rollback from 4 hours to 90 seconds saves about 30 lost orders. More importantly, the team can ship small changes often, and smaller changes mean smaller breakages. You can also ask any vendor to state in the contract how quickly they can roll back.

If you want to do this too

Bottom line: start with release directories; full automation can come later.

  • Move shared config and uploads into a shared directory first; that's where most teams get stuck.
  • Health-check only the 3 most critical URLs at the start.
  • Agree as a team that column drops always lag two releases, and put it in the PR template.
  • Rehearse a rollback every quarter to make sure the script really works.

Where we've used it

Bottom line: two anonymized cases, both rolled back before the client noticed.

  • Booking site: A package upgrade broke login; the health check caught it within 40 seconds and switched back automatically, and the fix shipped that afternoon.
  • B2B ordering system: A price column was split into two over two deploys, with one rollback along the way, and distributors never stopped ordering.

FAQ

Can a broken website update be rolled back?

Yes, if old releases are kept at deploy time. With release directories and a link switch, rollback takes about 90 seconds.

Will rolling back delete new orders?

No. Rollback reverts code; orders and member data already in the database stay.

What is an add-only migration?

Database changes add new columns without dropping old ones right away, so old and new code both work; old columns go once things are stable.

Does this work on shared hosting?

Usually not. It needs a VPS or cloud server that can run commands and create symlinks.

Next step

ScriptWalker (a Taiwanese Laravel/Flutter custom development studio) includes this deploy-and-rollback process in its maintenance plans, from NT$2,000 a month. If you want one-click rollback for your site, or want to join a team that writes its processes down, get in touch:

Share:
Behind the Scenes Back to Blog