Setting up rollback before you need it

In professional software engineering, deploying code to production is easy; safely recovering from a critical production incident in under 60 seconds is what separates elite engineering teams from amateurs.

The most common mistake we observe when auditing legacy codebases is a one-way deployment pipeline. Code is pushed to `main`, migrations run destructively against production SQL tables, and static assets are overwritten. If a critical bug slips past automated tests, the team is forced to scramble and write emergency forward-fix patches under immense stress.

The Golden Rule of Safe Migrations: Expand and Contract

Database migrations are the #1 cause of botched deployments that cannot be rolled back cleanly. If version `v2.0` of your application renames a database column from `full_name` to `first_name` and `last_name`, and you deploy both the SQL migration and code simultaneously, an instant outage occurs if you need to revert the frontend code back to `v1.9`.

To achieve zero-downtime rollbacks, every database change must follow the Expand and Contract pattern across three distinct releases:

Phase 1 (Expand): Add the new columns (`first_name`, `last_name`) as nullable fields. Existing code continues reading/writing `full_name` without disruption.

Phase 2 (Migrate & Backfill): Update the application code to write to BOTH old and new columns, and run a background worker to backfill historical rows.

Phase 3 (Contract): Only after `v2.0` has been stable in production for days, drop the deprecated `full_name` column in a cleanup release.

-- SAFE EXPAND MIGRATION (Can be rolled back instantly)
ALTER TABLE users ADD COLUMN first_name TEXT NULL;
ALTER TABLE users ADD COLUMN last_name TEXT NULL;

-- DANGEROUS DESTRUCTIVE MIGRATION (Breaks previous app versions!)
-- ALTER TABLE users RENAME COLUMN full_name TO first_name; -- DO NOT DO THIS

Immutable Artifacts & Instant Blue-Green Routing

For frontend and API layers running on modern infrastructure like Vercel, AWS ECS, or Docker containers, deployments must be immutable. Each commit builds a self-contained container image or static bundle hash.

When a deployment triggers higher error rates or latency spikes in Datadog/Sentry, rolling back shouldn't require running `git revert` and waiting 8 minutes for a full Docker build cycle. Instead, our CI/CD pipelines maintain the previous stable container image hot and ready. A rollback is simply a 2-second DNS or load balancer weight adjustment shifting 100% of traffic back to the immutable previous release artifact.