The rollback plan for most deployments is “redeploy the previous version.” For stateless code that works: the old container starts, serves requests, and the bug is gone. But code isn’t the only thing a release changes. If version 1.9 ran a database migration, wrote rows in a new shape or published events in a new format, redeploying 1.8 doesn’t undo any of that. You end up running old code against new data, a combination nobody tested.
Interviewers raise this when you describe a deployment pipeline with a confident “and if it breaks we roll back.” The follow-up is “what about the database?” A good answer explains why rollback and schema changes conflict, and how teams sequence changes so the previous release keeps working.
The rename that can’t be rolled back
Release 1.9 renames customers.full_name to display_name. The migration runs during deploy, and the new code reads the new column. Then 1.9 turns out to have an unrelated bug and you roll back to 1.8. Here’s that sequence in SQLite with Python:
import sqlite3
def v1_read(db):
# Version 1.8 of the app
return db.execute("SELECT id, full_name FROM customers").fetchall()
db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE customers (id INTEGER PRIMARY KEY, full_name TEXT)")
db.execute("INSERT INTO customers (full_name) VALUES ('Ada Lovelace')")
db.execute("ALTER TABLE customers RENAME COLUMN full_name TO display_name")
print("1.9 reads:", db.execute("SELECT id, display_name FROM customers").fetchall())
try:
print("rolled back to 1.8:", v1_read(db))
except sqlite3.OperationalError as e:
print("rolled back to 1.8: sqlite3.OperationalError:", e)1.9 reads: [(1, 'Ada Lovelace')]
rolled back to 1.8: sqlite3.OperationalError: no such column: full_nameThe rollback made things worse. Release 1.9 was partly broken; release 1.8 against the migrated schema is completely broken, since every query touching customer names fails.
The instinctive answer, “write a down migration,” has problems. Down migrations are rarely tested. They often can’t restore information: if 1.9 dropped a column, the data is gone. They take time on large tables, which is the worst moment to be holding locks. And during a rolling deploy, old and new versions run at the same time anyway, so even without a rollback, 1.8 pods are serving requests against the 1.9 schema for a few minutes.
Expand and contract
The pattern that solves this (also called parallel change) splits one breaking change into several steps, each compatible with the version before it:
- Expand. Add the new column. Change the code to write both columns and read the old one. Backfill existing rows.
- Migrate reads. Switch the code to read the new column, still writing both.
- Contract. Once no running or rollback-candidate version uses the old column, stop writing it, then drop it in a later release.
Here’s the expand step for the same table:
db.execute("ALTER TABLE customers ADD COLUMN display_name TEXT")
db.execute("UPDATE customers SET display_name = full_name WHERE display_name IS NULL") # backfill
db.execute("INSERT INTO customers (full_name, display_name) VALUES ('Grace Hopper', 'Grace Hopper')")1.9 reads: [(1, 'Ada Lovelace'), (2, 'Grace Hopper')]
rolled back to 1.8: [(1, 'Ada Lovelace'), (2, 'Grace Hopper')]Release 1.8 still works, because full_name is still there and still being written. You can roll back freely at every step, because each step is compatible with the one before. The cost is time and a little temporary duplication: a rename that looked like one migration becomes three or four releases over days or weeks. For a table with millions of rows, the backfill should run in batches, not as one UPDATE, to avoid long locks and replication lag.
The general rule is that a schema change and the code that depends on it must ship in separate releases. Schema changes go out first and must be backwards compatible: adding nullable columns, adding tables, adding indexes (built concurrently on Postgres). Removing or renaming happens last, only after the code stopped needing the old shape.
Data written by the new version
Schema isn’t the only problem. Release 1.9 may write values that 1.8 doesn’t understand, even into an unchanged schema. Suppose 1.9 adds partial refunds and writes a new order status:
LABELS_1_8 = {"paid": "Paid", "shipped": "Shipped", "refunded": "Refunded"}
orders = [{"id": 1, "status": "paid"}, {"id": 2, "status": "partially_refunded"}]
for o in orders:
try:
print(o["id"], LABELS_1_8[o["status"]])
except KeyError as e:
print(o["id"], "KeyError:", e)1 Paid
2 KeyError: 'partially_refunded'After the rollback, every page that lists that order crashes. The same thing happens with serialized formats: a new required field in a JSON blob, a new event type on a Kafka topic that old consumers reject, a cache entry in a new format.
The defences mirror expand and contract. Ship readers before writers: release N teaches the code to read and tolerate the new value, release N+1 starts writing it. Make readers tolerant of unknown values (fall back to a generic label, skip unknown event types and log them). Put new behaviour behind a feature flag so you can stop writing new values without a rollback. For event streams, a schema registry with compatibility rules can enforce that consumers can read what producers write.
Making rollback a tested path
A rollback plan is only real if it’s exercised:
- Check compatibility in CI. Run the previous release’s test suite, or at least its smoke tests, against the database after the new release’s migrations have run. If that fails, the release isn’t rollback-safe.
- Separate migration from deploy. Run migrations as their own pipeline step with their own approval, so you can see which releases change data and roll code back without touching the schema.
- Know when rollback isn’t the answer. Sometimes the safe move is forward: ship a fix, or turn off the feature with a flag. A release that ran an irreversible data transformation (re-encrypting values, merging records) needs a restore-from-backup plan instead, and someone should know before deploy that this release is one of those.
- Keep backups and know how long a restore takes. A point-in-time restore loses everything written since, so it’s a last resort, but you need to know it works.
Interview practice
Your team wants to change orders.amount from a floating-point column in euros to an integer column in cents, named amount_cents. The service runs as a rolling deployment of 20 pods, and the team requires that any release can be rolled back within an hour of going out. Plan the change.
Answer
A direct conversion breaks two ways. During the rolling deploy, old pods would read a column that’s gone or has changed meaning, and a rollback would leave the previous release reading data it doesn’t understand. So the change has to be split.
Release 1 (expand): add a nullable amount_cents column. Code writes both amount and amount_cents (computed as a rounded integer) on every insert and update, and still reads amount. A batched background job backfills amount_cents for existing rows. Rolling back to the previous release is safe, since it ignores the new column.
Release 2 (switch reads): after the backfill completes and a check confirms amount_cents = round(amount * 100) for every row, the code reads amount_cents and keeps writing both. Rolling back to release 1 is safe because it reads amount, which is still current.
Release 3 (stop writing the old column): the code no longer writes amount. Before this, make the column nullable or give it a default so inserts don’t fail. Rolling back to release 2 would read amount_cents, which is fine. The one thing that would break is rolling back two releases to release 1, which reads amount, so this step should only go out after release 2 has been stable well past the one-hour window.
Release 4 (contract), days later: drop amount. At every step, CI runs the previous release’s smoke tests against the migrated schema, so a non-compatible step can’t ship by accident.