A schema is a promise between whoever writes data and whoever reads it. In a single program you change both sides together and recompile. In a real system the writers and readers are different processes, deployed at different times, sometimes by different teams, and some readers are looking at data written months ago. Schema evolution is the discipline of changing that promise without breaking anyone who still relies on the old version.
Interviewers usually probe it through a concrete change: “you need to rename this column” or “this event field needs a different type”. The answer they want is a sequence of safe steps, not a single migration.
Why one migration isn’t enough
Consider a rolling deploy of an orders service running on 12 instances. Kubernetes replaces them a few at a time over, say, eight minutes. During those eight minutes some instances run version 41 and some run version 42, against the same database and the same topics. If anything goes wrong, you roll back to 41, which means 41 has to keep working against whatever schema 42 left behind.
Now suppose version 42 renames orders.amount to orders.amount_cents. If the migration runs before the deploy, every version-41 instance starts failing with column "amount" does not exist. If it runs after, every version-42 instance fails until it does. If you roll back after the migration, 41 is broken again. There’s no ordering that works, because the change is breaking in both directions at once.
The same problem exists for anything written to durable storage: Kafka topics retain old messages, caches hold serialized objects from the previous version, and a nightly export written last month may be replayed next year. You’re never dealing with exactly one version of the data.
Expand, migrate, contract
The standard answer is to split a breaking change into a series of changes that are each backwards and forwards compatible. Martin Fowler’s site calls this parallel change; it’s also known as expand-contract. For the amount to amount_cents rename, in Postgres syntax (illustrative, not tied to a particular ORM):
-- Release 1 (expand): add the new column. Old code ignores it.
ALTER TABLE orders ADD COLUMN amount_cents bigint;
-- Release 2: application writes both columns on every insert and update.
-- Then backfill old rows in batches to avoid one huge locking transaction:
UPDATE orders SET amount_cents = round(amount * 100)
WHERE id IN (SELECT id FROM orders WHERE amount_cents IS NULL LIMIT 10000);
-- repeat until no rows are updated
-- Release 3: application reads amount_cents, still writes both.
-- Release 4: application stops writing amount.
ALTER TABLE orders ALTER COLUMN amount DROP NOT NULL; -- if it had a NOT NULL constraint
-- Release 5 (contract): once no deployed version reads it and rollback is out of reach.
ALTER TABLE orders DROP COLUMN amount;Each release is safe to roll back to the one before it. After release 2, version 1 still finds amount populated. After release 3, rolling back to 2 is fine because both columns are still written. The only irreversible step is the final drop, and you take it only after every service, report and ad hoc script that reads amount has moved over.
A few details matter in practice:
- Adding a column with a default. In Postgres 11 and later,
ADD COLUMN ... DEFAULT 0with a constant default doesn’t rewrite the table. A volatile default likeDEFAULT gen_random_uuid()does, and on a large table that’s a long exclusive lock. MySQL behaviour depends on version and storage engine, so check before assuming it’s instant. - Backfill in batches. A single
UPDATEover 200 million rows holds locks, bloats the write-ahead log and can stall replicas. Batches of a few thousand rows with a short pause keep replication lag flat. - Dual writes in one transaction. Writing both columns in the same statement keeps them consistent. Writing them from different code paths invites drift.
- Verify before switching reads. A query like
SELECT count(*) FROM orders WHERE amount_cents IS DISTINCT FROM round(amount * 100)should return zero before release 3.
Yes, it’s five releases instead of one. That’s the cost of zero downtime, and interviewers want to hear you say so, and say when it’s not worth it (an internal admin table with one consumer and a maintenance window).
Messages and tolerant readers
Events have the same problem with a twist: the readers aren’t yours. A dozen services may consume OrderPlaced, and you can’t redeploy them all at once.
The first defence is the tolerant reader: consumers read the fields they need and ignore everything else. That makes adding fields safe. It doesn’t make renaming safe, and the way it fails is worth seeing. This Node 22 script feeds three versions of an event to three consumers: a strict v1 reader that rejects unknown fields, a tolerant v1 reader, and a v2 reader that prefers the new field and falls back to the old one.
const v1 = { orderId: 'o-1', amount: 49.99, currency: 'USD' };
const oneStep = { orderId: 'o-2', amountCents: 4999, currency: 'USD' }; // rename in one deploy
const expand = { orderId: 'o-3', amount: 49.99, amountCents: 4999, currency: 'USD' }; // write both
const strictV1 = (e) => {
const allowed = ['orderId', 'amount', 'currency'];
const extra = Object.keys(e).filter((k) => !allowed.includes(k));
if (extra.length) throw new Error(`unknown field(s): ${extra.join(', ')}`);
return Math.round(e.amount * 100);
};
const tolerantV1 = (e) => Math.round(e.amount * 100); // ignores unknown fields
const v2 = (e) => e.amountCents ?? Math.round(e.amount * 100); // prefers new, falls back to old
for (const [name, read] of Object.entries({ strictV1, tolerantV1, v2 })) {
for (const [label, evt] of Object.entries({ v1, oneStep, expand })) {
let out; try { out = read(evt); } catch (err) { out = 'ERROR ' + err.message; }
console.log(name.padEnd(10), label.padEnd(7), '->', out);
}
}strictV1 v1 -> 4999
strictV1 oneStep -> ERROR unknown field(s): amountCents
strictV1 expand -> ERROR unknown field(s): amountCents
tolerantV1 v1 -> 4999
tolerantV1 oneStep -> NaN
tolerantV1 expand -> 4999
v2 v1 -> 4999
v2 oneStep -> 4999
v2 expand -> 4999Three lessons sit in that table. The strict reader breaks on any addition, even the safe expand step, which is why strict “reject unknown fields” validation on consumers makes evolution nearly impossible. The tolerant reader survives the expand step, but a one-step rename gives it NaN, and it doesn’t throw. That’s the worst kind of failure: a revenue report quietly fills with NaN or zero, and nobody notices for a week. And the v2 reader, which handles both shapes, is what you deploy to consumers before producers stop sending the old field.
So for events the sequence mirrors the database one: producers add the new field while keeping the old one, consumers upgrade to read the new field with a fallback, and only when every consumer has moved do producers drop the old field. Usually that last step happens by publishing a new event version or topic, because old messages stay in the log for its retention period, and any consumer replaying from the start will meet them.
Letting tooling enforce the rules
People forget steps, so put the rules in CI. With Avro, Protobuf or JSON Schema plus a schema registry (Confluent Schema Registry is the common one for Kafka), each subject has a compatibility mode. BACKWARD means consumers on the new schema can read data written with the old one; FORWARD is the reverse; FULL is both. The registry rejects a schema that violates the mode, so a breaking change fails the build instead of a consumer.
Protobuf has its own rules worth knowing: never reuse a field number, mark removed fields as reserved, and remember that renaming a field is wire-compatible in the binary format (only the number goes on the wire, though the JSON mapping uses names) while changing its type usually isn’t. Avro needs a default on any added field for readers on the new schema to handle old data.
Practice question: splitting a name
A users table has a full_name column read by the web app, a mobile backend, and a nightly analytics export run by another team. Product wants separate given_name and family_name fields. Lay out the rollout.
Answer. Treat it as expand-contract with an extra wrinkle: splitting a name isn’t a lossless transformation. Names don’t reliably split on the first space (“Mary Ann Smith”, “Juan Carlos de la Cruz”, names with one part), so the backfill can only make a best guess.
Release 1 adds given_name and family_name as nullable columns. Release 2 makes the application write all three on every change: the new fields from the updated form, and full_name composed from them so the mobile backend and export keep working. Backfill existing rows in batches with a best-effort split, and flag them (name_source = 'derived') so the UI can ask those users to confirm their name. Release 3 switches the web app’s reads to the new columns. Then coordinate with the mobile team and the analytics team, since you can’t drop full_name until both have moved; give them a date, and check the database’s query statistics (for example pg_stat_statements) for any remaining reads of the column. Only then stop writing full_name and, after one more release so rollback stays possible, drop it. Arguably you keep full_name for good as a display name, since many cultures don’t fit a given/family split, and that’s a good point to raise with the interviewer.