Schema Evolution and Backward/Forward Compatibility

How to change the shape of your data while old and new code run side by side: backward and forward compatibility, the rules for Protobuf, Avro and SQL tables, and the order to roll changes out.

Overview

Applications change, and so does the shape of their data: a new field, a field no one needs, a wider number type. In a real system you cannot change every program and every stored record at the same moment. Servers are upgraded a few at a time, mobile apps stay old for months, and data written years ago is still on disk. Schema evolution is the practice of changing a data format so that every version still running can keep working.

Why old and new code meet

During a rolling upgrade, some servers run the new version while others still run the old one. Both read and write the same database and the same message topics. So new code reads data written by old code, and old code reads data written by new code.

The data also outlives the code. A row written five years ago is still read by today's code unless someone rewrote it. So compatibility is not only about deploys; it is about everything ever stored.

Backward and forward compatibility

Who reads what — new code reads old data — old code reads new data
Why you need it — old records and old messages still exist — old servers and old clients are still running
How it is usually kept — new fields have defaults — old code ignores fields it does not know
Difficulty — easier: you know the old format — harder: old code must tolerate what it has never seen

Full compatibility means both at once. It is what you want when producers and consumers upgrade in any order.

Rules per format

The safe changes follow from how each format finds a field: Protobuf and Thrift by field number, Avro by field name, JSON by key.

Add a field — safe; old code skips the unknown number — safe if the new field has a default — safe if old code ignores unknown keys
Remove a field — safe; reserve its number — safe if the removed field had a default — safe if readers do not require it
Rename a field — safe on the binary wire; breaks JSON and text encodings — use an alias in the reader's schema — breaking
Change a type — only between compatible types (for example int32, int64, uint32, uint64, bool); values may be cut — only allowed promotions (int to long, float or double) — depends on every reader
Reuse an old field number — never: old data is misread — not applicable — not applicable

Protobuf: numbers are forever

In Protobuf the field number is the field's identity on the wire. When you delete a field, reserve its number (and its name) so no one can reuse them later. Reusing a number means old data written with the old meaning is read with the new meaning, which can cause parse errors, leaked data or silent corruption.

Proto2 had required fields, and its guide warns that required is forever: changing a required field to optional is nearly impossible to do safely, because old readers reject messages without it. Proto3 (and the newer editions) dropped required. Put such checks in application code instead.

Avro: defaults do the work

In Avro, compatibility is checked between the writer's schema and the reader's schema, matching fields by name. A field the reader expects but the writer did not write is filled from the reader's default; without a default, decoding fails. Here the new version adds a nullable country field with a default, so it can read data written by the old version.

Old readers of new data simply skip the country field, because it is not in their schema. So this change is compatible in both directions. To rename a field, give the reader's field an alias listing the old name.

Enforce it with a schema registry

People forget the rules, so a registry checks them. Confluent Schema Registry compares each new schema with earlier ones and rejects a change that breaks the subject's compatibility setting. The setting also tells you which side to upgrade first.

BACKWARD (the default) — consumers on the new schema read data from the previous one — the last version — consumers
FORWARD — consumers on the previous schema read data from the new one — the last version — producers
FULL — both — the last version — either, in any order
..._TRANSITIVE — the same guarantee — all earlier versions — as above
NONE — no check — nothing — coordinate by hand

With plain BACKWARD, version 3 is only checked against version 2. If a consumer might replay a topic from the start, it may meet version 1 data too; that is what BACKWARD_TRANSITIVE is for.

Databases: expand, then contract

A database table has a schema too, and the same idea applies: never make a change that the currently running code cannot live with. Split a breaking change into safe steps. First expand (add the new thing so old and new code both work), then migrate the data and the code, then contract (remove the old thing).

In PostgreSQL, adding a column with no default, or with a constant default, only changes the table's metadata; the table is not rewritten. Setting NOT NULL normally scans the whole table under an ACCESS EXCLUSIVE lock, but the scan is skipped if a valid CHECK constraint already proves there are no NULLs. Validating that constraint takes a weaker lock that does not block writes.

1 — ana@example.com — US
2 — bo@example.com — US

What you will be able to do

Define backward and forward compatibility and say which one a given deploy order needs.
List which changes are safe in Protobuf, Avro and JSON, and why each rule follows from how the format finds fields.
Remove or rename a field without breaking running readers.
Choose a schema registry compatibility setting and the matching upgrade order.
Plan a zero-downtime database column change with expand and contract.

Key concepts

Backward compatibility: New code can read data written by old code. Needed because old records and old messages still exist.
Forward compatibility: Old code can read data written by new code. Needed because old servers and clients are still running.
Full compatibility: Both backward and forward, so producers and consumers can upgrade in any order.
Reserved field: A Protobuf declaration that retires a field number or name so it can never be reused with a new meaning.
Default value: The value an Avro reader uses for a field the writer did not write. Without one, adding a field breaks reading of old data.
Alias: An old name listed on an Avro reader field, so data written under the old name still matches.
Compatibility mode: A schema registry setting (BACKWARD, FORWARD, FULL, their transitive forms, NONE) that decides which schema changes are accepted.
Expand and contract: Making a breaking change in safe steps: add the new form, migrate data and code, then remove the old form.

When to use it

Designing an event stream where producers and consumers are deployed by different teams at different times.
Planning a rolling deploy that changes a message, API payload or table column.
Explaining how to remove, rename or retype a Protobuf or Avro field in production.
Choosing a schema registry compatibility mode for a Kafka topic.
Planning a database migration on a large table without downtime.

Interview questions

You need to rename a Protobuf field. How do you do it safely?
Producers and consumers of a Kafka topic are deployed by different teams. How do you keep schema changes from breaking them?
What goes wrong when old code reads and rewrites a record written by new code?
How do you add a NOT NULL column to a large PostgreSQL table without downtime?
Why is forward compatibility harder than backward compatibility?