An event schema looks like a serialization detail until a real system has to change it. Then the schema becomes a public API shared by producers, consumers, dashboards, replay jobs, data warehouses, and incident tools that may not be owned by the same team.
The difficult part of schema evolution is not adding a field to a JSON object. It is coordinating old consumers, new producers, replayed events, ownership, and rollback while messages continue to move through the system.
The principle I rely on is simple: compatibility is a release discipline, not a registry setting. A registry can reject an obviously incompatible schema, but it cannot tell you whether every consumer handles the meaning of a new field or whether a rollback will produce valid business behavior.
An event is a public API with a long memory
A synchronous API call is usually tied to the code that makes it. An event is different. It can be stored for days, replayed months later, copied into another system, or consumed by a service that no one remembered during the release.
That means an event contract has at least two audiences. Current consumers need to understand the next message. Historical consumers and replay jobs need to understand messages produced in the past. A change that looks harmless in the live path can fail when a backfill reads old data through new code.
The contract should include more than field names and types. It should state the event purpose, ownership, semantic meaning, identity fields, ordering assumptions, retention, privacy classification, and whether a consumer is allowed to ignore unknown fields.
{
"event_type": "order.shipped",
"event_version": 2,
"event_id": "evt_9d1a",
"occurred_at": "2026-08-14T08:10:00Z",
"producer": "fulfillment-service",
"data": {
"order_id": "ord_4821",
"carrier": "atlas",
"tracking_number": "AT-8821"
}
}The envelope gives consumers stable metadata while the data section can evolve under an explicit compatibility policy. Versioning is not a substitute for compatibility, but it makes the contract and migration path easier to discuss.
Backward, forward, and full compatibility
Compatibility answers a specific question: can one version of the application safely read data produced by another version?
| Mode | Practical question | Typical use |
|---|---|---|
| Backward | Can the new consumer read old events? | Deploy consumers before producers |
| Forward | Can the old consumer read new events? | Deploy producers before consumers |
| Full | Can old and new consumers read old and new events? | Safer rolling migrations |
| Transitive variants | Does the rule hold across multiple historical versions? | Long-lived topics and replay |
The names are useful, but teams often misuse them. A schema may be structurally compatible while its meaning has changed. A field can remain a string while changing from “local time” to “UTC time.” A default can make deserialization succeed while causing a consumer to take the wrong business action.
Schema checks should therefore run alongside semantic tests. The registry protects the shape; consumer tests protect the behavior.
Safe changes are still changes
Adding an optional field is usually easier than removing or renaming one. But “usually” is not a guarantee. Some consumers reject unknown fields. Some deserialize into strict classes. Some downstream jobs assume a fixed column count. A field that appears optional in the schema may be required by an undocumented dashboard.
Renaming is particularly dangerous because it is both a structural and semantic change. A consumer may interpret the missing old field as a real absence rather than a rename. Removing a field can break a replay path months after the original release.
For a simple additive change, a tolerant sequence often works:
- Make consumers able to ignore or safely default the new field.
- Deploy the consumer change and verify it against old events.
- Register the new compatible schema.
- Deploy the producer that begins populating the field.
- Measure consumer behavior and confirm the new meaning.
- Remove compatibility code only after the retention and replay window passes.
The sequence matters more than the exact tooling. The system should remain understandable during the period when old and new messages coexist.
Treat migration as a contract between teams
An event owner should be able to answer who is allowed to change the schema, which consumers exist, what compatibility mode applies, and how a rollback works. If the answer is “the registry will tell us,” the contract is incomplete.
A lightweight change proposal can include:
| Question | Decision to record |
|---|---|
| What changed? | Field addition, rename, removal, type or semantic change |
| Who consumes it? | Services, jobs, dashboards, external integrations |
| Which versions coexist? | Producer and consumer rollout order |
| What is the compatibility rule? | Backward, forward, full, or custom policy |
| How is old data handled? | Replay, backfill, quarantine, or ignore |
| What proves success? | Consumer tests, metrics, state checks |
| How do we roll back? | Code rollback, producer stop, bridge, or data repair |
This document is not bureaucratic overhead. It is the shared memory that prevents a schema decision from living only in a pull request.
Compatibility CI should test real consumers
A registry check is a valuable first gate. It should reject changes that violate the selected structural policy before they reach the topic. But production safety requires a second gate: run representative old events through current consumers and new events through old consumers when the rollout requires it.
The test corpus should include ordinary events, boundary values, missing optional fields, unknown fields, old versions, malformed records, and events produced during a partial deployment. For high-impact workflows, the assertion should inspect the resulting state transition, not only whether deserialization succeeded.
This catches failures such as a new enum value that parses but falls into a dangerous default branch, or a timestamp that passes type validation but shifts a billing date.
Rollback is not simply “deploy the old producer”
A code rollback and a data rollback are different things. If a new producer has already emitted events with a new field or meaning, stopping the producer does not erase those events. An old consumer may still read them, or it may fail when the replay window reaches them.
A rollback plan should answer three questions:
- Can the old consumer safely read events already written by the new producer?
- Can the producer return to the old schema without changing the meaning of records already emitted?
- If a bad event caused state changes, how will the state be repaired?
Sometimes the safest move is not an immediate schema rollback. It may be to stop new writes, quarantine a consumer, deploy a bridge that normalizes versions, or replay events into a new topic after repairing the data.
The important distinction is between reversing code and reversing facts. Facts already published to an event log need an explicit correction strategy.
Versioning should explain meaning, not excuse breaking changes
An event_version field is useful when consumers need to choose different parsing or business rules. It is not useful when every incompatible change receives a new number and the team stops thinking about migration.
Prefer one of two clear approaches. Either keep a stable event type with a compatible evolution policy, or create a new event type when the business meaning has genuinely changed. Avoid making consumers guess whether order.updated.v2 is a small additive change or an entirely different fact.
If a field changes meaning, give it a new name. A new name makes the semantic break visible to code review, dashboards, and operators. A version number alone can hide a dangerous assumption behind a familiar field.
Data contracts need ownership and lifecycle
A data contract is alive for as long as the event can be consumed. It needs an owner, a change process, a compatibility policy, and a retirement plan.
The owner does not need to approve every consumer implementation. The owner does need to publish the intended meaning and announce changes that affect downstream behavior. Consumers should declare whether they are strict or tolerant, whether they support historical replay, and which fields they actually depend on.
This creates an honest conversation about coupling. A consumer that silently depends on undocumented field behavior is already coupled; the contract makes that coupling visible before the producer changes it.
A production checklist
Before merging an event schema change, confirm that the business meaning is written down, all known consumers are inventoried, structural compatibility has been checked, representative old and new events have been tested, and the rollout order is explicit. Confirm what happens during partial deployment, how long compatibility code must remain, and how a bad event can be quarantined or repaired.
The final question should be uncomfortable but concrete: if we publish one million events with the wrong meaning, what exactly will we do next? If the answer includes a tested replay or repair path, the system is ready for change. If it only says “we can roll back,” the data contract still needs work.
Event-driven architecture becomes powerful when teams can evolve it without fear. That confidence does not come from never changing schemas. It comes from making compatibility, ownership, rollout, and rollback explicit enough that a change remains reversible in practice—not just in a deployment dashboard.
Comments (0)