Data migration
Data migration changes the schema or data stock, while software versions and accesses can exist in parallel. Extension, filling and later conversion arrange this transition.
- Step by stepSmall compatible changes instead of a big jump.
- Parallel versionsOld and new run at the same time.
- Mandatory too earlyA mandatory column before filling breaks versions.
Context
Why this is important
During a rollout, old and new versions run simultaneously. If the new version assumes a field that does not write the old one, gaps or failures occur. Small compatible steps avoid this.
This building block deals with data and schema transitions. Deployment coordinates versions; Recovery checks recovery of a data status.
At a glance · Migration in four phases
Invoice ID: Which app version runs in which phase
The shop adds an invoice identifier for orders. First, the column is optionally created, then filled by the application and only later mandatory read.
This is how you read the picture
- Step by step You go through four phases; in the phases 1 up to 3 Both versions run without errors.
- Parallel versions The lines old and new app version show who is running in which phase.
- Mandatory too early In the “duty too early” phase, all three lines are red.
The invoice identifier column shall be created as an option.
- Old app versionrunning – ignores the new column
- New App Versionrunning – can write the identifier
- Rollback to old versionrunning – possible, scheme remains compatible
Benefits & limitations
Helps if:
- Large datasets benefit from controllable partial transfers with visible progress.
- Compatible schema changes keep old and new software versions operational during continuous delivery.
Boundaries
- Dual-write paths diverge; reliable propagation and reconciliation are needed before switchover.
- Destructive steps such as dropping a column close off rollback; verify all consumers and recovery options beforehand.
- Large tables make backfills take longer; with a shared table (Shared Database), the complete set of readers may also be unknown.
In practice
- Design a compatible extension. Add new structures so active versions can keep working. Define how values are initially written and read. This creates a schema that supports the planned parallel operation.
- Control the backfill. Bound batch size and load, store progress and define resumption after interruption. Handle concurrent changes transparently. This keeps progress and production impact visible.
- Verify completeness and plan the contract phase. Reconcile data against business rules, check readers and monitor access to the old form. Then decide when to remove the old path. This provides a verifiable transition with a clear cleanup boundary.
Read the detailed explanation
A data migration changes stored values, their structure or their location. During continuous operation, old and new software versions may run in parallel. Expand-Migrate-Contract first adds the new structure, transfers existing data in a controlled way, switches consumers, then removes the old form. It relates to SQL & NoSQL and compatible Deployment & Rollout.
Backfills need bounded batches, progress markers, restartability and rules for concurrent changes. Counts, samples and business constraints verify the transfer. Dual writes can diverge and require atomic coupling or reliable propagation. The contract phase follows evidence that active readers no longer use the old form.
Check your understanding
Test yourself in three stages
ReadingCheck in the picture
What is red in the “duty too early” phase?
View the solution with justification
All three lines: The old version cannot save, the new one reads empty identifiers, a rollback is impossible.
ApplyingTransferred to a new situation
An old worker still generates orders without an invoice identifier. Can you create the column as a duty?
View the solution with justification
No. First, create it as an option, update all the writing paths and fill the stick. It will only become binding in step 4.
EvaluateWeighing an alternative
The team is considering doing column, filling, and duty in a single deployment. How do you rate that?
View the solution with justification
A deployment is faster and has fewer intermediates. In parallel versions, however, it risks errors and leaves no safe way back, while the individual steps remain compatible.
Explore further
These building blocks continue the thoughts of this page.
References
- Martin Fowler – Evolutionary Database Design (opens in new tab)Describes versioned database changes and schema and application collaboration.
- Martin Fowler – Parallel Change (opens in new tab)Explains expand-and-contract for compatible transitions during a migration.
- Prisma – Data migration using expand and contract (opens in new tab)Shows schema extension, data transfer and subsequent removal of old structures.