Writing series · Complete · 5 parts
Replayable by Design
Practical notes on pipelines that can be retried, backfilled and recovered safely.
Complete · 5 parts
Backfills Are a Data Model Problem, Not an Airflow Feature
A scheduler can recreate yesterday's run. Only the data model can make that run safe to publish again.
Idempotent Writes Are Not Replay Safety
A table can converge perfectly while the workflow around it still repeats the wrong effect.
Late Data Without Endless Reprocessing
A correction window should bound automatic work without turning older truth into dropped data or a full-history rebuild.
Validation Is Not a Publish Protocol
Passing checks matters only when the evidence, candidate and reader-visible commit belong to the same revision.
Proving a Pipeline Is Replayable
Replayability becomes credible when controlled failures produce independently observed evidence that named invariants still hold.