Silent failures: watermarks, vacuum, and pipelines that look green

Green ticks are not a test

A long-running pipeline can succeed and still skip a day, double a batch, or vacuum the only copy of last month. Schedulers report 'OK' because the process exited 0. Finance notices six weeks later. We put guardrails at the grain: row counts versus yesterday, hash of the business keys, a watermark that only moves when the new high-water mark is real.

Watermarks, not timestamps you hope for

If you increment on 'updated_at' from a source that back-dates corrections, you will miss the correction forever. If you increment on load time, you will reprocess yesterday until the table explodes. We store the watermark per source, per table, and we only advance it after the target reconcile matches. Late data gets a replay window with a cap, not an infinite backfill.

Vacuum and delete are not chores

Delta and Iceberg vacuum remove files that the table no longer references. Run it too soon and a reader still on an old snapshot fails. Run a DELETE as a way to 'fix duplicates' and you have rewritten history without an audit. We treat vacuum retention as a policy: enough days for time travel and for the slowest downstream job, then a named person who can restore. We never vacuum in the same run as a load we have not reconciled.

A retailer we supported was deleting 'old' partitions from a lake to save storage. The Power BI dataset still pointed at them. Visuals went blank on a Monday. The fix was boring: retention in writing, vacuum on a separate schedule, and a canary query after every destructive job.

Guardrails we actually install

No overlapping runs. Alert on zero-row success. Alert on 20 percent row swing without a release note. Quarantine files that fail schema. Do not let a notebook publish to gold. Gold is a job with tests. That is how silent failure becomes a ticket at 07:10, not a board paper in month three.