Your orchestrator will run that load again. Nothing anywhere checks what the second run did.

Most of this is free, and the free versions are good

Every page in this section starts by naming what you can get for nothing, because a service that hides its free competition is not worth buying. Here the free competition is unusually strong.

A language model will review your pipeline properly

Paste in a DAG, a dbt project or a Glue script and you will get a competent critique in seconds — the anti-patterns, an idempotency checklist, a corrected merge statement, and a clear explanation of every concept on this page. Pipeline code is text, and text is what models are best at. Do that first. If it finds something obvious, fix it and you may not need this at all.

dbt ships tests, and Great Expectations and Soda Core are open source

unique, not_null, relationships, accepted_values, plus anything you write yourself — free, in the tool your team already runs. The open-source frameworks go considerably further than anything a consultant would hand-write. Install them yourself; you do not need me for that.

Airflow gives you retries, backfill and catchup, and documents the right answer

Its best-practices page tells you plainly what to do about all of this. So does the Glue documentation. The advice is free, correct and public, and it is quoted below rather than paraphrased.

My own free tools give you nothing at all here, and that is worth saying plainly

There are seventeen checkers on this site and not one of them touches a data pipeline. No finding, no id, no report. Every other page in this section starts by pointing you at a free checker of mine; this one has none to point at, because the fault is not visible from outside the system it lives in.

Notice what none of that list contains: a way to find out whether your pipeline survives being run twice. Every entry either describes the problem or tests the data at rest. None of them runs your load again and counts what changed.

A pipeline is not code. It is code plus the state it has already written.

The first run writes into an empty table. Every run after that writes into a table already holding the result of the last one — and that is the case nobody rehearses, because the only place it exists is the place nobody wants to experiment in.

It has to be caused, and the cause is routine. A retry after a transient failure. A backfill after a bad day. A late-arriving batch. Somebody re-triggering the job because the numbers looked wrong. The orchestrator does this automatically and by design — and the state that breaks a rerun does not exist until the pipeline has been running long enough to have written something.

And the damage is downstream, where nobody is looking. A second run does not only touch the target table. It can rebuild a materialised view something is reading, re-trigger a webhook, re-send an export, or re-push rows into a system that emails your customers. Almost nobody has written that list down, which is why it is one of the two things a drill hands back.

Three vendors document this limitation in their own products

Not a competitor comparison and not a criticism. These are good tools, and each of them is straightforward about the same boundary. Quoted rather than paraphrased, with the page that carries each sentence linked so you can check it in one click.

• AWS — Glue job bookmarks, the feature built for rerun safety When you rewind or reset a bookmark, AWS Glue does not clean the target files because there could be multiple targets and targets are not tracked with job bookmarks. Only source files are tracked with job bookmarks. This is the sentence the page rests on. The mechanism sold as rerun safety is explicitly one-sided, and AWS says so in the documentation for that mechanism. The same page adds that in some cases, you might have enabled AWS Glue job bookmarks but your ETL job is reprocessing data that was already processed in an earlier run — so it sometimes falls short on the half it does cover. Tracking processed data using job bookmarks
• Apache Airflow — best practices, “Creating a task” Airflow can retry a task if it fails. Thus, the tasks should produce the same outcome on every re-run. Read the modal verb. Should, not does. The retry is automatic; whether it is safe is entirely the author’s responsibility, and nothing in Airflow checks it. The scheduler will run a double-inserting task twice quite happily. The same section is specific about how: Do not use INSERT during a task re-run, an INSERT statement might lead to duplicate rows in your database. Replace it with UPSERT. It also says Read and write in a specific partition. Never read the latest available data in a task. and that the Python now() function should never be used inside a task, especially to do the critical computation, as it leads to different outcomes on each run. Airflow best practices
• dbt — incremental models, two documented defaults If you add a column to your incremental model, and execute a dbt run, this column will not appear in your target table. That is the default behaviour and it is silent. dbt states on the same page that this is on_schema_change: ignore, which is set by default. Separately, on merge keys: Columns used in this way should not contain any nulls, or the incremental model may fail to match rows and generate duplicate rows. Carry that sentence with its condition or not at all. dbt says may, it is written about a unique_key supplied as a list of column names, and not every incremental strategy even uses a unique_key — insert_overwrite works on partitions rather than rows and does not. dbt incremental models

Weight them honestly. The Glue sentence is the primary one, because it is a property of the system rather than a setting anyone can change. Airflow’s should and dbt’s two defaults support it. All three are quoted from the page that carries them, and all three were re-opened and checked against the source text on 23 August 2026.

Six ways a second run changes something it should not

None of these is exotic. Every one is a documented default or a habit that reads as perfectly normal in review, which is exactly why reading the code is not enough to find them.

An INSERT where the target needed a merge

The commonest one by a distance, and the one Airflow tells you about on its own best-practices page. The first run inserts. The retry inserts again. Nothing errors, because inserting rows is exactly what the statement was asked to do. The repair is an upsert keyed on something that actually identifies a row.

A merge key that can be null

The write is already a merge, so this one looks fixed. But a null in the key means the match silently fails and the row is written as new. dbt documents this against its own unique_key and the wording is careful — it says the model may generate duplicate rows. The repair is either a key that cannot be null or a single surrogate key, which is what dbt itself recommends.

A now() inside a transformation

The load is scoped to “today”, so a rerun tomorrow writes to a different bucket than the run it was meant to repeat, and a backfill writes every historical day into the present. Airflow’s own guidance is that this function should never be used inside a task for critical computation. The repair is a fixed interval passed in, not read from the clock.

Reading “the latest” instead of a fixed partition

The task reads whatever is newest at the moment it runs. Rerun it a week later and it reads a week’s more data, so the second run legitimately produces a different answer and neither is wrong. The repair is reading and writing one named partition, which is the same advice from the same page.

A new column that never arrives

Not a duplicate-row fault at all — the opposite. Add a column to an incremental model and the default behaviour writes the table without it, with no error. The rows look right and a field is quietly missing. The repair is choosing an on_schema_change value on purpose rather than inheriting the one that stays quiet.

A sink nobody counted

The target table is rarely the only thing a load writes. A materialised view rebuilds, a BI extract refreshes, a webhook fires, a partner export is dropped again, a reverse-ETL sync pushes rows into a system that emails people. The repair starts with the list, because you cannot fence what you have not written down.

Every one of those is fixable and most are a small change. The hard part is never the repair — it is knowing which one you have, on which stage, before it costs you a quarter of restated numbers.

Two artifacts, and here is what both of them look like

Rendered rather than described, because a service page that describes its own output is asking you to take the interesting part on trust.

The rerun ledger

One scheduled load, run a second time against a clone carrying the previous run’s data. Per stage, never as a total — the total is the number that hides everything, because a stage that duplicates and a stage that drops can sum to a delta that looks fine.

The expected column is written down by you, in your own words, before the drill runs. That is deliberate and it is half the deliverable: the gap between what you expect a second run to change and what it actually changes is the finding. Agreeing it afterwards would let either of us describe whatever happened as the intended outcome.

Look at the reported column. Every stage above reported success, including the two that were wrong. That is the whole problem in one column, and it is the same shape as a deployment rollback that the pipeline calls finished while the old version is still serving.

The blast list

Everything a second run touches, not just the table everyone thinks of — and which of them are safe to fire again, decided one at a time before anything is rerun.

Both tables above are an example of the shape rather than anybody’s real system — but the columns are the real columns. The numbers are illustrative and are not a claim about a client, a project or a result. What is real is the structure: rows before, rows after, expected against actual, per stage, against what the job reported; and every sink named, classified and decided.

What is free here, and what is not

Once, in one place, rather than a locked row beside every paragraph.

Nothing here is gated, blurred or held back behind an email address. There is no scanner on this page and therefore no result to withhold — the free column above is not a sample of the paid one, it is genuinely everything that can be given away without touching your system. What costs money is running your load, which nobody can do from a web page.

And if the drill cannot run, that is a check that did not happen — not a tier you failed to buy. If your warehouse cannot produce a clone carrying the previous run’s state, the honest answer is that the drill is not available on that target yet, said on the day and before anything is committed to. Dressing an incomplete measurement as a paid upgrade is the one thing that would make this page dishonest.

This asks to run a production load a second time, so the limits are in writing

It is a bigger request than anything else on this site — bigger than changing a DNS record, bigger than rolling an application back. A bad rerun corrupts data, and data does not roll back the way a container image does.

It looks like the deployment rollback review, and it is not one

Both do a dangerous thing on purpose, inside an agreed window, and hand back a measurement. If you have read that page the resemblance is real and worth putting side by side rather than glossing over.

That last row is the honest one and it cuts against this page rather than for it. The boundary between the two is in the subject matter, not in any shared code — there is no finding on either side of it, because there is no finding at all. If what you actually need is your release path timed rather than your load counted, go and read that page instead; it is backed by evidence this one does not have.

Bulk document data extraction is the only other data-shaped page here and there is no overlap: it creates a dataset out of documents, while this one tests what happens when a load that already exists runs again.

Four ways in, and the second one is where most loads end up

Everyone starts with the drill, because until a rerun has been counted nothing after it can be aimed at anything — including by me.

The monthly tier is not monitoring and it is not data observability. It produces no graph and nothing that wakes anybody up. It runs the same drill, counts the same rows, and compares this month’s answer with last month’s. If continuous watching is what you want, that is a different service.

There are no figures on this page, and that is deliberate rather than coy. A fixed number would be too high for a three-stage load writing one table and too low for a load feeding a CRM, a partner export and six downstream models. Tell me the shape of it in the form — which load, how many sinks, whether any of them reaches a customer — and you get a figure back before anything is committed to.

What this does not do

Stated before the tiers rather than after them, because a limit found afterwards is an excuse.

• A rerun drill does not prove your pipeline is correct. It proves what a second run does. Whether the first run produced the right numbers is a separate question, and it needs your business logic rather than my measurement.
• A clone is not production, and the one thing it cannot reproduce is concurrency. If something else writes to that table while the load runs, the drill did not see it. That is a real limit and it is written into the report rather than left for you to assume.
• Stubbed sinks are not tested sinks. Whatever was disabled for safety was not exercised, and the blast list names every one of them.
• Idempotent is not the same as cheap. A load made safe to rerun may read or write more than it did before. That trade gets stated up front rather than discovered in a bill.
• This is not data quality testing and it is not observability. It is one question answered with a number.
• A clean result says nothing about future runs. A pipeline that reruns cleanly today is a pipeline nobody has edited yet, which is the entire argument for the monthly tier and is not a claim that anything is permanently solved.

What I need before anything can be quoted

Most of this you can answer in the contact form in a couple of minutes, and the answers change the price more than anything I could guess at.

• Which single scheduled load is in scope, and what it writes.
• A clone, snapshot or restore of the target carrying the previous run’s data — or the access to create one. If that is not possible on your warehouse, say so early; it changes the shape of the work.
• The list of every sink, including the ones that are not tables. This is the question most likely to come back incomplete, and that is itself worth knowing.
• Whether any sink is customer-facing. Asked before anything is quoted, because it is the single biggest driver of the size of the job.
• Read access to the pipeline definition — a DAG, a dbt project, a Glue job, a scheduled script, whatever it actually is.
• A backup taken by you and verified by you. Not sold here, not vouched for here.
• What you expect to change if the load runs twice, in your own words, in writing, before the drill. Half the deliverable is the gap between that and what happens.

Things this is not, and where they live instead

• Building the pipeline. Reviewing and repairing rerun behaviour is in scope. Greenfield construction is a different conversation.
• Data quality testing and observability — dbt tests, Great Expectations, Soda, freshness alerting. Free products, better than hand-written ones, and yours to install.
• Deployment pipelines and rollback — that is the deployment and rollback review, and the difference is in the table above.
• Backups and proving a restore — the resilience review.
• Turning documents into a dataset — bulk document data extraction.
• Warehouse cost reduction, BI modelling, and migrating between warehouses. Real work, not this page.

What I can show you rather than tell you: travel-etl-pipeline is a public repository — a serverless ETL pipeline for NYC travel data on GCP, ending in a Tableau reporting layer. Ingest, transform, warehouse, visualise. Read the code rather than taking my word for anything.

Questions