That is the moment the AWS Amplify vs Firebase decision actually gets made. Not on day one, when both platforms look like a schema file, a typed client and a deploy that just works. On day two hundred, when the read path you committed to without noticing starts charging rent.

This post is a comparison, but not a feature tour. Both vendors publish good feature tables and neither of them is lying. What they do not tell you is which parts of the platform you can walk away from cheaply and which parts you are stuck with. So this is organised around the four things that decide that: the data model, the billing shape, the auth layer, and the exit cost.

The failure mode that bites: your read path is decided on day one

Both platforms hand you a pleasant abstraction over a NoSQL store. Firebase gives you Firestore, a document database. Amplify’s data layer, by default, gives you AppSync sitting in front of DynamoDB. In both cases you write a schema, get a typed client, and start shipping.

Here is what neither schema file makes obvious: it is not a description of your data. It is a bet on the queries you will need. Change the bet later and you are not editing a schema, you are backfilling.

In Firestore, every query shape needs an index that supports it. Filter on two fields and sort on a third and you need a composite index. The console will helpfully offer to create it for you, which is why most teams never notice they are accumulating a per-query-shape index budget. There are no joins. If an order needs its customer’s name, either you denormalise the name into the order document or you do a second fetch per row.

In Amplify, relationships between models are resolved by AppSync, which usually means an extra round trip per relationship. Secondary indexes have to be declared in the schema so the underlying table can be built to serve them. Same constraint, different vocabulary.

The good news is that both vendors have finally admitted this is a problem. Firebase Data Connect puts a managed Cloud SQL for PostgreSQL instance behind a GraphQL schema and generates typed SDKs against it. On the AWS side, Amplify’s data layer can be pointed at a SQL database rather than DynamoDB, and the newer Blocks approach lets you compose Postgres and other capabilities into an existing Amplify backend. If your application has anything resembling reporting, joins or ad-hoc filtering in its future, start there rather than starting on documents and migrating under pressure.

AWS Amplify: where it wins and where it doesn’t

Amplify Gen 2 defines the whole backend in TypeScript. Auth, data, storage and functions all live as code in an amplify/ directory, get reviewed in pull requests, and compile down through CDK to CloudFormation.

// amplify/data/resource.ts
import { type ClientSchema, a, defineData } from '@aws-amplify/backend';

const schema = a.schema({
  Order: a
    .model({
      region: a.string().required(),
      status: a.enum(['PENDING', 'SHIPPED', 'CANCELLED']),
      total: a.float(),
    })
    .authorization((allow) => [allow.owner()]),
});

export type Schema = ClientSchema<typeof schema>;

export const data = defineData({ schema });

Two things in that snippet matter more than they look. The authorization call is not a comment about intent, it generates the actual resolver-level access rules, so an access change is a code change with a diff. And ClientSchema flows the backend types into the frontend, which means removing a field breaks the build rather than breaking production at 3am.

Development happens in a per-developer cloud sandbox. It watches the amplify/ folder and redeploys on save:

# Deploy and watch your own isolated backend stack
npx ampx sandbox

# Tear it down when you're finished - it is real infrastructure
npx ampx sandbox delete

Where Amplify wins

• You are already on AWS. If your organisation has accounts, IAM boundaries, a VPC design and a compliance story, Amplify slots into all of it. Firebase would mean a second cloud to audit.
• The escape hatch is real. Because the backend compiles to CDK, you can drop into raw constructs and attach any AWS service without leaving the framework. That ceiling is much higher than Firebase’s.
• Environments are branches. Git branch to deployed environment is the native model, so staging is not a thing you build, it is a thing you push.
• Type safety end to end. This genuinely catches a class of bug that Firebase projects tend to find in production.

Where Amplify doesn’t

• Gen 1 is on a clock. Amplify Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. This matters because a lot of the tutorials, Stack Overflow answers and agency codebases you will encounter are Gen 1, and Gen 1 to Gen 2 is a migration, not an upgrade. AWS ships tooling for it, but the guide describes a blue/green run with an irreversible refactor step. Budget for it properly.
• “Local” development is not local. The sandbox is a real CloudFormation stack in a real account. It is high fidelity, which is the point, but your inner loop now includes deploy time and your dev environment costs money.
• Errors surface as AWS errors. A mistake in a TypeScript schema can come back as a CloudFormation rollback referencing a resource you never named. Debugging Amplify means being comfortable reading AppSync, DynamoDB and CloudWatch.
• Smaller community. Fewer answers exist, and a meaningful share of the ones that do exist are for the previous generation.

Google Firebase: where it wins and where it doesn’t

Firebase’s advantage is not that it is simpler. It is that the whole loop, from writing a rule to seeing a client behave, is shorter. The emulator suite runs the pieces on your machine:

# Run auth, Firestore and functions locally, no cloud project touched
firebase emulators:start --only auth,firestore,functions

The thing you will spend the most time on is security rules, and it is worth understanding why they are shaped the way they are:

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /orders/{orderId} {
      allow read: if request.auth != null
                  && request.auth.uid == resource.data.ownerUid;

      allow create: if request.auth != null
                    && request.auth.uid == request.resource.data.ownerUid;

      allow update, delete: if false;
    }
  }
}

The distinction people get wrong is resource.data versus request.resource.data. The first is the document as it exists in the database, so it protects the read path. The second is the document the client is trying to write, so it protects the write path. Use the wrong one in a create rule and you are checking ownership on a document that does not exist yet, which evaluates to nothing and quietly lets the write through. That is the kind of bug that ships.

Where Firebase wins

• Mobile. Crashlytics, Cloud Messaging, Remote Config, A/B testing and analytics are one SDK away and genuinely good. AWS has equivalents, but not assembled like this.
• Offline and realtime. Firestore’s offline persistence and listener model handle flaky connectivity without you designing for it. If your users are on trains or in warehouses, this is a serious argument.
• Emulators. Being able to run the backend on a laptop, in CI, with no cloud project, is a real productivity difference against Amplify’s cloud sandbox.
• Time to first screen. For a prototype or an MVP that needs to exist by Friday, Firebase usually wins on speed alone.

Where Firebase doesn’t

• Configuration drifts out of git. The console is convenient and it is also a second source of truth. Indexes get created from an error link, a provider gets enabled by hand, and six months later nobody can rebuild the project from the repository. Terraform and the Firebase CLI both help, but you have to decide to use them.
• Security rules are a language, not a checkbox. They are also evaluated on the server for every access, which has cost implications when rules perform document lookups.
• Features you assume are included are an upgrade. Multi-factor auth, SAML and OIDC, blocking functions, audit logging and multi-tenancy live behind the Identity Platform upgrade, which comes with a different pricing model. On the free Spark plan, upgrading also introduces a daily active user cap. Find this out during design, not during an enterprise sales cycle.
• The ceiling is lower. When you outgrow Firebase you are usually moving to Google Cloud proper, which is a different set of tools and a different mental model.

What actually generates the bill

Rates change, so treat published numbers as current-at-time-of-reading and go to the vendor pricing pages for the arithmetic. What is stable, and what you should actually reason about, is the shape of the meters.

Firestore charges per document, and the details are where teams get caught:

• A query that returns 200 documents is 200 reads, whether you use two fields from each one or forty.
• A realtime listener is charged a read each time a document in its result set is added or updated. Attach a listener to a busy collection and you have built a meter that runs while nobody is looking at the screen.
• Where your SDK exposes query offsets, skipped documents are still billed. Paginate with cursors, not offsets.
• Aggregations such as count() are billed per batch of index entries read, with a minimum of one document read. Cheap, but not free, and not zero when you run them per page load.
• Stored bytes include indexes and metadata, so index sprawl shows up twice: once in write amplification, once in storage.

Amplify has no single meter, because Amplify is not a service. The bill is assembled from AppSync (query and mutation operations, realtime update messages, and connection-minutes for open WebSockets), DynamoDB, Lambda invocations and duration, CloudWatch log ingestion and retention, Cognito monthly active users, and Amplify Hosting build minutes plus data served.

That difference is the practical one. Firebase gives you a bill you can read in the console in thirty seconds. AWS gives you eight line items that will not tell you which feature caused them unless you tagged resources properly on the way in. If you go the AWS route, set up cost allocation tags and a budget alarm before launch, not after the first invoice. Pushing CloudWatch metrics into Grafana alongside your application metrics is worth the afternoon it takes, because cost anomalies and traffic anomalies are usually the same incident viewed twice.

Auth is the component you cannot cheaply replace

Firebase Authentication and Amazon Cognito are both fine. Neither is the reason to pick a platform. But auth is the layer where switching hurts most, because you are not migrating data, you are migrating sessions and identities, and every user notices.

Both let you export users, including password hashes, so a migration is technically possible. What is hard is everything hanging off the user ID: rules that compare against it, resolver authorisation that assumes it, third-party integrations keyed on it, and support tooling that looks users up by it.

One habit removes most of that pain regardless of which platform you pick: store your own internal user record from the first commit, with your own primary key, and treat the provider’s UID as an external identifier on that record. It costs you one table and one lookup. It converts an auth migration from a rewrite into a mapping exercise.

Hosting is the layer nobody should agonise over

Amplify Hosting deploys from a git branch and supports server-rendered frameworks. Firebase now has two: the original Hosting for static sites, and App Hosting for modern server-rendered apps, which is generally available and runs on Cloud Run with Cloud CDN in front.

Both are good. Neither should influence your decision much, because hosting is the one layer with genuine competition and near-zero switching cost. A Next.js or Nuxt app moves to Vercel, Netlify, Cloudflare Pages or a container on a VPS from a provider like InterServer or DigitalOcean in an afternoon. Put a CDN such as Cloudflare in front and the origin becomes an implementation detail your users never see.

Spend your deliberation budget on the data layer instead. That is where it pays.

Exit cost, ranked

If you rank the layers by what it costs to leave, the ordering is the same on both platforms and it is the opposite of the ordering people use when choosing:

1. Hosting. Swap it in a day. Barely counts as lock-in.
2. Functions. Cloud Functions and Lambda handlers are mostly your own code with a different signature wrapped around it. A rewrite, but a bounded one.
3. Auth. A user migration project with a communications plan attached.
4. The data model. You do not migrate this. You rebuild the read path. Firestore’s managed export produces Firestore-format files in Cloud Storage, and DynamoDB exports to S3, and in both cases what lands is denormalised documents shaped by decisions you made two years ago, not a relational dataset you can point a reporting tool at.

There is a fifth item with no clean export at all: realtime. Firestore listeners and AppSync subscriptions have no portable equivalent, so if realtime is core to your product, whichever you choose is close to permanent. That is not automatically an argument against either. It is an argument for knowing you are making that commitment when you make it.

AWS Amplify vs Firebase: how I’d actually decide

Not a scorecard. A procedure, in order, stopping at the first clear answer:

1. Write down your three hardest read queries. Not the CRUD. The reporting screen, the search, the admin filter. If you cannot write them, you are not ready to choose a backend.
2. Do any of them need joins, aggregates or filters you cannot predict? If yes, start relational: Firebase Data Connect, Amplify over a SQL data source, or plain Postgres behind an API you own. Do not start on documents planning to fix it later.
3. Is this mobile-first with real offline requirements? Firebase, and it is not close.
4. Does your organisation already run on AWS? Existing accounts, IAM, VPCs, audit posture and a security team who will ask questions about a second cloud all point to Amplify, even if Firebase would be faster this quarter.
5. Who is maintaining this in two years? Amplify Gen 2 expects TypeScript fluency and some tolerance for CDK and CloudFormation. Firebase expects less. Match the platform to the team you will actually have.
6. Any data residency or regulatory constraints? Check region availability for every service you plan to use, not just the headline one, before you commit.
7. Still tied? It is a prototype. Pick the one your team can ship this week, and write down explicitly that the data model is the part you will have to revisit.

Mistakes I see repeatedly

• Choosing on SDK ergonomics. Both SDKs are pleasant. You are picking a data model and a billing model, and the SDK is the layer you would replace most easily.
• Building reporting on the transactional store. Both platforms punish this. Stream to a warehouse or a read replica and let the operational store do one job.
• Launching without a budget alarm. A missing limit(), a listener on a busy collection, or one post that gets traction can turn a hobby bill into a real one overnight. Set the alarm before you need it.
• Treating authorisation as a later task. On both platforms, access control is coupled to the data model. Retrofitting it usually means reshaping documents.
• Following Amplify Gen 1 material. The commands, the directives and the mental model are all different. Check which generation a tutorial targets before you follow it.
• Letting console clicks become infrastructure. Anything you enabled by hand is something you cannot rebuild. Get it into the CLI config or into Terraform.
• Assuming the free tier is the plan. It is a trial of the plan. Model your costs at ten times current traffic and see whether you still like the answer.

Frequently asked questions

Is AWS Amplify or Firebase cheaper?

It depends entirely on your access patterns, and anyone who answers this without seeing them is guessing. Firebase tends to be cheaper and far more predictable at small scale, partly because the free allowances are generous and partly because there is one meter to watch. AWS tends to win when you have steady, high-volume traffic and the engineering discipline to tune DynamoDB capacity and Lambda sizing. The bigger practical difference is legibility: a Firebase bill tells you what happened, an AWS bill tells you what was consumed.

Can I move from Firebase to AWS Amplify later?

Yes, but price it by layer rather than as one project. Hosting and functions move cheaply. Auth is a real migration with user impact. The data model is usually a rebuild rather than a move, because a Firestore export gives you denormalised documents, not a relational dataset. Teams that manage this well are the ones that kept business logic out of security rules and out of client code from the beginning.

Is Amplify Gen 1 still supported?

Gen 1 is in maintenance mode and reaches end of life on 1 May 2027. It receives critical fixes and security patches, not new features. If you are starting something new, use Gen 2. If you have a Gen 1 application in production, schedule the migration deliberately using the official tooling and guide, and test thoroughly before the refactor step, which is difficult to reverse.

Do I need Firebase Data Connect, or is Firestore enough?

Firestore is enough when your reads are predictable, mostly key-based, and denormalise cleanly. The moment you need joins, aggregates across collections, or filters that product will keep changing, Data Connect’s managed Postgres is the better foundation. It is also a lot easier to adopt at the start than to migrate onto after a year of document design.

Which one is better for a mobile app?

Firebase, in most cases. The offline persistence model, the crash reporting, the push messaging and the remote configuration are all mature and designed to work together. Amplify has mobile libraries and they work, but you are assembling the surrounding pieces from separate AWS services rather than getting them as a set.

What about a self-hosted alternative?

Worth considering if lock-in is your main worry. Supabase and Appwrite both offer a comparable developer experience over Postgres, and either can run on your own infrastructure. Understand what you are taking on, though: backups, restore testing, upgrades, patching and monitoring all become yours. That is a real operational load, and it is the trade most teams are implicitly paying a managed vendor to avoid.

Can I use both together?

You can, and some teams do run Firebase for analytics and messaging alongside an AWS backend. Be honest about the cost: two identity models, two billing accounts, two audit trails and two on-call runbooks. Do it because one platform is clearly better at a specific job, not because it was easier than making a decision.

The one thing worth remembering

The AWS Amplify vs Firebase question is almost never settled by features, and it is never settled by which SDK feels nicer in a tutorial. Both platforms will get you to a working application quickly, and both will hold up in production for the workloads they were designed for.

What separates them, months later, is the read path you committed to on day one and the layer you cannot afford to leave. Write down your three hardest queries before you write any schema. Decide whether you need relational storage while it is still a decision rather than a migration. Keep your own user record from the first commit. Set a budget alarm before launch.

Do those four things and either platform is a reasonable choice. Skip them and neither one saves you.

Need a second opinion before you commit?

Backend platform decisions are cheap to review and expensive to reverse. Things I help with in this area:

• Reviewing a Firestore or DynamoDB data model against the queries you actually need, before the design hardens
• Cost modelling both platforms against your real traffic assumptions, with the meters broken out so you can see what drives the bill
• Amplify Gen 1 to Gen 2 migration planning, including the blue/green sequence and what to verify before the irreversible step
• Auditing Firestore security rules and Amplify authorisation rules for gaps between the read path and the write path
• Setting up cost allocation tags, budget alarms and CloudWatch or Grafana dashboards so a runaway query shows up in hours rather than on the invoice
• Exit planning: working out what a migration off your current platform would genuinely cost, layer by layer

If you want a concrete answer rather than a general one, send me the thing itself: your schema file, your security rules, a screenshot of last month’s billing breakdown, or the three queries you are worried about. That is usually enough to tell you something useful.

The post AWS Amplify vs Firebase: Choosing a Backend You Can Still Leave appeared first on John Nessime.

That is the shape of most API pipeline incidents. Not a crash. A success that isn’t one.

This post covers running Apache Airflow on AWS specifically for SaaS and API workloads: pulling from HubSpot, Salesforce, Stripe, Zendesk, Shopify, an internal partner API, whatever. It is organised by failure family rather than by feature, because the Airflow documentation already explains what an operator is and does a poor job of explaining which of these things will page you at 3am. I will cover choosing a deployment model, the concurrency trap that catches almost everyone, retry design, incremental state, secrets, and where the money actually goes.

Why SaaS and API sources break differently

When your source is a database you control, failure is loud: connection refused, deadlock, disk full. When it is somebody else’s SaaS API, three things change.

• You are a guest. The vendor decides your rate limit, and they can change it without telling you. Your pipeline’s correctness now depends on a number in someone else’s config file.
• Errors arrive as valid HTTP. A 429, a 200 with a truncated page, a 200 with an error object in the body. Your HTTP client is happy. Your data is not.
• Tasks spend most of their life waiting. API extraction is I/O bound almost end to end. That sounds harmless and is the root of the most expensive mistakes.

Pick the deployment model before you write a DAG

This decision constrains everything after it and is harder to reverse than people expect. Three realistic options.

Amazon MWAA, provisioned

AWS runs the scheduler, web server, workers, triggerer and metadata database on Fargate; you drop DAGs into an S3 bucket and they get picked up.

Where it wins: real Airflow, custom providers, custom plugins, full control over environment configuration. If your DAGs need arbitrary Python libraries, this option will not fight you.

Where it doesn’t: the environment bills by the hour whether or not anything is running. There is no scale to zero on the base environment. If you sync six APIs once a day and each run takes twenty minutes, you are paying for a mostly idle cluster around the clock. The mw1.micro class exists precisely for the small case, but it collapses the scheduler and worker into a single Fargate task and caps worker autoscale low, so treat it as a dev or isolation tier rather than a cheap production tier.

Amazon MWAA Serverless

You submit workflow definitions and AWS runs each task in its own Fargate container, billing per task duration with a one-minute minimum rather than per environment hour.

Where it wins: spiky or infrequent schedules. If the workload is “six syncs a day, nothing overnight,” the cost profile beats a permanently running environment by a wide margin. Each workflow also gets its own IAM execution role, which is a real security improvement over one shared role per environment.

Where it doesn’t: it leans on declarative YAML workflow definitions based on the DAG Factory format and a curated set of AWS operators. That is a deliberate trade: because the definition is declarative, the service can schedule tasks without executing your DAG code. It also means custom operators, exotic third-party providers and clever Python at parse time are not the sweet spot. It is also available in fewer regions than provisioned MWAA, so check your region before you design around it.

Self-managed on ECS, EKS or a VPS

On EKS with the Kubernetes executor you get per-task pods and tight cost control. On a single VPS from a provider like InterServer or Hetzner, a Docker Compose stack with a Postgres metadata database will run a modest set of API syncs for a fraction of any managed price.

Where it wins: cost at both extremes, and total control. Where it doesn’t: you now own metadata database upgrades, major version migrations, log retention and the 2am scheduler restart. Managed Airflow is a bet that your time is worth more than the hourly premium. For a solo engineer with three pipelines that bet often loses; for a data team of eight it usually wins. Astronomer is the main non-AWS managed option worth pricing alongside these.

Failure family one: the throttle that silently stops throttling

You start with a normal setup: an Airflow pool named crm_api with four slots, and every task that touches the vendor assigned to it. Four concurrent requests, comfortably under the vendor’s limit. This works.

Then you notice those tasks spend nearly all their runtime waiting on HTTP, burning worker slots to sit still. So you switch them to deferrable operators. A deferrable task suspends itself while waiting, releases its worker slot, and hands the waiting to the triggerer, which polls asynchronously. Worker pressure drops. Everything looks better.

And your rate limiting quietly stops working.

By default, a pool does not count tasks in the deferred state as occupying slots. That was deliberate, and the logic is sound in the abstract: a deferred task is not consuming a worker. But if you were using the pool to protect an external API rather than your own workers, it has just stopped doing the job you gave it. Every task can defer at once, and the vendor sees the full fan-out.

The fix is a per-pool flag, include_deferred, which tells the scheduler to count deferred tasks against the slot budget. It is off by default. You can set it when editing the pool in the Airflow UI, or through the API.

The failure signature is what makes this nasty. Nothing errors. Your DAG gets faster. The vendor starts returning 429s that your retry logic absorbs, and the only symptom is that runs take a little longer and occasionally a page goes missing. Weeks can pass. Two related traps in the same family:

• max_active_tasks at the DAG level has the same blind spot with deferred tasks, and there is no equivalent opt-in flag. If you need a hard external concurrency cap, use a pool with include_deferred enabled, not DAG-level concurrency.
• On MWAA, the triggerer runs alongside the scheduler on the same Fargate task, so scheduler count and triggerer capacity are linked. If you go heavily deferrable and your deferred tasks start stalling, scheduler capacity is the thing to look at.

Failure family two: retries that make the outage worse

The default instinct is to set retries high and move on. Against a rate-limited API, a fixed retry delay across many parallel tasks is just a slower version of the same stampede.

What you want is exponential backoff with a ceiling. The shape:

from datetime import timedelta

from airflow.sdk import dag, task

@dag(
    schedule="0 5 * * *",
    catchup=False,
    max_active_runs=1,          # never let two runs of this DAG overlap
    default_args={
        "retries": 5,
        "retry_delay": timedelta(seconds=30),
        "retry_exponential_backoff": True,   # 30s, 60s, 120s, 240s...
        "max_retry_delay": timedelta(minutes=15),  # stop doubling here
        "pool": "crm_api",      # shared budget across every task touching this vendor
    },
    tags=["crm", "extract"],
)
def crm_extract():

    @task(max_active_tis_per_dag=4)
    def fetch_page(page_token: str) -> str:
        ...

crm_extract()

The lines that matter:

• retry_exponential_backoff turns retry_delay into a base rather than a constant, so repeated failures spread out instead of hammering in lockstep.
• max_retry_delay caps the doubling. Without it, a task that fails five times can sit idle for hours and blow past the window you actually cared about.
• max_active_runs=1 is the one people skip. If a run overruns its schedule, the next one starts anyway, and now two runs are fetching the same pages from the same vendor with the same credentials. This is a common way to trigger a rate limit you have never hit before.
• max_active_tis_per_dag limits how many instances of that specific task run concurrently across DAG runs, which is the right knob for dynamically mapped extraction tasks.

One thing Airflow will not do for you: honour a Retry-After header. Airflow’s retry timing is computed from your config, not from the vendor’s response. If the API tells you exactly how long to wait, you have to catch that in your own code and sleep or reschedule accordingly. Ignoring a header the vendor bothered to send is a good way to get your API key throttled harder.

Failure family three: pagination, cursors and the empty page

Back to the Monday message. The specific bug behind most “the numbers are wrong but nothing failed” incidents is a loop that treats any non-error response as a terminating condition. Three rules prevent it:

1. Never infer “done” from an empty result. Terminate on the explicit signal the API gives you: a null next_cursor, a missing Link header, a page count. An empty array with a valid cursor still has more data behind it.
2. Assert the response shape before you use it. Check the status code explicitly and validate that the fields you depend on exist. A 200 carrying {"error": "..."} should raise, not return zero rows.
3. Land raw, transform later. Write the untouched API response to S3 first, then parse from S3. When the vendor changes a field type, you can replay from raw instead of re-extracting from an API that no longer serves that window.

Where to keep incremental state

The tempting pattern is to store the last-seen timestamp in an Airflow Variable and update it at the end of a run. Do not make that your source of truth. If a run dies midway, the Variable is in an undefined state, and clearing and re-running the DAG will not restore it. Airflow’s retry and backfill machinery has no idea it exists.

Better: make each run’s window a function of the run itself, and write output to a deterministic, run-scoped location such as s3://bucket/source=crm/dt=<logical-date>/. Re-running the same interval overwrites the same prefix. That is what makes a task idempotent, and idempotency is the difference between “clear the task and let it rerun” and a two-hour manual repair.

Then overlap your windows deliberately. Many SaaS APIs order results by modified time with eventual consistency, so a record edited at the boundary can appear after you have already moved on. Query a window slightly wider than your schedule interval and rely on an idempotent upsert downstream to absorb the duplicates. Late-arriving data is not an edge case with SaaS sources. It is the normal case.

Airflow’s asset-based scheduling is the clean way to trigger downstream DAGs from this: the extract DAG produces an asset, and the transform DAG runs when the asset updates, rather than being scheduled at a time you hope is late enough.

Failure family four: credentials

API tokens rotate, sometimes on the vendor’s schedule rather than yours. Storing an API key in an Airflow Connection through the UI works, and is the wrong long-term answer: the value lives in the metadata database and there is no rotation story. On AWS, point Airflow’s secrets backend at AWS Secrets Manager. On MWAA that is an environment configuration option:

secrets.backend
  airflow.providers.amazon.aws.secrets.secrets_manager.SecretsManagerBackend

secrets.backend_kwargs
  {"connections_prefix": "airflow/connections", "variables_prefix": "airflow/variables"}

With that in place, a connection lookup for crm_default resolves against the secret at airflow/connections/crm_default. Rotating the credential is a Secrets Manager operation with no Airflow deployment involved.

Two things to know before you turn it on. First, every connection and variable lookup becomes a Secrets Manager API call, and lookups fall through to the backend before hitting the metadata database, so a DAG that reads a Variable at parse time will generate a call on every parse cycle. Move those reads inside tasks. Second, the environment’s execution role needs explicit read permission on the relevant secret ARNs, and if you use a customer-managed KMS key, decrypt permission on that key too.

Worth knowing if you are on Airflow 3: task code can no longer reach the metadata database directly. All runtime interaction goes through the Task Execution API. If you inherited custom operators that open a session and query Airflow’s own tables, that is a migration blocker, not a warning.

Failure family five: the bill

Nobody is surprised by the environment line item. They are surprised by the other four.

• Idle time. A provisioned MWAA environment bills continuously. Compute the ratio of hours billed to hours doing work. If it is bad, that is the argument for MWAA Serverless or for consolidating several thin pipelines into one environment.
• NAT Gateway. This is the classic one. Private-subnet workers calling public SaaS APIs route through a NAT Gateway, which charges hourly and per gigabyte processed. A high-volume extraction pipeline can spend more on NAT than on Airflow. VPC endpoints remove that cost for AWS service traffic, but they do nothing for calls to a third-party API, which is exactly the traffic an API pipeline generates.
• CloudWatch Logs. Task logs go to CloudWatch, and ingestion is billed per gigabyte. Set the Airflow log level per component rather than globally at DEBUG, and set a retention policy on the log groups. The default is to keep logs forever.
• S3 requests. Landing raw API responses one small object per page generates a lot of PUTs. Batch pages into larger objects where you can.

Rates and dimensions change, so model your own workload against the current pricing page rather than trusting a number from a blog post. The point is knowing which four lines to look at.

Troubleshooting Apache Airflow on AWS when API pipelines misbehave

Tasks sit in “queued” and never start

Usually a slot problem, not a broken scheduler. Check, in order: is the pool full; has DAG-level max_active_tasks been hit; is worker autoscaling at its configured maximum. On MWAA, the container and queue utilisation metrics published to CloudWatch tell you which of the three it is far faster than reading scheduler logs.

DAG file changes don’t appear

On MWAA, DAGs sync from S3 on an interval; it is not instant. If a file has been there for several minutes and still hasn’t appeared, it almost always failed to parse. Check the DAG processing logs in CloudWatch rather than the scheduler logs, because a broken import raises there and never reaches the scheduler.

A new provider package won’t install

MWAA installs from your requirements.txt in the DAGs bucket, and from Airflow 2.7.2 onward that file must include a constraint line. Without one, MWAA picks a constraint for you, and pip is free to resolve a provider version that conflicts with the Airflow build in the image.

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-<AIRFLOW_VERSION>/constraints-<PYTHON_VERSION>.txt"

apache-airflow-providers-http
apache-airflow-providers-salesforce

Substitute the literal Airflow version your environment runs and the Python version bundled with it. MWAA does not expand shell variables in that file. Test the requirements file against a local Airflow image before you upload it, because a failed install on MWAA surfaces as a partially working environment rather than a clean error.

A backfill is stuck and you need to clear it

You do not need a web login token for this. MWAA exposes the Airflow REST API through a signed AWS API call, so you can drive it from CI or a runbook with normal IAM credentials:

aws mwaa invoke-rest-api 
  --name MyMWAAEnvironment 
  --path "/dags/crm_extract/clearTaskInstances" 
  --method POST 
  --body '{"dry_run": true}'

Start with dry_run set to true so the response tells you which task instances would be cleared before you actually clear them. Note that the resource paths differ between Airflow 2 and Airflow 3 environments, so confirm against the API version your environment exposes.

Deferred tasks stall forever

If deferred tasks stop resuming while the environment reports healthy, suspect the triggerer rather than your DAG. A triggerer that has lost its ability to process triggers can keep heartbeating normally, so the scheduler sees nothing wrong while every deferred task drifts toward timeout. This class of bug has been fixed and re-fixed upstream, so check your Airflow version’s release notes before assuming it is your code.

Common mistakes

• Switching to deferrable operators without enabling include_deferred on the pools that were protecting the API.
• Leaving max_active_runs unset, so a slow run and the next scheduled run compete for the same rate limit budget.
• Treating an empty response page as the end of pagination.
• Storing the incremental watermark in an Airflow Variable and updating it mid-run.
• Calling an API or reading a Variable at DAG parse time, which executes on every parse cycle rather than once per run.
• Transforming during extraction, so a vendor schema change means re-pulling data the API may no longer serve.
• Sizing the environment for peak concurrency when the actual constraint is the vendor’s rate limit.

Best practices

• One pool per vendor, sized to their published limit with headroom, and include_deferred enabled on every one of them.
• Land raw responses to S3 before parsing. Extraction and transformation are separate tasks with separate failure modes.
• Make every task idempotent and window-scoped, so “clear and rerun” is always a safe repair.
• Overlap extraction windows and deduplicate downstream rather than trusting a vendor’s timestamps to be exact.
• Secrets Manager for credentials, with the execution role scoped to specific secret ARNs.
• Alert on row counts and freshness, not just task state. A green DAG that produced 40% of yesterday’s rows is the failure you actually care about. Shipping Airflow’s StatsD metrics into Prometheus, Grafana Cloud or Datadog makes that a dashboard rather than a discovery.
• Define the environment in Terraform or OpenTofu. Recreating an MWAA environment by hand after a bad configuration change is a bad afternoon.

Frequently asked questions

Is MWAA worth it compared to self-hosting Airflow on EC2?

It depends almost entirely on how many people share the platform. MWAA’s premium buys you managed metadata database upgrades, patched images and version migration support. If one engineer maintains three DAGs, self-hosting on a modest VPS is cheaper and the operational load is real but small. Once several teams depend on the scheduler being up, the premium is easy to justify.

Should I use Step Functions instead of Airflow for API pipelines?

Step Functions is genuinely better for event-driven, AWS-service-centric orchestration with modest branching, and it scales to zero. Airflow wins when you need scheduled batch semantics, backfills over historical windows, dependencies between many pipelines, and a UI that non-platform engineers can use to see why last Tuesday failed. Backfill is usually the deciding feature.

Do deferrable operators reduce my AWS bill?

On provisioned MWAA, they reduce worker pressure, which reduces autoscaling into additional worker instances. The base environment cost is unchanged. On a Kubernetes executor setup where each task is a pod, the saving is more direct. Either way, do not adopt them purely for cost without revisiting your pool configuration first.

How do I handle a vendor with no documented rate limit?

Start conservative, one or two concurrent requests, and instrument the response status codes. Raise the pool size gradually and watch for 429s or rising latency. Latency creeping up under load is often the earlier signal, because some vendors throttle by slowing you down before they start rejecting.

Can Airflow read a Retry-After header automatically?

No. Airflow computes retry timing from retry_delay and the backoff settings on the task. If a vendor sends Retry-After, you need to handle it in your own request code or in a custom operator.

What breaks when upgrading to Airflow 3?

The big one for API pipelines is that task code can no longer access the metadata database directly; everything goes through the Task Execution API. Imports also move to the airflow.sdk namespace, and several core operators now live in the standard provider package. Audit custom operators first, since that is where direct database access hides. MWAA requires you to be on the latest Airflow 2 minor version before a major upgrade, so plan two steps.

How many DAGs can one MWAA environment handle?

The binding constraint is usually the metadata database and scheduler CPU, not DAG count. Watch metadata database memory and scheduler CPU utilisation; when either saturates, you either move up an environment class or split into multiple environments. Splitting also gives you blast-radius isolation, which matters more than people expect.

Wrapping up

Running Apache Airflow on AWS for SaaS and API pipelines is mostly not an Airflow problem. The scheduler works. The operators work. What bites is the gap between “the task succeeded” and “the data is correct,” and that gap lives in concurrency settings, pagination logic and retry design rather than anywhere Airflow will warn you about.

If you take one thing away: a green DAG is not a signal that your data is complete. Enable include_deferred on the pools protecting your vendors, terminate pagination on an explicit signal instead of an empty page, make every task idempotent, and alert on row counts. Those four things prevent most of the incidents that never show up as a failed task.

Need help with your Airflow pipelines on AWS?

I work with teams running data and API pipelines on AWS, usually somewhere between “it works but nobody trusts it” and “we need to move off cron.” Things I can help with:

• Reviewing existing DAGs for silent data loss: pagination logic, retry behaviour, pool and concurrency configuration.
• Choosing between MWAA provisioned, MWAA Serverless and self-managed Airflow, with a cost model for your actual schedule rather than a generic comparison.
• Building SaaS extraction pipelines that are idempotent and safely re-runnable, landing raw to S3 with incremental windows that survive failure.
• Cutting MWAA cost: environment right-sizing, NAT Gateway traffic, CloudWatch log volume and dependency install time.
• Airflow 2 to 3 migration audits, focused on custom operators and direct metadata database access.
• Data freshness and volume alerting in Grafana or CloudWatch, so you learn about a partial sync before the business does.

If something specific is broken, send me the DAG file, the task log, or the CloudWatch metrics for the run that went wrong. It is usually faster to look at the real thing than to describe it.

The post Apache Airflow on AWS: Building SaaS and API Pipelines That Don’t Lie to You appeared first on John Nessime.

The frustrating part is that the pipeline is fine. Webhooks are arriving, Lambda is running clean, S3 has the files, Athena returns rows. Nothing is broken in the way monitoring understands “broken”. The pipeline is faithfully recording something that is no longer true.

This post is about building a Shopify sales dashboard with AWS that survives that conversation. Not the wiring, which is well documented and mostly straightforward, but the design decisions that determine whether your numbers still hold up six months in. I’ll cover the three ingestion paths and when each one is the right call, why append-only pipelines drift, how to lay out S3 and Athena so recomputation is cheap, and what to do when the totals are already wrong.

Why a Shopify sales dashboard with AWS drifts from the admin

Here’s the thing that catches almost everyone: a Shopify order is not an event, it’s a mutable record.

An event pipeline assumes facts are immutable once written. A payment happened. A shipment left. You append it, you never touch it again, and the sum of the log is the truth. That model is why streaming architectures are so clean, and it’s exactly wrong for order data.

An order created on Monday can be edited on Tuesday, partially refunded on Friday, and fully refunded three weeks later. Every one of those changes belongs, financially, to Monday. If your pipeline appends the orders/create payload and never revisits it, Monday’s revenue is frozen at the moment of checkout and it will only ever be too high.

This is the invisible failure. Nothing alerts. No queue backs up. Your dashboard is confidently wrong, and the gap widens roughly in proportion to your return rate. A store with a two percent return rate takes a long time to notice. A fashion store running thirty percent returns notices in about a month, usually via an angry accountant.

The four adjustments that move historical numbers

• Refunds. Full or partial. A refund carries its own created_at, which is when the money moved back. The order it belongs to has a different, earlier date. You need both, and which one you attribute to depends on whether finance wants cash-basis or order-basis reporting. Ask before you build.
• Order edits. A merchant adds a line item or adjusts a quantity after the fact. The original payload is now stale. Shopify exposes both the original and the current totals precisely because of this.
• Cancellations. A cancelled order keeps existing in the API. If you filter only on payment status you will happily keep counting it.
• Test and draft orders. Test orders carry a flag marking them as such. Nobody remembers to filter these until a QA run during a quiet week produces a suspicious spike.

The design consequence is simple to state and annoying to implement: your pipeline must be able to recompute any past day. Every storage and partitioning decision below follows from that one requirement.

Getting data out of Shopify: three paths, three trade-offs

Before anything else: new Shopify apps are built on the GraphQL Admin API. The REST Admin API has been designated a legacy API and new public apps must use GraphQL. If you’re starting fresh, start there. If you inherited a REST integration, it probably still runs, but you’re on borrowed time and you should plan the migration rather than discover the deadline.

Path 1: EventBridge partner event source

Shopify can deliver webhooks straight into an Amazon EventBridge partner event bus in your account. No public endpoint, no API Gateway, no HMAC verification code, because verification only applies to HTTPS deliveries. Shopify’s own docs confirm EventBridge and Pub/Sub deliveries skip it.

You create the source in the Shopify app configuration using your AWS account ID, region and a source name, then associate it with an event bus in the EventBridge console and write rules to route it. The address you register with Shopify is the partner event source ARN, not the event bus ARN. That distinction accounts for a large share of the “I set it up and nothing arrives” threads on the Shopify forums.

A rule matching everything from the Shopify partner source looks like this. Start broad, then narrow once you’ve seen the real shape of an event:

{
  "source": [ { "prefix": "aws.partner/shopify.com" } ]
}

Send that to an SQS queue with a dead-letter queue attached rather than straight to Lambda. Buffering gives you a replay buffer when a downstream deploy goes wrong, and the DLQ means a bad payload parks itself instead of poisoning the whole rule. This is the path I reach for first for anything already on AWS.

Path 2: HTTPS webhooks into API Gateway and Lambda

The conventional route, and the right one if you need webhook delivery outside AWS too, or you want the payloads to pass through something you fully control. The cost is that you now own an internet-facing endpoint and the HMAC verification on it.

Verify against the raw request body, before any JSON parsing. Re-serialising the payload changes byte-for-byte content and the signature will never match. Use a constant-time comparison so the check doesn’t leak timing information:

import base64, hashlib, hmac

def verify(raw_body: bytes, header_hmac: str, secret: str) -> bool:
    digest = hmac.new(secret.encode(), raw_body, hashlib.sha256).digest()
    computed = base64.b64encode(digest).decode()
    return hmac.compare_digest(computed, header_hmac)

Shopify sends the signature in the X-Shopify-Hmac-SHA256 header, base64-encoded, computed with your app’s client secret over the raw body. Store that secret in Secrets Manager or as an SSM SecureString parameter, not in a Lambda environment variable.

Acknowledge fast. Shopify’s timeout is short and it retries with backoff over a finite window, so a handler that does real work inline will generate a wall of duplicate deliveries during a flash sale, exactly when you can least afford it. Return 2xx immediately, do the work asynchronously.

Path 3: scheduled GraphQL bulk pull

Webhooks give you low latency. They do not give you completeness. Anything that fails past its retry window is gone, and Shopify will eventually remove a subscription that keeps failing. That’s a silent data loss mode with no local symptom at all.

So run a scheduled reconciliation pull alongside the stream. Shopify’s GraphQL bulk operations are built for this: you submit a query, it runs asynchronously, and you fetch a JSONL result file when it finishes. That’s the right tool for backfills and nightly catch-up, rather than paginating thousands of pages against a points-based rate limiter and getting throttled halfway through.

A nightly job that re-pulls the last seven to fourteen days and overwrites those partitions costs almost nothing and quietly fixes every category of drift described above. If you build one thing from this post, build that.

Decide what “revenue” means before you write a line of SQL

This is where most reconciliation arguments actually live, and it isn’t an engineering problem at all until you’ve had the conversation.

Shopify’s own sales reporting builds total sales from gross sales, minus discounts, minus returns, plus taxes and shipping. Gift card sales sit outside that in a separate finance report. If your dashboard sums order totals and calls it revenue, you have built a different metric with the same name, and it will disagree with the admin forever no matter how good your pipeline is.

Write the definition down. Put it in the dashboard as a tooltip. When someone challenges a number, you want the argument to be about the definition, not about whether your infrastructure works.

The currency trap

If the store sells in more than one currency, the money fields split in two. Shopify exposes totals as a set containing both shop_money and presentment_money: the amount in the store’s base currency, and the amount the customer actually saw and paid.

Sum the presentment amounts across a multi-currency store and you get a number with no meaning at all, euros and yen added together as if they were the same unit. For a single reporting figure you want the shop-currency side. Keep the presentment amount and its currency code in the table anyway, because the day someone asks “how much did we actually sell in Germany”, you’ll want it and it is painful to backfill.

One caveat worth knowing: orders created through the API rather than through checkout can behave differently from native multi-currency checkout orders. If your store takes orders from an ERP or a marketplace integration, spot-check a few of those specifically.

Storage layout: partition by order date, never by arrival date

Two layers in S3. Keep them separate and keep them honest about what they are.

1. Raw. Every payload exactly as received, partitioned by ingestion date. Append-only, never edited. This is your audit trail and your rebuild source. Lifecycle it to a colder storage class after a few months, don’t delete it.
2. Curated. One row per order representing current state, in Parquet, partitioned by order date. This is what the dashboard queries. It is derived, disposable and rewritable.

The partitioning choice on the curated layer is the load-bearing decision in the whole design. If you partition by arrival date, which is what Amazon Data Firehose does by default because it buckets on the moment it writes the file, then a refund that arrives three weeks late lands in today’s partition. Correcting Monday now means finding and rewriting fragments scattered across twenty other partitions. Partitioned by order date, correcting Monday means overwriting exactly one prefix.

Firehose can do this with dynamic partitioning, which routes records by keys inside the payload rather than by write time. If you’re not using Firehose, extract the order date in your Lambda and write the prefix yourself.

Use partition projection so Athena stops guessing

The default Glue Data Catalog approach means running a crawler or issuing MSCK REPAIR TABLE to register new partitions. Forget one and you get a query that silently returns nothing for recent days. Nobody notices until Monday.

Partition projection removes the metastore lookup entirely. You tell Athena the shape of the partition keys and it calculates the prefixes at query time:

CREATE EXTERNAL TABLE shop_orders (
  order_id             bigint,
  order_number         string,
  created_at           timestamp,
  financial_status     string,
  cancelled_at         timestamp,
  is_test              boolean,
  total_shop           decimal(12,2),
  shop_currency        string,
  total_presentment    decimal(12,2),
  presentment_currency string
)
PARTITIONED BY (order_date string)
STORED AS PARQUET
LOCATION 's3://your-bucket/curated/orders/'
TBLPROPERTIES (
  'projection.enabled' = 'true',
  'projection.order_date.type' = 'date',
  'projection.order_date.format' = 'yyyy-MM-dd',
  'projection.order_date.range' = '2019-01-01,NOW',
  'projection.order_date.interval' = '1',
  'projection.order_date.interval.unit' = 'DAYS',
  'storage.location.template' =
    's3://your-bucket/curated/orders/order_date=${order_date}/'
);

Set the range start to your store’s actual first order month. Projection generates every prefix in the range, so a range starting a decade too early makes wide scans slower for no benefit.

Parquet matters here for the same reason. Athena bills on bytes scanned, so a columnar format with good compression cuts the bill directly, and a dashboard that only ever selects six columns from a forty-column table never touches the rest.

Net sales in one query

With refunds in their own table keyed by order and carrying their own date, attributing them back to the original order day is a left join and a subtraction:

SELECT
    o.order_date,
    SUM(o.total_shop)                                AS gross_shop,
    SUM(COALESCE(r.refunded_shop, 0))                AS refunded_shop,
    SUM(o.total_shop - COALESCE(r.refunded_shop, 0)) AS net_shop
FROM shop_orders o
LEFT JOIN (
    SELECT order_id, SUM(amount_shop) AS refunded_shop
    FROM shop_refunds
    GROUP BY order_id
) r ON r.order_id = o.order_id
WHERE o.order_date BETWEEN '2025-01-01' AND '2025-01-31'
  AND o.is_test = false
  AND o.cancelled_at IS NULL
GROUP BY o.order_date
ORDER BY o.order_date;

Note the two filters doing quiet work at the bottom. Those two lines are the difference between a number finance accepts and a number they don’t.

Choosing the dashboard layer

Once the data is correct, this part is genuinely a preference. All of these work.

• Amazon QuickSight, now delivered as part of Amazon Quick Suite, is the least-friction option if you’re already in AWS. Its in-memory SPICE layer means viewers aren’t firing an Athena query per chart interaction, which controls both latency and scan cost. Per-viewer pricing tends to be the deciding factor either way, so model it for your actual audience size before committing.
• Grafana with the Athena data source is a good fit if you’re already running Grafana for infrastructure and want commercial and operational panels on one screen. Grafana Cloud removes the hosting question if you’d rather not run it.
• Power BI makes sense when the finance team already lives in Microsoft 365 and models in DAX. The cross-cloud hop is real but manageable.
• Metabase or a self-hosted alternative on a small VPS from a provider like InterServer or Hetzner is the pragmatic answer for a handful of internal viewers, where per-seat BI licensing costs more than the entire pipeline.

The honest trade-off: managed BI costs more per month and saves you from becoming the person who patches the reporting server. Self-hosting inverts that. Neither is wrong, but pick deliberately rather than by inertia.

Troubleshooting: symptom to cause

Totals are consistently higher than the Shopify admin

Almost always refunds, cancellations or test orders. Check in that order. If the gap grows with the age of the reporting window, it’s refunds. If it’s a fixed offset on specific days, look for test orders or a QA run.

Totals are lower, and recent days are missing rows

Either partitions aren’t registered, which projection fixes permanently, or the webhook subscription has been dropped after repeated delivery failures. Check the subscription still exists before you go digging through Lambda logs. A nightly bulk pull would have masked this, which is another argument for having one.

Orders appear twice

Shopify’s delivery model is at-least-once, not exactly-once, and you may also have more than one subscription on the same topic. Deduplicate on the delivery ID header before you touch anything else, and make the write itself idempotent so a duplicate is a no-op rather than a second row.

Numbers are right on the daily view, wrong on the monthly

Timezone. Order timestamps carry an offset; your partition key is a date string. If you derive the date in UTC and the store reports in a local timezone, orders near midnight land on the wrong day. That averages out over a month, which is exactly why the discrepancy hides until month boundaries.

Athena costs jumped without more data

Someone built a dashboard with a filter that doesn’t hit the partition column, so every panel refresh scans the full table. Look at bytes scanned per query and check whether the BI tool is caching results or re-querying on every interaction.

Common mistakes

• Treating orders as immutable events and never revisiting a past day.
• Partitioning on arrival time because that’s the default, then discovering corrections are expensive.
• Verifying the HMAC against a re-serialised body instead of the raw bytes.
• Doing real work inside the webhook handler, generating duplicates under load.
• Summing presentment amounts across currencies.
• Registering the event bus ARN with Shopify instead of the partner event source ARN.
• Relying on webhooks alone with no scheduled reconciliation.
• Shipping a “revenue” number without ever defining what it includes.

Best practices

• Keep raw and curated layers separate. Raw is append-only; curated is rewritable.
• Make every partition idempotently rebuildable from raw. Test that path deliberately, before you need it.
• Buffer through SQS with a dead-letter queue. Free replay, free isolation of bad payloads.
• Run a nightly bulk pull over a rolling window and overwrite those partitions.
• Use partition projection. It removes an entire category of silent failure.
• Alarm on the absence of events, not just on errors. A CloudWatch alarm on zero orders processed in an hour during business hours catches broken subscriptions the same day.
• Store the API secret in Secrets Manager and scope the Lambda role to the exact prefixes it writes.
• Publish a reconciliation panel comparing your total to the admin’s for the same window. Surfacing the gap builds more trust than hiding it.

Frequently asked questions

Do I need a data warehouse, or is S3 and Athena enough?

For a single store’s order data, S3 with Athena is almost certainly enough, and it’s cheaper because you pay per query rather than for a running cluster. Redshift starts to earn its place when you’re joining Shopify data against several other large sources, or when concurrent query load makes Athena’s queue times noticeable.

How near-real-time can this be?

Events land within seconds. The practical floor is your buffering window, since writing one tiny file per order gives you a small-files problem that ruins query performance. A few minutes of buffering is the usual compromise. If you genuinely need sub-minute order counts, put a live counter in DynamoDB alongside the analytical pipeline rather than trying to make the data lake do both jobs.

Which webhook topics should I subscribe to?

At minimum, order creation, order update, order cancellation and refund creation. Update and refund topics are the ones people skip, and they’re exactly the ones carrying the corrections. Subscribe to fewer topics than you think you need and add rather than subscribing to everything, since every extra topic is volume you pay to store and process.

Can I skip AWS and use a connector tool?

Yes, and for many stores that’s the right answer. A managed connector into a hosted warehouse gets you a working dashboard in an afternoon. You’re paying a monthly fee to avoid owning any of this, and trading away control over the data model. Building it on AWS wins when you need Shopify data joined to systems the connector doesn’t cover, or when row-based connector pricing outgrows the infrastructure cost.

How do I backfill historical orders?

Use a GraphQL bulk operation rather than paginating the API. Submit the query, poll for completion, then stream the JSONL result into your raw bucket and run the same transformation your live pipeline uses. If backfill and live processing use different code paths, they will diverge, and you’ll spend an afternoon working out which one is lying.

What does a setup like this cost to run?

For a typical single store, the pipeline itself is small money: Lambda invocations, a few gigabytes in S3, and Athena billed on bytes scanned, which partitioning and Parquet keep low. The BI seats are usually the largest line item, which is why the dashboard layer decision deserves more thought than the ingestion one. Model it against current published rates rather than trusting any figure you read in a blog post, including this one.

The one thing to take away

A Shopify sales dashboard with AWS doesn’t fail because the pipeline breaks. It fails because the pipeline keeps working perfectly on data that has since changed underneath it.

Design for correction from the first commit. Partition by order date, keep the raw layer so you can always rebuild, run a scheduled pull to catch what the stream missed, and agree on what revenue means before anyone builds a chart. Do that and the Monday morning screenshot comparison becomes a non-event, which is the highest praise a reporting pipeline ever gets.

Need help with your Shopify data pipeline on AWS?

I design and build ecommerce data pipelines and reporting stacks on AWS. Typical engagements look like:

• Working out why an existing Shopify dashboard disagrees with the admin, and fixing the root cause rather than patching the query
• Building the ingestion layer end to end: EventBridge or API Gateway, Lambda, SQS with dead-letter handling, and a scheduled GraphQL bulk reconciliation job
• Designing the S3 layout, Glue schema and Athena tables so past days can be recomputed cheaply and partitions never go missing
• Migrating REST Admin API integrations to GraphQL before the deadline forces the issue
• Building the dashboard itself in QuickSight, Grafana or Metabase, including the metric definitions finance will actually sign off on
• Cutting Athena scan costs and BI licensing on a reporting stack that has grown more expensive than anyone planned

If you’re in the middle of one of these, send me the actual thing: the Athena query, the S3 prefix layout, the two totals that don’t match. It’s a much faster conversation than describing it in the abstract.

The post Shopify Sales Dashboard with AWS: Build One That Actually Reconciles appeared first on John Nessime.

That is usually the moment someone says “let’s just pull it into a warehouse.” A Jira analytics pipeline built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.

This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.

Where Jira’s own reporting genuinely stops

Give the built-in tooling its due first. Jira’s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is “how did this sprint go,” you do not need a pipeline.

Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.

The shape of the pipeline

The architecture is unremarkable, which is the point:

1. EventBridge Scheduler triggers a Lambda function on a schedule.
2. Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.
3. It writes Parquet files to S3, partitioned by load date.
4. The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.
5. Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.

No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.

The complexity is not in the wiring. It is in four places, and they are worth taking in order.

Failure family 1: the search endpoint you were probably going to use is gone

Almost every Jira extraction tutorial and a good number of client libraries still reach for GET /rest/api/3/search. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.

The replacement is /rest/api/3/search/jql, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:

Pagination is cursor-based, and there is no total

startAt is gone. You get a nextPageToken back and hand it to the next request. There is no total in the response, which means any progress bar, any “expected N rows” sanity check, and any loop that terminated on startAt >= total needs rewriting.

If you only need a count, there is a separate operation, POST /rest/api/3/search/approximate-count, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.

There have been persistent community reports of isLast behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust isLast as your loop condition. Terminate on the absence of nextPageToken, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.

def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")

You have to ask for fields explicitly

The new endpoint does not hand you every field by default. Omit fields and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.

Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:

FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]

Custom field IDs are per-site. Do not hardcode one you read in someone else’s blog post. Pull /rest/api/3/field once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.

Rate limits and the fifteen-minute wall

Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a Retry-After header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.

Lambda’s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:

• Shard the work. Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.
• Checkpoint and continue. Persist the current nextPageToken to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.

Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda’s ephemeral storage: the default /tmp allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.

Failure family 2: the change history is the whole point, and it truncates silently

Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.

Here is the trap. Requesting an issue with expand=changelog returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.

Two correct approaches:

• GET /rest/api/3/issue/{issueIdOrKey}/changelog and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.
• POST /rest/api/3/changelog/bulkfetch, which accepts issueIdsOrKeys and an optional fieldIds filter, with the same nextPageToken pagination. Far fewer round trips.

Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter fieldIds to status if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.

Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.

Failure family 3: the incremental load that duplicates everything

This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.

The obvious incremental design is a watermark: track the last successful run, then pull updated >= watermark on each run and append the results to S3. It is the standard pattern and it works for immutable event data.

Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: SELECT count(*) FROM jira_issues WHERE status = 'Open' counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.

Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.

Pick one of three fixes, deliberately

Full snapshot per load, dedupe at read time. Keep every version, partition by load_date, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer “what did the board look like in March” without any extra machinery. The cost is that every query pays for the deduplication.

CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date >= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;

The load_date filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.

Overwrite the affected partitions. Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.

Use an ACID table format. Apache Iceberg gives you real MERGE INTO semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.

For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.

Deletions and moves

A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal fields list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.

Failure family 4: the S3 layout that decides your Athena bill

Athena’s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.

Three things do almost all the work:

• Write Parquet, not JSON. Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.
• Avoid tiny files. A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.
• Use partition projection. This is the one people skip and then wonder why queries have a fixed few-second overhead.

Partition projection lets Athena calculate partition locations from table properties instead of calling GetPartitions against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.

CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);

One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a load_date filter will probe empty prefixes. Keep the range tight to what you actually hold.

Finally, set BytesScannedCutoffPerQuery on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running SELECT * against three years of data in a BI tool’s preview pane is the classic surprise line item, and this stops it at the source.

Modelling time in status

Once status changelog rows are landed flat, the core metric is a window function. Each transition’s duration is the gap to the next transition on the same issue:

WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date >= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;

The COALESCE handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.

Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes “In Review” to “Peer Review”. A small mapping table joined at query time keeps history comparable.

Troubleshooting

• Every analytical column is null. You did not pass fields, or you passed a custom field ID from another site. Log the resolved field list on every run.
• Counts higher than the same JQL in Jira. Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.
• Extraction loops forever. Do not terminate on isLast. Terminate on missing nextPageToken and keep a page cap.
• HTTP 410 from the API. You are still calling the removed /rest/api/3/search. Check your client library version too, not just your own code.
• Cycle times suspiciously low. Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.
• Athena returns zero rows but the files are there. Partition projection range does not cover the partition, or the S3 prefix does not match storage.location.template. Compare a real object key against the template character by character.
• Queries slow down as the table grows. Either you have no projection and Glue lookups dominate, or you have accumulated small files.
• Authentication failures overnight with no deploy. API token expired or was revoked. Alert on the specific status code rather than on “run failed”.

Common mistakes

• Appending incremental extracts without a deduplication strategy.
• Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.
• Hardcoding custom field IDs copied from documentation.
• Storing the API token in a Lambda environment variable instead of Secrets Manager.
• Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.
• Letting BI tools query base tables directly instead of curated views.
• Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.

Best practices for a Jira analytics pipeline that survives

• Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.
• Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.
• Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.
• Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.
• Scope the Jira token to a service account with read access to exactly the projects you need.
• Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year’s raw JSON.
• Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena’s query result reuse for repeated dashboard loads.

Frequently asked questions

Should I use Lambda or Glue for Jira extraction?

Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark’s distributed compute does nothing for, and you would be paying for a Glue job’s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.

How often should the pipeline run?

Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.

Can I use webhooks instead of polling?

You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.

Does this work with Jira Data Center or Server?

The AWS half is identical. The extraction half is not. Cursor pagination and the /search/jql endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based /rest/api/2/search. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.

How much does an Athena-based Jira pipeline cost to run?

The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.

Why not just use eazyBI or a Jira reporting app?

Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.

Should I load into Redshift instead of querying S3?

Only if you have concurrent BI users hitting the same tables constantly and Athena’s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena’s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.

The one thing to carry away

The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.

Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.

Need this built or fixed?

I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:

• Migrating an extractor off the removed /rest/api/3/search endpoint to cursor-based /search/jql without losing rows in the cutover
• Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL
• Building changelog-based cycle time and time-in-status models that survive workflow renames
• Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs
• Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM
• Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables

Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.

The post Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything) appeared first on John Nessime.

The answer is in there. It is one email, or one line in a site diary, sitting among forty thousand other files. Nobody can read forty thousand files, so the instinct is to point a language model at the pile and ask it. That instinct is right. The naive implementation of it is where the money goes.

Here is the failure mode that bites hardest, and it is invisible until an expert challenges you on it. You build retrieval over the document set, ask about notice of delay, and the system confidently returns a letter that discusses the delay event in great detail. It is a good letter. It is also dated eleven months after the event, written by the claims consultant during preparation of the claim itself. It scored highest precisely because it was written to argue the point. The contemporaneous notice, the thing you actually needed, was four badly typed lines in a routine progress email that mentioned the word “delay” once.

Semantic similarity has no concept of a deadline. That single gap is the difference between an AI construction claims platform that shortens a disclosure exercise and one that quietly manufactures a wrong answer with a citation attached to it.

This post covers how to build that platform on AWS: how to lay out ingestion, how to make retrieval respect the contractual clock, why schedule data must never go anywhere near your vector index, how to keep privileged material out of the same retrieval path as project records, and which AWS building blocks are actually the current ones now that several of the obvious candidates have been moved to maintenance mode.

What a claims platform actually has to answer

Before any architecture, be honest about the question shapes. They are not all the same problem and they do not all get solved by retrieval.

1. Chronology. What happened, in what order, and on what date was it recorded? This is a retrieval and metadata problem.
2. Entitlement. Which clause applies, and what did it require the parties to do? This is retrieval over the contract plus careful prompting.
3. Causation. Which event moved the critical path, and by how much? This is schedule data and date arithmetic. It is not a language problem at all.
4. Quantum. What did the disruption cost? This is cost and resource data, joined to the events above.

Treat all four as “ask the documents” and you will get fluent nonsense on two of them. The architecture below splits them deliberately.

Failure one: retrieval that finds the best match instead of the first one

Two corpora live in every dispute bundle and they look identical to an embedding model.

• Contemporaneous records. Site diaries, progress emails, minutes, early warnings, RFIs, instructions. Written while the project was running, by people with no idea a dispute was coming.
• Claim-era material. Narratives, expert reports, without-prejudice correspondence, internal analysis. Written afterwards, specifically to be persuasive about the same events.

Claim-era material wins on cosine similarity almost every time, because it is denser in exactly the terms you searched for. If your retriever cannot distinguish them, every answer is contaminated by the argument you were trying to test.

The fix is metadata, applied at ingestion, and it is cheap to get right and expensive to retrofit. Amazon Bedrock Knowledge Bases reads a sidecar file that sits next to each document in S3, named with the full original filename plus .metadata.json. So letter-0421.pdf gets letter-0421.pdf.metadata.json. The naming convention is the only link between them; there is no separate registration step.

{
  "metadataAttributes": {
    "doc_date": 20240314,
    "corpus": "contemporaneous",
    "doc_type": "site_correspondence",
    "matter_id": "matter-0007",
    "date_source": "email_header",
    "privileged": false
  }
}

Look closely at doc_date. It is an integer, not a string, and that is not a style choice. Bedrock Knowledge Bases metadata attributes support STRING, NUMBER, BOOLEAN and STRING_LIST. The range comparison operators, the ones you need to express “on or before the notice deadline”, only apply to NUMBER. Store the date as "2024-03-14" and your filter will not throw an error. It will just quietly match nothing, or match everything, depending on how you wrote it. You will find out weeks later when someone asks why a document they can see in the bundle never appears in results.

With the date as a sortable integer, a query filter can express the contractual window directly.

{
  "andAll": [
    { "equals":              { "key": "corpus",   "value": "contemporaneous" } },
    { "equals":              { "key": "matter_id","value": "matter-0007" } },
    { "greaterThanOrEquals": { "key": "doc_date", "value": 20240301 } },
    { "lessThanOrEquals":    { "key": "doc_date", "value": 20240329 } }
  ]
}

That is the whole trick. You are no longer asking “what is the most relevant document about this delay”. You are asking “what did the parties actually write during the window in which the contract required them to write it”. Those are different questions and only one of them is worth anything in a dispute.

Where the date comes from matters more than the date

Do not use the S3 object timestamp. It records when someone copied a folder, usually years after the fact and identical across ten thousand files. Derive the date from the document itself: the Date: header on an email, the printed date on a letter, the period covered by a diary entry.

Sometimes you cannot, because the scanned undated fax exists in every project archive. Record that honestly with a date_source attribute rather than guessing, and treat unknown-date documents as a separate review pile. An extension of time argument built on an inferred date is an argument you will lose.

Failure two: treating the programme like a document

This one is worse, because the output looks right.

Oracle Primavera P6 exports XER and PMXML files. Asta Powerproject and Microsoft Project have their own formats. XER in particular is a plain text dump of relational tables, so it goes through a text pipeline without complaint. Chunk it, embed it, and you now have vectors representing fragments of a table of activity codes with no relationships attached.

Ask that index how much float activity A1200 had at the March data date and you will get a number. It will be well formatted and it will be invented. Total float is the product of a forward and backward pass across the whole logic network under a specific calendar. It cannot be recovered from a retrieved fragment, and a language model asked to produce it will produce something plausible instead of admitting that.

Schedule data goes into a structured store, and the model queries it rather than reasoning about it.

1. Parse each programme file into tables. PyP6Xer handles XER from Python; MPXJ is a Java library that reads XER, PMXML, Asta Powerproject and MSPDI among others, which matters when the bundle contains four scheduling tools.
2. Load activities, logic links, calendars, resource assignments and WBS into Amazon Aurora PostgreSQL for interactive work, or into S3 with AWS Glue and Amazon Athena when you have hundreds of updates and want columnar scans.
3. Stamp every row with the data date of the update it came from. This is the single most important column in the whole platform. Without it you have a pile of schedules; with it you have a time series of the project’s own view of itself.
4. Run windows analysis, as-planned versus as-built comparison and float erosion in SQL or Python, deterministically, so the same inputs always give the same numbers.
5. Expose the results to the model as a tool it can call, or as generated SQL against a defined schema. The model turns a question into a query and narrates the result. It does not do the arithmetic.

A rough shape of the query that makes float erosion visible:

SELECT
    a.activity_id,
    a.data_date,
    a.total_float_days,
    a.total_float_days - LAG(a.total_float_days)
        OVER (PARTITION BY a.activity_id ORDER BY a.data_date)
      AS float_change
FROM   schedule_activities a
WHERE  a.project_id = 'PRJ-01'
  AND  a.data_date BETWEEN DATE '2024-01-01' AND DATE '2024-06-30'
ORDER BY a.activity_id, a.data_date;

Nothing clever there, and that is the point. Every number is traceable to a row that came from a named XER file. When an opposing expert asks where a figure came from, the answer is a file name and a query, not “the model said so”.

Be realistic about effort here. Programme parsing and normalisation across inconsistent updates is the hardest part of the build and the part clients always underestimate. Activity IDs get reused, calendars change mid-project, and someone will have re-baselined without telling anyone. Budget for it.

Failure three: an answer with no paper trail

In most RAG applications a citation is a nice touch. In dispute work it is the product. An answer that cannot be traced to a page of a disclosed document is not evidence, it is a rumour with good grammar.

Design for that from the ingestion layer, not the presentation layer.

• Keep page and position. Amazon Bedrock Data Automation returns confidence scores and bounding box data alongside extracted fields, and Amazon Textract returns geometry per block. Carry both through the pipeline so a citation resolves to a page and a region, not just a file.
• Route low confidence to humans. Handwritten site diaries and faxed variation orders will produce low-confidence extractions. Those should land in a review queue by default rather than silently entering the index.
• Reject ungrounded answers. Amazon Bedrock Guardrails includes contextual grounding checks that score whether a response is supported by the retrieved passages. It reduces confident invention. It does not eliminate it, and anyone who tells you otherwise is selling something.
• Keep an immutable evidential copy. S3 Versioning plus S3 Object Lock on the landing bucket means the file the platform indexed is provably the file that was disclosed.

One design rule underpins all of it: the platform shortlists evidence, it does not decide entitlement. Recognised frameworks for this work, the Society of Construction Law Delay and Disruption Protocol and AACE International’s Recommended Practice 29R-03 on forensic schedule analysis, both assume a named analyst applying a stated method and exercising judgement. A system that outputs “the contractor is entitled to 42 days” is not helping. A system that outputs “here are the eleven contemporaneous documents inside the notice window, here is the float movement across those updates, here is what is missing” is doing real work.

Failure four: one index for privileged and non-privileged material

Dispute bundles contain legal advice, counsel’s opinions, without-prejudice correspondence and internal settlement analysis. Those must not be retrievable through the same path as project records.

The tempting shortcut is a privileged: false metadata filter on every query. Do not rely on that as your boundary. A metadata filter is a query parameter. One missing filter in one code path, one debug endpoint, one caching layer that drops it, and privileged material surfaces in a general search. The blast radius of that mistake is not a bug report.

Separate the indexes physically and separate the IAM roles that can reach them. Amazon S3 Vectors makes this practical: you can set a dedicated customer-managed KMS key per vector index, and you get a large number of indexes per vector bucket, so per-matter and per-sensitivity separation does not become an operational burden. Keep the metadata flag as well, because defence in depth is free, but make the identity boundary the one you actually trust.

Amazon Macie is worth pointing at the landing bucket to find personal data you did not expect, particularly in HR records and accident reports that get swept into project archives.

Choosing the AWS building blocks, including what not to build on

A lot of published architectures for this kind of platform are now pointing at services AWS has stopped developing. Two matter here, and the dates are the point.

• Amazon Kendra entered maintenance mode on 30 June 2026 and stops accepting new customers on 30 July 2026. Existing customers keep support and security fixes but no new capability. AWS directs new enterprise search and RAG work to Amazon Bedrock Knowledge Bases. If a tutorial or a proposal you are reading starts with a Kendra index, it predates that change.
• Amazon Bedrock Agents moved to maintenance mode in the same round of service availability changes, with Amazon Bedrock AgentCore as the successor for agentic orchestration. Check the current AWS service availability page before you commit an orchestration layer.

For the retrieval layer itself, Bedrock Knowledge Bases now comes in two shapes and the choice is a real trade-off rather than a marketing tier.

Managed Knowledge Base

AWS manages the vector store, embeddings model, re-ranker and retrieval orchestration as a single primitive, with native connectors for Amazon S3, SharePoint, Confluence, Google Drive, OneDrive and a web crawler, plus automatic parsing strategy selection and a retriever that decomposes multi-step queries. The connectors pull source permissions along with content, which matters when the document set lives in the client’s SharePoint rather than a bucket you control.

Where it wins: you get a working retrieval layer in an afternoon instead of a fortnight, and the parsing tuning that normally eats the first weeks of a build is done for you. For a first matter, or a proof of value before a client commits budget, this is the one I would reach for.

Custom Knowledge Base

You bring your own vector store and control chunking, embedding model and index layout.

Where it wins: claims work has awkward chunking requirements. A two-page letter split mid-sentence at a page boundary produces a chunk where the notice sentence has lost its date and its addressee. Controlling chunk boundaries around document structure, and controlling which index a document lands in, are both easier when you own the store. Where it doesn’t: you now own embedding model upgrades, re-indexing, sync failures and capacity, which is real ongoing work for a small team.

Start managed, build a retrieval evaluation set of real questions with known correct documents, and only move to custom when that set demonstrates the problem is chunking. Most teams migrate on a hunch and discover the problem was metadata all along.

Where the vector storage bill actually comes from

Rates change, so learn the billing mechanism rather than a number. Amazon S3 Vectors charges on three axes: upload volume by logical gigabyte, storage by logical gigabyte, and queries by data processed, where data processed scales with the size of the index being searched. Note that filtering does not reduce the data processed by a query.

That shape suits claims work unusually well. A dispute archive is enormous and cold: millions of chunks, queried by a handful of analysts a few hundred times a day, so you pay mostly for storage, which is the cheap axis. Compare that against Amazon OpenSearch Serverless, which prices on provisioned compute units and therefore rewards high query volume against a smaller index, or Aurora PostgreSQL with pgvector when you already need Aurora for the schedule tables and would rather run one system than two.

The practical lever is to split indexes per matter. Query cost scales with index size, so one giant index across every dispute you have ever run makes every query more expensive than it needs to be, on top of being a bad idea for confidentiality.

A reference pipeline

1. Everything lands in S3 under a per-matter prefix, with Versioning and Object Lock enabled on the evidential copy.
2. S3 event notifications trigger AWS Step Functions. Use Step Functions rather than a chain of Lambdas so that a failed extraction on page 300 of a 400-page bundle is visible and resumable.
3. Classify and split. Scanned bundles arrive as one PDF containing forty separate documents. Splitting them correctly is a prerequisite for dating them correctly.
4. Extract text with Amazon Bedrock Data Automation or Amazon Textract, keeping confidence scores and geometry.
5. Derive the document date and write the .metadata.json sidecar. Anything undated goes to the review queue.
6. Route by type: correspondence to the knowledge base, programme files to the XER parser and the relational store, cost data to its own tables.
7. Sync the knowledge base, then run your retrieval evaluation set before anyone uses it. A sync that succeeds is not the same as an index that answers correctly.
8. Serve through an API that refuses to return an answer without citations, and log every query with the filters that were applied.

Define the whole thing in Terraform or OpenTofu from the start. Matters are per-client and short-lived, and standing one up should be a variable file, not an afternoon in the console. Point Amazon CloudWatch, or Grafana Cloud if you already run Grafana elsewhere, at the Step Functions execution metrics so a silently failing extraction stage does not go unnoticed for a week.

Troubleshooting

Date filters return nothing, and no error

Almost always the date was stored as a string. Range operators need NUMBER. Convert to an integer in YYYYMMDD form and re-sync the affected documents.

A document is in the bucket but never appears in results

Check the sidecar filename first. It must be the complete original filename with .metadata.json appended, extension included. report.pdf.metadata.json works; report.metadata.json is a file the ingestion job will happily ignore. After that, check whether a filter in the query path is excluding it.

Answers cite the right document but the wrong passage

Chunking split the document somewhere structurally meaningful. Look at the raw chunks for that file. If the notice sentence and its date are in different chunks, no amount of prompt tuning fixes it. That is the signal to take control of chunking.

Float figures do not match the client’s own analysis

Check calendars before you check logic. Different activity calendars, a changed default calendar, or an update where someone applied a progress override will move float without any logic change. Reconcile activity counts between your parsed tables and the source file before trusting anything downstream.

Query costs jumped without more usage

An index grew. With storage-side vector search, query cost tracks the size of the index being scanned, so ingesting a large new bundle raises the price of every subsequent query against that index. Split by matter.

Common mistakes

• Using the file’s storage timestamp as the document date. It records the migration, not the event.
• Indexing claim narratives and contemporaneous records into the same corpus with no way to tell them apart.
• Embedding programme exports because they happen to be text files.
• Treating a metadata filter as a privilege boundary instead of an optimisation.
• Letting the model state entitlement conclusions rather than assembling and citing evidence.
• Building on services that have moved to maintenance mode because the tutorial you followed predates the change.
• Shipping without a retrieval evaluation set, so you have no way to know whether a change made things better or worse.
• One index for every matter, which is both a cost problem and a confidentiality problem.

Best practices

• Make the document date a first-class, numeric, filterable attribute, and record where it came from.
• Keep an immutable evidential copy separate from the working copy the pipeline mutates.
• Separate structured schedule and cost data from unstructured documents, and let the model query the former rather than reason about it.
• Build a retrieval evaluation set from real questions with known correct documents before you tune anything.
• Enforce citations at the API layer, so an uncited answer is impossible rather than discouraged.
• Isolate privileged material by index and by IAM role, with metadata as a second layer.
• Log every query with its filters, so you can reconstruct how any given answer was reached.
• Define infrastructure as code so a new matter is a deployment, not a project.

FAQ

Can an AI construction claims platform replace a delay expert?

No, and building toward that goal produces something unusable. Established forensic frameworks assume a named analyst applying a stated method whose reasoning can be tested. The platform’s value is compressing weeks of document review into hours and making the schedule data queryable, so the expert spends their time on judgement rather than searching.

Should I use Amazon Kendra for the search layer?

Not for a new build. Kendra entered maintenance mode on 30 June 2026 and closed to new customers on 30 July 2026, with AWS pointing to Bedrock Knowledge Bases for equivalent and more current capability. Existing Kendra deployments continue to be supported, so this is a migration assessment rather than an emergency, but starting there now means starting on a service with no roadmap.

How do I stop the model inventing float and delay figures?

Do not give it the chance. Keep schedule data in a relational or columnar store and have the model generate queries against a defined schema, or call a tool that runs a fixed calculation. The arithmetic happens in SQL or Python where it is deterministic and reproducible; the model only turns questions into queries and results into sentences.

Which vector store should I choose for a claims archive?

Match the store to your query pattern. Large, cold archives queried by a few analysts favour storage-priced options like Amazon S3 Vectors, where you mostly pay to keep the data. Smaller indexes hit constantly favour compute-priced options like Amazon OpenSearch Serverless. If you already run Aurora PostgreSQL for schedule data, pgvector alongside it is a legitimate way to avoid operating a second system.

How do I handle scanned and handwritten site records?

Extract them with confidence scores retained, set a threshold, and route everything below it to human review before indexing. Handwritten diaries are frequently the most probative documents in a delay claim and also the least reliable to read automatically, so the review queue is not an edge case. Plan capacity for it.

Where do documents come from if they are not already in S3?

Most project records live in a common data environment such as Procore, Autodesk Construction Cloud, Aconex or a client SharePoint tenancy. Bedrock Managed Knowledge Base has native connectors for SharePoint, Confluence, Google Drive and OneDrive that ingest permissions alongside content. For platforms without a native connector, export to S3 and keep the export manifest as part of the disclosure record.

The one thing worth remembering

An AI construction claims platform lives or dies on whether it understands time. Every hard requirement in this build traces back to that: numeric dates so you can filter to a contractual window, a data date on every schedule row so float movement is measurable, a corpus flag so contemporaneous records are not drowned out by material written to argue about them, and citations that resolve to a page so any answer can be checked.

Get the temporal metadata right at ingestion and the rest of the architecture is ordinary AWS work. Get it wrong and you have built a very expensive way to retrieve the most persuasive document instead of the true one.

Need help building this on AWS?

I design and build document and data platforms on AWS, and this kind of system sits squarely in that work. Things I can help with:

• Designing the ingestion pipeline: S3 landing zones with Object Lock, Step Functions orchestration, splitting and classifying scanned bundles, and confidence-based routing to human review.
• Getting the temporal metadata model right, including date derivation, sidecar generation and filter design against Amazon Bedrock Knowledge Bases.
• Parsing Primavera P6 XER and PMXML exports into queryable tables in Aurora PostgreSQL or S3 with Glue and Athena, with a data date on every row.
• Choosing and sizing the vector layer across Amazon S3 Vectors, OpenSearch Serverless and pgvector, based on your actual query pattern rather than a benchmark.
• Building index and IAM separation for privileged material, plus KMS key strategy and Macie scanning of landing buckets.
• Setting up retrieval evaluation, citation enforcement, query audit logging and CloudWatch or Grafana dashboards over the pipeline so failures surface early.

If you are partway into something like this already, send me a sample metadata sidecar, a Step Functions execution history, or a query that returns the wrong document, and I will tell you what I think is going on.

The post Building an AI Construction Claims Platform on AWS That Holds Up Under Scrutiny appeared first on John Nessime.