That is usually the moment someone says “let’s just pull it into a warehouse.” A Jira analytics pipeline built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.

This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.

Where Jira’s own reporting genuinely stops

Give the built-in tooling its due first. Jira’s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is “how did this sprint go,” you do not need a pipeline.

Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.

The shape of the pipeline

The architecture is unremarkable, which is the point:

1. EventBridge Scheduler triggers a Lambda function on a schedule.
2. Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.
3. It writes Parquet files to S3, partitioned by load date.
4. The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.
5. Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.

No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.

The complexity is not in the wiring. It is in four places, and they are worth taking in order.

Failure family 1: the search endpoint you were probably going to use is gone

Almost every Jira extraction tutorial and a good number of client libraries still reach for GET /rest/api/3/search. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.

The replacement is /rest/api/3/search/jql, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:

Pagination is cursor-based, and there is no total

startAt is gone. You get a nextPageToken back and hand it to the next request. There is no total in the response, which means any progress bar, any “expected N rows” sanity check, and any loop that terminated on startAt >= total needs rewriting.

If you only need a count, there is a separate operation, POST /rest/api/3/search/approximate-count, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.

There have been persistent community reports of isLast behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust isLast as your loop condition. Terminate on the absence of nextPageToken, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.

def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")

You have to ask for fields explicitly

The new endpoint does not hand you every field by default. Omit fields and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.

Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:

FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]

Custom field IDs are per-site. Do not hardcode one you read in someone else’s blog post. Pull /rest/api/3/field once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.

Rate limits and the fifteen-minute wall

Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a Retry-After header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.

Lambda’s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:

• Shard the work. Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.
• Checkpoint and continue. Persist the current nextPageToken to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.

Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda’s ephemeral storage: the default /tmp allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.

Failure family 2: the change history is the whole point, and it truncates silently

Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.

Here is the trap. Requesting an issue with expand=changelog returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.

Two correct approaches:

• GET /rest/api/3/issue/{issueIdOrKey}/changelog and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.
• POST /rest/api/3/changelog/bulkfetch, which accepts issueIdsOrKeys and an optional fieldIds filter, with the same nextPageToken pagination. Far fewer round trips.

Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter fieldIds to status if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.

Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.

Failure family 3: the incremental load that duplicates everything

This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.

The obvious incremental design is a watermark: track the last successful run, then pull updated >= watermark on each run and append the results to S3. It is the standard pattern and it works for immutable event data.

Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: SELECT count(*) FROM jira_issues WHERE status = 'Open' counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.

Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.

Pick one of three fixes, deliberately

Full snapshot per load, dedupe at read time. Keep every version, partition by load_date, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer “what did the board look like in March” without any extra machinery. The cost is that every query pays for the deduplication.

CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date >= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;

The load_date filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.

Overwrite the affected partitions. Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.

Use an ACID table format. Apache Iceberg gives you real MERGE INTO semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.

For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.

Deletions and moves

A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal fields list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.

Failure family 4: the S3 layout that decides your Athena bill

Athena’s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.

Three things do almost all the work:

• Write Parquet, not JSON. Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.
• Avoid tiny files. A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.
• Use partition projection. This is the one people skip and then wonder why queries have a fixed few-second overhead.

Partition projection lets Athena calculate partition locations from table properties instead of calling GetPartitions against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.

CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);

One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a load_date filter will probe empty prefixes. Keep the range tight to what you actually hold.

Finally, set BytesScannedCutoffPerQuery on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running SELECT * against three years of data in a BI tool’s preview pane is the classic surprise line item, and this stops it at the source.

Modelling time in status

Once status changelog rows are landed flat, the core metric is a window function. Each transition’s duration is the gap to the next transition on the same issue:

WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date >= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;

The COALESCE handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.

Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes “In Review” to “Peer Review”. A small mapping table joined at query time keeps history comparable.

Troubleshooting

• Every analytical column is null. You did not pass fields, or you passed a custom field ID from another site. Log the resolved field list on every run.
• Counts higher than the same JQL in Jira. Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.
• Extraction loops forever. Do not terminate on isLast. Terminate on missing nextPageToken and keep a page cap.
• HTTP 410 from the API. You are still calling the removed /rest/api/3/search. Check your client library version too, not just your own code.
• Cycle times suspiciously low. Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.
• Athena returns zero rows but the files are there. Partition projection range does not cover the partition, or the S3 prefix does not match storage.location.template. Compare a real object key against the template character by character.
• Queries slow down as the table grows. Either you have no projection and Glue lookups dominate, or you have accumulated small files.
• Authentication failures overnight with no deploy. API token expired or was revoked. Alert on the specific status code rather than on “run failed”.

Common mistakes

• Appending incremental extracts without a deduplication strategy.
• Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.
• Hardcoding custom field IDs copied from documentation.
• Storing the API token in a Lambda environment variable instead of Secrets Manager.
• Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.
• Letting BI tools query base tables directly instead of curated views.
• Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.

Best practices for a Jira analytics pipeline that survives

• Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.
• Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.
• Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.
• Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.
• Scope the Jira token to a service account with read access to exactly the projects you need.
• Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year’s raw JSON.
• Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena’s query result reuse for repeated dashboard loads.

Frequently asked questions

Should I use Lambda or Glue for Jira extraction?

Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark’s distributed compute does nothing for, and you would be paying for a Glue job’s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.

How often should the pipeline run?

Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.

Can I use webhooks instead of polling?

You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.

Does this work with Jira Data Center or Server?

The AWS half is identical. The extraction half is not. Cursor pagination and the /search/jql endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based /rest/api/2/search. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.

How much does an Athena-based Jira pipeline cost to run?

The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.

Why not just use eazyBI or a Jira reporting app?

Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.

Should I load into Redshift instead of querying S3?

Only if you have concurrent BI users hitting the same tables constantly and Athena’s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena’s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.

The one thing to carry away

The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.

Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.

Need this built or fixed?

I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:

• Migrating an extractor off the removed /rest/api/3/search endpoint to cursor-based /search/jql without losing rows in the cutover
• Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL
• Building changelog-based cycle time and time-in-status models that survive workflow renames
• Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs
• Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM
• Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables

Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.

The post Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything) appeared first on John Nessime.