<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Engineering Metrics | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/engineering-metrics/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/engineering-metrics/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Thu, 29 Oct 2026 06:58:01 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Engineering Metrics | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/engineering-metrics/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go</title>
		<link>https://john-nessime.com/blog/devops/ci-cd-pipeline-bottlenecks/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 01 Oct 2026 06:02:12 +0000</pubDate>
				<category><![CDATA[Continuous Delivery]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Troubleshooting]]></category>
		<category><![CDATA[Actions Runner Controller]]></category>
		<category><![CDATA[Build Cache]]></category>
		<category><![CDATA[BuildKit]]></category>
		<category><![CDATA[Cache Mounts]]></category>
		<category><![CDATA[CI/CD]]></category>
		<category><![CDATA[Engineering Metrics]]></category>
		<category><![CDATA[Flaky Tests]]></category>
		<category><![CDATA[GitHub Actions]]></category>
		<category><![CDATA[GitLab CI]]></category>
		<category><![CDATA[Multi-Stage Builds]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Platform Engineering]]></category>
		<category><![CDATA[Queue Time]]></category>
		<category><![CDATA[Required Status Checks]]></category>
		<category><![CDATA[Test Parallelization]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=706</guid>

					<description><![CDATA[<p>The build duration graph looks fine and shipping is still slow. Most CI/CD pipeline bottlenecks are waiting, not working: queue time, caches that report a hit and deliver nothing, stage ordering nobody revisited, and required checks stuck at pending. Here is how to find yours.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/ci-cd-pipeline-bottlenecks/">CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The pipeline takes eleven minutes. The pull request took two days to merge.</p>



<p class="wp-block-paragraph">That gap is where the real cost lives, and it is almost never on anyone&#8217;s dashboard. The build duration graph looks fine. The team still says shipping is slow. Somebody proposes buying bigger runners, and six weeks later the graph has moved a little and the complaints are identical.</p>



<p class="wp-block-paragraph">Most CI/CD pipeline bottlenecks are not the thing people go looking for. Compute is usually the smallest slice. The expensive parts are minutes spent waiting for a runner, a cache that reports a hit and delivers nothing, a stage blocking four jobs that do not depend on it, and a required check sitting at &#8220;Pending&#8221; until someone notices. This post is organized by those failure families rather than by tool. Examples use GitHub Actions and GitLab CI, but the shapes apply to Jenkins, Buildkite, CircleCI and Argo Workflows too.</p>



<h2 class="wp-block-heading">First, split the number you are staring at</h2>



<p class="wp-block-paragraph">&#8220;The build takes eleven minutes&#8221; is not a measurement. It is three measurements added together and then thrown away:</p>



<ul class="wp-block-list">
<li><strong>Queue time.</strong> From job creation to a runner picking it up. You pay this in wall clock, not compute.</li>

<li><strong>Compute time.</strong> The job actually running. The only part most dashboards show.</li>

<li><strong>Gate time.</strong> Waiting on a human, an approval, a shared environment, or a check that never fired.</li>
</ul>



<p class="wp-block-paragraph">Until you split those, every optimization is a guess. On GitHub Actions the jobs API hands you the split directly, because each job carries a creation timestamp and a start timestamp:</p>



<pre class="wp-block-code"><code>gh api repos/OWNER/REPO/actions/runs/RUN_ID/jobs 
  --jq '.jobs[] | {name, created_at, started_at, completed_at}'</code></pre>



<p class="wp-block-paragraph">Creation to start is queue. Start to completion is compute. Run that across your last fifty runs on the default branch and the answer is usually sitting there in plain sight. GitLab reports the same idea on its jobs API, with a queued duration alongside the run duration. If the two numbers are roughly equal, you have a capacity problem rather than a build problem, and faster CPUs will fix half of nothing.</p>



<h2 class="wp-block-heading">Queue time: the minutes you pay for twice</h2>



<p class="wp-block-paragraph">Queue time has two causes and they need different fixes.</p>



<h3 class="wp-block-heading">Not enough runners</h3>



<p class="wp-block-paragraph">On hosted runners you are bounded by the concurrency your plan allows. On self-hosted you are bounded by whatever fleet you provisioned, which for most small teams is a fixed number of always-on boxes that sit idle overnight and saturate at 4pm.</p>



<p class="wp-block-paragraph">If you run your own fleet on Kubernetes, Actions Runner Controller is where most people land. It creates runner scale sets that scale on workflow demand and can drop to zero when the queue is empty, with ephemeral pods so every job starts clean. That last part matters more than the scaling: a persistent runner accumulates state, and state is where &#8220;works on my machine&#8221; comes back as a CI flake.</p>



<p class="wp-block-paragraph">If Kubernetes is more machinery than you want, a couple of dedicated boxes from a provider like Contabo or InterServer running the runner agent under systemd gets you a long way, with hosted runners as overflow when the queue backs up. Managed runner services such as Depot and BuildJet sell faster hardware behind the same interface. Their honest pitch is that you stop maintaining a fleet. The honest counterpoint is another vendor in the path of every merge, and a bill that scales with commit volume.</p>



<h3 class="wp-block-heading">Runners busy on work that is already obsolete</h3>



<p class="wp-block-paragraph">Check this one first because it is free. Push three commits to a branch in five minutes and by default you get three pipelines. The first two are answering a question nobody is asking anymore, and they are holding the runners the third one needs.</p>



<p class="wp-block-paragraph">On GitHub Actions, a concurrency group keyed on the ref does it:</p>



<pre class="wp-block-code"><code>concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true</code></pre>



<p class="wp-block-paragraph">Put that at workflow level on PR builds. Do not put it on a deploy workflow with cancellation enabled, or you will eventually kill a release halfway through and spend an afternoon working out what actually shipped.</p>



<p class="wp-block-paragraph">GitLab splits the same job across two settings that have to agree. The <code>interruptible</code> keyword marks a job as safe to kill and defaults to false. The workflow-level auto-cancel setting decides what happens on a new commit:</p>



<pre class="wp-block-code"><code>workflow:
  auto_cancel:
    on_new_commit: interruptible

test:
  interruptible: true
  script:
    - make test

deploy:
  interruptible: false
  script:
    - make deploy</code></pre>



<p class="wp-block-paragraph">The default mode is conservative, which means once any non-interruptible job has started, the whole pipeline stops being cancelable. Setting it to <code>interruptible</code> cancels only the jobs that opted in and leaves the rest running. That distinction is the difference between a policy that works and one that quietly does nothing, and it is the usual reason teams turn auto-cancel on and see no change.</p>



<h2 class="wp-block-heading">Caches that look like they are working</h2>



<p class="wp-block-paragraph">My favorite family of CI/CD pipeline bottlenecks, because the failure is completely silent. The step is green. The log says the cache was restored. The install takes exactly as long as before.</p>



<h3 class="wp-block-heading">Branch scoping</h3>



<p class="wp-block-paragraph">GitHub Actions caches are scoped to the branch that created them, plus the default branch as a fallback. A feature branch reads its own caches and the default branch&#8217;s caches. It cannot read a sibling branch&#8217;s cache.</p>



<p class="wp-block-paragraph">The consequence catches a lot of people: if your caching step only runs on pull requests, the default branch never populates a cache, so every new branch starts cold forever. Make sure at least one workflow on the default branch writes it. Runs triggered from fork pull requests get read-only cache access, so those jobs restore but never save.</p>



<h3 class="wp-block-heading">Keys that never hit, or hit and never update</h3>



<p class="wp-block-paragraph">Cache entries are immutable. Once a key exists, writing to it again does nothing. So a key like <code>node-modules-v1</code> gets populated once, on some machine, in some state, and then freezes. A key that hashes the lockfile is correct but creates a brand new entry on every dependency change, which burns through storage.</p>



<p class="wp-block-paragraph">The pattern that survives contact is an exact key plus a prefix fallback: hash the lockfile for the primary key, then use restore keys to fall back to the most recent entry sharing that prefix. You get a partial restore and a fast install instead of a cold one.</p>



<p class="wp-block-paragraph">Then watch the ceiling. Repositories have a default cache storage limit, entries untouched for a week are removed, and once you hit the limit the least recently accessed entries get evicted. Push enough large caches and you get thrashing, where entries are created and evicted at roughly the rate you create them. The tell is a hit rate that looks random. Check the repository&#8217;s cache usage in Actions settings before assuming the config is wrong.</p>



<h3 class="wp-block-heading">Docker layer cache that does not cross machines</h3>



<p class="wp-block-paragraph">On ephemeral runners the local BuildKit cache dies with the machine, so it has to be exported somewhere. Registry-backed cache is the one I reach for first:</p>



<pre class="wp-block-code"><code>docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:${GIT_SHA} 
  --push .</code></pre>



<p class="wp-block-paragraph">Two details do the work. <code>mode=max</code> exports intermediate stage layers, not just the ones that landed in the final image. If the expensive work happens in a builder stage that gets discarded, the default mode caches none of it, which is exactly the case you were trying to fix. Inline cache, the kind that ships alongside the image, only supports the minimal mode, so it cannot help multi-stage builds either.</p>



<p class="wp-block-paragraph">The second detail is a trap. BuildKit cache mounts, the <code>RUN --mount=type=cache</code> kind, are not part of the exported layer cache. They live on the builder. Restoring a registry cache does not restore them, so if the instruction runs again the package manager starts from an empty download cache:</p>



<pre class="wp-block-code"><code># syntax=docker/dockerfile:1
FROM node:22-alpine
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci</code></pre>



<p class="wp-block-paragraph">That line is excellent on a persistent builder and close to useless on a runner destroyed after every job. Know which one you have before you credit it with a speedup.</p>



<h2 class="wp-block-heading">Pipelines shaped like a staircase</h2>



<p class="wp-block-paragraph">Stages are a scheduling constraint most pipelines inherit by accident. In a stage-based model nothing in the test stage starts until everything in the build stage finishes. Put a forty-second lint job in the same stage as a six-minute compile and your lint feedback arrives six minutes late for no reason.</p>



<p class="wp-block-paragraph">GitLab&#8217;s fix is <code>needs</code>, which lets a job start as soon as its own dependencies finish and ignore stage order entirely:</p>



<pre class="wp-block-code"><code>lint:
  stage: test
  needs: []
  script:
    - make lint

unit:
  stage: test
  needs: ["build"]
  script:
    - make unit</code></pre>



<p class="wp-block-paragraph"><code>needs: []</code> means start immediately, wait for nothing. Lint now reports in under a minute while the compile is still running.</p>



<p class="wp-block-paragraph">GitHub Actions has the opposite default: jobs run in parallel unless you add <code>needs</code>. So the equivalent problem there is over-declared dependencies, usually added early to force a readable graph and never revisited. Go read your <code>needs</code> lines and ask which are real data dependencies and which are narrative ordering.</p>



<p class="wp-block-paragraph">The trade-off is real. A wide graph surfaces every failure at once and costs more compute per run. A narrow graph fails fast and saves minutes, but developers make several round trips to find three problems. For PR pipelines, wide is usually right. For expensive nightly jobs, narrow is.</p>



<h2 class="wp-block-heading">The test suite is not one bottleneck</h2>



<p class="wp-block-paragraph">&#8220;Tests are slow&#8221; gets treated as a single problem. It is usually three, and they want different fixes.</p>



<ul class="wp-block-list">
<li><strong>The suite is serial.</strong> Split by shard across parallel jobs. The cost is per-job setup overhead paid N times, so past a point more shards makes things slower. Find your own break-even instead of assuming more is better.</li>

<li><strong>The suite is flaky.</strong> A retry costs more than one rerun. It costs attention, a context switch, and eventually the team&#8217;s belief that red means anything. Quarantine flaky tests into a non-blocking job with an owner and a deadline. Leaving them in the required path while everyone knows to just re-run is worse than deleting them.</li>

<li><strong>The suite tests the wrong layer.</strong> A test that boots a database and a message broker to assert a validation rule is a unit test wearing a costume. Slowest fix, largest payoff.</li>
</ul>



<p class="wp-block-paragraph">Before touching any of it, get per-test timing out of your test runner and sort descending. Most suites have a handful of tests that dominate, and they are rarely the ones people suspect.</p>



<h2 class="wp-block-heading">The wait that nobody is measuring</h2>



<p class="wp-block-paragraph">Here is the one that costs days rather than minutes, and it appears in no CI metric because from CI&#8217;s point of view nothing is happening.</p>



<p class="wp-block-paragraph">If a workflow is skipped because of path filtering, branch filtering, or a skip directive in the commit message, its checks stay pending. Not skipped. Pending. And if that check is required, the pull request cannot merge. Nothing fails, nothing alerts, the PR just sits there looking almost ready until someone investigates by hand.</p>



<p class="wp-block-paragraph">This bites hardest in monorepos, which is exactly where path filters are most attractive. You add <code>paths</code> to skip the service test suite on documentation-only changes, you save real minutes, and you accidentally build a merge trap.</p>



<p class="wp-block-paragraph">The distinction that matters: a job skipped by an <code>if</code> condition inside a workflow that did run reports success. A workflow that never ran at all reports pending. So either keep the workflow triggering on everything and move the filtering inside, or pair each filtered workflow with a companion using the same name and inverted path filters that does nothing and passes. GitHub documents that second pattern and it works, but it means two files that must stay in sync.</p>



<p class="wp-block-paragraph">The other gate-time offenders are less subtle and just as expensive: one shared staging environment serializing every team, and approvals routed to a single person asleep in another time zone. Plot merge timestamps against last-commit timestamps for a month. The shape of that distribution tells you which problem you have.</p>



<h2 class="wp-block-heading">Troubleshooting: finding your bottleneck in an afternoon</h2>



<ol class="wp-block-list">
<li>Pull the last fifty runs on your default branch and compute queue time and compute time separately. If queue is a meaningful share, fix capacity and cancellation before anything else.</li>

<li>Read your slowest run as a graph, not a list. Wide gaps between a dependency finishing and its dependent starting are either queue or bad ordering.</li>

<li>Compare install duration on a cache hit against a cache miss. If they are similar, the cache is decorative.</li>

<li>Check repository cache storage usage. Thrashing looks like an erratic hit rate with no config changes behind it.</li>

<li>Count retries. The rate of second attempts on the same commit is your flakiness tax, and it is usually higher than anyone guesses.</li>

<li>Measure merge latency, last commit to merge, across a month. The difference between that and pipeline duration is gate time, and that is where the days are.</li>
</ol>



<p class="wp-block-paragraph">Pipeline run data is worth shipping somewhere queryable. Most CI platforms emit webhooks on job completion; feeding those into Prometheus and Grafana, or a hosted platform like Grafana Cloud or Datadog if you would rather not run the storage, turns &#8220;the build feels slow lately&#8221; into a graph you can argue with.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Buying bigger runners before splitting queue time from compute time. The most common wasted spend in CI.</li>

<li>Turning on auto-cancel without setting the per-job interruptible flags, so the setting is technically on and functionally off.</li>

<li>Caching a directory the tool does not read from, then declaring victory because the step went green.</li>

<li>Using the default Docker cache export mode on a multi-stage build and wondering why the builder stage rebuilds every time.</li>

<li>Adding path filters to workflows that back required checks, creating pull requests that can never merge.</li>

<li>Optimizing the pipeline that runs on merge instead of the PR pipeline developers actually wait on.</li>
</ul>



<h2 class="wp-block-heading">Practices that hold up</h2>



<ul class="wp-block-list">
<li>Treat queue time as a first-class metric with its own threshold, not a footnote on build duration.</li>

<li>Cancel superseded runs on branches, never on the default branch and never on deploys.</li>

<li>Make the default branch populate every cache other branches depend on.</li>

<li>Order jobs by how fast they fail, not by how the file reads top to bottom. Cheap checks first, always unblocked.</li>

<li>Give every flaky test an owner and an expiry date, or take it out of the blocking path.</li>

<li>Re-measure after every change and keep the number. Pipelines rot, and the bottleneck moves as soon as you fix the current one.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How do I find the bottleneck in my CI/CD pipeline?</h3>



<p class="wp-block-paragraph">Split every job into queue time and compute time using the timestamps your platform&#8217;s API already exposes, then compare pipeline duration against real merge latency. Those two comparisons put you in one of three buckets: capacity, build work, or gates. Fixing the wrong bucket is why most optimization efforts show no result.</p>



<h3 class="wp-block-heading">Are self-hosted runners faster than hosted ones?</h3>



<p class="wp-block-paragraph">Per job, often yes, because you pick the hardware and can keep caches warm. Across a working day, frequently no, because a fixed fleet queues under load while a hosted pool does not. The right answer depends on which number is hurting you, compute or queue, which is why measurement comes first. A hybrid setup, self-hosted by default with hosted overflow, avoids committing to either.</p>



<h3 class="wp-block-heading">Why is my cache not speeding anything up?</h3>



<p class="wp-block-paragraph">Usual suspects in order of frequency: the cache is scoped to a branch that never populated it, the key is static so it froze at whatever went in first, the cached directory is not the one the tool reads, or entries are evicted before the next run. Compare install duration on a hit against a miss. Identical duration means the restore is doing nothing regardless of what the log says.</p>



<h3 class="wp-block-heading">Should I parallelize tests or make them faster?</h3>



<p class="wp-block-paragraph">Parallelize first because it is cheap and reversible, then fix the slow tests, because parallelism has a floor. Every shard pays setup cost, so a suite split too many ways spends more time booting containers than running assertions. Sharding buys time to do the real work. It is not the real work.</p>



<h3 class="wp-block-heading">Why does my pull request say pending forever?</h3>



<p class="wp-block-paragraph">Almost certainly a required status check attached to a workflow that was skipped by path filtering, branch filtering, or a commit message directive. A skipped workflow reports pending rather than skipped, so the merge requirement is never satisfied. Either drop the path filter and skip inside the workflow instead, or add a companion workflow with the same name that always passes.</p>



<h3 class="wp-block-heading">Do bigger runners actually help?</h3>



<p class="wp-block-paragraph">They help work that is genuinely CPU or memory bound, such as compilation and container builds. They do nothing for queue time, network-bound dependency downloads, tests waiting on external services, or gate time. Before upgrading, confirm the slow job is saturating the resource you are about to buy more of.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Most CI/CD pipeline bottlenecks are waiting, not working. Queue time, cache misses that report as hits, jobs blocked behind stages they do not depend on, and checks that sit pending because a workflow was never triggered. None of them show up on a build duration graph, and all of them are cheap to fix once you can see them. Split the number before you spend anything; the measurement takes an afternoon and usually points at a config change rather than a purchase order.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need someone to find where your pipeline is losing time?</h2>



<p class="wp-block-paragraph">This is a large part of what I do as a freelance DevOps engineer. Typical work looks like:</p>



<ul class="wp-block-list">
<li>Instrumenting your pipeline so queue time, compute time and merge latency become three separate numbers you can watch over time</li>

<li>Auditing GitHub Actions and GitLab CI caching to find the restores that report success and deliver nothing</li>

<li>Restructuring stage-based pipelines into dependency graphs so cheap checks report first</li>

<li>Setting up autoscaling self-hosted runners, on Kubernetes with Actions Runner Controller or on plain VMs, with hosted overflow</li>

<li>Fixing Docker build caching for multi-stage images on ephemeral runners, including registry-backed export</li>

<li>Untangling required checks and path filters in monorepos so pull requests stop getting stuck</li>
</ul>



<p class="wp-block-paragraph">Send me a workflow file, a job timeline, or the output of your slowest run, and I will tell you where the minutes are going before we talk about anything else.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/ci-cd-pipeline-bottlenecks/">CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</title>
		<link>https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 04 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Athena]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Glue]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Business Intelligence]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Engineering Metrics]]></category>
		<category><![CDATA[ETL]]></category>
		<category><![CDATA[Jira]]></category>
		<category><![CDATA[Partition Projection]]></category>
		<category><![CDATA[Pipeline Design]]></category>
		<category><![CDATA[Python]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[SQL]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=125</guid>

					<description><![CDATA[<p>Jira's built-in reports stop at the board boundary. This guide walks through a Jira analytics pipeline built on AWS Lambda, S3 and Athena, organised around the four failure families that actually bite: the removed search endpoint, silently truncated changelogs, incremental loads that duplicate rows, and an S3 layout that quietly inflates your query bill.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">Someone in a delivery review asks why cycle time went up last quarter. You open the Jira dashboard, and it can tell you what is in progress right now, roughly. It cannot tell you what &#8220;in progress&#8221; meant three months ago, how long each ticket sat in code review, or whether the increase came from one team or all six. The control chart resets when the board configuration changes, the sprint report only knows about sprints, and the CSV export tops out somewhere unhelpful.</p>



<p class="wp-block-paragraph">That is usually the moment someone says &#8220;let&#8217;s just pull it into a warehouse.&#8221; A <strong>Jira analytics pipeline</strong> built on AWS Lambda, S3 and Athena is a reasonable answer to that, and it is genuinely cheap to run. It is also easy to build a version that looks correct for two weeks and then quietly reports numbers that are thirty percent wrong.</p>



<p class="wp-block-paragraph">This post covers the extraction and modelling problems that actually cost you time: the search endpoint Atlassian removed, the change history that truncates without erroring, the incremental load pattern that duplicates rows across partitions, and the S3 layout decisions that decide whether Athena costs you pennies or hundreds. Code where it clarifies something, and honest notes on what I would skip.</p>



<h2 class="wp-block-heading">Where Jira&#8217;s own reporting genuinely stops</h2>



<p class="wp-block-paragraph">Give the built-in tooling its due first. Jira&#8217;s velocity, burndown and control charts are fine for a single team inspecting its own recent work, they need no infrastructure, and they update instantly. Marketplace apps like eazyBI and Custom Charts cover a lot of ground without you writing a line of Python. If your question is &#8220;how did this sprint go,&#8221; you do not need a pipeline.</p>



<p class="wp-block-paragraph">Where it stops is anything that crosses a boundary. Comparing lead time across projects that use different workflows. Joining ticket data to deploy events from your CI system or incident data from PagerDuty. Retaining a consistent view of history after someone renames a status or archives a board. Answering a question nobody anticipated when the board was configured. Those need the raw data somewhere you control, in a shape you decide.</p>



<h2 class="wp-block-heading">The shape of the pipeline</h2>



<p class="wp-block-paragraph">The architecture is unremarkable, which is the point:</p>



<ol class="wp-block-list">
<li>EventBridge Scheduler triggers a Lambda function on a schedule.</li>

<li>Lambda reads a Jira API token from Secrets Manager and pages through the Jira Cloud REST API.</li>

<li>It writes Parquet files to S3, partitioned by load date.</li>

<li>The Glue Data Catalog holds the table definitions, with partition projection so nothing has to crawl.</li>

<li>Athena queries S3 directly. Grafana, Power BI, Metabase or QuickSight sit on top of Athena.</li>
</ol>



<p class="wp-block-paragraph">No cluster, no always-on database, nothing to patch. The whole thing costs about as much as a small EC2 instance for a mid-sized Jira site, and most of that is S3 storage. Terraform or CloudFormation to define it, GitHub Actions to deploy it.</p>



<p class="wp-block-paragraph">The complexity is not in the wiring. It is in four places, and they are worth taking in order.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 1: the search endpoint you were probably going to use is gone</h2>



<p class="wp-block-paragraph">Almost every Jira extraction tutorial and a good number of client libraries still reach for <code>GET /rest/api/3/search</code>. Atlassian removed it from Jira Cloud. It returns 410 Gone. If you are copying a script from a blog post that predates the change, this is the first thing that breaks, and at least it breaks loudly.</p>



<p class="wp-block-paragraph">The replacement is <code>/rest/api/3/search/jql</code>, available as both GET and POST. Use POST for anything real, because JQL strings get long and you avoid URL encoding entirely. Three behavioural changes matter more than the URL:</p>



<h3 class="wp-block-heading">Pagination is cursor-based, and there is no total</h3>



<p class="wp-block-paragraph"><code>startAt</code> is gone. You get a <code>nextPageToken</code> back and hand it to the next request. There is no <code>total</code> in the response, which means any progress bar, any &#8220;expected N rows&#8221; sanity check, and any loop that terminated on <code>startAt &gt;= total</code> needs rewriting.</p>



<p class="wp-block-paragraph">If you only need a count, there is a separate operation, <code>POST /rest/api/3/search/approximate-count</code>, which takes a JQL body and returns an approximate figure without paging through results. It is genuinely useful as a reconciliation check: run it before extraction, compare against the row count you actually wrote, and alert on a large gap.</p>



<p class="wp-block-paragraph">There have been persistent community reports of <code>isLast</code> behaving unreliably on this endpoint, including tokens that chain without ever terminating. Do not trust <code>isLast</code> as your loop condition. Terminate on the absence of <code>nextPageToken</code>, and put a hard page cap in as a circuit breaker so a bad token cannot burn your entire Lambda budget in one invocation.</p>



<pre class="wp-block-code"><code>def search_issues(session, base_url, jql, fields, max_pages=2000):
    """Page through /search/jql. Terminates on missing nextPageToken,
    not on isLast, which has been reported as unreliable."""
    token = None
    for _ in range(max_pages):
        body = {"jql": jql, "fields": fields, "maxResults": 100}
        if token:
            body["nextPageToken"] = token
        r = session.post(f"{base_url}/rest/api/3/search/jql",
                         json=body, timeout=60)
        r.raise_for_status()
        page = r.json()
        for issue in page.get("issues", []):
            yield issue
        token = page.get("nextPageToken")
        if not token:
            return
    raise RuntimeError("page cap hit, refusing to loop further")</code></pre>



<h3 class="wp-block-heading">You have to ask for fields explicitly</h3>



<p class="wp-block-paragraph">The new endpoint does not hand you every field by default. Omit <code>fields</code> and you get essentially nothing back. This is the failure that looks like success: the pipeline runs, files land in S3, row counts look plausible, and every analytical column is null.</p>



<p class="wp-block-paragraph">Be explicit and be narrow. Every field you request costs response size, and Jira sites accumulate hundreds of custom fields nobody uses. Name what you need:</p>



<pre class="wp-block-code"><code>FIELDS = [
    "summary", "status", "issuetype", "project", "priority",
    "assignee", "reporter", "created", "updated", "resolutiondate",
    "labels", "components", "parent",
    "customfield_10016",   # story points on this site, verify yours
]</code></pre>



<p class="wp-block-paragraph">Custom field IDs are per-site. Do not hardcode one you read in someone else&#8217;s blog post. Pull <code>/rest/api/3/field</code> once, find the field by name, and either store the mapping in config or resolve it at runtime and log what it resolved to. When a Jira admin rebuilds a field, an ID-based pipeline goes null and a name-resolving pipeline keeps working.</p>



<h3 class="wp-block-heading">Rate limits and the fifteen-minute wall</h3>



<p class="wp-block-paragraph">Jira Cloud applies cost-based rate limiting and returns HTTP 429 when you exceed it, typically with a <code>Retry-After</code> header. Respect that header rather than inventing your own backoff. A naive retry loop that ignores it turns a brief throttle into a sustained one.</p>



<p class="wp-block-paragraph">Lambda&#8217;s hard ceiling is fifteen minutes. A full historical backfill of a large Jira site will not finish in one invocation, and the ugly failure mode is a function that times out at minute fifteen having written half its data with no record of where it stopped. Two ways out:</p>



<ul class="wp-block-list">
<li><strong>Shard the work.</strong> Fan out one Lambda invocation per project key, or per month of created date. Each one is small, independently retryable, and finishes well inside the limit.</li>

<li><strong>Checkpoint and continue.</strong> Persist the current <code>nextPageToken</code> to DynamoDB or S3 after each page. When the function is close to its deadline, stop cleanly and let Step Functions re-invoke it from the checkpoint.</li>
</ul>



<p class="wp-block-paragraph">Sharding is simpler and I reach for it first. Checkpointing is what you need when a single project is itself too large. Either way, watch Lambda&#8217;s ephemeral storage: the default <code>/tmp</code> allocation is 512 MB, and buffering a large Parquet write there will fail before your API calls do. Raise it or stream to S3 in chunks.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 2: the change history is the whole point, and it truncates silently</h2>



<p class="wp-block-paragraph">Current issue state answers almost none of the interesting questions. How long a ticket spent waiting for review, how many times it bounced back from QA, when it actually entered development rather than when someone remembered to drag the card, all of that lives in the changelog. Without it you have a list of tickets. With it you have a process.</p>



<p class="wp-block-paragraph">Here is the trap. Requesting an issue with <code>expand=changelog</code> returns a capped number of history entries, commonly the first hundred, and it does not tell you it truncated. Well-worn tickets with lots of field edits blow past that easily. Your data does not error, it just quietly loses the later transitions, which are usually the ones near completion. Cycle time comes out looking better than reality.</p>



<p class="wp-block-paragraph">Two correct approaches:</p>



<ul class="wp-block-list">
<li><code>GET /rest/api/3/issue/{issueIdOrKey}/changelog</code> and page it properly. Correct, but it is one request per issue, which is brutal against rate limits on a large site.</li>

<li><code>POST /rest/api/3/changelog/bulkfetch</code>, which accepts <code>issueIdsOrKeys</code> and an optional <code>fieldIds</code> filter, with the same <code>nextPageToken</code> pagination. Far fewer round trips.</li>
</ul>



<p class="wp-block-paragraph">Bulk fetch is the one I would use, with a caveat: it has carried an experimental designation, so pin your expectations and keep the per-issue path available as a fallback. Filter <code>fieldIds</code> to <code>status</code> if status history is all you model. That cuts the response size enormously, because most changelog volume is description edits and label churn nobody analyses.</p>



<p class="wp-block-paragraph">Store changelog as its own narrow table, one row per field change, not nested inside the issue record. Athena can handle nested structures, but flat is dramatically easier to reason about in SQL and much cheaper to scan.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 3: the incremental load that duplicates everything</h2>



<p class="wp-block-paragraph">This is the one that bites hardest, because nothing fails. Everything runs green for weeks and the numbers are wrong the entire time.</p>



<p class="wp-block-paragraph">The obvious incremental design is a watermark: track the last successful run, then pull <code>updated &gt;= watermark</code> on each run and append the results to S3. It is the standard pattern and it works for immutable event data.</p>



<p class="wp-block-paragraph">Jira issues are not immutable. A ticket created in January and touched every week appears in every weekly extract. If you append each run into its own partition, that one issue now exists as a row in twenty partitions. Athena is doing exactly what you asked: <code>SELECT count(*) FROM jira_issues WHERE status = 'Open'</code> counts twenty things. Your open bug count is inflated, your throughput is inflated, and the inflation is proportional to how much a ticket gets edited, which correlates with how contentious it was. The busiest work is the most over-counted.</p>



<p class="wp-block-paragraph">Nobody catches this from the dashboard, because the numbers move in the right direction and look approximately sane. It surfaces months later when someone reconciles against a JQL query in Jira and the two disagree.</p>



<h3 class="wp-block-heading">Pick one of three fixes, deliberately</h3>



<p class="wp-block-paragraph"><strong>Full snapshot per load, dedupe at read time.</strong> Keep every version, partition by <code>load_date</code>, and always read through a view that takes the latest row per issue key. Storage is cheap, history is free, and you can answer &#8220;what did the board look like in March&#8221; without any extra machinery. The cost is that every query pays for the deduplication.</p>



<pre class="wp-block-code"><code>CREATE OR REPLACE VIEW jira_issues_current AS
SELECT * FROM (
  SELECT
    i.*,
    ROW_NUMBER() OVER (
      PARTITION BY issue_key
      ORDER BY load_date DESC, updated DESC
    ) AS rn
  FROM jira_issue_snapshot i
  WHERE load_date &gt;= date_format(current_date - interval '7' day, '%Y-%m-%d')
) WHERE rn = 1;</code></pre>



<p class="wp-block-paragraph">The <code>load_date</code> filter inside the view matters. Without it the deduplication window scans the entire table on every query, which is the single most common way a cheap Athena setup becomes an expensive one.</p>



<p class="wp-block-paragraph"><strong>Overwrite the affected partitions.</strong> Partition by something stable, usually issue created month, and rewrite whole partitions when any issue in them changes. Clean reads, no dedupe cost, but you now own read-modify-write logic in Lambda and a concurrency problem if two runs overlap.</p>



<p class="wp-block-paragraph"><strong>Use an ACID table format.</strong> Apache Iceberg gives you real <code>MERGE INTO</code> semantics on S3, and Athena supports it natively. This is the right answer if you are already running Iceberg elsewhere or if the pipeline will grow to a dozen sources. It is not worth adopting solely to load one Jira site, because you inherit compaction and snapshot expiry as ongoing maintenance.</p>



<p class="wp-block-paragraph">For a single Jira site feeding a handful of dashboards, snapshot plus a dedupe view is what I would build. It has the fewest moving parts and it gives you point-in-time history as a side effect, which you will want the first time someone asks a retrospective question.</p>



<h3 class="wp-block-heading">Deletions and moves</h3>



<p class="wp-block-paragraph">A JQL watermark query never returns deleted issues, so they persist in your data forever. Same for issues moved out of scope or into an archived project. Periodically reconcile: pull the full set of issue keys with a minimal <code>fields</code> list, compare against what you hold, and mark the difference. Monthly is usually enough. Skip this and your historical counts drift upward permanently.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family 4: the S3 layout that decides your Athena bill</h2>



<p class="wp-block-paragraph">Athena&#8217;s standard pricing model bills on bytes scanned, rounded up, with a small per-query minimum. DDL statements are free. That means your storage layout, not your SQL, is what determines cost.</p>



<p class="wp-block-paragraph">Three things do almost all the work:</p>



<ul class="wp-block-list">
<li><strong>Write Parquet, not JSON.</strong> Columnar storage lets Athena read only the columns your query touches. A dashboard selecting five columns from a forty-column table scans a small fraction of what the equivalent JSON would. Snappy or Zstd compression on top of that.</li>

<li><strong>Avoid tiny files.</strong> A Lambda that runs hourly and writes one small file per invocation produces thousands of objects. Athena spends more time opening files than reading them. Batch writes so files land in the low hundreds of megabytes, or run a periodic compaction job.</li>

<li><strong>Use partition projection.</strong> This is the one people skip and then wonder why queries have a fixed few-second overhead.</li>
</ul>



<p class="wp-block-paragraph">Partition projection lets Athena calculate partition locations from table properties instead of calling <code>GetPartitions</code> against the Glue Data Catalog. On a table with a couple of years of daily partitions, that lookup is real latency on every single query, and it grows as the table does. Projection removes it, and removes your need for a Glue crawler entirely, which is one less scheduled job and one less IAM role.</p>



<pre class="wp-block-code"><code>CREATE EXTERNAL TABLE jira_issue_snapshot (
  issue_id        string,
  issue_key       string,
  project_key     string,
  issue_type      string,
  status          string,
  status_category string,
  assignee_id     string,
  created         timestamp,
  updated         timestamp,
  resolutiondate  timestamp,
  story_points    double
)
PARTITIONED BY (load_date string)
STORED AS PARQUET
LOCATION 's3://example-jira-lake/issue_snapshot/'
TBLPROPERTIES (
  'projection.enabled'                = 'true',
  'projection.load_date.type'         = 'date',
  'projection.load_date.format'       = 'yyyy-MM-dd',
  'projection.load_date.range'        = 'NOW-3YEARS,NOW',
  'projection.load_date.interval'     = '1',
  'projection.load_date.interval.unit'= 'DAYS',
  'storage.location.template'         =
    's3://example-jira-lake/issue_snapshot/load_date=${load_date}'
);</code></pre>



<p class="wp-block-paragraph">One caution that surprises people: projection describes partitions Athena will look for, not partitions that exist. Set a range wider than your data and queries without a <code>load_date</code> filter will probe empty prefixes. Keep the range tight to what you actually hold.</p>



<p class="wp-block-paragraph">Finally, set <code>BytesScannedCutoffPerQuery</code> on the Athena workgroup. It kills any query that exceeds a scan threshold. One analyst running <code>SELECT *</code> against three years of data in a BI tool&#8217;s preview pane is the classic surprise line item, and this stops it at the source.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Modelling time in status</h2>



<p class="wp-block-paragraph">Once status changelog rows are landed flat, the core metric is a window function. Each transition&#8217;s duration is the gap to the next transition on the same issue:</p>



<pre class="wp-block-code"><code>WITH transitions AS (
  SELECT
    issue_key,
    to_status,
    changed_at,
    LEAD(changed_at) OVER (
      PARTITION BY issue_key ORDER BY changed_at
    ) AS next_changed_at
  FROM jira_changelog
  WHERE field_id = 'status'
    AND load_date &gt;= date_format(current_date - interval '90' day, '%Y-%m-%d')
)
SELECT
  issue_key,
  to_status,
  SUM(date_diff('second', changed_at,
                COALESCE(next_changed_at, current_timestamp))) / 3600.0
    AS hours_in_status
FROM transitions
GROUP BY issue_key, to_status;</code></pre>



<p class="wp-block-paragraph">The <code>COALESCE</code> handles the current status, which has no successor transition. Two modelling decisions to make consciously: whether to subtract non-working hours, and how to treat a ticket that moves backwards through the workflow. Both are business questions, not technical ones, and both should be settled in a documented view rather than reimplemented in each dashboard.</p>



<p class="wp-block-paragraph">Map raw status names to a stable category early. Teams rename statuses constantly, and a metric keyed on the literal string breaks the moment someone changes &#8220;In Review&#8221; to &#8220;Peer Review&#8221;. A small mapping table joined at query time keeps history comparable.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>Every analytical column is null.</strong> You did not pass <code>fields</code>, or you passed a custom field ID from another site. Log the resolved field list on every run.</li>

<li><strong>Counts higher than the same JQL in Jira.</strong> Duplicate rows across partitions. Check whether your query goes through the dedupe view or straight at the base table.</li>

<li><strong>Extraction loops forever.</strong> Do not terminate on <code>isLast</code>. Terminate on missing <code>nextPageToken</code> and keep a page cap.</li>

<li><strong>HTTP 410 from the API.</strong> You are still calling the removed <code>/rest/api/3/search</code>. Check your client library version too, not just your own code.</li>

<li><strong>Cycle times suspiciously low.</strong> Truncated changelog. Move to the dedicated changelog endpoint or bulk fetch.</li>

<li><strong>Athena returns zero rows but the files are there.</strong> Partition projection range does not cover the partition, or the S3 prefix does not match <code>storage.location.template</code>. Compare a real object key against the template character by character.</li>

<li><strong>Queries slow down as the table grows.</strong> Either you have no projection and Glue lookups dominate, or you have accumulated small files.</li>

<li><strong>Authentication failures overnight with no deploy.</strong> API token expired or was revoked. Alert on the specific status code rather than on &#8220;run failed&#8221;.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Appending incremental extracts without a deduplication strategy.</li>

<li>Building on current issue state and adding changelog later. Retrofitting history is far more work than including it from the start.</li>

<li>Hardcoding custom field IDs copied from documentation.</li>

<li>Storing the API token in a Lambda environment variable instead of Secrets Manager.</li>

<li>Running an hourly schedule when the dashboard is read once a day. You pay in small files and API quota for freshness nobody uses.</li>

<li>Letting BI tools query base tables directly instead of curated views.</li>

<li>Never reconciling against Jira. A scheduled check comparing approximate count to your row count catches drift within a day.</li>
</ul>



<h2 class="wp-block-heading">Best practices for a Jira analytics pipeline that survives</h2>



<ul class="wp-block-list">
<li>Land raw API responses to S3 before transforming. When your parsing is wrong, and it will be, you replay from raw rather than re-hammering the API.</li>

<li>Make the load idempotent. Re-running for the same date should produce the same result, not a second copy.</li>

<li>Define the pipeline in Terraform or CloudFormation and deploy it from CI. The IAM policy for Lambda and Athena is fiddly and you do not want to rebuild it from memory.</li>

<li>Alert on a run that succeeds with zero rows, not just on runs that error. Silent empty loads are the more common failure.</li>

<li>Scope the Jira token to a service account with read access to exactly the projects you need.</li>

<li>Put an S3 lifecycle policy on the raw zone. It grows faster than you expect and nobody queries last year&#8217;s raw JSON.</li>

<li>Expose curated views, not tables, to Grafana, Metabase, Power BI or QuickSight, and enable Athena&#8217;s query result reuse for repeated dashboard loads.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Should I use Lambda or Glue for Jira extraction?</h3>



<p class="wp-block-paragraph">Lambda, for an API-driven pull. The work is mostly waiting on HTTP responses, which Spark&#8217;s distributed compute does nothing for, and you would be paying for a Glue job&#8217;s minimum billing on something that is idle. Glue earns its place downstream, if you have heavy joins across several sources. Athena CTAS often covers that too.</p>



<h3 class="wp-block-heading">How often should the pipeline run?</h3>



<p class="wp-block-paragraph">Match the decision cadence, not the data cadence. Delivery metrics are reviewed weekly or in sprint ceremonies, so daily is almost always enough and produces far better file sizes. Reserve hourly for something with a real-time consumer, and know that you are buying that freshness with small-file overhead and API quota.</p>



<h3 class="wp-block-heading">Can I use webhooks instead of polling?</h3>



<p class="wp-block-paragraph">You can, and for near-real-time reaction it is the right tool. For analytics it is a poor primary source, because a missed delivery leaves a permanent hole you have no way to detect. The pattern that works is webhooks for freshness plus a scheduled reconciliation pull as the source of truth. If you only build one, build the scheduled pull.</p>



<h3 class="wp-block-heading">Does this work with Jira Data Center or Server?</h3>



<p class="wp-block-paragraph">The AWS half is identical. The extraction half is not. Cursor pagination and the <code>/search/jql</code> endpoint are Cloud-only changes, so self-hosted instances still use the older offset-based <code>/rest/api/2/search</code>. Write the extractor behind an interface if you need to support both, and expect the auth model to differ as well.</p>



<h3 class="wp-block-heading">How much does an Athena-based Jira pipeline cost to run?</h3>



<p class="wp-block-paragraph">The mechanism matters more than any figure I could quote. You pay for S3 storage, Lambda invocation time, and Athena per byte scanned with a small per-query minimum. Jira issue data compresses extremely well as Parquet, so storage stays small. The variable is dashboard query volume, which is why partitioning and workgroup scan limits matter. Model it against current AWS rates and your own expected query count.</p>



<h3 class="wp-block-heading">Why not just use eazyBI or a Jira reporting app?</h3>



<p class="wp-block-paragraph">Often you should. If your questions stay inside Jira, a Marketplace app gets you there in an afternoon with no infrastructure. The case for a pipeline is joining Jira to data that lives elsewhere, retaining history the app does not, or needing your data in a warehouse you already run. Build the pipeline when the app has actually failed you, not in anticipation.</p>



<h3 class="wp-block-heading">Should I load into Redshift instead of querying S3?</h3>



<p class="wp-block-paragraph">Only if you have concurrent BI users hitting the same tables constantly and Athena&#8217;s per-query latency is a real complaint. Jira data volumes are small, the query pattern is bursty, and Athena&#8217;s zero-idle-cost model fits that far better. Redshift makes sense as a consolidation layer across many sources, not for one issue tracker.</p>



<h2 class="wp-block-heading">The one thing to carry away</h2>



<p class="wp-block-paragraph">The hard part of a Jira analytics pipeline is not the AWS wiring. Lambda, S3, Glue and Athena will be working within a day. The hard part is that Jira issues are mutable, so an incremental load that appends is an incremental load that duplicates, and it does so without a single error in your logs.</p>



<p class="wp-block-paragraph">Decide your deduplication strategy before you write the first extract, not after someone reconciles a dashboard against JQL and finds a gap. Snapshot with a read-time dedupe view, partition projection so queries stay fast as history accumulates, and a scheduled reconciliation check that alerts on drift. Get those three right and the rest is plumbing.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need this built or fixed?</h2>



<p class="wp-block-paragraph">I design and run serverless data pipelines on AWS, and Jira extraction has more sharp edges than most sources. Things I can help with directly:</p>



<ul class="wp-block-list">
<li>Migrating an extractor off the removed <code>/rest/api/3/search</code> endpoint to cursor-based <code>/search/jql</code> without losing rows in the cutover</li>

<li>Auditing an existing Jira pipeline for duplicate rows and reconciling your numbers back against JQL</li>

<li>Building changelog-based cycle time and time-in-status models that survive workflow renames</li>

<li>Restructuring an S3 layer with Parquet, partition projection and file compaction to cut Athena scan costs</li>

<li>Packaging the whole thing as Terraform or CloudFormation with CI deployment and least-privilege IAM</li>

<li>Connecting Athena to Grafana, Power BI, Metabase or QuickSight with curated views instead of raw tables</li>
</ul>



<p class="wp-block-paragraph">Send me your extractor code, a Glue table definition, or an Athena query that is scanning more than it should, and I will tell you what I would change.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/jira-analytics-pipeline-aws-lambda-athena/">Building a Jira Analytics Pipeline with AWS Lambda and Athena (Without Double-Counting Everything)</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
