They probably never will, and that is not a criticism of the product. SageMaker Unified Studio is a front door. It is not a runtime, it is not a billing boundary, and it does not replace your IAM design. Teams who adopt it expecting consolidation do get consolidation, but only of attention. Everything underneath stays where it was, on the same services, on the same bill, behind the same permission model.

This post covers what it genuinely replaces, what it leaves untouched no matter how the migration deck was worded, and what it quietly adds to your operational surface that you will own indefinitely. If you are signing off on an adoption plan, the third part is the one that matters.

SageMaker Unified Studio is a front door, not a runtime

Start here, because every surprise downstream follows from this. When you run a query in the SQL editor, Athena or Redshift runs it. When you build a visual ETL job, Glue runs it. When you train a model, SageMaker AI runs it. When you orchestrate something, Airflow runs it. The studio decides who may open which tool, inside which project, against which data. It executes nothing itself.

AWS is explicit about this in its access control documentation: opening a project is equivalent to signing into an account in a specific Region and assuming one of that project’s IAM roles. The tools you then see are the tools that role can reach. There is no new execution layer. There is a new authorisation layer, and a new set of nouns to describe it. Hold that model and the rest is predictable.

What it actually replaces

The pile of standalone studios and query editors

This is the real win, worth stating plainly before the caveats start. AWS assembled the studio by pulling together the separate studios, query editors and visual tools that previously lived inside Athena, EMR, Glue, Redshift, Managed Workflows for Apache Airflow and the existing SageMaker Studio, then added a Bedrock-backed environment for generative AI work.

The problem it solves is not tab count. It is context loss. A data engineer moving from a Glue job, to the table it wrote, to the notebook consuming that table crossed three consoles, three permission surfaces and three mental models of what a “workspace” meant. That handoff is where mistakes lived. Collapsing it into one project boundary is the reason to adopt at all.

Per-team domain sprawl

The older pattern for isolating business units was a SageMaker domain each, then reconciling identity, networking and tagging across all of them by hand. Projects replace most of that. A project is a multi-user boundary with its own roles, storage and members, and it can span accounts, which the old shared-space model could not do cleanly.

The access request spreadsheet

There are two supported ways to give a project access to data: attach IAM permissions directly to the project role, or publish the asset to the catalog and let members subscribe through a request and approval flow.

The second one is the point of the product. If you are only ever going to use the first, you have bought a portal and skipped the governance engine you are now paying to run.

What it doesn’t replace

Your bill, or your ability to explain it

There is no standalone charge for the studio itself. AWS bills you for each underlying service you reach through it, plus usage of SageMaker Catalog, which is metered separately on metadata ingestion and related operations. Nothing consolidates.

What changes, and this is the part that bites, is attribution. Before, an engineer created a Glue job and it inherited whatever tags your pipeline applied. Now a project profile deploys a bundle of resources on that engineer’s behalf, and unless you have deliberately configured project-level custom tags, those resources land in your cost report under a project name that means nothing to finance.

Sort this out before migration. Decide your tag keys, wire them into the project configuration, and confirm they appear in Cost Explorer and the Cost and Usage Report before you onboard a second team. If you run a cost platform such as CloudZero or Vantage on top of the CUR, check that your allocation rules cope with resources whose creator is a service role rather than a person, because that is the shape everything arrives in now.

Your IAM and Lake Formation design

Project roles are ordinary IAM roles. Lake Formation grants are ordinary Lake Formation grants. Bucket policies still apply. If your data access is broken at any of those layers, the studio will surface the same denial you always got, with slightly less context about which principal was being used at the time.

Nobody in the adoption meeting will say “this fixes our permissions mess.” Plenty of people will quietly assume it. Say it out loud early.

Your infrastructure as code

This one catches DevOps teams hardest. AWS ships an open-source CI/CD CLI for the studio that promotes applications across dev, test and production by mapping each pipeline stage to an isolated project and substituting stage-specific values at deploy time. It is a good tool and it slots into GitHub Actions, GitLab CI or Jenkins without much argument.

It also does not create domains or projects. The documentation says so directly and points you at CloudFormation, CDK or Terraform for the infrastructure underneath, and the same limitations page notes that native notebook deployment is not covered. The substrate is still yours to build and version. You get an application deployment tool, not a platform provisioning tool. Plan for both.

SageMaker AI

The original SageMaker was renamed SageMaker AI and remains available on its own. If your team’s work is model training and inference and nothing else, you are allowed to keep using it directly and skip the governance layer entirely. That is a supported choice, not a holdout position.

What SageMaker Unified Studio quietly adds

Here is the section that rarely makes it into the migration plan.

You now operate DataZone, intentionally or not

SageMaker Catalog is built on Amazon DataZone, and the governance model comes with it: domains, domain units, projects, project profiles, blueprints, environments, glossaries, metadata forms, subscription requests. Domain identifiers still carry a dzd prefix, and the console for creating one still lives under the DataZone console path.

You can see the shape of what you own from the CLI, which is worth doing on day one. These commands read the DataZone control plane sitting behind the studio:

# Every domain in the account and Region, with its status
aws datazone list-domains 
  --query 'items[].{id:id,name:name,status:status}' 
  --output table

# Projects inside one domain (domain IDs look like dzd_xxxxxxxx)
aws datazone list-projects 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The project profiles an admin has published, which is the
# real menu of what users can create
aws datazone list-project-profiles 
  --domain-identifier dzd_EXAMPLE 
  --output table

# The environments actually deployed under one project
aws datazone list-environments 
  --domain-identifier dzd_EXAMPLE 
  --project-identifier PROJECT_ID

Run these monthly against a domain someone else administers and you will learn more about your real governance posture than any dashboard tells you.

Project creation is a CloudFormation deploy wearing a friendly hat

Blueprints are backed by CloudFormation templates. Project profiles bundle blueprint configurations, and choosing a profile triggers a stack deployment that provisions the environment behind the project.

This is the invisible failure mode. To a user, project creation is a form and a spinner. To you, it is a stack that can roll back on a missing permission, an unavailable subnet, a service quota or a name collision. When somebody reports that project creation “just doesn’t work,” do not reproduce it in the UI. Read the stack events:

# Find recently failed or rolled-back stacks
aws cloudformation list-stacks 
  --stack-status-filter CREATE_FAILED ROLLBACK_COMPLETE ROLLBACK_IN_PROGRESS 
  --query 'StackSummaries[].{name:StackName,status:StackStatus}' 
  --output table

# Then read the actual reason, which is almost always
# in the first CREATE_FAILED event, not the last
aws cloudformation describe-stack-events 
  --stack-name STACK_NAME 
  --query 'StackEvents[?ResourceStatus==`CREATE_FAILED`].[LogicalResourceId,ResourceStatusReason]' 
  --output table

The first failed resource causes the rollback. Everything after it is noise. This one habit turns most “the platform is broken” tickets into five-minute answers.

A domain-type fork you will live with for a long time

Domains can authenticate through IAM or through IAM Identity Center. AWS positions the Identity Center configuration as the governance-heavy option, preserving individual user identity through the session and supporting the formal publish and subscribe approval workflow with identity-based audit trails. The IAM configuration is positioned toward lighter-weight developer productivity.

Parity between the two has moved over time and is still moving. Before committing an organisation to one, check the current documentation for the specific capabilities you care about, in the Region you are deploying to. Do not take a blog post’s word for it, including this one. This is the decision that is genuinely expensive to reverse, because it is entangled with how every user signs in.

Where adoption actually goes wrong

“It works in my console but not in the project”

This is the single most common support ticket, and it is almost never a bug. The engineer tested with their own federated role, which has broad permissions, then opened a project running under a narrow project role. Two different principals, two different answers.

Confirm which principal you are before debugging anything else. Run this from a terminal inside the project environment, not from your laptop:

aws sts get-caller-identity

If the ARN is not the one you assumed, stop. You have found the problem.

The bill went up after a migration that was supposed to save money

Usually because the migration created things rather than moving them. New project environments provision new resources, catalog metadata ingestion is metered, and per-project orchestration environments are not free just because nobody chose them explicitly. Compare like for like by tag rather than by service total, and give it a full billing cycle before drawing conclusions.

Nobody can find the data that is definitely there

Existing in a Glue database is not the same as being published to the catalog. If an asset was never published, it will not appear in search, and no amount of IAM permission will make it discoverable. Check publication state before you check permissions.

Common mistakes

• Treating it as a migration rather than an addition. Nothing is decommissioned. Your Glue jobs, Athena workgroups and Redshift clusters stay where they were.
• Letting users self-serve projects on day one. Every project is a stack deploy and a set of billable resources. Gate it until profiles and tagging are settled.
• Choosing the domain authentication type in a hurry. It is the hardest thing here to change later.
• Skipping the catalog. Grant everything through direct IAM attachments and you carry the governance overhead with none of the benefit.
• Assuming the CI/CD CLI provisions your platform. It deploys applications into projects that already exist.
• Not tagging project resources from the start. Retrofitting attribution across dozens of blueprint-created resources is miserable work.

How I’d decide

A rough order of operations for anyone weighing this up:

1. Ask what problem you are solving. If it is “our teams cannot discover or safely share each other’s data,” this is the right tool. If it is “we have too many consoles,” it is an expensive way to buy fewer bookmarks.
2. Count your teams. The value curve is steep with cross-team sharing and nearly flat for one team in its own account. One team, one account, mostly model training? Stay on SageMaker AI.
3. Decide the authentication model first. Identity Center for per-user audit trails and approval workflows. IAM if you are optimising for a small group moving fast and accept a different feature set.
4. Write the project profiles yourself. The defaults are demonstrations. Production profiles should encode your networking, tags and quotas.
5. Prove the teardown path before the build path. Create a project, use it, delete it, confirm nothing is orphaned. Do this before any real team touches it.
6. Keep the old consoles. They are your escape hatch when the abstraction hides something you need to see, and cost nothing to leave enabled.

One more piece of judgement: wire your existing observability in from the start rather than relying on the studio’s own views. CloudWatch, Amazon Managed Grafana or a Grafana Cloud stack you already run, it does not much matter which. The failures that matter here are stack failures and job failures, and both are visible in the places you already watch.

Frequently asked questions

Does SageMaker Unified Studio replace SageMaker Studio?

It supersedes it as the recommended experience, and AWS provides an upgrade path that imports existing domains, user profiles and spaces without redeploying infrastructure. JupyterLab and Code Editor spaces, training jobs, pipelines, models and endpoints are preserved, and administrators control when access to the older experience is turned off. It is a migration you schedule, not one that happens to you.

Do I still need Amazon DataZone separately?

Not as a separate purchase. The catalog is built on DataZone, and existing DataZone domains can be upgraded into unified domains, which requires establishing the domain execution and service roles the new model expects. Functionally you are already running DataZone; the studio is the interface on top of it.

How much does SageMaker Unified Studio cost?

The interface has no standalone charge. You pay for each AWS service you use through it on that service’s normal terms, plus SageMaker Catalog usage, metered on metadata ingestion and related operations. Model it by listing what a project profile actually provisions and applying current rates to each item, rather than hunting for a per-seat number that does not exist.

Can I manage domains and projects with Terraform?

AWS’s own documentation directs you to CloudFormation, CDK or Terraform for creating the domain and project substrate, since the CI/CD CLI deliberately does not do it. Coverage in each tool tracks the underlying API, so check the current provider or construct documentation for the specific resources you need before designing around it.

What happens to my Glue jobs and Athena queries?

Nothing. They keep running as Glue jobs and Athena queries. The studio gives you a different place to author and run them and a governance layer that decides who may do so. The execution, the quotas and the billing are unchanged.

Is it worth adopting for a small team?

Often not. The governance machinery is the product, and a team of three sharing one account has little to govern. The threshold is roughly where you have more than one team, data one team owns that another needs, and a real answer required to who approved that access.

Can I run both the old and new experiences at once?

Yes, and you should during transition. The upgrade path lets teams keep access to both while they adapt, with the administrator deciding when to close the older door. Running in parallel for a full billing cycle is also the cleanest way to see what the change actually costs.

The one thing worth remembering

SageMaker Unified Studio replaces an interface problem. It does not replace an infrastructure problem, a permissions problem, or a cost attribution problem, and it adds a governance system you are now responsible for operating.

That trade is worth it when you have data crossing team boundaries and no credible answer to who approved access to it. It is not worth it when the pitch was “fewer tabs.” The teams who are happy a year in went in knowing they were buying a governance engine and budgeted the operational time to run it. The unhappy ones expected a consolidation that was never on offer.

Ask which one you are before the first project gets created, not after the fortieth.

Need a second opinion on your adoption plan?

This is a decision that is cheap to get right at the start and expensive to unwind two quarters later. Things I can help with directly:

• Reviewing a domain design before rollout, especially the authentication model choice and how it meets your existing identity setup
• Writing production project profiles and blueprints that encode your VPC layout, tagging standard and quotas instead of the defaults
• Building the cost attribution layer so project-created resources land in reporting with tags finance can actually use
• Wiring the CI/CD CLI into an existing GitHub Actions, GitLab CI or Jenkins pipeline, plus the CloudFormation or Terraform layer underneath that the CLI does not cover
• Diagnosing failed project creations and access denials that trace back to project roles or Lake Formation grants
• Sanity-checking a migration plan from SageMaker Studio or an existing DataZone domain, including what runs in parallel and for how long

Send me a project profile, a failed stack event log, or a tagged cost export and I will tell you what I see.

The post SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn’t appeared first on John Nessime.

That gap, between a Dockerfile that looks correctly ordered and a build that behaves as if it isn’t, is where most Dockerfile layer caching problems live. The ordering advice everyone repeats is real, but it’s maybe a third of the story. The rest is about what actually invalidates a layer, where the cache physically lives, and whether the machine running your build can reach it at all.

So this is organized by failure family rather than by feature. Ordering that looks right and isn’t. A build context that quietly poisons your checksums. A cache that exists but is unreachable from CI. Cache mounts that promise more than they deliver. Multi-stage layouts and the COPY --link argument. Then how to prove which one you’re hitting instead of guessing.

How the builder actually decides to reuse a layer

Two different rules are running, and mixing them up causes half the confusion.

For most instructions, the builder compares the instruction string against the cached layer. That is the entire check. It does not look inside the container to see what changed. RUN apt-get update matches a cached layer whenever the text matches, regardless of what the package mirrors are serving today.

For COPY, ADD, and RUN instructions that use a bind mount, the builder computes a cache checksum from the metadata of the files involved. Modification time is deliberately excluded, which is why a fresh git clone on a CI runner doesn’t invalidate every copy by itself.

Then the rule that ties it together: once a layer misses, every layer after it rebuilds. No skipping ahead, no partial reuse. One bad instruction near the top costs you the whole tail of the file.

Three consequences catch people out. A RUN line that installs OS packages caches indefinitely, so rebuilding months later reproduces the original package versions and “just rebuild the image” is not a security response on its own. Build arguments do invalidate the cache, so an ARG carrying a commit SHA near the top of the file is expensive. And build secrets do not: their contents are excluded from the checksum, though their IDs and mount paths are part of it.

One more that bites teams chasing reproducible builds. WORKDIR respects the SOURCE_DATE_EPOCH build argument. Set it to a git commit timestamp for provenance and you invalidate WORKDIR and everything below it on every single commit. Pin it to a fixed value if you want deterministic timestamps without paying that.

Failure family one: ordering that looks right

The canonical fix is to split the copy so dependency installation sits above the source code. Cheap to write, and it works:

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim
WORKDIR /app

# Manifest and lockfile only. Changes rarely.
COPY package.json package-lock.json ./
RUN npm ci

# Source. Changes constantly. Everything below rebuilds, and that is fine.
COPY . .
RUN npm run build

What breaks this in real repositories is usually not the copy order. It’s something above it that nobody thinks of as a layer.

• An ARG near the top. A build arg whose value changes per build invalidates everything that follows. Declare args as low in the file as the build allows.
• Labels carrying commit metadata. LABEL lines that embed a revision are cheap to run but they are still layers. Put them at the very bottom.
• Wildcard copies that pick up more than you think. COPY package*.json ./ looks tidy until a generated file starts matching the glob. Name the files explicitly.
• Monorepo manifests scattered across packages. If installing needs twelve manifest files from twelve directories, copy them individually or accept you’re copying a subtree and lean hard on .dockerignore.

The general test: walk the file top to bottom and ask, for each line, “what makes this change?” Anything answering “every commit” belongs below anything expensive.

Failure family two: the build context poisons the checksum

When you run COPY . ., the checksum covers everything in the context that isn’t excluded. That means .git, node_modules, coverage output, local .env files, and whatever your last test run left behind. A colleague running the test suite before pushing is enough to change the checksum and rebuild the tail of your image.

A .dockerignore file in the root of the context fixes it, and the rules apply to subdirectories too:

.git
node_modules
coverage
dist
*.log
.env
.env.*
Dockerfile
.dockerignore

This pays twice. Smaller contexts transfer faster to the builder, and unrelated local files stop ending up inside your published image. If you have ever found a .git directory or a stray credentials file in a production image, this is the line of defense that was missing.

There’s a second option when the source is only needed to produce an artifact and shouldn’t live in the image at all. A bind mount makes the context available for the duration of one RUN without adding it to any layer:

# syntax=docker/dockerfile:1
FROM golang:1-bookworm
WORKDIR /build
RUN --mount=type=bind,target=. go build -o /app/server ./cmd/server

Two rules keep this from surprising you. The mount is read-only by default, so the output has to be written outside the mount target, which is why /app/server sits outside /build. And mounted files are not persisted into the final image or the build cache. Only what the RUN produced survives.

Failure family three: Dockerfile layer caching that never reaches CI

This is the invisible one, and it’s behind most “my Dockerfile is ordered correctly and CI is still slow” tickets.

The build cache is internal to the builder instance. Not to your account, not to your registry. To the builder. Switch builders and the cache doesn’t follow. Start a fresh ephemeral runner and there is no cache at all, so every instruction runs from scratch no matter how beautifully the file is ordered. Nothing warns you. The build just takes as long as a cold build, because it is one.

The fix is an external cache. --cache-to exports the cache somewhere durable, --cache-from imports it at the start of the next build.

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:latest 
  --push .

Four details decide whether this works or quietly does nothing:

• mode=max versus the default. The default exports only the layers of the resulting image. In a multi-stage build the expensive work happens in a stage that never ships, so the default exports almost nothing useful. mode=max includes intermediate stages. It costs storage and upload time, and it is usually the right trade.
• Use a separate ref for the cache. A different tag or repository, anything that isn’t the tag you push the image to.
• Registry compatibility. The registry cache defaults to writing an image index and not every registry accepts one, with Amazon ECR the usual casualty. Setting image-manifest=true alongside oci-mediatypes=true writes a single image manifest instead.
• The default driver is limited. The plain docker driver supports the inline, local, registry and GitHub Actions backends only when the containerd image store is enabled. Otherwise create a builder with the docker-container driver first.

On GitHub Actions the equivalent lives in the build action:

- uses: docker/setup-buildx-action@v4

- uses: docker/build-push-action@v7
  with:
    push: true
    tags: ghcr.io/acme/app:latest
    cache-from: type=gha
    cache-to: type=gha,mode=max

The GitHub Actions backend is convenient because it needs no credentials, but it draws from a per-repository storage budget shared with every other cache in that repo, and eviction is silent. A build that was fast last week can go cold with nothing changed in your Dockerfile. When that starts happening, move to a registry backend, or to an S3 or Azure Blob backend if you’d rather pay object storage rates than registry rates.

Failure family four: cache mounts that don’t survive the job

Layer caching is all or nothing per instruction. Add one dependency, the install layer misses, and you re-download the four hundred packages you already had. Cache mounts fix exactly that: a directory that persists across builds even when the layer around it rebuilds.

# syntax=docker/dockerfile:1
FROM python:3-slim
WORKDIR /app
COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip 
    pip install -r requirements.txt

Point the target at whatever directory that tool uses for its own cache. Go wants its module and build caches, Rust the Cargo directories, .NET the NuGet package folder. Apt is the one needing care, because it expects exclusive access to its data:

RUN --mount=type=cache,target=/var/cache/apt,sharing=locked 
    --mount=type=cache,target=/var/lib/apt,sharing=locked 
    apt-get update && apt-get install --no-install-recommends -y gcc

Without sharing=locked, parallel builds write to the same apt cache simultaneously and you get an inconsistent package database, intermittently, which is a genuinely unpleasant thing to debug.

Now the part that isn’t in most tutorials. Cache mounts live on the builder’s local storage and they are not exported by --cache-to. The external cache carries layers. It does not carry cache mount contents. On a fleet of ephemeral CI runners, every cache mount you add is a no-op: a fresh runner starts with an empty mount, fills it during the build, then throws the machine away.

They’re still worth adding for local development, where they pay for themselves immediately. Making them pay off in CI needs a builder with persistent storage behind it: a self-hosted runner that keeps its builder between jobs, a long-lived buildkitd on a VPS from somewhere like Contabo or InterServer reached over the remote driver, the Kubernetes driver with a volume attached to the builder pods, or a managed build service such as Docker Build Cloud or Depot that holds persistent cache volumes for you.

One hard rule regardless: never put something the image needs into a cache mount. Installing into /app/node_modules behind a cache mount produces an image that works on your machine and has no dependencies in production, because the mount contents are discarded when the instruction finishes. Cache the package manager’s download directory, not its output.

Failure family five: multi-stage builds and the COPY –link argument

Multi-stage builds help caching for a reason that has nothing to do with image size. They let you put expensive, rarely-changing work in a stage whose inputs are stable, and copy only the artifact forward.

# syntax=docker/dockerfile:1
FROM node:22-bookworm-slim AS deps
WORKDIR /app
COPY package.json package-lock.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

FROM node:22-bookworm-slim AS build
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
RUN npm run build

FROM node:22-bookworm-slim AS runtime
WORKDIR /app
ENV NODE_ENV=production
COPY --from=deps /app/node_modules ./node_modules
COPY --from=build /app/dist ./dist
CMD ["node", "dist/index.js"]

The catch wastes an afternoon if you hit it cold: export cache in the default mode and the deps stage isn’t in the export, because it isn’t part of the final image. Multi-stage plus external cache means mode=max, or the whole structure buys you nothing in CI.

Then there’s COPY --link, which comes up in every thread on this topic and deserves an honest treatment rather than a recommendation. The case for it is real: a linked copy creates an independent snapshot layer that doesn’t depend on the filesystem beneath it, then merges on top. The layer can be reused with --cache-from even when earlier instructions in the same stage changed, and it enables rebasing an image onto an updated base without re-running the build.

The case against is also real, and it’s mostly semantics you didn’t ask to change. Linked layers start from an empty filesystem, so a destination that resolved through a symlink created in an earlier layer no longer behaves the same, and ambiguous destinations resolve differently. Use explicit trailing slashes for directories. There are also reports of cache accounting misbehaving with linked layers, and a long-standing open report that the cache behavior doesn’t match the announcement.

Where I land: worth trying on artifact copies into a final stage, where destinations are plain directories and the win is largest. I wouldn’t blanket-apply it across a repository on the strength of a blog post, mine included. Measure a warm rebuild before and after. If the numbers don’t move, you’ve added a semantic footgun for nothing.

Troubleshooting: proving which layer missed and why

Stop guessing. The build output tells you, if you ask for the plain progress format:

docker buildx build --progress=plain --tag app:dev .

1. Find the first step not marked CACHED. Everything after it is guaranteed to rebuild, so everything after it is noise. There is exactly one problem and it is that step.
2. If it’s a COPY or ADD, the inputs changed. List what the context actually contains, compare against .dockerignore, and look for generated files or lockfiles being rewritten by a pre-build script.
3. If it’s a RUN, something above it changed. Scroll up. A build arg, a label, a base image tag that moved, or SOURCE_DATE_EPOCH set from a commit timestamp.
4. If the earliest steps miss in CI but not locally, the import didn’t happen. Check the log for the cache manifest being imported. A missing or unreadable cache ref usually isn’t a hard failure, the build just proceeds cold.
5. If it’s intermittent, suspect eviction. Garbage collection or a cache storage budget removed what you were relying on.

Two commands earn their place here. The first shows what the builder is currently holding, the second clears it so you can measure a genuine cold build:

docker buildx du
docker buildx prune --all

And when you want one stage to run fresh while keeping the rest cached, target it by name instead of nuking everything:

docker build --no-cache-filter deps --tag app:dev .

Common mistakes

• Assuming the Dockerfile is the whole problem. If CI has no external cache configured, no amount of reordering helps. Check that first.
• Adding cache mounts to fix CI. Great locally, invisible in the pipeline unless the builder persists.
• Exporting cache in the default mode with a multi-stage build. The expensive stage isn’t in the export. This is the most common reason external caching looks configured and doesn’t help.
• Squashing everything into one giant RUN. Fewer layers is a size optimization, not a caching one. Collapsing distinct steps means any change reruns all of them.
• Treating a cached rebuild as a patched rebuild. A cached package install gives you the versions from the original build. If you rebuilt for a CVE, you need to break that cache on purpose.
• Using latest as a base tag. The digest moves under you, invalidating everything below unpredictably and making builds non-reproducible at the same time.

Best practices worth the effort

• Order instructions by how often their inputs change, not by what feels logical. Stable and expensive at the top, volatile and cheap at the bottom.
• Write the .dockerignore before the Dockerfile. Thirty seconds of work that prevents both cache churn and accidental secret inclusion.
• Configure an external cache in CI on day one, with mode=max if the build has more than one stage.
• Add cache mounts for every package manager in the build. One line each, and the difference between “added one dependency” and “re-downloaded the world”.
• Track warm build duration as a metric rather than a vibe. A Grafana panel showing build time per branch turns “the build feels slow” into something you can point at.
• Keep a scheduled no-cache build in the pipeline. It proves the Dockerfile still works from scratch and picks up base image updates that cached builds silently skip.

Frequently asked questions

Why does my Docker build ignore the cache even though nothing changed?

Usually because something did change and it wasn’t the file you were watching. Build with --progress=plain and find the first step not marked CACHED. If it’s a COPY, a gap in .dockerignore is letting local artifacts into the context. If it’s early in the file, check build args, labels carrying commit metadata, or a base image tag that moved. In CI, the usual answer is that there was no cache to hit at all.

Do fewer layers make Dockerfile layer caching better?

No, and the advice is often applied backwards. Combining commands into one RUN reduces image size and stops intermediate junk being committed into separate layers. It also means any change reruns all of it. Combine steps that genuinely belong together, like an update-and-install pair, and keep independent work in separate instructions.

Do cache mounts work in GitHub Actions?

Not usefully on the default hosted runners. Cache mounts live on the builder’s local storage and are not exported by --cache-to, so an ephemeral runner starts with an empty mount every time. You need a builder with persistent storage: a self-hosted runner keeping its builder between jobs, a remote buildkitd on your own host, the Kubernetes driver with a volume, or a managed build service.

Registry cache backend or the GitHub Actions one?

Start with the GitHub Actions backend because it needs no credentials and no extra infrastructure. Move to a registry backend when you outgrow the repository cache budget, when you want the cache shared across repositories or with local developer builds, or when silent evictions make build times unpredictable. Registry caching costs a round trip and some storage in exchange for being yours.

Does changing a build secret invalidate the cache?

No. Secret contents are deliberately excluded from the cache checksum, so rotating a token leaves every layer valid. The secret’s ID and mount path do participate, so renaming one invalidates. If a changed secret needs to force a rebuild, pair it with a build argument you also change, since build args do invalidate.

Why does a cached image still contain old, vulnerable packages?

Because a RUN instruction is matched on its text, not on what it would produce today. The package install layer stays valid indefinitely, so rebuilding without breaking that cache reproduces the original package set. Force it with --no-cache, target the relevant stage with --no-cache-filter, or move the base image tag so the layers below invalidate.

Is COPY –link safe to use everywhere?

Not everywhere. It changes destination semantics because linked layers start from an empty filesystem, so anything relying on symlinks created by earlier layers can break and ambiguous destinations resolve differently. It’s most defensible on artifact copies into a final stage with explicit directory paths. Try it, measure a warm rebuild, keep it only if the numbers justify the change.

The one thing worth remembering

Dockerfile layer caching fails in two categories that need completely different fixes. Either a layer is being invalidated when it shouldn’t be, which is a Dockerfile and build context problem, or the cache was never there to begin with, which is a builder and pipeline problem. Reordering instructions solves the first and does nothing at all for the second.

So before you spend an afternoon rearranging COPY lines, run one build with plain progress output and find the first step that isn’t CACHED. If it’s step one, your problem isn’t the Dockerfile. It’s that the machine running the build has never seen your cache and nobody told it where to look.

Need a second pair of eyes on your build pipeline?

Slow container builds are usually three or four small problems stacked, not one big one, and they unpick quickly once you can see the whole pipeline. I help teams with:

• Auditing a Dockerfile end to end for invalidation traps, oversized build contexts and stages that never cache
• Setting up external cache backends in GitHub Actions, GitLab CI or Jenkins, including registry and object storage backends and the awkward ECR manifest cases
• Standing up a persistent remote buildkitd on your own infrastructure so cache mounts actually pay off in CI
• Restructuring multi-stage builds so the expensive stages are cacheable and the runtime image stays small
• Adding build duration and cache hit visibility to your existing dashboards so regressions get caught early
• Separating “rebuild for a code change” from “rebuild for a security patch” so cached layers stop hiding unpatched packages

Send me your Dockerfile and a build log with --progress=plain output, and I’ll tell you which step is costing you and why.

The post Dockerfile Layer Caching: Why Every Build Reinstalls Everything appeared first on John Nessime.

The sync worker was healthy. Zero errors in the last twenty-four hours. CPU flat, memory flat, no restarts, no failed jobs. Every dashboard was green because every dashboard was measuring the wrong thing. The integration had stopped receiving anything from Zoho four days earlier, and an integration that receives nothing produces no errors at all.

That is the whole problem with monitoring Zoho integrations. The failures that hurt do not throw exceptions. They show up as silence, as slowly degrading throughput, or as HTTP 200 responses that quietly did not do what you asked. This post covers how to instrument for those three failure families using CloudWatch, Grafana and Loki, and why the obvious metrics miss all three.

The three ways Zoho integrations actually break

Before touching a dashboard, it helps to be honest about what you are defending against. In my experience these fall into three groups, and they need genuinely different instrumentation:

1. Silent stop. Zoho stops sending you events. Nothing fails, because nothing happens.
2. Silent throttle. You run out of API credits or hit a concurrency ceiling. Throughput degrades before anything hard-fails.
3. Silent drift. Calls succeed, records do not land, and the two systems diverge without a single error log.

An error-rate panel catches none of these. That is not a criticism of error-rate panels, it is just what they are for.

Failure family one: the integration that just stops

If you push data out of Zoho CRM using the Notifications API, you subscribe a notify_url to module events and Zoho posts to it when records change. It works well. It also has a detail that catches people out repeatedly.

Notification channels expire. Zoho’s documentation is explicit: the expiry can be set to a maximum of one week from the time the channel is enabled, and if you do not specify one, or you ask for longer than a week, the default expiry is one hour. So the failure mode is not “someone forgot to renew after a year.” It is “someone deployed a subscription script without an expiry field and the integration worked beautifully for sixty minutes.”

What makes this genuinely hard rather than just annoying is that there is no endpoint to list your currently active channels. You cannot ask Zoho what it thinks it is subscribed to. You can only ask whether events are still arriving. That single fact should shape your entire monitoring design: you are not monitoring for errors, you are monitoring for absence.

Emit a heartbeat, then alarm on its absence

The receiving end of the webhook should emit a metric on every delivery, including deliveries you decide to discard. Embedded Metric Format is the cheapest way to do this from Lambda or a container, because you write structured JSON to stdout and CloudWatch extracts the metrics asynchronously. No PutMetricData call, no synchronous latency in your request path.

{
  "_aws": {
    "Timestamp": 1710000000000,
    "CloudWatchMetrics": [
      {
        "Namespace": "ZohoIntegration",
        "Dimensions": [["Module", "Channel"]],
        "Metrics": [
          { "Name": "NotificationsReceived", "Unit": "Count" },
          { "Name": "ChannelSecondsRemaining", "Unit": "Seconds" }
        ]
      }
    ]
  },
  "Module": "Leads",
  "Channel": "leads-prod",
  "NotificationsReceived": 1,
  "ChannelSecondsRemaining": 486000,
  "record_id": "5725767000002166520",
  "operation": "insert"
}

Note what is a dimension and what is not. Module and Channel are bounded sets, so they are safe dimensions. record_id sits in the log body only. EMF creates a separate custom metric for every unique dimension combination, so promoting a record ID to a dimension is how you turn a monitoring project into a billing incident. AWS says this plainly in the EMF documentation and it is worth reading twice before you ship.

Now the alarm. A normal CloudWatch alarm goes to INSUFFICIENT_DATA when the metric stops reporting, and INSUFFICIENT_DATA does not notify. That is precisely backwards for this failure. You want missing data treated as a breach:

aws cloudwatch put-metric-alarm 
  --alarm-name zoho-leads-notifications-silent 
  --namespace ZohoIntegration 
  --metric-name NotificationsReceived 
  --dimensions Name=Module,Value=Leads Name=Channel,Value=leads-prod 
  --statistic Sum 
  --period 3600 
  --evaluation-periods 2 
  --threshold 1 
  --comparison-operator LessThanThreshold 
  --treat-missing-data breaching 
  --alarm-actions arn:aws:sns:eu-west-1:111122223333:ops-alerts

The --treat-missing-data breaching flag is the entire point of that command. Without it, an integration that goes completely dead produces an alarm that sits quietly in an unknown state forever.

On the Loki side the equivalent is absent_over_time, which returns 1 when a stream has produced no entries in the window. Run it from the Loki ruler so it fires even when nobody has a browser open:

groups:
  - name: zoho-integration
    interval: 5m
    rules:
      - alert: ZohoNotificationsSilent
        expr: absent_over_time({job="zoho-webhook", module="leads"}[2h])
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "No Zoho notifications for Leads in 2h"
          description: "Channel may have expired. Re-subscribe and check channel_expiry."

One caveat worth knowing: absent_over_time only works for label combinations you name explicitly. It cannot tell you that a stream you never thought about has gone quiet. That is a real limitation, and it means your channel inventory has to live in your own configuration, not be discovered from Loki.

Because you cannot list channels through the API, treat renewal as a scheduled job with its own deadman alert, and emit ChannelSecondsRemaining at each renewal. Alarm when it drops below roughly a day. That gives you warning before the silence starts rather than after somebody notices.

Failure family two: throttling that is not rate limiting

Most engineers instrument Zoho as if it were a requests-per-minute API. It is not, and that mismatch produces dashboards that look fine right up until calls start failing.

Zoho CRM meters API usage in credits, not calls. Different operations deduct different amounts, and bulk operations are metered per batch of records rather than per request. The budget resets on a rolling twenty-four hour window measured from each call, not at midnight. If your dashboard has a daily reset at 00:00, it is drawing a line that does not exist.

The gauge that appears halfway through the tank

Here is the detail that makes credit monitoring awkward. Zoho returns remaining credits in an X-API-CREDITS-REMAINING response header, but only once usage has reached fifty percent or more of the daily allowance. Below that threshold the header simply is not there.

Read that again in monitoring terms. An absent header means either “you are comfortably under half your budget” or “your instrumentation is broken and you have no idea.” Those two states look identical, and one of them is fine while the other is a blind spot. So record both facts separately:

• A counter for responses where the header was present, which effectively tells you when you crossed the halfway mark.
• A gauge of the value itself, which only has meaning while the header exists.
• Your own client-side count of calls made, which works regardless and is the only thing you control.

The first crossing of that threshold is a useful early warning in its own right. Alert on the transition, not just on a low remaining value.

Concurrency is a separate ceiling

Alongside credits, Zoho applies a concurrency limit: the number of calls active simultaneously per org. There is also a sub-concurrency limit covering the expensive operations such as COQL queries, composite requests, lead conversion, and inserts or updates above a small record count. Exceeding either returns TOO_MANY_REQUESTS.

Two consequences matter for monitoring. First, since the limit is on simultaneous calls rather than calls per unit time, a requests-per-minute graph will never predict a concurrency failure. You need a gauge of in-flight requests, incremented before the call and decremented in a finally block. Second, the sub-concurrency ceiling is the same across every CRM edition, so upgrading the plan raises the general limit but does not buy headroom on the expensive calls. Batch those and queue them rather than fanning out.

One more source of consumption that surprises people: integration tasks inside Deluge functions consume credits from the same org budget as your external API calls. If your credit graph climbs while your service is idle, someone’s workflow rule is spending your budget. Zoho’s API Dashboard breaks consumption down by application and function, which is where to look before blaming your own code.

Failure family three: successful calls that did nothing

Bulk and multi-record operations return a per-record status inside the response body. The HTTP status can be perfectly healthy while individual records were rejected for validation rules, mandatory fields, duplicate checks or picklist values that do not exist in the target layout.

If your client only checks the HTTP status, you will log a success, increment a success counter, and move on. Weeks later somebody asks why the CRM is missing a slice of records that your logs insist were written.

The fix is to stop counting requests and start counting records, on both sides:

• Records submitted versus records confirmed in the per-record response status.
• A breakdown of rejection reasons as a low-cardinality label, so a spike in one specific validation error is visible.
• A periodic reconciliation job comparing source counts against Zoho counts for a bounded window, which is the only thing that catches drift you failed to anticipate.

In LogQL, the divergence between the two counters is the panel that actually earns its place on the wall:

sum(count_over_time({job="zoho-sync"} | json | status = "submitted" [1h]))
  -
sum(count_over_time({job="zoho-sync"} | json | status = "confirmed" [1h]))

A healthy integration holds that near zero. A number that grows steadily is drift, and it is visible weeks before anyone in sales notices.

Deciding which signal goes where

Running CloudWatch, Grafana and Loki together only helps if each one has a job. The split I reach for first:

• CloudWatch holds the numbers you alarm on, especially where the workload already runs in AWS and EMF gives you metrics for free out of log output.
• Loki holds the full request and response context: which record, which module, which validation error, which channel. This is what you actually read at three in the morning.
• Grafana is the single pane over both, with CloudWatch and Loki added as separate data sources so one dashboard can show the credit gauge next to the rejection reasons.

The trap that spans both backends is cardinality, and it bites in the same way for different reasons. In CloudWatch, each unique dimension combination becomes a billable custom metric. In Loki, each unique label combination becomes a separate stream, and Loki was designed for a small number of long-lived streams. Record IDs, email addresses and timestamps belong in neither.

Loki’s answer is structured metadata: high-cardinality fields attached to log lines without creating streams or entering the index. It needs enabling per tenant:

limits_config:
  allow_structured_metadata: true
  volume_enabled: true

Put module, environment and job in labels. Put the Zoho record ID and the correlation ID in structured metadata. Query with a label selector first and filter on the rest.

On collection, one scheduling note that does age: Promtail reached end of life on 2 March 2026, with its collection capabilities folded into Grafana Alloy. If you are still shipping through Promtail, that migration is now maintenance debt rather than an optional upgrade, and Grafana published a conversion tool for existing configurations. New pipelines should start on Alloy.

Where you run all this is a real decision. Self-hosting Loki on a VPS from a provider like Contabo or InterServer is cheap and entirely workable for a single integration’s log volume, and it keeps the data under your control. Grafana Cloud or Amazon Managed Grafana removes the operational surface if you would rather not babysit an ingester. Both are defensible. The honest trade-off is that self-hosted Loki is easy to stand up and less easy to keep healthy once retention and compaction start to matter.

Troubleshooting the monitoring itself

• EMF metrics never appear. The JSON must be a single line and structurally valid. A pretty-printed or truncated log line is ingested as an ordinary log event with no error, so check the raw event in the log stream rather than the metrics console.
• The alarm never fires despite obvious silence. Look at the alarm state. Sitting in INSUFFICIENT_DATA means the missing-data treatment was left at its default.
• absent_over_time returns nothing. That is the success case. It emits a value only when the stream is empty, which makes it feel broken while everything is working.
• Loki queries slow to a crawl. Check stream count before blaming resources. A label added “temporarily” during debugging is the usual cause.
• Webhook deliveries stop after a deployment. Check whether the subscription script reran and created a channel without an expiry. An hour later it will look identical to a network problem.
• Credit consumption does not match your call count. Expected. Operations deduct different credit amounts, and Deluge integration tasks draw from the same pool.

Common mistakes

• Alerting on error rate alone, which is blind to every failure described above.
• Creating notification channels without an explicit channel_expiry and inheriting the one-hour default.
• Assuming an absent X-API-CREDITS-REMAINING header means healthy usage rather than sub-threshold usage.
• Building a daily credit dashboard when the limit is a rolling window.
• Putting record IDs into EMF dimensions or Loki labels.
• Treating an HTTP 200 on a bulk write as confirmation that every record landed.
• Running alert rules only in the Grafana UI, so nothing evaluates if that instance is down.

Best practices for monitoring Zoho integrations

1. Start with a deadman alert per channel. If you build one thing, build this.
2. Set channel_expiry explicitly, renew on a schedule, and monitor the renewal job as carefully as the integration.
3. Count records, not requests, on both sides of every write.
4. Keep a client-side call counter so you are never dependent on a header that may not be sent.
5. Gauge in-flight requests to see concurrency pressure before it becomes TOO_MANY_REQUESTS.
6. Attach a correlation ID to every record as structured metadata so one identifier joins logs and metrics.
7. Evaluate critical rules server-side, in the Loki ruler or as CloudWatch alarms.
8. Test by disabling a channel in a sandbox org and confirming the alert fires. An untested deadman alert is decoration.

Frequently asked questions

Why did my Zoho webhook stop working on its own?

Almost always channel expiry. Zoho CRM notification channels last a maximum of one week, and if the expiry is omitted or set beyond that maximum, the channel defaults to one hour. Nothing is logged on your side because the deliveries simply stop arriving.

Can I list my active Zoho notification channels?

There is no endpoint for enumerating active channels, which is a long-standing complaint in Zoho’s developer community. Keep the inventory in your own configuration and treat the receipt of events as the only reliable liveness signal.

Do I need both CloudWatch and Loki?

No. Either can carry the whole job. The pairing makes sense when your integration already runs on AWS, because EMF gives you alarmable metrics from log output at no extra instrumentation cost, while Loki holds richer request context more cheaply. If you are starting fresh outside AWS, one backend is simpler and simpler is worth a lot.

How do I monitor Zoho API credits properly?

Record the X-API-CREDITS-REMAINING header when it appears, alert on the moment it first appears in a window, and maintain your own call counter as the fallback. Model the budget as a rolling twenty-four hour window rather than a daily reset.

What causes TOO_MANY_REQUESTS if I am under my credit limit?

Concurrency, not consumption. Credits and concurrency are independent ceilings. Expensive operations such as COQL queries and composite requests share a tighter sub-concurrency pool, so a burst of parallel queries can trip it while your credit balance is untouched.

Should I still use Promtail for shipping integration logs?

No. Promtail reached end of life on 2 March 2026 and its functionality moved into Grafana Alloy. Existing deployments will keep running, but they are unsupported, and Grafana provides a tool to convert Promtail configuration to Alloy.

How long should the deadman alert window be?

Base it on the quietest legitimate period for that module. A leads webhook that goes quiet overnight and at weekends needs a longer window than an order sync, or you will train the team to ignore it. Measure the real gap distribution for a fortnight before picking a number.

The one thing worth remembering

Monitoring Zoho integrations is mostly the discipline of detecting absence. Expired channels, exhausted credits and silently rejected records all present as nothing happening, and nothing happening is invisible to the instrumentation most teams reach for first.

So invert the default. Alarm when the count is too low rather than when the errors are too high. Set --treat-missing-data breaching. Use absent_over_time. Count records on both sides and watch the gap. If you build only one thing from this post, build the deadman alert on notification delivery, because that is the failure that will otherwise be reported to you by a salesperson wondering where the leads went.

Need help instrumenting a Zoho integration?

Most of the work here is judgement about what to measure rather than volume of code. If you are staring at a sync that keeps going quiet, this is the sort of thing I take on:

• Building deadman alerting for Zoho notification channels, including renewal jobs that monitor themselves.
• Instrumenting credit and concurrency headroom so throttling is visible before it becomes TOO_MANY_REQUESTS.
• Designing EMF dimensions and Loki label schemas that will not turn into a cardinality bill.
• Adding reconciliation jobs that compare record counts across systems and surface drift early.
• Standing up Loki and Grafana Alloy, or migrating an existing Promtail pipeline that is now unsupported.
• Reviewing an existing Grafana dashboard and cutting the panels that look reassuring but detect nothing.

Send me a sample webhook payload, a redacted response header set, or a screenshot of the dashboard you currently trust, and I will tell you what it would miss.

The post Monitoring Zoho Integrations: Catching the Failures That Never Throw an Error appeared first on John Nessime.

That is the failure worth writing about. Drift itself is normal. Somebody fixes production at 2am, a managed service rotates a value, a policy engine stamps a tag. The part that breaks is the alert, and it breaks quietly, because a muted channel looks identical to a quiet one.

Switching on Terraform drift detection takes one flag. Building Terraform drift detection that engineers still read six months later means deciding what not to send: which plan mode you run, what you classify as noise, how you route by blast radius, and whether the message names who made the change. A detector firing daily with the same content is worse than none, because no detector leaves you knowingly blind while a muted one leaves you blind with a green dashboard. Every section below asks the same question differently: does this message change what a human does in the next ten minutes?

Pick the right plan mode or you will alert on your own backlog

This is the biggest source of false drift, and it catches almost everyone building their first check. A normal terraform plan compares three things: your configuration, your state, and the real infrastructure. It returns a non-zero exit code when any of them disagree, so it fires for real drift and just as loudly for a pull request that merged this morning and has not been applied. Your drift channel becomes a notification feed for your own deployment backlog, and people learn that most messages are work in progress.

Refresh-only mode narrows the question. HashiCorp documents it as a plan whose goal is only to update state and root module outputs to match changes made to remote objects outside of Terraform. That is the actual definition of drift.

terraform init -input=false

terraform plan -refresh-only -detailed-exitcode -lock=false -input=false -no-color

# exit 0  state matches the real infrastructure
# exit 1  the check itself failed
# exit 2  something changed outside Terraform

Two flags there matter more than they look. -detailed-exitcode is what makes the result machine-readable at all: without it a successful plan returns zero whether or not anything changed, so your job passes forever. And -lock=false is safe here specifically because a plan never writes state. Only terraform apply -refresh-only updates the state file. A read-only detector that grabs the state lock will collide with real applies, and the loser of that race is the engineer trying to ship.

The trade-off, stated plainly: refresh-only will not tell you that merged code has not shipped, and it will not find resources created outside Terraform, because it can only refresh what is already in state. I run both modes and route them separately. Refresh-only feeds the alert. Normal plan feeds a weekly digest tied to the repository, where unapplied changes are a delivery problem rather than a security one.

Classify the noise before you route anything

Most recurring drift falls into a few families. Sorting yours takes an hour and decides most of your routing rules.

Provider normalization

A JSON policy document comes back with keys reordered. An identifier returns in different case. An empty block reads back as null instead of an empty map. Terraform sees a difference because it compares a stored representation against an API response, and that round trip is not always lossless. Fix these in the configuration where you can, because a phantom diff that follows you into every plan also hides real diffs on the same resource.

Attributes another system legitimately owns

The classic is autoscaling capacity. Your config says two instances, a scaling policy moved it to five during a spike, and the next plan proposes taking it back to two. Applying that kills three instances mid-spike. The drift is real; reverting it is the wrong move.

resource "aws_autoscaling_group" "web" {
  min_size         = 2
  max_size         = 10
  desired_capacity = 2

  lifecycle {
    # Owned by the scaling policy after creation.
    # Reverting this during a spike removes live capacity.
    ignore_changes = [desired_capacity]
  }
}

Understand what you bought. ignore_changes stops Terraform proposing an update for that attribute, and it applies only after creation, not on the first apply. It also delegates the attribute permanently. If someone sets desired capacity to zero at 3am, your check will not raise a hand. Fair trade for capacity. Terrible trade for an ingress rule.

Tags stamped by something else

Organization tag policies, cost allocation tooling, backup schedulers and compliance agents all add tags. The AWS provider exposes both the tags you set and the effective set including defaults, and ignoring the effective set silences all tag drift at once. Convenient, and slightly dangerous if you use tags for access control, because tag-based policy conditions stop being watched. The same logic covers values the platform rotates or defaults on its own, like a managed secret version: they drift on a schedule that has nothing to do with your team, and belong in a digest.

The goal here is not a clean plan. A clean plan is a side effect. The goal is a decision per family: see it immediately, see it weekly, or never see it again.

Route by blast radius, not by count

“Drift detected: 12 resources changed” makes a human open a terminal before they know whether to care, and that friction is where attention goes to die. Split the output by what each resource can do if it is wrong:

• Wake someone. Anything governing exposure or access: security groups and firewall rules, IAM policies and roles, key policies, public access blocks, bucket policies, DNS records pointing at production.
• Review today. Instance sizes, scaling bounds, database parameters, retention settings, anything that costs money or changes capacity.
• Weekly digest. Tags, descriptions, names, and every family you classified as platform noise.

To split them you need structure, not text. Terraform’s machine-readable output emits a message for each resource that changed outside Terraform, carrying the resource address and the action it would take.

terraform plan -refresh-only -no-color -json 
  | jq -c 'select(.type == "resource_drift") | .change'

Check the exact field names inside that object against the machine-readable output reference for your Terraform version before building matching logic on them. Keep the severity mapping in the repository next to the code it describes. A resource type moving from digest to page should be a pull request, not something changed in a monitoring UI at midnight.

Make the alert answer “who did this”

Terraform tells you what changed. It has no idea who changed it, and that gap is where the investigation time goes. Half the drift you find is a legitimate incident fix nobody wrote down. If the alert names the actor, the response is a two-word reply in the thread instead of twenty minutes of archaeology. On AWS the audit trail can be queried by resource:

aws cloudtrail lookup-events 
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0123456789abcdef0

CloudTrail’s console event history only reaches back a limited window, so a weekly check may be hunting an event that has aged out; a trail delivering to S3, or CloudTrail Lake, covers the gap. Azure exposes the equivalent through Activity Logs, Google Cloud through Cloud Audit Logs. Wiring the lookup into the alert body does more for adoption than any amount of formatting.

Decide the remediation path before you automate it

There are three answers to a drift finding, and picking the wrong one can be worse than the drift.

1. Revert. Apply the configuration and push reality back. Right when the change was unauthorized or wrong.
2. Adopt. Change the code to match what is running. Right when the change was correct and the code was stale, which happens more than people admit.
3. Tolerate. Suppress the attribute deliberately, with a comment and an owner. Right when another system owns it.

The trap is automating option one. Auto-approving a remediation apply sounds like the natural endpoint of a drift pipeline, and it is fine right up until the drifted attribute is one that forces replacement. Then “restore the desired state” means destroying and recreating a live resource, unattended, at whatever hour the schedule runs. If you automate anyway, gate it hard: an allowlist of resource types, a check that the planned action is an in-place update rather than a replace, and a notification either way.

Troubleshooting a drift job that quietly stopped working

The nastiest failure mode is not a noisy check. It is a check reporting “no drift” because it never ran the plan successfully.

Exit code 2 kills your script

With set -e, a non-zero exit terminates the script. Exit code 2 means drift, which is the case you care about, so a naive script dies exactly when it has something to report. Capture the code without letting the shell act on it:

set -euo pipefail

terraform plan -refresh-only -detailed-exitcode -lock=false 
  -input=false -no-color > plan.txt 2>&1 && code=0 || code=$?

case "$code" in
  0) echo "no drift" ;;
  2) notify_drift plan.txt ;;
  *) notify_broken plan.txt; exit 1 ;;
esac

Note that the failure branch notifies too. Exit code 1 deserves the same urgency as drift, because a broken detector and a clean estate produce identical silence.

Credentials that rot without complaining

Drift checks run unattended, so nobody notices when a role trust policy changes, an OIDC subject claim stops matching after a repository rename, or a long-lived key gets rotated out. Add a deadman check: if the job has not reported within its expected interval, that absence is the alert. Same reasoning as heartbeat monitoring for backups.

A provider upgrade that drifts everything

Bump a provider and a new attribute default can appear across every resource of a type. Overnight the check reports hundreds of changed resources and whoever is on call assumes the tool is broken, which it sort of is. Commit the dependency lock file, pin the provider version, run the drift job on the same Terraform version as your apply pipeline, and treat provider upgrades as a reviewed change rather than something a bot merges.

Refresh times that outgrow the schedule

Refresh queries the API for every resource in state. Large workspaces get slow, then overlap their own schedule, then time out. The durable fix is splitting state along boundaries that already exist in your team structure. Terraform does offer targeting to narrow a plan, but it is documented as an exceptional-use option and prints a warning for good reason, so treat it as a stopgap rather than the design.

When to stop building this and buy it

A scheduled job in GitHub Actions or GitLab CI, or a systemd timer on a small always-on box from a provider like Contabo or InterServer, covers a handful of workspaces perfectly well and costs almost nothing. I would not reach for a platform before that stops working. What eventually stops working is the surrounding machinery: credentials for a dozen accounts across providers, concurrency against real applies, and retaining history so you can see that the same resource has drifted repeatedly this quarter, which is a process problem no single alert reveals.

That is where HCP Terraform’s health assessments, or a platform like Spacelift, Scalr or env0, starts earning its fee, since drift detection becomes a workspace setting with history and notifications attached rather than code you maintain. Two honest caveats: HCP Terraform gates health assessments behind its paid editions and requires remote or agent execution, so it is not a drop-in for local-state workflows, and any platform will happily send you the same undifferentiated wall of drift the homegrown script did. Severity tiering is a decision about your infrastructure that no vendor can make for you. Buying the runner does not buy the judgement.

Common mistakes

• Using normal plan mode for the alert. You notify yourself about unapplied work and teach people the channel is noise.
• Alerting on a count instead of a resource. “Nine resources changed” makes someone open a terminal to find out if they should care.
• Treating exit code 1 as a non-event. A detector that errors is indistinguishable from an estate with no drift.
• Taking the state lock for a read-only check. Monitoring should not be able to block a deployment.
• Auto-approving remediation across all resource types. One force-replacement attribute and the fix becomes the outage.

Best practices worth the effort

• Run refresh-only for drift and normal plan for unapplied changes, routed to different destinations.
• Keep the severity mapping in the repository, version controlled and reviewed like code.
• Give every suppression a comment naming the system that owns the attribute and a date to revisit.
• Alert on the detector failing and on the detector going quiet, not only on findings.
• Review recurring drift monthly. The same resource drifting repeatedly is a process gap, not an infrastructure one.

Terraform drift detection FAQ

What is the difference between a normal plan and refresh-only?

A normal plan compares configuration, state and reality, so it flags genuine drift and any committed change that has not been applied. Refresh-only compares state against reality only, which matches what most people mean by drift. Use refresh-only for the alert and normal plan for a delivery-focused report.

Does a refresh-only plan modify my state file?

No. Plan operations do not write state. Only applying a refresh-only plan updates the state file to match the drifted remote objects, and even then it makes no changes to infrastructure. That is why a detection job can safely skip the state lock.

How often should Terraform drift detection run?

Tie the interval to how long you would tolerate not knowing. Daily for production and anything holding customer data, weekly for lower environments, less often for slow-moving workspaces where refresh is expensive. Frequency matters less than whether the message gets read.

How do I find resources created outside Terraform?

Not with a plan. Terraform refreshes only what is already in state, so a resource it has never seen is invisible. You need a cloud-side inventory such as AWS Config, Azure Resource Graph or Cloud Asset Inventory, compared against your state, and import blocks to bring the results under management. Worth knowing that driftctl, the open-source tool most often recommended here, has moved into maintenance mode.

Does ignore_changes hide drift I actually care about?

Yes, and that is the point of using it. It stops Terraform proposing updates for that attribute regardless of what changed it, so a deliberate suppression and a malicious change look identical afterwards. Scope it to specific attributes, never whole resources you still care about, and document the owner.

Why did drift suddenly appear across hundreds of resources?

Almost always a provider or Terraform version change rather than something happening in your cloud account. A new provider release can add or reshape an attribute and every resource of that type reports a difference at once. Compare the versions used by the drift runner and the apply pipeline before investigating the infrastructure.

The one thing worth keeping

Terraform drift detection is not a monitoring problem, it is an attention problem. The flag that finds drift took an afternoon. Everything determining whether anyone still reads the output next quarter is judgement: refresh-only rather than normal plan, noise families sorted and suppressed deliberately, severity by blast radius rather than resource count, an actor name in the message, and an alert when the detector itself goes dark.

Judge it by one test: when a message lands in that channel, does anyone open it? If the honest answer is no, your detection is working and your alert is not.

Need help making your drift alerts worth reading?

This is common work once a Terraform estate grows past one team. Things I can help with:

• Auditing an existing drift job and finding the false positives that trained everyone to ignore it
• Building scheduled refresh-only checks in GitHub Actions, GitLab CI or a systemd timer, with correct exit code handling and a deadman alert
• Sorting recurring drift into families and deciding, per attribute, what to fix, adopt or suppress
• Building severity routing from the plan’s structured output so exposure changes page and tag changes do not
• Enriching alerts with CloudTrail, Activity Log or Cloud Audit Log lookups so findings arrive with an actor attached
• Splitting oversized state so refresh finishes inside its schedule, and judging whether a managed platform is worth it at your size

If you want a second opinion, send me a week of your drift output or the workflow file that produces it, and I will tell you what I would cut first.

The post Terraform Drift Detection That Nobody Mutes appeared first on John Nessime.

That is the shape of most EKS upgrade incidents. The part everyone books a maintenance window for, the control plane bump, is the part AWS handles and almost never goes wrong. The damage happens in the data plane rollout, because of decisions made months earlier that nobody connected to upgrades.

This post covers the three failure families that actually break EKS version upgrades: removed APIs you did not know you were calling, add-on updates that silently discard your configuration, and pod disruption budgets that deadlock the drain. For each one: what it looks like, how to catch it beforehand, and what the escape hatch costs.

Why the control plane is the easy half

An EKS cluster has two versions that must stay compatible. AWS manages the control plane; you manage the data plane, which in practice means the kubelet version on your nodes.

Two constraints shape everything else. In-place upgrades move one minor version at a time, so being three versions behind means three sequential upgrades and three data plane rollouts. And the kubelet cannot run ahead of the API server, so once the control plane moves, the clock starts on getting nodes to follow.

Failure one: the API calls you did not know you were making

Deprecation and removal are different events, and the gap between them is where people get comfortable. An API version gets deprecated, everything keeps working, the warning scrolls past in CI, and several releases later the version is gone and whatever depended on it starts getting 404s.

Your own manifests are usually fine; someone fixed those when the warning appeared. What breaks is the third-party agent nobody owns: a metrics collector, an ingress controller, a Datadog or New Relic agent installed by a Helm chart pinned two years ago. These call the API on a watch loop, so they hit removed APIs immediately, and they tend to fail quietly rather than crash-loop.

Cluster insights, and what it is actually reading

aws eks list-insights --cluster-name my-cluster

aws eks describe-insight 
    --cluster-name my-cluster 
    --id a1b2c3d4-5678-90ab-cdef-EXAMPLE11111

EKS runs upgrade readiness checks and exposes them through the insights API. The deprecated API check reads your control plane audit logs on a thirty-day rolling window, scanned about once a day. That has a consequence people trip over constantly: after you fix a deprecated call, the insight stays in ERROR until the last matching log entry ages out. The work is done, the check still fails, and if you gate automation on insight status your pipeline is blocked for weeks on a solved problem. Force a fresh scan rather than waiting for the daily one:

aws eks start-insights-refresh --cluster-name my-cluster
aws eks describe-insights-refresh --cluster-name my-cluster

The refresh re-reads the window; it does not shorten it. Audit-log detection also only sees APIs that were actually called, so a monthly CronJob or a chart sitting unapplied in a repo never appears. Insights tells you what your cluster did, not what it could do.

Ask the API server directly

The API server keeps its own counter, and it names the release that removes each API. Fastest check available:

kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis

Read the removed_release label first: it tells you whether this is urgent for the next upgrade or a warning for the one after. Scrape the metric into Prometheus, Grafana Cloud or Amazon Managed Grafana and you get a standing signal instead of a scramble.

For the callers, go to the audit logs. With control plane audit logging on, CloudWatch Logs Insights gives you the identity behind each deprecated call:

fields @message
| filter `annotations.k8s.io/deprecated` = "true"

The user.username field is what you need. It usually points at a service account in a namespace you did not install, which is your cue to find the chart’s owner. Static scanners like kubent and pluto cover the other half, the manifests that have not run recently. Use both.

Failure two: add-ons and the conflict resolution you picked once

Managed add-ons cover what a cluster cannot run without: the VPC CNI, CoreDNS, kube-proxy, the EBS CSI driver. They are version-pinned separately from the cluster, and their compatible ranges move with the Kubernetes version. The failure here is rarely the version, though. It is the conflict resolution mode.

When you update an add-on, EKS wants to apply its own field values. If you edited those fields in-cluster, that is a conflict, resolved by what you pass:

• NONE leaves your value alone and the update may simply fail. Safe, loud, and it stops your upgrade.
• OVERWRITE reverts your customization to the EKS default. The update succeeds. Your custom Corefile, your CNI tuning, your tolerations, gone, with no error.
• PRESERVE keeps your value through the update.

OVERWRITE is the one that hurts, because it is also the one people reach for when an update fails and they want it to stop failing. It is the standard fix when migrating a self-managed add-on to a managed one, so it gets learned there and carried into the update path, where it means something different. A CoreDNS update with OVERWRITE will replace a Corefile carrying your stub zones and forwarders, and resolution for those zones stops cluster-wide while every health check stays green.

aws eks update-addon 
    --cluster-name my-cluster 
    --addon-name coredns 
    --addon-version <compatible-version> 
    --resolve-conflicts PRESERVE

One more trap: the same value means different things on create and on update. On create, PRESERVE behaves close to NONE, leaving an existing self-managed add-on alone rather than adopting your settings. Check the operation, not the word.

The durable fix is to stop editing add-on resources with kubectl at all. Put customizations in the add-on’s configuration values so EKS owns them and there is nothing left to conflict over. An afternoon of work, and this failure family goes away permanently.

Failure three: PDB deadlocks and the fifteen-minute drain clock

This is the one that stalls upgrades halfway, and the mechanics matter because the fix depends on where you are stuck. A managed node group update runs in four phases: setup, scale up, upgrade, scale down. On the default strategy, scale up launches replacements before anything is terminated, which is what preserves capacity. Then the upgrade phase does this, per node:

1. Pick a node to replace, up to the configured maxUnavailable.
2. Cordon it once a new node reaches Ready.
3. Drain the pods off it.
4. Wait sixty seconds after eviction, then ask the Auto Scaling group to terminate it.
5. Repeat until nothing is left on the old launch template version.

Step three has a fifteen minute budget. If pods have not left in that time and you did not pass the force flag, the whole update fails with PodEvictionFailure and you are left with a half-migrated node group.

The deadlock nobody sees coming

A PDB says how many pods must stay available during voluntary disruption, and eviction is voluntary disruption. So the arithmetic is simple and, once you see it, obvious: a deployment with two replicas and a budget requiring two available pods can never be evicted. Not slowly. Never. The drain retries until the clock runs out.

One replica with a one-pod minimum is the same deadlock in friendlier clothing, as is a percentage budget that rounds up to the full replica count. And if two PDBs select the same pod, the more restrictive one wins, producing deadlocks invisible in either manifest read alone. The pre-flight check is cheap, and I run it before every upgrade:

kubectl get pdb --all-namespaces

Read the ALLOWED DISRUPTIONS column, not the configured minimum. Any budget sitting at zero is a drain that will hang. Fix those before touching the node group, by raising replicas or loosening the budget so at least one disruption is permitted.

There is a second cause of PodEvictionFailure with nothing to do with PDBs. The upgrade taints nodes as unschedulable, so after eviction the node should be empty. Anything tolerating every taint keeps it non-empty, and the drain never satisfies its exit condition. If your budgets look fine and eviction still stalls, look at what tolerates NoSchedule unconditionally.

What the force flag actually costs

aws eks update-nodegroup-version 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup 
    --force

Force does not negotiate with budgets. It deletes the pods and terminates the node regardless. For a stateless service behind a load balancer with several replicas, that is fine and the upgrade completes. For a quorum-based system it is a bad idea: if the budget in front of your etcd, Consul, Kafka or ZooKeeper pods is doing its job, forcing past it can take the cluster below quorum. The budget was the protection; force is the decision to ignore it.

My rule is that force is reasonable on a node group I know holds only stateless workloads, and never a first response on a mixed one. Reaching for it because you do not know what is on the node is the actual problem.

If you run Karpenter, the levers are different

Karpenter has no update phases. It detects drift when a NodePool or EC2NodeClass stops matching a node’s configuration, then replaces the drifted nodes. Disruption budgets control the pace and can be scheduled, which is how you build a maintenance window without booking one:

spec:
  disruption:
    budgets:
      - nodes: "0"
        reasons: ["Drifted"]
        schedule: "0 6 * * mon-fri"
        duration: 12h

Two things bite people. Schedules are evaluated in UTC, so a window that looks like business hours to you may not be. And any period not covered by a budget is unrestricted, so gaps are not “no disruption”, they are “unlimited disruption”.

The other lever, terminationGracePeriod on the NodePool, overrides everything. Once it elapses, remaining pods are force-deleted regardless of PDBs or the karpenter.sh/do-not-disrupt annotation. It is a deadlock breaker, not a safety setting, and if it is shorter than your budget schedule you can get forced terminations inside a window you thought was protected.

The rollback window changes the calculation

Upstream Kubernetes has no control plane rollback. That single fact is why so many teams built heavyweight approval processes around upgrades. EKS now supports reverting an in-place control plane upgrade to the previous minor version within a seven day window, preserving etcd data and running workloads. It covers in-place upgrades only, which makes sense: with blue/green the old cluster is still there and rollback is a DNS change.

What changes is the shape of the risk. An upgrade used to be a one-way door, so the rational response was exhaustive testing beforehand. Now the first week is the test, under real traffic, with a way back. That argues for upgrading more often on smaller jumps, and treating the week after as an active watch period rather than a closed ticket.

It is not a general undo. Once the window closes, or you upgrade again, the previous version is gone, and it does not roll back add-ons or application state. Blue/green still has a place when you are several versions behind, since sequential in-place hops multiply exposure. The cost is running two clusters during migration, worth modelling first in Cost Explorer with both tagged, or in a tool like Vantage or CloudZero if you already run one.

Troubleshooting a stalled or failed upgrade

When a node group update fails, the reason comes back as an error code. Start there rather than guessing:

aws eks describe-nodegroup 
    --cluster-name my-cluster 
    --nodegroup-name my-nodegroup

• PodEvictionFailure. A budget blocked the drain, or something tolerates all taints. Check ALLOWED DISRUPTIONS first, then tolerations.
• NodeCreationFailure in scale up. New nodes never reached Ready. Usually no capacity for your instance type in an Availability Zone, an EC2 instance limit in the account, or custom user data breaking the bootstrap.
• Slow bootstrap. A node has fifteen minutes to join, measured from when the Auto Scaling group size increases. Heavy user data, large images pulled at boot or a slow init path push past it.
• Pods stuck terminating with no budget in sight. Look for blocked I/O. A pod with an NFS mount whose server is unreachable will not terminate, and eviction pressure changes nothing.
• Insights still in ERROR after the fix. The thirty day audit window. Refresh, confirm another way, move on.

One structural note: if the Cluster Autoscaler scales the node group during the scale down phase, the upgrade workflow exits without restoring the group to its original size. Check your desired and max counts afterwards.

Common mistakes

• Treating insights as the only pre-flight check. It reads audit logs, so it cannot see code paths that have not run.
• Reaching for OVERWRITE to make a failing add-on update succeed. It succeeds by discarding your configuration.
• Upgrading add-ons and the control plane in one unbroken pipeline run, so you cannot tell which change broke things.
• Leaving maxUnavailable at its default of one on a large node group, then wondering why the rollout takes hours.
• Forgetting self-managed components. Cluster Autoscaler, the AWS Load Balancer Controller and metrics-server are versioned against Kubernetes too, and nothing reminds you.

How I sequence EKS version upgrades

1. Read the release notes for the target version. Not the summary post, the removals section.
2. Run all three deprecation checks: insights for what the cluster did, the API server metric for what it does now, a static scanner for what sits in the repo.
3. Audit PDBs and fix every zero. Highest-value ten minutes in the process.
4. Update add-ons with PRESERVE to versions compatible with both the current and target Kubernetes version, so components talking to the API server are ready before it moves.
5. Upgrade the control plane and stop. Watch error rates and API server latency long enough to mean something.
6. Roll the least critical node group. Confirm pods reschedule cleanly and nothing depended on a node label or an AMI-specific detail.
7. Roll the rest, confirm every kubelet matches the control plane, then treat the following week as the real test. That is what the rollback window is for.

FAQ

Can you do EKS version upgrades with zero downtime?

For stateless workloads with sensible replica counts, spread across zones, with budgets allowing at least one disruption, yes. The default node group strategy launches replacements before terminating anything, so capacity holds throughout. Stateful workloads can too, but only if the application handles rolling leadership changes and the budget permits exactly one disruption at a time.

Why does the update fail with PodEvictionFailure when my PDBs look fine?

Two likely causes. Multiple budgets may select the same pod, in which case the most restrictive applies and neither manifest reveals it alone. Or something on the node tolerates every taint, so the node never becomes empty and the drain never completes. Check ALLOWED DISRUPTIONS across all namespaces, then check tolerations.

Is the force update option safe?

Safe for stateless workloads with multiple replicas. Not safe for quorum-based systems, because it deletes pods regardless of the budget protecting quorum. Use it when you know what is on the node group and the answer is nothing that minds being killed.

Does Karpenter make upgrades easier than managed node groups?

More automatic, which is not the same thing. Drift detection replaces outdated nodes without you running a command, but you inherit a different set of controls to get wrong: budgets evaluated in UTC, uncovered schedule gaps that mean unlimited disruption, and a grace period that overrides PDBs entirely. Neither is easier. They fail differently.

The one thing worth remembering

EKS version upgrades do not fail at the moment you run the command. They fail on decisions already sitting in the cluster: a budget written for a two-replica deployment, an add-on customized with kubectl rather than through its configuration values, a chart pinned before an API was removed.

Which means the pre-flight checks are the upgrade. Ten minutes reading ALLOWED DISRUPTIONS and the deprecated API metric tells you more about how tomorrow goes than any amount of care during the rollout. The rollback window gives you a second chance on the control plane. It will not un-delete a pod that force-eviction pulled out of a quorum.

Need a second pair of eyes on your upgrade path?

Most of this work is unglamorous and specific. What I would actually do on a cluster facing a version bump:

• Audit every PDB against its replica count and flag the ones that will deadlock a drain.
• Trace deprecated API calls back to the chart, controller or service account making them, from audit logs rather than guesswork.
• Move add-on customizations into managed configuration values, so conflict resolution stops being a decision made under pressure.
• Write a repeatable upgrade runbook: pre-flight checks, sequencing, and the rollback path for your specific workloads.
• Alert on the deprecated API metric and node group update events, so the next version is a scheduled task rather than an incident.
• Review Karpenter disruption budgets and grace periods for the gaps that turn into unplanned rollouts.

If you are staring at a failed node group update right now, send me the output of kubectl get pdb --all-namespaces and the error code from describe-nodegroup. That is usually enough to say what is holding the drain.

The post EKS Version Upgrades Without a Maintenance Window: APIs, Add-Ons and PDB Deadlocks appeared first on John Nessime.

That gap is where the real cost lives, and it is almost never on anyone’s dashboard. The build duration graph looks fine. The team still says shipping is slow. Somebody proposes buying bigger runners, and six weeks later the graph has moved a little and the complaints are identical.

Most CI/CD pipeline bottlenecks are not the thing people go looking for. Compute is usually the smallest slice. The expensive parts are minutes spent waiting for a runner, a cache that reports a hit and delivers nothing, a stage blocking four jobs that do not depend on it, and a required check sitting at “Pending” until someone notices. This post is organized by those failure families rather than by tool. Examples use GitHub Actions and GitLab CI, but the shapes apply to Jenkins, Buildkite, CircleCI and Argo Workflows too.

First, split the number you are staring at

“The build takes eleven minutes” is not a measurement. It is three measurements added together and then thrown away:

• Queue time. From job creation to a runner picking it up. You pay this in wall clock, not compute.
• Compute time. The job actually running. The only part most dashboards show.
• Gate time. Waiting on a human, an approval, a shared environment, or a check that never fired.

Until you split those, every optimization is a guess. On GitHub Actions the jobs API hands you the split directly, because each job carries a creation timestamp and a start timestamp:

gh api repos/OWNER/REPO/actions/runs/RUN_ID/jobs 
  --jq '.jobs[] | {name, created_at, started_at, completed_at}'

Creation to start is queue. Start to completion is compute. Run that across your last fifty runs on the default branch and the answer is usually sitting there in plain sight. GitLab reports the same idea on its jobs API, with a queued duration alongside the run duration. If the two numbers are roughly equal, you have a capacity problem rather than a build problem, and faster CPUs will fix half of nothing.

Queue time: the minutes you pay for twice

Queue time has two causes and they need different fixes.

Not enough runners

On hosted runners you are bounded by the concurrency your plan allows. On self-hosted you are bounded by whatever fleet you provisioned, which for most small teams is a fixed number of always-on boxes that sit idle overnight and saturate at 4pm.

If you run your own fleet on Kubernetes, Actions Runner Controller is where most people land. It creates runner scale sets that scale on workflow demand and can drop to zero when the queue is empty, with ephemeral pods so every job starts clean. That last part matters more than the scaling: a persistent runner accumulates state, and state is where “works on my machine” comes back as a CI flake.

If Kubernetes is more machinery than you want, a couple of dedicated boxes from a provider like Contabo or InterServer running the runner agent under systemd gets you a long way, with hosted runners as overflow when the queue backs up. Managed runner services such as Depot and BuildJet sell faster hardware behind the same interface. Their honest pitch is that you stop maintaining a fleet. The honest counterpoint is another vendor in the path of every merge, and a bill that scales with commit volume.

Runners busy on work that is already obsolete

Check this one first because it is free. Push three commits to a branch in five minutes and by default you get three pipelines. The first two are answering a question nobody is asking anymore, and they are holding the runners the third one needs.

On GitHub Actions, a concurrency group keyed on the ref does it:

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

Put that at workflow level on PR builds. Do not put it on a deploy workflow with cancellation enabled, or you will eventually kill a release halfway through and spend an afternoon working out what actually shipped.

GitLab splits the same job across two settings that have to agree. The interruptible keyword marks a job as safe to kill and defaults to false. The workflow-level auto-cancel setting decides what happens on a new commit:

workflow:
  auto_cancel:
    on_new_commit: interruptible

test:
  interruptible: true
  script:
    - make test

deploy:
  interruptible: false
  script:
    - make deploy

The default mode is conservative, which means once any non-interruptible job has started, the whole pipeline stops being cancelable. Setting it to interruptible cancels only the jobs that opted in and leaves the rest running. That distinction is the difference between a policy that works and one that quietly does nothing, and it is the usual reason teams turn auto-cancel on and see no change.

Caches that look like they are working

My favorite family of CI/CD pipeline bottlenecks, because the failure is completely silent. The step is green. The log says the cache was restored. The install takes exactly as long as before.

Branch scoping

GitHub Actions caches are scoped to the branch that created them, plus the default branch as a fallback. A feature branch reads its own caches and the default branch’s caches. It cannot read a sibling branch’s cache.

The consequence catches a lot of people: if your caching step only runs on pull requests, the default branch never populates a cache, so every new branch starts cold forever. Make sure at least one workflow on the default branch writes it. Runs triggered from fork pull requests get read-only cache access, so those jobs restore but never save.

Keys that never hit, or hit and never update

Cache entries are immutable. Once a key exists, writing to it again does nothing. So a key like node-modules-v1 gets populated once, on some machine, in some state, and then freezes. A key that hashes the lockfile is correct but creates a brand new entry on every dependency change, which burns through storage.

The pattern that survives contact is an exact key plus a prefix fallback: hash the lockfile for the primary key, then use restore keys to fall back to the most recent entry sharing that prefix. You get a partial restore and a fast install instead of a cold one.

Then watch the ceiling. Repositories have a default cache storage limit, entries untouched for a week are removed, and once you hit the limit the least recently accessed entries get evicted. Push enough large caches and you get thrashing, where entries are created and evicted at roughly the rate you create them. The tell is a hit rate that looks random. Check the repository’s cache usage in Actions settings before assuming the config is wrong.

Docker layer cache that does not cross machines

On ephemeral runners the local BuildKit cache dies with the machine, so it has to be exported somewhere. Registry-backed cache is the one I reach for first:

docker buildx build 
  --cache-from type=registry,ref=registry.example.com/app:buildcache 
  --cache-to type=registry,ref=registry.example.com/app:buildcache,mode=max 
  --tag registry.example.com/app:${GIT_SHA} 
  --push .

Two details do the work. mode=max exports intermediate stage layers, not just the ones that landed in the final image. If the expensive work happens in a builder stage that gets discarded, the default mode caches none of it, which is exactly the case you were trying to fix. Inline cache, the kind that ships alongside the image, only supports the minimal mode, so it cannot help multi-stage builds either.

The second detail is a trap. BuildKit cache mounts, the RUN --mount=type=cache kind, are not part of the exported layer cache. They live on the builder. Restoring a registry cache does not restore them, so if the instruction runs again the package manager starts from an empty download cache:

# syntax=docker/dockerfile:1
FROM node:22-alpine
WORKDIR /app
COPY package*.json ./
RUN --mount=type=cache,target=/root/.npm npm ci

That line is excellent on a persistent builder and close to useless on a runner destroyed after every job. Know which one you have before you credit it with a speedup.

Pipelines shaped like a staircase

Stages are a scheduling constraint most pipelines inherit by accident. In a stage-based model nothing in the test stage starts until everything in the build stage finishes. Put a forty-second lint job in the same stage as a six-minute compile and your lint feedback arrives six minutes late for no reason.

GitLab’s fix is needs, which lets a job start as soon as its own dependencies finish and ignore stage order entirely:

lint:
  stage: test
  needs: []
  script:
    - make lint

unit:
  stage: test
  needs: ["build"]
  script:
    - make unit

needs: [] means start immediately, wait for nothing. Lint now reports in under a minute while the compile is still running.

GitHub Actions has the opposite default: jobs run in parallel unless you add needs. So the equivalent problem there is over-declared dependencies, usually added early to force a readable graph and never revisited. Go read your needs lines and ask which are real data dependencies and which are narrative ordering.

The trade-off is real. A wide graph surfaces every failure at once and costs more compute per run. A narrow graph fails fast and saves minutes, but developers make several round trips to find three problems. For PR pipelines, wide is usually right. For expensive nightly jobs, narrow is.

The test suite is not one bottleneck

“Tests are slow” gets treated as a single problem. It is usually three, and they want different fixes.

• The suite is serial. Split by shard across parallel jobs. The cost is per-job setup overhead paid N times, so past a point more shards makes things slower. Find your own break-even instead of assuming more is better.
• The suite is flaky. A retry costs more than one rerun. It costs attention, a context switch, and eventually the team’s belief that red means anything. Quarantine flaky tests into a non-blocking job with an owner and a deadline. Leaving them in the required path while everyone knows to just re-run is worse than deleting them.
• The suite tests the wrong layer. A test that boots a database and a message broker to assert a validation rule is a unit test wearing a costume. Slowest fix, largest payoff.

Before touching any of it, get per-test timing out of your test runner and sort descending. Most suites have a handful of tests that dominate, and they are rarely the ones people suspect.

The wait that nobody is measuring

Here is the one that costs days rather than minutes, and it appears in no CI metric because from CI’s point of view nothing is happening.

If a workflow is skipped because of path filtering, branch filtering, or a skip directive in the commit message, its checks stay pending. Not skipped. Pending. And if that check is required, the pull request cannot merge. Nothing fails, nothing alerts, the PR just sits there looking almost ready until someone investigates by hand.

This bites hardest in monorepos, which is exactly where path filters are most attractive. You add paths to skip the service test suite on documentation-only changes, you save real minutes, and you accidentally build a merge trap.

The distinction that matters: a job skipped by an if condition inside a workflow that did run reports success. A workflow that never ran at all reports pending. So either keep the workflow triggering on everything and move the filtering inside, or pair each filtered workflow with a companion using the same name and inverted path filters that does nothing and passes. GitHub documents that second pattern and it works, but it means two files that must stay in sync.

The other gate-time offenders are less subtle and just as expensive: one shared staging environment serializing every team, and approvals routed to a single person asleep in another time zone. Plot merge timestamps against last-commit timestamps for a month. The shape of that distribution tells you which problem you have.

Troubleshooting: finding your bottleneck in an afternoon

1. Pull the last fifty runs on your default branch and compute queue time and compute time separately. If queue is a meaningful share, fix capacity and cancellation before anything else.
2. Read your slowest run as a graph, not a list. Wide gaps between a dependency finishing and its dependent starting are either queue or bad ordering.
3. Compare install duration on a cache hit against a cache miss. If they are similar, the cache is decorative.
4. Check repository cache storage usage. Thrashing looks like an erratic hit rate with no config changes behind it.
5. Count retries. The rate of second attempts on the same commit is your flakiness tax, and it is usually higher than anyone guesses.
6. Measure merge latency, last commit to merge, across a month. The difference between that and pipeline duration is gate time, and that is where the days are.

Pipeline run data is worth shipping somewhere queryable. Most CI platforms emit webhooks on job completion; feeding those into Prometheus and Grafana, or a hosted platform like Grafana Cloud or Datadog if you would rather not run the storage, turns “the build feels slow lately” into a graph you can argue with.

Common mistakes

• Buying bigger runners before splitting queue time from compute time. The most common wasted spend in CI.
• Turning on auto-cancel without setting the per-job interruptible flags, so the setting is technically on and functionally off.
• Caching a directory the tool does not read from, then declaring victory because the step went green.
• Using the default Docker cache export mode on a multi-stage build and wondering why the builder stage rebuilds every time.
• Adding path filters to workflows that back required checks, creating pull requests that can never merge.
• Optimizing the pipeline that runs on merge instead of the PR pipeline developers actually wait on.

Practices that hold up

• Treat queue time as a first-class metric with its own threshold, not a footnote on build duration.
• Cancel superseded runs on branches, never on the default branch and never on deploys.
• Make the default branch populate every cache other branches depend on.
• Order jobs by how fast they fail, not by how the file reads top to bottom. Cheap checks first, always unblocked.
• Give every flaky test an owner and an expiry date, or take it out of the blocking path.
• Re-measure after every change and keep the number. Pipelines rot, and the bottleneck moves as soon as you fix the current one.

Frequently asked questions

How do I find the bottleneck in my CI/CD pipeline?

Split every job into queue time and compute time using the timestamps your platform’s API already exposes, then compare pipeline duration against real merge latency. Those two comparisons put you in one of three buckets: capacity, build work, or gates. Fixing the wrong bucket is why most optimization efforts show no result.

Are self-hosted runners faster than hosted ones?

Per job, often yes, because you pick the hardware and can keep caches warm. Across a working day, frequently no, because a fixed fleet queues under load while a hosted pool does not. The right answer depends on which number is hurting you, compute or queue, which is why measurement comes first. A hybrid setup, self-hosted by default with hosted overflow, avoids committing to either.

Why is my cache not speeding anything up?

Usual suspects in order of frequency: the cache is scoped to a branch that never populated it, the key is static so it froze at whatever went in first, the cached directory is not the one the tool reads, or entries are evicted before the next run. Compare install duration on a hit against a miss. Identical duration means the restore is doing nothing regardless of what the log says.

Should I parallelize tests or make them faster?

Parallelize first because it is cheap and reversible, then fix the slow tests, because parallelism has a floor. Every shard pays setup cost, so a suite split too many ways spends more time booting containers than running assertions. Sharding buys time to do the real work. It is not the real work.

Why does my pull request say pending forever?

Almost certainly a required status check attached to a workflow that was skipped by path filtering, branch filtering, or a commit message directive. A skipped workflow reports pending rather than skipped, so the merge requirement is never satisfied. Either drop the path filter and skip inside the workflow instead, or add a companion workflow with the same name that always passes.

Do bigger runners actually help?

They help work that is genuinely CPU or memory bound, such as compilation and container builds. They do nothing for queue time, network-bound dependency downloads, tests waiting on external services, or gate time. Before upgrading, confirm the slow job is saturating the resource you are about to buy more of.

The one thing worth remembering

Most CI/CD pipeline bottlenecks are waiting, not working. Queue time, cache misses that report as hits, jobs blocked behind stages they do not depend on, and checks that sit pending because a workflow was never triggered. None of them show up on a build duration graph, and all of them are cheap to fix once you can see them. Split the number before you spend anything; the measurement takes an afternoon and usually points at a config change rather than a purchase order.

Need someone to find where your pipeline is losing time?

This is a large part of what I do as a freelance DevOps engineer. Typical work looks like:

• Instrumenting your pipeline so queue time, compute time and merge latency become three separate numbers you can watch over time
• Auditing GitHub Actions and GitLab CI caching to find the restores that report success and deliver nothing
• Restructuring stage-based pipelines into dependency graphs so cheap checks report first
• Setting up autoscaling self-hosted runners, on Kubernetes with Actions Runner Controller or on plain VMs, with hosted overflow
• Fixing Docker build caching for multi-stage images on ephemeral runners, including registry-backed export
• Untangling required checks and path filters in monorepos so pull requests stop getting stuck

Send me a workflow file, a job timeline, or the output of your slowest run, and I will tell you where the minutes are going before we talk about anything else.

The post CI/CD Pipeline Bottlenecks: Where Your Build Minutes Actually Go appeared first on John Nessime.

It granted iam:PassRole on one role ARN and lambda:CreateFunction in the same document. That is not a wildcard and it does not read as administrative access. It is administrative access anyway, to whatever that role can do, for anyone who can write a Lambda function.

This is the shape of the problem with AI-generated IAM policies. The failure is almost never the obvious one. Models have absorbed enough style guidance to avoid "Action": "*", so what comes back passes the eyeball test and the linter and still hands over more than you meant to give. The dangerous output is the one that looks careful.

This post covers what policy generation genuinely does well, the failure shapes that survive human review, and where the hard boundary belongs: not in the review step, which is fallible, but in a mechanism that cannot be talked out of its answer.

What AI-generated IAM policies get right

Start with the honest case, because it is a strong one. IAM has thousands of actions across hundreds of services and the naming is inconsistent in ways nobody holds in their head. Some services use Describe, some use Get, some use both for different things. ARN formats vary per service and per resource type inside a service. Condition keys apply to some actions and not others.

A model is good at that recall problem, and at shape work: splitting a monolithic statement into per-resource statements, adding conditions, converting an inline policy into Terraform. Paste an AccessDenied message and you get a plausible read of which layer refused. So the argument is not that the tooling is useless. It is that recall and reasoning about consequences are different capabilities, and IAM is a system where the second one protects you.

The failure modes that survive review

Permissions that compose into something bigger

Every action is individually defensible. The escalation lives in the combination.

The canonical case is iam:PassRole beside any service that runs code: Lambda, EC2, ECS, Glue. Passing a role to a compute service means your code executes with that role’s permissions, so your ceiling is not your own policy, it is whatever the most powerful passable role can do. Rhino Security Labs catalogued a long list of these chains and every one of them is built from permissions that look unremarkable alone.

Another that catches people: iam:CreatePolicyVersion on a policy you are attached to. A new version normally needs to be set as default to take effect, but the create call accepts a flag that makes it default immediately, and that flag does not require iam:SetDefaultPolicyVersion. A permission that reads as “can update policy documents” reads to an attacker as “can write myself an admin policy.”

Ask a model whether a policy grants admin and it will check the statements and say no, correctly, one at a time. Composition is the part it does not reliably do, and it is the part that matters.

Actions that do not exist

Plausible action names that no service publishes. s3:ListBucketContents instead of s3:ListBucket. IAM accepts a policy containing an unrecognized action string because the document is syntactically valid, so nothing fails at attach time.

What happens instead is worse. The application throws AccessDenied in production, somebody debugging under pressure decides the granular list must be wrong, and the fix is a service wildcard. The invented action does not grant too much. It causes a human to grant too much a week later, and by then nobody connects the two events.

Resource ARNs that look constrained and are not

An ARN with real characters in it feels safer than a star. Often it is not. The common versions are a trailing wildcard placed one segment too high, and an action whose resource element the service ignores. s3:ListAllMyBuckets is account-scoped no matter what you put in Resource. Several ec2:Describe actions behave the same way. A policy can read as resource-scoped throughout and be account-wide for the actions that count.

The related trap is the missing condition: cross-account trust policies without an external ID, bucket policies without aws:PrincipalOrgID, integration roles without aws:SourceIp where the caller has a stable egress address. Generated policies include conditions when you ask and omit them when you do not, because an absent constraint never produces an error.

Four things the model cannot know

Some of this is not a capability gap you can prompt past. It is missing input:

• Which resources exist. A bucket name is a guess unless you supplied it. Guessed names either fail closed or, if the wildcard is loose enough, match something you never intended.
• Which resources are sensitive. Your backup bucket and your public assets bucket are indistinguishable strings. Sensitivity lives nowhere in the document.
• What else is attached. Effective permissions come from identity policies, resource policies, boundaries, SCPs and session policies together. Reviewing one document tells you very little.
• What the blast radius is. “Can delete objects” means one thing for a scratch bucket and another for the only copy of a client’s data.

The training material is also skewed. Public IAM examples, including plenty of vendor documentation and countless forum answers, are permissive because permissive examples work on the first try. A model that has read the internet has read a lot of over-broad policies presented as correct.

Where the hard boundary belongs

Review is not a boundary. Review is a filter with a pass rate that drops the more policies you look at in a week, and generated policies are cheap to produce, so volume rises exactly when attention falls.

The boundary has to be something that says no without reading the document. AWS gives you two mechanisms and they are not interchangeable.

A permissions boundary is a managed policy attached to a user or role that caps what identity-based policies can grant it. It grants nothing itself. Effective permissions become the intersection of the identity policy and the boundary, so an identity policy allowing iam:* against a boundary that omits IAM produces no IAM access at all. If no boundary is attached, none is evaluated, so this is opt-in per identity.

A service control policy sets the maximum available permissions for every principal in an account or organizational unit, and also grants nothing. It is the right home for things nobody in that account should ever do, and it applies whether or not somebody remembered to attach a boundary.

The division I use: SCPs carry account-wide invariants, such as denying escalation-relevant IAM write actions to everyone outside the break-glass role and denying regions you do not operate in. Boundaries carry the per-workload ceiling, and they are what makes it safe to let a pipeline role create other roles. You can permit that role to create roles while requiring, through a condition on the boundary key, that every role it creates carries your boundary. The pipeline can then generate whatever policy it likes, because the ceiling is enforced at evaluation time rather than at review time.

If the only thing between a generated policy and production is a person reading JSON, you do not have a boundary. You have a habit.

The gate I would actually run

Mechanical checks in a pipeline, failing the build, not a checklist somebody is supposed to remember.

1. Lint for nonsense. Parliament catches unknown action names, resource formats that cannot match the action, and type mismatches. Cheapest way to catch an invented action before it becomes a wildcard.
2. Run AWS policy validation. Access Analyzer knows service-specific rules a generic linter does not.
3. Assert the escalation actions are absent. Name the actions you never want granted and fail if any appear. This is the check that maps to the composition failure.
4. Compare against the current policy. For edits, check whether the proposed document grants access the existing one does not. This one is billed per call, so run it on changed policies rather than the whole repository.
5. Scan for known risky patterns. Cloudsplaining scores policies against categories including privilege escalation, data exfiltration and resource exposure, and produces something you can hand to a client.

# 1. Lint. Exit status is non-zero when there are findings.
parliament --file policy.json

# 2. AWS-side validation
aws accessanalyzer validate-policy 
  --policy-document file://policy.json 
  --policy-type IDENTITY_POLICY

# 3. Fail if any escalation-relevant action is granted
aws accessanalyzer check-access-not-granted 
  --policy-document file://policy.json 
  --access actions="iam:PassRole","iam:CreatePolicyVersion","iam:AttachRolePolicy" 
  --policy-type IDENTITY_POLICY

# 4. Does the proposed policy grant more than the current one?
aws accessanalyzer check-no-new-access 
  --existing-policy-document file://current.json 
  --new-policy-document file://proposed.json 
  --policy-type IDENTITY_POLICY

# 5. Risk-categorised report
cloudsplaining scan-policy-file --input-file policy.json

Step three is the one to add first if you only add one. It turns “we reviewed it” into “the build fails.”

Attaching the boundary is a single call, and it is the part people skip because the role already works without it:

aws iam put-role-permissions-boundary 
  --role-name app-deploy-role 
  --permissions-boundary arn:aws:iam::123456789012:policy/WorkloadCeiling

Generate from observed behavior instead

There is a better source of truth than a model’s guess about what an application needs, and it is the application. Access Analyzer policy generation reads CloudTrail events for a role over a chosen window and writes a policy from what was actually called. Know its limits before relying on it:

• It analyzes up to ninety days of history and needs a trail already logging for the account, so a role that only exercises certain paths quarterly produces an incomplete policy.
• Coverage is per-service. For some services it identifies individual actions; for others it can only tell you the service was used and prompts you to fill in the actions yourself.
• It does not produce action-level detail for data events such as S3 object-level operations.
• iam:PassRole is not included in generated policies. If your workload needs it, a human adds it back by hand, so the most escalation-relevant line in the document is the one the generator did not write.

Behavior-derived policies also fail in a useful direction. A missing permission produces a denial you can see in CloudTrail, which beats a permission you did not know you granted.

Arguments that do not survive contact

• “We review every policy before merge.” Review quality is a function of volume and attention, and generation raises volume. Composition failures are also the hardest to spot by reading, because spotting them means knowing what other roles exist and what they can do.
• “Access Analyzer came back clean.” Validation checks the document. It does not know that the role ARN you may pass happens to have AdministratorAccess attached. Clean findings on a policy that grants a dangerous chain is the normal case, not an anomaly.
• “A permissions boundary is overkill for a small project.” Small projects are where this bites hardest. One account, one deploy role, no organization, nothing between a bad policy and everything you own. The boundary is one managed policy and one CLI call.
• “We will tighten it after launch.” Nobody does. If you genuinely intend to, drive it from Access Analyzer’s unused access findings so you have a work queue instead of an intention.
• “Just prompt the model to follow least privilege.” That changes the output surface, not the reasoning. Fewer wildcards, better statements, no evaluation of what the combination permits.

How I would decide

My rule is about which actions are in the document, not about who or what wrote it.

Ship generated policies freely when the document touches only data-plane actions on named resources, the identity has a boundary, and the pipeline checks pass. Read a bucket prefix, publish to a topic, write to a table. That is most policies, and hand-writing them wastes your afternoon.

Treat generated output as a first draft when the policy includes any IAM write action, any PassRole, any trust policy edit, any resource policy readable from outside the account, or any KMS key policy. Use it to save typing, then verify every statement against the service authorization reference yourself.

Do not generate at all for the boundary policies and SCPs. Those are the mechanism. Something that constrains everything else should be short, hand-written, understood by whoever owns the account, and changed rarely.

Two notes for small engagements. If you run application infrastructure on a VPS from a provider like Contabo or InterServer and pull AWS in for storage or mail, that server holds long-lived credentials somewhere your AWS controls do not reach, so scope its policy to a bucket prefix and attach a boundary before worrying about anything else. And if the credentials go to a SaaS connector, a marketing platform such as GoHighLevel or a payments integration, you are handing keys to a system whose behavior you cannot audit. Those are the policies to write by hand.

Frequently asked questions

Are AI-generated IAM policies safe to use in production?

For data-plane permissions on named resources, generally yes, provided the identity carries a permissions boundary and the policy passes automated validation. For anything involving IAM write actions, PassRole, trust policies or key policies, treat the output as a draft and verify each statement yourself. What matters is the blast radius of the actions, not the authorship.

Why does a generated policy pass validation and still grant too much?

Validation examines one document in isolation. Privilege escalation usually comes from a combination of individually reasonable permissions, or from the interaction between the policy and something outside it, such as which roles are passable and what those roles can do. A document can be internally correct and still open a path.

What is the difference between a permissions boundary and an SCP?

Both cap permissions and neither grants any. A boundary attaches to a single IAM user or role and limits what that identity’s policies can grant it. An SCP applies to every principal in an account or organizational unit and needs AWS Organizations. With one account and no organization, boundaries are what you have. With an organization, use both.

Can IAM Access Analyzer replace writing policies by hand?

Partly. Generation from CloudTrail activity beats a guess because it reflects what the workload actually called. It is limited by the analysis window, by per-service coverage differences, by the absence of action-level detail for data events, and by iam:PassRole being excluded. Treat it as a strong draft that needs a human pass over anything the generator could not see.

Which IAM actions should I block outright in a small account?

The ones that let a principal rewrite its own permissions or borrow another identity’s: creating and setting default policy versions, attaching and putting inline policies on users, roles and groups, creating access keys for other users, and passing roles to compute services. Deny them for everything except a break-glass identity, then grant them back deliberately where a workload genuinely needs one.

How do I catch an invented action name before production?

Run a linter that carries the IAM action catalogue. Parliament flags unknown actions and exits non-zero when it finds anything, so it drops straight into a pipeline step. Catching it early matters less because the fake action is dangerous and more because the eventual fix for the denial tends to be a service-level wildcard.

The one thing worth remembering

AI-generated IAM policies are not more dangerous because a model wrote them. They are more dangerous because they arrive faster than you can think about them, and because they arrive looking finished.

So stop trying to make the review step better. Put the boundary somewhere that does not depend on anyone paying attention: a permissions boundary on every identity a pipeline can create or modify, an SCP denying the escalation actions to everyone who does not need them, and a build step that fails when a forbidden action shows up. Then let generation happen at whatever speed it wants.

The policy approved in ninety seconds is not the failure. The absence of anything underneath it was.

Need help drawing that boundary in your account?

Most of the IAM work I take on is exactly this: an account that grew organically, permissions nobody wants to touch in case something breaks, and no ceiling underneath any of it. Things I can help with:

• Auditing existing roles and policies for privilege escalation chains, including the passable-role paths that never show up in a single-document review
• Designing and attaching permissions boundaries for pipeline, workload and human identities, including the delegation pattern that lets a deploy role create roles safely
• Writing the SCP set for a small organization: escalation-relevant IAM actions, region restrictions, and protection for logging and billing resources
• Adding Access Analyzer validation and custom policy checks to a CI pipeline so forbidden actions fail the build instead of relying on review
• Replacing over-broad roles with policies generated from CloudTrail activity, then closing the gaps the generator cannot see
• Scoping down credentials handed to third-party SaaS integrations and to servers outside AWS, where your account controls do not apply

If you want a second opinion on something specific, send me the policy JSON, the output of a failing check-access-not-granted run, or the role list from your account, and I will tell you what I would change and why.

The post AI-Generated IAM Policies: Where the Hard Boundary Belongs appeared first on John Nessime.

That gap is the entire problem. Run status tells you the code executed. It tells you nothing about whether data arrived. Data pipeline freshness monitoring is the signal that closes the gap, and most teams add it only after an analyst finds the stale table first.

This post covers the three signals worth instrumenting for a batch or streaming pipeline: how current the data is, how long it took to get there, and how often the thing breaks. For each one I’ll cover what to measure, where to measure it, the queries and config that produce it, and the specific ways the measurement lies to you.

The signal that lies to you first: run status

Job success is a statement about your code path, not about your data. A pipeline that reads an empty S3 prefix, transforms zero rows, writes zero rows and exits cleanly has succeeded. So has one whose upstream API silently started returning an empty page after a token rotation. So has one whose incremental watermark got stuck and now re-reads the same already-loaded slice on every run.

All three are green. All three are producing stale data. This is why the ordering matters: freshness is the signal that fires first in a real incident, and run status is usually the last one to notice anything at all.

So keep run status. It’s cheap and it catches crashes. Just stop treating it as your top-line health indicator.

Freshness: measure the table, not the job

Freshness is the age of the newest record in a dataset. It’s measured against the dataset itself, not against the process that filled it, and that distinction is what makes it useful. If your job disappears entirely, freshness keeps climbing and keeps alerting. If your job succeeds while doing nothing, freshness keeps climbing and keeps alerting. It’s the one signal that survives both failure modes.

Two clocks, and you need both

Every row usually carries two timestamps: when the event happened in the source system, and when your pipeline wrote it. Track the maximum of each.

• Max event time tells you whether the upstream system is still producing. If this stops moving, the problem is upstream of you.
• Max load time tells you whether your pipeline is still writing. If this stops moving while event time is fine, the problem is yours.

Watching only one of them means every stale-data page starts with twenty minutes of figuring out which side of the boundary the fault sits on. Watching both answers that in the alert body.

A freshness probe is a small scheduled query. This one is PostgreSQL syntax, and it returns both clocks plus a volume check in a single round trip:

select
  'orders' as dataset,
  extract(epoch from max(event_ts))  as max_event_ts,
  extract(epoch from max(loaded_at)) as max_loaded_at,
  count(*) filter (where loaded_at >= now() - interval '1 hour') as rows_last_hour
from analytics.orders;

The row count matters. A pipeline can advance its load timestamp while writing almost nothing, which is what a partially broken source looks like. Freshness alone will not catch that; freshness plus volume will.

Run that probe on a schedule that’s independent of the pipeline, push the results as gauges, and alert on age in PromQL:

# Newest row is older than 90 minutes
time() - max by (dataset) (dataset_max_loaded_timestamp_seconds) > 5400

# The probe itself has stopped reporting: a deadman check
absent(dataset_max_loaded_timestamp_seconds{dataset="orders"})

That second rule is the one people forget. A freshness metric that disappears looks identical to a healthy silence on a graph. absent() is what turns “no data” into a page.

Where dbt fits

If you’re already running dbt, source freshness is built in and worth using before you write anything custom. You declare thresholds per source, and dbt queries the maximum of your timestamp column and compares it to now.

sources:
  - name: raw_shop
    schema: raw
    config:
      loaded_at_field: _loaded_at
      freshness:
        warn_after: {count: 2, period: hour}
        error_after: {count: 6, period: hour}
    tables:
      - name: orders
      - name: refunds
        config:
          freshness:
            warn_after: {count: 12, period: hour}
            error_after: {count: 24, period: hour}

Three details that trip people up. First, dbt source freshness is a separate command; dbt build does not run it, so a green build says nothing about source staleness. Second, it exits non-zero when a source hits its error_after threshold, which makes it a natural gate at the top of a job: fail fast rather than building models on stale input. Third, results land in target/sources.json, which is the artifact you parse if you want to distinguish warn from error, or ship the numbers into Prometheus rather than just failing the run.

dbt source freshness --select source:raw_shop --output target/freshness.json

One caveat on the YAML above: dbt moved these keys under a config: block in recent releases, and loaded_at_field followed later. Older projects nest them directly under the source. Check what your project’s version expects before copying, because a misplaced key fails quietly by simply not calculating freshness at all.

Latency: name the clock before you name the number

“Our pipeline latency is twelve minutes” is meaningless until you say which two points you measured between. There are at least three plausible definitions, and teams routinely argue past each other because they’re each using a different one.

• Run duration. Start to end of the job. Easy, and mostly useless for anyone downstream.
• Ingestion latency. Source extract to target load. This is what you control.
• End-to-end latency. Event time to the moment the row is queryable. This is what the business actually feels, and it includes queue wait, scheduler delay, and every upstream hop you don’t own.

Publish end-to-end as the headline and keep the others as breakdown. If you only track run duration, a scheduler backlog that delays every run by forty minutes is completely invisible to you: each individual run still takes eleven minutes.

Instrument per stage with a histogram so you can ask percentile questions later without re-instrumenting:

histogram_quantile(
  0.95,
  sum by (le, stage) (rate(pipeline_stage_duration_seconds_bucket[6h]))
)

Use the median for capacity planning and p95 or p99 for the SLO. Averages hide the exact tail that generates the complaints, and on a pipeline that runs a few dozen times a day, a single pathological run is a real fraction of your day.

For streaming, the equivalent is consumer lag expressed in time rather than offsets. Offset lag of fifty thousand messages means nothing without a rate; two hundred seconds of lag means something to everyone. Kafka Lag Exporter popularised this by interpolating a time estimate from observed offset and timestamp samples, exposing kafka_consumergroup_group_max_lag_seconds alongside the offset-based kafka_consumergroup_group_lag. Worth knowing: that project’s repository has been archived and is read-only, so if you’re starting fresh, check whether your broker vendor or a maintained fork covers it before you deploy something unmaintained into the critical path.

Failure rate: decide what counts as a failure

Failure rate is trivially easy to compute and surprisingly easy to compute wrongly. The denominator and the definition both need a decision.

Count runs, not tasks. A DAG with sixty tasks where one flaps on a transient network error looks catastrophic at task level and fine at run level. The run is the unit the consumer cares about.

Count a run that succeeded on its third retry as a success for availability and a failure for a separate reliability metric. Both are true and they answer different questions. If your only metric folds retries into success, you will never see the slow degradation of an upstream API until it stops responding entirely.

sum by (pipeline) (rate(pipeline_runs_total{result="failure"}[6h]))
/
sum by (pipeline) (rate(pipeline_runs_total[6h]))

And add a category label for the failure reason at push time: source unavailable, schema mismatch, permission denied, timeout, validation failed. The rate tells you something is wrong. The category tells you who to wake up. Without it, every failure alert costs you a log dive before you can even route the incident.

Getting metrics out of a job that exits

Prometheus scrapes. Batch jobs finish and vanish. Pushgateway bridges that: the job pushes before exiting, and Pushgateway holds the values for Prometheus to scrape on its own schedule.

#!/usr/bin/env bash
set -euo pipefail

JOB="orders_load"
PGW="http://pushgateway.internal:9091"

start=$(date +%s)
if python /opt/pipelines/load_orders.py; then
  result=0
else
  result=1
fi
end=$(date +%s)

{
  cat <<EOF
# TYPE pipeline_run_duration_seconds gauge
pipeline_run_duration_seconds $((end - start))
# TYPE pipeline_last_run_timestamp_seconds gauge
pipeline_last_run_timestamp_seconds $end
# TYPE pipeline_last_run_success gauge
pipeline_last_run_success $((1 - result))
EOF
  if [ "$result" -eq 0 ]; then
    cat <<EOF
# TYPE pipeline_last_success_timestamp_seconds gauge
pipeline_last_success_timestamp_seconds $end
EOF
  fi
} | curl --fail --data-binary @- "$PGW/metrics/job/$JOB"

exit $result

The conditional block is doing real work. curl’s --data-binary issues a POST, and a POST to Pushgateway replaces only the metrics whose names appear in the payload, leaving the rest of the group intact. So a failing run updates the run timestamp and the success flag while leaving the previous pipeline_last_success_timestamp_seconds exactly where it was. That’s what lets time() - pipeline_last_success_timestamp_seconds keep climbing across consecutive failures. Send a PUT instead and you replace the whole group, wiping the value you needed.

On the Prometheus side, one setting is not optional:

scrape_configs:
  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway.internal:9091']

Without honor_labels: true, Prometheus overwrites the job label your pipeline pushed with the scrape job’s own name, and every pipeline in your estate collapses into one indistinguishable series called pushgateway.

The trap worth internalising: Pushgateway never forgets. Metrics persist until something explicitly deletes them or the process restarts. A pipeline you decommissioned last quarter is still cheerfully reporting a success timestamp and a duration, and it looks alive on every dashboard. Treat Pushgateway as a cache of past executions, not a picture of current state, and use the push_time_seconds gauge that Pushgateway attaches to each group to tell the difference between a fresh push and a fossil.

If you’re on Airflow, the SLA feature is gone

This one will bite anyone upgrading. The sla and sla_miss_callback parameters were removed in Airflow 3.0, and the replacement, Deadline Alerts, arrived in 3.1. DAGs carrying the old configuration need manual migration; they don’t quietly keep working.

from datetime import timedelta

from airflow.sdk import AsyncCallback, DAG, DeadlineAlert, DeadlineReference
from airflow.providers.slack.notifications.slack_webhook import SlackWebhookNotifier

with DAG(
    dag_id="orders_load",
    deadline=DeadlineAlert(
        reference=DeadlineReference.DAGRUN_QUEUED_AT,
        interval=timedelta(minutes=45),
        callback=AsyncCallback(
            SlackWebhookNotifier,
            kwargs={"text": "orders_load has not finished 45 minutes after queuing."},
        ),
    ),
):
    ...

Note the reference point. Measuring from when the run was queued, rather than from its logical date, means scheduler backlog counts against the deadline. That’s usually what you want, because a run that sat in a queue for an hour is late to its consumers regardless of how fast it executed once it started.

There’s a structural weakness here worth naming: any alert that lives inside the orchestrator dies with the orchestrator. If the scheduler is down, nothing evaluates your deadline and nothing notifies anyone. The freshness probe from earlier is the answer, and it needs to run somewhere else. A small VPS from a provider like Contabo or InterServer running Prometheus and Alertmanager, or a hosted option like Grafana Cloud, gives you a watcher outside the blast radius of the thing being watched. This is the single highest-value piece of monitoring most data teams are missing.

Alerting without burning your on-call

The fastest way to make all of this worthless is to alert on every threshold crossing. Three rules keep it survivable.

1. Set thresholds from observed behaviour, not from wishes. Look at a month of actual freshness values and set warn_after a comfortable margin above the normal worst case. A source that habitually passes at eleven hours against a twelve-hour threshold is not healthy, it’s one upstream hiccup from paging you.
2. Page on consumer impact, not internal events. A task retry is not an incident. A dataset breaching the freshness commitment its consumers rely on is. Route everything else to a channel someone reads in the morning.
3. Use burn rate over multiple windows for SLOs. A short window catches fast breakage, a long window catches slow erosion, and requiring both to fire filters out the transient spikes that generate most false pages.

One more: alert on the absence of your own telemetry. A freshness gauge that stops updating is indistinguishable from one that’s fine, right up until someone asks about the numbers.

Troubleshooting

Freshness alert fires but the data looks current. Almost always a timezone problem. Your loaded_at column is in local time, now() is in UTC, and the offset shows up as a constant bias in the age. Store load timestamps in UTC and cast explicitly at read time.

Freshness flaps in and out of breach. Your check runs too close to the expected arrival. If loads land around six and your probe runs at five past, a fifteen-minute upstream delay produces an intermittent failure that trains everyone to ignore the alert. Move the probe later or widen the threshold.

Every pipeline reports as one series. honor_labels: true is missing from the Pushgateway scrape config.

A decommissioned pipeline still shows healthy. Stale group in Pushgateway. Delete the group and add a check on push_time_seconds so the next one surfaces on its own.

Latency looks fine but consumers say data is late. You’re measuring run duration and they’re feeling end-to-end. Add queue wait and event-time-to-load and the gap will be obvious.

Freshness passes, row count is near zero. The load timestamp advanced without meaningful data. This is a broken source or a stuck watermark, and it’s the reason the volume check belongs in the same probe.

Common mistakes

• Treating job success as the health signal and discovering staleness through a human.
• Running the freshness check inside the same pipeline it’s meant to police.
• Tracking freshness without volume, so a zero-row load reads as healthy.
• Reporting latency without saying which two timestamps it spans.
• Putting run IDs, batch IDs or timestamps into Prometheus labels, which multiplies your series count without bound. Those belong in traces or logs.
• Alerting on task-level failures instead of run-level outcomes, then muting the whole channel a week later.
• Assuming Pushgateway reflects current state rather than the last thing anyone pushed.

Best practices for data pipeline freshness monitoring

• Start with your three most-used tables. Full coverage is a project; three tables is an afternoon and catches most of the pain.
• Write down the freshness commitment per dataset in plain language, then encode it. “Yesterday’s orders are complete by 07:00” converts directly into a threshold.
• Name metrics consistently across pipelines. A shared prefix and a stable label set is what makes one dashboard work for all of them.
• Keep labels low-cardinality: pipeline, dataset, stage, environment, result. Nothing unbounded.
• Emit a failure category alongside every failure so alerts route themselves.
• Run the deadman check on infrastructure that doesn’t share a failure domain with the pipeline.
• Link every alert to a runbook that names the owner and the first three things to check.

Frequently asked questions

What is the difference between pipeline latency and data freshness?

Latency measures how long a specific batch of data took to travel from source to destination. Freshness measures how old the newest available record is right now, regardless of whether anything is currently running. A pipeline can have excellent latency and terrible freshness if it stopped being triggered.

How often should freshness checks run?

Frequently enough that you find out before your consumers do. A useful rule is roughly a quarter of your tolerance window: if data may be up to four hours old, check hourly. Freshness probes are cheap single-aggregate queries, so the limiting factor is usually warehouse billing rather than load.

Do I need a data observability platform for this?

Not to start. Freshness, latency and failure rate for a handful of critical datasets is a scheduled query, a push, and a few alert rules. Commercial platforms earn their cost when you need automatic column-level lineage, anomaly detection across hundreds of tables, or coverage of assets nobody has explicitly instrumented. That’s a real problem at scale, and a genuinely expensive one to build yourself. It’s just not the problem you have on day one.

How do I monitor freshness for a table that only updates weekly?

Set the threshold from the schedule plus a delivery margin, and add a deadman rule so a missing metric alerts on its own. For genuinely static reference tables, disable freshness explicitly rather than leaving a check that always warns; in dbt that means setting freshness to null for the table.

Should freshness checks fail the pipeline or just warn?

Both, in different places. Checking source freshness at the start of a job and failing hard prevents you building models on stale input, which is the cheapest bug to prevent and the most expensive to unwind. Checking output freshness after the fact should alert rather than fail, because the run is already over.

What percentile should I use for a latency SLO?

p95 for most internal analytics pipelines, p99 where downstream systems make automated decisions on the data. Track the median separately for capacity planning. Never use the average as the SLO number, because it hides exactly the tail that produces complaints.

Does this work for streaming pipelines too?

Yes, with different plumbing. Freshness becomes consumer lag measured in time, latency becomes event-time to availability-time, and failure rate becomes connector and task state plus dead-letter volume. The reasoning is identical; only the source of the numbers changes.

Conclusion

If you remember one thing, make it this: measure the data, not the job. Run status, task counts and duration all describe your code. Only freshness describes what your consumers actually receive, and it’s the one signal that stays honest when the pipeline succeeds at doing nothing.

Good data pipeline freshness monitoring is not a platform purchase. It’s a scheduled query that reports the age and volume of your most important tables, pushed somewhere durable, with a deadman rule so that silence is treated as a failure rather than as health. Add latency broken down by stage and failure rate categorised by reason, and you can answer “is the data good right now” without opening a single log file.

Need help instrumenting your pipelines?

Most of my consulting work in this area is retrofitting observability onto pipelines that already exist and can’t be paused. Specifically:

• Defining freshness and latency commitments per dataset, then translating them into thresholds and alert rules that hold up on-call.
• Adding freshness and volume probes to existing warehouses without touching the pipelines themselves.
• Wiring batch jobs into Prometheus through Pushgateway, including the grouping-key and stale-metric problems that bite six months later.
• Migrating Airflow DAGs off the removed SLA feature onto Deadline Alerts, or onto external checks that survive a scheduler outage.
• Building the Grafana dashboard that answers “is the data good right now” in one screen, with drill-down by stage and failure category.
• Cutting alert noise on pipelines where the channel has already been muted, by moving from task-level events to consumer-impact SLOs.

If you want a concrete starting point, send me a DAG file, a scrape config, or a screenshot of the dashboard you don’t trust, and I’ll tell you what I’d instrument first and why.

The post Green Pipeline, Stale Data: Tracking Latency, Freshness and Failure Rates appeared first on John Nessime.

That’s the signature failure of telemetry pipelines on AWS. Transport worked, authentication worked, the endpoint answered 200. It kept part of the payload and dropped the rest, because something in the batch broke a limit your collector has no idea exists.

Getting started with OpenTelemetry on AWS is not the hard part. AWS Distro for OpenTelemetry (ADOT) will have you shipping traces in an afternoon. The hard part is four decisions underneath it: where the collector sits, which door telemetry uses to get into AWS, what happens to your metrics on the way through, and whether a second backend later costs you a config line or a migration. This post covers those four decisions and how each one fails without announcing itself.

What ADOT actually buys you

ADOT is not a separate protocol or agent. It’s a downstream build of the upstream OpenTelemetry Collector plus AWS-flavoured SDKs, tested by AWS and covered by AWS Support. The config syntax is the upstream syntax, so a config written for vanilla OTel runs on ADOT and vice versa. The difference is which components are compiled in: ADOT ships the X-Ray exporter, the CloudWatch EMF exporter, the SigV4 extension, and ECS metric receivers already there.

So the choice is narrower than it looks. Take ADOT for a build someone else validates and supports. Take upstream Contrib if you need a component ADOT hasn’t bundled, or you run one collector image across AWS, another cloud, and bare metal. Neither choice touches your instrumentation, which is the entire point.

Decision one: where the collector runs

Three shapes, trading the same three things: blast radius, cost, and how much processing happens before data leaves your network.

Sidecar

One collector container per ECS task or pod. The application talks to localhost, so no service discovery and no network hop to get wrong, and failure stays contained to one workload. The cost: you pay for that container everywhere, batches stay small because batching is per-instance, and any config change is a redeploy of every task.

Agent plus gateway

A light collector per node (a DaemonSet on EKS) forwarding to a small pool of gateways. This is the shape I reach for first on anything past a handful of services. The agent does host-level enrichment, the gateway does the expensive work: large batches, tail sampling, redaction, fan-out. Config changes hit the gateway only.

The catch: a gateway pool is a thing you now operate. It needs autoscaling and its own alerting, and undersized it starts refusing data at exactly the moment you have an incident and volume spikes.

No collector at all

The ADOT SDKs can export straight to the CloudWatch OTLP endpoints, signing requests with credentials already available to the process. For a handful of Lambda functions this is the right answer: nothing to run, nothing to scale, no extra container in the cold start path. What you give up is control. Batching, retries, sampling policy, redaction, and backend routing all move into application processes and their environment variables, so changing any of them is a fleet redeploy rather than a config push. Good starting point, poor steady state.

Decision two: which door into AWS

Two ways to hand telemetry to AWS, and they behave very differently.

The older path uses AWS-specific exporters. awsxray converts OTLP spans into X-Ray segment documents and calls the X-Ray API. awsemf converts OTLP metrics into CloudWatch Embedded Metric Format and writes them as log events. Both work, both are well trodden, and both reshape your data into an AWS-native format on the way out.

The newer path is native OTLP. CloudWatch exposes OTLP endpoints per signal, reached with the plain otlphttp exporter and a SigV4 signer. Same exporter you’d point at Grafana Cloud or Honeycomb, different URL and authenticator. That’s what “export anywhere” means in practice, and it’s why I default to it on new work.

The endpoints follow a per-service pattern:

• Traces: https://xray.<region>.amazonaws.com/v1/traces
• Metrics: https://monitoring.<region>.amazonaws.com/v1/metrics
• Logs: https://logs.<region>.amazonaws.com/v1/logs

Three constraints there will each cost you an afternoon if you don’t know them going in.

HTTP only. No gRPC. If your collector exports over otlp on 4317 you can’t just change the URL, you have to switch exporters. Receiving gRPC from applications is fine, it’s the outbound leg that must be HTTP.

SigV4 required. That means the sigv4auth extension, configured per signal because the signing service name differs: xray, monitoring, and logs respectively. Bearer tokens are an alternative for metrics and logs, but not for traces.

Logs need headers, not just a URL. Target log group and stream travel in x-aws-log-group and x-aws-log-stream headers. Omit them and the request has nowhere to land.

A working shape for traces and logs looks like this:

extensions:
  sigv4auth/traces:
    region: "us-east-1"
    service: "xray"
  sigv4auth/logs:
    region: "us-east-1"
    service: "logs"

exporters:
  otlphttp/traces:
    compression: gzip
    traces_endpoint: https://xray.us-east-1.amazonaws.com/v1/traces
    auth:
      authenticator: sigv4auth/traces

  otlphttp/logs:
    compression: gzip
    logs_endpoint: https://logs.us-east-1.amazonaws.com/v1/logs
    headers:
      x-aws-log-group: MyApplicationLogs
      x-aws-log-stream: default
    auth:
      authenticator: sigv4auth/logs

service:
  extensions: [sigv4auth/traces, sigv4auth/logs]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlphttp/traces]
    logs:
      receivers: [otlp]
      exporters: [otlphttp/logs]

Note traces_endpoint and logs_endpoint rather than endpoint. This one bites people. The endpoint setting is a base URL and the exporter appends the signal path itself, so endpoint: https://monitoring.us-east-1.amazonaws.com/v1/metrics actually requests /v1/metrics/v1/metrics. The per-signal settings take a full path. If you’re getting 404s from a URL you’re certain is right, this is why.

Also worth a comment in your config: upstream has renamed the component to otlp_http and otlphttp is now a deprecated alias scheduled for removal. Both work today.

Traces need Transaction Search turned on first

This is the single most common reason a correctly configured trace pipeline produces nothing. The X-Ray OTLP endpoint requires Transaction Search to be enabled on the account, which redirects span ingestion into CloudWatch Logs:

aws xray update-trace-segment-destination --destination CloudWatchLogs

Spans then land in a log group named aws/spans, with a percentage indexed in X-Ray as trace summaries for search. The default index rate is one percent, enough to find traces while all spans stay queryable as structured logs. The caller needs xray:UpdateTraceSegmentDestination and xray:UpdateIndexingRule plus log group creation rights.

Two consequences to plan for. Span ingestion is billed separately from log ingestion, so it’s a line item rather than a rounding error. And AWS recommends always_on sampling in the SDK on this path, because the indexing rule already handles reduction. Sample in both places and your service map goes patchy.

The X-Ray SDK clock is running

The dates matter here, so plainly: the X-Ray SDKs and daemon entered maintenance mode on 25 February 2026 (security fixes only, no new instrumentation support), with end of support on 25 February 2027. The X-Ray service is fine and still gaining features. It’s the client libraries and the UDP daemon winding down. If you run aws-xray-sdk and a daemon sidecar, that is technical debt with a published expiry, and either the collector or the CloudWatch agent replaces the daemon.

Decision three: metrics, temporality, and the cardinality bill

Metrics are where the silent drops live, because collector defaults and endpoint limits disagree.

The metrics endpoint caps a single request at 1 MB uncompressed and 1,000 datapoints, counted across resource, scope, and metric levels combined. The upstream batch processor’s default is far larger. Leave it alone and you build oversized requests, and the response to an oversized or partially invalid request is not always a clean failure: it can come back 200 with some metrics accepted and others rejected or throttled. So set the batch size deliberately. AWS’s own examples use a conservative value:

processors:
  batch:
    send_batch_size: 200
    timeout: 10s

The other limits shape attribute design more than config:

• 150 labels maximum across resource, scope, and datapoint attributes per datapoint
• 40 KB combined label and value size per series per datapoint
• One million new series creatable per ten-minute window, per account
• Timestamps no more than ten minutes in the future or fourteen days in the past

That new-series ceiling catches teams out. Attach a request ID, a session ID, or a raw URL path to a metric attribute and every request mints a fresh series. You’ll hit a million faster than you expect, and it presents as metrics randomly going missing rather than a quota error. Unbounded identifiers belong on spans and logs. Metric attributes should be values you could enumerate on a whiteboard.

Temporality is the other conscious decision. The OpenTelemetry SDK spec defaults to cumulative, while CloudWatch’s metric model is delta-shaped: it wants what happened this period, not the total since your process booted. Counters that look like ever-climbing staircases instead of rates are that mismatch. The cumulativetodelta processor converts in the pipeline, keeping the decision in the collector rather than scattered across SDK environment variables in every service.

Decide once and centrally either way. Mixed temporality across one account produces dashboards that are subtly wrong for months before anyone notices. And once volume climbs, pointing a cost tool such as Vantage or CloudZero at the CloudWatch line items earns its setup time, because telemetry spend grows in steps nobody approved.

Decision four: a second backend without a second instrumentation

Here’s the payoff. A collector pipeline holds multiple exporters, and adding one is a config change:

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/traces, otlphttp/vendor]

The same spans now reach CloudWatch and Grafana Cloud, Honeycomb, Datadog, or a self-hosted Tempo, with no application redeploy and no second agent. That’s the cashable value of instrumenting once against an open protocol: the switching cost of a backend drops from a migration project to a pull request.

Be honest about the price. You pay ingest twice, and egress from the gateway is real money at volume. Collector memory scales with exporter queue count, so a fan-out gateway needs headroom. And if one backend slows, backpressure can reach the pipeline feeding the other, which argues for separate pipelines per destination when their reliability differs.

It works in reverse too. Because the metrics and logs endpoints accept bearer tokens, a machine with no AWS credentials at all can ship into CloudWatch: a CI runner, another cloud, or a VPS at a provider like Contabo or InterServer running part of your stack. One collector config, one destination, regardless of who owns the hardware. Traces are the exception and still require SigV4.

Why OpenTelemetry on AWS fails quietly

Most breakages here don’t throw. How to recognise the common ones:

Data is missing but nothing errors

Almost always a limit breach inside an accepted request. Check batch sizes against the per-signal caps first, then attribute counts. Turn on the collector’s internal telemetry and compare the exporter’s sent counters against its failed counters. If sent looks healthy and data is still missing, the loss is happening server-side after acceptance, which narrows it to limits.

403 with a signature mismatch

A SigV4 problem, not an IAM problem. The usual cause is the wrong signing service name in the extension, since xray, monitoring, and logs are not interchangeable. If you sign requests yourself rather than using the extension, note that the traces endpoint is stricter about which headers land in the signed set, so sign a minimal stable set rather than everything the HTTP client added.

Backfilled data vanishes

Every endpoint enforces a timestamp window, and fourteen days in the past is the outer edge for all three signals. Replaying an old queue past that boundary gets rejected. Not a bug, but it looks like one when the replay job appears to succeed and produces nothing.

Traces arrive but the service map is empty

Usually a missing or inconsistent service.name resource attribute, or double sampling between the SDK and the indexing rule. Confirm what’s actually leaving the collector before hunting in the console: add the debug exporter to a copy of the pipeline, set verbosity: detailed, and read the resource attributes on real spans.

Common mistakes

• Pointing an OTLP gRPC exporter at a CloudWatch endpoint. They’re HTTP only, and the error won’t say so clearly.
• Using endpoint with a full signal path instead of the per-signal traces_endpoint, metrics_endpoint, or logs_endpoint.
• Leaving the batch processor at its default size on the metrics pipeline.
• Sampling in the SDK and relying on Transaction Search indexing rules, halving visibility twice over.
• Putting request IDs, user IDs, or raw paths into metric attributes.
• Running a gateway pool with no alerting on the collector’s own health, so saturation is invisible until you need it.

Best practices

• Keep instrumentation vendor-neutral. Plain OpenTelemetry SDK APIs, with the collector owning every AWS-specific decision.
• Set service.name, service.version, and deployment.environment as resource attributes everywhere. Almost every correlation feature depends on them.
• Alert on exporter failure counters and queue depth, and route those alerts somewhere that doesn’t depend on the pipeline being healthy.
• Version the collector config in Git and deploy it like application code, with a staging pipeline you can break safely.
• Redact in a processor before export. Once telemetry reaches a backend, removing it is a support ticket.
• Test a second exporter early, even to a throwaway account. A vendor migration is the wrong time to learn the fan-out path doesn’t work.

Frequently asked questions

Do I still need a collector if CloudWatch accepts OTLP directly?

Not to get data in, no. You need one to control what happens before it leaves: batching to fit endpoint limits, tail sampling, redaction, and fan-out. Small serverless estates can reasonably skip it. Anything with a dozen services will want one.

Can I send OTLP to CloudWatch over gRPC?

No. The CloudWatch OTLP endpoints are HTTP 1.1 only, accept binary or JSON payloads, and support gzip or no compression. Your collector can still receive gRPC from applications, it just can’t forward over it.

What’s the difference between ADOT and the upstream collector?

Same codebase, different build. ADOT is AWS’s tested distribution with AWS components bundled and AWS Support behind it; upstream Contrib carries a wider component set and moves faster. Configuration is compatible either way, so it’s a support and packaging choice, not an architectural one.

Do I have to enable Transaction Search to send traces?

For the X-Ray OTLP endpoint, yes. It’s a prerequisite, and enabling it routes span ingestion through CloudWatch Logs. Using the awsxray exporter against the classic X-Ray API instead doesn’t require it, but you give up the span-level analytics Transaction Search provides.

Can I ship telemetry to CloudWatch from outside AWS?

For metrics and logs, yes, using bearer token authentication instead of SigV4, which removes the need for AWS credentials on the host. Traces still require SigV4. Never hardcode the token in the config; read it from a mounted secret file or an injected environment variable.

Is it urgent to migrate off the X-Ray SDK?

Not an emergency, but it’s on a clock: maintenance mode now, end of support 25 February 2027. Existing applications keep working, they just won’t get new library instrumentation. Plan the migration on your schedule rather than someone else’s.

The one thing worth remembering

A pipeline that returns 200 is not a pipeline that works. Judge every design decision in OpenTelemetry on AWS against one question: when this drops data, do I find out from a dashboard or from a colleague asking why a service vanished off the map?

Size batches against the published limits, keep unbounded identifiers off metric attributes, decide temporality in one place, and instrument with plain OpenTelemetry so the export target stays a config line. Do that and “instrument once, export anywhere” stops being a slogan and becomes a property you can test.

Need help with your OpenTelemetry pipeline on AWS?

Most of this work is unglamorous and specific, which is why it gets deferred. Things I can help with:

• Auditing an ADOT or upstream collector config for silent drops, oversized batches, and limit breaches
• Designing collector topology for ECS, EKS, or Lambda, including gateway sizing and autoscaling
• Migrating X-Ray SDK and daemon workloads to OpenTelemetry ahead of end of support
• Cutting telemetry spend through sampling policy, cardinality control, and attribute pruning
• Dual export to CloudWatch and a third-party backend, so a future switch is a config change
• Meta-monitoring that tells you the pipeline is broken before your users do

Send me a collector config, an exporter log, or a screenshot of the gap in your dashboard and I’ll tell you what I’d look at first.

The post Instrument Once, Export Anywhere: OpenTelemetry on AWS With ADOT appeared first on John Nessime.

That conversation is usually the first sign that an organisation bought two Oracle products without deciding what each one was for. It is a common outcome, and it is not because the products are bad. It’s because they are sold as a suite, they overlap on purpose, and the sales conversation is about capability rather than authority.

This post is a working comparison of the four Oracle Construction and Engineering products people most often confuse: Primavera P6 vs Unifier vs Aconex, plus Textura. One profile each, what it genuinely owns, where it wins, where it doesn’t, and then the part nobody covers: how to decide which system holds the authoritative number when three of them offer a cost module.

The question that actually sorts them

Feature lists will not separate these products. All four manage documents. Three of them track cost. Two of them hold a schedule. If you compare capability columns you will conclude that you need all four, which is exactly the conclusion the suite is designed to produce.

The question that does separate them is: whose record is this, and who else needs to write to it?

Two of these products are single-organisation systems. You run them, you own the data, and outside parties get in only if you give them a seat and they accept your governance. One of them is deliberately multi-organisation, where every company on the project holds its own workspace and decides what it shares. And one is a transactional network that touches money leaving your bank account, which makes it a finance system wearing a construction badge.

Get that axis right and the rest of the decision mostly falls out.

Primavera P6: the schedule, and only the schedule

P6 is a critical path method scheduling engine with resource and cost loading on top. It exists to answer questions about time: what drives the finish date, what happens if this activity slips, what does the resource histogram look like if we accelerate.

Two things confuse people here. First, P6 comes in two shapes. P6 Professional is the desktop client that planners actually live in. P6 EPPM is the enterprise web layer with the shared database, portfolio views and web services. Most organisations run both against the same database, and most planners will fight you to keep the desktop client.

Second, Oracle Primavera Cloud is a separate product, not P6 in a browser. Oracle says so on its own product page, which is unusually direct for vendor copy. Primavera Cloud shares a lineage with P6 and is positioned as the cloud destination, but the data model is not identical and things planners rely on in P6 do not all have equivalents. If someone tells you a move to Primavera Cloud is a hosting change, they have not tried it.

Where P6 wins

• Contractual programme scheduling, where the schedule is a deliverable and gets audited
• Forensic delay work, because the XER and PMXML formats are what every expert and every third-party analysis tool expects
• Resource-loaded planning across a portfolio with a shared resource pool
• Anywhere the client’s contract names it, which is more places than you would expect

Where P6 doesn’t

• Cost control. P6 has cost fields and they are useful for cash flow shape, but a resource-loaded schedule is not a commitment ledger. It has no concept of a purchase order lifecycle, a variation approval or a payment certificate
• Correspondence and document control. There is nothing here for transmittals, RFIs or drawing registers
• Collaboration across companies. Subcontractors do not get P6 logins, and you would not want them writing to your programme if they did

Primavera Unifier: a business process engine that happens to know about projects

Unifier is the one people describe worst, usually as “the cost one”. That undersells it and also sets bad expectations.

Unifier is a configurable workflow platform for capital programmes. You define business processes: a change order, a funding request, an invoice, a payment application, a site instruction. Each one has a form, a workflow with approval steps and routing rules, and a defined effect on the cost sheet. Oracle ships preconfigured processes and a no-code designer called uDesigner for changing them or building your own. Alongside the business processes sit the managers, including the Cost Manager, Schedule Manager and Document Manager.

It’s sold as base products, principally Project Controls and Facilities and Asset Management, which matters because Unifier’s remit runs past handover into operating the asset.

The honest trade-off with Unifier is this: its configurability is the reason to buy it and the reason implementations run long. A platform that can model any approval workflow will not tell you what your approval workflow should be. If your organisation has not written its cost control process down, Unifier will faithfully automate the argument you are having instead of the process you wanted.

Where Unifier wins

• Owner-side capital programme control, where budget, funding, commitments, forecast and actuals need to roll up across many projects
• Anywhere approvals need an audit trail with defined routing, delegation and thresholds
• Public sector and utility programmes with fund tracking, where money has strings attached and the strings must be reportable
• Organisations that want one governed process applied to every project rather than per-project improvisation

Where Unifier doesn’t

• Supply chain collaboration. It’s your instance, your governance, your seats. External parties can be given access, but they are guests in your system and they behave like it
• Drawing and model control at scale. The Document Manager is real, but it is not a common data environment and it is not built for design coordination
• Fast starts. Configuration is a project in its own right, and treating it as an install is the most common way to burn a first year

Aconex: the project record, owned by nobody in particular

Aconex is a common data environment for correspondence, documents, workflows and models across every organisation on a project. Document control, transmittals, RFIs, submittals, drawing registers, model coordination, field inspections.

Its defining property is not a feature. It is the data ownership model. Every organisation on the project gets its own workspace and controls what it shares and when. That neutrality is the entire commercial argument, and it is the thing you cannot replicate by inviting subcontractors into your instance of something else.

This matters more than it sounds. A designer who believes the main contractor can read their internal drafts will keep their real work somewhere else, and your “single source of truth” quietly becomes a partial one. The reason Aconex accumulates a complete project record is that nobody is afraid of it. Oracle’s own materials lean on that neutrality hard, and in this case the marketing claim and the engineering consequence are the same thing.

On the model side, Aconex Model Coordination is design-neutral and built on open BIM standards including IFC and BCF, so it sits alongside the authoring tools rather than replacing them.

Where Aconex wins

• Multi-party projects where the correspondence record will be read by lawyers later
• Design-build and joint venture delivery, where no single party should hold everyone’s data
• Document and model volume, which Oracle’s datasheets describe as unbounded on data and participants
• Handover, because the audit trail and the linked document set are the deliverable

Where Aconex doesn’t

• Portfolio-level capital planning. Aconex is organised around projects, not around a programme of investment decisions
• Asset lifecycle after handover. The record ends where the project ends
• Deep enterprise cost governance. Connected Cost exists and is genuinely useful for project cost, but if your organisation needs funding, capital planning and multi-year rollup, that is Unifier’s shape

Textura: money leaving the building

Textura came into Oracle by acquisition and still feels different from the rest of the suite, because it is solving a different class of problem.

Textura Payment Management runs the construction payment cycle: initiating a draw, subcontractor pay applications, markups and approvals, owner billing, collecting lien waivers and compliance documents, enforcing compliance holds, payment authorisation and disbursement. It’s a network product. Your subcontractors transact in it, and the value comes from them being there.

The compliance side is the part that decides whether it’s worth buying. If your risk is a subcontractor’s insurance lapsing quietly and nobody noticing until a claim, an automated payment hold tied to a missing document is a control you cannot easily build with a spreadsheet and good intentions.

Where Textura wins

• General contractors with a wide subcontractor base and a monthly draw cycle that eats accounts payable alive
• Jurisdictions with real lien exposure, where waiver collection is a legal control rather than paperwork
• Compliance enforcement at the point of payment, which is the only moment anyone reliably pays attention
• Audit trails on who approved what, signed what and released what

Where Textura doesn’t

• It is not a cost control system. It handles the payment leg, not the budget, forecast or commitment ledger
• It does not replace your ERP. It integrates with it, and that integration is part of the implementation cost
• Adoption is a supply chain problem, not a software problem. If your subcontractors will not onboard, you own a slower process than the one you replaced

The overlap trap, and why the cost module is where it bites

Here’s the failure mode that is invisible at procurement and expensive eighteen months later.

Three of these products can hold cost. P6 has cost-loaded activities. Unifier has a full Cost Manager. Aconex has Connected Cost. Nobody sets out to run three cost systems, but it happens gradually: the schedule gets cost-loaded because the client asked for an S-curve, the finance team runs the commitment ledger in Unifier, and the project team starts logging variations in Aconex because that’s where the correspondence already is.

Now you have three forecasts. They disagree. Not because anyone is wrong, but because they are answering different questions with different cut-off dates and different accrual rules. And the reconciliation lands in a spreadsheet maintained by one person, which is the outcome every one of these products was bought to prevent.

The fix is a decision, not a tool. Before anything is configured, write down one sentence per data class: which system is authoritative, and what the others are allowed to hold.

• Schedule dates and logic: P6 is authoritative. Everything else receives dates, nothing else edits them
• Budget, commitments and forecast: pick one. If you own Unifier, it is Unifier. If you don’t, Connected Cost is a reasonable home, but say so out loud
• Correspondence and controlled documents: Aconex if it’s on the project, and if it is, do not run a parallel drawing register anywhere else
• Subcontractor payment and compliance status: Textura, with the resulting actuals flowing to whichever system owns cost
• Everything the authoritative system does not own: read-only, refreshed by integration, and clearly labelled in reports as a copy

That list takes an afternoon and saves a year. It’s also the single most common thing missing from Oracle Primavera implementations I get asked to look at.

Integration is the real cost, and it has a shape

Once you own two of these, you own an integration. Budget for it as a workstream, not a checkbox.

Oracle offers more than one path. Primavera Gateway is the established broker for P6 and Unifier, with built-in providers per application and a canonical intermediate format so mappings survive version changes. In the P6 to Unifier direction, Gateway maps the P6 Project ID to Unifier’s Project Number, and what flows depends on how the schedule is built: a duration-based schedule sends durations, a resource-loaded one sends durations and units. Oracle also has a lighter integration platform for exchanging data between Unifier, P6, Aconex and Primavera Cloud, so check which one your version and deployment model actually supports before designing anything.

Aconex Connected Cost takes a different route, talking to P6 over its web services API. Two constraints from that integration are worth knowing before you draw an architecture diagram:

• You can configure both a P6 and a Primavera Cloud integration for the organisation, but an individual project can only link to one of them at a time
• Self-hosted or third-party-hosted P6 servers have to be approved by Oracle and reachable over the internet before the integration can be established. Oracle-hosted P6 skips that approval step

That second one has teeth if you are running P6 EPPM on your own infrastructure. “Internet accessible” and “exposed” are not the same thing, and the gap between them is where your security review lives. If you’re standing up middleware or a reverse proxy in front of an on-premise P6 or Unifier deployment, that box needs to be treated as production: a dedicated host from a provider like Contabo or InterServer rather than a corner of an existing server, TLS terminated properly, access restricted to known networks, and remote administrative access over a business VPN such as NordLayer or Surfshark rather than a port forward someone opened for a consultant and forgot about.

Back up the database host properly too, and test a restore. On-premise P6 and Unifier estates are frequently the last thing anyone thinks about until they need it, and imaging tools from vendors like O&O Software earn their keep on exactly this kind of long-lived, rarely-touched server.

How I’d decide

Work through this in order and stop at the first answer that fits. Buying more than you need is far more common than buying too little.

1. Name the pain in one sentence. “We cannot defend our programme in a delay claim” points at P6. “We cannot tell the board what the capital programme will cost” points at Unifier. “We cannot find the approved revision” points at Aconex. “Subcontractor payments take six weeks and compliance is a filing cabinet” points at Textura
2. Count the organisations that must write to the system. One, and you are choosing between P6 and Unifier. Many, and you are choosing between Aconex and Textura
3. Ask whether the problem outlives the project. Assets, maintenance and multi-year capital planning are Unifier’s territory. Anything that ends at handover is not
4. Check what the contract already mandates. Plenty of clients name a scheduling tool or a project CDE in the contract documents. That decision is made, so spend your effort elsewhere
5. Ask the reseller how each product meters. Per named user, per project, per transaction or per value processed changes the architecture, not just the invoice. A per-user product pushes you towards fewer, more senior users, and that changes who does data entry
6. Only then decide on a second product, and write the system-of-record sentence for every data class the two of them share before signing anything

Arguments that don’t survive contact

• “Unifier can do document control, so we don’t need Aconex.” Technically true, practically false on multi-party projects. The obstacle is not features, it is that other companies will not put their real correspondence in your instance
• “Aconex has cost, so we don’t need Unifier.” Fine for project-level cost. It falls down when you need portfolio funding, capital planning and asset lifecycle
• “Primavera Cloud is just P6 in the browser.” Oracle itself states that Primavera Cloud is not P6. Treat any migration as a data model exercise with a gap analysis, not a lift and shift
• “We’ll integrate it later.” Later means after users have created data under two incompatible breakdown structures. Agree the project and cost breakdown codes on day one, even if the integration itself waits
• “The suite integrates out of the box.” Connectors exist and they are good. Mapping your codes, your workflow states and your calendars is still work, and it is work that needs someone who understands both sides

Practices worth adopting early

• Standardise the project ID before anything else. Gateway maps P6’s Project ID to Unifier’s Project Number, and a free-text naming convention will hurt on the day you connect them
• Keep the schedule in one place and let it flow outward. Bidirectional date editing across two systems is a support ticket generator
• Pilot Unifier configuration on one real process end to end before configuring twenty. uDesigner makes it easy to build the wrong thing quickly
• Onboard the supply chain before go-live on Aconex or Textura. Network products fail on adoption, not on uptime
• Export your own data on a schedule. XER and PMXML for schedules, and regular extracts from the cloud products, so you are not negotiating from zero at renewal
• Write the system-of-record statement into the implementation charter, where people will actually see it

FAQ

What is the difference between Primavera P6 and Unifier?

P6 is a critical path scheduling tool that answers questions about time and resources. Unifier is a configurable business process and cost platform that answers questions about money, approvals and governance across a capital programme. They overlap slightly because P6 can hold cost data and Unifier has a Schedule Manager, but neither is a substitute for the other.

Do I need Aconex if I already have Unifier?

If the project involves several independent organisations exchanging drawings and correspondence, usually yes. Unifier’s Document Manager works well for your own records, but external parties are guests in your instance. Aconex gives every organisation its own workspace and control over what it shares, which is what makes the project record complete rather than partial.

Is Oracle Primavera Cloud replacing P6?

Oracle positions Primavera Cloud as the cloud platform and publishes migration guidance from P6, while continuing to sell and support P6 EPPM. Oracle also states plainly that Primavera Cloud is not P6. Plan on the two coexisting for now, and treat any migration as a data model exercise with a proper gap analysis rather than a hosting change.

Can Textura replace our accounting system?

No. Textura handles the construction payment cycle: pay applications, approvals, lien waivers, compliance holds and disbursement. It integrates with ERP systems rather than replacing them, and that integration is a real line item in the implementation.

Which product should a general contractor start with?

It depends on which pain is loudest. If schedules are being disputed, P6. If the project record is scattered across email, Aconex. If the monthly draw cycle and subcontractor compliance are consuming the finance team, Textura. Contractors reach for Unifier less often than owners do, because Unifier’s strengths are programme-level funding and governance.

How hard is it to integrate P6 with Unifier?

The mechanism is well trodden through Primavera Gateway, which ships providers for both. The hard part is not the connector, it’s agreement: matching project identifiers, deciding which system owns dates, and reconciling breakdown structures that were designed independently. Expect the mapping work to take longer than the technical setup.

What data should I be exporting for portability?

For schedules, take regular XER and PMXML exports, since those formats are what independent analysis tools and delay experts read. For the cloud products, set up scheduled extracts of documents, transaction records and audit trails. The point is not distrust of the vendor, it’s that a project record you cannot read without an active subscription is a liability during a dispute.

The one thing worth remembering

The Primavera P6 vs Unifier vs Aconex comparison, with Textura alongside them, is not really a features question. Each of these products is good at the thing it was built for, and Oracle is not trying to trick anyone by giving three of them a cost module.

The damage comes from owning two of them and never deciding which one is authoritative. Sort the products by whose record they hold and how many organisations write to them, pick the smallest set that covers your loudest pain, and write down the system of record for every data class before a single workflow gets configured.

Do that and the monthly report matches the cost system. Skip it and you’ll meet the spreadsheet.

Need help untangling this?

Most of the work in this area is not clicking through screens, it’s the plumbing and the decisions around it. That part I can help with:

• Writing the system-of-record map for your data classes before you commit to a second Oracle product, so cost and dates have one owner each
• Designing and building integrations between P6, Unifier, Aconex, Textura and your ERP, including the field mapping and error handling nobody scopes
• Hardening on-premise P6 EPPM or Unifier deployments that need to be reachable for a cloud integration, with proper TLS, network restrictions and audited access
• Automating schedule and cost data extraction into a warehouse or BI layer, so reporting stops depending on someone manually exporting on a Friday
• Building repeatable backup, restore-testing and disaster recovery for self-hosted Primavera estates
• Second-opinion reviews on an implementation or integration design that already exists and is not behaving

If you’re stuck on something concrete, send it over: an integration mapping spec, a Gateway error, a network diagram for an on-premise P6 that needs to talk to a cloud service. Easier to give you something useful when I can see the actual thing.

The post Primavera P6 vs Unifier vs Aconex vs Textura: Which Oracle Product Do You Actually Need? appeared first on John Nessime.