The answer was one API call away, and nobody had made it. Snapshot management had been failing for the entire table since a developer added an Iceberg branch for a backfill experiment. Compaction kept running happily. Old snapshots never expired, the files they pinned never became eligible for deletion, and the bucket grew. No alarm, no console banner, no email.

That is the shape of most S3 Tables compaction problems in production. The managed maintenance is good at what it does. What it is bad at is telling you when it has stopped doing it. This post covers the failure modes that bite, the storage tail that keeps billing after you think data is gone, how the compaction charge accumulates, and the handful of knobs you actually control.

What managed compaction actually does

S3 Tables runs three maintenance jobs, all on by default. Compaction and snapshot management are per table. Unreferenced file removal is per table bucket and cannot be overridden per table.

Compaction is the Iceberg rewriteDataFiles routine with scheduling taken off your hands. It combines small objects into larger ones, applies pending row-level deletes on the way through, and writes the result as the table’s newest snapshot. Target file size defaults to 512MB and accepts 64MB to 512MB. Parquet row groups inside use a 128MB default.

There are four strategies. auto is the default and picks for you: sort order defined in metadata means sort, otherwise binpack. binpack packs small files into bigger ones. sort orders rows hierarchically by your chosen columns so engines prune more files. z-order interleaves several columns into one sortable value, for queries filtering on multiple dimensions rather than one leading column.

Both sort and z-order need a sort_order in your Iceberg table properties and the s3tables:GetTableData permission to set. They also cost more per byte processed than binpack, because they do more work.

Failure family one: maintenance that fails without telling anyone

This is the one worth reading the post for. Maintenance jobs have four states: Successful, Failed, Disabled, and Not_Yet_Run. There is exactly one place that state is exposed on demand, and it is an API call you have to remember to make.

aws s3tables get-table-maintenance-job-status 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table

The response carries a block for each of icebergCompaction, icebergSnapshotManagement, and icebergUnreferencedFileRemoval, each with its own status, a lastRunTimestamp, and a failureMessage. The three are independent. Compaction reporting Successful tells you nothing about whether the other two are running.

The failure messages are worth knowing before you see them, because several of them describe conditions your own pipeline creates.

Tags and branches disable snapshot expiry for the whole table

If any user-defined tag or branch exists on the table, snapshot management fails. Not partially. The whole table stops expiring snapshots, and the message says snapshot management does not support user-defined tags or references. This applies even when the tag or branch has a short retention period of its own.

Branches are a normal Iceberg workflow. Someone creating one for a write-audit-publish flow or a backfill is not doing anything wrong. They are just silently turning off garbage collection for that table, and the effect shows up on next month’s bill rather than in their terminal.

Iceberg retention properties conflict with the managed config

If history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep is set as an Iceberg table property, snapshot management fails for the entire table regardless of the value. The managed service will not reconcile its own configuration against properties in metadata.json. It refuses instead.

This catches teams migrating a self-managed Iceberg table into a table bucket, because those properties are exactly what a self-managed setup uses. Remove them and let the maintenance API own retention.

ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.max-snapshot-age-ms');
ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.min-snapshots-to-keep');

gc.enabled set to false

Setting the Iceberg gc.enabled table property to false stops both snapshot management and unreferenced file removal. Leave it unset or explicitly true. Teams sometimes set it defensively during a migration and then forget, which produces a table that compacts forever and cleans up never.

KMS key policy missing the maintenance principal

If the table is encrypted with a customer managed key, that key’s policy has to grant access to the service principal maintenance.s3tables.amazonaws.com. Without it you get an insufficient-access failure and all maintenance stops. Catch this in Terraform review rather than production, because the table reads and writes perfectly while maintenance is dead.

Failure family two: the storage tail you keep paying for

Deleting rows from an Iceberg table does not delete bytes from S3. Neither does compacting them. Understanding the cascade is the difference between predicting your storage bill and being surprised by it.

When compaction rewrites files, the originals stay because older snapshots still reference them. They become removable only once every referencing snapshot expires. Snapshot management drives that expiry from minSnapshotsToKeep (default 1) and maxSnapshotAgeHours (default 120). Both conditions must be satisfied before a snapshot goes.

When a snapshot expires, S3 marks objects referenced only by it as noncurrent. It does not delete them. They sit noncurrent for nonCurrentDays (default 10), then are deleted permanently.

Add that up. On defaults, a byte compaction superseded today is still on your bill roughly fifteen days from now. Files never referenced by any snapshot take a parallel path through unreferencedDays (default 3) into the same noncurrent window.

Two things follow. A table under heavy rewrite pressure carries a persistent storage overhang that is a function of your write rate and these windows, not of your logical table size. And those windows are your fastest cost lever if you do not need long time travel, configured in two different places.

# Per table: how long snapshots survive
aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergSnapshotManagement 
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"minSnapshotsToKeep":3,"maxSnapshotAgeHours":48}}}'

# Per bucket: how long dead files linger before deletion
aws s3tables put-table-bucket-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --type icebergUnreferencedFileRemoval 
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":2,"nonCurrentDays":3}}}'

Be deliberate. Shortening these windows shortens your time travel range and your recovery options. Deletion of noncurrent objects is permanent, and recovering them means a support case, if it is possible at all. I would rather pay for a few extra days of overhang on a table I might need to roll back than discover the limit during an incident.

Failure family three: write amplification and the compaction charge

S3 Tables bills storage, requests, a per-object monitoring fee for every object in the bucket, and, when compaction is on, both the objects and the bytes processed. Rates change and vary by region, so check the current S3 pricing page rather than a number from a blog post. What matters is the shape.

The shape is that compaction charges scale with how badly your writers behave, not with how big your table is. A batch job writing a few large files an hour barely registers. A streaming pipeline committing every few seconds produces a huge number of tiny objects, and every byte gets read and rewritten at least once, sometimes more as small compacted files merge again into larger ones.

People reach for targetFileSizeMB first. Wrong lever. Changing the target changes the destination, not how much data has to move to get there.

The lever that works is the commit pattern on the write side. Buffering so the pipeline commits larger files less often reduces object count, monitoring fees, compaction object charges, and bytes processed all at once. This is the trade S3 Tables makes explicit: the service takes the maintenance work off your plate and prices it, so a writer that generates more work pays more for it.

Because compaction is a separate billing dimension, break it out in whatever cost tooling you already use. Cost Explorer with usage-type grouping will show it, and platforms like Vantage or CloudZero make it easier to watch alongside the rest of your S3 spend.

Failure family four: files compaction quietly skips

Compaction supports Parquet, Avro, and ORC. It does not support the Fixed data type, nor the brotli or lz4 compression codecs.

That last one matters because lz4 is a reasonable choice for a write-heavy pipeline where encoding speed counts. Pick it and you get exactly the outcome you were avoiding: a table full of small files that the compaction you are paying for will not consolidate.

Compaction also writes Parquet by default even when source files are Avro or ORC. To keep the original format, set the write.format.default table property to avro or orc. Most people want Parquet for read performance, but if a downstream consumer expects a specific format, this is where you say so.

Failure family five: compaction and your writers fighting over the table

Iceberg uses optimistic concurrency with conflict detection to arbitrate commits. As far as the table is concerned, compaction is just another writer. When a compaction commit and one of yours touch overlapping state, one loses.

When compaction loses you see Failed with a message about failing to commit because of out-of-date metadata, and a note that it will retry. Benign in isolation. Seen persistently, it means your write pattern and the maintenance schedule collide often enough that compaction is not keeping up, and the small-file problem quietly worsens while status flips between Failed and Successful.

When your job loses, you get a commit failure in your pipeline. AWS explicitly recommends retry logic on your side, and that is not boilerplate. A pipeline without commit retries, against a table with managed compaction on, will fail intermittently for reasons that have nothing to do with your code.

Turning managed compaction off, and when that is right

Start with the honest case for leaving it on. Table maintenance is real operational work: a Spark cluster or scheduler, a job that has to run, alerting for when it does not, and someone who understands why it failed at 3am. Managed compaction removes all of that for a per-use fee. For most teams with a handful of tables and batch-shaped ingest, that is the right call. The cost is small and the alternative has a salary attached.

The case against is control. You cannot influence when compaction runs, hint from the write side about what to prioritise, or make it work harder after a large backfill. The published observability is a status field and a timestamp per job. If your workload has hot partitions needing attention now and cold partitions that never need touching, managed compaction cannot express that distinction and you pay to rewrite both.

Disabling is a per-table change, and it does not require zeroing the settings.

aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergCompaction 
  --value '{"status":"disabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'

From there you run maintenance yourself. Trino exposes an optimize procedure through ALTER TABLE ... EXECUTE, and Spark has the standard Iceberg rewrite_data_files procedure. Both let you scope work to specific partitions on your own schedule, which is the entire point of taking it back.

Do this with your eyes open. You now own the scheduling, the compute, the failure alerting, and the snapshot expiry. Teams that disable managed compaction and never build the replacement end up worse off than if they had left it alone. If cost rather than control is driving this, fix the write pattern first and re-measure. That is usually cheaper than the engineering you are about to sign up for.

Troubleshooting S3 Tables compaction and maintenance

Work through these in order. The first step resolves more cases than the rest combined.

1. Check all three job statuses, not one. Run get-table-maintenance-job-status and read every key in the response. A green compaction status next to a failed snapshot management status is the classic growing-bucket signature.
2. Read the failure message literally. It names the exact property or condition at fault. Tags or references means someone created a branch or tag. Incompatible table properties means history.expire.* is set.
3. Status is Disabled or Not_Yet_Run unexpectedly. Someone replaced the configuration. Confirm with get-table-maintenance-configuration, and remember put-table-maintenance-configuration replaces rather than merges.
4. Compaction runs but file count stays high. Check codec and data types against the unsupported list. Files using brotli or lz4, or containing the Fixed type, are outside what compaction touches.
5. Insufficient-access failures on an encrypted table. Check the KMS key policy grants the maintenance.s3tables.amazonaws.com service principal, and that the key is enabled and still exists.
6. Storage still growing with everything green. Do the arithmetic on your retention windows. Snapshot age plus noncurrent days is the floor on how long superseded data bills, and defaults put that at roughly two weeks.

For anything historical, S3 Tables maintenance operations appear in CloudTrail management events, which is where you go to answer “when did this start” rather than “what is broken now”.

Common mistakes

• Treating the three maintenance jobs as one thing. They fail independently and for different reasons. “Maintenance is enabled” is not a status.
• Creating branches or tags on a managed table. A normal Iceberg workflow that switches off snapshot expiry for the whole table.
• Tuning targetFileSizeMB to reduce cost. It changes where the data lands, not how much has to be moved. Fix the commit pattern instead.
• Setting sort or z-order without a sort order defined. The strategy needs sort_order in table properties and the s3tables:GetTableData permission, and it costs more per byte than binpack.
• Running write pipelines with no commit retry. Compaction is a concurrent writer. Conflicts are expected behaviour, not an anomaly.

Best practices for running S3 Tables compaction in production

• Poll job status on a schedule and alert on it. A small scheduled function calling get-table-maintenance-job-status across your tables, emitting a metric per job type, closes the largest observability gap in the service. Alert on Failed and on a stale lastRunTimestamp, on whatever dashboard you already watch, Grafana or CloudWatch.
• Make maintenance config part of your IaC. These are settings someone can change with one CLI call and no review. Codify them so drift is visible.
• Set retention windows from a real requirement. Ask how far back anyone needs to time travel, then set maxSnapshotAgeHours and nonCurrentDays to that plus a margin, rather than accepting defaults by omission.
• Batch on the write side before tuning anything else. Fewer, larger commits improve object count, monitoring cost, compaction cost, and query performance simultaneously.
• Choose the strategy from the query pattern. One dominant filter column suggests sort. Several equally weighted predicates suggest z-order. Neither suggests leaving it on auto.
• Track the compaction billing dimension separately. It moves independently of storage and it is the earliest signal that a writer’s behaviour has changed.

Frequently asked questions

How do I know if S3 Tables compaction is actually running?

Call get-table-maintenance-job-status and read the icebergCompaction block. It returns Successful, Failed, Disabled, or Not_Yet_Run, plus a last-run timestamp and a failure message. No dashboard surfaces this by default, so to know without asking you have to poll it and emit your own metric.

Why is my S3 Tables storage growing when compaction says it succeeded?

Two likely causes. Either snapshot management is failing separately, which stops old snapshots expiring and pins their data files indefinitely, or your retention windows are longer than you assumed. On defaults, superseded data sits for the snapshot age window plus the noncurrent window, around two weeks. Check the other two job statuses first.

Can I run my own compaction alongside the managed one?

You can, but it is a poor idea. Both are writers competing under Iceberg’s optimistic concurrency, so you generate commit conflicts and pay twice to rewrite the same data. Pick one. If you need engine-side control, disable managed compaction for that table first.

Does disabling compaction stop all S3 Tables charges?

No. It stops the compaction object and byte-processing charges for that table. Storage, requests, and the per-object monitoring fee continue, and that monitoring fee scales with object count, which is exactly what stops improving once compaction is off. Disabling it on a small-file-heavy table can raise monitoring cost over time.

Is sort or z-order compaction worth the extra cost?

It depends on whether queries filter on the sorted columns. Sorted layout lets engines skip files using Parquet column statistics, so the saving shows up in query cost and latency, not storage. Dominant queries filtering on one column favour sort. Several dimensions in varying combinations favour z-order. Queries that mostly scan everything justify neither.

Can I recover data deleted by S3 Tables maintenance?

Deletion of noncurrent objects is permanent. AWS documents contacting support to view or recover objects marked noncurrent, which is not a path to depend on. Treat retention window changes as a one-way door, set from a stated requirement rather than trimmed until the bill looks better.

The one thing worth remembering

S3 Tables compaction is a good trade for most teams. The service does real work you would otherwise be scheduling and babysitting, and it does it well enough that you can stop thinking about small files as a daily concern.

What it does not do is tell you when it stops. No alarm fires when a branch someone created for an afternoon experiment switches off snapshot expiry, and no banner appears when a KMS policy change kills maintenance across a bucket. The price of the convenience is that you build the one thing the service did not: a scheduled check on get-table-maintenance-job-status that tells you when the automation stopped automating. That is an afternoon of work. Skipping it is how a flat workload turns into a bill that grows every month for reasons nobody can explain.

Need help with Iceberg tables on AWS?

I work with teams running Apache Iceberg on S3, and most problems are the ones above rather than anything exotic. Things I can help with:

• Auditing every table in a bucket for failed or disabled maintenance jobs, and finding which have stopped expiring snapshots
• Building the scheduled job-status check and alerting that S3 Tables does not ship with, wired into CloudWatch or Grafana
• Working out where a bucket’s storage growth actually comes from, and setting retention windows against a real requirement
• Reworking streaming and CDC write patterns so commits produce fewer, larger files before compaction charges scale with them
• Deciding table by table whether managed or engine-side maintenance is right, and implementing whichever you pick
• Migrating self-managed Iceberg tables into table buckets without carrying over the properties that break managed maintenance

If something is not adding up, send me the output of get-table-maintenance-job-status for the table and a rough description of your write pattern. That is usually enough to say what is going on.

The post S3 Tables Compaction in Production: The Cost of Convenience appeared first on John Nessime.

That is the shape of the risk. When people ask whether a SageMaker Lakehouse migration is worth it, they picture data moving. Almost none of it does. What moves is the permission model, and that is the part nobody tests.

This post covers what the migration actually changes underneath, the failure mode that bites hardest, where the lakehouse earns its place, where staying on Glue plus Athena is the better call, how the cost levers shift, and a decision procedure you can run in an afternoon.

First, understand what you are actually migrating

The framing “SageMaker Lakehouse versus Glue and Athena” is misleading, and it drives bad decisions.

SageMaker Lakehouse is built on the AWS Glue Data Catalog and AWS Lake Formation. It is not a replacement metastore. It is a reorganisation of the catalog you already have, plus connectors, plus an Apache Iceberg REST interface, plus a console to look at all of it through.

The unit it introduces is the catalog: a logical container of databases, tables and views, nestable to mirror the shape of the source system. A managed catalog holds data you own in S3 or Redshift Managed Storage. A federated catalog points at something external such as Redshift, DynamoDB, PostgreSQL, Oracle, BigQuery or Snowflake, and lets you query it in place.

Your existing databases and tables do not disappear. They sit under the account catalog and keep working. AWS states in its own FAQ that you do not have to migrate, and that Athena, EMR, Glue and MWAA continue to work as they do today. That is an unusual thing for a vendor to publish about a flagship product, and it is the most useful sentence in the documentation set.

So the question is not “should we move our data”. It is “should we move our governance”.

The failure mode that actually bites

Lake Formation grants Super permission to a virtual group called IAMAllowedPrincipals on every existing Data Catalog database and table by default. That group effectively means “anyone whose IAM policy already lets them in”. It exists purely for backward compatibility. While it holds Super on a resource, access is decided by IAM and S3 policies exactly as it was before Lake Formation existed. Your Lake Formation grants are recorded, but they are not what gets enforced.

So the first time someone writes a careful column-level policy and tests it, the policy appears not to work. The analyst can still see the column. The natural next move is to revoke IAMAllowedPrincipals so the grants take effect.

That revoke is the moment everything relying on IAM-only access stops working. Glue jobs. Notebooks. Scheduled dbt runs. The QuickSight dataset nobody has looked at since the person who built it left. None of them had a Lake Formation grant, because none of them ever needed one.

It is invisible in advance because nothing warns you. Lake Formation cannot know which principals are silently depending on the compatibility group. There is no dry run.

Hybrid access mode is the way through

Hybrid access mode exists exactly for this. It supports two permission pathways to the same catalog objects at once. Principals you explicitly opt in are governed by Lake Formation grants. Everyone else keeps reaching the data through their IAM and S3 policies, untouched.

The detail worth internalising: when an S3 location is registered in hybrid access mode, Lake Formation enforces only CREATE_TABLE, CREATE_PARTITION and UPDATE_TABLE by default. The write path gets governed first. The read path stays on IAM until you opt a principal in. That asymmetry is what lets you migrate one team at a time instead of one account at a time.

The safe sequence: register the location in hybrid mode, opt in one low-stakes principal, verify its queries respect the fine-grained grants, verify a non-opted-in principal still works, repeat. Only after every consumer is opted in and verified do you consider revoking the compatibility group. Plenty of teams never need that last step.

Where the lakehouse genuinely wins

• One permission surface across engines. If governance today is “Lake Formation for the lake, Redshift GRANTs for the warehouse, IAM for DynamoDB”, you maintain three access models that disagree with each other. Catalogs let you express column and row-level policy once and have Athena, Redshift, EMR and Spark honour it.
• Tag-based access control across federated sources. LF-TBAC extends to federated catalogs covering S3 Tables, Redshift and external sources. If you hand-maintain named-resource grants across dozens of databases, tag expressions are a real operational win.
• Querying across boundaries without copying. A federated catalog lets one statement touch a Redshift table and an S3 Iceberg table, with no ETL job whose only purpose is moving rows so they can sit next to each other. Every pipeline you delete is a pipeline that cannot break at 3am.
• The Iceberg REST interface. Exposing the catalog through Iceberg REST APIs lets engines outside AWS read your tables over a standard protocol. If you run Trino, Spark or DuckDB elsewhere, this is what makes the catalog genuinely portable rather than nominally open.
• Cross-account sharing without replication. For data mesh setups, sharing a catalog instead of shipping a copy per consumer removes a whole category of staleness bug. Cross-account grants on federated catalog objects require the cross-account data sharing version set to 4 or higher.

Notice what these share. Every one is a governance or interoperability problem. None is a query performance problem, and none is a cost problem.

Where Glue plus Athena is still the right answer

• Single account, S3 only, one team. Catalogs solve a problem you do not have. You would be adding a governance layer to govern one group.
• Your access control is coarse and that is correct. Not every dataset needs column masking. If bucket-level IAM boundaries match your actual risk model, Lake Formation enforcement adds a failure mode without adding safety.
• You have heavy IAM-only automation. Every Glue job, Lambda, Airflow DAG and third-party tool touching the catalog is a principal you must eventually opt in and re-verify. That inventory is the real cost of migration, and it is always larger than the first estimate.
• Your bottleneck is query cost, not governance. Partitioning, file sizes, compaction and columnar layout drive Athena spend. A lakehouse migration touches none of them. Fix the small-files problem first; it pays back faster and it is reversible.
• You depend on a specific engine behaviour today. Fine-grained access control support varies by engine and table format. Confirm your exact combination is supported rather than assuming parity across Athena, Redshift, EMR and Glue Spark.

The cost levers that move, and the ones that don’t

Pricing pages change, so here are the billing mechanisms rather than numbers.

Unchanged: metadata storage and API requests follow existing Glue Data Catalog pricing, including free tier allowances. Reorganising into catalogs does not put you on a new metadata meter. Athena still bills on data scanned. S3 storage and request charges are what they were.

New line items:

• Lambda invocations behind federated catalogs. When a query hits a federated table, Lake Formation vends credentials that invoke a Lambda function defined in the Glue connection to retrieve metadata from the source. That is compute you did not previously pay for, and it is latency on the metadata path. Chatty BI tools that re-resolve schemas on every refresh are the ones to watch.
• Service-managed Redshift Serverless compute. Queries against a federated catalog pointing at an existing Redshift warehouse execute through a service-managed Redshift Serverless workgroup, which stages results before returning them. That compute is billed, and it does not appear where you expect it.
• Redshift Managed Storage. Managed catalogs backed by RMS rather than S3 put you on Redshift storage and Redshift Serverless compute economics, which behave nothing like S3 plus Athena.

The service-managed workgroup is the one that catches people, because the spend lands under Redshift Serverless with no obvious link to the analyst who ran the query. AWS publishes a generated cost allocation tag for exactly this, which you must activate in the billing console before it produces anything:

# Cost allocation tag on lakehouse-managed Redshift Serverless compute.
# Activate under Billing -> Cost allocation tags before it appears in reports.

Key:   aws:redshift-serverless:LakehouseManagedWorkgroup
Value: "True"

Activate it on day one, not after the first surprising invoice. Tags only apply to usage recorded after activation, so a late start leaves a blind period you cannot reconstruct. If you already run a cost visibility layer such as Vantage, CloudZero or Datadog’s cloud cost views, build a view isolating this tag before you create your first federated catalog. If you do not, the AWS Cost and Usage Report grouped by that tag key is enough to start.

Is a SageMaker Lakehouse migration worth it? A decision procedure

Run these in order. Stop at the first one that gives a clear answer.

1. Count your non-S3 governed sources. Redshift clusters, DynamoDB tables, operational databases and external warehouses that analysts need and that each carry their own access model today. Zero or one means the federated catalog story does not apply to you yet. Three or more and the unified permission surface starts paying for itself.
2. Inventory every principal touching the Data Catalog. Pull CloudTrail for Glue and Lake Formation API calls across a full billing cycle, long enough to catch monthly jobs. Every distinct role in that list is work. This number, not your data volume, is the migration estimate.
3. Ask whether column or row-level control has actually been requested. If the answer is a compliance requirement with a name attached, proceed. If it is “it would be nice”, you are buying a permission model to solve a hypothetical.
4. Check engine coverage against your real workloads. Confirm fine-grained access control support for each production engine in current documentation. One unsupported critical path makes hybrid mode permanent rather than transitional, which is a legitimate outcome but should be a decision, not a discovery.
5. Price the federated path against a copy. For your highest-frequency cross-source query, estimate metadata Lambda invocations plus any service-managed compute, then compare against a nightly job that lands a copy in S3. Federation wins on freshness and on one less pipeline. It does not automatically win on cost.
6. Decide the destination, not just the direction. “Hybrid access mode indefinitely, Lake Formation enforcement on the two regulated datasets” is a good end state. Write it down first, because migrations without a finish line become permanent half-migrations.

Three things to run before you commit

Enumerate what Athena can see today. This is your before picture, and it exposes a tooling gap worth knowing up front:

aws athena list-data-catalogs

# Returns catalogs registered directly with Athena, for example:
# {
#   "DataCatalogsSummary": [
#     { "CatalogName": "AwsDataCatalog", "Type": "GLUE" },
#     { "CatalogName": "cw_logs_catalog", "Type": "LAMBDA" }
#   ]
# }

S3 Tables catalogs use a s3tablescatalog/bucket-name naming convention and are not returned by this call, even though you can query them by naming the catalog explicitly. The console shows them through mechanisms the public API does not expose. Any inventory or drift-detection tooling built on list-data-catalogs will quietly under-report, so enumerate S3 Tables catalogs separately.

Next, confirm the role you intend to use as data lake administrator can create catalogs at all. If it is not already a data lake admin, it needs an explicit grant:

aws lakeformation grant-permissions 
  --cli-input-json 
  '{
     "Principal": {
       "DataLakePrincipalIdentifier": "arn:aws:iam::123456789012:role/Admin"
     },
     "Resource": { "Catalog": {} },
     "Permissions": [ "CREATE_CATALOG", "DESCRIBE" ]
   }'

This avoids the common false start where catalog creation works in the console for an admin user and fails for the automation role you actually meant to use.

Finally, if you read lakehouse tables from a Glue Spark job, fine-grained access control is not on by default. The job parameters that enable it:

--datalake-formats                          iceberg
--enable-lakeformation-fine-grained-access  true

Without the second one, the job reads through its IAM role and your column-level grants are not applied. That is a governance gap that looks exactly like a working pipeline, which is the worst kind.

Arguments that don’t survive contact

• “It’s the same Glue Data Catalog, so migrating is low risk.” True about the metadata, false about enforcement. The catalog is the same object. The thing deciding whether a query returns rows is not.
• “We’ll migrate to avoid falling behind.” AWS states plainly that you can keep using Athena, EMR, Glue and MWAA as you do today. Nothing here is a deprecation. Migrating on an invented schedule is how you end up owning a permission model nobody needed.
• “Federated catalogs mean we can retire our ETL.” Some of it, sometimes. Federation removes copies that existed only for co-location. It does not remove transformation, deduplication, late-arriving data handling or historical snapshots. Those jobs stay.
• “Iceberg REST support means no lock-in.” The table format is portable and that is real. The Lake Formation grants, LF-Tags, hybrid mode opt-ins and catalog hierarchy are not, and they become the hardest part of your platform to reproduce elsewhere. Portability moved down a layer; it did not appear.
• “We can test this in a sandbox.” You can test the mechanics. You cannot test the thing that actually breaks, which is the set of production principals depending on IAM-only access. That inventory does not exist in a sandbox.

How I’d decide

S3-only, one account, small team: I would not migrate. The same effort spent on partitioning and compaction returns more, and it keeps returning.

Multiple governed sources, several consuming teams, a named compliance requirement for fine-grained control: migrate, and treat hybrid access mode as the destination rather than a waypoint. Enforce Lake Formation on the datasets that need it, leave the rest on IAM. The pressure to reach a “clean” fully-enforced state is aesthetic, not operational.

Either way, start with a federated catalog over one source that is currently painful to reach. Prove the query path and the cost, and let the rest follow evidence rather than a plan drawn before anyone had numbers.

Frequently asked questions

Does SageMaker Lakehouse replace AWS Glue and Athena?

No. It is built on the Glue Data Catalog and Lake Formation, and Athena remains one of the engines you query it with. AWS documentation states you can continue using Athena, EMR, Glue and MWAA without migrating. It is a governance and connectivity layer, not a replacement stack.

Will a SageMaker Lakehouse migration break my existing Athena queries?

Reorganising into catalogs does not break queries on its own. What breaks them is revoking Super from IAMAllowedPrincipals before every consuming principal has a Lake Formation grant. Use hybrid access mode, opt principals in one at a time, and existing queries keep working throughout.

Do I have to move data into Redshift Managed Storage?

No. Managed catalogs can be backed by S3, including S3 Tables, and existing S3 data lakes integrate directly. RMS is a storage choice with different cost and performance behaviour, not a requirement of the architecture.

How much extra does it cost?

The catalog layer itself is not a new meter: metadata storage and API requests follow existing Glue Data Catalog pricing including free tier allowances. New costs come from what you build on it, mainly Lambda invocations for federated metadata retrieval, service-managed Redshift Serverless compute for federated Redshift queries, and RMS storage if you choose it.

Is SageMaker Unified Studio required?

No. Unified Studio is the integrated console experience. The catalogs are addressable from Athena, Redshift, EMR, Glue and Iceberg-compatible engines directly. Adopting the catalog structure without adopting the studio is a valid and common choice.

What is the smallest useful first step?

Create one federated catalog over a single non-S3 source that analysts currently reach through an awkward workaround. Grant read access to one team. Measure query latency and the resulting Lambda and compute charges. That gives you real numbers on your own workload, and it is trivially reversible.

Conclusion

A SageMaker Lakehouse migration is not a data migration. The Glue Data Catalog underneath is the same catalog, the tables are the same tables, Athena is still Athena. What changes is which system decides whether a query returns rows.

That makes the call simpler than the marketing implies. Several governed sources, multiple consuming teams and a real fine-grained control requirement, and the unified permission surface is worth the work. Without them, you are taking on a permission model and a Lambda-backed metadata path in exchange for a nicer console.

Whichever way you go: register in hybrid access mode, opt principals in one at a time, and do not revoke IAMAllowedPrincipals until you can name every principal that will notice.

Need help deciding, or unpicking a migration already underway?

An outside read on the actual inventory usually saves more time than any amount of architecture discussion. Things I help with:

• Building the principal inventory from CloudTrail so you know exactly who depends on IAM-only Data Catalog access before anything is revoked
• Designing a hybrid access mode rollout that opts teams in incrementally, with a verification step and a rollback position at every stage
• Modelling federated catalog costs against the copy-the-data alternative for your query patterns, including the service-managed compute nobody budgets for
• Diagnosing Lake Formation access denied errors across Athena, Glue Spark and Redshift, including the fine-grained access flags that silently bypass your grants
• Cost allocation tagging so lakehouse spend is attributable per catalog and per team from the first query rather than retrofitted later
• Honest second opinions on whether a migration is worth doing at all, including the answer that it isn’t

Send me a failing query, a Lake Formation grant listing or a CloudTrail export and I’ll tell you what I’d do with it.

The post SageMaker Lakehouse vs Glue + Athena: Is Migration Worth It? appeared first on John Nessime.

That slope is the signature. CloudWatch rarely fails loudly, it accumulates: a log group created automatically by a service you enabled once, retention left at the default, a metric filter with a dimension nobody thought about, and a dashboard quietly calling the API every few seconds forever.

This post breaks down where CloudWatch costs actually come from, why the same log line gets charged more than once, how cardinality multiplies your bill without your traffic changing, and how to attribute spend to specific log groups before you delete anything. It is organised by cost lever rather than by feature, because that is the order you work in when the invoice lands.

The four meters behind your CloudWatch bill

CloudWatch is not one product with one price. It is a set of independent meters that happen to share a console. Working out which meter is running is most of the job, because the fix for each is completely different.

• Ingestion. Per GB of log data accepted into a log group. Usually the largest component on log-heavy accounts.
• Storage. Per GB-month for what is retained afterwards. Cheap per unit, but it compounds because it never stops unless you tell it to.
• Analysis. Logs Insights bills on data scanned by a query, not on rows returned and not per query.
• Metrics and API calls. Custom metrics bill per unique time series per month. Separately, calls like GetMetricData bill per metric requested, which is how third-party dashboards end up on an AWS invoice.

The thing worth internalising: a single log line can be billed three times. Once when it arrives, again every month it sits in storage, and again every time a query scans the block it lives in. None of that is hidden, but nothing in the console surfaces it either.

Ingestion is the meter that actually hurts

Uncompressed in, compressed out

Ingestion is metered on the uncompressed volume of events you send. Storage is billed on the compressed size CloudWatch keeps. Text logs compress well, so the stored figure can be a small fraction of what you were charged to get the data in.

That asymmetry is why people look at a log group holding a few gigabytes, decide it is harmless, and miss that it cost several times that to arrive. The storedBytes value shown on a log group is the compressed number. It tells you about your storage line, not your ingestion line.

The log groups you never created

Most runaway ingestion comes from things switched on with a checkbox. VPC Flow Logs pointed at CloudWatch. EKS control plane logging with every type enabled. Lambda functions emitting a full structured payload per invocation. CloudTrail delivering to a log group as well as S3. RDS general query logs left on after a debugging session.

None of those feel like a decision at the time, and all of them scale with traffic rather than headcount. That is why the slope keeps rising while the team stays the same size.

The most effective lever here is not a CloudWatch setting at all. It is cutting verbosity at the source, before the bytes leave your application. Dropping debug-level logging in production, sampling health check lines, trimming repeated stack frames. Filtering after ingestion does nothing for this meter, because you already paid.

Retention: the one-line fix almost nobody applies

Log groups default to Never Expire. Not thirty days, not a year. Forever. Every log group created implicitly by another AWS service inherits that default, so an account accumulates hundreds of them without a single deliberate retention decision.

Find the offenders first. This lists every log group with no retention policy, alongside its compressed size:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output table

The ?!retentionInDays filter selects entries where the field is absent, which is how the API represents indefinite retention. There is no explicit “never” value to match on.

aws logs put-retention-policy 
  --log-group-name /aws/lambda/order-processor 
  --retention-in-days 30

Three things are worth knowing before running that across an account.

1. The value is an enum, not a free integer. Accepted values run 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731 and upward to ten years. Asking for 45 days is rejected.
2. Deletion is not instant. Expired events are marked for deletion and typically removed within about three days. They stop contributing to your storage charge once marked, and are excluded from the reported storedBytes, so the billing effect lands before the data physically goes.
3. Reverting means removing the policy, not setting zero. To go back to indefinite retention you call delete-retention-policy rather than passing a zero value.

Dry run before you bulk-apply. This prints what would change without touching anything:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].logGroupName' 
  --output text | tr 't' 'n' | while read -r lg; do
    echo "would set 30-day retention on: $lg"
  done

Retention is where compliance and cost pull against each other, and cost should lose. If an auditor needs seven years of logs, seven years is the answer. The move is to export them to S3 and apply lifecycle rules there, rather than paying CloudWatch rates for archival data you will query twice a decade.

Cardinality: the multiplier nobody sees coming

This is the part that surprises people, because it breaks the intuition that cost tracks traffic. Cardinality can multiply your metrics bill by an order of magnitude while request volume stays flat.

CloudWatch treats every unique combination of namespace, metric name and dimension values as a separate metric. Not a variant of one metric. A separate, separately billed metric. Publish OrderLatency with a Region dimension holding three values and you have three metrics. Add a CustomerId dimension and you have three times however many customers you have.

The dangerous dimension values are the ones that look like identifiers: request IDs, trace IDs, pod names, session tokens, customer IDs, IP addresses, URL paths with IDs embedded in them. Each new value is a new billable series that persists and accrues.

Metric filters have a guard rail, and it bites

Metrics extracted from logs by a metric filter bill as custom metrics. A filter can carry up to three dimensions, and AWS documents an explicit safety mechanism: CloudWatch Logs disables a metric filter that generates a thousand distinct dimension name/value pairs within a period.

That guard rail is useful, but read what it means operationally. Hitting it does not just cost money, it silently stops the filter, so the metric you built an alarm on stops receiving data. If that alarm treats missing data as not breaching, it goes quiet rather than firing. You lose the alert and keep the bill.

Audit what exists before adding more, and count what a namespace actually produces:

aws logs describe-metric-filters 
  --log-group-name /aws/lambda/order-processor

aws cloudwatch list-metrics 
  --namespace MyApp/Orders 
  --query 'length(Metrics)'

If that count is materially larger than the number of distinct things you consciously measure, you have found your multiplier.

EMF does exactly what you tell it

Embedded Metric Format is a good design: you emit structured events, CloudWatch extracts metrics from declared fields, and you get high-cardinality context in the logs with low-cardinality metrics for alarming. That split is the point.

The failure mode is promoting a high-cardinality field into the dimension set. AWS documents this plainly: declare something like a request ID as a dimension and EMF will, by design, create a custom metric for every unique combination. Keep identifiers in the log body as properties, and keep dimensions to values you would put on a dashboard axis.

Anomaly detection alarms cost more than they look

An anomaly detection alarm does not evaluate one metric. It evaluates the source metric plus the upper and lower bounds of the predicted band, and those bounds are metrics too. That is roughly a threefold multiple against a standard threshold alarm, per alarm.

It earns its keep on genuinely seasonal signals where a static threshold produces constant noise. Switching an entire alarm estate to it “to be safe” triples a line item for very little detection benefit.

Query cost spikes exactly when you need it most

Logs Insights bills on the volume a query scans. Not rows returned. Not query count. Scanned bytes. A query returning twelve lines that scans a terabyte costs the same as one returning a million lines that scans a terabyte.

During an incident, an engineer runs a dozen exploratory queries across a wildcard log group selection and a wide time window. The pricing model charges you hardest at the moment the tool is most valuable, which is an uncomfortable incentive. Two habits reduce it without reducing your ability to debug:

• Narrow the time window first, then the filter. Scanning is bounded by the selected range across the selected groups. Starting at fifteen minutes and widening is far cheaper than starting at a week and narrowing.
• Select specific log groups, not patterns. Querying across every Lambda log group multiplies scanned bytes by the number of functions when you only care about one.

Watch for scheduled queries too. A dashboard widget or automation running an Insights query every few minutes against a large log group is a fixed monthly cost nobody remembers creating.

Finding out where your CloudWatch costs come from

Attribute before you cut. Deleting log groups because they look big is how teams lose the one log they needed three weeks later.

1. Split the bill by meter. In Cost Explorer, filter service to CloudWatch and group by usage type. The region-prefixed usage types map to the meters directly: DataProcessing-Bytes is log ingestion, TimedStorage-ByteHrs is log storage, DataScanned-Bytes is Logs Insights, and CW:Requests and CW:GMD-Metrics cover API calls.
2. Rank log groups by ingestion. The IncomingBytes metric in the AWS/Logs namespace, dimensioned by log group, is the per-group ingestion figure. Graph it with the Sum statistic. One documented gotcha: use an absolute range covering exactly thirty days, because a relative range or a different length produces a number that does not correspond to a billing month.
3. Check stored bytes separately. A log group can be small in ingestion but large in storage because it has accumulated since the account was created. That is a retention problem, not a verbosity problem, and the fix differs.
4. Count custom metrics per namespace. Compare the count against your mental model of what you measure. The gap is cardinality.
5. Look inside the noisiest group. Once you know which log group dominates, find out which streams are responsible before changing any application code.

For that last step, a Logs Insights query over a short window is enough. Keep the range tight, because this query is itself billed on what it scans:

stats count(*) as events, sum(strlen(@message)) as approx_bytes by @logStream
| sort approx_bytes desc
| limit 20

That approximates byte volume per stream, which is usually enough to identify the chatty component. It is an approximation, not a billing figure, because it counts message characters rather than the metered event size.

If you manage several accounts and want attribution done for you, cost platforms such as Vantage and CloudZero break CloudWatch spend down by usage type and resource without you building dashboards. Worth the spend if client reporting is part of the job. For one account and one engineer, Cost Explorer plus the steps above gets the same answer.

Cheaper destinations, and what they cost you

Once you know your ingestion is dominated by logs you rarely read, there are three real options. None is free of trade-offs.

• The Infrequent Access log class. Set at the log group level, it ingests at a materially lower per-GB rate in exchange for a reduced feature set. Historically that meant no metric filters, no alarming from logs and no Live Tail, though the capability list has expanded since launch. Check the current log classes documentation rather than assuming, and note that a log group you extract metrics from is not a candidate.
• Subscription filter to S3 via Amazon Data Firehose. Sends log data to object storage where lifecycle rules and Athena queries apply. You trade Insights and the console experience for much cheaper long-term retention. Good for audit and forensic logs, poor for anything you troubleshoot weekly.
• A separate log stack. Grafana Cloud, or self-hosted Loki and Prometheus on a VPS from a provider like Contabo or InterServer, moves cost from per-GB metering to a fixed monthly box you administer. The economics can be dramatically better at volume. The price is that you now own retention, backups, upgrades and the availability of your own observability during an outage, which is exactly when you least want to be debugging it.

The honest version: CloudWatch is expensive partly because it is operated for you and wired into everything. At a few dozen gigabytes a month, migrating away costs more engineering time than it saves. At terabytes, the arithmetic flips hard.

Troubleshooting a bill that will not go down

You set retention everywhere and storage barely moved. Check whether storage was ever the dominant cost. If your usage type breakdown was mostly DataProcessing-Bytes, retention was never going to fix it.

Custom metric count grows with no code changes. Something is emitting a dimension whose value set is open-ended. Look for deployment identifiers, container IDs, or a version string appended to a dimension.

An alarm went quiet and the bill jumped the same week. Suspect a disabled metric filter. Check the filter still exists and is producing data, and verify the alarm’s missing data treatment. Alarms treating missing data as not breaching fail silently by design.

API request charges you cannot place. External dashboards are the usual answer. A Grafana instance polling CloudWatch, or any tool calling GetMetricData on a short interval, bills per metric requested. Widening the refresh interval on dashboards nobody watches live is a quick win.

Costs dropped, then came back. Look at your infrastructure as code. If Terraform or CloudFormation creates log groups without a retention argument, every apply restores the default. Fixing it in the console fixes it until the next deploy.

Common mistakes

• Treating storage as the problem because it is the number the console shows. Ingestion is usually larger and invisible in the log group list.
• Adding a dimension because it would be useful to filter by, without checking how many distinct values it can take.
• Filtering logs after they reach CloudWatch. The ingestion charge already applied.
• Deleting log groups to cut cost without checking whether anything alarms on them.
• Setting retention in the console while the IaC that owns the log group has none configured.
• Enabling every EKS control plane log type when only the audit log is consumed.

Best practices for controlling CloudWatch costs

• Make retention a required argument. If log groups come from Terraform or CloudFormation, treat a missing retention setting as a policy violation and fail the plan. Checkov and similar scanners have rules for exactly this.
• Decide verbosity per environment, not per service. Debug in staging, info in production, with a documented way to raise it temporarily during an incident.
• Keep dimensions bounded, identifiers in the message body. If you cannot enumerate a dimension’s possible values on a whiteboard, it does not belong in the dimension set.
• Alarm on the bill itself. A billing alarm on estimated charges, or an anomaly monitor scoped to CloudWatch, turns a quarterly surprise into a same-week signal.
• Review the top five log groups by ingestion monthly. Five minutes with the IncomingBytes graph catches new noise before it compounds.
• Route archival logs out deliberately. Anything kept for compliance rather than operations belongs in S3 with lifecycle rules from the start.

Frequently asked questions

Why is my CloudWatch bill so high when my traffic has not changed?

Two causes account for most cases. Storage accumulating because log groups default to indefinite retention, so the bill grows every month even at constant ingestion. Or metric cardinality, where a dimension with an open-ended value set keeps creating new billable series. Neither tracks request volume, which is why the bill and the traffic graph disagree.

Does deleting a log group reduce my bill immediately?

It stops future storage charges for that data and nothing else. Ingestion already billed is spent, and deleting the group does not stop whatever was writing to it, which will simply recreate it. Turn off the source first, then delete.

What counts as one custom metric for billing?

One unique combination of namespace, metric name and dimension values. The same metric name published with three different dimension value sets is three billable metrics. This is the single most important sentence in CloudWatch pricing and the easiest one to skim past.

Should I switch log groups to the Infrequent Access class?

For logs you keep but rarely read, it is usually a straightforward saving on ingestion. For any group you extract metrics from, alarm on, or stream live, it is not a candidate. The feature gap between the classes has narrowed since launch, so check current documentation rather than ruling it out on old information.

Is Logs Insights charged per query or per gigabyte?

Per gigabyte scanned. Rows returned are irrelevant. A tightly filtered query over a wide range across many log groups is expensive; a broad query over fifteen minutes on one log group is cheap. Narrow the range before you narrow the filter.

Do VPC Flow Logs have to go to CloudWatch?

No. Flow logs can be delivered to S3 instead, avoiding CloudWatch ingestion and storage entirely, at the cost of losing Insights queries over them. If you analyse flow logs occasionally with Athena rather than continuously in the console, S3 is the better destination.

The one thing to take away

CloudWatch costs are not one number with one cause. They are four meters running independently, and each has a different fix: cut verbosity at the source for ingestion, set retention for storage, narrow scope for queries, bound your dimensions for metrics. Applying the wrong fix to the wrong meter is why so many cost-reduction efforts produce a bill that barely moves.

So attribute first. Split by usage type, rank log groups by IncomingBytes, count custom metrics per namespace. Ten minutes of that tells you which lever you are actually pulling, and stops you deleting the log group that turns out to be the only place a production alarm was getting its data.

Need a hand with your CloudWatch spend?

This is well-defined work with a measurable outcome, which makes it a good fit for a short engagement. Things I can help with:

• Breaking your CloudWatch bill down by meter and by log group, so you know what you are paying for before anything changes
• Auditing custom metric cardinality and identifying which dimensions are multiplying your series count
• Building a retention policy that satisfies your compliance requirement without paying CloudWatch rates for archival data
• Enforcing retention and log class in Terraform or CloudFormation so the fix survives the next deployment
• Designing a subscription filter and S3 archive path for logs you need to keep but rarely query
• Reviewing alarms and dashboards for anomaly detection and API polling costs that are not earning their keep

If you want a second opinion, send me your Cost Explorer breakdown grouped by usage type, or the output of describe-log-groups for the account. That is usually enough to say where the money is going before we talk about anything else.

The post The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality appeared first on John Nessime.

That gap between what a table logically contains and what the object store physically holds is what Iceberg table maintenance exists to close. Iceberg never overwrites a file in place. Deletes and updates write new files and a new pointer, and the old files sit where they are until something explicitly removes them. If nothing does, the bill only moves one way.

This post covers where those bytes hide, why expiring snapshots often frees nothing, how orphan files accumulate without showing up in any metadata you would think to check, and the order to run cleanup so you do not corrupt a table while shrinking it.

Why deleting rows does not shrink anything

Every write produces a snapshot: an immutable view of the table at a point in time, pointing at a manifest list, which points at manifests, which point at data files. Nothing in that chain is mutated. A new commit builds a new chain that reuses most of the old files.

That design is what gives you time travel, rollback and safe concurrent writes. It is also why storage grows silently. A data file only becomes a deletion candidate once no retained snapshot references it. Delete a year of rows and you free nothing, because the snapshot from before the delete still points at every one of those files.

So maintenance is not an optimization. It is the mechanism by which deletes become real.

The four places your storage is actually sitting

When a lakehouse bucket balloons, the bytes are in one of four categories. Each needs a different tool, and confusing them is how teams end up running the same job for weeks with no effect.

1. Data files held alive by snapshots you never expired

This is the one everyone knows about and usually the largest share. Snapshots accumulate until something expires them, and nothing expires them by default.

Before changing anything, look at the table’s own metadata. Iceberg exposes it as queryable tables:

-- How many snapshots exist, and how far back do they go?
SELECT committed_at, snapshot_id, operation
FROM db.orders.snapshots
ORDER BY committed_at;

-- How much data does the CURRENT snapshot reference?
SELECT count(*) AS file_count,
       sum(file_size_in_bytes) AS bytes
FROM db.orders.files;

Compare that second number against what your object store reports for the table prefix. The difference is your maintenance debt. On a neglected table the referenced set is often a minority of the physical bytes.

Expiring snapshots is a Spark stored procedure:

CALL my_catalog.system.expire_snapshots(
  table => 'db.orders',
  older_than => TIMESTAMP '2000-01-01 00:00:00',
  retain_last => 10,
  stream_results => true
);

Three arguments matter. older_than sets the age cutoff. retain_last is the floor: it keeps that many ancestor snapshots regardless of age, which is your rollback safety net. stream_results streams the delete list back to the driver instead of collecting it at once, which on a table with long history is the difference between the job finishing and the driver running out of memory.

The trap is that Iceberg treats these conditions as guidance rather than a strict contract, leaning toward retention when they conflict. An aggressive older_than plus a generous retain_last gives you the generous answer. Safe, but it surprises people who expected the timestamp to win.

Branches and tags are separate: they are not subject to snapshot age at all. A tag created for a regulatory export a year ago pins its entire snapshot chain, and no amount of expiring will touch it. Reference age is governed by history.expire.max-ref-age-ms.

2. Orphan files that no metadata points at

This is the invisible one, and the reason many teams conclude that maintenance “does not work.”

When a Spark task writes a data file and the job then fails before the commit lands, that file is already on disk. It is in no manifest, no snapshot, no metadata file. Expiry will never find it, because expiry compares the file sets of expired snapshots against retained ones, and a file that was never in a snapshot is in neither set.

Streaming writers, speculative execution, retried stages and killed compaction jobs all produce these. The only way to find them is to list the storage prefix and diff it against the metadata:

-- Always dry run first. This lists candidates without deleting.
CALL my_catalog.system.remove_orphan_files(
  table => 'db.orders',
  dry_run => true
);

Read that output before you run it live. Recent timestamps or unfamiliar paths mean stop and investigate.

The safety mechanism is older_than, which defaults to three days. That default is not arbitrary caution. Orphan removal cannot distinguish a file abandoned by a dead job from one being written right now by a live one. Set the interval shorter than your longest-running write and the procedure can delete a file a job is about to commit, corrupting the table. Shorten it only when you know your write durations, and never below your longest backfill.

The second hazard is less obvious. Iceberg compares file paths as strings. If your metadata records s3a://bucket/... and your listing returns s3://bucket/..., or you migrated between HDFS namenodes and the authority changed, every live file looks like an orphan. Running that deletes the table. The procedure defends against it by erroring on a prefix mismatch by default, via prefix_mismatch_mode with ERROR, IGNORE and DELETE as options. Resolve a mismatch with equal_schemes or equal_authorities. Do not reach for DELETE to silence it.

3. Metadata JSON files and manifests

Every commit writes a new table metadata JSON file. On a batch table that is a handful a day. On a streaming table committing every minute it is a lot of small objects, and those cost you in request charges and planning time as much as in bytes.

ALTER TABLE db.orders SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '20'
);

The first property is off by default. The second defaults to 100 and controls how many previous metadata files are tracked in the metadata log. The catch is that cleanup only deletes tracked files. If you have been running with deletion disabled, files that fell out of the log are already untracked, and turning the property on later will not reach back for them. They are orphans now, and only orphan file removal clears them.

You can see the tracked set with SELECT timestamp, file FROM db.orders.metadata_log_entries.

Manifests rarely dominate the bill, but a table with thousands of tiny ones plans queries slowly. rewrite_manifests consolidates them. Treat it as a query performance job and run it after the deletion work, so it is not reorganizing metadata about to be thrown away.

4. The object store’s own hidden inventory

Most Iceberg guides skip this category, and it often explains a bill nobody can account for. Iceberg has no visibility into any of it.

• Incomplete multipart uploads. Large Parquet files upload in parts. When a writer dies partway, the completed parts sit in the bucket, billed as storage, and they do not appear in a normal object listing. Nothing removes them automatically.
• Noncurrent object versions. With versioning enabled, every file your maintenance job “deletes” becomes a noncurrent version you keep paying for, indefinitely, unless a rule expires it.
• Delete markers. Deletes on a versioned bucket leave markers behind. Small individually, but on a table with millions of expired files they add up and slow listings down.

A lifecycle configuration covers all three. On S3 the shape looks like this:

{
  "Rules": [
    {
      "ID": "abort-stale-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    },
    {
      "ID": "expire-noncurrent-versions",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 14 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}

Now the warning that matters most here. Never put an age-based expiration or storage class transition rule on the prefix holding live Iceberg data files. A rule deleting objects older than ninety days will happily delete data files a current snapshot still references, and the table breaks with missing-file errors on the next read. Transitioning them to an archive class is just as bad, because reads then fail or stall on retrieval.

The rules above act only on things the table does not reference: abandoned upload parts, superseded versions, tombstones. Iceberg decides what live data to delete; lifecycle rules clean up after that decision.

This applies equally on S3-compatible storage. If you run a lakehouse on Cloudflare R2, Backblaze B2, Wasabi or Contabo Object Storage to dodge egress charges, check which lifecycle actions the provider actually implements. Support for aborting incomplete multipart uploads varies, and a provider that silently ignores the rule leaves you paying for parts you cannot see.

Running Iceberg table maintenance in a safe order

Order matters, and the reason is not stylistic. These procedures read each other’s inputs.

1. Compact first, if you are compacting. rewrite_data_files writes new files and leaves the old ones referenced by older snapshots. Doing it after expiry means the space you just freed is immediately replaced.
2. Expire snapshots. This is what turns unreferenced data files into deleted ones. It must happen before orphan removal, because expiry needs to read the manifests belonging to the snapshots it is dropping.
3. Remove orphan files. Only after expiry has committed. Running the two concurrently is a genuine race: orphan removal can delete files the in-flight expiry job still needs to read, and expiry fails.
4. Rewrite manifests, if planning is slow. Optional, and worth measuring before and after rather than running on faith.

On cadence: expiry is cheap and should run often, daily or better on high-commit tables. Orphan removal is expensive because it lists the entire table prefix, and the official guidance is to run it periodically but not frequently. Weekly on a busy table, monthly on a quiet one. Where supported, the prefix_listing option makes that scan substantially cheaper on object stores, where recursive directory listing is the slow part.

Letting table properties do the boring part

Procedures are what you run. Properties are what you set once so the procedures behave sensibly without a config file per table.

ALTER TABLE db.orders SET TBLPROPERTIES (
  'history.expire.max-snapshot-age-ms' = '604800000',   -- 7 days
  'history.expire.min-snapshots-to-keep' = '10'
);

The snapshot age default is five days and the minimum to keep defaults to one. Setting both means a generic maintenance job can call expire_snapshots with no tuning and each table gets its own policy. A regulated reporting table keeps ninety days; a staging table keeps an hour. That beats maintaining per-table retention in your scheduler.

The other property worth knowing is gc.enabled. Setting it false blocks physical deletion entirely. It exists for tables whose files are shared with something else, and it is correct for those. It is also a common accidental cause of “maintenance runs successfully and frees nothing.” Check it before debugging anything else.

Troubleshooting: the job succeeded and nothing shrank

• The counters were zero. expire_snapshots returns counts of deleted data files, delete files, manifests and manifest lists. All zero means nothing qualified. Check whether retain_last or the table’s minimum-to-keep property is holding everything.
• A branch or tag is pinning history. Refs do not age out with snapshots. Look at the table’s refs metadata table and see what is anchored.
• Garbage collection is off. Either gc.enabled is false, or the table was created by the snapshot procedure, which is barred from expiry precisely because it does not own its data files.
• Files were deleted but the bucket did not shrink. Versioning. The deletes created noncurrent versions. This is the most common cause of healthy counters and a flat bill.
• Orphan removal hit a file limit. Some managed implementations cap deletions per run. On a backlogged bucket you may need several passes before the number stops dropping.
• The driver died. On a table with very long history, add stream_results and raise max_concurrent_deletes so deletes spread across a thread pool instead of serializing.

Common mistakes

• Running orphan file removal with a short retention window on a table that has long-running writers. This is the one that can actually destroy data.
• Setting a bucket-wide expiration rule to “clean up the lake” and taking live data files with it.
• Enabling metadata deletion after the fact and expecting it to collect the already-untracked backlog.
• Scheduling expiry and orphan removal as two independent cron jobs that can overlap.
• Skipping the dry run because the last twenty were fine. The one that matters is the run after somebody changes the storage endpoint.
• Setting retention from a storage target rather than a recovery target. Ask how far back you need to roll back, then keep that much.

Best practices worth adopting

• Express retention as a table property, not a job argument. It travels with the table and survives whoever wrote the scheduler.
• Track referenced bytes against physical bytes per table as a metric, not an occasional query. The ratio drifting upward is the early signal; the invoice is the late one. Any time series backend you already run will do.
• Give maintenance its own compute and its own credentials. It is the only workload in the lake that needs delete permissions.
• Log the returned counters from every run. When someone asks in six months where the storage went, that log is the answer.

Frequently asked questions

How often should I expire Iceberg snapshots?

As often as your commit rate justifies. A streaming table committing every minute benefits from daily or hourly expiry; a nightly batch table is fine with weekly. Set the window from how far back you would realistically need to roll back, then keep a few extra snapshots as a floor.

Does expiring snapshots delete my data?

It deletes data files that no retained snapshot references. Current table contents are never affected. What you lose is the ability to time travel to or roll back to the expired snapshots, so agree the window with anyone who queries the table as of a date.

Why do orphan files exist if Iceberg commits are atomic?

The commit is atomic; the writes preceding it are not part of that atomicity. Data files are written first, then the commit makes them visible. If the job dies between those steps, the files exist and the commit never happened. Atomicity guarantees readers never see a partial table, not that failed writes clean up after themselves.

Can I use an S3 lifecycle rule instead of Iceberg maintenance?

No, and trying is how tables get broken. Lifecycle rules act on object age. Whether Iceberg still needs a file has nothing to do with age; a data file written two years ago can be referenced by the current snapshot. Lifecycle rules are right for upload parts, noncurrent versions and delete markers, and wrong for anything the table might still point at.

Do I need Spark to run Iceberg table maintenance?

Spark has the most complete set of procedures, but it is not the only route. Flink has its own maintenance API, Trino exposes expiry and orphan removal as table procedures, and several managed catalogs run scheduled maintenance for you. The mechanics are identical regardless of engine, because the work is defined by the table format rather than the compute. Only the syntax and the exposed arguments change.

Is orphan file removal safe to automate?

Yes, once you have verified that the retention window comfortably exceeds your longest write and that the paths in your metadata match those your storage listing returns. Automate it after watching a few dry runs, not before. Erroring on a prefix mismatch is a stop signal, not something to configure away.

The one thing to take away

Iceberg table maintenance is not housekeeping you get to postpone. It is the step where logical deletes become physical ones, and until it runs, storage only grows. The bill is the visible symptom; the same neglect also makes query planning slower and recovery windows fuzzier.

Set retention as a table property so the policy lives with the data. Expire snapshots on a schedule matching your commit rate. Remove orphan files less often, always after expiry, always with a window longer than your slowest write. Then add the storage lifecycle rules that clean up what Iceberg cannot see. Any one alone leaves money on the table.

Need help getting your lakehouse storage under control?

Most of the work here is not writing the procedures. It is working out which of the four categories your bytes are in, and building a maintenance path that will not break a table at three in the morning. That is the kind of thing I do:

• Audit an existing lake and produce a per-table breakdown of referenced bytes against physical bytes, so you know where the money is before touching anything
• Build a scheduled maintenance pipeline with correct sequencing, dry-run gates and per-table retention driven by table properties
• Set retention against real rollback and time travel requirements, gathered from the people who depend on them
• Design storage lifecycle rules that clear incomplete uploads, noncurrent versions and delete markers without touching live data files
• Diagnose maintenance jobs that report success and free nothing, including prefix mismatches, pinned refs and disabled garbage collection
• Add table health metrics and alerting so drift shows up as a graph rather than a surprise on the invoice

If you want a second opinion, send me the output of your snapshots metadata table, your current table properties, or the counters from a run that did not do what you expected. That is usually enough to tell you where the bytes are.

The post Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing appeared first on John Nessime.

Then someone in finance asks what the reporting pipeline costs. You open Cost Explorer, group by Project, and over half the bill lands in a bucket called “No tag key”.

Nothing is broken. The tags are there. They just aren’t on the bill, because tagging a resource and activating a cost allocation tag are two separate operations, and AWS never warns you when you’ve done only the first.

This post covers getting AWS cost allocation tags to cover a messy account: the activation gap, what backfill will and won’t repair, where coverage leaks even with decent discipline, and a retrofit order that converges.

A resource tag and a cost allocation tag are not the same thing

A resource tag is metadata on the resource. A cost allocation tag is a billing dimension. Getting the second means taking a key that already exists on your resources and explicitly switching it on in the Billing and Cost Management console.

Two details make this bite. First, activation lives in the management account. From a member account, the page tells you tags are managed at the payer level and that’s the end of it. Plenty of teams have a good tagging standard and no access to the one screen that makes it count. Related trap: an account moving between organizations has its activated keys reset to inactive.

Second, the lag. AWS documents up to 24 hours for a newly applied key to appear on the activation page, and up to another 24 hours after you activate before it takes effect. The loop between “we agreed a standard” and “I can see it in Cost Explorer” is two days at best.

Past a handful of keys, drive it from the API. Listing the inactive ones is the fastest way to find what you forgot:

# Every key AWS knows about that is NOT currently billing-active
aws ce list-cost-allocation-tags --status Inactive

# Switch a set of them on
aws ce update-cost-allocation-tags-status --cli-input-json '{
  "CostAllocationTagsStatus": [
    {"TagKey": "Project",     "Status": "Active"},
    {"TagKey": "Environment", "Status": "Active"},
    {"TagKey": "Owner",       "Status": "Active"}
  ]
}'

User-defined keys appear in the report with a user: prefix and AWS-generated ones carry the reserved aws: prefix, so they never collide. There’s a cap on active keys, raisable through Service Quotas, which you almost certainly don’t need to raise.

What the backfill repairs, and what it quietly doesn’t

Activation used to be strictly forward-looking, so activating late permanently orphaned everything before it. That’s no longer true. The management account can ask AWS to reapply current activation status across up to twelve months of historical cost data.

# Must be the first day of a month, no earlier than twelve months back
aws ce start-cost-allocation-tag-backfill 
  --backfill-from 2025-01-01T00:00:00Z

• Start date must be the first day of a month, and can’t reach back more than twelve months.
• One request per 24 hours, and you can’t submit another while one is processing.
• It backfills activation status, not tag values.

That last point catches people. If the resource wasn’t carrying the tag when the cost was incurred, backfill gives you nothing for those months. It fixes “I forgot to activate.” It does not fix “I forgot to tag.” A tag applied in June and activated in November backfills to June and no further, whatever start date you hand it.

It also cuts both ways. Backfill applies the current status of every key, so a key sitting inactive right now gets stripped out of historical data too. If you’re pruning an overgrown tag list, backfill before you deactivate the old keys, not after, or you’ll erase the history you were trying to keep.

Where AWS cost allocation tags actually leak coverage

With activation sorted and every Terraform-managed resource tagged, a stubborn slice of the bill still refuses to attribute. It matters which slice, because the fixes have nothing in common.

Charges no resource can carry

Support charges, refunds, credits, tax, some Marketplace subscriptions. There’s no resource to hang a tag on, so tagging discipline never touches them. On accounts with a large support plan or heavy credit activity, that’s a visible fraction of the invoice.

The answer is account tags. AWS added support for using AWS Organizations account tags as a cost allocation dimension, applying to all metered usage inside the tagged account and flowing into Cost Explorer, Budgets, Cost Anomaly Detection and Cost Categories. Because attribution happens at the account level, it picks up exactly what resource tags can’t reach.

This is recent enough that most tagging guides predate it, and it’s the highest-value item here if your accounts already map roughly onto teams.

Resources created by other resources

This is the quiet one, and usually the biggest surprise gap in an otherwise well-run account. Terraform’s provider-level default_tags applies to resources Terraform creates. It does not reach the EC2 instances an Auto Scaling group launches, because Terraform didn’t create those, the ASG did. Your ASG is tagged. Your instances aren’t. Same for volumes attached at launch, ENIs, snapshots and AMIs.

data "aws_default_tags" "current" {}

resource "aws_autoscaling_group" "app" {
  # ... other config ...

  # Push provider default tags down to launched instances
  dynamic "tag" {
    for_each = data.aws_default_tags.current.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_launch_template" "app" {
  # ... other config ...

  tag_specifications {
    resource_type = "instance"
    tags          = data.aws_default_tags.current.tags
  }

  tag_specifications {
    resource_type = "volume"
    tags          = data.aws_default_tags.current.tags
  }
}

For a standalone instance, volume_tags on aws_instance covers its attached volumes. General rule: whenever one AWS resource provisions another on your behalf, assume tags don’t propagate until you’ve checked the created resource in the console, not the plan output.

Shared infrastructure that belongs to everyone

NAT gateway data processing, VPC interface endpoints, a Transit Gateway, a shared load balancer, the central logging bucket. You can tag all of these perfectly and the tag says Project=platform, which is accurate and useless for chargeback.

Transit Gateway is worth calling out because it supports tagging the gateway resource itself, so data processing and transfer charges pick up the tag rather than only attachment hours. That gap used to swallow a lot of network spend.

Beyond that, stop chasing tags and do allocation math instead. Cost Categories support split charge rules that redistribute a shared bucket across your other categories, using proportional splits, fixed percentages or an even division. Proportional is the sane default; fixed percentages are for when finance already agreed a number and the report has to match it.

Container spend hiding behind one instance tag

A shared EKS or ECS cluster is one tagged compute footprint running fifteen teams’ workloads. The instance tag gives you the total and nothing about who caused it. Split cost allocation data is the native answer: opt in and the cost and usage export gains rows breaking cluster cost down to individual pods or tasks. It won’t account for idle headroom you’re deliberately carrying, but it beats one line item labelled “the cluster”.

Case drift and near-duplicate keys

Keys and values are case-sensitive. Environment, environment and ENV are three distinct keys producing three columns, each holding a partial and misleading picture. Tag Policies in AWS Organizations exist for this: define the key including its capitalization, define permitted values, and get an org-wide compliance report. Cheapest governance on this list, because it costs nobody a deploy.

A retrofit order that converges

Retrofits stall because people start at the bottom, sweeping untagged resources by hand while the creation path keeps producing more. This order avoids that.

1. Measure first. Group Cost Explorer by your intended key and record what percentage of spend falls into “No tag key”. That’s your baseline and your only honest progress metric.
2. Cut the key set down. Three to five keys: owner, environment, and a project or cost centre. Coverage on five beats partial coverage on thirty.
3. Activate immediately, even with terrible coverage. Both clocks start now: the activation delay and the rolling twelve-month backfill window. There’s no downside to activating a key that’s barely applied.
4. Rank the gap by spend, not resource count. Sort untagged line items by cost descending. It’s usually a few NAT gateways, a couple of oversized databases and one forgotten bucket. Ten thousand untagged security groups cost nothing.
5. Fix the creation path before the backlog. IaC defaults, launch template tag specifications, ASG propagation. Sweeping first means sweeping again next month.
6. Sweep the stragglers in bulk. The Resource Groups Tagging API works across services and regions, which beats going service by service.
7. Backfill, then re-measure. Compare against step one. If the number barely moved, you fixed the wrong resources.

# Find what already carries a Project tag, to diff against inventory
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project

# Apply tags to a specific set of ARNs
aws resourcegroupstaggingapi tag-resources 
  --resource-arn-list arn:aws:rds:eu-west-1:111122223333:db:reporting-01 
  --tags Project=reporting,Environment=prod

Run it against a couple of ARNs and confirm in the console before looping over a list. Bulk tagging is trivially easy to point at the wrong account.

Enforcing tags without becoming the tag police

Four mechanisms, in increasing order of blast radius. Most teams need the first three and should be careful with the fourth.

• IaC defaults. Provider-level default tags plus required module variables. Cheapest, catches the majority, nobody thinks about it.
• CI validation. A linter rule that fails the pull request on missing required tags. The failure lands on the person who can fix it, when they can fix it.
• Detective controls. The AWS Config managed rule for required tags flags non-compliant resources without blocking anything. Tells the truth, breaks nothing.
• Preventive controls. A service control policy denying creation when a required key is absent, keyed on a null check against the request tag.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRDSCreateWithoutProjectTag",
      "Effect": "Deny",
      "Action": "rds:CreateDBInstance",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    }
  ]
}

The honest trade-off: service control policies are where tagging programmes go to die. Not every API supports tag-on-create, and not every legitimate workflow passes tags through the way you expect. Roll a broad deny across the org on day one and you break somebody’s deployment on day two, the ticket lands on you, and the policy is gone by day three along with everyone’s appetite for the project.

Scope it to one service at a time, starting in sandbox. I’d rather run Config plus Tag Policies for a quarter and fix the top three offenders in the IaC. The deny is a finishing move, not an opening one.

Measuring coverage so it doesn’t rot

Cost Explorer gives the headline number. For a per-service breakdown, query the cost and usage export in Athena. One caveat: tag column naming differs between export versions, older ones giving a flattened column per activated key and newer formats a map column. Run a DESCRIBE first rather than guessing.

-- Confirm the actual column names first
DESCRIBE your_cur_database.your_cur_table;

-- Then: untagged spend, ranked by service
SELECT   line_item_product_code,
         SUM(line_item_unblended_cost) AS untagged_cost
FROM     your_cur_database.your_cur_table
WHERE    your_project_tag_column = ''
  AND    line_item_unblended_cost > 0
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

That ranked list is your work queue; everything below the top twenty is noise. The list call also returns a last-used timestamp per key, the cleanest way to find keys nobody has applied to anything in months.

If you’d rather not maintain the dashboard yourself, this is a fair place to buy rather than build. Grafana Cloud sits on Athena comfortably if you already run Grafana, and platforms like CloudZero, Vantage and nOps handle allocation modelling out of the box. Their real value is the shared-cost and commitment maths. Their real cost is another vendor between you and your billing data, and several price against a percentage of the spend they watch, so run that number first.

Troubleshooting

• The key isn’t in the activation list. It must exist on at least one resource first, and propagation takes up to a day. Past that, check you’re in the management account and the key matches exactly, including case.
• Activated, but everything still shows untagged. Give it a full billing cycle. A key switched on mid-month leaves the first half of that month untagged until you backfill.
• Tags stopped working after an account move. Activation doesn’t survive a move between organizations. Reactivate from the new management account, then backfill.
• Cluster costs still land on one team. Instance tags can’t split a shared cluster. Opt into split cost allocation data, or treat the cluster as a shared cost in a split charge rule.
• You’ve hit the active key limit. Almost always too many keys rather than a real need for a quota increase. Use last-used timestamps to find dead ones, and backfill before deactivating.

Common mistakes

• Assuming a tagged resource is a tagged cost. It isn’t until you activate the key.
• Waiting for good coverage before activating. Activate first, improve coverage second, backfill third.
• Deactivating stale keys and then backfilling, which wipes their history out of the data.
• Measuring coverage by resource count instead of by spend.
• Rolling out a broad tag-enforcing SCP in production before anyone has read a compliance report.
• Trusting default_tags to reach resources Terraform didn’t directly create.
• Putting anything sensitive in a tag value. Tags surface in billing exports that finance and third-party tools read.

Best practices

• Keep the mandatory set to three to five keys, with defined allowed values rather than free text.
• Pin casing in the standard and enforce it with Tag Policies, because AWS won’t do it for you.
• Tag accounts as well as resources. Account tags cover charges resource tags structurally cannot.
• Treat coverage as a tracked percentage of spend with an owner and a monthly trend, not a project that finishes.
• Accept an irreducible remainder. Chasing the last few percent by hand costs more than it returns.
• Put allocation rules in Cost Categories rather than inventing tag values that encode a split.

Frequently asked questions

Are AWS cost allocation tags retroactive?

Partly. Activation applies going forward, but the management account can request a backfill reapplying current activation status across up to twelve months. The catch is that the resource must have carried the tag at the time, so backfill recovers late activation, never late tagging.

Why don’t my tags appear in Cost Explorer?

In order of likelihood: the key was never activated in the Billing console; you’re looking from a member account; the activation delay hasn’t elapsed; or the key differs in case from the one you activated.

How many cost allocation tags can I activate?

There’s a documented per-account cap on active keys, raisable through Service Quotas, plus a separate per-resource cap on total tags. Check current numbers in the AWS billing quotas docs. In practice the binding limit is human: past roughly a dozen keys, consistency collapses.

Should I use account tags or resource tags?

Both, for different jobs. Account tags handle top-level attribution and cover untaggable charges like support, credits and refunds. Resource tags handle granularity inside an account. If your accounts already map cleanly to teams, account tags are the cheaper win and worth doing first.

Is this worth doing on a small account?

Scale the effort to the shape of the bill. If you run a handful of services under one account and can name every resource from memory, account tags plus two resource keys is plenty. Worth noticing too that a fixed-price VPS from a provider like Contabo or InterServer has trivially perfect cost allocation, because the bill is one predictable line. Elastic infrastructure buys flexibility and pays for it in attribution work.

How do I allocate shared costs like NAT gateways?

Not with tags. Put them in a Cost Categories split charge rule and redistribute proportionally across consuming categories. Tag the shared resource to identify the bucket, then let the rule do the allocation.

The one thing worth remembering

AWS cost allocation tags fail silently, and that’s what makes them frustrating. Nothing errors. No alarm fires. Your resources look correctly tagged in every console you check, the bill quietly disagrees, and you find out when someone asks a question you can’t answer.

So check one thing now: open the cost allocation tags page in your management account and see whether the keys you’ve been diligently applying are actually switched on. If they aren’t, activate them today, then backfill. That twelve-month window rolls, and every month you wait is history you can’t recover.

Everything else here is incremental. Coverage is a percentage you improve, not a state you reach.

Need help getting your AWS bill to explain itself?

Cost attribution is unglamorous and easy to postpone until the quarter someone actually needs the numbers. If your account is already messy, it goes faster with someone who has untangled one before. I help with:

• Auditing tag coverage as a percentage of spend, broken down by service, so you know what you’re actually fixing
• Designing a minimal tag schema and the Tag Policies to keep it consistent across an organization
• Closing propagation gaps in Terraform and CloudFormation, around Auto Scaling groups, launch templates and attached storage
• Setting up Cost Categories with split charge rules so shared infrastructure stops distorting every team’s number
• Building cost and usage export pipelines in Athena, partitioned so the dashboard isn’t expensive to load
• Planning a staged tag enforcement rollout that doesn’t break anyone’s deployment pipeline

Send me a Cost Explorer screenshot grouped by your main tag key, or the output of a list call against your active tags, and I’ll tell you where the gap actually is.

The post AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill appeared first on John Nessime.

The invoice is where it surfaces, and not only on the big outputs. The 240p ladder rung went up. The audio-only rendition went up. Every output in every job that template touches went up, including the ones that never needed the feature.

That is not a billing bug. It is the documented rule working as designed, and it is the most common reason a bill moves without anyone changing how much video goes through it. Understanding MediaConvert cost per minute starts with accepting that the unit you pay for is not a job, not a source file, and not a minute of compute.

What follows is how that charge is assembled, what happens to it when a job fails, and how to build recovery around AWS Elemental MediaConvert. Rates are deliberately absent; AWS changes them and they vary by region. The mechanism is what catches people.

How MediaConvert cost per minute is actually assembled

Three rules stack on top of each other. Miss one and your cost model is wrong by a multiple, not a rounding error.

You pay per output, not per job

An eight-rung ABR ladder plus an audio-only rendition is nine billable items, each charged for its own duration. AWS bills the duration of the finished assets, not the transcoding time, in hundredth-of-a-minute increments. A ten-minute source producing nine outputs bills as ninety minutes of output.

Ladder design is therefore a cost decision, not only a quality one. The bottom rungs are not cheap because they are small. They cost a full pass of the source duration at whatever multiplier their settings attract, whether anyone streams them or not.

Normalized minutes, and why multipliers compound

There is no flat rate per output minute. MediaConvert converts each output into normalized minutes, a unit that scales with the work the encode demanded. Resolution, frame rate, codec and number of encoding passes each carry a multiplier, and those multipliers combine. Single-pass to multi-pass is one. 30fps to 60fps is another. Do both to a UHD rung and you have stacked three.

Volume discounts then apply to your accumulated normalized minutes for the month, per region, with basic-tier and professional-tier usage accruing separately toward their own thresholds. Two consequences: splitting a workload across regions splits the accrual and can leave you paying a higher effective rate in both, and under consolidated billing in AWS Organizations usage accrues across accounts, so a per-account model understates the discount you actually get.

The job-wide setting that reprices everything

Back to the opening. Each output is assessed for its billing tier separately, but only when the job has no job-wide professional-tier feature enabled. Turn one on, accelerated transcoding being the usual culprit, and every output in that job bills at pro-tier rates. An SD rung that would have billed at basic tier gets repriced because a different output in the same job needed to finish faster.

If a subset of content genuinely needs acceleration, split it out. Long, visually complex assets get a template with the feature; routine web renditions get one without. Two templates is less tidy than one, and considerably cheaper than a single checkbox repricing your catalogue.

Failed jobs are free, which changes how you should retry

MediaConvert charges only when a job reaches COMPLETED. Jobs ending in ERROR are not billed. That sounds like it removes the urgency from failure handling. It does the opposite.

A job that fails on output seven of nine may still have written outputs one through six to S3. Those partial artefacts cost storage, and if a naive retry writes to the same prefix you get a mix of old and new segments that a player will serve quite happily until someone notices the ladder is inconsistent. Free at the transcode line, expensive everywhere downstream.

“Not billed” also makes blind retries feel harmless, so people write them. A loop against a deterministic failure, a corrupt source or a settings mismatch, will never succeed. It burns concurrency paying jobs need and produces identical alerts until the team ignores the channel. The service already retries some conditions itself and exposes a retry count, so read that field before adding an attempt of your own.

The queue model is a throughput control, not a cost control

On-demand queues bill per output minute. How many jobs run at once is governed by a service quota for concurrent jobs per account in the region, a quota per individual queue, and a concurrent-jobs setting on each queue. Creating a second queue does not conjure capacity out of the account; you raise it through Service Quotas. Jobs sitting in SUBMITTED rather than moving to PROGRESSING is the signal you have hit the ceiling.

Reserved queues work on a different axis. You buy reserved transcode slots against a twelve-month commitment and pay for that capacity regardless of use. Each slot has its own dedicated compute, so one slot runs a given job just as fast as ten would. More slots means more jobs in parallel, not faster jobs. Three things about the commitment deserve a second read before anyone clicks agree:

• You cannot cancel it, and you cannot decrease the slot count.
• Adding capacity does not extend the existing term. It starts a fresh twelve-month commitment for the larger count, from the day you add it.
• When the plan expires, the queue persists and keeps accepting jobs. MediaConvert just does not run them.

That last one is the invisible failure in this whole topic. Your submission code gets a job ID back. The API call succeeded. Nothing errors. The jobs simply never progress, and unless you alarm on queue standby time or on the gap between jobs submitted and completed, the first person to notice is whoever asks why the library stopped updating. Set auto-renew deliberately, either way, and diary the term end date.

Queue hopping is the pressure valve. Give a job a wait time and a destination queue, and if it is still waiting when the timer expires it moves. The usual shape is reserved to on-demand during a spike. Billing follows the queue the job actually runs in, so a hopped job becomes on-demand spend.

{
  "Settings": { "OutputGroups": [], "Inputs": [] },
  "HopDestinations": [
    {
      "WaitMinutes": 10,
      "Queue": "arn:aws:mediaconvert:us-west-2:111122223333:queues/ondemandqueue",
      "Priority": 25
    }
  ]
}

WaitMinutes accepts 1 to 4320. Priority runs from -50 to 50 and defaults to 0, and the optional priority under HopDestinations lets a hopped job land higher or lower in the destination queue than it sat in the original. Hopping is a cost decision as much as a throughput one: it converts capacity you have already paid for into incremental on-demand spend.

The failure families that actually bite

MediaConvert publishes numbered error codes with documented causes. Rather than reproduce the table, group them by what you have to do about them, because the correct response differs sharply between groups.

Access failures

The service could not open an input or output location. Causes cluster around IAM permissions on the role the job runs under, cross-account bucket policies when the source lives elsewhere, keys that do not exist, and files whose headers cannot be read. Retry these once with a delay, because uploads still in flight are a real condition. If the second attempt fails identically, it is configuration, and no amount of retrying fixes configuration.

Settings failures that only appear on some inputs

The nastiest to diagnose, because the template has been fine for the last thousand assets. An encoder initialisation failure on one rung, triggered by an unusual source aspect ratio or frame rate meeting a fixed constraint in the template, is the classic shape. Never retry these; the job fails identically forever. The fix is usually to loosen the constraint rather than pin it harder, because letting the service derive a value the source cannot satisfy removes the whole class of failure.

Warnings, which are not errors and are easy to miss

MediaConvert also returns warning codes for problems that do not stop the job. It completes, you get billed, and something about the output is wrong. Video padding inserted to keep audio and video aligned, black frames carried through from the source, a QVBR score well below the rest of the catalogue: none raise an error, all are visible in metrics. Alert only on ERROR and this entire family is invisible to you.

Pinned engine versions that quietly expire

MediaConvert groups its releases into job engine versions so you can hold production on one while validating another. Access is requested through AWS Support rather than open to every account, so check whether it applies to you first. The behaviour worth knowing is expiry: requesting an expired version does not fail. The job runs as though no version had been specified, and the field reporting the version used comes back empty. An invalid version, by contrast, returns a 400 and creates no job. The failure mode of pinning is silence.

Building failure recovery that does not cost you twice

The pattern that holds up is event-driven, classified and idempotent. In order:

1. Subscribe to job state changes in EventBridge rather than polling. The ERROR event carries the code and message in its detail block.
2. Route by error code, not by status: one target for retryable access conditions, another for deterministic settings failures that need a human, and a suppression path for anything already seen on that asset.
3. Write a retry’s outputs to a new prefix, or clear the old one first. Never let attempt two land beside attempt one’s partial results.
4. Send a client request token on job creation. Reusing the same token within a minute of a successful request returns the original job instead of creating a duplicate, which protects you when your orchestration retries a create call whose response it never saw.
5. Cap attempts and land the remainder in a dead letter queue with the original event attached. A failure nobody can see is a failure nobody fixes.
6. Carry your own identifiers through user metadata on the job, so the returning event joins back to the asset without a lookup table.

{
  "source": ["aws.mediaconvert"],
  "detail-type": ["MediaConvert Job State Change"],
  "detail": {
    "status": ["ERROR"]
  }
}

The matching event delivers a detail object containing the queue ARN, job ID, status, an errorCode and an errorMessage. Route on errorCode. That one decision is the difference between a recovery system and a retry loop.

Measuring what you spend before the invoice tells you

MediaConvert sends metrics to CloudWatch at the end of every job, several mapping directly onto the billing dimensions. The ones I reach for first:

• SDOutputDuration, HDOutputDuration, UHDOutputDuration, 8KOutputDuration and AudioOutputDuration, in milliseconds per queue. Summed over a month, that is billable output volume by resolution class, days ahead of Cost Explorer.
• JobErrorsByCode, counting jobs that failed with a particular code. Build the dashboard on this one; it shows which failure family is growing.
• StandbyTime, how long a job waits before processing starts. Alarm on it and you catch backlogs, and you catch an expired reserved plan.
• JobsSubmittedCount against JobsCompletedCount. A widening gap is the earliest honest signal something upstream is broken.
• QVBRAvgQualityLowBitrate and QVBRMinQualityHighBitrate, for quality regressions that never raise an error.

For attribution rather than volume, enable cost allocation tags and set the billing tags source on your jobs so per-output costs land against a tag you chose. Pushing that into a wider FinOps view through Vantage, CloudZero or a Grafana Cloud dashboard fed from CloudWatch gets you per-title cost without building the aggregation yourself. Either way, these metrics let you model MediaConvert cost per minute continuously instead of discovering it monthly.

Transcoding is only part of the line. Storage for sources and renditions in S3, and egress through Amazon CloudFront or whichever CDN fronts delivery, are separate charges and often the larger ones. A cost model that stops at the encoder will be wrong.

Troubleshooting a failed job

1. Get the error code first. Everything else follows from which family it belongs to.
2. Ask whether other assets in the same bucket succeeded in the same window. Yes points at the file, no points at permissions or configuration.
3. For a settings failure, bisect the job. Strip it to a single video output, confirm it runs, then add outputs and features back one at a time. Tedious, and faster than reasoning about it.
4. Check warnings on jobs that completed nearby. Warnings often precede errors from the same root cause.
5. If nothing ran at all, look at the queue before the job: plan status on a reserved queue, standby time on an on-demand one.

Common mistakes

• Modelling cost from source duration instead of total output duration across the ladder.
• Enabling a job-wide professional-tier feature in a shared template and repricing every output in it.
• Creating extra on-demand queues for throughput when the account quota is the real ceiling.
• Buying reserved slots off a peak-week estimate and paying for idle capacity the other fifty-one.
• Retrying every failure identically, including the deterministic ones that cannot succeed.
• Alerting on ERROR only, and never looking at warnings or quality scores.

Best practices worth the effort

• Separate templates by cost profile rather than content type. Anything with a job-wide pro-tier feature gets its own.
• Audit the ladder against playback data at least once. Rungs nobody streams are pure cost.
• Keep a workload in one region so volume discounts accrue against a single pool.
• Run a representative job on an on-demand queue with the reserved simulation setting enabled before committing to slots.
• Use job priority instead of extra queues when the goal is for urgent work to jump ahead.
• Derive client request tokens from the asset, not from a timestamp, so retries are genuinely idempotent.

Frequently asked questions

Do I pay for MediaConvert jobs that fail?

No. Billing applies only to jobs reaching COMPLETED status, so a job ending in ERROR is not charged. You may still be paying for partial outputs that reached S3 before the failure, and the storage they occupy until something cleans them up.

What is a normalized minute?

The billing unit MediaConvert uses so a minute of demanding output counts for more than a minute of simple output. Resolution, frame rate, codec and encoding passes each contribute a multiplier, and those combine into the normalized total your volume discount tier is measured against.

Is a reserved queue cheaper than on demand?

Only if utilisation is genuinely steady. You pay for reserved transcode slots regardless of use, across a twelve-month commitment you cannot cancel or shrink. Bursty catalogues usually do better on demand, where volume discounts apply automatically as monthly usage climbs. Reserved pricing also excludes some features, so confirm everything your templates use is available there before committing.

Why are my jobs stuck in SUBMITTED?

On an on-demand queue, it is already running its maximum number of concurrent jobs; raise the quota through Service Quotas rather than adding queues. On a reserved queue, check whether the pricing plan has expired, because an expired queue keeps accepting jobs and never processes them.

Does adding queues give me more transcoding capacity?

No. Concurrency is bounded by account-level and queue-level quotas in the region. Extra queues isolate workloads from each other, for instance keeping one client’s jobs from starving another’s. Capacity increases come from Service Quotas.

How do I stop one setting from repricing my whole job?

Split the job. Job-wide professional-tier features apply their tier to every output, so isolate the assets that need them into their own template and keep routine renditions in a template with no job-wide pro-tier features enabled.

Should I run FFmpeg on my own servers instead?

Sometimes. Predictable, modest volume on dedicated hardware from a provider like Contabo or InterServer can undercut per-minute pricing, and you get exact control over encoder flags. What you take on is capacity planning, codec licensing questions, and building the retry and observability machinery the managed service hands you for free. That trade favours MediaConvert for spiky workloads and broadcast-grade features, and self-hosting for steady, simple ladders where an engineer already owns the pipeline.

The one thing to take away

MediaConvert cost per minute is not a rate you look up. It is a product: output count, times duration, times a stack of feature multipliers, with a single job-wide setting able to reprice the lot. Get that model into whatever you plan from and the bill stops being a monthly surprise.

On the failure side, the codes are documented and the events are structured, so there is no excuse for a retry loop that cannot tell a permissions problem from a settings problem. Classify before you retry. And put an alarm on queue standby time today, because the failure that costs you most is the one where nothing errors at all.

Need a second pair of eyes on your transcoding pipeline?

I work with teams running video-on-demand pipelines on AWS, usually at the point where the bill or the failure rate has become someone’s problem. What I actually help with here:

• Auditing job templates and ABR ladders for outputs and multipliers costing more than they return.
• Building a cost model from your CloudWatch output-duration metrics so you can forecast rather than react.
• Designing EventBridge-driven failure handling that classifies by error code and retries only what can succeed.
• Working the reserved versus on-demand decision against your real utilisation, including whether queue hopping earns its keep.
• Debugging settings failures that only hit certain sources, and loosening the constraints causing them.
• Wiring transcoding, S3 lifecycle and CDN delivery into one cost and reliability picture instead of three.

Send me a job template, an error code, or a month of output-duration metrics and I will tell you what I see.

The post MediaConvert Cost Per Minute: How the Bill Is Built and How Jobs Fail appeared first on John Nessime.

The expensive stuff is not the running instances. It’s the access keys nobody can attribute, the CloudTrail trail that quietly stopped delivering months ago, the Terraform state that no longer matches reality, and the region somebody spun a proof of concept up in two years ago. None of that comes up on a kickoff call. All of it comes up in week three, after you’ve quoted.

So I run an AWS account health check before I give anyone a number. It’s read-only, it takes a couple of hours, and it’s the cheapest insurance in consulting. This post is the actual check: the six passes, the commands, what each finding does to the estimate, and what to do when the client won’t give you the access to run it.

Why the AWS account health check comes before the quote

A quote is a bet on how much unknown there is. You’re pricing the work in the brief plus everything you’ll find while doing it, and the second half is invisible until you look. Two accounts can look identical and differ by a factor of three in effort. The difference is nearly always archaeology: how much has to be reverse engineered before you can safely change anything.

There’s a second reason that matters more than pricing. If you touch an account that already had a compromised key sitting in it, the incident becomes yours by proximity. Establishing what the account looked like before you arrived isn’t paranoia. It’s the only version of events you’ll be able to evidence later.

The access I ask for, and what a refusal tells me

I ask for a cross-account IAM role that the client creates and my account assumes. Not an IAM user. Not keys in an email. A role, with an external ID the client generates.

The external ID solves one specific problem. Without it, a trust policy that names my account can be used by anyone who talks me into assuming a role on their behalf, and I end up acting inside a third party’s account without knowing it. AWS calls this the confused deputy problem. It’s the piece of cross-account setup people skip most often.

On that role I ask for two AWS managed policies: SecurityAudit and ViewOnlyAccess. The pairing is the point. ViewOnlyAccess lets you enumerate resources across services, so you can see what exists. SecurityAudit lets you read configuration and policy documents, so you can see how it’s wired together. Neither alone gives you the picture, and neither lets you change anything. Billing is separate again: Cost Explorer read access has to be added explicitly and the feature has to be enabled in the account.

The reply to that request tells you as much as the check does.

• “Here’s the role ARN and external ID” means somebody there understands AWS identity. Good sign.
• “I’ll just make you an admin user, easier” means nobody is managing access. Expect that looseness everywhere else.
• “Our previous developer set it up and we can’t reach him” means you’re inheriting credentials nobody can account for. Price the rotation work as its own line.
• “We’d rather not grant access before signing” is fair. Offer a paid discovery engagement. If they won’t pay for two hours of assessment, they won’t pay for the fix.

The six passes

Each pass answers one question. Everything here is read-only, and the order matters, because each pass changes how you read the next.

Pass 1: Who can do what

Confirm where you actually are first. That sounds trivial until you’re in an organization with a dozen similarly named accounts.

aws sts get-caller-identity
aws iam list-account-aliases
aws organizations describe-organization

If the third command fails because the account isn’t in an organization, that’s a finding by itself. Standalone means no service control policies, no organization trail, and no central place to revoke anything.

Then the credential surface. The summary gives counts; the credential report gives per-principal detail, including when each password and key was last used.

aws iam get-account-summary

aws iam generate-credential-report
aws iam get-credential-report --query Content --output text | base64 --decode > credential-report.csv

The report generates asynchronously, so the second call may need a retry the first time. What I read from it: how many IAM users exist at all, how many have keys that have never been used, and how many have keys last used a long time ago that are still active.

An unused key is worse than a busy one. A key in daily use is attributable to something. A key last used eighteen months ago and still enabled belongs to a person or script nobody remembers, which means nobody will notice when someone else starts using it.

Then check whether anyone has ever asked this question before you:

aws accessanalyzer list-analyzers

IAM Access Analyzer offers external access analyzers, which flag resources shared outside a zone of trust you define, and unused access analyzers, which surface unused roles, keys and passwords. Unused access analysis is billed per IAM role and user analyzed per month, so it isn’t free, but it answers the least privilege question far faster than reading policy documents by hand. An empty list means nobody has ever asked this account who can reach what.

Root closes the pass. In an AWS Organizations setup, centralized root access management lets the management account or a delegated IAM administrator remove root credentials from member accounts entirely, so there’s no root password, key or MFA device to manage per account. Live root credentials on every member account is a real finding, and one of the few security fixes that’s genuinely quick.

Pass test: you can name every principal that can reach production, and say when each last did.

Pass 2: Would anyone notice

This is the pass that most often changes my number, because it’s the failure that stays invisible until it isn’t. Logging that exists but has silently stopped working is more dangerous than no logging, since everybody believes they’re covered.

aws cloudtrail describe-trails 
  --query 'trailList[].{Name:Name,MultiRegion:IsMultiRegionTrail,Org:IsOrganizationTrail}'

aws cloudtrail get-trail-status --name TRAIL_NAME_HERE

aws configservice describe-configuration-recorder-status

The first command tells you a trail is configured. The second tells you whether it’s delivering, which is the part that matters. A trail whose last delivery failed because someone edited the destination bucket policy still shows up in the console as a configured trail. The status call is where the truth is.

Multi-region matters for the same reason region sprawl does: a single-region trail records nothing about the region where somebody is quietly running compute. Check GuardDuty too, and how long logs are actually retained. Ninety days of retention against a one-year obligation is a project, not a task, and it belongs in the quote rather than in week four.

One naming trap worth knowing: AWS renamed the original Security Hub to Security Hub CSPM and launched a separate, newer service under the Security Hub name. When a client says “we have Security Hub,” ask which one, because the findings format and the configuration are not the same thing.

Pass test: a trail is delivering right now, covers every region, and its logs are kept for at least as long as the business needs them.

Pass 3: How wide the blast radius is

People forget regions. Not occasionally, constantly. A demo in Singapore, a test bucket in Ireland, a Lambda deployed from a tutorial that defaulted to us-east-1.

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  count=$(aws ec2 describe-instances --region "$r" 
    --query 'length(Reservations[].Instances[])' --output text)
  echo "$r $count"
done

Left without --all-regions, describe-regions returns the regions enabled for the account, which is exactly what you want. Adding the flag pulls in disabled regions too, and the per-region calls against those just error out.

Run the same loop shape for anything carrying state or cost: RDS instances, load balancers, EBS volumes, NAT gateways. NAT gateways are the classic. They bill hourly whether or not anything routes through them, and a forgotten one in a region nobody visits is pure waste. Then the two exposure checks I never skip:

aws ec2 describe-security-groups 
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 
  --query 'SecurityGroups[].{Id:GroupId,Name:GroupName,VpcId:VpcId}'

aws s3api list-buckets --query 'Buckets[].Name' --output text

For each bucket that matters, check its public access block configuration and its policy. An open security group on 443 in front of a load balancer is fine. An open security group on a database port is a different conversation, and you want that conversation before you own the account.

Pass test: you have a list of every region with resources in it, and nothing on that list surprises the client.

Pass 4: Whether any of this is described in code

The question isn’t “do they use Terraform.” It’s whether the code still describes the account. Those diverge quietly. Somebody makes an urgent console change during an incident, never brings it back into code, and from then on the repo is fiction that plans cleanly right up until the day it destroys something.

terraform plan -refresh-only

terraform plan -detailed-exitcode

The refresh-only plan shows where real infrastructure has drifted from state without proposing to change anything. The detailed exit code version is the one for scripts: it exits 0 for no changes, 1 for an error and 2 when there’s a diff, so you can gate on drift instead of reading output by eye.

Three things I want to know here:

• What fraction of live resources are managed by code? Half a codebase is often worse than none, because it creates false confidence about which half.
• Where does state live and who can write to it? Remote state in a versioned, locked S3 bucket is a different risk profile from a state file on a laptop.
• Does the pipeline that applies changes still run? Credentials that expired six months ago mean everything since went in by hand.

Tagging belongs here too. With nothing tagged you can’t attribute cost, can’t scope a change safely, and can’t tell a production resource from an abandoned experiment. Untagged environments add hours to everything that follows.

Pass test: a refresh-only plan comes back clean, or the drift is a short list you can read out loud.

Pass 5: What shape the spend is in

I don’t care much about the total. I care about the shape: which services dominate, whether the curve is flat or climbing, and whether anything jumped without an explanation.

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '6 months ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=SERVICE

The date -d syntax is GNU coreutils; on macOS use the BSD equivalent or a fixed start date. Worth knowing before you loop this: the Cost Explorer API bills per paginated request, unlike the console view, so a wide query across several dimensions is a small but real charge on the client’s bill. Keep it narrow and mention it in your notes.

• Data transfer in the top three usually means an architecture problem rather than a rate problem. Cross-AZ chatter and NAT gateway egress are the usual suspects.
• A step change with no matching launch deserves a direct question. Sometimes it’s a feature. Sometimes it’s a runaway job. Occasionally it’s mining on a stolen key.
• No budgets and no anomaly monitors means nobody finds out about a spike until the invoice arrives.
• Heavy commitment purchases against workloads you’re about to re-architect are a constraint on your design, and need to be on the table before you propose anything.

If the client wants ongoing cost visibility past what Cost Explorer offers, platforms like Vantage or CloudZero sit on the Cost and Usage Report and give per-team allocation and unit economics that native tooling makes awkward. Raise it as a separate line, not part of the engineering work.

Pass test: you can explain the top three cost lines in a sentence each, and account for every month-on-month jump.

Pass 6: What happens at two in the morning

Short pass, about escalation. What support plan is the account on, and would anything wake a human up?

Support plan matters more than people expect. On Basic you get the service limits checks and a small subset of Trusted Advisor security and fault tolerance checks, manual refresh only, and no technical support cases at all. The full check library and the Trusted Advisor API need a paid plan. If your engagement depends on opening a case during a cutover, that’s a prerequisite, not a nice to have.

There’s one date worth carrying in your head, because it’s a real deprecation rather than a version number. AWS has announced that Developer Support, Business Support and Enterprise On-Ramp are discontinued on 1 January 2027, with Business Support+ as the replacement tier. If you’re scoping anything that runs past that boundary, check which plan the client is on and whether their support cost is about to move.

Then monitoring. CloudWatch alarms wired to an SNS topic with no confirmed subscribers are the same failure as the dead trail: configured, visible, useless. If they run anything beyond alarms, whether that’s Grafana Cloud, Datadog or a self-hosted Prometheus stack, find out who holds the credentials and whether the routing has been tested this year.

Pass test: you can name the human who gets woken up, and the path that reaches them.

The findings that actually move the number

• Unattributable credentials. Each one has to be rotated or removed, and each needs a conversation with whoever might break. That’s coordination time, which is the hardest kind to compress.
• No working audit trail. No baseline means you can’t prove what you did or didn’t change. Add the cost of standing logging up before the real work starts.
• Significant drift between code and reality. Reconciliation is slow, fiddly and carries destruction risk. The single largest multiplier I see.
• No tags. A tax on every subsequent task rather than a task of its own, since every scoping decision becomes an investigation.
• Resources in unmentioned regions. Each is a small parallel environment with its own security groups, its own logging gap and its own bill.
• One shared admin identity. Nothing in CloudTrail is attributable to a person, so your first task is identity work the client didn’t ask for.

I don’t present these as criticism. I present them as line items with an hour estimate against each, and let the client decide what’s in scope. That conversation is much easier before a contract exists than after.

If you only run five checks

Sometimes you get twenty minutes and a screen share.

1. Pull the credential report and count active keys that haven’t been used recently.
2. Confirm a CloudTrail trail exists, is multi-region, and is currently delivering.
3. Loop the enabled regions for running instances and NAT gateways.
4. Run a refresh-only plan against whatever infrastructure code exists.
5. Pull six months of cost grouped by service and look at the shape.

Those cover identity, evidence, blast radius, delivery speed and money. The rest of the check refines a picture these five already gave you.

Where this check goes wrong

• Treating it as an audit. You’re scoping work, not issuing a compliance report. A twenty page findings document makes a prospect defensive and costs you a day.
• Checking one region. The default region in your profile is not where the problems are.
• Trusting the console over the API. Configured and working are different states, and the console is better at showing the first.
• Taking long-lived keys because it’s faster. Accept an admin access key by email to save fifteen minutes and you’ve become part of the problem you were hired to assess.
• Running the loops unannounced. A burst of API calls from an unfamiliar principal is exactly what GuardDuty exists to flag. Tell them you’re doing it and roughly when.

Frequently asked questions

How long should an AWS account health check take?

Two to three hours for a single account once you have access, including the write-up. Multi-account organizations take longer but not linearly, because the interesting variation concentrates in one or two accounts and the rest are copies of a landing zone pattern.

Should I charge for the assessment?

For a short engagement, folding it into the proposal is fine. For anything substantial, or where the client is shopping several consultants, quote it as a small standalone piece with a written deliverable. It qualifies the client, protects your time, and the findings have value to them even if they hire someone else.

What if the client won’t give me read-only access before signing?

Quote time and materials, or quote fixed price with an explicit assumptions list naming every unknown. Do not quote a fixed price on an account you haven’t seen. The pattern that works is a small paid discovery phase, then a fixed price set once you know what you’re dealing with.

Is ReadOnlyAccess enough on its own?

It’s broader than you probably want. ReadOnlyAccess extends to reading data in some services, not just configuration, which is more than you need for scoping and more than many clients will be comfortable granting. The SecurityAudit and ViewOnlyAccess pairing is narrower and easier to justify in writing.

Can I automate the whole thing?

The enumeration, yes, and it’s worth scripting since you’ll run it repeatedly. The interpretation, no. Whether five unused IAM users are a problem depends entirely on what the client is trying to do, and that judgement is most of what you’re selling.

What if I find something actively compromised?

Stop and tell them immediately, in writing, with timestamps. Don’t remediate unilaterally: you’re read-only, and containment steps can destroy evidence. Your job is a clear description of what you saw and when, so they can involve whoever they need to.

Does this apply to small accounts too?

Especially to small accounts. Larger organizations usually have some governance, even imperfect. A three year old account at a small business that four different contractors have touched is where the surprises live, and it’s exactly the account that gets described as simple.

The one thing worth remembering

An AWS account health check isn’t a security exercise dressed up as sales. It’s the only reliable way to price uncertainty, and uncertainty is what you’re actually quoting on.

If you take one habit from this, take the second pass. Everything else on the list is something you’d eventually trip over while doing the work. A logging gap is different, because it stays invisible while it costs you, and by the time it matters you’re the person who was in the account when it happened. Check the trail is delivering. Then check the rest.

Need a second pair of eyes on an AWS account?

I do this for consultancies scoping engagements and for businesses who inherited an account they don’t fully understand. Usually that looks like:

• A read-only AWS account health check with a written findings list, hour estimates per item, and a clear split between urgent and later.
• Cross-account audit role setup, with a trust policy and external ID you can hand to any consultant without granting more than you meant to.
• Credential archaeology: working out what every active access key belongs to, then rotating or removing the ones that belong to nothing.
• Getting CloudTrail, AWS Config and log retention into a state where they’re delivering rather than just configured.
• Terraform drift reconciliation, including importing unmanaged resources back under code without a destructive plan.
• Cost shape reviews: the forgotten NAT gateways, orphaned volumes and idle regions, plus budgets and anomaly alerts that reach a real inbox.

If you’ve got a credential report, a terraform plan output or a Cost Explorer screenshot you can’t make sense of, send it over and I’ll tell you what I’d look at next.

The post The AWS Account Health Check I Run Before Quoting a Project appeared first on John Nessime.