That slope is the signature. CloudWatch rarely fails loudly, it accumulates: a log group created automatically by a service you enabled once, retention left at the default, a metric filter with a dimension nobody thought about, and a dashboard quietly calling the API every few seconds forever.

This post breaks down where CloudWatch costs actually come from, why the same log line gets charged more than once, how cardinality multiplies your bill without your traffic changing, and how to attribute spend to specific log groups before you delete anything. It is organised by cost lever rather than by feature, because that is the order you work in when the invoice lands.

The four meters behind your CloudWatch bill

CloudWatch is not one product with one price. It is a set of independent meters that happen to share a console. Working out which meter is running is most of the job, because the fix for each is completely different.

• Ingestion. Per GB of log data accepted into a log group. Usually the largest component on log-heavy accounts.
• Storage. Per GB-month for what is retained afterwards. Cheap per unit, but it compounds because it never stops unless you tell it to.
• Analysis. Logs Insights bills on data scanned by a query, not on rows returned and not per query.
• Metrics and API calls. Custom metrics bill per unique time series per month. Separately, calls like GetMetricData bill per metric requested, which is how third-party dashboards end up on an AWS invoice.

The thing worth internalising: a single log line can be billed three times. Once when it arrives, again every month it sits in storage, and again every time a query scans the block it lives in. None of that is hidden, but nothing in the console surfaces it either.

Ingestion is the meter that actually hurts

Uncompressed in, compressed out

Ingestion is metered on the uncompressed volume of events you send. Storage is billed on the compressed size CloudWatch keeps. Text logs compress well, so the stored figure can be a small fraction of what you were charged to get the data in.

That asymmetry is why people look at a log group holding a few gigabytes, decide it is harmless, and miss that it cost several times that to arrive. The storedBytes value shown on a log group is the compressed number. It tells you about your storage line, not your ingestion line.

The log groups you never created

Most runaway ingestion comes from things switched on with a checkbox. VPC Flow Logs pointed at CloudWatch. EKS control plane logging with every type enabled. Lambda functions emitting a full structured payload per invocation. CloudTrail delivering to a log group as well as S3. RDS general query logs left on after a debugging session.

None of those feel like a decision at the time, and all of them scale with traffic rather than headcount. That is why the slope keeps rising while the team stays the same size.

The most effective lever here is not a CloudWatch setting at all. It is cutting verbosity at the source, before the bytes leave your application. Dropping debug-level logging in production, sampling health check lines, trimming repeated stack frames. Filtering after ingestion does nothing for this meter, because you already paid.

Retention: the one-line fix almost nobody applies

Log groups default to Never Expire. Not thirty days, not a year. Forever. Every log group created implicitly by another AWS service inherits that default, so an account accumulates hundreds of them without a single deliberate retention decision.

Find the offenders first. This lists every log group with no retention policy, alongside its compressed size:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output table

The ?!retentionInDays filter selects entries where the field is absent, which is how the API represents indefinite retention. There is no explicit “never” value to match on.

aws logs put-retention-policy 
  --log-group-name /aws/lambda/order-processor 
  --retention-in-days 30

Three things are worth knowing before running that across an account.

1. The value is an enum, not a free integer. Accepted values run 1, 3, 5, 7, 14, 30, 60, 90, 120, 150, 180, 365, 400, 545, 731 and upward to ten years. Asking for 45 days is rejected.
2. Deletion is not instant. Expired events are marked for deletion and typically removed within about three days. They stop contributing to your storage charge once marked, and are excluded from the reported storedBytes, so the billing effect lands before the data physically goes.
3. Reverting means removing the policy, not setting zero. To go back to indefinite retention you call delete-retention-policy rather than passing a zero value.

Dry run before you bulk-apply. This prints what would change without touching anything:

aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].logGroupName' 
  --output text | tr 't' 'n' | while read -r lg; do
    echo "would set 30-day retention on: $lg"
  done

Retention is where compliance and cost pull against each other, and cost should lose. If an auditor needs seven years of logs, seven years is the answer. The move is to export them to S3 and apply lifecycle rules there, rather than paying CloudWatch rates for archival data you will query twice a decade.

Cardinality: the multiplier nobody sees coming

This is the part that surprises people, because it breaks the intuition that cost tracks traffic. Cardinality can multiply your metrics bill by an order of magnitude while request volume stays flat.

CloudWatch treats every unique combination of namespace, metric name and dimension values as a separate metric. Not a variant of one metric. A separate, separately billed metric. Publish OrderLatency with a Region dimension holding three values and you have three metrics. Add a CustomerId dimension and you have three times however many customers you have.

The dangerous dimension values are the ones that look like identifiers: request IDs, trace IDs, pod names, session tokens, customer IDs, IP addresses, URL paths with IDs embedded in them. Each new value is a new billable series that persists and accrues.

Metric filters have a guard rail, and it bites

Metrics extracted from logs by a metric filter bill as custom metrics. A filter can carry up to three dimensions, and AWS documents an explicit safety mechanism: CloudWatch Logs disables a metric filter that generates a thousand distinct dimension name/value pairs within a period.

That guard rail is useful, but read what it means operationally. Hitting it does not just cost money, it silently stops the filter, so the metric you built an alarm on stops receiving data. If that alarm treats missing data as not breaching, it goes quiet rather than firing. You lose the alert and keep the bill.

Audit what exists before adding more, and count what a namespace actually produces:

aws logs describe-metric-filters 
  --log-group-name /aws/lambda/order-processor

aws cloudwatch list-metrics 
  --namespace MyApp/Orders 
  --query 'length(Metrics)'

If that count is materially larger than the number of distinct things you consciously measure, you have found your multiplier.

EMF does exactly what you tell it

Embedded Metric Format is a good design: you emit structured events, CloudWatch extracts metrics from declared fields, and you get high-cardinality context in the logs with low-cardinality metrics for alarming. That split is the point.

The failure mode is promoting a high-cardinality field into the dimension set. AWS documents this plainly: declare something like a request ID as a dimension and EMF will, by design, create a custom metric for every unique combination. Keep identifiers in the log body as properties, and keep dimensions to values you would put on a dashboard axis.

Anomaly detection alarms cost more than they look

An anomaly detection alarm does not evaluate one metric. It evaluates the source metric plus the upper and lower bounds of the predicted band, and those bounds are metrics too. That is roughly a threefold multiple against a standard threshold alarm, per alarm.

It earns its keep on genuinely seasonal signals where a static threshold produces constant noise. Switching an entire alarm estate to it “to be safe” triples a line item for very little detection benefit.

Query cost spikes exactly when you need it most

Logs Insights bills on the volume a query scans. Not rows returned. Not query count. Scanned bytes. A query returning twelve lines that scans a terabyte costs the same as one returning a million lines that scans a terabyte.

During an incident, an engineer runs a dozen exploratory queries across a wildcard log group selection and a wide time window. The pricing model charges you hardest at the moment the tool is most valuable, which is an uncomfortable incentive. Two habits reduce it without reducing your ability to debug:

• Narrow the time window first, then the filter. Scanning is bounded by the selected range across the selected groups. Starting at fifteen minutes and widening is far cheaper than starting at a week and narrowing.
• Select specific log groups, not patterns. Querying across every Lambda log group multiplies scanned bytes by the number of functions when you only care about one.

Watch for scheduled queries too. A dashboard widget or automation running an Insights query every few minutes against a large log group is a fixed monthly cost nobody remembers creating.

Finding out where your CloudWatch costs come from

Attribute before you cut. Deleting log groups because they look big is how teams lose the one log they needed three weeks later.

1. Split the bill by meter. In Cost Explorer, filter service to CloudWatch and group by usage type. The region-prefixed usage types map to the meters directly: DataProcessing-Bytes is log ingestion, TimedStorage-ByteHrs is log storage, DataScanned-Bytes is Logs Insights, and CW:Requests and CW:GMD-Metrics cover API calls.
2. Rank log groups by ingestion. The IncomingBytes metric in the AWS/Logs namespace, dimensioned by log group, is the per-group ingestion figure. Graph it with the Sum statistic. One documented gotcha: use an absolute range covering exactly thirty days, because a relative range or a different length produces a number that does not correspond to a billing month.
3. Check stored bytes separately. A log group can be small in ingestion but large in storage because it has accumulated since the account was created. That is a retention problem, not a verbosity problem, and the fix differs.
4. Count custom metrics per namespace. Compare the count against your mental model of what you measure. The gap is cardinality.
5. Look inside the noisiest group. Once you know which log group dominates, find out which streams are responsible before changing any application code.

For that last step, a Logs Insights query over a short window is enough. Keep the range tight, because this query is itself billed on what it scans:

stats count(*) as events, sum(strlen(@message)) as approx_bytes by @logStream
| sort approx_bytes desc
| limit 20

That approximates byte volume per stream, which is usually enough to identify the chatty component. It is an approximation, not a billing figure, because it counts message characters rather than the metered event size.

If you manage several accounts and want attribution done for you, cost platforms such as Vantage and CloudZero break CloudWatch spend down by usage type and resource without you building dashboards. Worth the spend if client reporting is part of the job. For one account and one engineer, Cost Explorer plus the steps above gets the same answer.

Cheaper destinations, and what they cost you

Once you know your ingestion is dominated by logs you rarely read, there are three real options. None is free of trade-offs.

• The Infrequent Access log class. Set at the log group level, it ingests at a materially lower per-GB rate in exchange for a reduced feature set. Historically that meant no metric filters, no alarming from logs and no Live Tail, though the capability list has expanded since launch. Check the current log classes documentation rather than assuming, and note that a log group you extract metrics from is not a candidate.
• Subscription filter to S3 via Amazon Data Firehose. Sends log data to object storage where lifecycle rules and Athena queries apply. You trade Insights and the console experience for much cheaper long-term retention. Good for audit and forensic logs, poor for anything you troubleshoot weekly.
• A separate log stack. Grafana Cloud, or self-hosted Loki and Prometheus on a VPS from a provider like Contabo or InterServer, moves cost from per-GB metering to a fixed monthly box you administer. The economics can be dramatically better at volume. The price is that you now own retention, backups, upgrades and the availability of your own observability during an outage, which is exactly when you least want to be debugging it.

The honest version: CloudWatch is expensive partly because it is operated for you and wired into everything. At a few dozen gigabytes a month, migrating away costs more engineering time than it saves. At terabytes, the arithmetic flips hard.

Troubleshooting a bill that will not go down

You set retention everywhere and storage barely moved. Check whether storage was ever the dominant cost. If your usage type breakdown was mostly DataProcessing-Bytes, retention was never going to fix it.

Custom metric count grows with no code changes. Something is emitting a dimension whose value set is open-ended. Look for deployment identifiers, container IDs, or a version string appended to a dimension.

An alarm went quiet and the bill jumped the same week. Suspect a disabled metric filter. Check the filter still exists and is producing data, and verify the alarm’s missing data treatment. Alarms treating missing data as not breaching fail silently by design.

API request charges you cannot place. External dashboards are the usual answer. A Grafana instance polling CloudWatch, or any tool calling GetMetricData on a short interval, bills per metric requested. Widening the refresh interval on dashboards nobody watches live is a quick win.

Costs dropped, then came back. Look at your infrastructure as code. If Terraform or CloudFormation creates log groups without a retention argument, every apply restores the default. Fixing it in the console fixes it until the next deploy.

Common mistakes

• Treating storage as the problem because it is the number the console shows. Ingestion is usually larger and invisible in the log group list.
• Adding a dimension because it would be useful to filter by, without checking how many distinct values it can take.
• Filtering logs after they reach CloudWatch. The ingestion charge already applied.
• Deleting log groups to cut cost without checking whether anything alarms on them.
• Setting retention in the console while the IaC that owns the log group has none configured.
• Enabling every EKS control plane log type when only the audit log is consumed.

Best practices for controlling CloudWatch costs

• Make retention a required argument. If log groups come from Terraform or CloudFormation, treat a missing retention setting as a policy violation and fail the plan. Checkov and similar scanners have rules for exactly this.
• Decide verbosity per environment, not per service. Debug in staging, info in production, with a documented way to raise it temporarily during an incident.
• Keep dimensions bounded, identifiers in the message body. If you cannot enumerate a dimension’s possible values on a whiteboard, it does not belong in the dimension set.
• Alarm on the bill itself. A billing alarm on estimated charges, or an anomaly monitor scoped to CloudWatch, turns a quarterly surprise into a same-week signal.
• Review the top five log groups by ingestion monthly. Five minutes with the IncomingBytes graph catches new noise before it compounds.
• Route archival logs out deliberately. Anything kept for compliance rather than operations belongs in S3 with lifecycle rules from the start.

Frequently asked questions

Why is my CloudWatch bill so high when my traffic has not changed?

Two causes account for most cases. Storage accumulating because log groups default to indefinite retention, so the bill grows every month even at constant ingestion. Or metric cardinality, where a dimension with an open-ended value set keeps creating new billable series. Neither tracks request volume, which is why the bill and the traffic graph disagree.

Does deleting a log group reduce my bill immediately?

It stops future storage charges for that data and nothing else. Ingestion already billed is spent, and deleting the group does not stop whatever was writing to it, which will simply recreate it. Turn off the source first, then delete.

What counts as one custom metric for billing?

One unique combination of namespace, metric name and dimension values. The same metric name published with three different dimension value sets is three billable metrics. This is the single most important sentence in CloudWatch pricing and the easiest one to skim past.

Should I switch log groups to the Infrequent Access class?

For logs you keep but rarely read, it is usually a straightforward saving on ingestion. For any group you extract metrics from, alarm on, or stream live, it is not a candidate. The feature gap between the classes has narrowed since launch, so check current documentation rather than ruling it out on old information.

Is Logs Insights charged per query or per gigabyte?

Per gigabyte scanned. Rows returned are irrelevant. A tightly filtered query over a wide range across many log groups is expensive; a broad query over fifteen minutes on one log group is cheap. Narrow the range before you narrow the filter.

Do VPC Flow Logs have to go to CloudWatch?

No. Flow logs can be delivered to S3 instead, avoiding CloudWatch ingestion and storage entirely, at the cost of losing Insights queries over them. If you analyse flow logs occasionally with Athena rather than continuously in the console, S3 is the better destination.

The one thing to take away

CloudWatch costs are not one number with one cause. They are four meters running independently, and each has a different fix: cut verbosity at the source for ingestion, set retention for storage, narrow scope for queries, bound your dimensions for metrics. Applying the wrong fix to the wrong meter is why so many cost-reduction efforts produce a bill that barely moves.

So attribute first. Split by usage type, rank log groups by IncomingBytes, count custom metrics per namespace. Ten minutes of that tells you which lever you are actually pulling, and stops you deleting the log group that turns out to be the only place a production alarm was getting its data.

Need a hand with your CloudWatch spend?

This is well-defined work with a measurable outcome, which makes it a good fit for a short engagement. Things I can help with:

• Breaking your CloudWatch bill down by meter and by log group, so you know what you are paying for before anything changes
• Auditing custom metric cardinality and identifying which dimensions are multiplying your series count
• Building a retention policy that satisfies your compliance requirement without paying CloudWatch rates for archival data
• Enforcing retention and log class in Terraform or CloudFormation so the fix survives the next deployment
• Designing a subscription filter and S3 archive path for logs you need to keep but rarely query
• Reviewing alarms and dashboards for anomaly detection and API polling costs that are not earning their keep

If you want a second opinion, send me your Cost Explorer breakdown grouped by usage type, or the output of describe-log-groups for the account. That is usually enough to say where the money is going before we talk about anything else.

The post The Real Cost of CloudWatch: Logs, Metrics, Retention and Cardinality appeared first on John Nessime.

Then someone in finance asks what the reporting pipeline costs. You open Cost Explorer, group by Project, and over half the bill lands in a bucket called “No tag key”.

Nothing is broken. The tags are there. They just aren’t on the bill, because tagging a resource and activating a cost allocation tag are two separate operations, and AWS never warns you when you’ve done only the first.

This post covers getting AWS cost allocation tags to cover a messy account: the activation gap, what backfill will and won’t repair, where coverage leaks even with decent discipline, and a retrofit order that converges.

A resource tag and a cost allocation tag are not the same thing

A resource tag is metadata on the resource. A cost allocation tag is a billing dimension. Getting the second means taking a key that already exists on your resources and explicitly switching it on in the Billing and Cost Management console.

Two details make this bite. First, activation lives in the management account. From a member account, the page tells you tags are managed at the payer level and that’s the end of it. Plenty of teams have a good tagging standard and no access to the one screen that makes it count. Related trap: an account moving between organizations has its activated keys reset to inactive.

Second, the lag. AWS documents up to 24 hours for a newly applied key to appear on the activation page, and up to another 24 hours after you activate before it takes effect. The loop between “we agreed a standard” and “I can see it in Cost Explorer” is two days at best.

Past a handful of keys, drive it from the API. Listing the inactive ones is the fastest way to find what you forgot:

# Every key AWS knows about that is NOT currently billing-active
aws ce list-cost-allocation-tags --status Inactive

# Switch a set of them on
aws ce update-cost-allocation-tags-status --cli-input-json '{
  "CostAllocationTagsStatus": [
    {"TagKey": "Project",     "Status": "Active"},
    {"TagKey": "Environment", "Status": "Active"},
    {"TagKey": "Owner",       "Status": "Active"}
  ]
}'

User-defined keys appear in the report with a user: prefix and AWS-generated ones carry the reserved aws: prefix, so they never collide. There’s a cap on active keys, raisable through Service Quotas, which you almost certainly don’t need to raise.

What the backfill repairs, and what it quietly doesn’t

Activation used to be strictly forward-looking, so activating late permanently orphaned everything before it. That’s no longer true. The management account can ask AWS to reapply current activation status across up to twelve months of historical cost data.

# Must be the first day of a month, no earlier than twelve months back
aws ce start-cost-allocation-tag-backfill 
  --backfill-from 2025-01-01T00:00:00Z

• Start date must be the first day of a month, and can’t reach back more than twelve months.
• One request per 24 hours, and you can’t submit another while one is processing.
• It backfills activation status, not tag values.

That last point catches people. If the resource wasn’t carrying the tag when the cost was incurred, backfill gives you nothing for those months. It fixes “I forgot to activate.” It does not fix “I forgot to tag.” A tag applied in June and activated in November backfills to June and no further, whatever start date you hand it.

It also cuts both ways. Backfill applies the current status of every key, so a key sitting inactive right now gets stripped out of historical data too. If you’re pruning an overgrown tag list, backfill before you deactivate the old keys, not after, or you’ll erase the history you were trying to keep.

Where AWS cost allocation tags actually leak coverage

With activation sorted and every Terraform-managed resource tagged, a stubborn slice of the bill still refuses to attribute. It matters which slice, because the fixes have nothing in common.

Charges no resource can carry

Support charges, refunds, credits, tax, some Marketplace subscriptions. There’s no resource to hang a tag on, so tagging discipline never touches them. On accounts with a large support plan or heavy credit activity, that’s a visible fraction of the invoice.

The answer is account tags. AWS added support for using AWS Organizations account tags as a cost allocation dimension, applying to all metered usage inside the tagged account and flowing into Cost Explorer, Budgets, Cost Anomaly Detection and Cost Categories. Because attribution happens at the account level, it picks up exactly what resource tags can’t reach.

This is recent enough that most tagging guides predate it, and it’s the highest-value item here if your accounts already map roughly onto teams.

Resources created by other resources

This is the quiet one, and usually the biggest surprise gap in an otherwise well-run account. Terraform’s provider-level default_tags applies to resources Terraform creates. It does not reach the EC2 instances an Auto Scaling group launches, because Terraform didn’t create those, the ASG did. Your ASG is tagged. Your instances aren’t. Same for volumes attached at launch, ENIs, snapshots and AMIs.

data "aws_default_tags" "current" {}

resource "aws_autoscaling_group" "app" {
  # ... other config ...

  # Push provider default tags down to launched instances
  dynamic "tag" {
    for_each = data.aws_default_tags.current.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_launch_template" "app" {
  # ... other config ...

  tag_specifications {
    resource_type = "instance"
    tags          = data.aws_default_tags.current.tags
  }

  tag_specifications {
    resource_type = "volume"
    tags          = data.aws_default_tags.current.tags
  }
}

For a standalone instance, volume_tags on aws_instance covers its attached volumes. General rule: whenever one AWS resource provisions another on your behalf, assume tags don’t propagate until you’ve checked the created resource in the console, not the plan output.

Shared infrastructure that belongs to everyone

NAT gateway data processing, VPC interface endpoints, a Transit Gateway, a shared load balancer, the central logging bucket. You can tag all of these perfectly and the tag says Project=platform, which is accurate and useless for chargeback.

Transit Gateway is worth calling out because it supports tagging the gateway resource itself, so data processing and transfer charges pick up the tag rather than only attachment hours. That gap used to swallow a lot of network spend.

Beyond that, stop chasing tags and do allocation math instead. Cost Categories support split charge rules that redistribute a shared bucket across your other categories, using proportional splits, fixed percentages or an even division. Proportional is the sane default; fixed percentages are for when finance already agreed a number and the report has to match it.

Container spend hiding behind one instance tag

A shared EKS or ECS cluster is one tagged compute footprint running fifteen teams’ workloads. The instance tag gives you the total and nothing about who caused it. Split cost allocation data is the native answer: opt in and the cost and usage export gains rows breaking cluster cost down to individual pods or tasks. It won’t account for idle headroom you’re deliberately carrying, but it beats one line item labelled “the cluster”.

Case drift and near-duplicate keys

Keys and values are case-sensitive. Environment, environment and ENV are three distinct keys producing three columns, each holding a partial and misleading picture. Tag Policies in AWS Organizations exist for this: define the key including its capitalization, define permitted values, and get an org-wide compliance report. Cheapest governance on this list, because it costs nobody a deploy.

A retrofit order that converges

Retrofits stall because people start at the bottom, sweeping untagged resources by hand while the creation path keeps producing more. This order avoids that.

1. Measure first. Group Cost Explorer by your intended key and record what percentage of spend falls into “No tag key”. That’s your baseline and your only honest progress metric.
2. Cut the key set down. Three to five keys: owner, environment, and a project or cost centre. Coverage on five beats partial coverage on thirty.
3. Activate immediately, even with terrible coverage. Both clocks start now: the activation delay and the rolling twelve-month backfill window. There’s no downside to activating a key that’s barely applied.
4. Rank the gap by spend, not resource count. Sort untagged line items by cost descending. It’s usually a few NAT gateways, a couple of oversized databases and one forgotten bucket. Ten thousand untagged security groups cost nothing.
5. Fix the creation path before the backlog. IaC defaults, launch template tag specifications, ASG propagation. Sweeping first means sweeping again next month.
6. Sweep the stragglers in bulk. The Resource Groups Tagging API works across services and regions, which beats going service by service.
7. Backfill, then re-measure. Compare against step one. If the number barely moved, you fixed the wrong resources.

# Find what already carries a Project tag, to diff against inventory
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project

# Apply tags to a specific set of ARNs
aws resourcegroupstaggingapi tag-resources 
  --resource-arn-list arn:aws:rds:eu-west-1:111122223333:db:reporting-01 
  --tags Project=reporting,Environment=prod

Run it against a couple of ARNs and confirm in the console before looping over a list. Bulk tagging is trivially easy to point at the wrong account.

Enforcing tags without becoming the tag police

Four mechanisms, in increasing order of blast radius. Most teams need the first three and should be careful with the fourth.

• IaC defaults. Provider-level default tags plus required module variables. Cheapest, catches the majority, nobody thinks about it.
• CI validation. A linter rule that fails the pull request on missing required tags. The failure lands on the person who can fix it, when they can fix it.
• Detective controls. The AWS Config managed rule for required tags flags non-compliant resources without blocking anything. Tells the truth, breaks nothing.
• Preventive controls. A service control policy denying creation when a required key is absent, keyed on a null check against the request tag.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRDSCreateWithoutProjectTag",
      "Effect": "Deny",
      "Action": "rds:CreateDBInstance",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    }
  ]
}

The honest trade-off: service control policies are where tagging programmes go to die. Not every API supports tag-on-create, and not every legitimate workflow passes tags through the way you expect. Roll a broad deny across the org on day one and you break somebody’s deployment on day two, the ticket lands on you, and the policy is gone by day three along with everyone’s appetite for the project.

Scope it to one service at a time, starting in sandbox. I’d rather run Config plus Tag Policies for a quarter and fix the top three offenders in the IaC. The deny is a finishing move, not an opening one.

Measuring coverage so it doesn’t rot

Cost Explorer gives the headline number. For a per-service breakdown, query the cost and usage export in Athena. One caveat: tag column naming differs between export versions, older ones giving a flattened column per activated key and newer formats a map column. Run a DESCRIBE first rather than guessing.

-- Confirm the actual column names first
DESCRIBE your_cur_database.your_cur_table;

-- Then: untagged spend, ranked by service
SELECT   line_item_product_code,
         SUM(line_item_unblended_cost) AS untagged_cost
FROM     your_cur_database.your_cur_table
WHERE    your_project_tag_column = ''
  AND    line_item_unblended_cost > 0
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

That ranked list is your work queue; everything below the top twenty is noise. The list call also returns a last-used timestamp per key, the cleanest way to find keys nobody has applied to anything in months.

If you’d rather not maintain the dashboard yourself, this is a fair place to buy rather than build. Grafana Cloud sits on Athena comfortably if you already run Grafana, and platforms like CloudZero, Vantage and nOps handle allocation modelling out of the box. Their real value is the shared-cost and commitment maths. Their real cost is another vendor between you and your billing data, and several price against a percentage of the spend they watch, so run that number first.

Troubleshooting

• The key isn’t in the activation list. It must exist on at least one resource first, and propagation takes up to a day. Past that, check you’re in the management account and the key matches exactly, including case.
• Activated, but everything still shows untagged. Give it a full billing cycle. A key switched on mid-month leaves the first half of that month untagged until you backfill.
• Tags stopped working after an account move. Activation doesn’t survive a move between organizations. Reactivate from the new management account, then backfill.
• Cluster costs still land on one team. Instance tags can’t split a shared cluster. Opt into split cost allocation data, or treat the cluster as a shared cost in a split charge rule.
• You’ve hit the active key limit. Almost always too many keys rather than a real need for a quota increase. Use last-used timestamps to find dead ones, and backfill before deactivating.

Common mistakes

• Assuming a tagged resource is a tagged cost. It isn’t until you activate the key.
• Waiting for good coverage before activating. Activate first, improve coverage second, backfill third.
• Deactivating stale keys and then backfilling, which wipes their history out of the data.
• Measuring coverage by resource count instead of by spend.
• Rolling out a broad tag-enforcing SCP in production before anyone has read a compliance report.
• Trusting default_tags to reach resources Terraform didn’t directly create.
• Putting anything sensitive in a tag value. Tags surface in billing exports that finance and third-party tools read.

Best practices

• Keep the mandatory set to three to five keys, with defined allowed values rather than free text.
• Pin casing in the standard and enforce it with Tag Policies, because AWS won’t do it for you.
• Tag accounts as well as resources. Account tags cover charges resource tags structurally cannot.
• Treat coverage as a tracked percentage of spend with an owner and a monthly trend, not a project that finishes.
• Accept an irreducible remainder. Chasing the last few percent by hand costs more than it returns.
• Put allocation rules in Cost Categories rather than inventing tag values that encode a split.

Frequently asked questions

Are AWS cost allocation tags retroactive?

Partly. Activation applies going forward, but the management account can request a backfill reapplying current activation status across up to twelve months. The catch is that the resource must have carried the tag at the time, so backfill recovers late activation, never late tagging.

Why don’t my tags appear in Cost Explorer?

In order of likelihood: the key was never activated in the Billing console; you’re looking from a member account; the activation delay hasn’t elapsed; or the key differs in case from the one you activated.

How many cost allocation tags can I activate?

There’s a documented per-account cap on active keys, raisable through Service Quotas, plus a separate per-resource cap on total tags. Check current numbers in the AWS billing quotas docs. In practice the binding limit is human: past roughly a dozen keys, consistency collapses.

Should I use account tags or resource tags?

Both, for different jobs. Account tags handle top-level attribution and cover untaggable charges like support, credits and refunds. Resource tags handle granularity inside an account. If your accounts already map cleanly to teams, account tags are the cheaper win and worth doing first.

Is this worth doing on a small account?

Scale the effort to the shape of the bill. If you run a handful of services under one account and can name every resource from memory, account tags plus two resource keys is plenty. Worth noticing too that a fixed-price VPS from a provider like Contabo or InterServer has trivially perfect cost allocation, because the bill is one predictable line. Elastic infrastructure buys flexibility and pays for it in attribution work.

How do I allocate shared costs like NAT gateways?

Not with tags. Put them in a Cost Categories split charge rule and redistribute proportionally across consuming categories. Tag the shared resource to identify the bucket, then let the rule do the allocation.

The one thing worth remembering

AWS cost allocation tags fail silently, and that’s what makes them frustrating. Nothing errors. No alarm fires. Your resources look correctly tagged in every console you check, the bill quietly disagrees, and you find out when someone asks a question you can’t answer.

So check one thing now: open the cost allocation tags page in your management account and see whether the keys you’ve been diligently applying are actually switched on. If they aren’t, activate them today, then backfill. That twelve-month window rolls, and every month you wait is history you can’t recover.

Everything else here is incremental. Coverage is a percentage you improve, not a state you reach.

Need help getting your AWS bill to explain itself?

Cost attribution is unglamorous and easy to postpone until the quarter someone actually needs the numbers. If your account is already messy, it goes faster with someone who has untangled one before. I help with:

• Auditing tag coverage as a percentage of spend, broken down by service, so you know what you’re actually fixing
• Designing a minimal tag schema and the Tag Policies to keep it consistent across an organization
• Closing propagation gaps in Terraform and CloudFormation, around Auto Scaling groups, launch templates and attached storage
• Setting up Cost Categories with split charge rules so shared infrastructure stops distorting every team’s number
• Building cost and usage export pipelines in Athena, partitioned so the dashboard isn’t expensive to load
• Planning a staged tag enforcement rollout that doesn’t break anyone’s deployment pipeline

Send me a Cost Explorer screenshot grouped by your main tag key, or the output of a list call against your active tags, and I’ll tell you where the gap actually is.

The post AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill appeared first on John Nessime.

That is where most cost work goes wrong. The pressure is on the total, so people attack the biggest line item first. The biggest line item is almost always compute, and compute is the one place where a wrong guess shows up as latency at peak. Three weeks later the bill is lower and the p99 is worse, and nobody connects the two, because the change that caused it was a Tuesday afternoon instance swap that “went fine.”

This post is about how to cut AWS costs without breaking production by ordering the work differently: not by how much each change saves, but by how far the damage travels if the change is wrong. I will go through the levers in that order, with the traps in each one, the commands I use to find the waste, and the handful of cuts that look harmless and are not.

The cuts that break production are rarely the ones you fear

The line items that look most alarming on a bill are usually networking and storage overhead: NAT Gateway processing, public IPv4 hours, log ingestion, snapshots nobody owns. They feel dangerous to touch because they sit in the middle of the VPC. In practice, almost all of them can be removed or rerouted with zero behavioural change to your application. A gateway endpoint for S3 does not alter a single byte your code sees. Deleting an Elastic IP associated with nothing cannot affect anything, by definition.

The changes that feel safe are the ones with real blast radius. Dropping an instance one size. Switching a volume type. Moving objects to a colder storage class. Each is a single API call, reversible on paper, and each quietly narrows a performance envelope you were relying on without knowing it. The failure does not show up at deploy time. It shows up the next time you hit the load level that used to fit inside the old headroom.

So the rule I work by: sequence cost work by blast radius, not by savings size. Take everything production cannot possibly notice first. Then the things that move a performance envelope, one workload at a time, with a metric to watch. Then, last, the financial commitments, which cannot break your uptime at all but can lock you into paying for architecture you are about to throw away.

First, find out what you are actually paying for

The default Cost Explorer view groups by service, which is close to useless here. “EC2 – Other” is not a thing you can act on. It is a bucket holding EBS volumes, snapshots, NAT Gateway hours, NAT Gateway data processing, data transfer and public IPv4 charges, all mixed together.

Group by usage type instead

Usage type names the actual mechanism being billed. Strings like NatGateway-Bytes, NatGateway-Hours, PublicIPv4:InUseAddress and EBS:VolumeUsage.gp2 tell you what to go fix. This pulls last month’s usage types sorted by cost:

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '1 month ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=USAGE_TYPE 
  --output json 
  | jq -r '.ResultsByTime[0].Groups[] | [.Keys[0], .Metrics.UnblendedCost.Amount] | @tsv' 
  | sort -k2 -gr | head -30

That uses GNU date, so run it on Linux unless you have coreutils on macOS. Cost Explorer API calls are themselves billed per request, which surprises people the first time they wrap this in a cron job.

Billing data cannot tell you where your bytes went

This is the most useful thing to internalise about networking spend. Cost data knows how many gigabytes crossed the NAT Gateway. It has no idea whether they were headed to S3, to ECR, to CloudWatch or to a third-party API. There is no destination breakdown in billing data.

VPC Flow Logs are the missing half. Turn them on for the private subnets, then query them grouped by destination address with Athena or CloudWatch Logs Insights and map the ranges back to AWS services. Now you have the split: this much to S3, this much to ECR, this much genuinely to the internet. That is what tells you which endpoint is worth adding. Flow Logs cost money to ingest, so scope them tightly and turn them off once you have the answer. A week on the right subnets is enough.

Tier zero: savings production cannot notice

Everything in this tier changes billing without changing behaviour. Do all of it before you touch a single instance type.

The orphan sweep

AWS charges for allocated resources, not used ones. A volume attached to nothing bills at full rate. An Elastic IP attached to nothing bills by the hour, and has done ever since AWS began charging for public IPv4 addresses whether or not they are in use.

# Elastic IPs associated with nothing
aws ec2 describe-addresses 
  --query 'Addresses[?AssociationId==`null`].[PublicIp,AllocationId]' 
  --output text

# EBS volumes in the "available" state, meaning attached to no instance
aws ec2 describe-volumes --filters Name=status,Values=available 
  --query 'Volumes[].[VolumeId,Size,VolumeType,CreateTime]' 
  --output text

Run the same pass over load balancers with no healthy targets, snapshots whose source volume is gone, and old AMIs still holding snapshots behind them. Snapshot a volume before deleting it. The snapshot costs a fraction of the volume and buys you a rollback if something did care after all.

NAT Gateway is usually the largest risk-free win

A NAT Gateway bills three separate ways: an hourly charge for existing, a per-gigabyte charge for every byte it processes, and then whatever data transfer applies to where that traffic actually went. Teams look at the hourly rate, decide it is a rounding error, and never look at the processing line. The processing line is the one that grows with traffic.

The part that stings: traffic from a private subnet to S3 or DynamoDB in the same region is charged NAT processing even though it never leaves the AWS network. Gateway endpoints for those two services are free, with no hourly and no per-gigabyte charge. If your private subnets talk to S3 through a NAT Gateway, this is the best ratio of saving to risk available anywhere on the platform.

aws ec2 create-vpc-endpoint 
  --vpc-id vpc-0123456789abcdef0 
  --service-name com.amazonaws.eu-west-1.s3 
  --vpc-endpoint-type Gateway 
  --route-table-ids rtb-0123456789abcdef0

A gateway endpoint works by adding a prefix-list route to the route tables you name, so the main failure mode is naming the wrong tables and leaving traffic on the old path. Nothing breaks; you just do not save anything. The real caveat is policy: if you have bucket policies or endpoint policies that restrict access by source, check them first, because requests arriving through an endpoint present differently.

Interface endpoints for other services (ECR, CloudWatch Logs, Secrets Manager, SSM) are a different calculation. They charge per hour per availability zone plus per gigabyte, so they only pay off above a certain volume for that one service. Use the Flow Logs breakdown to decide. Pulling container images from ECR through NAT on every scale-out is the case that usually justifies one.

Log groups that never expire

CloudWatch Logs bills mostly on ingestion and secondarily on stored volume. A log group created without an explicit retention setting keeps data forever, so every debug group someone created during an incident years ago is still accruing storage.

# Log groups with no retention policy set
aws logs describe-log-groups 
  --query 'logGroups[?!retentionInDays].[logGroupName,storedBytes]' 
  --output text

aws logs put-retention-policy 
  --log-group-name /aws/lambda/my-function 
  --retention-in-days 30

Retention is the safe half. Reducing ingestion is a tier-one change, because you are removing evidence you might need during an incident. If you need a long retention window cheaply, exporting to S3 with a lifecycle rule beats paying CloudWatch storage rates indefinitely, and a log platform like Grafana Loki or a hosted equivalent gives you a cheaper long tail while CloudWatch keeps the recent window.

Incomplete multipart uploads

Failed large uploads leave parts behind in S3. Those parts are billed as storage and do not appear in a normal object listing, which is why some buckets bill for far more than they seem to contain. One lifecycle rule ends it:

{
  "Rules": [
    {
      "ID": "abort-incomplete-multipart",
      "Status": "Enabled",
      "Filter": {},
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    }
  ]
}

Apply it with aws s3api put-bucket-lifecycle-configuration. That call replaces the entire lifecycle configuration on the bucket, so fetch the existing one and merge rather than overwrite.

Tier one: savings that move a performance envelope

Now a wrong call has consequences. Everything here goes one workload at a time, with a named metric you watch for at least a full traffic cycle afterwards.

gp2 to gp3, and the throughput trap

This is presented everywhere as a free win, and for most volumes it is. gp3 costs less per gibibyte and gives every volume a baseline of 3,000 IOPS regardless of size, where gp2 scales at 3 IOPS per GiB and only reaches 3,000 sustained at 1 TiB. For a small database volume quietly burning through burst credits, gp3 is a straight upgrade in both cost and stability.

The trap is throughput, not IOPS. gp3’s baseline throughput is 125 MiB/s. A large gp2 volume can deliver up to 250 MiB/s. Migrate a big sequential-read volume to gp3 with the defaults and you halve its throughput ceiling while the cost report still shows a saving. Log processing, backup jobs and analytics scratch space are exactly the workloads that hit this, and exactly the ones where the symptom is “the nightly job got slower” rather than an alert.

Check VolumeReadBytes and VolumeWriteBytes in CloudWatch first, and provision throughput explicitly where the volume needs it:

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 
  --volume-type gp3 --iops 3000 --throughput 250

aws ec2 describe-volumes-modifications --volume-id vol-0123456789abcdef0 
  --query 'VolumesModifications[].[ModificationState,Progress]' 
  --output text

Elastic Volumes changes the type in place with the instance running, so there is no downtime. Two things to watch anyway. If the root volume is declared inline in a CloudFormation block device mapping, changing its type there can replace the instance rather than modify the volume; modify through the EC2 API instead. And there is a cooldown before a volume can be modified again, so plan on getting it right rather than iterating. On RDS the same change also runs without downtime, but the instance then enters a storage optimization state during which further storage changes are blocked.

Rightsizing without guessing

AWS Compute Optimizer reads CloudWatch metrics and returns recommendations with a confidence rating. Better than eyeballing CPU graphs, with one important gap: without the CloudWatch agent installed it cannot see memory. A JVM service sitting at 12% CPU and 88% heap looks like an obvious downsize candidate and is not.

aws compute-optimizer get-ec2-instance-recommendations 
  --query 'instanceRecommendations[?finding==`OVER_PROVISIONED`].[instanceArn,currentInstanceType]' 
  --output text

My working rules:

• Look at a window that includes your worst week, not a rolling average. Averages hide the peak that justifies the size.
• Change one dimension at a time, family or size, so you know what caused any regression.
• Treat burstable families as a separate decision. A T-family instance that exhausts CPU credits under sustained load either throttles or silently bills you for unlimited mode. Neither is what you wanted.
• Moving to Graviton usually saves more than downsizing x86, but it is an architecture change. Images need to be multi-arch and every compiled dependency needs to build. That belongs in a sprint, not in a cost-cutting afternoon.
• Write down the metric you expect to move and the threshold at which you revert, before you make the change.

S3 storage classes and the small-object trap

Intelligent-Tiering is the low-thought option and it works well for what it was designed for: objects of meaningful size with access patterns you cannot predict. It charges a small per-object monitoring fee, moves objects between tiers automatically, and applies no retrieval fee on the automatic access tiers.

Where it fails is object count. Objects below the eligibility threshold of 128 KB are not auto-tiered at all. They sit in the frequent access tier permanently while still paying monitoring. A bucket holding hundreds of millions of thumbnails, metadata records or small JSON documents can pay for monitoring that produces no saving whatsoever. Check average object size first, from S3 Storage Lens or an inventory report: total bytes divided by object count.

Lifecycle transitions have their own trap, since each transition is a billed request. Moving a billion small objects to a colder class can cost more in transition requests than the storage saving returns in a year. Transitions pay off on size, not on count. Expiry is the exception: deleting costs nothing and removes the storage entirely. While you are in there, check non-current versions in versioned buckets, which accumulate invisibly and need their own rule. And if your workload is genuinely egress-heavy, an object store with no egress charge such as Cloudflare R2 changes the arithmetic enough to model, in exchange for cross-provider latency and an operational split.

Tier two: commitments, which break budgets rather than uptime

Savings Plans and Reserved Instances cannot cause an outage. They are billing constructs. What they can do is hold you to a spend rate for one or three years after the architecture that justified it has changed. What I would want anyone to know before signing:

• Savings Plans cannot be cancelled during the term; AWS documentation is explicit. There is a limited return path with eligibility conditions and quotas per billing family, but plan as though there is no exit.
• Standard Reserved Instances can be listed on the Reserved Instance Marketplace, which is a real exit but not a guaranteed one, since somebody has to buy. Convertible RIs cannot be sold, only exchanged for a different configuration.
• Commit after rightsizing, never before. A plan applied to an oversized fleet locks in the waste and removes the incentive to fix it.
• Savings Plans do not reserve capacity. If you need a guarantee that instances will be available in a specific zone, that is an On-Demand Capacity Reservation, a separate mechanism the discount can then apply to.
• Savings Plans do not apply to Spot usage or to usage already covered by an RI. If much of your fleet is Spot, your committable baseline is smaller than the bill suggests.
• Compute Savings Plans survive architectural drift: family changes, size changes and moves between EC2, Fargate and Lambda stay covered. EC2 Instance Savings Plans and Standard RIs discount more deeply and tolerate drift poorly. Choose based on how stable your architecture actually is, not how stable you would like it to be.

Most teams land on a layered shape: commit at the baseline you are confident will still exist in a year, leave the variable top of the curve on demand or on Spot. Flat usage and a frozen architecture justify deeper, longer commitments. Mid-migration, mid-container-adoption or mid-anything, take the smaller discount for the shorter term.

One honest question worth asking here: does the workload need to be on AWS at all? Steady, non-elastic things like build runners, staging environments and internal tooling are often cheaper on a plain VPS from a provider such as Contabo or InterServer, while genuinely elastic workloads are often cheaper on AWS. The trade is operational, because you take back patching, backups and capacity planning that AWS was doing for you.

Troubleshooting: the bill went down and something broke

The hard part of diagnosing a cost regression is the delay. The change lands on Tuesday, the symptom appears at month-end batch time, and by then three other things have shipped. Work backwards from the symptom:

• Latency got worse but CPU is fine. Look at storage. VolumeQueueLength climbing with throughput flat at a round number means you are sitting on a provisioned ceiling, which is the gp3 default-throughput signature. On gp2, a falling BurstBalance is the tell, and it usually starts hours before anything is user-visible.
• Intermittent timeouts after rightsizing. Check whether the new instance type has lower network or EBS bandwidth, not just fewer vCPUs. Baseline bandwidth scales with size, and smaller sizes burst their network the way T instances burst CPU.
• Something stopped reaching a service. If you added VPC endpoints, check whether an endpoint policy is denying it. Interface endpoints also need the endpoint’s own security group to allow inbound from your workload, which is the most common reason a new endpoint appears to break connectivity.
• Objects are slow or missing from an application path. Check whether a lifecycle rule moved them into a class with retrieval latency. The Glacier tiers inside Intelligent-Tiering are opt-in for exactly this reason, and enabling them turns instant reads into restore requests.

Keep a change log for cost work specifically: a dated line per change, what you expected to save, what metric you were watching. Five minutes each, and it is the difference between diagnosing this in an hour and in a week.

Common mistakes

• Starting with the biggest number. Compute is the biggest line and the riskiest to touch. It is the last tier, not the first.
• Committing before rightsizing. This locks in the exact waste you were trying to remove, for years.
• Accepting gp3 defaults on high-throughput volumes. Cheaper storage, halved throughput ceiling, no alert.
• Enabling Intelligent-Tiering on buckets full of tiny objects. Monitoring fees on objects that will never move a tier.
• Turning off monitoring to save on monitoring. Understandable at 2am on a budget deadline, indefensible during the next incident. Cut cardinality and retention, not coverage.
• Doing it once. Cost drifts back within two quarters unless something structural stops it.

Best practices that make the savings stick

• Make cost visible before it is a crisis. AWS Budgets with alerts at a percentage of forecast, plus Cost Anomaly Detection, catch the runaway job on day two rather than on the invoice. Platforms like Vantage, CloudZero or Kubecost for per-namespace Kubernetes attribution earn their place once several teams share an account structure.
• Tag at creation, in the IaC. Tags applied afterwards do not backfill historical cost data. If it was not tagged when it was created, that spend is unattributable forever.
• Put non-production on a schedule. Dev and staging running nights and weekends is the most common waste in any account, and stopping them has no production blast radius at all.
• Make the cheap thing the default in the module. gp3, sensible log retention, gateway endpoints and lifecycle rules belong in your Terraform or OpenTofu modules, so the next environment is born correct rather than remediated later.
• Review commitment coverage on a schedule. Coverage and utilisation are different numbers and both matter. High coverage with low utilisation means you over-committed.

Frequently asked questions

What is the fastest way to cut AWS costs without breaking production?

Add gateway endpoints for S3 and DynamoDB, delete unassociated Elastic IPs and unattached EBS volumes, and set retention on log groups that have none. All four are billing-only changes with no behavioural effect on your application, and together they often account for a large share of the surprise on a bill.

Why is my “EC2 – Other” line item so large?

It is a mixed bucket: EBS volumes and snapshots, NAT Gateway hours and data processing, data transfer, and public IPv4 address charges. Regroup Cost Explorer by usage type rather than by service to see which of those is driving it.

Is migrating from gp2 to gp3 always cheaper?

No. gp3 has a lower per-gibibyte price, but IOPS above the included baseline and throughput above the included baseline are billed separately. A volume needing high provisioned throughput can cost more on gp3 than it did on gp2. Model the volumes that genuinely need extra performance instead of assuming a flat saving across the fleet.

Can I cancel a Savings Plan if my usage drops?

Assume no. AWS documents Savings Plans as non-cancellable for the term. There is a limited return mechanism with eligibility rules and quotas, and no secondary marketplace of the kind that exists for Standard Reserved Instances. Size the commitment to the baseline you are confident about, not the peak you hope for.

Should I use Spot Instances to cut costs?

For fault-tolerant, interruptible work such as CI runners, batch jobs and stateless workers behind a queue, the discount is hard to ignore. For anything holding state in memory or serving a request that cannot be retried, the interruption notice is short and the failure is abrupt. Spot is an architecture decision, not a billing toggle.

How do I find out where my NAT Gateway traffic is going?

Billing data cannot tell you. Enable VPC Flow Logs on the relevant subnets, query them with Athena or CloudWatch Logs Insights grouped by destination address, and map the results back to AWS service ranges. That gives you the split between S3, ECR, other AWS services and the genuine internet, which is what determines whether an endpoint pays for itself.

The one thing worth remembering

If you take one idea from this, take the ordering. You cut AWS costs without breaking production by sorting the work by blast radius rather than by savings size, and by accepting that the frightening-looking networking and storage line items are usually the safe ones, while the innocuous single-API-call changes are the ones that bite.

Do the tier-zero sweep first and completely. It is free, it is fast, and it buys you the standing to say “not yet” when someone asks you to shrink a production database on a Thursday. Then move one performance envelope at a time, with a metric and a revert threshold. Commit last, at the baseline you actually believe in.

Need a second pair of eyes on your AWS bill?

This is work I do regularly, and it goes faster with someone who has no attachment to how the account got this way. Things I can help with:

• A usage-type level breakdown of your bill, with savings ranked by blast radius, so you know what is safe this week and what needs a change window.
• NAT Gateway and data transfer analysis from VPC Flow Logs, ending in a specific list of which endpoints to add and what each is worth.
• EBS and S3 review: gp3 candidates with throughput checked properly, lifecycle rules, version cleanup, and storage class decisions based on your real object size distribution.
• CloudWatch and logging spend reduction that keeps the signal you need during incidents, including moving long-tail retention off CloudWatch storage rates.
• Savings Plan and Reserved Instance modelling against your actual baseline, including what happens to coverage if the migration you are planning goes ahead.
• Guardrails so it stays fixed: Budgets, anomaly alerting, tagging enforced in Terraform or OpenTofu, and cheap defaults baked into your modules.

If you want a concrete starting point, send me a Cost Explorer export grouped by usage type, or the output of the orphan sweep commands above, and I will tell you what I would touch first and what I would leave alone.

The post Cut AWS Costs Without Breaking Production: A Blast-Radius Playbook appeared first on John Nessime.

The expensive stuff is not the running instances. It’s the access keys nobody can attribute, the CloudTrail trail that quietly stopped delivering months ago, the Terraform state that no longer matches reality, and the region somebody spun a proof of concept up in two years ago. None of that comes up on a kickoff call. All of it comes up in week three, after you’ve quoted.

So I run an AWS account health check before I give anyone a number. It’s read-only, it takes a couple of hours, and it’s the cheapest insurance in consulting. This post is the actual check: the six passes, the commands, what each finding does to the estimate, and what to do when the client won’t give you the access to run it.

Why the AWS account health check comes before the quote

A quote is a bet on how much unknown there is. You’re pricing the work in the brief plus everything you’ll find while doing it, and the second half is invisible until you look. Two accounts can look identical and differ by a factor of three in effort. The difference is nearly always archaeology: how much has to be reverse engineered before you can safely change anything.

There’s a second reason that matters more than pricing. If you touch an account that already had a compromised key sitting in it, the incident becomes yours by proximity. Establishing what the account looked like before you arrived isn’t paranoia. It’s the only version of events you’ll be able to evidence later.

The access I ask for, and what a refusal tells me

I ask for a cross-account IAM role that the client creates and my account assumes. Not an IAM user. Not keys in an email. A role, with an external ID the client generates.

The external ID solves one specific problem. Without it, a trust policy that names my account can be used by anyone who talks me into assuming a role on their behalf, and I end up acting inside a third party’s account without knowing it. AWS calls this the confused deputy problem. It’s the piece of cross-account setup people skip most often.

On that role I ask for two AWS managed policies: SecurityAudit and ViewOnlyAccess. The pairing is the point. ViewOnlyAccess lets you enumerate resources across services, so you can see what exists. SecurityAudit lets you read configuration and policy documents, so you can see how it’s wired together. Neither alone gives you the picture, and neither lets you change anything. Billing is separate again: Cost Explorer read access has to be added explicitly and the feature has to be enabled in the account.

The reply to that request tells you as much as the check does.

• “Here’s the role ARN and external ID” means somebody there understands AWS identity. Good sign.
• “I’ll just make you an admin user, easier” means nobody is managing access. Expect that looseness everywhere else.
• “Our previous developer set it up and we can’t reach him” means you’re inheriting credentials nobody can account for. Price the rotation work as its own line.
• “We’d rather not grant access before signing” is fair. Offer a paid discovery engagement. If they won’t pay for two hours of assessment, they won’t pay for the fix.

The six passes

Each pass answers one question. Everything here is read-only, and the order matters, because each pass changes how you read the next.

Pass 1: Who can do what

Confirm where you actually are first. That sounds trivial until you’re in an organization with a dozen similarly named accounts.

aws sts get-caller-identity
aws iam list-account-aliases
aws organizations describe-organization

If the third command fails because the account isn’t in an organization, that’s a finding by itself. Standalone means no service control policies, no organization trail, and no central place to revoke anything.

Then the credential surface. The summary gives counts; the credential report gives per-principal detail, including when each password and key was last used.

aws iam get-account-summary

aws iam generate-credential-report
aws iam get-credential-report --query Content --output text | base64 --decode > credential-report.csv

The report generates asynchronously, so the second call may need a retry the first time. What I read from it: how many IAM users exist at all, how many have keys that have never been used, and how many have keys last used a long time ago that are still active.

An unused key is worse than a busy one. A key in daily use is attributable to something. A key last used eighteen months ago and still enabled belongs to a person or script nobody remembers, which means nobody will notice when someone else starts using it.

Then check whether anyone has ever asked this question before you:

aws accessanalyzer list-analyzers

IAM Access Analyzer offers external access analyzers, which flag resources shared outside a zone of trust you define, and unused access analyzers, which surface unused roles, keys and passwords. Unused access analysis is billed per IAM role and user analyzed per month, so it isn’t free, but it answers the least privilege question far faster than reading policy documents by hand. An empty list means nobody has ever asked this account who can reach what.

Root closes the pass. In an AWS Organizations setup, centralized root access management lets the management account or a delegated IAM administrator remove root credentials from member accounts entirely, so there’s no root password, key or MFA device to manage per account. Live root credentials on every member account is a real finding, and one of the few security fixes that’s genuinely quick.

Pass test: you can name every principal that can reach production, and say when each last did.

Pass 2: Would anyone notice

This is the pass that most often changes my number, because it’s the failure that stays invisible until it isn’t. Logging that exists but has silently stopped working is more dangerous than no logging, since everybody believes they’re covered.

aws cloudtrail describe-trails 
  --query 'trailList[].{Name:Name,MultiRegion:IsMultiRegionTrail,Org:IsOrganizationTrail}'

aws cloudtrail get-trail-status --name TRAIL_NAME_HERE

aws configservice describe-configuration-recorder-status

The first command tells you a trail is configured. The second tells you whether it’s delivering, which is the part that matters. A trail whose last delivery failed because someone edited the destination bucket policy still shows up in the console as a configured trail. The status call is where the truth is.

Multi-region matters for the same reason region sprawl does: a single-region trail records nothing about the region where somebody is quietly running compute. Check GuardDuty too, and how long logs are actually retained. Ninety days of retention against a one-year obligation is a project, not a task, and it belongs in the quote rather than in week four.

One naming trap worth knowing: AWS renamed the original Security Hub to Security Hub CSPM and launched a separate, newer service under the Security Hub name. When a client says “we have Security Hub,” ask which one, because the findings format and the configuration are not the same thing.

Pass test: a trail is delivering right now, covers every region, and its logs are kept for at least as long as the business needs them.

Pass 3: How wide the blast radius is

People forget regions. Not occasionally, constantly. A demo in Singapore, a test bucket in Ireland, a Lambda deployed from a tutorial that defaulted to us-east-1.

for r in $(aws ec2 describe-regions --query 'Regions[].RegionName' --output text); do
  count=$(aws ec2 describe-instances --region "$r" 
    --query 'length(Reservations[].Instances[])' --output text)
  echo "$r $count"
done

Left without --all-regions, describe-regions returns the regions enabled for the account, which is exactly what you want. Adding the flag pulls in disabled regions too, and the per-region calls against those just error out.

Run the same loop shape for anything carrying state or cost: RDS instances, load balancers, EBS volumes, NAT gateways. NAT gateways are the classic. They bill hourly whether or not anything routes through them, and a forgotten one in a region nobody visits is pure waste. Then the two exposure checks I never skip:

aws ec2 describe-security-groups 
  --filters Name=ip-permission.cidr,Values=0.0.0.0/0 
  --query 'SecurityGroups[].{Id:GroupId,Name:GroupName,VpcId:VpcId}'

aws s3api list-buckets --query 'Buckets[].Name' --output text

For each bucket that matters, check its public access block configuration and its policy. An open security group on 443 in front of a load balancer is fine. An open security group on a database port is a different conversation, and you want that conversation before you own the account.

Pass test: you have a list of every region with resources in it, and nothing on that list surprises the client.

Pass 4: Whether any of this is described in code

The question isn’t “do they use Terraform.” It’s whether the code still describes the account. Those diverge quietly. Somebody makes an urgent console change during an incident, never brings it back into code, and from then on the repo is fiction that plans cleanly right up until the day it destroys something.

terraform plan -refresh-only

terraform plan -detailed-exitcode

The refresh-only plan shows where real infrastructure has drifted from state without proposing to change anything. The detailed exit code version is the one for scripts: it exits 0 for no changes, 1 for an error and 2 when there’s a diff, so you can gate on drift instead of reading output by eye.

Three things I want to know here:

• What fraction of live resources are managed by code? Half a codebase is often worse than none, because it creates false confidence about which half.
• Where does state live and who can write to it? Remote state in a versioned, locked S3 bucket is a different risk profile from a state file on a laptop.
• Does the pipeline that applies changes still run? Credentials that expired six months ago mean everything since went in by hand.

Tagging belongs here too. With nothing tagged you can’t attribute cost, can’t scope a change safely, and can’t tell a production resource from an abandoned experiment. Untagged environments add hours to everything that follows.

Pass test: a refresh-only plan comes back clean, or the drift is a short list you can read out loud.

Pass 5: What shape the spend is in

I don’t care much about the total. I care about the shape: which services dominate, whether the curve is flat or climbing, and whether anything jumped without an explanation.

aws ce get-cost-and-usage 
  --time-period Start=$(date -d '6 months ago' +%Y-%m-01),End=$(date +%Y-%m-01) 
  --granularity MONTHLY 
  --metrics UnblendedCost 
  --group-by Type=DIMENSION,Key=SERVICE

The date -d syntax is GNU coreutils; on macOS use the BSD equivalent or a fixed start date. Worth knowing before you loop this: the Cost Explorer API bills per paginated request, unlike the console view, so a wide query across several dimensions is a small but real charge on the client’s bill. Keep it narrow and mention it in your notes.

• Data transfer in the top three usually means an architecture problem rather than a rate problem. Cross-AZ chatter and NAT gateway egress are the usual suspects.
• A step change with no matching launch deserves a direct question. Sometimes it’s a feature. Sometimes it’s a runaway job. Occasionally it’s mining on a stolen key.
• No budgets and no anomaly monitors means nobody finds out about a spike until the invoice arrives.
• Heavy commitment purchases against workloads you’re about to re-architect are a constraint on your design, and need to be on the table before you propose anything.

If the client wants ongoing cost visibility past what Cost Explorer offers, platforms like Vantage or CloudZero sit on the Cost and Usage Report and give per-team allocation and unit economics that native tooling makes awkward. Raise it as a separate line, not part of the engineering work.

Pass test: you can explain the top three cost lines in a sentence each, and account for every month-on-month jump.

Pass 6: What happens at two in the morning

Short pass, about escalation. What support plan is the account on, and would anything wake a human up?

Support plan matters more than people expect. On Basic you get the service limits checks and a small subset of Trusted Advisor security and fault tolerance checks, manual refresh only, and no technical support cases at all. The full check library and the Trusted Advisor API need a paid plan. If your engagement depends on opening a case during a cutover, that’s a prerequisite, not a nice to have.

There’s one date worth carrying in your head, because it’s a real deprecation rather than a version number. AWS has announced that Developer Support, Business Support and Enterprise On-Ramp are discontinued on 1 January 2027, with Business Support+ as the replacement tier. If you’re scoping anything that runs past that boundary, check which plan the client is on and whether their support cost is about to move.

Then monitoring. CloudWatch alarms wired to an SNS topic with no confirmed subscribers are the same failure as the dead trail: configured, visible, useless. If they run anything beyond alarms, whether that’s Grafana Cloud, Datadog or a self-hosted Prometheus stack, find out who holds the credentials and whether the routing has been tested this year.

Pass test: you can name the human who gets woken up, and the path that reaches them.

The findings that actually move the number

• Unattributable credentials. Each one has to be rotated or removed, and each needs a conversation with whoever might break. That’s coordination time, which is the hardest kind to compress.
• No working audit trail. No baseline means you can’t prove what you did or didn’t change. Add the cost of standing logging up before the real work starts.
• Significant drift between code and reality. Reconciliation is slow, fiddly and carries destruction risk. The single largest multiplier I see.
• No tags. A tax on every subsequent task rather than a task of its own, since every scoping decision becomes an investigation.
• Resources in unmentioned regions. Each is a small parallel environment with its own security groups, its own logging gap and its own bill.
• One shared admin identity. Nothing in CloudTrail is attributable to a person, so your first task is identity work the client didn’t ask for.

I don’t present these as criticism. I present them as line items with an hour estimate against each, and let the client decide what’s in scope. That conversation is much easier before a contract exists than after.

If you only run five checks

Sometimes you get twenty minutes and a screen share.

1. Pull the credential report and count active keys that haven’t been used recently.
2. Confirm a CloudTrail trail exists, is multi-region, and is currently delivering.
3. Loop the enabled regions for running instances and NAT gateways.
4. Run a refresh-only plan against whatever infrastructure code exists.
5. Pull six months of cost grouped by service and look at the shape.

Those cover identity, evidence, blast radius, delivery speed and money. The rest of the check refines a picture these five already gave you.

Where this check goes wrong

• Treating it as an audit. You’re scoping work, not issuing a compliance report. A twenty page findings document makes a prospect defensive and costs you a day.
• Checking one region. The default region in your profile is not where the problems are.
• Trusting the console over the API. Configured and working are different states, and the console is better at showing the first.
• Taking long-lived keys because it’s faster. Accept an admin access key by email to save fifteen minutes and you’ve become part of the problem you were hired to assess.
• Running the loops unannounced. A burst of API calls from an unfamiliar principal is exactly what GuardDuty exists to flag. Tell them you’re doing it and roughly when.

Frequently asked questions

How long should an AWS account health check take?

Two to three hours for a single account once you have access, including the write-up. Multi-account organizations take longer but not linearly, because the interesting variation concentrates in one or two accounts and the rest are copies of a landing zone pattern.

Should I charge for the assessment?

For a short engagement, folding it into the proposal is fine. For anything substantial, or where the client is shopping several consultants, quote it as a small standalone piece with a written deliverable. It qualifies the client, protects your time, and the findings have value to them even if they hire someone else.

What if the client won’t give me read-only access before signing?

Quote time and materials, or quote fixed price with an explicit assumptions list naming every unknown. Do not quote a fixed price on an account you haven’t seen. The pattern that works is a small paid discovery phase, then a fixed price set once you know what you’re dealing with.

Is ReadOnlyAccess enough on its own?

It’s broader than you probably want. ReadOnlyAccess extends to reading data in some services, not just configuration, which is more than you need for scoping and more than many clients will be comfortable granting. The SecurityAudit and ViewOnlyAccess pairing is narrower and easier to justify in writing.

Can I automate the whole thing?

The enumeration, yes, and it’s worth scripting since you’ll run it repeatedly. The interpretation, no. Whether five unused IAM users are a problem depends entirely on what the client is trying to do, and that judgement is most of what you’re selling.

What if I find something actively compromised?

Stop and tell them immediately, in writing, with timestamps. Don’t remediate unilaterally: you’re read-only, and containment steps can destroy evidence. Your job is a clear description of what you saw and when, so they can involve whoever they need to.

Does this apply to small accounts too?

Especially to small accounts. Larger organizations usually have some governance, even imperfect. A three year old account at a small business that four different contractors have touched is where the surprises live, and it’s exactly the account that gets described as simple.

The one thing worth remembering

An AWS account health check isn’t a security exercise dressed up as sales. It’s the only reliable way to price uncertainty, and uncertainty is what you’re actually quoting on.

If you take one habit from this, take the second pass. Everything else on the list is something you’d eventually trip over while doing the work. A logging gap is different, because it stays invisible while it costs you, and by the time it matters you’re the person who was in the account when it happened. Check the trail is delivering. Then check the rest.

Need a second pair of eyes on an AWS account?

I do this for consultancies scoping engagements and for businesses who inherited an account they don’t fully understand. Usually that looks like:

• A read-only AWS account health check with a written findings list, hour estimates per item, and a clear split between urgent and later.
• Cross-account audit role setup, with a trust policy and external ID you can hand to any consultant without granting more than you meant to.
• Credential archaeology: working out what every active access key belongs to, then rotating or removing the ones that belong to nothing.
• Getting CloudTrail, AWS Config and log retention into a state where they’re delivering rather than just configured.
• Terraform drift reconciliation, including importing unmanaged resources back under code without a destructive plan.
• Cost shape reviews: the forgotten NAT gateways, orphaned volumes and idle regions, plus budgets and anomaly alerts that reach a real inbox.

If you’ve got a credential report, a terraform plan output or a Cost Explorer screenshot you can’t make sense of, send it over and I’ll tell you what I’d look at next.

The post The AWS Account Health Check I Run Before Quoting a Project appeared first on John Nessime.