Then someone in finance asks what the reporting pipeline costs. You open Cost Explorer, group by Project, and over half the bill lands in a bucket called “No tag key”.

Nothing is broken. The tags are there. They just aren’t on the bill, because tagging a resource and activating a cost allocation tag are two separate operations, and AWS never warns you when you’ve done only the first.

This post covers getting AWS cost allocation tags to cover a messy account: the activation gap, what backfill will and won’t repair, where coverage leaks even with decent discipline, and a retrofit order that converges.

A resource tag and a cost allocation tag are not the same thing

A resource tag is metadata on the resource. A cost allocation tag is a billing dimension. Getting the second means taking a key that already exists on your resources and explicitly switching it on in the Billing and Cost Management console.

Two details make this bite. First, activation lives in the management account. From a member account, the page tells you tags are managed at the payer level and that’s the end of it. Plenty of teams have a good tagging standard and no access to the one screen that makes it count. Related trap: an account moving between organizations has its activated keys reset to inactive.

Second, the lag. AWS documents up to 24 hours for a newly applied key to appear on the activation page, and up to another 24 hours after you activate before it takes effect. The loop between “we agreed a standard” and “I can see it in Cost Explorer” is two days at best.

Past a handful of keys, drive it from the API. Listing the inactive ones is the fastest way to find what you forgot:

# Every key AWS knows about that is NOT currently billing-active
aws ce list-cost-allocation-tags --status Inactive

# Switch a set of them on
aws ce update-cost-allocation-tags-status --cli-input-json '{
  "CostAllocationTagsStatus": [
    {"TagKey": "Project",     "Status": "Active"},
    {"TagKey": "Environment", "Status": "Active"},
    {"TagKey": "Owner",       "Status": "Active"}
  ]
}'

User-defined keys appear in the report with a user: prefix and AWS-generated ones carry the reserved aws: prefix, so they never collide. There’s a cap on active keys, raisable through Service Quotas, which you almost certainly don’t need to raise.

What the backfill repairs, and what it quietly doesn’t

Activation used to be strictly forward-looking, so activating late permanently orphaned everything before it. That’s no longer true. The management account can ask AWS to reapply current activation status across up to twelve months of historical cost data.

# Must be the first day of a month, no earlier than twelve months back
aws ce start-cost-allocation-tag-backfill 
  --backfill-from 2025-01-01T00:00:00Z

• Start date must be the first day of a month, and can’t reach back more than twelve months.
• One request per 24 hours, and you can’t submit another while one is processing.
• It backfills activation status, not tag values.

That last point catches people. If the resource wasn’t carrying the tag when the cost was incurred, backfill gives you nothing for those months. It fixes “I forgot to activate.” It does not fix “I forgot to tag.” A tag applied in June and activated in November backfills to June and no further, whatever start date you hand it.

It also cuts both ways. Backfill applies the current status of every key, so a key sitting inactive right now gets stripped out of historical data too. If you’re pruning an overgrown tag list, backfill before you deactivate the old keys, not after, or you’ll erase the history you were trying to keep.

Where AWS cost allocation tags actually leak coverage

With activation sorted and every Terraform-managed resource tagged, a stubborn slice of the bill still refuses to attribute. It matters which slice, because the fixes have nothing in common.

Charges no resource can carry

Support charges, refunds, credits, tax, some Marketplace subscriptions. There’s no resource to hang a tag on, so tagging discipline never touches them. On accounts with a large support plan or heavy credit activity, that’s a visible fraction of the invoice.

The answer is account tags. AWS added support for using AWS Organizations account tags as a cost allocation dimension, applying to all metered usage inside the tagged account and flowing into Cost Explorer, Budgets, Cost Anomaly Detection and Cost Categories. Because attribution happens at the account level, it picks up exactly what resource tags can’t reach.

This is recent enough that most tagging guides predate it, and it’s the highest-value item here if your accounts already map roughly onto teams.

Resources created by other resources

This is the quiet one, and usually the biggest surprise gap in an otherwise well-run account. Terraform’s provider-level default_tags applies to resources Terraform creates. It does not reach the EC2 instances an Auto Scaling group launches, because Terraform didn’t create those, the ASG did. Your ASG is tagged. Your instances aren’t. Same for volumes attached at launch, ENIs, snapshots and AMIs.

data "aws_default_tags" "current" {}

resource "aws_autoscaling_group" "app" {
  # ... other config ...

  # Push provider default tags down to launched instances
  dynamic "tag" {
    for_each = data.aws_default_tags.current.tags
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

resource "aws_launch_template" "app" {
  # ... other config ...

  tag_specifications {
    resource_type = "instance"
    tags          = data.aws_default_tags.current.tags
  }

  tag_specifications {
    resource_type = "volume"
    tags          = data.aws_default_tags.current.tags
  }
}

For a standalone instance, volume_tags on aws_instance covers its attached volumes. General rule: whenever one AWS resource provisions another on your behalf, assume tags don’t propagate until you’ve checked the created resource in the console, not the plan output.

Shared infrastructure that belongs to everyone

NAT gateway data processing, VPC interface endpoints, a Transit Gateway, a shared load balancer, the central logging bucket. You can tag all of these perfectly and the tag says Project=platform, which is accurate and useless for chargeback.

Transit Gateway is worth calling out because it supports tagging the gateway resource itself, so data processing and transfer charges pick up the tag rather than only attachment hours. That gap used to swallow a lot of network spend.

Beyond that, stop chasing tags and do allocation math instead. Cost Categories support split charge rules that redistribute a shared bucket across your other categories, using proportional splits, fixed percentages or an even division. Proportional is the sane default; fixed percentages are for when finance already agreed a number and the report has to match it.

Container spend hiding behind one instance tag

A shared EKS or ECS cluster is one tagged compute footprint running fifteen teams’ workloads. The instance tag gives you the total and nothing about who caused it. Split cost allocation data is the native answer: opt in and the cost and usage export gains rows breaking cluster cost down to individual pods or tasks. It won’t account for idle headroom you’re deliberately carrying, but it beats one line item labelled “the cluster”.

Case drift and near-duplicate keys

Keys and values are case-sensitive. Environment, environment and ENV are three distinct keys producing three columns, each holding a partial and misleading picture. Tag Policies in AWS Organizations exist for this: define the key including its capitalization, define permitted values, and get an org-wide compliance report. Cheapest governance on this list, because it costs nobody a deploy.

A retrofit order that converges

Retrofits stall because people start at the bottom, sweeping untagged resources by hand while the creation path keeps producing more. This order avoids that.

1. Measure first. Group Cost Explorer by your intended key and record what percentage of spend falls into “No tag key”. That’s your baseline and your only honest progress metric.
2. Cut the key set down. Three to five keys: owner, environment, and a project or cost centre. Coverage on five beats partial coverage on thirty.
3. Activate immediately, even with terrible coverage. Both clocks start now: the activation delay and the rolling twelve-month backfill window. There’s no downside to activating a key that’s barely applied.
4. Rank the gap by spend, not resource count. Sort untagged line items by cost descending. It’s usually a few NAT gateways, a couple of oversized databases and one forgotten bucket. Ten thousand untagged security groups cost nothing.
5. Fix the creation path before the backlog. IaC defaults, launch template tag specifications, ASG propagation. Sweeping first means sweeping again next month.
6. Sweep the stragglers in bulk. The Resource Groups Tagging API works across services and regions, which beats going service by service.
7. Backfill, then re-measure. Compare against step one. If the number barely moved, you fixed the wrong resources.

# Find what already carries a Project tag, to diff against inventory
aws resourcegroupstaggingapi get-resources --tag-filters Key=Project

# Apply tags to a specific set of ARNs
aws resourcegroupstaggingapi tag-resources 
  --resource-arn-list arn:aws:rds:eu-west-1:111122223333:db:reporting-01 
  --tags Project=reporting,Environment=prod

Run it against a couple of ARNs and confirm in the console before looping over a list. Bulk tagging is trivially easy to point at the wrong account.

Enforcing tags without becoming the tag police

Four mechanisms, in increasing order of blast radius. Most teams need the first three and should be careful with the fourth.

• IaC defaults. Provider-level default tags plus required module variables. Cheapest, catches the majority, nobody thinks about it.
• CI validation. A linter rule that fails the pull request on missing required tags. The failure lands on the person who can fix it, when they can fix it.
• Detective controls. The AWS Config managed rule for required tags flags non-compliant resources without blocking anything. Tells the truth, breaks nothing.
• Preventive controls. A service control policy denying creation when a required key is absent, keyed on a null check against the request tag.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRDSCreateWithoutProjectTag",
      "Effect": "Deny",
      "Action": "rds:CreateDBInstance",
      "Resource": "*",
      "Condition": {
        "Null": {
          "aws:RequestTag/Project": "true"
        }
      }
    }
  ]
}

The honest trade-off: service control policies are where tagging programmes go to die. Not every API supports tag-on-create, and not every legitimate workflow passes tags through the way you expect. Roll a broad deny across the org on day one and you break somebody’s deployment on day two, the ticket lands on you, and the policy is gone by day three along with everyone’s appetite for the project.

Scope it to one service at a time, starting in sandbox. I’d rather run Config plus Tag Policies for a quarter and fix the top three offenders in the IaC. The deny is a finishing move, not an opening one.

Measuring coverage so it doesn’t rot

Cost Explorer gives the headline number. For a per-service breakdown, query the cost and usage export in Athena. One caveat: tag column naming differs between export versions, older ones giving a flattened column per activated key and newer formats a map column. Run a DESCRIBE first rather than guessing.

-- Confirm the actual column names first
DESCRIBE your_cur_database.your_cur_table;

-- Then: untagged spend, ranked by service
SELECT   line_item_product_code,
         SUM(line_item_unblended_cost) AS untagged_cost
FROM     your_cur_database.your_cur_table
WHERE    your_project_tag_column = ''
  AND    line_item_unblended_cost > 0
GROUP BY 1
ORDER BY 2 DESC
LIMIT    20;

That ranked list is your work queue; everything below the top twenty is noise. The list call also returns a last-used timestamp per key, the cleanest way to find keys nobody has applied to anything in months.

If you’d rather not maintain the dashboard yourself, this is a fair place to buy rather than build. Grafana Cloud sits on Athena comfortably if you already run Grafana, and platforms like CloudZero, Vantage and nOps handle allocation modelling out of the box. Their real value is the shared-cost and commitment maths. Their real cost is another vendor between you and your billing data, and several price against a percentage of the spend they watch, so run that number first.

Troubleshooting

• The key isn’t in the activation list. It must exist on at least one resource first, and propagation takes up to a day. Past that, check you’re in the management account and the key matches exactly, including case.
• Activated, but everything still shows untagged. Give it a full billing cycle. A key switched on mid-month leaves the first half of that month untagged until you backfill.
• Tags stopped working after an account move. Activation doesn’t survive a move between organizations. Reactivate from the new management account, then backfill.
• Cluster costs still land on one team. Instance tags can’t split a shared cluster. Opt into split cost allocation data, or treat the cluster as a shared cost in a split charge rule.
• You’ve hit the active key limit. Almost always too many keys rather than a real need for a quota increase. Use last-used timestamps to find dead ones, and backfill before deactivating.

Common mistakes

• Assuming a tagged resource is a tagged cost. It isn’t until you activate the key.
• Waiting for good coverage before activating. Activate first, improve coverage second, backfill third.
• Deactivating stale keys and then backfilling, which wipes their history out of the data.
• Measuring coverage by resource count instead of by spend.
• Rolling out a broad tag-enforcing SCP in production before anyone has read a compliance report.
• Trusting default_tags to reach resources Terraform didn’t directly create.
• Putting anything sensitive in a tag value. Tags surface in billing exports that finance and third-party tools read.

Best practices

• Keep the mandatory set to three to five keys, with defined allowed values rather than free text.
• Pin casing in the standard and enforce it with Tag Policies, because AWS won’t do it for you.
• Tag accounts as well as resources. Account tags cover charges resource tags structurally cannot.
• Treat coverage as a tracked percentage of spend with an owner and a monthly trend, not a project that finishes.
• Accept an irreducible remainder. Chasing the last few percent by hand costs more than it returns.
• Put allocation rules in Cost Categories rather than inventing tag values that encode a split.

Frequently asked questions

Are AWS cost allocation tags retroactive?

Partly. Activation applies going forward, but the management account can request a backfill reapplying current activation status across up to twelve months. The catch is that the resource must have carried the tag at the time, so backfill recovers late activation, never late tagging.

Why don’t my tags appear in Cost Explorer?

In order of likelihood: the key was never activated in the Billing console; you’re looking from a member account; the activation delay hasn’t elapsed; or the key differs in case from the one you activated.

How many cost allocation tags can I activate?

There’s a documented per-account cap on active keys, raisable through Service Quotas, plus a separate per-resource cap on total tags. Check current numbers in the AWS billing quotas docs. In practice the binding limit is human: past roughly a dozen keys, consistency collapses.

Should I use account tags or resource tags?

Both, for different jobs. Account tags handle top-level attribution and cover untaggable charges like support, credits and refunds. Resource tags handle granularity inside an account. If your accounts already map cleanly to teams, account tags are the cheaper win and worth doing first.

Is this worth doing on a small account?

Scale the effort to the shape of the bill. If you run a handful of services under one account and can name every resource from memory, account tags plus two resource keys is plenty. Worth noticing too that a fixed-price VPS from a provider like Contabo or InterServer has trivially perfect cost allocation, because the bill is one predictable line. Elastic infrastructure buys flexibility and pays for it in attribution work.

How do I allocate shared costs like NAT gateways?

Not with tags. Put them in a Cost Categories split charge rule and redistribute proportionally across consuming categories. Tag the shared resource to identify the bucket, then let the rule do the allocation.

The one thing worth remembering

AWS cost allocation tags fail silently, and that’s what makes them frustrating. Nothing errors. No alarm fires. Your resources look correctly tagged in every console you check, the bill quietly disagrees, and you find out when someone asks a question you can’t answer.

So check one thing now: open the cost allocation tags page in your management account and see whether the keys you’ve been diligently applying are actually switched on. If they aren’t, activate them today, then backfill. That twelve-month window rolls, and every month you wait is history you can’t recover.

Everything else here is incremental. Coverage is a percentage you improve, not a state you reach.

Need help getting your AWS bill to explain itself?

Cost attribution is unglamorous and easy to postpone until the quarter someone actually needs the numbers. If your account is already messy, it goes faster with someone who has untangled one before. I help with:

• Auditing tag coverage as a percentage of spend, broken down by service, so you know what you’re actually fixing
• Designing a minimal tag schema and the Tag Policies to keep it consistent across an organization
• Closing propagation gaps in Terraform and CloudFormation, around Auto Scaling groups, launch templates and attached storage
• Setting up Cost Categories with split charge rules so shared infrastructure stops distorting every team’s number
• Building cost and usage export pipelines in Athena, partitioned so the dashboard isn’t expensive to load
• Planning a staged tag enforcement rollout that doesn’t break anyone’s deployment pipeline

Send me a Cost Explorer screenshot grouped by your main tag key, or the output of a list call against your active tags, and I’ll tell you where the gap actually is.

The post AWS Cost Allocation Tags: Getting Real Coverage on a Messy Bill appeared first on John Nessime.

That’s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn’t the one that throws. It’s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.

This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.

Why a backfill that finishes cleanly can still be wrong

A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it’s over. The only evidence it leaves behind is a set of objects in a bucket.

So the question you need to answer months later isn’t “did the job succeed?” It’s “is this window complete, and how do I know?” If your design can’t answer that from artifacts on disk, you don’t have a backfill. You have a large amount of data of unknown provenance.

Failure family 1: the source moved while you paginated

This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.

If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.

What to do instead

• Prefer keyset or cursor pagination. If the API offers an opaque cursor or a “records after ID X” parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can’t shift it.
• Sort by an immutable column. If you’re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.
• Bound every request on both ends. Don’t ask for “everything before now.” Ask for a closed interval with an upper bound already in the past. A closed window can’t grow while you read it.
• Know which timestamp the filter applies to. Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.

That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it’s worth writing down which one you chose.

Failure family 2: the retry that wrote the same page twice

Backfills retry. They have to, because you’re hammering an API for hours and something will time out. The problem is what the retry writes.

If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.

import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise

Two things in there are worth explaining rather than copying.

IfNoneMatch="*" is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 PreconditionFailed otherwise. It’s supported on PutObject and CompleteMultipartUpload. That turns “did I already write this page?” into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 ConditionalRequestConflict, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.

The same thing from the CLI, useful for testing the behaviour by hand:

aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"

The retries block matters for a reason that trips people up. In botocore, max_attempts means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a Config object it counts retries on top of the first. total_max_attempts exists only in a Config object and always means total, so using it removes the ambiguity. There’s also an adaptive mode that layers client-side rate limiting on top of standard; AWS still labels it experimental, so I reach for standard plus my own concurrency cap first.

Failure family 3: a layout designed for writing, not reading

The prefix structure you pick during the backfill is the one you’re stuck with. Reorganising later means rewriting every object, and copy requests aren’t free.

Partition on event time, not ingestion time

The tempting layout is the date you ran the job. It’s easy, it’s monotonic, and it’s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:

s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-<token>.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-<n>.parquet

Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.

Don’t let the write path throttle itself

S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is partitioned. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 SlowDown responses even when your sustained rate looks well inside the documented ceiling.

For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise within a single day and drive thousands of writes per second into one dt= prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.

Register partitions without a scan

A backfill can create thousands of partitions at once. MSCK REPAIR TABLE handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.

Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.

TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)

Replace START_DATE with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching storage.location.template, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.

Failure family 4: the seam between backfill and live pipeline

Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.

The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don’t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API’s own filtering will all conspire to open a hole at exactly T.

Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.

Failure family 5: the storage class you can’t undo

Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.

• S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.
• Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you’re charged the remainder.
• Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.

Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you’re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.

What I’d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.

One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It’s a five-minute change people usually discover after the bill.

Proving the backfill is complete

Here’s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.

{
  "dataset": "orders",
  "window": "<dt value>",
  "filter_field": "created_at",
  "range_start_inclusive": "<iso8601>",
  "range_end_exclusive": "<iso8601>",
  "pages_written": "<int>",
  "records_written": "<int>",
  "source_reported_total": "<int or null>",
  "last_cursor": "<opaque token>",
  "schema_fingerprint": "<sha256 of sorted field names>"
}

The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose records_written disagrees with source_reported_total is suspect. A window whose schema_fingerprint differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.

Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.

Troubleshooting a backfill that already ran

1. Count objects per window and look at the shape. Use S3 Inventory rather than a live ListObjectsV2 walk on a large bucket. Plot objects per dt= partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.
2. Reconcile record counts against the source. Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what’s in S3. If you kept manifests, this is one query.
3. Check for duplicate natural keys within a partition. A GROUP BY on the record ID with HAVING COUNT(*) > 1 finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren’t deterministic.
4. Look at boundary hours, not whole days. Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.
5. Verify the layout still matches the table definition. If you’re using partition projection and a query returns zero rows, the first suspect is storage.location.template not matching the actual prefix, trailing slashes included.

Common mistakes

• Treating an exit code of zero as evidence of completeness.
• Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.
• Partitioning by ingestion date because it’s what the job naturally knows.
• Making the backfill and live-ingest boundary exact instead of overlapping.
• Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.
• Running MSCK REPAIR TABLE over thousands of new partitions and trusting that it completed.
• Backfilling on a last-modified field while assuming you captured creation events.
• Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.

Best practices when you backfill historical API data into S3

• Make every window independently re-runnable. If re-running one day is routine rather than scary, you’ll actually fix problems when you find them.
• Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.
• Bound every API request on both ends, and never let the upper bound be “now”.
• Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn’t possible.
• Emit a manifest per window, written last, and reconcile it against source counts.
• Compact to Parquet before applying any lifecycle transition.
• Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won’t sleep halfway through.
• Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.

FAQ

Should I write raw JSON or convert to Parquet during the backfill?

Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don’t want to re-fetch years of data because of a type error.

How large should each object be?

In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.

Can I run the backfill in parallel across date windows?

Yes, and that’s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API’s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or Retry-After header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.

Do I need AWS Glue or Airflow for this?

Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.

What if the API caps how far back I can query?

Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it’s usually faster and less rate-limited than paginating.

Does any of this change with S3-compatible storage instead of AWS?

The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with If-None-Match, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you’re depending on against that provider’s documentation rather than assuming parity with S3.

How do I know the backfill is actually finished?

When every expected window has a manifest, every manifest’s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.

The one thing worth remembering

A backfill isn’t finished when the job exits. It’s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.

If you’re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.

Need a hand with a backfill?

This is a large part of what I do. If you’re staring at a historical range and an API with a rate limit, I can help with:

• Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.
• Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.
• Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.
• Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.
• Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.
• Wiring the seam between the backfill and your live pipeline so there’s no gap at the handover, plus the monitoring that would have caught it.

Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That’s usually enough to spot where the gap is.

The post Backfilling Historical API Data into S3 Without Silent Gaps appeared first on John Nessime.