That is the failure mode that makes this comparison worth writing about. The Parquet vs JSON vs CSV question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.

This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else’s benchmark, and an honest look at where CSV and JSON still beat the columnar option.

The billing model is the whole story

Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.

Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery’s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.

That last part catches people. LIMIT 10 does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.

So the question is not “which format is smaller on disk.” It is “how few bytes can the engine get away with reading.” Those diverge fast.

Four levers, not one

When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.

1. Column pruning

Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.

2. Block skipping

Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.

The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group’s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.

3. Splittability

This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.

Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.

If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.

4. Request overhead

Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine’s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.

CSV: still the right answer more often than people admit

Where it wins. Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.

Where it doesn’t. No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.

CSV is an excellent transport format and a poor storage format. That distinction is the useful one.

JSON: the most expensive way to store a table

Where it wins. Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.

Where it doesn’t. Every record repeats every key name. Store ten million events with a field called customer_identifier and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.

JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.

Parquet: cheap reads, real obligations

Where it wins. Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.

Where it doesn’t. This is the part the enthusiastic posts skip.

• You cannot append. Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.
• Schema mismatches across files are a runtime error, not a warning. Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.
• Timestamps are a compatibility minefield. The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.
• It is opaque. You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.
• Writing costs more. Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.

None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.

Measure it on your data, in about twenty minutes

Published benchmarks tell you about somebody else’s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.

1. Get a real byte count for what you have now. Not an estimate, the actual prefix.

aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable

The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.

2. Convert a representative slice, not the whole thing. One partition is enough. In Athena, a CTAS statement writes the output for you.

CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';

Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.

3. Run your three most common queries against both tables and compare bytes scanned. Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.
4. On BigQuery, price it before you run it. A dry run costs nothing and returns the byte estimate.

bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'

5. Inspect the Parquet file you just wrote. Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.

python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"

That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.

6. Put a guardrail up before anyone else touches it. Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.

Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.

Arguments that don’t survive contact

• “We gzip our CSVs, so we already save the money.” Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.
• “Parquet is always faster.” Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.
• “We’ll just convert everything to Parquet.” Converting a landing zone that receives appended events all day means fighting the format’s biggest weakness. Land raw, convert on a schedule, query the converted copy.
• “Storage is cheap so format doesn’t matter.” Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.
• “Our BI tool caches, so nobody re-scans.” Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.

How I’d decide

Not a scorecard. A short sequence of questions, in order.

• Is this data leaving your organisation? CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.
• Is it arriving continuously, one record at a time? Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.
• Is it read more than once, by more than one person, with filters? Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.
• Is the whole dataset smaller than a few hundred megabytes? Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.
• Are you fighting append semantics, concurrent writers, or schema evolution? You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.

The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.

Frequently asked questions

How much smaller is Parquet than CSV?

It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.

Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?

Less than you would think, because once data is loaded into a BigQuery native table it lives in Google’s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.

Is ORC better than Parquet?

They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet’s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.

Why did my query cost the same after I converted to Parquet?

Usually one of three things. You are running SELECT *, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.

Should I use Snappy, GZIP or ZSTD for Parquet?

Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.

Can I query CSV and Parquet in the same table?

Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.

What about the cost of the conversion job itself?

Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.

The one thing worth remembering

The Parquet vs JSON vs CSV decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.

Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.

Need help cutting a data lake bill?

Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:

• Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves
• Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses
• Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control
• Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges
• Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types
• Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end

If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.

The post Parquet vs JSON vs CSV: What Each Format Actually Costs You appeared first on John Nessime.

The answer was one API call away, and nobody had made it. Snapshot management had been failing for the entire table since a developer added an Iceberg branch for a backfill experiment. Compaction kept running happily. Old snapshots never expired, the files they pinned never became eligible for deletion, and the bucket grew. No alarm, no console banner, no email.

That is the shape of most S3 Tables compaction problems in production. The managed maintenance is good at what it does. What it is bad at is telling you when it has stopped doing it. This post covers the failure modes that bite, the storage tail that keeps billing after you think data is gone, how the compaction charge accumulates, and the handful of knobs you actually control.

What managed compaction actually does

S3 Tables runs three maintenance jobs, all on by default. Compaction and snapshot management are per table. Unreferenced file removal is per table bucket and cannot be overridden per table.

Compaction is the Iceberg rewriteDataFiles routine with scheduling taken off your hands. It combines small objects into larger ones, applies pending row-level deletes on the way through, and writes the result as the table’s newest snapshot. Target file size defaults to 512MB and accepts 64MB to 512MB. Parquet row groups inside use a 128MB default.

There are four strategies. auto is the default and picks for you: sort order defined in metadata means sort, otherwise binpack. binpack packs small files into bigger ones. sort orders rows hierarchically by your chosen columns so engines prune more files. z-order interleaves several columns into one sortable value, for queries filtering on multiple dimensions rather than one leading column.

Both sort and z-order need a sort_order in your Iceberg table properties and the s3tables:GetTableData permission to set. They also cost more per byte processed than binpack, because they do more work.

Failure family one: maintenance that fails without telling anyone

This is the one worth reading the post for. Maintenance jobs have four states: Successful, Failed, Disabled, and Not_Yet_Run. There is exactly one place that state is exposed on demand, and it is an API call you have to remember to make.

aws s3tables get-table-maintenance-job-status 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table

The response carries a block for each of icebergCompaction, icebergSnapshotManagement, and icebergUnreferencedFileRemoval, each with its own status, a lastRunTimestamp, and a failureMessage. The three are independent. Compaction reporting Successful tells you nothing about whether the other two are running.

The failure messages are worth knowing before you see them, because several of them describe conditions your own pipeline creates.

Tags and branches disable snapshot expiry for the whole table

If any user-defined tag or branch exists on the table, snapshot management fails. Not partially. The whole table stops expiring snapshots, and the message says snapshot management does not support user-defined tags or references. This applies even when the tag or branch has a short retention period of its own.

Branches are a normal Iceberg workflow. Someone creating one for a write-audit-publish flow or a backfill is not doing anything wrong. They are just silently turning off garbage collection for that table, and the effect shows up on next month’s bill rather than in their terminal.

Iceberg retention properties conflict with the managed config

If history.expire.max-snapshot-age-ms or history.expire.min-snapshots-to-keep is set as an Iceberg table property, snapshot management fails for the entire table regardless of the value. The managed service will not reconcile its own configuration against properties in metadata.json. It refuses instead.

This catches teams migrating a self-managed Iceberg table into a table bucket, because those properties are exactly what a self-managed setup uses. Remove them and let the maintenance API own retention.

ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.max-snapshot-age-ms');
ALTER TABLE mydb.mytable UNSET TBLPROPERTIES ('history.expire.min-snapshots-to-keep');

gc.enabled set to false

Setting the Iceberg gc.enabled table property to false stops both snapshot management and unreferenced file removal. Leave it unset or explicitly true. Teams sometimes set it defensively during a migration and then forget, which produces a table that compacts forever and cleans up never.

KMS key policy missing the maintenance principal

If the table is encrypted with a customer managed key, that key’s policy has to grant access to the service principal maintenance.s3tables.amazonaws.com. Without it you get an insufficient-access failure and all maintenance stops. Catch this in Terraform review rather than production, because the table reads and writes perfectly while maintenance is dead.

Failure family two: the storage tail you keep paying for

Deleting rows from an Iceberg table does not delete bytes from S3. Neither does compacting them. Understanding the cascade is the difference between predicting your storage bill and being surprised by it.

When compaction rewrites files, the originals stay because older snapshots still reference them. They become removable only once every referencing snapshot expires. Snapshot management drives that expiry from minSnapshotsToKeep (default 1) and maxSnapshotAgeHours (default 120). Both conditions must be satisfied before a snapshot goes.

When a snapshot expires, S3 marks objects referenced only by it as noncurrent. It does not delete them. They sit noncurrent for nonCurrentDays (default 10), then are deleted permanently.

Add that up. On defaults, a byte compaction superseded today is still on your bill roughly fifteen days from now. Files never referenced by any snapshot take a parallel path through unreferencedDays (default 3) into the same noncurrent window.

Two things follow. A table under heavy rewrite pressure carries a persistent storage overhang that is a function of your write rate and these windows, not of your logical table size. And those windows are your fastest cost lever if you do not need long time travel, configured in two different places.

# Per table: how long snapshots survive
aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergSnapshotManagement 
  --value '{"status":"enabled","settings":{"icebergSnapshotManagement":{"minSnapshotsToKeep":3,"maxSnapshotAgeHours":48}}}'

# Per bucket: how long dead files linger before deletion
aws s3tables put-table-bucket-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --type icebergUnreferencedFileRemoval 
  --value '{"status":"enabled","settings":{"icebergUnreferencedFileRemoval":{"unreferencedDays":2,"nonCurrentDays":3}}}'

Be deliberate. Shortening these windows shortens your time travel range and your recovery options. Deletion of noncurrent objects is permanent, and recovering them means a support case, if it is possible at all. I would rather pay for a few extra days of overhang on a table I might need to roll back than discover the limit during an incident.

Failure family three: write amplification and the compaction charge

S3 Tables bills storage, requests, a per-object monitoring fee for every object in the bucket, and, when compaction is on, both the objects and the bytes processed. Rates change and vary by region, so check the current S3 pricing page rather than a number from a blog post. What matters is the shape.

The shape is that compaction charges scale with how badly your writers behave, not with how big your table is. A batch job writing a few large files an hour barely registers. A streaming pipeline committing every few seconds produces a huge number of tiny objects, and every byte gets read and rewritten at least once, sometimes more as small compacted files merge again into larger ones.

People reach for targetFileSizeMB first. Wrong lever. Changing the target changes the destination, not how much data has to move to get there.

The lever that works is the commit pattern on the write side. Buffering so the pipeline commits larger files less often reduces object count, monitoring fees, compaction object charges, and bytes processed all at once. This is the trade S3 Tables makes explicit: the service takes the maintenance work off your plate and prices it, so a writer that generates more work pays more for it.

Because compaction is a separate billing dimension, break it out in whatever cost tooling you already use. Cost Explorer with usage-type grouping will show it, and platforms like Vantage or CloudZero make it easier to watch alongside the rest of your S3 spend.

Failure family four: files compaction quietly skips

Compaction supports Parquet, Avro, and ORC. It does not support the Fixed data type, nor the brotli or lz4 compression codecs.

That last one matters because lz4 is a reasonable choice for a write-heavy pipeline where encoding speed counts. Pick it and you get exactly the outcome you were avoiding: a table full of small files that the compaction you are paying for will not consolidate.

Compaction also writes Parquet by default even when source files are Avro or ORC. To keep the original format, set the write.format.default table property to avro or orc. Most people want Parquet for read performance, but if a downstream consumer expects a specific format, this is where you say so.

Failure family five: compaction and your writers fighting over the table

Iceberg uses optimistic concurrency with conflict detection to arbitrate commits. As far as the table is concerned, compaction is just another writer. When a compaction commit and one of yours touch overlapping state, one loses.

When compaction loses you see Failed with a message about failing to commit because of out-of-date metadata, and a note that it will retry. Benign in isolation. Seen persistently, it means your write pattern and the maintenance schedule collide often enough that compaction is not keeping up, and the small-file problem quietly worsens while status flips between Failed and Successful.

When your job loses, you get a commit failure in your pipeline. AWS explicitly recommends retry logic on your side, and that is not boilerplate. A pipeline without commit retries, against a table with managed compaction on, will fail intermittently for reasons that have nothing to do with your code.

Turning managed compaction off, and when that is right

Start with the honest case for leaving it on. Table maintenance is real operational work: a Spark cluster or scheduler, a job that has to run, alerting for when it does not, and someone who understands why it failed at 3am. Managed compaction removes all of that for a per-use fee. For most teams with a handful of tables and batch-shaped ingest, that is the right call. The cost is small and the alternative has a salary attached.

The case against is control. You cannot influence when compaction runs, hint from the write side about what to prioritise, or make it work harder after a large backfill. The published observability is a status field and a timestamp per job. If your workload has hot partitions needing attention now and cold partitions that never need touching, managed compaction cannot express that distinction and you pay to rewrite both.

Disabling is a per-table change, and it does not require zeroing the settings.

aws s3tables put-table-maintenance-configuration 
  --table-bucket-arn arn:aws:s3tables:REGION:ACCOUNT_ID:bucket/BUCKET_NAME 
  --namespace my_namespace 
  --name my_table 
  --type icebergCompaction 
  --value '{"status":"disabled","settings":{"icebergCompaction":{"targetFileSizeMB":256}}}'

From there you run maintenance yourself. Trino exposes an optimize procedure through ALTER TABLE ... EXECUTE, and Spark has the standard Iceberg rewrite_data_files procedure. Both let you scope work to specific partitions on your own schedule, which is the entire point of taking it back.

Do this with your eyes open. You now own the scheduling, the compute, the failure alerting, and the snapshot expiry. Teams that disable managed compaction and never build the replacement end up worse off than if they had left it alone. If cost rather than control is driving this, fix the write pattern first and re-measure. That is usually cheaper than the engineering you are about to sign up for.

Troubleshooting S3 Tables compaction and maintenance

Work through these in order. The first step resolves more cases than the rest combined.

1. Check all three job statuses, not one. Run get-table-maintenance-job-status and read every key in the response. A green compaction status next to a failed snapshot management status is the classic growing-bucket signature.
2. Read the failure message literally. It names the exact property or condition at fault. Tags or references means someone created a branch or tag. Incompatible table properties means history.expire.* is set.
3. Status is Disabled or Not_Yet_Run unexpectedly. Someone replaced the configuration. Confirm with get-table-maintenance-configuration, and remember put-table-maintenance-configuration replaces rather than merges.
4. Compaction runs but file count stays high. Check codec and data types against the unsupported list. Files using brotli or lz4, or containing the Fixed type, are outside what compaction touches.
5. Insufficient-access failures on an encrypted table. Check the KMS key policy grants the maintenance.s3tables.amazonaws.com service principal, and that the key is enabled and still exists.
6. Storage still growing with everything green. Do the arithmetic on your retention windows. Snapshot age plus noncurrent days is the floor on how long superseded data bills, and defaults put that at roughly two weeks.

For anything historical, S3 Tables maintenance operations appear in CloudTrail management events, which is where you go to answer “when did this start” rather than “what is broken now”.

Common mistakes

• Treating the three maintenance jobs as one thing. They fail independently and for different reasons. “Maintenance is enabled” is not a status.
• Creating branches or tags on a managed table. A normal Iceberg workflow that switches off snapshot expiry for the whole table.
• Tuning targetFileSizeMB to reduce cost. It changes where the data lands, not how much has to be moved. Fix the commit pattern instead.
• Setting sort or z-order without a sort order defined. The strategy needs sort_order in table properties and the s3tables:GetTableData permission, and it costs more per byte than binpack.
• Running write pipelines with no commit retry. Compaction is a concurrent writer. Conflicts are expected behaviour, not an anomaly.

Best practices for running S3 Tables compaction in production

• Poll job status on a schedule and alert on it. A small scheduled function calling get-table-maintenance-job-status across your tables, emitting a metric per job type, closes the largest observability gap in the service. Alert on Failed and on a stale lastRunTimestamp, on whatever dashboard you already watch, Grafana or CloudWatch.
• Make maintenance config part of your IaC. These are settings someone can change with one CLI call and no review. Codify them so drift is visible.
• Set retention windows from a real requirement. Ask how far back anyone needs to time travel, then set maxSnapshotAgeHours and nonCurrentDays to that plus a margin, rather than accepting defaults by omission.
• Batch on the write side before tuning anything else. Fewer, larger commits improve object count, monitoring cost, compaction cost, and query performance simultaneously.
• Choose the strategy from the query pattern. One dominant filter column suggests sort. Several equally weighted predicates suggest z-order. Neither suggests leaving it on auto.
• Track the compaction billing dimension separately. It moves independently of storage and it is the earliest signal that a writer’s behaviour has changed.

Frequently asked questions

How do I know if S3 Tables compaction is actually running?

Call get-table-maintenance-job-status and read the icebergCompaction block. It returns Successful, Failed, Disabled, or Not_Yet_Run, plus a last-run timestamp and a failure message. No dashboard surfaces this by default, so to know without asking you have to poll it and emit your own metric.

Why is my S3 Tables storage growing when compaction says it succeeded?

Two likely causes. Either snapshot management is failing separately, which stops old snapshots expiring and pins their data files indefinitely, or your retention windows are longer than you assumed. On defaults, superseded data sits for the snapshot age window plus the noncurrent window, around two weeks. Check the other two job statuses first.

Can I run my own compaction alongside the managed one?

You can, but it is a poor idea. Both are writers competing under Iceberg’s optimistic concurrency, so you generate commit conflicts and pay twice to rewrite the same data. Pick one. If you need engine-side control, disable managed compaction for that table first.

Does disabling compaction stop all S3 Tables charges?

No. It stops the compaction object and byte-processing charges for that table. Storage, requests, and the per-object monitoring fee continue, and that monitoring fee scales with object count, which is exactly what stops improving once compaction is off. Disabling it on a small-file-heavy table can raise monitoring cost over time.

Is sort or z-order compaction worth the extra cost?

It depends on whether queries filter on the sorted columns. Sorted layout lets engines skip files using Parquet column statistics, so the saving shows up in query cost and latency, not storage. Dominant queries filtering on one column favour sort. Several dimensions in varying combinations favour z-order. Queries that mostly scan everything justify neither.

Can I recover data deleted by S3 Tables maintenance?

Deletion of noncurrent objects is permanent. AWS documents contacting support to view or recover objects marked noncurrent, which is not a path to depend on. Treat retention window changes as a one-way door, set from a stated requirement rather than trimmed until the bill looks better.

The one thing worth remembering

S3 Tables compaction is a good trade for most teams. The service does real work you would otherwise be scheduling and babysitting, and it does it well enough that you can stop thinking about small files as a daily concern.

What it does not do is tell you when it stops. No alarm fires when a branch someone created for an afternoon experiment switches off snapshot expiry, and no banner appears when a KMS policy change kills maintenance across a bucket. The price of the convenience is that you build the one thing the service did not: a scheduled check on get-table-maintenance-job-status that tells you when the automation stopped automating. That is an afternoon of work. Skipping it is how a flat workload turns into a bill that grows every month for reasons nobody can explain.

Need help with Iceberg tables on AWS?

I work with teams running Apache Iceberg on S3, and most problems are the ones above rather than anything exotic. Things I can help with:

• Auditing every table in a bucket for failed or disabled maintenance jobs, and finding which have stopped expiring snapshots
• Building the scheduled job-status check and alerting that S3 Tables does not ship with, wired into CloudWatch or Grafana
• Working out where a bucket’s storage growth actually comes from, and setting retention windows against a real requirement
• Reworking streaming and CDC write patterns so commits produce fewer, larger files before compaction charges scale with them
• Deciding table by table whether managed or engine-side maintenance is right, and implementing whichever you pick
• Migrating self-managed Iceberg tables into table buckets without carrying over the properties that break managed maintenance

If something is not adding up, send me the output of get-table-maintenance-job-status for the table and a rough description of your write pattern. That is usually enough to say what is going on.

The post S3 Tables Compaction in Production: The Cost of Convenience appeared first on John Nessime.

That is the failure mode worth understanding, and it stays invisible until you go looking for it. The conversion was real. The files did get much smaller. But the dashboard queries were still selecting every column from an unpartitioned table, so the engine still had to open every column chunk in every file. Columnar storage only pays out when the query gives the reader something to skip.

This is a working comparison of Parquet vs JSON vs CSV on the two things that end up on an invoice: bytes read, and time spent reading them. Each format gets a profile with the cases where it wins and the cases where it does not, followed by the places where the promised savings quietly evaporate and a procedure for modelling the change on your own data before you spend an engineering week on it.

Your billing model decides how much the format matters

Before comparing formats, work out what you are actually charged for. The same conversion produces very different savings depending on which of these you are on.

• Bytes scanned from object storage. Amazon Athena is the clearest case. You are billed for the bytes a query reads out of S3, rounded up, with a minimum charge per query. Here a format change converts almost directly into money.
• Logical bytes processed. BigQuery’s on-demand model bills on the uncompressed logical size of the columns a query references in a native table, not on the compressed size sitting on disk. Compressing harder does not lower that bill. Touching fewer columns does.
• Compute time. Snowflake, Databricks, EMR, and anything you run on your own hardware charge you for how long a machine is awake. The format shows up as wall clock time, which is a real cost but a much softer one.

The BigQuery detail catches people out. Google documents that on-demand cost is calculated from logical, uncompressed bytes for native tables, but that when you query external data stored as Parquet or ORC, the bytes charged are limited to the columns BigQuery actually reads. The same file format therefore sits on opposite sides of the billing line depending on whether the data was loaded or is being referenced in place. Both Athena and BigQuery also apply a 10 MB minimum per query, and BigQuery applies it per table referenced, so a swarm of small queries against small tables costs far more than the data volume suggests.

CSV: cheap to produce, expensive to interrogate

CSV has no schema, no statistics, and no internal structure. Every value is text. The number 1000000000 takes ten bytes instead of the four it would take as a 32-bit integer, and because the layout is row-wise, there is nothing for a column-aware compressor to exploit.

Where CSV wins

• A human needs to open it. Nothing else comes close.
• Export from a legacy system that offers no other option.
• Small data. Below a few megabytes, the metadata and encoding overhead of a columnar file is not repaid.
• Streaming appends. You can write a line at a time and stop whenever you like.

Where CSV does not

Two problems, and the second is worse than the first.

The first is that the reader has to parse every byte, even the bytes it intends to throw away. The DuckDB team published a comparison on TPC-H data at scale factor 20, run on a laptop, reporting the median of five runs, and it is the most honest set of numbers I have seen on this. Loading the lineitem table took roughly 11.8 seconds from CSV against 5.2 seconds from Snappy-compressed Parquet, with the CSV file about five times larger on disk. That is a 2x gap on load, not the order of magnitude most people assume. Modern CSV readers are genuinely fast.

The gap opens when you query the files directly rather than loading them. On the same setup, TPC-H Q1 ran in about 6.7 seconds against CSV and 0.9 seconds against Parquet, and the join-heavy Q21 took roughly 20 seconds against 2.1. That difference is not parsing throughput. It is that the Parquet reader can skip row groups using statistics and skip unreferenced columns entirely, while the CSV reader cannot skip anything, because nothing in the file tells it what it would be skipping.

The second problem is dialect. There is an RFC for CSV and it is widely ignored. Quoting, escaping, embedded newlines, date formats, and whether NA means null or Namibia are conventions rather than rules, so every consumer re-guesses and sooner or later one guesses differently from the others. That failure does not raise an error. It produces wrong numbers in a report nobody questions.

JSON: you pay for the schema on every single row

Newline-delimited JSON is what most event pipelines emit, because a producer can add a field without coordinating with anyone downstream. That flexibility is worth something real. It is also why JSON is usually the most expensive format on a bytes-scanned bill: every record repeats every key name, so a field called customer_subscription_status carries those thirty-odd bytes on every row, forever.

Where JSON wins

• Landing zone for data whose shape you do not control. Accept it, keep the raw copy, argue about the schema later.
• Genuinely nested or sparse records, where flattening into columns would give you a table that is mostly nulls.
• Debugging. Being able to decompress one file and read a record with your eyes is worth a lot at three in the morning.
• API and webhook payloads, where it is simply what arrives.

Where JSON does not

As a query target at any serious volume. You are paying scan charges on repeated key names, on quote characters, and on the fact that every number is stored as text. Compressing the files helps storage and helps the scan bill on Athena, but it cannot give the engine anything to skip, and gzip brings a separate problem covered below.

The nested-data argument for keeping JSON has also weakened. Parquet now has a ratified Variant type for semi-structured data, with support arriving across Delta Lake, Iceberg, and Spark, plus a shredding scheme that pulls frequently accessed fields out into real columns while leaving the rest flexible. If you are keeping JSON purely because your records have unpredictable shape, that reason is worth revisiting rather than treating as settled.

Parquet: what the file is actually doing

Understanding the layout is what lets you predict whether a migration will pay for itself, so it is worth thirty seconds.

A Parquet file is divided into row groups. Inside each row group, every column is stored as its own contiguous column chunk, and each chunk is split into pages. The footer at the end of the file carries the schema and, per column chunk, statistics such as minimum, maximum, and null count. That structure is what makes the two headline optimisations possible:

• Column pruning. The reader consults the footer, works out the byte ranges of the three columns you asked for, and never issues a read for the other forty-seven.
• Predicate pushdown. If a row group’s maximum value for event_date falls below your filter’s lower bound, the entire row group is skipped without being read.

Two later additions refine this. The page index stores per-page minimum and maximum values in a structure near the footer, so a reader doing a selective scan can locate matching pages without walking every page header, which previously meant pulling most of the column off disk anyway. Bloom filters cover the case statistics cannot help with at all: a high-cardinality column such as a user ID, where every row group’s min and max span the whole value range and pruning by range is useless. A bloom filter can say that a value is definitely not present in a row group. DuckDB even exposes parquet_bloom_probe so you can check which row groups a given value would eliminate.

Encoding does the rest of the work. Because a column chunk holds values of a single type, dictionary and run-length encoding apply before any general-purpose compressor starts. Low-cardinality columns such as country codes or status flags compress extremely well. UUIDs and free text do not, and writers often disable dictionary encoding for them. This is why compression ratios quoted in blog posts are close to meaningless for your data: the ratio is a property of your cardinality, not of the format.

Where Parquet does not win

• Point lookups. Fetching one complete row by ID means touching every column chunk. A row format reads a single contiguous block instead.
• Row-level updates. Parquet files are immutable. Changing one row means rewriting the file, which is precisely the problem Iceberg and Delta Lake exist to solve on top of it.
• Streaming writes. The schema lives in the footer, so a writer has to finish the file before it is readable. Small-batch writers produce small files, which carries its own tax.
• Tiny datasets. Below a few megabytes, footer and row group overhead is not worth paying.
• Human inspection. It is binary. You need tooling, and someone will eventually ask you for a CSV export anyway.

Where the savings quietly disappear

Every one of these has cost someone a migration that looked excellent in the design document.

• Small files. Each file costs a metadata read, an object storage request, and reader setup. Thousands of 2 MB Parquet files will lose to a handful of large CSVs. The commonly cited target is roughly 128 MB to 1 GB per file, and you need a compaction step to get there if your writer emits micro-batches.
• One enormous row group. Parallelism in most engines is per row group. A file containing a single row group is processed by a single thread no matter how many cores you have. Check this rather than assuming the writer got it right.
• Select-everything queries in the reporting layer. Column pruning cannot help a query that asks for all columns. This is the one from the opening paragraph, and it is usually a BI tool doing it rather than a person.
• No partitioning. Format reduces bytes per row. Partitioning reduces how many rows are considered at all. The two multiply, and partitioning is normally the larger lever. Converting without partitioning leaves most of the money on the table.
• Gzipped text. AWS documents that Parquet and ORC are always splittable, because they compress sections independently and carry metadata pointing at those sections, while most text compression formats are not. Bzip2 and LZO are splittable; gzip is not. One 5 GB gzipped CSV cannot be divided among workers, so it is read start to finish by one reader. The compression made storage cheaper and the query slower.
• Uppercase file extensions. A small operational trap worth knowing: Athena determines the compression type of CSV and JSON data from the file extension, does not recognise uppercase extensions such as .GZ, and treats a file with no extension as uncompressed plain text.

How to model the change before you migrate

Do not migrate on faith. This takes an afternoon and answers the question properly.

1. Find out what you scan today. Every Athena query execution carries a DataScannedInBytes statistic. Pull it for a representative window and rank queries by bytes scanned. Almost always a handful of them are most of the bill, and those are the only ones worth optimising.
2. Convert one day of data, not all of it. A single partition is enough to measure with.
3. Look inside the resulting file before trusting it. Row group count, row group size, and compression codec decide whether the file behaves.
4. Re-run the same queries against the converted partition and compare bytes scanned. Measured, not estimated.
5. Price the write side. Conversion is not free. Glue jobs, EMR time, and Lambda invocations all cost money, and if a dataset is queried twice a month the conversion may never pay back.
6. Check who else reads the files. One downstream consumer that only speaks CSV turns a clean migration into a dual-write pipeline.

For step three, DuckDB is the fastest way to inspect a file without standing up a cluster. It runs happily on a laptop or a small VPS from a provider like Contabo or InterServer, which is usually all the compute this kind of analysis needs:

-- one row per column chunk: row groups, sizes, codec
SELECT row_group_id, row_group_num_rows, row_group_bytes,
       path_in_schema, compression
FROM parquet_metadata('events.parquet')
LIMIT 20;

-- convert a sample and control the row group size
COPY (SELECT * FROM 'events.json')
TO 'events.parquet'
(FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 1000000);

The parquet_metadata function reads the footer and returns a row per column chunk, so you can see straight away whether your writer produced one row group or two hundred, and which codec it used. If you would rather stay in Python, pyarrow.parquet.ParquetFile exposes the same footer through its metadata attribute, including per-column statistics and whether a bloom filter offset is present.

If the data already lives in S3 and you want the engine you are already paying for to do the conversion, Athena’s CTAS handles it in one statement. The AWS documentation lists GZIP and SNAPPY as the compression options for Parquet output here, with GZIP as the default:

CREATE TABLE events_parquet
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://your-bucket/events-parquet/',
  partitioned_by = ARRAY['event_date']
) AS
SELECT event_id, user_id, event_type, amount, event_date
FROM events_json;

Three things to know before running it. Partition columns have to come last in the SELECT list. CTAS refuses to write into a location that already contains data, so a re-run needs the prefix cleared first. And Athena has a write limit of 100 partitions per CTAS statement, which means a large backfill has to be chunked rather than fired off in one go.

How I would decide

• Keep the raw landing zone in whatever arrives. Usually JSON. Do not convert on ingest and discard the original; you will want it the first time a schema assumption turns out to be wrong.
• Convert once, at the point where data becomes queryable. Partitioned Parquet for anything a dashboard or an analyst touches repeatedly.
• Partition before you optimise the format. If there is budget for exactly one change, partition on the column your queries actually filter on.
• Choose the codec by read frequency. Zstd where scanned bytes are the bill and the data is read often, Snappy where decompression speed matters more, gzip only when a consumer demands it.
• Leave small CSVs alone. Reference tables, seed data, config. Converting a 4 MB lookup file to Parquet is busywork.
• Watch the scan bill, not the storage bill. Storage is cents. Scans are dollars. Athena workgroups can cap bytes scanned per query, and that limit is worth setting before someone discovers what a full table scan costs.

Frequently asked questions

Is Parquet always cheaper than CSV?

On a bytes-scanned billing model, for analytical queries that touch a subset of columns, almost always. It is not cheaper for point lookups, for very small datasets, for row-level updates, or when the query selects every column and filters on nothing. It also costs compute to produce, which matters when the data is rarely read.

How much smaller is Parquet than CSV in practice?

It depends entirely on cardinality, and anyone quoting a single ratio is guessing about your data. In the DuckDB TPC-H comparison the Parquet file came out around five times smaller than the CSV. Columns with few distinct values do far better than that; UUID and free-text columns do far worse. Measure on one partition of your own data instead of trusting a headline number.

Does compressing my JSON or CSV give me the same savings?

Partially. On Athena you are billed for bytes scanned before decompression, so compression does reduce the bill. What it cannot do is let the engine skip columns or row groups, and if you reach for gzip you lose splittability, which can make queries slower even as they get cheaper. Compression and columnar layout solve different halves of the problem.

Should I use Parquet or ORC?

For most teams that difference is much smaller than the difference between either of them and CSV. Parquet has wider support across engines and languages, which is usually the deciding factor. ORC has a long history in Hive-centric stacks and remains a reasonable choice if that is where you already live. Both are splittable and both support predicate pushdown.

Why did my costs not drop after converting to Parquet?

The usual suspects, in the order I would check them: the table is not partitioned, the queries select all columns, the files are too small, or the writer produced one enormous row group. Pull DataScannedInBytes for the same query before and after. If the number barely moved, the engine was not able to skip anything, and the problem is the query or the layout rather than the format.

Is JSON still worth keeping now that Parquet has a Variant type?

As a raw landing format and a transport format, yes. As the thing analysts query directly, the case is weaker than it used to be. Variant support is still spreading across engines, so confirm your query engine handles it before designing around it rather than assuming it is available everywhere.

What file size should I aim for?

Roughly 128 MB to 1 GB per file is the widely used guidance, with row groups sized so a file holds several of them rather than one. Below that range you pay per-file overhead on every query; above it you can starve parallelism if the row group layout is wrong.

The one thing worth remembering

The Parquet vs JSON vs CSV question is not really about the formats. It is about how much of your data the engine is permitted to ignore. Parquet hands an engine the most opportunities to skip work, JSON gives it almost none, and CSV sits between the two while making every reader re-guess the schema.

An opportunity to skip work is not the same as skipping it, though. If the query asks for every column, if the table is not partitioned, if the files are too small or the row groups too large, you have paid the conversion cost and bought a smaller file with the same bill attached. Measure bytes scanned before and after on a single partition. That one number tells you the truth in an afternoon, and it beats any comparison table you will read, including this one.

Need a second pair of eyes on your data lake costs?

Most format work starts as a cost problem and ends as a layout problem. Things I can help with:

• Auditing which queries actually drive your Athena or BigQuery bill, and whether a format change would move them at all
• Designing the partition scheme and file sizing before the conversion, so the migration is worth running
• Building the CSV or JSON to partitioned Parquet conversion as a Glue, Spark, or DuckDB job, including compaction for small-file pipelines
• Diagnosing conversions that did not pay off: row group layout, dictionary encoding, splittability, gzip traps
• Setting scan limits, workgroup controls, and alerting so one bad query cannot produce a surprise invoice
• Retrofitting Iceberg or Delta Lake where the real requirement is row-level updates rather than a different file format

If you want a concrete opinion rather than a general one, send me a query’s execution statistics along with the table schema and a listing of one partition, and I will tell you where your bytes are going.

The post Parquet vs JSON vs CSV: Where the Money Actually Goes appeared first on John Nessime.

That’s the shape of the problem. When you backfill historical API data into S3, the failure to worry about isn’t the one that throws. It’s the one that writes fewer records than it should and reports success anyway. Bytes transferred look plausible. Nothing goes red. The data is quietly wrong, and you find out when someone downstream trusts it.

This post covers the failure families that produce that outcome and the design decisions that make them impossible rather than merely unlikely: pagination that drifts under a mutating source, retries that write the same page twice under different keys, a prefix layout chosen while thinking about writes instead of reads, the seam where the backfill meets the live pipeline, and the storage class trap that makes a cheap-looking archive expensive to fix.

Why a backfill that finishes cleanly can still be wrong

A live pipeline has a natural error signal. Data stops arriving, a lag metric climbs, someone notices. A backfill has none of that. It runs once, against a range nobody is watching in real time, and then it’s over. The only evidence it leaves behind is a set of objects in a bucket.

So the question you need to answer months later isn’t “did the job succeed?” It’s “is this window complete, and how do I know?” If your design can’t answer that from artifacts on disk, you don’t have a backfill. You have a large amount of data of unknown provenance.

Failure family 1: the source moved while you paginated

This one bites hardest and shows up least. Most REST APIs paginate historical queries with an offset or page number. You ask for page 1, page 2, page 3. Meanwhile the underlying table is still accepting writes.

If a record is inserted into the middle of the result set between your page 4 and page 5 requests, everything after it shifts down by one. The row that would have started page 5 is now the last row of page 4, which you already fetched. You request page 5 and that record is gone. Not errored. Gone. Insertions cause skips, deletions cause duplicates, and the job still exits zero.

What to do instead

• Prefer keyset or cursor pagination. If the API offers an opaque cursor or a “records after ID X” parameter, use it. The read position is anchored to a value rather than a count, so inserts elsewhere in the set can’t shift it.
• Sort by an immutable column. If you’re stuck with offsets, sort by something that never changes after insert, usually the primary key or the creation timestamp. Sorting by a last-modified column while records are being modified is the worst case: rows physically move within the ordering as you walk it.
• Bound every request on both ends. Don’t ask for “everything before now.” Ask for a closed interval with an upper bound already in the past. A closed window can’t grow while you read it.
• Know which timestamp the filter applies to. Filtering on created-at gives you records that came into existence in that window. Filtering on updated-at gives you records touched in it, which for a mutable source is a different set entirely.

That last point causes arguments. Backfill on created-at, then run incremental syncs on updated-at, and your historical range never picks up edits made to old records. That may be correct for your use case. It is not correct by default, and it’s worth writing down which one you chose.

Failure family 2: the retry that wrote the same page twice

Backfills retry. They have to, because you’re hammering an API for hours and something will time out. The problem is what the retry writes.

If your object key contains a UUID, a run identifier or a wall-clock timestamp, a retried page produces a second object with a different key and the same records inside. You now have duplicates that listing will never reveal, because both objects are legitimately present and neither is obviously a copy. Derive the key from the request instead of the run, so the same window plus the same cursor always produces the same key and a retry overwrites rather than accumulates.

import hashlib
import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

s3 = boto3.client(
    "s3",
    config=Config(
        retries={"total_max_attempts": 8, "mode": "standard"},
        max_pool_connections=32,
    ),
)

def page_key(dataset, window, cursor):
    # Deterministic: the same page always maps to the same key.
    token = hashlib.sha256(cursor.encode()).hexdigest()[:16]
    return f"raw/{dataset}/dt={window}/page-{token}.json.gz"

def put_once(bucket, key, body):
    try:
        s3.put_object(Bucket=bucket, Key=key, Body=body, IfNoneMatch="*")
        return "written"
    except ClientError as err:
        if err.response["Error"]["Code"] == "PreconditionFailed":
            return "already_present"
        raise

Two things in there are worth explaining rather than copying.

IfNoneMatch="*" is a conditional write. S3 accepts the PUT only if no object exists at that key and returns a 412 PreconditionFailed otherwise. It’s supported on PutObject and CompleteMultipartUpload. That turns “did I already write this page?” into a single request with no read-before-write race. If two workers hit the same key concurrently, S3 can instead return a 409 ConditionalRequestConflict, which AWS documents as retryable, so handle the two codes differently: retry on 409, skip on 412.

The same thing from the CLI, useful for testing the behaviour by hand:

aws s3api put-object 
  --bucket your-bucket 
  --key raw/orders/dt=WINDOW/page-abc123.json.gz 
  --body page.json.gz 
  --if-none-match "*"

The retries block matters for a reason that trips people up. In botocore, max_attempts means different things depending on where you set it: in the AWS config file or the environment variable it counts total requests including the first, but in a Config object it counts retries on top of the first. total_max_attempts exists only in a Config object and always means total, so using it removes the ambiguity. There’s also an adaptive mode that layers client-side rate limiting on top of standard; AWS still labels it experimental, so I reach for standard plus my own concurrency cap first.

Failure family 3: a layout designed for writing, not reading

The prefix structure you pick during the backfill is the one you’re stuck with. Reorganising later means rewriting every object, and copy requests aren’t free.

Partition on event time, not ingestion time

The tempting layout is the date you ran the job. It’s easy, it’s monotonic, and it’s useless. Every query anyone writes against historical data filters on when the thing happened, not when you happened to fetch it. If your partitions are ingestion dates, every one of those queries becomes a full scan. Use Hive-style keys so query engines prune without extra configuration:

s3://your-bucket/raw/orders/dt=YYYY-MM-DD/page-<token>.json.gz
s3://your-bucket/curated/orders/dt=YYYY-MM-DD/part-<n>.parquet

Keep the raw landing zone and the curated zone separate. Raw is whatever the API gave you, unmodified, so you can re-derive everything if your parsing logic turns out to be wrong. Curated is columnar and typed. Merging them saves nothing and costs you the ability to reprocess.

Don’t let the write path throttle itself

S3 documents at least 3,500 PUT/COPY/POST/DELETE and 5,500 GET/HEAD requests per second per partitioned prefix, with no limit on the number of prefixes. The word doing the work is partitioned. That rate applies once S3 has split the keyspace out for that prefix, and AWS is explicit that the scaling is gradual rather than instant. During the ramp you can see 503 SlowDown responses even when your sustained rate looks well inside the documented ceiling.

For a backfill this is mostly good news, since date-partitioned keys spread writes across many prefixes as you work through the range. It becomes a problem when you parallelise within a single day and drive thousands of writes per second into one dt= prefix. Two mitigations: back off with jitter instead of retrying 503s immediately, and shard workers across windows rather than within them.

Register partitions without a scan

A backfill can create thousands of partitions at once. MSCK REPAIR TABLE handles that badly. AWS documents that it only adds partitions and never removes them, that it can take a long time on large tables, and that it can time out leaving the catalog partially updated. Partial success with no error is exactly the failure mode this whole post is about.

Partition projection sidesteps the catalog. Athena computes the partition list from table properties instead of looking it up, so a backfilled range is queryable the moment the objects exist.

TBLPROPERTIES (
  'projection.enabled'          = 'true',
  'projection.dt.type'          = 'date',
  'projection.dt.format'        = 'yyyy-MM-dd',
  'projection.dt.range'         = 'START_DATE,NOW',
  'projection.dt.interval'      = '1',
  'projection.dt.interval.unit' = 'DAYS',
  'storage.location.template'   = 's3://your-bucket/curated/orders/dt=${dt}/'
)

Replace START_DATE with the earliest date in your backfill. The trade-off is worth stating plainly: projection generates every partition in the range whether or not data exists there, so an over-wide range on a sparse dataset wastes planning effort. And if the S3 layout ever stops matching storage.location.template, queries return nothing rather than erroring. Set the range to your actual data, not to a comfortable margin.

Failure family 4: the seam between backfill and live pipeline

Two processes write to the same table: a backfill walking backwards through history, and a live ingest walking forwards. They meet somewhere, and that meeting point is where records go missing.

The instinct is to make the handoff exact. Backfill up to timestamp T, start streaming from T. Don’t. Clock skew between your runner and the API, records that arrive late at the source, and rounding in the API’s own filtering will all conspire to open a hole at exactly T.

Overlap deliberately instead. Run the backfill past the point where live ingest started, by a margin generously larger than any plausible source lag, then deduplicate on a natural key during curation. Duplicates you can remove with a query. Gaps you can only fix by going back to the API, and by then the retention window may have closed.

Failure family 5: the storage class you can’t undo

Historical data feels like archive data, so the instinct is to write it straight into a cold class or transition it immediately. Read the mechanics first, because the billing rules for cold classes are structural and pro-rated against you.

• S3 Standard-IA and One Zone-IA carry a 30-day minimum storage duration and a 128 KB minimum billable object size.
• Glacier Instant Retrieval and Glacier Flexible Retrieval carry a 90-day minimum duration; Deep Archive carries 180 days. Delete, overwrite or transition earlier and you’re charged the remainder.
• Objects in Glacier Flexible Retrieval and Deep Archive are billed for 40 KB of extra per-object metadata on top of the object itself.

Read those together and the conclusion is uncomfortable. A backfill that writes millions of small JSON pages into an archive class pays a size floor on every one of them, plus metadata overhead, plus a minimum duration that makes fixing the mistake expensive. Those pages are also the objects you’re most likely to want to reprocess, which is exactly the overwrite that triggers the early-deletion charge.

What I’d actually do: land raw pages in Standard, compact them into larger Parquet files in the curated zone, verify completeness, and only then apply a lifecycle rule to the raw prefix. Compaction is what makes cold storage sane, because it turns a size floor per tiny object into a rounding error on a large one. It also fixes the small-file problem in Athena, which is a separate cost line.

One more cost note: if your backfill runs from EC2 in a private subnet, every byte pulled from the API and pushed to S3 may route through a NAT gateway with per-GB processing charges. A gateway VPC endpoint for S3 removes the S3 half of that. It’s a five-minute change people usually discover after the bill.

Proving the backfill is complete

Here’s the part most backfills skip, and the part that makes everything above verifiable rather than hopeful. Write a manifest object per window, alongside the data, describing what that window contains.

{
  "dataset": "orders",
  "window": "<dt value>",
  "filter_field": "created_at",
  "range_start_inclusive": "<iso8601>",
  "range_end_exclusive": "<iso8601>",
  "pages_written": "<int>",
  "records_written": "<int>",
  "source_reported_total": "<int or null>",
  "last_cursor": "<opaque token>",
  "schema_fingerprint": "<sha256 of sorted field names>"
}

The manifest turns completeness into something you can query. A window with no manifest was never finished. A window whose records_written disagrees with source_reported_total is suspect. A window whose schema_fingerprint differs from its neighbours tells you the API changed shape partway through your range, which is otherwise close to undetectable.

Write the manifest last, after every page for that window is confirmed in S3. A manifest written optimistically at the start is worse than no manifest at all, because it asserts a completeness nobody checked.

Troubleshooting a backfill that already ran

1. Count objects per window and look at the shape. Use S3 Inventory rather than a live ListObjectsV2 walk on a large bucket. Plot objects per dt= partition. Real data has a rhythm, weekday peaks and weekend troughs. A window that breaks the rhythm without a business reason is your gap.
2. Reconcile record counts against the source. Many APIs return a total count on the first page of a filtered query. Pull that for a sample of windows and compare with what’s in S3. If you kept manifests, this is one query.
3. Check for duplicate natural keys within a partition. A GROUP BY on the record ID with HAVING COUNT(*) > 1 finds retry duplicates immediately. Duplicates clustered at window boundaries mean pagination drift; duplicates spread evenly mean your keys weren’t deterministic.
4. Look at boundary hours, not whole days. Cursor drift and clock skew concentrate at the edges of each request window. Aggregate to the hour and the missing slice usually becomes obvious even when daily totals look fine.
5. Verify the layout still matches the table definition. If you’re using partition projection and a query returns zero rows, the first suspect is storage.location.template not matching the actual prefix, trailing slashes included.

Common mistakes

• Treating an exit code of zero as evidence of completeness.
• Putting a run ID or timestamp in the object key, which makes retries additive instead of idempotent.
• Partitioning by ingestion date because it’s what the job naturally knows.
• Making the backfill and live-ingest boundary exact instead of overlapping.
• Writing millions of small objects straight into an archive class and meeting the minimum object size and duration rules afterwards.
• Running MSCK REPAIR TABLE over thousands of new partitions and trusting that it completed.
• Backfilling on a last-modified field while assuming you captured creation events.
• Sharding workers within a single day, concentrating writes on one prefix and inviting 503s.

Best practices when you backfill historical API data into S3

• Make every window independently re-runnable. If re-running one day is routine rather than scary, you’ll actually fix problems when you find them.
• Derive object keys deterministically from window plus cursor, and use conditional writes so a retry is a no-op.
• Bound every API request on both ends, and never let the upper bound be “now”.
• Write raw responses untouched before parsing anything. Parsing bugs are common; re-fetching a closed API window often isn’t possible.
• Emit a manifest per window, written last, and reconcile it against source counts.
• Compact to Parquet before applying any lifecycle transition.
• Run the job somewhere it can survive for hours. A long backfill on a laptop over hotel wifi is a bad plan; a small dedicated VPS from a provider like Contabo or InterServer, or a spot-priced EC2 instance, costs little and won’t sleep halfway through.
• Watch progress somewhere other than the terminal. Pushing records-per-window into CloudWatch or Grafana gives you a curve you can eyeball, and an unusually flat stretch is a gap you can catch while the source data is still retrievable.

FAQ

Should I write raw JSON or convert to Parquet during the backfill?

Both, in two zones. Write the raw response body first so the fetch step has no dependency on your schema assumptions, then convert as a separate job. Converting inline couples an expensive, rate-limited API walk to a parsing step that will eventually throw on an unexpected field, and you don’t want to re-fetch years of data because of a type error.

How large should each object be?

In the raw zone, one object per API page is fine and keeps idempotency simple. In the curated zone, aim for files in the tens to low hundreds of megabytes. Query engines pay a fixed overhead per file, so thousands of tiny Parquet files scan slower and cost more than a few large ones holding identical data.

Can I run the backfill in parallel across date windows?

Yes, and that’s the right axis, because each window is independent and lands on its own prefix. The binding constraint is almost always the source API’s rate limit rather than S3. Add a global concurrency cap and honour any documented limit or Retry-After header. Getting your credentials throttled or suspended mid-backfill is a worse outcome than finishing slowly.

Do I need AWS Glue or Airflow for this?

Not to start. A single script that takes a window as an argument, writes pages and a manifest, and is safe to re-run covers most backfills. Orchestration earns its place when you need retries across hundreds of windows, dependency ordering, or a visible run history. Glue and Step Functions fit AWS-native stacks; Airflow gives you more control and more to operate. Pick after the script works, not before.

What if the API caps how far back I can query?

Then the retention boundary is your real deadline and it should drive priority. Fetch the oldest reachable data first, not the most recent, because the newest data is still available tomorrow and the oldest may not be. If the vendor offers a bulk export or reporting endpoint separate from the standard API, it’s usually faster and less rate-limited than paginating.

Does any of this change with S3-compatible storage instead of AWS?

The layout, idempotency and manifest patterns carry over unchanged to Cloudflare R2, Backblaze B2, MinIO and similar. What varies is API-level feature support. Conditional writes with If-None-Match, request-rate behaviour and storage class semantics differ by provider, so verify the specific behaviour you’re depending on against that provider’s documentation rather than assuming parity with S3.

How do I know the backfill is actually finished?

When every expected window has a manifest, every manifest’s record count reconciles against the source, and a duplicate check on natural keys returns nothing. Not when the script exits.

The one thing worth remembering

A backfill isn’t finished when the job exits. It’s finished when you can prove the range is complete without re-reading the source. Deterministic keys, conditional writes, closed windows, manifests and deliberate overlap all exist to make that proof cheap.

If you’re about to backfill historical API data into S3, spend the extra hour on the key derivation and the manifest before you start the run. The alternative is learning about a gap from a chart, months later, with the source retention window already closed behind you.

Need a hand with a backfill?

This is a large part of what I do. If you’re staring at a historical range and an API with a rate limit, I can help with:

• Designing an idempotent, re-runnable backfill job with deterministic keys and per-window manifests, so a failed day is a one-command fix.
• Choosing a partition layout that serves your actual query patterns, and setting up Athena or Glue tables with partition projection so thousands of backfilled partitions become queryable without catalog scans.
• Auditing a backfill that already ran, reconciling record counts against the source and finding the gaps and duplicates before your dashboards do.
• Handling awkward source APIs: offset-only pagination, aggressive rate limits, undocumented result caps, schema drift partway through a historical range.
• Compaction and lifecycle design, so cold storage actually saves money instead of hitting minimum object size and minimum duration charges.
• Wiring the seam between the backfill and your live pipeline so there’s no gap at the handover, plus the monitoring that would have caught it.

Send me something concrete: a sample API response, your current key layout, or the object counts per partition. That’s usually enough to spot where the gap is.

The post Backfilling Historical API Data into S3 Without Silent Gaps appeared first on John Nessime.