That is the failure mode that makes this comparison worth writing about. The Parquet vs JSON vs CSV question almost never shows up as a performance complaint first. It shows up as a bill, weeks later, from a decision somebody made in an afternoon because CSV was easy to write.

This post breaks down where the money and the milliseconds actually go: what each format costs you on read, on write, and on the request path in between. Then a procedure for measuring it on your own data instead of trusting somebody else’s benchmark, and an honest look at where CSV and JSON still beat the columnar option.

The billing model is the whole story

Most serverless query engines bill on bytes read, not bytes returned. That one sentence explains almost every surprise bill I have seen on a data lake.

Amazon Athena charges for bytes scanned, rounded up to the nearest megabyte, with a 10 MB minimum per query. DDL statements are free. Cancelled queries are charged for whatever was scanned before the cancel landed. BigQuery’s on-demand model works the same way in spirit: charges round up to the nearest MB, with a 10 MB minimum per table referenced and a 10 MB minimum per query, and you are billed on the columns you select even when you put a LIMIT on the result.

That last part catches people. LIMIT 10 does not make a query cheap; the engine still reads enough to know which ten rows to give you, and on a row-based format that means reading whole rows.

So the question is not “which format is smaller on disk.” It is “how few bytes can the engine get away with reading.” Those diverge fast.

Four levers, not one

When people compare formats they usually compare compressed file sizes and stop. There are four levers that move real cost, and file size is the weakest of them.

1. Column pruning

Row formats store a whole record contiguously. To get one field you read the record. Columnar formats store each column separately, so an engine reads only the column chunks your query names. Query two columns out of forty and you skip most of the file. This is the single biggest lever and it only exists in columnar formats.

2. Block skipping

Parquet splits data into row groups, typically sized in the tens to hundreds of megabytes, and stores per-column statistics in the file footer: min, max, null count. An engine reads the footer first and skips any row group whose min/max range cannot possibly match your filter. Newer Parquet writers also emit page-level indexes and optional Bloom filters, which push the same idea down to a finer granularity.

The catch: this only works if your data is sorted or clustered on the column you filter by. Write rows in random order and every row group’s min/max spans the whole range, so nothing gets skipped and you paid for the statistics for nothing. Sorting before write is what turns the feature on.

3. Splittability

This is the one that bites silently. A gzip stream cannot be split at an arbitrary byte offset, so a distributed reader cannot hand different chunks of one .gz file to different workers. One gzipped CSV file means one task, on one core, no matter how large your cluster is. Spark has carried a long-standing issue about exactly this confusion, and the standard workarounds are ugly: decompress first, split the file up beforehand, or recompress with something splittable like bzip2 and eat the CPU cost.

Parquet sidesteps this entirely. Compression is applied per page inside the file, not across the whole stream, so a Parquet file stays splittable at row-group boundaries regardless of which codec you chose. Snappy is the usual default; ZSTD is the common pick when you want a better ratio and can spend a bit more CPU.

If your job takes an hour and one executor is doing all the work while the rest idle, check the file extension before you check anything else.

4. Request overhead

Object storage bills per request as well as per byte, and listing is slow. Ten thousand tiny files cost more in GET and LIST calls than one properly sized file, and those charges never appear under your query engine’s line item. It hits Parquet users hardest, because each file carries footer metadata that must be read separately before any data is. A thousand 1 MB Parquet files can be slower and dearer than a hundred 10 MB ones holding the same rows. The format is not a substitute for file sizing.

CSV: still the right answer more often than people admit

Where it wins. Every tool on earth reads it. A human can open it, diff it, grep it, and eyeball it in a terminal. It appends trivially: concatenate two files and the result is still valid. For interchange with a client, a finance team, or a vendor whose ingest spec you do not control, CSV is the format that never generates a support ticket. And for datasets small enough that the 10 MB query minimum dominates anyway, the whole debate is theatre.

Where it doesn’t. No types, no schema, no statistics. Every value is text until something guesses otherwise, and the guesses are where pipelines break: leading zeros stripped off ZIP codes, dates parsed in the wrong locale, a free-text field with an embedded comma. There is no column pruning, so a two-column query pays for a forty-column file. And gzipped CSV, the most common shape in the wild, is the splittability trap above.

CSV is an excellent transport format and a poor storage format. That distinction is the useful one.

JSON: the most expensive way to store a table

Where it wins. Nested and irregular data. If records genuinely differ, if fields come and go, if there are arrays of objects inside objects, JSON models that honestly and CSV cannot. It is self-describing per record, so a consumer that has never seen your schema can still make sense of a line. Newline-delimited JSON also appends cleanly, which is why it dominates log shipping and webhook capture. Most event pipelines are right to land raw JSON first.

Where it doesn’t. Every record repeats every key name. Store ten million events with a field called customer_identifier and you have stored that string ten million times. Compression claws a lot of that back, but the engine still pays to decompress and parse it, and parsing JSON costs meaningfully more per byte than splitting on a comma. No column pruning, no statistics, no skipping.

JSON as a landing format is fine. JSON as the format your analysts query every day is a recurring charge you are choosing to pay.

Parquet: cheap reads, real obligations

Where it wins. Everything above. Column pruning, footer statistics, per-column encoding chosen to suit the data (dictionary encoding for low-cardinality strings, run-length and delta encoding for sorted integers), and splittability regardless of codec. Compressing a string column and an integer column separately beats compressing rows that mix both. On a wide table with selective queries, the reduction in bytes scanned is not marginal.

Where it doesn’t. This is the part the enthusiastic posts skip.

• You cannot append. Adding rows means rewriting a file or adding another one. Streaming small batches straight to Parquet is how you end up with the small-files problem, which is why table formats like Apache Iceberg exist on top of it.
• Schema mismatches across files are a runtime error, not a warning. Write one file with a column as INT32 and another as a string, point a reader at the directory, and the job fails on type conversion. Debugging means finding which file in a prefix is the odd one out.
• Timestamps are a compatibility minefield. The old INT96 physical type carried nanosecond timestamps for historical reasons, is deprecated, and is still what some writers emit for compatibility with older engines. Readers expecting the INT64 logical timestamp types will refuse it. If timestamps arrive as garbage, check this first.
• It is opaque. You cannot grep it, tail it, or read it without tooling, and that costs real debugging time.
• Writing costs more. Encoding, statistics, and compression are work. If you write far more than you read, that trade may not pay off.

None of these are reasons to avoid Parquet for analytics. They are reasons to stop treating the conversion as free.

Measure it on your data, in about twenty minutes

Published benchmarks tell you about somebody else’s data. Yours has different cardinality, different width, different query patterns. Here is the procedure I would run before committing to a conversion.

1. Get a real byte count for what you have now. Not an estimate, the actual prefix.

aws s3 ls s3://your-bucket/raw/events/ --recursive --summarize --human-readable

The summary at the end gives you total objects and total size. Object count matters as much as size here, because it tells you whether you have a small-files problem before you add to it.

2. Convert a representative slice, not the whole thing. One partition is enough. In Athena, a CTAS statement writes the output for you.

CREATE TABLE events_parquet_sample
WITH (
  format = 'PARQUET',
  write_compression = 'ZSTD',
  external_location = 's3://your-bucket/experiments/events-parquet/'
) AS
SELECT * FROM events_csv WHERE dt = '2020-01-01';

Pick a date that exists in your data. Athena can read Parquet compressed with LZ4 or LZO but will not write those codecs, so stick to Snappy, GZIP or ZSTD on the write path.

3. Run your three most common queries against both tables and compare bytes scanned. Athena reports this per query in the console and in the query execution statistics. This number, not the wall-clock time, is your bill.
4. On BigQuery, price it before you run it. A dry run costs nothing and returns the byte estimate.

bq query --use_legacy_sql=false --dry_run 
  'SELECT user_id, amount FROM `project.dataset.events` WHERE dt = "2020-01-01"'

5. Inspect the Parquet file you just wrote. Check row group count and sizes, because a file full of tiny row groups will not perform the way the marketing says.

python3 -c "
import pyarrow.parquet as pq
f = pq.ParquetFile('part-0000.parquet')
print(f.metadata)
print(f.metadata.row_group(0))
"

That prints the number of row groups, rows per group, and compressed and uncompressed sizes. If you see dozens of row groups in a small file, your writer is flushing per batch and you need to buffer more before writing.

6. Put a guardrail up before anyone else touches it. Athena workgroups support a per-query data scan limit that cancels a query which exceeds it. You still pay for what was scanned before the cancel, but it turns an unbounded mistake into a bounded one.

Running the comparison locally works too. DuckDB reads CSV, JSON and Parquet directly from a path, and a laptop or a small VPS from a provider like Contabo or InterServer will happily handle datasets that people reflexively push into a cluster.

Arguments that don’t survive contact

• “We gzip our CSVs, so we already save the money.” Compression reduces bytes scanned, and that is real. It does not give you column pruning, and it costs you parallelism. You have optimised one lever and broken another.
• “Parquet is always faster.” Not for full-table scans of narrow tables, not for tiny files where footer reads dominate, and not on the write path. Selectivity and table width are what make it win.
• “We’ll just convert everything to Parquet.” Converting a landing zone that receives appended events all day means fighting the format’s biggest weakness. Land raw, convert on a schedule, query the converted copy.
• “Storage is cheap so format doesn’t matter.” Storage is cheap. Scanning is not, and scanning is billed per query, repeatedly, forever. The recurring cost is the one to model.
• “Our BI tool caches, so nobody re-scans.” Until the cache misses, someone changes a filter, or a scheduled refresh runs hourly against a table nobody remembers owning.

How I’d decide

Not a scorecard. A short sequence of questions, in order.

• Is this data leaving your organisation? CSV, unless the recipient asked for something else. Interoperability beats efficiency at a boundary you do not control.
• Is it arriving continuously, one record at a time? Newline-delimited JSON or CSV into a landing prefix. Do not stream into Parquet.
• Is it read more than once, by more than one person, with filters? Parquet, partitioned on whatever people filter by most, sorted within partitions on the next most common filter.
• Is the whole dataset smaller than a few hundred megabytes? Use whatever is convenient. At that size the per-query minimums and human time dominate the byte costs.
• Are you fighting append semantics, concurrent writers, or schema evolution? You have outgrown bare files. That is the problem a table format like Iceberg solves, and it will still store its data as Parquet underneath.

The pattern most teams land on is not one format. It is raw JSON or CSV in a landing zone with a short lifecycle policy, Parquet in a curated zone that analysts actually query, and a scheduled job in between. Storage on the cheap tier, scanning on the cheap format.

Frequently asked questions

How much smaller is Parquet than CSV?

It depends entirely on your data, and anyone quoting a single multiplier is guessing about yours. Wide tables with repetitive low-cardinality strings compress dramatically because dictionary encoding does most of the work. Narrow tables of high-entropy values barely improve. Convert one partition and measure it; that takes minutes and gives you a number you can defend.

Does Parquet vs JSON vs CSV matter if I use BigQuery native tables?

Less than you would think, because once data is loaded into a BigQuery native table it lives in Google’s own columnar storage and on-demand billing is calculated from the logical size of the columns you select, based on their data types, rather than from whatever format you loaded from. The format question applies to your source files and to external tables over Cloud Storage. Batch loading itself is not charged.

Is ORC better than Parquet?

They are close enough that ecosystem support should decide it for you. ORC carries statistics at file, stripe and row-group levels; Parquet’s page indexes and Bloom filters have narrowed that gap. ORC has deeper roots in the Hive world, Parquet has broader support across Spark, Trino, DuckDB, Polars and the Python data stack. Pick the one your engines read best and stop optimising the choice.

Why did my query cost the same after I converted to Parquet?

Usually one of three things. You are running SELECT *, which reads every column and throws away the main advantage. Your data is unsorted, so no row group can be skipped. Or the table is not partitioned, so the engine still lists and touches everything. Parquet gives the engine the ability to skip work; your query and layout have to give it a reason to.

Should I use Snappy, GZIP or ZSTD for Parquet?

Snappy is the common default and favours speed over ratio. ZSTD gives a better ratio at moderate CPU cost and is a sensible pick for data you keep a long time and read occasionally. GZIP compresses well but decompresses slowly. Because Parquet compresses at page level, none of these break splittability, so this really is just a CPU-versus-bytes trade.

Can I query CSV and Parquet in the same table?

Not within one table definition, since a table declares a single input format. What you can do is keep them as separate tables over separate prefixes and union them in a view during a migration, then drop the CSV side once the backfill is verified. Mixing compression codecs within a single format is fine; Athena can read a Parquet table where some files are Snappy and others GZIP.

What about the cost of the conversion job itself?

Real, and worth modelling. A conversion reads all the data once and writes it again, whether through Athena CTAS, a Glue job, or a script on a box you own. If a table is queried a handful of times a month, the conversion may never pay for itself. If it backs a dashboard that refreshes hourly, it pays for itself almost immediately. Divide the one-off cost by the per-query saving and you have your break-even in queries.

The one thing worth remembering

The Parquet vs JSON vs CSV decision is not about disk space. It is about how many bytes an engine is forced to read to answer a question, how many workers it can put on the job, and how many requests it makes getting there. CSV and JSON force the engine to read everything. Parquet lets it read almost nothing, provided you sorted, partitioned and sized the files so it can.

Land raw in whatever arrives. Curate to Parquet on a schedule. Point people at the curated copy. Then go and look at your actual bytes-scanned numbers, because that is where the argument gets settled.

Need help cutting a data lake bill?

Most of the savings on a lake are unglamorous: layout, file sizes, and a handful of queries nobody has looked at since they were written. Things I can help with:

• Auditing which queries and tables drive your bytes-scanned spend, and which conversions would actually pay for themselves
• Designing partition and sort layouts around your real filter patterns instead of a default date partition nobody uses
• Building the raw-to-curated conversion pipeline, whether that is Athena CTAS, a Glue job, or something running on a box you control
• Fixing small-files and row-group sizing problems that quietly cap your read throughput and inflate request charges
• Diagnosing Parquet schema and timestamp compatibility failures across engines that disagree about types
• Setting up scan limits, cost alerting, and guardrails so the next surprise bill is caught in hours rather than at month end

If you want a concrete second opinion, send me a query execution history, a sample file, or the output of a bytes-scanned report and I will tell you where the money is going.

The post Parquet vs JSON vs CSV: What Each Format Actually Costs You appeared first on John Nessime.

That is the failure mode worth understanding, and it stays invisible until you go looking for it. The conversion was real. The files did get much smaller. But the dashboard queries were still selecting every column from an unpartitioned table, so the engine still had to open every column chunk in every file. Columnar storage only pays out when the query gives the reader something to skip.

This is a working comparison of Parquet vs JSON vs CSV on the two things that end up on an invoice: bytes read, and time spent reading them. Each format gets a profile with the cases where it wins and the cases where it does not, followed by the places where the promised savings quietly evaporate and a procedure for modelling the change on your own data before you spend an engineering week on it.

Your billing model decides how much the format matters

Before comparing formats, work out what you are actually charged for. The same conversion produces very different savings depending on which of these you are on.

• Bytes scanned from object storage. Amazon Athena is the clearest case. You are billed for the bytes a query reads out of S3, rounded up, with a minimum charge per query. Here a format change converts almost directly into money.
• Logical bytes processed. BigQuery’s on-demand model bills on the uncompressed logical size of the columns a query references in a native table, not on the compressed size sitting on disk. Compressing harder does not lower that bill. Touching fewer columns does.
• Compute time. Snowflake, Databricks, EMR, and anything you run on your own hardware charge you for how long a machine is awake. The format shows up as wall clock time, which is a real cost but a much softer one.

The BigQuery detail catches people out. Google documents that on-demand cost is calculated from logical, uncompressed bytes for native tables, but that when you query external data stored as Parquet or ORC, the bytes charged are limited to the columns BigQuery actually reads. The same file format therefore sits on opposite sides of the billing line depending on whether the data was loaded or is being referenced in place. Both Athena and BigQuery also apply a 10 MB minimum per query, and BigQuery applies it per table referenced, so a swarm of small queries against small tables costs far more than the data volume suggests.

CSV: cheap to produce, expensive to interrogate

CSV has no schema, no statistics, and no internal structure. Every value is text. The number 1000000000 takes ten bytes instead of the four it would take as a 32-bit integer, and because the layout is row-wise, there is nothing for a column-aware compressor to exploit.

Where CSV wins

• A human needs to open it. Nothing else comes close.
• Export from a legacy system that offers no other option.
• Small data. Below a few megabytes, the metadata and encoding overhead of a columnar file is not repaid.
• Streaming appends. You can write a line at a time and stop whenever you like.

Where CSV does not

Two problems, and the second is worse than the first.

The first is that the reader has to parse every byte, even the bytes it intends to throw away. The DuckDB team published a comparison on TPC-H data at scale factor 20, run on a laptop, reporting the median of five runs, and it is the most honest set of numbers I have seen on this. Loading the lineitem table took roughly 11.8 seconds from CSV against 5.2 seconds from Snappy-compressed Parquet, with the CSV file about five times larger on disk. That is a 2x gap on load, not the order of magnitude most people assume. Modern CSV readers are genuinely fast.

The gap opens when you query the files directly rather than loading them. On the same setup, TPC-H Q1 ran in about 6.7 seconds against CSV and 0.9 seconds against Parquet, and the join-heavy Q21 took roughly 20 seconds against 2.1. That difference is not parsing throughput. It is that the Parquet reader can skip row groups using statistics and skip unreferenced columns entirely, while the CSV reader cannot skip anything, because nothing in the file tells it what it would be skipping.

The second problem is dialect. There is an RFC for CSV and it is widely ignored. Quoting, escaping, embedded newlines, date formats, and whether NA means null or Namibia are conventions rather than rules, so every consumer re-guesses and sooner or later one guesses differently from the others. That failure does not raise an error. It produces wrong numbers in a report nobody questions.

JSON: you pay for the schema on every single row

Newline-delimited JSON is what most event pipelines emit, because a producer can add a field without coordinating with anyone downstream. That flexibility is worth something real. It is also why JSON is usually the most expensive format on a bytes-scanned bill: every record repeats every key name, so a field called customer_subscription_status carries those thirty-odd bytes on every row, forever.

Where JSON wins

• Landing zone for data whose shape you do not control. Accept it, keep the raw copy, argue about the schema later.
• Genuinely nested or sparse records, where flattening into columns would give you a table that is mostly nulls.
• Debugging. Being able to decompress one file and read a record with your eyes is worth a lot at three in the morning.
• API and webhook payloads, where it is simply what arrives.

Where JSON does not

As a query target at any serious volume. You are paying scan charges on repeated key names, on quote characters, and on the fact that every number is stored as text. Compressing the files helps storage and helps the scan bill on Athena, but it cannot give the engine anything to skip, and gzip brings a separate problem covered below.

The nested-data argument for keeping JSON has also weakened. Parquet now has a ratified Variant type for semi-structured data, with support arriving across Delta Lake, Iceberg, and Spark, plus a shredding scheme that pulls frequently accessed fields out into real columns while leaving the rest flexible. If you are keeping JSON purely because your records have unpredictable shape, that reason is worth revisiting rather than treating as settled.

Parquet: what the file is actually doing

Understanding the layout is what lets you predict whether a migration will pay for itself, so it is worth thirty seconds.

A Parquet file is divided into row groups. Inside each row group, every column is stored as its own contiguous column chunk, and each chunk is split into pages. The footer at the end of the file carries the schema and, per column chunk, statistics such as minimum, maximum, and null count. That structure is what makes the two headline optimisations possible:

• Column pruning. The reader consults the footer, works out the byte ranges of the three columns you asked for, and never issues a read for the other forty-seven.
• Predicate pushdown. If a row group’s maximum value for event_date falls below your filter’s lower bound, the entire row group is skipped without being read.

Two later additions refine this. The page index stores per-page minimum and maximum values in a structure near the footer, so a reader doing a selective scan can locate matching pages without walking every page header, which previously meant pulling most of the column off disk anyway. Bloom filters cover the case statistics cannot help with at all: a high-cardinality column such as a user ID, where every row group’s min and max span the whole value range and pruning by range is useless. A bloom filter can say that a value is definitely not present in a row group. DuckDB even exposes parquet_bloom_probe so you can check which row groups a given value would eliminate.

Encoding does the rest of the work. Because a column chunk holds values of a single type, dictionary and run-length encoding apply before any general-purpose compressor starts. Low-cardinality columns such as country codes or status flags compress extremely well. UUIDs and free text do not, and writers often disable dictionary encoding for them. This is why compression ratios quoted in blog posts are close to meaningless for your data: the ratio is a property of your cardinality, not of the format.

Where Parquet does not win

• Point lookups. Fetching one complete row by ID means touching every column chunk. A row format reads a single contiguous block instead.
• Row-level updates. Parquet files are immutable. Changing one row means rewriting the file, which is precisely the problem Iceberg and Delta Lake exist to solve on top of it.
• Streaming writes. The schema lives in the footer, so a writer has to finish the file before it is readable. Small-batch writers produce small files, which carries its own tax.
• Tiny datasets. Below a few megabytes, footer and row group overhead is not worth paying.
• Human inspection. It is binary. You need tooling, and someone will eventually ask you for a CSV export anyway.

Where the savings quietly disappear

Every one of these has cost someone a migration that looked excellent in the design document.

• Small files. Each file costs a metadata read, an object storage request, and reader setup. Thousands of 2 MB Parquet files will lose to a handful of large CSVs. The commonly cited target is roughly 128 MB to 1 GB per file, and you need a compaction step to get there if your writer emits micro-batches.
• One enormous row group. Parallelism in most engines is per row group. A file containing a single row group is processed by a single thread no matter how many cores you have. Check this rather than assuming the writer got it right.
• Select-everything queries in the reporting layer. Column pruning cannot help a query that asks for all columns. This is the one from the opening paragraph, and it is usually a BI tool doing it rather than a person.
• No partitioning. Format reduces bytes per row. Partitioning reduces how many rows are considered at all. The two multiply, and partitioning is normally the larger lever. Converting without partitioning leaves most of the money on the table.
• Gzipped text. AWS documents that Parquet and ORC are always splittable, because they compress sections independently and carry metadata pointing at those sections, while most text compression formats are not. Bzip2 and LZO are splittable; gzip is not. One 5 GB gzipped CSV cannot be divided among workers, so it is read start to finish by one reader. The compression made storage cheaper and the query slower.
• Uppercase file extensions. A small operational trap worth knowing: Athena determines the compression type of CSV and JSON data from the file extension, does not recognise uppercase extensions such as .GZ, and treats a file with no extension as uncompressed plain text.

How to model the change before you migrate

Do not migrate on faith. This takes an afternoon and answers the question properly.

1. Find out what you scan today. Every Athena query execution carries a DataScannedInBytes statistic. Pull it for a representative window and rank queries by bytes scanned. Almost always a handful of them are most of the bill, and those are the only ones worth optimising.
2. Convert one day of data, not all of it. A single partition is enough to measure with.
3. Look inside the resulting file before trusting it. Row group count, row group size, and compression codec decide whether the file behaves.
4. Re-run the same queries against the converted partition and compare bytes scanned. Measured, not estimated.
5. Price the write side. Conversion is not free. Glue jobs, EMR time, and Lambda invocations all cost money, and if a dataset is queried twice a month the conversion may never pay back.
6. Check who else reads the files. One downstream consumer that only speaks CSV turns a clean migration into a dual-write pipeline.

For step three, DuckDB is the fastest way to inspect a file without standing up a cluster. It runs happily on a laptop or a small VPS from a provider like Contabo or InterServer, which is usually all the compute this kind of analysis needs:

-- one row per column chunk: row groups, sizes, codec
SELECT row_group_id, row_group_num_rows, row_group_bytes,
       path_in_schema, compression
FROM parquet_metadata('events.parquet')
LIMIT 20;

-- convert a sample and control the row group size
COPY (SELECT * FROM 'events.json')
TO 'events.parquet'
(FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 1000000);

The parquet_metadata function reads the footer and returns a row per column chunk, so you can see straight away whether your writer produced one row group or two hundred, and which codec it used. If you would rather stay in Python, pyarrow.parquet.ParquetFile exposes the same footer through its metadata attribute, including per-column statistics and whether a bloom filter offset is present.

If the data already lives in S3 and you want the engine you are already paying for to do the conversion, Athena’s CTAS handles it in one statement. The AWS documentation lists GZIP and SNAPPY as the compression options for Parquet output here, with GZIP as the default:

CREATE TABLE events_parquet
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://your-bucket/events-parquet/',
  partitioned_by = ARRAY['event_date']
) AS
SELECT event_id, user_id, event_type, amount, event_date
FROM events_json;

Three things to know before running it. Partition columns have to come last in the SELECT list. CTAS refuses to write into a location that already contains data, so a re-run needs the prefix cleared first. And Athena has a write limit of 100 partitions per CTAS statement, which means a large backfill has to be chunked rather than fired off in one go.

How I would decide

• Keep the raw landing zone in whatever arrives. Usually JSON. Do not convert on ingest and discard the original; you will want it the first time a schema assumption turns out to be wrong.
• Convert once, at the point where data becomes queryable. Partitioned Parquet for anything a dashboard or an analyst touches repeatedly.
• Partition before you optimise the format. If there is budget for exactly one change, partition on the column your queries actually filter on.
• Choose the codec by read frequency. Zstd where scanned bytes are the bill and the data is read often, Snappy where decompression speed matters more, gzip only when a consumer demands it.
• Leave small CSVs alone. Reference tables, seed data, config. Converting a 4 MB lookup file to Parquet is busywork.
• Watch the scan bill, not the storage bill. Storage is cents. Scans are dollars. Athena workgroups can cap bytes scanned per query, and that limit is worth setting before someone discovers what a full table scan costs.

Frequently asked questions

Is Parquet always cheaper than CSV?

On a bytes-scanned billing model, for analytical queries that touch a subset of columns, almost always. It is not cheaper for point lookups, for very small datasets, for row-level updates, or when the query selects every column and filters on nothing. It also costs compute to produce, which matters when the data is rarely read.

How much smaller is Parquet than CSV in practice?

It depends entirely on cardinality, and anyone quoting a single ratio is guessing about your data. In the DuckDB TPC-H comparison the Parquet file came out around five times smaller than the CSV. Columns with few distinct values do far better than that; UUID and free-text columns do far worse. Measure on one partition of your own data instead of trusting a headline number.

Does compressing my JSON or CSV give me the same savings?

Partially. On Athena you are billed for bytes scanned before decompression, so compression does reduce the bill. What it cannot do is let the engine skip columns or row groups, and if you reach for gzip you lose splittability, which can make queries slower even as they get cheaper. Compression and columnar layout solve different halves of the problem.

Should I use Parquet or ORC?

For most teams that difference is much smaller than the difference between either of them and CSV. Parquet has wider support across engines and languages, which is usually the deciding factor. ORC has a long history in Hive-centric stacks and remains a reasonable choice if that is where you already live. Both are splittable and both support predicate pushdown.

Why did my costs not drop after converting to Parquet?

The usual suspects, in the order I would check them: the table is not partitioned, the queries select all columns, the files are too small, or the writer produced one enormous row group. Pull DataScannedInBytes for the same query before and after. If the number barely moved, the engine was not able to skip anything, and the problem is the query or the layout rather than the format.

Is JSON still worth keeping now that Parquet has a Variant type?

As a raw landing format and a transport format, yes. As the thing analysts query directly, the case is weaker than it used to be. Variant support is still spreading across engines, so confirm your query engine handles it before designing around it rather than assuming it is available everywhere.

What file size should I aim for?

Roughly 128 MB to 1 GB per file is the widely used guidance, with row groups sized so a file holds several of them rather than one. Below that range you pay per-file overhead on every query; above it you can starve parallelism if the row group layout is wrong.

The one thing worth remembering

The Parquet vs JSON vs CSV question is not really about the formats. It is about how much of your data the engine is permitted to ignore. Parquet hands an engine the most opportunities to skip work, JSON gives it almost none, and CSV sits between the two while making every reader re-guess the schema.

An opportunity to skip work is not the same as skipping it, though. If the query asks for every column, if the table is not partitioned, if the files are too small or the row groups too large, you have paid the conversion cost and bought a smaller file with the same bill attached. Measure bytes scanned before and after on a single partition. That one number tells you the truth in an afternoon, and it beats any comparison table you will read, including this one.

Need a second pair of eyes on your data lake costs?

Most format work starts as a cost problem and ends as a layout problem. Things I can help with:

• Auditing which queries actually drive your Athena or BigQuery bill, and whether a format change would move them at all
• Designing the partition scheme and file sizing before the conversion, so the migration is worth running
• Building the CSV or JSON to partitioned Parquet conversion as a Glue, Spark, or DuckDB job, including compaction for small-file pipelines
• Diagnosing conversions that did not pay off: row group layout, dictionary encoding, splittability, gzip traps
• Setting scan limits, workgroup controls, and alerting so one bad query cannot produce a surprise invoice
• Retrofitting Iceberg or Delta Lake where the real requirement is row-level updates rather than a different file format

If you want a concrete opinion rather than a general one, send me a query’s execution statistics along with the table schema and a listing of one partition, and I will tell you where your bytes are going.

The post Parquet vs JSON vs CSV: Where the Money Actually Goes appeared first on John Nessime.