That is the failure mode worth understanding, and it stays invisible until you go looking for it. The conversion was real. The files did get much smaller. But the dashboard queries were still selecting every column from an unpartitioned table, so the engine still had to open every column chunk in every file. Columnar storage only pays out when the query gives the reader something to skip.

This is a working comparison of Parquet vs JSON vs CSV on the two things that end up on an invoice: bytes read, and time spent reading them. Each format gets a profile with the cases where it wins and the cases where it does not, followed by the places where the promised savings quietly evaporate and a procedure for modelling the change on your own data before you spend an engineering week on it.

Your billing model decides how much the format matters

Before comparing formats, work out what you are actually charged for. The same conversion produces very different savings depending on which of these you are on.

• Bytes scanned from object storage. Amazon Athena is the clearest case. You are billed for the bytes a query reads out of S3, rounded up, with a minimum charge per query. Here a format change converts almost directly into money.
• Logical bytes processed. BigQuery’s on-demand model bills on the uncompressed logical size of the columns a query references in a native table, not on the compressed size sitting on disk. Compressing harder does not lower that bill. Touching fewer columns does.
• Compute time. Snowflake, Databricks, EMR, and anything you run on your own hardware charge you for how long a machine is awake. The format shows up as wall clock time, which is a real cost but a much softer one.

The BigQuery detail catches people out. Google documents that on-demand cost is calculated from logical, uncompressed bytes for native tables, but that when you query external data stored as Parquet or ORC, the bytes charged are limited to the columns BigQuery actually reads. The same file format therefore sits on opposite sides of the billing line depending on whether the data was loaded or is being referenced in place. Both Athena and BigQuery also apply a 10 MB minimum per query, and BigQuery applies it per table referenced, so a swarm of small queries against small tables costs far more than the data volume suggests.

CSV: cheap to produce, expensive to interrogate

CSV has no schema, no statistics, and no internal structure. Every value is text. The number 1000000000 takes ten bytes instead of the four it would take as a 32-bit integer, and because the layout is row-wise, there is nothing for a column-aware compressor to exploit.

Where CSV wins

• A human needs to open it. Nothing else comes close.
• Export from a legacy system that offers no other option.
• Small data. Below a few megabytes, the metadata and encoding overhead of a columnar file is not repaid.
• Streaming appends. You can write a line at a time and stop whenever you like.

Where CSV does not

Two problems, and the second is worse than the first.

The first is that the reader has to parse every byte, even the bytes it intends to throw away. The DuckDB team published a comparison on TPC-H data at scale factor 20, run on a laptop, reporting the median of five runs, and it is the most honest set of numbers I have seen on this. Loading the lineitem table took roughly 11.8 seconds from CSV against 5.2 seconds from Snappy-compressed Parquet, with the CSV file about five times larger on disk. That is a 2x gap on load, not the order of magnitude most people assume. Modern CSV readers are genuinely fast.

The gap opens when you query the files directly rather than loading them. On the same setup, TPC-H Q1 ran in about 6.7 seconds against CSV and 0.9 seconds against Parquet, and the join-heavy Q21 took roughly 20 seconds against 2.1. That difference is not parsing throughput. It is that the Parquet reader can skip row groups using statistics and skip unreferenced columns entirely, while the CSV reader cannot skip anything, because nothing in the file tells it what it would be skipping.

The second problem is dialect. There is an RFC for CSV and it is widely ignored. Quoting, escaping, embedded newlines, date formats, and whether NA means null or Namibia are conventions rather than rules, so every consumer re-guesses and sooner or later one guesses differently from the others. That failure does not raise an error. It produces wrong numbers in a report nobody questions.

JSON: you pay for the schema on every single row

Newline-delimited JSON is what most event pipelines emit, because a producer can add a field without coordinating with anyone downstream. That flexibility is worth something real. It is also why JSON is usually the most expensive format on a bytes-scanned bill: every record repeats every key name, so a field called customer_subscription_status carries those thirty-odd bytes on every row, forever.

Where JSON wins

• Landing zone for data whose shape you do not control. Accept it, keep the raw copy, argue about the schema later.
• Genuinely nested or sparse records, where flattening into columns would give you a table that is mostly nulls.
• Debugging. Being able to decompress one file and read a record with your eyes is worth a lot at three in the morning.
• API and webhook payloads, where it is simply what arrives.

Where JSON does not

As a query target at any serious volume. You are paying scan charges on repeated key names, on quote characters, and on the fact that every number is stored as text. Compressing the files helps storage and helps the scan bill on Athena, but it cannot give the engine anything to skip, and gzip brings a separate problem covered below.

The nested-data argument for keeping JSON has also weakened. Parquet now has a ratified Variant type for semi-structured data, with support arriving across Delta Lake, Iceberg, and Spark, plus a shredding scheme that pulls frequently accessed fields out into real columns while leaving the rest flexible. If you are keeping JSON purely because your records have unpredictable shape, that reason is worth revisiting rather than treating as settled.

Parquet: what the file is actually doing

Understanding the layout is what lets you predict whether a migration will pay for itself, so it is worth thirty seconds.

A Parquet file is divided into row groups. Inside each row group, every column is stored as its own contiguous column chunk, and each chunk is split into pages. The footer at the end of the file carries the schema and, per column chunk, statistics such as minimum, maximum, and null count. That structure is what makes the two headline optimisations possible:

• Column pruning. The reader consults the footer, works out the byte ranges of the three columns you asked for, and never issues a read for the other forty-seven.
• Predicate pushdown. If a row group’s maximum value for event_date falls below your filter’s lower bound, the entire row group is skipped without being read.

Two later additions refine this. The page index stores per-page minimum and maximum values in a structure near the footer, so a reader doing a selective scan can locate matching pages without walking every page header, which previously meant pulling most of the column off disk anyway. Bloom filters cover the case statistics cannot help with at all: a high-cardinality column such as a user ID, where every row group’s min and max span the whole value range and pruning by range is useless. A bloom filter can say that a value is definitely not present in a row group. DuckDB even exposes parquet_bloom_probe so you can check which row groups a given value would eliminate.

Encoding does the rest of the work. Because a column chunk holds values of a single type, dictionary and run-length encoding apply before any general-purpose compressor starts. Low-cardinality columns such as country codes or status flags compress extremely well. UUIDs and free text do not, and writers often disable dictionary encoding for them. This is why compression ratios quoted in blog posts are close to meaningless for your data: the ratio is a property of your cardinality, not of the format.

Where Parquet does not win

• Point lookups. Fetching one complete row by ID means touching every column chunk. A row format reads a single contiguous block instead.
• Row-level updates. Parquet files are immutable. Changing one row means rewriting the file, which is precisely the problem Iceberg and Delta Lake exist to solve on top of it.
• Streaming writes. The schema lives in the footer, so a writer has to finish the file before it is readable. Small-batch writers produce small files, which carries its own tax.
• Tiny datasets. Below a few megabytes, footer and row group overhead is not worth paying.
• Human inspection. It is binary. You need tooling, and someone will eventually ask you for a CSV export anyway.

Where the savings quietly disappear

Every one of these has cost someone a migration that looked excellent in the design document.

• Small files. Each file costs a metadata read, an object storage request, and reader setup. Thousands of 2 MB Parquet files will lose to a handful of large CSVs. The commonly cited target is roughly 128 MB to 1 GB per file, and you need a compaction step to get there if your writer emits micro-batches.
• One enormous row group. Parallelism in most engines is per row group. A file containing a single row group is processed by a single thread no matter how many cores you have. Check this rather than assuming the writer got it right.
• Select-everything queries in the reporting layer. Column pruning cannot help a query that asks for all columns. This is the one from the opening paragraph, and it is usually a BI tool doing it rather than a person.
• No partitioning. Format reduces bytes per row. Partitioning reduces how many rows are considered at all. The two multiply, and partitioning is normally the larger lever. Converting without partitioning leaves most of the money on the table.
• Gzipped text. AWS documents that Parquet and ORC are always splittable, because they compress sections independently and carry metadata pointing at those sections, while most text compression formats are not. Bzip2 and LZO are splittable; gzip is not. One 5 GB gzipped CSV cannot be divided among workers, so it is read start to finish by one reader. The compression made storage cheaper and the query slower.
• Uppercase file extensions. A small operational trap worth knowing: Athena determines the compression type of CSV and JSON data from the file extension, does not recognise uppercase extensions such as .GZ, and treats a file with no extension as uncompressed plain text.

How to model the change before you migrate

Do not migrate on faith. This takes an afternoon and answers the question properly.

1. Find out what you scan today. Every Athena query execution carries a DataScannedInBytes statistic. Pull it for a representative window and rank queries by bytes scanned. Almost always a handful of them are most of the bill, and those are the only ones worth optimising.
2. Convert one day of data, not all of it. A single partition is enough to measure with.
3. Look inside the resulting file before trusting it. Row group count, row group size, and compression codec decide whether the file behaves.
4. Re-run the same queries against the converted partition and compare bytes scanned. Measured, not estimated.
5. Price the write side. Conversion is not free. Glue jobs, EMR time, and Lambda invocations all cost money, and if a dataset is queried twice a month the conversion may never pay back.
6. Check who else reads the files. One downstream consumer that only speaks CSV turns a clean migration into a dual-write pipeline.

For step three, DuckDB is the fastest way to inspect a file without standing up a cluster. It runs happily on a laptop or a small VPS from a provider like Contabo or InterServer, which is usually all the compute this kind of analysis needs:

-- one row per column chunk: row groups, sizes, codec
SELECT row_group_id, row_group_num_rows, row_group_bytes,
       path_in_schema, compression
FROM parquet_metadata('events.parquet')
LIMIT 20;

-- convert a sample and control the row group size
COPY (SELECT * FROM 'events.json')
TO 'events.parquet'
(FORMAT parquet, COMPRESSION zstd, ROW_GROUP_SIZE 1000000);

The parquet_metadata function reads the footer and returns a row per column chunk, so you can see straight away whether your writer produced one row group or two hundred, and which codec it used. If you would rather stay in Python, pyarrow.parquet.ParquetFile exposes the same footer through its metadata attribute, including per-column statistics and whether a bloom filter offset is present.

If the data already lives in S3 and you want the engine you are already paying for to do the conversion, Athena’s CTAS handles it in one statement. The AWS documentation lists GZIP and SNAPPY as the compression options for Parquet output here, with GZIP as the default:

CREATE TABLE events_parquet
WITH (
  format = 'PARQUET',
  write_compression = 'SNAPPY',
  external_location = 's3://your-bucket/events-parquet/',
  partitioned_by = ARRAY['event_date']
) AS
SELECT event_id, user_id, event_type, amount, event_date
FROM events_json;

Three things to know before running it. Partition columns have to come last in the SELECT list. CTAS refuses to write into a location that already contains data, so a re-run needs the prefix cleared first. And Athena has a write limit of 100 partitions per CTAS statement, which means a large backfill has to be chunked rather than fired off in one go.

How I would decide

• Keep the raw landing zone in whatever arrives. Usually JSON. Do not convert on ingest and discard the original; you will want it the first time a schema assumption turns out to be wrong.
• Convert once, at the point where data becomes queryable. Partitioned Parquet for anything a dashboard or an analyst touches repeatedly.
• Partition before you optimise the format. If there is budget for exactly one change, partition on the column your queries actually filter on.
• Choose the codec by read frequency. Zstd where scanned bytes are the bill and the data is read often, Snappy where decompression speed matters more, gzip only when a consumer demands it.
• Leave small CSVs alone. Reference tables, seed data, config. Converting a 4 MB lookup file to Parquet is busywork.
• Watch the scan bill, not the storage bill. Storage is cents. Scans are dollars. Athena workgroups can cap bytes scanned per query, and that limit is worth setting before someone discovers what a full table scan costs.

Frequently asked questions

Is Parquet always cheaper than CSV?

On a bytes-scanned billing model, for analytical queries that touch a subset of columns, almost always. It is not cheaper for point lookups, for very small datasets, for row-level updates, or when the query selects every column and filters on nothing. It also costs compute to produce, which matters when the data is rarely read.

How much smaller is Parquet than CSV in practice?

It depends entirely on cardinality, and anyone quoting a single ratio is guessing about your data. In the DuckDB TPC-H comparison the Parquet file came out around five times smaller than the CSV. Columns with few distinct values do far better than that; UUID and free-text columns do far worse. Measure on one partition of your own data instead of trusting a headline number.

Does compressing my JSON or CSV give me the same savings?

Partially. On Athena you are billed for bytes scanned before decompression, so compression does reduce the bill. What it cannot do is let the engine skip columns or row groups, and if you reach for gzip you lose splittability, which can make queries slower even as they get cheaper. Compression and columnar layout solve different halves of the problem.

Should I use Parquet or ORC?

For most teams that difference is much smaller than the difference between either of them and CSV. Parquet has wider support across engines and languages, which is usually the deciding factor. ORC has a long history in Hive-centric stacks and remains a reasonable choice if that is where you already live. Both are splittable and both support predicate pushdown.

Why did my costs not drop after converting to Parquet?

The usual suspects, in the order I would check them: the table is not partitioned, the queries select all columns, the files are too small, or the writer produced one enormous row group. Pull DataScannedInBytes for the same query before and after. If the number barely moved, the engine was not able to skip anything, and the problem is the query or the layout rather than the format.

Is JSON still worth keeping now that Parquet has a Variant type?

As a raw landing format and a transport format, yes. As the thing analysts query directly, the case is weaker than it used to be. Variant support is still spreading across engines, so confirm your query engine handles it before designing around it rather than assuming it is available everywhere.

What file size should I aim for?

Roughly 128 MB to 1 GB per file is the widely used guidance, with row groups sized so a file holds several of them rather than one. Below that range you pay per-file overhead on every query; above it you can starve parallelism if the row group layout is wrong.

The one thing worth remembering

The Parquet vs JSON vs CSV question is not really about the formats. It is about how much of your data the engine is permitted to ignore. Parquet hands an engine the most opportunities to skip work, JSON gives it almost none, and CSV sits between the two while making every reader re-guess the schema.

An opportunity to skip work is not the same as skipping it, though. If the query asks for every column, if the table is not partitioned, if the files are too small or the row groups too large, you have paid the conversion cost and bought a smaller file with the same bill attached. Measure bytes scanned before and after on a single partition. That one number tells you the truth in an afternoon, and it beats any comparison table you will read, including this one.

Need a second pair of eyes on your data lake costs?

Most format work starts as a cost problem and ends as a layout problem. Things I can help with:

• Auditing which queries actually drive your Athena or BigQuery bill, and whether a format change would move them at all
• Designing the partition scheme and file sizing before the conversion, so the migration is worth running
• Building the CSV or JSON to partitioned Parquet conversion as a Glue, Spark, or DuckDB job, including compaction for small-file pipelines
• Diagnosing conversions that did not pay off: row group layout, dictionary encoding, splittability, gzip traps
• Setting scan limits, workgroup controls, and alerting so one bad query cannot produce a surprise invoice
• Retrofitting Iceberg or Delta Lake where the real requirement is row-level updates rather than a different file format

If you want a concrete opinion rather than a general one, send me a query’s execution statistics along with the table schema and a listing of one partition, and I will tell you where your bytes are going.

The post Parquet vs JSON vs CSV: Where the Money Actually Goes appeared first on John Nessime.