S3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For

Managed Iceberg is not a storage upgrade, it is a decision about who owns compaction, snapshot expiry and orphan cleanup. A practical comparison of S3 Tables against Iceberg in general purpose S3 buckets, covering the maintenance you actually get, the Lake Formation trap in the access path, how the two bills are shaped, and a seven-step decision procedure you can run against a single table.

Continue ReadingS3 Tables vs Iceberg on S3: When Managed Compaction Is Worth Paying For

Athena Partitioning: Why Your Queries Are Still Slow

Partitioning an Athena table is not the same as getting the benefit of it. A walkthrough of the three failure families behind slow queries on partitioned external data: predicates that never prune, catalog lookups that dominate planning, and file layouts that undo everything. Includes partition projection trade-offs, Glue partition indexes, compaction patterns, and how to tell which problem you have.

Continue ReadingAthena Partitioning: Why Your Queries Are Still Slow

SageMaker Unified Studio: What It Replaces, and What It Quietly Doesn’t

SageMaker Unified Studio is a front door, not a runtime. It consolidates the interface while leaving your bill, your IAM design and your infrastructure as code exactly where they were, and it quietly hands you a DataZone governance system to operate. Here is what it genuinely replaces, what it doesn't, the failure modes that bite during adoption, and how to decide whether it is worth it for your team.

Continue ReadingSageMaker Unified Studio: What It Replaces, and What It Quietly Doesn’t

Building a Hotel Data Lake on AWS That Agrees With the Night Audit

Hotel source data is mutable in the past, so an append-only pipeline drifts away from the PMS without anyone noticing until month close. A practical guide to room-night grain, bitemporal modeling with Apache Iceberg, PMS ingestion, guest data scope, and file physics at hotel volumes.

Continue ReadingBuilding a Hotel Data Lake on AWS That Agrees With the Night Audit

Backfilling Historical API Data into S3 Without Silent Gaps

A backfill that exits zero can still be missing a week of data, and nothing will tell you. This is a practical guide to the failure families behind silent gaps: pagination drift under a mutating source, retries that duplicate pages, prefix layouts designed for writes instead of reads, the seam where backfill meets live ingest, and the storage class rules that make mistakes expensive. Includes deterministic key derivation, S3 conditional writes, Athena partition projection, and a per-window manifest pattern that turns completeness into something you can query.

Continue ReadingBackfilling Historical API Data into S3 Without Silent Gaps

Grafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem

A Glue job that stops running emits no metrics, so the dashboard stays green and the alert quietly resolves itself. Here is why Grafana monitoring for AWS data pipelines misses that failure, and the heartbeat, metric math and no-data configuration that closes the gap, along with the CloudWatch query costs and IAM boundaries nobody warns you about.

Continue ReadingGrafana Monitoring for AWS Data Pipelines: The Green Dashboard Problem

Building a GraphQL Data Ingestion Pipeline on AWS That Doesn’t Lie to You

A GraphQL source can hand you a 200 OK, a populated data block, and a quietly broken column in the same response. Here is how to build a GraphQL data ingestion pipeline on AWS that catches partial errors, respects cost-based rate limits, resumes cleanly from a cursor, and notices when the schema moves under you.

Continue ReadingBuilding a GraphQL Data Ingestion Pipeline on AWS That Doesn’t Lie to You