The dashboard usually isn’t lying. It’s answering a slightly different question, and the cause is almost always the same one: the pipeline recorded what the property management system said on the day it read it, and the PMS has since changed its mind. A reservation got shortened. A rate was corrected. A rebate landed against a folio that closed two weeks ago. The lake never went back to look.

This post is about building a hotel data lake on AWS that survives that. Not a service tour, because the AWS documentation already covers S3, Glue and Athena better than a blog post can. What it covers instead is the handful of modelling and ingestion decisions that determine whether your numbers reconcile with the night audit six months from now, and what each of those decisions costs you.

The failure that shows up three weeks late

Most first attempts at hotel analytics are append-only. Pull yesterday’s reservations, write Parquet under a folder keyed by the day you pulled it, register the partition, done. Yesterday reconciles perfectly. So does the day after. Confidence goes up.

The problem is that hotel source data is mutable in the past, and it stays mutable for a long time. Group blocks get released. Stays get extended at the desk. Adjustments and rebates get posted against nights that already closed. Commission corrections come back from an OTA weeks after checkout. Night audit itself is a rewrite operation: it closes the business date and posts room and tax charges for every occupied room, and if the auditor reruns it or corrects a posting, the figures for that date change after you already stored them.

An append-only lake captures a diagonal. For each business date you keep the first thing you ever saw about it, and nothing after. Every individual day looks right on the day you check it. The month-to-date total drifts. Nobody notices until close, and by then you have no idea which nights moved or why.

That is the whole reason to be careful about grain and table format. Everything else in a hospitality pipeline is ordinary data engineering.

Fix the grain before you fix the pipeline

The core hotel metrics are per-night metrics. Occupancy is rooms sold divided by rooms available. ADR is room revenue divided by rooms sold. RevPAR is room revenue divided by rooms available, which is the same as ADR multiplied by occupancy. All three are defined against a single night.

What the PMS hands you is reservations. A reservation spans nights, carries one confirmation number, and may have different rates on different nights of the same stay. If you store reservations and try to compute occupancy from them, you end up writing date-range arithmetic into every query, and two analysts will write it differently.

Explode to room-nights once, in the pipeline

The fact table that makes everything else easy has one row per room, per night, per reservation. Stay dates, rate code, market segment, source of business, channel, room type, rate amount and the taxes as separate columns. Every downstream metric becomes a filter and a sum instead of a range join.

Do the explosion in the transform layer, not in the BI tool. If a range join lives in a Power BI model or a QuickSight dataset, it exists once per report and drifts once per report.

The denominator is a decision, not a fact

“Rooms available” sounds like a constant. It isn’t. Rooms out of order for renovation, rooms out of service for a night, house-use rooms and comp rooms all get treated differently by different properties, and the PMS report you’re reconciling against has already made a choice.

Same question on the numerator. Whether no-show revenue counts as room revenue, whether cancellation fees count, whether package components get split out of the room rate, whether ADR is quoted net or gross of tax. Hotel accounting conventions follow the Uniform System of Accounts for the Lodging Industry, and most PMS vendors document how their own reports treat each case. Read that documentation before you write a single SQL definition.

Write the definitions down in the repository next to the transform code. When the lake and the PMS disagree later, the first question is always whether it’s a bug or a definition, and without written definitions you cannot answer that question quickly.

Model restatements with two dates, not one

Every row needs a business date, which is the night the revenue belongs to, and an as-of timestamp, which is when you learned it. Two time axes. That’s what makes the difference between “the numbers changed” and “here is exactly what changed, when, and by how much”.

There are two ways to store that. Keep every version of every row and let the reader pick a version, or keep the current truth in a table you update in place and keep the history separately. The second is easier to query and easier to explain, and it’s the one I reach for first. It needs a table format that supports updates and deletes, which in practice on AWS means Apache Iceberg.

Iceberg gives you three things that matter here: row-level MERGE, snapshots you can query as of a point in time, and schema evolution without rewriting the dataset. Athena, Glue ETL, EMR and Redshift all read Iceberg tables through the Glue Data Catalog, so you’re not tying the storage to one engine.

A curated room-night table in Athena looks roughly like this. Note the partitioning, which I’ll come back to.

CREATE TABLE curated.room_night (
  property_code      string,
  business_date      date,
  confirmation_id    string,
  room_number        string,
  room_type_code     string,
  rate_code          string,
  market_segment     string,
  source_of_business string,
  room_revenue       decimal(12,2),
  room_tax           decimal(12,2),
  reservation_status string,
  guest_key          string,
  as_of_ts           timestamp,
  source_system      string
)
PARTITIONED BY (property_code, month(business_date))
LOCATION 's3://example-lake-curated/room_night/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

Loading is a MERGE from a staging table holding the latest extract. Matched rows get overwritten with the newer version, unmatched rows get inserted, and rows that vanished from the source get marked rather than deleted.

MERGE INTO curated.room_night AS t
USING staging.room_night_batch AS s
  ON  t.property_code   = s.property_code
  AND t.business_date   = s.business_date
  AND t.confirmation_id = s.confirmation_id
  AND t.room_number     = s.room_number
WHEN MATCHED AND s.as_of_ts > t.as_of_ts THEN
  UPDATE SET
    room_revenue       = s.room_revenue,
    room_tax           = s.room_tax,
    rate_code          = s.rate_code,
    reservation_status = s.reservation_status,
    as_of_ts           = s.as_of_ts
WHEN NOT MATCHED THEN
  INSERT (property_code, business_date, confirmation_id, room_number,
          room_revenue, room_tax, rate_code, reservation_status, as_of_ts)
  VALUES (s.property_code, s.business_date, s.confirmation_id, s.room_number,
          s.room_revenue, s.room_tax, s.rate_code, s.reservation_status, s.as_of_ts);

The as_of_ts comparison in the MATCHED clause is doing real work. Extracts arrive out of order more often than you’d like, particularly when a backfill runs alongside the scheduled job, and without that guard a late-arriving older snapshot overwrites newer truth. It’s a silent corruption, which is the worst kind.

Keep a restatement log, not just the current state

Before the MERGE runs, write the delta out to an append-only table: key, field changed, old value, new value, as-of timestamp. It costs almost nothing and it turns “the March numbers moved” into a query that returns rows.

You can also lean on Iceberg snapshots and time travel for this, and they’re useful for spot checks. I still prefer an explicit log, because snapshot retention will eventually expire the snapshot you want, and finance will eventually ask about a night that’s older than your retention window.

Landing the sources without building a point-to-point mess

A property runs more systems than people expect. PMS, POS, channel manager, revenue management, spa and golf, parking, loyalty, and whatever the marketing team bought last year. Land each of them raw and immutable first, then transform. The raw zone is your audit trail and your ability to reprocess without going back to a vendor API.

The PMS is the hard one

For OPERA Cloud, the Oracle Hospitality Integration Platform is the supported route. It exposes REST APIs over OAuth, and registration is self-service rather than going through the older partner validation process. Two things to plan for: the API surface is large and reservation, financial and configuration data live in different areas, and access depends on the property’s licensing, so confirm what the property actually has before you scope the work.

Legacy on-premise OPERA is a different world. Integration there goes through the older interface stack and the validation programme, and scheduled report exports to a file drop are frequently the pragmatic answer. Cloud-native systems like Cloudbeds and Mews expose REST APIs directly and are far less work.

Reading the PMS database directly, where you can even reach it, is tempting and usually a bad idea. Schema changes arrive with vendor upgrades and nothing tells you in advance, and depending on the contract it can put support out of scope. Check the agreement before you build anything on top of direct database access.

The relay box, if you need one

Properties with on-premise systems usually need something that picks up scheduled exports and pushes them to S3. Keep it boring: a small always-on Linux host, a scheduled job, checksum on write, and an SNS or CloudWatch alarm when a file doesn’t turn up. A cheap VPS from somewhere like Contabo or InterServer does this fine, and it is much easier to hand over at the end of an engagement than a machine sitting under the front desk.

For reaching that host without opening inbound ports on a hotel network, an outbound-only tunnel such as Cloudflare Tunnel is a clean fit. If staff are hitting admin interfaces from guest or public Wi-Fi, a consumer VPN client like NordVPN or Surfshark protects that transport, though it is not a substitute for a real network path into the property.

Time zones will bite you

The hotel business date is not a calendar day and it is not UTC. It’s a property-local concept that rolls at night audit, which might be at 2am or 4am depending on the property, and it can be held open when the audit runs late.

Store the source system’s own business date as the partition key. Store event timestamps in UTC alongside it. Never derive the business date by truncating a UTC timestamp, because for a group with properties in several time zones that quietly shifts revenue between nights, and the error is small enough to look like noise.

Keep card data out and plan for erasure on day one

Guest records are personal data and hotels hold a lot of it: names, addresses, passport and ID details in some jurisdictions, loyalty identifiers, stay history. Payment card data sits under a separate regime again.

The single highest-value decision is negative. Do not bring cardholder data into the analytics platform at all. Nothing in occupancy, ADR or segment analysis needs a card number, and pulling one in drags the whole lake, its query engines and everyone with access into PCI DSS scope. Take the payment token or a masked reference from the PMS and stop there.

For guest identity, pseudonymise at ingestion. Keep a surrogate guest key in the analytics tables and hold the mapping to real identity in one small, separately governed place with its own access controls. Amazon Macie can scan the raw zone for personal data that shouldn’t be there, which is useful precisely because someone will eventually land a CSV export nobody reviewed.

Erasure needs thinking about before you have to do it. Iceberg makes row-level deletes straightforward, but a deleted row is still present in older snapshots, and time travel will happily return it. Deletion isn’t complete until snapshot expiration has run past the snapshots containing that row and the underlying files have been removed. Set retention deliberately, know what your window is, and be able to state it. Crypto-shredding, where the identity mapping is encrypted with a per-subject key you can destroy, is the cleaner design if the requirement is strict.

The same discipline applies to workstations. Analysts pull extracts, extracts contain guest names, and laptops get replaced. A secure erasure tool such as O&O SafeErase covers that end of it, and it belongs in the same policy as the lake.

For access control, Lake Formation is the right layer. Define column and row filters once and they’re enforced across Athena, EMR and Redshift rather than reimplemented per engine. A group operator can then give each property’s management team row access to its own data from the same tables, which is far less work than building a schema per property.

File physics at hotel scale

Hotel data is small. A single property produces a few hundred room-nights a day. Even a large group is nowhere near the volumes that data lake tutorials assume, and that changes the advice in one specific direction: your problem is too many tiny files, not too much data.

Two habits create it. Partitioning by day when a property generates kilobytes per day, and polling the PMS every fifteen minutes so each run writes another sliver. Both feel careful and both make queries slower and more expensive, because the engine spends its time opening files rather than reading them.

• Partition curated tables by property and month, not by day. Iceberg prunes from its own metadata, so you get file-level skipping without the partition explosion.
• Batch the writes. Buffer frequent polls and MERGE on a schedule that matches how the numbers are actually consumed, which for most operational reporting means hourly at most.
• Run compaction. The Glue Data Catalog offers managed table optimizers for Iceberg tables covering compaction, snapshot retention and orphan file deletion, configurable from the console, CLI or API. Check the current format and engine support against the AWS docs before you rely on it.
• Or drive it from Athena directly with OPTIMIZE to rewrite data files and VACUUM to expire snapshots and remove orphans.

For the raw zone, where tables are usually plain Hive-style Parquet rather than Iceberg, Athena partition projection is worth knowing about. It computes partition values from a configured pattern instead of listing them from the catalog, which removes a metadata round trip on tables with many partitions. It does not apply to Iceberg tables, which handle pruning through their own manifests.

Athena bills on data scanned and Glue on job runtime, so cost tracks file layout and query shape rather than how much data you own. That’s a good thing at this scale, but it does mean a single badly written dashboard query on a schedule can cost more than the entire storage bill. Tools like Vantage or CloudZero make that visible per workload if the AWS cost console isn’t giving you enough granularity.

Troubleshooting a hotel data lake on AWS

The lake and the PMS report disagree by a small, consistent amount. Almost always a definition, not a bug. Compare the room count first: if rooms sold matches and revenue doesn’t, you’re including or excluding tax, packages or fees differently. If rooms sold doesn’t match, check house-use and comp rooms.

Yesterday reconciles, last month doesn’t. The append-only failure. Query your restatement log for that period. If you don’t have one, compare the current curated table against an older Iceberg snapshot for the same business dates.

Numbers went backwards after a backfill. An older extract overwrote a newer one. Check that the as-of guard is present in the MERGE and that the staging table is deduplicated to one row per key before the merge runs, keeping the highest as-of value.

Occupancy exceeds one hundred percent. Usually duplicate room-nights from a reservation that was modified into a new record while the old one stayed active, or a room moved mid-stay and counted in both rooms for the same night. Add a uniqueness check on property, business date and room number to the pipeline and fail the load rather than publishing it.

Queries got slow and nothing changed. Look at file counts before anything else. Compaction that stopped running, or a new high-frequency job appending small files, explains most of these.

A daily load silently produced nothing. This is the one that hurts, because an empty successful run looks like a healthy pipeline. Alert on the absence of expected rows per property per business date, not just on job failure. A deadman check in CloudWatch, or in Grafana Cloud if you already run dashboards there, catches it the same morning instead of at month end.

Common mistakes

• Storing reservations instead of room-nights, then rebuilding the date logic in every report.
• Treating the ingestion date as the business date. They diverge the first time a load fails and gets rerun the next morning.
• Building the reporting layer before agreeing the metric definitions with the finance and revenue teams.
• Pulling the full guest profile because the API returns it, rather than the fields the analysis needs.
• Skipping the raw zone and transforming on ingest, which means a logic bug is unrecoverable without re-extracting from the vendor.
• Partitioning by day out of habit, then wondering why a small dataset queries slowly.
• Relying on time travel as the audit trail, and discovering retention expired the snapshot finance asked about.

Best practices worth the effort

• One row per room, per night, per reservation. Everything else derives from it.
• Business date and as-of timestamp on every fact row, always.
• Raw zone immutable, curated zone merged, consumption zone shaped for the tool that reads it.
• Metric definitions in version control, next to the SQL that implements them.
• A daily reconciliation query against one PMS report, checked automatically, with a tolerance and an alert.
• Pseudonymise guest identity at ingestion and keep the mapping somewhere separate.
• Compaction and snapshot retention configured before the table grows, not after someone complains.

Frequently asked questions

Do I need Iceberg, or is plain Parquet enough?

Plain Parquet is fine for anything genuinely append-only, such as raw extracts and event logs. It stops being fine the moment you need to correct history, because updating a row means rewriting whole partitions and coordinating that yourself. Since correcting history is the defining characteristic of hotel data, the curated layer wants a table format with row-level updates.

Should the data lake match the PMS exactly?

It should match one named PMS report, on one named metric, within a tolerance you’ve agreed. Trying to match every report is a trap, because vendor reports use different denominators and different treatment of taxes and fees, and several of them disagree with each other. Pick the report finance already trusts and reconcile to that one.

How often should the pipeline pull from the PMS?

Match the decision it feeds. Financial reporting is a daily cycle anchored on night audit and needs nothing faster. Operational views like arrivals and room status benefit from intraday refresh, but that’s a different table with a different retention policy, not a reason to run the whole lake hourly.

Can guest names go in the data lake?

Sometimes, with a lawful basis, retention limits and access controls. But most of the analysis people actually want, occupancy, rate, segment, channel mix, pace, works perfectly on a pseudonymous key. Start without names and add them back only for a use case that genuinely requires them. Card data is a different matter and the answer there is simply no.

Athena or Redshift for the reporting layer?

At single-property and small-group volumes, Athena over Iceberg is usually enough, and there’s no cluster to size or keep warm. Redshift earns its place when you have high-concurrency BI users, heavy joins across many domains, or an existing warehouse investment. Either way, keep the storage open so the choice stays reversible.

How do I handle a group with many properties?

One set of tables with property code as the leading partition key, and Lake Formation row filters for per-property access. Separate schemas per property look tidy at three properties and become unmanageable at thirty, because every schema change and every metric fix has to be applied thirty times.

Where does benchmarking data fit?

Land it in the same lake as a separate domain, at whatever grain the provider supplies, usually a market aggregate rather than a room-night. Keep it in its own tables and join at report time. Mixing external benchmark figures into your own fact table is how index numbers end up in a revenue total.

The one thing to remember

A hotel data lake on AWS is not hard because of the volume. It’s hard because the past keeps changing, and an architecture that assumes otherwise fails quietly and slowly rather than loudly and fast.

Get the room-night grain right, carry a business date and an as-of timestamp on every row, merge instead of appending, and write down what each metric means before anyone builds a dashboard on it. The S3, Glue, Iceberg and Athena assembly is the easy part, and it’s well documented. Reconciling with the night audit six months later is the part that decides whether anyone trusts the thing.

Working on this with me

Most of the hospitality data work I get asked about is one of a small number of shapes. Here’s where I’d be useful:

• Designing the room-night model and the bitemporal load, including the MERGE logic and the restatement log, so your numbers reconcile with the night audit and you can show why they moved.
• Getting data out of the PMS reliably, whether that’s OHIP against OPERA Cloud, a REST API on a cloud-native system, or a scheduled export relay for an on-premise property.
• Auditing an existing lake that has quietly drifted, finding which business dates restated and which metric definitions diverged from the reports finance uses.
• Cutting Athena and Glue spend on a small dataset that behaves like a big one, usually through partition layout, compaction and killing scheduled queries nobody reads.
• Setting up Lake Formation row and column controls for a multi-property group so each property sees its own data from shared tables.
• Building the boring safety net: freshness and row-count checks per property per business date, deadman alerts, and a daily automated reconciliation against a named PMS report.

If you’ve got a discrepancy you can’t explain, send me the two figures and the query behind one of them, and I’ll tell you which side is wrong.

The post Building a Hotel Data Lake on AWS That Agrees With the Night Audit appeared first on John Nessime.