You look at the CRM reporting tab. It cannot do that. So now you are picking a platform, and within about ten minutes someone will say “we should build a data lake,” and someone else will say “just put it in the warehouse,” and the deciding argument will be about how much data there is.

That argument is almost always the wrong one. This post covers data lake vs data warehouse specifically for CRM analytics: why volume rarely decides it, what actually does, a decision procedure you can run in an afternoon, and where zero-ETL has changed the maths recently enough that older advice is stale.

Start with the number, then stop using it

Go and count your rows. Not estimate. Count.

A mid-sized company’s entire Salesforce org is usually a few million rows across the objects anyone actually reports on: accounts, contacts, opportunities, activities, cases. Add HubSpot, Zendesk and a billing system and you are still typically in the low tens of gigabytes once it is written as compressed Parquet. Activity and event history is the one object that can genuinely run large, and it is worth checking separately.

Numbers at that scale do not stress either architecture. A data warehouse will not break a sweat. A lake will not break a sweat. The lake-versus-warehouse debate as it is usually framed, the one about petabytes and elasticity and separating storage from compute, is a debate about a scale problem you do not have.

So count the rows, confirm you are small, and then stop using volume as an argument. Everything below is about what decides it instead.

Data lake vs data warehouse: the three options, honestly

Warehouse only

Load CRM objects straight into Redshift, Snowflake, BigQuery or Postgres. Model with SQL. Point the BI tool at it.

Where it wins: it is the fastest path to an analyst being productive, and that is not a small thing. Query performance on joins is better than a lake gives you without tuning. Access control is mature and row-level. Every BI tool connects natively. The people who will maintain this already know SQL and probably do not know Spark.

Where it doesn’t: semi-structured data is awkward, and CRM payloads are full of it. Custom field bags, JSON blobs, email bodies, call transcripts. You can store JSON in a modern warehouse but you will not enjoy querying it at any depth. Storing many years of daily snapshots gets expensive on warehouse storage in a way it does not on object storage. And you are more locked in: moving between warehouses is a migration project, not a repoint.

Lake only

Land raw API responses to S3, catalogue them, query with Athena or Spark. Iceberg or Delta as the table format if you want anything resembling a database.

Where it wins: raw retention is nearly free, so keeping every version of every record for years is a storage line item rather than a decision. Schema evolution is genuinely easier: when a SaaS admin adds a field, the lake absorbs it. Unstructured content lives alongside structured tables, which matters more now that people want to run models over call notes and support tickets. And an open table format means the data is not hostage to one query engine.

Where it doesn’t: you have taken on work a warehouse does for you. Small-file compaction, partitioning, table maintenance, catalogue hygiene. Query latency on interactive dashboards is worse unless you invest in layout. Governance is a build, not a setting. And the per-query cost model punishes exactly the workload CRM analytics generates: lots of small, repeated, unpredictable queries from a BI tool. Athena bills on data scanned with a per-query minimum, which means a dashboard refreshing thirty tiles is thirty billable scans whether or not anyone is looking.

Lakehouse

Raw and historical data in object storage as Iceberg tables; curated marts in the warehouse or queried in place by a warehouse engine. This is where most serious setups end up, and it is what AWS, Databricks and Snowflake have all converged on.

Where it wins: you get cheap deep history and expensive fast queries in the right places. Raw payloads stay replayable when a vendor changes a field type. Analysts still get warehouse-shaped tables.

Where it doesn’t: it is two systems. Two cost models, two sets of permissions, two places a table can be defined and drift. For a five-person company with one CRM, this is architecture as procrastination. The complexity is real and it should be earning something.

What actually decides it

Four levers, in the order they usually matter for CRM data.

1. How much history you need, and whether deletes matter

This is the big one and it is the one people skip.

CRM systems mutate in place. An opportunity’s stage, amount and close date are overwritten as the deal moves. If you only ever sync current state, you can report on the pipeline as it is today and you cannot report on what it looked like last March. Every question about pipeline movement, forecast accuracy, stage velocity or “what changed since the QBR” needs point-in-time history, and the CRM will not give it to you retroactively. Deleted records are worse: they are simply gone.

If you need history, you need somewhere cheap to keep daily snapshots or a change feed. That pushes toward object storage, or toward a warehouse with a genuinely cheap storage tier. If a rolling twelve months of current-state data covers every question anyone asks, the warehouse handles it alone.

2. How fast the schema churns

Count the custom fields added to your main CRM object in the last six months. If it is two, schema churn is not your problem. If it is forty, and half of them are from a business unit you do not talk to, a rigid warehouse schema becomes a standing maintenance tax and the lake’s tolerance for drift starts paying for itself.

3. The shape of the queries

Scheduled dashboards hitting the same tables repeatedly favour a warehouse, because per-second compute amortises across many queries and results are cached. Redshift Serverless, for example, bills in processing-unit-hours per second with a short minimum and shuts down when idle, which suits bursty BI traffic.

Exploratory one-off analysis across wide, rarely-touched history favours a lake, because you only pay when someone asks. Most CRM analytics is overwhelmingly the first kind. That fact alone settles a lot of these arguments.

4. Who is going to maintain it

The least technical criterion and often the most decisive. If the answer is “a RevOps analyst who is very good at SQL and has no platform team behind them,” build the warehouse. A lake handed to someone who cannot maintain it becomes an unqueryable pile of Parquet in about eight months. That is not a hypothetical failure mode; it is the normal one.

A decision procedure you can run this afternoon

1. Count rows per object for everything anyone reports on. Note which objects are over about ten million rows. Usually it is activities and nothing else.
2. Write down the ten questions the business actually asks. Not the ones you imagine. Mark each one as needing current state or point-in-time history.
3. Count schema changes on your main objects over the last six months.
4. Estimate query volume: number of dashboards, tiles per dashboard, refresh frequency, plus rough ad-hoc queries per week. This is the number that drives cost, not storage.
5. Model both costs against those numbers. For the lake: bytes scanned per query times queries per month. For the warehouse: compute-hours per month plus storage. Do it on a napkin. At CRM scale the answer is usually obvious and usually favours the warehouse.
6. Name the maintainer. If you cannot name a person, pick the option that needs less of them.

Read together, most CRM-only shops land on: warehouse for the curated layer, cheap object storage for raw history, and no ambition beyond that until something forces it. That is a lakehouse, but the small honest kind, and it is a very different commitment from standing up a full lake platform.

Where zero-ETL changed the maths

A lot of writing on this topic predates a genuine shift, so it is worth stating plainly.

Managed zero-ETL integrations now replicate CRM and support platforms into AWS targets without you building a pipeline. AWS Glue supports SaaS sources including Salesforce, SAP, ServiceNow, Zendesk and Zoho CRM, landing into Amazon Redshift, the SageMaker lakehouse, or S3 Tables with Iceberg underneath. Salesforce has pushed the other direction with zero-copy federation, letting Data Cloud read Iceberg tables in Databricks, Snowflake, BigQuery or Redshift in place, and letting those platforms read Data Cloud objects back.

Two consequences for this decision:

• Ingestion is no longer the differentiator. “The lake is easier to load into” was a real argument when you were hand-rolling connectors. It is much weaker when the same managed integration targets either destination.
• Iceberg makes the choice less permanent. If your raw layer is Iceberg tables in object storage, adding a warehouse engine later is a configuration change rather than a migration. That is a genuine reason to prefer open table formats even if you never build a “lake” as such.

Two honest caveats. Zero-ETL integrations cover a curated list of sources and regions, so check yours before designing around them. And a managed replica is still current-state replication plus change capture, not a modelled history; you still have to decide how snapshots and slowly changing dimensions get built.

Arguments that don’t survive contact

• “We need a lake because we’ll do AI later.” Reasonable in spirit, wrong in sequence. Build the thing that answers this quarter’s questions, keep raw payloads in object storage so you have not destroyed your options, and revisit. Raw retention is the cheap insurance; a lake platform is not.
• “Storage is cheaper in a lake.” True and almost always irrelevant. At CRM volumes, storage is a rounding error next to compute and, more to the point, next to the salary of whoever maintains the thing.
• “The warehouse is a single point of vendor lock-in.” Fair, and it is why open table formats matter. But if you replicate raw data to object storage, the lock-in is on your transformation logic, not your data, and that is portable if you keep it in SQL and version control.
• “We’ll just query the CRM API directly and skip both.” Works until a dashboard fans out into rate limits and the whole thing becomes a data pipeline you did not design. If more than one person is asking questions, you need a copy.
• “Let’s do both properly from day one.” This is the expensive mistake. Two systems, two cost models, and tables that quietly disagree, in service of a workload that fits comfortably in one.

Frequently asked questions

Is a data lake overkill for one CRM?

For a single CRM with current-state reporting needs, usually yes. The point at which it stops being overkill is when you need years of point-in-time history, when unstructured content like call transcripts becomes part of the analysis, or when you are integrating enough systems that a common raw layer is cheaper than a dozen warehouse loaders.

How do I get historical CRM snapshots if I only have current state?

You mostly cannot get it retroactively, which is why this decision is urgent rather than academic. Start capturing now: either daily full snapshots to partitioned object storage, or a change feed if the platform offers one, then build slowly changing dimension tables from that. Field history tracking inside the CRM covers a limited number of fields and is not a substitute.

Does the lakehouse make this comparison obsolete?

It blurs the storage layer, not the operational reality. You still choose a query engine, a cost model, a governance model and a maintainer. The useful question has shifted from “lake or warehouse” to “how much of this do I want to operate myself,” which is a better question anyway.

Where does reverse ETL fit?

If you plan to push computed values back into the CRM, such as a lead score or account health field, that pushes toward the warehouse. Reverse ETL tools connect to warehouses far more readily than to query engines over object storage, and the write path wants a stable, modelled table rather than a raw one.

Can I start with one and move later?

Moving from lake to warehouse is straightforward: you are loading modelled tables into a new engine. Moving from warehouse to lake is harder, because the history only exists inside the warehouse and the transformations are written in its dialect. The asymmetric fix is to land raw data in object storage regardless of which you query from. That single habit preserves the option cheaply.

What about just using the CRM’s own analytics product?

Worth pricing seriously before building anything. It is genuinely the right answer when all your questions are about one system, and it removes an entire pipeline from your life. It stops being the right answer the moment a question spans CRM plus billing plus product usage, because you will be paying to ingest the other systems into a platform priced for CRM data.

The short version

In the data lake vs data warehouse question for CRM analytics, volume is the argument everyone reaches for and it is the one that matters least. Your CRM data is small. Both options handle it.

What decides it is how much history you need, how fast the schema moves, what shape the queries are, and who is going to keep it running. Answer those four and the architecture falls out on its own. And whatever you pick, land the raw payloads in object storage from day one, because that is the cheap decision that keeps every other decision reversible.

Need a second opinion on your CRM analytics stack?

I help teams make this call without over-building, and unpick it when it has already been over-built. Things I can help with:

• Running the sizing and cost model above against your real query volume, not a vendor calculator.
• Designing point-in-time history capture for CRM objects, including deletes and slowly changing dimensions.
• Setting up zero-ETL or connector-based ingestion from Salesforce, HubSpot, Zendesk or Zoho into Redshift, Snowflake or an Iceberg lake.
• Migrating an unmaintainable pile of Parquet into something an analyst can actually query.
• Cutting query costs on an existing setup: partitioning, table layout, dashboard refresh patterns.
• Building the curated CRM models that BI tools sit on, in SQL and in version control.

If you are mid-argument about this, send me your row counts and the ten questions the business wants answered. That is usually enough to settle it.

The post Data Lake vs Data Warehouse for CRM Analytics: Volume Is the Wrong Question appeared first on John Nessime.

That gap between what you think you are paying for and what you are actually paying for is what makes Redshift cost optimization awkward in a SaaS analytics product. You are not running one nightly batch against a warehouse that sleeps the rest of the day. You are serving hundreds of small, latency-sensitive queries that fire whenever a customer opens a dashboard, plus ingestion, plus whatever your BI layer and your connection pool are doing when nobody is watching.

This post covers the levers that genuinely move that number: how the meter works, why idle-looking connections still bill, how to work out which tenant is expensive, and which isolation choices cost more than they save. Where the popular advice is wrong for SaaS specifically, I will say so.

How Amazon Redshift actually charges you

Three buckets, and they behave very differently.

• Compute. On Redshift Serverless this is RPU-hours, metered per second. On provisioned clusters it is node-hours, plus separate line items for concurrency scaling and Spectrum.
• Storage. Redshift Managed Storage, billed by GB per month, independent of compute. Snapshots are storage too.
• Everything else. Cross-region data sharing and snapshot replication, machine learning, data transfer outside the usual in-region S3 paths.

In a SaaS analytics workload compute dominates, often overwhelmingly. And the important part: compute is a function of how long the warehouse is awake and at what capacity, not how many rows you touched. Two teams can scan identical data volumes and get bills that differ by a factor of five, purely because of how their queries arrive.

The billing mechanic that catches SaaS teams out

Read the serverless billing notes properly once and a lot of mysterious spend stops being mysterious. The parts that matter:

• The minimum charge is 60 seconds of resource usage, metered per second beyond that. This is a minimum for the warehouse, not for each individual query.
• Usage is recorded when a transaction completes, rolls back, or is stopped. A transaction that runs for hours shows up in your usage view only at the end.
• Cancel a query before it finishes and you still pay for the time it ran.
• Querying system tables is billed like any other query. Your monitoring loop is a workload.
• After a burst, capacity can stay elevated for a period after the load drops. Scale-down is not instant.

Put those together and you reach a conclusion that irritates most engineers: on serverless, ten small queries crammed into one minute are cheaper than the same ten queries spread across ten minutes. Every wake-up costs you a minimum billing window multiplied by your base capacity. That is the opposite of the instinct you have from tuning an OLTP service, where you smooth load out to protect tail latency.

Before you change anything, get the real numbers out of the warehouse rather than out of Cost Explorer, which lags and aggregates.

-- Daily billed RPU-seconds converted to RPU-hours.
-- Multiply by your region's on-demand RPU-hour rate for dollars.
SELECT trunc(start_time) AS day,
       sum(charged_seconds) / 3600::double precision AS rpu_hours
FROM   sys_serverless_usage
GROUP  BY 1
ORDER  BY 1 DESC;

charged_seconds is the column to build cost reporting on. compute_seconds is informative but it is not what the invoice is derived from, and the two can disagree within a given interval. Two constraints worth knowing before you wire this into a dashboard: the view holds roughly a week of history, and it is visible only to superusers. If you want month-over-month trends, UNLOAD it to S3 on a schedule and query the archive with Amazon Athena instead.

Lever one: connections that look idle and are not

This is the one that bites hardest and shows up last, because there is nothing to see. AWS documents it plainly: Redshift Serverless treats all incoming queries as billable user activity, including lightweight health-check queries sent by connection pools. It does not matter whether the statement came from your application, a JDBC driver, or a pooling framework doing its job.

So a pool that fires SELECT 1 every thirty seconds to validate connections is a warehouse that never gets to sleep. Your product has no users at 3am and you are still paying the minimum window, over and over, multiplied by base capacity. HikariCP, Apache Commons DBCP and PgBouncer all have some form of this behaviour, and the defaults are tuned for OLTP databases where a validation query costs nothing.

Open transactions are the same problem wearing a different hat. A BEGIN without a matching COMMIT or ROLLBACK keeps consuming RPUs until the session ends. Session timeouts exist precisely because this happens.

What I would check, in this order:

1. Disable the pool’s validation or heartbeat query entirely if the driver allows it. If it does not, stretch the interval as far as your failure tolerance permits.
2. Drop idle pool size to something honest. A pool sized for peak that stays warm overnight is pure waste on this pricing model.
3. Fix any code path that opens a transaction and returns early on error without ending it.
4. Set a session timeout per application role so a leaked connection cannot bill indefinitely.

-- Cap idle sessions for the application role.
-- Value is in seconds; the documented range is 60 to 1,728,000.
ALTER USER analytics_app SESSION TIMEOUT 1800;

-- Cap how many connections a single role can hold open at once.
ALTER USER analytics_app CONNECTION LIMIT 40;

-- What is connected right now, and with what timeout.
SELECT * FROM stv_sessions;

Session timeout changes apply to new sessions only, so recycle the pool afterwards or you will conclude the setting does nothing.

Lever two: base capacity, max capacity and usage limits are three different things

These get conflated constantly, and two of them will not save you a cent on their own.

• Base capacity (base RPU). The floor. It multiplies every billed second, including that 60-second minimum. Halving base capacity roughly halves the cost of a warehouse dominated by short queries. It also halves the compute those queries get, so watch p95 latency alongside the bill.
• Max capacity (MaxRPU). A ceiling on how far automatic scaling can go. It caps compute available to the workgroup, it does not stop queries and it does not interrupt anything running. Useful as a guard rail against a runaway scan, useless as a budget.
• Usage limits. An actual budget, expressed in RPU-hours over a daily, weekly or monthly period. The breach actions are: log to a system table, raise an SNS alert, or turn off user queries.

Only the third one can stop you spending money, and only the third one can take your product down at 2pm on a Tuesday. Set it to alert first, live with it for a full billing cycle so you learn the shape of a normal week, then decide whether you are genuinely willing to have queries turned off. In a customer-facing SaaS product the answer is usually no, and the limit stays as an alarm feeding PagerDuty or whatever you already page from.

There is also the price-performance target, the slider that hands scaling decisions to AWS in exchange for a stated cost or speed preference. AWS recommends it for mid-range base capacities and advises against it at the very bottom and very top of the RPU scale, so check the current guidance against your base setting before enabling it. It is worth trying on a staging workgroup with a replayed query mix; it is not worth switching on blind in production.

On provisioned clusters the equivalent controls are per-feature usage limits: concurrency scaling measured in time, Spectrum measured in data scanned, cross-region data sharing, and extra compute for automatic optimization. Each takes a breach action of log, emit a metric, or disable the feature. Concurrency scaling also earns free credits as the main cluster runs, which is why a moderately bursty provisioned cluster often shows no concurrency scaling charge at all until it suddenly does.

Lever three: Redshift cost optimization starts with knowing which tenant is expensive

Be clear-eyed about what is possible here. On serverless you cannot get an exact dollar figure per query, because billing happens at the warehouse level and the minimum charge is shared across whatever else was running in that window. What you can build is a defensible apportionment, and that is enough to find the customer whose scheduled export is quietly eating your margin.

Start by labelling every statement your API issues on a tenant’s behalf.

-- Set in the pool's per-checkout init SQL, or per request.
SET query_group TO 'tenant_4417';

SELECT metric_date, sum(events)
FROM   fact_events
WHERE  tenant_id = 4417
  AND  metric_date >= dateadd(day, -30, current_date)
GROUP  BY 1;

RESET query_group;

The label lands in the query log and surfaces as query_label in the SYS monitoring views. Keep it short: the older query log views truncate the label to 30 characters, so a tenant slug beats a UUID with prefixes bolted on.

-- Seven days of activity grouped by tenant label.
-- Note: time columns in the SYS views are microseconds;
-- confirm units before converting anything to money.
SELECT trim(query_label)   AS tenant,
       count(*)            AS queries,
       sum(execution_time) AS exec_time,
       sum(queue_time)     AS queue_time
FROM   sys_query_history
WHERE  start_time > dateadd(day, -7, sysdate)
  AND  query_label LIKE 'tenant_%'
GROUP  BY 1
ORDER  BY exec_time DESC;

Three columns in that view earn their keep beyond the obvious ones. result_cache_hit tells you which dashboard queries are already free, which is often a bigger share than people expect. The split between queue_time and execution_time tells you whether you have a tuning problem or a capacity problem, and those have opposite fixes. And user_query_hash groups repeated queries with different literals, which is exactly what an embedded dashboard produces, so it is the fastest way to find the one panel that fifty tenants are running badly.

From there, apportion the day’s charged_seconds by each tenant’s share of execution time. It is an approximation and you should label it as one when you show it to finance. It is still the difference between “Redshift costs us a lot” and “eleven percent of our warehouse spend is one customer pulling an unbounded date range every fifteen minutes.”

Lever four: the isolation model you picked is a cost decision

AWS’s SaaS guidance describes three partitioning models, and each one has a distinct cost signature on Redshift.

• Pool. All tenants share tables with a tenant identifier column. Cheapest by a wide margin, one warehouse to keep warm, one set of statistics. You pay for it in noisy-neighbour risk and in the access-control work you now have to do yourself.
• Bridge. Separate schemas or databases inside one cluster. Sounds like a compromise, behaves like neither. AWS’s own whitepaper is fairly blunt that the isolation profile does not usually justify it, since cluster-level access grants reach across the databases anyway.
• Silo. A warehouse per tenant. Clean boundaries and per-tenant cost visibility for free. On serverless it is also the most expensive thing you can do, because every workgroup carries its own base capacity floor and its own 60-second minimums. Twenty small tenants means twenty warehouses waking up independently.

Data sharing sits between these and is the pattern I reach for when workload interference is the real problem. One producer handles ingestion and transformation; consumers read the shared data without copying it, and a consumer’s load does not touch the producer. Genuinely useful for separating a heavy ETL window from customer-facing reads. But be honest about the arithmetic: every consumer is its own billable warehouse. Data sharing buys you performance isolation, not cheaper compute.

In a pooled model, the thing I set up first is a sort key that leads with the tenant identifier followed by the time column everyone filters on. That lets Redshift prune blocks before it reads them instead of scanning broadly and filtering afterwards. Combine it with row-level security so the tenant predicate cannot be forgotten by an application bug, and you have removed both the largest cost driver and the scariest failure mode in one change.

Lever five: scan less, refresh less

Classic warehouse hygiene still applies, it just pays differently here. Shorter queries mean fewer billed seconds at your base capacity.

• Sort keys that match your real predicates. Not the ones from the design doc. Pull the top twenty query hashes and read their WHERE clauses.
• Materialized views for the panels every tenant loads. Real savings on the read path, but refresh is compute you pay for. A view refreshed every five minutes and read twice an hour is a net loss.
• Let the result cache work. Identical query text against unchanged data is free. Anything your BI layer does that injects a timestamp or a random parameter into otherwise identical SQL is throwing that away. Worth checking in Amazon QuickSight, Metabase or whatever sits in front.
• Tune zero-ETL refresh intervals. The refresh interval on the target database is adjustable via ALTER DATABASE. Shorter is fresher and more expensive. For reporting and historical analysis, a longer interval is usually the right call and nobody notices.
• Keep cold history out of managed storage. Partitioned Parquet or Apache Iceberg tables in S3, catalogued in AWS Glue, queried through the lake. On serverless those queries bill at the same RPU rate rather than as a separate Spectrum line, so the win is in scan efficiency and storage cost, not in dodging a charge.

One reassuring detail: the automatic optimization work Redshift does in the background is not billed by default. It becomes billable only if you explicitly enable extra compute resources so those operations can run during busy periods. That is a deliberate trade, not an accident, and it is worth knowing before you turn it on.

Provisioned or serverless: how I would decide

Both have a genuine case and the honest answer depends on the shape of your load, not on which is newer.

Serverless wins when demand is spiky or concentrated in business hours, when you cannot forecast capacity, and for dev and test environments that sit idle most of the week. It also folds concurrency scaling and data-lake queries into a single rate, which removes two line items people routinely forget to model.

Provisioned RA3 wins when load is steady around the clock, because a reserved commitment on nodes can beat accumulated on-demand RPU-hours, and because you get the full workload management surface: queues, query priority, query monitoring rules with the complete set of controls. If you need to guarantee that a tenant’s export can never starve the interactive path, that machinery is more expressive than a price-performance slider.

Commitment discounts now exist on both sides, including reservations for serverless managed at the payer account level. Rates and terms change, so price it against your own numbers rather than a blog post.

The tell is simple. Pull a week of charged_seconds bucketed by hour and plot it. A flat line means you are paying serverless rates for provisioned behaviour. A sawtooth with long dead zones means the opposite.

Troubleshooting: the bill moved and nothing shipped

1. Get hourly billed seconds first. Aggregate charged_seconds by hour from the usage view. If the increase is spread evenly across all 24 hours, it is background activity: a pool, a monitor, a health check. If it is concentrated, it is a workload.
2. Check for anything running or queued right now. A single stuck statement explains a lot of otherwise inexplicable spend.
3. Look for transactions that never ended. A deploy that changed error handling can leave transactions open without a single failed request in your logs.
4. Compare query counts against query cost. Flat count with rising cost points at base capacity changes, scale-down lag, or data growth making the same queries slower.
5. Group by user_query_hash and diff against last week. New shapes appearing means a shipped change. Old shapes getting slower means data or statistics.
6. Only then look at storage. Managed storage grows quietly and it is rarely the cause of a sudden jump, but it is often the cause of a slow one.

-- Anything currently running or waiting.
SELECT user_id, query_id, transaction_id, session_id, status,
       trim(database_name) AS database_name,
       start_time, queue_time, execution_time
FROM   sys_query_history
WHERE  status IN ('running','queued')
ORDER  BY start_time;

Common mistakes

• Treating max capacity as a spending cap. It caps compute, not cost, and it will not stop a workload that simply runs for a long time.
• Optimising individual slow queries while ignoring a connection pool that wakes the warehouse every thirty seconds all night.
• Smoothing scheduled jobs out across the hour to be gentle on the warehouse. On serverless this is backwards; batching into fewer windows costs less.
• Building cost dashboards on compute_seconds instead of charged_seconds, then wondering why the totals never reconcile with the invoice.
• Giving every tenant their own workgroup for isolation, then discovering that base capacity floors and minimum charges multiply by tenant count.
• Setting a usage limit to “turn off user queries” on the first day, before anyone knows what a normal week looks like.
• Leaving the monitoring loop itself unbounded. Polling system views every few seconds is a workload that bills like any other.

Best practices worth the effort

• Label every tenant-originated query with query_group from day one. Retrofitting attribution is far more painful than adding a SET statement to your pool’s init SQL.
• UNLOAD the serverless usage view to S3 on a schedule. Seven days of retention is not enough to argue about a monthly invoice.
• Keep at least one usage limit configured as an alert, permanently, even if you never set a hard cap.
• Review base capacity quarterly against p95 latency, not just against cost. The right number moves as your workload changes.
• Put a hard date bound on every customer-facing query in the application layer. Unbounded ranges are the single most common source of surprise spend in embedded analytics.
• Model concurrency scaling and data-lake charges explicitly if you are on provisioned. They are the line items people forget until they appear.
• Tag workgroups and clusters consistently so cost tooling, whether that is AWS Cost Explorer or something like CloudZero or Vantage, can split spend by environment without guesswork.

FAQ

Does Redshift Serverless really charge me when nobody is using the product?

Idle time itself is not billed, but anything that sends a query is. AWS states explicitly that health-check queries from connection pools count as billable user activity. If your pool validates connections on a timer overnight, you are paying minimum billing windows all night. Check the pool before you conclude the pricing model is broken.

How do I calculate the cost of a single query?

You cannot, exactly. Serverless bills the warehouse, and the 60-second minimum is shared with whatever else ran in that window. The workable approach is apportionment: take charged_seconds for a period and divide it by each labelled tenant’s share of execution time from the query history view. Useful for finding outliers, not precise enough for per-customer invoicing.

Should I lower base capacity to save money?

Often yes, and it is the single highest-leverage change for a workload made of many short queries, because base capacity multiplies every billed second including the minimum. The catch is that it also reduces the compute each query gets. Change it in one step, watch p95 latency and queue time together for a full week, then decide whether to go further.

Is a warehouse per tenant a good idea?

Only when tenants are large enough to keep a warehouse genuinely busy, or when a contract requires that level of separation. For a long tail of small tenants it is the most expensive option available, since each warehouse carries its own capacity floor and its own minimum charges. Pooled tables with row-level security and a tenant-leading sort key gets you most of the isolation for a fraction of the compute.

Does concurrency scaling cost extra?

On Redshift Serverless, no, scaling is included in the RPU rate. On provisioned clusters it is a separate charge, offset by credits that accrue while the main cluster runs. That difference catches out teams migrating between the two, in both directions.

Will a usage limit take my product down?

It will if you configure the breach action to turn off user queries. The logging and alerting actions are safe and are what you want in a customer-facing system. Treat the hard stop as a deliberate business decision about which is worse, an unexpected invoice or an outage, rather than as a default setting.

Why does my cost report never match the AWS invoice?

Usually one of three things: using compute_seconds rather than charged_seconds, forgetting that usage is recorded only when a transaction completes so long transactions land in a later interval, or leaving storage and cross-region transfer out of the model entirely.

The one thing worth remembering

Redshift cost optimization for a SaaS analytics product is mostly not a query tuning exercise. It is a question of how often something wakes the warehouse up and at what capacity. Query tuning matters, sort keys matter, materialized views matter, but a connection pool with default settings will quietly outspend all of them combined.

So start at the meter. Pull hourly billed seconds, look at the overnight hours when your product has no users, and see whether the line goes to zero. If it does not, you have found your first and cheapest win before touching a single line of SQL.

Need help getting your Redshift bill under control?

I work with SaaS and data teams on exactly this problem, usually somewhere between the warehouse and the application that is hammering it. Things I can help with:

• Auditing an existing Redshift Serverless or RA3 workload and producing a ranked list of what is actually driving spend, with the numbers pulled from your own system views.
• Building per-tenant cost attribution: query labelling, a usage archive in S3, and a dashboard your product and finance teams can both read.
• Fixing the connection and session layer, including pool configuration, validation queries, session timeouts and transaction hygiene.
• Right-sizing base and max capacity against measured latency, and setting usage limits and alerts that warn without risking an outage.
• Reviewing multi-tenant data models: sort and distribution keys, row-level security, and whether data sharing or a pooled model fits your tenant mix.
• Deciding between provisioned and serverless with a workload profile behind the recommendation rather than a rule of thumb.

If you have a week of usage data, a suspicious hourly cost chart, or a pool configuration you are not sure about, send it over and I will tell you what I see in it.

The post Redshift Cost Optimization for SaaS Analytics: The Levers That Actually Move the Bill appeared first on John Nessime.

That is where embedded analytics on AWS stops being a front-end task and becomes an architecture decision. The awkward part is that the choice you make first, how the viewer is identified, quietly decides which isolation mechanisms remain available to you afterwards. Get that order backwards and you rebuild the data layer, not the iframe.

This post covers the four decisions that determine whether the build holds: identity model, tenant isolation, query mode, and session economics. Then the embed handshake itself, the errors you will actually see in the browser console, and what to check first when it fails.

Before anything else: the product got renamed

Amazon QuickSight was folded into a broader platform called Amazon Quick Suite, and the BI product inside it is now called Amazon Quick Sight. AWS documentation has since moved again under an “Amazon Quick” umbrella. You will land on all three naming conventions depending on which search result you click, which makes finding the right doc page genuinely annoying.

The practical upshot: the APIs, SDKs and IAM action names did not change. You are still calling quicksight:GenerateEmbedUrlForRegisteredUser against ARNs in the quicksight namespace, and the JavaScript SDK is still published as amazon-quicksight-embedding-sdk. Nothing in your code breaks. Only your bookmarks do. I mention it because half the confusion in a first embedded build comes from following a doc page that describes a UI menu that has since been reorganised.

Decision one: registered users or anonymous sessions

Two API operations generate embed URLs. GenerateEmbedUrlForRegisteredUser issues a session for a user who exists inside the BI account. GenerateEmbedUrlForAnonymousUser issues a session for someone who does not, and never will.

This reads like a convenience choice. It is not. Row-level security using session tags, the mechanism most SaaS products want, is supported only for anonymous embedding. It does not work with GenerateEmbedUrlForRegisteredUser, it does not work with the older GetDashboardEmbedUrl operation, and it is not supported with the IAM identity type. That constraint is documented, easy to miss, and it is the single most expensive thing to discover late.

So the fork is really this. If you register every viewer, you get per-user features (bookmarks, threshold alerts, scheduled snapshots) and you enforce isolation with username or group rules on the dataset. You also inherit the job of provisioning, deprovisioning and reconciling a user directory that mirrors your own. If you go anonymous, you skip all of that and filter with session tags at embed time, but per-user features are off the table because there is no persistent user to hang them on.

The registered-user request body is small. Everything interesting is in ExperienceConfiguration:

POST /accounts/<aws-account-id>/embed-url/registered-user

{
  "UserArn": "arn:aws:quicksight:<region>:<account>:user/default/<user>",
  "SessionLifetimeInMinutes": 60,
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": {
      "InitialDashboardId": "<dashboard-id>",
      "FeatureConfigurations": {
        "Bookmarks": { "Enabled": true }
      }
    }
  }
}

One trap on the anonymous path that deserves its own sentence. Anonymous sessions belong to a namespace, and any dashboard shared with that namespace is reachable by a session in it, whether or not you listed the dashboard in AuthorizedResourceArns. If you were treating that parameter as your allowlist, it is not. Namespace membership is the real boundary.

Decision two: where tenant isolation actually lives

There are three places you can put the filter, and only one of them scales.

• A dashboard per tenant. Works for five customers. Becomes a deployment problem at fifty and a change-management disaster at five hundred, because every visual fix is now a fan-out.
• A dataset per tenant, filtered in SQL. Better isolation guarantees, genuinely defensible in a compliance review, but you multiply refresh jobs and in-memory footprint by tenant count.
• One dashboard, one dataset, row-level security. The standard answer. One artifact to maintain, filtering applied per session.

With anonymous embedding, RLS is driven by tags. You declare tag keys against columns on the dataset, then supply values at embed time. The filter is evaluated server-side against the session, so a viewer poking at the iframe cannot lift it.

POST /accounts/<aws-account-id>/embed-url/anonymous-user

{
  "Namespace": "default",
  "SessionLifetimeInMinutes": 60,
  "AuthorizedResourceArns": [
    "arn:aws:quicksight:<region>:<account>:dashboard/<dashboard-id>"
  ],
  "SessionTags": [
    { "Key": "tenant_id", "Value": "acme-corp" },
    { "Key": "region",    "Value": "emea" }
  ],
  "AllowedDomains": ["https://app.example.com"],
  "ExperienceConfiguration": {
    "Dashboard": { "InitialDashboardId": "<dashboard-id>" }
  }
}

The value in SessionTags must come from your server-side session, never from a request parameter, a cookie your client can write, or a JWT claim you have not verified. This is the whole security boundary. Tag rules support combining conditions, so a manager who should see several sites is expressible without a second dashboard.

One quiet limit worth knowing before it bites: when RLS is applied to in-memory datasets, each field has a maximum length in Unicode characters, and fields exceeding it are truncated during ingestion rather than rejected. If your tenant identifiers are long opaque strings, test that a truncated value cannot collide with another tenant’s. Silent truncation plus a prefix collision is exactly the kind of bug that produces a cross-tenant data leak with no error anywhere in the logs.

Decision three: SPICE or direct query against your AWS data

Every dataset runs in one of two modes, and the difference shows up on a bill somewhere else in your account.

Direct query sends a live query to the source each time a visual renders. Against Amazon Athena that means an S3 scan per dashboard open, billed by bytes scanned. Against Amazon Redshift it means a concurrent query slot per viewer. Freshness is perfect. The failure mode is that dashboard load is now coupled to warehouse load, and your analytics traffic competes with everything else running there. Two hundred people opening a dashboard at 9am is two hundred queries, and Redshift concurrency is finite.

SPICE imports a snapshot into an in-memory engine and serves every viewer from it. One scan on refresh, then arbitrarily many reads. For an embedded product where the same aggregate is served to thousands of sessions, this is usually the right call, and the Athena cost difference between “scan once per refresh” and “scan once per pageview” is not subtle. What you give up is freshness, bounded by your refresh schedule, plus a capacity dimension to manage and incremental refresh to configure if the dataset is large.

The pattern I reach for first on a data-lake backend is a hybrid: recent partitions in SPICE with an incremental refresh on a look-back window, historical data left on direct query for the rare deep query. It costs more design effort up front and it is the thing most teams skip, but it is the only shape that keeps both the bill and the load time flat as history grows.

Whichever you pick, note that visual generation has a timeout, and data-source-specific timeouts apply on top of it. A query that is merely slow in a console tab renders as a broken visual in a customer’s browser. Model your worst partition, not your average one.

Decision four: what a session actually costs

I am not going to quote figures, because AWS changes them and you should read the current pricing page. The mechanism is what matters, and it is genuinely different from seat-based BI licensing.

• A reader session is a fixed 30-minute window. Not a pageview, not a query. Reopening the dashboard twenty minutes later is still the same session.
• Per-user pricing charges per session with a monthly cap per reader. Predictable when the same people return daily.
• Capacity pricing buys sessions in bulk with no user provisioning at all. This is the model built for embedding, and it is the one that pairs with anonymous sessions.
• Capacity pricing is also the prerequisite for programmatic dashboard refresh, so if near-real-time rendering is a product requirement, that decision is already made for you.
• Annual commitments to capacity unlock removing the “Powered by” attribution footer. If white-labelling is a contractual requirement, factor that in early rather than discovering it during a customer demo.
• Enabling certain Pro-tier and generative Q&A capabilities triggers an account-level monthly infrastructure fee that exists whether or not anyone uses the feature.

The cost failure mode nobody plans for is architectural rather than commercial. If you embed the dashboard on a tab that loads by default, you bill a session for every user who lands on that page and looks at something else. Lazy-load the iframe on explicit interaction. That one change is often the largest single lever on the bill, and it costs an afternoon.

The embed handshake, and the three things that break it

The flow is short. Your backend authenticates the user with your own identity system, calls the embed URL API with the right tags or user ARN, returns the URL to the browser, and the SDK mounts an iframe against it.

1. The generated URL carries a temporary bearer token valid for five minutes, and it is single use once redeemed. Generate it per page load from your backend. Never cache it, never put it in a build artifact, never log it.
2. Session lifetime is separate from URL validity, set with SessionLifetimeInMinutes, and ranges from fifteen minutes to ten hours with ten hours as the default. Ten hours is almost never what you want for a customer-facing product. Match it to your own session, or shorter.
3. Domains must be allowed explicitly. An administrator configures static domains in the admin menu, and AllowedDomains on the API call can override that with up to three domains or subdomains per request. Add an AllowedEmbeddingDomains condition to the IAM policy of the calling role, or any developer with that permission can list any domain on the internet.

On the browser side, the v2 SDK creates an embedding context (which appends its own zero-pixel iframe to body for message passing) and then mounts the experience:

import { createEmbeddingContext } from 'amazon-quicksight-embedding-sdk';

const context = await createEmbeddingContext();

await context.embedDashboard(
  {
    url: embedUrl,                        // fetched from your backend, just now
    container: '#analytics',
    height: '600px',                      // acts as loading height below
    resizeHeightOnSizeChangedEvent: true,
  },
  {
    toolbarOptions: { export: false, undoRedo: false, reset: false },
    attributionOptions: { overlayContent: true },
    onMessage: async (event) => {
      if (event.eventName === 'ERROR_OCCURRED') {
        console.error(event.message.errorCode);
      }
    },
  }
);

Two details in there earn their place. resizeHeightOnSizeChangedEvent turns the height value into a loading placeholder and lets the frame grow to fit content, which is what stops the dashboard rendering into a 600px letterbox with an inner scrollbar. And overlayContent tells the layout to overlay the attribution footer rather than reserve extra height at the bottom for it.

Troubleshooting embedded analytics on AWS

Almost every failure lands in one of these. Read the error code out of the ERROR_OCCURRED message before doing anything else.

• Forbidden means the URL’s authentication code expired. You held the URL longer than five minutes, or you served it from a cache, or a retry redeemed it twice. Fix the generation path, not the permissions.
• Unauthorized means the session obtained from that code expired. Different problem, different fix: your SessionLifetimeInMinutes is shorter than how long people keep the tab open. Handle it by re-fetching a fresh URL and re-mounting rather than letting the frame sit there dead.
• Frame never appears at all. Check the onChange handler for NO_CONTAINER or INVALID_CONTAINER, which usually means you mounted before your target element existed, and for INVALID_URL, which means the URL shape does not match the experience method you called.
• Frame appears, dashboard does not. Nine times out of ten this is the domain allowlist. The request domain has to match what was allowed, including scheme and any subdomain, and a staging hostname that nobody added is the usual culprit.
• Modals render off-screen. A known consequence of auto-resizing height: an export dialog can open above the visible viewport. Listen for MODAL_OPENED and scroll the parent page to the frame position.
• Toolbar features silently missing. Bookmarks, threshold alerts and scheduling require both the SDK toolbar flag and the matching entry under FeatureConfigurations in the embed URL request, and they only exist on the registered-user path. Setting the client flag alone does nothing.
• First render is slow, later ones are fine. Direct query against a cold warehouse. Compare the same query in Athena or Redshift directly to confirm before blaming the BI layer.

Common mistakes

• Choosing registered-user embedding for the identity story, then discovering session-tag RLS is unavailable on that path.
• Treating AuthorizedResourceArns as the security boundary instead of namespace membership.
• Deriving a session tag value from anything the client can influence.
• Generating the embed URL at build time, or caching it in a CDN, and then not understanding the Forbidden errors.
• Leaving session lifetime at the ten-hour default in a customer-facing app.
• Putting the dashboard on a default-loaded tab and paying for sessions nobody asked for.
• Building the first version on direct query against Athena because it is quicker to wire up, then meeting the scan bill.
• Forgetting that embedding and row-level security sit in the Enterprise tier, so a Standard-tier proof of concept proves nothing.

Best practices

• Decide the identity model before you build a single dataset. Everything downstream inherits it.
• Put the embed URL call behind one server-side endpoint that reads tenant scope from your own session and nowhere else. One function, one place to audit.
• Constrain the calling IAM role with an AllowedEmbeddingDomains condition and scope resources to specific namespaces rather than a wildcard.
• Write an automated test that requests tenant A’s embed URL and asserts tenant B’s rows are absent. Run it on every dataset change, because RLS breaks silently.
• Default to SPICE with a refresh schedule matched to a stated freshness SLA, and only reach for direct query where the SLA genuinely demands it.
• Track refresh failures as a first-class alert in CloudWatch or whatever you already run, whether that is Grafana, Datadog or something in-house. A stale dashboard that still renders is worse than one that errors, because nobody notices.
• Lazy-load the iframe on user intent, not on page mount.
• Keep the embedded surface read-only unless authoring is a real product requirement. Console embedding is a much larger permissions surface than dashboard embedding.

Is managed BI even the right call?

Worth asking honestly, because the answer is not always yes. The case for the AWS-native route is real: no connector layer to maintain against Athena, Redshift, S3 and Aurora, IAM you already understand, and a usage-based cost model that beats per-seat licensing when your viewers are bursty. If most of your data already sits in AWS, that adds up.

The case against is equally real. Visual customisation is limited compared to charting directly against your own API, the attribution footer needs a commitment to remove, and if you want full control of the front end you may be better served by Apache Superset or Metabase self-hosted, or by Grafana where the workload is closer to operational metrics than customer-facing BI. Those come with an operational burden you now own. That is the trade: you either run the BI layer or you rent it, and renting it means living inside its constraints.

Frequently asked questions

Do my users need AWS accounts to view an embedded dashboard?

No. With anonymous embedding they need no AWS account and no BI user record at all. Your application authenticates them however you already do, and your backend maps that identity to session tags when it requests the embed URL.

Can I use row-level security with registered-user embedding?

Yes, but only with username or group based rules, not with session tags. Tag-based RLS is restricted to the anonymous embedding path. If you need tags, you need anonymous sessions.

How long does an embed URL stay valid?

The URL itself carries a bearer token valid for five minutes and usable once. The session it opens is separate and lasts between fifteen minutes and ten hours depending on SessionLifetimeInMinutes, defaulting to ten hours.

Should I use SPICE or direct query for embedded analytics on AWS?

SPICE for anything with many viewers per refresh, which describes most embedded products. Direct query where the data must be current to the second, or where the dataset exceeds what you want to hold in memory. A hybrid split by data age is often the right answer and is under-used.

Why do I get a Forbidden error when the dashboard worked yesterday?

Forbidden points at the URL, not at permissions. The most common causes are caching the URL, generating it more than five minutes before use, or a client retry redeeming the same single-use token twice. If it is Unauthorized instead, the session expired and you need a fresh URL.

Can I white-label the embedded dashboard completely?

Largely. Themes control colours and typography, the SDK hides toolbar controls, and parameters let your own UI drive the dashboard. Removing the attribution footer entirely is tied to an annual capacity commitment, so confirm that against current terms before you promise it to a customer.

Does natural-language querying work in an embedded context?

Yes. The SDK exposes a generative Q&A experience alongside dashboards and visuals, driven by curated topics rather than raw tables. It is billed on its own capacity dimension and gates behind the Pro tiers, so treat it as a separate cost decision rather than a free addition.

The one thing worth remembering

Embedded analytics on AWS is not a rendering problem. The iframe is the last five percent. The part that decides whether the build survives contact with a second customer is the identity model, because it silently determines which isolation mechanism you are allowed to use, and that in turn shapes your dataset design, your refresh strategy and your bill.

Pick that first. Write the cross-tenant test before you write the dashboard. Everything else is recoverable in an afternoon.

Need help with an embedded analytics build?

This is the kind of work I do. Things I can help with directly:

• Reviewing an existing embed integration for cross-tenant leakage, including the session-tag path and the namespace boundary.
• Designing the identity and row-level security model before you commit to a dataset layout.
• Cutting Athena scan and Redshift concurrency cost by moving the right datasets into SPICE with incremental refresh.
• Building the backend embed-URL service with scoped IAM roles, domain conditions and sane session lifetimes.
• Setting up refresh failure alerting so a stale dashboard does not quietly serve last week’s numbers.
• Automated cross-tenant isolation tests wired into CI, so an RLS regression fails the build instead of the customer.

Send me the actual thing: your embed URL request payload with secrets stripped, the browser console error, or the dataset RLS rules. It is much faster to reason about a real payload than a description of one.

The post Embedded Analytics on AWS: The Four Decisions That Bite Later appeared first on John Nessime.

Acme has nineteen. The assistant retrieved five chunks that happened to mention Acme, because five is roughly how many chunks a retrieval returns, and then it counted them. There is no bug. Nothing failed. Similarity search returned the most similar things and the model described what it was given.

That is the defining problem with preparing CRM data for Amazon Bedrock, and it is upstream of chunk sizes and embedding models. CRM data is structured. Retrieval-augmented generation is built for unstructured text. Feed a table of opportunities through an embedding pipeline and you get a system that is genuinely good at “what did we discuss with this account” and quietly terrible at “how many”, “how much” and “since when”.

This covers the split that fixes it, what a good document actually looks like, the permissions problem nobody notices until it is a problem, and how to know whether any of it works.

Route the question before you build anything

Your users will ask two kinds of question and they need two different mechanisms.

• Narrative questions. “What were the objections on the Acme renewal?” “Summarise our history with this account.” “What did the customer say about pricing?” These live in notes, emails, call summaries and descriptions. Semantic retrieval is exactly right for them.
• Analytical questions. “How many open deals over fifty thousand?” “What is total pipeline this quarter?” “Which accounts have had no activity in ninety days?” These need aggregation, filtering and joins. Retrieval cannot do them and will not tell you it cannot.

The useful part is that you do not have to build the second half yourself. Bedrock Knowledge Bases supports structured data retrieval: you point it at Amazon Redshift or a Glue Data Catalog, and it generates and executes SQL from natural language against your data in place, with no copying or embedding. The counting question goes to SQL, the narrative question goes to the vector store, and both answers are right for the right reasons.

So the first architectural decision is not which chunking strategy to use. It is: which of your fields are text worth embedding, and which are facts worth querying. Get that wrong and no amount of tuning downstream will save it.

One practical note if you take the structured route. Bedrock’s execution role authenticates to Redshift as an IAM identity, and you grant it access the same way you would any other reader. Point it at curated views rather than base tables, exactly as you would a BI tool:

-- Give the knowledge base the same narrow access you'd give a
-- reporting user. A view is a contract; a schema grant is not.
GRANT USAGE ON SCHEMA analytics
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

GRANT SELECT ON analytics.v_opportunities
  TO "IAMR:AmazonBedrockExecutionRoleForKnowledgeBase";

Named columns and clear table names matter more here than anywhere else, because the model is reading your schema to write the SQL. A column called flag_2 will produce exactly the query quality it deserves.

What a good document looks like

For the narrative half, the instinct is to export records to CSV and let the chunker deal with it. That produces chunks that are half a row of one record and half a row of another, with no context about which is which.

Build documents deliberately instead. Three rules that do most of the work:

One record, one document. An opportunity and its notes become a single file. A chunk from it is then always about one thing.

Denormalise the context in. A note that says “they want a two-year term” is useless in isolation. Embed the account name, the opportunity name, the stage and the owner into the same document, so a chunk carries enough context to be retrieved and to be understood once retrieved.

Write it as prose, not as fields. Embedding models were trained on language. Stage__c: Negotiation embeds worse than “The opportunity is at the Negotiation stage.” This feels silly and it measurably improves retrieval.

On chunking itself: start with the default strategy and change it only when evaluation tells you to. If your documents are one record each and reasonably short, semantic or hierarchical chunking buys you less than people expect. The gains that matter come from document construction and metadata, not from chunk size.

The permissions problem

This is the one that turns a nice project into an incident, and it is easy to miss because nothing about it looks broken.

Salesforce has an elaborate sharing model: role hierarchies, sharing rules, territory management, field-level security. A rep sees their accounts and not everyone else’s. Your vector store has none of that. Once a record is embedded, it is a vector like any other, and anyone who can query the knowledge base can retrieve it.

So a rep asks about a competitor deal they were never on the team for, and the assistant helpfully summarises it. Or someone asks about compensation-adjacent records and gets an answer. No alert fires, because from the system’s perspective retrieval worked perfectly.

Three ways out, in increasing order of effort:

1. Only embed what everyone may see. Blunt, easy, and often correct for a first version. Exclude anything sensitive and be explicit that the assistant does not know about it.
2. Filter at retrieval time using metadata. Attach owner, team or region to every document, and pass a filter with each query derived from the caller’s identity. This is the practical answer for most teams.
3. Separate knowledge bases per audience. Heavier to operate, and the only option that gives you a hard boundary rather than a filter you have to remember to apply.

Whichever you pick, decide it before ingestion, because metadata has to be configured at ingest time. Retrofitting a filter you did not plan for means rebuilding the index.

Metadata is most of the value

Metadata filtering narrows the candidate set before similarity search runs. That improves relevance, reduces tokens in the prompt, and gives you the access control lever above. For S3 sources it lives in a sidecar file named after the object it describes:

// acme-renewal.txt.metadata.json, alongside acme-renewal.txt in S3
{
  "metadataAttributes": {
    "object_type": "Opportunity",
    "owner_id": "0051t00000XYZ",
    "account_id": "0011t00000ABC",
    "region": "EMEA",
    "is_closed": false,
    "last_modified_epoch": 1750000000
  }
}

Include, at minimum: what kind of record it is, who owns it, which account it belongs to, whether it is still open, and when it last changed. Those five turn “search everything” into “search the open opportunities this person owns, changed recently”, which is a different product.

Freshness, and the vectors that outlive the record

An embedding is a copy. When the source record changes, the copy is stale. When the source record is deleted, the copy is not.

The consequences are worse in a CRM than in a document store, because CRM records change constantly and confidently answering with last quarter’s stage is indistinguishable from answering correctly. An assistant that cites a deal which closed-lost in March, or a contact who left the company, is not obviously wrong to the person reading it.

• Sync on a schedule that matches how the data is used. Daily is fine for account summaries and useless for live pipeline questions, which should be going to SQL anyway.
• Handle deletes explicitly. Whatever removes a record from your lake must also remove its document from the source bucket, or the vector survives.
• Put the record’s last-modified timestamp in the metadata and surface it in the answer. “As of three days ago” is a small change that prevents a large class of misunderstanding.
• Reconcile document count against source record count periodically. Divergence means orphans.

What not to embed

CRM free-text fields are where people put things. Notes fields contain personal details, occasionally payment information, opinions about customers that were never meant to leave a private conversation, and years of accumulated noise.

Once embedded, all of it is retrievable by anyone with query access, and a vector is not something you can easily grep for a mistake.

• Run detection over free-text fields before ingestion and redact rather than hope.
• Exclude fields nobody asks questions about. Every extra field is noise competing for retrieval slots.
• Consider excluding records closed long ago; they dilute results and rarely answer anything.
• Add Bedrock Guardrails on the output side as a second layer, not as your only control.
• Keep the whole thing on private networking with VPC endpoints if the data warrants it.

There is a governance question here too, and it is worth raising before someone else does: your customers’ personal data is now in a vector store feeding a language model. Whether your privacy notice covers that is not an engineering decision, but it is an engineering responsibility to ask.

Evaluate it before you show anyone

A demo proves the pipeline runs. It does not tell you whether the answers are right, and this is a system whose failure mode is being wrong fluently.

Write thirty real questions from the people who will use it, with correct answers you have verified by hand. Include the analytical ones specifically, because those are where retrieval fails silently. Then run the set after every meaningful change to chunking, metadata or the document format, and compare.

Use the Retrieve API rather than RetrieveAndGenerate while you are tuning. Seeing which chunks came back tells you whether a bad answer is a retrieval problem or a generation problem, and those have completely different fixes.

Common mistakes

• Embedding structured records and expecting aggregation to work.
• Exporting to CSV and letting the chunker split rows arbitrarily.
• Field-value dumps instead of prose, so embeddings have little language to work with.
• No metadata, so every query searches everything and access control is impossible.
• Assuming Salesforce sharing rules follow the data into the vector store.
• Deciding on filtering after ingestion, then discovering it has to be configured at ingest.
• Syncing changes but never removing documents for deleted records.
• Embedding every field because it was easier than choosing.
• No PII detection over free-text notes.
• Tuning chunk size before fixing document construction.
• Judging the system on a demo instead of a scored question set.
• Granting the execution role a whole schema rather than specific views.

Best practices

• Route analytical questions to structured retrieval and narrative questions to the vector store.
• One record per document, with context denormalised in, written as prose.
• Metadata on every document: type, owner, account, open state, last modified.
• Decide the access model before ingestion, and filter at retrieval time by the caller’s identity.
• Redact free-text fields before embedding, and exclude fields nobody queries.
• Delete documents when source records are deleted, and reconcile the counts.
• Surface record age in answers so staleness is visible rather than implied.
• Expose curated views to the execution role, never whole schemas.
• Meaningful table and column names, because the model reads your schema to write SQL.
• A scored evaluation set, run on every change, including questions you expect it to fail.

FAQ

Why does it get counts wrong?

Because retrieval returns the most similar chunks, not all matching records, and the model counts what it was handed. Counting is a query, not a similarity search. Send those questions to structured retrieval over your warehouse instead.

Should I use the Salesforce connector or export to S3 first?

The connector is quicker to stand up. Going via S3 gives you control over document construction, metadata and redaction, which is where most of the quality lives. Start with the connector to learn what people ask, then move to S3 once you know what the documents should look like.

Which chunking strategy is best?

Usually the default, honestly. If each document is one record, chunking has less work to do. Change it in response to evaluation results rather than in advance, because chunking is the most over-discussed and least decisive variable in a CRM RAG setup.

How do I stop people seeing records they shouldn’t?

Metadata filters applied per query based on the caller’s identity, separate knowledge bases per audience, or simply not embedding sensitive records. There is no automatic inheritance of CRM sharing rules, and assuming otherwise is the most consequential mistake in this post.

How often should I re-sync?

Match the questions. Account histories and notes tolerate a daily sync comfortably. Anything needing current numbers should not be coming from embeddings at all. Whatever the interval, make sure deletions propagate.

Do I need a vector database at all?

Only for the narrative half. If every question your users ask is analytical, structured retrieval over Redshift answers all of them with no embedding pipeline, no vector store and no staleness. Plenty of CRM assistants would be better products if they had noticed that.

The one thing to remember

The model will answer either way. It will answer the narrative question well and the counting question badly, in the same tone, with the same citations, and nothing in the response distinguishes them.

So the preparation work is mostly triage: decide which questions are retrieval and which are queries, build documents that carry their own context, attach metadata before you ingest rather than after, and test with real questions whose answers you already know. Chunk size is the last thing to worry about and the first thing everybody argues about.

Building one of these?

The pipeline is usually the easy part; the decisions before it are where these projects succeed or quietly disappoint. Work I take on:

• Designing the split between semantic retrieval and structured querying, so counting questions get correct answers.
• Building the document preparation pipeline from CRM data: record-level documents, denormalised context, metadata, redaction.
• Access control design for knowledge bases, including per-caller metadata filtering and audience separation.
• Freshness and deletion handling so the index does not diverge from the source.
• Curated Redshift views and IAM grants for structured data retrieval.
• Evaluation harnesses with scored question sets, run on every change rather than once before launch.

Send me ten questions your users would actually ask, and I will tell you which of them retrieval can answer.

The post It Counted the Chunks: Preparing CRM Data for Amazon Bedrock appeared first on John Nessime.

You pull a sample of the extra rows and they all look fine. Real opportunity IDs, real amounts, real owners. You paste one into Salesforce and get nothing. The record was deleted eight months ago. So were the others.

Your incremental extract has never deleted a row in its life. It asks Salesforce for everything where SystemModstamp is greater than the last run, and a record that no longer exists cannot come back in that result set. So the lake only ever grows, the gap widens a little every week, and nothing anywhere reports an error.

That is the defining problem when you build a Salesforce data lake on AWS, and it is barely mentioned in the tutorials, which mostly stop once data lands in S3. Landing the data is the easy part. Keeping it equal to the source is the work.

This covers the ingestion options and their honest trade-offs, how to structure the lake, how to handle deletes properly (harder than it sounds), the fields that go stale without telling you, staying inside your API budget, and the query layer on top.

Why Salesforce is an awkward source

Four properties make it different from replicating a database, and every design decision below follows from them.

• Deletes are soft, then permanent. A deleted record moves to the Recycle Bin, where it is visible for a limited window and then gone. Your ability to detect a deletion expires.
• Some fields are computed at read time. Formula fields are not stored. Salesforce works them out when you query, which means what you extracted is a snapshot, not a value that stays true.
• The API is a metered resource. Your org has a daily request allowance shared with every other integration. A careless extract can starve the tools sales actually uses.
• The schema changes without warning. Admins add fields as part of their normal job. Nobody tells the data team.

Choosing how to get the data out

Four realistic paths, roughly in order of how much you have to operate yourself.

Amazon AppFlow is the native option: a managed connector that moves Salesforce objects into S3 on a schedule or on events, with no infrastructure to run. It handles both scheduled batch extracts and Salesforce Change Data Capture event flows, and it supports AWS PrivateLink to Salesforce, so the traffic does not have to cross the public internet. That last point matters more than it usually gets credit for.

Where it gets thin: schema drift handling is basic, and mapping large numbers of objects through the console gets tedious fast. Define the flows in Terraform or CloudFormation from the start rather than clicking them, or you will end up with production configuration nobody can reproduce.

A managed ELT vendor such as Fivetran or Airbyte handles schema drift, deletes and incremental logic for you, which is genuinely most of the hard work in this post. You pay per row or per connector, and on a large Salesforce org that number gets attention. Worth pricing against the engineering time before dismissing it.

Rolling your own on the Bulk API gives you total control and makes you responsible for everything: pagination, retries, API budget, delete detection, schema evolution. I would only choose this when an existing tool genuinely cannot express what you need, and I would expect it to be a real service with monitoring rather than a Lambda someone wrote in an afternoon.

Zero-copy sharing via Salesforce Data Cloud avoids replication entirely by querying Salesforce-held data from your warehouse. Attractive when compliance objects to copying CRM data, and it carries its own licensing and latency questions. Check what your Salesforce contract actually includes before designing around it.

Structure the lake so mistakes are cheap

Two layers, and the discipline to keep them separate.

Raw is append-only and never edited. Every extract lands as a new partition, in Parquet, exactly as Salesforce returned it. You do not deduplicate here, you do not fix types here, and you certainly do not delete here. Raw is your ability to rebuild everything downstream when you discover a logic bug six months in, which you will.

# Partition by extraction date, not by a business date. A bad run
# is then one partition to drop rather than a full-table repair.
s3://acme-lake/raw/salesforce/opportunity/extract_date=YYYY-MM-DD/

Curated is what people query: deduplicated to current state, typed properly, deletes applied. Build it as views over raw to begin with, and only materialise the ones that are genuinely too slow. Materialising early gives you a second copy of the truth to keep in sync, which is the problem you are already trying to solve.

-- Current state from an append-only raw layer: keep the most
-- recent version of each record and drop anything flagged deleted.
CREATE OR REPLACE VIEW curated.opportunity AS
SELECT *
FROM (
    SELECT o.*,
           ROW_NUMBER() OVER (
               PARTITION BY id ORDER BY systemmodstamp DESC
           ) AS rn
    FROM raw.opportunity o
)
WHERE rn = 1
  AND is_deleted = false;

Register both in the Glue Data Catalog so Athena and Redshift Spectrum see the same definitions. If you expect frequent updates and deletes rather than pure appends, an open table format like Apache Iceberg is worth the extra setup: row-level deletes, schema evolution and time travel are exactly the operations this workload needs, and they are painful to hand-roll on plain Parquet.

Handling deletes properly

Here is the section that matters. There are three mechanisms and you need more than one, because each has a hole.

1. Query the Recycle Bin

Standard SOQL hides deleted records. The REST API’s queryAll endpoint, the SOAP queryAll() call, or ALL ROWS in Apex all return them, and IsDeleted exists on virtually every object even though it does not appear in Setup.

-- Against /services/data/vXX.X/queryAll, not /query.
-- Substitute your own high-water mark for the placeholder.
SELECT Id, IsDeleted, SystemModstamp
FROM Opportunity
WHERE IsDeleted = true
  AND SystemModstamp > {last_run_utc}

The hole: records sit in the Recycle Bin for fifteen days by default. Salesforce Classic offers an extended retention setting that pushes it to thirty. But the bin also has a capacity tied to your org’s storage allocation, and when it fills, the oldest records are purged early to make room. No alert, no warning. So your detection window is not fifteen days. It is fifteen days or less, and you find out which during a mass-delete week.

2. Change Data Capture

Salesforce CDC publishes create, update, delete and undelete events as they happen, and AppFlow can consume them. This is the cleanest mechanism, because a delete arrives as an event rather than being inferred from an absence.

The hole: it is a streaming subscription, so if your consumer is down long enough, you miss events. Event replay windows are finite. CDC is a good primary mechanism and a bad only mechanism.

3. Periodic full ID reconciliation

This is the safety net, it catches everything the other two miss, and it is the one people skip because it feels crude. Pull nothing but the Id column for the whole object. One narrow field over the Bulk API is cheap even on millions of rows.

SELECT Id FROM Opportunity

Then diff it against the lake. Whatever is on your side and not on theirs no longer exists, whatever the reason:

-- Rows the lake still believes in and Salesforce does not.
SELECT l.id
FROM curated.opportunity l
LEFT JOIN staging.opportunity_ids s
       ON s.id = l.id
WHERE s.id IS NULL;

Run it weekly on your important objects. Soft-delete the differences in the curated layer rather than hard-deleting from raw, so you keep the audit trail.

One more case this catches that nothing else does cleanly: merges. When an admin merges two Accounts, Contacts or Leads, the losing record is deleted and gets a MasterRecordId pointing at the survivor. If your lake keeps the loser, you are double-counting a customer, and the arithmetic is wrong in a way that looks like a data quality problem rather than a pipeline problem.

The other thing that goes stale silently

Formula fields are not stored anywhere. Salesforce computes them when you ask. So the value that landed in your lake is a snapshot from extraction time, and Salesforce will happily return something different tomorrow without the record’s SystemModstamp moving at all.

Cross-object formulas are the worst version. A formula on Opportunity that references a field on its Account recalculates when the Account changes. The Opportunity did not change, so your incremental extract never picks it up. That column in your lake can be wrong indefinitely and nothing detects it.

Three ways out, in order of preference:

1. Do not extract them. Pull the underlying fields and reimplement the calculation in your transformation layer, where it is version-controlled and testable. More work up front, correct forever after.
2. Refresh them on a schedule. A narrow periodic extract of just the ID plus the formula columns, full-table. Cheap if the column list is short.
3. Accept it and document it. Fine for a formula nobody reports on. Not fine for anything in a revenue calculation.

Keep a list of which columns in your lake are formula-derived. Six months from now, when a number disagrees with Salesforce, that list is the first thing you will want and the last thing anybody wrote down.

Stay inside the API budget

Your Salesforce org has a daily API request allowance determined by edition and licence count, and it is shared. Exceed it and everything integrated with Salesforce stops working, not just your pipeline. That includes whatever marketing and support depend on, which is a conversation you would rather not have.

• Use the Bulk API for extracts, not the REST query API. Bulk is designed for volume and consumes the allowance very differently.
• Incremental by default, full refresh by exception. A nightly full extract of a large object is the single most common way to blow the budget.
• Give the pipeline its own integration user. That way the API usage reports attribute consumption to it, and you can see your own footprint instead of arguing about it.
• Watch consumption as a metric. Salesforce exposes API usage; graph it. Finding out by outage is expensive.

One thing that surprises people: field-level security on that integration user determines which fields come back. A field can exist in Salesforce, be populated, and arrive in your lake as consistently null because the integration user’s profile cannot see it. Check the profile before debugging the pipeline.

Query layer and access

Athena over the Glue catalog is the default answer and a good one: no cluster to run, pay per query, and it reads the curated views directly. Redshift Spectrum makes sense when you already run Redshift and want to join CRM data to warehouse tables.

Cost control on Athena is mostly about scan volume, which means partitioning and columnar storage rather than query tuning. Parquet plus sensible partitions does more than any amount of SQL cleverness.

On access: you have just replicated your customer database into object storage. Contacts and Leads are personal data, and Salesforce’s own field-level security does not follow it across. Encrypt the bucket, keep it private, use Lake Formation for column-level grants if different teams need different views, and decide the retention policy deliberately rather than by default. “We keep everything forever in S3 because it’s cheap” is a defensible engineering position and a poor compliance one.

Troubleshooting

Row counts drift upward over time

Deletes are not being applied. Run the full ID reconciliation and see how many rows come back. If it is a lot, also check for merged records via MasterRecordId before assuming the extract is at fault.

A column is always null in the lake, populated in Salesforce

Field-level security on the integration user’s profile. Log in as that user, or check the profile’s field permissions, before touching the pipeline.

A value disagrees with Salesforce but the record looks current

Almost certainly a formula field, especially a cross-object one. Check whether the column is formula-derived; if it is, the record’s modstamp never moved and your incremental extract had no reason to refetch it.

Extract fails partway on a large object

Query timeouts or governor limits. Chunk by date range or by ID range rather than pulling the object in one go, and make each chunk independently retryable so a failure costs you one slice rather than the whole run.

New Salesforce fields never appear

Most connectors map fields explicitly at configuration time and do not add new ones on their own. Poll the object’s describe metadata on a schedule and alert when the field list changes, so schema drift is a notification rather than a discovery.

Athena queries are slow or expensive

You are scanning too much. Check partitioning first, then whether the files are Parquet rather than JSON or CSV, then whether many tiny files are being read per query. Small-file proliferation is the usual culprit on frequently-run extracts, and compaction fixes it.

Common mistakes

• Building incremental sync on SystemModstamp alone and never handling deletes.
• Assuming the Recycle Bin gives you a guaranteed fifteen-day detection window.
• Treating CDC as sufficient on its own, with no reconciliation behind it.
• Ignoring merges, so a merged customer is counted twice forever.
• Extracting formula fields and treating them as durable values.
• Nightly full extracts of large objects, and the API outage that follows.
• Sharing an integration user with other tools, so nobody can attribute API consumption.
• Mutating the raw layer, which destroys your ability to rebuild.
• Clicking AppFlow flows together in the console with no infrastructure as code.
• Landing JSON or CSV instead of Parquet, then paying for it on every Athena query.
• No alerting on schema drift, so new fields are found by a user asking.
• Replicating Contacts and Leads into S3 with no retention policy or access controls.

Best practices

• Append-only raw layer, curated views on top, and never edit raw.
• Use at least two delete mechanisms, one of which is periodic full ID reconciliation.
• Soft-delete in curated rather than hard-deleting, so the audit trail survives.
• Reimplement formula logic downstream instead of trusting extracted formula values.
• Bulk API for extracts, incremental by default.
• A dedicated integration user with a documented, deliberately scoped profile.
• Partition by extraction date and store Parquet, or use Iceberg if you need row-level updates.
• Define flows, catalogs and permissions as code.
• Monitor API consumption and row-count delta against Salesforce as first-class metrics.
• Alert on schema changes rather than discovering them.
• Encrypt, restrict and set a retention policy on personal data the day you land it, not later.

FAQ

How do I handle deleted Salesforce records in a data lake?

Combine mechanisms. Query the Recycle Bin with queryAll and IsDeleted for recent deletions, consume Change Data Capture events for real-time coverage, and run a periodic full ID reconciliation as the backstop. No single one of the three is complete on its own.

Should I use AppFlow or a third-party ELT tool?

AppFlow if you want to stay inside AWS, value PrivateLink connectivity, and are willing to handle schema drift and deletes yourself. A managed vendor if you would rather buy those solved and can live with per-row pricing. Price both against the engineering time honestly; the build-it-yourself option is usually costed at zero and is not.

Parquet or Iceberg?

Parquet with date partitions is fine for append-only raw. Iceberg earns its complexity in the curated layer, where you want row-level updates and deletes, schema evolution and time travel. Salesforce data changes constantly, so that need is real rather than theoretical.

How often should I sync?

Match the decisions people actually make with the data. Hourly incrementals suit most reporting; sub-minute freshness usually means CDC and a real streaming consumer, which is a much larger commitment. Schedule the ID reconciliation separately and less often, weekly is normally enough.

Will this use up my Salesforce API limit?

It can, and the failure is shared with every other integration on the org. Use the Bulk API, stay incremental, give the pipeline its own user so consumption is attributable, and graph usage. Do not find out from an outage.

Do I need Salesforce Data Cloud?

Not to build a lake on AWS. It becomes interesting when compliance objects to copying CRM data at all, since zero-copy sharing avoids replication. Check what your existing Salesforce agreement covers before designing around it, because the licensing is a real factor.

Why do my numbers differ from Salesforce reports?

Check in this order: missing deletes, merged records counted twice, stale formula fields, fields the integration user cannot see, and only then your own transformation logic. The first two account for most of it, and the last is where people look first.

The one thing to remember

Getting Salesforce data into S3 is a configuration exercise you can finish in an afternoon. Keeping that data equal to Salesforce is an ongoing engineering problem, and the parts that break do not raise errors. They produce numbers that are slightly wrong, in a consistent direction, for months.

So build the reconciliation before you build the dashboards. A weekly job that pulls every ID and diffs it against the lake is unglamorous, cheap, and the only thing that will tell you the truth when someone asks why the two systems disagree.

Building or fixing one of these?

Most of the Salesforce lakes I get asked to look at work correctly for the first quarter and then quietly diverge. Work I take on:

• Building a Salesforce to S3 pipeline end to end: AppFlow or Bulk API ingestion, Glue catalog, Athena or Redshift Spectrum query layer.
• Auditing an existing lake for drift, and reporting exactly how many rows disagree with Salesforce and why.
• Implementing proper delete handling: CDC consumption, Recycle Bin queries and scheduled ID reconciliation.
• Reworking formula-derived columns into version-controlled transformations that stay correct.
• API budget work: moving extracts to Bulk, splitting integration users, and monitoring consumption before it causes an outage.
• Access and retention design for CRM data in S3, including encryption, Lake Formation grants and deletion policies.

Tell me which objects you replicate and how you currently detect deletes, and I will tell you where the drift is coming from.

The post The Row Count That Only Goes Up: Building a Salesforce Data Lake on AWS appeared first on John Nessime.

Nothing failed. Sync Design ran exactly as designed. What it did not do is fetch any new column information, because there is an unresolved mismatch sitting in the connection, on a different table, from a rename somebody did two months ago that nobody noticed. Until that is cleared, design sync will keep running and keep declining to pick up anything new.

That behaviour is documented, not a bug, and it is the single most useful thing to know about a Zoho Analytics Redshift schema mismatch. Mismatches are not independent little problems you can leave lying around. One of them jams the mechanism for all of them.

This walks through the three families these problems come in, names, types and time, the fix for each, and the structural change that stops them recurring: stop pointing the BI tool at your base tables.

First: work out which mode you are actually in

Zoho Analytics connects to Redshift two completely different ways, and half the confusion in this area comes from people reading advice written for the other one. Whoever set the connection up may have left, so check rather than assume.

Data Import copies the data into Zoho Analytics on a schedule. Reports are fast because they run against Zoho’s own storage. In this mode:

• Column additions and deletions are synchronised automatically.
• You can change a column’s data type inside Zoho Analytics, but the type has to stay compatible with the Redshift column or subsequent syncs fail. Zoho’s own guidance is to change it in both places, which is worth taking literally.
• You can create query tables, and import a filtered subset using a custom query.

Live Connect keeps nothing locally and queries Redshift when a report loads. It is available on the paid tiers only. In this mode:

• Column additions, deletions and renames are not synchronised automatically. You have to trigger Sync Design from the Edit Redshift Settings page.
• You cannot change a column’s data type in Zoho Analytics at all. Whatever Redshift says, that is what you get.
• No query tables, and you cannot pull other data sources into that workspace.
• Foreign keys defined in Redshift become lookup relationships automatically, which is a genuine advantage over Data Import, where you build those by hand.
• Report loading time is your Redshift cluster’s problem now.

The practical consequence: in Live Connect, every schema fix has to happen in Redshift. There is no BI-side escape hatch. That constraint sounds annoying and is actually the thing that pushes you toward the right architecture, which is the last section of this post.

Family 1: names

Zoho keeps its own copy of the table and column names it expects. When Redshift’s names drift away from that copy, the difference shows up in the Mismatch tab of the connection settings, and the two most common causes are both silent.

Case folding

Redshift lowercases unquoted identifiers. Somebody writes what looks like a camel-case column name, Redshift stores something else, and the BI tool is now looking for a column that does not exist under that name.

-- These two do NOT create the same column.
ALTER TABLE analytics.orders ADD COLUMN DiscountAmount DECIMAL(12,2);
-- ...stored as: discountamount

ALTER TABLE analytics.orders ADD COLUMN "DiscountAmount" DECIMAL(12,2);
-- ...stored as: DiscountAmount

Pick one convention, lowercase with underscores, and enforce it. Mixed quoting across a schema means some columns are case-sensitive and some are not, and you will spend an afternoon working out which.

See what Redshift actually has

Before touching anything in Zoho, get the ground truth. Use SVV_COLUMNS rather than PG_TABLE_DEF, because the latter only returns rows for schemas that happen to be in your search_path and silently returns nothing otherwise, which has wasted a lot of people’s time.

SELECT table_name,
       column_name,
       ordinal_position,
       data_type,
       character_maximum_length,
       numeric_precision,
       numeric_scale
FROM svv_columns
WHERE table_schema = 'analytics'
ORDER BY table_name, ordinal_position;

Export that, put it next to the Mismatch tab, and work down the list. Guessing from memory is how you resolve four mismatches and leave the fifth.

Renames and drops

A rename upstream reads to Zoho as one column disappearing and an unrelated one appearing. If a report or formula referenced the old name, you will also see the alert about a view that cannot be accessed because of changes made to the table. The fix there is to re-synchronise the table from the connection settings, but re-syncing will not help while the Mismatch tab still has entries.

So the order is fixed: clear every mismatch first, then Sync Design, then fix reports. Doing it in any other order produces the “I clicked sync and nothing happened” experience.

Family 2: types

Redshift has a rich type system. A BI tool has maybe a dozen column types. The mapping is lossy in places, and the losses are quiet.

TEXT and BPCHAR are not what they look like

This one catches people migrating from PostgreSQL, where TEXT is unbounded. In Redshift it is an alias that becomes VARCHAR(256), and BPCHAR becomes CHAR(256). Longer values get rejected or truncated depending on how they arrive, and the column reaching Zoho is a 256-character string rather than the free text you thought you had.

-- Looks unbounded. Is not.
CREATE TABLE staging.notes (body TEXT);        -- VARCHAR(256)

-- Say what you mean. 65535 bytes is the VARCHAR maximum.
CREATE TABLE staging.notes (body VARCHAR(65535));

VARCHAR length is measured in bytes

Not characters. An accented Latin character costs two bytes, most CJK characters three, an emoji four. A VARCHAR(50) holds fifty English letters or twelve emoji. Names, addresses and free-text fields with international data hit this constantly, and the symptom in the dashboard is a truncated string rather than an error.

-- LENGTH counts characters, OCTET_LENGTH counts bytes.
-- The second number is the one that has to fit.
SELECT MAX(LENGTH(customer_name))       AS max_chars,
       MAX(OCTET_LENGTH(customer_name)) AS max_bytes
FROM analytics.customers;

Widening a VARCHAR is one of the few in-place alterations Redshift allows. Narrowing one, or changing a column’s type outright, generally means rebuilding the table, so size these deliberately at creation rather than planning to fix them later.

Numbers and precision

A DECIMAL(38,10) is a perfectly reasonable warehouse column and an awkward BI column. Currency stored as a float is worse, because you get rounding that appears only in the total row and only sometimes, which is a genuinely unpleasant thing to debug in front of a finance team.

Cast money to a fixed scale before it leaves Redshift. Two decimal places, DECIMAL not FLOAT, decided once in the warehouse rather than per-report in the BI tool.

SUPER, and anything else with no BI equivalent

Semi-structured SUPER columns, VARBYTE, GEOMETRY, HLLSKETCH: there is no sensible column type on the other side. Do not expose them. Flatten what you need into typed scalar columns in a view and leave the rest in the warehouse.

If you are in Data Import mode

You have the option of overriding a column’s type on the Zoho side. Use it sparingly. Zoho’s requirement is that the type stays compatible with Redshift’s, and “compatible” is doing quiet work in that sentence: an override that works today breaks the next sync when a value arrives that the Zoho type cannot hold. Changing it in both places, as Zoho recommends, is the version that keeps working.

Family 3: time

This one does not appear as a mismatch anywhere. It appears as a reconciliation problem, which is worse, because you spend the first hour looking for missing rows.

Redshift has TIMESTAMP, which carries no timezone and means whatever the writer intended, and TIMESTAMPTZ, which is stored in UTC. A BI tool has a timezone setting of its own. Between those, a row written at 23:40 local time can be counted on a different day at each end.

The tell is specific and worth memorising: daily totals match, monthly totals do not. Nothing is missing. A few hours’ worth of rows at each month boundary are being attributed to the neighbouring month.

-- Run the dashboard's aggregate directly against Redshift and
-- compare. Relative bounds so this keeps working next month.
SELECT DATE_TRUNC('day', created_at) AS day,
       COUNT(*)                      AS orders,
       SUM(total_amount)             AS revenue
FROM analytics.orders
WHERE created_at >= DATEADD(month, -1, DATE_TRUNC('month', GETDATE()))
  AND created_at <  DATE_TRUNC('month', GETDATE())
GROUP BY 1
ORDER BY 1;

The fix is to stop making the BI tool guess. Convert in Redshift, expose both the UTC instant and a pre-computed local date, and build every report on the pre-computed one:

-- CONVERT_TIMEZONE is the Redshift idiom. Doing this once here
-- beats doing it in every report and getting it right in most.
SELECT
    created_at                                            AS created_at_utc,
    CONVERT_TIMEZONE('UTC', 'Europe/London', created_at)  AS created_at_local,
    CAST(CONVERT_TIMEZONE('UTC', 'Europe/London', created_at) AS DATE)
                                                          AS order_date_local
FROM analytics.orders;

Named zones rather than fixed offsets, so daylight saving is handled for you. A hardcoded offset is correct for roughly half the year.

The structural fix: give Zoho a contract, not your tables

Everything above is treatment. This is prevention, and it is the part worth doing even if nothing is currently broken.

Pointing a BI tool at base tables means every upstream change is a potential BI incident. Someone widening a column, renaming a field, or adding a SUPER column for a new feature has no idea a dashboard depends on it. Put a view layer in between and that stops being true: the view is the interface, the tables underneath are free to change, and you decide when the interface changes.

CREATE OR REPLACE VIEW analytics.v_orders_bi AS
SELECT
    -- Explicit casts pin the types Zoho will see, so an upstream
    -- change cannot quietly alter the shape of the report.
    CAST(o.order_id      AS BIGINT)         AS order_id,
    CAST(o.order_status  AS VARCHAR(64))    AS order_status,
    CAST(o.total_amount  AS DECIMAL(18,2))  AS total_amount,

    o.created_at                            AS created_at_utc,
    CAST(CONVERT_TIMEZONE('UTC','Europe/London', o.created_at) AS DATE)
                                            AS order_date_local,

    -- SUPER flattened to something a BI column can hold.
    CAST(o.attributes.channel AS VARCHAR(64)) AS channel
FROM analytics.orders o
WITH NO SCHEMA BINDING;

WITH NO SCHEMA BINDING creates a late-binding view: it does not hold a dependency on the underlying table, so you can drop and recreate analytics.orders without Redshift refusing or the view vanishing. For a warehouse where tables get rebuilt by a nightly load, that is the difference between a maintenance window and a broken dashboard.

List columns explicitly. Never SELECT * in a view a BI tool depends on, because then any upstream column addition changes the contract without anybody deciding to.

Give the connection its own read-only Redshift user, granted access to the views and nothing else. That also means the credentials in the BI tool cannot read tables you did not intend to publish:

CREATE USER zoho_reader PASSWORD 'use-a-generated-one';
GRANT USAGE ON SCHEMA analytics TO zoho_reader;
GRANT SELECT ON analytics.v_orders_bi TO zoho_reader;

The honest cost: a view layer is a thing to maintain, and adding a column now means editing the view as well as the table. That is the point. The friction is the control. If your views are getting numerous, managing them with dbt or an equivalent gives you version control and review on what is otherwise a pile of undocumented SQL.

A repeatable resolution procedure

1. Get ground truth from Redshift. Run the SVV_COLUMNS query and save the output.
2. Open the Mismatch tab in the connection settings and list every entry, including ones on tables nobody reports on.
3. Resolve every mismatch. All of them. A single leftover blocks design sync for everything else.
4. Trigger Sync Design and confirm the new columns actually appear before moving on.
5. Check types, not just names. A column can sync successfully and still be the wrong type. Spot-check the ones carrying money and dates.
6. Reconcile a known number. Run the same aggregate in both places for a closed period. Daily and monthly. If daily matches and monthly does not, go back to the timezone section.
7. Fix broken reports last, once the data underneath them is right.

Troubleshooting

Sync Design runs but the new column never appears

An unresolved mismatch is blocking it, almost certainly on a table you were not looking at. Clear the Mismatch tab completely, then sync again.

“This view cannot be accessed due to some changes made in the table”

Something the report depends on was renamed or deleted in Redshift. Re-synchronise that table from the connection settings. If it recurs after every deployment, that is the argument for the view layer.

The connection failed entirely

Check three things in order: whether the Redshift database was renamed or dropped, whether Zoho’s IP addresses are still allowlisted in your security group, and whether the credentials still work. A renamed database needs the connection edited; a dropped one means starting over.

Reports show old data after a schema fix

In Live Connect, caching can be enabled per workspace with its own refresh interval, and it applies to reports rather than tables. If the numbers look stale after you fixed something, check that setting before you conclude the fix did not work.

Text is truncated in the dashboard but complete in Redshift

Byte-length truncation on multi-byte characters, or a TEXT column that quietly became VARCHAR(256). Compare LENGTH against OCTET_LENGTH and widen the column.

Totals are close but not equal

Timezone if the gap sits at period boundaries. Precision if it is a consistent tiny drift across everything. Filters that differ between the report and your reconciliation query if it is neither. Check them in that order, because the first two are far more common than a genuinely missing row.

Common mistakes

• Fixing one mismatch, running Sync Design, and assuming the rest can wait.
• Not knowing whether the connection is Data Import or Live Connect, and applying advice for the wrong one.
• Overriding a column type in Zoho without changing it in Redshift, so the next sync fails.
• Using TEXT in Redshift and expecting PostgreSQL behaviour.
• Sizing VARCHAR by character count when the limit is in bytes.
• Storing currency as a float.
• Mixing quoted and unquoted identifiers, so some column names are case-sensitive and some are not.
• Letting reports use raw timestamps and setting the timezone per report.
• Using a fixed UTC offset instead of a named timezone.
• Pointing the BI tool at base tables that a nightly job rebuilds.
• SELECT * in a view that a dashboard depends on.
• Connecting with an admin-level Redshift user because it was quicker.
• Declaring the fix done without reconciling a number against the warehouse.

Best practices

• Expose late-binding views to the BI tool, never base tables.
• Cast every column explicitly in the view so the types are decided, not inferred.
• Do timezone conversion in Redshift and publish a pre-computed local date.
• Lowercase, underscore-separated identifiers everywhere, unquoted.
• Fixed-scale DECIMAL for money, never floating point.
• Size VARCHAR against OCTET_LENGTH of real data, with headroom.
• A dedicated read-only Redshift user granted access only to the reporting views.
• Treat the Mismatch tab as a queue to empty, not a list to triage.
• Reconcile at least one aggregate against the warehouse after every schema change.
• Version-control the view definitions, with dbt or just a repository of SQL files.
• Tell whoever owns the upstream tables that a view depends on them.

FAQ

What exactly is a mismatch in Zoho Analytics?

A disagreement between the table and column names Zoho Analytics expects and the ones Redshift currently has. They are listed in the Mismatch tab of the Redshift connection settings. The important property is that leaving one unresolved stops Sync Design from fetching new column information at all.

Why does Sync Design not pick up my new column?

Because there is at least one mismatch outstanding. Clear the Mismatch tab entirely and run it again. It is not a caching issue and re-running it more times will not help.

Should I use Data Import or Live Connect?

Data Import when you want fast dashboards, query tables, and the ability to blend data, and can accept the data being as fresh as the last sync. Live Connect when the numbers must be current and you would rather not duplicate the data, accepting that report speed becomes a Redshift performance question and every schema change needs a manual sync.

Can I change a column’s data type in Zoho Analytics?

In Data Import, yes, provided it stays compatible with the Redshift type. In Live Connect, no. Either way the durable fix is to cast the column correctly in a view on the Redshift side, so both ends agree without anyone having to remember an override exists.

Why do my dashboard totals not match the warehouse?

If daily figures agree and monthly ones do not, it is timezone handling at period boundaries. If everything is off by a tiny consistent amount, it is numeric precision. Genuinely missing rows are the least likely of the three and the one people check first.

How do I handle SUPER columns?

Do not expose them. Extract the specific fields you report on, cast them to scalar types in a view, and let the rest stay in the warehouse. A BI tool has nowhere to put a nested document.

Will a view layer slow down Live Connect reports?

A little, since the casts and conversions run per query. In practice the dominant cost is how much data the query scans, so sort keys and distribution keys on the underlying tables matter far more than the view. Measure before optimising, and if a particular view is genuinely expensive, materialise it as a table refreshed by your load job.

The one thing to remember

A Zoho Analytics Redshift schema mismatch is not a small isolated problem you can leave in the queue. One unresolved entry stops new columns arriving at all, which is why the symptom people report is almost never “there’s a mismatch” and almost always “I added a column and nothing happened”.

Clear them all, then sync, then reconcile a real number rather than trusting that it worked. And once it is working, spend the afternoon putting a view layer in between, because the alternative is having this conversation again the next time someone upstream renames a field they had no idea you were reading.

Need this sorted out properly?

Warehouse-to-BI connections tend to be set up once, by someone who has since moved on, and then quietly degrade. Work I take on:

• Auditing an existing Redshift to Zoho Analytics connection and clearing the mismatch backlog properly.
• Building a reporting view layer in Redshift with explicit casts, timezone handling and late binding, so upstream changes stop breaking dashboards.
• Reconciling dashboard figures against the warehouse and finding where the difference comes from.
• Redshift schema work: type corrections, column sizing, flattening SUPER data into reportable columns.
• Least-privilege database users and network access for BI tools, including security group and allowlist configuration.
• Putting the view definitions under version control with dbt so schema changes get reviewed instead of discovered.

Send me the output of the SVV_COLUMNS query above and a screenshot of your Mismatch tab, and I will tell you what is actually wrong.

The post The Column That Never Shows Up: Fixing Redshift and Zoho Analytics Schema Mismatches appeared first on John Nessime.

What just happened is that your data warehouse now has an endpoint on the public internet, reachable on port 5439, and a third-party SaaS platform holds credentials to it. Usually those credentials belong to whatever user was to hand, which on a lot of clusters means the admin account that can read every schema you have.

That is not an argument against doing it. It is an argument for doing it on purpose. Most guides on how to connect Amazon Redshift to Zoho Analytics walk you through six screenshots and never mention the two decisions that actually matter: which user this connection runs as, and how much of your network it needs.

This covers the whole setup in order, with those decisions made deliberately: the mode you pick and why it is hard to change later, a read-only user scoped to almost nothing, network access done narrowly, and what to actually expose once it works.

What the connection actually requires

Strip away the interface and the mechanics are simple. Zoho Analytics runs on Zoho’s infrastructure. To reach your cluster it opens an inbound TCP connection from Zoho’s own IP addresses to your Redshift endpoint, on the Redshift port. That is it.

Which means three things have to be true, and they are the three places setup fails:

• The endpoint has to be reachable from outside your VPC. A cluster in a private subnet with no public endpoint cannot be reached by a SaaS tool, full stop.
• The security group has to allow it. Zoho publishes the IP addresses it connects from, and those need an inbound rule on the Redshift port. The default port is 5439, though it can be changed at cluster creation, so check yours rather than assuming.
• The credentials have to work and the user has to be able to see the schemas you want to report on.

Zoho keeps the current allowlist on a dedicated help page. Do not copy a list of addresses out of a blog post, including this one. They change, and a stale allowlist produces an intermittent failure that looks like everything except what it is.

Decide the mode before you start

Zoho offers two fundamentally different connection types, and switching later means rebuilding the workspace. Get this one right the first time.

Data Import copies your data into Zoho Analytics on a schedule and reports run against their storage. Dashboards are fast, you can create query tables, and you can import a filtered subset using a custom query. Column additions and deletions sync automatically. The trade is freshness: your numbers are as current as the last sync.

Live Connect stores nothing and queries Redshift each time a report loads. Numbers are always current and your data does not leave AWS. It is available on the paid tiers only. The trade is bigger than it looks: report speed becomes a Redshift performance problem, schema changes need a manual sync each time, you cannot change column types on the Zoho side at all, and no query tables.

My default for most teams is Data Import, because the operational surface is smaller and a scheduled sync is usually fresh enough for a dashboard people look at once a day. Live Connect earns its cost when the numbers genuinely need to be current, or when duplicating the data into a third-party system is a compliance problem.

Step 1: create a user that cannot do much

Do this before you open any network access, so that when the connection works you already know its blast radius.

-- A dedicated user for this connection and nothing else.
CREATE USER zoho_reader PASSWORD 'generate-a-long-random-one';

-- Reach the schema, then read the objects in it.
-- Both grants are needed; USAGE alone gets you nothing.
GRANT USAGE ON SCHEMA analytics TO zoho_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA analytics TO zoho_reader;

-- The grant above covers what exists today. This covers what
-- gets created tomorrow, which is the part people forget and
-- then debug as a mysterious missing table three months later.
ALTER DEFAULT PRIVILEGES IN SCHEMA analytics
  GRANT SELECT ON TABLES TO zoho_reader;

Then check what you actually granted, rather than trusting that the statements did what you meant:

SELECT table_schema, table_name, privilege_type
FROM information_schema.table_privileges
WHERE grantee = 'zoho_reader'
ORDER BY table_schema, table_name;

If that returns rows from schemas you did not intend to publish, fix it now. Once the connection is live, someone will build a report on whatever is visible and removing access becomes a conversation instead of a command.

The password goes into a SaaS platform, so treat it accordingly: long, random, generated, stored in your password manager, and rotated when people leave.

Step 2: open exactly as much network as you need

First find out what your cluster currently looks like, because the console tells you less at a glance than this does:

aws redshift describe-clusters 
  --cluster-identifier my-warehouse 
  --query 'Clusters[0].{Endpoint:Endpoint.Address,Port:Endpoint.Port,Public:PubliclyAccessible,SecurityGroups:VpcSecurityGroups}'

That gives you the hostname Zoho needs, the real port, whether the cluster is currently public, and which security groups govern it. Redshift Serverless has an equivalent setting on the workgroup rather than a cluster, but the same reasoning applies.

Then add one narrow inbound rule per Zoho range. Not 0.0.0.0/0, which is what happens when someone is in a hurry and it never gets tightened afterwards:

aws ec2 authorize-security-group-ingress 
  --group-id sg-0123456789abcdef0 
  --protocol tcp 
  --port 5439 
  --cidr 203.0.113.0/24

Put a description on each rule in the console so that in a year somebody can tell which ranges belong to which vendor. An undocumented allow rule is one nobody will ever dare remove.

While you are in the cluster parameter group, turn on require_ssl so connections that do not use TLS are refused rather than merely discouraged. Parameter group changes need a cluster reboot, so do it in a window rather than mid-afternoon.

Test from outside before you touch Zoho

# Does anything answer on the port, from a machine outside your VPC?
nc -zv my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 5439

# Then prove the credentials work, over TLS, as the new user.
psql "host=my-warehouse.abc123.eu-west-1.redshift.amazonaws.com 
      port=5439 dbname=analytics user=zoho_reader sslmode=require"

If both of those work from your laptop, the remaining variable is Zoho’s IP ranges rather than anything structural. If the first one hangs, it is the security group or public accessibility. If it connects but authentication fails, it is the user. Separating those two takes thirty seconds and saves an hour of guessing inside a web form that reports one generic error for both.

If you cannot make the cluster public

Plenty of organisations have a policy against it, and that policy is not unreasonable. Zoho’s answer for databases behind a firewall is Zoho Databridge, a lightweight agent you install on a machine inside your own network. It opens an outbound connection on 443 to Zoho and waits for requests, so nothing inbound has to be allowed at all. It runs on Windows, macOS and Linux, and it is designed for exactly this situation.

Worth checking before you commit to it: Databridge is documented against a long list of relational databases, and Redshift is not always named explicitly in that list. Since Redshift speaks the PostgreSQL wire protocol and has a JDBC driver, it is usually workable, but confirm with Zoho support for your plan rather than taking my word for it. If it fits, an outbound-only agent is a meaningfully better architecture than a public endpoint, and almost nobody setting this up knows the option exists.

Step 3: make the connection

With the groundwork done, this part is genuinely a form. In Zoho Analytics, create a workspace, choose to import from or connect to Amazon Redshift, and supply the endpoint hostname, the port, the database name, and the zoho_reader credentials.

Two expectations worth setting so you do not diagnose a non-problem:

• The first fetch takes a while, depending on volume and how fast your cluster responds. Zoho emails you when it finishes.
• An empty workspace before that fetch completes is normal. If you open it early it will show nothing, which looks exactly like a failure and is not one.

Step 4: choose what to expose, carefully

This is the screen people click through fastest and the one with the longest consequences. Whatever tables you select here become the interface between your warehouse and your dashboards, and every future upstream change becomes a potential BI incident.

Select views, not base tables. A reporting view with explicit casts, timezone conversion done in Redshift, and semi-structured columns already flattened gives you a stable contract. The tables underneath stay free to change.

CREATE OR REPLACE VIEW analytics.v_orders_bi AS
SELECT
    CAST(order_id     AS BIGINT)        AS order_id,
    CAST(order_status AS VARCHAR(64))   AS order_status,
    CAST(total_amount AS DECIMAL(18,2)) AS total_amount,
    created_at                          AS created_at_utc,
    CAST(CONVERT_TIMEZONE('UTC','Europe/London', created_at) AS DATE)
                                        AS order_date_local
FROM analytics.orders
WITH NO SCHEMA BINDING;

WITH NO SCHEMA BINDING makes it a late-binding view, so a nightly job can drop and rebuild the underlying table without Redshift refusing or the view disappearing. On a warehouse with a rebuild-style load, that alone prevents a recurring class of broken-dashboard morning.

Grant your reader access to the views specifically, and skip the blanket grant on the whole schema if you can. Fewer objects visible means fewer things somebody can accidentally build a report on.

Step 5: schedule it, then verify a real number

In Data Import mode, set the sync schedule to match how the data is actually produced. Syncing hourly when the warehouse loads once at 03:00 just means twenty-three pointless queries a day against your cluster, and Redshift bills by uptime and workload.

Then do the step everyone skips. Pick a closed period, run the same aggregate in both places, and compare:

SELECT DATE_TRUNC('day', created_at) AS day,
       COUNT(*)                      AS orders,
       SUM(total_amount)             AS revenue
FROM analytics.v_orders_bi
WHERE created_at_utc >= DATEADD(month, -1, DATE_TRUNC('month', GETDATE()))
  AND created_at_utc <  DATE_TRUNC('month', GETDATE())
GROUP BY 1
ORDER BY 1;

A connection that returns data is not the same as a connection that returns correct data. If daily figures agree but monthly ones do not, you have a timezone problem rather than a connection problem, and it is far easier to find now than six weeks into someone’s board pack.

Troubleshooting

Connection times out

Network, not credentials. A timeout means nothing answered. Check public accessibility, then the security group rule, then whether Zoho’s allowlist has changed since you configured it. A refused connection rather than a timeout usually means you reached the right host on the wrong port.

Authentication failed

The network is fine, which is genuine progress. Test the same credentials with psql from your own machine. Also check the database name: Redshift clusters often have both a default database and the one you actually use, and connecting to the wrong one authenticates fine and then shows you nothing.

Connected, but no tables listed

A permissions gap. USAGE on the schema and SELECT on the objects are separate grants and you need both. Run the information_schema.table_privileges query as a check rather than re-running the grants and hoping.

Worked on setup, fails intermittently later

Classic symptom of a partially stale IP allowlist: some of Zoho’s ranges reach you and some do not, so syncs succeed or fail depending on which host tries. Re-check the published list against your security group rules.

Sync succeeds but a new column never appears

In Live Connect, schema changes need a manual Sync Design, and an unresolved mismatch anywhere in the connection stops it fetching new column information at all. Clear the Mismatch tab completely, then sync again.

Reports are slow

In Live Connect that is your cluster, not Zoho. Look at the queries hitting Redshift and at sort and distribution keys on the underlying tables. Zoho also offers per-workspace caching for Live Connect with a configurable refresh interval, which trades freshness for speed.

Common mistakes

• Connecting with the cluster admin user because it was already in the password manager.
• Opening the Redshift port to 0.0.0.0/0 to make the test pass, then never tightening it.
• Copying an IP allowlist out of a blog post instead of Zoho’s own page.
• Granting USAGE without SELECT, or the reverse, and concluding the connector is broken.
• Forgetting ALTER DEFAULT PRIVILEGES, so tables created later are invisible.
• Pointing the connection at base tables rather than reporting views.
• Picking Live Connect for the freshness without accounting for the manual sync on every schema change.
• Assuming the setup failed because the workspace is empty during the initial fetch.
• Scheduling syncs far more often than the warehouse is actually loaded.
• Not enforcing TLS on the cluster.
• Declaring it done without reconciling a single number against Redshift.
• Leaving no note anywhere about which security group rules belong to which vendor.

Best practices

• A dedicated read-only user per integration, never a shared or admin account.
• Grant access to reporting views only, not whole schemas, where you can.
• One narrow security group rule per vendor range, described and dated.
• Enforce TLS with require_ssl rather than trusting the client to ask for it.
• Prefer an outbound-only agent over a public endpoint if your plan and setup support it.
• Decide Data Import versus Live Connect deliberately, because changing it means rebuilding.
• Expose late-binding views with explicit casts and timezone conversion done in Redshift.
• Match the sync schedule to your load schedule, not to how fresh you wish the data were.
• Reconcile at least one aggregate before anyone builds a dashboard on it.
• Rotate the connection password when people leave, and document where it lives.
• Test connectivity and credentials separately, from outside the VPC, before blaming the connector.

FAQ

Does Redshift have to be publicly accessible?

For a direct connection, yes: Zoho reaches your cluster inbound from its own IP addresses, so there has to be an endpoint it can resolve and reach. The alternative is Zoho Databridge, an agent inside your network that connects outbound on 443, which removes the inbound requirement entirely. Confirm Redshift support for it with Zoho before planning around it.

Which port does Zoho need open?

Whichever port your cluster listens on. 5439 is the Redshift default, but it can be set to something else when the cluster is created, so read it from describe-clusters rather than assuming.

Should I use Data Import or Live Connect?

Data Import for most cases: faster dashboards, query tables, automatic column syncing, and a smaller operational surface. Live Connect when the numbers must be current or copying data into a third party is a compliance issue, accepting slower reports and manual schema syncs. Decide before you build, because switching means starting the workspace over.

What permissions does the Zoho user need?

USAGE on the schema and SELECT on the objects you want reported on. Nothing else. Add ALTER DEFAULT PRIVILEGES so objects created later are covered without anybody having to remember.

Why is the workspace empty after I finish setup?

The initial fetch has not finished. Zoho emails you when it completes, and the workspace shows nothing until then. Give it time before you start pulling the configuration apart.

Can I limit which tables Zoho can see?

Yes, and you should. Grant the reader access only to a set of reporting views in a dedicated schema. That is a stronger control than the table picker in the interface, because it holds even if somebody later edits the connection.

Will this increase my Redshift bill?

Live Connect can, since every report load is a query. Data Import costs you one scheduled query run per sync. If cost matters, Data Import on a schedule matched to your load window is the cheaper shape by a wide margin.

The one thing to remember

Getting Zoho Analytics to talk to Redshift takes about twenty minutes. Getting it to talk to Redshift without handing a SaaS platform a superuser account and an open port takes about an hour, and that hour is the entire difference between a connection you can defend in an audit and one you quietly hope nobody asks about.

So do it in this order: least-privilege user first, narrow network access second, connection third, views rather than tables fourth, and a reconciled number before anyone builds a dashboard on it. Every step after the first is easier when the first one is already done.

Want this set up properly the first time?

Most of these connections get built under time pressure and inherit whatever shortcuts made the test pass. Work I take on:

• Setting up a Redshift to Zoho Analytics connection end to end, with least-privilege credentials and scoped network access.
• Reviewing an existing connection and reporting what it can actually reach, then narrowing it without breaking reports.
• Building the reporting view layer in Redshift: explicit casts, timezone handling, flattened semi-structured columns, late binding.
• Network architecture for BI access, including outbound-agent options where a public endpoint is not acceptable.
• Choosing between Data Import and Live Connect based on your actual freshness, cost and compliance constraints.
• Reconciliation checks so dashboard numbers are verified against the warehouse rather than assumed.

Tell me whether your cluster is currently public and which user the connection runs as, and I will tell you what I would change first.

The post Connect Amazon Redshift to Zoho Analytics Without Putting Your Warehouse on the Internet appeared first on John Nessime.