That is the shape of most trouble with SAP data integration with AWS Glue. The connector does not usually crash. It returns an empty or partial result set, the job exits zero, and your data lake quietly diverges from the source system. By the time anyone notices, you have no idea which run was the last correct one.

This post covers the parts that bite after the proof of concept: how delta tokens actually behave, why deleted records keep showing up in your reports, where partitioning stops working, and the licensing distinction between two things that both get called “ODP” and are not the same thing at all. There is a troubleshooting section at the end for when a job is already broken.

First, get the ODP terminology right, because licensing depends on it

Operational Data Provisioning is SAP’s framework for exposing extraction interfaces in a delta-aware way. A provider (a BW extractor, an ABAP CDS view, SLT, a HANA information view) writes changes into an Operational Delta Queue, and subscribers pull from that queue. This part is uncontroversial.

What matters is that there are two ways to talk to ODP, and SAP treats them very differently. SAP Note 3255746 restricts third-party and customer applications from using the RFC modules of the ODP Data Replication API against ABAP sources. That note has been sharpened over several revisions, from “unsupported” to “not permitted”, and SAP has stated it reserves the right to introduce technical measures and audit unpermitted use. A number of established extraction tools were built directly on ODP-RFC and have had to move.

The AWS Glue SAP OData connector takes the other path. It talks OData over HTTPS to an activated SAP Gateway service, and uses the ODP framework through that OData layer rather than through RFC. That is the route SAP points customers at. If you are being asked in an architecture review whether your pipeline is on the wrong side of Note 3255746, this distinction is the whole answer, and it is worth having the note number written down before the meeting.

Two honest caveats. First, notes get revised, so read the current version rather than a vendor blog summarising it, including this one. Second, “permitted” is not the same as “free”: you still need the OData service activated, a service user with the right authorisations, and a Basis team willing to own the queue growth on their side.

Failure family one: the delta token is shared state you do not own

This is the one that produces silent drift, and it is the reason the finance report was short.

When you read an ODP-enabled entity with change data capture switched on, the connector asks SAP to initialise a delta queue subscription. SAP hands back a delta token. That token is your bookmark. On the next run you pass it back, and you get only what changed since.

sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "ENABLE_CDC": "true"
    },
    transformation_ctx="initial_load"
)

ENABLE_CDC defaults to false. Leave it off and you get a full load every single run, which is expensive but at least obvious. Turn it on and the token appears in the DELTA_TOKEN column, populated only on the last row of the frame. That last row is not a business record. It is a carrier for the token, and if you write your frame straight to S3 without stripping it you have just added a junk row to your table.

The subsequent run passes the token back:

sapodata_df = glueContext.create_dynamic_frame.from_options(
    connection_type="SAPOData",
    connection_options={
        "connectionName": "sap-prod-odata",
        "ENTITY_NAME": "entityName",
        "DELTA_TOKEN": last_known_token
    },
    transformation_ctx="incremental_load"
)

Why this goes wrong

A delta token is bound to a service collection and a user. If a new initialisation runs with ENABLE_CDC set true for that same collection and user, SAP invalidates every token previously issued from the earlier initialisation. Not “expires eventually”. Invalidates, immediately.

So picture two Glue jobs sharing one service account, both reading the same extractor. Someone redeploys job B with the CDC flag still set from testing. Job B re-initialises. Job A’s stored token is now dead, and the next time job A runs you get:

Could not open data access via extraction API RODPS_REPL_ODP_OPEN

An error is the good outcome. The bad outcome is code that catches the exception, falls back to a full load, and writes it over the top of an incremental table with no dedupe. Or code that re-initialises on failure, which fixes job A and breaks job B, and now the two jobs take turns invalidating each other forever while both keep exiting zero.

Practical rules that come out of this:

• One SAP service user per consuming pipeline. Sharing a user is what couples independent jobs together.
• Persist the token somewhere durable and versioned. DynamoDB, Parameter Store, S3 with versioning enabled, all fine. Job bookmarks are not a substitute, because the token is source state, not read position.
• Never auto re-initialise on error. Fail loudly, page someone, and make the reinitialisation a deliberate human action with a documented backfill.
• Alert on zero-change runs. A pipeline that returns no changes for several consecutive runs against a busy table is telling you something, and it is rarely “business is quiet”.

Retention on the SAP side is a hard deadline

The delta queue is not infinite. In transaction ODQMON the Basis team configures retention, and a reorganisation job cleans up data past that window. If your pipeline is down longer than the retention period, the changes you missed are gone from the queue and no token will bring them back. You are into a full reload and a reconciliation.

Two things to find out before you go live: what the retention setting is, and whether the cleanup job is actually scheduled. Then make sure your pipeline’s alerting escalates well inside that window. A weekend outage on a queue with a short retention is a very expensive Monday.

Related: if anyone has been testing extraction with RODPS_REPL_TEST, that program registers as a real additional subscriber rather than simulating one. Because the queue is shared and data is only cleaned once every subscriber has collected it, forgotten test subscriptions hold delta data indefinitely and the underlying queue tables grow. Ask whoever ran the tests to reset those subscriptions in ODQMON.

Failure family two: deletes arrive as a column, not as a deletion

CDC extraction from ODP entities returns a couple of extra fields alongside your business columns. DML_STATUS carries UPDATED for inserts and updates, and DELETED for records removed at source. GLUE_FETCH_SQ is a per-record sequence derived from an epoch timestamp, in receipt order, and it is your tie-breaker when the same key changes twice inside one delta.

Nothing acts on those fields for you. If your job appends the frame to S3 and calls it done, a deleted sales order is now a row in your lake with DML_STATUS = DELETED and every other column populated exactly as before. Any query that does not filter on that column counts it. Cancelled orders keep appearing in revenue. Closed accounts stay open. The pipeline is green, the numbers are wrong, and the discrepancy grows slowly enough that nobody catches it early.

You have to apply the changes rather than accumulate them. Broadly two options:

1. Merge into a table format that supports it. Apache Iceberg, which Glue supports natively, gives you a real merge with delete handling. Land the raw delta, then merge on the business key with GLUE_FETCH_SQ deciding the winner within a batch, deleting where the status says deleted.
2. Keep an append-only history and build a current-state view. Cheaper to write, and it preserves an audit trail. It also means every downstream consumer has to use the view, and someone eventually will not.

I reach for Iceberg first on anything that feeds reporting, because the correctness lives in the table rather than in the discipline of whoever writes the next query. Append-only is the right call when you genuinely need history, or when the consumer is a downstream warehouse like Amazon Redshift that will do its own merge anyway.

Worth knowing: not every ODP provider emits deletes reliably. Some older extractors express them as reversal records rather than a delete flag. Confirm the behaviour for your specific entity with the functional owner before you build merge logic on an assumption.

Failure family three: pagination and partitioning do not compose

The connector’s default page size is 50,000 records, and you can raise it up to 500,000. Anything above that is ignored and the maximum is used instead. Raising the page size cuts the number of round trips, which is usually the dominant cost on a wide table across a slow link.

The catch is what your SAP Gateway will tolerate. A large page means a large response the ABAP stack has to assemble and hold, and Gateway or an intervening load balancer may cut it off before it completes. Symptoms are timeouts, connection resets, or a job that appears to paginate without ever finishing. Tune the page size with your Basis team, not in isolation, and treat it as a negotiation between round trips and response size rather than a value you set once.

On partitioning, there is a specific documented limit that catches people out: ODP entities are not compatible with record-based partitioning, because pagination is already handled through skip and delta tokens. For record-based partitioning, the default concurrency is forced to null regardless of what you asked for. So the obvious lever for making a slow ODP extraction faster is not available.

What you can still do on non-ODP OData entities is field-based partitioning, splitting a read across a bounded range:

• PARTITION_FIELD is the field to split on, typically a date.
• LOWER_BOUND is inclusive, UPPER_BOUND is exclusive. Getting that backwards is how you duplicate or drop a boundary day.
• NUM_PARTITIONS defaults to 1, so leaving it out means you did not partition at all.
• Where a limit and a partition are both applied, the limit wins. That is the usual cause of “I partitioned it and got fewer rows”.

For non-ODP entities you can also do timestamp-based incremental reads with filteringExpression, pointing at whatever last-changed field the entity exposes. Note the casing: that option is camelCase while most of the others are upper snake case, which trips people up when a filter appears to be ignored.

Failure family four: connection, auth and network

These fail early and loudly, which makes them the easy ones. The trap is that they fail during setup, when you have three people on a call and someone suggests loosening something to move on.

The connector supports two authentication types: CUSTOM, which is SAP basic authentication, and OAUTH2. The custom secret in AWS Secrets Manager takes this shape:

{
  "basicAuthUsername": "<your SAP username>",
  "basicAuthPassword": "<your SAP password>",
  "basicAuthDisableSSO": "<True/False>",
  "customAuthenticationType": "CustomBasicAuth"
}

For OAuth 2.0 the client secret goes into Secrets Manager under USER_MANAGED_CLIENT_APPLICATION_CLIENT_SECRET, and the client ID, authorisation code URL, token URL and scopes are supplied on the connection itself.

Basic auth is genuinely easier for a scheduled pipeline: no interactive redirect, no refresh token to keep alive, and rotation is a Secrets Manager operation. OAuth gets you short-lived tokens and revocation you can actually reason about, which is what a security review will want. Both are defensible. What is not defensible is one shared service user across every pipeline, for the delta token reasons above as much as the audit ones.

A few things that reliably waste a morning:

• The service path on the connection is the catalog service path, not the path to your object. The entity path goes in ENTITY_NAME on the job.
• The OData service has to be activated in the SAP system. Until it is, nothing you do on the AWS side matters.
• For a non-VPC connection, the application host URL must be reachable from the public internet. If your SAP system sits in a private network, configure VPC connectivity properly rather than opening it up.
• The Glue role needs Secrets Manager read plus the EC2 network interface permissions when a VPC is involved. A missing ec2:CreateNetworkInterface shows up as a connection failure that looks like a network problem.
• Private certificate authorities need the chain in a form Glue accepts. If you are wiring this up yourself, expect it to take a pass or two.

Choosing your path: Glue jobs, zero-ETL, or something else

The same connector is reachable through more than one product surface, and the choice is really about how much state you want to own.

• Glue Studio or script jobs. You control page size, filters, output format, transformation and scheduling, and you own delta token persistence. Pick this when you need real transformation in flight, or when the merge logic has to live next to the extraction.
• Glue zero-ETL integrations. AWS manages token state internally, runs the initial full load and then the incremental cycle, and preserves the token on failure so retries do not lose data. Less code, less control. Worth noting that once an incremental update field is chosen for a connection and entity, a later integration on the same pair cannot pick a different one.
• Amazon AppFlow. Simpler, flow-shaped, well suited to modest volumes and quick prototypes. Less appropriate once you need Spark-scale processing.
• Commercial replication tools. Fivetran, Airbyte and SAP-specialist vendors all play here. They cost money and add a vendor, and in exchange they absorb exactly the state management this post is about. If your team has no SAP depth, that trade is often correct.

On cost, Glue bills for the resources a job consumes for as long as it runs, so the levers are the same as any Spark workload: fewer round trips, less data pulled, fewer idle executors, and not re-reading history you already have. Rates change and vary by region, so model it against current pricing rather than a figure from a blog post. If your incremental run is pulling the same volume as your initial load, the delta mechanism is not working and the bill is the symptom.

Also worth writing back. The connector supports write operations to SAP OData, using INSERT and UPDATE, with ID_FIELD_NAMES required on update to identify the external ID field. Useful for enrichment loops, and something to treat with a good deal more caution than a read.

Troubleshooting

“Could not open data access via extraction API RODPS_REPL_ODP_OPEN”

Your token is invalid or expired. Find out why before you re-initialise, because re-initialising destroys the evidence. Check ODQMON for a recent initialisation on that collection and user, check whether another job or a test program shares the credentials, and check whether the request you wanted has aged out of retention. Then re-initialise deliberately and reconcile the gap.

The job succeeds but returns no rows

Most often a stale or wrong token, a filter predicate that matches nothing, or CDC never having been initialised so the connector has nothing to hand back. Run once without ENABLE_CDC against a small filtered slice and confirm you get data at all. That separates “no changes” from “not reading”.

Pagination never terminates

Usually the SAP side enforcing its own page limit under your requested size, so the connector keeps requesting more. Reduce the page size, verify the filter is actually being applied server side rather than after retrieval, and have Basis confirm what Gateway is capping.

Row counts drift over time

Almost always unhandled DML_STATUS = DELETED records, or a token invalidation that silently turned an incremental into a partial. Count distinct business keys in your target against a direct count in SAP for a bounded date range. Make that comparison a scheduled check, not something you run when someone complains.

Duplicates after a rerun

A delta was consumed, the write failed, and the token advanced anyway. Store the token only after the write commits, never before. If you are appending rather than merging, add a dedupe on business key ordered by GLUE_FETCH_SQ.

Common mistakes

• Sharing one SAP service user across multiple pipelines, which couples their delta tokens together.
• Automatic re-initialisation in an exception handler. It converts a loud failure into permanent silent drift.
• Writing the token-carrier row into the target table as if it were data.
• Appending CDC output and ignoring DML_STATUS.
• Persisting the token before the write succeeds.
• Assuming an entity is ODP-enabled because it looks analytical. Confirm it, because the incremental mechanism differs.
• Leaving test subscriptions from RODPS_REPL_TEST registered against the queue.
• Building the pipeline without ever asking Basis what the ODQ retention window is.

Best practices for SAP data integration with AWS Glue

• Treat the delta token as production state. Versioned storage, a documented recovery procedure, and a written-down answer to “how do we backfill”.
• One service user per pipeline, one connection per environment. Isolation is the cheapest fix for the worst failure.
• Land raw, merge second. Keep the untouched connector output so you can replay a merge without going back to SAP.
• Alert on absence, not just errors. Zero rows for several consecutive runs against a busy entity should page someone. Push Glue metrics into CloudWatch and forward them wherever you already look, whether that is Grafana Cloud, Datadog or your own Prometheus.
• Reconcile on a schedule against SAP for a bounded window, and record the result. Drift you can see is a bug; drift you cannot see is an incident.
• Start with a filtered slice and a small page size while you are building. Get it correct, then tune throughput.
• Write down what each field means while the functional owner is still on the call. SAP field names do not explain themselves later.

Frequently asked questions

Does the AWS Glue SAP OData connector fall under SAP Note 3255746?

The note restricts the RFC modules of the ODP Data Replication API. The Glue connector uses OData over HTTPS against an activated Gateway service, which is the interface SAP directs customers toward. Read the current revision of the note yourself and get it confirmed with your SAP account team before you rely on any summary, including this one.

Where should I store the delta token?

Anywhere durable and versioned that your job can write to transactionally: DynamoDB, Parameter Store, or S3 with versioning. What matters more than the store is the ordering. Write it only after the target write commits, and keep the previous value so you can roll back after a bad run.

Can I run two Glue jobs against the same SAP entity?

Yes, but give them separate SAP service users. A delta token is scoped to the service collection and the user, so two jobs on one user will invalidate each other’s tokens whenever either re-initialises.

Why do deleted SAP records still appear in my data lake?

Because the connector reports them rather than applying them. Deletes come back as rows with DML_STATUS set to DELETED. If your pipeline appends without a merge, those rows sit in the target looking like live data. Merge into Iceberg, or filter through a current-state view that every consumer actually uses.

What page size should I use?

The default is 50,000 and the ceiling is 500,000. Larger pages mean fewer round trips but bigger responses for SAP to assemble, and Gateway or a load balancer may cut them short. Start at the default, raise it while watching for timeouts, and agree the ceiling with your Basis team.

Can I partition an ODP extraction for speed?

Not with record-based partitioning. ODP entities handle pagination through skip and delta tokens, so concurrency is forced to null there regardless of your input. Field-based partitioning on a bounded range is available for non-ODP OData entities. For ODP, tune page size and narrow the entity instead.

Can Glue write data back into SAP?

Yes. The SAP OData connector supports insert and update operations, with the external ID field named through ID_FIELD_NAMES on updates. Test that against a sandbox client with a functional owner watching, because a bad write into an ERP system is a very different class of incident to a bad read.

The one thing worth remembering

SAP data integration with AWS Glue is not hard because the connector is difficult to configure. Configuration takes an afternoon. It is hard because the correctness of your pipeline depends on a token held in a queue on a system your team does not administer, subject to a retention policy you may not have been told about, and shared with every other subscriber using the same credentials.

Design for that. Isolate the users, persist the token deliberately, apply deletes rather than collecting them, and alert on silence as loudly as you alert on errors. Do that and the connector is a good piece of engineering that will run for a long time without your attention. Skip it and you get a job that stays green while the numbers quietly stop being true.

Need a hand with an SAP to AWS pipeline?

This is the kind of work I take on regularly. If you are standing one up, or you have one that has started drifting, I can help with:

• Designing delta token state management that survives failed runs, redeploys and concurrent pipelines.
• Building merge logic that applies DML_STATUS deletes correctly into Iceberg or Redshift instead of accumulating ghost rows.
• Diagnosing RODPS_REPL_ODP_OPEN failures, silent zero-row runs and row count drift, and working out how far back the damage goes.
• Setting up SAP OData connections properly: Secrets Manager, least-privilege IAM, VPC connectivity and private certificate authorities.
• Tuning page size, filters and job sizing so incremental runs cost what an incremental run should cost.
• Adding reconciliation and absence alerting so drift shows up in a dashboard rather than in a finance report.

Send me a job log, a connection config, or the row counts that stopped matching, and I will tell you what I think is going on.

The post SAP Data Integration with AWS Glue: The Failure Modes Nobody Demos appeared first on John Nessime.