<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>API Gateway | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/api-gateway/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/api-gateway/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 02 Nov 2026 07:03:28 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>API Gateway | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/api-gateway/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>AWS WAF Rate-Based Rules: Stop the Flood, Not Your Best Customer</title>
		<link>https://john-nessime.com/blog/devops/aws-waf-rate-based-rules/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 02 Nov 2026 07:03:28 +0000</pubDate>
				<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Web Security]]></category>
		<category><![CDATA[Amazon CloudFront]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Application Load Balancer]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS CLI]]></category>
		<category><![CDATA[AWS WAF]]></category>
		<category><![CDATA[Bot Mitigation]]></category>
		<category><![CDATA[Brute Force Protection]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[DDoS Protection]]></category>
		<category><![CDATA[HTTP 429]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[Throttling]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<category><![CDATA[X-Forwarded-For]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=1023</guid>

					<description><![CDATA[<p>Your rate limit worked, and it also blocked your biggest customer's office NAT. How to pick aggregation keys, scope rules per endpoint, return 429 instead of 403, and roll out AWS WAF rate-based rules in Count mode first.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-waf-rate-based-rules/">AWS WAF Rate-Based Rules: Stop the Flood, Not Your Best Customer</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads something like this: <em>&#8220;Your API started returning 403 to our integration around nine this morning. Nothing changed on our side.&#8221;</em> You check the application logs and there&#8217;s nothing there, because the requests never reached the application. They died at the edge, in a WAF rule somebody added last month to stop a credential-stuffing run on the login page.</p>



<p class="wp-block-paragraph">That rule worked. It also counted every request from the customer&#8217;s office NAT, their ERP connector and their warehouse scanners as one source, and at the start of the working day that one source crossed the line. The customer who sends you the most traffic is, almost by definition, the one most likely to look like an attacker to a counter that only sees IP addresses.</p>



<p class="wp-block-paragraph">This post is about building <strong>AWS WAF rate-based rules</strong> that catch floods, scrapers and brute-force attempts without throttling the people you actually want. I&#8217;ll go through the four ways these rules go wrong (the key, the scope, the response and the timing), a rollout order that keeps you out of trouble, and the troubleshooting steps for when a real user gets caught anyway.</p>



<h2 class="wp-block-heading">How AWS WAF rate-based rules actually count</h2>



<p class="wp-block-paragraph">A rate-based rule does three things. It decides which requests it cares about (the scope-down statement), it groups those requests into buckets (the aggregation key), and it counts each bucket over a sliding look-back window. When a bucket goes over your limit, the rule action applies to requests in that bucket until the rate falls back under.</p>



<p class="wp-block-paragraph">By default the bucket is the source IP address, and the window is five minutes. You can change both. AWS lets you pick a one, two, five or ten minute evaluation window, and you can aggregate on things other than IP: a forwarded IP header, a header value, a cookie, a query argument, the URI path, the HTTP method, an ASN, a JA3 or JA4 TLS fingerprint, or a label namespace set by an earlier rule. You can combine several of these into a composite key.</p>



<p class="wp-block-paragraph">Most bad outcomes come from one of those choices being made with the attacker&#8217;s traffic in mind and never checked against the customer&#8217;s.</p>



<h2 class="wp-block-heading">Failure family 1: the key groups the wrong people together</h2>



<h3 class="wp-block-heading">Shared egress IPs look like one very busy client</h3>



<p class="wp-block-paragraph">IP aggregation assumes one IP is roughly one client. That stopped being true a long time ago. A company office sits behind one NAT address. Mobile carriers put thousands of subscribers behind carrier-grade NAT. People browsing through a commercial VPN such as NordVPN or Surfshark share exit nodes with strangers. A B2B customer&#8217;s integration server, their staff browsers and their scheduled exports can all arrive from the same address.</p>



<p class="wp-block-paragraph">None of that is abuse, and it all lands in one bucket. A limit sized for &#8220;one heavy user&#8221; is really a limit on everyone behind that address, combined.</p>



<h3 class="wp-block-heading">The proxy in front of WAF hides the client entirely</h3>



<p class="wp-block-paragraph">This one is invisible until it isn&#8217;t. If your web ACL sits on an Application Load Balancer and CloudFront (or another CDN like Cloudflare) sits in front of that load balancer, the source IP that WAF sees on the ALB is the CDN&#8217;s edge address, not the visitor&#8217;s. A plain IP-aggregated rule is now counting &#8220;requests from this CloudFront edge location&#8221;. Under normal load that stays under the limit, so testing looks fine. Under a traffic spike, legitimate users sharing a busy edge get blocked together.</p>



<p class="wp-block-paragraph">There are two fixes, and they aren&#8217;t equal. The cleaner one is to attach the rate-based rule to a web ACL on the CloudFront distribution itself, where the source IP is the viewer. The other is forwarded IP aggregation on the ALB, which reads the client address from a header such as <code>X-Forwarded-For</code>. That brings its own problem.</p>



<h3 class="wp-block-heading">Forwarded IP aggregation trusts the part the client wrote</h3>



<p class="wp-block-paragraph">For rate-based rules, AWS WAF uses the <strong>first</strong> address in the forwarded header. Proxies append to <code>X-Forwarded-For</code>, so the first entry is whatever the client sent before any proxy touched the request. A client that sets its own <code>X-Forwarded-For</code> value, and rotates it on every request, lands in a new bucket every time. Your rate limit never fires for that client, while it keeps working perfectly for honest users who don&#8217;t forge headers.</p>



<p class="wp-block-paragraph">Two related details from the forwarded IP documentation matter here. If the header is missing, the rule doesn&#8217;t apply to that request at all, not even the fallback behavior. And the fallback (match or no match) only kicks in when the header is present but holds a malformed address. So forwarded IP aggregation is safe only when something upstream that you control overwrites the header rather than appending to it. If you can&#8217;t guarantee that, move the rule to the edge.</p>



<h3 class="wp-block-heading">Composite keys exclude requests that lack a component</h3>



<p class="wp-block-paragraph">Custom keys are how you stop punishing shared IPs. Aggregate on an API key header, a session cookie, or IP plus something else, and each customer gets their own bucket. The catch is in one line of the docs: every component in the key must be present, or the request isn&#8217;t evaluated by the rule at all.</p>



<p class="wp-block-paragraph">So a rule keyed on <code>x-api-key</code> rate-limits authenticated clients precisely and ignores anonymous traffic completely. The attacker who simply leaves the header off is invisible to it. You always need a second rule for the requests the first one can&#8217;t see, usually an IP-keyed rule scoped to requests without the header.</p>



<p class="wp-block-paragraph">Here&#8217;s the header-keyed rule. The limit is a placeholder; you&#8217;ll derive a real one from your own logs in a moment.</p>



<pre class="wp-block-code"><code>{
  "Name": "rate-per-api-key",
  "Priority": 20,
  "Statement": {
    "RateBasedStatement": {
      "Limit": 2000,
      "EvaluationWindowSec": 300,
      "AggregateKeyType": "CUSTOM_KEYS",
      "CustomKeys": [
        {
          "Header": {
            "Name": "x-api-key",
            "TextTransformations": [
              {
                "Priority": 0,
                "Type": "NONE"
              }
            ]
          }
        }
      ]
    }
  },
  "Action": {
    "Count": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "rate-per-api-key"
  }
}</code></pre>



<p class="wp-block-paragraph">Note the action is <code>Count</code>, not <code>Block</code>. More on why below. Also remember that a header value is only as trustworthy as your API validates it to be. A made-up key still gets its own bucket in WAF, so the backend has to reject it.</p>



<h2 class="wp-block-heading">Failure family 2: one limit for traffic with very different shapes</h2>



<p class="wp-block-paragraph">A single site-wide rule applies one number to everything. But a page load might fetch dozens of assets, an SPA can fire a burst of API calls on first render, and a login form should see a handful of posts per person per session. Any limit loose enough for the first case is useless for the last one.</p>



<p class="wp-block-paragraph">The answer is several narrow rules instead of one wide one, each with a scope-down statement:</p>



<ul class="wp-block-list">

<li><strong>Login, signup, password reset and OTP endpoints:</strong> tight limit, IP or forwarded IP key. This is where credential stuffing and SMS pumping live.</li>


<li><strong>Search, pricing and other expensive endpoints:</strong> moderate limit. Scrapers go here, and each request costs you real backend work.</li>


<li><strong>Authenticated API:</strong> key on the credential, sized from actual per-customer usage.</li>


<li><strong>Everything else:</strong> one generous catch-all on IP, meant for floods, not for shaping.</li>

</ul>



<p class="wp-block-paragraph">Static assets are often best left out of rate counting entirely, especially when a CDN is serving them from cache and they never reach anything expensive. Counting them mainly inflates the numbers for legitimate browsers.</p>



<p class="wp-block-paragraph">A scoped login rule looks like this:</p>



<pre class="wp-block-code"><code>{
  "Name": "rate-login",
  "Priority": 10,
  "Statement": {
    "RateBasedStatement": {
      "Limit": 100,
      "EvaluationWindowSec": 300,
      "AggregateKeyType": "IP",
      "ScopeDownStatement": {
        "ByteMatchStatement": {
          "SearchString": "/login",
          "FieldToMatch": {
            "UriPath": {}
          },
          "TextTransformations": [
            {
              "Priority": 0,
              "Type": "LOWERCASE"
            }
          ],
          "PositionalConstraint": "STARTS_WITH"
        }
      }
    }
  },
  "Action": {
    "Count": {}
  },
  "VisibilityConfig": {
    "SampledRequestsEnabled": true,
    "CloudWatchMetricsEnabled": true,
    "MetricName": "rate-login"
  }
}</code></pre>



<p class="wp-block-paragraph">The <code>LOWERCASE</code> transformation matters more than it looks. Without it, <code>/Login</code> may route to the same handler in your app but fall outside the scope-down match.</p>



<h3 class="wp-block-heading">Exempting a partner: Allow is heavier than it looks</h3>



<p class="wp-block-paragraph">When a known partner keeps tripping a limit, the tempting fix is an IP set rule with an <code>Allow</code> action placed above the rate rule. It works, but <code>Allow</code> is terminating. That partner&#8217;s traffic now skips every rule below it, including your managed rule groups for injection and known bad inputs. If their integration server is ever compromised, you&#8217;ve pre-approved the attacker.</p>



<p class="wp-block-paragraph">The narrower fix is to exclude the IP set inside the rate rule&#8217;s own scope-down, with a <code>NotStatement</code> around an IP set reference, combined with your path match in an <code>AndStatement</code>. The partner is exempt from counting and nothing else.</p>



<h2 class="wp-block-heading">Failure family 3: the response breaks the client instead of slowing it</h2>



<p class="wp-block-paragraph">The default Block action returns a 403. To a well-behaved API client, 403 means &#8220;you&#8217;re not allowed&#8221;, and many clients treat that as permanent: they stop, raise an alert, or worse, retry immediately in a loop that keeps them over the limit.</p>



<p class="wp-block-paragraph">A rate limit should say what it is. AWS WAF supports a custom response on Block, so you can return 429 with a <code>Retry-After</code> header, which HTTP clients and SDKs generally understand as &#8220;back off and try later&#8221;:</p>



<pre class="wp-block-code"><code>{
  "Action": {
    "Block": {
      "CustomResponse": {
        "ResponseCode": 429,
        "ResponseHeaders": [
          {
            "Name": "Retry-After",
            "Value": "60"
          }
        ]
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">You can also reference a custom response body defined on the web ACL, such as a small JSON error, so API clients get something parseable.</p>



<p class="wp-block-paragraph">What about Challenge and CAPTCHA? They&#8217;re genuinely useful on human-facing pages, because a real browser passes a silent challenge and a basic script doesn&#8217;t. On API endpoints they&#8217;re a trap. A mobile app, a server-to-server integration or a webhook sender can&#8217;t complete a browser challenge, so for them Challenge behaves like a block with a confusing response. Keep those actions to routes where you know a browser is on the other end.</p>



<h2 class="wp-block-heading">Failure family 4: expecting precision the service doesn&#8217;t promise</h2>



<p class="wp-block-paragraph">AWS is explicit that WAF rate limiting is meant to control high request rates, not to enforce exact quotas. It estimates the rate with an algorithm that weights recent requests more heavily, so it acts <em>near</em> your limit rather than exactly on it. There&#8217;s also lag both ways: a burst can run over the limit for a while before it&#8217;s caught, and a client that has slowed down can stay limited for a short time after. AWS&#8217;s own caveats page gives the typical delay as under 30 seconds, with longer possible.</p>



<p class="wp-block-paragraph">Two consequences follow:</p>



<ul class="wp-block-list">

<li><strong>Don&#8217;t use WAF to enforce billing quotas or contractual API limits.</strong> If &#8220;1,000 calls per customer per hour&#8221; is in a contract, count it in your application, where you control the number exactly and can show it to the customer. API Gateway usage plans help, but AWS describes their throttling and quotas as best effort too.</li>


<li><strong>Editing a live rule resets its counters.</strong> Changing the limit, window, key, forwarded IP config or scope-down resets the rule&#8217;s counts and can pause its rate limiting for up to a minute. Tuning during an active attack gives the attacker a free minute each time.</li>

</ul>



<h2 class="wp-block-heading">Rolling out AWS WAF rate-based rules without collateral damage</h2>



<p class="wp-block-paragraph">The order below is the one I reach for first, because each step produces the evidence the next step needs.</p>



<ol class="wp-block-list">

<li><strong>Turn on WAF logging before anything else.</strong> Send logs to CloudWatch Logs, S3 or Firehose. The destination name has to start with <code>aws-waf-logs-</code>. Without logs you&#8217;re tuning blind. If you want a longer retention and audit setup around them, that&#8217;s its own job: <a href="/aws/logging-audit/">AWS logging and audit trails</a>.</li>


<li><strong>Measure real traffic per key.</strong> Look at the busiest sources per five-minute bucket over a normal week, including your peak hour and any batch windows. The query further down does this in CloudWatch Logs Insights.</li>


<li><strong>Deploy every new rate rule in Count.</strong> Count mode evaluates the rule and records matches without affecting the request. Give it at least one full business cycle, ideally including a month-end or a marketing send.</li>


<li><strong>Review what would have been blocked.</strong> For each counted match, ask whether it was an attacker, a customer, a partner, a crawler you want, or your own monitoring. The last one catches people out: uptime checks and synthetic tests often run from a small pool of addresses.</li>


<li><strong>Set the limit above the busiest legitimate bucket, with headroom.</strong> The goal is to sit clearly between your heaviest real client and the flood, not to hug the average.</li>


<li><strong>Switch to Block with a 429 response,</strong> and add an alarm on the rule&#8217;s blocked request metric so a sudden jump reaches a human.</li>

</ol>



<p class="wp-block-paragraph">Here&#8217;s the Logs Insights query for step two, run against the WAF log group. It shows the heaviest client IPs per five-minute bin:</p>



<pre class="wp-block-code"><code>fields httpRequest.clientIp
| stats count(*) as requests by httpRequest.clientIp, bin(5m)
| sort requests desc
| limit 25</code></pre>



<p class="wp-block-paragraph">Swap <code>httpRequest.clientIp</code> for <code>httpRequest.uri</code> to find the hottest paths, or filter on a URI to size a scoped rule. At high volume, logs in S3 queried with Athena are usually cheaper to explore than repeated Logs Insights scans.</p>



<h2 class="wp-block-heading">Troubleshooting a customer who got rate limited</h2>



<p class="wp-block-paragraph">When the ticket arrives, work through it in this order:</p>



<ol class="wp-block-list">

<li><strong>Confirm it was WAF.</strong> A WAF block never reaches your application logs. Search WAF logs for the customer&#8217;s IP or API key and check <code>terminatingRuleId</code> and <code>action</code>.</li>


<li><strong>List what the rule is currently limiting.</strong> For rules aggregated only on IP or forwarded IP, AWS WAF can return the addresses currently being limited. The command is shown below. It won&#8217;t work for custom-key rules; for those, the logs are your only view.</li>


<li><strong>Check which address WAF saw.</strong> If it&#8217;s a CloudFront or CDN range, your rule is counting the proxy. If it&#8217;s the customer&#8217;s office NAT, the limit is fine for a person and wrong for an organization.</li>


<li><strong>Check for traffic you didn&#8217;t expect.</strong> A retry loop, a misconfigured cron job or a client polling far more often than it needs to will look exactly like abuse. Sometimes the fix is on their side, and the logs give you the evidence to say so politely.</li>


<li><strong>Fix the scope or key, not just the number.</strong> Raising the limit for everyone to fit one heavy customer weakens the rule for every attacker too.</li>

</ol>



<pre class="wp-block-code"><code>aws wafv2 get-rate-based-statement-managed-keys 
  --scope REGIONAL 
  --region eu-west-1 
  --web-acl-name my-web-acl 
  --web-acl-id 11111111-2222-3333-4444-555555555555 
  --rule-name rate-login</code></pre>



<p class="wp-block-paragraph">For a CloudFront web ACL, use <code>--scope CLOUDFRONT</code> with <code>--region us-east-1</code>. If the rule sits inside a rule group, add <code>--rule-group-rule-name</code> with the name of the rule group reference in your web ACL. A single rate-based rule instance limits at most 10,000 addresses at once; beyond that, the highest-rate addresses are the ones limited.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">

<li>Putting an IP-keyed rate rule on an ALB behind a CDN and never checking which IP WAF sees.</li>


<li>Trusting the first address in <code>X-Forwarded-For</code> when nothing you control overwrites it.</li>


<li>Keying on a header or cookie with no companion rule for requests that don&#8217;t carry it.</li>


<li>One site-wide limit instead of scoped rules for login, expensive endpoints and the API.</li>


<li>Going straight to Block without a Count period on real traffic.</li>


<li>Returning 403 for a rate limit, or Challenge on endpoints non-browser clients call.</li>


<li>Exempting a partner with Allow and silently removing every protection below it.</li>


<li>Tweaking a live rule repeatedly during an incident and resetting its counters each time.</li>

</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">

<li>Put IP-based rate rules on the web ACL closest to the real client, usually the CloudFront distribution.</li>


<li>Size limits from logged peak traffic per key, not from guesses about &#8220;normal&#8221; users.</li>


<li>Alarm on sudden changes in blocked requests per rule, in CloudWatch or whatever monitoring stack you already watch, such as Grafana or Datadog. <a href="/aws/monitoring-alerting/">Alerts that fire only when customers are affected</a> is the goal.</li>


<li>Keep contractual quotas in the application; use WAF for floods and abuse.</li>


<li>Consider AWS Shield Advanced only if you actually need its DDoS response support and cost protection; rate-based rules alone handle a lot of ordinary abuse.</li>

</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">What is a rate-based rule in AWS WAF?</h3>



<p class="wp-block-paragraph">It&#8217;s a rule that counts requests per aggregation key (by default, per source IP) over a look-back window and applies an action, such as Block or Count, to keys that go over a limit you set. It&#8217;s aimed at floods, brute force and scraping rather than exact quota enforcement.</p>



<h3 class="wp-block-heading">Why is AWS WAF blocking legitimate users?</h3>



<p class="wp-block-paragraph">Usually because many real users share one key: an office NAT, carrier-grade NAT, a VPN exit node, or a CDN or load balancer whose address WAF sees instead of the client&#8217;s. Check which IP appears in the WAF log for the blocked request.</p>



<h3 class="wp-block-heading">Should I use IP or forwarded IP aggregation behind CloudFront?</h3>



<p class="wp-block-paragraph">Where you can, put the rule on the CloudFront web ACL and use plain IP aggregation, since WAF sees the viewer&#8217;s address there. Forwarded IP on an ALB uses the first address in the header, which the client can forge unless something you control overwrites it.</p>



<h3 class="wp-block-heading">How do I return 429 instead of 403 from AWS WAF?</h3>



<p class="wp-block-paragraph">Use a Block action with a custom response: set the response code to 429 and add a <code>Retry-After</code> response header. You can also attach a custom response body defined on the web ACL.</p>



<h3 class="wp-block-heading">Why didn&#8217;t my rate-based rule block at exactly the limit?</h3>



<p class="wp-block-paragraph">AWS WAF estimates request rates and doesn&#8217;t guarantee an exact limit match. There&#8217;s also a detection delay, and editing the rule resets its counts. For exact per-client quotas, count in the application.</p>



<h3 class="wp-block-heading">Can I see which clients a rate-based rule is currently blocking?</h3>



<p class="wp-block-paragraph">Yes, for rules that aggregate only on IP or forwarded IP, via <code>GetRateBasedStatementManagedKeys</code>. Rules with custom keys don&#8217;t support that call, so use WAF logs instead.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The one thing worth remembering about AWS WAF rate-based rules: the limit is the easy part. What decides if you block the attacker or your best customer is <em>what counts as one client</em>, and that choice is made by the aggregation key, the scope-down and where the web ACL sits. Log first, count before you block, give each kind of traffic its own rule, and answer with a 429 that tells well-behaved clients how to back off.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help tuning AWS WAF rate limits?</h2>



<p class="wp-block-paragraph">I help teams put rate limiting in front of AWS workloads that stops abuse and leaves real traffic alone. That usually covers:</p>



<ul class="wp-block-list">

<li>Reviewing an existing web ACL to find rules that count proxies, shared NATs or spoofable headers as single clients.</li>


<li>Moving IP-based rate rules to the right layer when CloudFront or another CDN sits in front of an ALB or API Gateway.</li>


<li>Designing scoped rules for login, signup, OTP and expensive endpoints, with composite keys for authenticated APIs.</li>


<li>Setting up WAF logging and the queries that size limits from your real peak traffic.</li>


<li>Running new rules in Count mode, reviewing what they&#8217;d catch, and switching to Block with 429 responses and alarms.</li>


<li>Untangling a live incident where customers or partners are being rate limited and you need to know why.</li>

</ul>



<p class="wp-block-paragraph">If a customer is getting blocked right now, send me the rule JSON and one redacted WAF log entry for the blocked request. That&#8217;s usually enough to see which of the four failure families you&#8217;re in.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://john-nessime.com/aws/waf-protection/">See the AWS WAF protection service</a></div>



<div class="wp-block-button is-style-outline is-style-outline--1"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-waf-rate-based-rules/">AWS WAF Rate-Based Rules: Stop the Flood, Not Your Best Customer</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>ALB vs API Gateway: Pick the Limits You Can Live With</title>
		<link>https://john-nessime.com/blog/devops/alb-vs-api-gateway/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 25 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Networking]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Application Load Balancer]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[JWKS]]></category>
		<category><![CDATA[JWT Validation]]></category>
		<category><![CDATA[Listener Rules]]></category>
		<category><![CDATA[Load Balancing]]></category>
		<category><![CDATA[OIDC]]></category>
		<category><![CDATA[Private Integration]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Serverless]]></category>
		<category><![CDATA[Web Application Firewall]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=550</guid>

					<description><![CDATA[<p>Choosing between an Application Load Balancer and API Gateway is not a feature comparison. It is a choice of hard limits, a billing curve and an authentication story. Here is how each one bills you, which constraints surface months later, what ALB's native JWT validation changes, and a decision procedure you can run in ten minutes.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said the reporting endpoint returns a 504 after about thirty seconds. The application log said the job finished in forty-one seconds and wrote a perfectly good response to a client that had already hung up. Nothing crashed, nothing scaled wrong, nobody deployed anything that week.</p>



<p class="wp-block-paragraph">The integration timeout had been sitting in front of that API since the day someone created it. It only became a bug when a customer&#8217;s dataset got big enough to cross it.</p>



<p class="wp-block-paragraph">That is the real shape of the ALB vs API Gateway decision. You&#8217;re not picking a feature list. You&#8217;re picking a set of hard limits, a billing curve and an authentication story, then living inside them. The failure mode that bites isn&#8217;t a bad Friday, it&#8217;s finding the constraint months later, once the thing in front of your app has become load-bearing.</p>



<p class="wp-block-paragraph">Below: how each one bills you, which limits surface late, what changed recently on the load balancer side, and a decision procedure.</p>



<h2 class="wp-block-heading">The old rule of thumb is shakier than it looks</h2>



<p class="wp-block-paragraph">The advice you find everywhere is: containers and instances behind an Application Load Balancer, serverless behind API Gateway. Wrong often enough to cost you. ALB takes Lambda functions as targets. API Gateway reaches containers through private integrations with an NLB, an ALB, or Cloud Map. Compute type barely narrows the field.</p>



<p class="wp-block-paragraph">What separates them is who&#8217;s calling and what needs enforcing before a request reaches your code. A browser session against your own web app is a different problem from a partner&#8217;s server calling a documented endpoint with a key you issued and a quota you promised. One needs routing and a session. The other needs a contract.</p>



<h2 class="wp-block-heading">Application Load Balancer: routing you own the position of</h2>



<p class="wp-block-paragraph">ALB is a Layer 7 router inside your VPC. You give it subnets and security groups, attach listeners, and write rules that match on host, path, header, query string, method or source IP, then forward, redirect, return a fixed response, or authenticate.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>Long requests.</strong> The idle timeout is an attribute you set, defaulting to 60 seconds. Report generation and slow third-party calls become a number in Terraform instead of an architecture problem.</li>

<li><strong>Big bodies and streaming.</strong> Against instance, IP and container targets there&#8217;s no small payload ceiling to design around, and WebSocket upgrades pass through.</li>

<li><strong>Browser authentication.</strong> An HTTPS listener rule can run an <code>authenticate-oidc</code> or <code>authenticate-cognito</code> action before forwarding. The load balancer does the redirect dance, sets a session cookie, and hands your target the verified claims in a header. Your app reads a header instead of implementing OAuth.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>No per-client rate limiting.</strong> No API keys with quotas attached. Rate limiting means AWS WAF rate-based rules in front, counting requests per source over a window. Blunter than a per-consumer plan, and a separate bill.</li>

<li><strong>No request validation or response caching.</strong> A malformed body reaches your handler and burns your compute. A cacheable GET is recomputed unless you put a CDN in front.</li>

<li><strong>Lambda targets are constrained.</strong> Request body capped at 1 MB, response JSON capped at 1 MB, WebSocket upgrades rejected with a 400. If your function returns a large document, that ceiling arrives as a 502 in production, not an error at deploy time.</li>

<li><strong>You pay for it while it sleeps.</strong> An hourly charge per load balancer whether or not a request arrives. Three environments, three floors.</li>
</ul>



<h2 class="wp-block-heading">Amazon API Gateway: the contract layer</h2>



<p class="wp-block-paragraph">API Gateway isn&#8217;t a load balancer with extra features. It&#8217;s a place to define what your API promises: which routes exist, who may call them, how often, and in what shape. That framing explains most of its trade-offs.</p>



<p class="wp-block-paragraph">Choosing it is really two decisions, because REST APIs and HTTP APIs are different products under one name.</p>



<ul class="wp-block-list">
<li><strong>REST APIs</strong> carry the full management surface: API keys, usage plans, per-client throttling, request validation, response caching, AWS WAF integration, the private endpoint type, execution logs and X-Ray tracing.</li>

<li><strong>HTTP APIs</strong> are the stripped-down, cheaper option. They add a built-in JWT authorizer, automatic deployments and Cloud Map private integrations. They offer none of the six management features listed above.</li>
</ul>



<p class="wp-block-paragraph">That second line is the one people get wrong. HTTP APIs aren&#8217;t &#8220;REST APIs but cheaper&#8221;, they&#8217;re a different product that happens to serve HTTP. Pick them to save money, then discover you need metered access, and you&#8217;re rebuilding usage plans in your own code.</p>



<h3 class="wp-block-heading">Where it wins</h3>



<ul class="wp-block-list">
<li><strong>You&#8217;re selling or governing access.</strong> API keys tied to usage plans, on REST APIs, are the only native answer to &#8220;customer B must not burn customer A&#8217;s capacity&#8221;.</li>

<li><strong>Rejecting bad requests before they cost you.</strong> Request validation against a model means a malformed payload never invokes your function. On per-invocation compute that&#8217;s a security boundary and a cost lever at once.</li>

<li><strong>Zero idle cost.</strong> No hourly floor, so a dev stage nobody calls is free. For genuinely spiky traffic this is often the whole argument, and there&#8217;s no network to design either.</li>
</ul>



<h3 class="wp-block-heading">Where it doesn&#8217;t</h3>



<ul class="wp-block-list">
<li><strong>The integration timeout.</strong> The default is 29 seconds. It can be raised, but AWS notes that going beyond 29 seconds may require reducing your Region-level throttle quota. You&#8217;re buying request duration with request rate.</li>

<li><strong>Payload ceilings.</strong> There&#8217;s a documented maximum payload size, well below what a file upload service wants. Large objects want presigned S3 URLs either way, but here the gateway forces that conversation early.</li>

<li><strong>Cost scales with request count.</strong> Per request plus data transfer. A health check hammered by a monitoring tool shows up directly on the line item.</li>

<li><strong>The account throttle is shared.</strong> The default steady-state rate and burst quota applies per Region across your APIs in that account. Raisable, but a shared ceiling by default rather than a per-API one.</li>
</ul>



<h2 class="wp-block-heading">What changed: the load balancer validates JWTs itself now</h2>



<p class="wp-block-paragraph">For years the clean argument for API Gateway on machine-to-machine APIs was token validation. If a partner&#8217;s service calls you with a bearer token, someone has to check signature, issuer and expiry, and you&#8217;d rather that not be a library duplicated in every service.</p>



<p class="wp-block-paragraph">ALB does this natively now. A rule on an HTTPS listener can carry a <code>jwt-validation</code> action that runs before any routing action. The load balancer fetches signing keys from a JWKS endpoint you configure, checks the signature, and requires <code>iss</code> and <code>exp</code>. It also validates <code>nbf</code> and <code>iat</code> when present, and you can configure up to ten additional claims. Valid tokens forward unchanged. Invalid ones never reach your VPC.</p>



<p class="wp-block-paragraph">This isn&#8217;t <code>authenticate-oidc</code>, which is built for humans in browsers and works by redirecting. JWT validation does no redirect. It inspects the token and returns a yes or no, which is what service-to-service traffic needs. Validation action first, routing action second, and order matters:</p>



<pre class="wp-block-code"><code>aws elbv2 create-rule 
  --listener-arn "$LISTENER_ARN" 
  --priority 10 
  --conditions Field=path-pattern,Values="/api/*" 
  --actions '[
    {
      "Type": "jwt-validation",
      "JwtValidationConfig": {
        "JwksEndpoint": "https://issuer.example.com/.well-known/jwks.json",
        "Issuer": "https://issuer.example.com"
      },
      "Order": 1
    },
    {
      "Type": "forward",
      "TargetGroupArn": "$TARGET_GROUP_ARN",
      "Order": 2
    }
  ]'</code></pre>



<p class="wp-block-paragraph">Two constraints first. Only RS256 is supported, and the JWKS endpoint has a maximum response size and key count. A provider publishing a large key set fails that check, and the failure mode is that requests stop reaching your targets. Fetch your provider&#8217;s JWKS document and look at it before committing.</p>



<p class="wp-block-paragraph">So &#8220;we need token validation at the edge&#8221; is no longer by itself a reason to reach for API Gateway. What the gateway still owns is everything downstream of that yes or no: keys, quotas, per-consumer throttles, shape validation, caching.</p>



<h2 class="wp-block-heading">How each one actually bills you</h2>



<p class="wp-block-paragraph">This is where the argument usually goes wrong, because people compare unit prices instead of curve shapes.</p>



<p class="wp-block-paragraph">API Gateway is a slope with no floor: per request, plus data out. Zero traffic costs nothing, ten times the traffic costs roughly ten times as much.</p>



<p class="wp-block-paragraph">ALB is a floor plus a stranger slope. An hourly rate per load balancer, plus capacity units measured across four dimensions at once:</p>



<ul class="wp-block-list">
<li>New connections per second</li>

<li>Active connections per minute</li>

<li>Processed bytes per hour</li>

<li>Rule evaluations per second, where the first ten processed rules per request are free</li>
</ul>



<p class="wp-block-paragraph">You&#8217;re charged on the highest of the four in a given hour, not the sum. That mechanic explains most surprises. Clients that open a fresh connection per call can bill you on new connections while your bandwidth graph looks flat. A listener with dozens of path rules can bill you on rule evaluations while everything else idles.</p>



<p class="wp-block-paragraph">Two details worth knowing: Lambda targets get a smaller processed-bytes allowance per capacity unit than instance, container and IP targets, so identical traffic consumes more units. Enabling mutual TLS reduces the active-connections allowance per unit.</p>



<p class="wp-block-paragraph">Before modelling anything, look at what you already consume:</p>



<pre class="wp-block-code"><code>aws cloudwatch get-metric-statistics 
  --namespace AWS/ApplicationELB 
  --metric-name ConsumedLCUs 
  --dimensions Name=LoadBalancer,Value=app/my-alb/0123456789abcdef 
  --start-time "$(date -u -d '7 days ago' +%Y-%m-%dT%H:%M:%SZ)" 
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%SZ)" 
  --period 3600 
  --statistics Sum Maximum</code></pre>



<p class="wp-block-paragraph">Run that against a representative week and you have the real input. For API Gateway, the equivalent is request count per stage. Across several accounts, a cost visibility tool like Vantage or CloudZero attributes this per environment faster than Cost Explorer, though disciplined tagging gets you most of the way for free.</p>



<p class="wp-block-paragraph">Rough shape: low steady traffic favours API Gateway because you skip the floor. High-volume, byte-heavy, long-lived-connection traffic favours ALB because the per-request slope disappears. Chatty traffic with tiny payloads is where the curves cross, and that one is worth calculating.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">A decision procedure for ALB vs API Gateway</h2>



<p class="wp-block-paragraph">Run these in order, stop at the first clear answer.</p>



<ol class="wp-block-list">
<li><strong>Does any request legitimately run past about thirty seconds?</strong> If yes and you can&#8217;t make it asynchronous, ALB. Raising the gateway timeout is possible but trades against your regional throttle quota.</li>

<li><strong>Do you need per-consumer quotas or metered access?</strong> If yes, API Gateway REST APIs. Nothing else here gives you keys and usage plans natively.</li>

<li><strong>Do requests or responses carry large payloads?</strong> If yes, ALB with instance or container targets. Both options constrain size, but the Lambda-target and gateway ceilings arrive sooner.</li>

<li><strong>Is traffic occasional, spiky, or mostly zero?</strong> If yes, API Gateway. An hourly floor for a few thousand requests a day is real cost with no matching benefit.</li>

<li><strong>Is the caller a browser session against your own app?</strong> If yes, ALB, with an authenticate action if you need login. The gateway&#8217;s management features are dead weight here.</li>

<li><strong>Is the caller another service with a bearer token and nothing more?</strong> Either works now. Decide on cost curve and payload shape, because JWT validation stopped being the tiebreaker.</li>
</ol>



<p class="wp-block-paragraph">Still torn? Choose the limits you&#8217;d rather explain to a customer in a year. Usually ALB for applications, API Gateway for products. Whichever you pick, alarm on the constraint rather than the symptom: integration latency approaching the timeout, or consumed capacity units trending up. Grafana Cloud or plain CloudWatch alarms both do the job. Then send a payload just over your ceiling in staging and confirm what the client actually receives.</p>



<h2 class="wp-block-heading">Arguments that don&#8217;t survive contact</h2>



<h3 class="wp-block-heading">&#8220;It&#8217;s serverless, so it&#8217;s cheaper&#8221;</h3>



<p class="wp-block-paragraph">Per-request billing is cheaper at low volume and more expensive at high volume. Arithmetic, not philosophy. The crossover depends on your request size and connection reuse, which is why the capacity-unit metric matters more than anyone&#8217;s price table.</p>



<h3 class="wp-block-heading">&#8220;We&#8217;ll put API Gateway in front of it later&#8221;</h3>



<p class="wp-block-paragraph">You can, through a private integration, and it&#8217;s a legitimate pattern. But &#8220;later&#8221; means a second hop, a second bill, a second set of timeouts and a second place to look during an incident. Adding it deliberately is fine. Assuming it&#8217;s free is not.</p>



<h3 class="wp-block-heading">&#8220;ALB has no rate limiting, so it&#8217;s insecure&#8221;</h3>



<p class="wp-block-paragraph">ALB has no <em>native</em> rate limiting, but WAF rate-based rules in front of it are a real control, and a source-based limit is often closer to what you wanted than a per-key quota. Cloudflare in front of the load balancer is another common answer when you already want its caching and bot controls. What ALB genuinely lacks is the per-consumer contract, which matters when consumers are customers and doesn&#8217;t when they&#8217;re your own front end.</p>



<p class="wp-block-paragraph">Check the pairing before you rely on it, though: WAF associates with ALB and with API Gateway REST APIs, but not with HTTP APIs. And the amount of request body WAF reads is configurable for some resource types and fixed for the load balancer. Anything past that window isn&#8217;t scanned.</p>



<h3 class="wp-block-heading">&#8220;The front door has to be an AWS service&#8221;</h3>



<p class="wp-block-paragraph">For one service with predictable traffic, a plain VPS running Nginx or Caddy does the same job with far less surface. Providers like Contabo or InterServer are reasonable homes for a small internal API. It won&#8217;t scale into a multi-service platform and it moves patching and certificate renewal onto you, but &#8220;which AWS service&#8221; is sometimes the wrong question.</p>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I use API Gateway and ALB together?</h3>



<p class="wp-block-paragraph">Yes. API Gateway supports private integrations with an Application Load Balancer, so the gateway handles keys, quotas and validation while the load balancer routes inside the VPC. Sound pattern for a public product API in front of an existing internal platform. It also doubles the hops, the bills and the places a request can die, so use it when you need both jobs done.</p>



<h3 class="wp-block-heading">Which is cheaper, ALB or API Gateway?</h3>



<p class="wp-block-paragraph">Neither, universally. API Gateway has no idle cost and charges per request, so it wins at low or intermittent volume. ALB charges an hourly floor plus capacity units billed on the highest of four dimensions, so it wins once the per-request slope dominates. Pull your consumed capacity units and request counts and do the arithmetic on your own traffic.</p>



<h3 class="wp-block-heading">Can ALB do rate limiting?</h3>



<p class="wp-block-paragraph">Not natively. You get it by associating AWS WAF with the load balancer and using rate-based rules, which limit by source over a time window. If you need per-key quotas, that&#8217;s an API Gateway REST API usage plan, or logic you write yourself.</p>



<h3 class="wp-block-heading">Does ALB support JWT authentication now?</h3>



<p class="wp-block-paragraph">Yes, through a <code>jwt-validation</code> action on an HTTPS listener rule. It verifies the signature against a JWKS endpoint and checks issuer and expiry, plus optional additional claims. Note the RS256-only restriction and the JWKS size and key count limits before designing around it.</p>



<h3 class="wp-block-heading">Should I use HTTP APIs or REST APIs?</h3>



<p class="wp-block-paragraph">REST APIs if you need any of: API keys, usage plans, per-client throttling, request validation, response caching, WAF integration, or a private endpoint type. HTTP APIs when you need none of those and want the built-in JWT authorizer at a lower price. Migrating between them isn&#8217;t a config flag, so decide before you publish a URL.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">ALB vs API Gateway isn&#8217;t a question about serverless versus containers, and it stopped being a question about token validation. It&#8217;s a question about whether the thing in front of your application needs to enforce a contract with people outside your organisation.</p>



<p class="wp-block-paragraph">If it does, take the gateway and accept its timeout and payload ceilings as the price of keys, quotas and validation. If it doesn&#8217;t, take the load balancer and spend the saved effort on the network and the health checks. Either way, write down the three numbers that will eventually bite: timeout, payload ceiling, throttle quota. The 504 in that opening ticket wasn&#8217;t a bug. It was a decision nobody had written down.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second opinion on your front door?</h2>



<p class="wp-block-paragraph">Cheap to get right at the start, expensive to revisit. If you&#8217;re staring at this now, or already living with the wrong answer:</p>



<ul class="wp-block-list">
<li>Modelling ALB capacity units against API Gateway request counts on your actual traffic, so the comparison uses your numbers</li>

<li>Auditing an existing gateway or load balancer for the limits you&#8217;re closest to hitting: timeouts, payload ceilings, throttle quotas</li>

<li>Moving browser authentication off your application onto an ALB authenticate rule, or setting up JWT validation at the listener for service-to-service traffic</li>

<li>Designing the gateway plus private integration pattern when you need both layers, including where to terminate TLS and how to preserve client IPs</li>

<li>Building the Terraform or OpenTofu modules for either option, with listener rules, target groups, health checks and alarms in one place</li>

<li>Setting up alarms that fire on approaching limits rather than the 5XX that follows</li>
</ul>



<p class="wp-block-paragraph">Send me a listener rule dump, a stage configuration, or a week of consumed capacity unit metrics and I&#8217;ll tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/alb-vs-api-gateway/">ALB vs API Gateway: Pick the Limits You Can Live With</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</title>
		<link>https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 18:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[AgentCore]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Cognito]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[External Client Apps]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[Identity Propagation]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[OAuth]]></category>
		<category><![CDATA[Rate Limiting]]></category>
		<category><![CDATA[REST API]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=211</guid>

					<description><![CDATA[<p>The demo works, then someone sees records they shouldn't. Nothing errors. Here's how to build a Salesforce AI assistant with Amazon Bedrock that respects your sharing model: which doors AWS and Salesforce just closed, why identity propagation is the failure mode that bites, and how to shape the tool surface so the agent can't wander.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The demo went fine. Someone from the support team asked the assistant which accounts looked at risk this quarter, and it answered. Good answer, too. Right opportunities, right open cases, sensible summary.</p>



<p class="wp-block-paragraph">Then somebody noticed that the person who asked couldn&#8217;t see half those opportunities in the Salesforce UI. Different role, different branch of the hierarchy. They saw them through the assistant.</p>



<p class="wp-block-paragraph">Nothing errored. No 403, no exception in the logs, no alert. The integration user had access, so the API call succeeded, so the model got the records, so it summarised them. Every layer did exactly what it was configured to do. That is the failure mode that should worry you most when you build a Salesforce AI assistant with Amazon Bedrock: it is silent, it is correct-looking, and you will not find it in a load test.</p>



<p class="wp-block-paragraph">This post covers what actually goes wrong when you wire a large language model into a CRM: which door AWS closed on the entry path, whose permissions your agent is really running under, how to shape the tool surface so the model can&#8217;t wander, and what happens to your org&#8217;s API budget when an agent starts looping. There&#8217;s a build walkthrough, but the build is the easy part.</p>



<h2 class="wp-block-heading">Two doors closed while you were reading the old tutorial</h2>



<p class="wp-block-paragraph">Most of what ranks for this topic was written against an architecture you can no longer build in a fresh account. Both vendors changed the front door within a few months of each other, and the tutorials haven&#8217;t caught up. Check this before you follow anything, including this post.</p>



<h3 class="wp-block-heading">On the AWS side: Bedrock Agents is now Bedrock Agents Classic</h3>



<p class="wp-block-paragraph">AWS renamed the original Bedrock Agents service to Bedrock Agents Classic and put it into maintenance mode. It stopped accepting new customers on 30 July 2026. This one is worth stating with the date attached, because the date is the whole point.</p>



<p class="wp-block-paragraph">The mechanics matter more than the branding:</p>



<ul class="wp-block-list">
<li>Only <code>CreateAgent</code> and <code>InvokeInlineAgent</code> are restricted. Every other API stays available to everyone.</li>

<li>Allowlisting is per AWS account, based on whether that account used Bedrock Agents in the previous twelve months. Multi-account estates get a split result: your old sandbox works, your new production account doesn&#8217;t.</li>

<li>A non-allowlisted account calling <code>CreateAgent</code> gets an <code>AccessDeniedException</code> with HTTP 403, not a quota error. If you&#8217;re debugging IAM and seeing 403s, this is a plausible cause that no policy change will fix.</li>

<li>There is no exception process. AWS documents this plainly.</li>

<li>The model catalogue inside Classic is frozen. Bedrock itself keeps getting new models; the Classic orchestration layer does not.</li>
</ul>



<p class="wp-block-paragraph">The replacement is Amazon Bedrock AgentCore. It splits into a managed harness (config-driven, closest to the old experience) and code-defined agents on the AgentCore runtime (bring Strands, LangGraph, CrewAI, or your own loop). For a Salesforce assistant, the harness is usually enough, and you can move to code later without re-plumbing the tools.</p>



<p class="wp-block-paragraph">Existing Classic agents keep running. There&#8217;s no announced end-of-life. But a frozen model catalogue sets a ceiling on the thing, and if your team can&#8217;t recreate the agent in a new account, your disaster recovery story has a hole in it.</p>



<h3 class="wp-block-heading">On the Salesforce side: Connected Apps gave way to External Client Apps</h3>



<p class="wp-block-paragraph">Salesforce blocked creation of new Connected Apps through both UI and API, with package installation as the exception. External Client Apps are the replacement. Existing Connected Apps keep working and keep their OAuth flows, so this is a wall on new creation rather than a sunset.</p>



<p class="wp-block-paragraph">The practical symptom is that the <em>New Connected App</em> button is greyed out with no explanation. Every AWS-to-Salesforce tutorial that opens with &#8220;first, create a Connected App&#8221; stops working right there.</p>



<p class="wp-block-paragraph">One difference matters for agent work specifically: External Client Apps do not support the username-password OAuth flow at all. If your integration design assumed you&#8217;d stash a service account username and password somewhere and call it a day, that design is gone. Which is a good thing, and it forces the question the next section is about.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure family one: whose permissions is the assistant using?</h2>



<p class="wp-block-paragraph">This is the one from the opening, and it&#8217;s the reason most of these projects should slow down before they ship.</p>



<p class="wp-block-paragraph">Salesforce&#8217;s security model is genuinely good. An OAuth token runs in a user context, and object permissions, field-level security, and sharing rules all still apply to API calls. A token with the <code>api</code> scope does not bypass CRUD, FLS, or sharing. People hear that and relax.</p>



<p class="wp-block-paragraph">The catch is the phrase &#8220;a user context.&#8221; Which user?</p>



<p class="wp-block-paragraph">If you authenticate the gateway to Salesforce with client credentials, every request from every person runs as one integration user. Salesforce enforces that user&#8217;s permissions perfectly. It has no idea a different human asked the question. You&#8217;ve built what&#8217;s sometimes called a multiplexor: one licence fronting many people, with all the real access control now living in your application layer instead of the platform&#8217;s.</p>



<p class="wp-block-paragraph">At that point the sharing model is decoration. A prompt that talks the model into calling a query tool with a wider filter returns data the asker was never entitled to, and Salesforce logs it as a legitimate read by a legitimate user. There&#8217;s nothing to alert on, because nothing went wrong.</p>



<h3 class="wp-block-heading">Two-legged versus three-legged, and when each is honest</h3>



<p class="wp-block-paragraph">AgentCore Gateway supports both client credentials (two-legged) and authorization code (three-legged) flows for reaching downstream targets, and handles ingress and egress authorisation separately. That distinction is the control you need.</p>



<ul class="wp-block-list">
<li><strong>Two-legged is honest when the data is genuinely org-wide.</strong> Product catalogues, published knowledge articles, case deflection content. Nobody&#8217;s entitlement varies, so a single service identity represents the truth.</li>

<li><strong>Three-legged is the only correct answer for record data.</strong> Opportunities, accounts, contacts, custom objects with sharing rules on them. The token is minted for the actual human, Salesforce applies their sharing, and the answer is bounded by what they could have looked up themselves.</li>
</ul>



<p class="wp-block-paragraph">Three-legged costs you something real, and I&#8217;d rather say so than pretend it&#8217;s free. You need per-user consent, token storage, refresh handling, and a way to deal with a user whose token has expired mid-conversation. AgentCore Identity provides a token vault and handles the consent flow, including binding the authorisation URL to the session that started it so a shared link can&#8217;t be completed by someone else. It&#8217;s still more moving parts than a single secret in Secrets Manager.</p>



<p class="wp-block-paragraph">Pay the cost anyway for record data. The alternative is a system whose blast radius is &#8220;everything the integration user can see,&#8221; discovered by a customer or an auditor rather than by you.</p>



<p class="wp-block-paragraph">If you already run Okta, Auth0, or Ping as your identity provider, Gateway can sit in front of them for inbound auth rather than you standing up a parallel Cognito user pool. Reusing the IdP you already audit is worth more here than saving a config file.</p>



<h2 class="wp-block-heading">Failure family two: the shape of the tool surface</h2>



<p class="wp-block-paragraph">The second mistake is giving the model a tool called <code>run_soql</code> that accepts an arbitrary query string. It demos beautifully. Ask anything, get anything.</p>



<p class="wp-block-paragraph">You&#8217;ve now made the model your query planner and your authorisation layer at once. Every safety property of the system depends on prompt text. Guardrails help with content, not with whether a generated <code>WHERE</code> clause happened to be too generous.</p>



<p class="wp-block-paragraph">Narrow tools are the fix, and they&#8217;re boring on purpose. <code>get_open_cases_for_account(account_id)</code> beats <code>run_soql(query)</code> because the query shape is fixed in your code, the model only supplies an identifier, and you can log and rate-limit each tool independently. When something goes wrong you&#8217;re reading a parameter, not parsing SOQL to work out what the model asked for.</p>



<p class="wp-block-paragraph">AgentCore Gateway takes OpenAPI specs, Smithy models, and Lambda functions as tool input types, and exposes them to the agent as MCP tools. It also has one-click integrations for several SaaS products including Salesforce. Practically that gives you three routes:</p>



<ol class="wp-block-list">
<li><strong>The built-in Salesforce integration</strong> for the fastest path to something working. Good for a proof of concept. Look closely at which operations it exposes before you take it further, because breadth is exactly what you don&#8217;t want.</li>

<li><strong>An OpenAPI target</strong> describing a small purpose-built API in front of Salesforce. This is the one I reach for first. You control the surface completely, and the spec doubles as the tool description the model reads.</li>

<li><strong>Lambda targets</strong> when a tool needs to combine several Salesforce calls, or talk to something else at the same time.</li>
</ol>



<p class="wp-block-paragraph">MCP server targets are also supported, with the gateway aggregating tools from multiple targets into a single <code>tools/list</code> response. If you&#8217;re running your own MCP server as the Salesforce facade, it needs a stable HTTPS endpoint the gateway can reach; a small VPS from a provider like InterServer or Hetzner is plenty for the development and staging copies, with the production one behind whatever your standard ingress is.</p>



<h2 class="wp-block-heading">Failure family three: the write path</h2>



<p class="wp-block-paragraph">Reads leak. Writes destroy. They deserve different treatment and usually get the same treatment, which is how you end up with three hundred opportunities silently restaged.</p>



<p class="wp-block-paragraph">Start read-only and stay there longer than feels necessary. When you do add writes, the constraints that actually help:</p>



<ul class="wp-block-list">
<li><strong>One record per call.</strong> No bulk update tool. If the model wants to change twenty records it makes twenty calls, each individually logged and individually rate-limited. Slow is a feature here.</li>

<li><strong>Separate the write identity.</strong> A distinct External Client App and permission set for writes means you can revoke the write path at 2am without taking the assistant offline.</li>

<li><strong>Human confirmation on anything irreversible.</strong> The AgentCore harness supports inline function tools that pause the agent and hand control back to your client code, equivalent to return-of-control. That pause is where a person says yes.</li>

<li><strong>Stamp the source.</strong> A custom field recording that a change came from the assistant, with the session and actor ID, is the difference between a ten-minute rollback and an afternoon of forensics.</li>
</ul>



<p class="wp-block-paragraph">Salesforce field history tracking is on a limited number of fields per object. If the field your assistant writes to isn&#8217;t tracked, you have no before-value. Check that before go-live, not after.</p>



<h2 class="wp-block-heading">Failure family four: what happens under load</h2>



<p class="wp-block-paragraph">Salesforce meters API requests against a daily org-wide allocation that depends on your edition and licence count. Your assistant does not consume that budget the way a nightly sync does.</p>



<p class="wp-block-paragraph">A single conversational turn can be several tool calls. A model that doesn&#8217;t get a clean answer will often retry with a different filter. Ten support agents having ten conversations each becomes a request pattern nobody modelled, and the org-wide limit means the integration that breaks first might be your ETL job, your marketing platform, or your billing sync. Not the assistant. That&#8217;s what makes it hard to diagnose.</p>



<p class="wp-block-paragraph">Three things that help, in order of how much they buy you:</p>



<ul class="wp-block-list">
<li>Cap tool calls per session in your agent config, and return a clear message when the cap is hit rather than letting the loop run.</li>

<li>Cache anything org-wide and slow-moving. Picklist values, record type metadata, product catalogue. These are a surprising share of calls and almost never change mid-day.</li>

<li>Watch API usage as a leading indicator, not a post-mortem. Salesforce exposes remaining daily requests via the REST limits resource. Scrape it on a schedule into CloudWatch, Grafana Cloud, or whatever you already page from, and alert on the trend rather than the wall.</li>
</ul>



<h2 class="wp-block-heading">Wiring it up</h2>



<p class="wp-block-paragraph">With the design decisions settled, the build is short. The AgentCore CLI scaffolds and deploys the harness.</p>



<ol class="wp-block-list">
<li>Create an External Client App in Salesforce with the narrowest OAuth scopes the tools need. Assign it via permission set, not profile.</li>

<li>Build the tool layer: an OpenAPI spec for your purpose-built endpoints, or Lambda functions for composite operations.</li>

<li>Create an AgentCore Gateway, attach the target, and configure the credential provider in AgentCore Identity for the egress side.</li>

<li>Create the harness, point it at the gateway, and deploy.</li>
</ol>



<p class="wp-block-paragraph">Scaffolding a harness and attaching a gateway looks like this. The <code>--gateway-arn</code> value is the gateway you created in step three, and the <code>--type</code> tells the CLI this tool is a gateway rather than one of the built-in tools such as the browser or code interpreter:</p>



<pre class="wp-block-code"><code>agentcore create --name sf-assistant

agentcore add tool --harness sf-assistant 
  --type agentcore_gateway --name sf-gateway 
  --gateway-arn arn:aws:bedrock-agentcore:REGION:ACCOUNT_ID:gateway/GATEWAY_ID</code></pre>



<p class="wp-block-paragraph">Then declare the model and the system prompt, and deploy. Check the current model identifiers in the Bedrock console rather than copying one from any blog post, this one included, because they change:</p>



<pre class="wp-block-code"><code>agentcore add harness 
  --name sf-assistant 
  --model-id YOUR_MODEL_ID 
  --system-prompt "You answer questions about CRM records using only the provided tools. Never guess a record value." 
  --tools sf-gateway

agentcore deploy</code></pre>



<p class="wp-block-paragraph">Invoking it takes a session ID, and an actor ID if you want memory scoped per user. Skipping <code>--actor-id</code> is the memory equivalent of the integration user problem: one shared memory store, everyone&#8217;s context in it.</p>



<pre class="wp-block-code"><code>agentcore invoke --harness sf-assistant 
  --session-id "$(uuidgen)" --actor-id alice 
  "Which of my open opportunities have had no activity in three weeks?"</code></pre>



<p class="wp-block-paragraph">Note &#8220;my&#8221; in that question. With three-legged auth it means something. With an integration user it means nothing, and the model will cheerfully invent a definition.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong><code>AccessDeniedException</code> when creating an agent.</strong> If you&#8217;re calling <code>CreateAgent</code> or <code>InvokeInlineAgent</code> on the Classic API, check whether the account has prior Bedrock Agents usage before you touch IAM. No policy fixes this one. Build on AgentCore instead.</p>



<p class="wp-block-paragraph"><strong>The New Connected App button is greyed out.</strong> Expected. Create an External Client App. If you have a documented technical requirement that only a Connected App satisfies, that&#8217;s a Salesforce Support request, not a setting you can toggle.</p>



<p class="wp-block-paragraph"><strong>Tools appear in <code>tools/list</code> but calls fail.</strong> Gateway can cache tool schemas from a target without holding a working credential for calls. Discovery succeeding proves the schema synced, not that egress auth works. Look at the credential provider.</p>



<p class="wp-block-paragraph"><strong>The assistant returns fewer records than the user expects.</strong> Under three-legged auth this is often correct behaviour that looks like a bug. Have the user run the same query in the Salesforce UI. If the UI shows more, check FLS on the specific fields; if it shows the same, the assistant is right and the user&#8217;s mental model is wrong.</p>



<p class="wp-block-paragraph"><strong>Answers are confidently wrong about record values.</strong> Usually the model answering from training data instead of calling a tool. Tighten the system prompt to forbid answering record questions without a tool result, and check the traces to confirm which turns actually called out.</p>



<p class="wp-block-paragraph"><strong>Everything works, then breaks at the same time daily.</strong> Look at org-wide API consumption before you look at your own code. Something else in the org may be eating the allocation on a schedule.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using a System Administrator profile for the integration identity because it makes the errors go away.</li>

<li>Exposing a generic SOQL tool and calling the prompt a security control.</li>

<li>Testing only in a sandbox with one user, where the sharing model is invisible because there&#8217;s nothing to share.</li>

<li>Treating Guardrails as an access control layer. They filter content; they don&#8217;t know who&#8217;s asking.</li>

<li>Shipping writes in the first release because the read-only version felt underwhelming in the demo.</li>

<li>Building on Bedrock Agents Classic in a new account without checking allowlist status first.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Use three-legged OAuth for anything governed by sharing rules. Reserve client credentials for genuinely org-wide data.</li>

<li>Give each tool one job and a fixed query shape. The model supplies parameters, never structure.</li>

<li>Separate read and write identities so you can kill one without the other.</li>

<li>Scope memory per actor, not per deployment.</li>

<li>Test with at least three users at different levels of the role hierarchy, and diff the answers. Identical answers to &#8220;show me my pipeline&#8221; is the signal that identity isn&#8217;t propagating.</li>

<li>Keep end-to-end traces and ship them somewhere you already look, so agent behaviour lands in the same place as the rest of your telemetry.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I still use Bedrock Agents for a new Salesforce project?</h3>



<p class="wp-block-paragraph">Only if the AWS account already used Bedrock Agents in the previous twelve months. Accounts without that history get an <code>AccessDeniedException</code> on <code>CreateAgent</code>, and AWS has said there&#8217;s no exception process. For anything new, build on AgentCore.</p>



<h3 class="wp-block-heading">Do I need Agentforce to build a Salesforce AI assistant with Amazon Bedrock?</h3>



<p class="wp-block-paragraph">No. Agentforce is Salesforce&#8217;s own agent layer, and it can call out to AWS, which is useful when you want the assistant to live inside the Salesforce UI and inherit the Einstein Trust Layer. But an AgentCore agent calling Salesforce APIs through a gateway is a complete architecture on its own, and it&#8217;s the better fit when the assistant also needs to reach systems outside Salesforce.</p>



<h3 class="wp-block-heading">Does the model see all my CRM data?</h3>



<p class="wp-block-paragraph">It sees whatever your tools return. That&#8217;s determined by the OAuth identity and the tool definitions, not by the model. This is why the identity decision matters more than the model choice.</p>



<h3 class="wp-block-heading">Will Bedrock Guardrails stop the assistant leaking records?</h3>



<p class="wp-block-paragraph">No, and this is worth being blunt about. Guardrails filter content categories and can mask sensitive data patterns. They have no concept of which user is asking or which records that user is entitled to. A record correctly retrieved under an over-permissioned token passes every guardrail check, because there&#8217;s nothing objectionable about it. Guardrails are a good layer. They are not this layer.</p>



<h3 class="wp-block-heading">How do I keep the assistant from burning the org&#8217;s API limit?</h3>



<p class="wp-block-paragraph">Cap tool calls per session, cache org-wide reference data, and monitor remaining daily requests via the REST limits resource on a schedule. The allocation is org-wide, so the symptom of overuse usually shows up in some other integration first.</p>



<h3 class="wp-block-heading">Should I migrate an existing Bedrock Agents Classic setup?</h3>



<p class="wp-block-paragraph">There&#8217;s no deadline and no announced end-of-life, so there&#8217;s no emergency. The reasons to move are the frozen model catalogue and the fact that new accounts can&#8217;t recreate your agent. If you&#8217;re still actively developing the thing, move. If it&#8217;s stable and finished, it can wait. AWS publishes a capability mapping and a CLI import path for existing configurations.</p>



<h3 class="wp-block-heading">Is an External Client App harder to set up than a Connected App?</h3>



<p class="wp-block-paragraph">Slightly different rather than harder. The main structural change is that developer settings and admin policies are separated, so the app definition and the per-org access controls are configured independently. The one real removal is the username-password flow, which no longer exists in the ECA model.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A Salesforce AI assistant with Amazon Bedrock is not really a model problem. Model quality is the part that will take care of itself, and the part you can change with one config line later.</p>



<p class="wp-block-paragraph">The part that won&#8217;t take care of itself is identity. If the assistant answers as one integration user, you have quietly moved your entire access control model from a platform that enforces it well into a prompt that enforces it not at all, and nothing in your monitoring will tell you. Get that right first. Then argue about models.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building or auditing this?</h2>



<p class="wp-block-paragraph">Most of the work on these projects is unglamorous: deciding what the agent is allowed to touch, and proving it can&#8217;t touch anything else. That&#8217;s the part I&#8217;m useful for.</p>



<ul class="wp-block-list">
<li>Auditing an existing Salesforce AI integration for identity propagation gaps, including testing across role hierarchy levels to find silent over-permissioning.</li>

<li>Migrating a Bedrock Agents Classic deployment to AgentCore, including mapping action groups to gateway targets.</li>

<li>Designing the tool surface: replacing generic query tools with narrow, logged, individually rate-limited operations.</li>

<li>Setting up three-legged OAuth between AgentCore Gateway and Salesforce, including token vault and consent handling.</li>

<li>Moving integrations from Connected Apps to External Client Apps without breaking existing OAuth flows.</li>

<li>Instrumenting API consumption and agent traces so you see problems before an unrelated integration starts failing.</li>
</ul>



<p class="wp-block-paragraph">If you&#8217;ve got something concrete, send it over: a gateway target config, an agent trace, a tool schema, or just the error you&#8217;re stuck on. Easier to say something useful about a real setup than a hypothetical one.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/salesforce-ai-assistant-amazon-bedrock/">Building a Salesforce AI Assistant on Amazon Bedrock Without Leaking Your CRM</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</title>
		<link>https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 06 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Agentforce]]></category>
		<category><![CDATA[AI Agents]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[API Gateway]]></category>
		<category><![CDATA[Architecture]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[CloudWatch]]></category>
		<category><![CDATA[Data Integration]]></category>
		<category><![CDATA[Einstein Trust Layer]]></category>
		<category><![CDATA[EventBridge]]></category>
		<category><![CDATA[Generative AI]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[MCP]]></category>
		<category><![CDATA[Observability]]></category>
		<category><![CDATA[PrivateLink]]></category>
		<category><![CDATA[Salesforce]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=155</guid>

					<description><![CDATA[<p>Agentforce and AWS wire together in four standard patterns, and every one of them has a point where Salesforce's guarantees stop and yours start. This traces a single request across each boundary it crosses, covers the Trust Layer default most write-ups get wrong (LLM data masking is disabled for agents), and sets out what changes the moment a callout lands in your own account: retention, audit trail, and user identity that does not travel.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/">Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The architecture review goes fine right up until someone from security asks the only question that matters: when the agent answers a customer&#8217;s question, where exactly does that customer&#8217;s data go, and who keeps a copy?</p>



<p class="wp-block-paragraph">&#8220;Salesforce handles it&#8221; is not an answer. Neither is &#8220;it&#8217;s in our VPC,&#8221; because it usually isn&#8217;t, and the parts that are in your VPC are the parts nobody has threat-modelled.</p>



<p class="wp-block-paragraph">This post traces a single Agentforce request end to end and stops at each boundary it crosses. Agentforce and AWS get wired together in three or four standard patterns, and every one of them has a specific point where Salesforce&#8217;s guarantees stop and yours start. I will cover where that line sits, what changes about the security model when you cross it, and one default that surprises almost everyone.</p>



<h2 class="wp-block-heading">The boundaries a single request crosses</h2>



<p class="wp-block-paragraph">One user turn touches more systems than the diagrams suggest. Roughly:</p>



<ol class="wp-block-list">
<li><strong>Retrieval.</strong> The agent grounds itself in CRM records, Data Cloud objects and knowledge content.</li>

<li><strong>Reasoning.</strong> The request gets classified to a topic, then an action gets chosen. Both steps involve a model call.</li>

<li><strong>Generation.</strong> A prompt is assembled and sent through the LLM gateway to a model provider.</li>

<li><strong>Action.</strong> If the agent decides to do something, it calls out. That callout is where AWS usually enters the picture.</li>
</ol>



<p class="wp-block-paragraph">Steps one to three happen inside Salesforce&#8217;s trust boundary, and the Einstein Trust Layer applies. Step four does not, and it doesn&#8217;t. That sentence is the whole post, but the detail matters.</p>



<h2 class="wp-block-heading">Retrieval: the part that&#8217;s better than you&#8217;d expect</h2>



<p class="wp-block-paragraph">Grounding is where Agentforce gets its context, and the architecturally interesting decision is that vector search lives inside Data Cloud rather than in a separate vector database. Structured records and unstructured content are retrieved from the same layer, which removes a whole class of sync problem you would otherwise be building.</p>



<p class="wp-block-paragraph">The security-relevant property is that retrieval is permission-aware. The agent retrieves as the running user, so sharing rules and field-level security apply. If a rep cannot see an opportunity, the agent acting on their behalf should not be able to summarise it either.</p>



<p class="wp-block-paragraph">Two caveats worth putting in your notes. First, this only holds while retrieval goes through Salesforce; the moment you ground from an external source, you are enforcing access control yourself. Second, if your CRM sharing model is loose, the agent inherits that looseness and makes it much easier to exploit, because a model will cheerfully summarise a thousand records a human would never have opened one by one. Agentforce does not create the over-permissioning problem, it just removes the friction that was hiding it.</p>



<p class="wp-block-paragraph">Where AWS shows up here: if your analytical data lives in Redshift, an Iceberg lake or Databricks, zero-copy federation lets Data Cloud register those tables and query them in place rather than replicating them. Good for cost and duplication, and it moves an access-control decision into the federation configuration, which is a place people forget to audit.</p>



<h2 class="wp-block-heading">The Trust Layer, and the default nobody expects</h2>



<p class="wp-block-paragraph">The Einstein Trust Layer sits between every Salesforce-originated prompt and the model. Its components are well documented: secure grounding, data masking, system policies against prompt injection, the LLM gateway, zero data retention agreements with model providers, toxicity checks on the way back, and an audit trail of the whole journey.</p>



<p class="wp-block-paragraph">Here is the part that gets misreported constantly, including in a lot of otherwise decent write-ups: <strong>LLM data masking is disabled for agents.</strong> Salesforce says so plainly in its own documentation on the Trust Layer architecture and repeats it in the Trailhead module on masking. Masking remains available and configurable for embedded generative features like service replies and work summaries. For Agentforce agents, it is off.</p>



<p class="wp-block-paragraph">The reason is not hard to work out once you think about what an agent does. Masking swaps real values for placeholders. An agent that has to actually perform an action needs the real record ID, the real amount, the real email address, because those become tool arguments. You cannot pass <code>&lt;Person_0&gt;</code> to a Flow and expect it to update a contact. Masking and acting are in genuine tension, and Salesforce resolved it in favour of acting.</p>



<p class="wp-block-paragraph">What this changes in practice:</p>



<ul class="wp-block-list">
<li>Real PII reaches the model provider. The zero data retention commitment is what protects it, not de-identification. Those are different controls with different failure modes: masking is technical and verifiable, a retention agreement is contractual.</li>

<li>If your compliance position was &#8220;no personal data leaves our boundary in identifiable form,&#8221; Agentforce does not satisfy it the way you may have assumed, and this is worth raising before your security review rather than during it.</li>

<li>Controlling exposure moves upstream. What the model sees is now decided by what you ground on and what your sharing model permits, not by a masking filter downstream.</li>
</ul>



<p class="wp-block-paragraph">None of this makes the Trust Layer weak. Zero retention, the gateway, system policies and the audit trail all still apply to agent traffic, and the Trust Layer typically runs several times in a single user turn, once for topic classification and again for generation. It is a real control plane. Just know which control is doing the work.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Connecting Agentforce and AWS: four patterns</h2>



<p class="wp-block-paragraph">There are four ways these two systems talk, and they have different latency, failure and security characteristics.</p>



<h3 class="wp-block-heading">Synchronous outbound: agent action calls AWS</h3>



<p class="wp-block-paragraph">A custom agent action, implemented via External Services with an OpenAPI schema, an Apex invocable method, or a Flow, calls an endpoint you own. Behind it sits API Gateway and Lambda, and often a Bedrock agent or knowledge base.</p>



<p class="wp-block-paragraph">Authentication uses named credentials, which keep the secret out of your Apex and centralise the auth config. Use them; hardcoding a key into an Apex class is the version of this that ends up in a git history.</p>



<p class="wp-block-paragraph">The trade-off is that you have put a network call and a second model invocation inside a conversational turn. Users notice. Budget for it, and set the timeout deliberately rather than inheriting a default.</p>



<h3 class="wp-block-heading">Asynchronous outbound: Event Relay to EventBridge</h3>



<p class="wp-block-paragraph">Salesforce Event Relay delivers platform events from the Salesforce event bus to Amazon EventBridge without code, and EventBridge routes onward to Lambda, S3, Step Functions or a Bedrock agent.</p>



<p class="wp-block-paragraph">This is the right pattern for anything that does not need to complete inside the conversation: enrichment, downstream processing, long-running work. It also fails better, because a retry is a queue concern rather than a user watching a spinner.</p>



<h3 class="wp-block-heading">Inbound: AWS calls Agentforce</h3>



<p class="wp-block-paragraph">The Agent API lets something on the AWS side invoke an Agentforce agent, typically through EventBridge API destinations. That gives you bidirectional multi-agent flows: a Bedrock agent that detects a problem can open a case through Agentforce rather than writing to the Salesforce API directly and bypassing the agent&#8217;s own logic.</p>



<p class="wp-block-paragraph">Worth being deliberate here. An inbound path means an AWS workload can trigger agent reasoning, which means anything that can inject content into that workload can influence a prompt. Treat the payload as untrusted input, because it is.</p>



<h3 class="wp-block-heading">MCP: tools rather than integrations</h3>



<p class="wp-block-paragraph">Agentforce has a native MCP client, so an agent can connect to any MCP-compliant server without a bespoke integration, alongside a server registry intended to let admins govern which agents connect to which tools under central policy.</p>



<p class="wp-block-paragraph">This is the pattern that will cause the most trouble over the next couple of years, and not because the protocol is bad. It is because MCP makes adding a tool trivial, and every tool is a new path by which text from somewhere else reaches your agent&#8217;s context and by which your agent&#8217;s decisions reach someone else&#8217;s system. The registry exists precisely so that &#8220;which tools can this agent reach&#8221; is an administered answer rather than an emergent one. Use it as a control, not a catalogue.</p>



<h3 class="wp-block-heading">Keeping the traffic private</h3>



<p class="wp-block-paragraph">By default these callouts traverse the public internet with TLS. Salesforce Private Connect uses AWS PrivateLink to give you a private path to your own AWS endpoints instead. Whether that is required is a policy question rather than a technical one, but it is much easier to set up at design time than to retrofit after an auditor asks.</p>



<h2 class="wp-block-heading">What changes the moment you cross into AWS</h2>



<p class="wp-block-paragraph">This is the seam, and it is where most real incidents will come from. Four things stop being true simultaneously.</p>



<ul class="wp-block-list">
<li><strong>Zero retention becomes your policy, not Salesforce&#8217;s.</strong> If your Lambda logs its input at INFO, you now have customer PII in CloudWatch Logs with whatever retention that log group happens to have, which by default is forever. Your own debug logging is the single most likely leak path in this entire architecture, and it will not show up in any Salesforce audit report.</li>

<li><strong>The audit trail splits.</strong> Salesforce logs the prompt journey. AWS logs the invocation. Nothing correlates them unless you pass a request identifier through the callout and log it on both sides. Do that from day one; reconstructing an incident across two disconnected audit systems is genuinely unpleasant.</li>

<li><strong>User identity does not travel.</strong> This is the important one. Inside Salesforce the agent acts as the running user and sharing rules apply. Your Lambda runs under an IAM execution role with fixed permissions and no idea who asked. If that role can read a whole DynamoDB table, then every user of the agent can effectively read the whole table through it, regardless of their CRM permissions. That is a classic confused deputy, and it is easy to build without noticing. If the action needs to be scoped per user, pass the identity explicitly and enforce it in your own code.</li>

<li><strong>Model governance forks.</strong> A Bedrock model you invoke from your own Lambda is outside the Trust Layer. Its guardrails, logging and retention are configured in your account and are yours to get right. If you want prompt filtering or PII detection on that leg, you are building or buying it.</li>
</ul>



<p class="wp-block-paragraph">Note the difference between two things people conflate. Salesforce running on Hyperforce, which is built on public cloud infrastructure including AWS, means Salesforce&#8217;s own workloads run there under Salesforce&#8217;s controls. It does not mean your Salesforce data sits in your AWS account. When an agent action calls your Lambda, that is the first moment data lands in infrastructure you control and are responsible for.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Assuming masking protects agent traffic to the model. It does not; check the current documentation yourself rather than taking a blog&#8217;s word for it, including this one.</li>

<li>Logging full request payloads in Lambda during development and never turning it off.</li>

<li>Giving the Lambda execution role broad data access because &#8220;the agent is trusted,&#8221; when the agent is a proxy for every user who can talk to it.</li>

<li>Putting a slow AWS call inside a synchronous agent action when the work did not need to be synchronous.</li>

<li>Not passing a correlation identifier across the boundary, so Salesforce and CloudWatch tell two unlinked stories.</li>

<li>Treating content returned from an external tool or MCP server as trusted context rather than as untrusted input that will be read by a model.</li>

<li>Deploying agents against a permissive sharing model and discovering the blast radius later.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does Agentforce data get used to train models?</h3>



<p class="wp-block-paragraph">Not on the Salesforce leg. Salesforce&#8217;s zero data retention arrangements with model providers mean prompts and responses are not stored or used for training. That commitment covers traffic going through the LLM gateway. It says nothing about a model you invoke yourself from your own AWS account, where the retention and logging configuration is entirely yours.</p>



<h3 class="wp-block-heading">Can I use my own Bedrock model inside Agentforce?</h3>



<p class="wp-block-paragraph">There are two distinct things here. Bedrock-hosted models can be brought into Salesforce through the Einstein and Bedrock integration, in which case they sit behind the Trust Layer. Separately, you can call a Bedrock model or agent from a custom agent action, in which case you are outside the Trust Layer and own the controls. Be clear which one you are building, because they have completely different compliance stories.</p>



<h3 class="wp-block-heading">Sync or async for AWS integrations?</h3>



<p class="wp-block-paragraph">Default to async through Event Relay and EventBridge unless the agent genuinely needs the result to continue the conversation. Synchronous callouts put a network round trip and often a second inference call inside a user&#8217;s turn, and they fail in front of the user rather than into a retry queue.</p>



<h3 class="wp-block-heading">How do I stop the agent seeing data a user shouldn&#8217;t?</h3>



<p class="wp-block-paragraph">Inside Salesforce, fix the sharing model, because retrieval respects it and inherits its flaws. Outside Salesforce, sharing rules do not apply at all: pass the user identity through the callout and enforce authorisation in your own code, or scope the integration to data that is safe for every user of that agent.</p>



<h3 class="wp-block-heading">Do I need PrivateLink for this?</h3>



<p class="wp-block-paragraph">Technically no; callouts work over TLS on the public internet. Practically it depends on your regulatory position and whether your AWS endpoints should be publicly reachable at all. If they should not be, Private Connect over PrivateLink is the supported path and is far cheaper to design in than to add later.</p>



<h3 class="wp-block-heading">What should I log, and where?</h3>



<p class="wp-block-paragraph">On the AWS side, log the correlation identifier, the action name, the outcome and the timing. Do not log the payload. If you need payload-level debugging, gate it behind a flag that is off in production and set an explicit short retention on the log group. Ship the operational metrics somewhere you already watch, whether that is CloudWatch alarms, Grafana or Datadog, because an agent action that silently starts failing looks like nothing at all from the Salesforce side.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Wiring up Agentforce and AWS is not hard. The patterns are documented, the connectors exist, and a working proof of concept is a day&#8217;s work.</p>



<p class="wp-block-paragraph">What is hard is knowing precisely where Salesforce&#8217;s guarantees end. They end at the callout. Everything before it is covered by the Trust Layer, with the significant exception that masking is off for agents, so what reaches the model is real data protected by a retention agreement rather than by de-identification. Everything after it is yours: your retention, your audit trail, your authorisation, your logs.</p>



<p class="wp-block-paragraph">Draw that line on your architecture diagram before the security review, not during it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help designing or reviewing this?</h2>



<p class="wp-block-paragraph">I work on the AWS side of Salesforce integrations, which is usually the side nobody owns. Things I can help with:</p>



<ul class="wp-block-list">
<li>Threat-modelling the seam between Agentforce and your AWS account, and writing down what each side actually guarantees.</li>

<li>Building agent actions on API Gateway and Lambda with named credentials, sane timeouts and no payload logging.</li>

<li>Event-driven integration via Event Relay and EventBridge, including retry, dead-letter and replay design.</li>

<li>Scoping IAM execution roles so an agent action cannot become a confused deputy for every user who can reach it.</li>

<li>Correlated observability across Salesforce and CloudWatch so an incident can be reconstructed in one timeline.</li>

<li>Private Connect and PrivateLink setup where callouts should not touch the public internet.</li>
</ul>



<p class="wp-block-paragraph">If you have an integration already built, send me the agent action definition and the Lambda&#8217;s IAM policy. Those two together usually tell the whole story.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/agentforce-aws-architecture-security/">Agentforce and AWS: Where the Trust Layer Stops and Your Logs Begin</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
