<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Client Onboarding | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/client-onboarding/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/client-onboarding/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sat, 31 Oct 2026 07:06:41 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Client Onboarding | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/client-onboarding/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Configuration Drift Across 50 Client AWS Accounts: The Checks That Actually Fire</title>
		<link>https://john-nessime.com/blog/devops/aws-configuration-drift-detection-multi-account/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sat, 31 Oct 2026 07:06:41 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[Alert Fatigue]]></category>
		<category><![CDATA[AWS Config]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[CloudFormation]]></category>
		<category><![CDATA[Conformance Packs]]></category>
		<category><![CDATA[Confused Deputy]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[Delegated Administrator]]></category>
		<category><![CDATA[Drift Detection]]></category>
		<category><![CDATA[External ID]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Managed Service Providers]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Refresh-Only Mode]]></category>
		<category><![CDATA[Silent Failures]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[Terraform State]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=913</guid>

					<description><![CDATA[<p>Detecting configuration drift across dozens of client AWS accounts is not one tool, it is three detectors with three different blind spots. How to set up cross-account read-only access with per-client external IDs, what AWS Config, CloudFormation drift detection and refresh-only Terraform plans each fail to see, how to keep Config costs sane, and why automatic remediation is more dangerous than the drift itself.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-configuration-drift-detection-multi-account/">Configuration Drift Across 50 Client AWS Accounts: The Checks That Actually Fire</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A client emails on a Tuesday afternoon: &#8220;Our auditor is asking whether anything in the AWS account has changed since you handed it over. Can you confirm?&#8221; You built that account eleven months ago. You have the Terraform repo. You have no idea what has happened inside it since.</p>



<p class="wp-block-paragraph">Multiply that by forty-nine other clients and you have the actual problem. It is not &#8220;is there drift&#8221;, because there is always drift. It is that you cannot answer the question per account in under a minute, and you cannot tell a change someone made on purpose from a change nobody remembers making.</p>



<p class="wp-block-paragraph">This post covers how to build AWS configuration drift detection across a fleet of client accounts you do not own and cannot put into your own organization. The access model, the three detectors and what each one is blind to, how to stop the cost from running away, and why the automated fix is more dangerous than the drift.</p>



<h2 class="wp-block-heading">The failure mode: a green dashboard over an unwatched resource</h2>



<p class="wp-block-paragraph">Everything else in this post is downstream of one thing. Every drift detector on AWS reports on a subset of reality, and none of them tell you loudly where the subset ends. You get a clean report, you forward it to the client, and the thing that actually changed was never in scope.</p>



<p class="wp-block-paragraph">There are three distinct shapes of this blind spot, and they map exactly onto the three detectors:</p>



<ul class="wp-block-list">
<li><strong>The resource type is not being recorded.</strong> AWS Config only knows about resource types the recorder is configured to capture. Turn off a noisy type to save money and you have also turned off drift detection for it.</li>



<li><strong>The property is not in the template.</strong> CloudFormation drift detection compares only the resource properties explicitly written in your stack template. Rely on an AWS default and never declare it, and a change to it is invisible.</li>



<li><strong>The resource is not in state.</strong> A refresh-only Terraform plan compares state against reality. Anything a human created by hand in the console is not in state, so the plan is perfectly clean while the account fills up with unmanaged resources.</li>
</ul>



<p class="wp-block-paragraph">Once you internalise that, the design falls out of it. You do not pick a detector. You run more than one, on purpose, because each covers the other&#8217;s hole.</p>



<h2 class="wp-block-heading">Access comes first, because you do not own the organization</h2>



<p class="wp-block-paragraph">Most writing on multi-account drift assumes one AWS Organization with a delegated administrator and an organization-wide Config aggregator. That is the right answer inside a company. It is the wrong answer for consulting, because fifty clients are fifty separate organizations and none of them are joining yours.</p>



<p class="wp-block-paragraph">What you get instead is a read-only role in each client account, assumed from one account you control. The trust policy is where the security lives.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/drift-scanner" },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "unique-value-for-this-client-only"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The external ID is not decoration. AWS documents it specifically as the fix for the confused deputy problem: you hold credentials that can reach fifty client accounts, and without a per-client secret, one client who learns another client&#8217;s account ID could persuade your tooling to act on an account it should not touch. Generate a different external ID per client, store it with the client&#8217;s config, and never reuse one.</p>



<p class="wp-block-paragraph">For permissions, start with the two AWS managed policies built for exactly this. <code>arn:aws:iam::aws:policy/SecurityAudit</code> grants read access to security configuration across services. <code>arn:aws:iam::aws:policy/job-function/ViewOnlyAccess</code> adds the ability to list and describe resources generally. Between them you can run every detector below without holding a single write permission in a client account. That is worth insisting on in the contract, because it changes the conversation from &#8220;the consultant has admin&#8221; to &#8220;the consultant can look and nothing else&#8221;.</p>



<p class="wp-block-paragraph">If you want a further lock, run the scanner from a machine with a stable outbound address and add an <code>aws:SourceIp</code> condition to the trust policy. A small always-on box at a provider like Contabo or InterServer, or a fixed-IP egress from a business VPN such as NordLayer, is enough. The trade-off is real: you now have infrastructure to patch and a single point of failure. For fewer than ten accounts I would skip it. Past twenty, clients start asking for it during security review and it is easier to already have it.</p>



<h2 class="wp-block-heading">Detector one: AWS Config, and the cost trap inside it</h2>



<p class="wp-block-paragraph">AWS Config is the only one of the three that sees resources nobody wrote code for. It records a configuration item every time a supported resource changes, and it evaluates rules against those items. That makes it the right tool for the &#8220;somebody clicked something in the console&#8221; case.</p>



<p class="wp-block-paragraph">If a client does happen to run an Organization, you can enable Config as a trusted service and query everything from one place:</p>



<pre class="wp-block-code"><code># Run from the client's management account, once
aws organizations enable-aws-service-access 
  --service-principal=config.amazonaws.com

aws organizations list-aws-service-access-for-organization</code></pre>



<p class="wp-block-paragraph">Then the aggregator becomes queryable with a SQL-like expression. This is the single most useful command in the whole toolkit, because it turns &#8220;check compliance&#8221; into one call per client instead of one call per account per region:</p>



<pre class="wp-block-code"><code>aws configservice select-aggregate-resource-config 
  --configuration-aggregator-name client-aggregator 
  --expression "SELECT configuration.complianceType, COUNT(*) 
    WHERE resourceType = 'AWS::Config::ResourceCompliance' 
    GROUP BY configuration.complianceType"

# What is actually in the account, by type
aws configservice get-aggregate-discovered-resource-counts 
  --configuration-aggregator-name client-aggregator 
  --group-by-key RESOURCE_TYPE</code></pre>



<h3 class="wp-block-heading">The bill is the reason people turn it off</h3>



<p class="wp-block-paragraph">I am not going to quote rates, because AWS changes them and they vary by region. What matters is the shape of the billing, which has not changed: you pay per configuration item recorded, per rule evaluation, and per conformance pack evaluation. Three separate meters, all of which multiply by account and region.</p>



<p class="wp-block-paragraph">Two things drive the surprise bills. First, recording mode. Continuous recording captures every change as it happens; periodic recording captures a resource&#8217;s latest state once every twenty-four hours, and only when it differs from the last one recorded. Periodic costs more per item but produces far fewer items for high-churn types, so the total usually drops hard. You can set the default for the account and then override it per resource type, which is the setting most people never find.</p>



<p class="wp-block-paragraph">Second, and this is the one that catches people: compliance evaluations themselves generate configuration items. Every time a rule re-evaluates a resource and its compliance state is recorded, that is another item on the meter. In an account with conformance packs enabled across many resources, compliance records can outnumber the records of actual infrastructure changes. If a client&#8217;s Config bill looks wrong, that is the first place to look, not the resource count.</p>



<p class="wp-block-paragraph">The practical setting for consulting work: record the resource types that carry security meaning, and leave the churny ones off. Network interfaces, auto scaling lifecycle hooks and Lambda versions generate enormous item volume and almost never represent drift anyone cares about.</p>



<h2 class="wp-block-heading">Detector two: CloudFormation drift, and the property that was never declared</h2>



<p class="wp-block-paragraph">CloudFormation has native drift detection, and it is genuinely good at what it does. The call is asynchronous, so you fire it and poll:</p>



<pre class="wp-block-code"><code>ID=$(aws cloudformation detect-stack-drift 
  --stack-name client-network 
  --query StackDriftDetectionId --output text)

aws cloudformation describe-stack-drift-detection-status 
  --stack-drift-detection-id "$ID"

# Only the resources that actually moved
aws cloudformation describe-stack-resource-drifts 
  --stack-name client-network 
  --stack-resource-drift-status-filters MODIFIED DELETED</code></pre>



<p class="wp-block-paragraph">The output is the best of the three detectors. For each drifted resource you get <code>ExpectedProperties</code>, <code>ActualProperties</code>, and a <code>PropertyDifferences</code> list naming the exact path that changed and whether the difference is <code>ADD</code>, <code>REMOVE</code> or <code>NOT_EQUAL</code>. You can paste that into a client email without editing it.</p>



<p class="wp-block-paragraph">Now the blind spot, stated the way AWS states it: only resource properties explicitly defined in the stack template are checked for drift. Read that twice. If your template creates a security group and does not declare the ingress rules inline, drift detection on that stack will not tell you someone opened port 22 to the world. The stack is <code>IN_SYNC</code>. The account is not.</p>



<p class="wp-block-paragraph">The second blind spot is resource type coverage. Not every resource type supports drift detection. Types that do not are simply excluded from the results rather than flagged, and <code>describe-stack-resource-drifts</code> omits anything that was not checked. So a stack with twelve resources can return three drift records and you have no signal about the other nine unless you go looking. Compare the count of drift records against the count of stack resources on every run and alert when the gap grows.</p>



<p class="wp-block-paragraph">The other thing to know: drift detection tells you. It does not fix. Reconciling means either reverting the change by hand or bringing the current state into the template through resource import.</p>



<h2 class="wp-block-heading">Detector three: refresh-only Terraform plans, and their false positives</h2>



<p class="wp-block-paragraph">For accounts you built with Terraform or OpenTofu, the detector is already in the tool. A refresh-only plan compares state against the live provider APIs without proposing any configuration changes, and the detailed exit code turns the result into something a scheduler can branch on.</p>



<pre class="wp-block-code"><code>set +e
terraform plan -refresh-only -detailed-exitcode 
  -input=false -lock=false -no-color -out=drift.tfplan
CODE=$?
set -e

# 0 = no drift, 1 = the plan itself failed, 2 = drift detected
if [ "$CODE" -eq 2 ]; then
  terraform show -json drift.tfplan &gt; drift.json
fi</code></pre>



<p class="wp-block-paragraph"><code>-lock=false</code> matters more than it looks. This is a read-only operation running on a schedule, and you do not want a nightly drift job holding the state lock when someone needs to apply a fix. <code>-input=false</code> stops the job hanging forever on a prompt nobody is there to answer. And <code>terraform show -json</code> is what makes the result machine-readable; parsing the human-readable plan text is a trap that breaks on every minor version.</p>



<p class="wp-block-paragraph">Two honest caveats. Refresh-only plans have a history of returning exit code 2 while reporting no changes in the console output, and there are open issues against Terraform describing exactly that. If your drift alerts fire every night on an account nobody touched, check the JSON before you go hunting; you may be looking at a provider that rewrites a computed attribute on every read rather than at real drift. The second caveat is scale: this needs one plan per state file, and a client with fifteen root modules is fifteen invocations, each doing a full provider refresh. That is where a fifty-account nightly run stops being fast.</p>



<p class="wp-block-paragraph">On dedicated open-source drift tools: driftctl was the obvious answer for years, and it is worth knowing about because its model was good, comparing live resources against state and specifically surfacing unmanaged resources. It has been in maintenance mode since Snyk stepped back from active development, so treat it as a tool you can still run rather than one that will keep pace with new resource types. The commercial platforms in this space will do the fan-out for you and are a legitimate choice if you would rather buy than build. What you are paying for is the scheduling and the reporting, not the detection itself, which is the part you already have.</p>



<h2 class="wp-block-heading">Making the fan-out survivable</h2>



<p class="wp-block-paragraph">Fifty accounts is where a working script becomes an operational problem. The things that matter, in the order they bite:</p>



<ol class="wp-block-list">
<li><strong>Isolate every account.</strong> One client&#8217;s expired external ID must not stop the other forty-nine from being scanned. Catch per-account, record the failure as a result, keep going.</li>



<li><strong>Treat a failed scan as a finding.</strong> An account that has not been successfully scanned in a week is a worse state than an account with three drifted properties, because you know nothing about it. Report scan coverage before you report drift.</li>



<li><strong>Stagger, do not blast.</strong> Deep scans hit provider APIs hard and throttling is the normal outcome of running fifty in parallel. Spread them across the window and back off on retry.</li>



<li><strong>Store every run, not just the alerts.</strong> The value of drift history is the trend. &#8220;This account has drifted four times this quarter, always in the same security group&#8221; is a conversation with the client. A single alert is noise.</li>



<li><strong>Alert on new drift, not on drift.</strong> If a client accepted a manual change last month and you never reconciled the code, your job will report it every night forever until everyone stops reading the emails. Diff against the last run and only surface what is new.</li>
</ol>



<p class="wp-block-paragraph">For output, a per-client dashboard beats an inbox. Grafana Cloud or a self-hosted Grafana over a small results table is plenty; you are plotting a handful of numbers per account per day, not high-cardinality metrics. Whatever you pick, the client should be able to see their own account and nobody else&#8217;s.</p>



<h2 class="wp-block-heading">Do not let it fix things by itself</h2>



<p class="wp-block-paragraph">This is the part I would push back on hardest if a client asked for it. Automatic remediation on a drift detector means re-applying code over reality on a schedule, and the code does not know why reality changed.</p>



<p class="wp-block-paragraph">Somebody widened a timeout at two in the morning to get an incident under control. Your nightly job reverts it and the incident comes back, this time without the person who fixed it awake. Worse, some attribute changes force replacement rather than update in place, so an unattended apply can destroy a live resource in order to make it match the file.</p>



<p class="wp-block-paragraph">There are three legitimate responses to detected drift and a machine cannot choose between them: revert infrastructure to match the code, change the code to match reality, or record the difference as deliberate and exclude it from future comparisons. Detection is automatable. That decision is not. Keep the scanner on read-only credentials and let it open a pull request rather than run an apply.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>AccessDenied on assume-role for one client only.</strong> Almost always a mismatched external ID or a role ARN in the trust policy that changed when the client recreated your role. Check the exact principal ARN, not the account ID.</li>



<li><strong>Config returns no data for a region you know has resources.</strong> The recorder is per account and per region. Enabling Config in one region tells you nothing about the others, and an aggregator only aggregates regions you listed.</li>



<li><strong>Organization Config API calls fail from a member account.</strong> The organization rule and conformance pack APIs must be called from the management account or a registered delegated administrator. Calls from a member account are rejected outright.</li>



<li><strong>Drift detection returns fewer resources than the stack has.</strong> Expected. Unsupported types are not checked and not returned. Compare counts rather than assuming an empty result means clean.</li>



<li><strong>Exit code 2 every night with nothing in the plan.</strong> Read the JSON output. Either a provider is normalising an attribute on read, or a computed field is being refreshed. Both look like drift and neither is.</li>



<li><strong>The scan works manually but not on schedule.</strong> Almost always credentials. An interactive run picks up a profile from your local config that a scheduled run does not have.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Using one external ID across all clients, which removes the entire protection the external ID exists to provide.</li>



<li>Reading an empty drift report as proof of no drift, when it usually means nothing was in scope.</li>



<li>Enabling every managed rule and every conformance pack on day one, then being surprised by the evaluation charges.</li>



<li>Running the scanner with write permissions &#8220;just in case&#8221;, which turns a read-only monitoring credential into a blast radius.</li>



<li>Parsing plan text instead of JSON, which works until the day the output format changes.</li>



<li>Alerting on total drift rather than new drift, which trains everyone to ignore the alert inside a month.</li>
</ul>



<h2 class="wp-block-heading">Best practices for AWS configuration drift detection at fleet scale</h2>



<ul class="wp-block-list">
<li>Run at least two detectors per account so each covers the other&#8217;s blind spot. Config for unmanaged resources, template or state comparison for managed ones.</li>



<li>Declare security-relevant properties explicitly in templates even when the AWS default is what you want, because undeclared properties are not checked.</li>



<li>Give every client account its own external ID and rotate it when the engagement scope changes.</li>



<li>Report scan coverage alongside drift findings, so an account that silently stopped being scanned shows up as a red number.</li>



<li>Tune recording frequency per resource type rather than globally, and revisit it whenever a client&#8217;s Config bill moves.</li>



<li>Keep remediation manual and route it through code review, so the decision about why something changed is made by a person.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Can I detect drift across client accounts without AWS Organizations?</h3>



<p class="wp-block-paragraph">Yes, and for consulting it is the only workable model. You assume a read-only role in each client account from one account you control, guarded by a per-client external ID. You lose the organization aggregator, so you fan out and aggregate the results yourself, but every individual API call works the same way.</p>



<h3 class="wp-block-heading">Is AWS Config or CloudFormation drift detection better?</h3>



<p class="wp-block-paragraph">They answer different questions. CloudFormation tells you which declared properties of a managed stack no longer match the template, with exact before and after values. Config tells you what exists and whether it satisfies a rule, including resources no template ever created. If you only run one, run Config, because unmanaged resources are the bigger risk in an account you did not watch for a year.</p>



<h3 class="wp-block-heading">How often should drift detection run?</h3>



<p class="wp-block-paragraph">Nightly for the full fleet is the sensible default, because the cost is time rather than money and the alternative is finding out at audit. Tighten to hourly only for the narrow set of resources where a change is an incident, typically identity and network exposure. Everything else can wait until morning.</p>



<h3 class="wp-block-heading">Does terraform plan detect drift on resources created outside Terraform?</h3>



<p class="wp-block-paragraph">No. A plan compares state against reality, and a console-created resource is not in state, so it does not appear. This is the single most common misunderstanding about Terraform drift detection and the reason a clean plan is not a clean account.</p>



<h3 class="wp-block-heading">How do I keep AWS Config costs under control across many accounts?</h3>



<p class="wp-block-paragraph">Restrict the recorded resource types, set high-churn types to periodic recording, and be selective about rules and conformance packs rather than enabling everything. Remember that compliance evaluations create billable configuration items of their own, so adding rules increases two meters, not one.</p>



<h3 class="wp-block-heading">What permissions does a drift scanner actually need?</h3>



<p class="wp-block-paragraph">Read only. The <code>SecurityAudit</code> and <code>ViewOnlyAccess</code> managed policies cover the detectors described here. Anything that wants write access is doing remediation, which should be a separate role with a separate approval path.</p>



<h3 class="wp-block-heading">Should I buy a drift detection platform instead?</h3>



<p class="wp-block-paragraph">If the fan-out and the reporting are what you are short on, buying is defensible, and the good platforms connect drift to security context rather than just reporting divergence. What you should not buy is detection alone. The commands above are the detection, they are free, and a platform that wraps them still has the same three blind spots.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The hard part of AWS configuration drift detection across a client fleet is not running the commands. It is knowing what each command cannot see, and building the fleet view so a silent gap looks different from a clean result.</p>



<p class="wp-block-paragraph">If you remember one thing: an empty drift report and a healthy account are not the same claim, and until you can tell them apart per account, you cannot answer the client&#8217;s Tuesday email.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need this built across your client accounts?</h2>



<p class="wp-block-paragraph">Multi-account drift detection is most of a week of fiddly work the first time and an afternoon every time after. If you would rather have it running than build it, this is the kind of thing I do:</p>



<ul class="wp-block-list">
<li>Designing the cross-account read-only role and per-client external ID scheme, including the CloudFormation or Terraform your clients can review before they deploy it</li>



<li>Building the scheduled fan-out scanner with per-account isolation, throttling and scan-coverage reporting</li>



<li>Tuning AWS Config recording scope and frequency so the bill matches what the client is actually getting from it</li>



<li>Auditing existing CloudFormation stacks for undeclared properties that make drift detection report clean when it should not</li>



<li>Wiring drift results into a Grafana dashboard or a per-client report your account managers can send without editing</li>



<li>Sorting out the noisy false positives so the alerts stay worth reading after month two</li>
</ul>



<p class="wp-block-paragraph">If you already have something running, send me a scan output or the trust policy you are using and I will tell you what it is missing.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/aws-configuration-drift-detection-multi-account/">Configuration Drift Across 50 Client AWS Accounts: The Checks That Actually Fire</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Building a Secure Client Document Vault Your Accounting Firm Can Defend</title>
		<link>https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Fri, 28 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Accounting Technology]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Accounting Firms]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[Client Portal]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Classification]]></category>
		<category><![CDATA[Envelope Encryption]]></category>
		<category><![CDATA[GLBA Safeguards Rule]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Litigation Hold]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[Offboarding]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[Secure Disposal]]></category>
		<category><![CDATA[WISP]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=544</guid>

					<description><![CDATA[<p>Most accounting-firm document vaults do not fail at the bucket. They fail at the edges: a download link that outlives the engagement, a KMS key policy as wide as the bucket policy, a retention lock that cannot be undone, and an audit trail nobody switched on. Six failure families, and the control that closes each.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/">Building a Secure Client Document Vault Your Accounting Firm Can Defend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket usually reads like this: <em>client says the download link expired, can you send a new one.</em> Thirty seconds of work. Generate a fresh link, paste it into a reply, close the ticket.</p>



<p class="wp-block-paragraph">Nobody looks at the old link. If it was signed with a long expiry and mailed out, it is still live: in the client&#8217;s inbox, in the inbox of the bookkeeper on the CC line, and in whatever mail archive that firm runs. It keeps working for anyone holding the URL until the clock runs out, because a presigned S3 URL is a bearer token. AWS says exactly that in its own documentation. The URL does not know who you are. It knows that you have it.</p>



<p class="wp-block-paragraph">That is the shape of most document vault failures at a small accounting firm. Not a wide-open bucket. Not a stolen partner password. An ordinary delivery mechanism that leaks quietly and writes no log line that looks wrong six months later.</p>



<p class="wp-block-paragraph">This post covers how to build a <strong>secure client document vault</strong> for an accounting practice: what the regulation actually demands, the six failure families that decide whether it holds, and the control that closes each one. The examples are AWS because that is where most of these land, but the failure families are platform independent.</p>



<h2 class="wp-block-heading">Let the regulation write your requirements</h2>



<p class="wp-block-paragraph">Most firms treat compliance as paperwork done after the build. That is backwards. For a US accounting or tax practice, the FTC Safeguards Rule at 16 CFR Part 314 is the closest thing you will get to a free requirements document.</p>



<p class="wp-block-paragraph">It treats tax preparers, bookkeepers and accounting firms as financial institutions under the Gramm-Leach-Bliley Act. Size does not get you out of it. The parts that translate directly into architecture:</p>



<ul class="wp-block-list">
<li><strong>Access controls</strong> limiting each user to the customer information they need, and limiting customers to their own information.</li>



<li><strong>Encryption</strong> in transit over external networks and at rest, with compensating controls only where encryption is infeasible and your Qualified Individual signs off in writing.</li>



<li><strong>Multi-factor authentication</strong> for any individual accessing any information system, subject to the same written-approval exception.</li>



<li><strong>Secure disposal</strong>, with an outer bound of two years after last use unless retention is required by law or necessary for legitimate business purposes.</li>



<li><strong>Logging of authorized user activity</strong>, plus a named Qualified Individual who owns the program.</li>
</ul>



<p class="wp-block-paragraph">One more clause should shape your design more than any other. The Rule requires notifying the FTC within thirty days of discovering a security event involving the unencrypted information of at least five hundred consumers, and it counts encrypted information as unencrypted if the key was accessed by an unauthorized person.</p>



<p class="wp-block-paragraph">Read that twice. Encryption only helps at the breach-analysis stage if the key was genuinely out of reach. That is the entire argument for treating key custody as a control separate from storage.</p>



<p class="wp-block-paragraph">Separately, IRS Publication 4557 expects paid preparers to keep a Written Information Security Plan, and PTIN renewal asks you to confirm you have one. Treat the WISP as the document describing the vault you built, not a substitute for building it.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the link that outlives the engagement</h2>



<p class="wp-block-paragraph">Presigned URLs are the sensible answer to &#8220;let the client download this without an AWS account&#8221;. The problem is their expiry semantics, which almost nobody has read.</p>



<p class="wp-block-paragraph">Two clocks apply and the shorter wins: the expiry you asked for, and the lifetime of the credential you signed with. Long-lived IAM user credentials under Signature Version 4 reach seven days. A role session from <code>AssumeRole</code> dies when the session ends, one hour by default, regardless of what you requested. An EC2 instance profile is bound by metadata credential rotation. The console caps at twelve hours.</p>



<p class="wp-block-paragraph">So the two symptoms are opposites and both are common. Links dying early is a credential problem. Links living too long is a policy problem, and that is the one that hurts.</p>



<p class="wp-block-paragraph">The fix is not a smaller number. It is changing what goes in the email.</p>



<ol class="wp-block-list">
<li>Email a notification, not the file link. The client logs in with MFA and the portal mints a URL at click time.</li>



<li>Keep that expiry in minutes. The URL only has to survive a redirect and a download.</li>



<li>Scope every URL to one object and one verb.</li>



<li>If you truly need a long-lived share, use CloudFront signed URLs or signed cookies, where you can revoke by rotating the key group without touching the object.</li>
</ol>



<pre class="wp-block-code"><code># Sign a download URL valid for five minutes.
# --expires-in takes seconds; the default is 3600.
aws s3 presign s3://firm-client-vault/clients/acme-llc/workpapers.pdf 
    --expires-in 300</code></pre>



<p class="wp-block-paragraph">Run that under an assumed role, wait past the session expiry, and watch it fail well before five minutes. That is behaviour worth seeing once in a test rather than discovering during an incident review.</p>



<h2 class="wp-block-heading">Failure two: everyone inside the firm sees everything</h2>



<p class="wp-block-paragraph">This shows up in nearly every small-firm build, and the reasoning is always the same. Nine people, everyone helps with everything during filing season, and compartmentalising access is friction exactly when friction is expensive.</p>



<p class="wp-block-paragraph">The case for flat access is real, so say it out loud: a scoped model people fight gets worked around, and a workaround is worse than the flat model you started with. If your access control makes a senior accountant email a PDF to themselves at 11pm, you have made things less safe.</p>



<p class="wp-block-paragraph">So scope by engagement rather than by person, and automate the grant so nobody files a request. Make the key prefix the boundary, then grant the prefix instead of the bucket. Two statements, two jobs: the first controls what someone can enumerate, the second what they can fetch.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ListOnlyThisClient",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::firm-client-vault",
      "Condition": {
        "StringLike": { "s3:prefix": "clients/acme-llc/*" }
      }
    },
    {
      "Sid": "ReadWriteThisClient",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::firm-client-vault/clients/acme-llc/*"
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Without the prefix condition on <code>ListBucket</code>, a scoped user still gets a full inventory of your client list. Client names are confidential in this business. A directory listing that reads like a book of business is a disclosure even when no file opens.</p>



<p class="wp-block-paragraph">The other half of this family is standing administrative access. Somebody has to fix things, so somebody ends up with a permanent <code>s3:*</code> on the whole bucket. That principal is the highest-value target in the firm and usually has the weakest MFA, because it is a service account nobody wants to touch.</p>



<p class="wp-block-paragraph">Replace it with an assumable break-glass role: MFA required, short session, and an alert on every assumption. The alert is the point. Emergency access used twice a year is fine. Used weekly, it means your normal path is broken.</p>



<h2 class="wp-block-heading">Failure three: encrypted at rest, key sitting right next to it</h2>



<p class="wp-block-paragraph">Default bucket encryption gets ticked in week one and forgotten. It satisfies the letter of &#8220;encrypted at rest&#8221; and protects against exactly one threat: somebody walking out of a data centre with a disk. That is not your threat.</p>



<p class="wp-block-paragraph">Go back to the notification clause. If an attacker reaches a principal that can both read the object and use the key, your data is unencrypted for reporting purposes. The question is not &#8220;was the bucket encrypted&#8221;. It is &#8220;could whoever got in also use the key&#8221;.</p>



<ul class="wp-block-list">
<li>Use a customer managed KMS key. Only that gives you a key policy you control.</li>



<li>Name the principals allowed to call <code>Decrypt</code>, and keep that list shorter than the list allowed to call <code>GetObject</code>. Two different lists means an attacker needs two wins.</li>



<li>Use encryption context so grants can be conditioned per client, and so CloudTrail entries are readable later.</li>



<li>Enable S3 Bucket Keys to cut KMS request volume. KMS bills per API call, so a vault serving many small reads generates a surprising line item.</li>
</ul>



<p class="wp-block-paragraph">Encryption in transit needs its own control, because IAM policies say nothing about the connection. A bucket policy does:</p>



<pre class="wp-block-code"><code>{
  "Sid": "DenyPlaintextTransport",
  "Effect": "Deny",
  "Principal": "*",
  "Action": "s3:*",
  "Resource": [
    "arn:aws:s3:::firm-client-vault",
    "arn:aws:s3:::firm-client-vault/*"
  ],
  "Condition": {
    "Bool": { "aws:SecureTransport": "false" }
  }
}</code></pre>



<p class="wp-block-paragraph">An explicit deny beats every allow, including one you add by mistake later. Five lines that permanently remove a category of misconfiguration, which makes it the best value item on this page.</p>



<h2 class="wp-block-heading">Failure four: retention rules that fight your disposal duty</h2>



<p class="wp-block-paragraph">Here is the trap, and it is why I would not let an accounting firm switch on Object Lock without a conversation first.</p>



<p class="wp-block-paragraph">Object Lock has two retention modes. In governance mode, a principal holding <code>s3:BypassGovernanceRetention</code> can remove or shorten the lock. In compliance mode, nobody can. Not an administrator, not the account root user. AWS is explicit that the only way to delete a compliance-locked object before its retention date is to close the AWS account.</p>



<p class="wp-block-paragraph">Set that against the Safeguards Rule&#8217;s two-year outer bound on disposal of information you no longer need. Apply a blanket seven-year compliance-mode retention to everything landing in the vault and you have written a promise you cannot revoke, on documents you may later be obliged to destroy.</p>



<ol class="wp-block-list">
<li>Classify by document type before client. Filed returns, engagement letters and supporting workpapers have real statutory retention arguments. Raw intake uploads and scratch scans usually do not.</li>



<li>Default to governance mode. It stops accidental and casual deletion, which is what you need, while keeping a documented and audited escape hatch.</li>



<li>Give <code>s3:BypassGovernanceRetention</code> to almost nobody and alert on every use through CloudTrail.</li>



<li>Reserve compliance mode for records where an external mandate genuinely requires immutability and you chose the period deliberately.</li>



<li>Use legal hold, not a retention period, for litigation. A legal hold has no end date and stays on until explicitly removed, which is right when you cannot know how long a matter runs.</li>
</ol>



<p class="wp-block-paragraph">Two details catch people out. Object Lock requires versioning and normally has to be enabled at bucket creation, so it is a design decision rather than a retrofit. And lifecycle expiration will not delete a version still under retention, so lifecycle and Object Lock are two layers that must agree, not alternatives.</p>



<pre class="wp-block-code"><code># What lock configuration is this bucket actually running?
aws s3api get-object-lock-configuration --bucket firm-client-vault

# Freeze one object version for a matter with no known end date.
aws s3api put-object-legal-hold 
    --bucket firm-client-vault 
    --key clients/acme-llc/2023/return.pdf 
    --legal-hold Status=ON</code></pre>



<p class="wp-block-paragraph">For deletion itself, crypto-shredding is worth knowing. If a client&#8217;s documents sit under a per-client KMS key, scheduling that key for deletion renders every object unreadable without touching the objects. Clean, fast, provable, and irreversible once the waiting period ends, so it belongs behind two-person approval rather than in a script.</p>



<h2 class="wp-block-heading">Failure five: no evidence when somebody asks</h2>



<p class="wp-block-paragraph">Every control above is worth nothing if you cannot show it held. The Rule expects logging of authorized user activity and an annual written report from the Qualified Individual.</p>



<p class="wp-block-paragraph">The trap is that CloudTrail does not record S3 object-level reads and writes by default. Management events cover things like changing a bucket policy. Data events cover the actual <code>GetObject</code> calls, and they are opt-in and billed per event. Firms enable CloudTrail, see a trail in the console, assume they have a record of file access, then find during an incident that they have nothing at object level.</p>



<p class="wp-block-paragraph">Turn on data events for the vault bucket. At firm scale the volume is manageable, and it is the difference between &#8220;we believe two people opened that file&#8221; and being able to show it.</p>



<p class="wp-block-paragraph">Then decide where logs live. Access records for client financial data are themselves sensitive, so keeping them in the same account under the same administrators defeats the purpose. A separate logging account, or at minimum a separate bucket with its own key and retention, is the shape you want. If you already run Grafana Cloud or similar for infrastructure monitoring, pointing vault access logs there gives you dashboards and alerting without standing anything new up.</p>



<p class="wp-block-paragraph">The alerts worth having are boring and few: any use of the break-glass role, any change to the bucket or key policy, any bulk download outside normal working patterns, and any failed decrypt. Four alerts that fire almost never beat forty that everyone mutes.</p>



<h2 class="wp-block-heading">Failure six: the two ends of the pipeline</h2>



<p class="wp-block-paragraph">A vault only protects documents that are in it. The two ends are where firms quietly undo everything.</p>



<h3 class="wp-block-heading">Intake</h3>



<p class="wp-block-paragraph">If clients can still email a W-2 as an attachment, they will, because it is easier. Each attachment becomes customer information sitting in a mailbox, a sent-items folder and a mail archive, each with a different retention policy and access model than your vault.</p>



<p class="wp-block-paragraph">A policy memo will not fix this. Make the vault path faster than the email path: an upload link that works on a phone, no account creation for a one-off drop, and a mail rule that returns certain attachment types with the upload link instead. Then sweep the mailboxes you already have, because that backlog is real customer information and it counts.</p>



<h3 class="wp-block-heading">Offboarding</h3>



<p class="wp-block-paragraph">Seasonal staff are normal here, so offboarding happens in batches and gets rushed. Disabling the SSO login is the part everyone does. The parts that get missed:</p>



<ul class="wp-block-list">
<li>Active role sessions and refresh tokens, which survive the account being disabled until they expire on their own.</li>



<li>Presigned URLs that person generated, which stay valid because they are bound to the signing credential, not the login session.</li>



<li>Named principals still sitting in the KMS key policy.</li>



<li>Local copies on laptops and external drives. When hardware is retired, a certified erase tool such as O&amp;O SafeErase is the difference between disposal and hoping.</li>
</ul>



<p class="wp-block-paragraph">The same applies to service providers. The Rule makes you responsible for selecting providers capable of maintaining appropriate safeguards and holding them to it contractually. If an outsourced bookkeeping team touches your document workflow, that relationship is in scope, and direct network access means they need MFA too. Cloudflare Zero Trust or a business VPN such as NordLayer in front of the portal gives you somewhere to enforce that without rebuilding the application.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting a secure client document vault</h2>



<ul class="wp-block-list">
<li><strong>Links expire far earlier than the expiry you set.</strong> You signed with temporary credentials that expired first. Sign from a principal whose credential outlives the link, or shorten the link and mint on demand.</li>



<li><strong>Uploads fail with access denied despite an allow on PutObject.</strong> With a customer managed key, both the IAM policy and the key policy must permit the caller. Check <code>kms:GenerateDataKey</code>, not just S3 actions.</li>



<li><strong>A file will not delete and no policy seems to block it.</strong> Look for Object Lock retention or a legal hold on that version. They are independent and either one alone blocks the delete.</li>



<li><strong>CloudTrail shows the bucket being created but no file access.</strong> Data events are not enabled. Management events will never show <code>GetObject</code>.</li>



<li><strong>A scoped user can see every client folder name.</strong> The prefix condition is missing from <code>ListBucket</code>, or was applied to the object ARN instead of the bucket ARN.</li>



<li><strong>Lifecycle rules appear to do nothing.</strong> Versions under retention are not expired by lifecycle. Confirm lock state before assuming the rule is wrong.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes, and what to do instead</h2>



<ul class="wp-block-list">
<li><strong>Emailing the download link.</strong> Email a notification that a document is waiting, and mint the link after the client authenticates.</li>



<li><strong>Treating default bucket encryption as finished.</strong> Use a customer managed key and keep the decrypt list shorter than the read list.</li>



<li><strong>Granting bucket-wide list permissions.</strong> Condition <code>ListBucket</code> on the client prefix, or the roster leaks even when no document opens.</li>



<li><strong>Applying compliance-mode Object Lock broadly because it sounds safer.</strong> Write the retention schedule per document class first, get partners to sign it, then configure to match.</li>



<li><strong>Keeping a standing full-access principal.</strong> Replace it with an MFA-gated break-glass role that alerts on every assumption.</li>



<li><strong>Building a portal and leaving email intake open.</strong> Both channels then carry client data and only one is governed. Close the mail path and sweep the backlog.</li>



<li><strong>Storing access logs beside the data they describe.</strong> Ship S3 data events somewhere with different administrators and different keys.</li>



<li><strong>Never testing any of it.</strong> Verify a restore and a scoped-access denial quarterly. An unverified control is an assumption.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Does a small accounting firm really fall under the FTC Safeguards Rule?</h3>



<p class="wp-block-paragraph">Almost certainly, if it prepares returns or handles client financial data in the United States. The Rule&#8217;s definition of a financial institution covers activities financial in nature, and the FTC&#8217;s own examples include tax preparation firms. A narrow exception at 16 CFR 314.6 covers institutions holding information on fewer than five thousand consumers, but it exempts only four specific provisions. Encryption, MFA, access controls and disposal still apply. Confirm your own position with counsel, not a blog post.</p>



<h3 class="wp-block-heading">Should I build this or buy a client portal product?</h3>



<p class="wp-block-paragraph">Buy, if the product genuinely does what you need. A firm with no engineering capacity is safer on a maintained product than a bespoke stack nobody patches. Build when you need controls the product will not give you: per-client key isolation, your own retention logic, or integration with a pipeline you already run. The honest test is whether someone will still own the system in a year.</p>



<h3 class="wp-block-heading">Can I self-host a secure client document vault instead of using AWS?</h3>



<p class="wp-block-paragraph">Yes. Nextcloud or similar on a well-run VPS from a provider such as InterServer or Contabo can meet the same requirements, and it is often simpler to reason about. What you take on is patching, backups, TLS renewal and log retention. The regulation does not care where the data sits. It cares that the controls exist and that you can show they held.</p>



<h3 class="wp-block-heading">How long should client documents be retained?</h3>



<p class="wp-block-paragraph">There is no single number, which is why a schedule beats a default. Tax and professional standards drive minimums for filed returns and engagement documentation, and those vary by jurisdiction and engagement type. The Safeguards Rule pushes the other way. Write both directions down per document class, then configure to match.</p>



<h3 class="wp-block-heading">Is a presigned URL safe to send to a client?</h3>



<p class="wp-block-paragraph">It is safe to use, not safe to send. Treat it as a bearer token with a short life, minted after the client authenticates and consumed immediately. What travels over email should be a notification, not the credential.</p>



<h3 class="wp-block-heading">Does encrypting the bucket mean I avoid breach notification?</h3>



<p class="wp-block-paragraph">Only if the key stayed out of the attacker&#8217;s hands. The Rule treats encrypted customer information as unencrypted where the key was accessed by an unauthorized person. That is the whole argument for keeping key permissions narrower than storage permissions.</p>



<h3 class="wp-block-heading">What is the minimum viable version for a two-person practice?</h3>



<p class="wp-block-paragraph">One bucket with per-client prefixes, a customer managed key, an explicit deny on non-TLS transport, MFA on every human account, short mint-on-click links, S3 data events enabled, and a written retention schedule. That is a weekend of work and it closes five of the six failure families above.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">A secure client document vault is not defined by the storage you picked. It is defined by how narrow the path is between an attacker and a readable file, and by your ability to prove afterwards how many people walked it.</p>



<p class="wp-block-paragraph">Most firms get the storage right and lose at the edges: a link that outlives the engagement, a key policy as wide as the bucket policy, a retention setting that cannot be undone, an audit trail nobody turned on. Fix those four and the rest is ordinary engineering.</p>



<h2 class="wp-block-heading">Where I can help</h2>



<p class="wp-block-paragraph">I work with small firms and the MSPs that support them on exactly this kind of build. Things I take on regularly:</p>



<ul class="wp-block-list">
<li>Designing and building the vault itself: bucket layout, per-client prefix isolation, KMS key policy, and the portal glue that mints short-lived links.</li>



<li>Reviewing an existing setup against the Safeguards Rule&#8217;s technical elements and producing a gap list your Qualified Individual can work from.</li>



<li>Untangling retention: turning a vague policy into a per-document-class schedule, implemented with lifecycle rules, Object Lock and legal holds that do not conflict.</li>



<li>Standing up the evidence layer: S3 data events, a separate logging account, dashboards, and the small set of alerts worth waking up for.</li>



<li>Closing the email intake path without making life harder for clients, including mobile upload flows and a sweep of the historic mailbox backlog.</li>



<li>Offboarding and access reviews: break-glass roles, session revocation, key policy hygiene, and a repeatable quarterly check.</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send the thing itself: a bucket policy, an IAM policy, a CloudTrail event selector, or a description of how documents reach your clients today. It is much easier to say something useful about a real configuration than about a general question.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/secure-client-document-vault-accounting-firm/">Building a Secure Client Document Vault Your Accounting Firm Can Defend</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Twenty Client AWS Accounts and Not One Shared Admin Credential</title>
		<link>https://john-nessime.com/blog/devops/client-aws-account-access/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Thu, 27 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[AWS Organizations]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[CloudTrail]]></category>
		<category><![CDATA[Confused Deputy]]></category>
		<category><![CDATA[Cross-Account Access]]></category>
		<category><![CDATA[External ID]]></category>
		<category><![CDATA[IAM]]></category>
		<category><![CDATA[IAM Access Analyzer]]></category>
		<category><![CDATA[IAM Identity Center]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Managed Service Providers]]></category>
		<category><![CDATA[MFA]]></category>
		<category><![CDATA[Multi-Account Strategy]]></category>
		<category><![CDATA[Offboarding]]></category>
		<category><![CDATA[Permissions Boundaries]]></category>
		<category><![CDATA[Role Chaining]]></category>
		<category><![CDATA[Session Policies]]></category>
		<category><![CDATA[Source Identity]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=546</guid>

					<description><![CDATA[<p>Shared admin credentials work right up until someone resigns or an auditor asks who did what. Here's how to design client AWS account access around cross-account roles, unique external IDs and per-engineer attribution, so twenty accounts need zero shared keys and revocation is one action.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/client-aws-account-access/">Twenty Client AWS Accounts and Not One Shared Admin Credential</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">A client&#8217;s auditor asks one question: list every person at your company who can sign in to our AWS account, and show us how you know. If the honest answer starts with &#8220;whoever has the shared vault,&#8221; that isn&#8217;t an answer. It&#8217;s a hope.</p>



<p class="wp-block-paragraph">That question is what actually bites when you run infrastructure for twenty clients. Not a breach. An audit, or a resignation. Someone gives notice on a Friday and you spend Monday working out which accounts they still hold keys to, because the credential everybody shared was never attached to a person. You can&#8217;t revoke a person from a password. You can only rotate it and hope you got all twenty.</p>



<p class="wp-block-paragraph">This post covers how to design client AWS account access so that humans authenticate once against something you control, every client account holds a role that trusts exactly one thing, and every API call in a client&#8217;s CloudTrail can be traced back to a named engineer. It&#8217;s organised by failure family, because the mechanics are well documented and the failure modes are not.</p>



<h2 class="wp-block-heading">The shape: one identity plane, twenty separate trust decisions</h2>



<p class="wp-block-paragraph">The instinct that causes all the trouble is putting credentials in the middle. Client gives you a login, you store it somewhere shared, engineers take it out when they need it. Everything downstream of that decision is damage control.</p>



<p class="wp-block-paragraph">The shape that works is boring. One AWS account whose only job is identity. Your engineers exist there, once, in IAM Identity Center or federated from whatever directory you already pay for. Each client account holds an IAM role whose trust policy names one principal in that account and requires an external ID. Nobody stores a client credential, because there is no client credential. There&#8217;s a role your engineers can ask STS for a short-lived session against, and a client who can delete that role whenever they like.</p>



<p class="wp-block-paragraph">There is a second design worth knowing about: inviting client accounts into your own AWS Organization. It&#8217;s genuinely better in some ways. You get service control policies, resource control policies, and centralised root access management across the estate, all of which are organisation-scoped features you simply cannot use on an account you don&#8217;t own. It&#8217;s also a much bigger commitment. You take on consolidated billing, the client&#8217;s spend flows through you, and pulling an account back out of an organisation at the end of a contract is a real project rather than a policy change.</p>



<p class="wp-block-paragraph">For most consultancies and managed service providers, the client keeps their own organisation and you get a role. That&#8217;s the model the rest of this assumes.</p>



<h2 class="wp-block-heading">Failure family 1: credentials you can&#8217;t revoke</h2>



<p class="wp-block-paragraph">The shared IAM user with an access key is the default because it works immediately. It keeps working right up until one of three moments: an engineer leaves, a client asks who did something, or a laptop gets stolen.</p>



<p class="wp-block-paragraph">Access keys have no expiry. They&#8217;re bearer tokens with an indefinite lifetime, and the only way to invalidate one is to go into each account and delete it. At twenty accounts that&#8217;s twenty deliberate actions performed under time pressure, and the one you forget is the one that matters. A role session, by contrast, expires on its own. That&#8217;s the entire argument.</p>



<p class="wp-block-paragraph">A shared password manager still earns its place, just not for admin keys. Bitwarden or 1Password is the right home for per-client external IDs, account numbers, role ARNs and the client-side contacts you need when something breaks at two in the morning. Metadata, not credentials.</p>



<h2 class="wp-block-heading">Failure family 2: a trust policy that trusts more than you meant</h2>



<p class="wp-block-paragraph">This is the invisible one. The role gets created, the access works, everyone moves on, and the trust policy quietly grants far more than anyone intended.</p>



<p class="wp-block-paragraph">Two mistakes travel together. The first is naming your account root in the principal. Writing <code>arn:aws:iam::111122223333:root</code> does not mean &#8220;the root user of my account.&#8221; It means any principal in that account that has permission to call AssumeRole. Every engineer, every Lambda function, every CI runner, every future role you haven&#8217;t created yet. It delegates the decision to your own account&#8217;s IAM, which is fine when your IAM is tidy and quietly catastrophic when it isn&#8217;t.</p>



<p class="wp-block-paragraph">The second is skipping the external ID. AWS calls the problem it solves the confused deputy: you are a third party that holds the ability to assume roles in many customer accounts, and without a shared secret in the condition, one customer who learns another customer&#8217;s role ARN could persuade your systems to act against it. The external ID is the value that makes each trust relationship specific rather than generic.</p>



<p class="wp-block-paragraph">Generate it yourself, one per client, as a UUID or similar unguessable value. The client must not be the one choosing it, because if they can choose it they can choose someone else&#8217;s, and the control does nothing.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::111122223333:role/MspOperationsAccess"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": {
          "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">The principal names one role, not the account. If that role is compromised you have one thing to fix. If the account is named, you have an inventory problem you didn&#8217;t know you had.</p>



<h2 class="wp-block-heading">Failure family 3: CloudTrail says the role, not the human</h2>



<p class="wp-block-paragraph">You&#8217;ve done everything above and a client emails asking who deleted the staging load balancer. You open their CloudTrail and every event attributes to the same assumed role. The role is shared by your whole team by design, so the log tells you your company did it. That&#8217;s not what they asked.</p>



<p class="wp-block-paragraph">There are two knobs and they are not equivalent. The role session name is chosen freely by whoever calls AssumeRole, appears in the assumed-role ARN, and is trivially set to anything. It&#8217;s useful, but it&#8217;s a label, not evidence. Source identity is the one that carries weight: once set on a session it cannot be changed, it persists across chained role sessions, and it lands in CloudTrail in the session context of every subsequent call.</p>



<p class="wp-block-paragraph">The catch is that the role&#8217;s trust policy has to permit <code>sts:SetSourceIdentity</code> alongside <code>sts:AssumeRole</code>, and you can require it with a condition so a session without one is refused outright.</p>



<pre class="wp-block-code"><code>"Action": [
  "sts:AssumeRole",
  "sts:SetSourceIdentity"
],
"Condition": {
  "StringEquals": {
    "sts:ExternalId": "6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02"
  },
  "StringLike": {
    "sts:SourceIdentity": "*@yourcompany.com"
  }
}</code></pre>



<p class="wp-block-paragraph">If your engineers arrive through SAML federation, the identity provider can populate source identity from a directory attribute such as email, which means nobody has to remember to pass it. For direct CLI use it&#8217;s an explicit flag:</p>



<pre class="wp-block-code"><code>aws sts assume-role 
  --role-arn arn:aws:iam::444455556666:role/MspEngineerAccess 
  --role-session-name acme-investigation 
  --external-id 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02 
  --source-identity jane.doe@yourcompany.com</code></pre>



<p class="wp-block-paragraph">Getting this right is what turns &#8220;we take security seriously&#8221; into a sentence you can back with a log query. It is also, in my experience of what clients actually ask for, the single most persuasive thing on the list.</p>



<h2 class="wp-block-heading">Failure family 4: one identity, twenty blast radii</h2>



<p class="wp-block-paragraph">A hub account is a concentration of risk. That&#8217;s the trade you&#8217;re making, and it&#8217;s worth making, but only if the hub is held to a higher standard than the accounts it reaches into.</p>



<p class="wp-block-paragraph">Practically: not every engineer gets permission to assume every client&#8217;s role. Group permissions in the hub so the people on the Acme engagement reach Acme and nothing else. Use per-client role names rather than one shared name across all twenty, so a policy in the hub can&#8217;t wildcard its way into a client you stopped working with a year ago.</p>



<p class="wp-block-paragraph">Tier the roles by what they&#8217;re for, not by who&#8217;s asking. A read-only role for investigation covers most of the work most days. A change role carries the write permissions the engagement actually needs. Break-glass sits separate, with a short maximum session duration and an alarm on every use.</p>



<p class="wp-block-paragraph">Two extras are worth knowing about even if you don&#8217;t reach for them immediately. Session policies let the caller narrow a session further at assume time, which is useful for automation that only needs one bucket. Permissions boundaries cap what any role or user created inside the client account can do, which matters when part of your job is creating IAM there.</p>



<p class="wp-block-paragraph">If your team already routes through a business VPN or zero trust gateway with stable egress addresses, a source IP condition on the trust policy is a cheap extra layer. NordLayer, Tailscale and Cloudflare&#8217;s zero trust products all give you predictable egress. Be honest about the cost, though: the moment something runs from CI, a client&#8217;s network or a laptop on hotel wifi, an IP condition becomes a support ticket. I&#8217;d add it once everything else works, not before.</p>



<h2 class="wp-block-heading">Failure family 5: standing administrator access nobody remembers granting</h2>



<p class="wp-block-paragraph">Somewhere in the twenty accounts there is a role with AdministratorAccess attached, created during a migration, described in the ticket as temporary. It&#8217;s been eighteen months.</p>



<p class="wp-block-paragraph">Here&#8217;s the honest part: AWS does not give you a native just-in-time approval workflow for cross-account role assumption. There&#8217;s no built-in &#8220;request four hours of write access, get approved in Slack, access expires automatically&#8221; for this pattern. You either build that on top of STS and your ticketing system, or you accept standing access and compensate with tight session durations, aggressive monitoring and a genuine periodic review. Both are defensible. Pretending you have JIT when you have a wiki page is not.</p>



<p class="wp-block-paragraph">The cheap version that actually helps: set the maximum session duration on client roles to something short, alert on assumption of the break-glass role, and run a quarterly pass over service last accessed data to strip permissions nobody has used. IAM can generate that report per role.</p>



<pre class="wp-block-code"><code># Kick off the report, then fetch it with the returned job ID
aws iam generate-service-last-accessed-details 
  --arn arn:aws:iam::444455556666:role/MspEngineerAccess

aws iam get-service-last-accessed-details --job-id JOB_ID_FROM_ABOVE</code></pre>



<p class="wp-block-paragraph">Run that in each client account and the conversation about narrowing permissions stops being theoretical. You&#8217;re not guessing what the engagement needs; you&#8217;re reading it.</p>



<h2 class="wp-block-heading">Failure family 6: offboarding that only half happens</h2>



<p class="wp-block-paragraph">The whole point of the hub is that offboarding is one action. Disable the person in your directory and every path into every client account closes at once, because there was never a credential in a client account with their name on it.</p>



<p class="wp-block-paragraph">That&#8217;s true only if nothing crept in sideways. Things that creep in sideways:</p>



<ul class="wp-block-list">
<li>IAM users created directly in a client account during an incident, because it was faster</li>

<li>Access keys minted under a role by automation and never inventoried</li>

<li>A client&#8217;s own console user shared with your team &#8220;just for the migration&#8221;</li>

<li>Third party tools connected with their own cross-account roles that outlived the trial</li>

<li>A departing engineer&#8217;s laptop with cached SSO tokens and stale profiles still on disk</li>
</ul>



<p class="wp-block-paragraph">IAM Access Analyzer is the tool that finds most of the first four. Its external access findings show every principal outside the account that can reach into it, which is exactly the list a client wants to see, and it&#8217;s also the list that reveals the monitoring vendor nobody remembers signing up for. For the laptop, whatever your hardware disposal process is, make sure it involves an actual wipe rather than a factory reset; tools like O&amp;O SafeErase exist for that reason and are cheap next to the alternative.</p>



<h2 class="wp-block-heading">Rolling client AWS account access out across twenty accounts</h2>



<p class="wp-block-paragraph">Doing this by hand twenty times guarantees twenty subtly different configurations. Make it a template and a checklist.</p>



<ol class="wp-block-list">
<li>Write one CloudFormation template that creates the role, its trust policy and its permissions, parameterised by external ID and your operations account ID.</li>

<li>Generate a unique external ID per client and store it with the client&#8217;s account metadata, not in the template.</li>

<li>Send the client a launch stack link rather than asking for credentials. They deploy it in their own account, in their own console, and can read exactly what they&#8217;re granting.</li>

<li>Record the account ID, role ARN, external ID location, engagement scope and review date in a single inventory file in version control.</li>

<li>Generate everyone&#8217;s local AWS profiles from that inventory so nobody hand-edits a config file.</li>

<li>Verify each one with a call that proves identity and nothing else before you touch anything real.</li>
</ol>



<p class="wp-block-paragraph">The generated profile chains from your federated session into the client role. The AWS CLI reads these from the shared config file and handles the assume-role call for you:</p>



<pre class="wp-block-code"><code># ~/.aws/config
[profile msp-hub]
sso_session = company
sso_account_id = 111122223333
sso_role_name = MspOperationsAccess
region = eu-west-1

[profile acme-readonly]
role_arn = arn:aws:iam::444455556666:role/MspEngineerReadOnly
source_profile = msp-hub
external_id = 6f1c2e94-3a77-4c2b-9f0d-8e5a1b7c4d02
role_session_name = jane-doe
duration_seconds = 3600
region = eu-west-1</code></pre>



<p class="wp-block-paragraph">Then the two-second sanity check that has saved me from more mistakes than any other command:</p>



<pre class="wp-block-code"><code>aws sts get-caller-identity --profile acme-readonly</code></pre>



<p class="wp-block-paragraph">It returns the account ID and the assumed-role ARN. Read the account number before you run anything destructive. Twenty profiles in one config file is exactly the situation where muscle memory picks the wrong one.</p>



<p class="wp-block-paragraph">For scheduled work that runs without a human, the same trust relationship serves a Lambda function or an EC2 instance role in your hub. A low-cost VPS from Contabo or InterServer will also do the job, but note the trade: an instance profile inside AWS gets rotated credentials automatically, while a box outside AWS needs a long-lived credential to bootstrap from, which reintroduces exactly what this design removes. Inside AWS is the cleaner answer.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the failures you will actually hit</h2>



<h3 class="wp-block-heading">AccessDenied on AssumeRole</h3>



<p class="wp-block-paragraph">Three separate causes produce nearly the same message, and they need different fixes. Either the client&#8217;s trust policy doesn&#8217;t name your principal, or the external ID you sent doesn&#8217;t match what the condition expects, or your own identity in the hub lacks permission to call AssumeRole on that role ARN. Check them in that order, because the first two require the client and the third doesn&#8217;t.</p>



<h3 class="wp-block-heading">Switch Role in the console silently fails</h3>



<p class="wp-block-paragraph">This one wastes an afternoon if you don&#8217;t know it. The console&#8217;s switch role feature does not support roles that require an external ID. It&#8217;s a documented limitation, not a bug in your policy. Engineers who need console access to those accounts should reach them through federation rather than the switch role box.</p>



<h3 class="wp-block-heading">Sessions expiring after an hour when you asked for longer</h3>



<p class="wp-block-paragraph">When one role session assumes another role, that&#8217;s role chaining, and a chained session is capped at one hour regardless of the role&#8217;s maximum session duration setting. Since your engineers assume the client role from an already-assumed session in the hub, you&#8217;re chaining by definition. Plan around the hour rather than fighting it.</p>



<h3 class="wp-block-heading">An authorization message you can&#8217;t read</h3>



<p class="wp-block-paragraph">Some denials come back with a long encoded blob attached. STS will decode it if your identity has permission to, and the decoded version usually names the exact statement that denied you.</p>



<pre class="wp-block-code"><code>aws sts decode-authorization-message --encoded-message &lt;encoded-blob&gt;</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Naming the account root as the trust principal instead of the specific operations role</li>

<li>Letting the client pick the external ID, or reusing one value across several clients</li>

<li>Treating the role session name as an audit trail when it&#8217;s a caller-supplied string</li>

<li>Asking a client for console credentials because a template felt like too much effort</li>

<li>Granting AdministratorAccess for a migration and never revisiting it</li>

<li>Building the hub properly and then leaving one legacy IAM user in one account as a fallback</li>

<li>Assuming that removing someone from your directory also removed the keys they created by hand</li>
</ul>



<h2 class="wp-block-heading">Best practices worth the effort</h2>



<ul class="wp-block-list">
<li>One identity plane, federated, with MFA enforced at the point the human authenticates</li>

<li>A unique unguessable external ID per client, generated by you and stored with client metadata</li>

<li>Source identity required in the trust policy so CloudTrail names a person, not a role</li>

<li>Roles tiered by purpose: read-only for investigation, change for the engagement, break-glass alarmed</li>

<li>Client-owned policies, deployed by the client from your template, revocable by the client in one action</li>

<li>A version-controlled inventory of every account, role, scope and review date</li>

<li>Access Analyzer external access findings reviewed on a schedule, in every client account</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Can&#8217;t I just ask each client to create an IAM user for me?</h3>



<p class="wp-block-paragraph">You can, and it&#8217;ll work on day one. The problem is day four hundred. That user&#8217;s access key never expires, it isn&#8217;t attached to any individual on your team, and revoking it means someone in the client&#8217;s organisation remembering to act. A role costs the client the same five minutes to create and hands them a control they can exercise unilaterally.</p>



<h3 class="wp-block-heading">Do I need a different external ID for every client?</h3>



<p class="wp-block-paragraph">Yes. A shared external ID across your client base defeats the purpose, because the value&#8217;s entire job is to be specific to one trust relationship. Generate a UUID per client, treat it as a secret, and never let the client choose it.</p>



<h3 class="wp-block-heading">Should I bring client accounts into my own AWS Organization?</h3>



<p class="wp-block-paragraph">Only if you&#8217;re also taking on their billing and the commercial relationship supports it. Inside your organisation you get service control policies, resource control policies and centralised root access management, which are real advantages. Outside it, those stay the client&#8217;s tools and your job is to recommend them well. Most consulting engagements sit outside.</p>



<h3 class="wp-block-heading">How do I prove a departed engineer no longer has access?</h3>



<p class="wp-block-paragraph">Show the directory record disabling them, then show that no IAM user or access key in the client account carries their name, then show Access Analyzer external access findings listing only your operations role as an external principal. Three artefacts, all exportable. That&#8217;s the evidence pack.</p>



<h3 class="wp-block-heading">Does MFA carry across the cross-account role assumption?</h3>



<p class="wp-block-paragraph">Don&#8217;t assume it does. The reliable place to enforce MFA is where the human authenticates in your identity plane, and the reliable control at the client boundary is the combination of a narrowly scoped principal and the external ID condition. If you want an MFA condition in the client&#8217;s trust policy as well, test it end to end against your actual federation path before you rely on it, because the request context differs between direct assumption and chained sessions.</p>



<h3 class="wp-block-heading">What&#8217;s the minimum viable version if I only have three clients?</h3>



<p class="wp-block-paragraph">A hub account with federated engineers, one role per client scoped to a named principal with a unique external ID, and source identity required. Skip the tiering, skip the IP conditions, skip the JIT ambitions. Those three things give you revocation, attribution and a clean answer for the auditor, which is most of the value.</p>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Client AWS account access is not really a permissions problem. It&#8217;s a revocation and attribution problem wearing a permissions costume. Any design where a credential exists that isn&#8217;t tied to a specific human will fail you at exactly the wrong moment, and no amount of policy tuning fixes that, because the gap isn&#8217;t in the policy.</p>



<p class="wp-block-paragraph">Put the humans in one place you control. Put a narrowly scoped, externally identified role in each client account. Require source identity so the log names a person. Everything else on this page is refinement; those three are the structure. And the pleasant side effect is that &#8220;here&#8217;s how you revoke us, unilaterally, in one action&#8221; is a genuinely good thing to be able to say in a sales conversation.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your client access model?</h2>



<p class="wp-block-paragraph">This is the kind of work I do for consultancies and managed service providers who&#8217;ve grown past the point where a shared vault is survivable. Specifically:</p>



<ul class="wp-block-list">
<li>Reviewing existing cross-account trust policies for over-broad principals and missing external IDs</li>

<li>Designing a hub account and federation model that scales past a handful of clients without per-client special cases</li>

<li>Building the CloudFormation or Terraform template clients deploy themselves, so onboarding stops involving credentials</li>

<li>Turning on source identity end to end, including the identity provider side, so CloudTrail attributes to named engineers</li>

<li>Producing the offboarding evidence pack clients and auditors ask for, and automating it so it isn&#8217;t a scramble</li>

<li>Narrowing long-standing administrator roles down using service last accessed data, without breaking the engagement</li>
</ul>



<p class="wp-block-paragraph">If you want a concrete starting point, send me one trust policy and the AccessDenied message it produces. That&#8217;s usually enough to tell where the model is leaking.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/client-aws-account-access/">Twenty Client AWS Accounts and Not One Shared Admin Credential</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Automating Server Provisioning for New Clients Without Building a Snowflake Farm</title>
		<link>https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Wed, 12 Aug 2026 13:00:00 +0000</pubDate>
				<category><![CDATA[DevOps]]></category>
		<category><![CDATA[System Administration]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Ansible]]></category>
		<category><![CDATA[Automation]]></category>
		<category><![CDATA[Client Onboarding]]></category>
		<category><![CDATA[cloud-init]]></category>
		<category><![CDATA[Freelancing]]></category>
		<category><![CDATA[Infrastructure]]></category>
		<category><![CDATA[Infrastructure as Code]]></category>
		<category><![CDATA[OpenTofu]]></category>
		<category><![CDATA[Secrets Management]]></category>
		<category><![CDATA[Server Provisioning]]></category>
		<category><![CDATA[SOPS]]></category>
		<category><![CDATA[SSH]]></category>
		<category><![CDATA[Sysadmin]]></category>
		<category><![CDATA[Terraform]]></category>
		<category><![CDATA[VPS]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=192</guid>

					<description><![CDATA[<p>Hand-built client servers fail quietly: you only find out they were never reproducible on the day you need to rebuild one. Here is how to automate server provisioning for new clients using cloud-init for first boot, Ansible for convergence, per-client secrets and a verification gate that refuses to hand over a half-built box.</p>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The email is usually short. &#8220;Can you spin up a staging box that matches production?&#8221; You open your notes from eight months ago, find a half-finished shell history and a text file called <code>setup-notes-final2.txt</code>, and the honest answer is no. You can build something close. You cannot build the same machine.</p>



<p class="wp-block-paragraph">That is the failure that actually bites when you provision servers by hand for a handful of clients. It is invisible for months. Nothing alerts. Nothing breaks. Then a disk dies, or a client asks for a second region, or you want to hand the whole thing over, and the gap between &#8220;the server works&#8221; and &#8220;the server is reproducible&#8221; turns into a weekend.</p>



<p class="wp-block-paragraph">This post covers how to automate server provisioning for new clients in a way that survives that moment: what to put in the image, what to put in first boot, what to leave to configuration management, how to keep one client&#8217;s credentials from ever touching another client&#8217;s box, and how to prove a build is finished instead of assuming it.</p>



<h2 class="wp-block-heading">The four failure families worth designing against</h2>



<p class="wp-block-paragraph">Automation is not the goal. Automation is what you reach for once you have decided which of these you refuse to keep living with. Pick the ones that apply to you and let them shape the build, because trying to solve all four at once is how provisioning projects stall.</p>



<h3 class="wp-block-heading">1. You cannot rebuild it</h3>



<p class="wp-block-paragraph">The test is blunt: destroy the machine and rebuild it from your repository, with no shell access to the old one. If you cannot do that, you do not have a server, you have an artefact. Every hand-run <code>apt install</code>, every file edited over SSH at 1am, every &#8220;I&#8217;ll document that later&#8221; is a piece of the machine that exists nowhere but the machine.</p>



<p class="wp-block-paragraph">This is the one that stays invisible longest. A hand-built box can run happily for two years. The cost lands entirely in one bad afternoon.</p>



<h3 class="wp-block-heading">2. Every client is a slightly different server</h3>



<p class="wp-block-paragraph">Client A got fail2ban. Client B got it two months later with different jail settings. Client C never got it because that build happened on a Friday. Now a CVE lands in a package you installed on four of six boxes and you have to go look at each one to find out.</p>



<p class="wp-block-paragraph">Drift is not just untidy. It destroys your ability to reason about the fleet as a fleet. Any statement that starts &#8220;all our clients have&#8230;&#8221; becomes a guess.</p>



<h3 class="wp-block-heading">3. One credential, too much blast radius</h3>



<p class="wp-block-paragraph">This one is specific to consulting and it is the one people get wrong most often. If the same SSH key opens every client&#8217;s server, and the same API token provisions every client&#8217;s infrastructure, then a compromise of your laptop is a compromise of everybody. You are a shared dependency with none of the controls a shared dependency is supposed to have.</p>



<p class="wp-block-paragraph">Worse, it is contractually awkward. Plenty of client agreements have language about access control that a single reused key quietly violates.</p>



<h3 class="wp-block-heading">4. The handover you cannot perform</h3>



<p class="wp-block-paragraph">Engagements end. When one does, the client should be able to take a repository, run it, and get their infrastructure. If the only way to reproduce their environment is to hire you again, that is not leverage, it is a liability. It also makes you slower to say yes to the next client, because your capacity is tied up in machines only you understand.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">How to automate server provisioning for new clients</h2>



<p class="wp-block-paragraph">The stack that holds up in practice has four layers, and the important part is the boundary between them. Most broken provisioning setups I see are broken because someone pushed work into the wrong layer.</p>



<h3 class="wp-block-heading">Layer 1: the machine itself</h3>



<p class="wp-block-paragraph">Terraform or OpenTofu talks to your provider&#8217;s API and creates the server, the firewall rules, the volumes and the DNS records. Whether you run a VPS at InterServer, Hetzner, Vultr or DigitalOcean, or full cloud on AWS, the shape is the same: a declarative file that says what should exist.</p>



<p class="wp-block-paragraph">Worth knowing before you commit: HashiCorp moved Terraform from the MPL to the Business Source License, which is source-available rather than open source, and the community forked the last MPL release as OpenTofu under the Linux Foundation. For an independent engineer building client infrastructure, either is workable. Terraform has the larger provider and module ecosystem and the commercial backing. OpenTofu gives you a licence that will not be renegotiated under you, which matters more when the code you write ends up in a client&#8217;s repository and out of your control. The HCL, provider protocol and state model are shared, so the switching cost is low in both directions. I lean OpenTofu for client work for the licence certainty, and that is a judgement call, not a technical verdict.</p>



<p class="wp-block-paragraph">Keep state remote and locked from day one, and keep one state file per client. Shared state is the fastest way to turn a mistake on one engagement into an outage on another.</p>



<h3 class="wp-block-heading">Layer 2: first boot with cloud-init</h3>



<p class="wp-block-paragraph">cloud-init runs once, on first boot, before you have any way in. Its job is narrow: create the account you will connect as, install its key, lock down password and root login, and get out of the way. Everything else belongs in the next layer.</p>



<pre class="wp-block-code"><code>#cloud-config
hostname: web01
fqdn: web01.acme.internal
timezone: Etc/UTC

users:
  - name: deploy
    groups: [sudo]
    shell: /bin/bash
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3Nz... provisioning key for this client only

# Refuse passwords and direct root logins from the very first boot,
# so there is no window where the box is reachable with weaker settings.
ssh_pwauth: false
disable_root: true

package_update: true
packages:
  - python3
  - sudo</code></pre>



<p class="wp-block-paragraph">Two deliberate omissions. There is no <code>package_upgrade: true</code>, because a full upgrade at first boot can run for minutes, holds the package lock, and is the single most common cause of the race described in the troubleshooting section below. Do the upgrade in Ansible where you can see it fail. And there is no application setup, because anything in <code>runcmd</code> is a script you will never test again.</p>



<p class="wp-block-paragraph"><code>python3</code> is there so Ansible has an interpreter to work with. Some images ship it already; installing it is cheap insurance.</p>



<p class="wp-block-paragraph">Validate the file before it ever reaches a provider, because a YAML error in user-data fails silently and you find out by not being able to log in:</p>



<pre class="wp-block-code"><code>cloud-init schema --config-file user-data.yaml</code></pre>



<h3 class="wp-block-heading">Layer 3: convergence with Ansible</h3>



<p class="wp-block-paragraph">This is where the actual build lives: packages, users, firewall, TLS, web server, database, backup agent, monitoring agent, log shipping. Ansible is agentless and runs over SSH, which fits consulting well because you are not asking a client to accept a permanent daemon on their box.</p>



<p class="wp-block-paragraph">The layout that scales across clients separates the shared build from the per-client differences:</p>



<pre class="wp-block-code"><code>infra/
  roles/                  # shared, versioned, the same for everyone
    base/
    hardening/
    web/
    backup/
  clients/
    acme/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
    globex/
      inventory/hosts.ini
      group_vars/all.yml
      secrets/vars.sops.yml
      terraform/
  site.yml</code></pre>



<p class="wp-block-paragraph">Roles are the product. Client directories are configuration. If you find yourself copying a role into a client directory and editing it, that is a variable you have not extracted yet.</p>



<p class="wp-block-paragraph">Hand the inventory over from Terraform rather than maintaining it by hand. The simplest reliable version reads the outputs as JSON:</p>



<pre class="wp-block-code"><code>cd clients/acme/terraform

tofu output -json client_hosts 
  | jq -r '.value[] | "(.name) ansible_host=(.ipv4)"' 
  &gt; ../inventory/hosts.ini</code></pre>



<p class="wp-block-paragraph">There are richer options, including inventory plugins that read Terraform state directly, but a generated file has a property worth keeping: you can read it, diff it, and commit it. When a provisioning run targets the wrong host, that diff is how you find out.</p>



<h3 class="wp-block-heading">Layer 4: the verification gate</h3>



<p class="wp-block-paragraph">This is the layer almost everyone skips, and it is the one that turns provisioning from a script into a process. A playbook that finishes without errors has told you that no task raised an exception. It has not told you the server works.</p>



<p class="wp-block-paragraph">Write a separate play whose only job is to assert the finished state, and run it as the last step of every provision:</p>



<pre class="wp-block-code"><code>- name: Verify the host is genuinely ready
  hosts: all
  gather_facts: true
  tasks:
    # raw, not command: this may run before Python is guaranteed present
    - name: Wait for cloud-init to finish
      ansible.builtin.raw: cloud-init status --wait
      changed_when: false

    - name: Collect service state
      ansible.builtin.service_facts:

    - name: Core services are running
      ansible.builtin.assert:
        that:
          - ansible_facts.services['nginx.service'].state == 'running'
          - ansible_facts.services[ssh_unit].state == 'running'
        fail_msg: "Core services are not up on {{ inventory_hostname }}"

    - name: Site answers over TLS
      ansible.builtin.uri:
        url: "https://{{ client_domain }}/"
        status_code: 200
      delegate_to: localhost</code></pre>



<p class="wp-block-paragraph">Note <code>ssh_unit</code> as a variable rather than a literal: the SSH unit is <code>ssh.service</code> on Debian and Ubuntu but <code>sshd.service</code> on RHEL-family systems. Hard-coding it is how a verification play passes on one distro and fails confusingly on another.</p>



<p class="wp-block-paragraph">The point of the gate is not that it catches exotic bugs. It is that it catches the ordinary ones: the service that installed but did not enable, the certificate that issued but did not get loaded, the firewall rule that opened the wrong port.</p>



<h2 class="wp-block-heading">Keeping clients apart: secrets and access</h2>



<p class="wp-block-paragraph">Generate a fresh SSH keypair per client engagement. It costs you thirty seconds and it means revoking access to one client&#8217;s infrastructure is a single, contained action.</p>



<p class="wp-block-paragraph">For secrets in the repository you have two credible options. Ansible Vault is built in and needs nothing extra, but it encrypts whole files, so your diffs become meaningless blobs and rotating a key across several clients is tedious. SOPS encrypts values while leaving keys readable, which means a code review can still see that a variable changed without seeing what it changed to, and it supports age, PGP and the cloud KMS services. For a repo that a client might eventually inherit, that readability is worth the extra tool.</p>



<pre class="wp-block-code"><code># .sops.yaml at the repo root: different key per client
creation_rules:
  - path_regex: clients/acme/secrets/.*.ya?ml$
    age: age1acme...
  - path_regex: clients/globex/secrets/.*.ya?ml$
    age: age1globex...</code></pre>



<p class="wp-block-paragraph">Separate keys per client is the whole point. One shared key and you are back to a single blast radius with extra ceremony.</p>



<p class="wp-block-paragraph">For handing the initial credentials to a client, use something with an audit trail and an expiry. A shared vault item in 1Password or Bitwarden is fine. Email and chat are not, because the secret then lives in two message histories neither of you controls.</p>



<p class="wp-block-paragraph">If you want ongoing access without exposing SSH to the internet at all, an overlay network like Tailscale installed by the base role is a clean way to do it. The provisioning path still uses a public address for the first run, then the firewall closes it.</p>



<h2 class="wp-block-heading">Troubleshooting a provisioning run</h2>



<h3 class="wp-block-heading">Ansible connects before cloud-init has finished</h3>



<p class="wp-block-paragraph">The most common failure by a wide margin. SSH comes up before cloud-init is done, Ansible connects, and then the first package task fails because cloud-init still holds the package manager lock. The error points at apt or dnf, which sends you looking in entirely the wrong place.</p>



<p class="wp-block-paragraph">Fix it by blocking explicitly at the top of the first play, as in the verification snippet above. Check the state manually with:</p>



<pre class="wp-block-code"><code>cloud-init status --wait --long
sudo grep -i Traceback /var/log/cloud-init.log</code></pre>



<p class="wp-block-paragraph">One caveat that catches people: never put <code>cloud-init status --wait</code> inside a <code>runcmd</code> or <code>bootcmd</code> block. It will deadlock waiting for the process that is running it.</p>



<h3 class="wp-block-heading">cloud-init appears to have done nothing</h3>



<p class="wp-block-paragraph">Two usual causes. Either the user-data was malformed, in which case cloud-init skipped it rather than erroring loudly, or the modules already ran on a previous boot and the semaphore files in <code>/var/lib/cloud/instance/sem/</code> are preventing a rerun.</p>



<pre class="wp-block-code"><code># what actually ran, and how long each stage took
cloud-init analyze show

# the human-readable output of scripts and package installs
sudo less /var/log/cloud-init-output.log

# on a throwaway test box only: wipe state and start over
sudo cloud-init clean --logs --reboot</code></pre>



<p class="wp-block-paragraph">Never run <code>cloud-init clean</code> on a production host. It removes cached state and the following boot will behave as a first boot.</p>



<h3 class="wp-block-heading">Host key verification failed on a rebuilt server</h3>



<p class="wp-block-paragraph">Rebuild a machine on the same IP and every client with the old key in <code>known_hosts</code> refuses to connect. The tempting fix is to disable host key checking in <code>ansible.cfg</code>. Do not. That turns a real integrity check into a permanent no-op across every client you manage.</p>



<p class="wp-block-paragraph">Remove the stale entry and pin the new one as part of the provisioning flow:</p>



<pre class="wp-block-code"><code>ssh-keygen -R 203.0.113.10
ssh-keyscan -H 203.0.113.10 &gt;&gt; ~/.ssh/known_hosts</code></pre>



<p class="wp-block-paragraph">Better still, if your provider exposes the host key fingerprint through its API or console output, compare it before you accept. That closes the trust-on-first-use gap properly.</p>



<h3 class="wp-block-heading">The playbook reports changes on every single run</h3>



<p class="wp-block-paragraph">Almost always a <code>shell</code> or <code>command</code> task with no <code>creates</code>, <code>changed_when</code> or module equivalent. Ansible cannot know whether a raw command changed anything, so it assumes yes. Real changed counts are what let you spot drift, and a playbook that always reports twelve changes tells you nothing.</p>



<p class="wp-block-paragraph">Run against an already-provisioned host in check mode. Anything that reports a change is either genuine drift or a task you need to fix:</p>



<pre class="wp-block-code"><code>ansible-playbook -i clients/acme/inventory/hosts.ini site.yml --check --diff</code></pre>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Putting the application build in <code>runcmd</code>. It runs once, never gets tested, and is invisible to every later change.</li>

<li>One shared SSH key and one shared API token across all clients. Convenient right up to the moment it is not.</li>

<li>Copying a role into a client directory to make one small change, instead of extracting a variable.</li>

<li>Treating &#8220;the playbook exited zero&#8221; as proof the server is ready. It is proof no task threw.</li>

<li>Local Terraform state on your laptop. It is a single point of failure that also cannot be handed over.</li>

<li>Provisioning the server but not the DNS, TLS, monitoring and backups. A box nobody watches and nobody can restore is not finished.</li>

<li>Automating the build and then fixing production by hand. One SSH session and you are back to a snowflake.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li><strong>Rebuild before you need to.</strong> Destroy and rebuild a client&#8217;s staging host on a schedule. It is the only honest test of reproducibility.</li>

<li><strong>Version your roles and pin them per client.</strong> A shared role that everyone tracks on main means one commit changes six clients at once.</li>

<li><strong>Keep first boot minimal.</strong> If it can wait for Ansible, it should.</li>

<li><strong>Make monitoring part of provisioning.</strong> Whether that is Prometheus you run yourself, Grafana Cloud, or a lighter uptime service, the agent goes on in the base role, not as a follow-up ticket.</li>

<li><strong>Same for backups.</strong> An offsite target such as Backblaze B2 configured at build time, plus a restore test, or you have a backup you have never proven.</li>

<li><strong>Separate credentials per client, always.</strong> Keys, tokens, encryption keys, state backends.</li>

<li><strong>Write the runbook as you build.</strong> The provisioning repo&#8217;s README is the handover document. If it is accurate, the engagement can end cleanly.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Do I need Terraform if I only manage a few VPS instances?</h3>



<p class="wp-block-paragraph">Not necessarily. If you are creating three servers a year through a provider&#8217;s control panel, the honest answer is that Terraform adds a state file to look after for very little return. Start with cloud-init and Ansible, which give you most of the reproducibility. Add Terraform or OpenTofu when the infrastructure around the server, such as firewalls, volumes, load balancers and DNS records, starts to be the part you keep getting wrong.</p>



<h3 class="wp-block-heading">Should I build custom images instead of provisioning at boot?</h3>



<p class="wp-block-paragraph">Golden images cut boot time and remove dependency on package mirrors being up during a build. They also add a pipeline you have to maintain and rebuild for every security update. For client work at small scale, provisioning from a stock distribution image is usually the better trade, because the build stays readable and there is no stale image to forget about. Images earn their cost when you are launching machines often enough that boot time matters.</p>



<h3 class="wp-block-heading">Terraform or OpenTofu for client projects?</h3>



<p class="wp-block-paragraph">Both work, and the configuration language is shared, so this is not a lock-in decision in the way it looks. Terraform has the deeper ecosystem and the commercial support path. OpenTofu is under the Linux Foundation with a licence that is unlikely to change, which is the stronger argument when the code will live in a client repository long after you have moved on. Ask the client whether they have a standard before you pick.</p>



<h3 class="wp-block-heading">How do I automate server provisioning without exposing SSH to the internet?</h3>



<p class="wp-block-paragraph">The first run generally needs a reachable address, so scope the firewall rule to your own source IP for the duration of provisioning and close it in the same run. After that, reach the host through a bastion or an overlay network installed during the base role. Provider firewalls and host firewalls should both be in the automation, because relying on only one of them means a provider console change can silently open the box.</p>



<h3 class="wp-block-heading">How do I stop configuration drift once the server is live?</h3>



<p class="wp-block-paragraph">Run the playbook in check mode on a schedule and look at what it reports. Any change on a host you have not touched is drift, and it is far cheaper to find it that way than during an incident. This only works if your tasks are properly idempotent, which is why the always-reports-changed problem above matters more than it looks.</p>



<h3 class="wp-block-heading">What belongs in cloud-init versus Ansible?</h3>



<p class="wp-block-paragraph">cloud-init handles what must exist before you can connect: hostname, the login account, its SSH key, disabled password authentication, and a Python interpreter. Ansible handles everything you will ever want to change again. The dividing line is repeatability. cloud-init runs once and you cannot re-run it safely on a live host, so anything you might revisit belongs on the Ansible side.</p>



<h3 class="wp-block-heading">How long should provisioning a new client server take?</h3>



<p class="wp-block-paragraph">Wall-clock time is the wrong measure. The useful measure is how much of it needs you. A build that takes twenty minutes but runs unattended from one command is in far better shape than a five-minute build with four manual steps, because the manual steps are where the divergence between clients gets introduced.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">When you automate server provisioning for new clients, the deliverable is not the script. It is the ability to answer &#8220;can you rebuild this?&#8221; with yes, on any day, without opening a shell on the existing machine.</p>



<p class="wp-block-paragraph">Everything else follows from that. Reproducibility gives you consistency across the fleet, because there is one definition. It gives you a clean handover, because the repository is the environment. It gives you the ability to take the next client on without dreading the build. And it costs the same afternoon that a single unplanned rebuild would have cost you anyway, just spent on your own schedule instead of at 2am.</p>



<p class="wp-block-paragraph">Start with one client. Get their staging box rebuilding from scratch. Then make the second client&#8217;s build reuse the roles from the first. That is the whole path.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a provisioning setup you can actually hand over?</h2>



<p class="wp-block-paragraph">Most of the provisioning work I take on is not greenfield. It is a set of hand-built servers that work fine and nobody can reproduce. Things I can help with:</p>



<ul class="wp-block-list">
<li>Turning existing hand-built client servers into an Ansible role set, without a rebuild and without downtime</li>

<li>Building a client onboarding repository: Terraform or OpenTofu, cloud-init, roles, per-client inventory and secrets</li>

<li>Splitting shared credentials into per-client keys, tokens and state backends, with a revocation path that works</li>

<li>Writing the verification playbook that decides whether a build is finished, and wiring it into CI</li>

<li>Fixing idempotency so check-mode runs become a usable drift report instead of noise</li>

<li>Preparing an environment for handover: runbook, access review, and a documented rebuild that the client can run themselves</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion on something specific, send the cloud-config, the playbook output, or a check-mode diff you cannot explain, and I will tell you what I see.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/devops/automate-server-provisioning-new-clients/">Automating Server Provisioning for New Clients Without Building a Snowflake Farm</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
