<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Object Storage | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/object-storage/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/object-storage/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Mon, 28 Sep 2026 06:04:50 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.4</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Object Storage | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/object-storage/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>Property SaaS Document Storage: The Five Failures That Show Up in an Audit</title>
		<link>https://john-nessime.com/blog/cloud-security/property-saas-document-storage/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Mon, 28 Sep 2026 06:04:50 +0000</pubDate>
				<category><![CDATA[Cloud Security]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Real Estate Technology]]></category>
		<category><![CDATA[SaaS Engineering]]></category>
		<category><![CDATA[ABAC]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Audit Logging]]></category>
		<category><![CDATA[AWS KMS]]></category>
		<category><![CDATA[AWS STS]]></category>
		<category><![CDATA[Bucket Policy]]></category>
		<category><![CDATA[Cross-Tenant Leakage]]></category>
		<category><![CDATA[Crypto Shredding]]></category>
		<category><![CDATA[Data Minimization]]></category>
		<category><![CDATA[Document Retention]]></category>
		<category><![CDATA[Encryption Context]]></category>
		<category><![CDATA[File Upload Security]]></category>
		<category><![CDATA[GDPR]]></category>
		<category><![CDATA[Least Privilege]]></category>
		<category><![CDATA[Multi-Tenant]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[Presigned POST]]></category>
		<category><![CDATA[Presigned URLs]]></category>
		<category><![CDATA[Property Management Software]]></category>
		<category><![CDATA[Real Estate Data]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[S3 Object Lock]]></category>
		<category><![CDATA[SaaS Architecture]]></category>
		<category><![CDATA[Secure Disposal]]></category>
		<category><![CDATA[Stored XSS]]></category>
		<category><![CDATA[Tenant Isolation]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=898</guid>

					<description><![CDATA[<p>Storing tenant IDs, leases and referencing packs is easy until someone asks who actually read them. Five failure families in property SaaS document storage, with the IAM policies, presigned URL mechanics, upload controls and deletion patterns that close each one.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket that starts this is almost never &#8220;our document storage is insecure.&#8221; It&#8217;s a security questionnaire from a prospect, or a data subject access request, or an insurer asking one specific question: list everyone who has viewed this tenant&#8217;s ID document in the last twelve months.</p>



<p class="wp-block-paragraph">So you open the audit table. And you find it records who <em>requested</em> a link to the document. Not who opened it. The actual downloads never touched your application at all, because you handed out a signed storage URL and the browser went straight to the bucket. Your log is a record of intentions.</p>



<p class="wp-block-paragraph">That gap is typical of how property SaaS document storage goes wrong. The failures are rarely dramatic. Nobody dumps a bucket publicly. The guarantees you thought you had just turn out to be enforced in application code that ran once, long before the file was actually read. What follows is five failure families worth designing against when a platform stores tenant ID scans, signed leases, deposit certificates, right-to-rent evidence and referencing packs.</p>



<h2 class="wp-block-heading">What&#8217;s actually in the pile</h2>



<p class="wp-block-paragraph">Before architecture, be honest about the contents. A property platform&#8217;s document store is at least four risk classes wearing the same file extension:</p>



<ul class="wp-block-list">
<li><strong>Identity evidence.</strong> Passport and driving licence scans, visa documents, sometimes a selfie held next to the document. Highest blast radius, lowest genuine need for long retention.</li>



<li><strong>Contracts.</strong> Tenancy agreements, addenda, guarantor deeds, signed inventories. Legally driven retention, plus an integrity requirement: proving the file hasn&#8217;t changed matters more than hiding it.</li>



<li><strong>Financial and referencing material.</strong> Bank statements, payslips, credit reference reports. Sensitive, third-party sourced, and often covered by contractual restrictions from the referencing provider.</li>



<li><strong>Operational clutter.</strong> Inspection photos, safety certificates, maintenance invoices. Low sensitivity, high volume, and the thing that dominates your storage bill.</li>
</ul>



<p class="wp-block-paragraph">These want different retention clocks, different access rules and arguably different buckets. Treating them as one &#8220;documents&#8221; table is where most of the later pain originates.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure one: the access check and the access happen at different times</h2>



<p class="wp-block-paragraph">Presigned URLs are the right pattern. They keep the bucket private, take the transfer off your servers, and mean you never hand storage credentials to a browser. I reach for them first.</p>



<p class="wp-block-paragraph">The problem is what a presigned URL <em>is</em>. It&#8217;s a request you already authorised, signed, and then gave away. Anyone holding it can replay it until it expires. Your permission logic ran once and never runs again. Which means all of these keep working after the fact:</p>



<ul class="wp-block-list">
<li>A letting agent who left the company yesterday and whose account you disabled this morning.</li>



<li>A link forwarded into a WhatsApp group, pasted into a support ticket, or sitting in browser history on a shared office machine.</li>



<li>A document the tenant asked you to delete, if the object is still there behind a delete marker.</li>
</ul>



<h3 class="wp-block-heading">Make the URL die when the session does</h3>



<p class="wp-block-paragraph">Amazon&#8217;s own guidance is clearer on this than most people realise. A presigned URL created with temporary credentials expires when those credentials expire, <em>even if you asked for a longer expiry</em>. It also stops working if the signing credential is revoked, deleted or deactivated.</p>



<p class="wp-block-paragraph">That&#8217;s a lever, not a footnote. Sign with a long-lived IAM user key and the seven day maximum the SDK allows is a real seven days. Sign inside a short-lived assumed-role session and the URL cannot outlive that session, whatever number you passed in. So: never sign document URLs with static credentials. Assume a role per request, tag the session with the tenant, sign inside it.</p>



<pre class="wp-block-code"><code>import boto3

sts = boto3.client("sts")

# The tag key here must match the principal tag key used in the role's policy.
session = sts.assume_role(
    RoleArn="arn:aws:iam::111122223333:role/TenantDocumentAccess",
    RoleSessionName="doc-download",
    Tags=[{"Key": "TenantID", "Value": tenant_id}],
    DurationSeconds=900,
)["Credentials"]

s3 = boto3.client(
    "s3",
    aws_access_key_id=session["AccessKeyId"],
    aws_secret_access_key=session["SecretAccessKey"],
    aws_session_token=session["SessionToken"],
)

url = s3.generate_presigned_url(
    "get_object",
    Params={
        "Bucket": "prop-docs",
        "Key": f"tenants/{tenant_id}/identity/{doc_id}.pdf",
        "ResponseContentDisposition": 'attachment; filename="id-document.pdf"',
        "ResponseContentType": "application/octet-stream",
    },
    ExpiresIn=120,
)</code></pre>



<p class="wp-block-paragraph">Two things are doing work. <code>DurationSeconds</code> caps the real lifetime of the link regardless of <code>ExpiresIn</code>. And <code>ResponseContentDisposition</code> overrides how the object is served on this request, which matters for the serving problem further down.</p>



<p class="wp-block-paragraph">The trade-off is real: an STS call per download adds latency, adds a dependency, and STS has request limits you can hit at volume. AWS&#8217;s prescriptive guidance says as much. For identity documents I&#8217;d pay that cost. For inspection photos I wouldn&#8217;t.</p>



<h3 class="wp-block-heading">Log where the read actually happens</h3>



<p class="wp-block-paragraph">Your application log records issuance. To record retrieval you need logging at the storage layer: CloudTrail data events for object-level operations, or S3 server access logging. Data events are billed per event, so scope them to the identity and contract prefixes rather than the bucket holding thousands of inspection photos.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure two: isolation that lives in your ORM instead of your IAM policy</h2>



<p class="wp-block-paragraph">Ask most teams how tenant isolation works and you get a description of a WHERE clause. The key is built as <code>tenants/{tenant_id}/...</code> and the tenant ID comes from the session. Fine until one code path takes it from a request parameter, or a background job iterates without scoping, or someone writes an admin endpoint in a hurry.</p>



<p class="wp-block-paragraph">Move the boundary below your code. With attribute-based access control the tenant becomes a session tag and the IAM policy interpolates it into the resource ARN. One role serves every tenant, and the policy count doesn&#8217;t grow as you onboard.</p>



<pre class="wp-block-code"><code>{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteOwnTenantPrefix",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::prop-docs/tenants/${aws:PrincipalTag/TenantID}/*"
    },
    {
      "Sid": "ListOnlyOwnTenantPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::prop-docs",
      "Condition": {
        "StringLike": {
          "s3:prefix": "tenants/${aws:PrincipalTag/TenantID}/*"
        }
      }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Two statements because S3 splits the problem. The first governs the object, where the tenant prefix is part of the resource ARN. The second governs listing, a bucket-level action, so the prefix has to be constrained through a condition instead. Skip it and a tenant can enumerate every key in the bucket even though they can&#8217;t read any of them, and key names in this domain leak plenty on their own.</p>



<p class="wp-block-paragraph">The failure mode worth internalising: if a code path forgets the session tag, <code>${aws:PrincipalTag/TenantID}</code> resolves to nothing and the statement doesn&#8217;t match. That&#8217;s a deny, which is what you want. But it only holds if the role has no broader fallback statement. One well-meaning <code>s3:GetObject</code> on the whole bucket, added to unblock a migration script, quietly turns the scheme into decoration.</p>



<h3 class="wp-block-heading">When to use a bucket per tenant instead</h3>



<p class="wp-block-paragraph">A bucket or an account per tenant is a genuinely harder boundary and I wouldn&#8217;t dismiss it. With a few dozen large agency customers it&#8217;s defensible and makes offboarding trivially provable. It stops being defensible once you onboard self-serve landlords, because bucket and account counts run into service quotas and the operational surface grows with every signup. Pick on tenant count, not on which sounds more secure. And note that ABAC is only as trustworthy as the tenant claim: if the caller can influence the value reaching the tagging call, you&#8217;ve built an elegant no-op.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure three: the upload and serve path</h2>



<p class="wp-block-paragraph">Uploads here come from people you have no relationship with yet. Prospective tenants, guarantors, contractors. That&#8217;s an untrusted channel by definition, and it produces two separate problems.</p>



<h3 class="wp-block-heading">Constrain the upload itself</h3>



<p class="wp-block-paragraph">A presigned PUT URL lets the holder write whatever they like to that key, at whatever size. Presigned POST is the better tool for public-facing uploads: the signature covers a policy document, and the policy carries conditions the browser cannot alter.</p>



<pre class="wp-block-code"><code>post = s3.generate_presigned_post(
    Bucket="prop-docs",
    Key=f"tenants/{tenant_id}/inbox/{upload_id}",
    Fields={"Content-Type": "application/pdf"},
    Conditions=[
        {"Content-Type": "application/pdf"},
        ["content-length-range", 1, 15728640],
    ],
    ExpiresIn=600,
)</code></pre>



<p class="wp-block-paragraph">The <code>content-length-range</code> condition is the one people skip, and it&#8217;s the one that stops a stranger using your referencing portal as free storage for a 40 GB file. A declared <code>Content-Type</code> is a claim, not a fact, so land uploads in a quarantine prefix, verify the actual bytes, and only then move them to the tenant prefix.</p>



<p class="wp-block-paragraph">Scan on that path too. ClamAV in a container is the zero-cost option and covers known malware; GuardDuty Malware Protection for S3 and hosted APIs such as Cloudmersive trade money for not maintaining it. What you&#8217;re protecting is mostly the letting agent who opens the attachment on a Windows machine, so this is a control for your customers rather than your servers.</p>



<h3 class="wp-block-heading">Never serve tenant documents from your application origin</h3>



<p class="wp-block-paragraph">This is the one that turns a document store into an account takeover. SVG and HTML are formats browsers execute. If a file uploaded by a prospective tenant is served inline from the same origin as your app, any script inside it runs with your application&#8217;s cookies and session context. A long stream of real advisories follows exactly this shape, particularly in self-hosted deployments where the object store sits on the app&#8217;s own domain.</p>



<p class="wp-block-paragraph">Three controls, and they stack rather than substitute:</p>



<ol class="wp-block-list">
<li><strong>Force download.</strong> Serve with <code>Content-Disposition: attachment</code> and <code>X-Content-Type-Options: nosniff</code>. Set these as object metadata at upload time <em>and</em> override them per request. The override protects links you mint; the stored metadata protects anything that bypasses your minting path.</li>



<li><strong>Use a separate origin.</strong> Put user content on a distinct domain, not a subdomain of the app. GitHub&#8217;s separate apex for user content is the reference implementation. Same-origin policy then contains the blast radius if control one is missed somewhere.</li>



<li><strong>Normalise where you can.</strong> Rendering an uploaded ID to a flattened image server side removes the executable content entirely. It costs CPU and fidelity. For identity documents, where you rarely need the original bytes back, it&#8217;s the right trade.</li>
</ol>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure four: deleting the document doesn&#8217;t delete the document</h2>



<p class="wp-block-paragraph">A tenant asks you to erase their passport scan. Your app sets a flag, the file disappears from the UI, and you reply confirming deletion. Then count the copies that still exist.</p>



<ul class="wp-block-list">
<li>The previous object versions, if versioning is on. A delete without a version ID just writes a delete marker.</li>



<li>Cross-region replicas with their own versioning and lifecycle rules, plus any backup or snapshot taken before the deletion.</li>



<li>Derivatives you generated and forgot: thumbnails, page previews, OCR text in a search index. Run a passport through Amazon Textract and the extracted fields are a copy of the sensitive data in a different shape.</li>



<li>Copies that left the platform. A document attached to a CRM record in something like GoHighLevel, forwarded by email, or downloaded to a letting agent&#8217;s laptop.</li>
</ul>



<p class="wp-block-paragraph">The ICO&#8217;s position is that a valid erasure request extends to backups as well as live systems, and that where backup data cannot be immediately overwritten it must be put beyond use: isolated, unused for anything else, deleted when that backup cycles. Workable, but only if you know which backup sets contain documents and how long they cycle. If you can&#8217;t answer that, you can&#8217;t answer the request.</p>



<h3 class="wp-block-heading">Crypto-shredding, and what it actually costs</h3>



<p class="wp-block-paragraph">The one technique that makes deletion provable across copies you can&#8217;t reach is a per-tenant KMS key. Encrypt that tenant&#8217;s objects under their own customer managed key; when they offboard, schedule the key for deletion. Every ciphertext copy anywhere, backups included, becomes undecryptable. You delete the data by deleting the ability to read it. The honest costs:</p>



<ul class="wp-block-list">
<li>KMS bills per key per month plus per request, so per-tenant keys are a real line item. Check current rates and quotas first: there&#8217;s an account limit on customer managed keys that a self-serve signup flow will eventually meet.</li>



<li>Key deletion has a mandatory waiting period. Crypto-shredding is not instant, and your erasure response has to say so.</li>



<li>It&#8217;s all-or-nothing per key. If a tenant asks you to erase one document rather than offboard, the key doesn&#8217;t help.</li>
</ul>



<h3 class="wp-block-heading">Object Lock fights erasure, on purpose</h3>



<p class="wp-block-paragraph">S3 Object Lock is tempting for signed tenancy agreements, because write-once storage is a strong integrity story. But compliance mode means nobody, root included, can delete the object until retention lapses. That is the feature. It is also a direct collision with an erasure request, and you cannot engineer your way out of it afterwards.</p>



<p class="wp-block-paragraph">My line: Object Lock on executed contracts, where a statutory retention basis overrides erasure, and never on identity documents or referencing material. Different buckets, different rules, decided before anything is written. For per-document erasure elsewhere, lifecycle rules that expire noncurrent versions on the sensitive prefixes are worth setting up before you need them.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Failure five: storing what you never needed to store</h2>



<p class="wp-block-paragraph">The cheapest control on this list is not storing the file. The ICO is clear that UK GDPR sets no fixed retention periods and puts the burden on you to justify how long you keep personal data in identifiable form. That&#8217;s a design instruction, not a legal footnote.</p>



<p class="wp-block-paragraph">For identity documents, ask what your platform needs once the check is done. Usually it&#8217;s the result, not the evidence: document type, issuing country, expiry date, a check reference, who performed it and when. If your customer&#8217;s regulator requires the image itself, that&#8217;s a retention rule with a stated basis, which is a different conversation from keeping it because deleting felt risky.</p>



<p class="wp-block-paragraph">Practical version: give every document class a retention clock wired to a business event rather than upload date, and let lifecycle rules do the deleting. Tenancy end plus a defined period. Check completion plus a defined period. Not &#8220;whenever someone remembers.&#8221;</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Troubleshooting the breakages you&#8217;ll actually hit</h2>



<h3 class="wp-block-heading">Everything broke after someone enabled S3 Bucket Keys</h3>



<p class="wp-block-paragraph">This one catches teams who did the sophisticated thing. If your IAM or KMS key policies use the object ARN as the encryption context to scope access, enabling an S3 Bucket Key changes that context to the <em>bucket</em> ARN and those policies stop matching. AWS documents it explicitly as something to fix before flipping the setting.</p>



<p class="wp-block-paragraph">The quieter side effect: KMS CloudTrail events then log the bucket ARN instead of the object ARN, and there are far fewer of them because S3 caches the bucket-level key. If you were treating KMS CloudTrail volume as a proxy for document access, that signal degrades without anything appearing to break. Bucket Keys are still worth enabling for the request cost reduction. Move the conditions first, then re-check what your dashboards count.</p>



<h3 class="wp-block-heading">The presigned URL expires far sooner than requested</h3>



<p class="wp-block-paragraph">Almost always the signing credential is a role session shorter than the expiry you asked for. This is documented behaviour, not a bug. Either raise the session duration within the role&#8217;s maximum, or accept the shorter window as the security property it is.</p>



<h3 class="wp-block-heading">Listing works, reading returns Access Denied</h3>



<p class="wp-block-paragraph">Classic split between the bucket-level and object-level statements. Check both exist and reference the same tag key spelling. Session tag keys are case sensitive, and a <code>tenantId</code> in the code against a <code>TenantID</code> in the policy fails silently as a non-match.</p>



<h3 class="wp-block-heading">Access denied only for encrypted objects</h3>



<p class="wp-block-paragraph">An SSE-KMS object needs two grants, not one. The S3 action on the object, and <code>kms:Decrypt</code> on the key it was encrypted under. In a per-tenant key setup that second grant is usually also tag-conditioned, so a missing or mismatched tag on the key itself produces a denial that looks like an S3 problem and isn&#8217;t.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Signing document URLs with a long-lived IAM user access key, which makes every link independently long-lived.</li>



<li>Granting the tenant role a broad bucket-wide read statement alongside the tag-scoped one, usually to unblock a one-off script that then stayed.</li>



<li>Serving user uploads from a subdomain of the application and assuming that counts as origin separation. It doesn&#8217;t for cookie scope.</li>



<li>Enabling versioning for safety and never adding a noncurrent version expiry rule, so every deletion is reversible forever.</li>



<li>Building deletion that handles the object but not the OCR output, the thumbnail, or the search index entry.</li>



<li>Putting Object Lock on the bucket that also holds identity documents.</li>
</ul>



<h2 class="wp-block-heading">Design rules worth holding to</h2>



<ol class="wp-block-list">
<li><strong>Separate buckets by risk class,</strong> not by feature. Identity, contracts, financial, operational. Different encryption, retention, logging and lock settings follow naturally from that split.</li>



<li><strong>Enforce isolation in IAM,</strong> with application checks as defence in depth rather than the boundary itself.</li>



<li><strong>Sign short, sign from a session.</strong> Minutes, not days, and always from assumed-role credentials so revocation actually revokes.</li>



<li><strong>Deny unencrypted writes at the bucket policy,</strong> using the server-side encryption request headers as conditions, so a misconfigured client fails loudly instead of writing plaintext.</li>



<li><strong>Give every document class an expiry clock</strong> tied to a business event, implemented in lifecycle rules rather than a cron job someone maintains.</li>



<li><strong>Write the deletion runbook before launch,</strong> listing every place a copy lands. If you can&#8217;t enumerate them, that&#8217;s the finding.</li>



<li><strong>Test cross-tenant access in CI.</strong> A negative test that asserts tenant A&#8217;s session cannot read tenant B&#8217;s key is worth more than any amount of policy review.</li>
</ol>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How long should a presigned URL for a tenant ID document last?</h3>



<p class="wp-block-paragraph">Short enough that the user clicks it immediately. A couple of minutes covers a click-to-download flow. If the UI needs longer, re-mint on demand rather than extend. And remember the ceiling is set by the signing credential, so a short assumed-role session enforces the limit even if a later code change passes a bigger number.</p>



<h3 class="wp-block-heading">Do I need AWS, or will Backblaze B2 or Cloudflare R2 do?</h3>



<p class="wp-block-paragraph">They all speak the S3 API and presigned URLs work on all of them, so the core patterns transfer. Wasabi, Backblaze B2, Cloudflare R2 and DigitalOcean Spaces are reasonable for the operational tier, and R2&#8217;s egress model is attractive if agents download a lot. They differ in the depth of the IAM story: session-tag ABAC and per-tenant KMS keys are AWS-specific. If isolation and crypto-shredding are hard requirements, keep the sensitive prefixes on S3 even when the bulk photos live somewhere cheaper.</p>



<h3 class="wp-block-heading">Can I store scanned passports on a normal VPS instead?</h3>



<p class="wp-block-paragraph">You can, and plenty of platforms run the app tier on a VPS from Contabo or InterServer while keeping documents in object storage. What you lose on local disk is the deletion story and the access log. Proving an image is gone from a filesystem, its backups and its snapshots is far harder than scheduling a KMS key for deletion. If documents stay local, budget real effort for secure disposal, including on staff machines where tools like O&amp;O SafeErase have a place.</p>



<h3 class="wp-block-heading">What about e-signed tenancy agreements?</h3>



<p class="wp-block-paragraph">With a provider like DocuSign, the executed document and its audit trail live there as well as in your platform. That&#8217;s a second copy under a second retention regime, and it belongs in your deletion runbook and data map. It also strengthens your integrity story, which is a fair reason to skip Object Lock on contracts you retrieve from the provider.</p>



<h3 class="wp-block-heading">Should the CDN cache tenant documents?</h3>



<p class="wp-block-paragraph">Not the sensitive classes. A CDN in front of private documents adds a cache that can serve content after your access decision changed, and header handling at the edge is a common source of the inline-rendering bug. CloudFront with origin access control and signed URLs is legitimate if you need the performance, but for identity documents I&#8217;d go straight to the bucket. For inspection photos, cache away.</p>



<h3 class="wp-block-heading">How do I prove any of this to a prospect&#8217;s security team?</h3>



<p class="wp-block-paragraph">Artefacts, not assertions. The bucket policy, the ABAC policy, CI output showing cross-tenant denial, a sample of storage-layer access logs, and a retention schedule with a clock per document class. That set answers most questionnaires without a call.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">Good property SaaS document storage is mostly about closing the gap between when you check access and when access happens. Every failure family above is a version of that gap: the presigned link that outlives the permission, the isolation enforced once in code and never at the storage layer, the deletion that updates a flag while the bytes sit behind a delete marker.</p>



<p class="wp-block-paragraph">Close it by pushing the decision down: into IAM policies that evaluate on every request, into credentials short enough that revocation is real, into lifecycle rules that delete without anyone remembering to. Then the answer to &#8220;show us who read this tenant&#8217;s ID document&#8221; is a query, not a project.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need a second pair of eyes on your document storage?</h2>



<p class="wp-block-paragraph">Most of what I do on property and lettings platforms is this exact work. Things I&#8217;d help with here:</p>



<ul class="wp-block-list">
<li>Reviewing bucket and IAM policies for cross-tenant read paths, including the ones added to unblock a migration and never removed.</li>



<li>Moving tenant isolation from application code into session-tag ABAC, with a CI test that fails the build on cross-tenant access.</li>



<li>Rebuilding the upload path with presigned POST conditions, quarantine, content verification and scanning before anything lands in a tenant prefix.</li>



<li>Fixing the serving path so uploaded files can&#8217;t execute against your origin, including header handling through a CDN.</li>



<li>Designing retention clocks and lifecycle rules per document class, plus a deletion runbook that names every copy including derivatives and third-party systems.</li>



<li>Setting up storage-layer access logging scoped to the sensitive prefixes so you can actually answer an access request.</li>
</ul>



<p class="wp-block-paragraph">If it&#8217;s useful, send me your bucket policy and the IAM policy attached to your document role, with the account IDs redacted, and I&#8217;ll tell you what a determined tenant could reach.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-security/property-saas-document-storage/">Property SaaS Document Storage: The Five Failures That Show Up in an Audit</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing</title>
		<link>https://john-nessime.com/blog/technical-guides/iceberg-table-maintenance-storage-costs/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 20 Sep 2026 06:00:00 +0000</pubDate>
				<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[FinOps]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Apache Iceberg]]></category>
		<category><![CDATA[Apache Spark]]></category>
		<category><![CDATA[Cost Optimization]]></category>
		<category><![CDATA[Data Lake]]></category>
		<category><![CDATA[Lakehouse]]></category>
		<category><![CDATA[Object Storage]]></category>
		<category><![CDATA[Orphan Files]]></category>
		<category><![CDATA[S3 Lifecycle Rules]]></category>
		<category><![CDATA[Snapshot Expiration]]></category>
		<category><![CDATA[Snapshots]]></category>
		<category><![CDATA[Storage]]></category>
		<category><![CDATA[Table Compaction]]></category>
		<category><![CDATA[Table Maintenance]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=855</guid>

					<description><![CDATA[<p>Your lake bucket keeps growing even though nobody added data. Here is where the bytes actually sit: snapshots you never expired, orphan files no metadata points at, metadata JSON accumulation, and the object store's own hidden inventory. Plus the safe order to clean all four up.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/iceberg-table-maintenance-storage-costs/">Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The question usually comes from finance, not engineering. Someone points at the bucket holding the analytics lake and asks why it is twice the size it was two quarters ago. Nobody added a pipeline. The tables are the same tables. One of them had a large chunk of history deleted on purpose, and the bucket still grew.</p>



<p class="wp-block-paragraph">That gap between what a table logically contains and what the object store physically holds is what Iceberg table maintenance exists to close. Iceberg never overwrites a file in place. Deletes and updates write new files and a new pointer, and the old files sit where they are until something explicitly removes them. If nothing does, the bill only moves one way.</p>



<p class="wp-block-paragraph">This post covers where those bytes hide, why expiring snapshots often frees nothing, how orphan files accumulate without showing up in any metadata you would think to check, and the order to run cleanup so you do not corrupt a table while shrinking it.</p>



<h2 class="wp-block-heading">Why deleting rows does not shrink anything</h2>



<p class="wp-block-paragraph">Every write produces a snapshot: an immutable view of the table at a point in time, pointing at a manifest list, which points at manifests, which point at data files. Nothing in that chain is mutated. A new commit builds a new chain that reuses most of the old files.</p>



<p class="wp-block-paragraph">That design is what gives you time travel, rollback and safe concurrent writes. It is also why storage grows silently. A data file only becomes a deletion candidate once <em>no</em> retained snapshot references it. Delete a year of rows and you free nothing, because the snapshot from before the delete still points at every one of those files.</p>



<p class="wp-block-paragraph">So maintenance is not an optimization. It is the mechanism by which deletes become real.</p>



<h2 class="wp-block-heading">The four places your storage is actually sitting</h2>



<p class="wp-block-paragraph">When a lakehouse bucket balloons, the bytes are in one of four categories. Each needs a different tool, and confusing them is how teams end up running the same job for weeks with no effect.</p>



<h3 class="wp-block-heading">1. Data files held alive by snapshots you never expired</h3>



<p class="wp-block-paragraph">This is the one everyone knows about and usually the largest share. Snapshots accumulate until something expires them, and nothing expires them by default.</p>



<p class="wp-block-paragraph">Before changing anything, look at the table&#8217;s own metadata. Iceberg exposes it as queryable tables:</p>



<pre class="wp-block-code"><code>-- How many snapshots exist, and how far back do they go?
SELECT committed_at, snapshot_id, operation
FROM db.orders.snapshots
ORDER BY committed_at;

-- How much data does the CURRENT snapshot reference?
SELECT count(*) AS file_count,
       sum(file_size_in_bytes) AS bytes
FROM db.orders.files;</code></pre>



<p class="wp-block-paragraph">Compare that second number against what your object store reports for the table prefix. The difference is your maintenance debt. On a neglected table the referenced set is often a minority of the physical bytes.</p>



<p class="wp-block-paragraph">Expiring snapshots is a Spark stored procedure:</p>



<pre class="wp-block-code"><code>CALL my_catalog.system.expire_snapshots(
  table =&gt; 'db.orders',
  older_than =&gt; TIMESTAMP '2000-01-01 00:00:00',
  retain_last =&gt; 10,
  stream_results =&gt; true
);</code></pre>



<p class="wp-block-paragraph">Three arguments matter. <code>older_than</code> sets the age cutoff. <code>retain_last</code> is the floor: it keeps that many ancestor snapshots regardless of age, which is your rollback safety net. <code>stream_results</code> streams the delete list back to the driver instead of collecting it at once, which on a table with long history is the difference between the job finishing and the driver running out of memory.</p>



<p class="wp-block-paragraph">The trap is that Iceberg treats these conditions as guidance rather than a strict contract, leaning toward retention when they conflict. An aggressive <code>older_than</code> plus a generous <code>retain_last</code> gives you the generous answer. Safe, but it surprises people who expected the timestamp to win.</p>



<p class="wp-block-paragraph">Branches and tags are separate: they are not subject to snapshot age at all. A tag created for a regulatory export a year ago pins its entire snapshot chain, and no amount of expiring will touch it. Reference age is governed by <code>history.expire.max-ref-age-ms</code>.</p>



<h3 class="wp-block-heading">2. Orphan files that no metadata points at</h3>



<p class="wp-block-paragraph">This is the invisible one, and the reason many teams conclude that maintenance &#8220;does not work.&#8221;</p>



<p class="wp-block-paragraph">When a Spark task writes a data file and the job then fails before the commit lands, that file is already on disk. It is in no manifest, no snapshot, no metadata file. Expiry will never find it, because expiry compares the file sets of expired snapshots against retained ones, and a file that was never in a snapshot is in neither set.</p>



<p class="wp-block-paragraph">Streaming writers, speculative execution, retried stages and killed compaction jobs all produce these. The only way to find them is to list the storage prefix and diff it against the metadata:</p>



<pre class="wp-block-code"><code>-- Always dry run first. This lists candidates without deleting.
CALL my_catalog.system.remove_orphan_files(
  table =&gt; 'db.orders',
  dry_run =&gt; true
);</code></pre>



<p class="wp-block-paragraph">Read that output before you run it live. Recent timestamps or unfamiliar paths mean stop and investigate.</p>



<p class="wp-block-paragraph">The safety mechanism is <code>older_than</code>, which defaults to three days. That default is not arbitrary caution. Orphan removal cannot distinguish a file abandoned by a dead job from one being written right now by a live one. Set the interval shorter than your longest-running write and the procedure can delete a file a job is about to commit, corrupting the table. Shorten it only when you know your write durations, and never below your longest backfill.</p>



<p class="wp-block-paragraph">The second hazard is less obvious. Iceberg compares file paths as strings. If your metadata records <code>s3a://bucket/...</code> and your listing returns <code>s3://bucket/...</code>, or you migrated between HDFS namenodes and the authority changed, every live file looks like an orphan. Running that deletes the table. The procedure defends against it by erroring on a prefix mismatch by default, via <code>prefix_mismatch_mode</code> with <code>ERROR</code>, <code>IGNORE</code> and <code>DELETE</code> as options. Resolve a mismatch with <code>equal_schemes</code> or <code>equal_authorities</code>. Do not reach for <code>DELETE</code> to silence it.</p>



<h3 class="wp-block-heading">3. Metadata JSON files and manifests</h3>



<p class="wp-block-paragraph">Every commit writes a new table metadata JSON file. On a batch table that is a handful a day. On a streaming table committing every minute it is a lot of small objects, and those cost you in request charges and planning time as much as in bytes.</p>



<pre class="wp-block-code"><code>ALTER TABLE db.orders SET TBLPROPERTIES (
  'write.metadata.delete-after-commit.enabled' = 'true',
  'write.metadata.previous-versions-max' = '20'
);</code></pre>



<p class="wp-block-paragraph">The first property is off by default. The second defaults to 100 and controls how many previous metadata files are <em>tracked</em> in the metadata log. The catch is that cleanup only deletes tracked files. If you have been running with deletion disabled, files that fell out of the log are already untracked, and turning the property on later will not reach back for them. They are orphans now, and only orphan file removal clears them.</p>



<p class="wp-block-paragraph">You can see the tracked set with <code>SELECT timestamp, file FROM db.orders.metadata_log_entries</code>.</p>



<p class="wp-block-paragraph">Manifests rarely dominate the bill, but a table with thousands of tiny ones plans queries slowly. <code>rewrite_manifests</code> consolidates them. Treat it as a query performance job and run it after the deletion work, so it is not reorganizing metadata about to be thrown away.</p>



<h3 class="wp-block-heading">4. The object store&#8217;s own hidden inventory</h3>



<p class="wp-block-paragraph">Most Iceberg guides skip this category, and it often explains a bill nobody can account for. Iceberg has no visibility into any of it.</p>



<ul class="wp-block-list">
<li><strong>Incomplete multipart uploads.</strong> Large Parquet files upload in parts. When a writer dies partway, the completed parts sit in the bucket, billed as storage, and they do not appear in a normal object listing. Nothing removes them automatically.</li>



<li><strong>Noncurrent object versions.</strong> With versioning enabled, every file your maintenance job &#8220;deletes&#8221; becomes a noncurrent version you keep paying for, indefinitely, unless a rule expires it.</li>



<li><strong>Delete markers.</strong> Deletes on a versioned bucket leave markers behind. Small individually, but on a table with millions of expired files they add up and slow listings down.</li>
</ul>



<p class="wp-block-paragraph">A lifecycle configuration covers all three. On S3 the shape looks like this:</p>



<pre class="wp-block-code"><code>{
  "Rules": [
    {
      "ID": "abort-stale-multipart",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 7 }
    },
    {
      "ID": "expire-noncurrent-versions",
      "Status": "Enabled",
      "Filter": { "Prefix": "warehouse/" },
      "NoncurrentVersionExpiration": { "NoncurrentDays": 14 },
      "Expiration": { "ExpiredObjectDeleteMarker": true }
    }
  ]
}</code></pre>



<p class="wp-block-paragraph">Now the warning that matters most here. <strong>Never put an age-based expiration or storage class transition rule on the prefix holding live Iceberg data files.</strong> A rule deleting objects older than ninety days will happily delete data files a current snapshot still references, and the table breaks with missing-file errors on the next read. Transitioning them to an archive class is just as bad, because reads then fail or stall on retrieval.</p>



<p class="wp-block-paragraph">The rules above act only on things the table does not reference: abandoned upload parts, superseded versions, tombstones. Iceberg decides what live data to delete; lifecycle rules clean up after that decision.</p>



<p class="wp-block-paragraph">This applies equally on S3-compatible storage. If you run a lakehouse on Cloudflare R2, Backblaze B2, Wasabi or Contabo Object Storage to dodge egress charges, check which lifecycle actions the provider actually implements. Support for aborting incomplete multipart uploads varies, and a provider that silently ignores the rule leaves you paying for parts you cannot see.</p>



<h2 class="wp-block-heading">Running Iceberg table maintenance in a safe order</h2>



<p class="wp-block-paragraph">Order matters, and the reason is not stylistic. These procedures read each other&#8217;s inputs.</p>



<ol class="wp-block-list">
<li><strong>Compact first, if you are compacting.</strong> <code>rewrite_data_files</code> writes new files and leaves the old ones referenced by older snapshots. Doing it after expiry means the space you just freed is immediately replaced.</li>



<li><strong>Expire snapshots.</strong> This is what turns unreferenced data files into deleted ones. It must happen before orphan removal, because expiry needs to read the manifests belonging to the snapshots it is dropping.</li>



<li><strong>Remove orphan files.</strong> Only after expiry has committed. Running the two concurrently is a genuine race: orphan removal can delete files the in-flight expiry job still needs to read, and expiry fails.</li>



<li><strong>Rewrite manifests, if planning is slow.</strong> Optional, and worth measuring before and after rather than running on faith.</li>
</ol>



<p class="wp-block-paragraph">On cadence: expiry is cheap and should run often, daily or better on high-commit tables. Orphan removal is expensive because it lists the entire table prefix, and the official guidance is to run it periodically but not frequently. Weekly on a busy table, monthly on a quiet one. Where supported, the <code>prefix_listing</code> option makes that scan substantially cheaper on object stores, where recursive directory listing is the slow part.</p>



<h2 class="wp-block-heading">Letting table properties do the boring part</h2>



<p class="wp-block-paragraph">Procedures are what you run. Properties are what you set once so the procedures behave sensibly without a config file per table.</p>



<pre class="wp-block-code"><code>ALTER TABLE db.orders SET TBLPROPERTIES (
  'history.expire.max-snapshot-age-ms' = '604800000',   -- 7 days
  'history.expire.min-snapshots-to-keep' = '10'
);</code></pre>



<p class="wp-block-paragraph">The snapshot age default is five days and the minimum to keep defaults to one. Setting both means a generic maintenance job can call <code>expire_snapshots</code> with no tuning and each table gets its own policy. A regulated reporting table keeps ninety days; a staging table keeps an hour. That beats maintaining per-table retention in your scheduler.</p>



<p class="wp-block-paragraph">The other property worth knowing is <code>gc.enabled</code>. Setting it false blocks physical deletion entirely. It exists for tables whose files are shared with something else, and it is correct for those. It is also a common accidental cause of &#8220;maintenance runs successfully and frees nothing.&#8221; Check it before debugging anything else.</p>



<h2 class="wp-block-heading">Troubleshooting: the job succeeded and nothing shrank</h2>



<ul class="wp-block-list">
<li><strong>The counters were zero.</strong> <code>expire_snapshots</code> returns counts of deleted data files, delete files, manifests and manifest lists. All zero means nothing qualified. Check whether <code>retain_last</code> or the table&#8217;s minimum-to-keep property is holding everything.</li>



<li><strong>A branch or tag is pinning history.</strong> Refs do not age out with snapshots. Look at the table&#8217;s refs metadata table and see what is anchored.</li>



<li><strong>Garbage collection is off.</strong> Either <code>gc.enabled</code> is false, or the table was created by the <code>snapshot</code> procedure, which is barred from expiry precisely because it does not own its data files.</li>



<li><strong>Files were deleted but the bucket did not shrink.</strong> Versioning. The deletes created noncurrent versions. This is the most common cause of healthy counters and a flat bill.</li>



<li><strong>Orphan removal hit a file limit.</strong> Some managed implementations cap deletions per run. On a backlogged bucket you may need several passes before the number stops dropping.</li>



<li><strong>The driver died.</strong> On a table with very long history, add <code>stream_results</code> and raise <code>max_concurrent_deletes</code> so deletes spread across a thread pool instead of serializing.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Running orphan file removal with a short retention window on a table that has long-running writers. This is the one that can actually destroy data.</li>



<li>Setting a bucket-wide expiration rule to &#8220;clean up the lake&#8221; and taking live data files with it.</li>



<li>Enabling metadata deletion after the fact and expecting it to collect the already-untracked backlog.</li>



<li>Scheduling expiry and orphan removal as two independent cron jobs that can overlap.</li>



<li>Skipping the dry run because the last twenty were fine. The one that matters is the run after somebody changes the storage endpoint.</li>



<li>Setting retention from a storage target rather than a recovery target. Ask how far back you need to roll back, then keep that much.</li>
</ul>



<h2 class="wp-block-heading">Best practices worth adopting</h2>



<ul class="wp-block-list">
<li>Express retention as a table property, not a job argument. It travels with the table and survives whoever wrote the scheduler.</li>



<li>Track referenced bytes against physical bytes per table as a metric, not an occasional query. The ratio drifting upward is the early signal; the invoice is the late one. Any time series backend you already run will do.</li>



<li>Give maintenance its own compute and its own credentials. It is the only workload in the lake that needs delete permissions.</li>



<li>Log the returned counters from every run. When someone asks in six months where the storage went, that log is the answer.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">How often should I expire Iceberg snapshots?</h3>



<p class="wp-block-paragraph">As often as your commit rate justifies. A streaming table committing every minute benefits from daily or hourly expiry; a nightly batch table is fine with weekly. Set the window from how far back you would realistically need to roll back, then keep a few extra snapshots as a floor.</p>



<h3 class="wp-block-heading">Does expiring snapshots delete my data?</h3>



<p class="wp-block-paragraph">It deletes data files that no retained snapshot references. Current table contents are never affected. What you lose is the ability to time travel to or roll back to the expired snapshots, so agree the window with anyone who queries the table as of a date.</p>



<h3 class="wp-block-heading">Why do orphan files exist if Iceberg commits are atomic?</h3>



<p class="wp-block-paragraph">The commit is atomic; the writes preceding it are not part of that atomicity. Data files are written first, then the commit makes them visible. If the job dies between those steps, the files exist and the commit never happened. Atomicity guarantees readers never see a partial table, not that failed writes clean up after themselves.</p>



<h3 class="wp-block-heading">Can I use an S3 lifecycle rule instead of Iceberg maintenance?</h3>



<p class="wp-block-paragraph">No, and trying is how tables get broken. Lifecycle rules act on object age. Whether Iceberg still needs a file has nothing to do with age; a data file written two years ago can be referenced by the current snapshot. Lifecycle rules are right for upload parts, noncurrent versions and delete markers, and wrong for anything the table might still point at.</p>



<h3 class="wp-block-heading">Do I need Spark to run Iceberg table maintenance?</h3>



<p class="wp-block-paragraph">Spark has the most complete set of procedures, but it is not the only route. Flink has its own maintenance API, Trino exposes expiry and orphan removal as table procedures, and several managed catalogs run scheduled maintenance for you. The mechanics are identical regardless of engine, because the work is defined by the table format rather than the compute. Only the syntax and the exposed arguments change.</p>



<h3 class="wp-block-heading">Is orphan file removal safe to automate?</h3>



<p class="wp-block-paragraph">Yes, once you have verified that the retention window comfortably exceeds your longest write and that the paths in your metadata match those your storage listing returns. Automate it after watching a few dry runs, not before. Erroring on a prefix mismatch is a stop signal, not something to configure away.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The one thing to take away</h2>



<p class="wp-block-paragraph">Iceberg table maintenance is not housekeeping you get to postpone. It is the step where logical deletes become physical ones, and until it runs, storage only grows. The bill is the visible symptom; the same neglect also makes query planning slower and recovery windows fuzzier.</p>



<p class="wp-block-paragraph">Set retention as a table property so the policy lives with the data. Expire snapshots on a schedule matching your commit rate. Remove orphan files less often, always after expiry, always with a window longer than your slowest write. Then add the storage lifecycle rules that clean up what Iceberg cannot see. Any one alone leaves money on the table.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help getting your lakehouse storage under control?</h2>



<p class="wp-block-paragraph">Most of the work here is not writing the procedures. It is working out which of the four categories your bytes are in, and building a maintenance path that will not break a table at three in the morning. That is the kind of thing I do:</p>



<ul class="wp-block-list">
<li>Audit an existing lake and produce a per-table breakdown of referenced bytes against physical bytes, so you know where the money is before touching anything</li>



<li>Build a scheduled maintenance pipeline with correct sequencing, dry-run gates and per-table retention driven by table properties</li>



<li>Set retention against real rollback and time travel requirements, gathered from the people who depend on them</li>



<li>Design storage lifecycle rules that clear incomplete uploads, noncurrent versions and delete markers without touching live data files</li>



<li>Diagnose maintenance jobs that report success and free nothing, including prefix mismatches, pinned refs and disabled garbage collection</li>



<li>Add table health metrics and alerting so drift shows up as a graph rather than a surprise on the invoice</li>
</ul>



<p class="wp-block-paragraph">If you want a second opinion, send me the output of your snapshots metadata table, your current table properties, or the counters from a run that did not do what you expected. That is usually enough to tell you where the bytes are.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/iceberg-table-maintenance-storage-costs/">Iceberg Table Maintenance: Why Your Storage Bill Keeps Growing</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
