<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Structured Outputs | John Nessime</title>
	<atom:link href="https://john-nessime.com/blog/tag/structured-outputs/feed/" rel="self" type="application/rss+xml" />
	<link>https://john-nessime.com/blog/tag/structured-outputs/</link>
	<description>Cloud, DevOps, Data &#38; AI — Built, Tested, Explained</description>
	<lastBuildDate>Sun, 20 Sep 2026 13:12:04 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.2</generator>

<image>
	<url>https://john-nessime.com/blog/wp-content/uploads/2026/07/cropped-jn-32x32.png</url>
	<title>Structured Outputs | John Nessime</title>
	<link>https://john-nessime.com/blog/tag/structured-outputs/</link>
	<width>32</width>
	<height>32</height>
</image> 
	<item>
		<title>CV Parsing at Scale on AWS: Textract, Bedrock, and the Bias You Have to Document</title>
		<link>https://john-nessime.com/blog/cloud-computing/cv-parsing-aws-textract-bedrock-bias/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Sun, 30 Aug 2026 09:00:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Compliance]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Adverse Impact]]></category>
		<category><![CDATA[Algorithmic Bias]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[Bedrock Guardrails]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[EU AI Act]]></category>
		<category><![CDATA[HR Technology]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Local Law 144]]></category>
		<category><![CDATA[Model Invocation Logging]]></category>
		<category><![CDATA[OCR]]></category>
		<category><![CDATA[PII Redaction]]></category>
		<category><![CDATA[Resume Parsing]]></category>
		<category><![CDATA[Structured Outputs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=540</guid>

					<description><![CDATA[<p>A CV parsing pipeline stops being a parser the moment a score sorts a list. Where Textract and Bedrock actually fit, why redacting names does nothing for bias, and what to document before anyone asks you to explain a ranking.</p>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cv-parsing-aws-textract-bedrock-bias/">CV Parsing at Scale on AWS: Textract, Bedrock, and the Bias You Have to Document</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The ticket said &#8220;parser is working fine.&#8221; It was. Extraction confidence was high, the dead letter queue was empty, and recruiting liked the shortlists. Then someone in legal asked why candidate 4,412 ranked below candidate 118.</p>



<p class="wp-block-paragraph">Nobody could answer. Not because anything was broken, but because the system had quietly stopped being a parser somewhere between the S3 upload and the sorted list on a recruiter&#8217;s screen. It had become a selection procedure, and nothing in it was designed to explain itself.</p>



<p class="wp-block-paragraph">That is the failure mode worth leading with, because it never pages you. Your alarms stay green while the thing you built lands in a category of regulated system you never intended to build.</p>



<p class="wp-block-paragraph">This post covers CV parsing on AWS as it actually goes: what Amazon Textract gives you and what it doesn&#8217;t, where Bedrock belongs and where it becomes a liability, the bias problem you have to document rather than solve, and the failures that only surface at volume.</p>



<h2 class="wp-block-heading">When CV parsing on AWS becomes a hiring decision</h2>



<p class="wp-block-paragraph">Most of these pipelines start honestly. Someone wants structured JSON out of a PDF pile so the ATS populates fields instead of a coordinator retyping them. Extraction is a data problem with clean success criteria.</p>



<p class="wp-block-paragraph">Then a product manager asks for a match score. It seems small. You already have the skills list and the job description, so you ask a model how well they line up and emit a number. The API contract barely changes.</p>



<p class="wp-block-paragraph">What changed is everything. The moment a recruiter sorts by that number and works down from the top, the pipeline substantially assists a hiring decision. Under New York City&#8217;s Local Law 144 that is an automated employment decision tool, and the obligations attach to the employer using it, not the vendor who wrote the code. Under the EU AI Act, Annex III point 4 lists systems intended to analyse and filter job applications and evaluate candidates as high-risk by classification, not by how confident you are in your prompt.</p>



<p class="wp-block-paragraph">The EU date is worth stating plainly because it moved. Annex III obligations were originally due to apply from 2 August 2026; the Digital Omnibus package pushed stand-alone Annex III systems out to 2 December 2027. That is a reprieve on paperwork, not on design. Systems procured now will still be running then, and the separate prohibition on workplace emotion-recognition AI, plus existing GDPR duties around automated decisions, were never on that timetable.</p>



<p class="wp-block-paragraph">So the first architectural decision isn&#8217;t about services. It&#8217;s about where extraction stops and judgement begins, and whether that boundary is visible in the code or buried in a prompt.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Getting text out: what Textract actually does</h2>



<p class="wp-block-paragraph">Textract is good at CVs in a narrow way. It reads printed and handwritten text, detects tables and form-style key-value pairs, and with the Layout feature type it identifies structural elements such as titles, paragraphs, lists, headers and footers. For a two-column CV with a skills sidebar, layout awareness is the difference between usable text and word salad.</p>



<h3 class="wp-block-heading">Sync versus async is not a performance choice</h3>



<p class="wp-block-paragraph"><code>AnalyzeDocument</code> is synchronous and takes a single image or a document passed as bytes or an S3 object. <code>StartDocumentAnalysis</code> is the asynchronous entry point for multipage documents; it publishes completion status to an SNS topic you drain from SQS before calling <code>GetDocumentAnalysis</code> with the returned job ID.</p>



<p class="wp-block-paragraph">People pick sync because it&#8217;s easier to reason about, then discover half of real CVs are three pages. Go async from the start. The SNS-to-SQS hop isn&#8217;t overhead, it&#8217;s the only thing that gives you a retry surface when a job fails partway through a batch.</p>



<h3 class="wp-block-heading">Queries beat walking the block graph by hand</h3>



<p class="wp-block-paragraph">The Queries feature lets you ask natural-language questions of a document without knowing where the data sits on the page. Pass <code>QUERIES</code> in <code>FeatureTypes</code>, supply a <code>QueriesConfig</code>, then read the <code>QUERY_RESULT</code> blocks out of the response.</p>



<pre class="wp-block-code"><code>aws textract start-document-analysis 
  --document '{"S3Object":{"Bucket":"bucket","Name":"document"}}' 
  --feature-types '["QUERIES"]' 
  --queries-config '{"Queries":[{"Text":"Question"}]}'</code></pre>



<p class="wp-block-paragraph">Each query can carry an alias so answers map back to your schema without positional guessing. In Python, walk the returned blocks and pick the ones whose <code>BlockType</code> is <code>QUERY_RESULT</code>, each carrying answer text and a confidence score.</p>



<pre class="wp-block-code"><code>response = client.analyze_document(
    Document={'S3Object': {'Bucket': bucket, 'Name': document}},
    FeatureTypes=["TABLES", "FORMS", "QUERIES"],
    QueriesConfig={'Queries': [{'Text': question}]}
)

for block in response['Blocks']:
    if block["BlockType"] == "QUERY_RESULT":
        print(block["Text"])</code></pre>



<p class="wp-block-paragraph">Use Queries for facts that exist verbatim on the page: employer name, job title, dates, certification number. Not for anything requiring inference. &#8220;What is this person&#8217;s seniority level?&#8221; is a judgement, and pushing it through an extraction API hides that judgement inside something that looks like OCR.</p>



<h2 class="wp-block-heading">Where Bedrock belongs, and where it stops</h2>



<p class="wp-block-paragraph">Bedrock earns its place in one job: turning messy extracted text into a consistent schema. CVs are the worst kind of semi-structured document because the structure is a fashion trend. One person lists dates as &#8220;Mar 2019 to present,&#8221; another renders them as a bar chart. Normalising that is exactly the fuzzy transformation a language model handles well, and one you can verify.</p>



<ol class="wp-block-list">
<li>Constrain output to a schema and reject anything that fails validation. A job title you didn&#8217;t ask for, or an employment date absent from the source text, is a failed record rather than a creative flourish.</li>



<li>Turn on model invocation logging before you process a single real CV. It&#8217;s disabled by default, and it&#8217;s the only native record of what you sent and what came back.</li>
</ol>



<p class="wp-block-paragraph">That second point isn&#8217;t a nice-to-have. When someone asks a year from now why a candidate scored the way they did, the invocation log is your answer or you don&#8217;t have one.</p>



<pre class="wp-block-code"><code>aws bedrock put-model-invocation-logging-configuration 
  --logging-config '{
    "s3Config": {"bucketName": "my-bedrock-logs", "keyPrefix": "cv-pipeline/"},
    "textDataDeliveryEnabled": true
  }'</code></pre>



<p class="wp-block-paragraph">Logs go to CloudWatch Logs, S3, or both, and only destinations in the same account and Region are supported. Apply lifecycle rules and encryption up front, because these logs contain full CVs and they outlive the hiring round that produced them.</p>



<p class="wp-block-paragraph">Bedrock Guardrails fits here through sensitive information filters, which detect PII in prompts and responses and either block the payload or mask individual entities with identifier tags. There&#8217;s a detect mode that reports what a guardrail would have caught without acting, which is the right way to tune against real documents first. Don&#8217;t oversell it to yourself: masking is not anonymisation, and a filter firing on <code>NAME</code> and <code>EMAIL</code> does very little about bias, for reasons that are the whole point of the next section.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">The bias problem you must document, not solve</h2>



<p class="wp-block-paragraph">Every engineer&#8217;s instinct is to strip the protected attributes and declare it handled. Remove the name, the photo, the address, and the model can&#8217;t discriminate on what it can&#8217;t see. It doesn&#8217;t work, and understanding why separates a pipeline you can defend from one you can&#8217;t.</p>



<h3 class="wp-block-heading">Proxies survive redaction</h3>



<p class="wp-block-paragraph">A CV is dense with correlated signal. University name tracks geography and socioeconomics. Graduation year is an age proxy that survives every name redaction you can write. Employment gaps correlate with caregiving. Postcode correlates with almost everything. Add professional association membership, the language a qualification is written in, whether military service appears, the phrasing conventions of a non-native English speaker.</p>



<p class="wp-block-paragraph">None of those are protected attributes. All of them carry information about protected attributes, and a model optimising for similarity to your existing successful hires will use them. Redaction removes the label, not the signal, and it makes disparity harder to detect because you deleted the field you would have measured against.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow">
<p class="wp-block-paragraph">You can&#8217;t prove a screening pipeline is unbiased. You can measure its outcomes, record what you measured, and act when the numbers move.</p>
</blockquote>



<h3 class="wp-block-heading">What the four-fifths rule actually asks for</h3>



<p class="wp-block-paragraph">The Uniform Guidelines on Employee Selection Procedures set out the rule of thumb federal enforcement agencies apply: a selection rate for any race, sex or ethnic group falling below four-fifths of the rate for the highest group is generally treated as evidence of adverse impact. It sits at 29 CFR 1607.4(D).</p>



<ul class="wp-block-list">
<li><strong>It&#8217;s a rule of thumb, not a safe harbour.</strong> The same guidelines note smaller differences can still constitute adverse impact where they&#8217;re significant in statistical and practical terms, and courts have declined to treat clearing 0.8 as proof nothing is wrong.</li>



<li><strong>It applies to the selection procedure.</strong> That means your parser, your scorer and the recruiter&#8217;s sort order taken together. Auditing the model in isolation measures the wrong thing.</li>
</ul>



<p class="wp-block-paragraph">The computation is trivial, which is why there&#8217;s no excuse for skipping it.</p>



<pre class="wp-block-code"><code>SELECT
  demographic_group,
  COUNT(*) AS applicants,
  SUM(CASE WHEN advanced THEN 1 ELSE 0 END) AS selected,
  1.0 * SUM(CASE WHEN advanced THEN 1 ELSE 0 END) / COUNT(*) AS selection_rate
FROM screening_outcomes
WHERE requisition_id = :req
GROUP BY demographic_group;</code></pre>



<p class="wp-block-paragraph">Divide each selection rate by the largest and you have the impact ratio. Anything under 0.8 is a signal to stop and look, not a number to explain away.</p>



<h3 class="wp-block-heading">Where the demographic data comes from</h3>



<p class="wp-block-paragraph">This is the genuinely hard part and where most teams stall. You need demographic data to measure disparity, and you must not feed it into the screening decision. The pattern that works is a strict split: self-reported responses collected separately at application time, stored in a different bucket under a different KMS key, joined to outcomes only in an analysis account the scoring pipeline has no path to.</p>



<p class="wp-block-paragraph">NYC&#8217;s regime allows historical or test data, and requires the audit be performed by someone not involved in using or developing the tool. That independence requirement is easy to miss when you&#8217;re the consultant who built the thing. You can build the measurement harness. You can&#8217;t be the independent auditor of your own pipeline.</p>



<h3 class="wp-block-heading">Document the pipeline, not the model</h3>



<p class="wp-block-paragraph">The artifact that saves you is boring and written down. At minimum: which model and version produced scores in each date range, the scoring logic in force, every field that reached the scorer, every field withheld and why, where human review sits and what a reviewer can override, and the impact ratios from each measurement run.</p>



<p class="wp-block-paragraph">Keep it in version control beside the infrastructure code. A change to the prompt is a change to the selection procedure and should be reviewable as one.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Problems that only appear at volume</h2>



<ul class="wp-block-list">
<li><strong>Throttling arrives as data loss.</strong> Without retries using exponential backoff and jitter, a throttled call becomes a missing candidate rather than an error anyone sees. Assert on record counts at every stage boundary.</li>



<li><strong>Duplicates are the norm.</strong> The same person applies to three roles with two slightly different files. Deduplicate on content hash before you spend on extraction, and keep an idempotency key so a retried job doesn&#8217;t create a second candidate record.</li>



<li><strong>Cost is dominated by re-runs.</strong> Extraction is the expensive step and it&#8217;s deterministic on the same input. Store the raw Textract response in S3 keyed by document hash. Reprocessing after a schema change should cost storage, not inference.</li>



<li><strong>Retention outlives the requisition.</strong> Extracted text, invocation logs and intermediate JSON all contain personal data and all quietly accumulate. Set lifecycle rules at build time, not after the first data subject request.</li>



<li><strong>Scanned CVs are a different pipeline.</strong> Photographed documents behave nothing like exported PDFs. Route them separately with lower confidence thresholds and higher review rates.</li>
</ul>



<p class="wp-block-paragraph">On cost visibility, native tooling shows the shape of spend but is slow to attribute it. If this runs alongside other AI workloads, a dedicated platform such as Vantage or CloudZero makes per-pipeline attribution less painful than slicing Cost Explorer by tag every month. And if the upload front door is public, putting Cloudflare in front of it deals with bot traffic before it becomes Textract spend.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<ul class="wp-block-list">
<li><strong>High confidence, wrong fields.</strong> Almost always multi-column layouts read in the wrong order. Add the Layout feature type and check reading order in the block graph against what a human sees.</li>



<li><strong>Async jobs complete but results never arrive.</strong> Check the SNS topic policy allows publishing to your queue and that the role passed to Textract can publish. This fails silently and looks like a stuck pipeline.</li>



<li><strong>The model invents employment history.</strong> Your prompt is asking it to summarise rather than extract. Constrain the schema, require every value trace to source text, and fail validation instead of accepting a plausible answer.</li>



<li><strong>Scores shift after a model update.</strong> Pin the model identifier rather than tracking a moving alias, and re-run your measurement set against a held-out corpus before promoting any change.</li>



<li><strong>Guardrails block legitimate CVs.</strong> Sensitive information filters are probabilistic and context-dependent. Run detect mode against a real corpus and read the traces before switching to block.</li>
</ul>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating a match score as a data field rather than a decision, and skipping the governance that follows.</li>



<li>Redacting names and considering the bias question closed.</li>



<li>Enabling invocation logging after the pilot, leaving the first thousand decisions with no record.</li>



<li>Building human review that only sees the shortlist, which reviews the output of the bias rather than the bias.</li>



<li>Assuming a US-headquartered employer is out of scope. Both regimes reach based on where the candidate is, not where the company is.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Draw a hard line between extraction and scoring: separate services, separate IAM roles, separate logs.</li>



<li>Enable model invocation logging before the first real document, with encryption and lifecycle rules already applied.</li>



<li>Cache raw extraction output keyed by document hash so reprocessing never re-bills inference.</li>



<li>Collect demographic data separately, key it differently, and join it to outcomes only in an isolated analysis account.</li>



<li>Run impact ratios on a schedule rather than on request, and alert on movement instead of waiting for an audit.</li>



<li>Version prompts and scoring logic in the same repository as the infrastructure, reviewed as changes to a selection procedure.</li>
</ul>



<h2 class="wp-block-heading">Frequently asked questions</h2>



<h3 class="wp-block-heading">Is Textract or Bedrock the right tool for CV parsing?</h3>



<p class="wp-block-paragraph">Both, for different halves. Textract turns pixels into text with layout awareness and confidence scores. Bedrock turns that text into a consistent schema. Using a language model for the OCR step gives up per-field confidence, which is exactly the signal you need to decide what goes to human review.</p>



<h3 class="wp-block-heading">Does removing names and photos make a resume screening pipeline compliant?</h3>



<p class="wp-block-paragraph">No. Redaction handles direct identifiers while leaving proxies such as graduation year, institution, postcode and employment gaps intact. It also removes the fields you&#8217;d need to measure disparity, so it can make a pipeline harder to audit while feeling safer.</p>



<h3 class="wp-block-heading">Do these rules apply if my company isn&#8217;t in New York or the EU?</h3>



<p class="wp-block-paragraph">Possibly. Local Law 144 turns on whether the candidate resides in New York City, including for remote roles, rather than where the employer sits. The EU AI Act has similar extraterritorial reach. Check with counsel, not with an architecture diagram.</p>



<h3 class="wp-block-heading">Can I run the bias audit myself?</h3>



<p class="wp-block-paragraph">You can build the measurement pipeline and run it continuously, and you should. The formal NYC bias audit has to be performed by an independent party not involved in using or developing the tool, which rules out the team that built it.</p>



<h3 class="wp-block-heading">How do I keep extraction costs predictable at scale?</h3>



<p class="wp-block-paragraph">Deduplicate before extraction, cache raw responses by content hash, and choose feature types deliberately. Requesting tables, forms and queries on every document when you only need queries multiplies cost for nothing. Both services bill per unit of work, so the lever is doing less work rather than finding a cheaper tier.</p>



<h3 class="wp-block-heading">Should the pipeline auto-reject candidates?</h3>



<p class="wp-block-paragraph">Treat that as a question for counsel, not engineering. GDPR restricts solely automated decisions with significant effects on individuals, and the practical answer in most designs is a meaningful human review before any rejection. Build the human step in early; retrofitting it is painful.</p>



<h2 class="wp-block-heading">The one thing worth remembering</h2>



<p class="wp-block-paragraph">CV parsing on AWS is two systems wearing one name. The extraction half is an engineering problem with clean success criteria, and Textract plus Bedrock will carry you a long way given retries, caching and schema validation. The scoring half is a selection procedure, governed by rules that don&#8217;t care what you called the service.</p>



<p class="wp-block-paragraph">You won&#8217;t eliminate bias from a system trained on hiring history. What you can do is know where the boundary sits, log what crossed it, measure outcomes on a schedule, and be able to answer the question about candidate 4,412 when it arrives. Build the documentation while you build the pipeline. Reconstructing it later from CloudWatch and memory is a much worse job.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Work with me on document pipelines and AI governance</h2>



<p class="wp-block-paragraph">Most of the work here isn&#8217;t the extraction. It&#8217;s the boundary drawing, the evidence trail and the parts that only break at volume. Things I help with:</p>



<ul class="wp-block-list">
<li>Designing Textract and Bedrock pipelines with an auditable split between extraction and any scoring or ranking step.</li>



<li>Building the measurement harness that computes selection rates and impact ratios on a schedule, with demographic data isolated from the scoring path.</li>



<li>Setting up model invocation logging, retention, encryption and access controls so the evidence trail exists before you need it.</li>



<li>Fixing throughput and cost problems: throttling, deduplication, extraction caching and reprocessing that doesn&#8217;t re-bill inference.</li>



<li>Adding human-in-the-loop review that routes on confidence and score bands, including sampled rejections rather than shortlists only.</li>



<li>Writing the technical documentation an independent auditor or your legal team will actually ask for.</li>
</ul>



<p class="wp-block-paragraph">Send me a sample Textract response, a scoring prompt or an architecture sketch and I&#8217;ll tell you where the boundary is currently blurred.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/cloud-computing/cv-parsing-aws-textract-bedrock-bias/">CV Parsing at Scale on AWS: Textract, Bedrock, and the Bias You Have to Document</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
		<item>
		<title>Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</title>
		<link>https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/</link>
		
		<dc:creator><![CDATA[John Nessime]]></dc:creator>
		<pubDate>Tue, 18 Aug 2026 11:17:11 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[Data Engineering]]></category>
		<category><![CDATA[Technical Guides]]></category>
		<category><![CDATA[Amazon Bedrock]]></category>
		<category><![CDATA[Amazon S3]]></category>
		<category><![CDATA[Amazon Textract]]></category>
		<category><![CDATA[AWS]]></category>
		<category><![CDATA[AWS Lambda]]></category>
		<category><![CDATA[Bedrock Data Automation]]></category>
		<category><![CDATA[Confidence Scoring]]></category>
		<category><![CDATA[Contract Analysis]]></category>
		<category><![CDATA[Document Processing]]></category>
		<category><![CDATA[Human In The Loop]]></category>
		<category><![CDATA[Intelligent Document Processing]]></category>
		<category><![CDATA[Legal Tech]]></category>
		<category><![CDATA[Schema Design]]></category>
		<category><![CDATA[Step Functions]]></category>
		<category><![CDATA[Structured Outputs]]></category>
		<guid isPermaLink="false">https://john-nessime.com/blog/?p=392</guid>

					<description><![CDATA[<p>Valid JSON is not correct data. A practical guide to extracting clauses, parties, dates and obligations from contracts with Amazon Bedrock: the two build paths, schema design, the three failure families that produce confident wrong answers, and the validation layer that catches them.</p>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/">Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph">The extraction job finishes clean. Every contract in the batch produces valid JSON, every field is populated, the schema validator passes, and the results load into the obligation tracker without a single error. Three weeks later someone in legal asks why the tracker says the renewal notice window is 30 days when the contract says 90. That is the moment you learn the real problem with contract data extraction: the failure mode is not malformed output. It is well-formed output that is wrong.</p>



<p class="wp-block-paragraph">This post is a practical guide to how I&#8217;d extract contract data with Amazon Bedrock: the clauses, parties, dates, and obligations that feed contract lifecycle tools, obligation trackers, and compliance dashboards. It covers the two build paths Bedrock gives you, how to design the extraction schema, the three failure families that produce confident wrong answers, and the validation layer that catches them before they reach a system someone trusts.</p>



<h2 class="wp-block-heading">Two ways to extract contract data with Amazon Bedrock</h2>



<p class="wp-block-paragraph">Bedrock offers two genuinely different routes to the same destination, and picking the wrong one costs you either flexibility or months of pipeline plumbing you didn&#8217;t need to build.</p>



<h3 class="wp-block-heading">Path one: call a model directly</h3>



<p class="wp-block-paragraph">You send the contract to a foundation model, typically Anthropic Claude, through the Converse or InvokeModel API, with a prompt and a schema describing the fields you want back. Converse accepts PDF documents as a content block, so contract bytes go straight from S3 without a separate OCR step. One detail that catches people: full visual understanding of a PDF through Converse with Claude requires citations to be enabled. Without them the API falls back to plain text extraction, and anything living in a table or a scanned signature page silently disappears from the model&#8217;s view.</p>



<p class="wp-block-paragraph">The piece that changed this path from fragile to dependable is structured outputs. Instead of asking the model nicely for JSON and writing defensive parsing code for the day it prepends an apology, you attach a JSON Schema to the request and Bedrock enforces it during token generation through constrained decoding. The response cannot fail to parse. One mechanic worth knowing: the first request with a new schema triggers a grammar compilation step, and compiled grammars are cached for a limited time, so schema churn has a latency cost.</p>



<h3 class="wp-block-heading">Path two: Bedrock Data Automation</h3>



<p class="wp-block-paragraph">Bedrock Data Automation (BDA) is the managed intelligent document processing service. You define a blueprint, which is a list of fields with types and natural language instructions for how to find and normalize each one, attach it to a project, and invoke the async processing API against files in S3. BDA returns two things a raw model call does not give you out of the box: a confidence score per field and visual grounding, meaning bounding boxes pointing back to where on the page each value came from. A project can hold multiple document blueprints, so a mixed intake of NDAs, master service agreements, and amendments routes each document to the right extraction logic automatically.</p>



<p class="wp-block-paragraph">My honest read on the choice: BDA wins when you want the boring parts managed and you need per-field confidence and grounding for a human review loop, which for contracts you almost always do. The direct model path wins when you need reasoning the blueprint format can&#8217;t express, like interpreting a clause against the definitions section. Plenty of real pipelines use both: BDA for the structured pass, a direct Claude call for the clauses that need interpretation. Amazon Textract still earns its place for high-volume standardized forms, but contracts are exactly the variable, long-tail documents layout-based extraction was never great at.</p>



<h2 class="wp-block-heading">Design the schema before you touch the API</h2>



<p class="wp-block-paragraph">The schema is not a formality. With structured outputs, the field names, descriptions, and ordering you write become context the model generates against, so the schema is effectively part of the prompt. A vague schema produces structurally valid garbage: JSON that parses perfectly and means nothing.</p>



<p class="wp-block-paragraph">Three design rules that pay for themselves. First, one concept per field: &#8220;renewal terms&#8221; as a single string is unusable downstream; split it into whether renewal exists, whether it is automatic, the duration, and the notice period to prevent it. Second, write field descriptions like instructions to a junior reviewer, including what the field is not. Third, make every field nullable and tell the model to return null when the contract is silent. A model forced to fill a required field will fill it with something.</p>



<p class="wp-block-paragraph">Here is the shape I&#8217;d start with for a general commercial agreement, trimmed to the core:</p>



<pre class="wp-block-code"><code>{
  "type": "object",
  "properties": {
    "parties": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "legal_name": { "type": ["string", "null"],
            "description": "Full registered entity name from the preamble or signature block, not the defined shorthand" },
          "defined_term": { "type": ["string", "null"],
            "description": "The shorthand assigned in the preamble, e.g. Supplier, Client" },
          "role": { "type": ["string", "null"] }
        }
      }
    },
    "effective_date": { "type": ["string", "null"],
      "description": "Date the agreement takes effect. Null if only a signature date exists." },
    "governing_law": { "type": ["string", "null"] },
    "obligations": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "obligor": { "type": ["string", "null"] },
          "description": { "type": ["string", "null"] },
          "deadline_text": { "type": ["string", "null"],
            "description": "The deadline exactly as written, e.g. within 30 days of the Effective Date" },
          "source_clause": { "type": ["string", "null"],
            "description": "Section number where this obligation appears" }
        }
      }
    }
  }
}</code></pre>



<p class="wp-block-paragraph">Notice <code>deadline_text</code> keeps the deadline as written instead of asking the model to compute a calendar date. That is deliberate, and the reason is the second failure family below.</p>



<h2 class="wp-block-heading">Failure family one: parties that aren&#8217;t who they say they are</h2>



<p class="wp-block-paragraph">Contracts name their parties once, in the preamble, then never again. From that point on everything refers to &#8220;the Supplier&#8221; or &#8220;the Receiving Party.&#8221; A naive extraction returns the defined term as the party name, which is useless for matching against a vendor master or a CRM. Worse, the signature block sometimes names a different legal entity than the preamble, usually an affiliate signing on behalf of a subsidiary, and the model will happily pick whichever it saw last.</p>



<p class="wp-block-paragraph">The fix is structural, not clever prompting. Extract the legal name and the defined term as separate fields, as in the schema above, with instructions to take legal names from the preamble and cross-check the signature block. Then validate outside the model: if the two entities differ, flag the document for review rather than picking one. That mismatch is occasionally a genuine drafting error in the contract itself, which makes it exactly the kind of thing you want surfaced, not smoothed over.</p>



<h2 class="wp-block-heading">Failure family two: dates that mean different things</h2>



<p class="wp-block-paragraph">A contract carries several dates that look interchangeable and are not: execution, effective, commencement, expiry, and the deadlines derived from notice periods. The extraction failure that costs money is almost never a misread date string. It is a correctly read date assigned to the wrong concept, or a relative deadline flattened into a wrong absolute one.</p>



<p class="wp-block-paragraph">Relative dates are the invisible one. &#8220;Either party may terminate by written notice no later than ninety days before the end of the then-current term&#8221; contains no date at all. If your schema demands a date type, the model computes one, and its arithmetic rests on assumptions about the term start that may be wrong, especially when an amendment has reset the term. That computed date looks exactly as trustworthy in your tracker as a real one. This is why I extract deadline language verbatim and compute calendar dates in code, where the arithmetic is testable and the inputs are auditable.</p>



<p class="wp-block-paragraph">Cheap validation catches most of the rest. The effective date must not fall after the expiry. A notice period must be a plausible number of days. If the agreement auto-renews, a renewal duration should exist. Fields that fail these checks get flagged. None of this needs a model; it is twenty lines of code standing between extraction and the systems people trust.</p>



<h2 class="wp-block-heading">Failure family three: obligations, cross-references, and the amendment problem</h2>



<p class="wp-block-paragraph">Obligations are the hardest target because they are relational. &#8220;The Supplier shall deliver the reports described in Schedule 3 within the period set out in Section 4.2, except as provided in Section 9&#8221; is one obligation spread across three locations. Chunk the contract carelessly and the model sees the reference with no Section 4.2 in sight. It will either drop the deadline or invent a plausible one, and the second outcome is far worse because nothing downstream looks broken.</p>



<p class="wp-block-paragraph">Contracts resist chunking more than almost any other document type, because defined terms and cross-references make every section depend on distant ones. The practical answers, in order of preference:</p>



<ol class="wp-block-list">
<li>Don&#8217;t chunk. Modern model context windows fit most commercial agreements whole, and whole-document extraction is the single biggest accuracy win available. Reach for chunking only when a document genuinely doesn&#8217;t fit.</li>



<li>If you must split, prepend the definitions section and the table of contents to every chunk, and split on section boundaries rather than fixed token counts so no clause straddles a cut.</li>



<li>Capture the source clause reference for every obligation, as in the schema above, so a reviewer can jump straight to the language instead of re-reading the contract.</li>
</ol>



<p class="wp-block-paragraph">Then there are amendments. An amendment is a short document that quietly rewrites the base agreement, and if you extract each file independently your dataset will assert the original terms with full confidence long after they stopped being true. Treat the contract family, not the file, as the unit of extraction: process the base agreement, process each amendment, and resolve them in order so later documents override earlier ones. BDA&#8217;s multi-blueprint routing helps here, since amendments have a recognizably different shape and can carry their own blueprint focused on what changed.</p>



<h2 class="wp-block-heading">The validation layer that makes it trustworthy</h2>



<p class="wp-block-paragraph">Schema compliance tells you the output is well-formed. It says nothing about whether it is true. The trust comes from three checks layered after extraction.</p>



<ul class="wp-block-list">
<li><strong>Cross-field consistency.</strong> Dates in order, notice periods plausible, every obligation carrying an obligor that matches an extracted party. Pure code, no model.</li>



<li><strong>Grounding.</strong> Every extracted value should point back to a location in the document. BDA gives you this natively through bounding boxes; on the direct model path, requiring a source clause reference per field is the lightweight version. A value that cannot be traced to the page is a value you cannot audit.</li>



<li><strong>Confidence-based routing.</strong> Use BDA&#8217;s per-field confidence scores to route low-confidence extractions to human review and let high-confidence ones flow through. The goal is not eliminating review; it is spending reviewer minutes only where the model is unsure. Party names and governing law typically extract very reliably, while obligations from unusual templates are where automation honestly ends and review begins.</li>
</ul>



<p class="wp-block-paragraph">Operationally this looks like S3 event notifications triggering a Step Functions workflow: extract, validate, then branch to auto-accept or a review queue. Instrument it from day one. Track the auto-accept rate, per-field confidence distributions, and validation failure counts; a slow drift in confidence is usually the first sign a new contract template has entered the intake. I ship these to Grafana Cloud with the rest of the pipeline metrics, and since direct model calls bill by tokens while BDA bills per unit of content processed, a cost dashboard in something like Vantage or CloudZero shows quickly whether whole-document extraction is worth its token count for your mix. It usually is.</p>



<h2 class="wp-block-heading">Troubleshooting</h2>



<p class="wp-block-paragraph"><strong>Fields come back null for content that is clearly in the document.</strong> On the Converse path with PDFs, check whether you are getting text-only extraction; content in tables, stamps, or scanned pages needs the visual analysis mode. For scans, verify image quality before blaming the model.</p>



<p class="wp-block-paragraph"><strong>The same contract extracts differently on reruns.</strong> Set temperature to zero for extraction work, then look for instructions the model can resolve more than one way. Ambiguity in a schema description is the usual culprit.</p>



<p class="wp-block-paragraph"><strong>First requests with a new schema are slow.</strong> That is grammar compilation for structured outputs. Stabilize your schemas and warm them before batch runs rather than generating schemas dynamically per request.</p>



<p class="wp-block-paragraph"><strong>BDA accuracy is mediocre on your templates.</strong> Blueprint instructions are tunable, and BDA can refine them automatically from a small set of example documents with known correct values. Feed it your hardest real examples, not your cleanest ones.</p>



<h2 class="wp-block-heading">Common mistakes</h2>



<ul class="wp-block-list">
<li>Treating schema-valid JSON as verified data. Structure and truth are separate properties, and the second one needs its own checks.</li>



<li>Extracting the defined term as the party name and matching it against vendor records.</li>



<li>Letting the model compute calendar dates from relative deadline language instead of extracting the language and computing in code.</li>



<li>Chunking by token count through a document whose clauses reference each other across fifty pages.</li>



<li>Processing amendments as standalone documents and silently keeping stale base-agreement terms.</li>



<li>Making every schema field required, which converts &#8220;the contract is silent&#8221; into a fabricated answer.</li>
</ul>



<h2 class="wp-block-heading">Best practices</h2>



<ul class="wp-block-list">
<li>Extract whole documents whenever the context window allows. Every chunking strategy is a workaround, not a feature.</li>



<li>Keep a small labeled evaluation set of your own contracts and score every prompt or blueprint change against it before shipping.</li>



<li>Require a source reference for every extracted value, whether that is a BDA bounding box or a clause number from the model.</li>



<li>Route on confidence and validation results, and measure your auto-accept rate as a first-class metric.</li>



<li>Version your schemas and store the schema version alongside every extraction, so you can tell which records predate a fix.</li>



<li>Keep humans in the loop for high-stakes fields. The realistic goal is automating the bulk and focusing legal review, not replacing it.</li>
</ul>



<h2 class="wp-block-heading">FAQ</h2>



<h3 class="wp-block-heading">Should I use Bedrock Data Automation or call Claude directly for contract extraction?</h3>



<p class="wp-block-paragraph">BDA if you want managed parsing, per-field confidence scores, and visual grounding for a review workflow. Direct model calls if you need interpretive reasoning over clauses or conversational access to the document. Many production pipelines combine both.</p>



<h3 class="wp-block-heading">Do I still need Amazon Textract?</h3>



<p class="wp-block-paragraph">Usually not as the primary extractor for contracts, since layout-based extraction struggles with variable legal drafting. It remains a good fit for high-volume standardized forms, and some pipelines route documents between Textract and generative extraction by type.</p>



<h3 class="wp-block-heading">How accurate is contract data extraction in practice?</h3>



<p class="wp-block-paragraph">It varies sharply by field type. Parties, governing law, and explicit dates extract reliably. Obligations, renewal mechanics, and anything expressed through cross-references are meaningfully harder, which is why confidence routing and human review exist. Distrust any vendor quoting one accuracy number for &#8220;contracts.&#8221;</p>



<h3 class="wp-block-heading">How do I extract obligations specifically, not just metadata?</h3>



<p class="wp-block-paragraph">Model each obligation as its own object with obligor, description, deadline language, and source clause. Extract against the whole document so cross-references resolve, and validate that every obligor matches an extracted party.</p>



<h3 class="wp-block-heading">Can Bedrock handle scanned contracts?</h3>



<p class="wp-block-paragraph">Yes. BDA processes scanned documents as part of its parsing stage, and Claude&#8217;s visual document understanding reads scanned pages when invoked with visual analysis. Scan quality still matters; poor scans degrade every approach.</p>



<h3 class="wp-block-heading">Is it safe to put contracts through Bedrock?</h3>



<p class="wp-block-paragraph">Bedrock runs within your AWS account boundary, and AWS states that customer content is not used to train the underlying models. For sensitive agreements the usual controls apply: VPC endpoints for private connectivity, KMS encryption on the buckets, and IAM scoped to the pipeline roles. Confirm the specifics against your own compliance requirements.</p>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">If you remember one thing, make it this: when you extract contract data with Amazon Bedrock, valid output is the starting line, not the finish. Structured outputs and BDA blueprints have genuinely solved the formatting problem, which means the remaining risk is concentrated in meaning: the wrong entity, the wrong date concept, the obligation whose deadline lives two sections away. Design the schema around those failures, validate outside the model, ground every value to the page, and route on confidence. That is the difference between a demo and a pipeline a legal team will actually trust.</p>



<hr class="wp-block-separator has-alpha-channel-opacity"/>



<h2 class="wp-block-heading">Need help building a contract extraction pipeline on AWS?</h2>



<p class="wp-block-paragraph">I design and build document intelligence pipelines on AWS as a freelance DevOps engineer. If you&#8217;re extracting data from contracts or other legal documents, here&#8217;s what I can help with:</p>



<ul class="wp-block-list">
<li>Designing extraction schemas and BDA blueprints for your actual contract types, including amendments and multi-document families</li>



<li>Building the full serverless pipeline: S3 intake, Step Functions orchestration, extraction, validation, and review routing</li>



<li>Adding the validation and grounding layer that catches confident wrong answers before they reach your CLM or tracker</li>



<li>Setting up evaluation sets and accuracy measurement so prompt and blueprint changes are tested, not guessed</li>



<li>Locking down the security side: VPC endpoints, KMS encryption, and IAM scoping for sensitive documents</li>



<li>Cost tuning across model choice, whole-document versus chunked extraction, and BDA versus direct model calls</li>
</ul>



<p class="wp-block-paragraph">Send me a sample contract (redacted is fine), your target field list, or the output of a pipeline that isn&#8217;t behaving, and I&#8217;ll tell you what I&#8217;d do with it.</p>



<div class="wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex">
<div class="wp-block-button"><a class="wp-block-button__link wp-element-button" href="https://www.upwork.com/freelancers/~01f15a912ad84a6620" target="_blank" rel="noreferrer noopener">Work with me on Upwork</a></div>
</div>
<p>The post <a href="https://john-nessime.com/blog/technical-guides/extract-contract-data-amazon-bedrock/">Extracting Clauses, Parties, Dates and Obligations with Amazon Bedrock</a> appeared first on <a href="https://john-nessime.com/blog">John Nessime</a>.</p>
]]></content:encoded>
					
		
		
			</item>
	</channel>
</rss>
