The demo works. You ask the model to pull the invoice number, the date and the line items out of a document, it returns clean JSON, and everybody agrees this is going to save a great deal of time.
Then it goes to production and you learn what the demo was hiding. Roughly one document in forty comes back with a trailing comma. Occasionally the model explains what it has done before the JSON, so the response no longer parses. Once in a while a date arrives as 03/04/2025 and nobody knows whether that is March or April.
None of these are model failures in an interesting sense. They are the predictable result of asking for structure in prose and hoping.
Stop parsing prose
The first fix is the one people skip because it feels like a detail: do not ask the model to write JSON in its reply. Use the tool-use interface and define the output as a tool with a JSON Schema. We build extraction on Claude, where this is a first-class part of the Messages API rather than an add-on — you declare a tool, give it an input_schema, and the response arrives as a validated tool_use block.
The difference is not stylistic. When the shape is expressed as a schema on a tool, the response comes back as a structured object rather than text you have to salvage. Required fields are required. Enums constrain to the values you listed. Types are types. The class of bug where the model narrates before the payload disappears, because there is no longer a text channel for it to narrate into.
Every technique below is worth less than this one. Do this first.
Make the schema carry the meaning
A schema is not just a shape — it is the most reliable place to put instructions, because it sits directly beside the field it governs. A description on a property is read in the context of producing that property, where a line buried in a long prompt is not.
So spend the effort there. Not "date": {"type": "string"}, but a description saying the invoice issue date, ISO 8601, and what to do if the document shows more than one. Not "currency": {"type": "string"}, but an enum of the currencies you actually accept.
Two rules earn their keep repeatedly:
Enumerate wherever the set is closed. Any field with a known set of valid values should be an enum. This eliminates a whole category of downstream normalisation, and it converts “the model invented a category” from a silent data problem into a schema violation you can see.
Give absence somewhere to go. If a field can legitimately be missing, model that explicitly — nullable, or a sibling boolean, or a status enum with a not_found member. If the schema demands a value the document does not contain, the model has been left with a choice between violating the schema and inventing something, and it will usually invent something. Most hallucination in extraction is a schema design failure.
Ask for provenance
The highest-value field in an extraction schema is frequently not extracted data at all. It is source_text: the verbatim span the value came from.
It costs a few tokens and it changes the economics of review. A human checking a total against a quoted line from the document works far faster than one hunting through fourteen pages. It gives you a cheap automated check — if the quoted span is not present in the source, something is wrong, and you can detect that without a human at all. And when a value is disputed months later, you can show where it came from.
A confidence or needs_review field is worth adding for the same reason. It will not be perfectly calibrated, but it is good enough to sort a review queue, and sorting the queue is most of the value.
Validate as though it were a hostile input
Schema conformance means the shape is right. It does not mean the content is.
Validate business rules separately: the line items sum to the stated total, the date falls in a plausible range, the supplier reference exists in your master data. These checks are ordinary code, they are deterministic, and they catch the failures that matter — a plausible wrong number is far more dangerous than a malformed response, because the malformed one cannot get past your parser and the plausible one goes straight into a payment run.
Decide in advance what happens on failure. Retrying once with the validation error appended is cheap and works surprisingly often. Retrying indefinitely is how you build a system that spends money in a loop.
Extraction is a step, not a system
The framing that keeps this manageable: extraction is one task type inside a workflow, with a declared input, a schema’d output, and a defined failure path. It is not an AI project.
Treated that way, it composes. The extraction step writes into the run document under a declared key; the next step is a validation rule; the step after that routes to a human if confidence is low or the totals disagree. That routing is deterministic, auditable, and entirely free of model involvement.
The model does the thing it is uniquely good at — reading an unstructured document — and hands a typed object to a process that behaves like software. Everything that made the demo look fragile lives in the join between them, which is precisely where a schema belongs.
Designing the schema itself
Since the schema carries the meaning, it deserves the care usually spent on the prompt. A few patterns earn their place repeatedly.
Flat beats nested, until it does not
Deeply nested schemas produce more structural errors and are harder to validate partially. Keep the shape as flat as the domain allows. The legitimate exception is genuinely repeating structure — line items on an invoice are an array of objects and pretending otherwise creates parallel arrays that can fall out of alignment.
One field, one fact
A field called customer holding “Acme Ltd, 14 High St, London” is three facts in a string, and every downstream consumer will parse it slightly differently. Split them in the schema. It costs a few tokens and removes a whole category of ambiguity.
Model uncertainty explicitly
The most useful pattern in production extraction is a per-field wrapper rather than a bare value:
value— the extracted content, nullablesource_text— the verbatim span it came fromconfidence— a coarse band, not a false-precision decimalstatus—found,not_present,illegible,ambiguous
That status enum is doing quiet, important work. Without it, “the field is not in this document” and “the field is here but I cannot read it” collapse into the same null, and they require completely different handling — one is a valid outcome, the other is a scan quality problem.
Coarse confidence bands
Ask for high / medium / low rather than a number. A model asked for 0–1 will produce 0.87 with an air of precision it cannot support, and someone will inevitably build a threshold at 0.85. Three bands are about the resolution that is actually meaningful, and they map cleanly onto three actions: accept, review, reject.
Retry strategy that terminates
Retrying with the validation error appended works well and needs boundaries.
Retry at most once or twice. If two attempts with explicit feedback both fail, the third rarely succeeds, and you are now spending money in a loop.
Retry only on validation failure. Transport errors need a different policy — backoff, not re-prompting. Conflating them means retrying a rate limit immediately, which makes it worse.
Make the feedback specific. “The field `total` must be a number; you returned ‘1,234.56 GBP’. Return the numeric value only; currency belongs in `currency`.” A generic “that was invalid” mostly produces a differently invalid response.
Have a terminal state. After the retries, the item goes to human review with the failed output attached. Silently dropping it is how a batch quietly loses 3% of its records.
Evaluating extraction without a benchmark
Public benchmarks will not tell you whether this works on your documents. What works is a small, honest, growing set.
Start with fifty documents that reflect the real distribution — including the bad scans, the unusual layouts and the one supplier whose invoices are photographs of a screen. Label them by hand once. That is the ground truth.
Measure per field, not per document. Document-level accuracy hides that one field is at 99% and another at 60%, and the 60% field is the one causing all the rework. Per-field numbers tell you where to spend effort.
Then close the loop: every human correction in review is a new labelled example. Feed those back into the evaluation set, and gate every change — prompt, schema, model version — on it. Within a few months the evaluation set is a better description of your document estate than any specification, and it is the only thing that will tell you whether a model upgrade helped or quietly regressed the one field that matters.
EFTEDRA builds workflow automation on Claude. If you are putting document extraction into a regulated process and want the schema, validation and review design done properly, talk to us.