Schema-Guided Decisions for Document Extraction

An insurance submission may describe a continuity procedure in an application form and provide evidence of testing in an attachment. To answer whether the procedure is documented and tested, an extractor has to establish that both statements concern the same procedure, site and policy period. If the attachment concerns another site, the question remains open. If testing goes unmentioned, that silence cannot establish that it never happened.

This is a simplified example of the interpretation required in the document extraction system I delivered for AllegoryOS, an insurtech company. Underwriting information arrives across forms, emails and supporting documents. The system must produce structured values while preserving enough evidence for an underwriter to check what each value means and where it came from.

An experimental revision of the pipeline uses TypeSafe AI’s Jev to make decisions over explicit answer choices, with a generative model supplying values that must first be found in the documents. The revision improved results on a development submission and approximately halved estimated provider costs. Its design also raises a broader question: how does the architecture of an intelligent application change when small semantic decisions become inexpensive?

Answer spaces

The earlier pipeline used optical character recognition (OCR) and layout extraction, followed by broad annotation with a large language model (LLM). Annotation proposed field values and their evidence; subsequent lookup and finalization tasks mapped and reconciled those proposals. The intermediate records made the work inspectable, but generation was involved even where the schema already specified every possible answer.

Jev’s Choice interface accepts context, a question and defined alternatives, and returns a selection with a distribution over those alternatives. TypeSafe describes its training objective as reinforcement learning for calibrated decisions. For this application, the useful properties were the explicit answer space, access to uncertainty signals and the ability to ask independent questions together over shared context.

A categorical field can therefore go directly to Jev with its complete choices. This removes a candidate-generation step and avoids making the available answers depend on what an annotation model happened to propose. Each choice carries its definition, including the conditions under which it applies.

A procedure name needs a different route. Jev identifies relevant pages, and a narrow LLM extraction task proposes names with references to their sources. Competing names remain available for resolution. Jev then selects among those candidates, with access to the full source corpus. The distinction makes failures easier to investigate: a missing candidate is an extraction problem; selecting the wrong candidate is a resolution problem. Improving the selector cannot recover a name that never entered its answer space.

This division of work suited the schema. Many outputs were categories or catalog entries, while the remaining fields required names, amounts or descriptions. Jev could handle the repeated selection tasks, and generation could concentrate on discovering the open values.

Schema compilation

An explicit answer space still leaves the problem of interpreting the requirement. A field description can inherit qualifications from its parent section, refer to a particular entity, and require several conditions to hold together. Those details must survive every intermediate representation.

The revised pipeline compiles the schema when it changes. A coding agent translates descriptions into questions, choice definitions, dependencies and routing instructions. The resulting plans preserve inherited scope and are versioned against the source schema. Structural validation checks field coverage, choices and the dependency graph; semantic review checks whether the questions still mean what the schema says. The plans are then reused across submissions.

For the illustrative continuity field, suppose the allowed outcomes are meets, not_met and unknown. Meeting the requirement needs evidence of both documentation and testing. An explicit statement that either condition fails supports not_met; insufficient evidence leaves unknown.

The compiled plan asks about documentation and testing separately, then asks a third question to apply the complete definition. Each receives the target site and period. The final question also receives the original corpus and treats the earlier answers as fallible observations.

The first two questions can share a request; the third waits for their answers. Jev performs the final semantic composition, including checking that the observations concern the same procedure. This matters if documentation is established for Site A and testing for Site B: two affirmative intermediate answers would conceal the mismatch if their scope were discarded.

Decomposition has costs. Each question can be wrong, and dependencies add sequential work. Keeping the original corpus available lets the final decision revisit the evidence, but also increases input and exposes it to potentially irrelevant material. Simple fields remain single questions. The purpose of the graph is to make a complicated requirement inspectable without losing the context needed to interpret it.

Evidence and acceptance

Source identity has to survive this process too. OCR produces text and layout regions with document, page and block identifiers. Candidates retain references to those records. After resolution, attribution tasks select supporting regions, and their stored coordinates provide highlights in the document viewer. A reviewer can inspect the surrounding text and check whether it applies to the intended entity and period.

Support and completeness require different checks. An address can contain only words present in the source and still omit a building name required by the schema. Attribution may correctly establish support for everything included while leaving the omission undetected. The evaluated revision has no separate completeness gate, which remains a limitation of its acceptance policy.

The policy combines model signals with structural, attribution and consistency checks. Jev’s returned confidence statistic describes the concentration of its distribution; its relationship to correctness still needs evaluation on the intended workload. Thresholds in this experiment are exploratory. A schema-level unknown also remains distinct from runtime abstention caused by unresolved conflict, unreadable evidence or an unsupported candidate set. Those outcomes tell the reviewer different things about what prevented a decision.

Development results

The evaluation used one development submission with 80 approved field labels kept outside inference. It compared the historical pipeline with the best completed Jev run and an unchanged repeat.

Here G is the earlier pipeline’s annotation cost and C its total priced provider cost; each row has its own baseline. Narrow generation cost about one-third as much as broad annotation, while total provider cost fell to roughly one-half. The comparison includes OCR and priced downstream model work, using recorded usage and the experiment’s rates; it excludes infrastructure, embeddings and unrecorded historical failures. Several components changed together, so the comparison cannot isolate the contribution of Jev, decomposition or source selection.

The accuracy scores describe proposed field values. Results from one development submission, whose fields share evidence, are insufficient to establish production reliability.

The earlier pipeline took 389 seconds end to end. The revised estimate is 165–199 seconds, assembled from measured model stages and preparation and persistence assumptions. A fresh paired end-to-end run is still needed. Broader evaluation should also measure completeness and the amount of review required on new submissions.

Application economics

The cost reduction suggests a useful direction for applications with many semantic decisions. A schema can become a reusable program of questions, with its authoring and review costs spread across submissions. Generation remains available where the answer space must be discovered. Independent checks can share evidence in a request, reducing repeated input. Together, these choices can make detailed interpretation affordable at more points in a workflow.

There is evidence of this beyond insurance. A recent crash-narrative study reports coding 195,857 records with a 27-question Jev schema and auditing the results against human judgments. The authors measured processing costs and recalibrated probabilities. Their results concern short narratives and one schema, but the scale illustrates how decision models can make a substantial body of text practical to analyze.

Low decision cost also makes routing attractive, although the policy needs its own evaluation. In Entagl’s replay of customer conversations, confidence-based fallback improved some tasks and worsened one: the fallback performed poorly on the subset sent to it. An application needs to measure what additional inference buys on those cases. In the extraction revision described here, escalation is recorded for review; there is no generative fallback.

Decision models seem likely to expand the set of requirements that applications can check routinely. A system might examine whether evidence concerns the right entity, whether a candidate satisfies a definition, or whether two records warrant reconciliation, wherever that judgment earns its cost. This is particularly useful when requirements change often enough that maintaining a separately trained classifier for each task would be burdensome. Stable tasks with substantial labeled data may still favor specialized classifiers; comparative benchmarks give examples of that tradeoff.

The full cost includes defining the questions, evaluating each revision, processing evidence, resolving exceptions and correcting errors. Additional checks are worthwhile when they reduce mistakes or review effort enough to pay for themselves. For this extraction system, the next useful measurement is how much underwriter work remains at a specified error tolerance across representative submissions.