Design human review workflows and confidence calibration

Human review is a scarce resource, so the design question is how to spend it where the errors actually are — and how to know where that is.

Aggregate accuracy hides the failures that matter. A pipeline reporting 97% overall accuracy tells you almost nothing about risk, because the mean is dominated by the high-volume, easy segment. Underneath it a specific document type (handwritten forms, low-quality scans, a foreign-language variant) or a specific field (tax ID, line-item totals, dates in ambiguous formats) may be running far below the headline. Before reducing human review, analyze accuracy by document type and by field and confirm performance is consistent across all segments. Automating on the aggregate is the classic wrong answer: it automates precisely the segment where automation fails. This is measured, not hypothetical. Commercial gender classifiers with acceptable headline accuracy ran 0.8% error on lighter-skinned males and 34.7% on darker-skinned females.⁶⁷ Disaggregated reporting is a norm rather than an optional analysis, and a federal risk framework states it plainly: accuracy measurements may include disaggregation of results for different data segments.⁶⁸⁶⁹

Field-level confidence, calibrated against labeled data. Have the model output confidence per extracted field rather than one score per document. A document can have nine solid fields and one guess, so the useful routing decision is field-scoped. A verbalized score can be made to track probabilities, which is why eliciting one is worth doing at all.⁶¹ But raw scores are not trustworthy on their own. They must be calibrated using labeled validation sets until a threshold means something empirically. A calibration run might conclude, for instance, that "extractions above 0.92 have a measured 0.4% error rate on this document type". The threshold is an output of measurement, not a number someone chose. The named post-hoc method is temperature scaling, fitted against held-out labeled data.⁷² Building that labeled set is its own task, and it must cover the ambiguous cases where even humans struggle to reach consensus.⁶⁵

Route on two axes, not one. Confidence decides nothing by itself, because the score describes the model and the second axis describes the document:

Row 2 is the one exam items are built on. A high score means the model read the document consistently, not that the document says one thing: a page stating two different totals reaches a person whatever the score says. Prioritizing along both axes is what lets limited reviewer capacity cover the highest expected error. The deferral literature supports the general move: a pure confidence cutoff is not the right hand-off rule.⁷³ Read it honestly, though — its second axis is the human reviewer's competence, not document ambiguity. Ambiguity becomes checkable when each extracted value is bound to a verifiable span of the source document.⁶⁶

Keep measuring what you automated. High-confidence extractions that skip review still need stratified random sampling: sample across strata (document type, field, confidence band) rather than uniformly, so rare-but-risky segments are actually represented. Stratification is what lets you control precision within a stratum rather than only overall.⁷⁰⁷¹ This gives an ongoing error-rate estimate for the automated path — provided the per-stratum rates are recombined weighted by each stratum's share of that path. The sampler deliberately over-draws from small strata, so the raw pooled error rate across the sample is not the population error rate. It over-weights the rare segments it exists to surface. Detecting a novel pattern and estimating the overall rate want opposite allocations. The weighting is what lets one draw serve both. Sampling also detects novel error patterns that appear when document formats change upstream. Without sampling, a new failure mode is invisible until it reaches a customer, because everything the model was confident about was accepted unexamined.