Implement validation, retry, and feedback loops for extraction quality

Retry only works if the model learns what failed

A retry that re-sends the original prompt unchanged is a re-roll: same inputs, same distribution, no reason to expect a different outcome. Retry-with-error-feedback means the follow-up request carries three things:²¹

1. the original document,
2. the failed extraction exactly as the model produced it, and
3. the specific validation errors — not "invalid output" but "line_items sum to 1240.00 but stated_total is 1420.00" and "vendor_tax_id contains what appears to be an invoice number".

With those three, the second attempt is a self-correction task on a concrete defect rather than a fresh guess.³

Know which failures a retry can fix

Classifying the failure comes before choosing the response, and the guide treats that classification as its own skill.

Failure kind	What it looks like	Retry with error feedback?	What to do
Syntax	a markdown fence, a trailing comma, an unquoted key	not applicable	already eliminated upstream by tool use with a strict JSON schema (4.3)
Format or structural	a date in the wrong notation, a nested object flattened, a normalization rule ignored	yes — this is the class retry exists for	resend the three items above, on a bounded loop
Semantic	line items that do not sum to the stated total, a vendor tax ID sitting in the invoice-number field	yes	same loop, driven by your validator's errors
Information absent from the source	a figure that lives only in a referenced external appendix you never supplied	no	escalate, or fetch the missing source and re-extract

No amount of re-asking creates data the model never saw. Retrying an absent-information failure just multiplies cost and, worse, pressures the model toward fabrication.

Semantic vs syntax errors

The semantic row is the one your validator exists for: values that do not reconcile, values in the wrong field, internally contradictory records. Because the schema cannot catch these, your validator is the only thing standing between a plausible-looking record and a corrupted downstream system.

Pydantic is the guide's named example of the validator in this loop. You declare the record as a Pydantic model and it does the schema validation, while your own field and cross-field checks raise the semantic validation errors. The guide's phrasing is "when Pydantic or JSON schema validation fails, send a follow-up request".⁴⁷ The same validate → error → retry loop is what you build regardless of which of the two raised the failure. Keep the two levels distinct: schema-valid means the shape parses and the types match; semantically valid means the values are true of the source document.

Design the schema to help you validate

The strongest pattern is to make the model surface the evidence for its own consistency check:

• Extract calculated_total (sum the line items yourself, in the extraction) alongside stated_total (what the document prints). A mismatch is then a single comparison rather than a re-derivation.
• Add a conflict_detected boolean (with a note) so the model can report that the source is internally inconsistent, instead of silently picking one of two contradictory values.

Feedback loops beyond a single record

The same principle applies to review findings. Adding a detected_pattern field — the code construct that triggered the finding — turns individual dismissals into analyzable data. When developers dismiss findings, you can group by detected_pattern and see that, say, 90% of dismissals come from one construct, which points straight at the criteria to rewrite (4.1) or the negative few-shot example to add (4.2). Without that field you have a dismissal rate and no idea what to fix.

let extraction = await extract(document);            // tool_use, schema-compliant
let errors = validateSemantics(extraction);          // sums, field placement, conflicts

for (let attempt = 1; attempt <= 2 && errors.length > 0; attempt++) {
  // The follow-up request includes the document, the failed extraction,
  // and the specific errors -- this is what makes it a correction, not a re-roll.
  extraction = await extract(document, {
    failedExtraction: extraction,
    validationErrors: errors,
    // e.g. "line_items sum to 1240.00 but stated_total is 1420.00"
    //      "vendor_tax_id looks like an invoice number, not a tax id"
  });
  errors = validateSemantics(extraction);
}

if (errors.length > 0) {
  // Retrying cannot invent data the document never contained.
  return errors.every(isInformationAbsentFromSource)
    ? escalateToHuman(document, extraction, errors)
    : quarantine(document, extraction, errors);
}
return extraction;

{
  "stated_total": {
    "type": ["number", "null"],
    "description": "The total as printed on the document, verbatim"
  },
  "calculated_total": {
    "type": "number",
    "description": "The sum of line_items[].amount that you extracted"
  },
  "conflict_detected": {
    "type": "boolean",
    "description": "true when the source document itself is internally inconsistent"
  },
  "conflict_note": {
    "type": ["string", "null"],
    "description": "Required when conflict_detected is true: which values disagree"
  }
}

{
  "location": "src/billing/invoice.ts:88",
  "category": "correctness",
  "severity": "HIGH",
  "issue": "total computed before discount is applied",
  "suggested_fix": "compute subtotal, apply discount, then sum tax",
  "detected_pattern": "reduce_over_items_without_discount_call",
  "dismissed": true,
  "dismissal_reason": "discount applied by caller in applyPromotion()"
}