Domain 4 — Prompt Engineering & Structured Output

Explicit criteria are a decision rule; vague instructions are a wish

The exam's canonical contrast is "check that comments are accurate" versus "flag comments only when claimed behavior contradicts actual code behavior". The second one is not merely more words — it is a decidable predicate. The model can apply it to a comment/code pair and get the same answer twice. The first one forces the model to invent its own bar for "accurate", and it will invent a slightly different bar on every file, every run, and every PR. Inconsistent classification is the direct consequence of a criterion the model has to define for itself.

Why "be conservative" does not improve precision

Instructions like "be conservative" or "only report high-confidence findings" are meta-instructions about the model's internal certainty. They do not change what counts as a finding; they only ask the model to feel differently about findings it was already going to produce. There is no calibrated confidence threshold behind such a request, so the observable effect is usually tone and hedging rather than a moved decision boundary. The lever that actually moves precision is categorical: name the issue classes that are in scope (bugs, security vulnerabilities, correctness defects) and the classes that are explicitly out of scope (minor style, formatting, project-local patterns the team has deliberately adopted).

False positives are contagious

The reason precision matters more than raw recall in a CI review bot is a trust effect: a category with a high false positive rate undermines developer confidence in the categories that are accurate. Once a reviewer has dismissed eight of ten "possible null dereference" comments, they stop reading the security comments too. Precision is therefore a per-category property with a global blast radius.

That gives you two distinct engineering moves:

• Temporarily disable the offending category while you improve its prompt. You trade coverage you were not getting value from anyway for restored trust in the remaining categories. This is a legitimate, exam-relevant answer — not a cop-out.
• Define explicit severity criteria with a concrete code example for each level. Naming levels "critical / major / minor" is another vague criterion. Anchoring each level to a short code sample ("critical: an unchecked index into a caller-supplied array") makes classification consistent across runs and reviewers.

Reporting criteria (report vs skip) and severity criteria (how bad) are separate axes. Both need to be explicit, and neither should be delegated to the model's sense of confidence.

Few-shot is the fix when instructions alone are inconsistent

The guide is unusually direct here: few-shot examples are described as the most effective technique for getting consistently formatted, actionable output when detailed instructions alone produce inconsistent results. The mechanism is that an example demonstrates the target simultaneously along every axis you care about — field set, field order, granularity, tone, length, and what a "good" value looks like — whereas prose has to describe each axis separately and each description is itself open to interpretation. If your first instinct on inconsistent output is "write longer instructions", the exam expects the second instinct: show two to four examples.

Examples earn their place at the ambiguous boundary

The target count is small — 2 to 4 targeted examples — which means selection matters more than volume. Keep this count attached to its scope: 2–4 is the few-shot prompting number, for examples that teach ambiguous-case judgment inside a prompt. The sibling number you will also see on the exam is 2–3, which belongs to iterative refinement (3.5): concrete input/output pairs supplied to pin down a transformation the model keeps interpreting differently. Same idea, different statement and different job, so read which one the stem is describing before you pick a count. The examples that pay for themselves are the ambiguous ones: which tool to select for a request that plausibly maps to two tools; whether a partially covered branch counts as a test coverage gap; whether an unusual-looking code pattern is a genuine defect or an accepted project convention. Crucially, an ambiguous-case example should include the reasoning for why one action was chosen over a plausible alternative. That reasoning is the part that transfers: it teaches the model the principle behind the boundary rather than a single mapping, which is why well-chosen few-shot examples let the model generalize judgment to novel patterns instead of only matching the cases you pre-specified.

Format pinning and false positive reduction

Two concrete jobs show up repeatedly:

• Format consistency. Demonstrate the exact output shape you want — location, issue, severity, suggested fix — in the example, rather than describing those fields in prose.
• False positive reduction with generalization. Include an example of an acceptable pattern that must not be flagged alongside a superficially similar genuine issue. This teaches the exclusion boundary. Without a negative example the model can only guess where the line sits, and the guide notes that this pairing reduces false positives while still allowing generalization — you are not enumerating an allowlist.

Extraction: examples reduce hallucination

In extraction tasks, few-shot examples are the stated remedy for hallucination and for empty or null values in required fields. Documents differ structurally: citations may be inline or collected in a bibliography; methodology may have its own section or be buried in a results paragraph; measurements may be informal ("about a tablespoon"). An example per structural variant shows the model where to look and how to normalize, instead of letting it either invent a plausible value or give up and return nothing. Pair this with a nullable schema (4.3) so "genuinely absent" has a legal representation.

Tool use is the schema enforcement mechanism

The reliable way to get structured output is not to ask for JSON in the prompt. Instead you define a tool whose input_schema is your output schema, let the model call it, and read the structured data out of the tool_use block. Because the model emits tool input rather than free text, you get schema-compliant output and JSON syntax errors are eliminated: no markdown fences, no preamble, no trailing comma to repair. The extraction "tool" usually executes nothing on your side; its schema exists purely to constrain the shape of the answer.

This is the API-layer mechanism. When the surface is Claude Code in CI rather than a direct API call, the equivalent controls are --output-format json plus --json-schema (3.6). Same goal of a schema-enforced machine-readable result, expressed as CLI flags instead of a tool definition.

The three tool_choice modes

The distinction the exam tests:

Strict schemas stop syntax errors, not semantic ones

This is the highest-value nuance in the statement. JSON Schema's strict mode is the feature name to know here: it makes the provider enforce the declared shape rather than treat it as a hint. The guide lists it under JSON Schema as strict mode for syntax error elimination — syntax errors, and nothing more. A strict schema guarantees the shape: required keys present, values of the right type, enum values legal. It guarantees nothing about meaning. Line items can still fail to sum to the stated total; a vendor tax ID can land in the invoice-number field; a date can be wrong in perfect ISO 8601. Schema compliance is not correctness, and the remedy is semantic validation (4.4), not a stricter schema.

Schema design that does not induce fabrication

Two design decisions carry most of the weight:

• Required vs optional. If a source document may legitimately not contain a field, model it as optional or nullable. A field marked required forces the model to produce something, and what it produces is a fabricated value. Making absence representable is what converts a hallucination into an honest null.
• Extensible enums. Closed enums break on real-world data. Add an "unclear" value for genuinely ambiguous cases, and an "other" value paired with a detail string so an unanticipated category can be captured rather than forced into the nearest wrong bucket.

Finally, a schema constrains structure but says nothing about formatting. Source documents write dates, currencies and units inconsistently. Put format normalization rules in the prompt alongside the strict schema: "dates as YYYY-MM-DD; amounts as decimals with no currency symbol; for a range, record the lower bound and note it".

Retry only works if the model learns what failed

A retry that re-sends the original prompt unchanged is a re-roll: same inputs, same distribution, no reason to expect a different outcome. Retry-with-error-feedback means the follow-up request carries three things:

1. the original document,
2. the failed extraction exactly as the model produced it, and
3. the specific validation errors — not "invalid output" but "line_items sum to 1240.00 but stated_total is 1420.00" and "vendor_tax_id contains what appears to be an invoice number".

With those three, the second attempt is a self-correction task on a concrete defect rather than a fresh guess.

Know which failures a retry can fix

Classifying the failure comes before choosing the response, and the guide treats that classification as its own skill.

No amount of re-asking creates data the model never saw. Retrying an absent-information failure just multiplies cost and, worse, pressures the model toward fabrication.

Semantic vs syntax errors

The semantic row is the one your validator exists for: values that do not reconcile, values in the wrong field, internally contradictory records. Because the schema cannot catch these, your validator is the only thing standing between a plausible-looking record and a corrupted downstream system.

Pydantic is the guide's named example of the validator in this loop. You declare the record as a Pydantic model and it does the schema validation, while your own field and cross-field checks raise the semantic validation errors. The guide's phrasing is "when Pydantic or JSON schema validation fails, send a follow-up request". The same validate → error → retry loop is what you build regardless of which of the two raised the failure. Keep the two levels distinct: schema-valid means the shape parses and the types match; semantically valid means the values are true of the source document.

Design the schema to help you validate

The strongest pattern is to make the model surface the evidence for its own consistency check:

• Extract calculated_total (sum the line items yourself, in the extraction) alongside stated_total (what the document prints). A mismatch is then a single comparison rather than a re-derivation.
• Add a conflict_detected boolean (with a note) so the model can report that the source is internally inconsistent, instead of silently picking one of two contradictory values.

Feedback loops beyond a single record

The same principle applies to review findings. Adding a detected_pattern field — the code construct that triggered the finding — turns individual dismissals into analyzable data. When developers dismiss findings, you can group by detected_pattern and see that, say, 90% of dismissals come from one construct, which points straight at the criteria to rewrite (4.1) or the negative few-shot example to add (4.2). Without that field you have a dismissal rate and no idea what to fix.

What the Message Batches API buys and what it costs

The trade is explicit, and it runs on more axes than cost.

Most batches finish far sooner than the window, but you cannot build a promise on that. The only number you may plan against is 24 hours, which is why a pre-merge check that gates a developer's merge cannot sit in that queue.

The capability limit deserves reading precisely, because the exam tests its exact scope. The guide excludes the multi-turn case, so an agentic loop that needs to call a tool, read the result and continue belongs on the synchronous API. The wording implies that a one-shot request returning a tool_use block — the structured-extraction pattern from 4.3 — is not what the restriction is about, since no tool is executed mid-request. But the guide does not state that capability, so treat it as a reading of "multi-turn" rather than a fact to key an answer on. What is testable is the exclusion itself.

custom_id is the join key, and results are polled

Each request carries a custom_id and every result comes back with it. Results are not guaranteed to arrive in submission order, so custom_id — not array position — is how you correlate response to request, and how you identify precisely which documents failed.

Retrieval is a pull, not a push: you check the batch's status until it is done, then read the results. Nothing wakes your pipeline up when the work finishes, which is a second reason batch cannot sit under a blocking gate. Adding status polling does not rescue such a workflow either, because progress reporting shortens nothing.

Sizing submission cadence against an SLA

The guide's own worked example: you promise a 30-hour SLA and batch processing may take up to 24 hours. That leaves 6 hours of slack, which is the maximum time a document may wait in your queue before submission. Submitting every 4 hours therefore keeps the worst case at 4 + 24 = 28 hours, inside the promise with margin. The general recipe: maximum queue window = promised SLA − 24 hours, then pick a submission interval at or below that.

Handling failures and de-risking the first pass

When a batch ends with partial failures, resubmit only the failed custom_ids, with a modification appropriate to the cause — chunk the documents that exceeded context limits, fix the ones that hit a validation problem. Resubmitting the whole batch pays a second time for work that already succeeded.

Because a wrong prompt is amplified by volume, refine the prompt on a small sample set before batch-processing large volumes. Every first-pass failure becomes an iterative resubmission cost, so a sample run that lifts the first-pass success rate is usually the cheapest step in the whole pipeline.

Why a model is a poor reviewer of its own work

When a model generates code and is then asked, in the same session, to review it, it still holds the reasoning that produced that code. Every design decision already looks justified, because the justification is right there in context. The result is that self-review is less likely to question its own decisions — it tends to confirm rather than challenge. This is a context property, not a capability shortfall. That is why the guide is explicit that independent review instances, running without the generator's prior reasoning context, catch subtle issues more effectively than either a self-review instruction ("now critically review your work") or extended thinking. Adding more reasoning inside the same context does not remove the bias built into that context.

Practically: for generated code, spin up a second Claude instance that receives the code and the requirements but not the generation transcript. It has to reconstruct intent from the artifact, which is exactly what a human reviewer does and exactly what surfaces the assumption the generator never questioned.

Multi-pass review: the pass taxonomy and the merge rule

Why a single giant pass degrades is covered in full in 1.6 — attention dilution, contradictory findings inside one PR, and why a larger context window is not the remedy. Here the subject is the shape of the decomposition and what becomes of the findings afterwards.

The taxonomy has exactly two pass types, each defined by the question it can answer.

Because the two types answer different questions they get different prompts, and they produce two finding sets that must be merged rather than concatenated. The merge rule falls straight out of the taxonomy:

1. Dedupe on (location, detected_pattern).
2. Resolve each collision in favor of the pass that could see the answer, as the last column above records.
3. Pass through untouched any finding that only one pass type could have produced.

Verification passes with self-reported confidence

The third technique is a verification pass in which the model self-reports confidence alongside each finding, so findings can be routed by calibration. High-confidence findings post automatically as PR comments, low-confidence findings go to a human triage queue, and the rest are dropped or batched. Note the careful distinction against 4.1 — confidence is used downstream, as routing metadata on an already-generated finding, not upstream as the criterion that decides whether the model should report at all. Asking the model to filter by confidence during finding does not improve precision; asking it to label findings with confidence so your pipeline can route them does add value.

Two preconditions bound that value, and both live in Domain 5:

• The thresholds must be calibrated. A self-reported "high" is not a probability until you have measured what it means. The procedure is 5.5's: have the model emit field-level or finding-level confidence, then set the routing thresholds against a labeled validation set, so "auto-post above this level" reflects a measured accuracy rather than an assumption. An uncalibrated threshold is 4.1's anti-pattern wearing a number.
• Routing is not escalation. Confidence is never the trigger for handing a live conversation to a human (5.2). Official Sample Question 3 rejects precisely that option: the agent is already incorrectly confident on the hard cases, so a confidence threshold escalates the easy ones and keeps the hard ones. Explicit escalation criteria with few-shot examples are the credited fix there. Routing works here because a finding already exists and is being sorted offline against calibrated data; escalation fails because the decision must be made in-flight by the same miscalibrated model.