Domain 5 — Context Management & Reliability
Manage conversation context to preserve critical information across long interactions
What you need to know
- Pin the numbers outside the summary: a case-facts block, re-sent verbatim, one structured layer per issue.
- The API stores nothing. Coherence is the full history you resend, so whatever you drop is unrecoverable.
- Beginning and end are read; the middle is dropped. Summary first, detail below under headers.
- Trim at the tool boundary, not afterwards — 40 fields where 5 matter ride along on every future request.
- Upstream agents ship facts, citations and scores; a small downstream budget cannot afford narrative.
Show that the prompt sent on every request is deliberately assembled from distinct layers in a fixed order — pinned case facts first, summarized older turns in the middle where the lost-in-the-middle effect bites, the last N turns verbatim at the end — and that pinned case facts bypass compaction entirely while older turns and tool results are lossy on purpose.
Click a layer to see whether it is lossless or lossy on the next request.
Make clear that the cure for the "lost in the middle" effect is input layout — summary first, addressable sections below — rather than a stronger instruction to the model.
Worked examples
A case-facts block that survives compaction
Scenario 1 · Customer Support Resolution AgentA billing-dispute conversation runs 40 turns and the oldest turns get summarized. The disputed amount, the charge date and the promise the agent already made must not be inside that summary. The support agent extracts them into a structured case-facts object after every tool call and re-injects it, verbatim and first, on each request. The compacted history stays useful for tone and context; the facts block stays authoritative for arithmetic and policy.
typescript Facts are pinned; only the older history is compacted.
type CaseFacts = {
orderId: string;
orderTotal: string; // "USD 847.32" — never "about 850 dollars"
chargeDate: string; // "2026-03-14"
refundStatus: string; // "requested | approved | processed"
agentCommitments: string[]; // "refund within 5 business days"
};
// Rebuilt on every API request: facts pinned, older turns compacted.
function buildMessages(facts: CaseFacts, olderSummary: string, recentTurns: Message[]) {
const pinned = [
'## CASE FACTS (authoritative, never summarized)',
JSON.stringify(facts, null, 2),
'## EARLIER CONVERSATION (compacted — may be imprecise)',
olderSummary,
].join('\n\n');
return [{ role: 'user' as const, content: pinned }, ...recentTurns];
}Trimming lookup_order before it enters the transcript
Scenario 1 · Customer Support Resolution Agentlookup_order returns 40+ fields (warehouse routing, carrier scan events, marketing attribution). For a return decision only a handful are relevant. Trimming happens in a PostToolUse hook — the interception point for transforming a tool result before the model processes it (see 1.5) — so the full payload never reaches the conversation. The alternative is paying for those fields on every subsequent request for the rest of the session, and burying the fields that matter among ones that do not.
typescript Field allow-list applied in a PostToolUse hook.
const RETURN_RELEVANT = [
'order_id',
'order_date',
'order_total',
'currency',
'status',
'items',
'return_window_ends',
] as const;
/** The upstream response has 40+ fields; only these enter context. */
export function trimOrder(raw: Record<string, unknown>) {
return Object.fromEntries(
RETURN_RELEVANT.filter((k) => k in raw).map((k) => [k, raw[k]]),
);
}Laying out an aggregated research brief
Scenario 3 · Multi-Agent Research SystemThe coordinator receives four subagent reports and must synthesize them. Concatenating them in arrival order buries report 2 and 3 in the middle of a very long input. Instead the coordinator writes a key-findings block first, then the details under explicit headers, and requires each subagent to have supplied dates, source locations and methodological context so the synthesis step does not have to guess.
text Aggregated-input template that mitigates position effects.
# KEY FINDINGS (read first — one line per subagent)
1. Web search: adoption rose 34% in 2025 (3 sources, most recent 2026-01).
2. Document analysis: internal figures disagree with public ones (see S2).
3. Filings review: no disclosure found for FY2025 (valid empty result).
4. Competitor scan: two of five competitors ship the feature.
# SECTION 1 — Web search findings
source | date collected | claim | relevance
...
# SECTION 2 — Document analysis findings
document | publication date | excerpt | claim
...
# OPEN QUESTIONS AND COVERAGE GAPS
- FY2025 filings unavailable; conclusions about FY2025 are unsupported.Anti-patterns
- Truncating or summarizing the oldest turns blindly instead of first extracting transactional facts and prior commitments into a pinned layer because compaction destroys precisely the numbers, dates and expectations the agent must honor.
- Concatenating subagent reports in arrival order instead of leading with a key-findings summary and sectioning the detail because middle-of-input findings are the ones the model omits.
- Piping raw tool responses into the transcript instead of trimming them to the relevant fields because irrelevant fields are re-sent on every later request and crowd out the relevant ones.
- Having upstream agents forward verbose content and reasoning chains instead of structured facts, citations and metadata because a downstream agent with a small context budget cannot afford narrative it did not ask for.
How it is examined
- Stems describe a long support or research session where the agent "forgot" an amount, a date or a promise. The tempting-but-wrong option is a bigger context window or a better summarization prompt; the credited answer extracts the facts into a persistent layer outside the summarized history.
- When a stem mentions that findings from the middle of a long aggregated input were missing from the final answer, the mechanism being tested is "lost in the middle" — pick reordering plus explicit section headers, not a re-read instruction.
- Watch for token-cost stems built on a tool returning dozens of fields. The credited answer trims at the tool boundary; distractors trim after the fact, or compact the whole conversation, which also loses the case facts.
Why this is true
Long sessions fail in predictable ways, and the exam expects the mechanism behind each failure rather than the vocabulary.
Summarization loses precision first. Compression favors narrative over data. "USD 847.32 charged on March 14 for order A-99231" becomes "a billing issue from last month"; "a refund within 5 business days" becomes "the customer expects resolution soon". Numerical values, percentages, dates and customer-stated expectations are what a summarizer treats as noise, and what the agent needs to act correctly. The fix is not a better summarizer. It is to keep transactional facts outside the summarized region, in a case facts block re-included in every prompt. A session covering several complaints gets one structured issue layer each, so a second problem cannot blur into the first.
Statelessness is why that works. The Messages API keeps no memory of its own. Conversational coherence exists only because you pass the complete conversation history in each subsequent request. Compaction and trimming are deliberate edits to that resent payload, so whatever you drop is genuinely gone.
Position is a design choice. Models process the beginning and the end of a long input reliably and may simply omit findings sitting in middle sections — the "lost in the middle" effect. Layout is the mitigation, not exhortation. Assemble each request in layers, on purpose:
| Layer | Position | Lossy? | Why there |
|---|---|---|---|
| Case facts, or a key-findings summary of many subagent reports | First | Never summarized | Primacy; carries what every later decision depends on |
| Summarized older turns | Middle | Yes, by design | Fine for tone and context, never authoritative for numbers |
| Detailed results under explicit section headers | Middle | No | Headers keep each block addressable despite the position |
| Trimmed tool results | With their turns | Trimmed on purpose | Relevance, not completeness, is what earns a place in context |
| Last N turns verbatim | Last | Never | Strongest recency; the live state of the conversation |
Tool results dominate context. An order lookup may return 40+ fields when only 5 matter for a return decision, and each one is re-sent on every later request. Trim at the tool boundary: a PostToolUse hook is the named interception point that transforms a tool result before the model ever processes it (see 1.5). Apply the same discipline agent to agent. When the downstream context budget is small, require upstream agents to emit structured data (key facts, citations, relevance scores) plus metadata (dates, source locations, methodological context) instead of verbose content and reasoning chains.
Exam guide, verbatim — what is measured
Knowledge of
- Progressive summarization risks: condensing numerical values, percentages, dates, and customer- stated expectations into vague summaries
- The "lost in the middle" effect: models reliably process information at the beginning and end of long inputs but may omit findings from middle sections
- How tool results accumulate in context and consume tokens disproportionately to their relevance (e.g., 40+ fields per order lookup when only 5 are relevant)
- The importance of passing complete conversation history in subsequent API requests to maintain conversational coherence
Skills in
- Extracting transactional facts (amounts, dates, order numbers, statuses) into a persistent "case facts" block included in each prompt, outside summarized history
- Extracting and persisting structured issue data (order IDs, amounts, statuses) into a separate context layer for multi-issue sessions
- Trimming verbose tool outputs to only relevant fields before they accumulate in context (e.g., keeping only return-relevant fields from order lookups)
- Placing key findings summaries at the beginning of aggregated inputs and organizing detailed results with explicit section headers to mitigate position effects
- Requiring subagents to include metadata (dates, source locations, methodological context) in structured outputs to support accurate downstream synthesis
- Modifying upstream agents to return structured data (key facts, citations, relevance scores) instead of verbose content and reasoning chains when downstream agents have limited context budgets
Design effective escalation and ambiguity resolution patterns
What you need to know
- Valid triggers: an explicit request for a human, a policy exception or policy gap, and inability to make meaningful progress — not "the case looks complex".
- An explicit demand for a human is honored immediately, with no investigation attempt first.
- Frustration without an explicit request: acknowledge it, offer to resolve if the issue is in scope, and escalate only if the customer reiterates their preference.
- Sentiment detection and self-reported confidence scores are unreliable as a proxy for case complexity, or as a substitute for the explicit escalation triggers — unlike calibrated, field-level confidence, which legitimately prioritizes offline human review (5.5) or routes an already-generated finding (4.6).
- Multiple tool matches mean ask for another identifier — never pick by heuristic.
- Escalation criteria in the system prompt need few-shot examples of both escalating and resolving to be usable.
Fix the order in which triggers are evaluated: an explicit human request short-circuits everything, policy gaps escalate, multiple matches trigger clarification, and frustration alone leads to an offer to resolve.
Click a decision node to highlight its two outgoing branches and the cue on each label.
Worked examples
System-prompt escalation criteria with both branches
Scenario 1 · Customer Support Resolution AgentThe criteria list makes the triggers explicit, and the few-shot pairs demonstrate the boundary in both directions. Note that the "resolve" example is as important as the "escalate" one: without it the agent over-escalates and first-contact resolution collapses well below the 80% target.
text Excerpt from the support agent system prompt.
ESCALATE with escalate_to_human when ANY of these hold:
1. The customer explicitly asks to speak with a person.
-> Escalate immediately. Do NOT investigate first.
2. Policy does not cover the request, or an exception is required.
3. You cannot make meaningful progress after your available tools.
Do NOT escalate merely because the case is complex or the customer is upset.
Example A — escalate immediately
Customer: "Stop with the bot answers, transfer me to a human."
Action: escalate_to_human(reason="explicit_human_request")
Example B — resolve, do not escalate
Customer: "This is absurd, the jacket arrived torn and I want it gone."
Action: acknowledge the frustration, confirm the order is inside the
return window, and offer the return now. Escalate only if the customer
then repeats that they want a person.A policy gap, not a hard case
Scenario 1 · Customer Support Resolution AgentA customer asks for a price match against a competitor's listing. The refund policy documents price adjustments for purchases on your own site within 14 days and says nothing about competitors. The agent has enough information to answer — and that is the trap. Because the policy is silent, any autonomous answer either invents an entitlement or denies one that may exist. The correct behavior is to escalate as a policy gap, stating what the policy does cover so the human starts informed.
json Escalation call that carries the gap, not just a flag.
{
"tool": "escalate_to_human",
"input": {
"reason": "policy_gap",
"customer_request": "price match against competitor listing at USD 79.00",
"policy_coverage": "own-site price adjustments within 14 days only",
"policy_silent_on": "competitor price matching",
"case_facts": {
"order_id": "A-99231",
"order_total": "USD 129.00",
"order_date": "2026-07-22"
},
"agent_recommendation": "no autonomous decision made"
}
}Multiple customer matches: ask, do not disambiguate
Scenario 1 · Customer Support Resolution Agentget_customer on "J. Alvarez" returns three records. A heuristic such as "use the account with the most recent order" is wrong often enough to cause refunds against the wrong account and disclosure of one customer's order history to another. The instruction is to surface the ambiguity to the customer and request an additional identifier that is safe to ask for.
text Instruction pattern for multi-match tool results.
When get_customer returns more than one match:
- Do NOT select a record using recency, name similarity, or order volume.
- Ask the customer for ONE additional identifier: order number,
billing postal code, or the last four digits of the payment card.
- Re-run get_customer with the identifier before any account action.
- If still ambiguous after one clarification round, escalate with the
number of candidate matches and the identifiers already tried.Anti-patterns
- Investigating first when a customer has explicitly asked for a human instead of escalating immediately because continuing to work the case after a direct request reads as refusing the request.
- Using detected sentiment or a self-reported confidence threshold as the live escalation trigger instead of the explicit criteria because tone and uncalibrated introspection do not track actual case complexity — note this is narrower than "never use confidence": calibrated field-level confidence still prioritizes offline review (5.5).
- Selecting among multiple customer matches with a heuristic instead of asking for an additional identifier because acting on the wrong record causes wrong refunds and cross-customer data exposure.
- Answering a request the policy is silent about instead of escalating the policy gap because an invented entitlement (or an invented denial) is a decision the agent has no basis to make.
How it is examined
- Expect stems that quote a customer utterance. Look for an explicit request for a human: if present, the credited answer escalates immediately and every "investigate first" option is wrong.
- When a stem offers a sentiment classifier or a "confidence below 0.7" threshold as the escalation mechanism, that is the designed distractor — Sample Question 3 credits explicit criteria with few-shot examples instead because the agent is already incorrectly confident on the hard cases. Read the option carefully though: a calibrated field-level score prioritizing offline review (5.5) or routing an existing finding (4.6) is credited, not penalized.
- Policy-gap stems are worded so the agent could plausibly answer (e.g. competitor price matching). The credited answer escalates because the policy is silent, not because the case is hard.
Why this is true
Escalation is a design problem, not a fallback. The exam tests whether you can state which signals justify handing off to a human and which ones only look like they do.
The signals, evaluated in this order. The order carries as much weight as the list. An explicit request for a person short-circuits every check below it, and no lower signal overrides one above.
| Check, in this order | What fires it | What the agent does | Why not the alternative |
|---|---|---|---|
| 1 | The customer explicitly asks for a person | Call escalate_to_human now, with no investigation first |
Working the case after a direct request reads as refusing it |
| 2 | A policy exception, or the policy is silent on the request | Escalate as a policy gap, stating what the policy does cover | Any autonomous answer invents an entitlement, or invents a denial |
| 3 | No meaningful progress left with the available tools | Escalate, carrying what was already attempted | Another lap through the same tools returns the same non-answer |
| 4 | get_customer returns multiple matches |
Ask for one more identifier — order number, postal code, last four card digits — then re-run | Picking by "most recent activity" or "closest name" acts on the wrong person |
| 5 | Frustration, but no explicit request for a person | Acknowledge it, then offer to resolve now if the issue is in scope | Escalating on tone abandons a case the agent could have closed |
| Never | The case looks complex, or a self-reported score is low | Keep working | Complexity is what the agent is for |
Row 2 is the one to read twice. The guide's example is competitor price matching when the policy only addresses adjustments on your own site. The policy is silent, so no autonomous answer is correct and inventing one is worse than escalating. Row 5 escalates only if the customer then reiterates a preference for a person: the distinguishing question is whether an explicit request was made, not how negative the message sounds.
Why the two tempting proxies fail. Sentiment-based escalation confuses tone with difficulty: angry customers frequently have trivially solvable problems, and calm customers sometimes have unsolvable ones. Self-reported confidence scores are uncalibrated introspection. The model has no grounded basis for "I am 0.7 confident", and it produces numbers that do not track actual error rates. Official Sample Question 3 makes the failure precise. An agent that escalates straightforward damage replacements while attempting policy-exception cases autonomously is already incorrectly confident on the hard cases. A "confidence below threshold" router therefore escalates the easy tickets and keeps the hard ones. Both proxies waste human capacity and miss real escalations.
Scope this claim carefully, because it is the axis the exam disambiguates. Self-reported confidence is rejected here as a proxy for case complexity, and as a substitute for the explicit escalation triggers — not as a signal in general. Field-level confidence that has been calibrated against labeled validation sets is legitimate elsewhere: it prioritizes offline human review of extractions (5.5), and it can route a finding that has already been generated (4.6). What it cannot do is decide, live in a conversation, whether this case needs a human. And it must never act as an upstream filter on what counts as a finding at all (4.1).
Why row 4 asks instead of picking. Acting on the wrong customer record is a data-exposure and wrong-refund incident, not a UX inconvenience. A disambiguation heuristic is right often enough to feel safe, and wrong often enough to refund one customer while disclosing another customer's order history.
How you implement it. Put explicit escalation criteria in the system prompt and pair them with few-shot examples that demonstrate both branches — escalate here, resolve autonomously there. Criteria alone under-specify the boundary; the worked examples are what make the boundary learnable.
Exam guide, verbatim — what is measured
Knowledge of
- Appropriate escalation triggers: customer requests for a human, policy exceptions/gaps (not just complex cases), and inability to make meaningful progress
- The distinction between escalating immediately when a customer explicitly demands it versus offering to resolve when the issue is straightforward
- Why sentiment-based escalation and self-reported confidence scores are unreliable proxies for actual case complexity
- How multiple customer matches require clarification (requesting additional identifiers) rather than heuristic selection
Skills in
- Adding explicit escalation criteria with few-shot examples to the system prompt demonstrating when to escalate versus resolve autonomously
- Honoring explicit customer requests for human agents immediately without first attempting investigation
- Acknowledging frustration while offering resolution when the issue is within the agent's capability, escalating only if the customer reiterates their preference
- Escalating when policy is ambiguous or silent on the customer's specific request (e.g., competitor price matching when policy only addresses own-site adjustments)
- Instructing the agent to ask for additional identifiers when tool results return multiple matches, rather than selecting based on heuristics
Implement error propagation strategies across multi-agent systems
What you need to know
- A propagated error must carry failure type, what was attempted, partial results and suggested alternatives — that is what makes coordinator recovery intelligent rather than blind.
- Distinguish access failures (source never consulted; a retry decision applies) from valid empty results (query succeeded, nothing matched; that is evidence).
- Generic statuses like "search unavailable" are a distractor: they discard exactly the context the coordinator needs.
- Silent suppression and whole-workflow termination are both wrong; subagents recover transient failures locally and propagate only the unresolved ones.
- Synthesis output needs coverage annotations so gaps from unavailable sources are visible instead of reading as findings.
Trace one failure end to end: local recovery first, then a structured propagation that lets the coordinator re-route, and finally a coverage annotation in the output.
Cement the distinction that drives most items here, and show what each of the two anti-patterns costs the coordinator.
Worked examples
A structured subagent error contract
Scenario 3 · Multi-Agent Research SystemThe web-search subagent hits repeated timeouts on one provider. It retries locally with backoff, and only after exhausting its local budget does it return a structured failure that keeps everything the coordinator needs — including the two results it did manage to fetch and an alternative it cannot itself reach.
typescript Discriminated result type: success, empty, or failure — never conflated.
type SubagentResult<T> =
| { status: 'ok'; data: T[] }
| { status: 'empty'; query: string; note: string } // query ran, no matches
| {
status: 'failed';
failureType: 'timeout' | 'rate_limited' | 'auth' | 'not_found' | 'malformed';
attempted: string; // the exact query or target
localRecovery: string; // retries already performed
partialResults: T[]; // keep what was obtained
alternatives: string[]; // routes the coordinator could take
};
const result: SubagentResult<Finding> = {
status: 'failed',
failureType: 'timeout',
attempted: 'web_search("EU battery directive 2026 compliance costs")',
localRecovery: '3 attempts, exponential backoff, then narrowed query',
partialResults: [finding1, finding2],
alternatives: ['regulator PDF archive via document_agent', 'query without year filter'],
};The taxonomy earns its keep at the coordinator
Scenario 6 · Structured Data ExtractionAn extraction worker looks up a supplier VAT number in the reference service. "Not registered" and "reference service unreachable" must not reach the coordinator as the same status: the first is a finding the validator can act on, the second leaves the field unverified and needs a retry decision. Collapsing them means unverified records are silently promoted as verified-absent. Note the division of labor — the shape of the error payload a tool hands back is a tool-design question (2.2); what matters here is that the coordinator's recovery branch is genuinely different for the two outcomes, and that the difference survives into the record.
text One taxonomy, two recovery branches — and what collapsing them costs.
subagent outcome coordinator recovery branch
-----------------------------------------------------------------------
valid empty result accept as evidence; record "not registered";
(query ran, zero matches) do NOT retry; the field is complete
access failure field stays unverified; choose retry, alternate
(timeout, 429, 5xx, auth) source, or human review; annotate the record as
uncovered — never mark the field absent
both flattened into coordinator cannot tell them apart; it either
"lookup unavailable" wastes retries on a settled question or promotes
unverified records as verified-absentCoverage annotations in the synthesized report
Scenario 3 · Multi-Agent Research SystemTwo of five sources were unreachable. The report neither omits the affected topic nor states its conclusions with the same force as the rest. An explicit coverage section tells the reader which claims are well-supported and which topic areas have gaps due to unavailable sources — the difference between a limitation and a silent error.
markdown Report section that makes gaps legible.
## Coverage and confidence
**Well-supported** (3+ independent sources, all reachable)
- Adoption growth in the EU market, 2024-2026.
- Regulatory timeline for the 2027 compliance deadline.
**Partially covered** (single source, or partial results only)
- Cost-per-unit impact: 2 of 6 planned sources retrieved before timeout.
**Gaps due to unavailable sources**
- FY2025 filings: document store returned auth failures on 2 attempts.
No conclusion is drawn about FY2025 in this report.
- Competitor pricing: search provider rate-limited; retry scheduled.Anti-patterns
- Returning a generic status such as "search unavailable" instead of failure type, attempted query, partial results and alternatives because the coordinator then has nothing to base a retry-or-reroute decision on.
- Reporting an empty result as a failure (or a failure as an empty result) instead of distinguishing them because one wastes retries on settled questions and the other lets unverified gaps pass as evidence.
- Silently suppressing a subagent error and returning empty results as success because the coordinator will synthesize confident conclusions over data that was never retrieved.
- Terminating the whole workflow on a single subagent failure instead of recovering locally and propagating only unresolved errors because it discards every other subagent completed result for a recoverable problem.
How it is examined
- Look for stems where a subagent returns "no results" and the coordinator concludes the topic is settled. The credited answer separates access failures from valid empty results.
- Options that "fail fast and abort the workflow" and options that "return an empty array so the pipeline continues" are usually the paired distractors — the guide names both as anti-patterns.
- When the stem is about the final deliverable rather than the run, the answer is usually coverage annotations distinguishing well-supported findings from gaps, not a retry policy.
Why this is true
In a multi-agent system a subagent failure is information, and the reliability question is how much of that information survives the trip back to the coordinator.
Three outcomes, three recovery branches. The taxonomy is the statement. Label which outcome occurred and the coordinator knows what to do; collapse them into one status and it cannot.
| Subagent outcome | What it means | Who resolves it | What the coordinator does |
|---|---|---|---|
| Transient access failure — timeout, rate limit, 500, auth rejection | The source was never consulted, and the fault may clear on its own | The subagent, locally: bounded retry, backoff, a narrower query | Nothing; it never sees the ones that clear |
| Unresolved access failure | Local recovery is exhausted and the topic is still uncovered | Propagated upward, with its context attached | Retry, re-route to an alternative source, narrow the scope, or record a gap |
| Valid empty result | The query executed successfully and nothing matched | Nobody — there is nothing to repair | Accept it as evidence and treat the question as answered |
An empty result is not a broken one. This is the distinction the exam leans on hardest, and it is easy to lose because both outcomes look alike in a log line. Absence is itself a finding: it is what the system now knows. Retrying it spends budget on a settled question, and re-reporting it as a failure understates the coverage the run actually achieved. The reverse error is worse. An access failure dressed as an empty result puts a silent hole into the evidence base, and nothing downstream can tell it apart from a real answer.
What a propagated failure has to carry. Four things: the failure type, what was attempted (the actual query or target), any partial results already obtained, and alternative approaches the subagent can suggest. Four designs compete for the job, and three of them destroy part of that payload:
| Design | What reaches the coordinator | What it costs |
|---|---|---|
| Structured report upward | Failure type, attempted query, partial results, suggested alternatives | Nothing — it is the only design that supports an intelligent recovery decision |
| Generic status such as "search unavailable" | One opaque string | The coordinator cannot tell whether retrying is sensible, whether half the work is already done, or whether another source would answer |
| Empty results returned as success | Nothing; the gap is invisible | Synthesis draws confident conclusions over data that was never retrieved |
| Terminate the whole workflow | The failure, and nothing else | Every other subagent's completed work is thrown away over one recoverable problem |
The correct posture sits between silence and shutdown: resolve transient failures where they happen, and propagate only what you cannot resolve yourself.
Failures must reach the final artifact. Recovery is not complete when the coordinator handles the error; it is complete when the report is honest about it. Structure synthesis output with coverage annotations that distinguish well-supported findings from topic areas with gaps caused by unavailable sources. A reader who cannot see which sections rest on unavailable evidence will treat a hole as a conclusion.
Exam guide, verbatim — what is measured
Knowledge of
- Structured error context (failure type, attempted query, partial results, alternative approaches) as enabling intelligent coordinator recovery decisions
- The distinction between access failures (timeouts needing retry decisions) and valid empty results (successful queries with no matches)
- Why generic error statuses ("search unavailable") hide valuable context from the coordinator
- Why silently suppressing errors (returning empty results as success) or terminating entire workflows on single failures are both anti-patterns
Skills in
- Returning structured error context including failure type, what was attempted, partial results, and potential alternatives to enable coordinator recovery
- Distinguishing access failures from valid empty results in error reporting so the coordinator can make appropriate decisions
- Having subagents implement local recovery for transient failures and only propagate errors they cannot resolve, including what was attempted and partial results
- Structuring synthesis output with coverage annotations indicating which findings are well- supported versus which topic areas have gaps due to unavailable sources
Manage context effectively in large codebase exploration
What you need to know
- The tell-tale sign of context degradation is answers drifting from specific discovered classes to generic "typical patterns", plus inconsistency between turns.
- Scratchpad files persist key findings across context boundaries; the transcript does not.
- Delegate bounded investigations to subagents to isolate verbose exploration output and keep the main agent on coordination.
- Summarize each exploration phase and inject that summary into the next phase initial context instead of letting the next agents rediscover.
- Use /compact when the window fills with discovery output — after key findings are written down.
- Crash recovery = each agent exports state to a known location and the coordinator loads a manifest on resume.
Show the funnel from a broad question to a specific answer, and where verbose output is deliberately kept out of the main agent context.
Click a stage to highlight whether its output lands in a subagent context or the main one.
Show that resumability comes from agents exporting state to known locations and the coordinator loading a manifest and injecting recovered state, rather than from re-running the exploration.
Worked examples
Progressive narrowing instead of reading the repository
Scenario 2 · Code Generation with Claude CodeThe question is "how does the refund flow work?" in a 400k-line service. Reading broadly to build understanding fills the window with code that turns out to be irrelevant and leaves no room for the files that matter. Use the built-in tools in order of cost: Glob for path patterns and Grep for content search (neither loads a file body), then delegated investigations, then a handful of targeted Read calls guided by what the scratchpad says.
text Cheap breadth (Glob, Grep) before expensive depth (Read).
# 1. Breadth — path patterns first, then content, no file bodies loaded
Glob(pattern="src/**/*efund*.ts") # path matching: Refund and refund files
Grep(pattern="class .*Refund|function refund",
glob="src/**/*.ts",
output_mode="files_with_matches") # content search: where it lives
# 2. Delegate the verbose parts to subagents (their context, not yours):
# - "trace refund flow dependencies from RefundService to the gateway"
# - "find all test files covering refunds and list what they assert"
# 3. Record what came back in notes/refund-flow.md, then work from the file
- Entry: src/api/refunds/handler.ts -> RefundService.create()
- Policy gate: src/domain/refund/policy.ts (RefundWindowPolicy, 30 days)
- Gateway: src/infra/payments/StripeRefundClient.ts (idempotency key required)
- Tests: test/refund/*.spec.ts (12 files), no test for partial refunds
# 4. Only now Read the 3 files that actually matter, then /compactPhase summary injected into the next phase
Scenario 2 · Code Generation with Claude CodePhase 1 mapped the modules; phase 2 must assess test coverage. Rather than continuing in a context already full of phase-1 grep output, the coordinator condenses phase 1 into a short findings block and passes it as the opening context of each phase-2 subagent. The subagents inherit the specifics — real class names, real paths — which is exactly what degraded context loses.
text Findings block injected as initial context for phase 2.
PHASE 1 FINDINGS (authoritative; do not rediscover)
Refund entry point : src/api/refunds/handler.ts
Core service : src/domain/refund/RefundService.ts
Policy gate : RefundWindowPolicy (30-day window, src/domain/refund/policy.ts)
Payment gateway : StripeRefundClient, requires idempotency key
Known gap : no test covers partial refunds
YOUR TASK (phase 2)
Assess test coverage for the components listed above ONLY.
Write findings to notes/phase2-coverage.md before returning.
Return at most 20 lines: file, covered behavior, gap.Manifest-based crash recovery for a long run
Scenario 3 · Multi-Agent Research SystemAn overnight multi-agent exploration crashes after the third of six phases. Because each agent exported its state to a known path and registered it in a manifest, the coordinator resumes by loading the manifest, injecting completed state into prompts, and re-dispatching only the incomplete phases.
json Manifest the coordinator loads on resume.
{
"run_id": "explore-2026-08-04T22:10Z",
"updated_at": "2026-08-05T01:47Z",
"phases": [
{ "id": "p1-module-map", "status": "complete",
"state_path": "state/p1-module-map.json", "findings": "notes/modules.md" },
{ "id": "p2-refund-flow", "status": "complete",
"state_path": "state/p2-refund-flow.json", "findings": "notes/refund-flow.md" },
{ "id": "p3-test-coverage", "status": "partial",
"state_path": "state/p3-test-coverage.json",
"completed_units": 8, "total_units": 12,
"resume_hint": "re-dispatch units 9-12 only" },
{ "id": "p4-dependency-risk", "status": "pending", "state_path": null }
],
"inject_on_resume": ["notes/modules.md", "notes/refund-flow.md"]
}Anti-patterns
- Exploring a large codebase by reading files broadly instead of narrowing progressively with search and delegated investigations because the window fills with code that turns out to be irrelevant and the relevant files never fit.
- Keeping key findings only in the conversation instead of writing them to a scratchpad file because context degradation replaces the specific classes discovered earlier with generic "typical pattern" answers.
- Running verbose discovery in the main agent instead of delegating it to subagents because the coordinator loses the high-level context it needs precisely when the exploration gets long.
- Compacting or starting a fresh session to fix degradation without first persisting findings and injecting a phase summary because the next phase then rediscovers everything from scratch.
How it is examined
- The stem often describes the symptom rather than the cause: inconsistent answers and references to "typical patterns" late in a session. Name it as context degradation and pick scratchpad persistence plus delegation.
- Distinguish the two roles of subagents in this domain: here they exist to isolate verbose output, not to parallelize. Options praising speed are usually the weaker choice.
- Crash-recovery items reward structured state exports plus a coordinator-loaded manifest; distractors offer longer timeouts, bigger context windows, or simply restarting the run.
Why this is true
Exploring an unfamiliar codebase is the workload that exhausts context fastest, because discovery output is voluminous and most of it is disposable.
Recognize context degradation. The symptom in extended sessions is specific and testable. Answers turn inconsistent across turns, and the model starts referencing "typical patterns" — "usually a repository layer handles this" — instead of the concrete classes and files it discovered earlier. That shift from specific to generic is the signal that key findings have been pushed out of effective context, and no amount of re-asking will bring them back. The remedies are all about moving findings out of the transcript and into durable form.
Scratchpad files. Have the agent maintain a scratchpad file recording key findings — entry points, file paths, class names, invariants — and reference that file for subsequent questions. A finding written to disk survives a context boundary; the same finding held only in the transcript does not. This is what lets a later question be answered from the recorded specifics instead of from generic pattern-matching.
Subagent delegation as context isolation. Spawn subagents for bounded investigations ("find all test files", "trace the refund flow dependencies"). The subagent absorbs the verbose exploration output — dozens of file reads and grep dumps — and returns a compact answer, while the main agent's context stays reserved for high-level coordination. The point is isolation of verbosity, not parallel speed.
Phase boundaries and compaction. Before spawning the next phase's subagents, summarize the key findings of the phase that just finished. Inject that summary into the new agents' initial context so they start informed rather than rediscovering. Within an interactive Claude Code session, use /compact to reduce context usage once the window has filled with verbose discovery output — after the findings you care about are safely in the scratchpad, not before.
Crash recovery via manifests. Long unattended runs need structured state persistence. Each agent exports its state to a known location; on resume the coordinator loads a manifest and injects the recovered state into agent prompts. That converts a crash from "start the whole exploration again" into "resume the phases that did not complete".
Exam guide, verbatim — what is measured
Knowledge of
- Context degradation in extended sessions: models start giving inconsistent answers and referencing "typical patterns" rather than specific classes discovered earlier
- The role of scratchpad files for persisting key findings across context boundaries
- Subagent delegation for isolating verbose exploration output while the main agent coordinates high- level understanding
- Structured state persistence for crash recovery: each agent exports state to a known location, and the coordinator loads a manifest on resume
Skills in
- Spawning subagents to investigate specific questions (e.g., "find all test files," "trace refund flow dependencies") while the main agent preserves high-level coordination
- Having agents maintain scratchpad files recording key findings, referencing them for subsequent questions to counteract context degradation
- Summarizing key findings from one exploration phase before spawning sub-agents for the next phase, injecting summaries into initial context
- Designing crash recovery using structured agent state exports (manifests) that the coordinator loads on resume and injects into agent prompts
- Using /compact to reduce context usage during extended exploration sessions when context fills with verbose discovery output
Design human review workflows and confidence calibration
What you need to know
- A 97% aggregate can hide a document type or a field performing far worse; validate accuracy by document type and by field before reducing review.
- Confidence should be field-level, and thresholds must be calibrated on labeled validation sets rather than chosen intuitively.
- Route to humans on low confidence AND on ambiguous or contradictory source documents — high confidence over a self-contradicting document is still a review case.
- Stratified random sampling of high-confidence extractions keeps measuring the automated path and surfaces novel error patterns.
- The goal is prioritizing limited reviewer capacity toward the highest expected error, not reviewing more overall.
Show that routing is two-dimensional: confidence alone does not decide, because source ambiguity overrides a high score, and the automated cell still owes a sampled audit.
Click a cell to reveal the reason its route is what it is.
Make the masking effect concrete so the learner instinctively decomposes any headline accuracy figure by document type and field.
Worked examples
Segment analysis before switching off review
Scenario 6 · Structured Data ExtractionThe team wants to auto-approve high-confidence extractions. The aggregate says 97.1%. Breaking accuracy down by document type and field shows two segments that must keep full review, and the decision becomes segment-scoped rather than global.
text Accuracy by segment — the breakdown that changes the decision.
document type volume accuracy decision
--------------------------------------------------------------
typed PDF invoice 62% 99.2% eligible for automation
digital receipt 21% 98.4% eligible for automation
scanned invoice 12% 96.1% automate, sample at 5%
handwritten form 5% 71.3% keep 100% human review
field accuracy decision
--------------------------------------------------------------
invoice_number 99.4% automate
total_amount 98.9% automate
tax_id 82.0% keep human review (all doc types)
line_item_dates 91.2% automate only for typed PDFs
Aggregate: 97.1% <- would have justified automating everything.Field-level confidence with a calibrated threshold
Scenario 6 · Structured Data ExtractionThe extractor emits a confidence value per field plus flags describing the source evidence. Routing uses the calibrated threshold for that field and document type, and any source ambiguity overrides a high score.
json Extraction payload carrying per-field confidence and source flags.
{
"document_id": "inv-2026-08-0417",
"document_type": "scanned_invoice",
"fields": {
"invoice_number": { "value": "INV-88213", "confidence": 0.97,
"source_span": "p1:header", "source_flags": [] },
"total_amount": { "value": "1284.50", "confidence": 0.94,
"source_span": "p2:total-row",
"source_flags": ["two_conflicting_totals"] },
"tax_id": { "value": "DE811234567", "confidence": 0.61,
"source_span": "p1:footer", "source_flags": ["low_ocr_quality"] }
},
"routing": {
"invoice_number": "auto_approve",
"total_amount": "human_review (source conflict overrides confidence)",
"tax_id": "human_review (below calibrated threshold 0.92)"
}
}Stratified sampling of the automated path
Scenario 6 · Structured Data ExtractionAuto-approved extractions are not left unexamined. A sampler draws from each stratum — document type crossed with field and confidence band — so low-volume, high-risk strata are represented instead of being swamped by the dominant one. The sampled records go to reviewers as labeled data, feeding both the ongoing error-rate estimate and threshold recalibration.
typescript Stratified draw: rare strata get proportionally more attention.
type Stratum = { docType: string; field: string; band: '0.92-0.95' | '0.95-1.0' };
// Uniform sampling would spend ~62% of the budget on typed PDFs.
// Stratified sampling guarantees coverage of every risky stratum.
function stratifiedSample(pool: Extraction[], budget: number) {
const strata = groupBy(pool, keyOf); // docType x field x band
const perStratum = Math.max(5, Math.floor(budget / strata.size));
return [...strata.values()].flatMap((rows) =>
randomDraw(rows, Math.min(perStratum, rows.length)),
);
}
// Reviewer verdicts feed two loops:
// 1. error rate per stratum -> is the automated path still within tolerance?
// 2. new failure descriptions -> novel error patterns (e.g. a changed
// vendor template) that no existing validation rule would have caught.Anti-patterns
- Reducing human review on the strength of an aggregate accuracy figure instead of per-document-type and per-field analysis because the mean is dominated by the easy segment and hides the one that fails.
- Auto-approving everything above a confidence threshold without stratified sampling of that population because novel error patterns from changed document formats then stay invisible until a customer finds them.
- Asking for human review with no confidence signal or priority ordering because undifferentiated queues spend scarce reviewer capacity on the extractions least likely to be wrong.
- Trusting raw model confidence as a probability instead of calibrating thresholds against a labeled validation set because an uncalibrated score does not map to any measured error rate.
How it is examined
- Any stem quoting a single impressive accuracy number (97%, 99%) is testing aggregate masking — the credited answer breaks accuracy down by document type and field before automating.
- Watch the difference between "sample randomly" and "sample stratified". Uniform sampling is the plausible distractor; stratification is what surfaces rare segments and novel patterns.
- Routing items usually have two triggers hidden in the stem: low confidence and ambiguous or contradictory source documents. An option that only handles low confidence is incomplete.
Why this is true
Human review is a scarce resource, so the design question is how to spend it where the errors actually are — and how to know where that is.
Aggregate accuracy hides the failures that matter. A pipeline reporting 97% overall accuracy tells you almost nothing about risk, because the mean is dominated by the high-volume, easy segment. Underneath it a specific document type (handwritten forms, low-quality scans, a foreign-language variant) or a specific field (tax ID, line-item totals, dates in ambiguous formats) may be running far below the headline. Before reducing human review, analyze accuracy by document type and by field and confirm performance is consistent across all segments. Automating on the aggregate is the classic wrong answer: it automates precisely the segment where automation fails.
Field-level confidence, calibrated against labeled data. Have the model output confidence per extracted field rather than one score per document. A document can have nine solid fields and one guess, so the useful routing decision is field-scoped. Raw scores are not trustworthy on their own. They must be calibrated using labeled validation sets until a threshold means something empirically: "extractions above 0.92 have a measured 0.4% error rate on this document type". The threshold is an output of measurement, not a number someone chose.
Route on two axes, not one. Confidence decides nothing by itself, because the score describes the model and the second axis describes the document:
| Confidence against the calibrated threshold | Source evidence | Route | Why |
|---|---|---|---|
| Above | Clear and self-consistent | Auto-approve, then sample it | The measured error rate is acceptable; sampling is what keeps that true |
| Above | Ambiguous or contradictory | Human review | The model is confident about a value the document does not settle |
| Below | Clear and self-consistent | Human review | Below the calibrated threshold means an error rate nobody accepted |
| Below | Ambiguous or contradictory | Priority human review | Highest expected error — spend the scarcest capacity here first |
Row 2 is the one exam items are built on. A high score means the model read the document consistently, not that the document says one thing: a page stating two different totals reaches a person whatever the score says. Prioritizing along both axes is what lets limited reviewer capacity cover the highest expected error.
Keep measuring what you automated. High-confidence extractions that skip review still need stratified random sampling: sample across strata (document type, field, confidence band) rather than uniformly, so rare-but-risky segments are actually represented. This gives an ongoing error-rate estimate for the automated path and — importantly — detects novel error patterns that appear when document formats change upstream. Without sampling, a new failure mode is invisible until it reaches a customer, because everything the model was confident about was accepted unexamined.
Exam guide, verbatim — what is measured
Knowledge of
- The risk that aggregate accuracy metrics (e.g., 97% overall) may mask poor performance on specific document types or fields
- Stratified random sampling for measuring error rates in high-confidence extractions and detecting novel error patterns
- Field-level confidence scores calibrated using labeled validation sets for routing review attention
- The importance of validating accuracy by document type and field segment before automating high- confidence extractions
Skills in
- Implementing stratified random sampling of high-confidence extractions for ongoing error rate measurement and novel pattern detection
- Analyzing accuracy by document type and field to verify consistent performance across all segments before reducing human review
- Having models output field-level confidence scores, then calibrating review thresholds using labeled validation sets
- Routing extractions with low model confidence or ambiguous/contradictory source documents to human review, prioritizing limited reviewer capacity
Preserve information provenance and handle uncertainty in multi- source synthesis
What you need to know
- Claim-source mappings (source URL or document name plus relevant excerpt) must be created at discovery and preserved and merged through synthesis, never reconstructed later.
- Conflicting statistics from credible sources are annotated with attribution and passed up for the coordinator to reconcile — never resolved by arbitrary selection.
- Reports separate well-established from contested findings and keep each source original characterization and methodological context.
- Publication and data-collection dates in structured output stop temporal differences from being misread as contradictions.
- Match rendering to content type — forcing tables, prose and findings into one shape loses the structure.
Contrast the path that carries claim-source mappings as structured data through synthesis with the path that compresses findings into prose and loses attribution irreversibly.
Show why keeping both conflicting values with attribution, methodology and dates is the only option that lets the coordinator reconcile knowingly.
Worked examples
A claim-source mapping that survives synthesis
Scenario 3 · Multi-Agent Research SystemEach research subagent returns findings as records, not prose. The coordinator merges the arrays; the synthesis agent is instructed to carry the source, excerpt and date fields through into the report and to refuse to state any claim that has lost its mapping. That constraint is what makes the final citations verifiable rather than decorative.
json Structured finding record emitted by every subagent.
{
"findings": [
{
"claim": "EU adoption of the technology grew 34% year over year",
"source": "https://example-institute.org/reports/eu-adoption-2026",
"source_name": "EU Adoption Monitor 2026",
"excerpt": "adoption across the EU-27 rose 34% relative to 2025",
"publication_date": "2026-02-11",
"data_collection_period": "2025-07 to 2025-12",
"methodology": "self-reported survey, n=412 firms",
"relevance_score": 0.91
}
],
"synthesis_contract": "preserve claim, source, excerpt, dates and methodology verbatim; do not merge claims from different sources into one sentence"
}Two credible sources, two different numbers
Scenario 3 · Multi-Agent Research SystemThe document analyst finds 34% in a survey report and 21% in audited filings. Picking either one would hide a real methodological difference. The analyst completes its task with both values annotated, and the coordinator — which can see the methodology and the collection periods — decides whether they are a conflict, a time-series difference, or two measurements of different populations.
json Conflict passed upward with attribution intact.
{
"metric": "year-over-year adoption growth, EU",
"status": "conflicting_values",
"values": [
{ "value": "34%", "source_name": "EU Adoption Monitor 2026",
"publication_date": "2026-02-11", "collection_period": "2025-H2",
"methodology": "self-reported survey, n=412" },
{ "value": "21%", "source_name": "Regulator Filing Digest",
"publication_date": "2026-05-30", "collection_period": "FY2025",
"methodology": "audited filings, n=1,180" }
],
"analyst_note": "different populations and periods; likely not a contradiction",
"resolution": "deferred to coordinator; do not average or drop either value"
}Report structure that separates established from contested
Scenario 6 · Structured Data ExtractionThe extraction and reporting layer renders each content type in its natural form and keeps the certainty gradient visible. Financial figures go in a table where values are comparable; contested claims get their own section with both characterizations preserved; qualitative context stays as prose instead of being shredded into bullets.
markdown Synthesis output with a certainty gradient and type-appropriate rendering.
## Well-established findings
| Metric | Value | Source | Collected | Method |
|---|---|---|---|---|
| Revenue FY2025 | USD 1.28B | Annual Report 2025 | FY2025 | audited |
| Headcount | 4,120 | Annual Report 2025 | 2025-12-31 | audited |
## Contested findings
**YoY adoption growth — 34% vs 21%.** The EU Adoption Monitor 2026
(self-reported survey, n=412, collected 2025-H2) reports 34%. The Regulator
Filing Digest (audited filings, n=1,180, FY2025) reports 21%. The populations
and periods differ; both figures are retained.
## Context and developments (prose)
Coverage in Q1 2026 focused on the compliance deadline; no source disputes
the timeline itself, only the cost estimates attached to it.Anti-patterns
- Summarizing subagent findings into prose instead of preserving structured claim-source mappings because once the URL, document name and excerpt are compressed away no downstream agent can restore them and citations become guesses.
- Arbitrarily selecting one of two conflicting statistics instead of annotating both with source attribution and deferring reconciliation to the coordinator because it erases a real disagreement and overstates certainty.
- Omitting publication or collection dates from structured outputs because differently dated measurements of the same metric then look like sources contradicting each other.
- Converting every source into one uniform format instead of rendering financial data as tables, news as prose and technical findings as structured lists because flattening destroys comparability and methodological nuance.
How it is examined
- When a stem says the final report cited sources incorrectly or generically, the mechanism is attribution lost during an intermediate summarization step — the credited answer adds structured claim-source mappings that downstream agents must preserve.
- For conflicting-statistics stems, every option that picks a winner (most recent, most reputable, the average) is wrong: annotate both with attribution and let the coordinator reconcile.
- If the stem mentions figures that "contradict" each other and the sources are from different years, the answer is requiring publication or data-collection dates, not adjudicating the numbers.
Why this is true
When many sources are merged into one deliverable, two things are routinely destroyed: where each claim came from, and how certain it is. Both losses happen at the same place — the summarization step.
Attribution dies in compression, and the loss is one-way. A subagent finds a figure in a named report and writes "adoption grew 34%". Once that sentence is summarized into a coordinator-level brief, the URL, document name and excerpt are gone. No downstream agent can restore them, so the synthesis agent either drops the citation or invents a plausible one. This is why provenance has to travel as data through synthesis rather than be reconstructed from prose afterwards: every downstream agent preserves and merges the structured record instead of rewriting it.
Conflicts are annotated, not resolved by fiat. When two credible sources give different statistics, silently choosing one erases a real disagreement and makes the report look more settled than the evidence is. Complete the analysis with both values included and attributed, and let the coordinator reconcile before anything reaches synthesis. The report then separates well-established findings from contested ones, which is the certainty gradient a reader needs in order to weigh a claim.
What every finding carries, and what breaks without it. Each element is a separate failure mode, so a record missing one is not merely less detailed:
| Element in the structured finding | What it preserves | What breaks without it |
|---|---|---|
| Source URL or document name, plus the relevant excerpt | The claim-source mapping, still verifiable after synthesis | Citations become guesses — dropped, or invented to look plausible |
| Publication and data-collection dates | Which vintage of a measurement this is | A 2023 figure and a 2026 figure read as a contradiction instead of a time series |
| Methodology and each source's original characterization | Why two credible numbers can both be right | "Self-reported survey of 400 firms" and "audited filings" collapse into one unexplained gap |
| Both values when sources conflict, each attributed | A real disagreement, visible as a disagreement | Arbitrary selection overstates certainty and hides the conflict from the coordinator |
| Content type — financial data as tables, news as prose, technical findings as structured lists | Comparability of numbers, nuance of qualitative material | One uniform format loses both: narrative paragraphs bury the numbers, bullet fragments shred the nuance |
Exam guide, verbatim — what is measured
Knowledge of
- How source attribution is lost during summarization steps when findings are compressed without preserving claim-source mappings
- The importance of structured claim-source mappings that the synthesis agent must preserve and merge when combining findings
- How to handle conflicting statistics from credible sources: annotating conflicts with source attribution rather than arbitrarily selecting one value
- Temporal data: requiring publication/collection dates in structured outputs to prevent temporal differences from being misinterpreted as contradictions
Skills in
- Requiring subagents to output structured claim-source mappings (source URLs, document names, relevant excerpts) that downstream agents preserve through synthesis
- Structuring reports with explicit sections distinguishing well-established findings from contested ones, preserving original source characterizations and methodological context
- Completing document analysis with conflicting values included and explicitly annotated, letting the coordinator decide how to reconcile before passing to synthesis
- Requiring subagents to include publication or data collection dates in structured outputs to enable correct temporal interpretation
- Rendering different content types appropriately in synthesis outputs—financial data as tables, news as prose, technical findings as structured lists—rather than converting everything to a uniform format