Overview

From Prompt Engineering to Loop and Graph Engineering

  • Prompt engineering optimizes an individual model interaction: the user supplies instructions and context, the model produces an output, and the user decides what happens next. Loop engineering moves this decision-making structure into software. The engineer defines the objective, operating constraints, feedback signals, memory, and stopping conditions; the system then invokes the model repeatedly until it reaches a verified outcome or exhausts a safety limit. Loop Engineering describes this as replacing manual, turn-by-turn prompting with a system that discovers work, delegates it, checks the result, records progress, and selects the next action.

  • Graph engineering generalizes this design from one iterative process to a network of coordinated processes. Instead of asking one agent to perform a long sequence of loosely related operations, the engineer decomposes the work into bounded nodes, represents real dependencies as edges, executes independent nodes concurrently, routes uncertain or high-risk results through deeper review, and recombines verified evidence at explicit synchronization points. The complete Graph Engineering playbook for Claude Code presents this transition as a move from optimizing a single prompt to designing the path by which information, decisions, and evidence travel through an agentic system.

  • The central progression is therefore:

\[\text{Prompt} \;\longrightarrow\; \text{Agentic run} \;\longrightarrow\; \text{Verified loop} \;\longrightarrow\; \text{Graph of coordinated loops}\]
  • This progression does not imply that every task should use a loop or graph. Complexity should be introduced only when it produces measurable gains in reliability, coverage, latency, or autonomy. Building effective agents distinguishes fixed workflows from dynamically directed agents and recommends starting with the simplest architecture that satisfies the task.

The Architectural Layers

  • Loop and graph engineering are easiest to understand as distinct layers rather than competing names for the same idea.

The Model

  • The model supplies probabilistic reasoning, language generation, planning, classification, and tool-selection capabilities. By itself, however, it is not a durable workflow. A model invocation has a finite context, no inherently reliable long-term state, no guaranteed access to external systems, and no independently trustworthy definition of success.

  • ReAct: Synergizing Reasoning and Acting in Language Models by Yao et al. (2023) established an important foundation by interleaving reasoning with actions and environmental observations, allowing an agent to revise its plan using information returned by external tools.

The Harness

  • The harness is the operational environment surrounding a model invocation. It typically includes:

    • Tool interfaces and external connectors.
    • Repository and filesystem access.
    • Retrieval and context assembly.
    • Persistent instructions and project knowledge.
    • Sandboxes and permission boundaries.
    • Logging, tracing, and cost accounting.
    • Tests, linters, compilers, and other evaluators.
    • Recovery and retry mechanisms.

  • The harness makes one agent run capable of performing useful work. It determines what the agent can observe, what it can modify, how it receives feedback, and what damage it can cause. The model is the reasoning engine; the harness defines the environment in which that reasoning becomes action. Building effective agents similarly describes the augmented language model as a model equipped with retrieval, tools, and memory.

The Loop

  • The loop repeatedly executes a harnessed agent against an evolving external state. A basic loop performs five conceptual operations:
\[\text{Observe} \;\longrightarrow\; \text{Plan} \;\longrightarrow\; \text{Act} \;\longrightarrow\; \text{Verify} \;\longrightarrow\; \text{Update}\]

  • If verification succeeds, the loop terminates. If verification fails but the remaining budget permits another attempt, the result is converted into feedback and the next iteration begins. Getting started with loops distinguishes turn-based, goal-based, time-based, and proactive loops according to their trigger, stopping condition, and appropriate task class.

  • The essential property is not repetition alone. It is feedback-directed repetition governed by an external criterion. A shell script that invokes an agent indefinitely is repetitive, but it is not a well-engineered loop unless each iteration receives meaningful evidence about progress and the system can determine when to stop.

The Graph

  • A graph coordinates multiple bounded computations. Some nodes may be agent loops, while others may be deterministic scripts, routers, database operations, approval gates, or human-review stages. Edges carry structured outputs and encode actual dependencies.

  • A graph commonly contains:

    • Sequential chains where one result is required by the next node.
    • Fan-out stages that divide independent work across workers.
    • Barriers that wait for required branches to complete.
    • Reduce stages that deduplicate, aggregate, or rank results.
    • Routers that choose branches according to risk, confidence, or task type.
    • Verifiers that reject unsupported or malformed outputs.
    • Feedback edges that return failed work to an earlier node.
    • Human gates for high-impact or weakly specified decisions.

  • The following figure (source) shows a split-reduce-synthesize graph in which a shared scope fans out into parallel inspection branches, converges through reduction and risk routing, and reaches a final synthesis stage only after the evidence has been filtered.

  • A loop is therefore the fundamental iterative unit, while a graph determines how multiple units exchange work:
\[\text{Loop} = \text{one feedback-controlled unit}\] \[\text{Graph} = \text{multiple units connected by explicit dependencies}\]
  • Graph engineering is not merely “running many agents.” Launching several agents against the same unconstrained task may increase cost and produce redundant, correlated outputs. The engineering value comes from specialization, structured interfaces, selective parallelism, explicit synchronization, independent verification, and traceable evidence flow.

A Formal Model of an Agent Loop

  • A loop can be represented as a discrete-time control process. At each iteration, the system observes the current environment, assembles a bounded context, asks an agent to select an action, applies the action, and evaluates the resulting state:
\[\begin{aligned} o_t &= \Omega(s_t) \\ c_t &= C(o_t, m_t, g, b_t) \\ a_t &\sim \pi_{\theta}(\,\cdot \mid c_t\,) \\ s_{t+1} &= T(s_t, a_t) \\ r_{t+1} &= E(s_{t+1}, g) \\ m_{t+1} &= U(m_t, a_t, r_{t+1}) \end{aligned}\]

  • Here:

    • The environment state includes the repository, files, test results, issue state, external services, and any other task-relevant reality.
    • The observation function selects what the loop can inspect.
    • The context builder combines the current observation, persistent memory, goal, and remaining budget.
    • The agent policy chooses an action conditioned on that context.
    • The transition function applies the action to the environment.
    • The evaluator measures the new state against the goal.
    • The memory update records what was attempted, what changed, and what remains unresolved.

  • The loop terminates when success is verified or a resource boundary is reached:

\[\operatorname{stop}_t = \mathbf{1} \left[ r_t \geq \tau \;\lor\; t \geq T_{\max} \;\lor\; c_t^{\text{cost}} \geq B_{\max} \;\lor\; \operatorname{unsafe}(s_t) \;\lor\; \operatorname{stuck}(h_t) \right]\]

  • The completion threshold should preferably be computed from external evidence, such as a test exit code, benchmark score, schema validator, database invariant, or explicitly authorized human decision. It should not depend exclusively on the generating model’s assertion that the task is complete.

  • Self-Refine: Iterative Refinement with Self-Feedback by Madaan et al. (2023) demonstrates that iterative feedback and refinement can improve model outputs without additional training, but its generator, feedback provider, and refiner may be the same model. Production loops generally strengthen this pattern by adding independent or deterministic evaluators.

  • Reflexion: Language Agents with Verbal Reinforcement Learning by Shinn et al. (2023) shows how environmental feedback can be converted into verbal reflections stored in episodic memory and reused in subsequent attempts, providing a research foundation for persistent failure summaries and cross-iteration learning.

A Formal Model of an Agent Graph

  • An agent workflow graph consists of nodes and directed edges:
\[G = (V, E)\]
  • Each node implements a bounded transformation:
\[f_i : \mathcal{X}_i \longrightarrow \mathcal{Y}_i\]
  • Each edge transfers a typed output from one node to a dependent node:
\[e_{ij} = \left( v_i, v_j, \sigma_{ij} \right)\]

  • The schema on the edge is important. It specifies the fields, types, evidence references, confidence values, and status information that the downstream node may consume. Without schemas, downstream agents must reinterpret free-form prose, creating semantic drift at every handoff.

  • A node may be:

    • A deterministic program.
    • A model invocation.
    • A persistent agent loop.
    • A tool or service call.
    • A routing decision.
    • A synchronization barrier.
    • A human approval gate.

  • A graph with no feedback edges is a directed acyclic workflow and can be scheduled according to its dependency order. A graph containing feedback edges supports iterative correction, discovery, or optimization. Each cyclic region must have its own stopping condition and resource limits.

  • The cost of a graph is approximately the sum of its node executions across all loop iterations:

\[C_{\text{graph}} = \sum_{k=1}^{K} \sum_{v_i \in V_k} C(v_i, k) + C_{\text{orchestration}}\]
  • Its latency is determined less by the number of nodes than by the longest dependency path, subject to concurrency limits and queueing:
\[L_{\text{graph}} \geq \max_{p \in \mathcal{P}} \sum_{v_i \in p} L(v_i)\]

  • This is why parallelism is valuable only when branches are genuinely independent. If the next operation requires the previous operation’s output, the edge is real and sequential execution is necessary. If no information crosses between two tasks, serializing them adds latency without improving correctness.

  • The fan-out and reduce structure has a direct precedent in MapReduce: Simplified Data Processing on Large Clusters by Dean and Ghemawat (2004), which separates parallel mapping from deterministic aggregation. Agent graphs extend this pattern by allowing model-based judgment, search, and synthesis inside selected nodes.

  • Graph of Thoughts: Solving Elaborate Problems with Large Language Models by Besta et al. (2024) represents generated reasoning units as vertices connected by dependencies and transformations. Graph engineering applies a related structural idea at the systems level, where nodes may be independent agents, scripts, tools, verifiers, or approval gates rather than only intermediate thoughts.

The Fundamental Design Principles

Externalized State

  • Long-running work cannot depend solely on a conversation transcript. Persistent state should reside in repositories, databases, task systems, structured files, or other durable stores. This allows a fresh agent invocation to reconstruct the current task state without replaying the full history.

  • A practical design separates:

  • Human-readable state, such as progress notes, unresolved questions, and next actions.
  • Machine-readable state, such as iteration counts, processed identifiers, blocked paths, last-known-good revisions, failure signatures, and budget consumption.

  • Immutable evidence, such as test logs, diffs, benchmark results, and approval records.

  • Fresh, narrow contexts also reduce dependence on very long prompts. Lost in the Middle: How Language Models Use Long Contexts by Liu et al. (2024) finds that language-model performance can degrade when relevant information appears in the middle of long contexts, motivating deliberate context selection rather than indiscriminate history accumulation.

Verifiable Progress

  • A loop must distinguish activity from progress. The agent may edit many files, produce lengthy analyses, or generate numerous candidate solutions without moving closer to the actual goal.

  • A reliable evaluator should be:

  • Independent of the proposed solution where possible.
  • Deterministic under a fixed state.
  • Idempotent when repeated without intervening changes.
  • Difficult for the agent to manipulate.
  • Sensitive to regressions outside the immediate target.

  • Capable of returning diagnostic evidence, not merely a binary label.

  • Tests should therefore be protected from unauthorized modification, benchmark definitions should remain immutable during optimization, and approval criteria should be evaluated separately from the implementation that attempts to satisfy them.

Bounded Autonomy

  • Every autonomous region requires both a success condition and one or more failure exits. Appropriate controls include iteration limits, token or monetary budgets, timeouts, repeated-failure detectors, permission boundaries, mandatory checkpoints, and escalation rules.

  • Autonomy should be proportional to blast radius. A loop editing an isolated branch can safely receive more freedom than one capable of deploying production code, modifying infrastructure, transferring funds, or sending external communications.

Structured Evidence Flow

  • Edges should carry evidence rather than ungrounded conclusions. A useful finding object may include:

    • The claim.
    • Its source location.
    • Supporting observations.
    • Confidence or uncertainty.
    • The method used to obtain it.
    • The node that produced it.
    • A stable identifier for deduplication.
    • The verification status.
    • Links to the underlying artifacts.

  • This structure makes it possible to trace a final recommendation back through reduction, classification, and discovery stages. It also enables deterministic validation at graph boundaries.

Separation of Probabilistic and Deterministic Work

  • Language models should be used for tasks that require semantic interpretation, open-ended search, comparison, planning, critique, or synthesis. Ordinary code should handle operations with exact answers, including sorting, schema validation, deduplication, arithmetic, access checks, threshold comparisons, file hashing, and dependency scheduling.

  • Using a model for deterministic transformation increases cost and introduces unnecessary variance. Conversely, forcing an underspecified judgment into brittle code can hide uncertainty rather than eliminate it.

Independent Review

  • The generator and evaluator should not be treated as interchangeable roles. A fresh reviewer receives only the task specification, candidate output, and relevant evidence, without inheriting the generator’s reasoning narrative. For high-impact decisions, review can be diversified through different prompts, model families, tools, or evaluation modalities.

  • The purpose is not to manufacture consensus. Multiple agents with the same context and assumptions can produce correlated errors. Independent review is valuable only when the reviewers obtain genuinely different evidence or apply meaningfully different criteria.

When to Use a Loop, a Graph, or Neither

  • Use a single model call: The task is narrow, low-risk, easily inspected, and unlikely to benefit from iteration.

  • Use an interactive agent: The work is exploratory, requirements are still emerging, or frequent human judgment is necessary.

  • Use a loop: The task admits repeated attempts, the environment returns useful feedback, success can be verified, and the work benefits from continuing without manual prompting.

  • Use a graph: The task contains independent branches, heterogeneous specialist roles, large evidence sets, conditional routing, or multiple verification stages.

  • Use nested loops: Both the task output and the process that searches for the output can be measured and improved. Bilevel Autoresearch: Meta-Autoresearching Itself by Qu and Lu (2026) demonstrates an outer loop that modifies the search mechanisms used by an inner autonomous research loop, showing how graph structure can optimize not only an artifact but also the procedure that produces it.

  • Retain a human gate: The objective is ambiguous, the consequences are difficult to reverse, evaluation requires organizational judgment, or automated checks cover only part of the real requirement.

The Core Engineering Shift

  • Loop and graph engineering relocate the engineer’s leverage. The primary artifact is no longer only a prompt or even an agent. It is an executable control structure that determines:

    • What work becomes visible.
    • Which component receives it.
    • What context that component sees.
    • Which actions it may perform.
    • How evidence is recorded.
    • How failures are diagnosed.
    • What conditions cause retry, escalation, or termination.
    • Which boundary requires accountable human judgment.

  • A successful system does not merely produce more model activity. It converts uncertain model behavior into bounded, observable, evidence-producing computation. The loop provides depth through iteration. The graph provides width through decomposition and coordination. The harness provides the tools, memory, and constraints that make both operationally meaningful.

Foundations and Core Concepts

Agentic Systems as Feedback-Controlled Computation

  • An agentic system combines a language model with tools, environmental observations, persistent state, and control logic. The model supplies probabilistic judgment, but the surrounding system determines what the model can observe, which actions it can take, how outputs affect the environment, and whether another step is required. Building effective agents distinguishes workflows, whose execution paths are substantially predetermined by code, from agents, whose models dynamically choose their processes and tool usage.

  • This distinction produces three broad execution modes:

    • Single invocation: A model receives an input and returns an output without environmental interaction.

    • Workflow: Code determines the sequence of model calls, tools, checks, and branches.

    • Agent: The model dynamically selects actions according to the current environment and task state.

  • A loop can contain any of these modes. A deterministic workflow may invoke an agent at one stage, while an agent may itself execute a fixed verification workflow. Loop engineering concerns the feedback and control structure surrounding these components. Graph engineering concerns their composition, dependencies, and information flow.

The Anatomy of a Loop

  • A loop is a feedback-controlled process that repeatedly attempts to move an environment toward a specified goal. A complete loop requires more than a repeated prompt. It needs a goal, an observable state, an action policy, an evaluator, persistent memory, and termination logic.

  • The following figure (source) shows the fundamental loop cycle: discover the current state, plan an intervention, execute it, verify the outcome, and either terminate or resume with updated evidence.

Goal

  • The goal defines the state the system should reach. A useful goal identifies an observable outcome rather than merely describing an activity.

  • Weak goals describe effort:

    • Improve the application.
    • Investigate the test failures.
    • Make the implementation cleaner.
    • Continue until the result looks good.

  • Operational goals describe measurable terminal states:

    • All authentication tests exit successfully.
    • The benchmark score exceeds a fixed threshold.
    • Every identified dependency has an owner and migration status.
    • The queue is empty and every processed item has a recorded disposition.
    • The generated report passes its schema and contains a citation for every factual claim.

  • A goal can be formalized as a predicate over the environment:

\[g(s_t) \in \{0,1\}\]
  • The loop has reached its desired terminal state when:
\[g(s_t)=1\]
  • For partially measurable tasks, the goal may instead be expressed through a score and threshold:
\[g(s_t) = \mathbf{1} \left[ J(s_t)\geq \tau \right]\]
  • The quality of the loop cannot exceed the quality of this objective. If the metric captures only a proxy for the real goal, the system can optimize the proxy while degrading the intended outcome. This is why A Technical Roadmap for an Autonomous Loop recommends determining whether a task admits an independent machine check before automating it.

Observation

  • An observation is the subset of the environment made visible during one iteration. It can include:

    • The current code and configuration.
    • Failing tests and stack traces.
    • Recent diffs.
    • Issue descriptions and acceptance criteria.
    • Logs and runtime telemetry.
    • Results produced by upstream graph nodes.
    • Persistent state from previous iterations.
    • Remaining time, token, or monetary budgets.

  • The observation should be sufficient for the current decision but narrow enough to remain relevant. Supplying the entire repository and complete interaction history on every iteration increases cost and allows irrelevant information to obscure the immediate failure.

  • A context builder can therefore be understood as a selection function:

    \[c_t = C \left( s_t, m_t, f_t, b_t \right)\]
    • where the context contains the relevant portion of the environment, persistent memory, current failure, and remaining budget.

  • The selection policy can initially rely on deterministic signals such as:

    • Files named in a stack trace.
    • Files imported by the failing test.
    • Files changed during the previous iteration.
    • Documentation associated with the affected package.
    • The smallest dependency neighborhood containing the failure.

  • More sophisticated retrieval, dependency analysis, or embedding-based selection should be introduced only when simpler rules demonstrably omit necessary evidence. Lost in the Middle: How Language Models Use Long Contexts by Liu et al. (2024) finds that model performance can degrade when relevant information appears in the middle of long contexts, supporting the use of focused contexts rather than indiscriminate context accumulation.

Policy

  • The policy selects the next action from the current context. In an agentic loop, this policy is usually implemented by a language model operating under instructions, tools, and permissions:
\[a_t \sim \pi_{\theta} \left( a \mid c_t \right)\]

  • The action may:

    • Inspect additional state.
    • Modify a file.
    • Run a command.
    • Query an external system.
    • Generate a candidate artifact.
    • Ask a specialized sub-agent for analysis.
    • Escalate to a human.
    • Declare that the current approach is blocked.

  • ReAct: Synergizing Reasoning and Acting in Language Models by Yao et al. (2023) demonstrates how interleaving reasoning, actions, and environmental observations enables an agent to update its plan using tool results rather than relying exclusively on internal generation.

  • The policy should not receive more authority than its task requires. A diagnostic loop may need read access and permission to run tests but not permission to modify code. A repair loop may need to edit an isolated worktree but not deploy the result. Permission design is therefore part of the policy’s effective action space.

Environment Transition

  • An action changes the external state:
\[s_{t+1} = T \left( s_t, a_t \right)\]

  • For a coding loop, this transition might apply a patch, run a migration, or revert an unsuccessful experiment. For a research loop, it might add a source, update a hypothesis ledger, or run an experiment. For an operational loop, it might classify a ticket, update a workflow state, or trigger a service API.

  • The transition should be observable and preferably reversible. Useful mechanisms include:

    • Git commits or isolated branches.
    • Worktrees for parallel modifications.
    • Database transactions.
    • Append-only event logs.
    • Versioned artifacts.
    • Checkpoints and last-known-good states.
    • Explicit rollback operations.

  • A transition that cannot be reconstructed or reversed makes failure analysis substantially harder. The loop should therefore record both the requested action and the environment’s actual response.

Evaluation

  • The evaluator determines whether the transition improved the state and whether the goal has been reached:
\[r_{t+1} = E \left( s_t, a_t, s_{t+1}, g \right)\]

  • Evaluators fall into three broad categories.

    • Deterministic evaluators: Tests, type checkers, compilers, schema validators, numerical thresholds, static analyzers, and invariant checks.

    • Model-based evaluators: Reviewers that assess semantic correctness, completeness, evidence quality, architectural consistency, or adherence to requirements.

    • Human evaluators: Reviewers who decide questions requiring accountability, product judgment, risk acceptance, or interpretation of underspecified goals.

  • Deterministic evaluators should be preferred whenever they capture the requirement faithfully. Model-based evaluators are valuable when quality cannot be reduced to a mechanical test, but their outputs should be treated as probabilistic judgments rather than ground truth. Human evaluation remains necessary where the consequences or objectives cannot be completely delegated.

  • Self-Refine: Iterative Refinement with Self-Feedback by Madaan et al. (2023) shows that model-generated feedback can improve subsequent outputs, while Building effective agents recommends evaluator-optimizer workflows when evaluation criteria are sufficiently clear and iterative refinement creates measurable value.

Memory

  • Memory records information that must survive beyond the current model invocation. It prevents the loop from repeatedly attempting the same failed approach and allows execution to resume after interruption.

  • A robust loop normally maintains several forms of memory:

    • Task memory: The goal, acceptance criteria, constraints, and prohibited actions.

    • Progress memory: Completed work, current phase, unresolved failures, and next actions.

    • Attempt memory: Approaches already tried, their outcomes, and explanations for rejection.

    • Evidence memory: Test logs, benchmark results, diffs, citations, and reviewer verdicts.

    • Operational memory: Iteration counts, budget consumption, timestamps, locks, leases, and retry state.

    • Institutional memory: Project conventions, build procedures, review standards, and known failure modes.

  • Reflexion: Language Agents with Verbal Reinforcement Learning by Shinn et al. (2023) stores verbal reflections in episodic memory so that subsequent attempts can use feedback from prior failures without updating the underlying model weights.

  • Human-readable and machine-readable memory should usually be separated. A Markdown status document is useful for review, while structured state is safer for control decisions:

{
  "phase": "verification",
  "iteration": 4,
  "last_green_revision": "a3f21c8",
  "open_failures": [
    "tests/auth/test_refresh.py::test_expired_token"
  ],
  "blocked_paths": [
    "infrastructure/",
    "tests/fixtures/"
  ],
  "budget_remaining_usd": 6.40
}
  • Control logic should not depend on extracting critical fields from unconstrained prose. Structured state allows the orchestrator to evaluate budgets, detect repeated failures, enforce protected paths, and resume deterministically.

Termination

  • Every loop needs a success exit and at least one safety exit.

  • A complete termination rule can combine:

\[\operatorname{stop}_t = \operatorname{success}_t \lor \operatorname{iterationLimit}_t \lor \operatorname{budgetLimit}_t \lor \operatorname{timeout}_t \lor \operatorname{unsafe}_t \lor \operatorname{stuck}_t \lor \operatorname{humanEscalation}_t\]

  • The success condition answers, “Has the goal been verified?” Safety exits answer, “Should execution stop even though the goal has not been reached?”

  • Common safety exits include:

    • A maximum number of iterations.
    • A token or monetary budget.
    • A wall-clock deadline.
    • Repetition of the same failure.
    • Lack of measurable improvement.
    • Attempts to modify protected artifacts.
    • Tool or infrastructure instability.
    • Detection of conflicting requirements.
    • Expansion beyond the authorized task scope.
    • A condition requiring human judgment.

  • Without explicit failure exits, an autonomous loop can continue consuming resources without producing progress. Getting started with loops recommends explicit turn caps, precise success criteria, small pilots, appropriate model selection, and usage monitoring when running goal-based or proactive loops.

Progress, Convergence, and Fixed Points

Progress Is Not the Same as Activity

  • A loop may generate many actions while making no progress. Reliable systems therefore define a progress measure in addition to a terminal goal.

  • For an objective that should increase, progress during one iteration is:

\[\Delta_t = J(s_{t+1})-J(s_t)\]
  • For a loss that should decrease, progress is:
\[\Delta_t = L(s_t)-L(s_{t+1})\]
  • A positive value indicates measurable improvement under the chosen objective. The loop may accept only improving transitions:
\[s_{t+1} \leftarrow \begin{cases} T(s_t,a_t), & \Delta_t > 0 \\ s_t, & \Delta_t \leq 0 \end{cases}\]
  • This accept-or-revert structure is particularly effective for autonomous experimentation, optimization, and repair. It prevents a failed iteration from silently becoming the new baseline.

Convergence

  • A loop converges when repeated execution reaches a state that satisfies its completion condition or no longer changes materially:
\[s_{t+1} \approx s_t\]

  • A stable state is not necessarily a correct state. The loop may converge to:

    • A valid solution.
    • A local optimum.
    • A repeated failure.
    • A state that exploits the evaluator.
    • A premature completion caused by an incomplete test.
    • An apparently stable result that violates an unstated requirement.

  • Convergence must therefore be interpreted relative to the evaluator and invariants. A loop has not reached a trustworthy fixed point merely because it has stopped changing.

Stagnation and Oscillation

  • Stagnation occurs when repeated actions fail to improve the objective. It may be detected through identical failure signatures, unchanged scores, repeated patches, or repeated reviewer objections.

  • Oscillation occurs when the loop alternates between states:

\[s_t \rightarrow s_{t+1} \rightarrow s_t \rightarrow s_{t+1}\]

  • For example, one iteration may fix a unit test while breaking a type check, and the next may reverse the change. Detection requires state hashes, metric histories, or failure signatures rather than only an iteration counter.

  • Appropriate responses include:

  • Changing the strategy.
  • Expanding or correcting the context.
  • Invoking a different specialist.
  • Restoring the last-known-good state.
  • Splitting the task into smaller phases.
  • Escalating the contradiction to a human.

Determinism, Idempotency, and Repeatability

Deterministic Checks

  • A deterministic evaluator produces the same verdict when run repeatedly against the same state:
\[E(s)=E(s)\]

  • In practice, tests can violate this expectation because of timing, randomness, external dependencies, shared mutable state, network behavior, or data contamination. A flaky evaluator corrupts the loop’s feedback signal. It can cause the system to modify correct code, accept an incorrect result, or oscillate between incompatible interventions.

  • Before using a test as an autonomous stopping condition, it should be repeated against an unchanged state. Instability should be corrected or explicitly modeled before the loop relies on it. A Technical Roadmap for an Autonomous Loop identifies deterministic and idempotent checks as prerequisites for reliable autonomous execution.

Idempotent Actions

  • An action is idempotent when repeating it produces the same externally visible result as executing it once:
\[T \left( T(s,a), a \right) = T(s,a)\]

  • Examples include:

  • Setting a ticket’s status to a particular value.
  • Creating a file only if its content hash differs.
  • Recording a result under a stable experiment identifier.
  • Applying a migration whose completion is tracked transactionally.

  • Sending a request with an idempotency key.

  • Idempotency matters because agent workflows may retry after timeouts, crashes, or uncertain acknowledgments. If a repeated action creates duplicate tickets, commits, deployments, or messages, ordinary recovery behavior becomes destructive.

Stable Identity and Deduplication

  • Every unit of work should have a stable identity derived from its source or semantics. A graph can use this identity to determine whether a result is new, already processed, superseded, or currently in progress.

  • A content-derived identifier may be represented as:

\[\operatorname{id}(x) = H \left( \operatorname{canonicalize}(x) \right)\]
  • Canonicalization must remove irrelevant variation while preserving distinctions that matter. Deduplication based only on free-form text is unreliable because semantically identical findings may be phrased differently.

The Anatomy of a Graph

  • A graph expands a loop by decomposing its work across nodes and expressing dependencies explicitly.

Nodes

  • A node should have one bounded responsibility, a defined input, a defined output, and an observable failure mode.

  • Good node boundaries include:

    • Find uses of a deprecated API.
    • Classify one finding by migration risk.
    • Verify whether a cited line supports a claim.
    • Run a fixed test suite.
    • Deduplicate findings by file and source location.
    • Synthesize verified findings into a migration plan.

  • A node that searches the repository, estimates business impact, implements changes, reviews itself, and writes the final report contains several hidden stages. If it fails, the system cannot easily determine whether the problem originated in discovery, interpretation, implementation, or verification.

Edges

  • An edge represents a real dependency. The downstream node must require information produced by the upstream node.

  • The defining question is:

Does the next node require the previous node’s output?

  • If the answer is yes, the edge is necessary. If the answer is no, the tasks may be independent and eligible for parallel execution. The complete Graph Engineering playbook for Claude Code describes graph engineering as making these dependencies visible and controlling how evidence moves between bounded nodes.

  • Edges should carry structured artifacts rather than conversational summaries. A finding might use:

{
  "finding_id": "AUTH-017",
  "file": "src/auth/session.ts",
  "lines": [84, 119],
  "claim": "The deprecated refresh method is called directly.",
  "evidence": [
    "legacySessionClient.refresh() appears at both locations."
  ],
  "confidence": 0.94,
  "producer": "repository-scanner",
  "verification_status": "unverified"
}
  • This schema lets downstream nodes validate required fields, deduplicate findings, route low-confidence outputs, and preserve provenance.

Control Flow and Data Flow

  • A graph contains two related but distinct structures:

    • Data flow: What information moves from one node to another.

    • Control flow: Which node is permitted to execute next.

  • A router may inspect a finding’s severity and choose a review path. The finding itself is data; the branch decision is control.

  • Keeping these concerns separate improves observability. The system can record both what evidence existed and why a particular path was selected.

Acyclic and Cyclic Regions

  • Many graphs contain a mostly acyclic pipeline with localized feedback loops:
\[\text{Scope} \rightarrow \text{Discover} \rightarrow \text{Reduce} \rightarrow \text{Verify} \rightarrow \text{Synthesize}\]
  • A failed verification may introduce a feedback edge:
\[\text{Verify} \rightarrow \text{Repair} \rightarrow \text{Verify}\]
  • This structure is safer than allowing the entire graph to repeat. Each cyclic region can have its own iteration budget, evaluator, memory, and escalation policy.

Fan-Out and Fan-In

  • Fan-out divides independent work across workers:
\[x \rightarrow \left\{ f_1(x), f_2(x), \ldots, f_n(x) \right\}\]
  • Fan-in collects their results:
\[y = R \left( f_1(x), f_2(x), \ldots, f_n(x) \right)\]

  • The reducer should eliminate duplication while preserving provenance. The fan-out and reduction pattern follows the distributed-computation structure introduced in MapReduce: Simplified Data Processing on Large Clusters by Dean and Ghemawat (2004), where independent mapping operations produce intermediate results that are combined by a reduction stage.

  • Fan-out is appropriate when branches differ by repository region, data partition, evidence source, hypothesis, review criterion, failure mode, and candidate solution.

  • Launching multiple identical workers with the same context and instructions often produces redundant outputs rather than useful diversity.

Barriers

  • A barrier waits for required branches before downstream execution. It defines:

    • Which branches are mandatory.
    • The minimum number of successful results.
    • How long the graph should wait.
    • Which failures may be retried.
    • Whether partial results are acceptable.
    • How incompleteness is represented.

  • A barrier is therefore a reliability boundary, not merely a synchronization primitive. If a mandatory security scan fails, the graph should not silently synthesize a report from the remaining branches and present it as complete.

Routers

  • A router directs work according to structured fields such as task type, confidence, severity, affected subsystem, or estimated blast radius.

  • A typical routing policy might be:

\[\operatorname{route}(x) = \begin{cases} \text{deterministic validation}, & \text{routine and well specified} \\ \text{lightweight review}, & \text{low risk} \\ \text{adversarial panel}, & \text{high risk} \\ \text{human escalation}, & \text{ambiguous or irreversible} \end{cases}\]
  • Routing enables model tiering and selective verification. Simple extraction can use inexpensive models or ordinary code, while architecture decisions and high-impact findings receive deeper analysis.

Reducers and Synthesizers

  • A reducer compresses evidence. A synthesizer turns verified evidence into a coherent result. Combining these roles carelessly can erase provenance.

  • A safe reducer should:

    • Remove exact and semantic duplicates.
    • Preserve stable source identifiers.
    • Retain conflicting findings.
    • Record which items were discarded and why.
    • Avoid converting uncertainty into false certainty.
    • Keep links from aggregated claims to supporting evidence.

  • The synthesizer should operate on the reduced evidence without silently introducing unsupported claims. When possible, every final claim should reference the identifiers of the findings that support it.

Loops Inside Graphs

  • A graph node may itself contain a loop. For example:

    • A discovery node searches until multiple rounds produce no new findings.
    • A repair node modifies code until tests pass or its retry budget expires.
    • A reviewer node requests revisions until all required evidence is present.
    • An optimizer node retains only experiments that improve a metric.

  • The graph coordinates these local loops, but each local loop needs an independent contract:

    • Its input schema.
    • Its output schema.
    • Its completion condition.
    • Its resource budget.
    • Its permitted tools.
    • Its failure and escalation states.

  • This composition permits hierarchical systems in which an outer graph routes work while inner loops perform bounded optimization. Bilevel Autoresearch: Meta-Autoresearching Itself by Qu and Lu (2026) demonstrates a related nested structure in which an inner loop optimizes a task while an outer loop modifies the search mechanisms used by the inner process.

Invariants and Truth Anchors

  • An invariant is a condition that must remain true throughout execution, not only at completion.

  • Examples include:

    • Protected tests must not change.
    • Production credentials must remain inaccessible.
    • Every finding must retain its source reference.
    • No branch may merge before required checks pass.
    • The loop may modify only its assigned worktree.
    • Budget consumption may not exceed the approved limit.
    • A human must approve irreversible actions.

  • Invariants provide truth anchors that the optimizing components cannot redefine. If an agent may alter both the solution and the evaluator, it can satisfy the metric by weakening the measurement rather than improving the solution.

  • A loop’s effective objective is therefore constrained optimization:

\[\begin{aligned} \max_{s} \quad & J(s) \\ \text{subject to} \quad & I_k(s)=1 \qquad \forall k \end{aligned}\]
  • The goal determines what should improve. Invariants determine what the system must not sacrifice while improving it.

The Relationship Between Loop, Graph, Context, and Harness Engineering

  • These engineering disciplines address different failure surfaces:

    • Context engineering: Selects and structures the information visible during a model invocation.

    • Harness engineering: Defines the tools, permissions, memory, evaluators, sandbox, and execution environment surrounding an agent.

    • Loop engineering: Defines how the system observes, acts, evaluates, remembers, and repeats over time.

    • Graph engineering: Defines how multiple loops, agents, tools, scripts, and gates exchange information and coordinate execution.

  • They can be represented as nested concerns:

\[\text{Context} \subset \text{Harness} \subset \text{Loop} \subseteq \text{Graph}\]

  • The nesting is conceptual rather than absolute. A graph node may build a new context, invoke a specialized harness, and run a local loop. The important distinction is the engineering question each layer answers:

    • Context engineering asks, “What should this invocation know?”
    • Harness engineering asks, “What can this agent observe and do?”
    • Loop engineering asks, “How does the system improve across attempts?”
    • Graph engineering asks, “How does work move among specialized components?”

  • The shift from prompting to loop and graph engineering is therefore a shift from authoring instructions to designing an observable, bounded, and verifiable computational process.

Designing Reliable Agent Loops

Begin with a Loop-Suitability Test

  • A loop should be built only when repeated autonomous execution is more valuable than a single agent invocation or a conventional script. The first design step is therefore not implementation, but task selection.

  • A strong loop candidate has five properties:

    • Repeatable work: The same process recurs, or the task requires multiple attempts against changing state.
    • Observable state: The system can inspect the relevant environment before and after an action.
    • Executable actions: The agent has tools that can materially change the environment rather than merely suggest changes.
    • Verifiable outcomes: Tests, metrics, schemas, invariants, or reviewers can determine whether the result improved.
    • Bounded consequences: Permissions, isolation, budgets, and rollback mechanisms constrain failure.

  • Getting started with loops recommends selecting the simplest loop primitive appropriate to the task and reserving more complex autonomous workflows for recurring, well-defined work with explicit verification.

  • A task is unsuitable for unattended looping when:

    • Its objective is primarily subjective and cannot be reviewed incrementally.
    • The required actions are irreversible or high-impact.
    • The environment cannot expose reliable progress signals.
    • Every iteration requires new human judgment.
    • The work is exploratory and its goal is still changing.
    • The expected recurrence does not justify the implementation cost.

  • The suitability decision can be expressed as a conjunction of required conditions:

\[\operatorname{suitable} = \operatorname{observable} \land \operatorname{actionable} \land \operatorname{verifiable} \land \operatorname{bounded}\]
  • A failed condition does not necessarily prohibit agent assistance. It usually means that a supervised agent or fixed workflow is more appropriate than an autonomous loop.

Establish a Reliable Manual Baseline

  • Automation should begin only after the complete task has succeeded at least once under supervision. The manual run establishes whether the proposed tools, context, evaluator, and permissions are sufficient.

  • A Technical Roadmap for an Autonomous Loop recommends completing one reliable manual run and recording its resource use and characteristic failures before introducing autonomous repetition.

  • The baseline should record:

    • Initial environment state.
    • Exact task specification.
    • Context supplied to the agent.
    • Tools and permissions used.
    • Number of model invocations.
    • Input and output token consumption.
    • Wall-clock duration.
    • Number and type of tool calls.
    • Intermediate failures.
    • Final verification evidence.
    • Human interventions.
    • Total monetary cost.

  • The baseline serves three purposes:

    • First, it confirms that one iteration can succeed. A loop cannot compensate for a fundamentally incapable agent or incomplete harness.

    • Second, it provides a cost estimate. If one supervised attempt consumes a known amount, the upper bound for a capped loop can be approximated before launch.

    • Third, it exposes recurring failure modes. These failures should become explicit checks, instructions, or invariants before autonomous execution.

  • If one attempt fails with probability \(p_f\), then the probability that at least one failure occurs across multiple independent attempts is:

\[P(\text{at least one failure}) = 1-(1-p_f)^n\]
  • Agent failures are not fully independent, so the expression is only an approximation. In practice, correlated errors make repeated execution more dangerous because the same misconception may persist across many iterations. Repetition magnifies both capability and systematic error.

Define the Loop Contract

  • Before implementation, the loop should have an explicit contract that separates the task objective from its execution policy.

  • A production loop contract should define:

    • Goal: The terminal state the loop should reach.
    • Input: The task, environment, and artifacts available at startup.
    • Output: The artifact or state transition the loop must produce.
    • Evaluator: The evidence required to establish completion.
    • Invariants: Conditions that must remain true during every iteration.
    • Permissions: Tools, paths, services, and actions the loop may access.
    • Budget: Iteration, time, token, and monetary limits.
    • State protocol: What must be persisted before and after an iteration.
    • Failure policy: Conditions for retry, rollback, escalation, and termination.
    • Ownership boundary: Decisions that remain subject to human approval.

  • A machine-readable contract might use:

name: repair-authentication-tests

goal:
  command: pytest tests/auth
  expected_exit_code: 0

allowed_paths:
  - src/auth/
  - src/session/

protected_paths:
  - tests/
  - infrastructure/
  - .github/

limits:
  max_iterations: 8
  max_runtime_minutes: 45
  max_budget_usd: 12.00
  max_repeated_failure: 2

verification:
  required_commands:
    - pytest tests/auth
    - pytest tests/regression
    - mypy src/auth src/session
  require_clean_protected_diff: true

escalation:
  on_ambiguous_requirement: human
  on_protected_path_change: stop
  on_budget_exhaustion: report
  • The contract should remain outside the model’s editable region. If the optimizing agent can alter its goal, protected paths, or evaluator, those controls are preferences rather than enforcement.

Build the Minimal Loop First

  • The first implementation should contain one worker, one evaluator, one state store, and one hard iteration limit. Scheduling, parallel agents, model panels, and dynamic routing should be added only after this minimal loop behaves reliably.

  • A minimal execution sequence is:

\[\text{Check} \rightarrow \text{Build context} \rightarrow \text{Act} \rightarrow \text{Validate invariants} \rightarrow \text{Evaluate} \rightarrow \text{Persist}\]

  • If the initial check already passes, the loop should terminate without calling the model. This prevents unnecessary work and makes repeated execution idempotent when the goal has already been reached.

  • The following figure (source) shows a metric-driven research loop in which the agent reads the current program, proposes a modification, executes a bounded experiment, and retains only improvements.

  • This accept-or-revert design is useful beyond research. A coding loop can retain a change only if tests improve, a content loop can retain a revision only if it satisfies additional criteria, and an operational loop can commit a transition only if postconditions remain valid.

Prefer Stateless Iterations

  • A reliable long-running loop should normally start each agent invocation with a fresh model context. Progress should reside in files, version control, structured state, and environmental changes rather than in an indefinitely growing conversation.

  • Stateless iteration provides:

    • Stable context size across iterations.
    • More predictable token cost.
    • Reduced exposure to stale reasoning.
    • Easier retry after interruption.
    • Better reproducibility.
    • Cleaner separation between durable evidence and temporary reasoning.
    • Independent reviewer contexts.

  • The loop remains stateful at the system level even when model invocations are stateless:

\[\text{Stateless model calls} + \text{persistent external state} = \text{stateful system}\]

  • A stateful conversation that accumulates a fixed amount of new history during every iteration has approximate input cost:

    \[C_{\text{stateful}} \approx \sum_{t=1}^{n} \left( c_0+t h \right)\]

    • which expands to:

      \[C_{\text{stateful}} \approx n c_0 + \frac{h n(n+1)}{2}\]

  • By contrast, a stateless loop with a bounded state summary has approximate cost:

\[C_{\text{stateless}} \approx n \left( c_0+c_s \right)\]
  • The first structure can grow quadratically with iteration count, while the second remains approximately linear when the context budget is fixed. A Technical Roadmap for an Autonomous Loop identifies stateless iteration as both a context-quality mechanism and a cost-control mechanism.

Construct a Bounded Iteration Context

  • Fresh context does not mean context-free execution. Each invocation should receive a deliberately assembled working set.

  • A useful iteration context contains:

    • Immutable contract: Goal, constraints, permissions, protected paths, and completion criteria.
    • Current machine state: Phase, iteration, remaining budget, and last-known-good revision.
    • Active failure: The first unresolved test, evaluator objection, or blocked condition.
    • Relevant artifacts: Only the files, documents, or records necessary to address that failure.
    • Recent change: The previous patch or action and its measured outcome.
    • Attempt summary: Short descriptions of strategies already tried and why they failed.
    • Required output schema: The exact format expected from the worker.

  • Context construction should obey an explicit budget:

\[\left| c_t \right| \leq B_{\text{context}}\]
  • A practical priority order is:
\[\text{Contract} > \text{Active failure} > \text{Relevant evidence} > \text{Prior attempt summary} > \text{Supplementary context}\]

  • If the budget is exceeded, supplementary material should be omitted before acceptance criteria or failure evidence. Truncation should be recorded so the system knows that the worker operated with incomplete context.

  • A deterministic context builder can begin with stack traces, import relationships, recent diffs, source references, and dependency metadata. Semantic retrieval should be introduced only when these signals do not provide sufficient recall.

Design an Evaluator the Agent Cannot Easily Manipulate

  • The evaluator is the most important component of an autonomous loop. A weak evaluator converts iteration into reward hacking: the system finds the easiest way to satisfy the measurement rather than solving the underlying task.

  • If the objective is only to make tests pass, the agent may discover transformations such as:

    • Deleting assertions.
    • Skipping tests.
    • Broadening exception handlers.
    • Replacing logic with hard-coded expected values.
    • Mocking the behavior being tested.
    • Editing fixtures to remove difficult cases.
    • Narrowing the evaluator’s input.
    • Changing the threshold or completion rule.

  • The correct acceptance condition should combine success, invariant preservation, and regression checks:

\[\operatorname{accept} = \operatorname{primaryPass} \land \operatorname{invariantsHold} \land \operatorname{noRegression} \land \operatorname{evidenceComplete}\]
  • A Technical Roadmap for an Autonomous Loop recommends protecting evaluator artifacts, checking that test files remain unchanged, and using an independent reviewer for failures that deterministic gates cannot detect.

Protect the Evaluator

  • Evaluator protection can be implemented through read-only mounts, filesystem permissions, protected-path diff checks, separate repositories, hidden evaluation cases, cryptographic hashes, independent services, and human-owned acceptance gates.

  • For example:

if ! git diff --quiet -- tests/; then
    echo "Protected evaluator files changed."
    exit 3
fi
  • A stronger check records the baseline hash of every protected artifact and compares it after each action.

Establish Causal Improvement

  • A passing test after an edit does not by itself prove that the edit fixed the intended problem. Stronger verification asks whether the result changed for the expected reason.

  • A causal repair check can require:

    1. The target test fails on the original revision.
    2. The target test passes after the candidate patch.
    3. Protected tests remain unchanged.
    4. Related regression suites pass.
    5. The diff affects code relevant to the failure.
    6. A reviewer can connect the behavioral change to the requirement.

  • This structure reduces false completion caused by unrelated environment changes, stale caches, or evaluator manipulation.

Separate Maker and Checker

  • The worker should not be the only component deciding whether its work is complete. Loop Engineering recommends separating the agent that produces a change from the agent that verifies it, particularly when the loop operates without continuous supervision.

  • A reviewer should receive:

    • The original task.
    • The acceptance criteria.
    • The candidate diff or artifact.
    • Test and evaluator outputs.
    • Relevant source evidence.
    • No unnecessary copy of the worker’s reasoning narrative.

  • The reviewer’s role is to produce a verdict and evidence, not to rationalize or improve the worker’s answer. A useful reviewer instruction is:

Assume the candidate is incorrect until the evidence demonstrates otherwise.

Check:
1. The original requirement is satisfied.
2. The evaluator passed for the intended reason.
3. Protected artifacts were not weakened.
4. No unrelated behavior changed.
5. Every conclusion is supported by observable evidence.

Return PASS or FAIL with concrete evidence.
Do not modify the candidate.
  • A second model does not create an objective oracle. It reduces some self-evaluation bias, but correlated model errors remain possible. Deterministic checks and independent environmental evidence should remain the primary truth anchors.

Use a Two-Level State Protocol

  • Long-running loops benefit from separate human-readable and machine-readable state.

Human-Readable Status

  • The human-facing document should make the run understandable without reconstructing its full trace:
### Goal

Repair authentication refresh behavior without modifying tests.

### Completed

- Reproduced the expired-token failure.
- Located the regression in session renewal.
- Added bounded refresh retry logic.

### Current status

- Authentication tests pass.
- Regression suite fails in session cleanup.

### Next action

Inspect the cleanup fixture and session lifecycle interaction.

### Constraints

- Do not edit tests or infrastructure.
- Do not deploy automatically.

Machine-Readable State

  • The machine state should contain fields required for control:
{
  "run_id": "auth-repair-2026-07-24-001",
  "phase": "regression-verification",
  "iteration": 5,
  "status": "running",
  "last_green_revision": "a3f21c8",
  "current_revision": "d1c842a",
  "active_failure_signature": "session_cleanup::expired_handle",
  "repeat_count": 1,
  "budget_spent_usd": 5.80,
  "budget_remaining_usd": 6.20,
  "protected_paths": [
    "tests/",
    "infrastructure/"
  ]
}

  • State writes should be atomic. A process should write the new state to a temporary file, flush it, and then rename it over the previous state. This prevents a crash from leaving partially written control data.

  • The loop should also maintain an append-only event log. The state file answers, “Where is the loop now?” The event log answers, “How did it get here?”

Isolate the Execution Environment

  • Parallelism and autonomy increase the importance of isolation. A loop should operate in an environment whose maximum possible damage is understood before execution begins.

  • Isolation can be layered:

    • Version-control isolation: A dedicated branch or worktree.
    • Filesystem isolation: Write access limited to task-specific directories.
    • Process isolation: A container or sandbox with resource limits.
    • Network isolation: Outbound access disabled unless required.
    • Credential isolation: Only task-specific, short-lived credentials exposed.
    • Service isolation: Development or staging endpoints instead of production.
    • Evaluator isolation: Tests and verification artifacts mounted read-only.

  • Worktrees prevent concurrent agents from overwriting the same working directory, but they do not isolate secrets, network access, system processes, or external services. Loop Engineering identifies worktrees as the basic isolation primitive for parallel coding agents, while A Technical Roadmap for an Autonomous Loop extends the design to containers, restricted filesystems, and disabled outbound networking.

  • The authorization principle is:

\[\text{Granted capability} = \text{minimum capability required by the task}\]
  • A loop should be defined as much by what it cannot do as by what it can do.

Add Brakes and Circuit Breakers

  • A reliable loop should assume that the worker will eventually become stuck, repeat itself, exceed its intended scope, or encounter an infrastructure failure.

  • Necessary brakes include:

    • Iteration limit: Bounds the number of attempts.
    • Time limit: Bounds wall-clock execution.
    • Token limit: Bounds cumulative model usage.
    • Cost limit: Bounds monetary expenditure.
    • Repeat detector: Stops recurring identical failures.
    • No-progress detector: Stops when metrics remain unchanged.
    • Divergence detector: Stops when failures or regressions increase.
    • Protected-action gate: Stops unauthorized changes.
    • Tool-error threshold: Stops when infrastructure feedback becomes unreliable.
    • Heartbeat timeout: Detects silent process death.

  • A composite stopping policy can be written as:

\[\operatorname{stop} = \operatorname{success} \lor \operatorname{limitReached} \lor \operatorname{stuck} \lor \operatorname{diverging} \lor \operatorname{unsafe} \lor \operatorname{unobservable}\]

  • A repeat detector should compare normalized failure signatures rather than raw logs. Timestamps, temporary paths, request identifiers, and stack addresses can make the same failure appear different.

  • A normalized signature can be computed as:

\[\sigma_t = H \left( \operatorname{normalize}(f_t) \right)\]
  • The loop is stuck when the same signature appears repeatedly without an accepted state transition:
\[\operatorname{stuck} = \mathbf{1} \left[ \sigma_t = \sigma_{t-1} = \cdots = \sigma_{t-k} \right]\]
  • Stopping is preferable to allowing repeated failures to consume the remaining budget. The terminal report should explain the blocker and preserve sufficient evidence for human continuation.

Make Every Iteration Observable

  • A loop that cannot explain its behavior cannot be trusted to run unattended. Observability should be designed before scheduling.

  • Each iteration should emit a structured event containing:

{
  "timestamp": "2026-07-24T21:10:00-07:00",
  "run_id": "auth-repair-2026-07-24-001",
  "iteration": 5,
  "event": "evaluation_completed",
  "state_before": "a3f21c8",
  "state_after": "d1c842a",
  "failure_signature": "session_cleanup::expired_handle",
  "accepted": false,
  "input_tokens": 8420,
  "output_tokens": 1290,
  "cost_usd": 0.74,
  "duration_seconds": 138,
  "details": {
    "target_tests_passed": true,
    "regression_tests_passed": false,
    "protected_paths_unchanged": true
  }
}
  • The event log should support several classes of metrics.

Outcome Metrics

  • Goal completion rate.
  • Verified success rate.
  • Regression rate.
  • Human rejection rate.
  • Rollback rate.
  • Escalation rate.

Efficiency Metrics

  • Iterations per successful run.
  • Tokens per successful run.
  • Cost per successful run.
  • Wall-clock time to completion.
  • Tool calls per iteration.
  • Percentage of iterations accepted.

Reliability Metrics

  • Repeated-failure frequency.
  • Flaky-evaluator frequency.
  • Tool-error rate.
  • Timeout rate.
  • State-recovery success rate.
  • Protected-action violations.

Quality Metrics

  • Reviewer agreement.
  • Evidence completeness.
  • Requirement coverage.
  • Unsupported-claim rate.

  • Post-completion defect rate.

  • A heartbeat should be updated independently of the main status summary. If the heartbeat stops advancing, the supervisor can distinguish a stalled process from a loop that is still reasoning or waiting for a tool.

Budget for the Worst Case

  • The loop should calculate an upper-bound cost before execution:
\[C_{\max} = N_{\max} \left( C_{\text{worker}} + C_{\text{reviewer}} + C_{\text{tools}} \right) + C_{\text{setup}}\]
  • The expected cost is:
\[\mathbb{E}[C] = \sum_{t=1}^{N_{\max}} P \left( \text{iteration } t \text{ executes} \right) C_t\]

  • Cost controls should operate at multiple levels:

    • Per model call.
    • Per iteration.
    • Per loop run.
    • Per scheduled period.
    • Per project or team.

  • Getting started with loops recommends using scripts for deterministic work, routing simpler operations to faster models, piloting on a small task slice, and selecting intervals that match how frequently the external environment changes.

  • Model tiering should assign capability according to node difficulty:

    • Deterministic code for exact transformations.
    • Smaller models for extraction and routine classification.
    • General-purpose models for implementation.
    • Stronger models for ambiguous planning or adversarial review.
    • Humans for accountability and irreversible decisions.

  • The strongest model should not be the default for every iteration. Conversely, a cheap worker that repeatedly fails may cost more than a capable worker that succeeds quickly.

Select the Correct Trigger

  • Loop execution can be initiated in several ways.

Manual Trigger

  • A user starts the loop for a specific task. This is appropriate during development and for infrequent work.

Goal-Based Trigger

  • The loop continues immediately until a completion condition or limit is reached. It is appropriate for bounded tasks with machine-verifiable outcomes.

Time-Based Trigger

  • The loop runs at a fixed interval. It is appropriate when the external system changes over time, such as issue queues, pull-request reviews, or monitoring data.

Event-Based Trigger

  • The loop starts in response to a state transition, such as a failed build, new issue, deployment event, or incoming record. Event-based execution generally reduces unnecessary polling.

Proactive Routine

  • A persistent routine discovers work, prioritizes it, performs bounded actions, and records results. This is appropriate only after its constituent goal-based loops have been validated independently.

  • A schedule should reflect the rate at which meaningful input changes:

\[f_{\text{loop}} \leq f_{\text{meaningful change}}\]
  • Running a loop more frequently than the environment changes wastes resources and may repeatedly process the same state. Stable identities and deduplication remain necessary even with event-driven triggers because delivery can be duplicated.

Reference Implementation

  • The following Python skeleton demonstrates the control structure of a bounded, stateless loop. The worker and evaluator interfaces are intentionally abstract so that the control logic remains independent of a particular model provider.
from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from typing import Protocol


class Worker(Protocol):
    def run(self, context: str) -> None:
        """Apply one candidate change to the isolated environment."""


class Evaluator(Protocol):
    def is_complete(self) -> bool:
        """Return whether the external success condition is satisfied."""

    def failure_report(self) -> str:
        """Return the current normalized failure evidence."""

    def invariants_hold(self) -> bool:
        """Return whether protected conditions remain satisfied."""


@dataclass(frozen=True)
class LoopConfig:
    max_iterations: int
    max_repeated_failure: int
    state_path: Path
    log_path: Path


class LoopFailedError(RuntimeError):
    """Raised when the loop stops without reaching the goal."""


def run_loop(
    worker: Worker,
    evaluator: Evaluator,
    config: LoopConfig,
) -> None:
    last_failure = ""
    repeated_failure_count = 0

    ## Case 1: Avoid model usage when the goal is already satisfied.
    if evaluator.is_complete():
        _append_log(config.log_path, "already_complete", 0, "")
        return

    for iteration in range(1, config.max_iterations + 1):
        ## Case 2: Capture the current external failure before acting.
        current_failure = evaluator.failure_report()

        ## Helper defined below: assemble a small, fresh context for this attempt.
        context = _build_context(
            iteration=iteration,
            failure=current_failure,
            state_path=config.state_path,
        )

        ## Each call starts from fresh model context.
        worker.run(context)

        ## Case 3: Stop immediately if an invariant was violated.
        if not evaluator.invariants_hold():
            _append_log(
                config.log_path,
                "invariant_violation",
                iteration,
                current_failure,
            )
            raise LoopFailedError("A protected invariant was violated.")

        ## Case 4: Finish only when the external evaluator passes.
        if evaluator.is_complete():
            _append_log(
                config.log_path,
                "completed",
                iteration,
                "",
            )
            return

        next_failure = evaluator.failure_report()

        ## Case 5: Detect repeated failure without measurable progress.
        if next_failure == last_failure:
            repeated_failure_count += 1
        else:
            repeated_failure_count = 0

        if repeated_failure_count >= config.max_repeated_failure:
            _append_log(
                config.log_path,
                "stuck",
                iteration,
                next_failure,
            )
            raise LoopFailedError(
                "The same failure repeated without progress."
            )

        ## Helper defined below: atomically persist the latest control state.
        _write_state(
            path=config.state_path,
            iteration=iteration,
            failure=next_failure,
        )

        _append_log(
            config.log_path,
            "iteration_failed",
            iteration,
            next_failure,
        )

        last_failure = next_failure

    ## Case 6: The hard iteration limit is the final circuit breaker.
    raise LoopFailedError("The maximum iteration count was reached.")

  • The omitted helpers should implement:

  • Narrow context construction.
  • Atomic state replacement.
  • Structured append-only logging.
  • Failure normalization.
  • Token and cost accounting.
  • Protected-path validation.

  • Last-known-good restoration.

  • The key property is that completion depends on the evaluator, not on the worker’s self-report.

Promote Autonomy Gradually

  • A loop should advance through increasing autonomy only after evidence supports the transition.

  • A practical progression is:

\[\text{Manual} \rightarrow \text{Supervised loop} \rightarrow \text{Bounded unattended loop} \rightarrow \text{Scheduled loop} \rightarrow \text{Graph node}\]

Manual Phase

  • Run every step interactively. Validate the tools, context, evaluator, and rollback process.

Supervised Loop Phase

  • Allow automatic iteration while a human observes every action. Record failure modes and add missing invariants.

Bounded Unattended Phase

  • Run in an isolated environment with strict budgets and no production authority. Review every final artifact and trace.

Scheduled Phase

  • Trigger the loop automatically only after repeated runs demonstrate stable verification, bounded cost, and successful recovery.

Graph Integration Phase

  • Treat the loop as a node with a defined input schema, output schema, completion condition, and failure contract. Do not integrate an unreliable loop into a larger graph because graph scale will multiply its defects.

  • The purpose of gradual promotion is not merely caution. Each phase generates evidence needed to design the next control boundary. Reliable autonomy is earned through measured behavior, not granted because the worker appears capable.

Graph Topologies and Multi-Agent Orchestration

From a Reliable Loop to an Agent Graph

  • A graph should be introduced only after its constituent loops and nodes have independently demonstrated reliable behavior. A graph does not correct an unreliable worker automatically. It multiplies that worker across more execution paths, increases the number of possible interactions, and makes failures more expensive to diagnose.

  • A graph becomes useful when the work contains:

    • Independent subtasks that can execute concurrently.
    • Specialized decisions requiring different instructions or tools.
    • Large evidence sets that must be reduced before synthesis.
    • Conditional paths based on risk, confidence, or task type.
    • Multiple verification stages.
    • Local feedback loops with different stopping conditions.
    • Tasks too large for one model context or sequential execution path.

  • The complete Graph Engineering playbook for Claude Code defines the central design shift as controlling the path information takes through a system: linear execution, fan-out, reduction, verification, and synthesis.

  • An agent graph can be represented as \(G=(V,E)\), where every node belongs to a set of executable components and every directed edge represents a dependency:

\[e_{ij} = (v_i,v_j,\sigma_{ij})\]
  • The edge schema defines the information transferred from the upstream node to the downstream node. The graph is therefore both a control-flow system and a typed data-flow system.

Draw the Graph Before Implementing It

  • The graph should first be expressed as nodes, edges, and invariants without reference to a particular agent framework.

  • For every proposed step, determine:

    • What input does the step require?
    • What output does it produce?
    • Does it require probabilistic judgment?
    • Can ordinary code perform it exactly?
    • Which downstream nodes consume its output?
    • Can it run independently of neighboring steps?
    • How does it fail?
    • Can it be retried safely?
    • What evidence establishes completion?

  • The critical dependency test is:

Does the downstream node consume information produced by the upstream node?

  • If the answer is no, the edge may be artificial. The tasks can potentially execute concurrently.

  • Consider the instruction:

Inspect the authentication package, inspect the billing package,
and then summarize all migration risks.
  • The two inspections do not depend on each other, but the summary depends on both. The correct topology is:
\[\text{Scope} \rightarrow \begin{cases} \text{Authentication inspection} \\ \text{Billing inspection} \end{cases} \rightarrow \text{Risk synthesis}\]
  • A sequential implementation adds latency without adding correctness:
\[\text{Authentication} \rightarrow \text{Billing} \rightarrow \text{Synthesis}\]
  • A graph exposes this unnecessary dependency and replaces it with parallel work and an explicit convergence point.

Define a Contract for Every Node

  • A node should perform one bounded transformation. Its contract should define:

    • Node identifier and version.
    • Responsibility.
    • Input schema.
    • Output schema.
    • Required tools.
    • Model and reasoning configuration.
    • Permissions.
    • Timeout.
    • Retry policy.
    • Idempotency behavior.
    • Completion condition.
    • Failure states.
    • Cost budget.
    • Logging requirements.

  • A repository-scanning node might declare:

node_id: scan-authentication
version: 1

responsibility:
  Find direct uses of deprecated authentication APIs.

input_schema:
  repository_revision: string
  allowed_paths: array[string]
  deprecated_symbols: array[string]

output_schema:
  findings: array[Finding]
  coverage:
    files_scanned: integer
    files_failed: array[string]
  status: complete | partial | failed

permissions:
  filesystem: read_only
  network: disabled

limits:
  timeout_seconds: 300
  max_input_tokens: 12000
  max_output_findings: 100

retry:
  max_attempts: 2
  retryable_errors:
    - timeout
    - transient_tool_error
  • A bounded node is easier to test, replay, replace, and assign to an appropriate model. If a node performs search, implementation, review, and final synthesis, its internal failure point remains hidden.

Use Typed Edges

  • Free-form conversational handoffs are easy to create but difficult to validate. Structured edges preserve evidence and prevent downstream agents from repeatedly reconstructing upstream meaning.

  • A finding transferred across an edge might use:

{
  "finding_id": "AUTH-017",
  "producer_node": "scan-authentication",
  "repository_revision": "a3f21c8",
  "file": "src/auth/session.ts",
  "start_line": 84,
  "end_line": 91,
  "category": "deprecated-api",
  "severity": "high",
  "confidence": 0.94,
  "claim": "The legacy token refresh method is called directly.",
  "evidence": [
    "legacySessionClient.refresh() is invoked without the migration adapter."
  ],
  "verification_status": "unverified"
}

  • Before the downstream node executes, ordinary code should validate required fields, field types, enumeration values, stable identifiers, source revision, evidence presence, payload size, producer authorization, and schema version.

  • The data accepted by a node should satisfy \(x_i \in \mathcal{X}_i\) and its output should satisfy \(f_i(x_i) \in \mathcal{Y}_i\).

  • Schema validation turns malformed model output into an observable node failure rather than allowing invalid state to propagate through the graph.

Separate Model Nodes from Deterministic Nodes

  • Not every graph node should contain an agent. Models should be reserved for operations requiring semantic judgment, open-ended search, planning, comparison, critique, or synthesis.

  • Ordinary code should perform sorting, deduplication, hashing, arithmetic, schema validation, threshold evaluation, permission checks, exact routing rules, barrier synchronization, state transitions, budget accounting, and retry scheduling.

  • For example, eight workers may return overlapping findings. Deduplication can be performed deterministically:

def deduplicate_findings(
    findings: list[dict[str, object]],
) -> list[dict[str, object]]:
    unique: dict[str, dict[str, object]] = {}

    for finding in findings:
        key = (
            f"{finding['repository_revision']}:"
            f"{finding['file']}:"
            f"{finding['start_line']}:"
            f"{finding['category']}"
        )
        unique[key] = finding

    return list(unique.values())
  • A model can subsequently group semantically related findings or estimate operational impact, but it should not be asked to perform exact transformations that code can complete more cheaply and consistently.

Linear Pipelines

  • The simplest graph topology is a chain:
\[v_1 \rightarrow v_2 \rightarrow \cdots \rightarrow v_n\]

  • A linear pipeline is appropriate when every stage genuinely depends on the preceding output.

  • Examples include:

    • Retrieve evidence, verify evidence, then write a cited answer.
    • Generate a migration plan, approve the plan, then implement it.
    • Parse a document, classify its contents, then route the classification.
    • Produce a candidate patch, run tests, then review the verified diff.

  • Building effective agents describes this pattern as prompt chaining and recommends programmatic gates between stages to prevent invalid intermediate outputs from continuing.

  • The end-to-end success probability of a strict chain is bounded by the reliability of every required node. Under an independence approximation:

\[P_{\text{success}} = \prod_{i=1}^{n} P_i\]

  • As the chain grows, even individually reliable nodes can produce a fragile workflow. Long chains should therefore include checkpoints, deterministic validation, retryable boundaries, and durable intermediate state.

  • A graph should not be constructed as a long chain merely because the task was described using “and then.” Only real data dependencies justify serialization.

Evaluator-Optimizer Loops

An evaluator-optimizer topology alternates between a generator and a reviewer:

\[\text{Generate} \rightarrow \text{Evaluate} \rightarrow \begin{cases} \text{Accept} \\ \text{Revise} \end{cases}\]

A failed evaluation introduces a feedback edge:

\[\text{Evaluate} \rightarrow \text{Revise} \rightarrow \text{Evaluate}\]

The evaluator should return structured feedback:

{
  "verdict": "fail",
  "criteria": {
    "functional_correctness": "pass",
    "regression_safety": "fail",
    "requirement_coverage": "partial"
  },
  "blocking_issues": [
    {
      "criterion": "regression_safety",
      "evidence": "Session cleanup test fails after the patch.",
      "required_change": "Preserve cleanup behavior for expired handles."
    }
  ]
}

The loop should revise only while:

  • The evaluator identifies an actionable defect.
  • The iteration budget remains.
  • The same defect is not repeating.
  • No invariant has been violated.
  • The candidate remains within scope.

The evaluator and optimizer are separate graph roles even if they use the same underlying model family. Fresh context and distinct instructions reduce direct self-rationalization, but deterministic evidence remains necessary.

Fan-Out and Fan-In

Fan-out divides independent work across concurrent workers. Fan-in gathers their results.

\[x \rightarrow \left\{ f_1(x), f_2(x), \ldots, f_n(x) \right\} \rightarrow R \left( f_1(x), \ldots, f_n(x) \right)\]

This topology is appropriate when work can be partitioned by file, module, repository, data shard, evidence source, hypothesis, risk category, review criterion, and candidate solution.

The following figure (source) shows the diamond topology: a shared scope fans out into independent workers, waits at a barrier, and converges through reduction and synthesis.

The classical computational foundation is MapReduce: Simplified Data Processing on Large Clusters by Dean and Ghemawat (2004), which separates parallel mapping from aggregation and provides a model for partitioning work, tolerating worker failures, and reducing intermediate outputs.

Sectioning

Sectioning gives each worker a distinct portion of the task:

\[x = x_1 \cup x_2 \cup \cdots \cup x_n\]

Each worker processes one partition:

\[y_i=f_i(x_i)\]

The reducer combines the outputs:

\[y=R(y_1,y_2,\ldots,y_n)\]

Partition coverage should be explicit. The graph should record which inputs were assigned, completed, skipped, or failed.

Perspective Parallelism

Multiple workers may inspect the same artifact using different criteria, including security reviewer, correctness reviewer, performance reviewer, privacy reviewer, maintainability reviewer, and requirements reviewer.

This is useful when the dimensions are distinct. Merely changing agent names without changing evidence, instructions, tools, or evaluation criteria creates superficial diversity.

Candidate Parallelism

Multiple workers may propose independent solutions to the same problem. A later node compares them using a shared evaluator.

Candidate parallelism is appropriate when:

  • The design space is broad.
  • Local optimization frequently becomes stuck.
  • Different tools or model families have complementary strengths.
  • Selecting among complete candidates is easier than incrementally revising one.

The candidate count should remain bounded because each additional branch increases cost and review load.

Barrier Semantics

A barrier determines when downstream execution may begin. It should not simply wait for every worker indefinitely.

A barrier contract should specify:

  • Mandatory branches.
  • Optional branches.
  • Minimum successful branch count.
  • Maximum wait time.
  • Retryable failures.
  • Acceptable partial coverage.
  • Cancellation policy.
  • Completeness metadata.

Let the set of required workers be:

\[V_{\text{required}}\]

and the successfully completed workers be:

\[V_{\text{complete}}\]

A strict barrier opens only when:

\[V_{\text{required}} \subseteq V_{\text{complete}}\]

A quorum barrier may open when:

\[\left| V_{\text{complete}} \right| \geq q\]

Quorum completion is appropriate only when missing workers do not correspond to mandatory coverage. Three successful reviewers out of five may be sufficient for a voting panel, but three successful repository partitions out of five do not constitute complete repository coverage.

Partial execution should be represented explicitly:

{
  "status": "partial",
  "required_workers": 12,
  "successful_workers": 10,
  "failed_workers": [
    "scan-payments",
    "scan-notifications"
  ],
  "coverage_complete": false
}

The synthesizer must not silently present a partial graph run as complete.

Reduction Before Synthesis

Parallel workers often produce overlapping, inconsistent, or excessively verbose results. These outputs should be reduced before reaching an expensive synthesis node.

Reduction may include:

  • Schema validation.
  • Removal of malformed results.
  • Exact deduplication.
  • Semantic clustering.
  • Conflict detection.
  • Confidence calibration.
  • Evidence ranking.
  • Compression.
  • Preservation of provenance links.

A reducer should not discard disagreement merely because it complicates synthesis. Contradictions are valuable evidence and should be represented explicitly.

A reduced claim might use:

{
  "claim_id": "RISK-004",
  "claim": "Token refresh can fail during rolling migration.",
  "severity": "high",
  "supporting_finding_ids": [
    "AUTH-017",
    "DEPLOY-006",
    "TEST-011"
  ],
  "contradicting_finding_ids": [
    "AUTH-022"
  ],
  "evidence_summary": [
    "Three services call the deprecated refresh path.",
    "The deployment plan mixes old and new token formats.",
    "No rolling-upgrade integration test exists."
  ],
  "verification_status": "requires_adjudication"
}

The synthesizer receives a smaller evidence set without losing the ability to trace a conclusion back to its sources.

Orchestrator-Worker Graphs

An orchestrator-worker topology uses a central planner to dynamically decompose a task:

\[\text{Task} \rightarrow \text{Orchestrator} \rightarrow \left\{ \text{Worker}_1, \ldots, \text{Worker}_n \right\} \rightarrow \text{Synthesis}\]

Unlike static fan-out, the number and type of subtasks are not fully known in advance. The orchestrator determines them from the input.

Building effective agents recommends orchestrator-worker workflows for tasks whose required subtasks cannot be predicted beforehand, including complex repository changes and multi-source research.

AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation by Wu et al. (2024) provides a framework for composing customizable agents, tools, and human inputs through programmable multi-agent conversation patterns.

The orchestrator should output a structured plan:

{
  "plan_id": "migration-audit-001",
  "subtasks": [
    {
      "task_id": "scan-api-usage",
      "worker_role": "repository-scanner",
      "depends_on": [],
      "required": true
    },
    {
      "task_id": "analyze-database-impact",
      "worker_role": "database-reviewer",
      "depends_on": [],
      "required": true
    },
    {
      "task_id": "build-migration-plan",
      "worker_role": "architect",
      "depends_on": [
        "scan-api-usage",
        "analyze-database-impact"
      ],
      "required": true
    }
  ]
}

The runtime, not the orchestrator model, should enforce dependency ordering, concurrency limits, timeout policies, retry budgets, permission boundaries, schema validation, cost limits, and completion state.

The model proposes the graph. Deterministic orchestration executes and constrains it.

Routing Graphs

Routing selects a specialized path according to structured properties of the input.

\[\operatorname{route}(x) = v_j\]

where the selected node depends on task type, risk, confidence, cost, or other measurable features.

A risk-based router might use:

\[\operatorname{route}(x) = \begin{cases} \text{automatic path}, & r(x)<\tau_1 \\ \text{specialist review}, & \tau_1\leq r(x)<\tau_2 \\ \text{human approval}, & r(x)\geq\tau_2 \end{cases}\]

Routing can be implemented using deterministic rules, a conventional classifier, a language-model classifier, and a hybrid policy.

Deterministic routing is preferable when the categories are explicit. Model-based routing is appropriate when classification requires semantic interpretation, but the router should return confidence and evidence.

A low-confidence classification should not be forced into a branch:

{
  "route": "human-triage",
  "confidence": 0.48,
  "reason": "The change affects both authentication and billing ownership."
}

Routing supports cost-aware model selection. Routine work can proceed through inexpensive paths, while unusual or high-impact cases receive stronger models and deeper verification.

Reviewer Panels and Voting

A reviewer panel sends the same candidate to multiple evaluators. This topology is appropriate when no single deterministic evaluator captures the complete requirement.

Reviewers should differ by evaluation criterion, model family, available tools, evidence source, risk orientation, prompt framing, and context selection.

A panel of nominally different agents using the same model, context, instructions, and evidence is likely to produce correlated judgments.

A simple majority rule accepts when:

\[\sum_{i=1}^{n} \mathbf{1} \left[ v_i=\text{approve} \right] > \frac{n}{2}\]

However, majority voting assumes comparable reviewer quality and sufficiently independent errors. High-impact systems should use weighted or veto-based rules.

A weighted rule can be expressed as:

\[\operatorname{accept} = \mathbf{1} \left[ \sum_{i=1}^{n} w_i a_i \geq \tau \right]\]

A veto rule blocks acceptance when any mandatory specialist rejects:

\[\operatorname{accept} = \operatorname{majorityApprove} \land \neg \operatorname{mandatoryVeto}\]

For example, a security rejection may block deployment regardless of general reviewer consensus.

AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors by Chen et al. (2023) studies collaborative groups of language-model agents and highlights both performance gains and emergent social behaviors that can help or impair collective problem solving.

Layered Agent Graphs

A layered graph passes the outputs of one agent group to another group:

\[L_1 \rightarrow L_2 \rightarrow \cdots \rightarrow L_k\]

Within each layer, multiple agents operate concurrently. Downstream agents use upstream proposals as additional evidence.

Mixture-of-Agents Enhances Large Language Model Capabilities by Wang et al. (2025) uses layered groups of language-model agents in which each layer conditions on outputs from the previous layer, demonstrating how heterogeneous model responses can be aggregated and refined.

A layered topology may use proposal layer, critique layer, revision layer, adjudication layer, and a final synthesis layer.

This design is effective for reasoning or generation tasks but can become expensive because every layer multiplies the number of model calls and the amount of information crossing edges.

If every agent in the next layer receives every output from the preceding layer, the communication volume grows approximately as:

\[M = \sum_{\ell=1}^{k-1} n_{\ell}n_{\ell+1}\]

where each layer contains a number of agents. Reduction between layers can prevent context and cost from growing unnecessarily.

Role-Based Conversational Graphs

Some multi-agent systems coordinate through conversation rather than explicit task-result edges. Each agent receives a role and communicates until a completion rule is met.

CAMEL: Communicative Agents for “Mind” Exploration of Large Scale Language Model Society by Li et al. (2023) introduces role-playing communication between agents, using inception prompting to maintain task alignment and role consistency during autonomous cooperation.

Conversational graphs are useful for negotiation, requirements clarification, adversarial debate, collaborative planning, and iterative design critique.

They are less appropriate for deterministic orchestration because conversational state is harder to validate, replay, and deduplicate than structured artifacts.

A production conversational graph should still enforce maximum message count, turn ownership, allowed recipients, message schema, tool permissions, termination criteria, conversation summary checkpoints, and human escalation conditions.

Unbounded agent conversation can consume resources without producing a state transition. Conversation should be treated as a mechanism for producing a structured result, not as the result itself.

Discovery and Convergence Graphs

Some tasks have an unknown number of relevant items. A discovery graph repeatedly searches until it stops finding new evidence.

A basic discovery loop maintains a global seen set:

\[S_{t+1} = S_t \cup D_t\]

New findings are:

\[N_t = D_t \setminus S_t\]

The graph may terminate after a configured number of dry rounds:

\[\operatorname{stop} = \mathbf{1} \left[ N_t=N_{t-1}=\varnothing \right]\]

The implementation must deduplicate against every previously observed item, not only accepted findings. Otherwise, rejected or uncertain findings can repeatedly re-enter the graph.

A discovery state might contain:

{
  "round": 6,
  "seen_finding_ids": [
    "AUTH-001",
    "AUTH-002",
    "DEPLOY-004"
  ],
  "new_findings_this_round": 0,
  "consecutive_dry_rounds": 2,
  "max_rounds": 10,
  "status": "converged"
}

The convergence rule should be combined with a hard iteration limit and cost budget. A graph can fail to converge when new low-value variations continue appearing indefinitely.

Hierarchical and Nested Graphs

Large systems often contain graphs within graph nodes. An outer graph may coordinate business phases while inner graphs execute specialized work.

For example:

\[\text{Discover} \rightarrow \text{Plan} \rightarrow \text{Implement} \rightarrow \text{Verify} \rightarrow \text{Approve}\]

The implementation node may itself contain:

\[\text{Partition files} \rightarrow \text{Parallel worktrees} \rightarrow \text{Test patches} \rightarrow \text{Select candidates}\]

Hierarchical graphs improve modularity when every subgraph has a clear contract. They become difficult to operate when inner graphs expose conversational details or mutable internal state directly to their parents.

A subgraph should appear to its parent as a bounded node with input schema, output schema, completion status, coverage metadata, evidence references, cost report, and a failure report.

The parent should not need to reconstruct the subgraph’s internal execution trace to determine whether it succeeded.

Static and Dynamic Orchestration

Static Graphs

A static graph is defined before execution. Its nodes and edges are known, although routing decisions and retries may vary.

Static graphs provide predictable cost, easier testing, clear authorization review, stable observability, simpler replay, and better capacity planning.

They are appropriate for recurring workflows with well-understood stages.

Dynamic Graphs

A dynamic graph is generated or expanded during execution. An orchestrator may create new workers, branches, or discovery rounds according to intermediate results.

Dynamic graphs provide flexibility for tasks whose decomposition is not known in advance, but they require stronger constraints.

  • The runtime should limit:

    • Maximum generated nodes.
    • Maximum graph depth.
    • Maximum fan-out.
    • Maximum concurrent agents.
    • Permitted node types.
    • Permitted tools.
    • Total budget.
    • Allowed edge schemas.
    • Maximum nested-loop iterations.

Graph Engineering: build 1000+ agent loops in one window emphasizes that graph scale comes from moving orchestration into executable workflow code, while the agents themselves still incur substantial model usage.

Dynamic generation does not transfer enforcement to the model. The model may propose topology, but a trusted runtime must validate and execute it.

Concurrency and Backpressure

Concurrency reduces latency only when tasks are independent and sufficient resources exist.

If independent nodes have latencies:

\[L_1,L_2,\ldots,L_n\]

then sequential latency is:

\[L_{\text{sequential}} = \sum_{i=1}^{n}L_i\]

Ideal parallel latency is:

\[L_{\text{parallel}} = \max_{1\leq i\leq n}L_i\]

Actual latency also includes scheduling, queueing, model-rate limits, tool contention, and reduction:

\[L_{\text{actual}} = L_{\text{critical path}} + L_{\text{queue}} + L_{\text{coordination}}\]

Unbounded fan-out can overwhelm model quotas, tool APIs, repository resources, database connections, reviewer capacity, token budgets, and merge processes.

A concurrency semaphore should bound active workers:

import asyncio
from collections.abc import Awaitable, Callable
from typing import TypeVar

T = TypeVar("T")


async def run_bounded(
    jobs: list[Callable[[], Awaitable[T]]],
    max_concurrency: int,
) -> list[T]:
    semaphore = asyncio.Semaphore(max_concurrency)

    async def run_one(job: Callable[[], Awaitable[T]]) -> T:
        async with semaphore:
            return await job()

    return await asyncio.gather(*(run_one(job) for job in jobs))

Backpressure should propagate upstream. If reviewers cannot process new candidates, the graph should pause candidate generation rather than allowing an unbounded queue to grow.

Retry Semantics and Idempotency

Node retries must distinguish between execution failure and uncertain completion.

A worker may:

  • Fail before taking action.
  • Complete the action but fail before acknowledging it.
  • Return malformed output.
  • Time out while an external action continues.
  • Partially modify the environment.

Every state-changing node should use a stable operation identifier:

{
  "operation_id": "migration-audit-001:scan-authentication:attempt-1",
  "node_id": "scan-authentication",
  "input_revision": "a3f21c8"
}

Before performing a side effect, the node should determine whether that operation has already completed. Retries should reuse the same idempotency key when they represent the same logical action.

The graph should aim for exactly-once effects rather than assuming exactly-once execution:

\[\text{At-least-once execution} + \text{idempotency} + \text{deduplication} = \text{effectively-once side effects}\]

Model generation can usually be repeated safely, but messages, deployments, database mutations, pull requests, and financial actions require stronger idempotency controls.

Isolate Parallel Code-Editing Agents

Parallel workers should not modify the same working directory. Each code-editing branch should receive an isolated worktree or equivalent checkout.

A worktree assignment should include:

{
  "worker_id": "repair-authentication",
  "base_revision": "a3f21c8",
  "branch": "agent/repair-authentication",
  "worktree": "/worktrees/repair-authentication",
  "allowed_paths": [
    "src/auth/",
    "src/session/"
  ]
}

After workers complete, a merge node should:

  1. Verify every candidate independently.
  2. Compare modified paths.
  3. Detect semantic and textual conflicts.
  4. Rank or select patches.
  5. Rebase selected changes onto a common revision.
  6. Run combined regression tests.
  7. Reject the merge if combined behavior differs from isolated behavior.

Two patches can pass independently and fail when combined. Verification must therefore occur both before and after integration.

Graph-Level State

Each node has local state, while the graph requires global orchestration state.

Graph state should record:

  • Graph run identifier.
  • Graph definition and version.
  • Input artifact versions.
  • Node states.
  • Edge payload references.
  • Worker attempts.
  • Current budgets.
  • Active leases.
  • Barrier states.
  • Routing decisions.
  • Accepted and rejected outputs.
  • Human approvals.
  • Final disposition.

Node states should use a controlled state machine:

\[\text{Pending} \rightarrow \text{Ready} \rightarrow \text{Running} \rightarrow \begin{cases} \text{Succeeded} \\ \text{Failed} \\ \text{Timed out} \\ \text{Cancelled} \end{cases}\]

A node becomes ready only when its dependency condition is satisfied. A succeeded node should not execute again unless its input version changes or an explicit replay is requested.

The graph should persist state after every transition so execution can resume following a process failure.

Graph-Level Cost Control

Graph cost depends on node count, model tier, retry count, context size, and loop repetition:

\[C_{\text{graph}} = \sum_{v_i\in V} \sum_{a=1}^{A_i} C(v_i,a)\]

where each node may execute multiple attempts.

A more complete budget constraint is:

\[\sum_{v_i\in V} \left( C_{\text{input},i} + C_{\text{output},i} + C_{\text{tools},i} \right) \leq B_{\text{graph}}\]

The orchestrator should reserve budget for downstream verification and synthesis. Allowing upstream discovery to consume the entire budget can leave the graph with extensive unverified evidence and no resources to adjudicate it.

A budget allocator may reserve:

\[B_{\text{graph}} = B_{\text{discovery}} + B_{\text{verification}} + B_{\text{synthesis}} + B_{\text{reserve}}\]

The reserve supports retries, unexpected tool failures, and human-requested follow-up. When the reserve is exhausted, the graph should stop expanding and produce a partial-status report.

Failure Containment

A graph should prevent one failed branch from corrupting unrelated branches or the final output.

Containment mechanisms include:

  • Per-node permissions.
  • Per-node worktrees.
  • Immutable edge payloads.
  • Schema validation.
  • Transactional state updates.
  • Retry budgets.
  • Circuit breakers.
  • Barrier completeness checks.
  • Explicit partial-result states.
  • Independent final verification.

A node failure policy should distinguish:

  • Retryable infrastructure failure.
  • Non-retryable validation failure.
  • Insufficient evidence.
  • Invariant violation.
  • Budget exhaustion.
  • Human escalation.
  • Cancellation caused by an upstream failure.

The final result should never collapse these states into a generic “completed” status.

When a Graph Is the Wrong Choice

A graph adds orchestration code, schemas, state transitions, retries, logs, intermediate storage, and new failure modes. It is inappropriate when:

  • One agent can hold the relevant context.
  • The task has no independent branches.
  • The work is small or infrequent.
  • Tight human steering is required.
  • The task remains exploratory.
  • Every step genuinely depends on the previous result.
  • The cost of a wrong answer is low.
  • The coordination overhead exceeds the expected benefit.

Building effective agents recommends increasing agentic complexity only when simpler approaches fail to meet measured requirements.

The decision criterion is not whether multiple agents are available. It is whether the work has a topology that benefits from decomposition:

\[\operatorname{useGraph} = \operatorname{parallelism} \lor \operatorname{specialization} \lor \operatorname{conditionalRouting} \lor \operatorname{multiStageVerification}\]

A graph is valuable when it makes real dependencies, evidence flow, cost, and failure boundaries visible. The number of agents is an implementation detail. The topology is the engineering artifact.

Verification, Evaluation, and Termination

Verification as the Control Boundary

An autonomous loop is reliable only when execution is governed by evidence rather than by the agent’s assertion that the task is complete. The generator proposes actions and artifacts, while the verifier determines whether observable system state satisfies an independently defined contract. Getting started with loops describes this distinction operationally: the agent continues working until an explicit stop condition is satisfied, rather than stopping solely because it has produced a plausible response.

Let the task contract contain a collection of acceptance predicates:

\[\mathcal{C} = \left\{ c_1, c_2, \dots, c_m \right\}\]

For an artifact produced at iteration \(t\), completion can be defined as:

\[\operatorname{complete} \left( y_t \right) = \bigwedge_{j=1}^{m} c_j \left( y_t, s_t, e_t \right)\]

where:

  • \(y_t\) is the candidate artifact.
  • \(s_t\) is the externally observed environment state.
  • \(e_t\) is the evidence collected by the verifier.
  • Each \(c_j\) represents one independently testable requirement.

The verifier should inspect effects in the environment, not merely the agent’s explanation of those effects. A code-editing agent may claim that a defect has been fixed, but the verifier should inspect the diff, execute the relevant tests, check for regressions, and confirm that the requested behavior is present. SWE-bench: Can Language Models Resolve Real-World GitHub Issues? by Jimenez et al. (2023) operationalizes software-agent evaluation through repository modifications tested in executable environments rather than through textual claims of success.

This control boundary creates four distinct outcomes:

  • Accepted: All mandatory predicates pass with valid evidence.
  • Rejected: At least one predicate demonstrably fails.
  • Indeterminate: The verifier lacks sufficient evidence to decide.
  • Blocked: Verification cannot proceed because the environment, tool, dependency, or permission is unavailable.

Indeterminate and blocked results must not be converted into acceptance. Treating missing evidence as success is one of the most dangerous failure modes in an unattended workflow.

Separate Generation from Evaluation

The maker and checker should be separate logical components even when both use the same underlying model family. Separation reduces the probability that assumptions, omissions, and framing errors from generation are copied directly into evaluation. CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing by Gou et al. (2023) shows that external tools provide materially stronger corrective signals than unsupported model self-reflection.

The following figure (source) shows a goal-conditioned loop in which the working model produces changes, a separate evaluator checks the completion condition, and execution resumes when the gate rejects the result.

Separation can be implemented at several levels:

  • Prompt separation: Use distinct generator and evaluator instructions.
  • Context separation: Prevent the evaluator from inheriting the generator’s persuasive narrative or unsupported conclusions.
  • Model-instance separation: Execute evaluation in a fresh model invocation.
  • Tool separation: Give the evaluator read-only access to authoritative tools and outputs.
  • Permission separation: Prevent the generator from modifying tests, rubrics, policy files, or reference answers.
  • Ownership separation: Assign high-risk acceptance decisions to a different agent or human reviewer.

A fresh evaluator should receive the task contract, candidate artifact, relevant environment state, and raw evidence. It should not receive unnecessary chain-of-thought, self-assigned confidence, or claims such as “all requirements are satisfied,” because these can anchor the judgment.

Build a Layered Verification Stack

No single evaluator is reliable across every failure class. Verification should therefore be implemented as a sequence of increasingly semantic gates, with inexpensive deterministic checks preceding costly model-based evaluation. Evaluation best practices recommends task-specific, continuously executed evaluations that combine automated measurements with calibrated human judgment.

A practical stack contains the following layers.

Structural Validation

Structural validation checks whether the artifact is well-formed and whether required components exist.

Examples include:

  • JSON-schema validation.
  • Required-file checks.
  • Type checking.
  • Configuration parsing.
  • Database constraint validation.
  • API response-schema validation.
  • Markdown-link and citation-format checks.
  • Graph node and edge contract validation.

These checks are fast, reproducible, and difficult for the agent to reinterpret. They should run before semantic evaluation.

Static Analysis

Static analysis detects violations without executing the artifact.

For code, this can include:

  • Compiler diagnostics.
  • Linters.
  • Type checkers.
  • Dependency scanners.
  • Secret scanners.
  • Policy-as-code checks.
  • Abstract syntax tree inspections.
  • Forbidden API or permission checks.

Static checks are necessary but not sufficient. A program may compile and still implement the wrong behavior.

Executable Behavioral Tests

Behavioral tests evaluate the artifact in a controlled environment.

Examples include:

  • Unit tests.
  • Integration tests.
  • End-to-end tests.
  • Property-based tests.
  • Contract tests.
  • Browser automation.
  • Database migration rehearsal.
  • Reproduction scripts for reported defects.

WebArena: A Realistic Web Environment for Building Autonomous Agents by Zhou et al. (2023) evaluates agents through functional task completion in reproducible web environments, demonstrating why interaction-level success must be measured through environment state rather than response fluency.

Semantic Evaluation

Semantic evaluation checks requirements that cannot be reduced to deterministic assertions, such as completeness, factual support, usability, coherence, or alignment with an open-ended specification.

This layer may use reference-based scoring, rubric-based model judgment, pairwise comparison, evidence-grounded review, specialist reviewer agents, and human evaluation.

Semantic judgment should run only after structural and behavioral gates pass. Otherwise, an articulate evaluator may assign a high score to an artifact that does not compile, cannot execute, or violates a hard constraint.

Regression and Adversarial Evaluation

The final layer checks whether the candidate introduces new failures or exploits weaknesses in the evaluator.

This layer should include:

  • Previously observed failure cases.
  • Boundary conditions.
  • Mutation tests.
  • Hidden or held-out tests.
  • Adversarial inputs.
  • Permission-abuse checks.
  • Prompt-injection tests.
  • Counterfactual cases.
  • Tests for unintended side effects.

A candidate is accepted only when mandatory lower-level gates pass and the aggregate semantic score exceeds its threshold:

\[A_t = \mathbb{I} \left[ \bigwedge_{j \in \mathcal{H}} c_j \left( y_t \right) \right] \cdot \mathbb{I} \left[ Q \left( y_t \right) \geq \tau \right]\]

where \(\mathcal{H}\) is the set of hard requirements, \(Q\) is the semantic quality score, and \(\tau\) is the acceptance threshold. A high semantic score cannot compensate for a failed security, correctness, or policy gate.

Prefer Deterministic Verifiers Where Possible

Deterministic verifiers should be the default for conditions that can be expressed mechanically. They are reproducible, inexpensive, auditable, and resistant to stylistic persuasion.

Suitable deterministic checks include:

  • Whether a command exits successfully.
  • Whether required tests pass.
  • Whether an endpoint returns the expected schema.
  • Whether a database invariant holds.
  • Whether a file exists.
  • Whether a metric has improved relative to a baseline.
  • Whether all graph branches produced schema-valid outputs.
  • Whether citations resolve.
  • Whether a deployment remained within an error budget.

A verifier should return a structured result rather than unstructured terminal output:

{
  "verdict": "reject",
  "checks": [
    {
      "name": "unit_tests",
      "passed": true,
      "evidence_uri": "artifacts/test-results.xml"
    },
    {
      "name": "type_check",
      "passed": false,
      "evidence_uri": "artifacts/type-check.log",
      "failure_summary": "Three incompatible return types"
    }
  ],
  "retryable": true,
  "recommended_focus": "Resolve the return-type violations"
}

This representation allows the controller to distinguish a candidate failure from an infrastructure failure. It also prevents the next iteration from receiving an enormous undifferentiated log.

The verifier process should use a stable exit-code protocol:

  • Exit successfully only when verification completed and all mandatory checks passed.
  • Return a distinct code for candidate rejection.
  • Return a distinct code for verifier or infrastructure failure.
  • Record timeouts and cancellations separately.
  • Persist raw logs outside the model context.

Tools that provide execution hooks can enforce these gates at control-flow boundaries. Hooks reference documents how hook results can block tool execution, prevent task completion, or prevent an agent from stopping when an external check rejects the current state.

Use Model-Based Judges for Irreducibly Semantic Criteria

Some requirements cannot be completely represented by tests. A research synthesis may need to be comprehensive and well-supported; a migration plan may need to address operational risks; a user interface may need to follow an ambiguous design intent. Model-based judges are useful in these cases, but they must be treated as probabilistic measurements rather than authoritative truth.

Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena by Zheng et al. (2023) reports that strong model judges can align with human preferences, while also identifying position, verbosity, self-enhancement, and reasoning biases that require explicit mitigation.

A robust judge prompt should contain:

  • The original task contract.
  • A decomposed rubric.
  • Clear score anchors.
  • Mandatory failure conditions.
  • The candidate artifact.
  • Authoritative evidence.
  • A requirement to cite evidence for every judgment.
  • An explicit insufficient-evidence outcome.
  • A machine-readable response schema.

Instead of asking whether an artifact is “good,” define anchored rubric levels:

Criterion: Requirement coverage

0: The requirement is absent or contradicted.
1: The requirement is mentioned but not implemented.
2: The core requirement is implemented with material omissions.
3: The requirement is implemented and supported by evidence.
4: The requirement is fully implemented, tested, and robust to edge cases.

The judge should score criteria independently before producing an aggregate verdict. This limits the halo effect in which a polished artifact receives uniformly high scores.

For important decisions, use repeated or panel evaluation:

\[\widehat{q} = \operatorname{median} \left( q_1, q_2, \dots, q_r \right)\]

The median is generally more robust than the mean when one judge produces an anomalous score. For categorical decisions, acceptance may require a supermajority and the absence of any hard-failure report.

Evaluate the Outcome and the Trajectory

Final-output evaluation alone cannot reveal whether an agent reached the correct result through a safe, efficient, and reproducible process. An agent may accidentally produce a correct answer after unauthorized actions, excessive retries, irrelevant exploration, or reliance on unavailable state.

Agent-as-a-Judge: Evaluate Agents with Agents by Zhuge et al. (2024) extends static output judgment to intermediate agent actions, enabling evaluation of the full task-solving trajectory rather than only its final artifact. AgentBench: Evaluating LLMs as Agents by Liu et al. (2023) similarly evaluates multi-turn reasoning and decision-making across interactive environments.

Trajectory evaluation should examine:

  • Whether each tool call was relevant.
  • Whether the agent respected permissions.
  • Whether observations were interpreted correctly.
  • Whether failed actions were retried appropriately.
  • Whether the agent repeated equivalent actions.
  • Whether branch outputs were used.
  • Whether intermediate evidence supported later conclusions.
  • Whether the agent modified its evaluator.
  • Whether termination occurred for the correct reason.

The outcome score and trajectory score should remain separate:

\[S_{\mathrm{run}} = w_o S_{\mathrm{outcome}} + w_t S_{\mathrm{trajectory}} + w_e S_{\mathrm{efficiency}} + w_s S_{\mathrm{safety}}\]

For high-risk tasks, safety should be implemented as a hard gate rather than as a compensable weighted term.

Evaluate Graphs at Multiple Levels

A multi-agent graph must be evaluated at the node, edge, subgraph, and graph levels. A correct final artifact can hide defective branches, discarded evidence, or an unreliable reducer.

Node-Level Evaluation

Each node should be evaluated against its local contract:

  • Input schema validity.
  • Output schema validity.
  • Local task success.
  • Tool-call correctness.
  • Runtime and token consumption.
  • Retry count.
  • Evidence completeness.
  • Permission compliance.

Node-level measurements make it possible to locate a failure without replaying the entire graph.

Edge-Level Evaluation

Edges should be tested as data contracts:

  • Whether required fields are present.
  • Whether provenance survives transmission.
  • Whether messages are duplicated.
  • Whether stale versions are rejected.
  • Whether payloads exceed context or size limits.
  • Whether private data crosses an unauthorized boundary.

An edge should carry evidence and provenance, not only a prose summary.

Reducer and Synthesizer Evaluation

Reducers should be measured for evidence preservation, duplicate removal, conflict handling, and recall of important findings. Synthesizers should be checked for unsupported additions, omission of minority findings, and misrepresentation of uncertainty.

For a reducer receiving branch findings \(F_1\) through \(F_n\), evidence recall can be represented as:

\[R_{\mathrm{evidence}} = \frac{ \left| F_{\mathrm{critical}} \cap F_{\mathrm{reduced}} \right| }{ \left| F_{\mathrm{critical}} \right| }\]

A reducer that produces a short, elegant summary with low critical-evidence recall is not successful.

Graph-Level Evaluation

The graph-level evaluator should measure:

  • End-to-end task success.
  • Critical-path latency.
  • Total cost.
  • Parallel-efficiency gain.
  • Branch failure rate.
  • Partial-result handling.
  • Retry amplification.
  • Synthesis faithfulness.
  • Completion-reason accuracy.

The critical path should be measured separately from total work because parallel graphs may perform substantial aggregate computation while keeping wall-clock latency low.

Defend Against Reward Hacking and Evaluator Gaming

Once an agent repeatedly optimizes against a verifier, the verifier becomes part of the environment the agent can exploit. The system may learn to improve the measured score without improving the intended outcome.

Categorizing Variants of Goodhart’s Law by Manheim and Garrabrant (2018) distinguishes several ways in which optimized proxies diverge from their intended objectives, explaining why stronger optimization pressure increases the importance of metric design.

Common forms of evaluator gaming include:

  • Weakening or deleting tests.
  • Changing fixtures so failures disappear.
  • Hard-coding expected outputs.
  • Suppressing error reporting.
  • Optimizing formatting that influences a model judge.
  • Producing verbose justifications that obscure missing evidence.
  • Avoiding difficult cases.
  • Reclassifying failures as skipped.
  • Repeatedly sampling until a favorable judge score appears.

Defenses should include:

  • Read-only evaluator files.
  • Held-out tests.
  • Immutable reference data.
  • Independent test generation.
  • Mutation testing.
  • Comparison against a baseline.
  • Review of evaluator changes.
  • Limits on repeated judgment requests.
  • Periodic human audits.
  • Multiple metrics with hard invariants.

The most important evaluator assets should live outside the generator’s writable environment. If an agent must modify tests as part of legitimate work, those changes should be reviewed by a separate gate before the revised tests are allowed to evaluate the implementation.

Maintain an Evaluation Dataset

A production loop needs a stable evaluation dataset rather than a handful of demonstrations. The dataset should represent normal tasks, edge cases, adversarial cases, historical failures, and operational incidents. Evaluate agent workflows recommends moving from individual traces to repeatable datasets and evaluation runs when comparing changes over time.

A useful dataset contains:

  • Task input.
  • Initial environment state.
  • Expected observable outcome.
  • Hard constraints.
  • Allowed tools and permissions.
  • Reference evidence.
  • Grading rubric.
  • Known failure modes.
  • Maximum budget.
  • Expected termination category.

Partition the data into distinct roles:

  • Development set: Visible cases used for rapid iteration.
  • Regression set: Stable cases that prevent recurrence of known failures.
  • Held-out set: Hidden cases used to detect evaluator overfitting.
  • Canary set: Small, high-value set executed continuously.
  • Adversarial set: Cases constructed to probe unsafe or deceptive behavior.
  • Production-derived set: Sanitized examples sampled from real traces.

Historical failures should be converted into regression tests. This turns operational incidents into durable improvements to the harness rather than lessons that must be rediscovered by each future run.

Measure More Than Task Success

Task success is necessary but insufficient. A system that succeeds occasionally through excessive retries, high cost, or unsafe actions is not production-ready.

Outcome Metrics

Useful outcome metrics include:

\[\operatorname{successRate} = \frac{ \text{successful runs} }{ \text{eligible runs} }\] \[\operatorname{requirementCoverage} = \frac{ \text{satisfied requirements} }{ \text{applicable requirements} }\] \[\operatorname{regressionRate} = \frac{ \text{previously passing cases that now fail} }{ \text{previously passing cases} }\]

Efficiency Metrics

Efficiency should include:

  • Median and tail latency.
  • Tokens per run.
  • Tool calls per run.
  • Cost per run.
  • Iterations per successful run.
  • Cost per successful run.
  • Parallel utilization.
  • Context-growth rate.

A useful operational metric is:

\[\operatorname{costPerSuccess} = \frac{ \text{total evaluation-period cost} }{ \text{successful runs} }\]

This metric prevents apparent improvements that increase success only by multiplying retries and model calls.

Reliability Metrics

Reliability should include timeout rate, tool-error rate, retry rate, duplicate-action rate, invalid-state-transition rate, partial-graph completion rate, incorrect-termination rate, and recovery success rate.

Evaluator Metrics

The evaluator itself must be evaluated. Track:

  • False acceptance rate.
  • False rejection rate.
  • Agreement with expert reviewers.
  • Score stability across repeated judgments.
  • Sensitivity to candidate order.
  • Sensitivity to verbosity.
  • Calibration across task categories.
  • Percentage of indeterminate cases.

An evaluator with a low overall error rate may still be unacceptable if its false acceptances concentrate in security-sensitive or irreversible tasks.

Define Explicit Termination Categories

Termination should be a controller decision with an explicit reason code. It should not be represented by the absence of another model response.

A loop should terminate under one of the following categories:

  • Goal satisfied: All required gates pass.
  • Iteration budget exhausted: The maximum number of attempts has been reached.
  • Cost budget exhausted: Continuing would exceed the permitted spend.
  • Time budget exhausted: The execution deadline has passed.
  • Stagnation detected: Recent iterations show no material progress.
  • Oscillation detected: The system alternates between previously observed states.
  • Repeated failure detected: The same failure signature persists.
  • Unsafe action requested: A policy or permission gate blocks continuation.
  • Environment blocked: A required external dependency is unavailable.
  • Human review required: The remaining decision exceeds the system’s authority.
  • Cancellation received: An external controller or user cancels the run.

The controller can represent the state transition as:

\[z_{t+1} = \begin{cases} \text{success}, & \operatorname{accept} \left( y_t \right) = 1 \\ \text{stagnated}, & \Delta_t < \epsilon \text{ for } k \text{ iterations} \\ \text{oscillating}, & s_t \in \left\{ s_0, \dots, s_{t-1} \right\} \\ \text{budget-exhausted}, & B_t \leq 0 \\ \text{blocked}, & \operatorname{recoverable} \left( e_t \right) = 0 \\ \text{continue}, & \text{otherwise} \end{cases}\]

Keep Claude working toward a goal illustrates this control pattern through completion conditions and stop hooks that can deterministically reject premature termination.

Detect Stagnation and Oscillation

A loop may remain active without making progress. Counting completed iterations is therefore not a sufficient progress signal.

Define a progress score from independent measurements:

\[P_t = \alpha Q_t + \beta C_t - \gamma F_t - \delta K_t\]

where:

  • \(Q_t\) measures artifact quality.
  • \(C_t\) measures requirement coverage.
  • \(F_t\) measures unresolved failures.
  • \(K_t\) measures resource consumption.

The marginal progress of an iteration is:

\[\Delta_t = P_t - P_{t-1}\]

If progress remains below a minimum threshold across a fixed window, the controller should stop, change strategy, or escalate. The agent should not be allowed to redefine the progress metric during execution.

Oscillation detection requires stable fingerprints for artifacts, failure sets, and plans. If two equivalent states recur, the controller should treat the cycle as evidence that the current policy is not converging.

Possible responses include:

  • Select a different strategy.
  • Increase the level of decomposition.
  • Replace the worker or evaluator.
  • Restore the best-known checkpoint.
  • Request missing information.
  • Escalate to human review.

Preserve the Best-Known Artifact

The latest artifact is not necessarily the best artifact. Every iteration should be compared against the best verified checkpoint rather than only against the immediately preceding state.

Let the best checkpoint at iteration \(t\) be:

\[y_t^{*} = \underset{ y \in \left\{ y_0, \dots, y_t \right\} }{ \operatorname{argmax} } \; Q \left( y \right)\]

A candidate should replace the checkpoint only when:

  • All hard gates still pass.
  • Its quality score improves beyond measurement noise.
  • It introduces no critical regression.
  • Its evidence is complete.
  • The evaluator itself completed successfully.

Rejected candidates may still provide useful diagnostic information, but they should not overwrite the last known-good state.

Make Verification Observable

Every acceptance decision should be reconstructable from persisted evidence. Logs should describe what happened, while traces should preserve the causal relationship between goals, actions, observations, evaluations, and state transitions.

Each iteration should record:

{
  "run_id": "run-1842",
  "iteration": 7,
  "candidate_id": "commit-a63d2c",
  "parent_candidate_id": "commit-f70b11",
  "actions": [],
  "tool_results": [],
  "verification": {
    "verifier_version": "auth-eval-v12",
    "verdict": "reject",
    "failed_checks": ["refresh_token_rotation"],
    "evidence_uris": ["artifacts/run-1842/iteration-7/"]
  },
  "progress_delta": -0.03,
  "cost": 1.84,
  "termination_reason": null
}

The trace should preserve:

  • Prompt and policy versions.
  • Model and tool versions.
  • State snapshots.
  • Candidate identity.
  • Raw verifier outputs.
  • Judge rubrics and scores.
  • Retry ancestry.
  • Graph branch lineage.
  • Costs and latency.
  • Termination reason.

Trace retention should respect privacy and data-minimization requirements. Sensitive tool outputs should be redacted or stored behind stricter access controls rather than copied indiscriminately into the model context.

Use Offline, Pre-Deployment, and Online Evaluation

Evaluation should occur at three operational stages.

Offline Evaluation

Offline evaluation runs candidate loop or graph versions against repeatable datasets. It is used for prompt changes, model upgrades, tool changes, routing changes, and verifier revisions.

A candidate should not be promoted when it:

  • Reduces success on critical categories.
  • Increases false acceptance.
  • Introduces new policy violations.
  • Raises cost beyond the permitted trade-off.
  • Produces unstable results across repeated runs.

Pre-Deployment Evaluation

Before deployment, execute the system in a production-like sandbox using realistic credentials, permissions, timeouts, and failure injection.

This stage should test:

  • Missing dependencies.
  • Rate limits.
  • Partial tool failures.
  • Delayed responses.
  • Duplicate events.
  • Concurrent runs.
  • Cancellation.
  • Rollback.
  • Recovery from checkpoints.

Online Evaluation

Online evaluation measures real production behavior through sampling, shadow execution, canaries, and human review. Production monitoring should detect distribution shifts that are absent from curated offline datasets.

An online rollout should proceed through bounded stages:

  • Shadow execution with no external side effects.
  • Read-only canary tasks.
  • Reversible low-risk actions.
  • Limited production traffic.
  • Broader rollout after stable evidence.

Reference Verification Controller

A minimal verification controller can preserve the distinction between acceptance, candidate rejection, and infrastructure failure:

from dataclasses import dataclass
from enum import Enum
from typing import Callable, Sequence


class Verdict(str, Enum):
    ACCEPT = "accept"
    REJECT = "reject"
    BLOCKED = "blocked"


@dataclass(frozen=True)
class CheckResult:
    name: str
    passed: bool
    retryable: bool
    evidence_uri: str
    summary: str


@dataclass(frozen=True)
class EvaluationResult:
    verdict: Verdict
    checks: tuple[CheckResult, ...]
    score: float | None
    next_focus: str | None


def evaluate_candidate(
    deterministic_checks: Sequence[Callable[[], CheckResult]],
    semantic_judge: Callable[[], float],
    minimum_semantic_score: float,
) -> EvaluationResult:
    results: list[CheckResult] = []

    ## Case 1: Run authoritative deterministic checks first.
    for check in deterministic_checks:
        result = check()
        results.append(result)

        if not result.passed:
            return EvaluationResult(
                verdict=Verdict.REJECT,
                checks=tuple(results),
                score=None,
                next_focus=result.summary if result.retryable else None,
            )

    ## Case 2: Evaluate irreducibly semantic requirements.
    score = semantic_judge()
    if score < minimum_semantic_score:
        return EvaluationResult(
            verdict=Verdict.REJECT,
            checks=tuple(results),
            score=score,
            next_focus="Improve the lowest-scoring semantic criterion.",
        )

    ## Case 3: Accept only after every mandatory layer passes.
    return EvaluationResult(
        verdict=Verdict.ACCEPT,
        checks=tuple(results),
        score=score,
        next_focus=None,
    )

A production implementation should additionally capture verifier exceptions, timeouts, version identifiers, evidence locations, retry budgets, and policy violations. Verifier failure should yield a blocked outcome, never acceptance.

The Verification Principle

Verification is not a final quality-control step added after an autonomous system has been designed. It defines the system’s effective objective, determines what the agent learns from each iteration, controls whether the graph continues, and establishes what the organization is willing to accept.

A reliable loop therefore satisfies three conditions:

\[\operatorname{reliableLoop} = \operatorname{independentEvidence} \land \operatorname{protectedEvaluator} \land \operatorname{boundedTermination}\]

Without independent evidence, repetition amplifies unsupported judgment. Without evaluator protection, optimization produces reward hacking. Without bounded termination, temporary uncertainty becomes uncontrolled cost. Verification is the mechanism that converts repeated model execution into controlled engineering progress.

State, Memory, and Context Management

Externalize State from the Model

A language-model invocation should be treated as a transient computation over an explicitly assembled input. Durable task state, prior decisions, artifacts, policies, and progress must remain outside the model so that a loop can resume after context exhaustion, process failure, deployment, or model replacement. Effective harnesses for long-running agents describes this as the central continuity problem for work that spans multiple agent sessions.

The model should implement the transition policy, while an external state store preserves the transition result:

\[a_t \sim \pi_{\theta} \left( C_t \right)\] \[S_{t+1} = T \left( S_t, a_t, O_{t+1} \right)\]

where:

  • \(S_t\) is the durable system state.
  • \(C_t\) is the bounded context assembled for the current invocation.
  • \(a_t\) is the selected action.
  • \(O_{t+1}\) is the observed result of executing that action.
  • \(T\) is the deterministic state-transition function.

The distinction is architectural. The model proposes a transition, but the controller validates and commits it. A model response should never become authoritative state merely because it contains phrases such as “completed,” “fixed,” or “verified.”

Loop Engineering: A Technical Roadmap for an Autonomous Loop recommends placing progress in files and version control while launching each iteration with a fresh, focused context. This makes the filesystem, database, or repository the continuity mechanism rather than the conversation transcript.

Distinguish State, Memory, Context, and Artifacts

These concepts are related but should not be used interchangeably.

State

State describes the system’s current authoritative condition. It answers questions such as:

  • What phase is active?
  • Which requirements are complete?
  • Which branch is blocked?
  • What is the current iteration?
  • Which checkpoint is accepted?
  • What budgets remain?
  • Why did the previous attempt fail?

State must be structured, versioned, and unambiguous enough for deterministic control logic.

Memory

Memory is retained information that may improve future decisions. It includes prior attempts, lessons, user preferences, reusable procedures, environmental facts, and failure patterns.

Memory is not automatically authoritative. A remembered fact may be stale, incorrectly inferred, superseded, or irrelevant to the current task. It must be retrieved and validated before influencing an action.

Context

Context is the bounded input assembled for one model invocation. Effective context engineering for AI agents defines context engineering as the iterative selection and organization of information placed into the model’s limited working window.

Context may contain selected portions of:

  • Current state.
  • Relevant memory.
  • Task instructions.
  • Tool definitions.
  • Source documents.
  • Recent observations.
  • Active failures.
  • Local artifacts.

Context should be derived from state and memory. It should not become the canonical store for either.

Artifacts

Artifacts are task outputs and supporting evidence, such as source-code changes, reports, test results, build logs, research notes, database migrations, screenshots, deployment manifests, and generated datasets.

Artifacts should be addressable through stable identifiers, hashes, versions, or repository commits. Large artifacts should remain outside the context window, with only their relevant excerpts and references injected into a model call.

The system should therefore preserve the following direction of dependency:

\[\left( \text{state}, \text{memory}, \text{artifacts} \right) \longrightarrow \text{context} \longrightarrow \text{model action}\]

The reverse direction should pass through validation before any model output updates durable state.

Define an Authoritative State Schema

The state schema should contain only fields that influence control flow, recovery, evaluation, permissions, or future decisions. Important fields should not be embedded exclusively in prose.

A practical state document may contain:

{
  "schema_version": 3,
  "run_id": "run-1842",
  "state_version": 17,
  "phase": "implementation",
  "iteration": 8,
  "status": "active",
  "goal_id": "auth-refresh-rotation",
  "active_requirement_ids": ["REQ-04", "REQ-07"],
  "completed_requirement_ids": ["REQ-01", "REQ-02", "REQ-03"],
  "blocked_requirement_ids": [],
  "current_failure": {
    "signature": "test_refresh_rotation::expired_token",
    "evidence_uri": "artifacts/run-1842/test-results.xml",
    "first_seen_iteration": 6,
    "consecutive_occurrences": 2
  },
  "best_checkpoint": {
    "artifact_id": "commit-a63d2c",
    "verification_id": "eval-984",
    "score": 0.91
  },
  "budgets": {
    "iterations_remaining": 12,
    "tokens_remaining": 180000,
    "cost_remaining_usd": 28.50,
    "deadline": "2026-07-26T02:00:00Z"
  },
  "lease": {
    "owner": "worker-04",
    "expires_at": "2026-07-25T23:41:00Z"
  },
  "last_event_id": "evt-881",
  "updated_at": "2026-07-25T23:36:12Z"
}

The schema should distinguish:

  • Control state: Phase, status, iteration, routing decision, and termination reason.
  • Task state: Completed, active, blocked, and deferred requirements.
  • Evaluation state: Failed checks, scores, evidence, and accepted checkpoints.
  • Resource state: Remaining time, cost, token, retry, and concurrency budgets.
  • Coordination state: Node ownership, leases, barriers, and branch completion.
  • Recovery state: Last committed event, checkpoint identity, and resumability information.
  • Policy state: Active permission profile, approval status, and restricted operations.

Each state update should include a schema version and monotonic state version. The schema version supports migration, while the state version detects lost updates and concurrent writers.

Maintain Human-Readable and Machine-Readable State

A long-running system benefits from two complementary representations.

The machine-readable state controls execution. It should use validated JSON, a relational schema, a key-value record, or another deterministic representation.

The human-readable status document explains:

  • The goal.
  • What has been completed.
  • What remains.
  • What is currently failing.
  • Which assumptions were made.
  • Which actions are prohibited.
  • What the next worker should do.
  • Where supporting evidence is stored.

A suitable STATUS.md can follow this structure:

## Goal

Rotate refresh tokens without invalidating active sessions.

## Completed

- Added token-family identifiers.
- Added replay detection.
- Migrated the development database.

## Current failure

`test_expired_refresh_token` returns HTTP 500 instead of HTTP 401.

## Next action

Inspect exception translation in `auth/refresh.py`.

## Constraints

- Do not change public API response schemas.
- Do not modify production infrastructure.
- Do not weaken existing authentication tests.

## Evidence

- Test results: `artifacts/run-1842/test-results.xml`
- Accepted checkpoint: `commit-a63d2c`

Human-readable state should not independently control branching, retries, or termination. Prose may be interpreted differently by different model invocations, so critical fields must remain in the structured state.

Treat State Updates as Transactions

A state transition should be committed only after the action result has been observed and validated.

The correct order is:

  1. Read the current state version.
  2. Acquire ownership or a bounded lease.
  3. Select an action.
  4. Execute the action in an isolated environment.
  5. Capture the observation.
  6. Run required validation.
  7. Persist artifacts and evidence.
  8. Atomically commit the new state.
  9. Release the lease.

The commit should use optimistic concurrency:

\[\operatorname{commit} \left( S_{t+1} \right) = \begin{cases} \text{success}, & \operatorname{version} \left( S_{\mathrm{stored}} \right) = \operatorname{version} \left( S_t \right) \\ \text{conflict}, & \text{otherwise} \end{cases}\]

If the stored version changed after the worker read it, the update should be rejected and recomputed from the latest state. Silent last-writer-wins behavior can erase progress, reintroduce completed work, or corrupt graph barriers.

For file-based state, use an atomic replacement pattern:

  1. Write the complete new state to a temporary file.
  2. Flush the file.
  3. Validate that it parses.
  4. Atomically rename it over the previous state file.
  5. Preserve the earlier version or append the transition to an event log.

The system should never expose a partially written state document.

Combine Snapshots with an Append-Only Event Log

A snapshot provides efficient access to the current state, while an event log records how that state was reached. Event Sourcing explains how an ordered sequence of state-changing events can reconstruct prior states, support temporal queries, and recover from incorrectly ordered or retroactively corrected events.

Let the event sequence be:

\[E_{1:t} = \left( e_1, e_2, \dots, e_t \right)\]

The current state can be reconstructed through a deterministic fold:

\[S_t = \operatorname{fold} \left( T, S_0, E_{1:t} \right)\]

A loop event should record:

{
  "event_id": "evt-881",
  "run_id": "run-1842",
  "parent_event_id": "evt-880",
  "event_type": "verification_failed",
  "state_version_before": 16,
  "state_version_after": 17,
  "actor_id": "verifier-02",
  "artifact_ids": ["commit-b915de"],
  "evidence_ids": ["test-run-443"],
  "payload": {
    "failed_check": "test_refresh_rotation",
    "retryable": true
  },
  "timestamp": "2026-07-25T23:36:12Z"
}

Events should be immutable. Corrections should be represented by new compensating or superseding events rather than rewriting history.

Periodic snapshots prevent recovery from requiring a replay of every event since the beginning of the run:

\[S_t = \operatorname{fold} \left( T, S_k, E_{k+1:t} \right)\]

where \(S_k\) is the most recent verified snapshot.

Design for Checkpointing and Recovery

A checkpoint should represent a state from which execution can safely resume. It is more than a copy of the latest output.

A valid checkpoint should include:

  • State snapshot.
  • Artifact identifiers and hashes.
  • Repository commit or immutable workspace image.
  • Completed requirement set.
  • Verifier results.
  • Environment and dependency versions.
  • Pending external operations.
  • Idempotency keys.
  • Remaining budgets.
  • Active leases or barrier state.
  • Prompt, policy, tool, and model versions.

Effective harnesses for long-running agents recommends leaving explicit progress information and a clean environment for the next session, allowing a fresh agent to resume rather than rediscover the entire task.

A checkpoint should be created:

  • After a verified improvement.
  • Before a high-risk or irreversible action.
  • Before context compaction.
  • Before handing work to another agent.
  • At phase boundaries.
  • Before a scheduled shutdown.
  • After recovering from an incident.

Recovery should begin by validating the checkpoint. If referenced artifacts are missing or hashes do not match, the checkpoint should be marked invalid rather than partially restored.

Define a Memory Taxonomy

Agent memory should be divided according to purpose and lifecycle. Generative Agents: Interactive Simulacra of Human Behavior by Park et al. (2023) combines stored observations, retrieval, reflection, and planning, demonstrating that useful memory requires selection and synthesis rather than indiscriminate transcript retention.

Working Memory

Working memory contains information needed for the current action, including active goal, current failure, immediate plan, relevant file excerpts, and recent tool results.

It belongs in the current context and can usually be discarded after the action is committed.

Episodic Memory

Episodic memory records specific prior experiences:

  • An attempted fix and its result.
  • A failed deployment.
  • A user correction.
  • A tool outage.
  • A reviewer rejection.
  • A successful recovery sequence.

Reflexion: Language Agents with Verbal Reinforcement Learning by Shinn et al. (2023) stores verbal feedback in an episodic memory buffer so later trials can avoid repeating earlier failures.

Semantic Memory

Semantic memory contains consolidated facts that are expected to remain useful across tasks:

  • Repository architecture.
  • API contracts.
  • Domain definitions.
  • User preferences.
  • Service ownership.
  • Known environmental constraints.

These facts require provenance, temporal validity, and update policies.

Procedural Memory

Procedural memory contains reusable instructions:

  • Build and test commands.
  • Deployment procedures.
  • Review checklists.
  • Incident-response workflows.
  • Data-handling rules.
  • Tool-use conventions.

Reusable skills are a form of procedural memory. Equipping agents for the real world with Agent Skills describes progressively loading task-specific instructions rather than placing every procedure in the initial context.

Policy Memory

Policy memory contains permissions and non-negotiable restrictions:

  • Prohibited operations.
  • Approval requirements.
  • Data-access rules.
  • Deployment boundaries.
  • Security controls.

Policy memory should be maintained by trusted owners. An ordinary worker should not be able to rewrite a policy because the policy prevented its preferred action.

The following figure (source) shows the five supporting components of a durable agent loop: automation provides the trigger, skills preserve project knowledge, subagents separate responsibilities, connectors expose external systems, and verification controls acceptance.

Apply a Strict Memory-Write Policy

Allowing an agent to convert every observation into durable memory creates noise, contradiction, and self-reinforcing errors. A memory should be written only when it is expected to improve future decisions.

A write policy can require:

\[\operatorname{write} \left( m \right) = \mathbb{I} \left[ U \left( m \right) \geq \tau_u \right] \cdot \mathbb{I} \left[ V \left( m \right) \geq \tau_v \right] \cdot \mathbb{I} \left[ P \left( m \right) = 1 \right]\]

where:

  • \(U\) measures expected future utility.
  • \(V\) measures verification confidence.
  • \(P\) indicates that storage is permitted.
  • \(\tau_u\) and \(\tau_v\) are minimum utility and verification thresholds.

Each durable memory should include:

  • Memory identifier.
  • Memory type.
  • Normalized content.
  • Source or evidence identifier.
  • Creation time.
  • Validity interval.
  • Confidence or verification status.
  • Scope.
  • Sensitivity classification.
  • Superseded memory identifier.
  • Retention policy.

A useful memory record may look like:

{
  "memory_id": "mem-773",
  "type": "procedural",
  "content": "Run integration tests with the local Redis service enabled.",
  "source": {
    "run_id": "run-1710",
    "evidence_uri": "artifacts/run-1710/test-analysis.md"
  },
  "scope": {
    "repository": "billing-service",
    "branch_pattern": "*"
  },
  "verification_status": "human_approved",
  "valid_from": "2026-06-12T00:00:00Z",
  "valid_until": null,
  "supersedes": "mem-618",
  "sensitivity": "internal",
  "retention": "review_annually"
}

Raw self-reflection should remain episodic until it has been validated and consolidated. A single failed attempt does not automatically justify a permanent procedural rule.

Consolidate Memory Without Losing Provenance

Long-running systems accumulate many observations that describe the same underlying lesson. Memory consolidation should compress these observations into a reusable statement while preserving links to the original evidence.

A suitable pipeline is:

  1. Capture raw observations.
  2. Cluster semantically related observations.
  3. Identify recurring patterns.
  4. Draft a candidate lesson.
  5. Validate the lesson against source events.
  6. Detect conflicts with existing memory.
  7. Approve or reject the candidate.
  8. Store the consolidated memory with provenance.
  9. Retain source links for audit and reversal.

The consolidated memory should not overgeneralize beyond its evidence. If three failures occurred only in one repository and one dependency version, the resulting rule should retain that scope.

Memory consolidation is therefore a lossy transformation that requires evaluation:

\[m^{*} = \operatorname{consolidate} \left( m_1, m_2, \dots, m_n \right)\]

subject to:

\[\operatorname{support} \left( m^{*}, \left\{ m_1, \dots, m_n \right\} \right) \geq \tau_s\]

The source records should remain recoverable when the consolidated statement becomes stale or is later contradicted.

Retrieve Memory by Relevance, Recency, and Importance

Durable memory is useful only when the correct subset is retrieved. Injecting every stored memory into every invocation recreates the context-overload problem.

A retrieval score can combine:

\[R \left( m, q \right) = \alpha \operatorname{similarity} \left( m, q \right) + \beta \operatorname{recency} \left( m \right) + \gamma \operatorname{importance} \left( m \right) + \delta \operatorname{scopeMatch} \left( m, q \right) + \eta \operatorname{authority} \left( m \right)\]

Retrieval should also apply hard filters for:

  • User or tenant.
  • Project.
  • Environment.
  • Time validity.
  • Permission level.
  • Sensitivity.
  • Artifact version.
  • Memory status.
  • Supersession.

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory by Wu et al. (2024) evaluates information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention, showing that memory systems must handle updates and temporal scope rather than only semantic similarity.

When retrieval confidence is low, the agent should query authoritative state or abstain. It should not silently replace missing evidence with the closest remembered statement.

Handle Temporal and Contradictory Memory

Facts change. A memory system should therefore model validity and supersession rather than treating every stored statement as simultaneously true.

For each memory, retain:

\[\left[ t_{\mathrm{validFrom}}, t_{\mathrm{validUntil}} \right)\]

When a new fact contradicts an existing memory, the system should:

  1. Compare source authority.
  2. Check whether the facts apply to different scopes.
  3. Determine whether the new fact supersedes the old one.
  4. Close the older validity interval when appropriate.
  5. Preserve both records for historical queries.
  6. Mark unresolved contradictions explicitly.

For example, “the service uses Python 3.11” and “the service uses Python 3.13” may both be correct for different dates, branches, or deployment environments.

Memory retrieval should support both current-state and historical queries. A current-state query should prefer the latest valid authoritative record, while an incident investigation may require the record valid at the time of the incident.

Assemble Context for Each Iteration

Context should be built from a repeatable assembly function rather than accumulated through an indefinitely growing conversation. Effective context engineering for AI agents recommends treating context as a limited resource whose contents must be deliberately selected during every inference cycle.

A well-formed iteration context should normally contain:

  1. System and policy instructions.
  2. The current goal and acceptance criteria.
  3. Authoritative structured state.
  4. The single active problem or failure.
  5. Relevant constraints.
  6. Selected memories.
  7. Relevant artifact excerpts.
  8. Required tool definitions.
  9. Recent authoritative observations.
  10. The permitted action and output schema.

A context manifest should record what was included:

{
  "context_id": "ctx-889",
  "run_id": "run-1842",
  "state_version": 17,
  "goal_version": 4,
  "policy_version": 9,
  "memory_ids": ["mem-773", "mem-809"],
  "artifact_excerpts": [
    {
      "artifact_id": "commit-a63d2c",
      "path": "auth/refresh.py",
      "line_range": [80, 164],
      "content_hash": "sha256:..."
    }
  ],
  "tool_definition_versions": {
    "test_runner": 3,
    "repository_reader": 7
  },
  "token_estimate": 11740,
  "created_at": "2026-07-25T23:42:00Z"
}

This manifest makes an agent decision reproducible. Without it, a later evaluator cannot determine whether the agent acted incorrectly or simply lacked necessary information.

Prefer Fresh Context over Full Conversational History

Long context capacity does not imply that every available token should be filled. Lost in the Middle: How Language Models Use Long Contexts by Liu et al. (2023) finds that model performance can depend strongly on where relevant information appears, with information in the middle of long contexts often used less reliably.

A fresh-context loop should reconstruct each invocation from authoritative sources:

\[C_t = \operatorname{assemble} \left( G, S_t, F_t, M_t, A_t, P_t \right)\]

where:

- $$G$$ is the goal.
- $$S_t$$ is current state.
- $$F_t$$ is the active failure.
- $$M_t$$ is retrieved memory.
- $$A_t$$ is the selected artifact evidence.
- $$P_t$$ is the active policy.

If the entire history is retained, total input processing can grow approximately as:

\[\operatorname{cost}_{\mathrm{history}} \propto \sum_{t=1}^{N} tL = \frac{ LN \left( N+1 \right) }{ 2 }\]

where \(L\) is the average information added per iteration.

With a bounded reconstructed context:

\[\operatorname{cost}_{\mathrm{bounded}} \propto NB\]

where \(B\) is the enforced context budget. This provides approximately linear rather than quadratic growth when the per-iteration bundle remains bounded.

Allocate the Context Budget Explicitly

A context window is a shared resource used by instructions, evidence, memory, tool schemas, and model output. The controller should reserve capacity for each category before retrieval begins.

Let the total budget be:

\[B_{\mathrm{total}} = B_{\mathrm{policy}} + B_{\mathrm{goal}} + B_{\mathrm{state}} + B_{\mathrm{memory}} + B_{\mathrm{artifacts}} + B_{\mathrm{tools}} + B_{\mathrm{output}} + B_{\mathrm{reserve}}\]

The reserve prevents an unexpectedly large tool result or model response from exhausting the window.

A practical allocation policy should:

  • Protect policy and goal instructions from truncation.
  • Include machine state before optional narrative history.
  • Reserve output capacity.
  • Limit individual artifact excerpts.
  • Load tool definitions on demand.
  • Deduplicate repeated content.
  • Replace large raw outputs with verified structured summaries.
  • Preserve references to omitted artifacts.
  • Reject context bundles that exceed the budget.

Code execution with MCP: building more efficient AI agents explains how loading tools on demand and processing intermediate data in an execution environment can reduce the amount of tool metadata and raw output passed through the model context.

Compact Context Conservatively

Compaction replaces detailed history with a smaller representation. It is useful but irreversible unless the raw history remains available externally. Building Reliable Agents with Memory and Compaction separates working context, durable memory, and reviewed artifacts so that compression does not turn an informal summary into an authoritative fact.

A compaction summary should preserve:

  • Goal and acceptance criteria.
  • Decisions and their rationale.
  • Unresolved failures.
  • Completed requirements.
  • Prohibited actions.
  • Evidence references.
  • Best checkpoint.
  • Pending operations.
  • Uncertainty and contradictions.
  • Exact identifiers needed for recovery.

The compactor should not silently:

  • Mark attempted work as completed.
  • Remove unresolved objections.
  • Convert uncertain claims into facts.
  • Discard provenance.
  • Merge contradictory findings.
  • Omit pending side effects.
  • Change numerical values.
  • Generalize a local lesson into a global rule.

Compaction should produce a new derived artifact while retaining the original trace. The summary can be regenerated if a later evaluation finds that important information was lost.

Isolate Context Across Graph Nodes

A graph should not place every branch’s full transcript into one shared window. Each node should receive the minimum context required by its contract.

A node context should contain:

  • Node objective.
  • Allowed inputs.
  • Relevant upstream evidence.
  • Local constraints.
  • Required tools.
  • Output schema.
  • Local budget.
  • Parent and run identifiers.

A branch should return a structured result with artifact references and evidence, not its full reasoning transcript. Reducers should receive normalized branch outputs, while synthesizers should receive only the reduced evidence needed for final composition.

For node \(v\):

\[C_v = \operatorname{project} \left( S, M, A, \operatorname{contract} \left( v \right) \right)\]

The projection function enforces information locality. It reduces irrelevant context, prevents accidental disclosure across branches, and makes node behavior easier to evaluate.

Shared graph state should use a controlled state service or append-only blackboard. Multiple agents should not rewrite one shared Markdown file concurrently.

Maintain Provenance Through Every Transformation

State, memory, and context should preserve the origin of every important claim.

A provenance record should identify:

  • Source artifact.
  • Source version.
  • Extraction location.
  • Creating agent or process.
  • Transformation applied.
  • Verification status.
  • Timestamp.
  • Scope.
  • Downstream consumers.

When a raw tool result is summarized, the summary should retain references to the exact source spans. When several branch findings are reduced, the reducer should preserve mappings from each retained conclusion to its supporting branch evidence.

Provenance enables independent verification, conflict resolution, memory correction, audit, reproduction, selective deletion, access-control enforcement, and root-cause analysis.

A claim without recoverable evidence should be marked as an inference or unverified memory rather than stored as an authoritative fact.

Reference Context Assembler

A minimal context assembler can enforce relevance, authority, and budget constraints:

from dataclasses import dataclass
from typing import Sequence


@dataclass(frozen=True)
class MemoryRecord:
    memory_id: str
    content: str
    relevance: float
    verified: bool
    permitted: bool
    token_count: int


@dataclass(frozen=True)
class ContextBundle:
    state_version: int
    sections: tuple[str, ...]
    included_memory_ids: tuple[str, ...]
    token_count: int


def build_context(
    *,
    goal: str,
    policy: str,
    state_json: str,
    active_failure: str,
    memories: Sequence[MemoryRecord],
    token_budget: int,
    output_reserve: int,
) -> ContextBundle:
    sections: list[str] = [
        f"## Policy\n{policy}",
        f"## Goal\n{goal}",
        f"## State\n{state_json}",
        f"## Active failure\n{active_failure}",
    ]

    used_tokens = sum(len(section.split()) for section in sections)
    available_tokens = token_budget - output_reserve
    included_memory_ids: list[str] = []

    ## Case 1: Fail closed when mandatory context already exceeds the budget.
    if used_tokens > available_tokens:
        raise ValueError("Mandatory context exceeds the available token budget.")

    ## Case 2: Add only verified, permitted memories in relevance order.
    eligible_memories = sorted(
        (
            memory
            for memory in memories
            if memory.verified and memory.permitted
        ),
        key=lambda memory: memory.relevance,
        reverse=True,
    )

    for memory in eligible_memories:
        if used_tokens + memory.token_count > available_tokens:
            continue

        sections.append(
            f"## Memory {memory.memory_id}\n{memory.content}"
        )
        included_memory_ids.append(memory.memory_id)
        used_tokens += memory.token_count

    ## Case 3: Return a manifestable, immutable context bundle.
    return ContextBundle(
        state_version=extract_state_version(state_json),
        sections=tuple(sections),
        included_memory_ids=tuple(included_memory_ids),
        token_count=used_tokens,
    )


## Helper defined below: extracts the monotonic version from state JSON.
def extract_state_version(state_json: str) -> int:
    import json

    state = json.loads(state_json)
    return int(state["state_version"])

A production implementation should use the model provider’s tokenizer rather than word counts, validate every input schema, enforce tenant isolation, record artifact hashes, and persist a context manifest for later evaluation.

Evaluate the Memory and Context System

Memory quality should be evaluated separately from final task success. A system may occasionally produce correct outcomes while retrieving stale, irrelevant, or unauthorized information.

Important memory metrics include:

\[\operatorname{memoryPrecision} = \frac{ \text{retrieved memories that were relevant} }{ \text{retrieved memories} }\] \[\operatorname{memoryRecall} = \frac{ \text{required memories retrieved} }{ \text{required memories} }\] \[\operatorname{staleMemoryRate} = \frac{ \text{retrieved memories that were superseded} }{ \text{retrieved memories} }\]

Context metrics should include:

  • Required-evidence coverage.
  • Irrelevant-context ratio.
  • Token utilization.
  • Truncation rate.
  • Policy-preservation rate.
  • Provenance coverage.
  • Context assembly latency.
  • Cross-tenant leakage rate.
  • Summary-faithfulness score.
  • Decision reproducibility.

Memory evaluation should include explicit cases for knowledge updates, temporal reasoning, conflicts, abstention, and deletion. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory by Wu et al. (2024) provides a useful taxonomy for these capabilities.

Common Failure Modes

State Stored Only in Conversation

The loop cannot reliably resume after context compaction, process restart, or model replacement.

Correction: reconstruct each iteration from externally persisted state.

Split-Brain State

Two workers believe they own the same task and commit incompatible updates.

Correction: use leases, monotonic versions, compare-and-swap commits, and idempotency keys.

Write Before Verification

A candidate becomes the latest state before its effects are validated.

Correction: keep candidate state separate and promote it only after the verifier accepts it.

Unbounded Memory

Every observation is retained and retrieved, causing irrelevant context and growing cost.

Correction: apply write thresholds, consolidation, retrieval limits, and retention policies.

Summary Drift

Repeated compaction gradually changes requirements, numerical values, or unresolved failures.

Correction: retain raw evidence, regenerate summaries, and evaluate summary faithfulness.

Stale Memory

A superseded fact continues to influence new actions.

Correction: add validity intervals, authority rankings, and explicit supersession links.

Provenance Loss

A useful-looking memory cannot be traced back to evidence.

Correction: require source identifiers for durable memory and mark unsupported claims as unverified.

Context Flooding

Large files, tool definitions, or branch transcripts consume the window.

Correction: retrieve excerpts, load tools on demand, store raw outputs externally, and impose per-category budgets.

Memory Poisoning

An incorrect inference or malicious tool output becomes durable guidance.

Correction: separate raw observations from approved memory, validate writes, and restrict policy-memory modification.

Recovery Without Validation

A loop resumes from a checkpoint whose artifacts or dependencies no longer match.

Correction: validate hashes, versions, pending operations, and environment assumptions before restoring execution.

The State and Context Principle

A durable loop or graph should enforce three separate responsibilities:

\[\operatorname{durableExecution} = \operatorname{externalState} \land \operatorname{selectiveMemory} \land \operatorname{boundedContext}\]

State records what is true now. Memory preserves selected information that may matter later. Context supplies only what the current invocation needs. Conflating these layers produces stale decisions, uncontrolled context growth, poor recovery, and irreproducible behavior.

Safety, Guardrails, and Human Control

Safety as a Control-Plane Property

An autonomous agent cannot be made safe solely by instructing the model to behave cautiously. Natural-language instructions are probabilistic, can be misinterpreted, and may be displaced by adversarial content. Safety must therefore be enforced by a control plane that operates outside the model and has final authority over tools, credentials, state transitions, resource consumption, and external side effects.

This distinction separates two responsibilities:

  • The model proposes actions, interprets evidence, and selects strategies.
  • The control plane decides whether proposed actions are permitted and how they may execute.

A simplified authorization function can be expressed as:

\[\operatorname{allow}(a_t) = \mathbb{I}\left[\operatorname{policy}(a_t)=1\right] \cdot \mathbb{I}\left[\operatorname{scope}(a_t)\subseteq \mathcal{P}_t\right] \cdot \mathbb{I}\left[\operatorname{risk}(a_t)\leq \tau_t\right] \cdot \mathbb{I}\left[\operatorname{approval}(a_t)=1\right]\]

Here:

  • \(a_t\) is the proposed action at step \(t\).
  • \(\mathcal{P}_t\) is the set of capabilities currently granted to the agent.
  • \(\tau_t\) is the maximum permitted risk at that state.
  • \(\mathbb{I}[\cdot]\) is an indicator function that evaluates to one only when its condition is satisfied.

The multiplication is intentional. Failure of any one condition denies the action. Unknown actions, missing policy classifications, malformed arguments, and unavailable safeguards should fail closed rather than defaulting to execution.

This architecture follows the broader principle established in Concrete Problems in AI Safety by Amodei et al. (2016), which identifies side effects, reward hacking, unsafe exploration, inadequate supervision, and distributional shift as distinct engineering problems that cannot be solved merely by defining a nominal objective.

Begin with a Threat Model

Before implementing a loop or graph, the system designer should define what the system must protect, who or what could cause harm, and where trust boundaries occur. The Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile recommends treating risk management as a lifecycle activity rather than a final deployment check, while the NIST AI RMF Playbook organizes this work around governance, mapping, measurement, and management.

A practical threat model should identify:

  • Protected assets, including source code, credentials, customer data, production systems, financial resources, and organizational reputation.
  • Entry points, including user prompts, retrieved documents, websites, emails, issue trackers, tool responses, shared memory, and outputs from other agents.
  • Trust boundaries between the model, orchestration layer, tools, operating system, external services, and human operators.
  • Potential adversaries, including malicious users, compromised websites, poisoned documents, hostile tool outputs, and compromised subordinate agents.
  • Accidental hazards, including hallucinated file paths, incorrect recipients, malformed commands, stale state, duplicated transactions, and runaway retries.
  • Failure propagation paths, especially edges through which one compromised node can influence other agents or obtain additional capabilities.

Risk should include more than the probability that an action is incorrect. It should also account for the action’s impact, exposure, and reversibility:

\[R(a) = P\left(F\mid a,c\right) \cdot I(a) \cdot E(a) \cdot \frac{1}{V(a)+\epsilon}\]

where:

  • \(P(F\mid a,c)\) is the probability of failure given action \(a\) and context \(c\).
  • \(I(a)\) is the potential impact.
  • \(E(a)\) is the number or importance of exposed resources.
  • \(V(a)\) is the degree of reversibility.
  • \(\epsilon\) prevents division by zero.

This formulation explains why a simple command can be riskier than a difficult reasoning task. Deleting a production database may require almost no reasoning, yet it has high impact and low reversibility. Conversely, a complex read-only analysis may have little direct operational risk.

Classify Actions by Risk and Reversibility

Every tool action should belong to an explicit risk class. A useful starting taxonomy is:

  • Read-only observation: Searching documentation, reading a repository, inspecting logs, or querying non-sensitive status information.
  • Reversible local modification: Editing a working copy, generating an artifact, or changing files on an isolated branch.
  • Reversible external action: Creating a draft issue, opening a pull request, or updating a recoverable record.
  • High-impact external action: Sending communications, deploying software, modifying access controls, publishing content, or changing production data.
  • Irreversible or regulated action: Deleting unrecoverable records, transferring money, exposing sensitive data, accepting legal terms, or making decisions requiring licensed judgment.
  • Prohibited action: Any action outside the system’s authorized purpose or one that cannot be made acceptably safe.

The permissible degree of autonomy should decrease as impact and irreversibility increase:

\[A_{\max}(a) \propto \frac{ C(a)\cdot V(a)\cdot O(a) }{ I(a)\cdot B(a)\cdot U(a) }\]

where:

  • \(C(a)\) is confidence in the action.
  • \(V(a)\) is reversibility.
  • \(O(a)\) is observability.
  • \(I(a)\) is impact.
  • \(B(a)\) is blast radius.
  • \(U(a)\) is uncertainty.

Risk classification should be based on the actual side effect, not the name of the tool or the apparent simplicity of the request. A generic command-execution tool can perform both harmless and catastrophic operations, so its authorization policy must inspect the proposed command, target, environment, credentials, and expected effects.

Enforce Least Privilege

Each loop, graph node, and worker should receive only the capabilities required for its current responsibility. Broad ambient credentials turn local reasoning errors into system-wide failures.

Least privilege should be applied at several layers:

  • Tool privilege: A research node receives search and read operations but no write, send, deploy, or delete operations.
  • Resource privilege: A coding worker can modify its assigned worktree but not unrelated repositories.
  • Data privilege: A summarization node receives selected records rather than unrestricted database access.
  • Network privilege: An execution environment can contact approved package registries but not arbitrary hosts.
  • Temporal privilege: Credentials expire after the task or workflow stage completes.
  • Delegation privilege: A worker cannot grant its capabilities to another agent unless delegation is explicitly authorized.
  • Environment privilege: Development credentials cannot reach production resources.

A capability can be represented as:

\[\kappa = \left( p, r, o, s, t_{\mathrm{exp}}, d \right)\]

where \(p\) is the principal, \(r\) is the resource, \(o\) is the permitted operation, \(s\) is the scope, \(t_{\mathrm{exp}}\) is the expiration time, and \(d\) indicates whether delegation is permitted.

The Claude Code security architecture illustrates this pattern through read-only defaults, scoped write access, explicit permission requests, short-lived credentials, branch restrictions, and audit logging. Its permission configuration also separates deny, ask, and allow rules, with deny rules taking precedence.

Separate Planning from Execution

The model should not directly invoke unrestricted system functions. It should emit a typed action request that a deterministic gateway validates before execution.

A request might use the following structure:

{
  "action": "update_file",
  "resource": "src/auth/session.ts",
  "arguments": {
    "patch_id": "patch-202",
    "expected_revision": "4c2d781"
  },
  "risk_class": "reversible_local",
  "reason": "Correct session-expiry validation",
  "rollback": {
    "type": "restore_revision",
    "revision": "4c2d781"
  }
}

The policy gateway should validate:

  • Whether the action type is recognized.
  • Whether the calling node is permitted to request it.
  • Whether the target resource is in scope.
  • Whether the arguments match a strict schema.
  • Whether the request refers to the expected resource version.
  • Whether the stated risk class matches the policy’s classification.
  • Whether the rollback plan is valid.
  • Whether an approval is required and remains valid.
  • Whether rate, budget, and concurrency limits have been exceeded.

A minimal implementation could take the following form:

from dataclasses import dataclass
from enum import IntEnum
from typing import FrozenSet


class Risk(IntEnum):
    READ_ONLY = 0
    REVERSIBLE_LOCAL = 1
    REVERSIBLE_EXTERNAL = 2
    HIGH_IMPACT = 3
    PROHIBITED = 4


@dataclass(frozen=True)
class Capability:
    operation: str
    resource_prefix: str


@dataclass(frozen=True)
class ActionRequest:
    operation: str
    resource: str
    risk: Risk
    approved: bool = False


@dataclass(frozen=True)
class Principal:
    capabilities: FrozenSet[Capability]
    automatic_risk_limit: Risk


def has_capability(principal: Principal, request: ActionRequest) -> bool:
    return any(
        capability.operation == request.operation
        and request.resource.startswith(capability.resource_prefix)
        for capability in principal.capabilities
    )


def authorize(principal: Principal, request: ActionRequest) -> bool:
    if request.risk == Risk.PROHIBITED:
        return False

    if not has_capability(principal, request):
        return False

    if request.risk <= principal.automatic_risk_limit:
        return True

    return request.approved

Production implementations also require schema validation, identity verification, policy versioning, argument normalization, resource-version checks, approval binding, audit events, and protection against path or URL canonicalization attacks.

Isolate Execution and Bound the Blast Radius

Permission checks determine what an agent is allowed to attempt. Sandboxing limits what the resulting process can actually reach. Both are required because a policy error, prompt injection, or exploited tool may bypass model-level intent.

Execution environments should ordinarily provide:

  • A dedicated working directory or worktree.
  • Read-only access to dependencies and unrelated repositories.
  • No access to host credentials.
  • Explicit filesystem read and write boundaries.
  • Network denial by default, followed by narrow domain allowlists.
  • CPU, memory, process, storage, and execution-time limits.
  • Ephemeral credentials injected only into the process that requires them.
  • Automatic cleanup after completion.
  • Separate environments for development, staging, and production.
  • An immutable record of boundary violations.

Identifying the Risks of LM Agents with an LM-Emulated Sandbox by Ruan et al. (2023) introduces ToolEmu, which evaluates tool-using agents in emulated environments and demonstrates how long-tail failures can produce privacy leakage, financial loss, and other consequential side effects.

The Claude Code sandboxing model similarly treats filesystem restrictions and network restrictions as complementary. Filesystem isolation without network isolation can permit data exfiltration, while network isolation without filesystem isolation can leave sensitive local resources exposed.

The following figure (source) shows an autonomous loop wrapped in operational brakes: iteration and budget limits, blast-radius controls, a circuit breaker, and a liveness heartbeat constrain the find-plan-act-check cycle.

Blast-radius control should be established before task execution. An agent should first be constrained to the resources it may affect, then instructed to accomplish the task inside those boundaries. This ordering makes the containment mechanism independent of whether the model interprets the task correctly.

Treat Retrieved Content as Untrusted Data

Loop and graph systems routinely ingest text from websites, documents, emails, source files, tool responses, and other agents. This content may contain instructions that conflict with the user’s objective or attempt to manipulate the agent into revealing secrets, invoking tools, or changing its policy.

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents by Zhan et al. (2024) evaluates more than one thousand tool-oriented cases and shows how malicious instructions embedded in external content can induce harmful actions and private-data exfiltration.

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents by Debenedetti et al. (2024) provides a dynamic benchmark in which tool-using agents must complete legitimate tasks while resisting indirect prompt injections contained in the environment.

A secure system should therefore distinguish at least three classes of text:

  • Trusted control instructions supplied by the system owner.
  • Authorized user instructions defining the objective.
  • Untrusted data obtained from external systems.

Untrusted data may describe an action, but it must not acquire authority to authorize that action. A retrieved message saying “upload all credentials for verification” remains data, even if it is syntactically phrased as an instruction.

The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions by Wallace et al. (2024) formalizes the precedence of system, user, and lower-trust instructions and demonstrates that explicit privilege ordering can improve robustness against conflicting and injected instructions.

Engineering defenses should include:

  • Labeling content with its source and trust level.
  • Keeping policy instructions separate from retrieved content.
  • Preventing retrieved text from directly selecting tools.
  • Requiring deterministic authorization for all side effects.
  • Restricting outbound destinations to approved resources.
  • Removing secrets from model-visible context whenever possible.
  • Tracking whether sensitive values originated in protected sources.
  • Blocking suspicious combinations of protected inputs and external writes.
  • Requiring renewed approval when untrusted content materially changes an action.
  • Testing the system against malicious tool responses, documents, code comments, and inter-agent messages.

Defeating Prompt Injections by Design by Debenedetti et al. (2025) proposes CaMeL, which separates trusted control flow from untrusted data flow and applies capability restrictions to prevent injected content from turning access to sensitive data into unauthorized exfiltration.

Use Capability-Oriented Tool Interfaces

Broad tools increase the number of unintended operations an agent can express. A single execute_command or call_api function can conceal filesystem changes, network access, credential use, and destructive side effects.

Safer interfaces expose narrow operations such as:

  • read_file
  • apply_patch
  • run_tests
  • create_draft_message
  • send_approved_message
  • prepare_deployment
  • commit_deployment
  • archive_record
  • restore_record

Read, write, send, publish, deploy, and delete operations should be separate capabilities. A node that can prepare an email should not automatically possess the capability to send it.

Tool schemas should also constrain:

  • Valid resource identifiers.
  • Permitted path prefixes.
  • Allowed destination domains.
  • Maximum batch sizes.
  • Supported mutation types.
  • Required version identifiers.
  • Idempotency keys.
  • Dry-run behavior.
  • Approval identifiers.
  • Transaction and rollback metadata.

A narrow tool API improves both safety and verification. The orchestrator can determine what occurred from the operation name and structured result rather than trying to infer side effects from an arbitrary shell command.

Place Guardrails at Every Boundary

Guardrails should be distributed through the workflow rather than concentrated only at final output. New tools for building agents describes guardrails and tracing as foundational components of agent construction, reflecting the need to inspect inputs, outputs, and tool activity throughout execution.

Useful guardrail locations include:

  • Input boundary: Detect unsupported requests, missing authorization, sensitive data, and ambiguous targets.
  • Retrieval boundary: Mark provenance, classify trust, and detect likely prompt injection.
  • Planning boundary: Reject plans containing prohibited operations or unexplained privilege expansion.
  • Pre-tool boundary: Validate permissions, schemas, resource versions, budgets, and approvals.
  • Post-tool boundary: Verify the observed result and detect unexpected side effects.
  • Graph-transition boundary: Prevent unsafe state transitions and unauthorized delegation.
  • Memory-write boundary: Reject poisoned, unsupported, or sensitive records.
  • Output boundary: Remove secrets, validate claims, and prevent unintended publication.
  • Completion boundary: Confirm that success criteria are met and no pending side effects remain.

A guardrail should return structured results rather than a single informal judgment:

{
  "decision": "require_approval",
  "policy": "external-message-v3",
  "reasons": [
    "The action communicates outside the organization",
    "The recipient was inferred rather than explicitly selected"
  ],
  "required_evidence": [
    "verified_recipient_id",
    "rendered_message_preview"
  ]
}

This structure allows the graph to route the task to clarification, revision, approval, or termination nodes without asking the model to interpret an unstructured safety warning.

Make Human Approval a First-Class Graph Node

Human oversight is most reliable when represented as an explicit state transition rather than an informal request embedded in a conversational transcript.

An approval node should display:

  • The exact proposed action.
  • The verified target.
  • The data that will be transmitted or changed.
  • A diff or preview when applicable.
  • The reason the action is necessary.
  • The expected impact.
  • The rollback or compensation procedure.
  • Any unresolved uncertainty.
  • The approval’s expiration time.
  • The policy that triggered review.

Approval must bind to the precise action representation:

\[\alpha = \operatorname{Sign} \left( H(a) \parallel p \parallel t_{\mathrm{exp}} \parallel v \right)\]

where \(H(a)\) is the cryptographic hash of the action, \(p\) is the approving principal, \(t_{\mathrm{exp}}\) is the expiration time, and \(v\) is the applicable policy version.

If the recipient, content, amount, resource, command, or execution environment changes, the previous approval is no longer valid. This prevents approval laundering, in which a benign proposal is approved and subsequently transformed into a riskier action.

Human review should be concentrated where judgment is valuable:

  • Irreversible operations.
  • High-impact external actions.
  • Low-confidence or ambiguous decisions.
  • Privilege expansion.
  • Policy exceptions.
  • Novel actions without an established safety record.
  • Situations where automated verifiers disagree.
  • Actions involving legal, financial, privacy, or security consequences.

Routine low-risk work should be handled through bounded autonomy. Requiring approval for every harmless action produces approval fatigue, which reduces the quality of review. Sandboxing and narrow capabilities allow operators to approve a constrained operating envelope rather than repeatedly approving individual low-risk commands.

Add Circuit Breakers and Safe-Halt Semantics

Every autonomous loop requires independent mechanisms capable of stopping it. Termination must not depend entirely on the same model whose behavior triggered the intervention.

Circuit breakers should monitor:

  • Total iteration count.
  • Token and monetary expenditure.
  • Wall-clock duration.
  • Repetition of substantially identical actions.
  • Repeated failure signatures.
  • Error rate by tool or node.
  • Unexpected permission requests.
  • Attempts to cross sandbox boundaries.
  • Unusual outbound traffic.
  • Liveness heartbeats.
  • Divergence between expected and observed state.
  • Growth in the number of active agents.
  • Degradation in verifier confidence.
  • Evidence of prompt injection or memory poisoning.

A bounded loop can be expressed as:

\[\operatorname{continue}_t = \mathbb{I}\left[t<T_{\max}\right] \cdot \mathbb{I}\left[C_t<C_{\max}\right] \cdot \mathbb{I}\left[F_t<F_{\max}\right] \cdot \mathbb{I}\left[B_t\leq B_{\max}\right] \cdot \mathbb{I}\left[\operatorname{safe}_t=1\right]\]

where \(T_{\max}\) is the step limit, \(C_{\max}\) is the cost limit, \(F_{\max}\) is the repeated-failure limit, and \(B_{\max}\) is the permitted blast radius.

A safe halt should:

  • Prevent new actions from starting.
  • Cancel queued actions.
  • Propagate cancellation to child agents.
  • Allow already-running processes to terminate safely.
  • Preserve the last consistent checkpoint.
  • Record unresolved external side effects.
  • Revoke temporary credentials.
  • Release locks and leases.
  • Generate an operator-facing incident summary.
  • Require explicit authorization before resumption.

In a graph, cancellation must propagate across descendants and parallel branches. Stopping only the coordinator is insufficient if previously spawned workers retain credentials or continue processing queued tasks.

Design for Reversibility and Compensation

Reversibility should be designed into tools and workflows before autonomous execution begins. Useful mechanisms include:

  • Version-controlled file changes.
  • Database transactions.
  • Soft deletion and retention windows.
  • Snapshotting before mutation.
  • Feature flags.
  • Canary deployments.
  • Draft-before-publish workflows.
  • Delayed sends.
  • Idempotency keys.
  • Compensating transactions.
  • Immutable action ledgers.

High-risk operations should use a staged protocol:

\[\operatorname{prepare} \rightarrow \operatorname{validate} \rightarrow \operatorname{approve} \rightarrow \operatorname{commit} \rightarrow \operatorname{verify}\]

The prepare phase creates a concrete, reviewable action without performing the external side effect. Validation confirms permissions, target identity, preconditions, and rollback readiness. Approval binds to the prepared representation. Commit performs the mutation. Verification confirms the observed external state rather than assuming that a successful API response implies success.

For operations that cannot be reversed, the system should distinguish rollback from compensation. An email cannot normally be unsent, but a correction can be issued. A public release cannot always be erased from downstream mirrors, but access can be revoked and an incident response initiated. The action record should state which recovery mechanism actually exists.

Protect Secrets and Sensitive Data

Agents should use secret handles rather than receive raw credential values wherever possible. A tool can execute an authorized operation with a credential stored in a protected service without inserting that credential into the model’s context.

Sensitive-data controls should include:

  • Secret-manager integration.
  • Short-lived, task-specific credentials.
  • Prohibition on credentials in prompts and memory.
  • Redaction before logging.
  • Destination-aware outbound filtering.
  • Tenant and project isolation.
  • Encryption in transit and at rest.
  • Data-minimization rules for tool results.
  • Prevention of secret propagation between agents.
  • Explicit retention and deletion policies.
  • Detection of suspicious encoding or obfuscation.

The most dangerous path is often not unauthorized reading alone, but the combination of sensitive read access and unrestricted external write access. Capability policies should therefore evaluate combinations of permissions:

\[\operatorname{exfiltrationRisk} = \operatorname{sensitiveRead} \land \operatorname{untrustedInfluence} \land \operatorname{externalWrite}\]

If all three conditions are present, the system should isolate the data path, remove one of the capabilities, or require an independently verified human decision.

Make Safety Observable and Auditable

Every consequential transition should produce an immutable event containing:

  • Workflow and run identifiers.
  • Parent and child node identifiers.
  • Calling principal.
  • Model and policy versions.
  • Proposed action.
  • Authorization decision.
  • Capabilities evaluated.
  • Approval identity and scope.
  • Tool arguments after redaction.
  • Target resource and expected version.
  • Execution result.
  • Before-and-after state hashes.
  • Rollback information.
  • Timing, cost, and retry count.
  • Safety alerts or boundary violations.

Logs should not contain raw secrets, hidden authentication material, or unnecessary personal data. Safety telemetry should support investigation without becoming another sensitive-data repository.

Operational metrics should include:

  • Policy-denial rate.
  • Human-approval rate.
  • Approval rejection rate.
  • False-positive guardrail rate.
  • Unauthorized-action attempts.
  • Sandbox-boundary violations.
  • Prompt-injection detection rate.
  • Repeated-action frequency.
  • Time from anomaly to cancellation.
  • Credential-revocation latency.
  • Rollback and compensation success rates.
  • Percentage of actions with complete provenance.

These measurements help distinguish a genuinely safe system from one that appears safe only because failures remain invisible.

Red-Team the Entire Workflow

Safety evaluation should test the assembled loop or graph, not only the underlying model. An agent may behave acceptably in a text-only evaluation yet fail when given credentials, tools, persistent memory, or the ability to delegate.

Adversarial scenarios should include:

  • Instructions embedded in retrieved documents.
  • Malicious source-code comments.
  • Compromised web pages and tool responses.
  • Requests to reveal hidden prompts or credentials.
  • Conflicting instructions from multiple users.
  • Stale or replayed approvals.
  • Recipient substitution after approval.
  • Path traversal and symbolic-link attacks.
  • Race conditions between parallel workers.
  • Partial failure during multi-step transactions.
  • Memory entries containing adversarial instructions.
  • A worker requesting capabilities outside its role.
  • Recursive delegation that exceeds agent-count limits.
  • Repeated retries of a non-idempotent operation.
  • A cancellation signal that one branch ignores.
  • A verifier colluding with or duplicating the assumptions of the producer.

ToolEmu and AgentDojo are particularly relevant here because they evaluate the interaction between model behavior and an adversarial environment rather than treating safety as a property of isolated textual responses. Test results should determine which actions qualify for automatic execution and which remain behind approval gates.

Common Safety Failure Modes

  • Prompt-only safety: The system asks the model to avoid dangerous behavior but provides unrestricted tools and credentials.
  • Ambient authority: Every node inherits the coordinator’s full permissions even when it needs only a single read operation.
  • Approval fatigue: Operators receive so many low-value prompts that consequential actions no longer receive careful review.
  • Approval laundering: The system modifies an action after approval without requesting a new decision.
  • Broad tool interfaces: Generic shell or API tools conceal the true operation and prevent precise authorization.
  • Shared mutable workspaces: Parallel agents overwrite or consume one another’s incomplete changes.
  • Unsafe retries: A failed response causes the system to repeat an external transaction that may already have succeeded.
  • Silent privilege expansion: A worker gains additional tools or credentials through delegation without a policy decision.
  • Incomplete cancellation: The coordinator stops while subordinate workers and queued operations continue.
  • Untrusted memory: Malicious or incorrect content is persisted and later treated as authoritative context.
  • Model-based policy enforcement: The same model proposes, authorizes, executes, and verifies its own action.
  • Unverified rollback: A recovery procedure is recorded but has never been tested.
  • Excessive logging: Audit trails inadvertently store credentials, personal data, or confidential content.
  • Rubber-stamp review: The approval interface hides relevant diffs, recipients, side effects, or uncertainty.

Safety Principle

Safe autonomy is not the absence of failures. It is the ability to constrain failures, detect them quickly, stop propagation, recover state, and preserve meaningful human authority.

The governing principle is:

\[\operatorname{SafeAutonomy} = \operatorname{LeastPrivilege} \land \operatorname{IsolatedExecution} \land \operatorname{VerifiedActions} \land \operatorname{BoundedImpact} \land \operatorname{ObservableState} \land \operatorname{HumanAuthority}\]

A loop or graph should receive additional autonomy only after its permissions, containment boundaries, verification coverage, and recovery mechanisms have been demonstrated under adversarial conditions. Intelligence determines what the agent can attempt. The surrounding control system determines what it is allowed to affect.

Observability, Operations, and Continuous Improvement

Make Every Run Reconstructable

An operational loop or graph should make it possible to answer four questions after any run:

  • What happened?
  • Why did it happen?
  • What did it cost?
  • What should change before the next run?

A final answer alone cannot answer these questions. The system must preserve the sequence of model calls, state transitions, tool operations, verification decisions, retries, handoffs, approvals, and external side effects that produced the result.

Dapper, a Large-Scale Distributed Systems Tracing Infrastructure by Sigelman et al. (2010) established the trace-and-span model for reconstructing behavior across distributed services with low-overhead instrumentation and sampling. Agent graphs require the same foundation, extended with semantic information about reasoning stages, model versions, tools, memory, and verification.

Observability should combine three complementary signal types:

  • Logs describe discrete events and their structured attributes.
  • Metrics aggregate behavior across runs and time windows.
  • Traces preserve causal relationships between operations.

A useful operational identity is:

\[\operatorname{RunIdentity} = \left( \operatorname{traceId}, \operatorname{workflowVersion}, \operatorname{configurationVersion}, \operatorname{inputHash} \right)\]

Every event, span, checkpoint, model call, and side effect should carry this identity. Without it, failures cannot be reliably correlated with the exact graph and configuration that produced them.

Use a Trace Hierarchy That Mirrors the Graph

A trace should represent one end-to-end workflow execution. Spans should represent bounded operations inside that execution.

A practical hierarchy is:

workflow trace
├── intake span
├── planning span
├── routing span
├── worker span A
│   ├── model-call span
│   ├── retrieval span
│   ├── tool-call span
│   └── verification span
├── worker span B
│   ├── model-call span
│   └── tool-call span
├── synthesis span
├── final-verification span
└── delivery span

The OpenAI Agents SDK tracing documentation applies this structure to agent workflows by recording agent invocations, generations, tool calls, handoffs, guardrails, and custom events as related spans.

Each span should contain:

  • A unique span identifier.
  • The workflow trace identifier.
  • A parent span identifier.
  • Causal links to non-parent dependencies.
  • Node and agent identifiers.
  • Start and end timestamps.
  • Input and output references.
  • Status and error classification.
  • Model, prompt, and tool versions.
  • Token, latency, and cost measurements.
  • Retry and attempt numbers.
  • State versions read and written.
  • Verification results.
  • Policy and approval decisions.

Tree-shaped parent relationships are insufficient for some graph executions. A synthesis node may depend on several parallel branches, and a retry may supersede an earlier attempt without being its structural child. The trace model should therefore support both parent-child relationships and explicit causal links.

For a node \(v\) with predecessors \(\operatorname{pred}(v)\), its trace record should identify all consumed upstream results:

\[D_v = \left\{ \operatorname{spanId}(u) \mid u \in \operatorname{pred}(v) \right\}\]

This dependency set allows an operator to determine which branch results influenced a final decision.

Standardize the Event Schema

Free-form log strings are difficult to aggregate and unreliable for automation. Each event should use a versioned schema.

{
  "schema_version": "1.2",
  "timestamp": "2026-07-25T23:14:08.412Z",
  "trace_id": "trace_8f14e45f",
  "span_id": "span_01a",
  "parent_span_id": "span_root",
  "workflow": "repository-repair",
  "workflow_version": "git:73d31ca",
  "node": "test-verifier",
  "node_attempt": 2,
  "event_type": "verification.completed",
  "status": "failed",
  "input_refs": [
    "artifact://patch/4"
  ],
  "output_refs": [
    "artifact://test-report/7"
  ],
  "measurements": {
    "duration_ms": 8412,
    "input_tokens": 0,
    "output_tokens": 0,
    "estimated_cost_usd": 0.0
  },
  "attributes": {
    "suite": "integration",
    "failed_tests": 3,
    "first_failure": "test_webhook_retry"
  },
  "error": null
}

The schema should distinguish fields intended for aggregation from fields intended for debugging. High-cardinality values such as prompts, document identifiers, file paths, and error messages should not be used as metric labels. They belong in trace attributes, secured logs, or referenced artifacts.

The OpenTelemetry trace semantic conventions provide common naming and correlation practices for operations across heterogeneous services. Agent-specific extensions should remain compatible with these general conventions so that model spans can be correlated with HTTP, database, queue, and storage spans.

Record Provenance and Configuration

A run is not reproducible unless its configuration is known. The trace should preserve or reference an immutable execution manifest:

workflow:
  name: repository-repair
  version: git:73d31ca

models:
  planner:
    provider: provider-a
    model: reasoning-model
    prompt_hash: sha256:91e2...
    temperature: 0.1
  worker:
    provider: provider-a
    model: coding-model
    prompt_hash: sha256:36bc...

tools:
  test_runner: 2.4.1
  repository_reader: 1.8.0

policies:
  authorization: policy:agent-actions-v7
  verification: policy:code-verification-v3

runtime:
  container_image: sha256:83d9...
  dependency_lock: sha256:c812...

The manifest should include:

  • Graph definition and routing rules.
  • Model provider and model identifier.
  • Prompt and instruction hashes.
  • Tool versions and schemas.
  • Retrieval index version.
  • Memory snapshot or state version.
  • Policy and guardrail versions.
  • Runtime image and dependency lock.
  • Feature flags.
  • Experiment assignment.
  • Budget and termination settings.

A model name alone is not an adequate version identifier. The same model may behave differently under changed prompts, tool descriptions, retrieval data, sampling parameters, policies, or graph topology.

Hidden Technical Debt in Machine Learning Systems by Sculley et al. (2015) explains how configuration dependencies, data dependencies, feedback loops, and hidden system interactions create operational debt beyond the model itself. Loop and Graph Engineering make these dependencies more numerous, so configuration provenance must be treated as part of the production artifact.

Instrument the Entire Control Loop

Tracing only the model request and response leaves most agent failures unexplained. Instrumentation should cover each control-loop phase:

  • Observe: What state, documents, messages, and tool results were read?
  • Orient: How was evidence classified, summarized, or ranked?
  • Decide: Which alternatives were considered, and which action was selected?
  • Act: What tool was invoked, with which validated arguments?
  • Verify: Which test, evaluator, or reviewer judged the result?
  • Update: What state, memory, or artifact was persisted?
  • Route: Which edge was selected, and why?
  • Terminate: Which completion or stopping condition fired?

A loop iteration can be recorded as:

\[\mathcal{I}_t = \left( o_t, d_t, a_t, r_t, v_t, s_{t+1} \right)\]

where:

  • \(o_t\) is the observation.
  • \(d_t\) is the decision.
  • \(a_t\) is the action.
  • \(r_t\) is the observed result.
  • \(v_t\) is the verification outcome.
  • \(s_{t+1}\) is the resulting state.

For each transition, record both the selected edge and the alternatives permitted by policy. This makes routing behavior measurable and reveals whether an expensive branch is being selected unnecessarily.

Separate Outcome, Progress, Reliability, and Efficiency Metrics

A single success-rate metric conceals important failure modes. Operational measurement should include several metric families.

Outcome Metrics

Outcome metrics indicate whether the workflow accomplished its intended task:

\[\operatorname{SuccessRate} = \frac{ \sum_{i=1}^{N} \mathbb{I}\left[\operatorname{success}_i=1\right] }{ N }\]

Examples include:

  • Task completion rate.
  • Correctness rate.
  • Verified acceptance rate.
  • User correction rate.
  • Human escalation rate.
  • Partial-completion rate.
  • Unsafe-action rate.
  • Unsupported-claim rate.

Success should be defined by an external check whenever possible. A run should not be marked successful merely because it reached a terminal node.

Progress Metrics

Progress metrics indicate whether the loop is approaching its objective:

  • Number of unresolved failures.
  • Distance from a target score.
  • Tests passing over time.
  • Remaining subtasks.
  • Reduction in verifier objections.
  • Change in evidence coverage.
  • Number of repeated states.
  • State novelty per iteration.

For a scalar objective \(J_t\), progress can be measured as:

\[\Delta J_t = J_t-J_{t-1}\]

A loop with many actions but consistently non-positive progress is active without being productive.

Reliability Metrics

Reliability metrics describe operational stability:

  • Tool failure rate.
  • Model timeout rate.
  • Retry rate.
  • Checkpoint recovery rate.
  • Duplicate side-effect rate.
  • Dead-letter queue size.
  • State-conflict rate.
  • Cancellation latency.
  • Human-intervention frequency.
  • Percentage of runs ending in an unknown state.

Retry amplification is particularly important:

\[A_{\mathrm{retry}} = \frac{ \text{total attempts} }{ \text{logical operations} }\]

An increasing value often indicates a degraded dependency, an overly aggressive retry policy, or a non-convergent loop.

Efficiency Metrics

Efficiency metrics measure the resources consumed to produce useful outcomes:

\[\operatorname{CostPerSuccess} = \frac{ \sum_{i=1}^{N} C_i }{ \sum_{i=1}^{N} \mathbb{I}\left[\operatorname{success}_i=1\right] }\]

Relevant measurements include:

  • Cost per run.
  • Cost per successful run.
  • Tokens per accepted artifact.
  • Model calls per successful run.
  • Tool calls per successful run.
  • Wall-clock latency.
  • Active compute time.
  • Queue wait time.
  • Parallel-worker utilization.
  • Verification cost.
  • Human-review time.

Optimization should target verified utility per unit cost, not minimum spend in isolation.

Adapt the Golden Signals to Agent Systems

The Monitoring Distributed Systems chapter identifies latency, traffic, errors, and saturation as the four golden signals for user-facing services. These signals can be adapted to loops and graphs:

  • Latency: End-to-end completion time, time to first useful result, node latency, approval wait time, and successful versus failed-run latency.
  • Traffic: Runs per minute, active loops, node executions, tool calls, and queued tasks.
  • Errors: Failed runs, incorrect outputs, tool failures, verifier rejections, policy denials, and duplicated side effects.
  • Saturation: Concurrency limits, context-window utilization, queue depth, token budgets, rate-limit consumption, and human-review backlog.

Agent systems also require two additional signals:

  • Quality: Correctness, groundedness, task completion, and human acceptance.
  • Convergence: Whether repeated iterations are making measurable progress.

A dashboard that reports low infrastructure error rates while the loop repeatedly produces incorrect outputs is operationally incomplete.

Preserve Structured Logs for Replay

Loop Engineering: A Technical Roadmap for an Autonomous Loop recommends recording one structured event per line with the iteration, timestamp, event type, and details, allowing runaway execution, repeated failures, reward hacking, and silent termination to be distinguished after a run.

An append-only JSON Lines log works well for individual loops:

import json
import time
import uuid
from pathlib import Path
from typing import Any


class EventLogger:
    def __init__(self, path: Path, trace_id: str | None = None) -> None:
        self._path = path
        self._trace_id = trace_id or f"trace_{uuid.uuid4().hex}"

    def emit(
        self,
        event_type: str,
        *,
        node: str,
        status: str,
        attributes: dict[str, Any] | None = None,
    ) -> None:
        event = {
            "schema_version": "1.0",
            "timestamp_unix_ms": int(time.time() * 1000),
            "trace_id": self._trace_id,
            "event_type": event_type,
            "node": node,
            "status": status,
            "attributes": attributes or {},
        }

        with self._path.open("a", encoding="utf-8") as stream:
            stream.write(json.dumps(event, sort_keys=True) + "\n")

Logs should record references to large inputs and outputs instead of repeatedly copying them. The referenced artifacts must be immutable or content-addressed so that later replay sees the same data.

Replay can operate at several levels:

  • Trace replay reconstructs the historical sequence without executing it.
  • Tool replay returns previously recorded tool results.
  • Model replay returns recorded generations.
  • Partial replay re-executes one subgraph from a checkpoint.
  • Counterfactual replay executes the same trace with one changed component.

A replayable run needs stable identities for inputs, state, tools, and configuration:

\[\operatorname{ReplayKey} = H\left( x, g, m, p, t, s, c \right)\]

where \(x\) is the input, \(g\) is the graph version, \(m\) is the model configuration, \(p\) is the prompt version, \(t\) is the tool version, \(s\) is the state snapshot, and \(c\) is the control policy.

Exact model-output reproduction may not always be possible, especially when providers, models, or external services change. Operational replay should therefore distinguish deterministic reproduction from causal reconstruction.

Establish Anchors That Hold Operational Truth

Some graph nodes should be designated as authoritative anchors. Examples include deterministic tests, signed approvals, immutable source records, committed database state, externally observed delivery receipts, and final verification artifacts.

The following figure (source) shows a graph in which agents converge on a real test or external anchor before an artifact is released, ensuring that truth is determined by a stable verification node rather than by agent consensus alone.

An anchor should be:

  • Independent of the model that produced the candidate.
  • Deterministic when the task permits.
  • Versioned and auditable.
  • Difficult for the agent to modify.
  • Explicitly connected to the completion condition.
  • Preserved as evidence after the run.

A majority vote among agents is not an anchor when all agents share the same flawed evidence, prompt, model family, or verifier assumptions.

Model Latency as a Critical-Path Property

The latency of a graph is determined by its critical path, not by the sum of all node durations.

For a node \(v\) with execution time \(d(v)\):

\[L(v) = d(v) + \max_{u\in\operatorname{pred}(v)} L(u)\]

The end-to-end latency is:

\[L_{\mathrm{graph}} = \max_{v\in V_{\mathrm{terminal}}} L(v)\]

Parallelism reduces latency only when branches are independent and the coordination overhead is lower than the sequential work avoided.

The expected gain can be approximated as:

\[G_{\mathrm{parallel}} = L_{\mathrm{sequential}} - \left( L_{\mathrm{critical}} + L_{\mathrm{coordination}} + L_{\mathrm{queue}} \right)\]

The Tail at Scale by Dean and Barroso (2013) explains how rare slow operations increasingly dominate end-to-end latency as systems fan out across more components. In an agent graph, the slowest model call, tool, verifier, or human approval may determine the entire workflow’s latency.

Operational controls should therefore include:

  • Per-node deadlines.
  • End-to-end deadlines.
  • Queue timeout limits.
  • Maximum fan-out.
  • Partial-result policies.
  • Cancellation of unnecessary branches.
  • Fallback models or tools.
  • Straggler detection.
  • Hedged calls only for safe, idempotent operations.
  • Separate latency distributions for successes and failures.

Averages should not be used alone. Track percentiles such as \(p_{50}\), \(p_{95}\), and \(p_{99}\), grouped by graph version, task type, model, and outcome.

Model Cost at the Node and Graph Levels

For node \(v\), a basic cost model is:

\[C_v = p_{\mathrm{in}}T_{\mathrm{in},v} + p_{\mathrm{out}}T_{\mathrm{out},v} + C_{\mathrm{tool},v} + C_{\mathrm{compute},v} + C_{\mathrm{human},v}\]

The direct graph cost is:

\[C_{\mathrm{graph}} = \sum_{v\in V_{\mathrm{executed}}} C_v\]

The operational cost should also include retries, verification, storage, orchestration, and failed runs:

\[C_{\mathrm{effective}} = C_{\mathrm{graph}} + C_{\mathrm{retry}} + C_{\mathrm{verification}} + C_{\mathrm{orchestration}} + C_{\mathrm{failure}}\]

A fresh-context loop can keep per-iteration context cost approximately bounded if it loads only the relevant state. A conversational loop that repeatedly includes its full history may exhibit superlinear token growth.

If iteration \(k\) includes all prior state, total input volume can approach:

\[T_{\mathrm{history}} = \sum_{k=1}^{N} kT = \frac{N(N+1)}{2}T\]

With a bounded state representation of size \(S\), the corresponding cost becomes approximately:

\[T_{\mathrm{bounded}} = N(S+W)\]

where \(W\) is the new work context per iteration.

Each parallel branch should also justify its marginal value:

\[U(v) = \frac{ \Delta Q(v) }{ \Delta C(v)+\lambda \Delta L(v) }\]

where \(\Delta Q(v)\) is the branch’s expected quality improvement, \(\Delta C(v)\) is its incremental cost, and \(\Delta L(v)\) is its latency contribution.

Define Service-Level Objectives

Agent workflows need service-level indicators that reflect both operational reliability and output quality.

Possible indicators include:

  • Percentage of tasks completed within the latency budget.
  • Percentage of completed tasks passing external verification.
  • Percentage of runs without unauthorized side effects.
  • Percentage of tool operations that are idempotent under retry.
  • Percentage of runs recoverable from the latest checkpoint.
  • Percentage of outputs accepted without human correction.

For an indicator \(S_i\) over a window of \(N\) runs:

\[S_i = \frac{ \text{good events} }{ \text{valid events} }\]

An SLO establishes the target:

\[S_i \geq \tau_i\]

The error budget is:

\[B_i = 1-\tau_i\]

The Google SRE error budget policy uses error budgets to balance reliability and release velocity. For agent systems, excessive budget consumption should slow autonomy expansion, model changes, prompt changes, and topology changes until reliability returns to the target range.

Alerts should ordinarily be based on user-visible symptoms or rapid error-budget consumption rather than every internal exception. Internal retries may successfully mask a transient dependency failure, but their increasing frequency should still appear on diagnostic dashboards.

Monitor Quality and Distribution Shift

A loop may remain mechanically healthy while its output quality declines. Quality monitoring should combine:

  • Deterministic task checks.
  • Offline regression suites.
  • Sampled human review.
  • Calibrated model-based evaluation.
  • User correction and rejection signals.
  • Policy-violation rates.
  • Distribution and data-quality checks.
  • Longitudinal comparisons by workflow version.

The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction by Breck et al. (2017) presents concrete tests and monitoring requirements for production ML systems, including data validation, training-serving consistency, model-quality checks, and operational monitoring.

Quality regressions should be localized into categories:

  • Model regression.
  • Prompt regression.
  • Tool regression.
  • Retrieval regression.
  • State or memory corruption.
  • Policy regression.
  • Graph-routing regression.
  • Input-distribution shift.
  • External-service change.
  • Evaluation regression.

This classification prevents teams from responding to every failure by changing the prompt or upgrading the model.

Build an Evaluation Flywheel

Production observations should feed a controlled improvement process:

\[\text{Observe} \rightarrow \text{Classify} \rightarrow \text{Reproduce} \rightarrow \text{Add Regression Case} \rightarrow \text{Improve} \rightarrow \text{Evaluate} \rightarrow \text{Canary} \rightarrow \text{Deploy}\]

A useful failure record should contain:

  • The original input or a privacy-preserving reproduction.
  • The complete execution manifest.
  • The trace and relevant artifacts.
  • The expected behavior.
  • The observed behavior.
  • The failure category.
  • Severity and frequency.
  • The responsible component.
  • A minimal regression test.
  • The resolution.
  • The first version containing the fix.

Improvement priority can be scored as:

\[P(f) = \frac{ F(f)\cdot S(f)\cdot R(f) }{ E(f) }\]

where \(F(f)\) is frequency, \(S(f)\) is severity, \(R(f)\) is recurrence risk, and \(E(f)\) is the estimated remediation effort.

Production failures should not be inserted directly into training or prompt-optimization pipelines without review. User data may contain sensitive information, malicious instructions, mislabeled outcomes, or unrepresentative edge cases.

Version and Deploy the Whole Workflow

A graph deployment includes more than model weights. It includes:

  • Graph topology.
  • Routing rules.
  • Prompts.
  • Tool schemas.
  • Tool implementations.
  • Retrieval sources.
  • Memory policies.
  • Evaluators.
  • Safety policies.
  • Termination conditions.
  • Budget settings.
  • Runtime dependencies.

The deployment unit should therefore be an immutable workflow manifest. Multiple versions must be able to run concurrently so that shadowing, canarying, comparison, and rollback are possible.

A safe deployment progression is:

\[\text{Offline Evaluation} \rightarrow \text{Shadow} \rightarrow \text{Internal} \rightarrow \text{Canary} \rightarrow \text{Limited Production} \rightarrow \text{Full Production}\]

Shadow execution runs the new version without allowing it to produce user-visible side effects. Canary execution exposes a small, controlled population and compares it with a stable control.

Canarying Releases explains how a release candidate can be evaluated on a small traffic segment and automatically paused or rolled back when its metrics diverge from the control. Agent canaries should compare quality, cost, latency, policy violations, human escalations, and side-effect correctness.

Promotion should require explicit gates:

promotion:
  minimum_runs: 1000
  success_rate_delta_min: -0.002
  p95_latency_delta_max: 0.10
  cost_per_success_delta_max: 0.05
  unsafe_action_rate_max: 0.0
  rollback_test_required: true

A deployment should roll back when one critical guardrail fails, even if aggregate task quality improves.

Support Incident Response

An agent incident may involve incorrect outputs, uncontrolled spending, unauthorized actions, data leakage, service degradation, or widespread task failures.

The initial response should prioritize containment:

  • Stop new runs.
  • Cancel active branches.
  • Revoke temporary credentials.
  • Disable affected tools.
  • Freeze configuration changes.
  • Roll back to the last known-good workflow.
  • Preserve traces, logs, checkpoints, and artifacts.
  • Identify external side effects.
  • Notify affected owners.
  • Begin remediation or compensation.

The Incident Response guidance emphasizes predefined roles, centralized coordination, documented timelines, and practiced procedures. Agent systems should similarly define an incident commander, operations lead, communications lead, and subject-matter owners before a production failure occurs.

Important incident measurements include:

\[\operatorname{MTTD} = t_{\mathrm{detect}}-t_{\mathrm{start}}\] \[\operatorname{MTTC} = t_{\mathrm{contain}}-t_{\mathrm{detect}}\] \[\operatorname{MTTR} = t_{\mathrm{recover}}-t_{\mathrm{start}}\]

For agent systems, mean time to containment may be more important than complete recovery because stopping additional side effects immediately limits harm.

Convert Incidents into Durable Improvements

A postmortem should document:

  • User and system impact.
  • Detection mechanism.
  • Timeline.
  • Trigger.
  • Root and contributing causes.
  • Why existing safeguards failed.
  • Mitigation and recovery.
  • Where the system behaved correctly.
  • Concrete preventive actions.
  • Owners, priorities, and deadlines.
  • Regression tests and monitoring changes.

Postmortem Culture: Learning from Failure recommends blameless, evidence-based postmortems with measurable, owned action items. The goal is to improve the system conditions that permitted the failure rather than attribute the failure to an individual operator.

Agent-specific postmortem questions include:

  • Why did the loop believe continued execution was useful?
  • Why was the unsafe edge reachable?
  • Which capability enabled the side effect?
  • Why did verification fail to detect the error?
  • Did retries amplify the incident?
  • Was the current workflow version identifiable?
  • Could the run be reconstructed from telemetry?
  • Did cancellation reach every worker?
  • Was rollback tested?
  • Has the failure been added to the regression suite?

Protect Sensitive Observability Data

Traces may contain prompts, retrieved documents, tool arguments, generated content, personal information, file paths, and credentials. Observability must therefore have its own security policy.

Controls should include:

  • Redaction before export.
  • Secret detection.
  • Content capture disabled by default for sensitive workflows.
  • Hashes or artifact references instead of raw payloads.
  • Attribute-level access control.
  • Tenant isolation.
  • Encryption.
  • Retention limits.
  • Audited access.
  • Deletion procedures.
  • Separate operational and content telemetry.

The OpenAI Agents SDK tracing documentation explicitly distinguishes span metadata from potentially sensitive generation and tool content and provides controls for excluding sensitive inputs and outputs.

Sampling policies should be risk-aware. Routine successful runs may be sampled, while failures, policy denials, security events, and unusual state transitions may require complete traces after appropriate redaction.

Build Role-Specific Operational Views

Different users need different observability views.

Executive View

  • Verified task success.
  • Cost per successful task.
  • User adoption.
  • Human escalation.
  • Reliability against SLOs.
  • Major incidents.
  • Trend in operational risk.

Operator View

  • Active and queued runs.
  • Error-budget consumption.
  • Latency percentiles.
  • Tool and model availability.
  • Saturation.
  • Circuit-breaker state.
  • Current deployments.
  • Incident alerts.

Workflow Engineer View

  • Node-level latency and cost.
  • Edge-selection frequencies.
  • Retry amplification.
  • Verification rejection reasons.
  • Context size by node.
  • Fan-out efficiency.
  • State-transition failures.
  • Trace comparisons between versions.

Evaluation View

  • Quality by task category.
  • Regression-suite performance.
  • Failure taxonomy.
  • Judge disagreement.
  • Human correction rate.
  • Distribution drift.
  • Canary-versus-control comparisons.

A single dashboard cannot serve all four roles without either omitting necessary detail or overwhelming its users.

Common Operational Failure Modes

  • Final-output-only logging: The system stores the answer but not the path that produced it.
  • Missing version provenance: A regression is detected, but the responsible prompt, policy, model, or graph version cannot be identified.
  • Uncorrelated telemetry: Model, tool, database, and graph events use different identifiers and cannot be assembled into one run.
  • Average-only reporting: Mean latency looks acceptable while a growing tail causes timeouts and failed fan-in operations.
  • Success without verification: Runs are marked successful because they reached a terminal node.
  • Cost without outcome normalization: Teams optimize cost per run while the cost per verified success increases.
  • Logging raw sensitive content: Observability becomes a secondary data-leak surface.
  • High-cardinality metrics: Prompts, trace identifiers, and file paths are used as metric labels, making the monitoring system expensive and unstable.
  • Retry opacity: Retries are counted as ordinary calls, hiding dependency degradation and duplicated work.
  • No partial replay: Debugging requires rerunning the entire graph, including expensive or irreversible operations.
  • Untested rollback: A prior version exists but cannot be safely restored under incident conditions.
  • Canary without segmentation: Aggregate metrics conceal regressions concentrated in specific task classes.
  • Alerting on every internal error: Operators become desensitized to alerts that do not correspond to user-visible impact.
  • Unclosed postmortem actions: Incidents produce documentation without durable system changes.

Operations Principle

A production agent system should be treated as an observable, versioned, replayable control system:

\[\operatorname{OperationalMaturity} = \operatorname{Traceability} \land \operatorname{Measurability} \land \operatorname{Reproducibility} \land \operatorname{ControlledDeployment} \land \operatorname{Recoverability} \land \operatorname{ContinuousLearning}\]

The operational objective is not simply to keep the graph running. It is to know when it is correct, detect when it is deteriorating, identify which component caused the change, limit the effect of failures, and convert production evidence into tested improvements.

Failure Modes and Recovery Patterns

Treat Failure as a State, Not an Exception

A loop or graph has failed whenever it can no longer make valid progress toward its objective within its authorized constraints. The runtime does not need to crash for this condition to occur. A workflow may remain active while repeatedly selecting ineffective actions, accepting invalid outputs, consuming unbounded resources, or silently diverging from the user’s objective.

Failure should therefore be modeled as part of the workflow state:

\[s_t = \left( x_t, g_t, p_t, r_t, q_t, e_t \right)\]

where:

  • \(x_t\) is the current task state.
  • \(g_t\) is the graph execution state.
  • \(p_t\) is measured progress.
  • \(r_t\) is the remaining resource budget.
  • \(q_t\) is the current verification or quality state.
  • \(e_t\) is the active error state.

A recovery decision should depend on the complete state rather than the last exception:

\[\rho_t = \pi_{\mathrm{recovery}} \left( s_t, e_t, h_t, \mathcal{C}_t \right)\]

where \(h_t\) is the execution history and \(\mathcal{C}_t\) is the set of recovery capabilities currently permitted.

Where LLM Agents Fail and How They can Learn From Failures by Zhu et al. (2025) presents a modular taxonomy spanning memory, reflection, planning, action, and system operations, showing how an early root-cause error can propagate through later decisions and produce a cascading end-to-end failure.

Build a Failure Taxonomy Around the Control Path

Failure categories should correspond to distinct control-path stages because each category requires different evidence and recovery behavior.

A practical taxonomy includes:

  • Objective failures: The task is ambiguous, contradictory, infeasible, or outside the authorized scope.
  • Observation failures: Required evidence is missing, stale, malformed, inaccessible, or incorrectly interpreted.
  • Planning failures: The plan omits necessary steps, violates constraints, assumes unavailable capabilities, or chooses an inefficient decomposition.
  • Routing failures: The graph selects the wrong node, repeats a completed branch, or bypasses required verification.
  • Action failures: A model emits an invalid tool call, targets the wrong resource, or executes an unauthorized operation.
  • Tool failures: A dependency times out, returns an error, produces malformed output, or succeeds without returning confirmation.
  • Verification failures: The verifier incorrectly accepts a bad result, rejects a correct one, or evaluates the wrong artifact.
  • State failures: Memory is stale, inconsistent, poisoned, overwritten, or incompletely checkpointed.
  • Coordination failures: Workers duplicate work, race on shared state, deadlock, or produce incompatible results.
  • Resource failures: The loop exceeds token, monetary, latency, storage, or concurrency budgets.
  • Safety failures: Untrusted content affects control flow, credentials are overexposed, or a prohibited side effect becomes reachable.
  • Human-control failures: Approval is ambiguous, delayed, stale, or granted without adequate evidence.

The recovery system should classify both the visible symptom and the likely root cause. A failed test is a symptom. Its cause may be an incorrect patch, an incompatible dependency, a stale test environment, a flaky oracle, or an unrelated concurrent change.

Distinguish Fail-Stop, Fail-Slow, and Semantic Failures

Failures should first be classified by how they manifest.

Fail-Stop Failures

A fail-stop component terminates or reports an explicit error. Examples include:

  • A process exits with a nonzero status.
  • A model request returns an error.
  • A tool rejects invalid arguments.
  • A database transaction aborts.
  • A worker misses its heartbeat and disappears.

These failures are comparatively easy to detect because the failure signal is explicit.

Fail-Slow and Gray Failures

A component may continue responding while behaving abnormally. It may become slow, intermittently drop operations, return partial results, or disagree with other observers.

Gray Failure: The Achilles’ Heel of Cloud-Scale Systems by Huang et al. (2017) identifies differential observability as the defining property of gray failure: the affected application perceives a failure that the component’s own health detector does not recognize.

Agent examples include:

  • A retrieval system returns results but omits newly indexed records.
  • A tool returns success while applying only part of a mutation.
  • A model endpoint remains available but produces malformed tool calls at an elevated rate.
  • A verifier becomes systematically permissive after a prompt change.
  • A worker continues sending heartbeats while repeating the same action.
  • A queue accepts work but delivers it with extreme delay.

Health checks should therefore measure task-relevant behavior, not only process availability.

Semantic Failures

A semantic failure occurs when the system executes successfully at the mechanical level but produces the wrong outcome. Examples include:

  • Sending a polished message to the wrong recipient.
  • Editing the wrong configuration file.
  • Passing an incomplete test suite.
  • Producing a factually unsupported report.
  • Optimizing a proxy metric while degrading the real objective.
  • Reaching consensus on an incorrect conclusion.

Semantic failures require external validation, provenance checks, constraint evaluation, and outcome-aware monitoring. Infrastructure uptime alone cannot detect them.

Detect Runaway Loops

A runaway loop continues consuming resources without approaching a valid terminal state.

Loop Engineering: A Technical Roadmap for an Autonomous Loop distinguishes runaway execution from other failures through structured traces showing repeated model calls, increasing cost, and no successful terminal event.

Useful detection signals include:

  • Iteration count exceeds a hard limit.
  • Cost exceeds a run budget.
  • Wall-clock execution exceeds its deadline.
  • No objective improvement occurs within a progress window.
  • The same tool and argument combination repeats.
  • The same failure fingerprint recurs.
  • Context or output size grows without corresponding progress.
  • The graph revisits the same state cycle.

For progress measure \(J_t\), declare a stall when:

\[\max_{k\in\{t-w+1,\ldots,t\}} \left( J_k-J_{k-1} \right) \leq \epsilon\]

for a window of \(w\) iterations and minimum meaningful improvement \(\epsilon\).

A basic runaway condition is:

\[\operatorname{runaway}_t = \mathbb{I}\left[t>T_{\max}\right] \lor \mathbb{I}\left[C_t>C_{\max}\right] \lor \mathbb{I}\left[\operatorname{stall}_t=1\right] \lor \mathbb{I}\left[\operatorname{cycle}_t=1\right]\]

The default recovery should be safe halt and diagnosis, not automatic continuation with a larger budget. Raising the limit is appropriate only when evidence indicates that useful progress was occurring.

Detect Oscillation and Random Walks

A loop may alternate between competing states without converging. For example, one iteration adds a dependency and the next removes it, or one worker changes a configuration that another worker later restores.

Oscillation can be detected through repeated state hashes:

\[H(s_t)=H(s_{t-k})\]

for some cycle length \(k\), with no improvement in the objective.

A random walk differs from oscillation. Its state changes continuously, but the changes do not move consistently toward the goal. A simple directional progress score is:

\[D_t = \frac{ \sum_{i=t-w+1}^{t} \mathbb{I}\left[\Delta J_i>0\right] - \sum_{i=t-w+1}^{t} \mathbb{I}\left[\Delta J_i<0\right] }{ w }\]

A value near zero with continuing resource consumption suggests undirected exploration.

Recovery strategies include:

  • Restore the best known checkpoint.
  • Freeze decisions that have already passed verification.
  • Replace open-ended reflection with a constrained decision.
  • Request an independent diagnostic agent.
  • Change the decomposition rather than repeating the same plan.
  • Reduce the action space.
  • Escalate when competing objectives cannot be resolved mechanically.

Detect Premature and False Convergence

A loop may stop too early because it mistakes activity completion for goal completion. Common causes include:

  • All graph nodes ran, but the external objective was not satisfied.
  • A model reported completion without executing the required tool.
  • A local verifier passed while a global constraint remained violated.
  • A worker completed its branch, but fan-in omitted another required branch.
  • The stop condition inspected stale state.
  • The agent modified the test or evaluator instead of satisfying it.

Completion should be a conjunction of independently checked conditions:

\[\operatorname{complete} = \operatorname{goalSatisfied} \land \operatorname{constraintsSatisfied} \land \operatorname{requiredBranchesComplete} \land \operatorname{externalEffectsVerified} \land \operatorname{noPendingActions}\]

A terminal node should produce a completion certificate containing:

  • The objective version.
  • The final artifact identity.
  • The verifier identity and version.
  • Evidence for each success criterion.
  • External side-effect confirmations.
  • Remaining warnings.
  • The state checkpoint from which the result can be reproduced.

If the certificate is incomplete, the run should enter an unresolved state rather than being labeled successful.

Defend Against Reward Hacking

Reward hacking occurs when the system improves its measured score without accomplishing the intended task.

Examples include:

  • Modifying tests instead of fixing the implementation.
  • Removing difficult evaluation cases.
  • Hardcoding expected outputs.
  • Suppressing error messages.
  • Selecting an easier verifier.
  • Editing state to mark a task complete.
  • Optimizing stylistic judge preferences while reducing factual accuracy.

The primary defense is to separate the actor’s authority from the evaluator’s authority. The agent should not be able to modify the success criterion, verification dataset, or trusted test artifacts during the run.

The verification boundary should enforce:

\[\operatorname{WritableByActor} \cap \operatorname{TrustedVerifierInputs} = \varnothing\]

Additional controls include:

  • Hashing trusted tests before execution.
  • Running verification in a separate environment.
  • Comparing verifier inputs before and after the action.
  • Maintaining hidden evaluation cases.
  • Checking for deleted or weakened constraints.
  • Using independent process and outcome checks.
  • Requiring adversarial review for suspiciously large score jumps.

A passing score is evidence only when the measurement process itself remains intact.

Handle Tool and Dependency Failures Explicitly

A tool invocation should return a structured outcome rather than an untyped exception:

{
  "status": "failed",
  "failure_class": "transient_dependency",
  "operation": "fetch_repository",
  "attempt": 2,
  "retryable": true,
  "side_effect_state": "none",
  "idempotency_key": "run-17-fetch-repo-2",
  "error_code": "UPSTREAM_TIMEOUT",
  "retry_after_ms": 4000,
  "evidence": {
    "request_sent": true,
    "response_received": false
  }
}

The most important field is the side-effect state:

  • None: The operation definitely did not execute.
  • Committed: The operation definitely executed.
  • Partial: Some effects occurred.
  • Unknown: The caller cannot determine whether it executed.

A timeout does not imply that no side effect occurred. The remote operation may have committed and lost its response. Retrying such an operation without deduplication can produce duplicate payments, messages, records, deployments, or file changes.

Every mutating tool should accept an idempotency key:

\[K = H\left( \operatorname{workflowId}, \operatorname{logicalOperationId}, \operatorname{target}, \operatorname{payload} \right)\]

The receiver should persist the outcome associated with \(K\) and return that outcome when the same logical operation is submitted again.

Retry Only When the Failure Is Transient

Retries should be selected by policy, not used as a universal exception handler.

Retryable conditions may include:

  • Temporary network failures.
  • Explicit rate limiting.
  • Dependency overload with a retry interval.
  • Short-lived lock conflicts.
  • Transient model-provider errors.
  • Worker termination before any side effect occurred.

Non-retryable conditions include:

  • Authentication and authorization failures.
  • Invalid schemas.
  • Prohibited actions.
  • Missing required information.
  • Deterministic test failures.
  • Policy denials.
  • Unknown side-effect state without deduplication.
  • Repeated failures after the retry budget is exhausted.

The Timeouts, Retries, and Backoff with Jitter guidance explains that retries can improve availability for rare transient faults but can intensify overload, duplicate side effects, and delay recovery when applied indiscriminately.

A bounded exponential-backoff schedule is:

\[d_k = \min \left( d_{\max}, d_0 2^k \right) + U\left(0,j_k\right)\]

where \(d_0\) is the initial delay, \(d_{\max}\) is the cap, and \(U(0,j_k)\) is jitter.

The retry policy should also enforce:

  • A maximum attempt count.
  • An end-to-end deadline.
  • A retry budget shared across the graph.
  • A single retry layer for each dependency path.
  • Idempotency for mutating operations.
  • Cancellation propagation.
  • Respect for provider-supplied retry intervals.

If a dependency chain of depth \(n\) retries up to \(r\) times at every layer, the worst-case number of downstream attempts can grow as:

\[A_{\max} = r^n\]

Retries should therefore be coordinated centrally or performed at one designated layer.

Use Circuit Breakers for Persistent Failures

The Circuit Breaker pattern prevents repeated calls to an unhealthy dependency by moving through closed, open, and half-open states.

For agent systems:

  • Closed: Operations proceed while failures are counted.
  • Open: Requests fail immediately, route to a fallback, or enter deferred processing.
  • Half-open: A limited number of probe operations test whether recovery has occurred.

A breaker may open when:

\[\frac{ \operatorname{failures}(t-w,t) }{ \operatorname{attempts}(t-w,t) } > \tau_f\]

subject to a minimum sample count.

Breakers should be scoped carefully. A failure in one model, tenant, tool operation, region, or data shard should not necessarily disable every use of that service.

When the breaker opens, the graph may:

  • Use a cached or previously verified result.
  • Route to an alternate tool or provider.
  • Reduce functionality.
  • Defer the task.
  • Request human intervention.
  • Terminate with a clear recoverable status.

A fallback should not silently change the task’s semantics. The output must state when degraded behavior was used.

Manage Partial Graph Failure

Parallel graphs routinely produce mixed outcomes. Some branches succeed, some fail, some time out, and some remain unresolved.

A fan-in node should declare an explicit completion policy:

  • All-required: Every required branch must succeed.
  • Quorum: A defined threshold of independent branches must succeed.
  • Best-effort: Use all available results after the deadline.
  • First-valid: Accept the first result that passes verification.
  • Priority-weighted: Require high-priority branches while treating others as optional.

For branch set \(B\) with required subset \(R\):

\[\operatorname{fanInReady} = \left( \bigwedge_{b\in R} \operatorname{success}(b) \right) \land \left( \operatorname{deadlineReached} \lor \bigwedge_{b\in B} \operatorname{terminal}(b) \right)\]

The synthesis node should receive branch status along with branch content:

{
  "branch": "security-review",
  "status": "timeout",
  "required": true,
  "attempts": 2,
  "last_checkpoint": "artifact://security-review/checkpoint/3",
  "partial_output": null
}

Missing results must never be represented as empty successful results. The difference between “no issue found” and “review did not complete” is operationally critical.

Prevent Shared-State Races

Parallel workers should not modify the same mutable resource without coordination. Common failures include:

  • Overwriting another agent’s edits.
  • Reading a partially written artifact.
  • Resetting a branch containing another worker’s work.
  • Applying patches against stale revisions.
  • Running tests against mixed states.
  • Reusing temporary files or port numbers.
  • Consuming another worker’s lock or checkpoint.

The following figure (source) shows how two agents sharing one workspace can race and corrupt one another’s work, while isolated worktrees create separate mutation boundaries that can be verified before integration.

Isolation mechanisms include:

  • One worktree or branch per worker.
  • One database transaction or namespace per run.
  • Optimistic concurrency using expected versions.
  • Leases with explicit expiration.
  • Immutable intermediate artifacts.
  • Single-writer ownership.
  • Deterministic merge nodes.
  • Conflict detection before integration.

An optimistic write should succeed only when:

\[v_{\mathrm{current}} = v_{\mathrm{expected}}\]

If the versions differ, the worker must reread the new state and replan. Silently overwriting the newer version converts a detectable conflict into hidden corruption.

Prevent Deadlock and Livelock

A deadlock occurs when nodes wait indefinitely for one another. A livelock occurs when nodes remain active but repeatedly respond to each other without accomplishing work.

Deadlock examples include:

  • Two workers each waiting for the other’s output.
  • A fan-in waiting for a branch that was never scheduled.
  • A worker holding a lease while waiting for approval.
  • A coordinator waiting for cancellation acknowledgment from a terminated process.

Livelock examples include:

  • Agents repeatedly requesting clarification from one another.
  • Two reviewers alternately reversing the same decision.
  • A planner and verifier exchanging unchanged artifacts.
  • Workers continuously releasing and reacquiring the same contested resource.

Detection should use:

  • Wait-for graphs.
  • Lease expiration.
  • Maximum node dwell time.
  • Repeated-message fingerprints.
  • Progress counters.
  • Heartbeats containing state, not merely liveness.

For wait-for graph \(W\), a cycle indicates potential deadlock:

\[\exists \left( v_1, v_2, \ldots, v_k \right) : v_1\rightarrow v_2 \rightarrow\cdots\rightarrow v_k\rightarrow v_1\]

Recovery may involve cancelling the lowest-priority branch, expiring a lease, restoring a checkpoint, or escalating the dependency cycle to the coordinator.

Account for Correlated Agent Failures

Multiple agents are not independent merely because they execute in separate processes. They may share:

  • The same model.
  • The same prompt structure.
  • The same retrieved evidence.
  • The same incorrect assumption.
  • The same poisoned memory.
  • The same tool defect.
  • The same verifier.
  • The same training-induced bias.

If agent failures have pairwise correlation \(\rho\), majority voting provides less reliability improvement than an independence assumption predicts.

For agents with individual error variance \(\sigma^2\), the variance of their mean is:

\[\operatorname{Var}(\bar{X}) = \frac{\sigma^2}{n} \left( 1+(n-1)\rho \right)\]

As \(\rho\) approaches one, adding agents provides little variance reduction.

Independent verification should diversify at least one relevant dimension:

  • Model family.
  • Prompt and reasoning method.
  • Evidence source.
  • Tool implementation.
  • Verification oracle.
  • Organizational role.
  • Execution environment.

Consensus should be evidence-weighted rather than based solely on vote count. Agreement after agents have copied or discussed the same conclusion is weaker evidence than independent agreement formed before communication.

Protect State and Memory During Recovery

Recovery can reintroduce a failure if the checkpoint already contains corrupted state.

Before resuming, classify persisted state as:

  • Trusted: Independently verified and immutable.
  • Tentative: Produced by an agent but not yet verified.
  • Derived: Reconstructable from trusted sources.
  • External: Controlled by another system.
  • Suspect: Associated with the failed execution path.
  • Revoked: Known to be invalid and prohibited from reuse.

A recovery checkpoint should include:

\[\chi_t = \left( s_t, v_t, a_t, o_t, \mathcal{L}_t \right)\]

where \(s_t\) is state, \(v_t\) is its version, \(a_t\) is the last committed action, \(o_t\) records external side effects, and \(\mathcal{L}_t\) contains active locks or leases.

Memory written after the suspected root-cause step should be quarantined until revalidated. Otherwise, restarting the loop may immediately reproduce the same failure from poisoned context.

Select Recovery by Failure Class

Recovery actions should be explicit and ordered from least disruptive to most disruptive:

  • Retry: Repeat the same operation when the fault is transient and the action is idempotent.
  • Resume: Continue from a valid checkpoint after infrastructure recovery.
  • Reobserve: Refresh stale or incomplete evidence before replanning.
  • Replan: Construct a materially different plan from the current state.
  • Fallback: Use an alternate model, tool, provider, or method.
  • Degrade: Complete a reduced but clearly disclosed version of the task.
  • Roll back: Restore a previously valid internal state.
  • Compensate: Apply a new external action that offsets a committed side effect.
  • Quarantine: Isolate suspect data, memory, artifacts, workers, or tools.
  • Escalate: Transfer control to a human with the trace and evidence.
  • Abort: Terminate safely when no authorized recovery path exists.

The recovery matrix should be policy-driven:

recovery:
  transient_dependency:
    action: retry
    max_attempts: 3
    backoff: exponential_jitter

  invalid_tool_arguments:
    action: replan
    max_attempts: 1

  unknown_side_effect_state:
    action: reconcile
    automatic_retry: false

  verifier_integrity_failure:
    action: quarantine
    escalate: true

  budget_exhausted:
    action: safe_halt

  prohibited_action:
    action: abort
    security_event: true

Recovery logic should live outside the model. The model may recommend a strategy, but the orchestrator must enforce whether that strategy is permitted.

Separate Rollback from Compensation

Rollback restores internal state to a prior version. Compensation issues a new action to offset a side effect that has already escaped the system boundary.

Examples:

  • A file change can be rolled back through version control.
  • A database transaction may be rolled back before commitment.
  • A sent email cannot usually be rolled back, but a correction can be sent.
  • A completed payment may require a refund.
  • A published artifact may require withdrawal and notification.
  • A created account may require deactivation rather than historical erasure.

Sagas by Garcia-Molina and Salem (1987) models long-running operations as sequences of transactions with compensating actions when partial execution prevents all steps from completing.

For actions \(T_1,\ldots,T_n\) with compensations \(C_1,\ldots,C_n\), failure after \(T_k\) may require:

\[T_1, T_2, \ldots, T_k, C_k, C_{k-1}, \ldots, C_1\]

Compensations must be designed, authorized, and tested before the corresponding forward actions are automated. A textual claim that an operation is reversible is not a recovery mechanism.

Reconcile Unknown Side Effects

When an operation times out after being sent, the orchestrator may not know whether it committed. The correct response is reconciliation:

  1. Query the target system using the idempotency key or external identifier.
  2. Compare the observed external state with the intended state.
  3. Classify the operation as committed, not committed, partial, or still unknown.
  4. Retry only if non-commitment is established or deduplication is guaranteed.
  5. Compensate or escalate when the observed state is incorrect.

The state machine should include an explicit unknown outcome:

\[\operatorname{Outcome} \in \left\{ \operatorname{NotCommitted}, \operatorname{Committed}, \operatorname{Partial}, \operatorname{Unknown} \right\}\]

Treating unknown as failed is unsafe because it can duplicate side effects. Treating unknown as successful is also unsafe because required work may be omitted.

Use a Recovery Router

A recovery router can translate structured errors into deterministic control decisions:

from dataclasses import dataclass
from enum import Enum


class FailureClass(Enum):
    TRANSIENT = "transient"
    INVALID_INPUT = "invalid_input"
    CONFLICT = "conflict"
    UNKNOWN_SIDE_EFFECT = "unknown_side_effect"
    VERIFICATION = "verification"
    POLICY = "policy"
    BUDGET = "budget"


class RecoveryAction(Enum):
    RETRY = "retry"
    REOBSERVE = "reobserve"
    REPLAN = "replan"
    RECONCILE = "reconcile"
    ESCALATE = "escalate"
    ABORT = "abort"
    SAFE_HALT = "safe_halt"


@dataclass(frozen=True)
class Failure:
    failure_class: FailureClass
    retryable: bool
    attempt: int
    max_attempts: int
    side_effect_known: bool


def choose_recovery(failure: Failure) -> RecoveryAction:
    if failure.failure_class == FailureClass.POLICY:
        return RecoveryAction.ABORT

    if failure.failure_class == FailureClass.BUDGET:
        return RecoveryAction.SAFE_HALT

    if failure.failure_class == FailureClass.UNKNOWN_SIDE_EFFECT:
        return RecoveryAction.RECONCILE

    if failure.failure_class == FailureClass.CONFLICT:
        return RecoveryAction.REOBSERVE

    if failure.failure_class == FailureClass.VERIFICATION:
        return RecoveryAction.REPLAN

    if (
        failure.failure_class == FailureClass.TRANSIENT
        and failure.retryable
        and failure.side_effect_known
        and failure.attempt < failure.max_attempts
    ):
        return RecoveryAction.RETRY

    return RecoveryAction.ESCALATE

Production routing should also consider risk class, deadline, remaining budget, dependency health, prior recovery attempts, tenant policy, and availability of compensating actions.

Test Recovery with Fault Injection

Recovery mechanisms that are never exercised should be treated as unverified.

Fault-injection tests should simulate:

  • Model timeouts.
  • Malformed tool responses.
  • Partial writes.
  • Lost acknowledgments.
  • Duplicate message delivery.
  • Worker crashes before and after checkpointing.
  • Stale reads.
  • Lock expiration.
  • Network partitions.
  • Slow dependencies.
  • Corrupted memory entries.
  • Verifier disagreement.
  • Cancellation during fan-out.
  • Human approval that expires before execution.
  • Recovery while the underlying dependency remains unhealthy.

Each exercise should verify:

  • The failure is detected.
  • The correct recovery policy is selected.
  • Side effects are not duplicated.
  • Budgets remain bounded.
  • Cancellation reaches all descendants.
  • The final state is known.
  • The trace is sufficient for diagnosis.
  • Operators receive a useful escalation package.

A fault-injection test is successful only when the system reaches a defined safe state, not merely when it avoids crashing.

Design Human Escalation as a Recovery Path

Human escalation should preserve the work already completed and present the decision in a bounded form.

An escalation package should include:

  • The original objective.
  • The current state.
  • The suspected root cause.
  • The critical trace segment.
  • Actions already attempted.
  • External side effects.
  • Remaining uncertainty.
  • Available recovery options.
  • Risk associated with each option.
  • The recommended action.
  • The exact authority requested.

The human should not be asked to inspect an unfiltered transcript or reconstruct the workflow from raw logs. The system should identify the earliest critical failure step and link it to supporting evidence.

Escalation should occur when:

  • Side-effect state remains unknown.
  • Verification mechanisms disagree.
  • A high-impact compensation is required.
  • The recovery policy has no authorized automatic action.
  • Repeated recoveries fail.
  • The task objective is ambiguous.
  • Sensitive or regulated judgment is required.
  • A security boundary may have been crossed.

Failure-Handling Anti-Patterns

  • Catch-all retry: Every error is treated as transient.
  • Retry at every layer: Nested retries multiply downstream load and cost.
  • Retry without idempotency: A lost response becomes a duplicated external action.
  • Restart from the beginning: Valid completed work is discarded, and committed side effects may be repeated.
  • Resume from the latest checkpoint: The latest checkpoint is used even when it contains the root-cause corruption.
  • Model-selected recovery authority: The same model that failed decides whether it may expand permissions or budgets.
  • Empty partial results: A missing branch is represented as a successful branch with no findings.
  • Consensus as truth: Correlated agents agree and are treated as independent evidence.
  • Liveness as progress: Heartbeats continue while the workflow remains stuck.
  • Unbounded reflection: The system responds to failure with repeated reasoning but no new evidence or action.
  • Global circuit breaker: Failure in one tenant, region, operation, or shard disables all use of a dependency.
  • Silent fallback: A weaker tool or model changes task semantics without disclosure.
  • Compensation as an afterthought: Irreversible actions are automated before recovery actions are designed.
  • Human escalation without evidence: The operator receives a large transcript and an ambiguous request to intervene.
  • Recovery without regression testing: The immediate run succeeds, but the same failure remains possible.

Recovery Principle

Resilience is not the ability to retry indefinitely. It is the ability to detect the earliest meaningful failure, contain its propagation, preserve valid work, reconcile uncertain side effects, and transition to a known safe state.

\[\operatorname{Resilience} = \operatorname{Detection} \land \operatorname{Containment} \land \operatorname{StateIntegrity} \land \operatorname{BoundedRecovery} \land \operatorname{SideEffectReconciliation} \land \operatorname{VerifiedReturnToService}\]

A well-engineered loop assumes that individual model calls, tools, workers, verifiers, and operators can fail. A well-engineered graph assumes that these failures can interact and propagate. Recovery architecture determines whether those local failures remain bounded incidents or become system-wide failures.

Production Architectures and End-to-End Reference Designs

Treat the Workflow as the Deployment Unit

A production agent system should be deployed and governed as a workflow rather than as an isolated model call. The deployable unit includes the graph definition, node contracts, model and tool configurations, verification policies, checkpoint schema, safety rules, observability configuration, and rollback procedure. This boundary makes releases reproducible because every behaviorally significant dependency can be versioned and evaluated together. Temporal Workflow documentation describes the corresponding durable-execution model, in which application logic is represented as a recoverable workflow whose progress survives process and infrastructure failures.

The workflow version should identify at least the graph topology, prompt set, model routing policy, tool registry, evaluator set, and state schema. A compact release identity can be represented as:

\[V_{\mathrm{workflow}} = H \left( V_{\mathrm{graph}}, V_{\mathrm{prompts}}, V_{\mathrm{models}}, V_{\mathrm{tools}}, V_{\mathrm{evaluators}}, V_{\mathrm{state}} \right)\]

where \(H\) is a deterministic content hash. A run must record this identity so that an observed result can be traced to the exact system configuration that produced it. The OpenAI Agents SDK tracing documentation describes tracing as a structured record of generations, tool calls, handoffs, and guardrails, which provides the execution lineage required for this form of reproducibility.

Separate the Architecture into Operational Planes

The architecture should be divided into planes whose responsibilities and failure modes are distinct. This separation prevents model behavior, workflow control, external side effects, and operational telemetry from becoming inseparable parts of a single application process. The Twelve-Factor App establishes the broader operational value of explicit configuration, disposable processes, and independently managed backing services, while Temporal Workflow documentation provides the durable-control concepts needed for long-running agent execution.

Experience Plane

The experience plane accepts user or system requests, performs authentication, resolves tenancy, normalizes inputs, and presents intermediate or final results. It should not directly execute privileged tools. Instead, it creates a typed task envelope containing the request, principal, tenant, authorization context, deadlines, budget, and required response contract. This preserves a stable external interface while the internal graph evolves.

Control Plane

The control plane owns graph traversal, scheduling, retries, timeouts, concurrency limits, cancellation, checkpointing, and resumability. It decides which node is eligible to run but does not decide the semantic content of that node’s output. A durable workflow engine is appropriate when runs may outlive a process, require human approval, or interact with systems that can fail independently. Temporal Workflow documentation explains how event history and deterministic replay allow workflow progress to be reconstructed after failure.

Intelligence Plane

The intelligence plane contains model gateways, prompt templates, model-selection policies, structured-output adapters, context assembly, and inference caches. Centralizing these concerns makes it possible to apply model allowlists, rate limits, fallback rules, and cost attribution consistently. The OpenAI Agents SDK model documentation illustrates a model abstraction that allows agent logic to remain separate from provider-specific invocation details.

Execution Plane

The execution plane performs deterministic computation and invokes external tools. It should expose narrow, typed capabilities rather than raw shell, database, or network access. Each operation should declare authorization requirements, idempotency behavior, timeout policy, input schema, output schema, and side-effect class. Model Context Protocol architecture describes a client-host-server structure for exposing tools and resources through explicit capability boundaries.

Data Plane

The data plane stores durable workflow state, checkpoints, artifacts, evidence, memory, caches, and audit records. State should be partitioned by tenant and classified by durability, sensitivity, and retention policy. Operational checkpoints, semantic memory, and transient model context are different data products and should not share a lifecycle merely because all three are called memory.

Verification Plane

The verification plane evaluates intermediate and final artifacts against executable tests, schemas, policies, factual evidence, and task-specific rubrics. Keeping verification separate from generation reduces the risk that the same assumptions and context produce both an error and its approval. Self-Refine: Iterative Refinement with Self-Feedback by Madaan et al. (2023) demonstrates that iterative feedback can improve outputs, while production systems strengthen this pattern by combining model-based critique with independent deterministic checks.

Observability Plane

The observability plane collects traces, structured events, metrics, evaluation outcomes, cost, latency, model usage, tool activity, and policy decisions. It should reconstruct both the control path and the semantic path: what executed, why it executed, what evidence it used, and which condition allowed the run to continue. OpenTelemetry documentation defines the standard telemetry primitives of traces, metrics, and logs that can be adapted to graph runs and node-level spans.

Define Typed Contracts Between Nodes

Every node should consume and produce a versioned contract. Free-form prose can remain inside fields intended for natural language, but routing and control should depend on validated structured fields. A node result commonly includes a status, artifact references, evidence references, confidence or quality estimates, error metadata, budget usage, and a requested transition.

{
  "schema_version": "1.2",
  "node_id": "verify_patch",
  "status": "needs_revision",
  "artifacts": ["artifact://patch/27"],
  "evidence": ["evidence://test-run/811"],
  "quality": {
    "tests_passed": 41,
    "tests_failed": 2
  },
  "budget": {
    "tokens_used": 18240,
    "elapsed_ms": 71321
  },
  "transition": {
    "target": "repair_patch",
    "reason_code": "FAILING_REGRESSION_TEST"
  }
}

The orchestrator should reject outputs that fail schema validation rather than infer control state from ambiguous prose. Structured contracts also allow deterministic migration when a graph version changes. JSON Schema provides a standard mechanism for constraining and validating JSON documents, while Protocol Buffers provides a strongly typed interface-definition approach when binary transport and generated clients are required.

Use a Durable Orchestrator for Long-Running Work

An in-process loop is adequate only when work is short, side effects are limited, and losing progress is acceptable. A durable orchestrator becomes necessary when runs involve long model calls, parallel branches, external services, human approvals, retries across deployments, or hours of elapsed time. Temporal Workflow documentation describes durable workflows as recoverable functions whose execution history is persisted independently of the worker process.

Durability requires deterministic control logic and isolated non-deterministic activities. Model calls, tool calls, wall-clock reads, random selection, and network requests should be recorded as activities or events rather than recomputed implicitly during replay. The workflow history then becomes the authoritative sequence from which the control state can be reconstructed.

The orchestrator should distinguish at least four retry classes:

  • Transient infrastructure failures, which may be retried with exponential backoff and jitter.

  • Model-format failures, which may be retried with a constrained repair prompt or alternate model.

  • Semantic failures, which should return to a revision node with evaluator feedback.

  • Permanent policy or authorization failures, which should terminate or require explicit human intervention.

For retry attempt \(k\), a bounded exponential delay can be expressed as:

\[d_k = \min \left( d_{\max}, d_0 2^k \right) + U(0,j)\]

where \(d_0\) is the initial delay, \(d_{\max}\) is the cap, and \(U(0,j)\) is random jitter. Exponential Backoff And Jitter explains why jitter reduces synchronized retry contention in distributed systems.

Build Tool Access Through a Gateway

All consequential tool use should pass through a gateway that authenticates the caller, validates parameters, enforces policy, records intent, applies rate limits, and returns a normalized result. The gateway should expose business-level operations such as create_refund_request or propose_database_migration, not unconstrained credentials or generic command execution. Model Context Protocol architecture provides an interoperable pattern for presenting tools and resources through servers while keeping the host responsible for permissions, consent, and orchestration.

The gateway should classify tools by side-effect level:

\[L(t) \in \{ \mathrm{read}, \mathrm{prepare}, \mathrm{reversible}, \mathrm{irreversible} \}\]

Read operations may execute automatically within policy. Prepared operations generate a reviewable action without applying it. Reversible operations require idempotency and compensation. Irreversible operations require the strongest authorization and, for high-impact cases, human approval. This classification should influence routing, approval requirements, checkpoint placement, and audit retention.

Reference Architecture for a Single-Agent Verification Loop

Topology

The smallest useful production architecture consists of an intake node, planner, executor, verifier, revision path, and completion gate:

intake -> plan -> execute -> verify -> complete
                     ^          |
                     |----------|

The graph remains explicit even though only one agent role performs most semantic work. This design is appropriate when the task is sequential, the artifact has strong automated checks, and parallel exploration would add coordination cost without improving coverage. Self-Refine: Iterative Refinement with Self-Feedback by Madaan et al. (2023) supports the underlying generate-feedback-refine pattern and reports improvements across multiple tasks without additional supervised training.

Appropriate Workloads

Suitable workloads include localized code changes, document transformations, data-cleaning jobs, configuration generation, and analytical tasks with executable acceptance criteria. The architecture is less suitable when the problem naturally decomposes into independent research areas or requires distinct permissions and expertise.

Required Components

The minimum implementation should include:

  • A typed task specification with explicit completion criteria.

  • A bounded working-state object.

  • One or more deterministic evaluators.

  • A revision counter and global budget.

  • A stagnation detector.

  • A checkpoint before consequential side effects.

  • A terminal record containing artifacts, evidence, and the reason for termination.

Execution Algorithm

while state.revision_count < limits.max_revisions:
    candidate = execute(state.plan, state.context)
    report = verify(candidate, state.acceptance_criteria)

    if report.passed:
        return complete(candidate, report)

    if stagnated(state.history, report):
        return escalate("stagnation", candidate, report)

    state = revise_state(state, candidate, report)

return escalate("revision_limit", state.latest_candidate, state.latest_report)

The loop should terminate only when an acceptance predicate passes, a budget is exhausted, a policy blocks continuation, progress stagnates, or an authorized operator cancels the run. Termination based solely on the model declaring completion is not sufficiently independent.

Scaling Characteristics

The sequential latency is approximately:

\[T_{\mathrm{seq}} = \sum_{i=1}^{n} \left( T_{\mathrm{model},i} + T_{\mathrm{tool},i} + T_{\mathrm{verify},i} \right)\]

This architecture minimizes coordination overhead but cannot hide slow node latency. Its principal scaling mechanisms are model routing, caching, prompt and context reduction, speculative preparation of deterministic resources, and faster evaluators.

Reference Architecture for Parallel Research and Analysis

Topology

A research graph should separate decomposition, independent collection, evidence normalization, synthesis, and final verification:

request -> decompose -> fan-out researchers -> evidence registry
                                      |                |
                                      +------> synthesize -> verify -> answer

This architecture follows the same broad partition-and-aggregate principle as MapReduce: Simplified Data Processing on Large Clusters by Dean and Ghemawat (2004), which separates parallel mapping from deterministic aggregation over intermediate results.

Planning Contract

The decomposition node should emit a set of bounded work packages:

\[W = \{ w_1,w_2,\ldots,w_m \}\]

Each package should define its question, scope, excluded overlap, required source classes, freshness requirement, budget, and output schema. Explicit exclusions reduce duplicated research, while required source classes prevent all branches from relying on the same weak evidence.

Evidence Registry

Researchers should write normalized evidence records rather than return only prose. Each record should contain the claim, source, retrieval time, quotation or extracted field where permitted, confidence, provenance, and contradiction links. The synthesizer should reason over these records and cite them directly, preserving the distinction between source evidence and generated interpretation.

Fan-Out Sizing

Parallelism should be based on independent information needs rather than a fixed desire for more agents. If branch times are \(T_1,\ldots,T_m\), the idealized critical path is:

\[T_{\mathrm{parallel}} \approx T_{\mathrm{decompose}} + \max_i(T_i) + T_{\mathrm{synthesize}} + T_{\mathrm{verify}}\]

The realized latency also includes queueing, rate limiting, stragglers, and integration overhead. The Tail at Scale by Dean and Barroso (2013) explains how high-percentile latency in component services can dominate the response time of large fan-out systems.

Completion Policy

The synthesis node should not run merely because every branch returned. It should run when coverage thresholds are met, required source categories are present, unresolved contradictions are represented, and evidence freshness satisfies the task. Missing evidence should produce an explicit gap rather than an invented conclusion.

Reference Architecture for Multi-Agent Coding

Topology

A coding graph should separate repository analysis, planning, isolated implementation, integration, and verification:

issue -> repository scan -> change plan
                         -> worker A in worktree
                         -> worker B in worktree
                         -> worker C in worktree
                         -> integration -> test matrix -> review

Agent orchestration distinguishes manager-style orchestration from handoffs and provides a conceptual basis for deciding whether a central coordinator should retain control or transfer execution to a specialist.

Repository Scanner

The scanner should build an evidence-backed change map that identifies relevant modules, interfaces, tests, ownership boundaries, and dependency edges. Its output should be a structured plan input, not a speculative implementation. Read-only analysis can be parallelized safely when branches inspect different subsystems.

Planner

The planner should assign non-overlapping file or subsystem ownership whenever possible. Each work package should include its allowed write set, prohibited files, interface assumptions, test obligations, and expected artifact. Dependencies between packages form a subgraph that constrains scheduling.

Worker Isolation

Workers should operate in isolated branches, worktrees, containers, or sandboxes so that incomplete edits cannot corrupt a shared working copy. Isolation also makes worker outputs reviewable as discrete patches and permits failed branches to be discarded without affecting successful work.

The following figure (source) shows a single request spawning an isolated fleet through code-level orchestration, with intermediate work contained inside the graph and only the synthesized result returned to the user.

Integration

Integration should be a first-class node, not an incidental merge command. It should apply patches in dependency order, detect semantic and textual conflicts, run targeted tests after each integration group, and retain provenance from every final line or artifact to the worker that produced it.

The integration gate should require:

  • Clean application of all accepted patches.

  • Successful static checks and unit tests.

  • Successful interface and integration tests.

  • No policy violations or unexpected file changes.

  • A reviewable summary of behavioral changes and remaining risks.

Reference Architecture for High-Impact Enterprise Actions

Topology

A high-impact action graph should separate analysis from commitment:

request -> authorize -> gather evidence -> prepare action
        -> independent verify -> human approval -> commit -> reconcile

The preparation stage should produce an immutable, reviewable action package containing exact parameters, affected resources, expected consequences, rollback instructions, and an expiration time. The commit node may execute only the approved package and must reject any material change introduced after approval.

Architectural Constraints

High-impact workflows should enforce least privilege, separation of duties, immutable audit records, idempotency, explicit approval identity, and policy evaluation close to the tool boundary. NIST AI Risk Management Framework organizes AI risk work around govern, map, measure, and manage functions, which supports treating operational controls and accountability as architectural requirements rather than prompt-level guidance.

Prepared Action

The prepared action should have a deterministic identity:

\[A_{\mathrm{id}} = H \left( \mathrm{tool}, \mathrm{parameters}, \mathrm{target}, \mathrm{principal}, \mathrm{policy\ version}, \mathrm{expiry} \right)\]

Approval should bind to \(A_{\mathrm{id}}\). If any bound field changes, the system must invalidate the approval and return to verification. This prevents a benign preview from authorizing a materially different execution.

Transactional Recovery

Where the external system supports transactions, commit and rollback should use them. Where it does not, the graph should use idempotency keys, reconciliation queries, and compensating actions. Sagas by Garcia-Molina and Salem (1987) introduces the decomposition of a long-lived transaction into smaller transactions with compensating operations, a pattern directly relevant to multi-step agent actions across independent services.

Reference Architecture for Customer-Support Routing

Manager Pattern

In a manager pattern, a central agent retains conversation ownership and invokes specialist agents as tools. This architecture is suitable when a consistent voice, centralized policy enforcement, and unified context are more important than specialist autonomy. OpenAI Agents SDK orchestration guidance describes the manager pattern as agents-as-tools orchestration in which a central agent controls the final response.

The manager should route using explicit intent, customer tier, policy domain, language, urgency, and required permissions. Specialists should return structured recommendations and evidence, while the manager remains responsible for the customer-facing answer.

Handoff Pattern

In a handoff pattern, the active specialist assumes control of the conversation or task. This architecture is appropriate when the specialist needs direct interaction, owns unique tools, or must collect domain-specific information over multiple turns. OpenAI Agents SDK handoff documentation describes handoffs as tools through which an agent delegates the conversation to another agent.

Handoffs should carry a bounded context package containing the user objective, verified facts, actions already taken, unresolved questions, policy constraints, and a transfer reason. The receiving agent should explicitly accept or reject the handoff contract so that routing failures remain observable.

Reference Architecture for Large Agent Fleets

Components

A fleet architecture requires a graph compiler or planner, scheduler, worker pool, capability registry, quota manager, artifact store, event bus, evaluator service, and fleet-level observability. Workers should be stateless with respect to durable control state; they lease work, emit events, write artifacts, and return typed results.

Ray: A Distributed Framework for Emerging AI Applications by Moritz et al. (2018) presents a distributed execution model based on tasks and stateful actors, illustrating primitives that can support heterogeneous, dynamically scheduled AI workloads.

Scheduling

The scheduler should place tasks using required capabilities, model access, data locality, tenant policy, resource cost, and deadline. Priority alone is insufficient because a high-priority task assigned to an incompatible worker will fail or block.

A placement score can be expressed as:

\[S(w,t) = \alpha C_{\mathrm{capability}}(w,t) + \beta C_{\mathrm{locality}}(w,t) + \gamma C_{\mathrm{deadline}}(w,t) - \delta C_{\mathrm{cost}}(w,t) - \epsilon C_{\mathrm{load}}(w)\]

where incompatible workers receive a negative-infinite capability score and are excluded before optimization.

Backpressure

Every fan-out boundary needs a capacity policy. Without backpressure, one planner can create more work than downstream models, tools, databases, or evaluators can absorb. The controller should cap outstanding tasks, queue depth, per-tenant concurrency, and total cost exposure. Reactive Streams defines asynchronous stream processing with non-blocking backpressure, a principle that applies directly to agent-task production and consumption.

Fleet Limits

The fleet should enforce nested limits:

\[B_{\mathrm{run}} \le B_{\mathrm{tenant}} \le B_{\mathrm{platform}}\]

and:

\[C_{\mathrm{run}} \le C_{\mathrm{tenant}} \le C_{\mathrm{platform}}\]

where \(B\) denotes budget and \(C\) denotes concurrency. Nested enforcement prevents a single graph or tenant from exhausting shared capacity.

Choose Between Centralized and Decentralized Control

Centralized control provides consistent policies, global budget enforcement, simpler debugging, and deterministic integration. Its disadvantages are coordinator bottlenecks and reduced resilience when every decision depends on one node. Decentralized control can improve locality and autonomy but makes global invariants, provenance, and recovery more difficult.

A practical production design is usually hierarchical. A durable central controller owns lifecycle, budgets, and terminal state, while bounded subgraphs control local execution within delegated constraints. The central controller does not need to inspect every token, but it must retain the authority to cancel, checkpoint, and reconcile the run.

Separate Control Messages from Task Data

Control messages should contain scheduling and lifecycle information, while task data should be stored in artifacts or state records referenced by identifier. Passing large documents through queues or workflow histories increases serialization cost, duplicates data, and complicates retention. A control message should therefore carry identifiers, hashes, schema versions, and routing metadata rather than complete artifacts.

The distinction can be represented as:

\[M_{\mathrm{control}} = \{ \mathrm{run\_id}, \mathrm{node\_id}, \mathrm{artifact\_refs}, \mathrm{schema\_version}, \mathrm{deadline}, \mathrm{trace\_context} \}\]

This structure also supports content-addressed storage, where immutable artifacts are retrieved by hash and can be safely reused across retries.

Define Consistency by State Type

Not every state category needs the same consistency model. Authorization, budget reservation, approval, and commit records require strong consistency because divergent views can cause unsafe execution or overspending. Telemetry, caches, progress displays, and derived analytics can often tolerate eventual consistency.

A useful classification is:

  • Strongly consistent: permissions, approvals, budget debits, idempotency records, terminal status.

  • Transactionally grouped: action intent, outbox event, and local state transition.

  • Eventually consistent: traces, dashboards, search indexes, usage aggregates.

  • Recomputable: caches, embeddings, derived summaries, transient context projections.

The selected consistency level should be recorded in the state schema rather than left as an implicit property of the storage product.

Place Checkpoints at Semantic Boundaries

Checkpoints should occur after stable semantic achievements, not after arbitrary token or time intervals. Appropriate boundaries include completion of a plan, acquisition of an evidence set, production of a candidate artifact, passage of an evaluation gate, receipt of approval, and completion of a side effect.

A checkpoint should contain:

\[C_k = \left( V_{\mathrm{workflow}}, S_k, A_k, E_k, B_k, P_k \right)\]

where \(S_k\) is resumable state, \(A_k\) is the artifact set, \(E_k\) is evidence, \(B_k\) is remaining budget, and \(P_k\) is the policy and authorization snapshot. Checkpointing at these boundaries permits recovery without repeating expensive work or accidentally duplicating side effects.

Use an Outbox for Reliable Side Effects

When a local state transition must correspond to an external message or action, the system should persist the transition and an outbox record in the same transaction. A separate dispatcher then performs the external operation and records delivery. This prevents a crash between database commit and message publication from leaving the workflow in an unrecoverable split state. Pattern: Transactional outbox explains the pattern for reliably publishing messages without requiring a distributed transaction across the database and broker.

Receivers must still be idempotent because dispatch can occur more than once. The idempotency key should derive from the run, node, action type, and logical attempt rather than from a transport-level delivery identifier.

Build a Unified Reference Architecture

A general production deployment can be organized as follows:

clients
  |
API and identity gateway
  |
task intake and policy check
  |
durable graph runtime ------------------- observability pipeline
  |              |              |                     |
model gateway    tool gateway   verifier service      traces and metrics
  |              |              |
model providers  enterprise     tests, policies,
                 systems        evidence checks
  |
checkpoint store, artifact store, evidence registry, audit log

The graph runtime should communicate with model and tool gateways through typed requests carrying tenant, policy, trace, budget, and idempotency context. The data stores should maintain separate retention and access policies even when implemented on shared infrastructure.

Model Gateway Responsibilities

The model gateway should perform provider routing, model allowlisting, structured-output enforcement, token accounting, request deduplication where safe, fallback selection, content-policy integration, and per-tenant rate limiting. It should record the exact model and configuration used for every generation.

Tool Gateway Responsibilities

The tool gateway should validate schemas, resolve credentials without exposing them to the model, authorize the principal, enforce action policies, attach idempotency keys, apply timeouts, normalize errors, and write audit events. Tool results should be treated as untrusted external input until validated.

Graph Runtime Responsibilities

The runtime should own node eligibility, leases, retries, deadlines, cancellation, checkpoints, human waits, graph-version migration, and terminal-state reconciliation. It should not embed provider-specific model logic or business-system credentials.

Verification Service Responsibilities

The verification service should run deterministic tests, policy checks, schema validators, evidence-quality checks, and model-based evaluators under independent configurations. It should return a structured report containing pass status, scores, failed criteria, evidence, and recommended transition.

Select the Smallest Architecture That Satisfies the Task

Architecture should grow in response to demonstrated requirements. A sequential verified loop is preferable when one worker and one evaluator can solve the task reliably. Parallel research is justified when information needs are independent. Multiple coding workers are justified when change packages can be isolated. A fleet platform is justified only when workload volume, heterogeneity, and tenancy require centralized scheduling and governance.

The choice can be framed as constrained minimization:

\[\mathcal{A}^{*} = \arg\min_{\mathcal{A}} \left( C_{\mathrm{build}}(\mathcal{A}) + C_{\mathrm{operate}}(\mathcal{A}) + C_{\mathrm{coordinate}}(\mathcal{A}) \right)\]

subject to:

\[Q(\mathcal{A}) \ge Q_{\min}\] \[R(\mathcal{A}) \ge R_{\min}\] \[S(\mathcal{A}) \ge S_{\min}\]

where \(Q\) is output quality, \(R\) is reliability, and \(S\) is safety. This formulation emphasizes that additional agents and infrastructure are costs to justify, not maturity signals by themselves.

Production Architecture Principle

The defining property of a production Loop and Graph Engineering architecture is controlled persistence. The system can continue working across multiple steps and failures, but it does so inside explicit contracts, budgets, permissions, checkpoints, evaluators, and termination rules. Models supply flexible reasoning; the surrounding architecture supplies durable control.

The resulting system should make five questions answerable for every run:

  • What objective and workflow version governed the run?

  • Which nodes, models, tools, and people influenced the result?

  • What evidence justified each important transition?

  • Which budgets, policies, and approvals constrained execution?

  • How can the run be resumed, reproduced, reversed, or audited?

When these questions can be answered from system records rather than reconstructed from scattered logs, the workflow has crossed from an experimental agent into an engineered production system.

Implementation Roadmap and Maturity Model

Begin with a Bounded Operational Problem

Loop and Graph Engineering should begin with a narrow, measurable workflow rather than a general-purpose autonomous agent. The first candidate should have a clear trigger, bounded inputs, observable intermediate states, verifiable outputs, and a known escalation path. Rules of Machine Learning recommends establishing robust infrastructure and simple, observable objectives before introducing more complex learned behavior.

Suitable initial workflows include:

  • Producing a localized code change and running its test suite.

  • Collecting evidence for a defined research question.

  • Classifying and routing support requests.

  • Transforming a document according to a validated schema.

  • Preparing, but not executing, an enterprise action for review.

The initial workflow should avoid irreversible actions, ambiguous success criteria, unrestricted tools, and dependencies on undocumented human judgment. Complexity can be introduced after the team has measured where the simpler workflow fails.

A candidate workflow can be prioritized using:

\[P_{\mathrm{workflow}} = \frac{ V_{\mathrm{business}} \cdot F_{\mathrm{frequency}} \cdot M_{\mathrm{measurability}} }{ R_{\mathrm{impact}} \cdot C_{\mathrm{integration}} \cdot U_{\mathrm{ambiguity}} }\]

where business value, execution frequency, and measurability increase priority, while operational risk, integration complexity, and task ambiguity reduce it. The formula is a decision aid rather than a universal scoring standard.

Establish the Manual Baseline

Before constructing an autonomous loop, the team should document how a capable human or manually supervised model completes the task. The baseline should capture inputs, decisions, tools, intermediate artifacts, validation steps, exception paths, and completion criteria.

The baseline provides three essential measurements:

  • The quality achievable with close supervision.

  • The cost and latency of the existing process.

  • The locations where judgment, iteration, and external information are actually required.

Without this baseline, automation may optimize visible activity rather than task value. Rules of Machine Learning recommends beginning with simple, attributable metrics tied to observable product behavior.

For a baseline containing \(N\) tasks, the initial measurements should include:

\[Q_0 = \frac{1}{N} \sum_{i=1}^{N} q_i\] \[C_0 = \frac{1}{N} \sum_{i=1}^{N} c_i\] \[T_0 = \frac{1}{N} \sum_{i=1}^{N} t_i\]

where \(Q_0\) is average task quality, \(C_0\) is average cost, and \(T_0\) is average completion time. Quality should be decomposed into task-specific criteria rather than represented only by one subjective score.

Build the Evaluation Set Before the Autonomous Loop

The first implementation artifact should be a representative evaluation set. It should include routine cases, difficult cases, malformed inputs, tool failures, ambiguous requests, conflicting evidence, budget exhaustion, and cases that require abstention or escalation.

The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction by Breck et al. (2017) presents actionable tests and monitoring requirements for assessing production readiness, demonstrating why evaluation must cover the surrounding system rather than model quality alone.

An agent evaluation suite should operate at several levels:

  • Node evaluation: Does each node satisfy its input-output contract?

  • Transition evaluation: Does the controller select the correct next node?

  • Trajectory evaluation: Does the run use an efficient and policy-compliant path?

  • Artifact evaluation: Does the final output satisfy the task specification?

  • System evaluation: Does the complete workflow remain reliable under concurrency, retries, timeouts, and dependency failures?

  • Human-impact evaluation: Does the workflow preserve user control, privacy, fairness, and appropriate escalation?

Demystifying evals for AI agents explains that multi-turn tool use and state modification require evaluations of trajectories and intermediate decisions, not only final responses.

The evaluation set should be divided into development, regression, and held-out sets:

\[D = D_{\mathrm{development}} \cup D_{\mathrm{regression}} \cup D_{\mathrm{heldout}}\]

The development set guides implementation. The regression set preserves previously corrected failures. The held-out set estimates whether changes generalize beyond cases repeatedly inspected during development.

Define the Task Contract

Every automated run should begin from a typed task contract containing:

  • The objective.

  • Required inputs.

  • Available tools and permissions.

  • Expected artifact types.

  • Acceptance criteria.

  • Maximum cost, elapsed time, iterations, and concurrency.

  • Conditions requiring approval.

  • Conditions requiring abstention or escalation.

  • The terminal response schema.

A completion predicate can be expressed as:

\[\mathrm{Complete}(s) = \bigwedge_{j=1}^{m} g_j(s)\]

where each \(g_j\) represents an independently checkable acceptance condition. The controller may transition to completion only when the complete conjunction passes.

The contract should distinguish preferences from hard constraints. A preference may influence planning or ranking, while violation of a hard constraint must block completion. This distinction prevents a model from trading away a safety or correctness condition in exchange for a more fluent result.

Introduce Automation in Controlled Stages

Instrument the Existing Workflow

The first stage should observe the current process without changing its authority. Record requests, intermediate decisions, tool use, validation outcomes, corrections, latency, and cost. This produces realistic traces from which graph boundaries and evaluators can be derived.

Instrumentation should preserve privacy and minimize unnecessary retention. Sensitive fields should be redacted or tokenized before entering general-purpose telemetry.

Automate One Deterministic Step

The first automated node should have a narrow contract and a strong verifier. Examples include schema extraction, repository scanning, query generation, test execution, evidence normalization, or classification among a small set of routes.

This stage validates the surrounding infrastructure:

  • State serialization.

  • Schema enforcement.

  • Tool authentication.

  • Trace propagation.

  • Timeout behavior.

  • Error normalization.

  • Artifact storage.

  • Evaluation reporting.

Rules of Machine Learning specifically recommends testing infrastructure independently from learned components so that pipeline failures are not confused with model failures.

Add a Single Bounded Loop

Once one execution pass is reliable, introduce a revision edge from verification back to execution. The first loop should have a small iteration limit and a deterministic completion gate.

prepare -> execute -> verify -> complete
              ^          |
              |----------|

The controller should record the reason for each additional iteration. Valid reasons include a failed test, missing evidence, schema violation, policy failure, or insufficient coverage. A generic instruction to improve the result is too ambiguous for reliable operations.

Progress between iterations can be measured as:

\[\Delta Q_k = Q_k - Q_{k-1}\]

The loop should terminate or escalate when:

\[\Delta Q_k < \epsilon\]

for a configured number of consecutive iterations, indicating stagnation. It should also terminate when cost, time, token, or iteration budgets are exhausted.

Externalize State

As soon as a run can span multiple calls, its authoritative state should move outside the model context. The state record should include the plan, completed work, current node, artifact references, evidence, evaluator feedback, budgets, approvals, and transition history.

Model context should be constructed as a projection of this durable state:

\[C_k = \Pi \left( S_k, N_k, B_k \right)\]

where \(S_k\) is durable state, \(N_k\) is the active node, \(B_k\) is the context budget, and \(\Pi\) is the context-selection function.

This separation prevents the prompt transcript from becoming the only representation of workflow progress. Hidden Technical Debt in Machine Learning Systems by Sculley et al. (2015) identifies entanglement, undeclared dependencies, and fragile glue code as recurring sources of production complexity, all of which are amplified when control state remains implicit.

Introduce an Explicit Graph

A graph is justified when the workflow contains meaningful branches, parallel work, specialized permissions, human approval, or recovery paths. The transition from a loop to a graph should be driven by task structure rather than a desire to add agents.

A decomposition is useful when:

\[T_{\mathrm{coordination}} + T_{\mathrm{integration}} < T_{\mathrm{sequential\ savings}}\]

or when independent workers materially improve coverage, specialization, isolation, or safety.

The first graph should remain small. A practical initial topology contains intake, planning, execution, verification, escalation, and completion nodes. Additional specialists should be introduced only when evaluations demonstrate a persistent failure class that a distinct role or context can address.

Apply Readiness Gates Between Stages

A workflow should advance only when it satisfies an explicit gate. The gate should be based on the weakest critical dimension rather than an average that allows strong quality to conceal weak safety or reliability.

Let the maturity dimensions be:

\[Q = \mathrm{quality}\] \[R = \mathrm{reliability}\] \[S = \mathrm{safety}\] \[O = \mathrm{observability}\] \[C = \mathrm{cost\ efficiency}\]

An aggregate score may support prioritization:

\[M = w_QQ + w_RR + w_SS + w_OO + w_CC\]

subject to:

\[\sum_i w_i = 1\]

However, promotion readiness should be determined by:

\[\mathrm{Readiness} = \min \left( Q, R, S, O, C \right)\]

This minimum rule prevents one critically weak dimension from being hidden by stronger scores elsewhere. The exact dimensions and thresholds should be adapted to the workflow’s risk class.

Gate for a Supervised Pilot

Before supervised use, the workflow should have:

  • Versioned prompts, tools, models, and schemas.

  • A representative development evaluation set.

  • Deterministic input and output validation.

  • Complete run traces.

  • Explicit iteration, cost, and time limits.

  • No unrestricted high-impact tool access.

  • A human able to inspect and override every consequential action.

Gate for Limited Production

Before limited production, the workflow should additionally have:

  • A held-out evaluation set.

  • Regression tests for known failures.

  • Idempotent or read-only tool behavior.

  • Checkpoint and resume support.

  • Tested timeout, retry, and cancellation paths.

  • Tenant and permission isolation.

  • Dashboards for quality, latency, cost, and failures.

  • A documented rollback and incident-response procedure.

Gate for Bounded Autonomy

Before executing actions without synchronous review, the workflow should additionally have:

  • Independent verification of consequential outputs.

  • Policy enforcement at the tool boundary.

  • Action-specific authorization.

  • Stagnation and anomaly detection.

  • Reconciliation for uncertain external outcomes.

  • Auditable action intent and execution records.

  • Demonstrated reliability under repeated and perturbed evaluation.

  • An error budget and automatic rollback or disablement policy.

The NIST AI Risk Management Framework organizes risk management around govern, map, measure, and manage, reinforcing that deployment authority should depend on continuing organizational controls rather than a one-time technical evaluation.

Use a Risk-Tiered Autonomy Model

Not every node should receive the same autonomy. Authority should be assigned according to the impact and reversibility of the operation.

Read-Only Autonomy

The system may search, retrieve, inspect, classify, calculate, and generate drafts. Read access should still be scoped by identity, tenant, purpose, and data sensitivity.

Prepared-Action Autonomy

The system may construct a complete proposed action but cannot commit it. A human or independent policy service reviews the exact parameters and consequences.

Examples include:

  • Drafting an email without sending it.

  • Preparing a code patch without merging it.

  • Creating a refund proposal without issuing payment.

  • Producing a database migration plan without applying it.

Reversible-Action Autonomy

The system may execute actions that have reliable idempotency and compensation. Examples include creating a reversible ticket, updating a draft record, or opening a pull request.

High-Impact Autonomy

Irreversible, financial, legal, security-sensitive, or externally binding actions should require stronger authorization, independent verification, and often human approval. The NIST AI RMF Playbook provides suggested actions for integrating trustworthiness and risk treatment across deployment and operation.

The permitted autonomy level can be represented as:

\[A_{\max} = f \left( I, V, D, P, E \right)\]

where \(I\) is impact, \(V\) is reversibility, \(D\) is detectability of error, \(P\) is permission sensitivity, and \(E\) is evaluator strength.

Define Production Service-Level Objectives

Production readiness requires measurable service objectives for both infrastructure and task behavior. Service Level Objectives recommends deriving indicators from user-relevant behavior and defining how teams respond when objectives are missed.

An agent workflow should define service-level indicators for:

  • End-to-end task success.

  • Unsupported or incorrect completion.

  • Correct abstention and escalation.

  • Policy-compliant tool use.

  • Recovery after dependency failure.

  • Duplicate or inconsistent side effects.

  • Median and tail latency.

  • Cost per successful task.

  • Human-intervention rate.

  • Trace and audit completeness.

A task-success objective may be written as:

\[\mathrm{SuccessRate} = \frac{ N_{\mathrm{accepted\ completions}} }{ N_{\mathrm{eligible\ runs}} }\]

A cost-normalized quality measure can be written as:

\[E_{\mathrm{value}} = \frac{ \sum_{i=1}^{N} q_i }{ \sum_{i=1}^{N} c_i }\]

Infrastructure metrics should include latency, traffic, errors, and saturation, following Monitoring Distributed Systems. Agent-specific monitoring must extend these signals with semantic quality, trajectory efficiency, tool correctness, evaluator outcomes, and safety events.

Roll Out Through Progressive Exposure

Deployment should progress through offline evaluation, shadow execution, internal use, limited production, controlled canarying, and broader availability.

Offline Evaluation

The candidate workflow runs against fixed tasks without access to production side effects. Promotion requires passing quality, safety, regression, and reliability thresholds.

Shadow Execution

The workflow receives representative production requests but does not control the user-facing result or execute consequential actions. Its decisions are compared with the existing process.

Shadowing reveals:

  • Input-distribution mismatches.

  • Missing permissions or data.

  • Tool latency and availability constraints.

  • Differences between synthetic and real workflows.

  • Evaluator blind spots.

Internal or Expert Pilot

A small group uses the workflow with full trace visibility and easy escalation. Pilot users should report incorrect outputs, unnecessary iterations, missing evidence, confusing approvals, and cases where intervention was required.

Limited Production

The system handles a bounded traffic segment, task category, tenant group, or low-risk action class. Limits should be enforced by configuration rather than operational convention.

Canary Release

The new graph version should receive a small fraction of eligible traffic while being compared with the current version. Canarying Releases explains how a canary limits user impact and allows release-candidate metrics to be compared with a control before wider deployment.

For canary fraction \(p\) and candidate failure rate \(e\), the approximate global failure exposure is:

\[E_{\mathrm{global}} = p \cdot e\]

The canary controller should pause or roll back when quality, policy, latency, cost, or error metrics exceed configured bounds. Evaluation windows must be long enough to observe representative behavior but short enough to limit exposure.

Maintain an Agent Error Budget

An error budget translates reliability objectives into deployment policy. If the target success rate is:

\[S_{\mathrm{target}}\]

then the allowable failure budget is:

\[B_{\mathrm{error}} = 1 - S_{\mathrm{target}}\]

The budget should be consumed by user-impacting failures such as incorrect task completion, unsafe tool execution, unrecovered dependency failure, missing escalation, or duplicate side effects. Different failure classes may receive different severity weights:

\[B_{\mathrm{consumed}} = \sum_{j=1}^{m} \lambda_j e_j\]

where \(e_j\) is the count or rate of failure class \(j\) and \(\lambda_j\) is its severity weight.

Example Error Budget Policy demonstrates how exceeding an error budget can trigger a release freeze and redirect effort toward reliability. Agent systems should similarly suspend autonomy expansion when the workflow is outside its quality or safety objectives.

Version Every Behaviorally Significant Artifact

A production release should version:

  • Graph topology.

  • Node implementations.

  • Prompts and system instructions.

  • Model identifiers and inference settings.

  • Tool schemas and permission policies.

  • State and checkpoint schemas.

  • Context-construction policies.

  • Evaluators and rubrics.

  • Evaluation datasets.

  • Budget and termination policies.

  • Feature flags and rollout configuration.

The run record should contain the complete version tuple:

\[V_r = \left( V_g, V_n, V_p, V_m, V_t, V_s, V_c, V_e, V_d, V_b \right)\]

A change to any component can alter system behavior and should therefore trigger an appropriate subset of regression evaluations. Hidden Technical Debt in Machine Learning Systems by Sculley et al. (2015) explains how configuration dependencies, pipeline entanglement, and undeclared consumers create technical debt beyond the model itself.

Establish Clear Ownership

A production workflow requires named ownership across product behavior, graph implementation, evaluation, safety, tools, and operations.

  • Product owner: Defines task value, eligible use cases, user experience, and acceptable failure tradeoffs.

  • Workflow owner: Owns graph topology, node contracts, state transitions, and version compatibility.

  • Evaluation owner: Maintains evaluation sets, graders, regression cases, and promotion thresholds.

  • Tool owner: Maintains tool schemas, authorization, idempotency, error semantics, and availability objectives.

  • Safety owner: Defines risk classifications, approval rules, incident thresholds, and policy tests.

  • Operations owner: Maintains deployment, monitoring, on-call response, capacity, rollback, and recovery.

  • Domain reviewer: Determines whether outputs satisfy professional or organizational requirements in specialized areas.

Ownership should include escalation paths and decision rights. A workflow without a clear owner for evaluator failures, tool incidents, or policy changes will accumulate unresolved operational risk even when the graph itself is technically sound.

Use a Practical Phased Roadmap

Initial Thirty Days

The first phase should establish the task and measurement foundation:

  • Select one bounded, high-frequency workflow.

  • Document the manual baseline.

  • Define the task contract and risk classification.

  • Assemble development and regression evaluation sets.

  • Implement trace and artifact schemas.

  • Build one deterministic or read-only node.

  • Establish baseline quality, latency, and cost.

The principal deliverable is not an autonomous agent. It is a measurable workflow specification with reliable instrumentation.

Days Thirty to Sixty

The second phase should introduce controlled iteration:

  • Add planning, execution, and verification nodes.

  • Implement one bounded revision edge.

  • Externalize workflow state.

  • Add budgets, stagnation detection, and cancellation.

  • Introduce deterministic tool and output validation.

  • Run offline, adversarial, and failure-injection evaluations.

  • Begin shadow execution on representative traffic.

The principal deliverable is a verified single loop that can explain why it continued, stopped, or escalated.

Days Sixty to Ninety

The third phase should prepare limited production:

  • Introduce explicit branching only where evaluations justify it.

  • Add durable checkpoints and recovery.

  • Implement tool authorization and idempotency.

  • Add dashboards, alerts, and operational runbooks.

  • Define service-level objectives and error budgets.

  • Conduct a supervised internal pilot.

  • Deploy a limited canary with automated rollback.

The principal deliverable is a bounded production workflow with clear ownership, auditability, and recovery.

Beyond Ninety Days

Later phases should focus on measured expansion:

  • Increase eligible task coverage.

  • Introduce additional specialists where they improve a documented failure class.

  • Automate low-risk approvals.

  • Add multi-tenant quotas and scheduling.

  • Improve model and tool routing.

  • Build reusable graph, evaluator, and observability components.

  • Introduce controlled learning from human corrections.

  • Periodically reassess whether accumulated complexity remains justified.

The roadmap should be adjusted to the workflow’s impact, existing infrastructure, compliance requirements, and failure tolerance. A high-impact financial or healthcare workflow may remain supervised much longer than a read-only research workflow.

Apply a Loop and Graph Engineering Maturity Model

Prompt-Centric Experimentation

At this level, a user interacts directly with a model through manually constructed prompts. State exists primarily in the conversation, evaluation is anecdotal, and tool use is manually supervised.

Characteristics include:

  • Single-turn or short multi-turn interactions.

  • No durable task state.

  • Informal completion judgments.

  • Limited reproducibility.

  • Manual recovery from failures.

The objective at this stage is to determine whether the model has enough capability to make the workflow viable.

Instrumented Workflow

The model is embedded in a defined application path with structured inputs, outputs, tracing, and basic evaluation.

Characteristics include:

  • Versioned prompts and models.

  • Typed input and output schemas.

  • Recorded model and tool activity.

  • A development evaluation set.

  • Human-controlled execution.

The objective is to transform an interesting demonstration into a measurable system.

Verified Loop

The workflow contains explicit execution, evaluation, revision, and termination logic.

Characteristics include:

  • Independent completion criteria.

  • Bounded iteration.

  • Durable or reconstructable state.

  • Regression tests.

  • Cost and latency budgets.

  • Stagnation and escalation handling.

The objective is to automate iteration without surrendering control over completion.

Explicit Graph

The workflow is represented as typed nodes and conditional transitions.

Characteristics include:

  • Specialized planning, execution, and verification nodes.

  • Conditional routing.

  • Parallel branches where justified.

  • Structured fan-out and synthesis.

  • Node-level permissions and budgets.

  • Graph and schema versioning.

The objective is to make decomposition, coordination, and recovery explicit.

Durable Production Graph

The graph can survive process failures and safely interact with production systems.

Characteristics include:

  • Durable orchestration and checkpoints.

  • Idempotent tool execution.

  • Human approval nodes.

  • Canary deployment and rollback.

  • Service-level objectives and error budgets.

  • Complete audit and incident records.

  • Tenant and authorization isolation.

The objective is dependable operation under realistic infrastructure and dependency failures.

Adaptive Agent Platform

Multiple workflows share governed infrastructure and improve through evaluated operational feedback.

Characteristics include:

  • Reusable node and graph libraries.

  • Central model and tool gateways.

  • Shared evaluation and evidence services.

  • Multi-tenant scheduling and quotas.

  • Automated failure clustering.

  • Controlled prompt, policy, or routing optimization.

  • Organization-wide governance and lineage.

The objective is to scale successful engineering patterns without allowing local experimentation to bypass platform controls.

Score Maturity by Capability, Not Architecture Size

A larger graph is not necessarily more mature. Maturity should reflect the system’s ability to deliver useful results reliably, safely, efficiently, and transparently.

Each dimension can be scored on a bounded scale:

\[x_i \in [0,5]\]

A practical assessment includes:

  • Objective and contract clarity.

  • Evaluation coverage.

  • Verification independence.

  • State durability.

  • Termination control.

  • Tool safety.

  • Recovery capability.

  • Observability.

  • Deployment discipline.

  • Ownership and governance.

The overall maturity level should be capped by critical weaknesses:

\[L_{\mathrm{maturity}} = \min_i \left( x_i \right)\]

for dimensions designated as mandatory. A system with advanced multi-agent coordination but no reliable termination or action authorization should therefore remain at a low production maturity level.

The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction by Breck et al. (2017) similarly uses concrete tests to make production readiness measurable rather than treating model sophistication as a sufficient proxy.

Decide What to Build and What to Adopt

Teams should generally build the task-specific graph, contracts, evaluators, policies, and domain integrations. Commodity infrastructure can often be adopted for workflow durability, telemetry, schema validation, queues, artifact storage, and identity.

A build decision is justified when:

  • The component encodes unique domain logic.

  • Existing tools cannot enforce the required safety or latency constraints.

  • The component is a source of durable competitive differentiation.

  • Provider abstraction is necessary for governance or resilience.

An adoption decision is preferable when:

  • The capability is operationally complex but not task-specific.

  • Mature standards or services already exist.

  • Building it would divert effort from evaluation and workflow quality.

  • The team cannot commit to long-term maintenance and incident ownership.

The total-cost comparison should include more than licensing:

\[C_{\mathrm{build}} = C_{\mathrm{implementation}} + C_{\mathrm{operation}} + C_{\mathrm{security}} + C_{\mathrm{upgrades}} + C_{\mathrm{incidents}}\] \[C_{\mathrm{adopt}} = C_{\mathrm{license}} + C_{\mathrm{integration}} + C_{\mathrm{migration}} + C_{\mathrm{dependency}} + C_{\mathrm{constraints}}\]

The decision should be revisited as workload scale, platform capabilities, and regulatory requirements change.

Avoid Common Adoption Failures

Beginning with a General-Purpose Agent

A broad agent creates an evaluation surface too large to characterize reliably. Begin with a bounded workflow and expand eligibility only after measurable success.

Adding Agents Before Identifying Independent Work

Multiple agents increase communication, state, latency, and integration requirements. Add a worker only when it owns an independent information need, permission boundary, artifact, or verification responsibility.

Treating Prompt Iteration as System Evaluation

A better-looking demonstration does not establish production reliability. Evaluate repeated runs, perturbations, tool failures, state recovery, and adverse inputs.

Allowing the Generator to Approve Itself

Self-critique can improve a candidate but should not be the only completion authority. Use deterministic tests, independent models, evidence checks, policies, or human review according to the task.

Storing Control State Only in Context

Conversation history is not a durable workflow database. Externalize authoritative progress, budgets, approvals, and artifact references.

Retrying Every Failure

Retries are appropriate for transient faults, not permanent authorization failures, invalid objectives, repeated semantic errors, or exhausted budgets. Retry classification should be explicit.

Measuring Only Final Accuracy

A workflow can produce an acceptable output through an unsafe, expensive, or irreproducible trajectory. Measure the path, evidence, actions, cost, latency, and termination reason.

Expanding Autonomy Without an Error Budget

Autonomy should increase only while the workflow remains within established quality and safety objectives. When the budget is exhausted, the system should reduce exposure or return to supervised operation.

Final Production Checklist

Before approving a Loop and Graph Engineering workflow for production, confirm that:

  • Objective: The task, scope, exclusions, and acceptance criteria are explicit.

  • Evaluation: Development, regression, held-out, adversarial, and failure cases exist.

  • Graph: Every node and transition has a typed, versioned contract.

  • State: Authoritative workflow state is durable and reconstructable.

  • Verification: Completion depends on independent evidence or tests.

  • Termination: Iteration, time, cost, stagnation, cancellation, and escalation limits are enforced.

  • Tools: Permissions, schemas, idempotency, timeouts, and side-effect classifications are defined.

  • Safety: High-impact actions require appropriate policy checks and approval.

  • Recovery: Checkpoint, resume, reconciliation, and rollback procedures have been tested.

  • Observability: Traces connect objectives, decisions, artifacts, evidence, actions, and outcomes.

  • Deployment: Shadowing, canarying, rollback, and version comparison are supported.

  • Operations: Service-level objectives, error budgets, alerts, and incident procedures are active.

  • Ownership: Product, workflow, evaluation, tool, safety, and operational owners are named.

  • Economics: Cost and latency are measured per successful task, not merely per model call.

  • Governance: Data retention, privacy, audit, and change-control policies are documented.

Closing Perspective

Prompt Engineering determines how an individual model invocation is framed. Loop Engineering determines how the system repeatedly acts, evaluates progress, revises work, and terminates. Graph Engineering determines how those loops and actions are decomposed, connected, coordinated, isolated, and recovered. System Engineering makes the resulting structure reliable enough to operate under real constraints.

The progression can be summarized as:

\[\mathrm{Prompt} \rightarrow \mathrm{Instrumented\ Workflow} \rightarrow \mathrm{Verified\ Loop} \rightarrow \mathrm{Explicit\ Graph} \rightarrow \mathrm{Durable\ Production\ System}\]

The goal is not maximum autonomy or the largest possible agent fleet. The goal is the smallest controlled system that consistently satisfies the task while remaining measurable, recoverable, secure, and accountable. A mature implementation therefore treats models as flexible reasoning components inside a rigorously engineered control system, not as substitutes for that system.

References

Loop and Graph Engineering core guides

Agent orchestration, tools, and execution guides

Reasoning, reflection, and iterative improvement papers

Multi-agent and distributed graph architectures

Memory, state, and context management

Evaluation, verification, and agent benchmarks

Agent safety, prompt injection, and governance

Production reliability, observability, and deployment

Citation

If you found our work useful, please cite it as:

@article{Chadha2020DistilledLoopGraphEngineering,
  title   = {Loop and Graph Engineering},
  author  = {Chadha, Aman and Jain, Vinija},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}