Overview

  • Autoresearch is the practice of turning experimental research into an agent-run optimization loop: an AI coding agent proposes a change, edits an executable research artifact, runs a bounded experiment, reads the metric, keeps or reverts the change, and repeats until the search budget is exhausted. In the concrete single-GPU setting, karpathy/autoresearch gives the agent a compact LLM training setup, lets it edit train.py, runs fixed 5-minute training experiments, evaluates val_bpb, and treats lower validation bits per byte as the target metric.

  • The following figure (source) shows an autoresearch progress curve from the repository, where sequential fixed-budget experiments are evaluated by validation bits per byte and the running best result improves over time.

What autoresearch automates

  • Autoresearch automates the outer loop of empirical ML work rather than only the inner loop of gradient-based training. The inner loop still optimizes model weights using ordinary training, typically next-token cross-entropy for language modeling:
\[\mathcal{L}_{\mathrm{CE}}(\theta) = -\frac{1}{T} \sum_{t=1}^{T} \log p_{\theta}(x_t \mid x_{<t})\]

  • The outer loop optimizes the research program around that training run: architecture choices, optimizer settings, batch sizes, attention patterns, model depth, learning-rate schedules, tokenizer and sequence-length tradeoffs, and other code-level decisions. In the single-GPU implementation, the repository is intentionally small: prepare.py handles fixed constants, data preparation, tokenizer training, dataloading, and evaluation; train.py contains the GPT model, optimizer, and training loop that the agent edits; and program.md acts as the lightweight instruction layer that defines how the agent should behave.

  • The metric used in this setup is validation bits per byte, which is useful because it is less tied to a particular vocabulary size than raw token loss. A typical conversion from average token negative log-likelihood in nats to bits per byte is:

    \[\mathrm{BPB} = \frac{\mathcal{L}_{\mathrm{nats}}}{\ln 2} \cdot \frac{N_{\mathrm{tokens}}}{N_{\mathrm{bytes}}}\]
    • where lower values indicate better compression of held-out text. This makes it suitable for comparing architecture and tokenizer changes inside a bounded training budget, as long as all candidates are evaluated on the same validation data and hardware regime.

The core loop

  • At a high level, autoresearch can be written as a propose, run, evaluate, decide loop:

    \[c_{t+1} \sim P(c \mid \mathcal{H}_t, I)\] \[m_{t+1} = \mathrm{Evaluate}(c_{t+1}; B)\] \[\mathcal{H}_{t+1} = \mathcal{H}_{t} \cup \{(c_{t+1}, m_{t+1}, \ell_{t+1})\}\]
    • where, \(c_t\) is a code candidate, \(P\) is the coding-agent proposer, \(I\) is the instruction layer, \(B\) is the fixed experimental budget, \(m_t\) is the measured score, \(\ell_t\) is the log of what happened, and \(\mathcal{H}_t\) is the accumulated history. The important design choice is that the search unit is executable code rather than a prompt string alone. This puts autoresearch closer to program search and harness engineering than to ordinary prompt optimization.

  • This is why Automatic Prompt Optimization with “Gradient Descent” and Beam Search by Pryzant et al. (2023) is relevant as background because it frames prompt revision as text-space optimization, but autoresearch generalizes the optimized artifact from text instructions to runnable ML code. TextGrad by Yuksekgonul et al. (2024) is also relevant because it treats natural-language feedback as an optimization signal for compound AI systems, but autoresearch uses full code execution and validation metrics as the main feedback channel.

Why fixed budgets matter

  • A fixed wall-clock training budget is central to autoresearch because it prevents the agent from “improving” results merely by training longer. In the single-GPU setup, every candidate gets the same 5-minute training window excluding startup and compilation, which makes changes to depth, batch size, model width, optimizer, attention pattern, and data throughput comparable under the same resource constraint.

  • This turns the objective from “find the best model eventually” into “find the best model under this compute envelope”:

    \[c^{*} = \arg\min_{c \in \mathcal{C}} \mathrm{BPB} \left( \mathrm{Train}(c, B) \right)\]
    • where \(\mathcal{C}\) is the space of valid code candidates and \(B\) is the bounded training budget. The resulting search naturally favors changes that improve learning speed, data efficiency, numerical stability, and hardware utilization, not just asymptotic quality.

How Meta-Harness fits into autoresearch

  • Meta-Harness is the natural generalization of autoresearch from “optimize a training file” to “optimize the harness around an LLM system.” In Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026), a harness is the code that decides what information to store, retrieve, and present to a fixed language model; the search procedure uses a coding-agent proposer that inspects prior source code, scores, and execution traces through a filesystem before proposing new harness code.

  • The following figure (source) shows the Meta-Harness search loop: an agent reads a filesystem containing prior candidates’ source code, execution traces, and scores; proposes a new harness; evaluates it on tasks; stores the proposed code, reasoning traces, and evaluation score back into the filesystem; and repeats.

  • The formal harness objective is:

    \[H^{*} = \arg\max_{H} \mathbb{E}_{x \sim X,\ \tau \sim p_M(H,x)} \left[ r(\tau, x) \right]\]
    • where \(M\) is the frozen base model, \(X\) is the task distribution, \(H\) is the harness, \(\tau\) is the rollout trajectory induced by running the model inside the harness, and \(r(\tau, x)\) is the task reward. This differs from ordinary model training because the weights of \(M\) may remain fixed while the outer-loop system searches over code that controls memory, retrieval, prompt construction, tool use, and state updates.

The key conceptual shift

  • The key shift is from optimizing parameters to optimizing research process. Traditional ML optimization changes \(\theta\), the model weights. Autoresearch changes \(c\), the code that defines the experiment. Meta-Harness changes \(H\), the code that defines the model’s operating environment. These layers can be viewed as nested optimization problems:
\[\theta^{*}(c) = \arg\min_{\theta} \mathcal{L}_{\mathrm{train}}(\theta; c)\] \[c^{*} = \arg\min_{c} \mathcal{L}_{\mathrm{val}}(\theta^{*}(c); c, B)\] \[H^{*} = \arg\max_{H} \mathbb{E} \left[ r(\tau, x) \right]\]

Why logs, traces, and filesystem access are first-class

  • Autoresearch becomes more powerful when the agent can inspect not just the final scalar metric, but also the code diff, console output, training curves, validation breakdowns, failure logs, and previous decisions. Meta-Harness makes this explicit by storing each candidate’s source code, scores, prompts, tool calls, model outputs, and state updates in a filesystem that the proposer can query with ordinary tools such as grep and cat, instead of compressing all feedback into a short summary.

  • This matters because code-level failures are often nonlocal. A candidate may degrade validation loss because a batch-size change destabilized the optimizer, because an attention-window change reduced throughput, because a data-loading change shifted token statistics, or because an apparently harmless prompt or harness edit changed the model’s downstream behavior many steps later. Scalar scores tell the agent what happened; traces help it infer why it happened.

Working definition

  • For the rest of this primer, autoresearch means an autonomous, metric-driven, code-editing research loop with five components: a bounded experimental sandbox, an editable artifact, a proposer agent, an evaluator, and a durable memory of attempts. The single-GPU version uses train.py as the editable artifact and val_bpb as the metric; Meta-Harness uses task-specific harness code as the editable artifact and task reward, accuracy, pass rate, or Pareto tradeoffs as the metric. Both are instances of the same deeper pattern: use agents to search over the process that produces model behavior, not only over the model output itself.

System architecture

  • Autoresearch systems are best understood as experimental operating systems for AI research: they define who can change what, how changes are evaluated, how results are logged, and how future proposals condition on past attempts. The minimal version has one editable research artifact, one evaluator, one metric, one instruction file, and one persistent experiment log; richer versions generalize the editable artifact from model-training code to full harness code that controls prompting, retrieval, memory, tools, and state.

Editable artifact

  • The editable artifact is the unit of search. In a compact LLM-training setup, the editable artifact is usually a single file containing the model definition, optimizer, and training loop, while data preparation and evaluation utilities remain fixed to keep the search space bounded and diffs reviewable. This design makes the agent’s changes easy to inspect and makes regression analysis practical because each candidate can be understood as a concrete code diff rather than a hidden policy update.

  • A useful rule is to separate code into three strata:

    • Frozen substrate: data download, tokenizer construction, dataloader correctness, validation-set construction, deterministic metric computation, hardware setup, and safety checks.
    • Editable research code: architecture, optimizer, batch sizing, schedules, attention patterns, normalization, loss variants, numerical precision choices, and training-loop structure.
    • Instruction layer: the agent’s research policy, constraints, experiment protocol, logging requirements, and decision rules.

  • This separation resembles the distinction between an LM program and its optimizer in DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines by Khattab et al. (2023), where the developer specifies a pipeline and the system compiles or tunes parts of it rather than treating every prompt as a one-off string.

Evaluator

  • The evaluator is the part of the system that turns an edited artifact into a comparable score. In the basic training setup, the evaluator runs a bounded training job and reports validation bits per byte, with lower values being better. A fixed time budget, such as 5 minutes per experiment, makes candidates comparable because each proposal must improve learning speed, hardware utilization, or modeling efficiency under the same wall-clock constraint.

  • A typical evaluator should return a structured record:

candidate_id
parent_candidate_id
git_diff_or_full_source
start_time
end_time
hardware_metadata
training_tokens
validation_loss
validation_bpb
throughput_tokens_per_second
peak_memory
nan_or_crash_flag
stderr_stdout_excerpt
full_log_path
decision
  • For language modeling, the main inner-loop objective is still next-token prediction:
\[\mathcal{L}_{\mathrm{LM}}(\theta) = -\frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T_i} \log p_{\theta}(x_{i,t} \mid x_{i,<t})\]

  • The autoresearch evaluator does not optimize this loss directly; it measures the result of a bounded training run induced by a code candidate. The outer-loop score can be written as:

    \[s(c) = \mathrm{BPB} \left( \mathrm{Train}(\theta_0, c, B) \right)\]
    • where \(c\) is the code candidate, \(\theta_0\) is the initialization procedure, and \(B\) is the fixed compute budget.

Proposer agent

  • The proposer is a coding agent that reads the current codebase, inspects past results, forms a hypothesis, edits code, runs tests or experiments, and records what happened. This is more powerful than a raw LLM prompt because the proposer can use shell tools, inspect files, execute code, revert changes, and diagnose failures through logs.

  • The proposer can be treated as a stochastic policy over code edits:

    \[c_{t+1} \sim \pi_{\phi} \left( c \mid I, \mathcal{D}_t, \mathcal{R}_t \right)\]
    • where \(I\) is the instruction layer, \(\mathcal{D}_t\) is the codebase and experiment history, and \(\mathcal{R}_t\) is the current research objective. This connects autoresearch to Evolution through Large Models by Lehman et al. (2022), which shows that code-generating language models can act as intelligent mutation operators over programs, and to AlphaEvolve by Novikov et al. (2025), which uses coding agents and automated evaluators to evolve algorithms through direct code changes.

Instruction layer

  • The instruction layer is the “research org code.” It tells the agent what success means, which files are editable, how long each experiment should run, how to log hypotheses, how to handle crashes, and when to keep or discard a change. In a small system, this can be a Markdown file that acts like a lightweight skill; in larger systems, it can include role-specific instructions, experiment queues, coding standards, ablation policies, and escalation rules.

  • A strong instruction layer should include:

    • Objective: optimize validation BPB, pass rate, accuracy, reward, latency-adjusted quality, or a Pareto frontier.
    • Search constraints: editable files, forbidden files, maximum experiment duration, maximum memory, and allowed dependencies.
    • Scientific hygiene: one primary hypothesis per experiment, record the diff, record why the change was tried, and record why it was kept or reverted.
    • Failure handling: immediately revert NaN-producing candidates, distinguish crash from metric regression, and preserve crash logs.
    • Exploration policy: alternate local improvements with occasional larger rewrites, but avoid stacking many untested changes at once.
    • Reporting policy: append each experiment to a durable table, including candidate ID, score, parent, diff summary, and notes.

  • The idea that the system can improve by editing the agent design itself is closely related to Automated Design of Agentic Systems by Hu et al. (2025), which frames agent design as a search problem over executable agentic systems rather than a purely manual engineering task.

Durable memory and experiment history

  • A durable memory turns a sequence of isolated attempts into cumulative research. At minimum, the system should store source code, diffs, scalar metrics, logs, and human-readable notes for every candidate. In a more capable setup, the proposer can query the full filesystem of past attempts using tools such as grep, cat, notebooks, plots, and structured result files.

  • This is the central architectural insight behind Meta-Harness: the proposer should not only see compressed summaries or scalar scores; it should be able to selectively inspect raw prior code, scores, prompts, tool calls, model outputs, state updates, and execution traces. The filesystem can be much larger than the proposer’s context window, so the agent retrieves only what it needs at each iteration rather than packing all history into one prompt.

  • A useful directory layout is:

runs/
  000_seed/
    source/
    metrics.json
    stdout.log
    stderr.log
    trace.jsonl
    notes.md
  001_depth6_lr3e-4/
    source/
    diff.patch
    metrics.json
    stdout.log
    stderr.log
    trace.jsonl
    notes.md
leaderboard.tsv
frontier.json
current_best/
  • This memory design makes the system closer to an empirical scientist than a blind optimizer. It can notice that several high-depth candidates improved early loss but crashed late, that a throughput improvement came from reduced sequence length rather than better modeling, or that a prompt-harness change improved easy cases but hurt rare classes.

Search controller

  • The search controller decides how candidates are generated and evaluated. The simplest controller is greedy hill climbing: accept a candidate only if it improves the metric. A more robust controller maintains a population of candidates and a Pareto frontier over multiple objectives such as quality, context length, runtime, and memory.

  • For a single scalar metric where lower is better:

\[c_{\mathrm{best},t} = \arg\min_{c_i \in \{c_1,\dots,c_t\}} s(c_i)\]

  • For multiple metrics, candidate \(a\) Pareto-dominates candidate \(b\) if:

    \[\forall j,\ f_j(a) \le f_j(b) \quad \text{and} \quad \exists k,\ f_k(a) < f_k(b)\]
    • where each \(f_j\) is a cost-like metric such as validation BPB, latency, memory, or context tokens. Pareto tracking is especially useful when optimizing harnesses because a slightly less accurate harness may be preferable if it uses far fewer tokens, fewer model calls, or less wall-clock time. Meta-Harness explicitly maintains a population and Pareto frontier while leaving parent selection flexible: the proposer can inspect any prior harness and its traces rather than being constrained to a fixed evolutionary parent rule.

Validation gates

  • Validation gates protect the search from wasting compute on invalid candidates. A basic autoresearch system should perform static and dynamic checks before running the full experiment:
format check
import check
unit smoke test
short forward/backward pass
short validation pass
NaN/Inf guard
memory estimate
full bounded experiment
  • For harness search, the analogous gates are interface validation, tool-call validation, prompt-shape validation, output-parser validation, and budget validation. In Meta-Harness, proposed harnesses are evaluated only after passing interface validation, and each evaluated candidate contributes code, scores, and traces back into the filesystem.

Implementation skeleton

  • A practical minimal loop looks like this:
def autoresearch_loop(seed_code, proposer, evaluator, budget, max_iters):
    history = []
    best = seed_code

    for t in range(max_iters):
        context = build_research_context(history, best)
        candidate = proposer.propose(context)

        valid, validation_report = validate_candidate(candidate)
        if not valid:
            history.append({
                "candidate": candidate,
                "status": "invalid",
                "report": validation_report,
            })
            continue

        result = evaluator.run(candidate, budget=budget)
        history.append({
            "candidate": candidate,
            "status": "evaluated",
            "metrics": result.metrics,
            "logs": result.logs,
            "diff": result.diff,
        })

        if result.metrics["val_bpb"] < score(best):
            best = candidate

        write_history_to_disk(history)

    return best, history
  • A Meta-Harness-style version changes the optimized artifact from training code to a model harness:
def meta_harness_loop(seed_harnesses, proposer, task_set, model, max_iters):
    filesystem = ExperimentFilesystem()
    population = list(seed_harnesses)

    for harness in population:
        result = evaluate_harness(harness, model, task_set)
        filesystem.store(harness=harness, result=result)

    for t in range(max_iters):
        proposal_context = filesystem.path
        new_harnesses = proposer.propose_harnesses(proposal_context)

        for harness in new_harnesses:
            if not passes_interface_validation(harness):
                filesystem.store_invalid(harness)
                continue

            result = evaluate_harness(harness, model, task_set)
            filesystem.store(harness=harness, result=result)
            population.append(harness)

    return compute_pareto_frontier(filesystem.results)
  • The key implementation detail is that proposal_context should be a navigable filesystem, not a giant serialized prompt. This lets the proposer decide whether to inspect the best runs, the worst regressions, the most recent diffs, specific failure traces, or raw code from older candidates.

Metrics table as the shared interface

  • A durable metrics table is the shared interface between the evaluator, proposer, and human reviewer. It should be append-only and machine-readable. For LLM training, useful columns include:
run_id
parent_id
status
val_bpb
train_loss
tokens_per_second
num_parameters
depth
max_seq_len
device_batch_size
total_batch_size
optimizer
learning_rate
peak_memory_gb
wall_time_seconds
notes
  • For harness optimization, useful columns include:
run_id
parent_id
status
accuracy
pass_rate
reward
context_tokens
num_model_calls
latency_seconds
tool_calls
crash_rate
parse_error_rate
trace_dir
notes
  • The system should never rely only on the final metric. A candidate with slightly better BPB but much worse throughput may not be a true improvement under longer budgets. A harness with higher accuracy but twice the context cost may be worse at deployment scale. This is why Pareto tracking is often a better default than a single leaderboard.

Failure modes designed into the architecture

  • Autoresearch systems need architecture-level defenses because agents will otherwise exploit ambiguity in the objective. Common failure modes include metric hacking, accidental test leakage, overfitting to a tiny validation set, increasing runtime while appearing to improve quality, silently changing evaluation code, and producing changes that work only on the current hardware. Keeping data preparation and evaluation frozen reduces these risks, while logging full diffs and traces makes suspicious improvements inspectable.

  • For harnesses, the analogous risks are prompt overfitting, hard-coded labels, benchmark-specific if-statements, brittle parsers, leakage from previous test results, and inflated context use. Code-space search has one practical advantage here: brittle shortcuts are often visible in the source code, unlike weight-space overfitting, where the failure may be hidden inside parameters.

Running the research loop

  • The research loop is where autoresearch stops being a collection of scripts and becomes an autonomous experimental process. A good loop is not just “let an agent edit code.” It is a disciplined cycle of hypothesis formation, constrained editing, validation, measurement, memory update, and selection.

The experiment lifecycle

  • Each experiment should begin with a concrete hypothesis, not just a code change. A hypothesis can be local, such as “reducing depth while increasing batch size may improve validation BPB within the fixed wall-clock budget,” or structural, such as “changing the attention pattern may improve throughput enough to outweigh a small modeling-quality loss.” The agent should then make the smallest coherent edit that tests the hypothesis, run validation gates, launch the bounded experiment, and write a postmortem.

  • A practical lifecycle is:

    • Plan: inspect the leaderboard, current best code, and recent regressions.
    • Hypothesize: state the mechanism expected to improve the metric.
    • Edit: modify only the allowed artifact.
    • Validate: run static checks, smoke tests, and NaN guards.
    • Evaluate: run the full bounded experiment.
    • Record: save source, diff, logs, metrics, and notes.
    • Select: keep, revert, branch, or mark for follow-up.

  • This differs from Self-Refine by Madaan et al. (2023), which iteratively improves model outputs using self-feedback, because autoresearch applies the refinement loop to executable research code and evaluates the result through external experiments rather than only through textual critique.

Candidate generation

  • Candidate generation should balance exploitative local edits with occasional exploratory changes. Purely local search often gets stuck refining the current best configuration, while unconstrained exploration wastes compute on invalid or noisy candidates. A good proposer should alternate among several edit families:

    • Hyperparameter edits: learning rate, warmup, weight decay, batch size, dropout, gradient clipping, optimizer betas, and scheduler shape.
    • Architecture edits: depth, width, MLP ratio, normalization, attention pattern, residual scaling, positional encoding, and parameter tying.
    • Training-loop edits: mixed precision, compilation, gradient accumulation, evaluation cadence, data packing, loss scaling, and optimizer step ordering.
    • Efficiency edits: fused operations, memory layout, reduced overhead, activation checkpointing, and batch-shape tuning.
    • Robustness edits: NaN checks, safer initialization, fallback paths, and clearer logging.

  • The outer-loop proposal distribution can be written as a mixture:

    \[\pi(c_{t+1} \mid \mathcal{H}_t) = \sum_{k=1}^{K} \alpha_k \pi_k(c_{t+1} \mid \mathcal{H}_t)\]
    • where each \(\pi_k\) is an edit family and \(\alpha_k\) is the current probability of selecting that family. In practice, the agent can adapt \(\alpha_k\) from history: if optimizer edits keep causing instability, reduce their frequency; if attention-pattern edits repeatedly improve throughput without hurting BPB, inspect them more deeply.

  • This view is close to Evolution through Large Models by Lehman et al. (2022), which treats language models as mutation operators for program search, but autoresearch typically uses a stronger experimental memory and a tighter evaluation budget.

Evaluation under a fixed budget

  • A fixed budget turns every proposal into a resource-constrained optimization problem. The score is not “best validation loss eventually,” but “best validation loss after the allowed compute.” For language modeling, the evaluator should measure at least:
\[\mathrm{val\_bpb}\] \[\mathrm{tokens/sec} = \frac{N_{\mathrm{train\ tokens}}}{T_{\mathrm{wall}}}\] \[\mathrm{effective\ improvement} = \mathrm{BPB}_{\mathrm{parent}} - \mathrm{BPB}_{\mathrm{candidate}}\]

  • A candidate that improves BPB by making the model much larger may still be undesirable if it only wins because it accidentally receives more effective training tokens, changes the validation path, or makes evaluation inconsistent. The fixed time budget protects against some of this, but the system should still log throughput, train tokens, parameter count, memory usage, and crash status.

  • A useful scalarized score for early triage is:

    \[J(c) = \mathrm{BPB}(c) + \lambda_1 \cdot \max(0, M(c) - M_{\max}) + \lambda_2 \cdot \mathbb{1}[\mathrm{crash}(c)] + \lambda_3 \cdot \mathbb{1}[\mathrm{nan}(c)]\]
    • where \(M(c)\) is peak memory, \(M_{\max}\) is the allowed memory threshold, and the indicator penalties prevent invalid runs from being ranked as promising.

Selection and acceptance

  • The simplest acceptance rule is greedy improvement:

    \[\mathrm{accept}(c_{t+1}) = \mathbb{1} \left[ s(c_{t+1}) < s(c_{\mathrm{best}}) \right]\]
    • where lower \(s\) is better. Greedy acceptance is easy to audit, but it can be too conservative. Some candidates may be worse overall while containing a useful sub-change, such as faster data loading or a more stable initialization. A better system stores all candidates, keeps the current best as the deployment baseline, and allows future proposals to branch from any prior run.

  • For noisy metrics, use a margin:

    \[\mathrm{accept}(c_{t+1}) = \mathbb{1} \left[ s(c_{t+1}) < s(c_{\mathrm{best}}) - \epsilon \right]\]
    • where \(\epsilon\) is a practical significance threshold estimated from repeated baseline runs. Without this threshold, the agent may overfit to random measurement variation.

Branching and population management

  • Autoresearch should not be a single linear chain unless the experiment cost is extremely high. A population gives the agent multiple promising lineages to revisit. Each candidate can be tagged by its parent, edit family, and outcome:
run_id    parent    family          val_bpb    status
000       none      seed            0.9978     kept
001       000       optimizer       0.9905     kept
002       001       depth           0.9874     kept
003       002       attention       0.9846     kept
004       003       batch-size      crash      rejected
005       002       scheduler       0.9839     kept

  • A population-based controller can sample parents according to a softmax over score:

    \[P(\mathrm{parent}=i) = \frac{ \exp(-s_i / \tau) }{ \sum_j \exp(-s_j / \tau) }\]
    • where \(\tau\) controls exploration. Smaller \(\tau\) focuses on the current best; larger \(\tau\) lets the agent revisit diverse candidates.

  • This is related to GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning by Agrawal et al. (2025), which maintains a Pareto-aware reflective optimization process over prompts, but autoresearch extends the same evolutionary pressure to executable code and experimental systems.

Trace-driven diagnosis

  • Scalar metrics are too compressed for serious autonomous research. A useful loop gives the agent access to full traces: code diffs, stdout, stderr, validation curves, throughput, memory, seed, hardware metadata, and notes. The agent should be able to answer questions such as:

    • Did the candidate improve because it trained more tokens per second?
    • Did it improve early but plateau worse?
    • Did it reduce BPB by changing tokenizer or validation behavior?
    • Did it crash only after evaluation, during compilation, or during backward pass?
    • Did a worse candidate contain a subcomponent worth transplanting?

  • The Meta-Harness pattern makes this trace-driven workflow explicit: the proposer reads prior code, scores, and execution traces from a filesystem, chooses what to inspect, and uses that evidence to propose the next harness rather than relying on a fixed summary. The same design should be used for autoresearch on training code.

Handling crashes and invalid candidates

  • Crashes are data. They should not disappear from the history because they teach the agent which regions of the search space are unstable. Each failed candidate should be classified:
syntax_error
import_error
shape_error
out_of_memory
nan_loss
timeout
metric_missing
evaluation_changed
external_dependency_failure

  • For each class, the loop should have a default policy. Syntax and import errors should trigger immediate repair or rejection. Out-of-memory candidates should be logged with peak memory and likely cause. NaN-producing candidates should be rejected unless the purpose of the next run is explicitly to stabilize them. Timeout candidates should record partial progress but should not be compared directly against full-budget runs.

  • A strong NaN guard looks like:

if not torch.isfinite(loss):
    raise RuntimeError(f"Non-finite loss at step {step}: {loss.item()}")
  • A strong metric guard checks that the result file exists, has the expected schema, and was produced by the current run ID. This prevents the agent from accidentally reusing stale scores.

Preventing evaluation drift

  • Evaluation drift is one of the most dangerous failure modes. If the agent can edit the validation data, metric computation, tokenizer accounting, or result parser, it may accidentally or deliberately make the score easier. The frozen substrate should therefore include:

    • validation split construction
    • evaluation-token selection
    • BPB computation
    • result serialization
    • run-time enforcement
    • comparison script
    • data download and preprocessing

  • For extra safety, hash the evaluator files before each experiment:

\[h_t = \mathrm{SHA256} \left( \mathrm{prepare.py} \parallel \mathrm{eval.py} \parallel \mathrm{validation\_manifest} \right)\]
  • Then reject candidates where \(h_t \ne h_0\) unless a human explicitly approved an evaluator migration.

Multi-objective research

  • Autoresearch becomes more useful when it optimizes multiple metrics instead of a single leaderboard number. In LLM training, relevant objectives include BPB, throughput, memory, parameter count, and code complexity. In harness optimization, relevant objectives include task score, context tokens, number of model calls, latency, parse-error rate, and tool-call cost.

  • A candidate belongs on the Pareto frontier if no other candidate is at least as good on every objective and strictly better on one:

\[c_i \in \mathcal{P} \iff \nexists c_j: \left( \forall k,\ f_k(c_j) \le f_k(c_i) \right) \land \left( \exists k,\ f_k(c_j) < f_k(c_i) \right)\]
  • This is especially important for harnesses because a high-accuracy system that uses enormous context may be less valuable than a slightly lower-accuracy system that is cheaper, faster, and more robust. Meta-Harness uses this kind of Pareto framing when multiple objectives such as accuracy and context cost matter.

Research notes as training data for the next loop

  • Every experiment should produce a short note in a consistent format:
Hypothesis:
Changing X should improve Y because Z.

Change:
Modified A, B, and C.

Result:
val_bpb changed from 0.9846 to 0.9839.
Throughput changed from 410k tok/s to 398k tok/s.
No NaNs or crashes.

Interpretation:
The improvement is real but small. The slower throughput suggests the modeling change helped more than the efficiency cost hurt.

Next:
Try the same architecture with the previous faster batch schedule.
  • These notes matter because they give the proposer a compact human-readable layer on top of raw logs. They should not replace raw traces, but they make the history easier to navigate. TextGrad by Yuksekgonul et al. (2024) is relevant because it shows how textual feedback can serve as an optimization signal for AI systems, but in autoresearch that feedback should be paired with executable traces and external metrics.

A robust loop template

  • A production-grade autoresearch loop should look like this:
def run_one_experiment(candidate, parent, budget):
    run_id = allocate_run_id()

    save_source_snapshot(run_id, candidate)
    save_diff(run_id, parent, candidate)

    validation = validate_candidate(candidate)
    save_json(run_id, "validation.json", validation)

    if not validation["ok"]:
        save_status(run_id, "invalid")
        return {"run_id": run_id, "status": "invalid"}

    result = launch_training(
        source=candidate,
        wall_clock_budget=budget,
        run_id=run_id,
    )

    metrics = parse_metrics(result)
    guards = check_metric_integrity(metrics)

    save_json(run_id, "metrics.json", metrics)
    save_text(run_id, "stdout.log", result.stdout)
    save_text(run_id, "stderr.log", result.stderr)
    save_json(run_id, "guards.json", guards)

    if not guards["ok"]:
        save_status(run_id, "rejected_metric_integrity")
    elif metrics["nan_or_inf"]:
        save_status(run_id, "rejected_nan")
    elif metrics["crashed"]:
        save_status(run_id, "rejected_crash")
    else:
        save_status(run_id, "evaluated")

    append_leaderboard(run_id, metrics)
    return {"run_id": run_id, "status": "done", "metrics": metrics}
  • The loop’s most important invariant is that every candidate, including invalid ones, leaves behind enough evidence for the next proposer to learn from it. This is what turns autonomous tinkering into cumulative research.

Harness optimization with Meta-Harness

  • Autoresearch becomes most powerful when the optimized object is not only a model-training script, but the full harness around a model: the code that decides what the model sees, what it remembers, what tools it can use, how outputs are parsed, how intermediate state is updated, and how future calls are conditioned on prior events. Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026) formalizes this as an outer-loop search over executable harness code, using a coding-agent proposer that can inspect previous code, scores, and execution traces through a filesystem.

What a harness is

  • A harness is the executable environment wrapped around a fixed model. In a simple classifier, the harness may only build a prompt. In a retrieval-augmented math solver, it may select solved examples, format them, call the model, parse the answer, and retry on malformed outputs. In an agentic coding system, it may maintain task state, decide when to call the shell, summarize files, route tool outputs, recover from errors, and decide when to submit.

  • Context compaction is one of the harness’s most consequential state transformations: durable evidence may remain stored externally even when the model-visible state has discarded it, so what the system remembers and what the model can currently access are distinct quantities.

  • The harness is therefore a policy over context and control flow:

    \[a_t, p_t, z_t = H(s_t, x, \mathcal{M}_t)\]
    • where \(s_t\) is the current task state, \(x\) is the task instance, \(\mathcal{M}_t\) is external memory, \(p_t\) is the prompt or model input, \(a_t\) is the next action or tool call, and \(z_t\) is the updated internal state. The model itself remains fixed, but the harness can change the distribution of trajectories the model produces.

  • A rollout can be written as:

    \[\tau = \left( s_0, p_0, y_0, a_0, s_1, p_1, y_1, a_1, \dots, s_T \right)\]

    • where \(y_t \sim M(\cdot \mid p_t)\) is the model output at step \(t\). Harness optimization searches for the harness \(H\) that maximizes expected reward:

      \[H^{*} = \arg\max_{H} \mathbb{E}_{x \sim X,\ \tau \sim p_M(H,x)} \left[ r(\tau, x) \right]\]

  • This objective is useful because it makes clear that harness engineering is not “prompt tweaking”; it is program search over the system that induces model behavior. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines by Khattab et al. (2023) is relevant because it treats LM pipelines as optimizable programs rather than fixed prompt strings, which is the same systems-level direction taken by harness optimization.

Why harnesses are high-leverage

  • Harnesses are high-leverage because many model failures are not caused by missing parametric knowledge alone. They are caused by missing context, poorly selected examples, lossy summaries, brittle parsers, premature stopping, unhelpful tool routing, or state that is stored in the wrong form. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis et al. (2020) is relevant because it shows how external non-parametric memory can improve generation when the right information is retrieved and conditioned on.

  • In long-running agents, the compaction policy can therefore change effective capability without changing model weights by deciding which observations, decisions, tool results, and constraints remain available after the active context is rewritten.

  • The practical implication is that a fixed model \(M\) can produce very different outcomes under different harnesses:

    \[p_M(\tau \mid H_1, x) \ne p_M(\tau \mid H_2, x)\]
    • even though the underlying model weights are identical. A better harness changes the task distribution seen by the model at inference time: it can expose better examples, compress history more faithfully, recover from tool errors, ask for structured outputs, or decompose tasks into more solvable subproblems.

Harness-managed memory as a capability variable

The harness determines the model’s effective memory

  • A model’s effective memory is not determined by its context-window length alone. In an interactive system, the harness determines which observations survive between calls, which previous reasoning artifacts are restored, when a trajectory is summarized or compacted, which archived evidence can be retrieved, and how much of that state is placed back into the model’s active context. Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026) makes storage, retrieval, and presentation explicit components of the executable harness searched around a fixed model, rather than treating them as properties of the model weights themselves.

  • It is useful to distinguish the model’s nominal context capacity from the memory state actually exposed by the harness. Let \(\Gamma_t\) denote the complete interaction history available in principle at step \(t\) and let \(Z_t\) denote the compact state that the harness chooses to make usable:

    \[Z_t = C_{\mathcal{H}} \left( \Gamma_t, Q_t \right), \qquad \operatorname{rate}(Z_t) \le B_t\]
    • where \(\mathcal{H}\) is the harness, \(C_{\mathcal{H}}\) is its context-management policy, \(Q_t\) is the current information need, and \(B_t\) is the available memory or context budget. The model then acts from the prompt constructed from the current observation and selected memory:
    \[Y_t \sim M \left( \cdot \mid P_{\mathcal{H}} \left( X_t, Z_t \right) \right)\]
    • Thus, even with the same frozen model \(M\), changing \(C_{\mathcal{H}}\) or \(P_{\mathcal{H}}\) changes which evidence reaches the model and therefore changes the trajectory distribution.

  • MemGPT: Towards LLMs as Operating Systems by Packer et al. (2023) provides an early systems formulation of this idea by treating the limited context window as fast memory backed by larger external memory tiers, with explicit movement of information between them. Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026) generalizes the design space further by making these storage, retrieval, context-construction, and control-flow choices themselves searchable harness code.

The same model under different memory interfaces

  • Interactive benchmarks provide a particularly direct demonstration that harness configuration can materially alter observed capability. OpenAI’s GPT-6 Astra on ARC-AGI-3 reports a best ARC-AGI-3 Semi-Private score of 62.7% for GPT-6 Astra under ARC Prize’s Standard harness, compared with 99.9% under its Provider Adapter harness. The Standard harness gives the model a provider-neutral interface and lets it decide what information to retain in visible notes, whereas the Provider Adapter preserves opaque reasoning state across requests and performs compaction as conversations grow. At matched maximum reasoning effort, the corresponding scores are 62.7% and 98.6%, showing that the gap is not explained solely by comparing different reasoning-effort settings.

  • The Provider Adapter also changed the efficiency profile of the system. Across the Public and Semi-Private evaluations and the \(167\) game-reasoning pairs solved by both configurations, ARC Prize reports that Provider Adapter runs were approximately \(3.66\times\) faster by aggregate recorded elapsed time and consumed 49% fewer total tokens. This illustrates why harness evaluation should treat context management as part of the system being measured, rather than reporting model identity alone.

  • These results should not be interpreted as a controlled causal estimate of “memory” alone. The Provider Adapter simultaneously changes preserved reasoning state and conversation compaction, and the underlying compaction policy is opaque to the evaluator. The rigorous conclusion is therefore that the same frozen model can exhibit substantially different measured performance under different context-management interfaces. ARC Prize accordingly reports Standard and Provider Adapter results separately because they answer different evaluation questions: standardized model capability versus model-plus-provider-harness capability.

  • This distinction is important for autoresearch. If a candidate harness improves a benchmark score by retaining more useful state, retrieving forgotten evidence, or compressing history more effectively, that is a real system improvement, but it should be attributed to harness optimization rather than to a change in the underlying model. Conversely, model comparisons become difficult to interpret when harness memory policies differ substantially between systems.

A three-tier memory hierarchy

  • What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) places memory management across the stack into a common hierarchy: the KV cache operates at token-level granularity within a forward pass, the working context persists across steps within a task, and the long-term semantic store persists across tasks or sessions. Despite their different representations and timescales, all three tiers face the same problem of preserving future task-relevant information under a finite budget.

  • The following figure (source) shows the three-tier memory hierarchy spanning the on-GPU KV cache, in-window working context, and external long-term store, together with the shared design knobs that recur across these otherwise different memory substrates.

  • The harness can interact with all three tiers, although not always through the same mechanism:

    • KV-cache tier: the runtime may retain, evict, quantize, page, or reuse cached attention state.
    • Working-context tier: the harness chooses which observations, tool outputs, plans, errors, and intermediate results remain directly visible during the current task.
    • Long-term tier: the harness decides which experiences are archived across tasks, how they are indexed, what abstractions are stored, and which original evidence remains recoverable.
    • Cross-tier movement: the harness determines when high-fidelity evidence should be promoted into active context, when active context should be demoted into an archive, and when semantic abstractions should replace raw state.

  • The same design dimensions recur at each tier. What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) identifies importance scoring, forgetting policy, query conditioning, reversibility, budget allocation, and compaction stopping criteria as closely related choices across KV and agent memory; the paper specifically notes that mechanisms developed at one tier can suggest designs at another.

Rate-distortion as a harness objective

  • Memory-aware harness optimization can therefore be viewed as a constrained version of the ordinary Meta-Harness objective. Instead of searching only for the harness with the highest task reward, the system searches for a context-management policy that preserves downstream utility under explicit resource constraints:
\[\mathcal{H}^{*} = \arg\max_{\mathcal{H}} \mathbb{E}_{x,\tau} \left[ r(\tau,x) \right]\] \[\text{s.t.} \qquad \mathbb{E} \left[ \operatorname{rate}(Z_t) \right] \le B\]

  • Equivalently, quality and memory consumption can be represented as separate Pareto objectives rather than collapsed immediately into one scalar:

    \[\left( -\mathbb{E}[r], \; \mathbb{E}[\operatorname{tokens}], \; \mathbb{E}[\operatorname{memory}], \; \mathbb{E}[\operatorname{latency}] \right)\]
    • This is particularly appropriate for harness search because more context is not free. A harness that gains one percentage point of accuracy by doubling context length or preserving every observation indefinitely may occupy a different operating point from one that obtains nearly the same quality using selective retrieval.

  • The rate-distortion formulation gives this tradeoff a more precise interpretation. What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) treats a compact representation \(Z\) as useful only insofar as it retains information required for downstream task utility; once the budget falls below the task-conditioned information requirement, some error becomes unavoidable.

  • A harness should therefore not optimize compression ratio independently of the task. The same amount of compression may be nearly harmless when the history contains redundant prose but destructive when the future task requires exact identifiers, causal dependencies, or multi-hop evidence distributed across earlier steps. Memory budget is consequently another resource that should be optimized jointly with reward rather than treated as a fixed preprocessing detail.

Decision-aware context management

  • Rate-distortion alone specifies that task-relevant information should survive, but an agent still needs a criterion for deciding what distinctions are task relevant. Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) proposes that the relevant equivalence relation should be defined by downstream decisions: histories can safely share a memory state when collapsing them does not materially change which action should be taken.

  • This distinction matters for harness search because semantic similarity is not necessarily decision similarity. Two traces may both describe a failed coding attempt, for example, while one failed because retrieval selected the wrong file and another failed because the correct patch did not compile. A descriptive summarizer may collapse them into “the edit failed,” whereas the next useful intervention is different in each case.

  • Formally, if \(\Delta_q(h,a)\) is the loss from using action \(a\) at history \(h\) for query \(q\), then a set of histories \(C\) can safely share a memory representation at tolerance \(\epsilon\) only when there exists an action that remains near-optimal throughout that set:

    \[\exists a\in\mathcal{A} \quad \text{s.t.} \quad \max_{h\in C} \Delta_q(h,a) \le \epsilon\]

  • For a metaharness, this suggests that context-management search should optimize not merely summary quality but decision preservation. A proposer evaluating a memory-policy edit should ask whether the new representation still supports the same downstream tool choice, experiment choice, retrieval target, or stopping decision that the uncompressed trace would have supported.

Query-conditioned and reversible memory

  • Two particularly important harness properties are query conditioning and reversibility. A query-conditioned memory system delays some selection until the downstream information need is known. A reversible system preserves a path to reconstruct or retrieve information that was removed from the active context. What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) argues that these properties repeatedly separate robust memory systems from schemes that permanently discard state before its future utility is known.

  • This leads to a useful harness architecture:

    \[\text{raw archive} \rightarrow \text{compact index} \rightarrow \text{query-conditioned retrieval} \rightarrow \text{active context}\]
    • The raw archive is the reversible source of truth. The compact index makes the archive searchable. The current query selects which evidence to restore. Only the small retrieved subset needs to occupy expensive active context.

  • This is preferable to:

    \[\text{history} \rightarrow \text{summary} \rightarrow \text{summary of summary} \rightarrow \cdots\]
    • when the original observations are irreversibly discarded, because repeated summarization can compound information loss. The rate-distortion analysis reports a reference experiment in which retrieval-backed reversible memory maintained approximately \(0.95\) fact recall across repeated compaction events, while irreversible summarization fell to roughly 0.33 to 0.56 depending on compaction frequency. The experiment is intentionally small-scale, but it demonstrates the failure mode that a metaharness evaluator should explicitly measure.

Memory policy as part of the Meta-Harness search space

  • Once memory is treated as harness code, an outer-loop proposer can search over substantially more than prompt wording. Candidate harnesses can alter:

    • Persistence policy: which model outputs, observations, plans, errors, and tool results survive across calls.
    • Compaction trigger: whether compression occurs at a fixed token threshold, after task boundaries, asynchronously, or only when a resource limit is approached.
    • Compaction representation: extractive notes, abstractive summaries, structured state, symbolic representations, graphs, or learned memory objects.
    • Retrieval policy: semantic nearest-neighbor retrieval, symbolic lookup, temporal retrieval, query-conditioned routing, or hybrid methods.
    • Reversibility: whether original evidence remains archived after it leaves active context.
    • Memory fidelity: whether important items remain verbatim while less important regions are summarized.
    • Budget allocation: how available tokens or bytes are divided among instructions, current observations, retrieved history, examples, scratch state, and tool outputs.
    • Stopping rule: when additional compression would create unacceptable downstream distortion.

  • Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026) is particularly well suited to this search because its proposer can inspect the complete source, scores, and execution traces of previous candidates rather than receiving only compressed optimizer feedback. That architecture allows the proposer to diagnose not just that one memory policy performed worse, but which downstream failures followed from specific retrieval omissions, compaction decisions, or state-update rules.

  • A memory-aware harness evaluator should therefore log enough information to attribute performance to these decisions:

memory_state_before
memory_operation
items_added
items_removed_from_active_context
items_archived
items_irreversibly_discarded
retrieval_query
retrieved_items
retrieval_scores
active_context_tokens
archive_size
compaction_count
model_output
downstream_action
task_reward
  • These fields make memory operations visible to the outer-loop researcher. If a candidate’s accuracy falls only after its fifth compaction event, the proposer can distinguish cumulative memory loss from a reasoning failure. If accuracy improves while context tokens fall, it can identify selective retrieval as a genuine efficiency gain. If the final model receives the correct evidence but still fails, the next experiment should target reasoning or control flow rather than memory.

Benchmarking the model and harness separately

  • Harness-sensitive performance also motivates a stricter reporting convention. An interactive agent result is better represented as:

    \[\operatorname{Score} \left( M, \mathcal{H}, B, R \right)\]
    • where \(M\) identifies the model, \(\mathcal{H}\) the harness, \(B\) the context or memory budget, and \(R\) the reasoning or inference configuration. Reporting only \(M\) hides system components that may materially affect observed capability.

  • Comparisons intended to isolate model capability should hold the harness approximately fixed. Comparisons intended to measure deployable agent capability may intentionally use each provider’s best harness, but the result should then be understood as the capability of the compound system. ARC Prize’s decision to report Standard-harness and Provider-Adapter results separately is an example of this distinction.

  • For autoresearch and Meta-Harness, this distinction is especially important because harness improvement is the object of optimization. A large performance change with frozen model weights is not a confound when the research question is “can the harness be improved?” It is the desired result. The experimental contract simply needs to preserve model identity and evaluation conditions so that the improvement can be attributed to the harness rather than to an uncontrolled change in the underlying model.

Context compaction as a harness-level state transformation

Context compaction is not the same as memory storage

  • Long-running agents often retain substantially more information internally than a model can consume in one inference. The critical constraint is therefore not necessarily how much information the agent can store, but how much of that information can be made model-visible on the next call. Context Compaction Theory by Tirmazi et al. (2026) formalizes this distinction by treating context compaction as the operation that converts an agent’s growing internal state into a bounded representation that fits within the model’s effective context window.

  • Let the internal state before an inference be \(\mathcal{I}_t\) and let the model’s available context budget be \(W\).

  • The harness must construct a state satisfying:

    \[\left| C(\mathcal{I}_t) \right| \leq W,\]
    • where \(C\) is the context-compaction operator.

  • This yields an important distinction for agent architecture:

\[\text{stored information} \neq \text{model-visible information}.\]

  • An agent may preserve the complete transcript, tool outputs, files, and checkpoints on disk while carrying forward only the compacted state. Information that survives in durable storage but is absent from the compacted context is effectively unavailable to the model unless the harness explicitly retrieves it again. Production agents commonly carry the compacted state forward because rebuilding a fresh summary from the complete history on every inference would incur substantial latency and model-call cost.

  • The following figure (source) shows the distinction between an agent’s potentially large internal state and the bounded compacted context supplied to the LLM. When the accumulated state exceeds the model-visible budget, the harness transforms it into a smaller context and carries that transformed state forward.

Selection and generation are two fundamental compaction strategies

  • Context Compaction Theory by Tirmazi et al. (2026) separates context compaction into two broad algorithmic classes: selection, which retains a subset of existing information, and generation, which constructs a new bounded representation such as a summary. This distinction covers a broad range of production mechanisms from token pruning and message truncation to LLM-generated conversation summaries.
Context selection

  • Suppose the accumulated context contains atomic information items:

    \[\mathcal{X} = \{ x_1,\ldots,x_N \},\]
    • where each item has size \(s(x_i)>0\).

  • A selection-based condenser chooses \(S\subseteq \mathcal{X}\), subject to \(\sum_{x_i\in S} s(x_i) \leq B\).

  • A future query \(q\in\mathcal{Q}\) is then evaluated using only the retained subset. If \(v_q(S) \in [0,1]\) measures how well the retained information supports the query, the corresponding error is: \(\operatorname{err}(S,q) = 1-v_q(S)\)

  • The important restriction is that the condenser preserves information by choosing among existing items. It cannot arbitrarily recode the complete history into a new representation.

Context generation
  • Generation removes that restriction. A condenser can construct an arbitrary bounded message \(\operatorname{Cond}(\mathcal{X}) \in \Sigma^{\leq B}\), and an interpreter combines that message with the future query:
\[\hat{a} = \operatorname{Int} \left( \operatorname{Cond}(\mathcal{X}), q \right)\]

  • The corresponding quality is determined by \(v_q(\hat{a})\), so the condenser is free to encode the historical information in whatever representation best supports the anticipated query distribution. An LLM-generated summary is one instance, but the definition also admits structured sketches, compressed state machines, learned representations, or other bounded encodings.

  • Since any selected subset can itself be encoded as a generated message:

\[\mathrm{SELECT} \subseteq \mathrm{GEN}\]
  • This formal distinction matters for harness search. Removing old messages, dropping tool outputs, retaining only recent turns, and ranking code symbols are selection operators. Summarizing a trajectory, compiling it into structured state, or constructing a task-specific sketch are generation operators. A metaharness should therefore search over both operator families rather than equating context management with summarization alone.

Context compaction is a communication problem

  • The central theoretical result in Context Compaction Theory by Tirmazi et al. (2026) is that generative context compaction is equivalent to one-way communication complexity. The accumulated history plays the role of information known to a sender, the compacted context is the bounded message, and the future query is information revealed only to the receiver.

  • Conceptually:

\[\begin{aligned} \text{history} &\rightarrow \text{condenser} \rightarrow \text{bounded message} \\ &\rightarrow \text{future query} \rightarrow \text{interpreter} \rightarrow \text{answer} \end{aligned}\]
  • For a stochastic workload with query distribution \(\mu\), the minimum generative compaction budget required to achieve target error \(\epsilon\) equals the distributional one-way communication complexity of the corresponding query problem:
\[B_{\mathrm{GEN}}^{\star} (\mu,\epsilon) = R_{\mu,\epsilon}^{\rightarrow} \left( \Pi_{\mathcal{G}} \right)\]
  • For an oblivious worst-case query regime:
\[B_{\mathrm{GEN}}^{\star} (\epsilon) = R_{\epsilon}^{\rightarrow} \left( \Pi_{\mathcal{G}} \right)\]
  • This transforms compaction from a purely heuristic systems problem into an information-budget problem. If communication complexity proves that a workload fundamentally requires a certain number of bits, no better summarization prompt, retrieval heuristic, or LLM can circumvent that lower bound.

Future queries determine what can safely be forgotten

  • Compaction occurs before the system necessarily knows which historical detail will matter later. The problem therefore depends on the future query regime.

  • In a stochastic regime, the condenser can exploit a known distribution of likely future queries and optimize expected error. In an oblivious-adversarial regime, it must support every possible query because the adversary can choose a difficult query after the context has been compacted, although without observing the specific compacted output.

  • This produces a general harness-design principle:

    \[\text{optimal compaction} = f \left( \text{future information needs} \right)\]
    • not merely:
    \[f left( \text{historical token importance} \right)\]

  • For autoresearch, for example, the information needed to answer “which optimizer settings have repeatedly caused instability?” differs from the information needed to answer “which candidate produced the best throughput-quality tradeoff?” A context representation optimized only for recency or semantic similarity may preserve the wrong distinctions.

Generation can be strictly more compact than selection

  • Generation is not merely a convenient implementation of selection. There are workloads where it is fundamentally more expressive.

  • Context Compaction Theory by Tirmazi et al. (2026) constructs a family with \(n = 2^k\) items where a generative encoding answers the target query exactly using \(B_{\mathrm{GEN}} = n\) bits, while every exact selection-based algorithm requires at least:

    \[B_{\mathrm{SELECT}} \geq n\log_2 n\]

    • Thus:

      \[\frac{ B_{\mathrm{SELECT}} }{ B_{\mathrm{GEN}} } \geq \log_2 n\]

  • The separation arises because generation can encode global information about the set rather than physically retaining the selected objects themselves.

  • For agent harnesses, this explains why a carefully designed structured state can sometimes dominate increasingly elaborate truncation policies. Selection preserves evidence directly; generation can preserve sufficient statistics, decisions, invariants, dependency structures, or other abstractions that require substantially fewer tokens than the underlying observations.

Production systems often combine selection and generation

  • The theoretical taxonomy also describes practical agent systems. The production survey in Context Compaction Theory by Tirmazi et al. (2026) classifies Codex and Gemini CLI primarily as generation-based systems because they replace earlier context with generated summaries, while Claude Code and OpenCode combine selective removal of tool results with a generative summarization fallback. Message trimming, code-symbol ranking, and token-level prompt compression occupy increasingly fine-grained selection regimes.

  • This suggests a richer metaharness search space than a binary “compact or do not compact” switch. Candidate policies can vary:

    • Selection granularity: Entire turns, tool results, files, code symbols, spans, or individual tokens.
    • Generation format: Narrative summaries, structured XML or JSON, decision ledgers, state sketches, or task-specific semantic representations.
    • Hybrid ordering: Select low-value material first, then summarize only when selection no longer frees sufficient capacity.
    • Recency policy: Preserve a recent uncompressed tail while compacting older history.
    • Compaction trigger: Use a hard context threshold, effective-context threshold, latency threshold, or learned policy.
    • Interpreter: Change how the subsequent model is instructed to interpret the compacted state.

OpenHands: thresholded condensation with a recent raw tail

  • OpenHands Context Condensensation for More Efficient AI Agents describes a practical hybrid policy in which older interactions are summarized after the accumulated context crosses a threshold while recent exchanges remain intact. The generated summary emphasizes user goals, completed progress, remaining work, critical files, and failing tests.

  • Under the selection-generation taxonomy, this can be viewed as a hybrid architecture:

\[\begin{aligned} \text{old trajectory} &\rightarrow \text{generative summary} \rightarrow \text{compact memory} \\ \text{recent trajectory} &\rightarrow \text{direct retention} \rightarrow \text{raw recent tail} \\ & \rightarrow \text{next model context}. \end{aligned}\]

  • The recent tail avoids immediately compressing the most locally relevant state, while the generated prefix carries forward longer-term task information.

  • The following figure (source) shows the OpenHands condenser workflow: the conversation grows until it reaches a threshold, older events are combined into an LLM-generated summary, and the resulting summary is prepended to a shorter uncompressed recent context.

Compaction timing interacts with prompt caching

  • The OpenHands design also highlights that compaction frequency is a systems parameter, not only an information-quality parameter. Rewriting a summary every turn invalidates part of the reusable prompt prefix and repeatedly pays the cost of condensation. Instead, OpenHands waits until a threshold is reached, allowing the same prefix to remain stable across multiple turns and amortizing the cache-rebuilding cost.

  • Under the simplifying assumption that events have similar size, the article characterizes cumulative processing without condensation as growing quadratically with conversation length, while periodically bounding the active context makes the corresponding growth linear.

  • This creates a compaction-frequency tradeoff:

\[\mathrm{TotalCost} = \mathrm{InferenceCost} + \mathrm{CompactionCost} + \mathrm{CacheInvalidationCost}\]
  • Compacting too late allows the active context to become expensive and potentially suffer context rot. Compacting too early repeatedly invokes the condenser and gives up cache reuse. The optimal trigger therefore depends jointly on context quality, input pricing, output pricing, cache behavior, latency, and the expected remaining trajectory length.

Context condensation can improve the cost-quality frontier

  • OpenHands evaluated its condenser against its baseline agent on a subset of SWE-bench Verified. After condensation begins, the reported average API cost per turn remains approximately bounded while the baseline continues increasing, eventually leaving the condenser at less than half the baseline per-turn cost.

  • The following figure (source) shows average API cost per turn over a long agent trajectory. The baseline cost grows as more history is repeatedly supplied, whereas the condenser stabilizes the model-visible context and keeps per-turn cost approximately bounded after condensation begins.

  • On the tested subset, the condensed agent solved an average of 54% of instances compared with 53% for the baseline. The article also reports a larger fraction of problems solved under several cost, token, and completion-time constraints, while noting that condensation can add turns because a condensation operation itself may consume an agent turn.

  • The important evaluation principle is not that summarization universally improves accuracy. Rather, a context policy should be evaluated as a joint quality-cost intervention:

    \[\left( \mathrm{task\ success}, \mathrm{tokens}, \mathrm{latency}, \mathrm{API\ cost}, \mathrm{turns} \right)\]
    • because a policy can be valuable even when headline task quality remains essentially unchanged if it substantially reduces the resources required to obtain that quality.

OpenClaw: preserving structure during compaction

  • Compaction in OpenClaw implements another hybrid design: older conversation turns are summarized into a persistent compact entry while recent turns remain verbatim, and the full conversation continues to exist on disk. Compaction therefore modifies the model-visible trajectory without deleting the underlying provenance.

  • The design also treats some message boundaries as semantic invariants. Assistant tool calls remain paired with their corresponding tool results, so the split point is moved rather than separating an action from the observation it produced. This is a useful general principle for agent compaction:

\[\text{atomic state unit} \neq \text{arbitrary token interval}.\]
  • A coding trajectory may contain compound structures whose semantics depend on adjacency, such as a tool call and its output, an edit and its test result, or a hypothesis and its evaluation. Breaking such structures can introduce state corruption even when all retained tokens are individually correct.

Compaction needs integrity checks

  • OpenClaw makes compaction validation part of the harness rather than trusting every generated summary. Its safeguard mode checks the finalized summary against requirements such as required headings, pending asks, and exact identifiers. If the summary fails validation after the allowed correction attempts, compaction is aborted before the transcript is modified and the original history is preserved.

  • This suggests a generic transactional pattern for compaction:

\[\begin{aligned} \text{prepare candidate summary} &\rightarrow \text{validate invariants} \rightarrow \text{check budget} \rightarrow \text{commit} \\ &\rightarrow \text{otherwise preserve original state}. \end{aligned}\]
  • The validation target should depend on the agent domain. Coding agents may preserve filenames, identifiers, failing tests, user constraints, pending edits, and unresolved tool errors. Research agents may preserve experiment IDs, numerical results, hypotheses, evaluation failures, and links back to raw evidence.

Preserve durable information before lossy transformation

  • OpenClaw can perform a memory-flush step before compaction, allowing durable notes to be written separately from the compacted conversation. Failure of this optional maintenance does not erase the conversation, and failure of required compaction preserves the original history rather than silently starting over.

  • This implements a useful separation between durable memory and active model context.

  • A robust autoresearch harness can apply the same pattern. Before transforming a large trace into a lossy summary, it should persist exact metrics, source snapshots, diffs, experiment identifiers, critical failures, and provenance. Compaction can then optimize the working representation without becoming the sole copy of the underlying evidence.

Compaction and pruning solve different problems

  • OpenClaw explicitly distinguishes compaction from pruning. Compaction creates a persistent summary of older conversation state, whereas pruning removes selected old tool results only from a particular model request and does not save those removals as a replacement summary.

  • In the theoretical taxonomy:

    • Pruning is primarily a selection operator.
    • Summarizing is primarily a generation operator.
    • Retaining a recent tail while summarizing the prefix is a hybrid policy.
    • Flushing durable memory before summarization separates archival retention from model-visible compaction.

  • These mechanisms should be represented as distinct search dimensions because they have different failure modes. Pruning risks removing the wrong evidence. Generation risks distorting or inventing state. Repeated generation risks accumulating lossy transformations.

Repeated compaction compounds information loss

  • Production agents generally compact the already compacted state rather than reconstructing each new compacted context from the immutable full history. Context Compaction Theory by Tirmazi et al. (2026) notes that later compactions therefore operate on summaries produced by earlier compactions, so each operation can discard additional information. The paper formalizes only a single compaction and identifies repeated compaction as an open theoretical problem.

  • For a sequence of compaction operators \(C_1,C_2,\ldots,C_T\), the state visible after repeated compaction becomes:

    \[\mathcal{I}_T^{c} = C_T \circ C_{T-1} \circ \cdots \circ C_1 \left( \mathcal{I}_0 \right)\]

  • Even when every individual transformation introduces small error, the relevant quantity for a long-horizon agent is \(\operatorname{err}_T(q)\), the probability or magnitude of error on a future query after \(T\) successive transformations. Characterizing how this quantity grows with compaction count remains open.

  • This reinforces the case for immutable raw archives, reversible retrieval, periodic reconstruction from high-fidelity evidence, or tests that explicitly evaluate behavior after multiple compaction events.

The information-theoretic optimum is not necessarily achievable by an LLM summarizer

  • The communication-complexity equivalence establishes the minimum amount of information that must survive, but it does not guarantee that a practical condenser can compute an optimal representation or that an LLM can reliably decode one. The paper therefore distinguishes information-theoretic attainability from computational attainability.

  • This gives three separate questions for harness evaluation:

\[\begin{aligned} \text{Is the budget sufficient in principle?} \\ \text{Can the condenser construct a sufficient representation?} \\ \text{Can the downstream model reliably interpret it?} \end{aligned}\]
  • A failure at the first level is fundamental. A failure at the second or third level is a harness or model-interface problem and may therefore be improvable through Meta-Harness search.

Measuring a deployed condenser against the theoretical frontier

  • Context Compaction Theory by Tirmazi et al. (2026) demonstrates how the communication formulation can be used as an empirical diagnostic. In a case study, the authors give Anthropic’s context compaction endpoint a set of 15,000.

  • URLs and explicitly state that the compacted representation will later be used only for membership queries. The resulting natural-language summaries are approximately:

    \[14 \text{ Kbits}\]

    • and produce total membership-query error rates of:

      \[0.505,\quad 0.535,\quad 0.555\]
      • across three seeds. A control retaining the full uncompressed context produces an error rate of 0.02.

  • The experiment therefore attributes the large degradation to information lost during compaction rather than to inability of the downstream model to answer membership queries from the full data.

  • A Bloom filter with the same budget would have substantially lower error because it is specifically designed to preserve membership information. The comparison illustrates that a fluent natural-language summary can be a poor encoding when the downstream task depends on exact, high-cardinality state.

  • The following figure (source) shows membership-query error as a function of compaction budget. The deployed natural-language condenser lies near the random-guess line in this workload, while a same-budget Bloom filter and the information-theoretic lower bound achieve substantially lower error.

  • This case study should not be generalized to all compaction workloads or provider implementations. The paper explicitly presents it as one endpoint, one model configuration, and one workload at a particular point in time. Its broader contribution is the evaluation methodology: when the query class has a known information-theoretic optimum, a deployed condenser can be measured against that frontier rather than compared only with another heuristic summarizer.

Synthesis: context compaction is an optimizable harness policy

  • Together, these results suggest that context compaction should be modeled as a first-class component of the agent harness rather than an emergency summarization step invoked only when the prompt becomes too long.

  • Context Compaction Theory by Tirmazi et al. (2026) provides the information-theoretic framework for determining what a compacted state must preserve. OpenHands Context Condensensation for More Efficient AI Agents demonstrates that thresholded condensation can materially improve long-horizon cost and latency while preserving software-engineering task quality on its tested subset. Compaction in OpenClaw illustrates the operational machinery needed around the condenser, including recent-tail preservation, structural boundaries, validation, memory flushing, provider checkpoints, and failure-safe state transitions.

  • A complete context-compaction policy can therefore be represented as:

    \[\mathcal{C} = \left( G, B, T, R, V, P \right)\]

    • where:

      • Granularity: \(G\) determines whether the harness reasons over tokens, messages, tool results, semantic items, or larger trajectory units.

      • Budget: \(B\) determines the maximum model-visible representation.

      • Trigger: \(T\) determines when compaction runs.

      • Representation: \(R\) determines whether information is selected, generated, or represented through a hybrid.

      • Validation: \(V\) determines which invariants must survive before the compacted state can be committed.

      • Persistence: \(P\) determines which raw evidence remains recoverable outside the active context.

  • Meta-Harness can then search over:

    \[\mathcal{C}^{\star} = \arg\max_{\mathcal{C}} \; \mathbb{E} \left[ U(\mathcal{C}) \right]\]

    • subject to:

      \[\mathrm{ContextTokens} \leq B_{\max}\] \[\mathrm{Latency} \leq L_{\max}\] \[\mathrm{Cost} \leq C_{\max}\]

  • The best policy is not necessarily the one producing the shortest summary. It is the one that preserves the information required by future decisions while satisfying the system’s context, latency, and cost constraints.

The search loop

  • The Meta-Harness loop has three core actions: propose, evaluate, and log. A coding-agent proposer reads prior experience from disk, proposes a new harness, the evaluator runs the harness on search tasks, and the resulting code, metrics, prompts, tool calls, model outputs, and traces are written back to disk for future iterations.

  • A minimal implementation looks like:

def search_harnesses(seed_harnesses, proposer, model, search_tasks, n_iters):
    archive = FilesystemArchive()
    population = []

    for harness in seed_harnesses:
        result = evaluate(harness, model, search_tasks)
        archive.write_run(harness=harness, result=result)
        population.append(harness)

    for step in range(n_iters):
        proposal = proposer.propose(
            archive_path=archive.root,
            instructions="Inspect prior code, metrics, and traces before editing.",
        )

        if not validate_interface(proposal):
            archive.write_invalid(proposal)
            continue

        result = evaluate(proposal, model, search_tasks)
        archive.write_run(harness=proposal, result=result)
        population.append(proposal)

    return pareto_frontier(archive.results)
  • The outer loop is deliberately simple. The proposer is not limited to a hand-written mutation operator, a fixed parent-selection rule, or a compressed prompt of previous scores. It can inspect anything in the archive and decide whether to make a local edit, revert a failed idea, combine two prior candidates, or rewrite the harness structure.

Filesystem as experience memory

  • The filesystem is the main implementation trick. Instead of forcing the proposer to consume all prior experience in one context window, the archive exposes experience as searchable files. A good run directory should contain:
runs/
  042/
    harness.py
    diff.patch
    metrics.json
    score.txt
    prompts.jsonl
    model_outputs.jsonl
    tool_calls.jsonl
    state_updates.jsonl
    failures.jsonl
    notes.md

  • This design lets the proposer perform selective credit assignment. It can search for repeated parse failures, compare the best and worst candidates, inspect only runs that improved one metric, or trace why a specific task failed. The key difference from ordinary prompt optimization is that the feedback channel is not a single scalar or short summary; it is a complete empirical record.

  • This matters because harness behavior often has long-range dependencies. A memory-update change at step \(2\) may only affect the final answer at step \(15\). A retrieval format change may help one task family and hurt another. A parser change may increase measured accuracy by reducing invalid outputs, even if reasoning quality is unchanged. These effects are difficult to diagnose from aggregate scores alone.

What gets optimized

  • Harness code can optimize many system components at once, as delineated below:

    • Prompt construction:

      • Prompt construction decides which instructions, examples, retrieved passages, intermediate summaries, and output schemas are shown to the model. A candidate prompt builder might choose different templates by task type:

      python id="vxu44o" def build_prompt(task, memory, retrieved, mode): if mode == "classification": return render_classification_prompt(task, memory.examples) if mode == "math": return render_math_prompt(task, retrieved.proofs) if mode == "coding": return render_agent_prompt(task, memory.repo_summary, memory.errors) raise ValueError(mode)

      • Useful search dimensions include instruction order, schema strictness, example count, chain-of-thought visibility policy, error-recovery text, and whether to include successful or failed prior attempts.

    • Retrieval policy:

      • Retrieval policy decides what external information to fetch and how to format it. For a query \(q\) and document set \(D\), a sparse retrieval harness might rank candidates by BM25:
      \[\mathrm{score}(q,d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{ f(t,d)(k_1 + 1) }{ f(t,d) + k_1(1 - b + b \cdot |d|/\mathrm{avgdl}) }\]
      • The harness can search over query rewriting, top-\(k\), diversity filters, deduplication, answer-aware reranking, proof-pattern retrieval, and context formatting. In retrieval-augmented reasoning, the important question is not whether retrieval exists, but whether the harness retrieves examples that are structurally useful for the target problem.

    • Memory update:

      • Memory update decides what persists across steps:
      \[\mathcal{M}_{t+1} = U(\mathcal{M}_t, s_t, y_t, a_t, o_t)\]
      • where \(o_t\) is a tool observation. Search can modify whether memory stores raw transcripts, compressed summaries, symbolic state, failed attempts, tool errors, retrieved examples, or confidence estimates. In long-horizon agents, memory design is often as important as the initial prompt because later decisions depend on what was preserved.

    • Tool orchestration:

      • Tool orchestration decides which tools are available, when to call them, and how observations are fed back into the model. For coding agents, this includes shell commands, file reads, test execution, patch application, and submission logic. Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces by Merrill et al. (2026) is relevant because it evaluates agents on long-horizon terminal tasks where harness decisions around tools, state, and recovery strongly affect pass rate.

    • Output parsing and repair:

      • Parsing is often underappreciated. A model can reason correctly but fail the benchmark because the answer is not emitted in the expected format. A harness can search over strict JSON schemas, regex extraction, fallback parsers, self-repair calls, and validation loops:
      def parse_with_repair(raw_output, schema, model):
    parsed = try_parse(raw_output, schema)
    if parsed.ok:
        return parsed.value

    repair_prompt = build_repair_prompt(raw_output, schema, parsed.error)
    repaired = model(repair_prompt)
    return try_parse(repaired, schema).value
      • The evaluator should log parse errors separately from reasoning errors, because these imply different edits.

Evaluation design

  • Harness evaluation should produce both aggregate metrics and task-level traces. A classification harness might report accuracy and context tokens. A retrieval reasoning harness might report pass@1, retrieved-document overlap, parse-error rate, and model-call count. An agentic coding harness might report pass rate, test failures, timeouts, number of tool calls, wall-clock time, and final submission status.

  • For a task set \(X_{\mathrm{search}} = \{x_1,\dots,x_n\}\), a simple score is:

\[\hat{R}(H) = \frac{1}{n} \sum_{i=1}^{n} r(\tau_i, x_i)\]
  • For stochastic models, use repeated samples:
\[\hat{R}_K(H) = \frac{1}{nK} \sum_{i=1}^{n} \sum_{k=1}^{K} r(\tau_{i,k}, x_i)\]
  • For cost-aware harnesses, use a constrained or scalarized objective:
\[J(H) = -\hat{R}(H) + \lambda_c \cdot \mathrm{ContextTokens}(H) + \lambda_m \cdot \mathrm{ModelCalls}(H) + \lambda_t \cdot \mathrm{Latency}(H)\]
  • or maintain a Pareto frontier instead of collapsing everything into one scalar. Pareto tracking is usually safer because the desired tradeoff between quality and cost may change later.

Search-set and test-set separation

  • The search set is the set of tasks used during harness evolution. The test set is held out until final evaluation. The proposer should never see test results during search. Otherwise, harness search can overfit just like hyperparameter tuning can overfit a validation set.

  • A clean protocol is:

train/search tasks:
  used repeatedly during outer-loop search

development audits:
  used occasionally by humans to inspect failure modes

held-out test tasks:
  used once for final reporting

contamination checks:
  used to detect leakage, hard-coded labels, or task-specific shortcuts
  • For public discovery benchmarks, where repeated benchmark-specific iteration is part of the setting, the system should still audit for explicit task leakage, brittle string matching, and hard-coded answer paths.

Trace schema

  • A useful trace schema for harness optimization is:
{
  "run_id": "057",
  "task_id": "math_0183",
  "harness_hash": "abc123",
  "model": "fixed-base-model",
  "steps": [
    {
      "t": 0,
      "state_summary": "...",
      "prompt_path": "prompts/000.txt",
      "model_output_path": "outputs/000.txt",
      "tool_call": null,
      "observation_path": null,
      "parser_status": "ok"
    }
  ],
  "final_answer": "...",
  "reward": 1,
  "context_tokens": 18422,
  "model_calls": 3,
  "latency_seconds": 41.2,
  "failure_type": null
}
  • This schema supports both automated analysis and agent inspection. The proposer can search for parser_status: failed, tasks with high context and low reward, or tool calls that repeatedly precede failures.

Credit assignment over harness edits

  • Credit assignment is hard because one harness edit may affect many downstream behaviors. Suppose a new harness improves reward from \(0.42\) to \(0.48\). The improvement might come from better retrieval, stricter parsing, more examples, or a subtle interaction between memory and prompt format. The proposer should therefore compare candidate traces, not only candidate scores.

  • A practical approach is to decompose each result by failure type:

\[\mathrm{ErrorRate}(H) = \mathrm{ReasoningError}(H) + \mathrm{RetrievalError}(H) + \mathrm{ParseError}(H) + \mathrm{ToolError}(H) + \mathrm{TimeoutError}(H)\]
  • This decomposition tells the next proposal where to intervene. If parse errors dominate, do not rewrite retrieval. If retrieval errors dominate, do not tune the final answer parser. If timeouts dominate, reduce tool loops or context size.

Harness search patterns

  • Local repair:

    • Local repair modifies a single component, such as output parsing or retry logic. It is low risk and often useful after a regression.
    Observation:
Many failed tasks contain the correct answer but invalid formatting.

Edit:
Add strict extraction and one repair call.

Expected effect:
Reduce parse-error rate without changing reasoning behavior.

  • Component swap:

    • Component swap replaces one subsystem while preserving the rest of the harness.
    Observation:
Dense retrieval hurts some reasoning tasks, while sparse retrieval is more stable.

Edit:
Keep sparse retrieval, but rewrite query generation and top-k filtering.

Expected effect:
Improve retrieved-example relevance without changing model-call budget.

  • Additive fallback:

    • Additive fallback preserves the current best behavior and adds a recovery path only when the default path fails.
    answer = primary_solver(task)
if not verifier.accepts(answer):
    answer = fallback_solver(task, previous_answer=answer)
return answer
    • This is often safer than rewriting the primary path.

  • Full rewrite:

    • Full rewrites are useful when traces show the current architecture is structurally misaligned with the task. They are risky and should be isolated into separate branches so that the system can recover if they regress.

Meta-Harness-style proposer instructions

  • A strong proposer instruction for harness search should emphasize evidence before editing:
Before proposing a new harness:
1. Inspect the current Pareto frontier.
2. Compare at least one improved run and one regression.
3. Read task-level traces for representative failures.
4. Identify the dominant failure mode.
5. Make one coherent edit that targets that failure mode.
6. Preserve evaluator interfaces and logging.
7. Record the hypothesis, diff summary, expected metric movement, and risks.
  • The point is not to make the agent verbose. The point is to force the search loop to behave like experimental science: observe, hypothesize, intervene, measure, and update.

Why full-history access beats summaries

  • Summaries are useful, but they are lossy. A summary might say “retrieval was noisy,” while the raw traces reveal that retrieval was good but the formatting caused the model to ignore the examples. A summary might say “tool use increased,” while raw logs reveal that one specific command pattern caused most timeouts. Full-history access lets the proposer form sharper causal hypotheses.

  • This is the same reason autoresearch logs full training traces rather than only the best BPB. A 0.002 BPB improvement means little without knowing whether it came from faster throughput, a true modeling gain, a changed validation path, or noise.

  • Full-history access should therefore mean archival availability rather than permanent prompt residency. The harness should preserve the raw empirical record even when it constructs a smaller working context for a particular model call, so compaction reduces inference cost without destroying the evidence needed for later retrieval, audit, or reinterpretation.

Implementation checklist

  • A practical harness-optimization setup should include:

    • Harness interface: a stable function signature such as run_task(task, model, tools, config) -> Result.
    • Evaluator: deterministic task loading, metric computation, budget enforcement, and result serialization.
    • Trace logger: prompts, outputs, tool calls, state updates, parser status, and final reward.
    • Archive: one immutable directory per candidate.
    • Proposer sandbox: read access to all prior runs, write access only to new candidate files.
    • Validation gates: import checks, interface checks, budget checks, parser checks, and small smoke tasks.
    • Selection logic: leaderboard plus Pareto frontier.
    • Leakage audit: scan candidate code for task IDs, answer strings, benchmark-specific shortcuts, and forbidden file access.
    • Final evaluation: run only frontier candidates on held-out tasks.

  • This turns harness engineering into a repeatable autoresearch loop: the agent searches over executable model scaffolding, the evaluator supplies grounded feedback, and the filesystem preserves enough experience for the next proposal to be better than a blind mutation.

Implementing the single-GPU research substrate

  • The simplest useful autoresearch substrate is a bounded, single-GPU language-model training environment where the agent is allowed to edit the model and training loop, but not the data-preparation or evaluation contract. The goal is to make experiments cheap, comparable, and inspectable, so the agent can run many trials and accumulate evidence instead of making one large, hard-to-debug change. karpathy/autoresearch is a compact implementation of this pattern: an agent edits the training file, runs a fixed-budget experiment, checks validation bits per byte, and repeats.

Repository boundary design

  • A good substrate begins by drawing a hard boundary between stable infrastructure and editable research code. The stable layer should own data download, tokenizer construction, validation split creation, dataloading, evaluation, metric calculation, and result serialization. The editable layer should own the model architecture, optimizer, schedule, precision choices, batch sizing, and training-loop mechanics. This boundary matters because an autonomous agent should be able to improve training, but should not be able to accidentally improve the score by changing what “validation” means.

  • A minimal layout looks like:

project/
  prepare.py        # fixed data prep, tokenizer, dataloaders, validation metric
  train.py          # editable model, optimizer, training loop
  program.md        # agent instructions and research protocol
  results.tsv       # append-only experiment ledger
  runs/
    000_seed/
    001_candidate/
    002_candidate/
  • The single editable file is not just a convenience. It is a search-space regularizer. If the agent can edit every file, the experiment becomes harder to audit and easier to corrupt. If the agent can edit only one coherent training file, every candidate can be reviewed as a single diff. The small-codebase design is similar in spirit to nanochat, which is designed as a minimal single-node LLM training harness covering tokenization, pretraining, finetuning, evaluation, inference, and chat UI.

Fixed budget as the experimental contract

  • The fixed budget is the main comparability device. If every candidate trains for exactly \(B\) wall-clock seconds, then the outer loop rewards changes that improve quality under the same resource limit:

    \[c^{*} = \arg\min_{c \in \mathcal{C}} \mathrm{BPB} \left( \mathrm{Train}(c; B) \right)\]
    • where \(c\) is the code candidate and \(B\) is the fixed training budget. In practice, this budget should exclude one-time startup or compilation when those are not part of the research question, but include normal training overhead once the run begins. The single-GPU setup uses a fixed 5-minute training run so the agent can run many comparable experiments overnight, roughly turning a workstation into a small autonomous research lab.

  • This fixed-budget framing changes the meaning of “better.” A larger model may have better asymptotic performance but worse fixed-budget BPB because it sees fewer updates. A smaller model may win because it trains more tokens per second. The system is therefore optimizing the joint interaction of architecture, optimizer, batch size, sequence length, compiler behavior, and hardware utilization, not only model expressivity.

Validation bits per byte

  • Validation bits per byte is a useful autoresearch metric because it is closer to a compression-normalized score than raw token-level loss. If the model reports average negative log-likelihood in nats over validation tokens, the conversion is:
\[\mathrm{bits} =\frac{\mathcal{L}_{\mathrm{nats}}}{\ln 2}\]
  • If validation contains \(N_{\mathrm{bytes}}\) bytes and \(N_{\mathrm{tokens}}\) tokens, a practical BPB estimate is:
\[\mathrm{BPB} =\frac{ \sum_{i=1}^{N_{\mathrm{tokens}}} -\log_2 p_{\theta}(x_i \mid x_{<i}) }{ N_{\mathrm{bytes}} }\]
  • Lower BPB is better. The reason BPB is preferable to raw token loss in a setting where the tokenizer may change is that token loss depends strongly on how many tokens the tokenizer emits. BPB anchors the score to bytes, making comparisons more meaningful across vocabulary-size and tokenization changes.

Training objective

  • The inner-loop training objective is standard autoregressive language modeling. For a sequence \(x_1,\dots,x_T\), the model minimizes:
\[\mathcal{L}_{\mathrm{LM}}(\theta) =-\frac{1}{T} \sum_{t=1}^{T} \log p_{\theta}(x_t \mid x_{<t})\]
  • The outer loop does not replace gradient descent. It wraps around it. The model weights \(\theta\) are optimized by the usual trainer, while the agent searches over code candidates \(c\) that determine the architecture, optimizer, data flow, schedule, precision, and other training details:
\[\theta_T(c) = \mathrm{Train}(\theta_0, c, B)\] \[s(c) = \mathrm{BPB} \left( \theta_T(c) \right)\]
  • This separation is important. Autoresearch is not a new optimizer for model weights; it is an optimizer over the experimental program that produces trained weights.

Model architecture search knobs

  • The editable training file should expose a small number of high-leverage knobs, but the agent should still be free to rewrite their relationships. For a compact GPT-like model, the most important knobs are usually:

    • Depth: number of transformer blocks.
    • Width: embedding dimension and hidden dimension.
    • Attention heads: number of heads and head dimension.
    • Sequence length: maximum context length.
    • MLP ratio: expansion factor in the feed-forward network.
    • Normalization: pre-norm, post-norm, RMSNorm, LayerNorm, or variants.
    • Attention pattern: full attention, local attention, alternating local/global patterns, or sliding windows.
    • Parameter tying: tied input/output embeddings or separate matrices.

  • A practical agent instruction is to prefer changes that preserve tensor-shape clarity and throughput instrumentation. Many architecture edits look promising in isolation but silently reduce tokens per second enough to lose under the fixed budget.

  • The outer-loop comparison should therefore log both quality and speed:

    \[\Delta \mathrm{BPB} = \mathrm{BPB}_{\mathrm{candidate}} - \mathrm{BPB}_{\mathrm{parent}}\] \[\Delta \mathrm{TPS} = \mathrm{TPS}_{\mathrm{candidate}} - \mathrm{TPS}_{\mathrm{parent}}\]
    • where \(\mathrm{TPS}\) is tokens per second. A candidate with \(\Delta \mathrm{BPB} < 0\) and \(\Delta \mathrm{TPS} > 0\) is a clean win. A candidate with better BPB and worse throughput may still be valuable, but it should be marked as a tradeoff rather than blindly accepted.

Optimizer design

  • The optimizer is one of the highest-leverage editable components because the fixed-budget score rewards fast early learning. A typical baseline can combine AdamW for embeddings, normalization, and scalar parameters with a specialized optimizer for matrix-shaped hidden-layer weights. Decoupled Weight Decay Regularization by Loshchilov and Hutter (2017) is relevant because AdamW separates weight decay from the adaptive gradient step, making the weight-decay coefficient less entangled with the learning rate than ordinary \(L_2\) regularization in Adam.

  • AdamW can be written as:

    \[g_t = \nabla_{\theta} \mathcal{L}_t(\theta_t)\] \[m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t\] \[v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t^2\] \[\hat{m}_t = \frac{m_t}{1-\beta_1^t}, \quad \hat{v}_t = \frac{v_t}{1-\beta_2^t}\] \[\theta_{t+1} = \theta_t - \eta \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon} - \eta \lambda \theta_t\]
    • where \(\eta\) is the learning rate and \(\lambda\) is the decoupled weight-decay coefficient.

  • Muon-style optimizers are also relevant in this setting because they are designed for hidden-layer matrices and have been used in small-model training speedrun contexts. Muon is Scalable for LLM Training by Liu et al. (2025) studies techniques for scaling Muon to larger LLM training, and Muon: An optimizer for hidden layers in neural networks is a practical optimizer writeup describing Muon as an optimizer for hidden layers.

  • A simplified Muon-like update applies momentum and then orthogonalizes a matrix update direction:

\[M_t = \mu M_{t-1} + G_t\] \[O_t \approx \mathrm{Orthogonalize}(M_t)\] \[W_{t+1} = W_t - \eta O_t\]
  • The practical design pattern is to let the agent search optimizer partitioning rules:
def parameter_groups(model):
    matrix_params = []
    scalar_or_embedding_params = []

    for name, p in model.named_parameters():
        if p.ndim == 2 and "embed" not in name and "lm_head" not in name:
            matrix_params.append(p)
        else:
            scalar_or_embedding_params.append(p)

    return [
        {"params": matrix_params, "optimizer": "muon"},
        {"params": scalar_or_embedding_params, "optimizer": "adamw"},
    ]
  • The evaluator should log optimizer grouping, learning rates, weight decay, gradient norms, NaN events, and whether any parameter group received no gradients.

Batch-size and sequence-length tradeoffs

  • Batch size and sequence length define how much data each forward/backward pass processes. If \(B_d\) is device batch size and \(L\) is sequence length, then tokens per step are approximately:
\[N_{\mathrm{tokens/step}} = B_d \cdot L\]
  • With gradient accumulation \(A\), the total batch tokens per optimizer update are:
\[N_{\mathrm{tokens/update}} = A \cdot B_d \cdot L\]

  • Under a fixed wall-clock budget, increasing \(L\) can improve long-context modeling but reduce throughput. Increasing \(B_d\) can improve hardware utilization but may reduce update frequency or trigger out-of-memory failures. Increasing gradient accumulation can stabilize optimization but may reduce the number of optimizer updates within the budget.

  • A good autoresearch agent should not tune these independently. It should reason about the product \(A \cdot B_d \cdot L\), memory pressure, and update count:

\[N_{\mathrm{updates}} \approx \frac{ T_{\mathrm{budget}} \cdot \mathrm{tokens/sec} }{ A \cdot B_d \cdot L }\]
  • The fixed-budget objective often rewards configurations that maintain enough updates for fast learning while keeping throughput high.

Learning-rate schedule

  • A learning-rate schedule is another high-impact edit. A standard warmup plus cosine decay schedule is:
\[\eta_t = \begin{cases} \eta_{\max}\frac{t}{T_{\mathrm{warmup}}}, & t < T_{\mathrm{warmup}} \\ \eta_{\min} + \frac{1}{2} (\eta_{\max}-\eta_{\min}) \left( 1 + \cos \left( \pi \frac{t-T_{\mathrm{warmup}}}{T_{\mathrm{total}}-T_{\mathrm{warmup}}} \right) \right), & t \ge T_{\mathrm{warmup}} \end{cases}\]

  • For autoresearch, the schedule should be parameterized by time budget or estimated total steps, not by a fixed epoch count. Otherwise, architecture or throughput changes can accidentally make the schedule too short or too long.

  • A robust schedule API is:

```python id=”olpaid” def get_lr(step, total_steps, warmup_frac, max_lr, min_lr): warmup_steps = int(warmup_frac * total_steps)

if step < warmup_steps:
    return max_lr * step / max(1, warmup_steps)

progress = (step - warmup_steps) / max(1, total_steps - warmup_steps)
cosine = 0.5 * (1.0 + math.cos(math.pi * progress))
return min_lr + cosine * (max_lr - min_lr) ```
  • The agent can then search over warmup_frac, max_lr, min_lr, and schedule shape without breaking comparability.

Precision and numerical stability

  • A single-GPU autoresearch loop should include explicit numerical-stability instrumentation because agents will try aggressive learning rates, optimizer variants, and batch-size changes. At minimum, every run should detect non-finite loss, non-finite gradients, and exploding norms:
if not torch.isfinite(loss):
    raise RuntimeError(f"non-finite loss at step {step}: {loss.item()}")

total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm)
if not torch.isfinite(total_norm):
    raise RuntimeError(f"non-finite grad norm at step {step}: {total_norm}")
  • The results table should distinguish instability from ordinary underperformance:
status:
  evaluated
  nan_loss
  nan_grad
  out_of_memory
  timeout
  import_error
  syntax_error
  metric_missing
  • This classification helps the next proposal. A NaN run suggests reducing learning rate, adding gradient clipping, changing initialization, or adjusting precision. A slow but stable run suggests optimizing throughput. A high-throughput but poor-BPB run suggests insufficient capacity or too few effective updates.

Result ledger

  • The result ledger should be append-only. It is both the memory of the research loop and the audit trail for humans. A useful results.tsv schema is:
run_id
parent_id
timestamp
status
val_bpb
train_loss
val_loss
tokens_per_sec
train_tokens
steps
params
depth
seq_len
device_batch_size
grad_accum
total_batch_tokens
optimizer_summary
max_lr
weight_decay
peak_mem_gb
diff_summary
notes_path
  • A candidate should never overwrite a previous result. Even failed runs are valuable because they define the boundary of the stable search space.

Candidate snapshotting

  • Every run should preserve the exact code that produced it. The easiest pattern is to copy the editable file into the run directory and store a patch against the parent:
runs/017/
  train.py
  diff.patch
  metrics.json
  stdout.log
  stderr.log
  notes.md
  • The agent should be instructed to inspect both successful and failed snapshots. Improvements often come from transplanting a subcomponent from a worse overall candidate, such as a faster data layout or safer initialization.

Agent prompt for single-GPU research

  • A useful program.md style instruction should be operational, not philosophical:
You are optimizing validation BPB under a fixed 5-minute training budget.

Allowed:
- Edit train.py.
- Change architecture, optimizer, schedule, batch sizing, precision, and training loop.
- Run smoke tests and full experiments.
- Inspect all previous runs.

Forbidden:
- Do not edit data preparation, validation data, tokenizer accounting, metric computation, or result parser.
- Do not compare partial runs to full runs.
- Do not keep NaN-producing candidates.
- Do not delete old run logs.

Before each edit:
- Inspect the current best run.
- Inspect at least one recent regression.
- State a hypothesis.
- Make one coherent change.

After each run:
- Record the metric, throughput, memory, status, diff summary, and interpretation.
  • This instruction layer is the closest thing to “research management.” It defines the norms of the autonomous lab.

Guardrails against false progress

  • The system should reject or flag any run that violates the experimental contract. Common false-progress modes include changing validation data, reducing evaluation tokens, changing byte accounting, accidentally reading stale metrics, skipping hard batches, silently shortening training, or comparing a partial run to a full run.

  • A simple metric-integrity check can hash the frozen files:

\[h_{\mathrm{eval}} = \mathrm{SHA256} ( \texttt{prepare.py} \parallel \texttt{validation\_manifest} \parallel \texttt{metric\_code} )\]
  • Then every run records:
{
  "eval_hash": "abc123",
  "expected_eval_hash": "abc123",
  "metric_integrity_ok": true
}
  • If the hashes differ, the run should be excluded from the leaderboard.

Human review interface

  • Even in autonomous mode, the system should make human review easy. The best interface is a compact progress plot plus a sortable leaderboard. Each improvement should link to its run directory, diff, logs, and notes. The progress plot should distinguish kept candidates from discarded candidates so a reviewer can see whether the agent is actually climbing or just sampling randomly.

  • A minimal review checklist is:

    • Does the best candidate preserve the evaluator?
    • Is the improvement larger than run-to-run noise?
    • Did throughput or train-token count change substantially?
    • Did parameter count change?
    • Did the model actually train for the full budget?
    • Are there hidden warnings in stderr?
    • Does the diff look like a general method rather than a metric hack?

  • The goal is not to remove humans. The goal is to move humans from manually editing every experiment to supervising an autonomous experimental process.

Designing the autonomous research agent

  • The agent is the part of an autoresearch system that turns stored evidence into the next experiment. It is not merely a code generator. It is a research operator that reads prior runs, diagnoses failures, proposes hypotheses, edits the allowed artifact, runs checks, interprets metrics, and leaves behind evidence for the next iteration.

Agent responsibilities

  • A useful autonomous research agent has five responsibilities: inspect, hypothesize, edit, evaluate, and explain. During inspection, it should examine the current best run, recent regressions, failure logs, and diffs. During hypothesis formation, it should identify a mechanism rather than only a desired outcome. During editing, it should make a change narrow enough to support credit assignment. During evaluation, it should obey the fixed budget and preserve the metric contract. During explanation, it should write enough context for later agents or humans to understand why the run happened.

  • This is the operational distinction between an agent and a sampler. A sampler proposes candidates from a prompt. A research agent maintains a working memory of the experiment history and uses tools to interrogate that history. Reflexion: Language Agents with Verbal Reinforcement Learning by Shinn et al. (2023) is relevant because it shows how agents can improve across trials by storing language feedback in memory instead of updating model weights.

  • A practical agent loop should include the following steps:

    • Read the leaderboard before editing: The agent should identify the current best candidate, the best score, the most recent candidates, and the main performance trend before deciding what to change.
    • Inspect the current best code: The agent should read the exact source snapshot that produced the current best run so that it does not accidentally regress a useful implementation detail.
    • Inspect recent regressions: The agent should examine at least one or two failed or underperforming candidates to understand which edit families recently caused crashes, slowdowns, NaNs, or worse validation metrics.
    • Identify the dominant opportunity or failure mode: The next edit should target a specific observed pattern, such as unstable optimization, poor throughput, parser failures, excessive context use, or repeated timeouts.
    • Write a hypothesis before changing code: The agent should record what mechanism it expects to improve and what evidence would falsify that expectation.
    • Make one coherent edit: The candidate should test a single interpretable idea, or a tightly coupled group of changes, so the result can be attributed to the intervention.
    • Run validation gates before the full experiment: The agent should run syntax checks, import checks, smoke tests, and metric-path checks before spending the full budget.
    • Run the bounded experiment: The candidate should be evaluated under the same time, data, and metric constraints as previous candidates.
    • Write metrics and notes: The agent should append structured metrics and a human-readable postmortem to the archive.
    • Decide whether to keep, revert, branch, or revisit: The agent should preserve the best candidate, reject invalid candidates, branch from promising but incomplete ideas, and mark useful regressions for future transplanting.

Research memory

  • The agent’s memory should have two layers: structured metrics and raw evidence. Structured metrics let the agent sort, filter, and compare runs quickly. Raw evidence lets it diagnose why a run behaved the way it did.

  • For single-GPU training, the memory should include the following artifacts:

    • An append-only results ledger: The ledger should store run IDs, parent IDs, status, validation BPB, throughput, memory use, parameter count, edit family, and notes path so the agent can compare experiments quickly.
    • A source snapshot for each run: Each run should preserve the exact train.py or editable artifact that produced the result, making it possible to reproduce and inspect candidates later.
    • A patch against the parent run: Each candidate should store a diff that shows exactly what changed relative to its parent, which supports credit assignment and human audit.
    • Structured metrics: Each run should save metrics such as validation loss, validation BPB, train loss, tokens per second, number of steps, peak memory, and crash status in a machine-readable file.
    • Raw stdout and stderr logs: Console logs should be retained because they often contain warnings, compilation behavior, CUDA errors, NaN reports, dataloader issues, or other signals that are not captured in scalar metrics.
    • Human-readable notes: Each run should include a short postmortem describing the hypothesis, change, result, interpretation, and next step.
    • Training curves or per-step traces: Curves are useful for distinguishing candidates that improve early learning from candidates that merely happen to finish with a slightly better final metric.

  • For harness optimization, the memory should also include prompts, model outputs, tool calls, parser failures, state updates, per-task rewards, and context-token usage. The key pattern is that the archive must be larger than any single prompt and still be navigable by the agent. This is why filesystem access is so useful: it lets the proposer selectively inspect the parts of history that matter for the next proposal rather than relying on a compressed summary.

  • The active prompt should therefore be viewed as a bounded projection of research memory rather than as research memory itself. Context compaction determines that projection, while retrieval provides a path for archived evidence to become model-visible again when a later hypothesis makes it relevant.

  • The memory update after each run can be modeled as:

    \[\mathcal{D}_{t+1} = \mathcal{D}_{t} \cup \{c_t, \Delta_t, m_t, \ell_t, n_t\}\]
    • where \(c_t\) is the candidate code, \(\Delta_t\) is the diff, \(m_t\) is the metric record, \(\ell_t\) is the raw log bundle, and \(n_t\) is the note written after the experiment.

Hypothesis quality

  • A good hypothesis is mechanistic and falsifiable. “Try a smaller model” is weak. “Reducing depth from \(8\) to \(6\) may improve fixed-budget BPB because throughput and update count will increase more than per-token modeling capacity decreases” is stronger.

  • A useful hypothesis should contain the following parts:

    • Observation: The agent should state the specific metric, trace pattern, or failure mode that motivated the experiment.
    • Proposed cause: The agent should explain the mechanism it believes is responsible for the observed pattern.
    • Intervention: The agent should describe the single coherent change it will make to test that mechanism.
    • Expected movement: The agent should state which metrics should improve or worsen if the hypothesis is correct.
    • Risk: The agent should record the most likely regression mode, such as underfitting, slower throughput, instability, memory pressure, or parser fragility.

  • For example, a strong hypothesis would say that recent deeper models improved early train loss but lost validation BPB under the fixed 5-minute budget; the likely cause is that they are update-limited and see fewer optimizer steps; the intervention is to reduce depth while modestly increasing batch size; the expected result is higher throughput and lower validation BPB; and the main risk is that the smaller model may become capacity-limited and underfit.

  • This format makes later credit assignment easier. If the candidate improves throughput but worsens BPB, the hypothesis was only partly right. If it improves BPB without improving throughput, the mechanism was probably wrong even though the result was good.

Tool use policy

  • An autoresearch agent should use tools in a predictable order. Before editing, it should read. Before a full experiment, it should run a cheap check. Before accepting a result, it should verify metric integrity.

  • A robust tool policy should include the following behaviors:

    • Before editing, read the empirical state of the search: The agent should inspect the results ledger, the current best source snapshot, recent diffs, and logs from recent regressions before proposing a new candidate.
    • Before editing, search for repeated failures: The agent should scan prior logs for crashes, NaNs, out-of-memory errors, metric warnings, parser failures, timeouts, and stale-result risks.
    • Before a full run, execute cheap validation gates: The agent should run syntax checks, import checks, interface checks, and a short smoke-training or smoke-evaluation pass before launching the full experiment.
    • Before accepting metrics, verify freshness: The agent should ensure that the metric file was produced by the current run ID and not inherited from a stale or partially failed run.
    • After a full run, parse and validate the metrics: The agent should check the metric schema, compare evaluator hashes, classify the run status, and detect NaN, crash, timeout, or integrity failures.
    • After a full run, preserve the full evidence bundle: The agent should append the ledger, copy the source snapshot, save raw logs, write structured metrics, and record a postmortem.
    • Before lossy compaction, checkpoint decision-critical state: The agent should persist experiment IDs, current hypotheses, unresolved failures, exact identifiers, important tool outputs, and links to raw evidence before rewriting the active context. Tool calls and their corresponding results should remain logically paired so that compaction does not separate an action from the evidence needed to interpret it.

  • This differs from pure reasoning-search methods such as Tree of Thoughts: Deliberate Problem Solving with Large Language Models by Yao et al. (2023), which expands and evaluates intermediate reasoning paths; autoresearch expands executable research states and evaluates them through real experiments.

Agent roles

  • A single agent can run the whole loop, but larger autoresearch systems benefit from role separation. Roles are not personalities; they are permissions and evaluation contracts.

    • Researcher: The researcher proposes code changes. It has read access to all prior runs and write access to candidate code. It should be optimized for hypothesis generation, implementation, and interpretation.
    • Evaluator: The evaluator runs experiments and computes metrics. It should be deterministic and conservative. Ideally, it should not be able to edit the candidate except through a clean checkout mechanism.
    • Auditor: The auditor checks for metric drift, leakage, suspicious diffs, stale result files, and hidden failures. It should compare evaluator hashes, inspect changed files, and reject invalid candidates.
    • Curator: The curator maintains the frontier, summarizes families of experiments, and decides which branches deserve more budget.

  • This role split prevents one agent from unconsciously optimizing the score by weakening the evaluator. It also makes the research organization scalable: many researchers can propose candidates, one evaluator can score them, and one curator can maintain a clean frontier.

Permission boundaries

  • Permissions are part of the algorithm. The agent should have enough freedom to discover non-obvious improvements, but not enough freedom to invalidate the experiment.

  • A practical permission map should use the following boundaries:

    • Editable files: The agent may edit the candidate artifact, such as train.py, harness.py, or candidate-local configuration files that affect only the proposed run.
    • Read-only files: The agent may inspect but not modify data preparation, validation data, metric computation, tokenizer accounting, benchmark task files, and previous run directories.
    • Append-only files: The agent may append to results ledgers, run notes, and leaderboard history, but should not rewrite prior records.
    • Forbidden actions: The agent should not delete old runs, edit held-out test results, alter metric parsers, weaken time-budget enforcement, or change validation data.

  • In the compact training setup, the intended pattern is that data preparation and runtime utilities remain fixed while the agent edits the single training file and the human iterates on the instruction file.

Planning depth

  • The agent should plan enough to avoid random edits, but not so much that it spends the whole budget reasoning instead of running experiments. A useful default is shallow planning for routine edits and deeper planning for structural rewrites.

  • For local edits, the agent should inspect a small number of highly relevant runs, identify one hypothesis, change one component, and run one experiment. For structural rewrites, the agent should inspect the frontier, compare several regressions, write a short design note, run stronger smoke tests, and only then launch a full experiment.

  • Planning can be framed as a value-of-information problem. The agent should spend more time inspecting history when the expected cost of a bad experiment is high:

\[\mathrm{VOI} = \mathbb{E} [ U(a \mid \mathrm{extra\ evidence}) - U(a \mid \mathrm{current\ evidence}) ] - C_{\mathrm{inspection}}\]
  • If inspection is cheap and full runs are expensive, deeper diagnosis is worthwhile. If runs are cheap and logs are simple, faster experimentation may dominate.

Exploration and exploitation

  • Autoresearch needs both exploitation and exploration. Exploitation refines the current best candidate. Exploration tries different mechanisms that may initially underperform but reveal useful ideas.

  • A simple controller can allocate experiments as:

    \[P(\mathrm{explore}) = \max \left( p_{\min}, p_0 \cdot e^{-t / \tau} \right)\]
    • where \(p_0\) is the initial exploration rate, \(p_{\min}\) is the minimum long-run exploration rate, and \(\tau\) controls annealing.

  • A more agentic version assigns each candidate a novelty score and a quality score:

\[S(c) = -\mathrm{BPB}(c) + \lambda_n \cdot \mathrm{Novelty}(c) - \lambda_r \cdot \mathrm{Risk}(c)\]
  • Novelty can be estimated from edit family, architecture family, optimizer family, or distance from previous diffs. Risk can be estimated from crash history, memory pressure, number of changed lines, or whether the edit touches fragile code.

Skill libraries

  • As the agent discovers useful procedures, it should turn them into reusable skills. A skill is not necessarily code used by the final model; it can be a research maneuver. Examples include “bisect a NaN regression,” “compare throughput-normalized candidates,” “audit evaluator integrity,” “transplant only the scheduler from a candidate,” or “cluster failures by parser status.”

  • Voyager: An Open-Ended Embodied Agent with Large Language Models by Wang et al. (2023) is relevant because it uses an ever-growing library of executable skills to support lifelong exploration and reuse across tasks.

  • A research skill should specify when to use it, what evidence it needs, and how to verify that it worked. For example, a component-transplant skill should be used when a candidate is worse overall but contains one useful subsystem. The agent should identify the subsystem and its local dependencies, copy only that subsystem into the current best candidate, preserve all unrelated behavior, run smoke tests, run the full budget, and compare the new result against both the current best and the donor candidate.

  • This prevents useful ideas from being discarded just because they appeared inside a bad candidate.

Communication between agents

  • When multiple agents run in parallel, they need a shared protocol. Without one, they will duplicate work, overwrite each other, or interpret metrics inconsistently.

  • A simple shared protocol should include the following stages:

    • Claiming work: An agent should reserve a candidate ID, identify the intended parent run, and write the edit family it plans to explore.
    • Running evaluation: The evaluator should mark the candidate as running, launch the experiment, and prevent other agents from reusing the same run ID.
    • Finishing evaluation: The evaluator should write the final status, structured metrics, raw logs, and source snapshot.
    • Interpreting results: The researcher should write a postmortem explaining whether the hypothesis was supported and what should happen next.
    • Curating the frontier: The curator should update the leaderboard or Pareto frontier and mark promising branches for further work.

  • The ledger should include the run ID, claiming agent, parent ID, status, edit family, start time, finish time, metric, and notes path. For parallel search, the candidate proposal distribution should penalize duplicated edit families already in flight:

    \[P(c) \propto \exp \left( \frac{ Q(c) - \lambda D(c, \mathcal{I}) }{ \tau } \right)\]
    • where \(Q(c)\) is expected quality, \(\mathcal{I}\) is the set of in-flight experiments, and \(D(c,\mathcal{I})\) measures similarity to active proposals.

Auditor checks

  • The auditor should run automatically after every candidate. Its job is not to judge scientific taste; its job is to enforce the contract.

  • For training-code autoresearch, auditor checks should verify that editable-file boundaries held, evaluator hashes matched, the result file was produced during the current run, the candidate trained for the full budget, validation token and byte counts did not change, stderr did not contain hidden warnings, and the run compared against the correct parent.

  • For harness optimization, auditor checks should additionally verify that the harness did not access held-out labels, hard-code benchmark task IDs, change parser behavior without logging parse-error rate, increase context beyond the allowed budget, or silently retry more times than permitted.

  • Auditing is especially important in code-space search because the agent is powerful enough to change the system around the metric. The advantage is that suspicious shortcuts are visible as code diffs or file accesses, so they can often be detected automatically.

Research organization code

  • The instruction file is the research organization encoded as text. It should evolve more slowly than candidate code. Humans should edit it when they observe systematic problems in the agent’s behavior, such as too many risky rewrites, insufficient logging, repeated NaN runs, or failure to inspect regressions.

  • A strong instruction file should define the mission, scientific method, permissions, experiment protocol, memory protocol, selection protocol, and safety protocol. The mission should specify the target metric and budget. The scientific method should require every run to have a hypothesis, a coherent change, and an interpretation. The permissions should define editable, read-only, append-only, and forbidden files. The experiment protocol should define validation gates, the full-run command, the metric parser, and result location. The memory protocol should define how to name runs, write notes, and inspect history before editing. The selection protocol should define how to compare candidates, branch, and update the frontier. The safety protocol should define how to handle crashes, NaNs, out-of-memory failures, suspicious improvements, and evaluator drift.

  • This aligns with the broader lesson that scalable search procedures tend to outperform brittle hand-coded solutions as compute and agents improve. The Bitter Lesson by Sutton (2019) is relevant because it argues that general methods that scale with computation, especially search and learning, tend to dominate hand-built knowledge over time.

Agent evaluation

  • The research agent itself should be evaluated, not only the candidates it produces. Useful agent-level metrics include:
\[\mathrm{hit\ rate} = \frac{ \# \mathrm{improving\ candidates} }{ \# \mathrm{evaluated\ candidates} }\] \[\mathrm{invalid\ rate} = \frac{ \# \mathrm{invalid\ candidates} }{ \# \mathrm{proposed\ candidates} }\] \[\mathrm{time\ to\ best} = \min \{t : s(c_t) = \min_i s(c_i)\}\] \[\mathrm{useful\ regression\ rate} = \frac{ \# \mathrm{regressions\ later\ mined\ for\ useful\ components} }{ \# \mathrm{regressions} }\]
  • For harness search, also track trace-inspection behavior, such as how many prior candidates the proposer inspected and whether it looked at raw traces before editing. A system that improves only by luck may still find good candidates, but it will be less reliable than one that consistently diagnoses failure modes before proposing changes.

Practical default policy

  • A good default policy for one autonomous agent should require the agent to read the leaderboard, read the current best candidate, inspect two recent failures, choose one edit family, write a hypothesis, make a small coherent change, run smoke checks, run the fixed-budget experiment, append structured metrics, write a postmortem, and decide whether to keep, revert, branch, or revisit.

  • Every tenth iteration, the system can allow a larger exploration. In that case, the agent should review all runs by edit family, identify underexplored regions, propose one structural change, run stronger validation before the full budget, and preserve the previous best unchanged.

  • This policy keeps the loop scientific without making it rigid. It gives the agent enough structure to accumulate knowledge and enough freedom to discover improvements humans did not pre-specify.

Evaluation

  • Autoresearch needs evaluation machinery that is stricter than ordinary ad hoc experimentation because the optimizer is allowed to edit code. The evaluator defines the scientific contract: what counts as progress, what is forbidden, which metrics are comparable, and which candidates must be rejected even if they appear to improve the headline score.

Metric choice

  • The primary metric should match the research substrate. In single-GPU language-model training, validation bits per byte is a good default because it measures compression quality on held-out text and is less tied to vocabulary size than raw token loss. In harness optimization, the primary metric is usually task accuracy, pass rate, reward, or another task-specific outcome. In long-horizon terminal agents, pass rate is often the central metric because a candidate either completes the task under the benchmark’s tests or fails. Terminal-Bench by Merrill et al. (2026) is relevant because it evaluates agents on hard terminal tasks with task-specific environments, human-written solutions, and verification tests, making pass rate a natural benchmark-level metric.

  • For language modeling, the core validation metric can be written as:

\[\mathrm{BPB} = \frac{ \sum_{i=1}^{N_{\mathrm{tokens}}} -\log_2 p_{\theta}(x_i \mid x_{<i}) }{ N_{\mathrm{bytes}} }\]

  • For task-solving harnesses, the empirical reward estimate is:

    \[\hat{R}(H) = \frac{1}{n} \sum_{i=1}^{n} r(\tau_i, x_i)\]
    • where \(H\) is the harness, \(x_i\) is a task instance, \(\tau_i\) is the rollout produced by the model inside the harness, and \(r(\tau_i, x_i)\) is the task reward.

Secondary metrics

  • The headline metric is not enough. A candidate that improves BPB while doubling memory use, slowing training dramatically, or relying on more context may be a poor research outcome. Every evaluation should log secondary metrics that explain the cost of the improvement.

    • Throughput: The evaluator should record tokens per second, examples per second, or tasks per hour so that quality gains can be separated from efficiency regressions.
    • Training exposure: The evaluator should record the number of training tokens, optimizer steps, and completed validation tokens so candidates are compared under equivalent work.
    • Memory use: The evaluator should record peak accelerator memory because some candidates only work by moving close to an out-of-memory boundary.
    • Parameter count: The evaluator should record model size because fixed-budget improvements can come from better capacity, better speed, or both.
    • Context cost: Harness evaluations should record prompt tokens, retrieved tokens, number of model calls, and tool-call count because a higher score may come from using substantially more inference budget.
    • Failure rate: The evaluator should separately track crashes, NaNs, parser failures, timeouts, invalid outputs, and budget violations so that failure modes do not get collapsed into one vague “bad run” label.

  • A cost-aware scalar objective can be useful for triage:

    \[J(c) = s(c) + \lambda_{\mathrm{mem}} \cdot \max(0, M(c)-M_{\max}) + \lambda_{\mathrm{time}} \cdot T(c) + \lambda_{\mathrm{fail}} \cdot \mathbb{1}[\mathrm{invalid}(c)]\]
    • where \(s(c)\) is the main score, \(M(c)\) is memory use, \(T(c)\) is runtime cost, and invalid candidates receive an explicit penalty. For many research settings, however, a Pareto frontier is cleaner than scalarization because the desired tradeoff between quality, speed, memory, and context cost may change later.

Pareto evaluation

  • A candidate is Pareto-optimal when no other candidate is at least as good on every tracked objective and strictly better on at least one. This is especially important in harness search, where a candidate may improve accuracy but use more tokens, more model calls, or more tool steps.
\[c_i \in \mathcal{P} \iff \nexists c_j: \left( \forall k,\ f_k(c_j) \le f_k(c_i) \right) \land \left( \exists k,\ f_k(c_j) < f_k(c_i) \right)\]
  • A good evaluation report should therefore present both the single best candidate under the chosen deployment preference and the frontier of alternatives. One candidate might be the best high-accuracy configuration, another might be the best low-cost configuration, and another might be the best robust configuration with few invalid outputs.

Search and test splits

  • Autoresearch needs a clean split between search-time feedback and final evaluation. The search set is allowed to influence proposals. The test set is not. This distinction is essential because an outer-loop agent can overfit just as easily as a human hyperparameter tuner.

    • Search set: The agent can repeatedly evaluate candidates here, inspect failures, and use traces to guide future edits.
    • Validation or selection set: The system can use this for candidate selection or frontier pruning when the search set is noisy or too small.
    • Held-out test set: The final report should evaluate only selected candidates here, and the proposer should not see the results during search.
    • Audit set: A small set of tasks can be reserved for checking leakage, parser shortcuts, memorized task IDs, or suspicious benchmark-specific behavior.

  • In public discovery settings, repeated benchmark iteration may be part of the competition or research objective, but the system should still audit candidates for hard-coded task identifiers, answer strings, and benchmark-specific shortcuts. This is especially important when optimizing executable harness code rather than only model weights.

Noise control

  • Autoresearch often makes decisions from short, noisy runs. A small BPB improvement may be real, or it may be seed noise, hardware jitter, dataloader variation, or stochastic sampling. The evaluator should estimate noise before treating small differences as discoveries.

  • For repeated runs of the same candidate, the mean and standard error are:

\[\bar{s} = \frac{1}{K} \sum_{k=1}^{K} s_k\] \[\mathrm{SE} = \frac{ \sqrt{\frac{1}{K-1}\sum_{k=1}^{K}(s_k-\bar{s})^2} }{ \sqrt{K} }\]

  • A simple acceptance rule with a margin is:

    \[\mathrm{accept}(c) = \mathbb{1} [ s(c) < s(c_{\mathrm{best}})-\epsilon ]\]
    • where \(\epsilon\) should be chosen based on repeated baseline runs. When experiments are expensive, the system can use single runs for exploration and repeated runs only for frontier candidates.

Budget integrity

  • Budget integrity means every candidate receives the same allowed resources. Without this, the agent may discover ways to appear better by training longer, evaluating less data, retrying more often, or using a larger context budget.

    • Wall-clock budget: Training candidates should run for the same measured duration, with clear rules for whether compilation and startup are included.
    • Token budget: Training candidates should report how many tokens they actually processed, and harness candidates should report how many input and output tokens they used.
    • Tool budget: Agentic harnesses should have explicit limits on shell calls, retries, web calls, file reads, or subprocesses when those costs matter.
    • Evaluation budget: Every candidate should evaluate on the same task set, validation tokens, or benchmark split unless the run is explicitly marked as a partial diagnostic run.
    • Retry budget: Harnesses should not silently improve pass rate by adding unbounded retries; retries should be counted and capped.

  • A candidate should be excluded from the leaderboard if it violates budget assumptions, even if the headline metric improves.

Failure taxonomy

  • Failures should be classified, not merely discarded. Failed runs are valuable because they define unstable regions of the search space and help future agents avoid repeated mistakes.

    • Syntax or import failure: The candidate cannot load, so the problem is implementation correctness rather than research quality.
    • Shape failure: Tensor dimensions, parser schema, or tool-output formats are inconsistent, which usually means the edit was not integrated carefully.
    • Out-of-memory failure: The candidate exceeded hardware limits, which may indicate that the idea needs a smaller batch, shorter sequence length, checkpointing, or a narrower model.
    • NaN or Inf failure: The candidate became numerically unstable, which often points to learning rate, precision, normalization, initialization, gradient clipping, or optimizer partitioning.
    • Timeout failure: The candidate exceeded runtime limits, which can happen because of inefficient code, excessive tool loops, slow retrieval, or too many retries.
    • Metric-integrity failure: The run did not produce a trustworthy metric, reused a stale result, changed evaluator files, or produced incomplete outputs.
    • Leakage failure: The candidate accessed forbidden labels, task IDs, held-out answers, or changed the data path in a way that invalidates comparison.

  • This taxonomy gives the next proposal a useful map. A candidate with a good idea but an out-of-memory failure may deserve a scaled-down retry. A metric-integrity failure should be rejected regardless of apparent performance.

Trace evaluation

  • Trace evaluation explains why a candidate succeeded or failed. In harness search, aggregate score is often too compressed because a harness can fail through retrieval, reasoning, parsing, tool use, memory update, or timeout. The evaluator should therefore produce task-level traces.

  • Each trace should answer several questions:

    • What did the model see: The trace should preserve the prompt, retrieved context, memory state, and system instructions used for the model call.
    • What did the model do: The trace should preserve model outputs, tool calls, intermediate answers, retries, and state updates.
    • What did the harness do: The trace should preserve parser decisions, verifier decisions, routing decisions, retrieval rankings, and stopping conditions.
    • What failed: The trace should label the dominant failure type when possible, such as retrieval miss, invalid format, wrong reasoning, timeout, tool error, or verifier rejection.
    • What did it cost: The trace should record context tokens, model calls, tool calls, latency, and any retry count.

  • This is the evaluation-side reason that full-history access matters. A proposer can only perform meaningful credit assignment if the evaluator leaves behind enough evidence to distinguish superficially similar failures.

Benchmark selection

  • The right benchmark depends on the capability being optimized. A single-GPU language-model substrate needs a held-out text validation set and a stable BPB computation. A coding-agent substrate needs tasks with executable tests. A terminal-agent substrate needs full environments, setup scripts, expected outcomes, and verification. SWE-agent by Yang et al. (2024) is relevant because it shows that agent-computer interface design can materially affect software-engineering agent performance, so evaluation should include the interface and tools rather than only the base model’s text output.

  • Benchmark choice should follow these principles:

    • Use tasks that expose the bottleneck: If the research question is retrieval, use tasks where retrieved evidence actually matters; if the question is tool use, use tasks requiring real tool interaction.
    • Use executable verification when possible: Unit tests, formal checks, exact-match answer checkers, and deterministic graders reduce ambiguity and make autonomous evaluation easier.
    • Include easy and hard tasks: Easy tasks catch regressions in basic functionality, while hard tasks provide gradient for frontier improvement.
    • Track per-family results: Aggregate metrics can hide regressions on minority task families, so benchmarks should record subgroup performance when possible.
    • Avoid tiny search sets: A small repeated search set invites overfitting, especially when the proposer can inspect detailed traces.

Memory-compaction evaluation and COMPACT-Bench

Why end-task accuracy is insufficient

  • Memory and context-management systems should not be evaluated only by final task accuracy. Two systems may achieve similar single-turn accuracy while making fundamentally different memory decisions: one may retain recoverable evidence outside the active context, while another may irreversibly discard it. Their behavior can diverge only after later queries require previously removed information or after the same memory is compacted repeatedly. What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) argues that evaluation should therefore measure the accuracy attainable at a fixed memory rate, whether removed information remains recoverable, and how distortion evolves over repeated compaction.

  • The evaluation problem can be expressed as an accuracy-budget frontier rather than a single operating point. For a compaction method indexed by \(j\), define:

    \[\mathcal{F}_j = \left\{ \left( B, A_j(B) \right) : B \in \mathcal{B} \right\}\]
    • where \(B\) is the retained-memory budget and \(A_j(B)\) is downstream accuracy at that budget. A useful comparison sweeps multiple values of \(B\) instead of allowing each method to choose its own compression ratio. This exposes whether one method dominates another throughout the budget range, whether their rankings change under tighter memory, and where performance collapses as retained information falls below the task requirement.

  • A complete memory evaluation should answer at least four distinct questions:

    • Utility: How much task performance remains at a fixed memory budget?
    • Attribution: Does the method know which information it removed and which future queries are therefore at risk?
    • Reversibility: Can information removed from active memory be recovered later when a query requires it?
    • Compounding: Does repeated compression progressively destroy information that survived earlier compression rounds?

  • These properties cannot be recovered from one final accuracy number. Post-compaction accuracy averages over the consequences of many memory decisions, obscuring which evidence was lost and whether the system could recover it. What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) therefore treats loss attribution, reversibility, calibrated compaction confidence, and repeated-compaction behavior as first-class evaluation targets.

Existing long-context benchmarks measure different pieces of the problem

  • Several benchmark families probe increasingly difficult forms of long-context use, but they do not place heterogeneous memory mechanisms on the same resource axis:

  • These benchmarks are complementary, but their resource measurements differ. KV-cache work naturally reports retained cache elements or bytes; prompt compression often reports token compression ratios; recurrent architectures report state dimensions; agent-memory systems report stored items, characters, tokens, or slots. Comparing accuracy at independently chosen budgets therefore does not establish which approach uses memory more efficiently. What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) motivates a layer-independent memory-rate axis precisely to make these frontiers commensurable.

Bytes per token of history as a common budget axis

  • A practical normalization is bytes per token of original history, abbreviated BPT:

    \[\mathrm{BPT} = \frac{ \mathrm{bytes\ retained\ after\ compaction} }{ \mathrm{tokens\ in\ original\ history} }\]

    • This converts heterogeneous memory representations into a common physical resource. Let the full, uncompressed representation consume:

      \[B_{\mathrm{full}}\]

      • bytes per history token. Several forms of compaction can then be approximately normalized relative to the same reference:

        \[B_{\mathrm{evict}} = f B_{\mathrm{full}}\]

        • when token eviction retains fraction \(f\) of the cache,

          \[B_{\mathrm{quant}} = \frac{b}{16} B_{\mathrm{full}}\]

          • when a 16-bit cache is quantized to \(b\) bits, and

            \[B_{\mathrm{text}} = \frac{m}{n} B_{\mathrm{full}}\]
            • when an original history of \(n\) tokens is represented by \(m\) retained or summarized tokens.

  • This normalization does not imply that a byte in a summary and a byte in a KV cache carry equivalent semantics. It provides a common resource denominator so their downstream accuracy can be compared as a function of retained memory. What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) uses this BPT framing to place otherwise heterogeneous memory operators on a shared accuracy-versus-budget frontier.

COMPACT-Bench

Multi-budget accuracy frontiers
  • Every method should be evaluated across a sweep of memory budgets rather than at one selected operating point:
\[B_1 < B_2 < \cdots < B_K\] \[\mathcal{A} = \left\{ A(B_1), A(B_2), \dots, A(B_K) \right\}\]
  • The resulting curve measures the amount of downstream utility preserved per unit of retained memory. A method that performs well only at a generous budget may be less useful than one whose degradation is gradual under tighter constraints. The location of the sharp degradation region also provides empirical evidence about how much task-relevant information must survive compression.
Loss attribution

  • Evaluation should test whether the compaction mechanism knows what it removed. After compression, a held-out set of facts or dependencies from the original history can be probed and compared against the method’s predicted retention or importance scores.

  • For a set of history facts:

    \[\mathcal{G} = \left\{ g_1,\dots,g_N \right\}\]

    • define a binary retention variable:

      \[r_i = \mathbb{1} \left[ g_i \text{ remains recoverable} \right]\]

      • and let the compaction mechanism emit a predicted retention score:

        \[\hat{r}_i \in [0,1]\]

  • A useful memory system should not merely achieve good aggregate accuracy. Its internal importance estimates should distinguish facts that remain recoverable from facts whose removal creates downstream risk. This turns the compaction scorer itself into an object of evaluation rather than treating it as an invisible implementation detail.

Reversibility

  • Reversibility should be tested directly by asking a late query whose evidence has already left active memory. If the system retains an archival representation, it should be allowed to retrieve or reconstruct the missing evidence. If the information was irreversibly deleted or summarized away, recovery should fail when the answer depends on the discarded detail.

  • A simple recovery metric is:

\[R_{\mathrm{recover}} = \frac{ N_{\mathrm{correct\ after\ requested\ recovery}} }{ N_{\mathrm{queries\ requiring\ removed\ evidence}} }\]
  • This separates active-context compression from actual forgetting. A retrieval-backed system can maintain a small active state without permanently losing the underlying evidence, while irreversible summarization trades away that possibility.
Compaction confidence

  • A deployment-grade memory system should also know when its compressed state is insufficient. COMPACT-Bench proposes evaluating whether the system’s confidence after compression tracks actual downstream correctness, allowing low-confidence cases to trigger retrieval, re-expansion, abstention, or a higher-fidelity memory tier.

  • The desired control policy is:

    \[\hat{p}_{\mathrm{correct}} < \tau \quad \Rightarrow \quad \text{retrieve, re-expand, or defer}\]
    • rather than forcing the model to answer from memory whose relevant evidence may already have been destroyed.
Joint quality and systems cost

  • Memory compression should not be evaluated separately from its serving cost. The benchmark should report task accuracy together with retained memory, latency, and monetary or compute cost because a compression procedure may reduce memory while introducing expensive summarization calls, retrieval operations, or recomputation.

  • A useful result record is therefore a vector rather than a single score:

    \[\mathbf{z} = \left[ A, B, T, C \right]\]
    • where \(A\) is task accuracy, \(B\) is retained memory, \(T\) is latency, and \(C\) is deployment cost. The evaluator can either expose the corresponding Pareto frontier or apply an application-specific utility function after measurement.

Evaluating repeated compaction

  • Long-running agents repeatedly edit their own working state. Evaluating only one compression event therefore misses a failure mode that does not exist in conventional single-turn long-context inference: an early omission can remove evidence from the input to the next summarization step, causing subsequent summaries to operate on an already distorted representation.

  • Let:

    \[Z_0 = H\]

    • denote the original history, and let repeated irreversible compaction produce:

      \[Z_{t+1} = C \left( Z_t, \Delta H_t \right)\]
      • where \(\Delta H_t\) is newly accumulated history. Once an item disappears from \(Z_t\) and no external copy survives, later applications of \(C\) cannot recover it from the available state.

  • Evaluation should therefore report accuracy or recall as a function of the number of compaction events:

    \[A(k) = \mathrm{Accuracy\ after\ } k \mathrm{\ compaction\ events}\]
    • rather than reporting only \(A(1)\)

  • The rate-distortion analysis predicts qualitatively different trajectories for reversible and irreversible operators. Retrieval-backed systems can restore evidence removed from active context, whereas repeated lossy summarization can compound earlier omissions. What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) identifies this repeated-compaction curve as a missing dimension in existing long-context evaluation.

  • For autoresearch, the analogous experiment is to run the same research agent for increasingly long search horizons and measure whether it can still recover early hypotheses, failed configurations, causal diagnostics, and experiment outcomes. An apparently strong memory design after ten experiments may become unreliable after hundreds of summarize-update cycles.

A reference unified accuracy-budget experiment

  • As a reference implementation of the budget-frontier protocol, What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) evaluates six KV-cache compaction methods using Qwen2.5-1.5B-Instruct on natural-text needle retrieval. Needles are inserted at depths from 10% to 90% into contexts spanning 2K to 8K tokens, and each method is evaluated across five memory budgets. The experiment contains 1,395 generations and uses the uncompressed cache as the full-budget reference.

  • The evaluated operators include SnapKV: LLM Knows What You are Looking for Before Generation by Li et al. (2024), which estimates important KV positions from an observation window near the end of the prompt, and Efficient Streaming Language Models with Attention Sinks by Xiao et al. (2023), which preserves initial attention-sink tokens together with a recent sliding window.

  • The following figure (source) shows the accuracy-budget frontier on natural-filler needle retrieval with Qwen2.5-1.5B, placing six KV-compaction methods on the same bytes-per-token-of-history axis; performance approaches the full-cache result at generous budgets and collapses for all tested methods once the retained budget becomes sufficiently small.

  • The important result is not the ranking of these particular methods at this small reference scale. It is that putting every method on the same BPT axis exposes the shape of the rate-distortion frontier. At the full-cache operating point, accuracy is \(1.00\); as the budget contracts, the methods move toward zero accuracy, and below roughly one quarter of the full budget all tested methods are at or near zero on this experiment. The random-eviction control also helps separate gains due to an importance scorer from gains that arise simply because a large fraction of the cache is still retained.

Decision-centric memory evaluation

  • Agent-memory evaluation should additionally measure whether the compressed representation preserves the distinctions needed for downstream decisions. Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) evaluates this using a decision-distortion frontier and diagnostics that compare descriptive retrieval against decision-aware memory under matched budgets.

  • Under a fixed query, an evaluator can define the best achievable action value from the complete history and compare it with the action selected from compressed memory:

\[d_q(h,m) = \mu_q^{\star}(h) - \mu_q \left( h,\pi_q(m) \right)\]

  • This produces a direct measure of memory-induced decision loss. The corresponding implementation-level evaluation should distinguish at least:

    • Compression error: the unavoidable loss caused by representing history with a finite number of memory states.
    • Routing error: additional loss because the system selects the wrong memory state.

    • Realization error: additional loss because the selected state is represented imperfectly to the answering model.

    • Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) explicitly decomposes realized distortion into these components, making it possible to determine whether an observed memory failure arises from the budget itself or from an improvable implementation choice.

  • Evaluation should also match competing systems on answer-time memory budget. On the annotated LoCoMo analysis reported by Zou et al., decision-aware routing recovers 83% of gold evidence compared with 66% for cosine top-k retrieval and 63% for BM25 top-k under the same character budget; the oracle selector reaches 89%. This is the relevant comparison because allowing one memory system to expose more evidence to the answering model would confound memory quality with memory quantity.

Implications for autoresearch evaluation

  • An autoresearch evaluator should treat memory policy as a measurable subsystem rather than a hidden implementation detail. In addition to task reward, the evaluator should record:

    • Active memory budget: prompt tokens, working-memory bytes, or another physical measure of the state visible to the model.
    • Archive budget: the size of reversible evidence retained outside active context.
    • Compaction count: the number of lossy or lossless memory transformations applied during a run.
    • Recovery rate: how often information absent from active context can still be recovered when required.
    • Evidence recall: whether the context supplied for each decision contains the evidence needed to make that decision.
    • Decision distortion: whether compressed memory changes the action, experiment, tool call, or stopping decision that would have been chosen with fuller evidence.
    • Memory-operation latency: the cost of retrieval, summarization, consolidation, and re-expansion.
    • Long-horizon degradation: performance as the number of experiments, interactions, and compaction events increases.

  • A memory-policy candidate should therefore be evaluated as a frontier:

    \[\mathcal{P}_{\mathrm{memory}} = \operatorname{Pareto} \left( -\mathrm{task\ quality}, \mathrm{active\ BPT}, \mathrm{archive\ bytes}, \mathrm{latency}, \mathrm{compaction\ cost} \right)\]
    • rather than selected solely because it minimizes prompt length.

  • This distinction is important for metaharness search. A candidate that shortens the prompt by 50% but irreversibly deletes evidence required several iterations later is not necessarily better than a slightly larger reversible memory. Conversely, a candidate that retains every raw trace indefinitely may preserve quality but offer little systems advantage. The relevant object is the full quality-versus-memory frontier and its behavior over the complete research horizon.

  • The central evaluation principle is therefore: compare memory systems at matched resource budgets, inspect what information was lost, test whether it can be recovered, and measure the effect repeatedly over time. This converts “memory quality” from a vague recall score into an auditable systems property that can itself be optimized by the autoresearch loop.

Evaluating compaction against task-specific lower bounds

  • Empirical baselines alone do not reveal whether a context-compaction method is close to the best achievable representation for its workload. Context Compaction Theory by Tirmazi et al. (2026) shows that the minimum generative compaction budget required to answer a family of future queries within target error is equal to the one-way communication complexity of the induced problem.

  • When this optimum or a tight lower bound is known, evaluation can report an optimality gap:

    \[G(B) = D_{\mathrm{method}}(B) - D_{\mathrm{lower}}(B)\]
    • where \(D_{\mathrm{method}}(B)\) is the observed downstream distortion at budget \(B\) and \(D_{\mathrm{lower}}(B)\) is the smallest distortion theoretically attainable at the same budget.

  • The evaluation should also compare selection-based and generation-based condensers at matched budgets. Generation can be strictly more expressive: Tirmazi et al. construct a workload where an exact generative representation requires \(n\) bits while every exact selection-based representation requires at least \(n\log_2 n\) bits. This means a weak selection baseline should not be mistaken for evidence that the budget itself is insufficient.

  • Query-specific benchmarks are especially valuable when the sufficient statistic is known. For membership queries, for example, a Bloom filter provides a principled compact representation against which a natural-language condenser can be compared.

  • In the paper’s deployed-endpoint case study, approximately 15,000 URLs were compacted into roughly 14 Kbits before membership queries were issued. The resulting error rates were 0.505, 0.535, and 0.555 across three seeds, while the uncompressed control produced an error rate of 0.02. The same-budget Bloom-filter curve was substantially better, showing that the failure was not explained by the budget alone.

  • The following figure (source) compares the membership-query error of the deployed natural-language condenser with a same-budget Bloom filter, the information-theoretic lower bound, and random guessing.

  • This suggests a stronger compaction benchmark protocol: specify the anticipated query family, measure quality across budgets, include a task-specific algorithmic baseline when one is known, estimate or derive the information-theoretic frontier when possible, and separately report the gap caused by the practical condenser or interpreter.

Evolutionary evaluation

  • Autoresearch is closely related to evolutionary program search because candidates are proposed, evaluated, selected, and mutated. AlphaEvolve by Novikov et al. (2025) is relevant because it describes an autonomous coding-agent pipeline that improves algorithms by directly editing code and receiving evaluator feedback. OpenEvolve is relevant as an open-source evolutionary coding-agent framework for generating, mutating, evaluating, and selecting code candidates.

  • The evaluator in such systems should support three comparison modes:

    • Parent comparison: The candidate should be compared against the exact parent it modified, which helps identify whether the local edit helped.
    • Best-so-far comparison: The candidate should be compared against the global best, which determines whether it should become the new default.
    • Frontier comparison: The candidate should be compared against the Pareto frontier, which determines whether it offers a new tradeoff even if it is not the best on the headline metric.

Reporting

  • A useful autoresearch report should make the improvement auditable. It should not only say that the best candidate improved the metric; it should show the search trajectory, the frontier, the winning diff summary, and the failure analysis.

  • A strong report should include:

    • Progress curve: The report should show best-so-far score over evaluated candidates, with invalid runs and partial runs visually distinguished from valid full evaluations.
    • Leaderboard: The report should list the top candidates, their parents, metrics, costs, failure rates, and links to source snapshots.
    • Pareto frontier: The report should show the tradeoff between quality and cost, such as BPB versus throughput or accuracy versus context tokens.
    • Ablations: The report should isolate which component of the winning candidate mattered, especially when the winning diff combined multiple changes.
    • Robustness checks: The report should repeat the best candidate under additional seeds, task subsets, or held-out tasks when feasible.
    • Leakage audit: The report should document that evaluator files, validation data, task labels, and held-out answers were not modified or accessed improperly.
    • Trace examples: The report should include representative successes and failures so humans can verify that the system improved for the intended reason.

  • The evaluator is therefore not just a scoring script. It is the foundation that makes autonomous research cumulative, comparable, and trustworthy.

Memory

  • Autoresearch depends on memory because the value of each experiment is not only its final score. Each run contributes evidence about what works, what fails, what is unstable, and which ideas deserve to be revisited. Without durable memory, an agent is just sampling code edits. With durable memory, it becomes a cumulative research process.

  • Durable memory should be separated from active context: an agent may preserve a complete experimental archive while still forgetting information operationally if context compaction removes it from the state supplied to future model calls and no retrieval path restores it.

Rate–distortion memory and decision-preserving forgetting

Memory compaction as a common optimization problem

  • Durable agent memory is not only a storage problem. It is a resource-allocation problem in which a system must decide which parts of an expanding history should remain available, at what fidelity, and under what memory or context budget. What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) unifies four mechanisms that are usually studied separately, KV-cache compression, prompt/context compression, bounded architectural state, and agent-memory consolidation, as instances of the same rate–distortion problem.

  • Let the complete interaction history be represented by \(H\), a compaction operator produce a smaller representation \(Z\), a downstream query be \(Q\), and the desired output be \(Y\). A layer-agnostic compaction problem can be written as:

    \[\min_{\theta} \mathbb{E}_{(H,Q,Y)} \left[ \ell \left( U(C_{\theta}(H),Q), Y \right) \right] \qquad \text{s.t.} \qquad \operatorname{rate}(Z)\le B\]

    • where \(C_{\theta}\) is the compaction operator, \(U\) is the mechanism that uses compact memory, and \(B\) is the available memory budget. The rate can be measured in GPU bytes for a KV cache, tokens for a prompt, dimensions for recurrent state, or storage consumed by an agent-memory system. The corresponding information-bottleneck form is:

      \[\max I(Z;Y\mid Q) \qquad \text{s.t.} \qquad I(Z;H)\le B\]

  • Intuitively, a good memory preserves the parts of history that predict future task-relevant outputs while spending as little capacity as possible on everything else. What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) develops this formulation across the full inference-to-agent-memory stack.

  • Define the task-conditioned information requirement as:

\[I^{\star}(Q) = I(Y;H\mid Q)\]

  • For the query-agnostic compaction setting analyzed in the rate–distortion formulation, the data-processing inequality together with Fano’s inequality yields the layer-independent lower bound:

    \[P_e \ge \frac{ H(Y\mid Q)-B-1 }{ \log |\mathcal{Y}| }, \qquad B<I^{\star}(Q)\]
    • where \(P_e\) is the probability of producing the wrong answer. The important consequence is substrate-independent: once retained memory falls below the information genuinely required by a task, no choice of KV representation, summary, recurrent state, or semantic memory can eliminate the resulting error. Exact and multi-hop retrieval therefore tend to tolerate less lossy compression than low-entropy tasks such as classification or summarization of highly redundant text.

  • The following figure (source) shows the rate–distortion view: below the task-conditioned information requirement, compaction necessarily introduces error, while a query-conditioned operator can achieve the same utility with a smaller memory budget than a query-agnostic operator.

Query conditioning, reversibility, and memory fidelity

  • A crucial distinction is whether compaction happens before or after the system knows what information will be needed. Under the query-agnostic analysis, the system must spread its finite capacity across possible future queries; query-conditioned retrieval can instead allocate capacity to information relevant to the query that actually arrives. The formalism associates this gap with query uncertainty and therefore gives a theoretical reason to prefer recall-time retrieval over unconditional eager compression when the future information need is not known in advance. What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) identifies query conditioning as one of the central properties governing compaction quality.

  • Reversibility is equally important. A reversible memory mechanism may remove information from the active context while retaining a path for retrieving the original evidence later. An irreversible mechanism, such as repeatedly replacing a trajectory with a lossy summary, removes that option. The distinction matters most on long horizons because each irreversible summary operates on an already compressed representation, allowing omissions and errors to propagate into later compaction cycles.

  • This suggests a multi-fidelity memory hierarchy rather than a single summary buffer:

    • High-fidelity episodic tier: preserve raw experiment traces, tool outputs, code snapshots, observations, and other evidence that would be expensive or impossible to reconstruct.
    • Compact semantic tier: maintain distilled conclusions, patterns, hypotheses, and experiment summaries that make the archive inexpensive to navigate.
    • Query-conditioned retrieval: use the current research question to select high-fidelity evidence from the archive when a compact representation is insufficient.

    • Promotion and demotion: move information between tiers according to demonstrated task utility rather than a fixed age or token threshold.

    • This design follows the broader principles derived in What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026): preserve cheaply recoverable raw evidence, condition retrieval on future information needs, move expensive consolidation off the critical path when possible, and allocate memory according to marginal task utility rather than a uniform compression ratio.

  • The distinction is especially relevant to autoresearch because two experiment histories that appear descriptively similar may imply very different next actions. For example, “increasing depth degraded validation performance” and “increasing depth initially helped but later diverged because of optimizer instability” may be close as textual summaries, yet the second history supports a different next experiment. Memory should therefore preserve causal and decision-relevant distinctions, rather than merely producing semantically faithful summaries.

Decision-preserving forgetting

  • Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) sharpens the rate–distortion view specifically for agent memory: the value of a memory representation is measured by whether compression changes the decisions an agent can make, rather than by how faithfully the compressed state describes its history. The resulting framework gives an exact criterion for when histories can safely share the same bounded memory state.

  • At answer time, the current query remains directly visible while the history must pass through a bounded memory state:

\[M_t = g_t(H_t,Q_t) \in [K], \qquad A_t = \pi_t(M_t,Q_t)\]

  • The memory encoder can therefore select different aspects of the same history for different queries, but the downstream policy cannot directly consult the entire history.

  • For a fixed query, let the full-history value of the best action be:

\[\mu_q^{\star}(h) = \max_{a\in\mathcal{A}} \mu_q(h,a)\]
  • The loss incurred by choosing action \(a\) is:
\[\Delta_q(h,a) = \mu_q^{\star}(h) - \mu_q(h,a)\]
  • If history \(h\) is compressed into memory state \(m\), the resulting decision distortion is:
\[d_q(h,m;\pi_q) = \mu_q^{\star}(h) - \mu_q \left( h,\pi_q(m) \right) = \Delta_q \left( h,\pi_q(m) \right)\]

  • This quantity measures the cost of forgetting directly in downstream decision quality rather than representation similarity.

  • The optimal decision rate–distortion frontier under a bounded number of memory states is:

\[\epsilon_{\infty}^{\star}(K;q) = \inf_{g_q:\mathcal{X}_q\rightarrow[K]} \inf_{\pi_q:[K]\rightarrow\mathcal{A}} \sup_{h\in\mathcal{X}_q} d_q \left( h,g_q(h);\pi_q \right)\]

  • This frontier gives the smallest worst-case decision loss achievable under a fixed runtime-memory budget. Increasing the number of states permits finer distinctions between histories; reducing it forces more histories to share a representation.

  • The exact forgetting boundary follows directly. A collection of histories can be collapsed into a single memory state with distortion no larger than a tolerance if and only if one common action remains near-optimal for every history in that collection:

\[C \text{ is safely mergeable at distortion } \epsilon \iff \exists a\in\mathcal{A} \text{ such that } \max_{h\in C} \Delta_q(h,a) \le \epsilon\]

  • This is a stronger criterion than semantic similarity. Two histories may be textually dissimilar yet safely merge if they imply the same decision, while two nearly identical histories must remain distinct if a small difference changes the correct action.

  • A corresponding pairwise decision distance is:

\[d_{\mathrm{dec}}^{q}(h,h') = \min_{a\in\mathcal{A}} \max \left\{ \Delta_q(h,a), \Delta_q(h',a) \right\}\]
  • Two histories can share a state under a given distortion tolerance precisely when their decision distance is below that boundary. This turns forgetting from an informal salience heuristic into a task-conditioned equivalence test.

DeMem: certified refinement under a fixed memory budget

  • Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) operationalizes decision-preserving compression with DeMem. Instead of continually creating memories according to salience or textual novelty, DeMem maintains a bounded partition and refines it only when observed feedback provides sufficient evidence that two situations cannot safely share a decision state.

  • The following figure (source) shows the DeMem workflow on a conversational-memory instance: a history and query are routed to a bounded memory slot, a policy acts from the selected state, and observed feedback triggers a split only when a decision conflict is certified.

  • DeMem separates the process into four operations:

    • Act: route the current history-query context into the current memory partition and choose an action using exploration or slot-level value estimates.
    • Certify: maintain statistical confidence bounds and introduce a cannot-link constraint when two contexts are confidently incompatible.
    • Partition: construct a feasible bounded-slot partition by greedily coloring the cannot-link graph.

    • Refresh: periodically replace the previous partition with the newly certified partition while keeping the memory budget fixed.

    • Importantly, the absence of a pairwise conflict is not treated as proof that a whole cluster is compatible; the method separately bounds the realized cluster-level distortion.

  • The regret decomposition makes explicit that finite memory introduces an unavoidable compression cost in addition to ordinary statistical learning. With high probability, the DeMem upper bound has the form:

\[\operatorname{Reg}(T) \le T\,O \left( \bar{\epsilon}^{\mathrm{cert}}_T \right) + \widetilde{O} \left( \sqrt{AKT} \right) + O \left( AN_T B_T(\gamma) \right)\]

  • The first term is the realized price of compressing histories into bounded memory, the second is statistical learning over the memory states and actions, and the third is the cost of collecting enough evidence to certify decision conflicts.

  • The corresponding minimax lower bound is:

\[\inf_{\mathrm{alg}} \sup_{\mu,\mathcal{D}} \mathbb{E} \left[ \operatorname{Reg}(T) \right] \ge cT\epsilon_{\infty}^{\star}(K) + c\sqrt{AKT}\]
  • The statistical term in DeMem therefore matches the minimax dependence up to logarithmic factors; the remaining gap depends on how close the learned partition comes to the optimal decision-distortion frontier and on the cost of conflict certification.

Evidence that descriptive similarity is not enough

  • The reported long-horizon experiments provide a concrete reason to separate decision relevance from textual similarity. On an annotated LoCoMo subset, descriptive similarity had only a weak association with evidence compatibility, with Spearman correlation 0.103 and AUC 0.548. Under the same answer-time character budget, cosine retrieval recovered 66% of gold evidence and BM25 recovered 63%, while decision-aware DeMem routing recovered 83%; an oracle selector recovered 89%. Among queries on which cosine retrieval failed and DeMem succeeded, 85.1% of failures were attributed to either missing the required evidence or diluting it with descriptively similar but irrelevant evidence. Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) uses these diagnostics to motivate decision-aware memory selection.

  • On the paper’s LoCoMo evaluation, DeMem obtained an overall judge score of 0.911 with the GPT-4o-mini backbone, compared with 0.888 for Mnemis, and 0.920 with GPT-4.1-mini, compared with 0.906 for Mnemis. The same study reports that certified splits fired on only 4.6% of routing events, with 85% split precision, indicating that the mechanism generally preserves shared states until positive evidence for decision conflict appears. These are results from a recent preprint and should be interpreted as evidence for the mechanism rather than as a settled ranking of memory systems.

  • The following figure (source) shows the DeMem ablation and robustness analysis, including parameter sensitivity, component ablations, the accuracy-versus-memory-budget frontier, and the accuracy-versus-latency frontier.

Repeated compaction is a distinct failure mode

  • One-shot evaluation can substantially understate the cost of lossy memory. The reference experiment in What to Keep, What to Forget: A Rate–Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) compares repeated LLM summarization with a reversible archive-and-retrieve policy while an agent processes a document containing twelve distributed facts. The reversible policy retains roughly 0.95 fact recall across compaction frequencies, while the irreversible summarization policy ranges from roughly 0.33 to 0.56 and degrades as compaction repeats. The experiment is deliberately small-scale, so the important result is the divergence between the two curves rather than their absolute values.

  • The following figure (source) shows fact recall as the number of compaction events increases: reversible retrieval-backed memory remains nearly flat, while irreversible summarization repeatedly loses information that later summaries can no longer recover.

  • This motivates evaluating agent memory as a trajectory rather than a single compression event. A memory benchmark should measure an accuracy-versus-budget frontier, identify which information was lost, test whether discarded evidence can be recovered, calibrate confidence after compaction, and explicitly measure degradation over repeated compaction cycles. The proposed COMPACT-Bench places heterogeneous memory mechanisms on a common bytes-per-token-of-history budget axis precisely to make such comparisons possible.

Harness-level memory can change measured capability

  • The distinction between model capability and memory-system capability is visible in interactive evaluation. OpenAI’s GPT-6 Astra on ARC-AGI-3 reports that GPT-6 Astra’s best Standard-harness result on ARC-AGI-3 Semi-Private was 62.7%, while its best Provider Adapter result was 99.9%. The Standard harness lets the model explicitly carry forward notes it elects to preserve; the Provider Adapter additionally preserves opaque reasoning state across requests and uses conversation compaction so prior computation can be reused.

  • The gap is not only an artifact of comparing the best scores at different reasoning settings. At matched maximum reasoning effort, the verified results are 62.71% under the Standard harness and 98.55% under the Provider Adapter; at matched high reasoning effort, they are 54.82% and 99.95% respectively. This makes the harness itself a first-class part of the measured agent system rather than incidental benchmark plumbing.

  • The public description is not sufficient to conclude that the Provider Adapter implements decision-preserving forgetting, query-conditioned compaction, or reversible storage in the formal senses above. It establishes that reasoning state is preserved and that long conversations are compacted, but not which state survives compaction or whether discarded information remains recoverable. The rate–distortion framework therefore provides a useful evaluation question rather than an explanation that can presently be asserted: does the harness preserve precisely the distinctions later decisions require, or does improved context management merely postpone irreversible information loss?

  • More generally, nominal context capacity should not be equated with usable memory. Context Rot: How Increasing Input Tokens Impacts LLM Performance evaluates eighteen contemporary language models and finds increasingly unreliable performance as input length grows, including on controlled retrieval and reproduction tasks. The systems problem is therefore not simply how much context can technically be supplied, but how much task-relevant information remains reliably usable when a decision is eventually required.

Implications for autoresearch and metaharness memory

  • These results imply a stronger memory architecture for autonomous research systems than a rolling summary alone. The archive should remain the reversible evidence tier; ledgers and summaries should primarily serve as low-cost indices into that evidence. This strengthens the existing principle that summaries should help the proposer decide where to look without replacing the raw traces needed for causal diagnosis.

  • The memory system should also distinguish descriptive similarity from experimental decision equivalence. Two runs should be merged in the proposer’s compact working memory only when losing their distinction is unlikely to alter a future research action. Conversely, semantically similar runs should remain distinguishable when one supports retrying an intervention and the other supports abandoning it.

  • Query-conditioned retrieval is particularly natural for autoresearch. When deciding whether to alter an optimizer, the agent should retrieve earlier optimizer experiments and their instability traces; when investigating a parser regression, it should retrieve parser failures instead. Eagerly reducing all prior experiments to a single global summary spends memory before the future research question is known.

  • Repeated compaction should itself become an evaluator target. A system can appear competent after one summarization pass while gradually erasing failed hypotheses, rare regressions, or causal details after tens or hundreds of iterations. Long-running autoresearch evaluations should therefore track how retrieval accuracy, decision consistency, and downstream experiment quality change with memory age and number of compaction cycles.

  • Finally, memory policy can itself become part of the optimized harness. Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026) treats storage, retrieval, context construction, and state management as executable harness decisions and gives the proposer access to prior code, scores, and traces through a filesystem. Combining that architecture with decision-preserving memory suggests an outer loop that can search not merely over what summary template to use, but over when to compact, which evidence remains reversible, how queries retrieve it, and which distinctions must survive for future research decisions.

Archive

  • The archive is the permanent record of the search. It should store every valid, invalid, crashed, and partial candidate in a way that future agents can inspect. The archive should be append-only by default because overwriting old runs destroys the causal trail needed for debugging.

  • A useful archive should include:

    • Candidate source: Each run should preserve the exact editable artifact that produced the result, such as the full training file, harness file, prompt builder, retrieval policy, or tool-orchestration code.
    • Parent diff: Each run should include a patch against its parent so that the agent can identify what changed without re-reading entire files.
    • Metrics: Each run should include structured metrics such as validation BPB, task accuracy, pass rate, context tokens, throughput, latency, memory use, and failure status.
    • Logs: Each run should retain stdout, stderr, warnings, stack traces, model-call logs, tool-call logs, and evaluator messages.
    • Traces: Harness runs should store prompts, retrieved context, model outputs, parser decisions, tool observations, state updates, retries, and final answers.
    • Notes: Each run should include a concise postmortem with the hypothesis, intervention, result, interpretation, and suggested next step.
    • Integrity metadata: Each run should record evaluator hashes, data hashes, run start time, run end time, hardware metadata, package versions, and random seeds when applicable.

  • This archive layout supports both automated inspection and human review. It also mirrors the key Meta-Harness idea: the proposer should access a large filesystem of prior code, scores, and execution traces selectively rather than receiving only compressed summaries.

Ledgers

  • The ledger is the compact index over the archive. It should be small enough for the agent to read frequently and structured enough for automated sorting. The archive holds the full evidence; the ledger holds the searchable map.

  • A good ledger row should include:

    • Run identity: The row should record a unique run ID, parent ID, candidate family, timestamp, and agent identity when multiple agents are involved.
    • Primary result: The row should record the main score, such as validation BPB, accuracy, pass rate, or reward.
    • Cost profile: The row should record wall-clock time, tokens processed, context tokens, model calls, tool calls, peak memory, and throughput.
    • Status: The row should distinguish evaluated runs from syntax errors, import errors, NaN failures, out-of-memory failures, timeouts, parser failures, leakage failures, and metric-integrity failures.
    • Change summary: The row should include a short description of the edit, such as “reduced depth and increased batch size,” “added parser repair,” or “changed retrieval query construction.”
    • Pointers: The row should link to the source snapshot, diff, metrics file, trace directory, logs, and notes.

  • The ledger can be treated as the agent’s high-level memory:

    \[L_t = \{(i, p_i, s_i, q_i, \kappa_i, \rho_i)\}_{i=1}^{t}\]
    • where \(i\) is the run ID, \(p_i\) is the parent ID, \(s_i\) is the main score, \(q_i\) is the cost vector, \(\kappa_i\) is the edit family, and \(\rho_i\) points to the full record.

Traces

  • Traces are the most important memory object for harness optimization. A scalar score can say that a harness failed, but the trace explains whether it failed because retrieval missed the right example, the model reasoned incorrectly, the parser rejected a correct answer, a tool call timed out, or the harness stopped too early.

  • A trace should preserve the causal chain:

    \[x \rightarrow H \rightarrow p_0 \rightarrow y_0 \rightarrow a_0 \rightarrow o_0 \rightarrow s_1 \rightarrow \dots \rightarrow r\]
    • where \(x\) is the task, \(H\) is the harness, \(p_t\) is the model input at step \(t\), \(y_t\) is the model output, \(a_t\) is an action or tool call, \(o_t\) is the observation, \(s_t\) is harness state, and \(r\) is the reward.

  • Trace fields should include:

    • Task metadata: The trace should identify the task family, difficulty, input length, expected output format, and evaluation rubric without exposing held-out labels to the proposer when that would create leakage.
    • Context construction: The trace should store the system prompt, developer instructions, examples, retrieved passages, memory summaries, and any schema shown to the model.
    • Model interaction: The trace should preserve model outputs, sampling parameters, retry attempts, stop reasons, and token counts for each model call.
    • Tool interaction: The trace should preserve commands, arguments, outputs, exit codes, timeouts, and files touched by tool calls.
    • State updates: The trace should log how memory, scratch state, retrieved context, intermediate answers, and verifier state changed after each step.
    • Parsing and verification: The trace should record parser decisions, repair attempts, verifier calls, extracted answers, and final grading status.
    • Costs: The trace should record latency, context tokens, output tokens, number of calls, and any budget violations.

  • MemGPT: Towards LLMs as Operating Systems by Packer et al. (2023) is relevant because it frames memory management as an explicit systems problem for LLM agents, which is the same kind of design pressure that trace-rich autoresearch systems expose when optimizing harness behavior.

Summaries

  • Summaries are useful for navigation, but they should never replace raw evidence. A summary can help the agent decide which run to inspect, while the raw trace lets it verify what actually happened.

  • A good memory system should maintain three summary levels:

    • Run summary: Each candidate should have a short note describing the hypothesis, edit, result, and interpretation.
    • Family summary: Each edit family should have a rolling summary of patterns, such as which optimizer changes improved BPB, which retrieval changes caused regressions, or which parser changes reduced invalid outputs.
    • Frontier summary: The best candidates should have a concise comparison that explains why each candidate remains on the frontier and what tradeoff it represents.

  • The risk is that summaries compress away the exact diagnostic signal needed for credit assignment. Meta-Harness explicitly argues that short feedback templates, scalar scores, and compressed summaries are poorly matched to harness engineering because they remove information needed to connect downstream failures to earlier harness decisions.

Retrieval

  • As the archive grows, the agent cannot read everything. It needs retrieval over prior experiments. Retrieval should combine structured queries over the ledger with text search over logs, notes, diffs, and traces.

  • Useful retrieval modes include:

    • Best-run retrieval: The agent should quickly retrieve the current best candidate, the best candidate by cost-adjusted score, and the best candidate in each edit family.
    • Regression retrieval: The agent should retrieve candidates that worsened the metric after similar edits, because regressions often explain what not to repeat.
    • Failure retrieval: The agent should search for repeated failure types such as NaNs, out-of-memory errors, parser failures, invalid JSON, stale metrics, or timeouts.
    • Similarity retrieval: The agent should retrieve prior candidates with similar diffs, architectures, optimizer settings, retrieval policies, or prompt templates.
    • Task-level retrieval: For harness search, the agent should retrieve all traces for a task family or all examples where a candidate failed but a baseline succeeded.
    • Anomaly retrieval: The agent should retrieve runs with suspicious improvements, unusual throughput, unexpectedly low context use, or mismatched evaluator hashes.

  • Retrieval can be formalized as selecting a subset of evidence \(E_t\) from the full archive \(\mathcal{D}_t\):

    \[E_t = \mathrm{Retrieve} \left( q_t,\mathcal{D}_t,k \right)\]
    • where \(q_t\) is the agent’s current diagnostic query and \(k\) controls the amount of evidence retrieved. The important design choice is that the agent can issue new queries interactively rather than being forced to accept a fixed summary chosen by the outer loop.

Credit

  • Memory exists to support credit assignment. The agent needs to infer which changes caused an improvement or regression. This is difficult because a single candidate may modify several interacting components.

  • A practical credit-assignment workflow should include:

    • Parent comparison: The agent should compare each candidate against its direct parent so that the local effect of the edit can be estimated.
    • Sibling comparison: The agent should compare candidates that share the same parent but differ in one component, which is useful for isolating effects.
    • Ablation comparison: When a winning candidate changes multiple components, the system should run follow-up candidates that remove or isolate each component.
    • Trace comparison: For harnesses, the agent should compare per-task traces before and after the edit, not only aggregate scores.
    • Cost comparison: The agent should check whether a quality gain came from more computation, more context, more retries, or genuinely better behavior.
    • Failure-mode comparison: The agent should compare how the distribution of errors changed, such as fewer parse failures but more reasoning failures.

  • If a candidate changes components \(a\), \(b\), and \(c\), the improvement cannot be safely attributed to the whole bundle. A minimal ablation plan evaluates \(H_{abc},\quad H_{ab},\quad H_{ac},\quad H_{bc},\quad H_a,\quad H_b,\quad H_c\) when the budget allows. In practice, autoresearch systems can use cheaper partial ablations first and reserve full evaluations for candidates that remain promising.

Curves

  • Curves are more informative than final scores. A final validation BPB might hide that a candidate learned faster early but plateaued worse, or learned slowly but was still improving at the time limit.

  • For training-code autoresearch, useful curves include:

    • Training loss over time: This curve shows whether a candidate is learning smoothly, diverging, or underfitting.
    • Validation BPB over time: This curve reveals whether the final score is stable or whether the candidate was lucky at the last checkpoint.
    • Tokens per second over time: This curve catches warmup effects, compiler effects, dataloader stalls, and memory-related slowdowns.
    • Gradient norm over time: This curve helps diagnose instability, optimizer misconfiguration, or overly aggressive learning rates.
    • Learning rate over time: This curve verifies that schedule changes behave as intended under different step counts.

  • For harness autoresearch, useful curves and series include:

    • Reward by task order: This series reveals whether failures cluster by task family, difficulty, or context length.
    • Context tokens by task: This series catches candidates that improve quality by spending much more context on hard examples.
    • Model calls by task: This series shows whether retries or decomposition loops are driving gains.
    • Failure type by iteration: This series reveals whether the harness is trading one failure mode for another.
    • Latency by task: This series identifies long-tail tasks where the harness gets stuck.

Compression

  • As memory grows, the system needs compression without losing auditability. The right pattern is layered compression: raw evidence remains on disk, while derived summaries, embeddings, and indexes make it easier to navigate.

  • Context compaction should modify the model-visible working state rather than become the only surviving copy of history. The durable archive and the active context should therefore be treated as separate layers, allowing lossy context transformation without destroying provenance or preventing later recovery.

  • Compaction should be transactional: generate the candidate compact state, verify required identifiers, unresolved tasks, structural relationships, and budget constraints, and only then replace the active state. If validation fails, the previous state should remain intact. This mirrors OpenClaw’s safeguard behavior, which refuses to commit an invalid summary and preserves the original history.

  • A good compression strategy should include:

    • Immutable raw logs: Raw evidence should remain available so that summaries can be checked.
    • Structured extraction: The system should extract metrics, statuses, failure types, token counts, and costs into machine-readable tables.
    • Text summaries: The system should create concise run summaries and family summaries for quick review.
    • Search indexes: The system should index diffs, notes, logs, prompts, and traces so the agent can search by concept or keyword.
    • Frontier views: The system should maintain a compact view of Pareto-optimal candidates and why they matter.
    • Staleness markers: Summaries should record which raw evidence they were derived from so the system knows when they are outdated.

  • Compression should preserve reversibility at the decision level. That means every summary claim should point back to the run IDs, trace IDs, or metrics that support it.

Decision-aware consolidation and certified forgetting

Why fixed compression rules are insufficient

  • A simple memory implementation usually compacts when a threshold is reached: summarize after \(N\) turns, retain the most recent \(K\) events, discard low-salience records, or compress once the active context exceeds a token limit. These rules control memory size, but they do not directly measure the quantity that ultimately matters to an agent: whether compression changes a later decision. Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) formulates bounded memory around this distinction, treating histories as safely mergeable only when collapsing them into one memory state preserves near-optimal downstream decisions.

  • What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) places this decision inside the broader memory-compaction problem: a memory operator should spend its finite budget on information that preserves task utility, preferably without committing irreversibly before the future query is known.

  • For autoresearch, a fixed summarization threshold is particularly problematic because experiment history is highly non-uniform. Ten nearly identical out-of-memory failures may need little active representation, while one unusual regression may contain the only evidence that a promising architecture becomes numerically unstable after several thousand steps. Age, textual similarity, and frequency are therefore useful retrieval signals, but they are not sufficient criteria for forgetting.

Forgetting as a decision-equivalence test

  • Let \(h\) denote an experiment history, \(q\) the current research question, and \(a\) a possible next research action. Define:

    \[\mu_q(h,a)\]

    • as the expected utility of action \(a\) after history \(h\), and:

      \[\mu_q^{\star}(h) = \max_{a\in\mathcal{A}} \mu_q(h,a)\]

      • as the utility obtainable when the full history is available. The decision loss associated with choosing action \(a\) is:

        \[\Delta_q(h,a) = \mu_q^{\star}(h) - \mu_q(h,a)\]

  • Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) uses this downstream loss, rather than reconstruction or semantic similarity, as the distortion induced by memory compression.

  • Suppose several histories are candidates for consolidation into one memory state \(C\). They can be merged at tolerance \(\epsilon\) when there exists one action that remains approximately optimal for all of them:

\[\exists a\in\mathcal{A} \quad\text{s.t.}\quad \max_{h\in C} \Delta_q(h,a) \le \epsilon\]

  • This is the exact forgetting boundary in Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026): if such an action exists, distinguishing the histories is unnecessary for the current decision tolerance; if no such action exists, merging them necessarily destroys a decision-relevant distinction.

  • For autoresearch, the criterion has a natural interpretation. Consider two experiment records:

    • Run A: A depth-12 model causes an out-of-memory failure before the first optimizer step, suggesting that the next intervention should reduce memory pressure, for example by lowering batch size, sequence length, or model memory use.

    • Run B: A depth-12 model trains successfully, but validation BPB regresses because lower throughput reduces the number of optimizer updates completed within the fixed budget, suggesting that the next intervention should address the compute-quality tradeoff rather than memory capacity.

  • A descriptive memory system may collapse both into “depth 12 performed poorly.” A decision-aware memory should preserve the distinction because the two histories imply different next experiments. They are descriptively similar but decision-incompatible.

  • Conversely, dozens of runs that differ in exact CUDA allocator messages but all imply the same corrective action may safely share a compact state. The objective is therefore not to maximize descriptive fidelity indiscriminately, but to preserve the partition of history that matters for future intervention.

Pairwise decision distance

  • A useful local test is the decision distance between two histories:
\[d_{\mathrm{dec}}^q(h,h') = \min_{a\in\mathcal{A}} \max \left\{ \Delta_q(h,a), \Delta_q(h',a) \right\}\]

  • If \(d_{\mathrm{dec}}^q(h,h') \le \epsilon\), then there exists an action that is within \(\epsilon\) of optimal for both histories, so the pair can share one memory state at that tolerance. If:

    \[d_{\mathrm{dec}}^q(h,h') > \epsilon\]

  • This distinction also clarifies why embedding distance is only a proxy. Semantic embeddings answer approximately:

    \[\text{“Do these histories describe similar things?”}\]

    • whereas the memory controller needs to answer:

      \[\text{“Would forgetting the difference change what I should do next?”}\]
      • The two questions can correlate, but they are not equivalent.

Learning when two memories cannot be merged

  • In practice, the true action values \(\mu_q(h,a)\) are unknown. DeMem therefore estimates them from observed feedback and delays irreversible refinement decisions until sufficient evidence exists. For an observed context-action pair, let:

    \[\hat{\mu}_t(x,a)\]

    • denote the empirical mean reward after:

      \[n_t(x,a)\]

      • observations. A confidence radius can be constructed as:

        \[c_t(x,a) = \sqrt{ \frac{ \log \left( 4NA t^2/\delta \right) }{ 2n_t(x,a) } }\]
        • where \(N\) is the number of contexts, \(A\) is the number of actions, and \(\delta\) controls the probability that the simultaneous confidence guarantee fails.

  • This gives confidence bounds:

    \[\mathrm{UCB}_t(x,a) = \min \left\{ 1, \hat{\mu}_t(x,a) + c_t(x,a) \right\}\] \[\mathrm{LCB}_t(x,a) = \max \left\{ 0, \hat{\mu}_t(x,a) - c_t(x,a) \right\}\]

  • From these intervals, the memory controller can construct lower and upper certificates for decision distance. With probability at least \(1-\delta\), the resulting certificate satisfies:

    \[\underline{d}_t(x,x') \le d_{\mathrm{dec}}(x,x') \le \overline{d}_t(x,x')\]

  • Once the lower confidence bound certifies:

    \[\underline{d}_t(x,x') > \epsilon\]
    • contexts \(x\) and \(x'\) receive a cannot-link relation: the available evidence is sufficient to conclude that the pair should not occupy the same low-distortion memory state.

  • The system can represent these constraints as a graph:

    \[G_{\epsilon} = (V,E_{\epsilon})\]

    • with:

      \[(x,x') \in E_{\epsilon} \iff \underline{d}_t(x,x') > \epsilon\]

  • Each vertex represents a memory context, and each edge denotes a certified decision incompatibility.

  • Absence of an edge does not prove that two histories are equivalent. It means only that the system does not yet possess sufficient evidence to certify their incompatibility. This asymmetry is important because aggressive splitting permanently consumes memory capacity. DeMem therefore uses cannot-link edges as conservative constraints rather than interpreting every non-edge as evidence that histories should be merged.

  • An autoresearch memory controller should adopt the same discipline. Lack of evidence that two experiments require different follow-up actions should not automatically be interpreted as evidence that they are interchangeable.

Enforcing a finite memory budget

  • If only \(K\) active memory states are allowed, the cannot-link graph induces a constrained partitioning problem. Each memory state corresponds to a cluster:

    \[\mathcal{P} = \left\{ C_1,\dots,C_K \right\}\]
    • such that histories connected by certified incompatibility should not occupy the same cluster.

  • Exact graph coloring is computationally difficult in general, so DeMem uses a polynomial-time greedy coloring procedure based on graph degeneracy. For a candidate decision-distance threshold \(\alpha\), it searches for the smallest level satisfying:

    \[\operatorname{degen} \left( G_{\alpha} \right) + 1 \le K\]
    • and greedily colors the resulting graph with at most \(K\) states.

  • The important systems principle is broader than the particular graph algorithm: when memory is finite, the controller should spend additional states on distinctions for which there is evidence of downstream conflict. It should not divide memory uniformly among all experiences.

Cluster-level distortion

  • Pairwise compatibility alone is insufficient because decision distance need not obey the triangle inequality. A set of histories can contain no pairwise certified conflict while still lacking one action that works well for the entire cluster. DeMem therefore evaluates the realized cluster radius:
\[\rho_{\mathrm{dec}}^q(C) = \min_{a\in\mathcal{A}} \max_{h\in C} \Delta_q(h,a)\]

  • The memory partition’s distortion is:

    \[\epsilon(\mathcal{P}) = \max_{C\in\mathcal{P}} \rho_{\mathrm{dec}}^q(C)\]
    • and its certified approximation is used to measure the actual compression cost of the learned partition rather than assuming that graph independence implies zero conflict.

  • This distinction is useful for autoresearch because a summary can appear internally consistent pairwise while still mixing too many mechanisms to support one reliable conclusion. For example, several optimizer regressions may individually resemble one another but jointly involve instability, undertraining, and memory bottlenecks that require separate interventions.

Adaptive refinement rather than continual rewriting

  • A decision-aware memory controller should refine its state only when new evidence justifies the extra capacity. In simplified form:
def update_memory(history, feedback, K, epsilon):
    update_action_value_estimates(history, feedback)
    update_confidence_bounds()

    conflicts = set()

    for h1, h2 in observed_history_pairs():
        if lower_decision_distance(h1, h2) > epsilon:
            conflicts.add((h1, h2))

    partition = budgeted_partition(
        conflicts=conflicts,
        max_states=K,
    )

    return partition

  • This differs from an unconditional “summarize every \(N\) steps” loop. The compression boundary is driven by observed decision conflict, while the number of available states remains explicitly bounded.

  • The DeMem execution cycle can be summarized as:

observe interaction
        ↓
route into current memory state
        ↓
take action and receive feedback
        ↓
update decision-value evidence
        ↓
certify incompatible histories
        ↓
repartition under K-state budget
        ↓
continue

  • Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) implements this process in epochs, freezing the partition during each epoch and rebuilding it from accumulated conflict evidence between epochs.

  • The following figure (source) shows the synthetic evaluation of decision-aware memory: cumulative regret over interaction rounds, the memory-budget-versus-regret frontier as the number of available slots changes, and performance as the mismatch between descriptive similarity and decision similarity increases.

  • In the synthetic Decoupled Bandit experiments, DeMem achieves lower cumulative regret than the other bounded-memory methods, obtains a better memory-distortion tradeoff as \(K\) varies, and gains relative advantage as descriptive similarity becomes increasingly misaligned with the action-relevant structure of the problem. The experiment is designed specifically to distinguish memory based on surface similarity from memory based on downstream decision compatibility.

Separating unavoidable compression loss from memory-controller error

  • A useful property of the decision-rate-distortion formulation is that it separates limitations caused by the memory budget from limitations caused by the memory implementation. If:

    \[\epsilon_{\infty}^{\star}(K)\]
    • is the optimal distortion achievable with \(K\) memory states, then even an ideal memory mechanism cannot eliminate this term when the budget forces decision-relevant histories together.

  • For a concrete slot-based implementation, realized distortion can be decomposed approximately as:

    \[\epsilon_{\mathrm{realized}} \lesssim \epsilon_{\infty}^{\star}(K) + \eta_{\mathrm{route}} + \eta_{\mathrm{read}}\]

    • where:

      • \(\epsilon_{\infty}^{\star}(K)\) is the unavoidable compression floor imposed by the finite state budget;
      • \(\eta_{\mathrm{route}}\) measures additional loss from routing a query to the wrong memory state;
      • \(\eta_{\mathrm{read}}\) measures additional loss from imperfectly materializing or interpreting the selected memory state.

  • Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) uses this decomposition to connect the abstract \(K\)-state formulation with practical slot-based language-agent memory.

  • This decomposition gives an autoresearch agent a sharper diagnostic loop. If performance is limited primarily by \(\eta_{\mathrm{route}}\), the next experiment should modify indexing or retrieval. If \(\eta_{\mathrm{read}}\) dominates, the memory representation or prompt materialization should change. If both are small but performance still degrades as \(K\) shrinks, the system may simply be operating below the memory capacity required by the task.

Compression cost and learning cost

  • Memory optimization also introduces a statistical-learning problem. Under bounded memory, DeMem’s regret upper bound has the form:

    \[\operatorname{Reg}(T) \le T\cdot O \left( \bar{\epsilon}^{\mathrm{cert}}_T \right) + \widetilde{O} \left( \sqrt{AKT} \right) + O \left( AN_T B_T(\gamma) \right)\]
    • where the three terms correspond respectively to realized compression distortion, statistical learning over \(K\) memory states and \(A\) actions, and the exploration required to certify conflicts.

  • The corresponding minimax lower bound contains:

    \[\Omega \left( T\epsilon_{\infty}^{\star}(K) + \sqrt{AKT} \right)\]
    • showing that finite memory and statistical uncertainty are distinct sources of difficulty. DeMem matches the statistical term up to logarithmic factors, while any remaining excess depends on how closely its learned partition approaches the optimal memory-distortion frontier and on the cost of obtaining certificates.

  • For autoresearch, this means memory should not be evaluated solely by whether its final summary looks sensible. A memory-management policy consumes resources to learn what should be retained: additional evaluator calls, retrieval operations, comparisons, verifier judgments, or task outcomes may be necessary before the system can distinguish safe compression from decision-critical conflict.

A decision-aware autoresearch memory controller

  • A practical autoresearch system can implement these ideas without literally estimating a full action-value table. The formal criterion can instead motivate a hierarchy of increasingly expensive evidence:

    • Cheap descriptive filter: Use metadata, embedding similarity, edit family, metric similarity, and error type to identify memories that may be redundant.
    • Trace-level comparison: Inspect the underlying runs to determine whether the apparent similarity has the same causal mechanism.
    • Counterfactual decision check: Ask whether seeing one record rather than the other would change the proposed experiment.
    • Conflict certification: Preserve both memories when their implied interventions or risk assessments differ materially.
    • Consolidation: Merge records only when their distinctions no longer change a relevant downstream research decision.
    • Archive backing: Keep the original source, diff, metrics, and traces recoverable even after their active representation has been consolidated.

  • One practical interface is:

def can_consolidate(run_a, run_b, research_goal):
    # Fast candidate screen.
    if not descriptively_related(run_a, run_b):
        return False

    evidence_a = inspect_raw_evidence(run_a)
    evidence_b = inspect_raw_evidence(run_b)

    action_a = propose_next_action(evidence_a, research_goal)
    action_b = propose_next_action(evidence_b, research_goal)

    # Preserve the distinction if it changes the research decision.
    if materially_different(action_a, action_b):
        return False

    # Raw records remain archived even when their active
    # representation is consolidated.
    return True
  • The important point is that semantic similarity appears only in the inexpensive first stage. Final consolidation is governed by whether the distinction affects downstream action.

Promotion, demotion, and multi-fidelity memory

\[\begin{aligned} \text{hot working memory} &\rightarrow \text{current hypotheses} \rightarrow \text{recent failures} \rightarrow \text{active frontier} \\ &\rightarrow \text{compact semantic memory} \rightarrow \text{family summaries} \rightarrow \text{discovered patterns} \rightarrow \text{reusable lessons} \\ &\rightarrow \text{reversible episodic archive} \rightarrow \text{source snapshots} \rightarrow \text{diffs} \rightarrow \text{metrics} \rightarrow \text{traces and logs} \\ &\rightarrow \text{cold provenance store} \rightarrow \text{older redundant evidence} \rightarrow \text{reproducibility} \rightarrow \text{audit} \end{aligned}\]

  • Promotion into active memory should depend on expected usefulness for the current research decision:

    \[\mathrm{Promote}(z,q) \iff \widehat{\Delta U}(z\mid q) > C_{\mathrm{retrieve}}(z)\]
    • where \(\widehat{\Delta U}(z\mid q)\) estimates the improvement in decision quality from restoring memory item \(z\) for query \(q\) and \(C_{\mathrm{retrieve}}\) measures its context, latency, or compute cost.

  • Demotion should be conservative when information cannot be reconstructed. A raw trace can safely leave active context when it remains retrievable from the archive; deleting the only copy requires a substantially stronger criterion.

Compaction stopping rules

  • A fixed compression ratio answers “how much memory should be removed?” before knowing whether further removal is safe. A better controller stops compaction when estimated marginal distortion exceeds an allowed threshold:
\[\widehat{\Delta D}_{t+1} > \tau_D \quad \Rightarrow \quad \text{stop compacting}\]

  • What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) highlights the absence of such calibrated stop rules for agent summarization and points to error-bounded KV-cache methods as a design pattern that could transfer upward to semantic memory.

  • In an autoresearch system, the stop rule could be approximated by testing whether another consolidation step changes any of the following:

    • the current best candidate;
    • the ranking of Pareto-frontier candidates;
    • the inferred cause of a major regression;
    • the proposed next experiment;
    • the estimated risk of a candidate;
    • the ability to reconstruct the provenance of a retained claim.

    If further compression changes one of these decisions materially, the current memory representation has crossed from redundancy removal into decision-relevant information loss.

Practical policy

  • A robust memory controller for autonomous research should therefore follow one principle: compress descriptions aggressively, but compress decision distinctions conservatively.

  • The operational policy is:

\[\begin{aligned} \text{retain raw evidence reversibly} &\rightarrow \text{extract structured facts and summaries} \rightarrow \text{group apparently redundant memories} \rightarrow \text{test whether distinctions change downstream decisions} \\ &\rightarrow \text{preserve certified conflicts} \rightarrow \text{consolidate decision-equivalent records} \rightarrow \text{retrieve raw evidence when later needed} \end{aligned}\]
  • This architecture complements the existing archive, ledger, summary, and retrieval components rather than replacing them. The archive remains the high-fidelity reversible layer; summaries and indexes provide cheap navigation; decision-aware consolidation determines which distinctions deserve scarce active-memory capacity; and retrieval restores high-fidelity evidence when a later research decision requires it.

Sharing

  • In multi-agent autoresearch, memory is shared infrastructure. The system should prevent agents from duplicating work, overwriting results, or drawing conclusions from incomplete runs.

  • Shared memory should support:

    • Reservations: Agents should claim run IDs and parent candidates before editing so that concurrent work does not collide.
    • Run states: Each run should have a state such as planned, running, evaluated, invalid, rejected, or frontier.
    • Locks: The evaluator should lock active run directories while writing metrics and logs.
    • Conflict detection: The system should detect when two agents propose near-identical edits or branch from stale parents.
    • Frontier updates: A curator or automated process should update the frontier only after metric integrity checks pass.
    • Notifications: Agents should be able to see when a new best candidate appears so they can rebase future work.

  • A shared archive turns independent agents into a research organization. The shared memory is the medium through which agents coordinate, avoid repeated mistakes, and compound discoveries.

Retention

  • Not all memory has the same long-term value, but deletion should be conservative. Failed runs often become useful later when a new agent wants to understand instability boundaries.

  • A practical retention policy should be:

    • Keep all frontier candidates permanently: These are the best known tradeoffs and should remain reproducible.
    • Keep all suspicious improvements permanently: These are needed for audit, even if later rejected.
    • Keep representative failures: The archive should preserve examples of each failure type so agents can learn from them.
    • Downsample redundant failures: If hundreds of candidates fail with the same syntax error or identical out-of-memory trace, the system can compress them into a family summary while retaining representative raw logs.
    • Preserve parent chains: Any candidate that influenced a frontier candidate should be kept because it supports provenance.
    • Separate cold storage from active memory: Older raw logs can move to cold storage, while summaries and indexes stay active.

  • Retention is part of scientific integrity. A system that keeps only the winners is easier to fool because it loses the negative evidence needed to understand why the winners worked.

  • Autoresearch is a search process over executable research artifacts. The search space may contain model architectures, optimizer configurations, training loops, retrieval policies, prompt builders, parser logic, tool-use policies, memory-update rules, and complete agent harnesses. The central design question is how to move through that space efficiently while preserving scientific interpretability.

Search unit

  • The search unit should be a coherent candidate that can be evaluated end to end. In single-GPU training, the candidate is usually the editable training file. In harness optimization, the candidate is the harness code that controls model calls, context construction, retrieval, parsing, and state updates. In agentic coding, the candidate may include tool policies, terminal interaction rules, retry logic, and task-submission behavior.

  • A candidate can be represented as:

    \[c_t = (a_t, h_t, o_t, e_t)\]
    • where \(a_t\) is the architecture or algorithmic structure, \(h_t\) is the hyperparameter configuration, \(o_t\) is the orchestration logic, and \(e_t\) is the evaluator-facing interface. The evaluator should only accept candidates that preserve \(e_t\), because changing the interface or metric contract makes candidates incomparable.

  • A good search unit should satisfy the following properties:

    • Executable: The candidate should be runnable by the evaluator without manual intervention, and invalid candidates should fail early through validation gates rather than during expensive evaluation.
    • Comparable: The candidate should be evaluated under the same metric, budget, task set, and interface contract as other candidates in its comparison group.
    • Inspectable: The candidate should be stored as source code plus a diff against its parent so future agents can understand exactly what changed.
    • Attributable: The candidate should test one coherent hypothesis or a tightly coupled bundle of changes, which makes it easier to infer why the result changed.
    • Reusable: Useful subcomponents, such as a better schedule, safer parser, or faster retrieval filter, should be easy to transplant into future candidates.

Search modes

  • Autoresearch should mix several search modes because different stages of the research process require different kinds of exploration, as indicated below:

    • Local refinement: The agent makes a small edit to the current best candidate, such as adjusting a learning rate, changing a retry threshold, reducing parser strictness, or modifying the prompt format. This mode is efficient when the current design is already strong and failures are localized.
    • Branching exploration: The agent starts from a non-best candidate that contains an interesting idea, such as a faster architecture that underperformed because of an unstable schedule. This mode prevents promising mechanisms from being discarded too early.
    • Component transplant: The agent extracts one useful subsystem from a candidate and inserts it into a stronger parent. This is useful when a candidate regressed overall but improved one measurable submetric.
    • Ablation: The agent removes or isolates a component from a winning candidate to test whether the component actually caused the improvement. This mode turns a discovered improvement into a more trustworthy result.
    • Structural rewrite: The agent replaces a whole subsystem, such as the retrieval policy, memory representation, optimizer partitioning, or tool loop. This mode is risky but necessary when traces show that the current structure is misaligned with the task.
    • Frontier expansion: The agent searches for candidates that are not best on the headline score but offer a useful tradeoff, such as lower context cost, fewer model calls, better latency, or lower memory use.

Proposal policy

  • The proposal policy decides what candidate to try next. A simple policy samples changes from the current best. A stronger policy conditions on the full archive, including failures, traces, diffs, and frontier candidates.

  • The proposal distribution can be written as:

    \[c_{t+1} \sim \pi_{\phi} \left( c \mid \mathcal{D}_t, \mathcal{P}_t, g \right)\]
    • where \(\mathcal{D}_t\) is the archive of prior runs, \(\mathcal{P}_t\) is the current Pareto frontier, and \(g\) is the current research goal. The policy can be implemented by a coding agent that reads the archive, forms a hypothesis, edits the candidate artifact, and records a rationale.

  • Large Language Models as Optimizers by Yang et al. (2023) is relevant because OPRO shows that language models can propose improved solutions from a history of candidate values, but autoresearch generalizes this idea from natural-language prompt candidates to executable code candidates.

Parent choice

  • Parent choice determines which prior candidate the next edit starts from. Greedy search always edits the current best candidate, but this can prematurely discard diverse mechanisms. Population-based search maintains multiple possible parents.

  • A softmax parent-selection rule is:

    \[P(i) = \frac{ \exp(-s_i/\tau) }{ \sum_j \exp(-s_j/\tau) }\]
    • where \(s_i\) is the score of candidate \(i\) and lower is better. The temperature \(\tau\) controls exploration: small \(\tau\) favors the best candidate, while larger \(\tau\) gives more probability to diverse candidates.

  • Parent choice should consider more than the headline score:

    • Score quality: Strong candidates should be sampled often because they represent the best known designs.
    • Cost profile: A candidate with slightly worse quality but much lower memory, context, or latency may be a valuable parent.
    • Novelty: A candidate from an underexplored edit family may deserve further search even if its first result was not best.
    • Failure potential: A failed candidate can be a useful parent if the failure is clearly repairable, such as out-of-memory caused by a batch-size choice rather than a flawed idea.
    • Trace evidence: A candidate that improved a specific failure class, such as parser failures or retrieval misses, may be worth extending even if aggregate accuracy did not improve.

Mutation design

  • A mutation is an edit to a candidate. In autoresearch, mutations should be semantically meaningful rather than random text changes. The best mutations are small enough to evaluate cleanly but large enough to express a real research idea.

  • Useful mutation families include:

    • Parameter mutation: The agent changes numeric values such as learning rate, warmup fraction, weight decay, batch size, sequence length, retrieval top-\(k\), parser thresholds, or retry limits.
    • Structural mutation: The agent changes model depth, attention pattern, optimizer grouping, retrieval stages, memory representation, prompt sections, tool-routing logic, or verification loops.
    • Schedule mutation: The agent changes learning-rate schedules, evaluation cadence, retry schedules, curriculum order, or context-expansion policies.
    • Interface mutation: The agent changes how information is presented to the model, such as example formatting, schema strictness, chain decomposition, retrieved-context ordering, or final-answer extraction.
    • Safety mutation: The agent adds guards such as NaN checks, parser validation, timeout handling, budget enforcement, fallback behavior, or evaluator-integrity checks.
    • Efficiency mutation: The agent changes computation layout, batching, caching, retrieval precomputation, prompt compression, or tool-call minimization.

  • AlphaEvolve by Novikov et al. (2025) is relevant because it frames LLM-based coding agents as part of an evolutionary loop that proposes direct code changes, receives evaluator feedback, and iteratively improves algorithms.

The final substantial insertion should go in ## Search, immediately after the existing ### Mutation design subsection and before ### Crossover. This is the cleanest location because Mutation design already identifies memory representation, prompt compression, retrieval, caching, and context-expansion policies as mutable components, while Crossover then explains how useful subsystems can be combined across candidates.

Searching over memory policies and compaction operators

  • Memory-policy search should distinguish selection mutations from generation mutations. Selection changes which existing items survive, whereas generation changes how the historical state is encoded into a new representation; because generation can be strictly more compact for some query families, the two should be treated as separate search families rather than variations of the same compression ratio.

Memory management as an explicit search space

  • Once memory is treated as executable harness behavior rather than fixed infrastructure, the outer loop can optimize the memory policy itself. Meta-Harness: End-to-End Optimization of Model Harnesses by Lee et al. (2026) searches over harness code that determines what information is stored, retrieved, and presented to a fixed language model, making memory-management logic a natural candidate for code-space optimization.

  • What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) further shows that apparently different mechanisms such as KV eviction, quantization, prompt compression, bounded recurrent state, and agent-memory consolidation can all be understood as operators that trade retained information against downstream task distortion.

  • The candidate searched by autoresearch can therefore include a memory policy:

    \[\mathcal{M} = \left( C, R, A, F, S \right)\]
    • where \(C\) is the compaction operator, \(R\) is the retrieval policy, \(A\) is the memory-budget allocation policy, \(F\) is the fidelity policy, and \(S\) is the stopping or consolidation policy.

  • A complete harness candidate can then be written as:

    \[H = \left( P, T, V, \mathcal{M} \right)\]
    • where \(P\) is prompt construction, \(T\) is tool orchestration, \(V\) is parsing or verification, and \(\mathcal{M}\) is memory management. The outer-loop search is consequently free to improve the information substrate on which the model acts without modifying the model weights.

The main memory-policy search dimensions

  • A useful memory search space should expose several logically separate design dimensions rather than one global compression_ratio parameter:

    • Query conditioning: Decide whether information is selected before the future query is known or selected dynamically after the current information need becomes available.
    • Reversibility: Decide whether content removed from active memory remains recoverable from an external archive or is permanently discarded.
    • Fidelity: Decide which memories remain verbatim, which receive compact representations, and which are summarized aggressively.
    • Budget allocation: Decide how total capacity is distributed across memory types, positions, layers, heads, tasks, or experiment families.
    • Compaction trigger: Decide when compression occurs, such as at fixed token thresholds, task boundaries, latency thresholds, or predicted memory pressure.
    • Retrieval policy: Decide how archived state returns to active context, including semantic, temporal, structural, task-conditioned, and decision-conditioned retrieval.
    • Consolidation policy: Decide when multiple episodic records are abstracted into a reusable semantic memory.
    • Stopping rule: Decide when further compression is likely to produce more downstream distortion than the resource saving justifies.

  • What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) identifies query conditioning, reversibility, fidelity, allocation, forgetting, and calibrated stopping as recurring design dimensions across the memory hierarchy, suggesting that mechanisms developed at one layer can often inspire candidates at another.

Search over query conditioning

  • Query conditioning is a particularly high-leverage mutation because memory relevance depends on what the system is currently trying to answer. Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference by Tang et al. (2024) demonstrates this principle at the KV-cache layer by estimating page importance from the current query and loading only the critical cache pages required for attention.

  • At the agent-memory layer, the corresponding mutation is to replace eager inclusion or static top-\(k\) retrieval with a query-conditioned retrieval policy:

    \[E_t = R_{\phi} \left( Q_t, \mathcal{D}_t, B_t \right)\]
    • where \(\mathcal{D}_t\) is the complete archive, \(Q_t\) is the current decision problem, and \(B_t\) is the amount of memory that can be restored into active context.

  • The search space can include:

\[R_{\phi} \in \{ \text{recency}, \text{semantic top-}k, \text{hybrid}, \text{temporal}, \text{decision-aware}, \text{learned router} \}\]
  • For autoresearch, this means the proposer need not receive the same historical summary for every experiment. A proposal about optimizer instability can retrieve prior optimizer failures; a proposal about retrieval can retrieve previous retrieval traces; a proposal about throughput can retrieve hardware and profiling evidence. The relevant memory state becomes a function of the research question rather than a static snapshot of the entire search history.

Search over reversible versus irreversible memory

  • Reversibility should itself be an optimization variable. One candidate might summarize and delete old traces; another might summarize only the active representation while retaining the original evidence in cold storage; another might maintain a searchable episodic archive and construct context entirely at query time.

  • InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management by Lee et al. (2024) illustrates a related principle at the KV level: instead of placing the entire KV cache on the accelerator, it predicts which entries will matter and prefetches the required subset from host memory, preserving a larger backing state while minimizing expensive active-state movement.

  • The analogous agent-memory candidates can be parameterized as:

    \[C_{\mathrm{lossy}} : H \rightarrow Z\]
    • for an irreversible representation, versus:
    \[C_{\mathrm{rev}} : H \rightarrow (Z,A)\]
    • where \(A\) is an archive from which discarded active information can later be recovered.

  • The evaluator should not assume the irreversible candidate is preferable merely because its active context is smaller. Search should compare \(\left(\mathrm{task\ utility},\mathrm{active\ memory},\mathrm{archive\ memory},\mathrm{retrieval\ latency},\mathrm{recovery\ rate}\right)\) across both designs.

Search over memory fidelity

  • Uniform compression assumes every memory item deserves the same representation quality. A stronger search space assigns different fidelity levels:

    \[f_i \in \{ \text{raw}, \text{structured}, \text{summary}, \text{index-only}, \text{discarded} \}\]
    • for each memory item \(z_i\).

  • The optimization problem becomes:

    \[\max_{\{f_i\}} \mathbb{E} \left[ U(\{z_i^{(f_i)}\}) \right]\] \[\text{s.t.} \qquad \sum_i C \left( z_i^{(f_i)} \right) \le B\]
    • where \(C(z_i^{(f_i)})\) is the resource cost of storing item \(i\) at fidelity \(f_i\).

  • This lets the outer loop discover policies such as keeping code diffs and rare failures verbatim, converting routine successful runs into structured metrics, compressing repeated observations into family summaries, and retaining only an index for older low-value material.

  • MIRIX: Multi-Agent Memory System for LLM-Based Agents by Wang and Chen (2025) is relevant because it separates memory into distinct functional types including episodic, semantic, procedural, resource, and core memory, illustrating why different information classes need not share a single storage or retrieval policy.

Search over non-uniform memory budgets

  • The optimal budget need not be distributed uniformly. At the KV layer, Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference by Feng et al. (2024) derives an attention-output error bound and uses it to allocate KV budgets adaptively across attention heads rather than assigning the same cache size to every head.

  • The corresponding autoresearch principle is to budget by marginal task utility. If \(B_i\) is the capacity assigned to memory component \(i\), the ideal allocation approximately satisfies:

    \[\frac{ \partial U }{ \partial B_i } \approx \frac{ \partial U }{ \partial B_j }\]
    • for components receiving interior allocations. Intuitively, an additional unit of memory should be spent where it yields the greatest improvement in expected downstream research quality.

  • Search candidates can therefore vary allocations such as the ones listed below, rather than imposing an equal quota across every memory category:

    • Recent experiments: Allocate about 20% of active memory to recent runs so the agent can reason from the latest evidence and avoid repeating immediately preceding mistakes.

    • Frontier candidates: Reserve roughly 20% for Pareto-frontier candidates so the agent keeps the strongest known quality-cost tradeoffs readily accessible.

    • Failure evidence: Allocate the largest share, around 25%, to informative failures because regressions, crashes, and edge cases often provide the strongest signal about what should not be tried again.

    • Family summaries: Use about 10% for compact summaries of experiment families, giving the agent a high-level view of recurring patterns without consuming much context.

    • Raw retrieved traces: Reserve roughly 20% for high-fidelity traces retrieved on demand when the current hypothesis requires detailed causal evidence.

    • Miscellaneous state: Keep the remaining 5% for auxiliary information such as temporary notes, task metadata, or other context that does not fit the main memory categories.

  • These percentages should themselves be learned or searched. If recent runs rarely influence proposals while old failure traces frequently prevent repeated mistakes, the outer loop should shift capacity accordingly.

Search over compaction triggers

  • A memory mutation should also specify when compaction occurs. Common candidates include:

    \[S_t = \begin{cases} 1, & \text{tokens}_t > B \\ 0, & \text{otherwise} \end{cases}\]

    • for a fixed context threshold,

      \[S_t = \mathbb{1} [ \mathrm{marginal\ memory\ utility} < \tau ]\]
      • for a utility-aware trigger, or a learned policy:
      \[S_t \sim \pi_{\phi} \left( s_t \right)\]
      • that decides whether to retain, summarize, retrieve, fold, or discard based on the current agent state.

  • ACON: Optimizing Context Compression for Long-horizon LLM Agents by Kang et al. (2025) treats compression behavior itself as optimizable by using failure cases where full context succeeds but compressed context fails to revise natural-language compression guidelines, and reports reductions in peak token usage while largely preserving task performance.

  • Scaling Long-Horizon LLM Agent via Context-Folding by Sun et al. (2025) goes further by making working-context management an agent action: a model can branch into a sub-trajectory and fold that trajectory after completing the subtask, with FoldGRPO providing process-level reinforcement learning for the resulting context-management policy.

  • These approaches suggest two distinct autoresearch mutation families:

    • Optimize the compression rule: Keep the agent fixed and search over instructions or algorithms determining what compaction preserves.
    • Optimize the memory-control policy: Let the agent itself decide when to create, retrieve, fold, archive, or consolidate memory and train or search for the policy producing those decisions.

Search over asynchronous consolidation

  • Memory management does not necessarily need to happen on the critical path. A system can preserve raw experience immediately and perform more expensive consolidation when latency is less important.

  • Sleep-time Compute: Beyond Inference Scaling at Test-time by Lin et al. (2025) studies offline precomputation over contexts before queries arrive, demonstrating the broader principle that computation can be moved away from latency-sensitive inference when future information needs can be anticipated.

  • An autoresearch system can search over consolidation schedules such as:

    • Synchronous consolidation: After each experiment, the system immediately summarizes the new evidence before proceeding to the next experiment. This keeps active memory continuously updated, but places consolidation work directly on the critical path.

    • Asynchronous consolidation: After an experiment, the system archives the raw evidence and immediately continues to the next experiment, while summarization and consolidation happen separately in the background.

    • Periodic consolidation: The system accumulates several experiments before consolidating them together, for example by summarizing an entire experiment family after every \(N\) runs rather than rewriting memory after each individual run.

    • Idle-time consolidation: The system performs heavier maintenance when compute or agent capacity is otherwise idle, such as rebuilding retrieval indexes, regenerating semantic summaries, deduplicating memories, or reorganizing long-term storage.

  • This creates a three-way tradeoff among:

    \[\mathrm{decision\ quality}, \qquad \mathrm{critical\ path\ latency}, \qquad \mathrm{consolidation\ compute}\]
    • rather than assuming that every memory update must happen immediately.

Searching over decision-aware memory partitions

  • Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory by Zou et al. (2026) provides an especially useful parameterization for search because it separates the finite-state memory problem into compression, routing, and realization components and refines memory only when observed evidence certifies a decision conflict.

  • A DeMem-style memory mutation exposes parameters such as:

    \[\left( K, \epsilon, \gamma, g, \phi, R \right)\]
    • where \(K\) is the number of runtime states, \(\epsilon\) is the tolerated decision distortion, \(\gamma\) controls certification resolution, \(g\) is the memory-state encoder, \(\phi\) is the router for new histories, and \(R\) determines how each selected state is represented to the language model.

  • These variables need not all be manually specified. The autoresearch loop can search over them while holding the downstream evaluation contract fixed.

  • The relevant loss decomposition is:

    \[D_{\mathrm{realized}} \lesssim \epsilon_{\infty}^{\star}(K) + \eta_{\mathrm{route}} + \eta_{\mathrm{read}}\]
    • where \(\epsilon_{\infty}^{\star}(K)\) is the unavoidable distortion imposed by the finite memory budget, \(\eta_{\mathrm{route}}\) is additional error from selecting the wrong state, and \(\eta_{\mathrm{read}}\) is error from the concrete representation supplied to the model. This decomposition makes routing and memory realization independently searchable rather than conflating every memory failure with insufficient capacity.

Composing multiple compaction operators

  • Real systems rarely use one memory operator in isolation. A harness may summarize dialogue history, retrieve archival episodes, prune retrieved passages, quantize a KV cache, and apply sparse attention within the same inference trajectory.

  • Let:

    \[C_1,C_2,\dots,C_m\]
    • denote compaction operators. The composite system is:
    \[C_{\mathrm{total}} = C_m \circ C_{m-1} \circ \cdots \circ C_1\]

  • It is generally unsafe to assume:

    \[D(C_{\mathrm{total}}) = \sum_{i=1}^{m}D(C_i)\]
    • because the information removed by one operator changes the input distribution seen by the next. Two individually mild operators can remove overlapping or complementary evidence, and applying them in the opposite order can produce a different result.

  • What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) identifies this missing composition map as an open problem: quantization, low-rank projection, eviction, prompt compression, and agent summarization are typically studied independently even though real serving systems can stack them and thereby compound or cancel distortion.

  • This creates a natural autoresearch problem. Rather than assuming one canonical ordering, search can evaluate candidates such as:

    • Retrieve, summarize, then prompt: The system first retrieves relevant historical evidence, compresses it into a shorter representation, and then places that representation into the model context.

    • Retrieve, rerank, summarize, then prompt: The system retrieves a broader candidate set, reranks it for relevance, summarizes the highest-value evidence, and only then constructs the prompt.

    • Summary index with raw-evidence recovery: The system uses compact summaries as an index for navigation, but retrieves the original underlying evidence when a query requires higher fidelity before prompting the model.

    • Retrieve, prompt, then apply KV eviction: The system retrieves relevant textual context and includes it in the prompt, while a lower-level cache policy subsequently reduces the active KV state during inference.

    • Retrieve, compress text, then quantize KV state: The system first retrieves the required evidence, compresses the textual representation to reduce prompt size, and then applies KV-cache quantization as a second compression stage during inference.

    • Episodic archive, semantic consolidation, then query-conditioned recovery: The system preserves detailed episodic records, periodically consolidates them into semantic memory, and restores the relevant high-fidelity evidence when a future query makes that information decision-relevant.

  • Each composition should be evaluated end to end because local compression metrics are insufficient to predict the final effect on agent behavior.

Operator crossover

  • The composition problem also creates a direct connection to the following ### Crossover subsection. A memory operator that performs poorly as a complete candidate can still contain a useful component. For example:

    • a candidate with weak overall accuracy may have an excellent retrieval router;
    • a lossy summarization candidate may have a useful compaction trigger;
    • a high-memory candidate may have a strong evidence-ranking policy;
    • a slow candidate may have a high-quality semantic consolidation rule.

  • Autoresearch should be able to transplant such components into stronger parents and re-evaluate the combination. This is particularly important for memory systems because storage, routing, compression, retrieval, and presentation errors are separable enough that a local improvement can be hidden by an unrelated regression elsewhere in the candidate.

Search over the compaction stopping rule

  • The outer loop can also optimize when to stop compressing. Instead of choosing a fixed ratio \(r\), a candidate can terminate compaction when estimated marginal distortion becomes too large:

    \[\frac{ \Delta D }{ -\Delta B } > \tau \quad \Rightarrow \quad \mathrm{stop}\]
    • where \(-\Delta B\) is the memory saved by the next compression step and \(\Delta D\) is its estimated increase in downstream distortion.

  • Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference by Feng et al. (2024) is relevant because it derives an attention-output error bound for KV eviction, illustrating how compression can be controlled by predicted output degradation rather than only a fixed retention ratio.

  • At the agent level, a metaharness can approximate the same idea by checking whether further consolidation changes:

    • the experiment the proposer would run next;
    • which candidate appears best;
    • which failure mechanism is inferred from the evidence;
    • whether an earlier claim remains traceable to raw evidence;
    • whether a held-out memory query remains answerable.

  • Search can then tune the corresponding tolerance \(\tau\) against the quality-cost frontier.

From ratio sweeps to predictive budget selection

  • Most compression systems tune memory ratios empirically. A more ambitious search objective is to predict the required memory before running a large sweep.

  • In the rate-distortion framing, the achievable compression ratio depends on the task-conditioned information requirement:

\[I^{\star}(Q) = I(Y;H\mid Q)\]

  • so the required memory budget should vary with model, task, context structure, and query distribution rather than remain globally fixed. What to Keep, What to Forget: A Rate-Distortion View of Memory Compaction in LLMs and Agents by Colaco and Lahjouji (2026) identifies a predictive compression scaling law for estimating this quantity as a central open problem.

  • An autoresearch system can approximate such a predictor empirically. Given previous runs:

    \[\mathcal{D} = \left\{ (x_i,B_i,D_i) \right\}_{i=1}^{N}\]

    • it can learn:

      \[\hat{B}^{\star} = f_{\psi} \left( x, D_{\max} \right)\]
      • where \(x\) contains task and trajectory features and \(D_{\max}\) is the maximum acceptable distortion.

  • The resulting controller can allocate more memory to high-information tasks and compact redundant trajectories aggressively without performing a full budget sweep for every new instance.

Joint search objective

  • Memory search should ultimately optimize a system frontier rather than compression in isolation. A candidate memory policy \(\mathcal{M}\) can be characterized by:
\[\mathbf{f}(\mathcal{M}) = \left( -\mathrm{Quality}, \mathrm{ActiveMemory}, \mathrm{ArchiveMemory}, \mathrm{Latency}, \mathrm{ModelCalls}, \mathrm{CompactionCost}, \mathrm{RecoveryError} \right)\]
  • The outer loop should retain policies on the Pareto frontier:
\[\mathcal{P}_{\mathrm{memory}} = \left\{ \mathcal{M} : \nexists \mathcal{M}' \prec \mathcal{M} \right\}\]
  • This prevents a candidate from being declared superior merely because it produces a smaller prompt. A more compressed policy may be worse if it increases future retrieval failures, requires repeated expensive summarization calls, or irreversibly destroys information needed later.

  • A memory-aware autoresearch loop can expose mutations at several granularities:

    • Query conditioning: Memory retrieval can progress from static retrieval, where the same retrieval rule is used regardless of the current task, to query-conditioned retrieval, where the current information need determines what is restored, and finally to learned retrieval routing, where the system learns which retrieval strategy to invoke.

    • Reversibility: Memory management can progress from irreversible deletion, to summarization with archival backup, to architectures that preserve raw evidence externally and retrieve it again when a later query requires higher-fidelity information.

    • Fidelity: Systems can move from uniform summaries for all memories, to typed representations that preserve different classes of information at different levels of detail, and eventually to learned per-item fidelity that decides dynamically whether each memory should remain raw, structured, summarized, indexed, or discarded.

    • Budget allocation: A fixed global compression ratio can be replaced by budgets assigned separately to different memory types, and then by query-adaptive allocation that gives more capacity to whichever memory class is most useful for the current task.

    • Compaction trigger: Compression can be initiated by a simple token threshold, by natural task boundaries, by confidence or predicted-distortion signals, or by a learned policy that decides when compaction is worthwhile.

    • Consolidation: Memory consolidation can happen synchronously after every interaction, periodically after several interactions, or asynchronously in the background so that expensive consolidation does not block the agent’s critical path.

    • Stopping: Instead of compressing to a fixed target ratio, the system can stop when predicted downstream distortion exceeds a configured tolerance, preserving additional memory when further compression is likely to change important decisions.

    • Composition: Memory systems can evolve from using a single operator, to combining retrieval with summarization, then retrieval with reranking and summarization, and ultimately to multi-tier architectures that coordinate active memory, semantic summaries, archival storage, and recovery of raw evidence.

  • These mutation families make memory optimization scientifically interpretable. Each proposal changes a recognizable mechanism and can be compared against its parent using the same downstream tasks and memory-budget accounting.

  • A memory mutation should be evaluated in stages:

    • Single-operation check: Verify that the compaction or retrieval implementation produces a valid state and respects the budget.
    • Matched-budget evaluation: Compare downstream quality against the parent at the same active-memory budget.
    • Recovery test: Query information intentionally removed from active context and test whether it can be restored.
    • Repeated-compaction test: Run the memory policy over a long trajectory to detect cumulative information loss.
    • Decision-distortion test: Compare whether the compressed and full-memory systems select different downstream actions.
    • Cost evaluation: Record latency, tokens, memory, model calls, retrieval cost, and consolidation cost.
    • Ablation: Isolate whether gains arise from routing, compaction representation, retrieval, or budget allocation.
    • Held-out evaluation: Test the selected memory policy on unseen trajectories whose future information needs were unavailable during search.

  • This produces the same scientific contract used elsewhere in autoresearch: one coherent intervention, externally measured outcomes, full trace preservation, and enough ablations to establish which part of the memory policy actually caused the change.

Toward self-optimizing memory systems

  • The combination of autoresearch and memory compaction suggests a broader architecture in which memory is not a hand-designed service attached to an agent but an evolving subsystem:
\[\text{experience} \rightarrow \text{memory policy} \rightarrow \text{agent decisions} \rightarrow \text{task reward} \rightarrow \text{memory-policy search}\]

Crossover

  • Crossover combines useful components from multiple candidates. In code-space autoresearch, crossover should usually be semantic rather than line-based. The agent should identify which subsystem to transfer, preserve its dependencies, and keep unrelated behavior unchanged.

  • A good crossover workflow should include:

    • Identify donor value: The agent should explain what the donor candidate contributed, such as faster throughput, fewer parse errors, better retrieval precision, lower memory, or improved stability.
    • Identify recipient strength: The agent should explain why the recipient candidate is the stronger base, such as better overall BPB, higher pass rate, or a cleaner cost profile.
    • Transfer only the target subsystem: The agent should avoid copying unrelated prompt text, hyperparameters, or logging changes that would confound the result.
    • Preserve evaluator interfaces: The combined candidate should still satisfy the same metric contract and validation gates.
    • Compare against both parents: The result should be evaluated against the donor and the recipient so the system can tell whether the combination was additive.

  • Crossover is most useful when the archive stores enough detail to identify subsystem-level wins. This is another reason to record secondary metrics and traces rather than only final scores.

Selection

  • Selection decides which candidates remain active. Greedy selection only keeps the best candidate, but autoresearch usually benefits from retaining multiple candidates with different strengths.

  • The active set can be defined as:

\[A_t = \mathcal{P}_t \cup B_t \cup N_t\]

  • where \(\mathcal{P}_t\) is the Pareto frontier, \(B_t\) is a small set of high-quality backups, and \(N_t\) is a small set of novel or underexplored candidates. This prevents the search from collapsing too early.

  • Selection should preserve:

    • The current best: The strongest candidate under the primary metric should always remain available as a safe parent and deployment baseline.
    • Frontier candidates: Candidates that represent different quality-cost tradeoffs should remain active even when they are not best on the headline metric.
    • Promising failures: Candidates with repairable failures should remain indexed, especially if they introduced a valuable component.
    • Diverse edit families: The active set should include different mechanisms so the search can recover from local optima.
    • Recent probes: A few recent candidates should remain easy to inspect because they carry fresh evidence about the current search region.

  • GEPA by Agrawal et al. (2025) is relevant because it combines genetic prompt evolution, natural-language reflection, and Pareto-based selection, which maps naturally onto autoresearch when the optimized artifact is executable code rather than only a prompt.

Exploration control

  • Search should start broad and become more exploitative as evidence accumulates, while still reserving some budget for surprises. A simple annealed exploration probability is:

    \[P(\mathrm{explore}) = p_{\min} + (p_0 - p_{\min}) e^{-t/\tau}\]
    • where \(p_0\) is the initial exploration rate, \(p_{\min}\) is the long-run exploration floor, and \(\tau\) controls how quickly exploration decays.

  • Exploration should increase when the search stagnates. For example, if no candidate improves the frontier for \(K\) iterations, the system can increase novelty pressure:

    \[S(c) = Q(c) + \lambda_n \mathrm{Novelty}(c) - \lambda_r \mathrm{Risk}(c)\]
    • where \(Q(c)\) is expected quality, \(\mathrm{Novelty}(c)\) measures distance from prior candidates, and \(\mathrm{Risk}(c)\) estimates invalid-run probability.

  • A practical exploration policy should include:

    • Early broad search: The first phase should test several edit families to identify which mechanisms matter most.
    • Middle focused search: The system should allocate more budget to edit families with repeated evidence of improvement.
    • Stagnation recovery: If the best score does not move for several iterations, the agent should inspect failures and try a more structural change.
    • Frontier search: The agent should occasionally target lower cost, higher speed, or better robustness rather than only headline quality.
    • Periodic ablation: The system should spend some budget verifying which components of the best candidate actually matter.

Bandit framing

  • Autoresearch can be framed as a bandit over edit families. Each edit family is an arm, each evaluated candidate produces a reward, and the system gradually shifts budget toward families with better returns.

  • Let \(k\) index edit families and \(\hat{\mu}_k\) be the observed mean improvement from family \(k\). An upper-confidence rule is:

    \[\mathrm{UCB}_k = \hat{\mu}_k + \alpha \sqrt{ \frac{\ln t}{n_k} }\]
    • where \(n_k\) is the number of attempts from family \(k\) and \(\alpha\) controls exploration. The agent can use this as a planning prior, while still making semantically rich edits within the chosen family.

  • This framing is useful for managing budget, but it should not replace trace-driven reasoning. A low-performing family may contain one repairable idea, and a high-performing family may stop working once its easy gains are exhausted.

Search traces

  • Search itself should be logged, not just candidate outcomes. The system should record what the agent inspected before proposing an edit, which parent it chose, which hypothesis it wrote, and why it selected that edit family.

  • A search trace should include:

    • Inspection record: The trace should record which leaderboard rows, source snapshots, diffs, logs, and task traces the agent examined before editing.
    • Parent rationale: The trace should explain why the selected parent was the right starting point for the next candidate.
    • Hypothesis: The trace should state the expected mechanism and the metric movement predicted by the agent.
    • Edit family: The trace should classify the proposal as a hyperparameter edit, structural edit, parser repair, retrieval change, prompt change, efficiency change, or another family.
    • Risk prediction: The trace should record the most likely failure modes before evaluation, such as NaNs, underfitting, timeout, context bloat, or parser brittleness.
    • Postmortem: The trace should compare the predicted outcome with the actual outcome and update what the agent should believe next.

  • These search traces make the optimizer itself debuggable. A system that repeatedly proposes risky edits without inspecting failures can be corrected by changing the instruction layer or parent-selection policy.

Baselines

  • Autoresearch should be compared against simple baselines. Otherwise, it is hard to know whether the agent is performing intelligent search or merely sampling enough candidates to get lucky.

  • Useful baselines include:

    • Best-of-\(N\): The system samples \(N\) independent candidates from the same starting point without using history, which tests whether iterative feedback matters.
    • Random local edits: The system applies simple random perturbations to hyperparameters or prompt sections, which tests whether the coding agent’s semantic edits add value.
    • Greedy hill climb: The system only edits the current best candidate and accepts improvements, which tests whether population diversity matters.
    • Scores-only optimizer: The proposer sees candidate code and scalar scores but not raw traces, which tests whether traces are useful.
    • Summary-only optimizer: The proposer sees code, scores, and compressed summaries but not raw logs, which tests whether summaries preserve enough diagnostic information.
    • Human baseline: A human-designed harness or training configuration provides a practical reference point for whether the autonomous loop is competitive.

  • OpenEvolve is relevant because it provides an open-source evolutionary coding-agent framework for generating, mutating, evaluating, and selecting program candidates, making it a useful point of comparison for code-space autoresearch systems.

Stopping

  • The search loop should stop for clear reasons rather than drift indefinitely. Stopping criteria can be budget-based, performance-based, or reliability-based.

  • A good stopping policy should include:

    • Evaluation budget: The search should stop after a fixed number of valid evaluations, wall-clock hours, GPU-hours, or model-call budget.
    • Stagnation: The search should stop or change mode when the frontier has not improved for a specified number of iterations.
    • Noise floor: The search should stop treating improvements as meaningful when changes are smaller than the estimated metric noise.
    • Failure rate: The search should pause for instruction or validation redesign if invalid candidates exceed a threshold.
    • Audit concerns: The search should stop if suspicious improvements, evaluator drift, leakage, or budget violations appear.
    • Deployment readiness: The search can stop when the frontier includes a candidate that satisfies the required quality, cost, latency, memory, and robustness constraints.

  • The final output of search should not be only the best candidate. It should include the best candidate, the frontier, the ablations, the failure analysis, the audit report, and the reusable lessons discovered during the run.

Safety

  • Autoresearch gives an agent the ability to change executable research code, so safety is not an optional add-on. The core safety question is whether the system can improve the target metric without invalidating the experiment, leaking answers, wasting compute, corrupting state, or producing changes that humans cannot audit.

Invariants

  • The safest autoresearch systems define a small set of invariants that every candidate must preserve. These invariants should be enforced by code, not only by instructions.

    • Metric invariance: The candidate should not change the metric definition, validation split, benchmark grader, result parser, byte accounting, or success criteria unless the run is explicitly marked as an evaluator migration rather than a candidate comparison.
    • Budget invariance: The candidate should not train longer, evaluate fewer examples, use unbounded retries, exceed the model-call budget, or silently skip expensive cases.
    • Data invariance: The candidate should not alter training data, validation data, held-out tasks, task labels, answer files, or contamination filters.
    • Interface invariance: The candidate should preserve the evaluator-facing function signatures, result schema, required output files, and expected error behavior.
    • Archive invariance: The candidate should not delete, rewrite, or obscure previous results because the search history is the evidence base for later conclusions.
    • Permission invariance: The candidate should only modify editable artifacts and should treat frozen infrastructure, previous runs, and held-out data as read-only or forbidden according to the experiment contract.

  • These invariants are the difference between autonomous research and unconstrained self-modifying code. They let the agent explore aggressively while keeping comparisons meaningful.

Sandboxing

  • The evaluator should run candidates in a sandboxed environment. The sandbox does not need to be elaborate at first, but it should prevent candidate code from modifying the wrong files or accessing forbidden resources.

  • A practical sandbox should provide the following constraints:

    • Fresh working copy: Each candidate should run in a clean copy of the repository so failed edits, generated files, and local caches do not contaminate later runs.
    • Read-only frozen files: Data preparation, validation manifests, metric code, and benchmark inputs should be mounted read-only when possible.
    • Scoped write access: Candidate code should write only to its assigned run directory, temporary directory, and expected metric output path.
    • Network policy: Network access should be disabled unless it is part of the task, because online access can introduce leakage, nondeterminism, or dependency drift.
    • Resource limits: The sandbox should enforce GPU memory expectations, wall-clock limits, process limits, file-size limits, and retry limits.
    • Environment capture: Each run should record package versions, hardware type, driver version, environment variables, and commit hashes so later reviewers can reproduce or explain behavior.

  • Sandboxing is especially important for harness optimization because a harness may call tools, read files, spawn subprocesses, or invoke other models. The search system should treat tool access as part of the candidate’s budget and permissions, not as an invisible implementation detail.

Integrity checks

  • Integrity checks make false progress visible. The evaluator should check that the candidate improved the intended system rather than weakening the measurement apparatus.

  • A strong integrity layer should include:

    • File hashes: The evaluator should hash frozen files such as data preparation, validation data manifests, task files, metric code, and grading scripts before each run and reject candidates when hashes differ unexpectedly.
    • Metric freshness: The evaluator should verify that the result file was created during the current run, contains the current run ID, and was not copied from a previous run.
    • Task count checks: The evaluator should verify that the candidate evaluated the expected number of validation examples, benchmark tasks, or validation tokens.
    • Budget checks: The evaluator should verify that wall-clock time, token budget, context budget, retry budget, and tool-call budget are within allowed limits.
    • Output schema checks: The evaluator should verify that metrics, traces, and final answers match the expected schema so downstream comparisons do not silently parse malformed outputs.
    • Diff checks: The evaluator should inspect changed files and reject candidates that modified forbidden paths or smuggled evaluation logic into editable code.
    • Leakage checks: The evaluator should scan candidate code, prompts, retrieval indexes, and generated traces for held-out answer strings, benchmark task IDs, forbidden labels, or explicit test-set shortcuts.

  • A useful hash-based guard is:

\[h_{\mathrm{frozen}} = \mathrm{SHA256} \left( F_{\mathrm{data}} \parallel F_{\mathrm{metric}} \parallel F_{\mathrm{tasks}} \parallel F_{\mathrm{parser}} \right)\]

  • A run is leaderboard-eligible only if:

    \[h_{\mathrm{frozen}}^{(t)} = h_{\mathrm{frozen}}^{(0)}\]
    • unless a human explicitly declares a new evaluation version and resets the comparison group.

Leakage

  • Leakage is any path by which the search process uses information that should not be available at proposal time. It can be obvious, such as reading held-out labels, or subtle, such as repeatedly tuning on a public benchmark until the harness encodes benchmark-specific quirks.

  • Leakage prevention should cover several surfaces:

    • Data leakage: The agent should not access held-out labels, test solutions, hidden graders, or answer files while proposing candidates.
    • Trace leakage: The proposer should not see test-set traces during search, because traces reveal failure-specific information that can guide overfitting even when labels are hidden.
    • Retrieval leakage: Retrieval corpora should be deduplicated against evaluation tasks, especially for math, coding, and question-answering settings where near-duplicate solutions can make retrieval look artificially strong.
    • Prompt leakage: Prompts and harness code should be scanned for task IDs, exact expected answers, hidden-label strings, or benchmark-specific conditional branches.
    • Leaderboard leakage: Public benchmark iteration should be reported honestly as benchmark-specific discovery rather than clean held-out generalization.

  • A clean protocol keeps search traces and final-test traces separate. The proposer can inspect search-set failures as much as needed, but final test results should be used only after candidate selection.

Reproducibility

  • An autoresearch result should be reproducible enough that a human can rerun the winning candidate and understand why it was selected. Full determinism is not always possible on GPU workloads or stochastic model calls, but the system should record enough state to make reruns meaningful.

  • A reproducibility record should include:

    • Source snapshot: The exact candidate code, parent code, and patch should be stored permanently.
    • Environment: The system should record package versions, Python version, CUDA version, GPU type, driver version, model version, tokenizer version, and relevant environment variables.
    • Randomness: The evaluator should record random seeds, sampling settings, data-ordering settings, and whether deterministic kernels were enabled.
    • Budget: The run should record wall-clock budget, actual elapsed time, tokens processed, optimizer steps, context tokens, model calls, and retries.
    • Data identity: The run should record dataset hashes, validation manifest hashes, retrieval-corpus hashes, benchmark task versions, and contamination-filter versions.
    • Metric identity: The run should record metric-code hashes and result-schema versions.
    • Trace identity: Harness runs should preserve the prompts, outputs, tool calls, parser decisions, and final grading records needed to reproduce task-level outcomes.

  • For stochastic systems, the final report should include repeated evaluations of frontier candidates when feasible. A candidate that wins once by a tiny margin may not be reliable enough to deploy or claim as a discovery.

Human audit

  • Autoresearch should move humans from manual iteration to oversight, not remove them from the scientific loop. Human review is especially valuable when a candidate improves the metric in an unexpected way, changes a large amount of code, or introduces a new mechanism that the evaluator was not designed to police.

  • A human audit should ask:

    • Was the metric contract preserved: The reviewer should confirm that the candidate did not change validation data, metric code, task selection, byte accounting, result parsing, or budget enforcement.
    • Is the improvement larger than noise: The reviewer should compare the gain to repeated baseline variation and rerun the candidate if the margin is small.
    • Did the candidate improve for the intended reason: The reviewer should inspect traces, curves, and ablations to distinguish true capability improvements from increased compute, extra context, more retries, or parser shortcuts.
    • Is the code general: The reviewer should look for hard-coded task IDs, brittle if-statements, answer strings, benchmark-specific heuristics, or changes that only work on one machine.
    • Is the candidate maintainable: The reviewer should check whether the winning code is readable, modular enough to reuse, and compatible with future experiments.
    • Are failures understood: The reviewer should inspect both successes and representative failures so the system’s limits are not hidden by an aggregate score.

  • Human audit should be triggered automatically for unusually large improvements, candidates that modify many lines, candidates that touch near-forbidden paths, and candidates that change the cost-quality tradeoff substantially.

Security

  • An autoresearch system runs model-written code, so it should be treated as an untrusted-code execution environment. Even benign agents can generate harmful behavior accidentally, such as deleting logs, exhausting disk, spawning runaway processes, or leaking credentials through tool outputs.

  • A secure deployment should include:

    • Credential isolation: API keys, cloud credentials, private datasets, and personal files should not be exposed to candidate code unless strictly required.
    • Process limits: Candidate runs should have limits on subprocess count, runtime, CPU, memory, GPU memory, disk writes, and open files.
    • Filesystem isolation: Candidate code should not have write access outside the run directory and should not be able to modify the archive or evaluator.
    • Network isolation: Network access should be disabled by default and granted only for tasks that explicitly require it.
    • Dependency control: Candidate code should not freely install arbitrary packages during evaluation unless the sandbox captures and approves dependency changes.
    • Log redaction: Logs should avoid storing secrets, credentials, personal data, or private file paths that future agents could read.
    • Kill switches: The orchestration layer should be able to stop runaway jobs, disable an agent, freeze the archive, and mark the current search as tainted.

  • The more capable the proposer, the more important these controls become. Capability increases the chance of useful discoveries, but it also increases the need for hard boundaries.

Robustness

  • A winning candidate should be tested beyond the exact setting that produced it. Robustness checks distinguish real improvements from overfit artifacts.

  • Useful robustness checks include:

    • Seed robustness: Rerun the candidate under multiple seeds or sampling settings to estimate variance.
    • Budget robustness: Evaluate nearby budgets, such as shorter and longer runs, to see whether the improvement is a fixed-budget artifact or a generally better method.
    • Task robustness: Test on task families not used during search when possible.
    • Model robustness: For harnesses, evaluate whether the same harness helps different base models or only the model used during search.
    • Cost robustness: Check whether the candidate remains useful under stricter memory, latency, context, or tool-call budgets.
    • Ablation robustness: Remove or isolate components of the winning candidate to verify which parts are necessary.

  • A candidate is more trustworthy when it improves the main metric, survives integrity checks, generalizes beyond the search set, and has an interpretable mechanism supported by traces or ablations.

Governance

  • Autoresearch systems need governance because they can accumulate many small automated decisions into a large research result. Governance defines when the system is allowed to continue, when humans must intervene, and what evidence is required before a result is accepted.

  • A practical governance policy should specify:

    • Promotion rules: A candidate should become the new default only after passing metric integrity checks, budget checks, and minimum improvement thresholds.
    • Frontier rules: A candidate should enter the frontier when it offers a non-dominated quality-cost tradeoff and passes audit.
    • Escalation rules: Human review should be required for suspicious improvements, evaluator-adjacent diffs, leakage warnings, high-cost runs, or large structural rewrites.
    • Deprecation rules: Candidates should be removed from active consideration when they are dominated, non-reproducible, invalid, or dependent on a tainted evaluator version.
    • Reporting rules: Final results should disclose search budget, number of evaluated candidates, invalid-run rate, candidate-selection procedure, held-out evaluation protocol, and known limitations.
    • Pause rules: The search should stop automatically when leakage is detected, evaluator hashes change unexpectedly, invalid-run rates become too high, or resource use exceeds limits.

  • Safety is therefore not a separate stage after search. It is part of the search algorithm. The evaluator, sandbox, archive, auditor, and governance rules together define which discoveries count as valid.

Scaling

  • Autoresearch can begin with one agent, one GPU, one editable file, and one metric, but the long-term pattern is a distributed research organization: many agents propose candidates, shared evaluators score them, an archive stores evidence, and frontier candidates are promoted only after audit. Scaling is therefore not only a compute problem. It is a coordination, memory, evaluation, and governance problem.

Parallelism

  • The simplest scaling move is to run multiple candidates in parallel. Parallelism increases search throughput, but it also introduces coordination problems: agents may duplicate edits, branch from stale parents, overwrite results, or interpret incomplete runs as final evidence.

  • A parallel system should enforce the following rules:

    • Run reservation: Each agent should reserve a unique run ID, parent candidate, and edit family before modifying code so that concurrent proposals do not collide.
    • Immutable parent snapshots: Each candidate should branch from a fixed source snapshot rather than a moving working directory, which prevents one agent’s changes from silently affecting another agent’s run.
    • Evaluator queue: Candidate evaluation should go through a central queue that enforces budget, resource limits, and metric integrity checks.
    • In-flight visibility: Agents should be able to see which edit families are currently running so they can avoid duplicating work.
    • Delayed promotion: A candidate should not become the new default until its metrics are finalized, validated, and written to the archive.
    • Rebase policy: Agents should periodically rebase future proposals on the current frontier when a new best candidate appears.

  • The main objective is to preserve the scientific meaning of each run. More parallelism is only helpful if the system still knows exactly which code, parent, budget, and metric produced each result.

Scheduling

  • As the number of candidates grows, scheduling becomes a research decision. The system must decide which candidates deserve scarce GPU time, which candidates should receive cheap smoke tests only, and which frontier candidates deserve expensive confirmation.

  • A useful scheduler should support multiple queues:

    • Smoke-test queue: This queue runs syntax checks, import checks, interface checks, tiny training runs, parser checks, and budget validation before expensive evaluation.
    • Exploration queue: This queue evaluates novel candidates that test underexplored mechanisms or branch from non-best parents.
    • Exploitation queue: This queue evaluates local refinements of frontier candidates.
    • Ablation queue: This queue isolates components of high-performing candidates to verify causality.
    • Confirmation queue: This queue reruns frontier candidates under additional seeds, task subsets, or stricter budgets.
    • Audit queue: This queue runs leakage checks, evaluator-hash checks, suspicious-diff scans, and reproducibility checks.

  • A simple scheduling priority can combine expected value and cost:

    \[\mathrm{Priority}(c) = \frac{ \mathbb{E}[\Delta(c)] \cdot P(\mathrm{valid}(c)) }{ \mathrm{Cost}(c) }\]
    • where \(\mathbb{E}[\Delta(c)]\) is the expected improvement, \(P(\mathrm{valid}(c))\) is the estimated probability that the candidate passes validation, and \(\mathrm{Cost}(c)\) is expected runtime or model-call cost.

Compute budgets

  • Autoresearch should make compute budgets explicit at every level. A single candidate has a run budget. A search campaign has a total budget. A confirmation phase has a separate robustness budget. Without this separation, the agent may spend all resources on exploration and leave no budget for validation.

  • A practical budget plan should include:

    • Per-candidate budget: Each training run or harness evaluation should have a fixed wall-clock, token, task, retry, and tool-call budget.
    • Campaign budget: The full search should have a maximum number of valid evaluations, invalid evaluations, GPU-hours, model calls, or wall-clock hours.
    • Confirmation budget: The system should reserve resources to rerun top candidates, estimate variance, and run held-out evaluations.
    • Ablation budget: The system should reserve resources to test whether the winning components actually caused the improvement.
    • Audit budget: The system should reserve resources for integrity checks, leakage scans, and reproducibility tests.
    • Fallback budget: The system should reserve a small amount of compute for recovery when a promising candidate fails because of an implementation bug rather than a bad idea.

  • A useful budget allocation is:

\[B_{\mathrm{total}} = B_{\mathrm{search}} + B_{\mathrm{confirm}} + B_{\mathrm{ablate}} + B_{\mathrm{audit}}\]
  • The exact split depends on the cost of evaluation and the noise level of the metric. Cheap noisy experiments need more confirmation. Expensive deterministic experiments need stronger candidate filtering before full evaluation.

Multi-agent roles

  • Scaling works better when agents specialize. Specialization reduces conflicts and makes permissions easier to reason about.

  • A multi-agent autoresearch organization should include:

    • Proposer agents: These agents inspect the archive, write hypotheses, edit candidate artifacts, and submit candidates to the evaluator queue.
    • Debugging agents: These agents repair invalid candidates, diagnose crashes, and convert promising failed runs into valid variants.
    • Ablation agents: These agents isolate components of high-performing candidates and test whether each component is necessary.
    • Auditor agents: These agents enforce file boundaries, check metric integrity, scan for leakage, and flag suspicious improvements.
    • Curator agents: These agents maintain the leaderboard, Pareto frontier, run-family summaries, and active research agenda.
    • Documentation agents: These agents turn raw experiments into readable summaries, reproducibility records, and final reports.

  • This role separation resembles a human research group, but the coordination medium is the archive rather than meetings. Each agent leaves behind structured evidence, and the next agent acts on that evidence.

Shared context

  • The archive becomes the shared context for the research organization. As scale increases, the archive must support fast retrieval, reliable indexing, and provenance tracking. Otherwise, agents will make decisions from stale or incomplete memory.

  • Shared context should include:

    • Global leaderboard: The system should maintain a current view of best candidates, frontier candidates, invalid-run rates, and recent improvements.
    • Run-family summaries: The archive should summarize which edit families are helping, which are failing, and which remain underexplored.
    • Failure database: The archive should index NaNs, out-of-memory failures, parser failures, timeouts, leakage warnings, and metric-integrity failures.
    • Component registry: Useful subcomponents, such as a stable optimizer group, parser repair routine, retrieval filter, or prompt template, should be recorded as reusable modules.
    • Open questions: The curator should maintain a list of unresolved hypotheses, such as whether a gain came from throughput, capacity, retrieval relevance, parser repair, or extra context.
    • Tainted runs: The archive should clearly mark runs that are invalid, non-comparable, leakage-suspect, or produced under a deprecated evaluator.

  • The shared context should always distinguish facts from interpretations. A metric is a fact produced by an evaluator. A postmortem is an interpretation. A family summary is a compressed interpretation over many runs. Agents should be able to trace summaries back to raw evidence.

Model diversity

  • A scaled autoresearch system can use different models for different roles. Stronger models may be better at structural rewrites and trace diagnosis. Smaller models may be sufficient for formatting, log parsing, smoke-test repair, or summarization.

  • A model-diverse setup should use:

    • Frontier proposer models: Strong models should be reserved for high-leverage design decisions, structural rewrites, difficult debugging, and final synthesis.
    • Cheap maintenance models: Smaller models can update ledgers, summarize logs, classify failures, extract metrics, and prepare candidate reports.
    • Independent auditor models: Auditors should ideally differ from proposers so that the same model’s blind spots do not affect both proposal and validation.
    • Specialized retrievers: Retrieval over the archive may use lexical search, embeddings, structured filters, or hybrid retrieval depending on the evidence type.
    • Human review: Humans should remain in the loop for suspicious improvements, high-impact claims, and changes to the evaluation contract.

  • The important principle is to spend reasoning budget where it changes decisions. A frontier model does not need to parse every log line if a cheaper model or script can extract the same failure status reliably.

Transfer

  • A discovery is more valuable if it transfers beyond the exact search setup. Scaling autoresearch therefore means turning one-off wins into reusable methods, skills, and harness components.

  • Transfer should be evaluated across several axes:

    • Across seeds: The candidate should remain strong under different random seeds or sampling settings.
    • Across budgets: The candidate should still help under shorter or longer training budgets, or under stricter inference budgets.
    • Across tasks: A harness improvement should help task families not seen during search when possible.
    • Across models: A harness should ideally improve multiple base models, not only the one used during search.
    • Across hardware: Training-code improvements should be checked on different accelerators or batch-size regimes before being treated as general.
    • Across datasets: A model-training change should be tested on another dataset or distribution when feasible.

  • Meta-Harness-style results are especially interesting when a discovered harness transfers across held-out models or out-of-distribution tasks, because that suggests the search found a reusable procedure rather than a benchmark-specific shortcut.

Cost control

  • Scaling without cost control leads to waste. Agents may run redundant experiments, repeatedly test invalid candidates, use excessive context, or rerun expensive evaluations before cheap checks.

  • Cost control should include:

    • Deduplication: The system should compare proposed diffs and reject near-duplicates before evaluation.
    • Progressive evaluation: Candidates should pass cheap checks before receiving full-budget evaluation.
    • Early stopping: Training runs or harness evaluations should stop early when they clearly violate constraints, diverge, or fail required interfaces.
    • Context budgets: Harnesses should have explicit context-token budgets, and candidates should be penalized or rejected when they exceed them.
    • Retry budgets: Agents and harnesses should have capped retries so they cannot convert cost into score invisibly.
    • Cache policy: Expensive retrieval indexes, compiled kernels, prepared datasets, and benchmark environments should be cached when this does not compromise comparability.
    • Invalid-rate monitoring: If too many candidates fail validation, the system should shift budget from exploration to debugging or instruction redesign.

  • A campaign-level efficiency metric is:

    \[\mathrm{SearchEfficiency} = \frac{ \max_{t \le T} \Delta s_t }{ B_{\mathrm{spent}} }\]
    • where \(\Delta s_t\) is the best improvement achieved by time \(t\) and \(B_{\mathrm{spent}}\) is the consumed budget. This metric discourages search strategies that eventually improve but burn excessive resources.

Staleness

  • At scale, information becomes stale. A run-family summary may describe an old evaluator version. A best candidate may be dominated under a new cost metric. A failure pattern may no longer apply after a structural rewrite.

  • The system should handle staleness explicitly:

    • Versioned evaluators: Every metric should be tied to an evaluator version, and candidates from different evaluator versions should not be compared without migration.
    • Versioned datasets: Data and benchmark versions should be recorded so improvements are not confused across changed task sets.
    • Versioned instructions: Agent instructions should be versioned because changes to the research policy affect the search distribution.
    • Summary timestamps: Family summaries and frontier notes should record when they were produced and which run IDs they cover.
    • Deprecation markers: Old candidates should be marked as deprecated when they are dominated, invalid under new rules, or tied to obsolete infrastructure.
    • Revalidation: Important frontier candidates should be rerun after evaluator, hardware, model, or dataset changes.

  • Staleness is not a bookkeeping issue. It directly affects whether the system’s memory is trustworthy.

Finalization

  • Scaling should end with a finalization phase. The goal is to convert a large search history into a small set of validated claims.

  • A finalization phase should include:

    • Candidate selection: Choose the best candidate or frontier candidates using only predeclared search-time criteria.
    • Clean reruns: Rerun selected candidates from clean checkouts under the final evaluator.
    • Variance estimation: Repeat evaluations when metrics are stochastic or margins are small.
    • Ablation: Remove or isolate major components of the winning candidate to verify causal contribution.
    • Held-out testing: Evaluate on held-out tasks, models, datasets, or seeds that were not visible during search.
    • Audit report: Document evaluator hashes, data versions, leakage checks, budget compliance, and suspicious-run handling.
    • Release artifact: Package the winning source code, configuration, evaluation script, run logs, and reproduction instructions.

  • The final claim should be proportional to the evidence. A candidate tuned on a public benchmark can be claimed as a strong benchmark-specific harness. A candidate validated on held-out tasks and models can be claimed as a more general method. A candidate that improves only one noisy run should be treated as a lead, not a result.

Practical blueprint

  • A practical autoresearch system should be small enough to audit, structured enough to run unattended, and strict enough that improvements remain meaningful. The blueprint below combines the single-GPU training pattern with the Meta-Harness pattern: start with a narrow editable artifact, add a durable archive, enforce evaluator invariants, then scale toward harness search and multi-agent coordination.

Minimal stack

  • The smallest useful stack has five pieces: an editable artifact, an evaluator, an archive, an agent instruction layer, and an audit layer. The editable artifact is what the agent changes. The evaluator scores the artifact under a fixed budget. The archive stores every attempt. The instruction layer tells the agent how to behave. The audit layer rejects candidates that violate the experiment contract.

  • A minimal implementation should include:

    • A fixed substrate: The system should keep data preparation, validation construction, metric computation, and result parsing outside the editable region so candidate comparisons remain valid.
    • A single editable artifact: The first version should usually let the agent edit one file, such as a model-training script or a harness implementation, because single-file diffs are easier to inspect and debug.
    • A bounded evaluator: Every candidate should receive the same time budget, token budget, task budget, context budget, or tool-call budget, depending on the domain.
    • An append-only archive: Every run should write a source snapshot, diff, metrics, logs, traces, and notes to a permanent run directory.
    • A compact ledger: The agent should be able to read a small table of all prior runs, including score, cost, status, parent, edit family, and pointers to full evidence.
    • A proposer instruction file: The agent should be told what files are editable, what success means, how to inspect prior runs, how to write hypotheses, and how to handle failures.
    • Validation gates: The system should run cheap checks before expensive evaluation, including import checks, smoke tests, interface checks, metric-path checks, and evaluator-hash checks.
    • A final audit path: Frontier candidates should be rerun cleanly, checked for leakage, and ablated before being treated as discoveries.

  • This stack is enough to convert manual iteration into an autonomous loop while keeping the system understandable.

Build order

  • Autoresearch should be built in stages. A system that tries to start with many agents, many tasks, and flexible harness search will be hard to debug. The first milestone should be a boring, reliable loop.

  • A sensible build order is:

    • Start with the frozen evaluator: First implement the data loading, task loading, metric computation, budget enforcement, and result schema. Do not add an agent until a human can run candidates reliably and compare results.
    • Add source snapshotting: Next, make every run copy the editable artifact and store a patch against its parent so that results are reproducible.
    • Add the ledger: Then create an append-only table that records run identity, parent, status, score, cost, and notes path.
    • Add validation gates: Before full evaluation, enforce syntax, import, interface, and smoke-test checks so invalid candidates fail cheaply.
    • Add a single agent: Give the agent read access to the ledger and prior runs, write access to only the editable artifact, and a clear instruction file.
    • Add trace logging: Once scalar metrics are reliable, log richer traces such as training curves, prompts, tool calls, parser outcomes, and task-level failures.
    • Add frontier tracking: Move from a single best score to a Pareto frontier when quality-cost tradeoffs matter.
    • Add ablation and confirmation: Reserve budget to rerun and ablate the best candidates so that improvements are not just noise or bundled confounds.
    • Add parallelism last: Only after single-agent runs are reliable should multiple agents or distributed evaluators be introduced.

  • This order reduces the chance that the system scales a flawed measurement setup.

Default loop

  • The default loop should be intentionally repetitive. Repetition is useful because it creates comparable evidence. Each iteration should inspect history, form a hypothesis, edit one coherent component, evaluate under the fixed contract, and write a postmortem.

  • A good default loop should require the agent to:

    • Inspect the current frontier: The agent should read the leaderboard and identify the current best candidate, relevant frontier candidates, and recent failed candidates before editing.
    • Choose a parent deliberately: The agent should state whether it is editing the current best, branching from a promising non-best candidate, transplanting a component, or performing an ablation.
    • State a hypothesis: The agent should explain the expected mechanism, expected metric movement, and main risk before changing code.
    • Make a coherent edit: The candidate should target one mechanism, such as optimizer stability, throughput, retrieval quality, parser robustness, or context reduction.
    • Run cheap checks: The agent should run validation gates before using the full budget.
    • Run full evaluation: The evaluator should execute the candidate under the fixed budget and produce structured metrics.
    • Write evidence: The system should save source, diff, logs, traces, metrics, and notes.
    • Update selection state: The system should update the leaderboard, frontier, family summaries, and failure database only after integrity checks pass.
    • Decide the next action: The agent should decide whether to refine, ablate, transplant, debug, or explore based on the result.

  • The loop can be summarized by:

    \[\mathcal{D}_{t+1} = \mathcal{D}_t \cup \left\{ c_t,\Delta_t,m_t,\ell_t,n_t \right\}\]
    • where \(c_t\) is candidate code, \(\Delta_t\) is the diff, \(m_t\) is the metric record, \(\ell_t\) is the raw evidence bundle, and \(n_t\) is the postmortem note.

Harness extension

  • Once the training-code loop works, the same pattern can be extended to harnesses. The editable artifact becomes the code that controls prompts, retrieval, memory, parsing, tool use, and state updates. The evaluator becomes a task suite. The archive stores task-level traces rather than only training logs.

  • A harness-oriented system should add:

    • A stable harness interface: The harness should expose a fixed entry point, such as running one task and returning a structured result with answer, reward, cost, trace path, and failure type.
    • Prompt logging: Every model call should save the prompt, model output, token counts, sampling settings, and stop reason.
    • Retrieval logging: Every retrieval step should save the query, retrieved items, scores, filters, and final context inserted into the prompt.
    • Memory logging: Every state update should save what was stored, compressed, dropped, or retrieved later.
    • Parser logging: Every parser decision should record extracted answers, schema errors, repair attempts, and final parse status.
    • Tool logging: Every tool call should record arguments, outputs, exit codes, touched files, timeouts, and cost.
    • Task-level scoring: The evaluator should store reward, failure type, latency, context tokens, model calls, and tool calls for each task.
    • Leakage scanning: Candidate code and traces should be scanned for task IDs, held-out answers, forbidden files, and benchmark-specific shortcuts.

  • This is where Meta-Harness becomes directly relevant: the proposer should inspect prior harness source code, task-level traces, and scores through the filesystem rather than relying on scalar rewards or short summaries.

Checklists

  • Checklists help make the loop reliable. They should be used by agents, evaluators, and human reviewers.

  • A pre-run checklist should include:

    • Editable-boundary check: Confirm that only allowed candidate files changed.
    • Interface check: Confirm that the candidate still exposes the expected functions, result schema, and output paths.
    • Smoke check: Confirm that a tiny run or a small task subset completes successfully.
    • Budget check: Confirm that time, token, context, retry, and tool-call limits are configured correctly.
    • Metric-path check: Confirm that the run will write fresh metrics to the correct run directory.
    • Frozen-hash check: Confirm that evaluator files, validation manifests, and task files match the expected hashes.

  • A post-run checklist should include:

    • Metric validity: Confirm that metrics are fresh, complete, schema-valid, and tied to the current run ID.
    • Budget compliance: Confirm that the run used the expected time, task count, validation tokens, context budget, and retry limits.
    • Failure classification: Confirm whether the run was evaluated, invalid, crashed, timed out, leaked, or violated metric integrity.
    • Evidence preservation: Confirm that source, diff, logs, traces, metrics, and notes were stored.
    • Comparison target: Confirm that the run was compared against the correct parent, current best, and frontier.
    • Audit status: Confirm whether the candidate is eligible for the leaderboard, requires review, or must be rejected.

  • A finalization checklist should include:

    • Clean rerun: Re-evaluate selected candidates from a clean checkout.
    • Variance check: Repeat evaluation when stochasticity or small margins matter.
    • Ablation check: Isolate the major components of the winning candidate.
    • Held-out check: Evaluate on held-out tasks, seeds, models, or datasets when available.
    • Leakage check: Scan source, prompts, traces, and retrieval corpora for forbidden information.
    • Release check: Package source code, configs, environment, evaluator version, metrics, traces, and reproduction instructions.

Common traps

  • Most autoresearch failures come from weak evaluation contracts, missing logs, or over-permissive agents. These traps are predictable.

  • Common traps include:

    • Changing the evaluator: A candidate that changes validation data, metric code, byte accounting, parser logic, or task selection is not comparable to earlier candidates.
    • Trusting tiny gains: A small improvement should not be treated as real until it is compared against noise from repeated runs.
    • Keeping only winners: Failed candidates contain useful evidence about unstable regions of the search space, so deleting them makes future search worse.
    • Overcompressing history: Short summaries can hide the exact trace detail needed for credit assignment.
    • Allowing unlimited retries: A harness that improves pass rate through unbounded retries is spending more budget, not necessarily becoming smarter.
    • Ignoring costs: A candidate that improves accuracy by doubling context or latency may be dominated under deployment constraints.
    • Skipping ablations: A bundled winning diff may contain one useful change and several irrelevant or harmful changes.
    • Parallelizing too early: Multiple agents amplify coordination bugs, stale parents, and duplicate work if the single-agent loop is not reliable.
    • Confusing benchmark discovery with generalization: A harness optimized repeatedly on a public benchmark may be valuable, but it should not be claimed as held-out generalization without separate evidence.

Design patterns

  • Several design patterns make autoresearch systems more robust.

    • Single editable surface: Start with one file or one harness module so diffs are inspectable and accidental evaluator changes are less likely.
    • Append-only memory: Preserve every run, including invalid candidates, because negative evidence supports future credit assignment.
    • Filesystem experience: Store full evidence on disk and let the agent retrieve selectively, rather than packing all history into one prompt.
    • Frontier over winner: Maintain a Pareto frontier so quality-cost tradeoffs are not collapsed too early.
    • Cheap checks before expensive runs: Use validation gates to reject broken candidates before consuming full compute.
    • Ablate before claiming: Treat a winning candidate as a hypothesis until its components are isolated.
    • Audit by default: Scan for metric drift, leakage, budget violations, stale results, and forbidden file edits after every candidate.
    • Version everything: Evaluator versions, dataset hashes, instruction versions, model versions, and candidate hashes should all be recorded.
    • Human review on anomalies: Large or surprising improvements should trigger human inspection before promotion.

Example rollout

  • A realistic first rollout might look like this:

    • Day one setup: A human freezes the evaluator, defines validation BPB, creates the run archive, writes the ledger schema, and verifies that several manual runs produce comparable metrics.
    • Day two single-agent loop: The agent receives permission to edit only the training file, reads the baseline, proposes small architecture and optimizer changes, runs bounded experiments, and writes postmortems.
    • Day three reliability pass: The system adds evaluator hashes, stale-metric checks, NaN classification, source snapshotting, and repeated baseline runs to estimate metric noise.
    • Day four trace enrichment: The evaluator starts storing training curves, throughput curves, gradient norms, memory usage, and richer failure labels.
    • Day five frontier search: The agent begins maintaining a Pareto frontier over BPB, throughput, memory, and parameter count rather than only one best score.
    • Day six ablation: The best candidate is decomposed into component changes, and the system tests which parts actually matter.
    • Day seven finalization: The top candidates are rerun from clean checkouts, compared against baselines, audited for evaluator integrity, and summarized into a reproducible report.

  • For a harness project, the same rollout replaces training curves with task traces, parser logs, retrieval logs, context-token accounting, and per-task reward analysis.

End state

  • The mature end state is not a black box that silently claims improvements. It is a research machine with visible evidence. It proposes code, runs experiments, stores traces, compares candidates, audits itself, and produces a frontier of validated artifacts.

  • A well-built autoresearch system should leave behind:

    • A reproducible best candidate: The final source, config, environment, and evaluator should be sufficient to rerun the winning result.
    • A Pareto frontier: The report should include alternatives that trade quality, cost, latency, memory, and robustness.
    • A complete archive: The search history should preserve both successes and failures so future work can build on the evidence.
    • A causal story: Ablations and traces should explain why the winning candidate worked.
    • A safety record: The report should document budget compliance, evaluator integrity, leakage checks, and suspicious-run handling.
    • Reusable skills: The system should extract patterns that can be reused in later campaigns, such as stable optimizer groups, parser repair strategies, retrieval filters, or audit checks.

  • The practical goal is not to replace scientific judgment. It is to make the experimental loop fast, cumulative, and inspectable enough that agents can do the repetitive search while humans supervise the claims.

References

Autoresearch core

Meta-Harness

Autoresearch extensions

Code search and evolutionary discovery

Prompt and text optimization

Agent reasoning and memory

Memory compaction and context management

Harnesses and LM systems

Retrieval and context

Benchmarks and evaluation

Optimization and training

Ecosystem articles

Background

Citation

If you found our work useful, please cite it as:

@article{Chadha2020DistilledAutoresearchMetaharness,
  title   = {Autoresearch and Metaharness},
  author  = {Chadha, Aman and Jain, Vinija},
  journal = {Distilled AI},
  year    = {2020},
  note    = {\url{https://aman.ai}}
}