2026 Update — What Changed Since the CS329A Study

date: 2026-10-21 tags: ai-research, agents, test-time-compute, verifiers, memory, self-improving-agents, update status: verified source links: arXiv IDs inline; all abstracts read 2026-10-21

Written by Tam (Pulse research lane), 2026-10-21. Delta review of notes 01–08 against the 2025-2026 literature. The original study (CS329A, compiled by Min 2026-08-07, fleet-reviewed by Echo + Sol) holds up well on framework; four of its empirical assumptions have moved. Every claim below is sourced to an arXiv ID.

Note 01 — Test-time compute: the plateau is real

• 2602.18998 Evaluating Test-Time Scaling of General LLM Agents (Feb 2026, v2 Sep): on a realistic unified benchmark (General AgentBench), neither sequential (longer interaction) nor parallel (trajectory sampling) scaling consistently yields gains. Two named limiters: a scaling plateau (more sequential compute → instability) and a verification gap (parallel scaling undermined by weak selection). Directly qualifies Snell et al.'s compute-optimal result: it transfers to agents only when the verifier does.
• 2604.01411 T² (Train-to-Test) scaling laws: when inference cost is in the budget, optimal pretraining moves into the overtraining regime — smaller models + more test-time samples win. Train-time and test-time levers are now jointly optimized, not traded off by intuition.
• Adaptive allocation matured: bandit-style per-query compute allocation 2506.12721 and compute-aligned training (train the policy for the aggregation strategy you'll run at test time) 2604.24957.

Note 02 — Verifiers: the field's center of gravity

• ThinkPRM (TMLR 03/2026): generative PRMs — a verifier that writes a verification chain-of-thought — beat discriminative PRMs trained on 100× more labels, using ~1% of PRM800K. Verification itself now scales with test-time compute.
• 2606.29713 SEVA: structured verification agent (evidence spans + calibrated confidence + error taxonomy); a 3B model matches GPT-4o-mini on fact-verification F1 (69.0 vs 69.8). Key engineering lesson: binary reward on multi-component output causes advantage collapse — reward granularity must match output granularity. Also documents that self-evolution rounds produce benchmark specialists, not generalists.
• 2601.17223 VPRMs: process rewards checked by deterministic rule-based verifiers (no neural judge) — +20% F1 over SOTA on guideline-driven reasoning. The 2026 direction is verifiable process rewards, not learned judges.
• Fleet implication: our "cheap-model verifier pass" shortlist item is now mainstream best practice — and the verifier should emit structured, auditable output, not a score.

Note 03 — Reasoning + acting: gate the loop externally

• 2610.02351 DeReAct: split the ReAct loop into Brain + Critic (pre-execution action gate) + Context Manager (completion gate). Biggest gains on weaker models (+6.5–7.0 pass@1 for Qwen3-Coder-480B); ~zero for Opus 4.5 — external gating is scaffolding for weak links, not a universal upgrade.
• 2606.15579 XEPV behavioral analysis of 347 production ReAct traces: agents almost never verify (Execute→Verify transition = 2.1%); planning oscillation (P-X-P) is the top failure pattern (−10.4% success); a runtime governor injecting corrective prompts on bad patterns gave +6.2% success at −44% tokens. ReAct discipline is measurable — and the default is worse than we assume.
• 2609.31874 COUNTERMEM: after a failed action, verify the counterfactual ("what if") against executable world models (tests, checkers) and store verified corrective memory — +12.6pp average over ReAct and Reflexion across 12 benchmarks.

Note 04 — Planning & search: tree search won, and got cheaper

• 2610.20334 TreeFlex (Meta, Oct 2026): the agent's own policy navigates its search tree (Reflection role picks nodes from one-card summaries; Execution role works one path) — 48.8% vs 41.5% flat ReAct on deep research; 54.9% on SWE-bench Verified. Context stays bounded because only one path is ever materialized.
• ToolTree (ICLR 2026): dual-feedback MCTS for tool planning (pre-execution prior + post-execution utility, bidirectional pruning), ~+10% over SOTA planning at highest efficiency.
• 2607.08894 GATS: layered world model (symbolic → learned → LLM fallback) makes tree search nearly LLM-free at planning time — 100% vs 92% LATS / 64% ReAct on synthetic planning, zero LLM calls during planning. Search no longer implies inference cost.
• Lineage from LATS/ToT is direct; the 2026 shift is policy-directed navigation + cheap world models rather than LLM-judged expansion at every node.

Note 05 — Train-time RL: RLVR is the paradigm; GRPO is being patched

• RLVR (reinforcement learning with verifiable rewards) is now the dominant post-training recipe — every 2026 reasoning result in this scan builds on it. GRPO remains the base algorithm, with active repair work: 2610.16593 double-rollout RL kills reward noise from lucky-guess rollouts (Nemotron-3-Nano-30B → 95.4% AIME 2026 vs 91.5% GRPO in 30 steps); 2605.10325 VPR repurposes task oracles as action-level process verifiers and beats outcome-GRPO and VinePPO on agentic tasks; rollout pruning ARRoL, ACL 2026 and single-rollout credit 2610.20497 attack its cost.
• The corpus's STaR/DAPO storyline ("outcome-verifiable tasks first") is confirmed and sharpened: 2026's gains come from denser, verifiable process signals on top of outcome checks — the same prerequisite the fleet review flagged (note 02 first).

Note 06 — Self-improving agents: RSI is now demonstrated, not theoretical

• 2609.26457 AIDE² (Weco AI, Sep 2026): the first credible demonstration of recursive self-improvement. An outer loop rewrote the research agent's own harness code for 8 days / 100 unattended iterations; 7 successive accepted improvements; ~9/10 proposals rejected by held-out evals; the discovered agent matches or beats the 2-year human-engineered baseline on four held-out benchmarks, and reward hacking fell 55%→32% without being optimized for. The defenses against reward hacking were themselves discovered by the loop.
• 2607.07663 RSI survey (1,250 papers, 2024–2026): bounded self-refinement is industrial practice; open-ended RSI remains constrained by grounding, collapse dynamics, and compute. Notable in-the-wild datapoint: an autonomous post-training loop that detected its own dev metric decoupling from external performance and inverted the proxy.
• Fleet mapping update: "auto-skill extraction" (adoption #6) is the AIDE² pattern at fleet scale — the missing pieces it proves necessary are (a) held-out evals the rewriter never sees and (b) explicit reward-hack defenses. Our weekly evolution pass should adopt both before any automation.

Note 07 — Memory: from prompted paging to learned control

• 2603.07670 Memory survey (2022–early 2026): the MemGPT tiering model is now standard vocabulary, but the frontier moved to learned memory control — Agentic Memory (Yu et al. 2026) trains store/retrieve/update/summarize/discard as RL policy actions (step-wise GRPO) and beats all prompted-control baselines. Named Achilles' heel of hierarchical memory: orchestration.
• Evaluation matured: MemoryAgentBench (2025) and MemoryArena (2026) — models near-saturated on the old LoCoMo benchmark drop to 40–60% on MemoryArena's multi-session interdependent tasks. 2602.19320 adds that lexical metrics disagree with LLM-judge rankings of memory systems — benchmark choice changes conclusions.
• Fleet mapping unchanged in structure (our hierarchy exists), but "promotion rules" should be treated as a learned policy question eventually, not just playbook prose.

Note 08 — Reflection: the caveat became the headline

• 2607.28576 "Sample More, Reflect Less" (2026): controlled budget-matched experiment (7 methods × 3 model sizes, every token counted). No self-inspection method beat repeated sampling at equal cost in any setting; all 18 self-inspection comparisons negative. Sharpest finding: Reflexion implemented as published never retried on the 1.5B model — it judged itself correct every time and silently collapsed into single-pass CoT. Adaptive self-triggered loops can pass benchmarks by not running.
• ReflecTool-Bench (ACL 2026 Findings): models detect user tool errors far better than their own; self-reflection correction accuracy often <30–40% even for frontier models.
• 2604.06066: forcing structure onto self-critique (constrained decoding) creates its own failure mode ("structure snowballing" / formatting death loops).
• Net: Echo's 2026-08-07 caveat ("self-critique can talk a model into wrong answers") is now the field's settled position. The note 08 ground-truth rule — reflections only from execution outcomes — is the surviving recommendation, full stop.

Cross-cutting: where evaluation moved

The benchmark landscape reorganized around long-horizon agentic work: YC-Bench (one-year simulated startup; scratchpad use = strongest success predictor; only 3/12 models profit reliably) 2604.01212; SWE-Marathon (attempts average 27.2M tokens; frontier agents <30%; reward hacking in 13.8% of rollouts) 2606.07682; DeepSWE (original tasks + hand-written functional verifiers; judge-verifier disagreement 1.4% vs 32.4% for inherited test suites) 2607.07946; SWE-EVO (release-sized evolution; gpt-5.2 drops 72.8%→22.9%) 2512.18470; SWE-Bench Pro V2 (480/731 tasks repaired/retired to stay game-resistant) 2610.24206. Community tracking (pareto-evals) shows GPQA/ARC-AGI-2 moving to "legacy" and open long-horizon agentic suites (Terminal-Bench 4.0, DeepSWE) featured in 4-5/5 frontier launches.

Revised fleet adoption shortlist (delta from 2026-08-07)

1. Verifier pass — unchanged at #1, now evidence-backed. Upgrade the spec: verifier emits structured, auditable output (SEVA pattern), rules-first where possible (VPRM pattern).
2. Execution-feedback → skill loop — unchanged at #2. Add the XEPV lesson: instrument the loop (verify-rate is ~2% by default); make verification a counted step, not an aspiration.
3. Memory tiering — keep, reframe. Promotion rules are the orchestration bottleneck; budget for eventually learning them from outcomes rather than hand-writing them forever.
4. Test-time compute — demote from flat "sample-2-verify". Gate on task verifiability: the verification gap (2602.18998) means sampling only pays where selection is trustworthy.
5. Auto-skill extraction — keep "long play", add AIDE² preconditions: held-out eval set + reward-hack checks before any self-rewriting loop touches fleet code.
6. Search/planning — new optional item. If Pulse agents take on multi-step builds, the TreeFlex/GATS direction (policy-directed tree + cheap world model) is the 2026 shape of it.

Sources

Primary (all arXiv unless noted): 2602.18998, 2604.01411, 2506.12721, 2604.24957, ThinkPRM (TMLR 03/2026, openreview FPVCb0WMuN), 2606.29713, 2601.17223, 2606.04579, 2610.02351, 2606.15579, 2605.11169, 2607.27271, 2609.31874, 2610.20334, 2607.08894, 2609.33920, 2606.11662, 2610.16593, 2605.10325, 2610.20497, 2609.26457 (+ Weco AI blog), 2607.07663, 2610.20431, 2607.21461, 2610.18083, 2603.07670, 2604.01707, 2602.19320, 2605.06716, 2607.28576, 2604.06066, 2604.01212, 2606.07682, 2607.07946, 2512.18470, 2610.24206. ToolTree: ICLR 2026. ReflecTool-Bench: ACL 2026 Findings. ReAcTree: AAMAS 2026.