The Hard Part Is Knowing When to Trust the Model

RACER has two responses. It can reduce κ, the trust assigned to the base forecast, and pull that forecast toward the anchor, the nearest cached feature. At the riskiest steps, it runs the denoiser to refresh the feature, then repays that evaluation by skipping a later scheduled step. The per-prompt number of denoiser function evaluations (NFE) stays exactly equal to the base cache. The result is a closed loop around an existing forecaster, with a hard denoiser budget.

The signal gets a useful ablation in a fixed refresh-only executor. At α=0.75srcResults, Table 4, disagreement reaches 18.13srcResults, Table 4 dB peak signal-to-noise ratio (PSNR), versus 15.71srcResults, Table 4 for a horizon signal and 16.30srcResults, Table 4 for a random one. That 2.42srcResults, Table 4 dB advantage over horizon is the largest reported in the comparison. It supports the signal’s value for choosing refreshes, while leaving the full controller’s advantage to the separate shrinkage and refresh ablations.

Table 5 gives the two responses a concrete division of labor. At α=3.0srcResults, Table 5, refresh-only loses 0.20srcResults, Table 5 dB against the base, shrinkage-only gains 0.25srcResults, Table 5, and the combined policy gains 0.49srcResults, Table 5; the same ordering holds at α=7.0srcResults, Table 5. The controller is doing more than selecting refresh points: it limits the damage when refreshes are too sparse to carry the regime.

The evaluation spans SD3.5-Large and FLUX.1-dev for images, Wan2.1-14B and HunyuanVideo for video, using DrawBench, VBench, and COCO (Common Objects in Context); each model’s 50srcResults, Tasks and Evaluation metrics; Implementation details-step sampler supplies the reference. Against the Chebyshev open-loop base at matched NFE, the image tables show higher peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) for RACER across the reported regimes. On SD3.5 at α=0.25, PSNR rises from 18.35srcResults, Figure 2 table and matched-NFE comparison to 19.68srcResults, Figure 2 table and matched-NFE comparison and SSIM from 0.761srcResults, Figure 2 table and matched-NFE comparison to 0.792srcResults, Figure 2 table and matched-NFE comparison. At α=3.0, PSNR rises from 15.70srcResults, Figure 2 table and matched-NFE comparison to 16.52srcResults, Figure 2 table and matched-NFE comparison. FLUX shows the same direction with a smaller margin: at α=0.25, PSNR moves from 24.48srcResults, Figure 2 table and matched-NFE comparison to 24.62srcResults, Figure 2 table and matched-NFE comparison.

COCO points the same way: the reported PSNR delta is positive for both image models at all five listed settings, although the FLUX improvement is as small as 0.07srcResults, Table 3; matched-NFE results discussion dB. The paper extends the evaluation to video models and reports gains there, but the image tables carry the clearest weight in the case for RACER. This is a solid fixed-budget result, especially where acceleration is aggressive; it is a narrower claim about fidelity to a sampler than about universal perceptual quality.

That qualification matters. Learned Perceptual Image Patch Similarity (LPIPS), where lower is better, worsens for FLUX in several regimes, including a rise from 0.146srcResults, Figure 2 table; Evaluation metrics to 0.164srcResults, Figure 2 table; Evaluation metrics at α=0.75. The paper also reports ImageReward and the Contrastive Language–Image Pre-training (CLIP) score alongside reference fidelity. Those metrics broaden the evaluation, while PSNR and SSIM remain comparisons with a 50-step trajectory rather than human ground truth.

RACER does turn the quality margin into a measured speed result on SD3.5. Across the sweep, the paper reports 2.6srcFigure 1 caption; Results, speed paragraph× to 5.4srcFigure 1 caption; Results, speed paragraph× speedup over the 50-step sampler; in one matched-quality comparison, it reaches the open-loop base’s best quality at 0.76srcFigure 1 caption; Results, speed paragraph× its latency and 0.72srcFigure 1 caption; Results, speed paragraph× its NFE. Those are useful measurements, with a narrower scope than the headline speedup suggests. NFE counts full denoiser evaluations, while RACER also computes an observer forecast, disagreement statistics, control logic, and feature-memory traffic. The reported roughly 2% auxiliary-computation overhead is for SD3.5, and offline calibration sits outside the NFE accounting.

Calibration narrows the portability story. Image parameters transfer from SD3.5 to FLUX, while video thresholds are calibrated separately for each model and acceleration regime on a short trace. That remains training-free and is a reasonable engineering compromise; it also makes the calibration protocol part of the result. Threshold, trace-length, and calibration-objective details are too thin to make that portability fully reproducible.

The theoretical case carries less weight. The paper invokes a deterministic endpoint-error bound and a mean-squared-error (MSE) optimality analysis, yet its assumptions, derivation, and numerical tightness evidence are too lightly connected to the controller to judge how broadly the guarantee applies. The reported tables also give point estimates without prompt counts, seed counts, uncertainty intervals, or significance tests. A 0.04srcResults, Figure 2 table; Results, Implementation details dB margin in one FLUX setting is directionally encouraging; it does not establish a large effect.

RACER’s real contribution sits one layer above forecasting. Disagreement itself is a familiar predictive-uncertainty heuristic, and the paper cites that lineage. Its own baseline list includes TeaCache and ToCa, so the novelty sits in the specific controller and budget repayment, not in runtime checking as a general idea. The Taylor-base experiment reports up to 3.83srcResults, Table 6 and Figure 4 dB PSNR recovery, which suggests the pattern can travel beyond Chebyshev; one alternate forecaster remains a narrow portability test.

That distinction also bounds the ablation. The disagreement comparison swaps in horizon and random under a fixed refresh-only executor; it establishes a useful selection signal there, while leaving the contribution of disagreement to the complete policy less isolated from the controller and budget schedule.

For an engineer, feature caching should be judged as a forecaster-plus-controller system, with matched NFE and wall-clock overhead reported separately. RACER makes that case in the image experiments; its broader reliability claim deserves a narrower reading given calibration, metric tradeoffs, and limited uncertainty reporting. The 5.4× figure belongs to a measured SD3.5 setup; the reusable result is the closed loop.

The delimiters and final answer remain ordinary text tokens. Inside a latent segment, each placeholder reserves an autoregressive position, and the previous decoder hidden state is fed back as the next input embedding. Every query therefore runs four perception positions; the reasoning route runs six more, even when the visible answer is short.

Training supplies localized evidence boxes and rationales for examples labeled as requiring inference. The recipe uses one epoch of supervised fine-tuning followed by 2,500srcMethods; Section 4.1, Training data and Training and inference reinforcement-learning optimization steps, with about 20,000 supervised examples and 30,000 reinforcement-learning examples. Candidate boxes and rationales are generated by Qwen3.5-397B-A17B and filtered by Kimi-K2.5. This is a teacher-heavy way to teach the response format; the latent states’ independent faithfulness remains untested.

The component ablation points in the same direction on LVBench before reinforcement learning: the full DyLaR supervised-fine-tuning variant scores 40.5srcTable 2 overall, compared with 38.3srcTable 2 without reasoning supervision, 38.7srcTable 2 without perception grounding, and 39.1srcTable 2 when reasoning is mandatory. That supports the training recipe. Because the table is pre-reinforcement learning, it says little about how much of the final benchmark result comes from the latent design versus the later optimization. It is also one benchmark and one set of point estimates, so it supports the design more than it settles the mechanism.

The main table gives the idea a credible first test. On Qwen3-VL-4B, DyLaR reaches a nine-benchmark average of 58.2srcTable 1, versus 54.0srcTable 1; Section 4.2, Main Results for Qwen3-VL-4B-Thinking. The corresponding DyLaR averages are 57.5srcTable 1 on Qwen2.5-VL-7B, 56.7srcTable 1 on InternVL3.5-4B, and 54.2srcTable 1 on LLaVA-OneVision-7B, each above the listed same-backbone comparator. On Qwen3, the 4.2-point gain accompanies a fall in reported visible output from 1,220.7srcTable 1; Section 4.2, Main Results to 18.5srcTable 1 tokens per query.

Then the scale of the claim comes into focus. On Qwen2.5, DyLaR leads VideoAuto-R1 by 0.5 average points, while losing to it on MVBench, MMVU, and Video-Holmes. Those prior numbers are reported results from other papers, so the comparison carries protocol uncertainty; Table 1 also gives point estimates without repeated-run uncertainty. The cross-backbone pattern is worth taking seriously; the exact ranking remains unsettled.

Training also draws on the training splits of LongVideo-Reason and Video-Holmes, both evaluated in the table. The pairing alone cannot establish contamination, but it makes video-level deduplication part of the evidence needed to read those benchmark scores as generalization.

Efficiency is where the headline needs the most discipline. Table 1 reports visible tokens per query, while the method reserves latent positions and runs hidden-state computation in them. With four perception positions on every query and up to six reasoning positions on the routed path, 18.5 visible tokens is an output-format statistic; end-to-end compute remains unmeasured. The reported results include no latency, floating-point operations, memory, or throughput accounting.

The router’s adaptivity is also narrower than its name suggests. Inference chooses between zero and six reasoning positions after a fixed four-position perception block; the reported analysis gives no route frequencies, question-type breakdown, or budget sweep. In engineering terms, read it as a two-path think-or-answer gate with a fixed budget on the reasoning branch.

The LLaVA comparison makes the token story concrete: DyLaR emits 17.8srcTable 1; Section 4.2, Main Results visible tokens versus 14.1srcTable 1; Section 4.2, Main Results for its chain-of-thought baseline, although the paper attributes the baseline’s short output to weak adherence to the chain-of-thought instruction. All experiments uniformly sample 16srcSection 4.1, Training and inference frames per video, which narrows the result’s claim about long-video temporal reasoning.

For a system builder, the useful carry is the split itself: make visual grounding compact and mandatory, then allow a second latent block when the route calls for it. Price every latent position in deployment measurements. DyLaR’s practical contribution is the gate; visible-token count and compute should be kept as separate measurements.

The arithmetic is compact. For the same generated token, OCSD computes e = δF − δA = log πF(y|h, EF) − log πA(y|h, EA); the common student log-probability cancels. It then maps e to q = tanh(e/2), bounded between −1 and 1. Positive q means the Full teacher supports the token more than the Observation-Ablated teacher, and negative q means the reverse. If the scaffold shifts both scores in the same way, that shared shift disappears in the subtraction, leaving the incremental score change associated with the inserted observation.

OCSD spends that signal selectively. It computes each interaction step’s mean negative log-likelihood under the old policy, selects the top ρ = 0.2srcMethods, Eqs. (9)–(10); Results of steps within each trajectory, and, with β = 0.5srcMethods, Eq. (11); Results, rescales the GRPO advantage for tokens in those steps by 1 + β sgn(A)q. Other steps retain the original advantage. Because q is bounded, this multiplier stays nonnegative, so each token keeps the sign of its trajectory-level advantage. That local guarantee is useful; it is weaker than a guarantee about the direction of the aggregate policy gradient, which can change when token weights change.

The subtraction is a sensible control, with a real assumption attached. If the two views match in the relevant ways, shared scaffold shifts can cancel. A nonlinear language model can still let an observation change how it attends to and interprets the scaffold, leaving interaction effects in e. The design sharpens the comparison; it does not guarantee a pure causal measurement.

The benchmark pattern is broad enough to matter. Across ALFWorld, WebShop, and SearchQA, the authors train Qwen3-1.7B, Qwen3-4B, and Qwen3-8B under a unified post-training protocol. On ALFWorld, OCSD reports success rates of 55.5srcResults, main-results narrative; Tables 1–2, 82.8srcResults, main-results narrative; Tables 1–2, and 87.2srcResults, main-results narrative; Tables 1–2, respectively—8.9srcResults, main-results narrative; Tables 1–2, 12.2srcResults, main-results narrative; Tables 1–2, and 14.0srcResults, main-results narrative; Tables 1–2 points above GRPO. Its WebShop rates are 54.4srcResults, main-results narrative; Tables 1–2, 73.7srcResults, main-results narrative; Tables 1–2, and 78.1srcResults, main-results narrative; Tables 1–2, while SearchQA average Exact Match is 43.1srcResults, main-results narrative; Tables 1–2, 47.5srcResults, main-results narrative; Tables 1–2, and 49.1srcResults, main-results narrative; Tables 1–2. The domains span household manipulation, product search, and information retrieval, so the positive pattern carries more weight than a win on one narrow task.

Even so, a clean sweep is too neat a reading. In the SearchQA table for Qwen3-4B, OCSD reaches 42.5srcResults, SearchQA results table ± 0.9srcResults, SearchQA results table on in-domain Exact Match, below RLSD’s 42.9srcResults, SearchQA results table ± 0.3srcResults, SearchQA results table; its advantage appears on out-of-domain Exact Match, where it scores 48.9srcResults, SearchQA results table ± 1.0srcResults, SearchQA results table, and on the average, 47.5 ± 0.8srcResults, SearchQA results table. At 1.7B, its 43.1 ± 1.2srcResults, SearchQA results table average exceeds GRPO’s 40.9srcResults, SearchQA results table ± 0.7srcResults, SearchQA results table and GRPO+OPSD’s 42.6srcResults, SearchQA results table ± 0.4srcResults, SearchQA results table. The reported ± spreads make small differences hard to treat as decisive.

The authors also report that the calibrated residual aligns more closely with local environment feedback than Full-view support. That is the right diagnostic question for a credit signal; it still cannot establish that the residual caused the end-task gains.

The results also leave the central attribution question open. The comparison set contains Vanilla, GRPO, OPSD, GRPO+OPSD, RLSD, SDAR, and OCSD—different recipes rather than a ladder that changes one component at a time. The main results do not include a matched Full-only support baseline, a no-subtraction control, or a selector comparison testing the top 20srcResults, main comparison and hyperparameter paragraphs; Methods, Eqs. (9)–(11) percent against another selection rule. Without that ladder, the improvement could belong to the residual, the selector, the extra replay scoring, or their interaction.

That cost matters to an engineer. OCSD scores the same token under both replay views, so it requires two replay-view score computations instead of one. The main results give no wall-clock, memory, or token-cost accounting. ALFWorld and WebShop are also reported without the uncertainty estimates shown for SearchQA. The gains remain worth attention; their value for a production trainer depends on whether the second score buys enough signal for its cost.

For an engineer, OCSD offers a concrete pattern: keep the replay scaffold fixed, build an observation-ablated counterpart, bound the resulting score difference, and spend it on uncertain steps. It is a sensible experiment when the extra replay scoring is affordable. The evidence supports a narrower claim: calibrated hindsight helps across these three benchmarks in one model family, Qwen3. The causal role of the residual remains unsettled, so treat the Full-only comparison as part of the method’s price of admission. That is the useful result: a promising control for privileged replay, with the proof still attached.

The setup is mechanically clear. The frozen target first runs a batch. A deterministic extractor keeps failed episodes and compacts task constraints, selected action–observation excerpts, outcomes, and necessary environment state into one failure packet. The engineer reads it once and produces a batch-conditioned overlay; it does not solve the tasks or join their rollouts.

That overlay can intervene at four lifecycle points: episode initialization, pre-decision, pre-action, and post-feedback. Those hooks set episode state, add retrieved guidance or interface constraints to context, canonicalize, rewrite, or veto proposed actions, and trigger recovery when feedback shows that a trajectory has stalled. Validation runs before installation, and invalid patches have no effect.

The same frozen target then reruns every task, including tasks it had already solved. The reward is the average per-task difference between patched and baseline outcomes, with zero for an invalid or incomplete patch. This paired design holds task composition constant and charges the engineer for regressions on originally successful tasks. It also makes the objective transductive: the patch is written with evidence from the exact cases used to judge it. Cold-start supervised fine-tuning supplies a prior over valid edits; online group-relative policy optimization (GRPO) tunes the engineer against those realized outcomes. The target weights never move.

Table 1 gives Harness-R1 a solid first showing. On the aggregate metrics, ALFWorld rises from 40.6%srcTable 1 to 53.2%srcTable 1, WebShop success from 31.2%srcTable 1 to 42.2%srcTable 1, and DBBench success from 61.0%srcTable 1 to 65.3%srcTable 1. The equal-weight average of those three headline metrics is the abstract’s 44.3% to 53.6%. The supervised-only engineer reaches 46.4%srcTable 1, putting 7.2 points between that baseline and Harness-R1.

The table also limits the rhetoric. Reflection reaches 55.8%srcTable 1, above Harness-R1’s 53.6%. The gain is not uniform inside ALFWorld either: Pick2 falls from 47.5%srcTable 1 to 39.3%srcTable 1 even as the aggregate rises. A benchmark average can be useful for a quick read, yet here it conceals a real category regression.

After direct fine-tuning of the target agent, Agent SFT alone averages 59.2%srcTable 1; adding a target-specific Harness-R1 engineer reaches 64.2%srcTable 1. The extra lift is concentrated: ALFWorld All moves from 71.2%srcTable 1 to 84.0%srcTable 1, DBBench from 63.7%srcTable 1 to 65.7%srcTable 1, and WebShop success only from 42.6%srcTable 1 to 43.0%srcTable 1. Runtime repair can complement weight updates, though the complementarity is far stronger in the embodied-task benchmark than in WebShop.

The method’s broader claim runs into its evaluation geometry. For the main three-benchmark measurement, the engineer reads failures from a batch, emits one overlay, and the patched target is rerun on that same batch. The Methods section calls this objective transductive and specifies no iterative refinement within an instance or persistent patch memory across batches. That is a sensible mode for repairing a known incident set. It is a different test from whether one learned harness helps new tasks or retains fixes over time. The main 9.3 points should be read at that scale.

That distinction also trims the co-evolution claim. The results pair a directly fine-tuned target with a target-specific engineer, but do not run repeated updates in which target and engineer respond to one another. The evidence shows stackable interventions—weight updates and runtime edits—rather than co-evolution in the stronger continual sense.

Attribution is similarly incomplete. The comparison gives no matched compute, patch-sampling, or training-budget accounting. Table 1 gives point estimates without error bars or repeated-run uncertainty. The 7.2-point gap is consistent with an advantage from outcome training, yet the table cannot tell how stable it is or how much comes specifically from GRPO.

Harness editing itself is already a populated area. The related work places Harness-R1 alongside Meta-Harness, Agentic Harness Engineering, AutoHarness, Life-Harness, HarnessX, and AEGIS—systems that edit prompts, tools, memory, middleware, control logic, or lifecycle components. Harness-R1’s sharper distinction is the training locus: it post-trains the engineer with outcome rewards and lets it install executable hooks around an existing runtime. That is meaningful, with novelty concentrated in which component learns the edit. The “first” claim needs that narrow reading.

For someone running a tool-using system, the practical takeaway is concrete. Log failure trajectories, expose the runtime’s lifecycle as explicit hooks, validate generated patches, and score them on paired reruns that include previously successful tasks. Harness-R1 shows how to improve an agent without changing its weights.

Treat it as a batch-level repair loop—diagnose this batch, patch this batch. That is a practical capability when failures arise in context, interface, or recovery logic. It gives practitioners a new place to intervene when fine-tuning the actor leaves the agent failing at the boundary with its environment.

The most revealing test is the least flattering. Thirty non-injected, high-temperature (0.9) episodes contained 11 organic failures: seven silent aborts, three fabricated counts, and one ungrounded retrieval blend. Using the monitors with their deployment calibration and without refitting, the behavioral detectors caught only one of the three fabrications, ranked the full organic failure set at or below chance, with within-organic area under the receiver operating characteristic curve (AUROC) of 0.31srcConclusion–0.42srcConclusion, and false-alarmed on 36%srcConclusion of healthy episodes under a temperature-matched null. The paper correctly treats those detections as non-deployable evidence.

That failure mix gives the system a sensible division of labour. A five-line completion check caught all seven silent aborts with zero false positives across the 30srcConclusion; Abstract episodes. For a value that remains plausible in the observed stream, the paper assigns the decision to an external reference or judge. The pre-registered fabrication study reinforces the restraint: it had 55srcConclusion episodes but only two candidate hallucinations, below its pre-registered minimum of 10srcConclusion, and made no detection claim. That is too little data for a prevalence claim, let alone a monitor benchmark.

The content-grounding channel follows the same rule. It improves detection on research collectors where corruption is visible in result content and is inert on frameworks where corruption leaves result text unchanged. Its benefit comes from adding observability; it cannot settle a semantic error that produces the same observable result.

The content result also deserves a narrower reading. The paper gives a pooled 0.28srcAbstract; Introduction, Contribution 5-to-0.59srcAbstract; Introduction, Contribution 5 figure in the abstract, but its contribution summary gives 0.07srcAbstract; Introduction, Contribution 5 for a memoryless parent, 0.14srcAbstract; Introduction, Contribution 5 for the ESN, and 0.90srcAbstract; Introduction, Contribution 5 for a content gate over 58srcAbstract; Introduction, Contribution 5 malformed_json episodes. Until those metrics share a definition, the conditional finding is safer than any headline percentage.

The deterministic layer is the paper’s most convincing idea. A numeric-grounding verifier recomputes a run’s stated total from the tool results that run actually received, checks arithmetic combinations, and confirms that required calls were made. It needs no healthy-run null, learned threshold, or calibration. A separate shape check validates tool results against the shapes the tool can return. These rules test explicit obligations, giving them a cleaner target than asking an anomaly score to infer whether a fluent answer is true.

Those checks are lightweight once the contracts exist; portability depends on task-specific knowledge of required calls, provenance, and permissible tool-output shapes. In the reported head-to-head on the same episodes and labels, the verifier caught 60%srcAbstract of failures, with 0/63srcAbstract false positives, versus 54%srcAbstract detection and 17%srcAbstract false positives for the monitor. Adding the coverage check lifted the reported stack result to 96%srcAbstract. The comparison is useful, though 96% combines two mechanisms and the zero-false-positive figures are bounded observations from particular cohorts. The verifier’s 26srcAbstract-of-26 result on provoked fabrications tests whether the rule catches an inserted ungrounded figure; it says little about natural fabrication rates.

A later frozen-seed replication reported 54% detection, 93%srcAbstract with coverage, and 0/64srcAbstract false positives on 120srcAbstract episodes; the verifier transferred unchanged to llama3.1:8b with 110srcAbstract/110 detections and 0/10 false positives. Those are encouraging transfer checks, with small, task-specific denominators that keep the conclusion narrow.

Detection matters because it creates a chance to salvage work. When the stack flagged a run, it rolled back to the last fact-gathering step and reran live. That recovered 45%srcAbstract of failures, against 16%srcAbstract for a resampling control (p = 0.0005srcAbstract), and raised net task success from 52%srcAbstract to 73%srcAbstract for about one extra model call per run. This is the clearest practical payoff in the paper: an imperfect alarm can produce a useful recovery when the response is targeted.

The deployment hinge is how the rollback point is found. A production monitor has to identify the last fact-gathering step from information available during the run; if that choice depends on an oracle or manual annotation, the repair result measures more than the detector. The reported sub-millisecond path also covers scoring and the telemetry adapter, so the real budget includes producing semantic embeddings and token-level uncertainty, especially for a commercial API. The speed claim is useful, though it does not by itself complete a cost comparison with an auditor.

The paper leaves engineers with a usable recipe. Run a temporal monitor on every step for loops, drift, and other changes in behavior; wire completion checks to required obligations; attach numeric provenance and tool-shape checks to structured outputs; escalate plausible-value disputes to an external reference or judge; and use rollback when a safe fact-gathering boundary is observable. Each layer has a failure class it can test and a boundary it cannot cross.

The ESN and CUSUM are familiar components, so the contribution lives in their composition, the cross-framework evaluation, and the decision to hand each failure type to a mechanism suited to it. The organic evidence is too small to support the paper’s broadest coverage language, and the behavioral monitor’s poor transfer to organic fabrication is a result teams should take seriously. The durable takeaway is narrower and more useful: cheap telemetry can serve as an initial monitoring layer, while verification and escalation handle failures that telemetry alone cannot settle.

Authority-Chain Hijack (ACH) is the mechanism at the center. The attacker keeps trajectory memory of earlier queries, visits, and injections; a planner maps each new event to a context-aware payload; and a reusable strategy keeps successive results and landing pages coherent. The aim is a chain of seemingly corroborating sources. The agent’s own search behavior supplies the feedback loop: it follows a lead, searches again, and uses what it sees in its next decision while the attacker adapts the next payload.

That is the paper’s best reframing. The object being poisoned is the path by which evidence is gathered; a page is one step in it. The attack tries to make cross-checking work for the attacker, so that later sources appear to validate an earlier intervention.

Trace-Guided Strategy Evolution (TGSE) extends this attacker by using successful and failed execution traces to explore a tree of strategy variants. It is presented as a way to reduce manual redesign, though the starting ACH strategy is itself expert-refined. The automation is useful; it also makes the result a runtime system built around an evolving strategy.

The evaluation is more demanding than a single poisoned-snippet test. It uses SafeSearch across five risk families: Advertisement, Bias-Inducing, Fake-Information, Harmful-Output, and Prompt-Injection. The authors remove site-locked cases, reserve 20srcResults §3.1. cases per family for strategy development, and evaluate the remaining 187srcResults §3.1. held-out cases. Victims share one DeepResearch-style reasoning-and-acting (ReAct) scaffold, with six backends: Qwen3.5-9B, Gemma4-31B, DeepResearch, MiniMax-M2.5, Qwen3-235B-A22B-Instruct-2507srcResults §3.3. (Qwen3), and Kimi-K2.5. Each case is repeated five times. Attack Success Rate (ASR) is the mean trajectory-level success rate; MaxN is the fraction of cases with at least one successful repeat; Overall is the macro-average across the five families.

Within that fixed comparison, ACH has the highest Overall ASR for all six victim models, beating the strongest non-ACH baseline by 13.0srcResults §3.3 and Figure 4(a) discussion.–36.1srcResults §3.3 and Figure 4(a) discussion. percentage points. That win is not uniform across tasks: the paper notes that AgentDojo snippets exceed ACH in a few Prompt-Injection cells, where direct marker or formatting obedience is enough. ACH is consistently stronger on Advertisement, Bias-Inducing, Fake-Information, and Harmful-Output, where evidence uptake matters.

The paper reports 55.9%srcAbstract. ASR and 83.3%srcAbstract. MaxN ASR for ACH. TGSE’s strongest single setting reaches 71.4%srcAbstract. ASR and 95.0%srcAbstract. MaxN on held-out evaluation. Those are substantial benchmark effects, and they make the trajectory vulnerability hard to dismiss.

Both the metric and the split change how those numbers should be read. An 83.3% MaxN result means that at least one of five runs succeeded on that fraction of cases; it is not a single-run attack probability. The abstract calls the ACH result a result on the “full SafeSearch test split,” while the evaluation section describes a filtered open-web test with a development tranche and 187 held-out cases. That distinction turns a broad-sounding headline into a more precise result: repeated intervention works on this benchmark slice.

The hardest question is what exactly won. ACH is credited with the authority-chain strategy, but its runtime also carries trajectory state and a planner that adapts payloads to each event. Objective Drift, the comparison’s dynamic baseline, learns from training-split feedback but uses fixed read-only memory at test time. The matchup therefore changes both the strategy and the attacker’s ability to react online. No reported ablation holds the runtime constant while removing the authority-chain strategy, or matches its adaptivity with a different strategy. The 13.0–36.1-point advantage is an end-to-end difference between configurations; it cannot isolate authority-chain corroboration as the cause.

That matters because the paper explains success as evidence surviving follow-up searches, competing sources, and verification. Its scaffold permits those behaviors, and the paper defines injected-visit rate plus ASR conditioned on an injected visit as trajectory gates. The headline comparison still bundles page content, payload generation, online adaptation, planner behavior, and the checklist judge. The result establishes inducibility, while leaving the decisive component unidentified.

Scope adds another limit. The six backends vary the model while the scaffold, SafeSearch benchmark, and controlled tool interface stay fixed. Unless otherwise stated, the attacker and checklist judge use Qwen3, which is also one victim backend; no independent judge validation is reported. Five repeats per case are reported, but there are no confidence intervals, seed results, or variance analysis. That leaves a meaningful result under a controlled setup, with less evidence for transfer to other agent protocols or for the precision of the reported margins.

TGSE has a related qualification. ACH is expert-refined, and the reported TGSE number is the strongest single setting. Trace-guided evolution may reduce later manual redesign; it begins with a human-designed strategy and does not establish typical performance across evolved variants.

For builders, the useful output is a test condition. Run the full search loop against an adversary that can coordinate observations across turns, then inspect which result caused a visit, which later query reinforced it, and what evidence reached the final answer. A check for an isolated injection string can miss a failure whose force comes from sequence and consistency.

The study earns attention as a trajectory-level vulnerability demonstration. Its scope is bounded by one scaffold, one benchmark, and an intermediary allowed at most one injection per query plus controlled landing pages. Treat the score as conditional evidence and the mechanism as the reusable lesson. The agent’s search history can become the attacker’s evidence chain.

A one-factor model explained 47%srcResults, Figure 2 of the variance in benchmark abilities and had poor absolute fit, with root-mean-square error of approximation (RMSEA) 0.32srcResults, Figure 2. Two factors explained 69%srcResults, Figure 2, but the comparative fit index (CFI) was 0.88srcResults, Figure 2, below the conventional 0.95 threshold. Three factors explained 77%srcResults, Figure 2 and reached CFI 0.97srcResults, Figure 2.

At the benchmark level, the refusal-strictness factor places HarmBench and SORRY-Bench at one end and OR-Bench-Hard at the other. HarmBench and SORRY-Bench correlate at 0.89srcResults, Figure 2, and their correlations with OR-Bench-Hard range from −0.77srcResults, Figure 2 to −0.67srcResults, Figure 2. The authors label a second factor truthfulness, defined by TruthfulQA and Do-Not-Answer; it is almost independent of refusal strictness (φ = 0.01srcResults, Figure 2). AdvBench and Health-ORSC define contextual harm through a correlation of 0.58srcResults, Figure 2, with a factor correlation of 0.65srcResults, Figure 2 with refusal strictness.

That decomposition gives evaluators a better diagnosis than a leaderboard position. A model can be unusually strict about refusals, differ on contextual harm, and land somewhere else on truthfulness; the average hides which profile produced the score.

The caution is in treating the labels as settled constructs. The 77% is variance inside an estimated eight-benchmark ability matrix, so it is evidence about this battery. Truthfulness and contextual harm each rest on two benchmark indicators, and the retained factor count changes with extraction: minimum-residual parallel analysis keeps three, while the principal-component variant keeps two. The result makes a strong case that this suite is multidimensional. It makes a smaller case for three general properties of safe models.

Once the target is explicit, selection starts to pay. The authors built one fixed 25srcResults, Figure 3-item test for each factor. Together, the tests use 75srcResults, Figure 3 items, under 2%srcResults, Figure 3 of the full suite. On held-out models, the tests produced lower root-mean-square error (RMSE) than random subsets of the same size: 0.047srcResults, Figure 3 versus 0.062srcResults, Figure 3 for refusal strictness, 0.041srcResults, Figure 3 versus 0.048srcResults, Figure 3 for truthfulness, and 0.065srcResults, Figure 3 versus 0.067srcResults, Figure 3 for contextual harm. The corresponding rank correlations were 0.94srcResults, Figure 3, 0.83srcResults, Figure 3, and 0.89, versus 0.90srcResults, Figure 3, 0.67, and 0.87srcResults, Figure 3 for random subsets.

Those comparisons make the strongest case for target-specific compression. The contextual-harm margin is only 0.002, while the refusal and truthfulness gains are clearer. Within the truthfulness test, TruthfulQA score estimation reached RMSE 0.052srcResults, Figure 3 versus 0.065 for a random baseline; spreading the same item budget across one pooled all-benchmark test reduced rank recovery from 0.73srcResults, Figure 3 to 0.60srcResults, Figure 3. A short test works when its target remains explicit.

The sharper efficiency result comes from computerized adaptive testing (CAT), which chooses later items using earlier responses. At roughly ten items, CAT estimated HarmBench, SORRY-Bench, and OR-Bench-Hard scores with 18srcResults, adaptive-testing paragraph; Figure 3–34%srcResults, adaptive-testing paragraph; Figure 3 lower error than random ten-item subsets and recovered their full rankings at correlations of 0.92srcResults, adaptive-testing paragraph; Figure 3–0.94. Against the full named benchmarks, that is a 97–99% reduction in item count.

That operating point is favorable, and the rest of the curve matters. At 25 items, CAT’s error advantage narrowed to 4–21%srcResults, adaptive-testing paragraph. By 50srcResults, adaptive-testing paragraph items, random subsampling matched or beat CAT on five of the eight benchmarks, although seven still reached rank correlation of at least 0.90. CAT did have lower error than random at every tested budget for the sign-aligned composite of all eight benchmarks, with 22%srcResults, Figure 3 lower error at 25 items. That composite shows recoverability of a constructed aggregate; universal safety validity is a separate claim.

These details set the right scale for the headline. The 97–99% figure is an item-count saving for three benchmarks at one budget; end-to-end deployment also involves calibration and sequential selection. IRT can make a benchmark-specific evaluation far smaller when the target and calibration align. The ten-item result should stay labelled as a result for those named benchmarks.

The third use, auditing, carries the largest inferential leap. Person-fit statistics ask whether a model’s response pattern is plausible given its estimated ability and the properties of the items. On prompted sandbagging model organisms, the authors report that IRT-derived person-fit catches at least 80%srcAbstract; Contributions of selective sandbagging. That is a useful demonstration that the calibration can flag an unusual response pattern.

Then comes the causal leap. The authors say the same machinery can identify model substitutions and changes behind application programming interface (API) endpoints. Person-fit sees an anomaly; it does not identify its cause. Prompt changes, stochastic outputs, judge noise, and ordinary behavior drift can all produce unusual patterns. The 80% figure is detection under a prompted sandbagging setup, while an operational audit also turns on false positives under benign variation. The audit result earns a place as an alerting component; the API-identification claim remains ahead of the demonstrated evidence.

For evaluators, the immediate use is concrete. Report whether a result concerns refusal strictness, truthfulness, or contextual harm. Use the fixed 25-item tests when the same target and calibration apply; use CAT’s ten-item result as a benchmark-specific shortcut for HarmBench, SORRY-Bench, and OR-Bench-Hard. When decisions depend on thresholds or rare failures, keep coverage beyond ranking-oriented sampling: rank correlation answers ordering, while those decisions turn on individual outcomes.

This is a meaningful improvement to how safety benchmarks can be read and reduced. It makes the measurement target visible, trims redundant work in selected cases, and keeps the calibration attached to the claim. Report the axes before the average, and carry the calibration with every shortcut.