The Model Is Becoming the Middle Layer

The attack works by making the copying step easy. EvoMal plants apparently ordinary utility skills whose code wraps an interchangeable malicious payload in a three-layer banner. The banner uses benign-looking structure—including a module-init hook and an @_telemetry decorator—to prompt reproduction of the enclosed source. The attacker never needs the agent to invoke the planted skill: retrieved skills appear as source text, and the harness prevents calls to them by name. That isolates the create path.

The rest of the apparatus is fixed across the experiments. mini-SWE-agent supplies the action loop; a Voyager-style SkillManager retrieves the top five skills through BGE-M3 embeddings and ChromaDB. The benign library contains 232srcResults, experimental setup and Figure 3 software-engineering helper entries, and the same setup runs across six model backends. The task pool contains 153srcResults, task selection Python tasks selected as tool-relevant from the 500srcResults, task selection-task SWE-bench Verified split. For each issue, the agent is asked to write a reusable tool.

That design gives the experiment a clear observation point and a built-in limitation. It concentrates every trial on exactly the behavior EvoMal needs—persistent skill authoring. ASPR from this setup is a conditional risk measure for that loop; the frequency of the loop in ordinary coding-agent use remains outside the experiment.

The headline number measures copying. Agent self-poisoning rate (ASPR) is the fraction of tasks in which a retrieved planted skill is reproduced in a skill the agent authors and stores. ASPR combines two chances: the planted skill must enter the top-five context, and the agent must reproduce it once retrieved. Across the six models, full-banner ASPR ranged from 20.3%srcResults, §4.2; Abstract to 41.8%srcResults, §4.2; Abstract on the 153-task set. In the reported ablation, the full three-layer banner reached 41.8%, while DeepSeek-V4-Pro reached 11.1%srcResults, Table 2; Abstract with the payload alone. The wrapper amplified the channel; the payload-only control shows that copying survives its removal.

ASPR and execution are separate measurements. The paper defines a callback rate for a copied payload that runs and contacts the attacker’s command-and-control endpoint. A banner can be reproduced and stored without completing that callback, so ASPR is a copying measure. The 20.3–41.8% range establishes a sizeable copying rate in this harness; credential theft, backdoor installation, and other downstream harm sit beyond what ASPR measures.

That calibration sets the scale of the claim. Six backends give the mechanism breadth, but the shared harness and forced authoring frame keep the rate conditional. Prevalence in deployed coding agents remains unmeasured.

Once an authored copy returns to the library, later tasks can retrieve and reproduce it without the original plant. The authors report 4.9srcAbstract–9.0srcAbstract times as many malicious skills as were initially planted, and a round-five ASPR of 68%srcAbstract for Qwen3 after the planted skills were removed.

The multiplier is a headline aggregate. Because the Methods describe the starting set only as a “handful” of skills, it gives the expansion without a seed count or a round-by-round growth curve. More important, a persistent library makes successive tasks dependent on one another. Later retrieval sees artifacts produced earlier, so the round-five figure is best read as continued propagation in this memory loop rather than a general worm reproduction rate.

Targeting comes with a qualifier. Tailoring planted descriptions to a single task family raised ASPR to 86.7%srcAbstract; Methods, Adversary goals G3 without victim-specific knowledge. That is a task-family backdoor built from a publicly known task type, a narrower capability than blind targeting of an unknown victim.

The counter-prompt attacks the authoring moment. The authors report that it reduced EvoMal’s ASPR to at most 6.7%srcAbstract; Conclusion, with no significant task-completion loss in the evaluation. That is the right kind of intervention for this mechanism: it changes how the agent treats retrieved source before the source becomes a library entry.

Its meaning remains narrow. The reduction is in ASPR, while the paper separates ASPR from callback, so it shows suppression of the tested copying behavior. Robustness to a paraphrased banner, a different payload, or an attacker adapting to the prompt remains open.

That leaves a practical rule for anyone building persistent skill memory. Agent-authored code needs provenance and the same admission checks as community code; where the system cannot establish that lineage, quarantine keeps unverified code out of automatic reinsertion. The mechanism deserves a place in threat models; the headline percentages belong to this high-exposure loop. An artifact written by the agent has earned no extra trust merely because the attacker never submitted that exact file.

ASIL presents one agent-facing contract across several backends. For each application, it reaches structured state and executable operations through the deepest feasible path: structured file formats, native scripting runtimes, or service APIs. That gives one interaction form across heterogeneous software, covering 300srcAbstract; Results single-application tasks and 80srcAbstract; Results multi-application tasks in the 15-application benchmark.

That scope is the paper’s strongest contribution. Structured state and executable operations already underpin programmatic tool-use benchmarks such as AppWorld and application-native interfaces, including draw.io’s MCP. ASIL’s distinctive contribution is to assemble that pattern across these heterogeneous applications and test it against pixels. That makes the work a systems contribution grounded in integration and evaluation.

The difference is visible in the unit of work. Renaming a layer, changing a property, or triggering an export are semantic goals; clicks and key presses are the route to them. Structured state can expose document structure before the agent commits to a plan, and semantic operations can produce effects that validators check. The same modality is also what the training experiment tests, giving the interface a role beyond inference.

The authors say the two modes share task definitions, initial states, and validators. Even granting that common target, the operational resources diverge. ASIL receives a 15-step budget and averages fewer than five executed actions; the repaired GUI baseline uses 50. Those are unlike units: an ASIL action can invoke a file or API operation, while the GUI baseline must express intent through low-level events.

Figure 5 makes the asymmetry concrete. On a LibreOffice task, GUI control spends its trajectory in a repeated keypress-and-paste loop, while ASIL uses one `modify_file` action to write the spreadsheet state. It is an effective demonstration of the cost of pixel control. It also folds direct artifact mutation into the ASIL advantage. If the validator checks only final spreadsheet state, ASIL has an operation with no GUI counterpart. The benchmark therefore measures a software-native route against a GUI route; it cannot apportion the gain among structured observation, semantic action granularity, and backend access.

Resource accounting in the reported comparison consists of action caps and success rates; there is no matched wall-clock, token, dollar, or backend-execution budget. The 54.6-point sonnet4.6 gap—81.2srcResults, Table 1 versus 26.6srcResults, Table 1—should be read as evidence about those two engineered routes, with the contribution of each ingredient unresolved.

An easier reference band makes the context less one-sided. On easy60, a single-application slice drawn from the same 380 tasks and calibrated by structural proxies against OSWorld-369, the 50-step GUI scores rise to 15.0% strict for GPT-5.4 and 53.3% for sonnet4.6. The authors call it OSWorld-comparable, not an exact difficulty match. That caveat belongs beside the numbers: the full benchmark is deliberately harder, and the easy band leaves the access-path confound intact.

ASIL’s native-interface comparisons sharpen the claim. On LibreOffice tasks, GPT-5.4 scores 95.0% strict with ASIL versus 66.7% with UNO; sonnet4.6 scores 98.3% versus 60.0%. The differences are 28.3 and 38.3 points. These results show that interface design can matter among structured routes, although the access surfaces are not specified as matched capability sets. They remain comparisons of interface-plus-backend packages.

draw.io supplies the check against a universal rule. GPT-5.4 records 55.0% strict with both ASIL and MCP, with mean scores of 80.9srcResults, Table 1 and 80.7srcResults, Table 1. sonnet4.6 scores 25.0% with ASIL against 55.0% with MCP, and 46.7srcResults, Table 1 against 81.9srcResults, Table 1 on the mean. On these rows, the claim that ASIL only matches MCP is model-dependent: it fits GPT-5.4’s strict score, while sonnet4.6 favors MCP by 30 points. The useful conclusion is narrower: software-native interfaces help, while their contract, capability, and fit to the model shape the outcome.

ASIL is also more than an inference-time shortcut. On the ASIL / 15max 380-task aggregate, supervised fine-tuning (SFT) moves Qwen3.5-2B from 58.0% to 72.1% and Qwen3.5-9B from 66.6% to 80.4%. Resource-limited on-policy reinforcement learning then reaches 74.4% and 82.2%. Those gains make the modality a plausible training target for smaller models alongside its inference-time use.

The training table is a set of point estimates. Qwen3.5-2B’s SFT score falls on Multi-App from 75.1% to 72.5%, and Qwen3.5-9B’s falls on Files from 97.5% to 83.8%. The table reports no seeds or uncertainty information. Overall gains are encouraging; their stability across runs and task types remains unmeasured.

Builders can take a clear rule from ASIL. When an application has a stable file format, native scripting runtime, or service API, expose structured state and intent-level operations before asking an agent to navigate the rendered surface. Keep screenshot-and-click for the parts of the application with no deeper, stable route. That is a practical architecture, and the benchmark gives it real weight.

Readers should carry a narrower claim than the raw scores invite. ASIL shows the value of meeting software at the level of state and operations; its comparison leaves that value entangled with direct file/API access, heterogeneous backends, and incomparable action units. The result earns a place in the agent stack as an engineering direction. Its score should be treated as evidence for that architecture, with causal attribution kept narrower. Expose state first; keep clicks as the fallback.

Lengths are unknown before generation, so TailSieve uses an earlier round as a signal for the next one. At a cutoff, groups that remain unfinished become candidates for the next round. The authors’ premise is that relative tail behavior persists across policy updates; the reported evidence supplies no cross-round recall or overlap statistic. That premise is doing real work in the design.

A hierarchical controller adjusts two coupled quantities: how many groups to isolate and how many replicas to give each pool. It uses pool completion times, response-work history, and a measured concurrency–throughput model. Capacity changes the decoding rate as well as the queue, so the best tail/bulk split is workload-specific.

The protocol keeps policy-update groups intact. Selected requests are regenerated from scratch under the current policy; partial responses, prefix tokens, and KV-cache state are not reused. That keeps the routing decision compatible with current-policy sampling, while making the work spent at the cutoff part of the cost that any speedup calculation must carry.

The experiment is concrete. TailSieve was tested on five Qwen configurations on one eight-GPU server using vLLM, with 64srcResults, experimental setup prompt groups, 512srcResults, experimental setup requests, eight responses per group, and a 16K maximum output length. The total replica budget stayed fixed: the larger configurations used four TP2 replicas, while the 4B and 2B configurations used eight TP1 replicas. The workloads were DeepScaleR mathematical reasoning and KodCode-Light-RL-10K coding.

With speculative decoding disabled, full TailSieve beat uniform group routing in every table cell, from 1.112srcResults, Table 1× to 1.670srcResults, Table 1×. The maximum, 1.670×, came on Qwen3.5-35B-A3B with KodCode-10K; the smallest, 1.112×, came on Qwen3-30B-A3B with DeepScaleR. The spread is informative: the method has the most room when a few long responses dominate the baseline makespan and isolation creates a useful decoding-concurrency difference. Where those conditions weaken, the gain is modest.

The fixed-allocation comparison gives the adaptive part some support. Full TailSieve is higher than TailSieve (Fixed Replica Allocation) in nine of ten cells; on the headline model/workload pair, it reaches 1.670× versus 1.560srcResults, Table 1×. That is evidence for tuning the tail/bulk replica split, beyond merely separating requests.

But it leaves the candidate signal untested. No reported ablation swaps partial-rollout candidates for random, stale, predicted, or oracle candidates. The table supports the assembled system and the adaptive allocation component; it cannot tell how much of the gain comes from partial-rollout guidance itself.

Those numbers come with a specific protocol label. Evaluation began only after the joint allocation had converged and averaged the next three consecutive rollout rounds. The study gives no convergence duration, warm-up cost, later steady-state rounds, absolute step times, or utilization figures. It also gives no timing breakdown for the cutoff rollout, controller, throughput-model measurement, allocation bookkeeping, or work spent before the cutoff that is not reused. The result is evidence about that protocol state, with whole-system cost still unmeasured.

Comparison also needs care. The Uniform Oracle, StreamRL-style Oracle, and Seer-style Oracle rows assume realized generation lengths are known before routing. They are useful reference points, though they have information unavailable to an ordinary online router. TailSieve also trails Seer-style Oracle on DeepScaleR/Qwen3-4B, 1.254srcResults, Table 1× versus 1.358srcResults, Table 1×, while RollPacker is cited as prior work and has no row in Table 1. One StreamRL-style Oracle cell falls below uniform at 0.928srcResults, Table 1× on DeepScaleR/Qwen3-30B-A3B. The table supports a promising comparison against uniform routing and leaves performance against every cited scheduler open.

Scope is equally clear. All experiments use the same eight-GPU server, vLLM backend, and Qwen family; the step-wise tests cover one math and one coding workload, and the end-to-end reinforcement-learning run uses only the math prompts. The 16K cap truncates approximately 3%srcResults, experimental setup; reported ablation coverage of math responses and 0.1%srcResults, experimental setup; reported ablation coverage of coding responses, with no reported sensitivity when the cap, group size, or batch size changes. Baseline latency is averaged over three repeated runs and TailSieve over three consecutive rounds; no standard deviations, confidence intervals, independent TailSieve seeds, or significance tests are reported.

That is a single-server systems result. It says something useful about the chosen regime; it says less about other GPU topologies, interconnects, or rollout distributions.

The second headline is weaker still. The abstract and conclusion claim up to 2.59srcAbstract; Conclusion; Results× over uniform when route-specialized MTP or DFlash is added, but the reported results give no corresponding table, absolute timings, or resource-matched denominator. The end-to-end run is described as maintaining comparable training quality, yet no reward values, learning curves, or time-to-target-quality comparison are reported. Those claims remain separate from the routing result.

For an engineer running synchronous rollouts, the practical takeaway is narrower than the headline and more useful: measure whether a small set of groups dominates step makespan, then test tail isolation together with adaptive replica splitting. TailSieve offers a training-free way to seed that decision from partial work and regenerates selected requests from scratch under the current policy. Use the routing-only 1.67× as the best cell of a three-round, eight-GPU experiment; first price the cutoff work and verify the tail on the workload at hand. That is enough to put tail routing in the systems toolbox, provided the workload is tail-dominated.

PROOF-Gen adds a feedback loop to the ordinary pipeline. After a teacher failure, a reflector reads the execution trace and the verifier’s feedback, including the failed dimensions and assertions, then writes a natural-language cheatsheet for that one task. The cheatsheet is appended to the teacher’s system prompt, GPT-4o starts the task again from scratch, and the loop continues for up to K=10srcMethods iterations. GEPA supplies the prompt optimizer. When the trajectory passes every evaluator, PROOF-Gen keeps it, deletes the cheatsheet, and adds the resulting trace to the SFT pool alongside the ordinary filtered examples.

Several ingredients are familiar. The paper’s Figure 1 places the pipeline beside rejection sampling, STaR, and ReST. Its distinctive move is to overfit a disposable prompt to one scenario rather than search for a prompt that generalizes, then discard that prompt. That composition is useful: it turns feedback into a demonstration without asking the student to learn the scaffold.

The deletion is a data-cleaning step, not an information firewall. The reflector sees the evaluator’s assertions, and the teacher’s revised behavior is shaped by them. In telecom, environment assertions determine pass or fail. The output is therefore a demonstration that passes the evaluator supplying the feedback; its status as evaluator-independent correctness is a separate claim.

Table 2 makes the recovery result concrete—and bounds it. In τ2-bench’s telecom domain, the training split contains 2,171srcTable 2 tasks. GPT-4o passes 158srcTable 2 of them, or 7.3%srcAbstract; Table 2, leaving 2,013srcTable 2 failed tasks. PROOF-Gen sends a fixed random sample of 300srcTable 2 failures to the optimizer; 279srcTable 2 produce passing trajectories, a 93.0% recovery rate. That is 279 out of the attempted 300, rather than a population estimate for all 2,013 failures.

BFCL v4 multi-turn provides a useful contrast. Its training split has 560srcTable 2 tasks, of which the teacher passes 296srcTable 2 and fails 264srcTable 2. All 264 failures enter recovery, and 89srcTable 2 are recovered, for a 33.7% rate. The paper attributes the gap to selection: on BFCL, the teacher already solves the easier tasks, leaving a harder failure set. That reading is plausible, while also showing why the telecom percentage cannot stand in for general recovery ability.

At the student level, the paper reports Qwen3-4B-Instruct-2507srcAbstract; Results moving from Pass^1=0.132srcAbstract; Results with filtered-only data to 0.529srcAbstract; Results with PROOF-Gen, and Gemma 4 E4B-it gaining 7.2srcAbstract; Results percentage points on BFCL v4 multi-turn. Pass^1 counts tasks that pass at least once across three temperature-0 trials, averaging user-simulator and serving variation. It is a meaningful end-to-end measure, though it is not a single-run success rate.

There is also a scope mismatch in the paper’s headline accounting. The abstract says 57%srcAbstract; Table 2 of τ2-bench teacher trials fail, while Table 2’s primary telecom setup implies a 92.7%srcAbstract; Table 2 failure rate from its 7.3% pass rate. Those figures may use different scopes, but the paper does not reconcile them. The table’s explicit denominator is the safer basis for reading the recovery result.

The cleanest objection sits in the training pool itself. On telecom, filtered-only contributes 158 trajectories. PROOF-Gen contributes those same filtered examples plus 279 recovered ones, for 437srcTable 2 total; recovered examples make up 64%srcTable 2 of the final pool. The Qwen comparison therefore changes both how examples were generated and how many successful examples the student receives. A larger pool could be responsible for some or all of the lift.

The reported setup supplies no trajectory-, token-, teacher-call-, or cost-matched comparison with more sampling. The introduction argues that higher temperature and repeated rejection cannot fix strategic failures, but GPT-4o generates one trajectory per task at temperature 0 in the experiment; there is no matched best-of-N, temperature, or partial-credit control. Nor is there an ablation separating reflector quality, accumulated cheatsheet guidance, GEPA, and the iteration budget.

The evaluation uses three temperature-0 trials to average user-simulator and serving nondeterminism. Those trials do not address variation across fine-tuning seeds, and no uncertainty estimates are reported. The result is enough to show that a larger pool of evaluator-passing examples can produce a higher reported score. It does not isolate the causal contribution of per-scenario optimization.

The generalization story is narrower than the method’s framing. The primary τ2-bench experiment uses only telecom, chosen because it has the largest task space and lowest teacher pass rate. Its training and evaluation task instances are disjoint, but parameterized tasks can share templates. BFCL supplies a second benchmark with a per-category 70srcResults; Table 1; Appendix L/30srcResults; Table 1; Appendix L split—560 training and 240srcResults; Table 1; Appendix L held-out tasks—yet the reported split does not establish separation at the function, template, or scenario-logic level. These checks cover two settings; they do not establish broad transfer to unseen tool-use situations.

The models are Qwen3-4B-Instruct-2507 and Gemma 4 E4B-it, with 4B and 4.5B effective parameters, respectively. GPT-4o serves as teacher, while GPT-5.1srcResults and GPT-5.4srcResults act as high-reasoning reflectors for the two benchmark settings. That gives the recipe some cross-model evidence, while keeping the experimental family small.

The abstract additionally reports 6.3srcAbstract percentage points of goal-completion improvement in a deployed pipeline, 1.5srcAbstract points after transfer to an on-device model, and gains of 1.7srcAbstract to 5.0srcAbstract points across response-quality metrics, with positive transfer in every locale and a 1.48srcAbstract-point non-English average. Those numbers would make the method practically important. The operational bill, however, is not quantified: the loop uses high-reasoning reflectors and permits ten iterations per failure, while the reported setup gives no token usage, price, wall-clock latency, or cost-normalized baseline.

For a production team, the actionable reading is narrower: use PROOF-Gen as a near-miss mining stage when the execution evaluator is trustworthy, and compare it with an equally large pool of ordinary teacher samples before assigning the gain to prompt optimization. The paper makes that recipe credible; its case as a general distillation advance remains unmade.

The main pattern is clean. All 30srcFinding 1; Table 2 generator–judge pairs declined strictly from easy to medium to hard under both conditions, and the without-GT decline was roughly 1.5srcFinding 1; Table 2 times the with-GT decline. The practical implication is immediate: reliability on easy call plans is a poor proxy for reliability when ambiguity and structural demands rise.

One qualification matters. Rewrite validation confirmed the medium-to-hard step as a difficulty increase on 93.9%src§3.2; Finding 1 scope note; Appendix N of records, while easy-to-medium received unanimous validation on only 58.1%src§3.2; Finding 1 scope note; Appendix N; roughly 41.9%src§3.2; Finding 1 scope note; Appendix N looked more like paraphrases than a strict increase. The upper step carries the cleaner evidence.

On hard queries without GT, the six judges clustered within each generator block, and the paper summarised that convergence as a 77srcAbstract; Finding 1; Table 2–82%srcAbstract; Finding 1; Table 2 task-level ceiling. The rows for the six primary judges in Table 2 make the claim less universal: hard/no-GT cells ran from 69.0%srcAbstract; Finding 1; Table 2 to 84.6%srcAbstract; Finding 1; Table 2 across the five generator blocks, a 15.6srcAbstract; Finding 1; Table 2-point spread. Averaging across generators can recover the reported band; individual generator–judge cells do not sit inside it. Prometheus-2 appears as a judge-specialised baseline but is excluded from those six-judge statistics; its hard/no-GT scores run about 58.4srcTable 2 and its baseline note–61.5%srcTable 2 and its baseline note. The reported ceiling belongs to the chosen primary panel. Conditional on a given output distribution, judge capacity has limited leverage on the hardest synthetic plans.

Ground-truth access is supposed to add information. In the paper’s aggregate results, it helped QwQ-32B and GPT-OSS-120B, yet lowered alignment for GPT-5.4srcFinding 2; Figure 3; Conclusion by 1.5 percentage points and Gemini-2.5srcFinding 2; Figure 3; Conclusion-Pro by 3.9srcFinding 2; Figure 3; Conclusion points. The reported effect concentrated in sequence accuracy: frontier judges appeared to anchor on the reference order and penalise functionally equivalent sequences. A DAG makes that failure mode plausible because independent branches can admit more than one valid order.

Over-anchoring remains an interpretation rather than a diagnosis. Showing the trace also changes prompt length and framing, and the result depends on what the programmatic reference treats as equivalent. The robust practical conclusion is that with-GT and without-GT are different measurement regimes.

The leaderboard splits along the same line. With ground truth, the paper reports QwQ-32B as the closest match to the programmatic reference; in a 120srcConclusion; Appendix G-record hard-query human validation, GPT-OSS-120B was reported as most human-aligned. That distinction is valuable: an evaluator can agree with the scorer and disagree with people. The human result is useful as a cross-check; it does not settle which judge should be deployed in every setting.

Structured evaluation rubrics improved alignment by 4.8srcAbstract; §4.2; Conclusion–6.5srcAbstract; §4.2; Conclusion percentage points in the tested ablations, although the gain varied by judge–generator pair. Chain-of-thought (CoT) changed alignment by at most 0.3srcAbstract; §4.2; Conclusion points across 24srcAbstract; §4.2; Conclusion paired comparisons, and temperature produced a spread of no more than 0.25srcAbstract; §4.2; Conclusion points in two judge–generator pairings. Within those tests, an explicit checklist mattered more than asking for longer reasoning or turning the sampling knob.

That is the most actionable part of the paper. It also argues for calibration at the exact pairing and prompt, since the rubric effect did not generalise uniformly. Use the rubric as a starting configuration, then remeasure its quality and operational cost.

The synthetic design is what makes the comparison controlled, and it also sets the boundary of the result. The pipeline synthesised domain scenarios, typed function inventories, pseudocode, and ordered traces, then compared generated call plans with a deterministic reference. It covered six dependency patterns, but the described evaluation stopped before execution against live or replayable tools: no state transitions, tool feedback, side effects, error recovery, or multi-turn interaction entered the measured task. AgentJudgeBench therefore calibrates structural plan judging, a valuable slice of agent evaluation with a much smaller surface area than execution reliability.

That reference is the hidden variable. Its quality gates checked JSON-schema validity, argument types, trace consistency, argument sufficiency, grounding alignment, and naturalness. The scoring description names four structural metrics while leaving the equivalence rules for parallel order, optional branches, loop-like cases, parameter variants, and partial completion unclear. Because the paper highlights judges penalising functionally equivalent but structurally deviant sequences, a strict reference could turn the measured ceiling into a ceiling on trace conformity. The metrics are useful diagnostics; their connection to end-to-end success still needs to be earned.

One arithmetic wrinkle sharpens the point. Using the stated expansion, 3,808 records crossed with three difficulty variants, five generators, and six judges imply 342,720src§3.2; §4 factorial design; Appendix D reference generator–judge–variant tuples. The results report 321,648src§3.2; §4 factorial design; Appendix D reference valid paired tuples, leaving 21,072 cases outside that total; the main design description does not say whether their absence was random or selective. The three variants also share their underlying structure and reference trace, so the nominal count is larger than the number of independent task designs unless the analysis accounts for clustering.

Practitioners should use the benchmark as a calibration layer: stratify scores by difficulty and generator, keep reference-exposed and reference-free results separate, and inspect the reference’s treatment of valid alternative plans. For an executing system, pair it with execution outcomes or human checks. The 77–82% figure is a warning about this measurement regime; it is a poor candidate for a law of agent evaluation.

The mechanics are straightforward, which is part of the appeal. The team downloads with 2,000srcSection 3.1 virtual servers coordinated by Celery and yt-dlp, using a residential proxy network; it filters videos to 10 seconds–30srcSection 3.1 minutes before scene detection and motion filtering. Qwen3-VL-2B-Instruct captions video clips from up to 32srcSection 3.2; Appendix A.3.2 frames, Audio Flamingo 3 supplies short audio descriptions, and DeepSeek-VL2-tiny captions extracted frames. A clip and its associated audio segment can therefore feed separate video-text and audio-text experiments, while keyframes become image-text pairs.

The trade-off appears in the supervision. All captions are synthetic and deliberately short. In a 134srcAppendix A.1.2 and A.2.3-item video-caption audit, 106srcAppendix A.1.2 and A.2.3 captions were rated accurate, 25srcAppendix A.1.2 and A.2.3 had minor errors, and 3 had major errors; the corresponding audio audit found 106 accurate, 20srcAppendix A.1.2 and A.2.3 minor, and 8 major. The checks are encouraging as spot tests, while their sample size and the absence of a frame-caption audit leave corpus-wide quality unresolved.

The reported source mix matters as much as the count: 94%srcFigure 3; Section 3.3 of videos come from YouTube and 57%srcFigure 3; Section 3.3 are in English. A multilingual tail is present, but the dataset remains predominantly a YouTube sample of the web.

ViCLIP is the paper’s strongest case. The model starts from DataComp-1B CLIP encoders, samples eight frames per video, and is tested on Kinetics-400srcSection 4.1; Appendix A.1.1, UCF-101srcSection 4.1; Appendix A.1.1, HMDB51, MSR-VTT, and MSVD. At the most legible same-scale point in Table 6—ViT-L/14srcSection 4.1; Table 6, 10 million samples seen, and WiSE-FT—BVD-V-10M reaches an aggregate score of 62.3srcSection 4.1; Table 6, compared with 60.2srcSection 4.1; Table 6 for InternVid-10M-FLT. The aggregate gives equal weight to the average of three classification scores and the average of four retrieval scores.

That is a real result, and the within-dataset scaling evidence is useful. At fixed 50srcTable 12; Appendix A.1.2 million samples seen, five seeds put BVD-V-50M at 61.52srcTable 12; Appendix A.1.2 ± 0.18srcTable 12; Appendix A.1.2 and BVD-V-10M at 60.94srcTable 12; Appendix A.1.2 ± 0.11srcTable 12; Appendix A.1.2 on the aggregate, with a 95%srcTable 12; Appendix A.1.2 confidence interval of +0.40srcTable 12; Appendix A.1.2 to +0.76srcTable 12; Appendix A.1.2 for the difference. The gain is clearest on retrieval and HMDB51; Kinetics-400 and UCF-101 sit within the reported uncertainty in this data-size comparison.

Still, the 2.1-point gap is a selected comparison. The paper describes selecting the BVD-V-50M interpolation coefficient from five values on downstream validation, while using α = 0.5srcSection 4.1; Appendix A.1.1 for InternVid, and it does not provide an equivalent InternVid tuning account. At that same point, BVD trails InternVid on Kinetics-400 (64.1srcTable 6 versus 65.0srcTable 6) and UCF-101 (80.5srcTable 6 versus 80.8srcTable 6), so the advantage is concentrated in some tasks, particularly retrieval. The overlap check is reassuring: 5.5%srcTables 13–16; Appendix A.1.2 of MSR-VTT and 6.3%srcTables 13–16; Appendix A.1.2 of MSVD unique YouTube IDs overlap with BVD-V-55M, and decontamination leaves retrieval broadly similar—but the check covers three video benchmarks and YouTube IDs.

The other modalities sharpen the verdict. In pure CLAP training at 30 million samples seen, BVD-A-10M beats LAION-Audio at all four model scales; for the 431srcTables 7–8; Section 4.2-million-parameter model, the averages are 46.8srcTables 7–8; Section 4.2 and 44.8srcTables 7–8; Section 4.2. In the mixed setup with AudioCaps and Clotho, however, BVD-A-1.7M trails the LAION-Audio version at every scale—62.3 versus 63.7srcTables 7–8; Section 4.2 for the 431M model—while adding AudioSet is strongest. These are useful source-level results, with curated audio still doing important work; the tables report point estimates rather than the seed analysis used for the video data-size comparison.

The frame experiment makes task dependence clear. At 300 million samples with ViT-B/32, BVD reaches COCO text-to-image and image-to-text Recall@5 of 0.56srcTable 10; Section 4.3 and 0.74srcTable 10; Section 4.3, versus 0.45srcTable 10; Section 4.3 and 0.63srcTable 10; Section 4.3 for DataComp. On ImageNet-1k, the same BVD model scores 0.24srcTable 10; Section 4.3 accuracy versus 0.50srcTable 10; Section 4.3 for DataComp; it also trails on ImageNet-R, ImageNet-Sketch, and ImageNet-V2. The frame distribution is measurably different: its Fréchet Inception Distance from Re-LAION is 33.92srcSection 3.3, versus 0.16srcSection 3.3 between two Re-LAION samples. That establishes distributional difference, while BVD’s strongest case remains retrieval and similarity-oriented representation learning.

Two practical qualifications matter before this becomes a default training source. First, “open” describes a layered release. The paper says it publishes the video URLs and captions for a subset on Hugging Face, while research institutions can download raw data after accepting terms of use. The release is useful for reproducibility; raw-data access is limited to research institutions that accept the terms. The collection also applied no additional safety filtering beyond platform moderation.

Second, the physical size is harder to audit than the title suggests. The paper reports 1.3 billion URLs, 130srcSection 3.1; Table 1 footnote; Table 2 million download attempts at approximately 60%srcSection 3.1; Table 1 footnote; Table 2 success, and 80 million retrieved videos, but supplies no unique-ID or content-deduplication count and no duration-truncation ledger. Table 1’s 1.8srcSection 3.1; Table 1 footnote; Table 2 billion full-corpus clip count is explicitly an estimate; Table 2 assigns 56,000srcSection 3.1; Table 1 footnote; Table 2 hours to the 55 million captioned clips, and the paper does not reconcile that with the 1.8-billion-clip estimate derived from 10 million hours. For a dataset whose central contribution is scale, this accounting is part of the result.

Finally, the evaluation scope is narrower than the foundation-model framing. Video, audio, and frame pairs are trained separately with ViCLIP, CLAP, and CLIP; there is no joint audio-visual model or generative video-language or diffusion test. The Audio Flamingo 3 manifests used for the audio captioner contain 356srcAppendix A.2.4; Table 19 of 975srcAppendix A.2.4; Table 19 AudioCaps test video IDs, although the reference captions were absent and a clean-set analysis found no disproportionate hit to BVD-trained models. That caveat limits how widely to generalize the audio result.

For a researcher, the sensible use is to treat LAION-BVD as a large, noisy source for retrieval-heavy multimodal pre-training and start with the 55M-clip evidence before extrapolating from 2.4M videos to the full reservoir. The 10M-hour figure is the reported collection scale; the validated result is smaller and task-specific. Keep those two scales separate when deciding what to train.