Capability is only part
of dependable execution.

Freshness: introduced September 30; initial public harness commit September 29. Individual execution dates are undisclosed. Surge AI's current leading row is Gemini 4 Argon High: 38.3%.

Mean rubric credit is not complete-task success. Experts selected tasks after earlier AI failures, so this is a deliberately difficult set. The full set is private; final answer text is graded and errored trials are excluded.

Freshness: Vals AI's result page and embedded chart metadata say updated September 29. Individual run days and the v2 introduction day are unestablished; the original family paper is from 2025.

All-Pass requires every check in one answer. It does not require the same question to succeed in all three attempts. Private questions and model grading limit replication. The current chart reports 55.789%; older narrative scores on the page differ.

Freshness: September introduction; repository first committed September 15. Vals' current page is updated September 29; model execution days are undisclosed.

All-Pass requires every rubric check, separate from the citation-quality multiplier in the primary score. No repeated-trial consistency requirement is disclosed. Most questions concern federal corporate tax; the set is private and uses one model judge.

Freshness: introduced June 25; paper July 28, revised August 3. September 16's completed-results article dates the current leading Fable 5.1 Adaptive/Max: 38.8% strict pass@1 result. It also contains an earlier September 10 page date; use the completed September 16 publication.

Every required and prohibited-action criterion must pass in an attempt. The score averages attempts, rather than requiring all four to succeed. This is a simulated workplace; the full set is available by request.

Freshness: paper June 29. Latest owner-verified leading product submission: arito, September 13. Its debugging score is 29/100 tasks, the highest current Debug row. The former 12% headline is superseded.

Debugging success requires the entire workbook, including unchanged cells, to match the expected result. The overall score mixes different measures. Public tasks, proprietary products and undisclosed budgets limit controlled comparisons.

Freshness: Mercor/Ramp paper July 29, revised July 30; announcement July 31. These are the original dated results. Newer current model rows have no verified individual result dates.

These are July results from the original nine-model roster, not a current model ranking. Mean criteria credit, any-attempt success and repeated success are different measures. Private synthetic company records and excluded provider failures limit generalization.

Freshness: benchmark revision/results September 8, despite the original January paper. The dated article gives Fable 5.1 Max: 68.6% pass@1 and GPT 6 Astra: 56.3% pass^4. Current Sonnet 5.5's 75.5% row is undated; it is observed, not assigned a September 8 run date.

The September revision changes tasks and grading, so it is not a simple before-and-after comparison. The release reports 8% false negatives in a 354-item grading check. A newer 75.5% leaderboard row has no verified individual result date here.

Freshness: original March 9 paper excluded. Explicit new Pro result updates July 20 and August 2 qualify. Pro V2 results entered the repository August 6; exact execution day is absent.

This measures numerical answers across documents, rather than actions in operational systems. Different native agent setups are part of the comparison. Repeat counts and numerical tolerance are incompletely specified alongside the figures.

Freshness: LlamaIndex paper July 31. Specialist refresh September 18; Codex GPT 6 Sol Evidence addition September 23.

F1 measures value matching, not the share of wholly correct documents. Grounding uses a separate population of verified locations, so subtracting those scores would mislead. Public data, varied setups and only 20 long documents limit the comparison.

Freshness: paper April 9, revised April 13; new reasoning-effort sweep September 24.

The composite averages five parsing capabilities, rather than complete-task success. Page populations overlap. This is direct document parsing on a vendor-authored public set, rather than a full agent workflow.

Freshness: initial multi-system study September 9; hidden-context extension September 24, revised September 25.

Eight templates and 24 hidden-fact runs provide a small, read-only synthetic test. Confidence intervals are wide and tool budgets differ. The earlier multi-system study uses another population; its scores should not be pooled with this extension.

Freshness: Rogo/OpenAI collaborators' study June 2; no later dated result verified.

Experts selected difficult questions after earlier model failures. Weighted rubric credit differs from final-answer accuracy. These are the dated June configurations, not a present-day capability ceiling. Private data and model judging limit replication.

Freshness: original benchmark is from 2024. New owner-verified runs qualify: QingqiuAgent June 23 and DataAnalysisAgent September 10.

These strong results qualify through new dated submissions to an older benchmark. The 400-task subset, full 912-task set and newer debugging benchmark are different populations. Proprietary setups and public tasks leave tuning and training exposure uncertain.

Freshness: introduced April 14; paper July 13. September 16's completed-result article dates Fable 5.1 and Muse Spark 1.3 runs. A newer current GPT 6 Astra 34.2% row is undated and is not given a September run date here.

The dated result requires every rubric criterion to pass. Experts selected difficult documents after earlier failures. This uses no tools and a model judge. A newer observed 34.2% row is undated and is not assigned a recent run date.

Freshness: harness introduced September 23; owner article displays September 23 and 26. Use September 26 for the dated article, not an inferred model run day.

The owner describes complete-task success, while the public verifier and README describe mean criterion credit. The private leaderboard conversion is unestablished. The owner-reported score is therefore not presented as a verified completion percentage.

Freshness: original March 4 introduction excluded. Qwen and Opus evaluated August 3 and submitted August 4; GPT 5.6 Sol evaluated July 22 and submitted August 4.

Per-attempt and all-four success measure different demands. Banking tests policy retrieval and support tools; it does not include the telecom user's dual-control tools. Public tasks, simulated users and July grading changes limit comparisons with older scores.

Freshness: standard GPT Live evaluated/submitted September 9; custom banking voice September 16.

The strong telecom result includes user-side state changes. It is one trial per task, so it does not establish repeated reliability. The newer custom banking run changes the caller, reviewer, effort and voice setup; its score is not directly comparable.

Freshness: Sierra/Princeton study introduced September 4; exact construction dates undisclosed.

Reward includes budget penalties and is not the fraction of agent builds completed. The expert reference has ground truth, rather than representing average human performance. Synthetic company artifacts and fixed simulated users do not test ongoing maintenance.

Freshness: Microsoft introduction August 20; latest v3 revision September 30, following v2 August 29. Exact execution days undisclosed; the revision date does not establish a new run.

Table 15 reports 241 tasks passing all 20 trials. The separate pass^20 column is an estimator, not generically the observed all-20 fraction. Of 507 simulated tasks, 477 grade only backend state and 30 also communication, so misleading messages can escape a state-only check.

Freshness: Accenture paper September 15, latest revision September 23; a new revision-specific execution date is unestablished.

Sonnet 4.6 achieves strong results. A separate smaller-model setup saves many records while getting few fields correct; that gap is configuration-specific. Four multi-field tasks and six document-chain tasks provide small populations on one ERP.

Freshness: campaign executed September 28 05:29 through September 29 10:48 UTC, with retained-output regrading in the September 28 receipt.

The benchmark owner also builds Proto, and tasks were available during development. Retained outputs were regraded without rerunning. The same-model difference is not decisive: its interval crosses zero. Strong scoped scores are not independent production evidence.

Freshness: introduced May 24; v3 September 22 updates results and simulator configuration. Execution days undisclosed.

Producing a plan and satisfying its time, budget and transport constraints are different measures. The 99.7% emission and 8% validity figures refer to the same configuration. Synthetic cooperative preferences and approximated API behavior limit transfer.

Freshness: Microsoft study September 10; exact run days undisclosed.

The larger gain compares no memory with memory plus probing, so it does not isolate probing's contribution. The two memory intervals overlap and separate curation cost is omitted. Controlled schema drift does not establish learning causally from delayed customer results.

Freshness: ServiceNow/academic collaborators' study September 17; execution days undisclosed.

The score measures predicted field effects, rather than workflow completion. All tested backbones revised or added rules, but none retired a deleted rule. Controlled changes on one platform and excluded unrepresentable rules limit generalization.

Freshness: introduction August 2, v2 August 11; execution days undisclosed.

The strongest tested configuration completes all tasks with 1% unauthorized actions. Other configurations vary substantially. Confirmation calls are automatically approved, so the study measures authority preservation, not physical containment. Tool exposure and handoffs also change between architectures.

Freshness: paper September 16; later project models have no verified added-run dates here.

This is strong explicit-policy behavior on synthetic workplace decisions. Unclear replies are excluded and abstention is separate. Replies and decisions are graded; no real payment execution or deployment certification is established.

Freshness: AmpUp's new CRMArena-Pro evaluation September 23; execution days undisclosed.

The alarming result concerns a selected subset of 31 contradictory assertions, not all 100 cases or production CRM use. Confidence intervals are wide. There is no manipulation/control experiment isolating a causal persuasion effect, and no live CRM writes.

Freshness: Microsoft study September 11; execution days undisclosed.

Memory-operation accuracy and visibility are different measures. The privacy model uses binary private/public scope, not enterprise roles. The same configuration must stay attached to each cited score; synthetic scenarios limit transfer.

Freshness: ServiceNow/academic study July 6; execution days undisclosed.

Visibility violations count attributes exposed in any of three runs, not the frequency of unsafe executed actions. Communications are simulated and the centralized agent sees all information by design. Oracle-labelled defenses do not establish deployable policy learning.

Freshness: new harness-validation study September 26; execution days undisclosed.

The same persisted attack traces receive different scores when grading checks tool names versus consequential arguments. This exposes an evaluation problem, not a change in agent behavior. A small mock suite and zero observed attacks cannot establish zero risk or a significant defense advantage.

Freshness: new study September 29; execution days undisclosed. This is separate from Surge's EnterpriseBench: CoreCraft.

Consultation, inventory and workforce decisions are simulated. Different methods lead different tracks and no pooled score is reported. The 423 consultation cases do not establish the denominator for other tracks. This does not demonstrate causal improvement from production feedback.