Actioneer v0.5 achieved 93.78% overall accuracy on DABstep, the Data Agent Benchmark for Multi-Step Reasoning, ranking first on the only real-world multi-step financial data benchmark currently in public circulation. DABstep benchmark AI agents financial data accuracy is now a third-party verified measure, not a vendor claim. This piece explains what the benchmark tests, what the leaderboard result means in practice, and what questions any enterprise buyer should ask before trusting an AI vendor's accuracy numbers.
Key Takeaways
- Ranked #1 on DABstep with 93.78% overall accuracy.
- Achieved 94.44% Hard Set Accuracy on the benchmark's most difficult tasks.
- Tested on 450 real-world financial data reasoning tasks developed by Hugging Face and Adyen.
- Demonstrates why hard-set accuracy matters more than polished product demos.
- Includes practical questions enterprise buyers should ask before evaluating AI accuracy claims.
In this article
- What DABstep Tests: Why It Is Harder Than Standard Benchmarks
- What the Results Show: Full Leaderboard, Actioneer #1
- What Hard Set Accuracy Means in Practice
- Why Grounding and the Multi-Agent Critique Layer Produce These Results
- What to Ask Any AI Vendor About Accuracy on Real-World Reasoning
- Frequently Asked Questions
A Founder or VP Revenue evaluating AI platforms for enterprise data workflows faces a structural problem. Every vendor shows a demo. Every demo works. Every accuracy figure in the sales deck comes from the vendor. Actioneer resolved this by submitting to DABstep, the Data Agent Benchmark for Multi-Step Reasoning, built by Hugging Face and Adyen on 450 real-world financial data tasks. In short, Actioneer v0.5 ranked first overall at 93.78% accuracy, including 94.44% on the hard set, ahead of Nvidia KGMON at 89.56%, Microsoft 365 Copilot at 68%, Sphinx AI at 65.56%, and Google DS-Star at 52%.
What DABstep Tests: Why It Is Harder Than Standard Benchmarks
DABstep tests AI agents on multi-step sequential reasoning across heterogeneous financial data, which is categorically different from the single-query benchmarks most vendors cite. The benchmark comprises 450 tasks drawn from a real financial analytics platform, developed by Hugging Face and Adyen. Each task requires the agent to reason across multiple tables and documents in sequence, apply company-specific metric definitions precisely, and commit to a single correct answer rather than a summary or a range.
Standard Benchmarks vs. DABstep
| Standard AI Benchmarks | DABstep |
|---|---|
| Retrieve a single fact | Multi-step sequential reasoning |
| One document or source | Multiple tables and documents |
| Simple queries | Complex analytical workflows |
| Generates plausible answers | Produces objectively verifiable answers |
| Often human-evaluated | Automatically scored for correctness |
| Optimised for demonstrations | Designed to resemble production workloads |
Standard benchmarks test whether a model can retrieve a fact or generate plausible text. DABstep tests whether an agent can navigate an unfamiliar data environment, maintain a consistent interpretation of metric definitions across multiple reasoning steps, and produce an answer that passes objective correctness checking at scale. The benchmark uses factoid-style scoring with automatic verification, which removes the subjectivity built into human-evaluated benchmarks.
The gap between easy and hard set performance is the meaningful signal. Before Actioneer's submission, the best-performing system on DABstep's hardest tasks achieved only 14.55% accuracy. o4-mini scored 76% on easy tasks and dropped to 14.5% on hard tasks requiring genuine multi-step reasoning. That collapse reveals whether a system is reasoning or pattern-matching on simpler task structures.
What the Results Show: Full Leaderboard, Actioneer #1
DABstep Leaderboard
| System | Overall | Easy Set | Hard Set |
|---|---|---|---|
| Actioneer v0.5 | 93.78% | 90.28% | 94.44% |
| Nvidia KGMON | 89.56% | — | — |
| Microsoft 365 Copilot | 68% | — | — |
| Sphinx AI | 65.56% | — | — |
| Google DS-Star | 52% | — | — |
Key Statistic
Actioneer achieved 94.44% Hard Set Accuracy, the benchmark's most commercially significant metric because hard tasks most closely resemble production enterprise workloads.
Actioneer v0.5 achieved 93.78% overall accuracy, 90.28% on the easy set, and 94.44% on the hard set, ranking first overall. The hard set result of 94.44% is the most commercially significant figure, because hard tasks represent the class of query that reaches data teams in production.
Across the leaderboard: Nvidia KGMON scored 89.56%, Microsoft 365 Copilot scored 68%, Sphinx AI scored 65.56%, and Google DS-Star scored 52%. Across two additional evaluations, the harness maintained consistent outperformance.
Additional Independent Benchmark Results
| Benchmark | Actioneer | Next Best |
|---|---|---|
| KramaBench | 78.8% | Claude Code – 66.3% |
| DA Bench | 72% | 65.5% |
On KramaBench, which tests schema exploration and data pipeline construction on 104 tasks evaluated in June 2026, Actioneer achieved 78.8% against Claude Code at 66.3%, Hugging Face DS-STAR at 55.8%, and Google DS-STAR at 45.0%. On DA Bench, developed by UC Berkeley and PromptQL, Actioneer achieved 72% against 65.5% for the next-ranked system.
Cost per Correct Answer
| System | Cost per Correct Answer |
|---|---|
| Actioneer | $0.28 |
| Claude Code | $0.47 |
Cost per correct answer is as important as accuracy. On KramaBench, Actioneer achieved $0.28 per correct answer against Claude Code at $0.47, approximately 40% lower cost at 12.5 percentage points higher accuracy.
Higher accuracy and lower cost per correct answer compound: a system that is both more accurate and cheaper per correct answer represents a material operational advantage at the query volumes enterprise data teams run.
One finding with direct implications for enterprise AI budgeting: moving from Opus 4.7 to Opus 4.8 produced a 2 percentage point accuracy improvement at approximately 50% higher inference cost. Harness design, not model version, is the more productive lever for improving performance.
What Hard Set Accuracy Means in Practice
Hard set accuracy measures whether a system handles the tasks that matter in production, specifically multi-step queries requiring precise reasoning across multiple sources with no tolerance for interpretation drift.
Three Structural Failure Modes
| Failure Mode | What Happens | Production Impact |
|---|---|---|
| Definition-shift | Wrong formula is applied | Incorrect analytical output |
| Action-bias | System proceeds instead of escalating ambiguity | Silent errors |
| Iteration-cap blowup | Tool budget exhausted before committing answer | Failed workflows |
Three failure modes account for most hard task failures in standard AI systems, each traceable to a specific architectural gap rather than a prompt artifact.
Definition-shift
The model reads a document, quotes the correct formula in its reasoning trace, then implements a different formula in code. This is a contract failure caused by long-context degradation as the executor's context fills with intermediate computation and buries the original specification.
Action-bias
The executor commits to a plausible interpretation of an ambiguous instruction rather than escalating. Tested across five separate runs with a minimal four-line prompt, a standard ReAct-loop executor invoked the escalation protocol zero times, despite genuine interpretation ambiguity in every run.
Iteration-cap blowup
In a baseline configuration without harness controls, 97 of 450 tasks, representing 21.6%, terminated without a committed answer because the executor exhausted its tool-call budget. Of those 97 tasks, 42 contained the correct value in an intermediate output. No role in the trajectory owned the decision to commit it.
These are not edge cases. They are structural failure modes that appear systematically on hard queries when the harness lacks specific controls to route around them. A system that passes a demo on three representative queries will produce all three failure modes at scale.
Why Grounding and the Multi-Agent Critique Layer Produce These Results
The Actioneer harness achieves 93.78% on DABstep because it treats each failure mode as an architectural problem with a specific structural solution.
Failure Mode → Structural Control
| Failure Mode | Structural Solution |
|---|---|
| Definition-shift | Frozen Specification |
| Action-bias | Structured Escalation Protocol |
| Iteration-cap blowup | Mandatory Escalation Rule |
A frozen specification eliminates definition-shift by treating the output of the planning step as a contract the executor cannot override mid-trajectory, even when the data or question phrasing suggests an alternative interpretation.
A structured escalation protocol eliminates action-bias by requiring the executor to externalise ambiguity through five mandatory headers — Step, Computed so far, Pseudo-code, Assumptions, Question — before the reviewer can re-ground the interpretation against source documents.
A mandatory escalation rule at iteration 15 routes around iteration-cap blowup by treating continued search past that empirically-observed threshold as a stronger signal of confusion than of progress.
The non-obvious insight from the evaluation data: the same GPT-5 checkpoint contributes to dramatically different outcomes depending on its role. In a standard ReAct loop, GPT-5 passes 20% of hard analytical tasks at $0.37 per task. Constrained to specification and review roles in a role-separated harness, that same checkpoint contributes to a system that passes 80% of the same tasks at $0.10 per task. The entire lift is attributable to role separation, not model quality. For enterprise buyers: the model matters substantially less than the harness around it.
What to Ask Any AI Vendor About Accuracy on Real-World Reasoning
The right question is not "what is your accuracy?" but "on what benchmark, tested by whom, with what scoring methodology, and against what class of task?"
Demo polish is not a proxy for production accuracy. According to Gartner's research on agentic AI adoption (June 2025), over 40% of agentic AI projects are predicted to be canceled by end of 2027. The most common gap is between demo performance and what holds in production. The NASSCOM AI Adoption Index found that only 29% of organisations can fully scale up to 30% of their AI proofs of concept. The gap between a pilot that works and a system that holds at scale is whether the harness controls for the failure modes that only appear under real production conditions.
Five questions that separate genuine accuracy from marketing:
- Is the benchmark third-party? Internal benchmarks can be constructed to favour the vendor's architecture.
- Is the leaderboard public? A result that cannot be verified by another party cannot be trusted.
- What is the hard set accuracy? Systems that perform well on easy tasks but collapse on hard tasks are not reliable in production.
- Is cost per correct answer reported? Query-level cost is not the relevant metric. Total cost to obtain a reliable answer to a specific class of query is.
- Does the system handle ambiguous instructions by escalating or by proceeding? A harness that never escalates under genuine ambiguity is generating silent errors.
Actioneer's benchmark methodology, harness architecture, and full evaluation results are available at actioneer.com.
Frequently Asked Questions
What is DABstep and how does it work?
DABstep (Data Agent Benchmark for Multi-Step Reasoning) is a public benchmark developed by Hugging Face and Adyen comprising 450 real-world financial data analysis tasks. It tests AI agents on multi-step sequential reasoning across heterogeneous data and documents, with objective scoring against known correct answers. It is the only real-world multi-step financial data benchmark currently in public circulation.
How did Actioneer perform on the DABstep leaderboard?
Actioneer v0.5 ranked first overall on DABstep with 93.78% overall accuracy, 90.28% on the easy set, and 94.44% on the hard set. Nvidia KGMON scored 89.56%, Microsoft 365 Copilot scored 68%, Sphinx AI scored 65.56%, and Google DS-Star scored 52%.
What is the difference between the DABstep easy set and hard set?
Easy set tasks test whether an agent can retrieve and process data with straightforward logic. Hard set tasks require multi-step sequential reasoning across multiple tables and documents, precise interpretation of company-specific metric definitions, and correct handling of ambiguity. Hard set accuracy is the more reliable predictor of how a system will perform in production.
Why do AI agents fail on multi-step financial data tasks?
Most agents fail due to three structural failure modes: definition-shift (applying a different formula than specified), action-bias (committing to a plausible interpretation without escalating on ambiguity), and iteration-cap blowup (exhausting the tool-call budget without committing a correct answer the system already computed in an intermediate step). These are architectural problems, not prompt engineering problems.
What does cost per correct answer mean for enterprise AI decisions?
Cost per correct answer is total inference cost divided by the number of tasks that return a passing result. A system that fails 30% of queries at low nominal cost per query is more expensive in practice than a system that fails 5% at higher nominal cost. Actioneer achieved $0.28 per correct answer on KramaBench against Claude Code at $0.47, approximately 40% lower cost at 12.5 percentage points higher accuracy.
Is it worth upgrading to a newer AI model to get better accuracy?
Harness design is a more effective lever than model version upgrades at current pricing. Actioneer's evaluation showed that moving from Opus 4.7 to Opus 4.8 produced a 2 percentage point accuracy improvement at approximately 50% higher inference cost. The gains from architectural role separation significantly exceed the gains from moving one model version forward, at a fraction of the cost.