By Sashank Vandrangi, Co-Founder & CPTO, Actioneer — June 2026
Frontier model inference costs have not fallen at the rate the industry anticipated. Provider endpoints have experienced meaningful reliability variation. And the operational reality for enterprise AI teams is increasingly that model selection decisions carry both accuracy and cost consequences that compound at scale. Against this backdrop, the question of where frontier intelligence is genuinely required and where it represents unnecessary spend has moved from a theoretical concern to a practical one.
Actioneer's approach to this problem is architectural. Rather than selecting a single model and applying it uniformly across every reasoning step in a data analysis workflow, we designed a harness that decomposes each task into distinct cognitive sub-tasks, assigns each sub-task to the model configuration that performs best on it, and maintains explicit controls against the failure modes that cause single-model systems to degrade on hard enterprise queries.
This post reports the results of running that harness across four benchmarks spanning multi-step analytical reasoning, schema exploration, pipeline construction, and curated enterprise problems. The findings are consistent: role-separated architecture with asymmetric model assignment outperforms monolithic frontier deployment on both accuracy and cost per correct answer.
The Problem with Uniform Model Application
Enterprise data analysis workflows are not cognitively uniform. A task that requires precise interpretation of a company-specific metric definition is a fundamentally different cognitive operation from the code generation that implements that interpretation. Treating both as equivalent, and routing both through the same frontier model in the same context window, introduces interference between the two that degrades performance on both dimensions.
Actioneer’s research documents this failure pattern rigorously on financial data analysis tasks. A single frontier model - GPT-5 running in a standard ReAct loop passes 20% of hard analytical tasks at $0.37 per task. The same GPT-5 checkpoint, constrained exclusively to the specification and review roles in a role-separated harness, contributes to a system that passes 80% of the same tasks at $0.10 per task. Same model. Different role. The lift is entirely attributable to role separation, not to model quality.
Three failure modes drive this gap, each traceable to a specific model incapability rather than a prompt artifact:
Definition-shift: the model reads a document and quotes the correct formula in its reasoning trace, then implements a different formula in code. This is a contract failure caused by long-context degradation as the executor's context fills with intermediate computation, burying the original specification.
Action-bias: the executor commits to a plausible interpretation of an ambiguous instruction rather than escalating for clarification. This is a model property under standard ReAct, not a scaffolding artifact. Tested across five separate runs with a minimal four-line prompt, the executor invoked the escalation protocol zero times despite genuine interpretation ambiguity in every run.
Iteration-cap blowup: 97 of 450 tasks in the baseline configuration (21.6%) terminated without a committed answer because the executor exhausted its tool-call budget. Of those 97, 42 had the correct value in an intermediate output but no role in the trajectory owned the decision to commit it.
The architectural response to each failure mode is specific. A frozen specification eliminates definition-shift by treating the output of the planning step as a contract the executor cannot override mid-trajectory. A structured escalation protocol with mandatory five-header formatting eliminates action-bias by externalising ambiguity before the reviewer can re-ground it against the source documents. A mandatory escalation rule at iteration 15 routes around the blowup failure by treating continued search past that empirically-observed threshold as a stronger signal of confusion than of progress.
How the Actioneer Harness Works
Actioneer's harness instantiates role separation across three distinct agent configurations. The router scores every incoming sub-task and dispatches it to the model that wins on accuracy per dollar for that sub-task type. Frontier reasoning capacity is reserved for planning and review. Execution runs on a smaller, faster model.
The Spec Agent reads source documents and emits a frozen computation plan — an ordered list of steps with expected output format. This plan is a contract. The executor cannot rewrite it mid-trajectory, even when the data or question phrasing suggests an alternative interpretation. The Review Agent runs in the same session as the Spec Agent, preserving the full document context without re-tokenisation cost. It re-reads the source documents before every reply and either confirms with a doc quote or issues a revised spec.
The executor's escalation protocol requires five verbatim headers — Step, Computed so far, Pseudo-code, Assumptions, Question — each of which earns its place. The Pseudo-code header surfaces structural bugs at a glance: misjoined tables, hallucinated columns, and incorrect filter predicates are immediately visible in pseudocode form where they are invisible in English prose. The Assumptions header forces the executor to externalise the silent decisions it made — null versus wildcard semantics, sum-vs-pick conventions, tie-break direction — before the reviewer re-grounds them against the source.
The harness also treats the worker as the centre of the pipeline rather than the frontier model. This means the executor can pull in superior intelligence from multiple sources as needed, whether that is a frontier language model for a specific reasoning step or a human reviewer for a decision that requires institutional context. The system is designed for human-in-the-loop intervention at the points where automated escalation is insufficient.
Benchmark Results
We evaluated the Actioneer harness across four benchmarks covering distinct dimensions of enterprise analytical reasoning. Across all four, the harness consistently outperforms the next highest-ranked configuration in its model weight class.
KramaBench: Actioneer internal evaluation, 104 tasks, June 2026. Accuracy = end-to-end pass rate. DABstep: Published benchmark, Hugging Face / Adyen, 450 tasks. DA Bench: Results pending external publication by UC Berkeley / PromptQL. All Actioneer results on Opus 4.7 unless noted.
KramaBench
KramaBench, developed at MIT, tests AI agents on schema exploration and data pipeline construction tasks drawn from messy real-world data lakes. It evaluates whether an agent can navigate an unfamiliar data environment, understand its structure, and build correct pipelines without prior documentation. On 104 tasks evaluated in June 2026, the Actioneer harness achieves a 78.8% end-to-end pass rate against Claude Code at 66.3%, Hugging Face DS-STAR at 55.8%, and Google DS-STAR at 45.0%.
Figure 1: KramaBench pass rate, June 2026 (104 tasks). End-to-end pass rate. Source: Actioneer internal evaluation.
DABstep
DABstep, developed by Hugging Face and Adyen, tests AI agents on 450 real-world financial data reasoning tasks requiring multi-step sequential reasoning across multiple tables and source documents. It is the most carefully-graded public benchmark in this category. The Actioneer harness achieves 95.8% overall accuracy and 95.8% on the easy split, ahead of Nvidia (89.56%), Microsoft Copilot (68%), and Google DS-Star (52%). Results are on the public dev set with consensus gold. This result is drawn from the peer-reviewed research by Actioneer
DA Bench
DA Bench, developed by UC Berkeley and PromptQL, is a curated collection of enterprise analytical problems representing the class of query that actually reaches data teams in production environments. Actioneer achieves 72% on DA Bench against an external result of 65.5% for the next-ranked system. Full external validation results will be published in the coming days.
AxnrBench
AxnrBench is Actioneer's internal benchmark constructed from anonymised production queries across our enterprise deployments. It is designed to evaluate harness reliability on the specific failure modes we observe in production — definition-shift on company-specific metric definitions, escalation behaviour under genuine ambiguity, and iteration behaviour on multi-step queries that require cross-source joins. Results on AxnrBench inform harness tuning and are not published externally.
Cost per Correct Answer: The More Useful Metric
Accuracy in isolation is an incomplete picture. A system that achieves 78.8% accuracy at $0.28 per correct answer is not directly comparable to a system that achieves 66.3% at $0.47 per correct answer without accounting for what each correct answer costs. Cost per correct answer — total inference cost divided by the number of tasks that return a passing result — is the metric that captures the trade-off accurately.
On KramaBench, the Actioneer harness running on Opus 4.7 costs $0.28 per correct answer against Claude Code on Opus 4.7 at $0.47 per correct answer. This is approximately 40% lower cost per correct answer at 12.5 percentage points higher accuracy. The two effects compound: a system that is both more accurate and cheaper per correct answer represents a significant operational advantage at the query volumes enterprise data teams run.
Figure 2: Cost per correct answer on KramaBench (June 2026) and the harness routing architecture. Source: Actioneer internal evaluation, 104 tasks, June 2026. Cost per correct answer = total cost divided by number of correct answers.
The routing architecture that produces this result is not complex in principle. The harness scores every incoming sub-task and dispatches it to the model that wins on accuracy per dollar for that sub-task type. Strategic reasoning and decomposition, which require frontier-grade document understanding and judgment, run on the frontier model. Sub-task execution, which generates the bulk of tokens but operates under a frozen specification, runs on a smaller, faster model. Verification and QA, which require pattern matching against known output formats, run on a fast, low-cost configuration. The net effect is frontier-grade output on the tasks that require frontier reasoning, without paying frontier prices for the tasks that do not.
Model Scaling Does Not Behave as Expected
One finding from our evaluation that has direct implications for enterprise AI budgeting is that newer, nominally more capable model versions do not reliably translate to better harness performance, and can actively degrade cost efficiency.
Across our benchmark evaluations, we ran the harness on Opus 4.5, Opus 4.6, Opus 4.7, and Opus 4.8. The accuracy ordering is Opus Fable > Opus 4.8 > Opus 4.7 > Opus 4.5. However, switching from Opus 4.7 to Opus 4.8 produced a 2 percentage point accuracy improvement at approximately 50% higher inference cost. At scale, that trade-off rarely justifies the upgrade. A harness running on Opus 4.7 that outperforms Claude Code on Opus 4.7 by 12.5 percentage points at 40% lower cost per correct answer represents a more material improvement than switching to the next model version.
This finding points to a broader principle for enterprise AI deployment: model version upgrades are one lever, and harness design is another. At the current state of the market, harness design appears to be the more productive lever. The accuracy gains from architectural role separation significantly exceed the gains from moving one model version forward, at a fraction of the cost.
Inference reliability is a second dimension. We observe meaningful variance in performance across time periods for the same model version at the same provider. Building harnesses that are tolerant of this variance, through escalation protocols, structured validation, and human-in-the-loop intervention points, is more robust than assuming stable endpoint behaviour.
What This Means for Enterprise Data Teams
The practical implications of these results fall into three areas.
Model allocation is an engineering decision, not a default. The instinct to apply the best available model to every step in a workflow produces neither the best accuracy nor the best cost structure. Mapping sub-task types to model configurations, routing frontier intelligence to the steps where it is most load-bearing, and running bulk token generation on smaller models is the design pattern that produces both.
Cost per correct answer, not cost per query. Query-level cost is the wrong unit for evaluating enterprise AI economics. A system that fails 30% of queries and charges less per query than a system that fails 5% of queries is not cheaper in practice. The relevant metric is the total cost to obtain a reliable answer to a given class of question.
Harness controls are not optional for production reliability. The iteration-cap blowup failure mode, in which an executor exhausts its budget without committing an answer that it had already computed correctly, is not an edge case. It accounts for 21.6% of tasks in an uncontrolled baseline configuration. A harness without a mandatory escalation rule at the empirically-observed failure threshold, and without structured escalation formatting that forces assumption externalisation, will produce this failure systematically on hard multi-step queries.
What We Are Building Next
The residual failure families in our current harness are predominantly cases where the spec reads internally consistent but applies the wrong interpretation of an ambiguous metric definition. The executor never escalates because there is no visible inconsistency, the wrong formula is applied correctly throughout the trajectory.
The next architectural extension addresses this pre-hoc rather than post-hoc. A Probe Agent, operating before the spec is finalised, generates two to three candidate interpretations of any docs-defined metric, probes the data against each, and commits to the interpretation whose output shape matches the expected format for the query type. This converts the harness from one that resolves ambiguity after the executor encounters it to one that resolves ambiguity before the spec freezes. We project 55 to 95 additional recoverable tasks from this extension on the DABstep public dev set, based on rule-level analysis of the residual failure families.
If you are evaluating AI infrastructure for enterprise data workflows and want to see the harness deployed on your own workloads, the conversation starts at actioneer.com.