Evaluating Agentic AI Architectures for Enterprise Applications: A Multi-Site Controlled Comparison of Orchestration Patterns

1.Introduction

Enterprise adoption of agentic systems has outpaced a rigorous understanding of which architectural choices trade reliability against cost. Organisations now routinely delegate multi-step work — reconciling invoices, triaging service tickets, assembling procurement packets — to language-model agents that plan, retrieve, and invoke tools with limited human supervision. The decision that governs the outcome of these deployments is rarely which model is selected, because model choice is increasingly commoditised and revised quarterly; it is how the agent's action space is structured. That decision is typically made once, early, by a platform team, and is expensive to reverse.

The published evidence available to support that decision is thin in three specific ways. First, most comparisons evaluate a single architecture against a static benchmark rather than several architectures against the same workload, which prevents any statement about relative performance. Second, benchmarks tend to reward short reasoning traces, whereas the enterprise workflows that motivate agent adoption commonly require six to twenty tool interactions with persistent intermediate state. Third, results are usually reported per model call, a unit that flatters architectures which fail quickly and cheaply while penalising those that complete work.

This paper addresses that gap with a paired, multi-site, controlled evaluation. We hold the model, retrieval corpus, tool inventory and scoring procedure fixed, and vary only the orchestration pattern. We frame orchestration as a measurable design decision with quantifiable consequences rather than an implementation detail, and we report results in the unit that matters to the organisations funding these systems: cost and latency per successfully completed task.

Our contributions are: (i) a paired evaluation protocol for orchestration architectures that controls the confounds we found most often conflated in prior work; (ii) effect estimates with confidence intervals across twelve real enterprise workflows at four organisations; (iii) a failure taxonomy showing that architectures differ principally in the recoverability of their errors rather than in error frequency; and (iv) a replication across three model checkpoints establishing that the ordering, though not the magnitude, is stable under model change.

2.Related Work

Work on agent evaluation divides broadly into capability benchmarking and reliability engineering. Capability benchmarks measure whether a model can in principle solve a task, and have driven rapid progress on reasoning and tool-use accuracy. Reliability studies instead ask whether a deployed system completes work repeatably under distribution shift, partial tool failure and ambiguous instructions. Our study belongs to the second tradition and adopts its central measurement commitment: repeated execution of the same task under controlled variation.

Prior comparisons of orchestration patterns have generally been architectural case reports, describing a single production system and the choices its authors made. These are informative about feasibility but cannot isolate the contribution of the orchestration layer, because model, prompt, corpus and workload all vary simultaneously with architecture. Okonkwo et al. established analytical reliability bounds for chained agents by treating each step as an independent Bernoulli trial, predicting the steep degradation with trace length that our empirical results confirm; our data additionally show that the degradation is not uniform across architectures, which their independence assumption cannot express.

Work on schema-constrained generation and tool validation has demonstrated substantial reductions in malformed action rates at the level of individual calls. What has been missing is whether these local gains survive composition across a long workflow, or are absorbed by retries and compensating logic elsewhere in the system. Our per-task measurements are designed to answer precisely that question.

3.Methods

Design. We used a fully paired within-workflow design. Every task instance was executed once under each of the three architectures, so each instance serves as its own control for prompt phrasing, retrieval content and data difficulty. Execution order was randomised to remove ordering effects arising from platform caching. The protocol, hypotheses and primary outcome were registered before data collection began.

Sites and workflows. Four organisations participated: a regional insurer, a public university IT service desk, a manufacturing procurement group and a hospital scheduling unit. Together they contributed twelve workflows requiring between four and twenty-two tool interactions, with a median of nine. Workflows were instrumented in production-adjacent environments containing de-identified copies of live data, so that task difficulty reflected real record quality rather than curated inputs.

Architectures. Unconstrained chaining passes free-form model output forward as the next step's input with no schema validation. Supervisor–worker delegation introduces a planning model that assigns sub-tasks to specialised workers and integrates their returns. Constrained tool-calling restricts every action to a typed, schema-validated tool signature, rejects malformed actions before execution, and requires explicit state transitions between steps. All three shared the same tool inventory and the same retrieval index.

Measurement. The primary outcome was binary task completion, scored against a workflow-specific rubric by two independent domain raters blind to architecture, with a third adjudicating disagreements; inter-rater agreement was κ = 0.83. Secondary outcomes were wall-clock latency, tokens per task, cost per completed task and a failure category assigned from a six-class taxonomy. We fitted mixed-effects logistic regression with random intercepts for workflow and organisation, reporting 95% confidence intervals from 10,000 cluster bootstrap resamples. The complete protocol was repeated on three model checkpoints released over five months.

Prompt-parity control. To separate architectural effect from prompt quality, we ran a secondary condition in which the strongest architecture's prompts were back-ported to the weaker architectures by an engineer given unlimited revision attempts against a held-out development split. This provides an upper bound on how much of the observed gap prompt engineering alone can recover.

4.Results

Completion. Constrained tool-calling completed 0.81 of task instances (95% CI 0.79–0.83), supervisor–worker delegation 0.74 (0.72–0.76) and unconstrained chaining 0.62 (0.60–0.64). The 19-point gap between the extremes corresponds to a 31% relative improvement and was significant at every site individually, with the smallest site-level gap at 14 points.

Prompt parity. Under the prompt-parity control, chaining rose from 0.62 to 0.65. Prompt engineering therefore recovered 3.1 of 19 percentage points, or roughly one-sixth of the gap, despite unlimited revision effort. We read this as strong evidence that the effect is located in the action boundary rather than in instruction quality.

Cost and latency. Constrained runs issued 4% more model calls and cost 4% more per call, owing to validation retries and explicit state transitions. Per completed task, however, they cost 17% less, because failed executions consume tokens without producing value. Median latency was 8% higher for constrained runs; the 95th percentile was 26% lower, since the long tail under chaining was dominated by runs that wandered before being abandoned.

Failure composition. Error counts were closer than error consequences. Chaining and constrained runs produced comparable numbers of first-step errors, but 71% of chaining failures involved irrecoverable state corruption at step four or later, against 22% under constraint. Constrained failures were concentrated in genuine task ambiguity and missing source data — categories a better architecture cannot fix.

Stability under model change. Across three checkpoints, absolute completion moved by up to 11 points for the same architecture and workflow, but the ordering of the three architectures never changed. This matters for procurement: conclusions drawn about architecture generalise across model upgrades, while conclusions drawn about absolute performance do not.

5.Discussion

The results support a simple reading. Long agentic workflows fail by accumulation, and the dominant determinant of accumulated failure is whether an invalid intermediate state can enter the trace at all. Constraint at the action boundary removes a class of error before it compounds; capability improvements in the underlying model reduce the probability of each individual error but leave the compounding structure intact. This explains why the architectural ordering survived model upgrades that shifted absolute performance substantially.

For practitioners, the operative recommendation is to change the unit of evaluation. Vendor comparisons and internal dashboards overwhelmingly report per-call cost and per-call success, and both metrics systematically favour architectures that fail early. Reporting cost and latency per completed task reversed the apparent ranking in eight of our twelve workflows.

For platform design, the finding that constrained failures concentrate in ambiguity and missing data suggests where remaining effort belongs: upstream data quality and explicit clarification protocols, not further orchestration sophistication. Two of our sites had planned significant investment in more elaborate planning layers; our failure taxonomy indicates that investment would have addressed under a quarter of their observed failures.

6.Limitations and Threats to Validity

Our four sites are English-language organisations in regulated or semi-regulated sectors, and all twelve workflows involve structured records. Agentic work over open-ended creative or exploratory tasks may not exhibit the same compounding structure, and we make no claim about it.

Completion rubrics were authored with site staff and therefore encode local definitions of acceptable work. We report rater agreement rather than construct validity, and we release rubrics with the dataset so that alternative scoring can be applied.

Cost figures depend on commercial pricing at the time of measurement and on our specific tool latencies; the direction of the per-completed-task result is robust to plausible repricing, but the magnitude is not portable. Finally, three checkpoints from a limited set of providers cannot establish stability across all future model families.

7.Conclusion

Across 4,800 paired task instances at four organisations, constraining the action boundary improved enterprise task completion by 31% relative to unconstrained chaining, reduced cost per completed task by 17%, and shifted residual failures from irrecoverable state corruption toward genuine data and specification gaps. The effect is architectural: unlimited prompt revision recovered only a sixth of it. We recommend that organisations evaluating agent platforms report outcomes per completed task, treat the action boundary as the primary design decision, and re-test magnitude — but not ordering — after each model upgrade.

Data availability. Task rubrics, failure taxonomy definitions, de-identified execution traces and analysis scripts are available to reviewers and, on publication, to readers under the same licence as this article.

8.References

1. [1] Okonkwo, R., Marsh, S. & Paxton, D. (2025). Reliability bounds for compositional LLM agents. Journal of Emerging Computing 18(1), 44–61.
2. [2] Marsh, S. & Hale, E. (2024). Tool-calling constraints and malformed action rates in production agents. Proceedings of APSEC, 202–215.
3. [3] Rahman, A. (2026). Evaluation cost models for multi-step inference workloads. JIPCET 1(1), 12–29.
4. [4] Vasquez, E. & Paxton, D. (2025). Paired designs for agent benchmarking. Empirical Methods in Applied AI 4(2), 133–158.
5. [5] Tan, M. & Haddad, J. (2025). Per-task accounting for serverless model invocation. Proceedings of SoCC, 288–301.
6. [6] Whitmore, O. (2025). Reliance, oversight and calibration in automated workflows. Human Factors 67(2), 210–229.
7. [7] Russo, G. & Whitfield, B. (2024). Quality proxies and the unit-of-measurement problem. Empirical Software Engineering 29(1), 77–98.
8. [8] Ferreira, L. (2026). Where recovered time goes: validation load after automation. JIPCET 2(3), 40–63.