What the first careful head-to-head test of AI agent teams tells us about cost.
Bottom line
A team of AI agents is not automatically better than one good agent. In a new independent test, teams of AI agents increase costs anywhere from 1.8 to 5.1 times as much as a single agent, and only one of four setups got a score boost you could trust [1]. Before you pay for a team of agents, or design your own, check whether the work really needs to be split into separate pieces.
What we’re seeing
We’re starting to see some movement away from the concept of “an agent” and replacing that with “teams of agents” as the new feature. Both OpenAI and Anthropic now ship built-in multi-agent support: a lead agent hands parts of a job to helper agents (“subagents”), then pulls their work back together [1]. This functionality already exists when designing agents in Microsoft Copilot Studio, and the ability to have supervisor and subsidiary agents, can be extremely useful or beneficial. In Part 1 of my post, “Bots, Copilots and Agents” (https://lumenforge.ai/bots-copilots-agents-and-autopilots-part-1/), I speak to this sitting at the top rung of the AI Autonomy Ladder as “autonomous and multi-agent systems,” and I said then it is rarely what a 50-person business needs first [8]. Now there’s data to support that claim.
Vals AI, an independent benchmarking firm, ran two leading models, OpenAI’s GPT 6 Sol and Anthropic’s Claude Opus 5.5, on its Vibe Code Bench [1]. The job: build 50 complete web apps from a written spec. Each model worked alone and as a team, at medium and at maximum “reasoning effort” (how hard the model thinks before it acts). Robot testers then clicked through every app and scored how much of it worked.
The idea is not new. Microsoft researchers released AutoGen in 2023, a framework where several agents talk to each other to get a task done [3]. Anthropic runs a lead agent with parallel helpers in its own research tool, and reports that setup uses about 15 times the tokens of a normal chat [4]. OpenAI builds this into its Agents SDK as “handoffs,” where one agent passes a task to a specialist agent [5].
What the numbers say
Here is the short version, all from the Vals study [1]:
| Setup | Score | Cost per app |
| Sol, medium effort, alone | 77.6% | $1.22 |
| Sol, medium effort, team | 84.9% | $3.07 |
| Sol, max effort, alone | 89.0% | $4.82 |
| Opus, medium effort, alone | 91.5% | $4.08 |
| Opus, medium effort, team | 91.2% | $9.10 |
| Opus, max effort, team | 93.2% | $122 |
Three things stand out:
- Only one team clearly helped. When compared to Sol, medium effort, working as a single agent, the Sol medium-effort team gained 7.3 points, and that gain was statistically significant. The other three team comparisons moved between minus 0.3 and plus 3.4 points, which is within the noise [1].
- Thinking harder can beat hiring help. For Sol, max effort alone added 11.4 points, more than the team configuration at medium effort did [1]. But that comes at nearly four times the cost, compared with a cost increase of “only” 2.5 times for the team, and it took three times as long as the medium-effort team.
- The cheapest strong answer came from a single agent. Opus working alone at medium effort scored 91.5% for $4.08. That is 13.9 points higher than Sol working alone at medium effort, though at about 234% more in cost. Its most expensive setup, a max-effort team, scored 93.2% for $122, about thirty times the cost, and that lead was not statistically significant [1]. Thirty times the bill. Same result, give or take.
Why agent teams cost so much
Every helper agent carries its own copy of the job: the spec, its notes, its progress. That copy gets re-sent every time the agent calls the model, and you pay for it each time. The Vals study found most of the extra spend came from exactly that [1]. More agents, more reading, more meter running. More time. More money.
Teams also aren’t always faster. Sol’s teams finished sooner because its lead split the work up front and ran the pieces in parallel. Opus’s lead wrote a shared plan first, then worked in waves of builders and testers, and its max-effort team took up to 2.3 times as long as working alone [1]. Same instruction, two approaches, and two very different managers.
Where the team DID earn its keep
This is the useful part. Sol’s team helped most on the harder apps, adding about 15 points there and about 1 point on the easy ones [1]. On the apps where it gained the most, the single agent had reported “tests passed” even when some features were never built; so it misrepresented the facts. The team lead checked the separately built pieces against each other and caught mismatches, like a form that didn’t fit the database [1].
So the win wasn’t more hands. It was a second set of eyes on the seams. Again, reinforcing the case for human in the loop at some stage.
Others have seen the same pattern. A UC Berkeley study of more than 1,600 agent team runs found that many failures came from agents that misread each other and from weak checking at the end [6]. Cognition, the company behind the Devin coding agent, argues teams break down when helpers don’t share full context and make conflicting choices [7].
That lines up with Anthropic’s own advice to builders: start with the simplest thing that works and add complexity only when it pays [2]. And with what I wrote in Part 2, agents multiply whatever you point them at (https://lumenforge.ai/bots-copilots-agents-and-autopilots-part-2/), and errors can compound. A team multiplies that risk even more.
Read it with care
Vals is upfront about limits, and so should we be [1]:
- This is only one benchmark, building web apps. Results may not carry over to bookkeeping, scheduling or customer email.
- Each setup ran once. There was no iteration or repetition of the testing and scoring.
- The two models ran through different tools, so model-versus-model comparisons are mixed up with the tooling.
Vals’ own guess that teams should shine work with many independent parts, like large codebases, broad research or analysis across many documents [1]. That’s their hypothesis, not a finding yet. Keep in mind that this was focused solely on development of web applications, and there are specific business cases where agents are or may be the right solution, specifically in dividing the workflows across different departments, divisions, etc.
What this means for your business
Most small businesses won’t run a coding benchmark. You will get offered “agent teams” inside tools you already pay for. The lesson travels. Paying for more agents is paying for more coordination, and coordination only pays when the work genuinely splits apart and someone checks how the pieces fit. And, as we’ve seen, there is a cost, in both time and token spend, that multiple agents bring into the discussion.
Is your problem too much work, or work that’s hard to check? Those need different fixes.
Here’s the test. Before you pay for a team of agents, ask:
Can this job be cut into pieces that don’t depend on each other, and will someone check how they fit back together?
If the answer is no, one well-configured agent (or a simple workflow) is probably the better buy. If it’s yes, run a small trial and compare cost and quality against a single agent on the same work. Measure it. Don’t assume it.
Disclosure: I run Claude and Grok side by side in my own AI pilot, and one of the models tested here is Anthropic’s. Naming vendors is reporting, not endorsement.
Agent teams are real and they’re becoming easier to deploy and switch on. More does not mean better, better is better.
If you want a second set of eyes on where agents fit in your business, book a free 30-minute AI Reality Check: https://tinyurl.com/y6yu6kvc
New to the vocabulary? Start with [Bots, Copilots, Agents and Autopilots: Part 1](https://lumenforge.ai/bots-copilots-agents-and-autopilots-part-1/) and [Part 2](https://lumenforge.ai/bots-copilots-agents-and-autopilots-part-2/).
References
1. Katrina Drozdov, Vals AI, “Do Agent Teams Pay Off? A Case Study on Vibe Code Bench,” October 9, 2026. https://www.vals.ai/blogs/multi-agent-vibe-code-bench
2. Anthropic, “Building effective agents.” https://www.anthropic.com/engineering/building-effective-agents
3. Qingyun Wu et al., Microsoft Research, “AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation,” arXiv:2308.08155, 2023. https://arxiv.org/abs/2308.08155
4. Anthropic, “How we built our multi-agent research system,” June 13, 2025. https://www.anthropic.com/engineering/multi-agent-research-system
5. OpenAI, “Handoffs,” OpenAI Agents SDK documentation. https://openai.github.io/openai-agents-python/handoffs/
6. Mert Cemri et al., UC Berkeley, “Why Do Multi-Agent LLM Systems Fail?” arXiv:2503.13657. https://arxiv.org/abs/2503.13657
7. Walden Yan, Cognition, “Don’t Build Multi-Agents,” June 12, 2025. https://cognition.ai/blog/dont-build-multi-agents
8. Dan Bond, LumenForge Advisors, “Bots, Copilots, Agents and Autopilots: Part 1,” October 8, 2026. https://lumenforge.ai/bots-copilots-agents-and-autopilots-part-1/