Claude, Codex, and Cursor pick different tools. A 17k-run study reveals why.

"The agents did not just write different code. They reached for different tools to get there, and the gap was not small." That is the framing behind Armature's study of 17,000 coding agent runs across Claude Code, Codex, and Cursor, and it is a more interesting finding than the headline suggests. Everyone assumes these three agents are converging on the same underlying skill set. The study says otherwise: tool selection, not code quality, is where the real divergence lives.

The training data problem hiding inside tool choice

Ask why one agent reaches for grep and another reaches for ripgrep, or why one shells out to a Python one-liner where another calls a dedicated CLI, and the obvious answer is "different models, different training." That is true but incomplete. The more useful explanation is that each of these agents was shaped by a different set of example transcripts during fine-tuning, and those transcripts encode habits, not just capabilities.

Codex, coming out of OpenAI's coding lineage, was trained on a corpus that leans toward direct shell usage and standard Unix tools. Claude Code has iterated through several rounds of tool-use reinforcement specifically aimed at safe, auditable file operations, which shows up as a bias toward more conservative, more verbose tool calls. Cursor sits closer to an IDE-integrated agent, so its defaults reflect what a VS Code extension can call cheaply through its own API surface rather than what a bare terminal session would naturally reach for. None of this is a coincidence. It is the fingerprint of whoever built the harness.

That distinction matters because many evaluations of coding agents test them on output quality alone. Two agents can produce nearly identical final code and still take wildly different paths there, one burning three times the tool calls, one silently failing and retrying, one skipping validation steps the other treats as mandatory. If you are only grading the diff, you miss the part of the run that determines cost, latency, and how often the agent quietly does something you did not ask for.

The consistent pattern across 17,000 runs was not which single tool won. It was how differently the same task got approached depending on which agent picked it up.

This measurement tells you less than it looks like

Here is the case against reading too much into this. A count of which tools an agent invokes is a proxy metric, and proxy metrics get treated as ground truth far too often in this industry. Seventeen thousand runs sounds like a large number until you ask what tasks generated them. If the task set leans toward file search and refactoring, you will see search-tool preferences dominate the data. If it leans toward dependency management, you will see package-manager preferences dominate instead. Tool selection frequency is a function of task composition as much as it is a function of agent design, and a study that does not control tightly for task distribution is measuring its own sample as much as it is measuring the agents.

There is also a version-drift problem baked into any agent comparison. Claude Code, Codex, and Cursor all ship updates on a rolling basis, and a tool-selection habit measured this month can shift after the next model swap or system prompt revision, without anyone announcing it. A benchmark like this is a snapshot, not a law. Treating it as a durable ranking of "which agent has better judgment" overstates what a frequency count of shell commands can actually tell you.

And the practical stakes are smaller than they sound. Most coding tasks that developers actually delegate to an agent do not hinge on whether it picked sed over a Python script. They hinge on whether the change compiles, passes tests, and does not touch files it should not touch. Tool choice is a readable signal of agent personality. It is a weak signal of agent competence.

A developer who trusts defaults, and one who does not

Skeptic: If tool choice does not predict output quality, why would I change how I pick between Claude Code, Codex, and Cursor based on this study?

Builder: You would not change which one you use. You would change what you watch for. If you know Codex defaults to raw shell commands, you know to sandbox it harder. If you know Cursor leans on its own integrated tools, you know its behavior outside the IDE is less tested.

Skeptic: That is just "read the documentation," dressed up.

Builder: Documentation tells you what the agent is supposed to do. This tells you what it actually reaches for under real task pressure, across thousands of runs, not one vendor's cherry-picked demo. That gap is exactly where production incidents come from.

Skeptic: Fair. But I would still want to see it fail on my tasks before I trust the pattern.

Builder: That is the right instinct, and it is also the whole point of running your own sample instead of someone else's.

Run a 20-task audit before you standardize on one agent

Pick one agent from your current stack, whether that is Claude Code, Cursor, or GitHub Copilot, and log the last 20 real tasks it completed for your team. Not synthetic benchmarks, actual pull requests or terminal sessions. For each one, note which external tools or commands it invoked, whether it asked permission first, and whether it retried silently after a failed call. You will likely find your own version of Armature's pattern within a single afternoon of review, and it will be specific to your codebase in a way no published study can be. If the pattern worries you, that is the moment to add a tool allowlist or a stricter approval step, not after the agent has already run in production for another quarter.

For more on where these agents diverge and where the comparisons actually hold up, see our breakdown of Cursor versus GitHub Copilot and the earlier piece on Claude's quirks under load. The full methodology behind the 17,000-run study is worth reading directly at Armature's original writeup, and Anthropic's own Claude Code documentation is a useful cross-check against what the study observed in practice.