We really need better harness benchmarks. It seems there's no reliable source that benchmarks the main harnesses against all open source models.

I also wish the discussion around Pi did not always use cost/token count as the metric. It's amazingly token efficient, but how does it stack up again opencode and others if you don't care about token count?

My experience is that the harness is mainly polish preventing failed tool calls, bad edits, stuff like that, but doesn't make much difference to the overall "intelligence". But that opencode seems slightly more robust against stupid errors than out of the box Pi due to the additional context it forces through every thread.

What matters more is that you use the tools that the target model was fine-tuned on.

E.g. for editing files with Claude models you should use Edit(file_path, old_string, new_string, replace_all) but with GPT models you should use apply_patch_call(patch) (where patch is a custom patch string with custom grammar).

It appears newer models are better at narive harness tool calls and worse at custom tools that look similar to default tools.

https://lucumr.pocoo.org/2026/7/4/better-models-worse-tools/

My own findings are in line with this research:

Having a coding harness is critical but the differences between them are overstated.

Personally, I’ve replaced OpenCode with a thin wrapper around Pydantic-AI as the pythonic analogue to Pi-Agent for headless use via Hermes

They’d all do the job - I just prefer to compartmentalize for access control.

Keeping the harness’ surface area tiny had the added benefit of preserving my understanding and being able to adapt it to my preferred workflow effortlessly

The term "harness" here is being overloaded for the term "agent", which is worrying. Putting that aside, there are many factors that matter. The "harness" context, the execution pattern (parallel vs sequential), the ability to delegate to other models, etc.

Optimal harnesses use concurrent execution + subagents and are not stuck on one model. Cost and performance are impacted GREATLY by these tactics, regardless of the native agent context (instruction). This kind of single-harness analysis is shallow and misleading, although the finding that "Provider-specific optimization does not guarantee the best pairing" is probably correct, depending on how you measure.

It is a starting point.

The most astonishing thing to me is that Pi harness is basically as efficient as the Codex/Claude.

I wonder if the same is true for the smaller models in the 9-32B range? I would expect that these models need more steering, but again I was not expecting this result either.

Claude Code/Codex charge the user for their extremely bloated one-size-fits-all system prompts (including safety instructions and other stuff users dont want).

In my experience if you're using OpenAI/Claude models and paying API costs, almost every other harness beats Claude Code/Codex in cost.

I think it’s really important, especially when you look at everything the tool does, from the execution loop and context management to feedback. The harness is basically the underlying source of truth.

With coding agents, what I’ve noticed is that a simple task can often be handled with a fairly simple harness. But the hidden cost is really around context. One of the more interesting things I’ve seen is that two different harnesses can make a similar number of model calls while consuming a very different amount of context.

I think I recently came across a paper comparing Claude Code and Pi that touched on this. More context, more tooling, focused context, simpler loops, all of these can lead to very different costs and performance, even when the number of model calls looks similar.

Say I'm using Claude Code or GPT Codex's harnesses but also sending some queries to the respective Anthropic and OpenAI models via OpenRouter.

Do harnesses and therefore sending the queries directly to the LLM providers have caching and other benefits that OpenRouter does not provide? Would I get any of those benefits if I simply proxied any requests to the major providers' harnesses through OpenRouter? Or only if the requests go straight from the harness to the provider's API?

I came to conclusion that all I really care about from the harness is interoperability. I want to be able to switch providers, models at any point in any session or between sessions in a single action, and I want my transcripts to be in one single format from the beginning till the end of time.

I want one place where I define project prompt, one place where I define project agents, MCPs, config. And all of this immediately rules out any lab-provided harness.

The problem is that Pi, on any third party harness, cannot really compete with codex or cc due to subscriptions.

I find that in many cases, you ARE the harness. Case in point, Terence Tao utilized the simple chat interface to find the Jacobian Conjecture counterexample. This is arguable no harness at all. I also find that many people will disagree with me but most of the time, they just want a button to press that will solve the problem. If that's the work you'd like to accomplish, then the harness matters, but as the article alludes to, it doesn't need to be complex.

> Models may perform better with other harnesses than with their own. So it turns out that your Claude models may not need Claude Code…

This is interesting. We built our own harness (CodeBot, an agent for Delphi) and it currently uses OpenAI models; we tuned it for specific behaviours and patterns and I find its behaviour better than Codex. Same models underneath.

Does this extend to open models like GLM 5.3? This would mean that simply changing the harness to Pi reduces cost in half?

Okay but...closed source harnesses change multiple times per week, sometimes per day.

And they also tune the inference of the models behind it.

You're using different tooling every day. Hard to benchmark.

I also test harness with "Ship Harness Bench", jcode with its browser integration gives the best results

If this is mostly because of the size of the system prompt, then perhaps in long horizon tasks the "tax" will be less obvious.

I have been using jcode for past two weeks, honestly I feel its much better compared to Opencode.

it seems like accuracy isn't affected much by the harness. can someone help me understand what's the value or the secret sauce in closed-source harnesses like claude code then?

As the model gets smarter, you need to tell it less

claude code feels mildly shitty to use in the way that every other vibe-coded-project-got-out-of-hand project does, which is like, not that bad, but it's fucking ridiculous for a 2 trillion dollar company's main companion product

It matters about as much as where you leave the electrician, he’s gonna use what tools he has to get the job done with what he has.

Much of the extra weight of Claude Code and Codex harness are (ostensibly?) for security and alignment purposes. Whether they are effective is an open question, but leaving those dimensions out and calling it a tax is disingenuous, just turning insecurity into a negative externality.

"Why pay the waste disposal tax? Dumping into the ocean is free!"

Pi actively omits any sort of guardrails and sandboxing in the name of speed and simplicity, so it is not shocking that it is faster and simpler.

Doubling the cost of something in the name of vague security is standard operating procedure for big enterprises, maybe even quite cheap.

[flagged]

[flagged]

[dead]

[dead]

[flagged]

[dead]

[dead]