419 points by Edward40 3 days ago | 133 comments | View on ycombinator
seizethecheese 3 days ago |
Zigurd 3 days ago |
xyzzy_plugh 3 days ago |
For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.
Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.
This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.
But I can't build a mobile app with this harness, or make a movie.
These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.
Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.
Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.
cpeth 3 days ago |
As someone who regularly uses Claude Opus 5.5 from Claude Code and from Copilot, I would say harness barely matters in terms overall performance and behavior. Opus is nearly identical in both harness. The opposite is also true, if I switch Copilot to something like Gemini 4 Argon it's completely different even though I'm in the same harness.
Just some andecdata.
bisonbear 3 days ago |
However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.
I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)
ford 3 days ago |
scubazealous 3 days ago |
ricardobeat 2 days ago |
fennecfoxy 3 days ago |
krawczstef 2 days ago |
Harnesses simply make different choices around certain context engineering things, what they name tools, how they compact, their system prompt, etc.
So it's not surprising that they work "better" for some models than others, but I'd bet that if you made each harness have same system prompt and tool names, the variance would shrink further...
So I would say know what you're optimizing for and construct the harness you need for it instead (it's just a while loop with an llm call and function calling after all) ...
lhk931122 2 days ago |
scosman 3 days ago |
I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.
0xbadcafebee 3 days ago |
Second thing to consider: ZCode was sending everyone's code to Alibaba... not sure I would use their harness just to save a few cents on coding.
evalstate 3 days ago |
GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.
hi_hi 3 days ago |
cronin101 3 days ago |
Can anyone confirm/deny this is the case?
If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
Majromax 3 days ago |
The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.
I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
zodwick 3 days ago |
For example I remember Codex having a different line number format than CC and every other harness had to have a two format tool to maximise performance.
I wonder if sucha thing exists still or do they RL on all possible combinations now
franzb 3 days ago |
elAhmo 3 days ago |
reversengineer 3 days ago |
augment_me 3 days ago |
If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.
d-sanoj 3 days ago |
samueloph 3 days ago |
Is this post mixing them up or is my understanding of these concepts off?
alok-g 3 days ago |
SAI_Peregrinus 3 days ago |
nosslin 1 day ago |
bob1029 3 days ago |
jrmg 3 days ago |
The page's title is: "Introducing FrontierHarness Eval v1.1"
andai 3 days ago |
I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.
And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)
anigbrowl 2 days ago |
Programmers, famous for not obsessively configuring/customizing their tools
colinmarc 3 days ago |
jedisct1 3 days ago |
But unsurprisingly, Codex works pretty well, too (and not just with OpenAI models), and Claude Code is just a waste of tokens and money.
0xcoffee 2 days ago |
bensyverson 3 days ago |
worik 3 days ago |
WorldPeas 3 days ago |
justincormack 3 days ago |
shevy-java 3 days ago |
winddude 3 days ago |
Let's not forget multiple departments like the Pentagon, FAA, Homeland Security etc, have been raising alarm that things like this were becoming a heightened risk since Trump and Musk cut ~300k jobs with DOGE, and slashed civilian contractors across nearly every department.
BugsJustFindMe 3 days ago |
maerF0x0 3 days ago |
We waste whatever is cheap to gain what is dear.
(Of course we iterate towards better, but that doesn't mean we delay shipment/usage once it's profitable / better than status quo)
aleqs 3 days ago |
brookritz 3 days ago |
underdeserver 3 days ago |
guanlan 3 days ago |
The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x
What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.
What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.
johnrc 3 days ago |
apishare_api 2 days ago |
haukebri 2 days ago |
xiaoliangliang 2 days ago |
undefined 3 days ago |
Viacol 2 days ago |
amarinhodev 3 days ago |
NamlchakKhandro 2 days ago |
shiqimei 3 days ago |
Olscore 3 days ago |
It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.
Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".
Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.