Basically if a Car A is performing better (be it speed, milage or in general sense) than Car B, then it is not necessarily because its engine. It could be because of better tires, better gearbox, lighter body, better usability of features, etc.

You can implement an AI feature (like AI for BI) in different ways even with the same model - via ReAct-loop, or plan-and-execute, or hybrid. You can make it stateless, stateful, RAG-based, etc. depending upon whether you want to prioritize result accuracy or depth of analysis. You can use LLM to generate either intent (requires lesser reasoning) or the queries itself (requires much more capable model).

Your harness can adapt to the underlying model's native capabilities, or can make up for its absence, e.g. query generation in above example requires your model to have MOE capabilities but intent generation wouldn't.

https://rahulmax.com/notes/how-i-keep-the-ai-bill-down/

In their case, context management pays off more the tighter your window. Their gap between managing and not managing is 35.7 points of success rate at 32k and 2.7 points at 128k. My version of that was a rule I stick to, as much as I can. I checkpoint a session at about 25-30% of the window, write the state out to a PROGRESS.md and a JSON file of the requirements, and start fresh. This restart costs me 30 seconds, since a bloated session doesn't get any cheaper the longer you stay in it.

Also worth knowing that the models are Nemotron-3 and Mistral-Medium, not the frontier models most people here are paying for.

I would have to imagine, that Luna would very much fall into the camp of "bash capable". At which point- it seems to me that adding any tools beyond just Bash requires some rigorous testing and verification that value is being added.

Is it the harness that enables the LLM to use tools and implement plans, or is it the short leash that, through many guardrails, ensures that the LLM follows the desired path?

I see both meanings used. The former is still plausible, but I see many people using the second interpretation.

> Planning improves success at additional cost for weaker models but mainly reduces cost, with small decreases in success rate, for stronger models.

> Predefined tools raise success rates for models with weak bash control, whereas bash-only yields higher success at lower cost for bash-capable models, most clearly on shell-centric task types.

> context management extends execution trajectories without substantially altering agent behavior and is most beneficial under tight context budgets

> planning sustains the trajectories of models that abandon tasks too early and trims repeated verification in models that verify too long

> structured tools support models with limited shell proficiency, while bash-only enables capable models to combine multiple code modifications in a single tool call

Seems fairly intuitive to me, based on feeling. But also fairly kind of obvious; bash-only tooling has higher success for bash-capable models, compared to using predefined tools for models that aren't good at bash? Yeah... They all seem a bit "duh" to me. The final piece of the conclusion is agreeable regardless of how they arrived at it though:

> Harness design is thus a conditional systems problem in which each component should be selected for the target model, task type, and resource budget rather than adopted as a default.

I think lots of people treat the harness/model/prompts combo as interchangeable, but in my experience the quality and efficiently depends heavily on the combo of the harness/model, and using the harness + model made by the same lab, has vastly better experience compared to more "general purpose" (for the lack of a better term) harnesses. Most likely because they use their own traces when training future model iterations.

Always looking for more harness evals, although I'm going broke running them across all these models and tasks.

Anthropic appears to agree frontier models don't need in-session planning tools.

https://github.com/anthropics/claude-code/issues/80487

I tried to get my lawyer-mom switched to Linux a bit ago, and she loved it in generally, but none of the Office competitors had good enough compat to work.

The only other thing keeping her on Windows is Adobe PDF pro, since it can do OCR where, when you edit it, it reflow the text in a font that matches the scanned in one to look like the original. (I also got weird "This feels like it enables fraud vibes" from this, but, no, turns out it's a totally common workflow for lawyers to need to do this ... I hate it.)

Cool to see 1/2 of the problems keeping her on Windows solved.