I hope the future of AI isn't this sort, where companies provide the user/customer with an interface to an AI that can do things for the user... I would much prefer that companies instead provide an interface FOR an AI, and the user brings their own AI which connects to that interface.

In other words, provide my AI with tools, instead of providing me an AI that uses your tools.

That way, my AI can bring all the context it needs, and I can bring all of the settings and knowledge about what I want with me. I don't want a fractured world of tons of AIs i interact with where I have to explain all the fundamental information about what I want and how I work every time.

This also has the benefit of sidestepping the issue the essay is talking about. You provide a consistent tool, and the AI weirdness is not your issue anymore. You don't have to worry about solving for all the weird ways people prompt the AI, or the ways they break.

Instead of anger, I think one of the few take aways from Peter Steinberg Mr Open Claw was his view on agentic empathy.

How would I try to compute things with the amount of context? But quickly it turns into a blame game, of at parts trying to give WAY too much context.

This is an amazing article. The problems it describes are exactly what we found when building a production system that used LLMs to (most of the time) produce reliable results. Extensive tests are necessary, and stakeholders have no idea how their suggestions fail in production. They just see the handful of times they tried something and had it work, not the long tail of cursed results. (“How hard can it be? Why don’t you just…”)

We didn’t get to the point of self-built prompts, as the article suggests, but it’s an intriguing idea.

As awesome as this is, because as much as I want to create an agent for my customer that is predictable and "deterministic", I can't help but think of how wasteful, expensive and not-fun this is.

Good for the author that it's fun for them, but for me it seems like I am in that "monkey ladder banana" experimemt: doing something because others are and the customer is giving me bana... sorry, money for it, convinced it will help him (the money would 100% stop if I started looping prompt optimizations like this).

If I have so many tools and MCPs as I do currently, and with each the behavior regresses and changes wildly, it seems I should either merge tools and do more automations and come back to the prompt. (The alternative being training my own model?)

One issue with this that we ran into is that it costs actual countable money to run the test suite, which is distinct from anything else I’m used to, so the notion that we’d do enough testing to generate a statistically significant gauge of performance - man, I know it’s correct, but I’m not sure my company will survive the process.

This was great! When you think about optimizing prompts with GEPA (or comparable methods and tools), do you consider each tool or skill separately? How do you think about the optimization of the system prompt for a large agentic system?

I imagine that you do a collection of passes to cover each overlapping set of what you want evaluated, but the system prompt makes all cases dependent on each other.

What I’ve done in the past is use the system prompt to extract subjective criteria for an LLM judge (like various system prompt statements that contribute to brand voice) and check individual traces with that for evaluation, but I’d like to move beyond including that in a prompt at all.

No idea how effective this is, but it sure looks a lot more like engineering than most of the 'prompt engineering' things I've seen in the past years. Kudos to the author for writing this concisely without aggrandizing his work.

> the textual nature of prompts leads us to take the intentional stance towards systems which aren’t conscious, and thus miss the essential nature of their non-meaning

I see LLMs as being capable of making useful distinctions and having a rich action space. They are widely used because their operation is useful, and that can only happen when semantics work well in practice. But useful things that pay for themselves don't need our "essential nature" blessing, they already have persistence by mutual entanglement with us.

> That is currently working, but since the fix is fully deranged I expect it’ll be disturbed again at some point.

This one made me laugh out loud

Am I the only one who waited to the end for, and was disappointed not to see, a peek into these optimized prompts? I so want to take a peek into that abyss, even if that risks the abyss looking back at me. I'm curious just how twisted they get relative to the original, in what alien ways.

The format of this article makes it almost impossible to read.

I gave up after about 10 "pages".

undefined

[flagged]

[flagged]

[flagged]

[dead]

[dead]

TL;DR: you need to do... essentially supervised learning on your prompts? I mean, if I wanted to do ML, I'd already have been doing it ten years ago.

[flagged]

[flagged]

Crafting a good prompt is what differentiates an expert's output from a beginner's. Of course you need good constraints too. But those good constraints are created precisely through good prompts.