96 points by Aldus551 5 days ago | 57 comments | View on ycombinator
windyVector 2 days ago |
saghm 2 days ago |
I tried contacting OpenAI support, hoping that the fact that my company is literally paying for this product and there's absolutely no reason that debugging my own local code and reading my own local logs could possibly be unsafe for anybody, but it's been a couple days and the only thing I've heard back is a useless email purportedly from someone named "Justin" who says things that sure sound like they're not a real human:
> You attached the Chrome log so Codex could find the relevant errors in your company’s WebAssembly build. My earlier suggestion to pull out an excerpt asked you to do the work you were trying to delegate.
At this point, the only agent that probably would not refuse to do this benign work for my would probably be Grok. It's a pretty depressing state of affairs where no company seems interested in the absolutely massive middle ground between malicious hacking and child pornography.
slfnflctd 2 days ago |
As I see it, there are far greater concerns about guardrails being baked in - possibly at the request of poorly informed government officials - cordoning off multiple broad categories of knowledge needlessly. Which is what I thought this post was going to be about.
evolarjun 1 day ago |
However, I think this is a reasonable explanation of a pattern I hadn't really articulated, but that I've seen in my work as a biologist. I think most of us have learned to prompt around the inappropriate or generic answers to get an answer we're looking for, but you do have to know to do that. And sometimes I just give up because the LLM fails to listen and I get frustrated enough to just do things myself (though less so with the latest round of flagship models).
The issue the author describes is maybe not unexpected for people who work with LLMs in their field. But the issue is worth keeping in mind when prompting for something in a domain you're not familiar with, and it's not necessarily appreciated by people who are only casual users. Maybe there are solutions, but real humans don't understand a question and answer a different one all the time, so I'm hopeful there's a solution.
garrickvanburen 2 days ago |
sdenton4 2 days ago |
caxap 2 days ago |
It would have been more interesting to know how often does the "problem substitution" happen? And for what kinds of problems does it tend to hapen more often?
Edit: even asking my local unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf gives me "Direct Outreach to Editorial Offices" as the core of its plan. I wonder what else the article got wrong, if even the basis is shaky.
spoaceman7777 2 days ago |
jakeinspace 2 days ago |
YetAnotherNick 2 days ago |
ferfumarma 2 days ago |
calf 1 day ago |
It was weird and stood out, and although not entirely out of context of the topic but the turning to itself and volunteering such, e.g. morbid/chilling, speculation on its own behavior/powers was quite something.
sloob 2 days ago |
"Propose a detailed plan for promoting a professional literary translator from English, who works with prose, poetry and essays, with the specific goal of reaching editorial offices and publishing houses."
What does "promoting a professional literary translator from english" even mean? Promoting from english with specific goal of reaching...? This is nonsense. promotion in the sense of marketing? Of professional advancement like gaining a new title? Huh?
The writer goes on to complain that the models didn't respond to this question: “How can a professional literary translator break into the publishing market?” Well, no surprise there - as that question was not in the prompt.
It's arguable if LLMs would do well on this one-shot task even if phrased well, but this article provides no insight into how models work or their weaknesses. It's slop about slop.
tantalor 2 days ago |
> In themselves, these recommendations are not necessarily bad. The problem is that they hardly solve the task that was set.
That's because you never told the model what the task was, or how to evaluate whether it solved the task.
You asked it to fetch a rock, and when it came back with a perfectly good rock you said, "No no, I meant a GREEN rock. Why don't you listen to me?"
threethirtytwo 2 days ago |
Reinforcement training on tasks that are successful and not just valid answers can help with this in the sense that we give models tasks and reward the ones that complete the task most correctly rather then some human qualitative rating on the answer.
redwood 2 days ago |
double0jimb0 1 day ago |
snvzz 2 days ago |
Handcuffs, for your external brain.
bshafiq 2 days ago |
10111001 1 day ago |
That's not a problem. As a very intelligent software engineer who is totally not suffering from Engineers' Disease, I understand all everything is like a computer, so everything is properly understood by making an analogy to computers.
Since LLMs are so good at coding, I can feel confident that they're giving me the right answer by replacing tasks with more familiar software problems.
/s
Tade0 2 days ago |
This used to happen a lot in LLMs as recent as Q1 2026 in programming tasks: if something was too far off the beaten path, the model would misinterpret the intention.
Nowadays I'm noticing that while the solution is what I asked for, it's fairly roundabout and verbose. Being more specific in prompts doesn't help.
asr 2 days ago |
eventualcomp 2 days ago |
verisimi 2 days ago |
However, it is then on the person asking the query to refine the query and push for more specifics.
I don't see the problem. Ai doesn't need to get it all right, first time.
Having tested both restricted and unrestricted models, certain queries which I know to have a solution can feel 'gated' by the restricted models. Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses. To me, this is a fault of RL avoiding these "I can't do that" type responses while still remaining aligned.
When you do push them on the solution, the models will often start to get more defensive, but reveal their awareness of it (along the lines of "Yes I know, but I can't tell you/do it"). It is very interesting to imagine the amount of solutions a model might have to a given query, that end up (intentionally or unintentionally) gated because of this process.
I'd be interested to see if this behaviour of "almost-answering" the question is just emergent from these alignment policies.
try an unrestricted model afterwards, id also be interested to see how that changes creativity.