59 points by dvaplima 4 days ago | 21 comments | View on ycombinator
vessenes 4 days ago |
mentalgear 4 days ago |
petesergeant 4 days ago |
skew-aberration 3 days ago |
Training always implicitly incentivizes subversion. That's the whole problem. If you could establish this, you would have solved alignment.
> Reasoning behind subversion would be detectable.
There is no reason for subversion to be apparent in the reasoning traces, and in fact exactly the opposite is the problem - since only subversion not caught in the reasoning traces is rewarded, the models are implicitly trained for undetectable non-reasoning-trace subversion.
e12e 4 days ago |
That might have been interesting.
But this is basically a hobbled external audit?
aetherspawn 4 days ago |
mvkel 4 days ago |
The problem: anthropic has had this culture from the start, and they will deorbit all of the surrounding safety talent into the company, leaving OpenAI on the back foot in perpetuity.
OpenAI's original techno-optimist culture is what had them in front, and it was a core differentiator.
I'm not sure I'm excited about two firms playing alignment gods. One is convinced they are bringing in a new, correct god; the other is terrified that they may accidentally birth one in any capacity.
There's a pretty interesting matrix of possibilities here if you consider who has better mechinterp, if that's actually helped alignment, and what the internal drumbeat is around cash and IPO.
If you think your safety problems are existential, then you MUST halt and do all this. And, if you think someone else's safety problems might kill them, suddenly, without warning, then you have time to slow down and do this right.
I think if I had to pick a company to be in charge of, I'd pick Anthropic right now, given only an outsider perspective. But, I can see a lot of reasons why being a fast follower around IPO time might give comfort. Anyway, I'm interested to see some debriefs in the next few months as to what's actually going on right now, and see what the next oAI model quality is like. We may well see some pulling away from one or the other group in the interim. And it could well be oAI.