Could there be a benefit to releasing a new model, slowly dumbing it down over a couple months, then releasing a new model that’s marginally if at all better than the original to create a perceived improvement when in reality there isn’t really one?

For an industry that’s stagnant in progress yet relies on new frequent releases to survive (non-progress being an existential risk), this could make sense.

I have no idea if that’s what’s happened, I completely pulled it out of my butt. And I have no idea is the actual frontier is stagnating.

The Office of Weights and Measures exists because, long before any of us were born, in 1836, companies were up to shady shit and consumers were paying for inconsistent products. I.E. Being scammed.

AI companies should be subject to the OWM like any other company that sells a product that varies in weight. Perhaps when a sane administration is re-elected; one that can read history books and comprehend why our regulations exist in the first place. Or have even a semblance of respect for its citizenry.

I have no hard data but I have a strong feeling this morning that something's wrong with Fable 5 compared to Friday evening.

Just an hour ago I had Fable correctly identify an unused method that could be deleted. I then immediately get a diff for an exact duplicate method, and then Fable outputting, "I accidentally duplicated <method> instead of deleting it. Removing both copies now."

The remaining morning complaints that makes it feel like something's off is that it will do a lot of "thinking" for simple things that previously took very little time. And it got very lost and completely mixed up DE-91M predicate names and implementations. Just absolute disaster code that I had over the past months come to generally expect it to do without issue.

Glad I carefully review everything. I think what I need is reliability and consistency. But it feels like picking a model from the list doesn't guarantee that: that the models' "brain" is open on the table and they're screwing with it.

Just yesterday I was thinking about gpt-5.6-luna. I made it my default model in Hermes during its fist week of launch. It was just as good as 5.5 which was my previous default. But over the last 2 or 3 weeks I've seen how dumb it is now. I have to be very explicit with it.

For example, I used to be able to prompt "Check the system logs on <server> for...." and it would just figure it out. Yesterday I asked "Did <service> on <server> complete the overnight job" and all it said was "that service is not installed on my host"

I had to tell it to ssh into the server and run journlctl to check it

Anecdotal, I know, but they all seem to be less capable with time.

_edit_ I use the same reasoning level of `medium`

I seem to recall Anthropic going on record saying that they don't do anything to model performance to stretch their compute capacity. I've anecdotally noticed massive peaks and troughs in performance week to week (albeit with Opus, not Fable).

I wonder what their official explanation for this behavior is.

Anecdotally, I have found the same. I spend a lot of time with these frontier models, brainstorming, etc. and the drop in performance from, say, week 1 to week 8 is often massive. Whereas in the beginning, it seemed like a capable research assistant, by the end of week 8 or so it starts acting like a puppy dog eager to make its 'master' happy for a few treats.

I've followed a few trackers, eg https://marginlab.ai/trackers/claude-code/ , for awhile. For Claude Code the trend, it seems to me at least, is fewer tokens to do the same or better job. Prompt changes, tool ergonomics changes, etc.; I'd be shocked if they didn't A/B every release. Less thinking as measured by tokens isn't necessarily bad if you can get the same results by making it think about the "right" things or structure. They obviously screw up sometimes, and I've always been suspicious with hidden tokens, but I haven't found evidence quality intentionally degrades over time.

All llm api providers should be compelled to return a checksum-like proof of quantization level of the model that served the request. Basic transparency should be the bare minimum.

It is clear by now to me that Anthropic is constantly trying to find a kind of “auto” degradation perhaps to save money on work it thinks does not require high reasoning. I always use max reasoning and I can clearly see differences between the models when they release and after 3-4 weeks. I think they give a kind of intelligence boost also for new accounts.

I'm always excited to run models on my local machine. I use Ollama for that.

Obviously. The standard pattern is that model X is basically AGI and wins all benchmarks, followed the next day by Y and Z, which both win all benchmarks, too.

Then weeks later people find out that they have been duped and complain that the models have been quantized or employ worse inference.

Buy decent coffee instead of your $200 subscription and sidestep all the scams.

This is a project i wanted to implement for a long time. It regularly benchmarks cloud hosted models with private benchmarks. Not just openai & anthropic, popular openrouter models too.

Tests their intelligence, not their diligence.

Sadly i cant think of a way to monetize the service. Also if it ever gets famous enough labs would try to game the system, it would be cat&mouse game that i am not willing to waste time on without any monetary gain.

> Instead of finding a nerfed model, after six weeks of reconstructing wire logs, parsing transcripts, analyzing output tokenization, and staring at data, I found a much deeper issue. The model identity had remained the same, but the inference regime being delivered behind that model had not.

Is this something specific that shows up in the wire log, or is this the author's intepretation? The fact that Claude Code versions change over time in the test is suspicious. Anthropic has stated in the past that the underlying model behavior does not change over time, but Claude Code will change from version to version and this is expected. So if it's just Claude Code more aggressively tuning some knob in its requests, that's a pretty different thing than the underlying model changing.

You should read this person's full article to understand what these charts are showing https://x.com/Lon/status/2101034933284417614

If you thought this was a repeated test of the same problems showing fluctuating performance, it's not. They set up a MITM proxy between Claude and the servers and ran analysis on the work they were doing.

So those ups and downs in the charts, which they plotted with sub-daily resolution, are just as much a function of their work changing from day to day. It's like plotting the miles per gallon of your car and blaming the gas station when the number goes up and down, without admitting that some days you drive to the grocery store on surface roads and other days you drive up a mountain on the freeway.

> The corpus analyzed in Charts 1-5 comes exclusively from Fable 5, at xhigh and max effort levels, during sustained production work across a diverse set of projects and workloads. Data was aggregated from transcripts and live wire logs

The analysis (which feels very vibe-slop) gets worse from there. In the second half they take thinking token counts for ARC-AGI-2, thinking problems designed to stress LLMs, and compare their average thinking-tokens-per-turn counts to that!

If you don't realize why this is so flawed: ARC-AGI-2 is a benchmark meant to collect problems thought to be extremely difficult, nearly impossible, for LLMs. If your goal was to cherry-pick a mislead example which would produce the highest number of thinking tokens, this is it!

Your daily coding work should not be producing a proportional number of thinking tokens on every invocation while it reads through some source code or edits a couple lines in a file.

You don't want to maximize the number of thinking tokens. You want problems solved accurately with the minimum number of tokens.

Confirmation bias runs deep on this topic so I assume few people read the analysis before posting, but as far as experiments go it's basically useless. Are they changing something on the server? I don't know, but this analysis isn't useful for answering that question.

Surely it's easy to verify by simply doing Benchmark tests every 2 or 3 days but not publishing them so they can't be gamed, then releasing all at once?

This is exactly what I have been experiencing and the difference is night and day! We have been advertised and given a taste of what Fable was and after that been served an exteme watered down version. It is so bad that sometimes chatgpt feels better.

Definitely noticed this before, but this parti6 time was very noticeable. I'm convinced it's to get the benchmarks in, then lower cost and prepare for the next release to look better relatively to users.

This thread pops up like clockwork when a new model is about to drop

The question I have is this only happening for a subset of users working in specific areas, such as AI or distributed systems (https://news.ycombinator.com/item?id=48742153), or is this across the board? I am working on distributed systems. Today Fable is mostly unusable. It resembles Opus, so I went looking to see if anyone else is having issues. Sure enough.

So in 5 years will they lose a suit for intentionally deceiving users? Or is something baked into the ToS by now that allows them to adjust things like this?

It's easy to imagine this only happening for subscription accounts rather than paid API usage. Any data on this?

How do you create repeatable tests in a non-deterministic system? Every time you send the same prompt you get a different answer.

I for sure felt that this was the case for a while now, but couldn’t explain it. Newly released feels great for the first couple of weeks, but then it starts to get worse.

Makes sense, no? Test time compute is something you can vary, so it makes sense that you start covertly reducing it once the model has already made it's splash.

The NERF is finally established, this should be part of the ever growing benchmark maxxing.

How do you measure thinking tokens? They don't send those back to the client.

Wasn’t there a website that was tracking that?

They want transparency from everyone else but not for them ... you don't say.

I strongly believe that the real Fable is the one we had for a few days in June. Then they nerfed the model a bit after the government pulled it off the market. What we have now is something less, but still good

undefined

Anthropic is straight up scamming its users at this point.

Perhaps RSI by virtue of feeding models their own slop as training data exponentially decays the ‘quality’ of the model?

This is great data, and good but somewhat flawed analysis.

The good part is showing that the drop in thinking tokens persists no matter what grouping you slice across. They make a very persuasive case that there's something systematic going on.

My usual complaint about these "they're nerfing the models, I feel it in my bones!" posts is that they don't account for the workload changing. From working on my own stuff, there are a series of evolutionary/de-evolutionary changes that happen in a heavily AI-written codebase. Initially everything goes great. Then the AI takes on too much technical debt. Improvements slow down and regressions creep up until it becomes a never-ending game of whack-a-mole just to keep up. So you direct some (probably AI) effort towards cleaning things up, reducing duplication, and removing patches for problems that are better fixed with a design change, or workarounds because the harness saw the wrong version or you incorrectly described a problem and it strenuously solved a non-problem. That gets you back up to cruising speed for a while, then the project exceeds some hidden threshold for size in latent space or something, and further progress has to rely on attending to one aspect of the codebase at a time. Once again, the architecture becomes the limiting factor, but in a subtly different way. My sense is that it all boils down to some sort of "attention capacity" -- is your codebase and problem space amenable to looking at one aspect at a time, or is it all snarled together? -- but that's an essay that I'd love to write but really don't have enough experience to do justice.

Anyway, the details don't matter. The point is that not only can you not assume that the difficulty presented to the AI is roughly constant over time, but also there's evidence to believe that it will be normally be increasing. (Unless you're constantly starting new projects instead of continuing old ones.)

That's why I like this writeup. Focusing on thinking trails doesn't eliminate the problem of snowballing difficulty, but it does sidestep the worst of it. In fact, I'd expect the same setup to think more as the complexity/sloppiness creeps up.

The flawed part that bothered me was that it feels like there's a little bit of a predetermined conclusion that thinking is a magic sauce that makes everything taste better if you spread it on everything. I want a high variance on thinking, especially between interactions. A smarter model would have a higher variance, in my opinion. So the accusatory tone (perhaps I should reread it? My first impressions are often wrong) around "look! it doesn't bother to think at all a lot of the time. That can't be right!" seems misguided to me. It should think when it needs to, and if its thinking was clear then it won't need to re-think over and over again; it's all still in the context.

Forgive the anthropomorphization, but consider those studies of chess experts vs novices. Novices have to work way harder, working through all kinds of things from scratch, while the expert instantly and effortlessly recognizes what's going on.

But anyway, the main takeaway fully survives this criticism. The models appear to systematically think less over time. It doesn't matter if a smarter model might be able to think less for the same quality; this is happening over the same model.

There's a lot of chatter on other forums and Reddit about the same thing happening to Astra over the last couple of weeks.

The ROI just isn't there. It feels like Fable is in the same place Opus was early last year; at best marginal improvement that's barely noticeable over the lower model, for 10x the cost. I'm sure it'll take over as the workhorse as Opus did once they get it down, but right now it just doesn't make sense

I'm always excited about running local models at home. This is one of the reasons. I pull it down from hugging face, and it continues to work at the same level of performance indefinitely.

The model improvements value are at the plateau of utility right now, peeling out small gains which is pretty “meh” in terms of business value

at this point frontier companies are just selling upgraded harnesses and tool calls with the rest of us

Don't they continuously tweak the models post-release?

Check gpt I think they recently started taking the same route

This is like shared clouds back in the day where if someone is using the CPU more it impacts you, just pool every one to the same service. There should be an SLA but for the intelligence of these models, otherwise, you are sold fable but with the intelligence of a table.

The smart takeaway is not skepticism or snark, but understanding that once the new datacenter buildout starts coming online, cheap and widespread access to even the current frontier models (without strict thinking limits) will blow the economy wide open.

(ie, even a pause in AI training isn't going to stop the train where AI flips the economy upside down, we've barely even seen the impact of the current frontier)

I stopped using Fable long time ago. It's worse than Sonnet. Opus is not much better.

This cycle of new model running at full quantisation and then nerfed few days / weeks after premiere should be called out. Anthropic should also drop the adaptive reasoning scam.

If I pay for Fable, I should get full, not nerfed model at honest pricing.

Regulators should investigate them.

OpenAI is no different. Astra has basically the same problem.

Gemini Chat is constantly throwing, "Pro is in high demand right now, a different model was used for this generation," too.

I'm thinking they're all running out of physical resources. It's the DotCom bubble all over again; rollout of the physical infrastructure that's necessary to keep all of the pie-in-the-sky promises will not happen on the timescales that investors can work with, and they will panic when they realize this.

EDIT: And, frankly, I can't wait. I'm tired of the sketchy and dishonest way these companies are behaving.

Opus 4.8 was smarter and possibly 10x faster than Opus 5, too. They are dumbing things down on purpose. I am praying for open models to become at least as smart as Fable soon so we can ditch these shitty, lying companies.

I was rooting for Anthropic 2 years ago, but now I have become an extremely bitter customer. Just another version of OpenAI, if not shittier.

[dead]

[flagged]

[dead]

Oh boy, a new "nerfed model" conspiracy theory, never seen THIS before