281 points by deenadz 2 days ago | 140 comments | View on ycombinator
tristanj 2 days ago |
horacemorace 2 days ago |
hetspookjee 2 days ago |
fwee 1 day ago |
I have no doubt that Google will adjust its training to find a better balance between speed and intelligence, just like they did with Gemini Flash (3.7 -> 3.8, slower but smarter).
The orchestrator and swarm modes are also interesting. Even though Carbon is slow, it can delegate tasks to other models. (They're both as slow as Carbon, but I think they have potential.)
Anyway, great article!
bagacrap 2 days ago |
> Unnecessary work. About half of Carbon's steps went on work the tickets never asked for: auditing code the fix didn't need and closing loopholes the ticket had already ruled out.
I have anecdotally noticed this, but I thought of it as Carbon "thinking" more "deeply" about the implications of a change. That can be a good thing! Instead of just doing exactly what I ask it, it is able to flag new bugs in far away code that would result, and it's able to find very subtly dead code in large codebases during redactors. My daily driver is Sol, which is exceedingly fast but is relatively focused on the task at hand. I think of my collection of models as similar to a team of humans who all have their strengths and weaknesses, and I'm much more likely to ask Carbon to review some work I've done than to help enact my ideas in the first place.
jedbrooke 1 day ago |
This is why I feel like instruction following is really all I care about in a LLM. I just want it to do what I asked (not what it thinks I actually meant). Intelligence can always be supplemented with tool calls, but an LLM that won’t do as it’s told is no good
timtam 2 days ago |
I only got around the issue by strictly setting an effort budget or a stopping rule.
Otterly99 1 day ago |
This is unfortunate. This was one of the most annoying behavior that appeared at Opus 4.6? 4.8? Not sure exactly, but one of the post 4.5 version started to be very proactive, on top of being very verbose. Thankfully, Anthropic did listen to their users and improved things greatly on this front.
It seems that Carbon is designed to have the same flaws, I would advise Google to listen to their users and get rid of the verbosity and proactivity.
chriddyp 1 day ago |
Looking into it, there are 4 areas that contribute to speed:
1. Time-to-first-token is slower than others: 5s (median) to start on a one-line prompt and 11s on a ~130K-token cached prompt, against 2-2.5s for Gemini 4 Hydrogen and 1-5s for GPT, Claude, Grok, Kimi and Qwen. Long tail though: one call took 50 s and one timed out.
2. Tokens/s is slower, but not that much: 3-9s for Carbon vs ~5s for Hydrogen.
3. It’s not wordy: thinking included, Carbon wrote ~500 output tokens per request vs ~780 for Hydrogen, and less over the whole exam (220K vs 259K). Slow-but-wordy models like Claude Fable 5.2 and Kimi K3.1 Pro write 1,600 to 2,200 per request.
4. More thorough, more tool calls: 10.3 model calls per question vs 7.7 for Hydrogen. On data-quality audits it made 17 vs 9, and on our "find the hidden pattern" questions it ran 41 queries on average vs a board median of ~25.
The 4th piece is pretty interesting as it’s more about the behavior than the infra. If the task is “find all of the issues”, and the model is very exhaustive in its search, then “time per task” calls it slow but in practice we’d call it thorough.
[1] https://plotly.com/blog/gemini-4-carbon-plotly-data-analytic...
p_l about 11 hours ago |
Lerc 2 days ago |
It really looks like something is broken more than just being inefficient.
totalhack 1 day ago |
logicchains 2 days ago |
bugzbenny 1 day ago |
Anecdotally, the Google harnesses (Gemini web/desktop, or Antigravity) tend to lose connections way more than their competitors.
I wanted to like Google's offering (since all my data is there) but I feel as though they've really dropped the ball.
lmeyerov 2 days ago |
My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy
While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models
My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem
undefined 2 days ago |
SubiculumCode 2 days ago |
seizethecheese 1 day ago |
Biggest differences:
- Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.
- Output tokens 16.4k for Carbon vs 6.8k for opponents.
- Tool calls: 21 for Carbon vs 11 for opponents.
All three of these are statistically significant.
The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).
Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.
- Both finished 83 carbon wins, 52 losses.
- Early votes 2 carbon wins, 40 losses!
Basically, Carbon is good when you're willing to wait.
[1] we're building a battle system on top of opencode. leaderboard: http://agentduel.com/leaderboard app is open source: https://github.com/bottomless/agent-duel
lp92 2 days ago |
That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.
With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.
undefined 1 day ago |
niltecedu 1 day ago |
mengqijin 2 days ago |
But with simple tasks, Carbon is way too slow, and much higher cost.
undefined 2 days ago |
5555watch 2 days ago |
In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.
So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!
Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.
Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.
oytis 2 days ago |
jrflo 2 days ago |
dada216 2 days ago |
It's pretty far beyond other models (Opus 5.5 is at 128k).
ahmadyan 2 days ago |
cybice 2 days ago |
timcobb 2 days ago |
user_7832 2 days ago |
I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.
Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).
nandanadileep29 2 days ago |
resters 2 days ago |
WeZZard 2 days ago |
kristjansson 2 days ago |
jaren 1 day ago |
gdevenyi 2 days ago |
It might do good work but I'm having a hard time figuring out if it is.
0xbadcafebee 1 day ago |
dario_amodei 2 days ago |
coldbreeze485 2 days ago |
sodowork 2 days ago |
georgestones 2 days ago |
ath3nd 1 day ago |
NoahZuniga 2 days ago |
erji77 2 days ago |
1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.
2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.
3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.