Hacker news

  • Top
  • New
  • Past
  • Ask
  • Show
  • Jobs

Why is Gemini 4 Carbon taking so long? (https://kernel.dev)

281 points by deenadz 2 days ago | 140 comments | View on ycombinator

tristanj 2 days ago |

It's pretty obvious that Gemini 4 Carbon is not token efficient. It also has relatively low tokens-per-second compared to other models. I can think of several reasons why this is the case:

1. Google has huge amounts of cash and compute. Their goal was to train a model that that has the highest quality & frontier performance, not create a model that's efficient to run and serve. They have other models for that, like Hydrogen.

2. Google created Carbon without caring for inference costs. DeepMind is subsidized by the rest of Google's empire. Selling inference is an insignificant part of Google's revenue. But for Anthropic and OpenAI, selling inference is their entire business model. Anthropic and OpenAI have enormous incentive to sell as much inference as they can in the most efficient manner. Google doesn't have this incentive.

3. They are one of the few companies that can be inefficient. Out of all the frontier labs, Google DeepMind is probably the least constrained by compute. Chinese labs have around 10 to 100 times less compute, so they must release highly optimized models.

horacemorace 2 days ago |

I was just using this one today for some coding tinkering and it was super slow. But the way it solves problems with a light touch while still driving toward sensible architecture is refreshing after claude’s bulldozer attitude. It was able to integrate repeated creeping scope nicely with what it had already done. Very different feel than sonnet/opus, and even gemini 3.8.

hetspookjee 2 days ago |

Might be the only one in the thread but ever since Gemini 4 Argon. I’ve become a huge fan of the Google models. I do use Carbon most of the time since it’s become available but notice that i also am falling back more and more on Argon again without much difference. For security purposes argon has far fewer triggers. I only use the Google models as orchestrator for how they talk and analyse. I let it delegate everything to Astra / Fable / Opus models

fwee 1 day ago |

Carbon is slow, but I've also realized that it works really well. I don't need to get involved or keep checking the results over and over again. I think it's pretty close to Opus 5.5 in terms of work quality.

I have no doubt that Google will adjust its training to find a better balance between speed and intelligence, just like they did with Gemini Flash (3.7 -> 3.8, slower but smarter).

The orchestrator and swarm modes are also interesting. Even though Carbon is slow, it can delegate tasks to other models. (They're both as slow as Carbon, but I think they have potential.)

Anyway, great article!

bagacrap 2 days ago |

This part actually seems like a feature:

> Unnecessary work. About half of Carbon's steps went on work the tickets never asked for: auditing code the fix didn't need and closing loopholes the ticket had already ruled out.

I have anecdotally noticed this, but I thought of it as Carbon "thinking" more "deeply" about the implications of a change. That can be a good thing! Instead of just doing exactly what I ask it, it is able to flag new bugs in far away code that would result, and it's able to find very subtly dead code in large codebases during redactors. My daily driver is Sol, which is exceedingly fast but is relatively focused on the task at hand. I think of my collection of models as similar to a team of humans who all have their strengths and weaknesses, and I'm much more likely to ask Carbon to review some work I've done than to help enact my ideas in the first place.

jedbrooke 1 day ago |

I noticed a similar issue going from Opus 4.6 to 5.5. I also however upgraded my harness during that time, and I found out that it was injecting something like “Be helpful, read in between the lines and proactively go beyond the user’s request” at each turn. I immediately patched that out and saw decent improvement in that area. TFA mentions they tried different harnesses though, so yeah it’s prob baked in to whatever RL setup they’re using to train the original model.

This is why I feel like instruction following is really all I care about in a LLM. I just want it to do what I asked (not what it thinks I actually meant). Intelligence can always be supplemented with tool calls, but an LLM that won’t do as it’s told is no good

timtam 2 days ago |

Using it through the prompt box, I kept running into 500 INTERNAL (TURN_STEPS_EXCEEDED). Gemini couldn’t finish this response I’m sorry, this request needed too many steps to finish in time. Please try asking a more specific question, or break it into smaller requests.

I only got around the issue by strictly setting an effort budget or a stopping rule.

Otterly99 1 day ago |

> Closing loopholes the ticket had already ruled out. Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

This is unfortunate. This was one of the most annoying behavior that appeared at Opus 4.6? 4.8? Not sure exactly, but one of the post 4.5 version started to be very proactive, on top of being very verbose. Thankfully, Anthropic did listen to their users and improved things greatly on this front.

It seems that Carbon is designed to have the same flaws, I would advise Google to listen to their users and get rid of the verbosity and proactivity.

chriddyp 1 day ago |

I've tested 75 models so far, and Gemini 4 Carbon is 68th/75 in terms of slowness [1]

Looking into it, there are 4 areas that contribute to speed:

1. Time-to-first-token is slower than others: 5s (median) to start on a one-line prompt and 11s on a ~130K-token cached prompt, against 2-2.5s for Gemini 4 Hydrogen and 1-5s for GPT, Claude, Grok, Kimi and Qwen. Long tail though: one call took 50 s and one timed out.

2. Tokens/s is slower, but not that much: 3-9s for Carbon vs ~5s for Hydrogen.

3. It’s not wordy: thinking included, Carbon wrote ~500 output tokens per request vs ~780 for Hydrogen, and less over the whole exam (220K vs 259K). Slow-but-wordy models like Claude Fable 5.2 and Kimi K3.1 Pro write 1,600 to 2,200 per request.

4. More thorough, more tool calls: 10.3 model calls per question vs 7.7 for Hydrogen. On data-quality audits it made 17 vs 9, and on our "find the hidden pattern" questions it ran 41 queries on average vs a board median of ~25.

The 4th piece is pretty interesting as it’s more about the behavior than the infra. If the task is “find all of the issues”, and the model is very exhaustive in its search, then “time per task” calls it slow but in practice we’d call it thorough.

[1] https://plotly.com/blog/gemini-4-carbon-plotly-data-analytic...

p_l about 11 hours ago |

Interestingly, agy 1.5.13 (but seen that with 1.5.12 already) now works hard to coalesce multiple steps in tool calls, just like Opus does - i.e. building small bash one liners with separators

Lerc 2 days ago |

A Theo vid suggests there's a time to first token issue. When doing subagent actions it can compound to be massive from a bunch of internal short queries.

It really looks like something is broken more than just being inefficient.

totalhack 1 day ago |

Carbon has been slow, but excellent at deep dive analysis of some of my projects, finding optimizations other models missed AND explaining them in a clear and concise manner, even as the conversation develops and really becomes four or five subthreads on the opportunities that it brought up. Maybe that massive context window helps.

logicchains 2 days ago |

Still gives far more value for $200 than an OpenAI subscription; Astra burns through limits way faster than Carbon, and Sol makes way more mistakes than Carbon (and Opus 5.5). Maybe it's only temporary though; the Google $200 subscription gives 4x then $100 one, while the OpenAI one only gives 2x.

bugzbenny 1 day ago |

Using any of the latest models within Antigravity is incredibly slow. I use Claude Desktop for sending off coding agents into the cloud and I get about 10x more done with it than I can do with Antigravity + Google models. The only advantage to the Google models over Claude so far that I've noticed is they are way better at writing copy.

Anecdotally, the Google harnesses (Gemini web/desktop, or Antigravity) tend to lose connections way more than their competitors.

I wanted to like Google's offering (since all my data is there) but I feel as though they've really dropped the ball.

lmeyerov 2 days ago |

It's been jarring & confusing to go from a year and a half of Claude Code max plan, and equivalent of Codex, to Google agy & their models

My flow is roughly a few days to max out Claude sub, a few days to max out codex, and 1 afternoon to max out agy

While Astra/fable, sol/opus, sonnet/Luna are largely interchangeable when jumping codex vs Claude Code on the same repo+skills... that hasn't been true when using agy with Google models

My next stop may be drop agy harness in favor of codex/opencode to see if the models are fine and agy is the problem

undefined 2 days ago |

undefined

SubiculumCode 2 days ago |

That isn't a good look, and either this is unmeasured uncontrolled behavior, or intentional under some longer term plan...but the latter possibility seems improbable.

seizethecheese 1 day ago |

We have only 146 battles [1] with Carbon, so I kicked off an analysis with skepticism we'd have anything to add here. I was wrong. Carbon is that slow.

Biggest differences:

- Time to complete: 6m 08s for Carbon vs 2m 04s for opponents.

- Output tokens 16.4k for Carbon vs 6.8k for opponents.

- Tool calls: 21 for Carbon vs 11 for opponents.

All three of these are statistically significant.

The ones that don't seem to be a huge issue, but have been mentioned here or in the blog are inference speed (52 tps vs 73 tps) latency (12s vs 8s) and agent turns (13 vs 8).

Interestingly enough, Carbon has a high win rate when the voter waits for both solutions to finish, but a terrible win rate from early voting.

- Both finished 83 carbon wins, 52 losses.

- Early votes 2 carbon wins, 40 losses!

Basically, Carbon is good when you're willing to wait.

[1] we're building a battle system on top of opencode. leaderboard: http://agentduel.com/leaderboard app is open source: https://github.com/bottomless/agent-duel

lp92 2 days ago |

>Carbon defended against cases the ticket's facts had already excluded, hardening the code against problems nobody had reported.

That's pretty interesting. Seems like it's over analyzing. Not the worst outcome though to have additional hardening.

With that said, I found Argon to be better for most coding tasks and Carbon for architecture/deep RCA analysis of complex issues.

undefined 1 day ago |

undefined

niltecedu 1 day ago |

I am pretty sure caching is busted, atleast on github copilot, I tried doing it via pi harness however cannot seem to get any cached inputs post 5 minutes.

mengqijin 2 days ago |

I tested with long and complicated tasks Carbon did perform better and took almost the same time as Hydrogen, roughly 50% more cost than Hydrogen.

But with simple tasks, Carbon is way too slow, and much higher cost.

undefined 2 days ago |

undefined

5555watch 2 days ago |

Carbon is extremely good for particular research questions I'm doing, and the only hope at the moment is that they don't eventually nerf it down, especially if topics like these about the speed might raise concerns.

In my several anecdotal cases it's miles ahead of Astra Ultra & Opus 5.5/Fable 5.1 (haven't tested on Astra 6.2 and Fable 5.2 yet). Importantly, the Astra and Opus criticizes the ideas proposed by Carbon as bad, they both use the same critiques. While from my eyes the Carbon makes good points.

So recently I've started ignoring Astra & Claude, and said, go on Carbon, finalize the proof of concept and let's see your results. Few nights of calculations, they do some good work in my eyes, I show the results and workflow to Astra & Claude, and only after seeing it all they say "This is substantial progress!". So yeah, duh!

Two points: First, it seems Claude and Astra are super correlated models; Gemini is less correlated.

Second, they criticize everything so much, that these critiques are evident in their workflow - they're too careful to do something risky, so they can't find good solutions if they're not super obvious. And I think if referees will be using these models to write reviews, they will nitpick on tiny details and will always find dozens of them.

oytis 2 days ago |

They claimed a precious domain I should say, shame that they are not using it for anything kernel-related

jrflo 2 days ago |

Is this due to Google wanting to catch up on benchmarks at all costs, and trains their models to be overly meticulous to do so?

dada216 2 days ago |

ITT it's that huge max output token of 1mln tokens.

It's pretty far beyond other models (Opus 5.5 is at 128k).

ahmadyan 2 days ago |

i'm having all sort of problems with the gemini carbon lag. using it through open-router is impossible (at least for me), the request times out after 5mins, and often it take more than 5mins for the first token to arrive. I don't know what batch size they are using, but this is not useful.

cybice 2 days ago |

Interesting that it looks like behaviour of cheap models, like deep seek flash overuse tool calls

timcobb 2 days ago |

really glad to see _this_ kind of analysis of these models. These type of analysis is much more practically useful than some context-less (especially with regard to time) number of "how good it is".

user_7832 2 days ago |

I am increasingly more convinced every day that every LLM company is just making their latest model to be the best at (coding) benchmarks, at the cost of pretty much everything else.

I use LLMs for non code purposes (probably not very common on HN) and Sonnet 3.5/3.6 still ranks as probably one of the best models I've used.

Can today's models put out tech demos and call more tools that make it way more impressive? Sure. But it's really hard to see any actual significant gains over the last few months, they're so mild (again, non coding).

nandanadileep29 2 days ago |

yeah i tried using it its not efficient at all. just burns up the token and not get the work done.

resters 2 days ago |

i wanted to test carbon but ran out of quota before i could get a definitive result.

WeZZard 2 days ago |

I didn't try but some people said Gemini 4 Carbon served by VertexAI is faster.

kristjansson 2 days ago |

Argon isn't much better. It just. keeps. going. OTOH when I asked it (in the agy cli) about some detail of how the permission system it was neck deep in decompiled go by the time I lost patience and killed the turn. So at least sometimes it's doing something with the time and persistence?

jaren 1 day ago |

any hypothesis why slower inference?

gdevenyi 2 days ago |

I just got a 6 month max subscription via the antigravity science program. I've tried to start using it and found both the harness to be terrible and the model to be very slow.

It might do good work but I'm having a hard time figuring out if it is.

0xbadcafebee 1 day ago |

Gemini is consistently the worst performing AI product, agent, model, etc I have used. I don't really understand who keeps using it.

dario_amodei 2 days ago |

how much?

coldbreeze485 2 days ago |

[dead]

sodowork 2 days ago |

[flagged]

georgestones 2 days ago |

[dead]

ath3nd 1 day ago |

[dead]

NoahZuniga 2 days ago |

Google pretty clearly says in their product documentation that carbon is just a version of carbon that thinks way longer.

erji77 2 days ago |

You idiot, stop pushing AI news to me from now on.