882 points by ilreb 4 days ago | 485 comments | View on ycombinator
simonw 4 days ago |
maplet 4 days ago |
seizethecheese 4 days ago |
And it’s a shame because the only reason it scores so low is that it’s slooooow. (Like 30 tokens per second vs opus near 80.) If you exclude voting, Qwen 3.8 does well.
(We allow early voting contrary to Arena.ai because I think it makes sense for agents, where it doesn’t for chat. This means our data takes speed into account.)
Also, it would be ideal if they served this with third parties. Part of the slowness with 3.8 is just not a lot of providers. Their open source version is much faster, but we couldn’t include it because it doesn’t handle images (another own goal by Alibaba).
hauntingseaweed 4 days ago |
dan_thinks 4 days ago |
Looking forward to the "Flash" and 27B versions.
pu_pe 4 days ago |
DeepSeek has a similar problem, I feel like their full model is hardly even worth using, while the Flash tier is absolutely amazing.
spwa4 4 days ago |
What has happened to artificial analysis is that they've begin to focus a lot on very broad tasks like legal reasoning, using very difficult to work with scientific software packages, correctly analyzing biology experiments (without or only tiny amounts of explanation, count cells, BUT leave it to the model to figure that cell that aren't round, or have a distorted cell membrane (ie. have burst) are dead), very difficult physics reasoning ("2 electrical wires are close together with power flowing in the same direction. What happens?", and you get points deducted if you don't realize the problem with high voltage AC power, where the wires will shake themselves loose if the phase doesn't match perfectly, meaning the wires need to be EXACTLY the same length or ...). If I'm being impolite I'd say the frontier models are trying to make sure they'll pass the "gotcha" interview questions for many jobs.
The 3 differences:
Qwen 4 is a huge jump from Qwen 3.8 in coding
Qwen 4 is a big jump in using it for agentic workflows
Qwen 4 also performs much better on avoiding hallucination
If the difference between Qwen 4 Max and Qwen 4 flash is similar to 3.8 and 3.8 flash next (ie. about 10%), then 3.8 flash next may just barely beat Opus 5.4 at coding.
nelaggy 4 days ago |
m00dy 4 days ago |
WASDx 4 days ago |
jhonof 4 days ago |
xpct 4 days ago |
anana_ 4 days ago |
andy_ppp 4 days ago |
husseinhb 4 days ago |
drnick1 4 days ago |
libpcap 4 days ago |
chriddyp 4 days ago |
Luke2642 4 days ago |
I suppose it could make actually worse at taking instructions and redirection mid-task, but that could be easy to work around too. Just hold back a bit of the spec and drip feed it in later?
I don't mean cheat on it, I just mean optimize for it harder!
mlmonkey 4 days ago |
akulbe 4 days ago |
dbg31415 4 days ago |
thatxliner 4 days ago |
grendelt 4 days ago |
It'll gush about how to make moon cakes and where the tallest and longest bridges are and how to run qwen on a raspberry pi - but it can't seem to manage some basic Wikipedia questions.
syntaxing 4 days ago |
expectfun 4 days ago |
There's plenty of competitors in the league where Max is, but few companies can improve the 27B level capabilities.
HarHarVeryFunny 4 days ago |
undefined 4 days ago |
trilogic 4 days ago |
The rest looks awesome.
nxtfari 4 days ago |
Onavo 4 days ago |
xiaoyu2006 4 days ago |
nixosbestos 4 days ago |
EDIT: oh, I guess this isn't open weights, so presumably I'll have to get it through Qwen or whoever they might partner with (Bedrock?).
Ikersl 4 days ago |
peter_d_sherman 3 days ago |
undefined 3 days ago |
sodowork 4 days ago |
zhuanxyz 4 days ago |
senko 4 days ago |
mipo256 4 days ago |
ankur_misra 4 days ago |
cricketfanatic 4 days ago |
Vivek-KY 4 days ago |
jacksonastone 4 days ago |
small_model 4 days ago |
bbor 4 days ago |
For comparison, here's one I got from Qwen3.8 Max (at default reasoning effort): https://simonwillison.net/2026/Jul/20/afraid-of-chinese-mode...
This inspired me to re-run pelicans through 3.8 Max - but OpenRouter has qwen3.8-max-prime and qwen3.8-max-0902 - prime is twice the price and faster - so I went with the 0902 one.
Turns out Qwen4-Max has a none setting, but 3.8 Max didn't allow disabling reasoning - you had to pick from these five levels instead: minimal, low, medium, high, xhigh.
Here are those 3.8 Max 0902 pelicans: https://tools.simonwillison.net/markdown-svg-renderer?url=ht... - they're mostly not as good as 4, but I prefer the 3.8 xhigh one.