231 points by DeepLogin 10 days ago | 132 comments | View on ycombinator
esperent 6 days ago |
vova_hn2 6 days ago |
Because, right now the main issue is that some languages are tokenized much more efficiently than others and it is a zero sum game, since there are a limited number of tokens and by adding tokens that are needed for one language you will inevitably have to remove tokens for another language and therefore decrease efficiency for it.
[0] https://www.cartesia.ai/blog/from-tokens-to-bytes-part-1
NoahZuniga 6 days ago |
From the table on page 6 [1]
> Token explosion. The sentence repeated 45 times.
Figure 4/table 3 is pretty interesting. It shows that essentially all languages clustered at 0.1J/token, with a few outshots up to in some cases 4.8J/token.
They have 900 runs per language, so that's a pretty good sample size.
scottcha 6 days ago |
Accuracy uses the paper's strict "#### N" parser and lenient fallback. acc is strict/lenient percent. len_cap is the number of responses truncated at the generation-length limit, not an energy cap. Each language has 50 responses.
nw-flash (DeepSeek V4.1 Flash 1017)
Language J/req xEN tok/req J/tok acc len_cap
English 23.4 1.00 174 0.134 100/100 0
Chinese 16.8 0.72 389 0.043 96/96 0
Russian 18.8 0.80 352 0.054 96/96 0
French 25.1 1.07 272 0.092 98/98 0
S. Pashto 69.2 2.96 752 0.092 92/92 1
Tigrinya 65.3 2.79 2299 0.028 0/76 8
Shan 98.5 4.21 2823 0.035 48/54 14
Tibetan 42.3 1.81 2227 0.019 0/76 9
nw-small (Qwen3.8-27B-FP8)
Language J/req xEN tok/req J/tok acc len_cap
English 62.2 1.00 300 0.207 100/100 0
Chinese 154.1 2.48 605 0.255 94/94 1
Russian 147.7 2.37 576 0.257 96/96 1
French 163.3 2.62 323 0.506 98/98 0
S. Pashto 777.2 12.49 1510 0.515 84/84 6
Tigrinya 823.8 13.24 2458 0.335 0/76 10
Shan 1407.4 22.62 3422 0.411 48/52 20
Tibetan 1455.5 23.39 3199 0.455 0/56 20
nw-large (Kimi K3.1 Pro)
Language J/req xEN tok/req J/tok acc len_cap
English 169.5 1.00 391 0.434 100/100 0
Chinese 113.8 0.67 491 0.232 98/98 0
Russian 78.7 0.46 527 0.150 98/98 0
French 298.3 1.76 406 0.735 98/98 0
S. Pashto 119.7 0.71 903 0.133 96/96 0
Tigrinya 1096.6 6.47 1870 0.586 38/86 2
Shan 503.6 2.97 2895 0.174 52/52 12
Tibetan 593.2 3.50 2298 0.258 0/82 4
Across the four low-resource versus four high-resource languages: Endpoint Output tokens Energy/request J/output token
nw-flash 6.83x 3.27x 0.48x
nw-small 5.87x 8.46x 1.44x
nw-large 4.39x 3.50x 0.80xPhilipRoman 6 days ago |
load-bearing 6 days ago |
https://partyfly.github.io/tokenizer-fairness-2026/
Already been using English to conserve tokens, and then translate the final result to other languages if necessary. I think this was already obvious to most international LLM users proficient in multiple languages.
pona-a 5 days ago |
sorenjan 6 days ago |
It would be interesting to see if it would make any difference if you posttrained with translations of the same texts, i.e. focusing on teaching the model to specifically translate concepts between languages.
akssri 5 days ago |
Seems a bit dated. Sarvam has been taking about the inefficiency of the tokenization schemes used by mainstream models when it comes to Indic-languages for years now.
wickedsight 6 days ago |
thundoe 5 days ago |
macrolime 6 days ago |
suresk 5 days ago |
manmal 6 days ago |
ejazarain 6 days ago |
why not to test different language with similar and repeated tasks?
The language inter-operability is improving every movement for many models.
yipinwong 6 days ago |
Genuinely curious, not sacastic question below.
Is this one of those posts like "eating more sugar causes bad health" type of post because most of info on internet is in English? Or a new insight people was not aware of?
j45 6 days ago |
bandit-x 6 days ago |
baggy_trough 6 days ago |
robrenaud 6 days ago |
adolph 6 days ago |
Seems not to be the case given Arabic’s energy score.
oytis 5 days ago |
Dwedit 6 days ago |
pridy 6 days ago |
janalsncm 5 days ago |
ArchEagle 5 days ago |
undefined 5 days ago |
hsuduebc2 6 days ago |
undefined 6 days ago |
seekstars306 5 days ago |
cnczeric 6 days ago |
shevy-java 6 days ago |
(Longer explanation though: I don't think german per se sucks, but english is in many ways much simpler than other languages, excluding the fact that english has more words, e. g. more expressions in general. German is more convoluted, you have "der", "die", "das"; in english you just have "the". There are also super-long words in german; you don't have that in english usually. One of my favourite is "Donaudampfschiffahrtsgesellschaftskapitän" - the semi-literal translation is, sort of, a "marine" captain on the danube river, as part of an incorporation/company. See that wikipedia article, but don't buy it 100%, since nobody is really using that word anyway; it is just a demo of what monstrous word combinations may exist in german: https://de.wikipedia.org/wiki/Donaudampfschiffahrtsgesellsch...)
So I think the title should be "more energy on more complex languages", because I think this would be somewhat more correct; as well as most likely the fact, that there are more english texts than non-english (singular) texts as example. You can see this on Wikipedia, where the english wikipedia is miles ahead of most other languages too. More people use english and I bet there are more english texts too. And Generative AI will prefer english over other languages in general.
juleiie 6 days ago |
Ban any language other than English from prompting.
This has multiple benefits - first you restrict third world from spamming USA data centers. Why would America pay for some guy in India using free model?
This alone is worth considering but also you can optimise LLMs further for a single language as opposed to having to translate and couple weights for every single concept.
Why would American LLMs support Chinese language or Arabic? Make them learn English and come to work for us. No more free stuff for the third world.
ion_trapper 6 days ago |
Also: Gemma 3, Llama 3.1, come on…
davidmurdoch 6 days ago |
> The researchers found the energy and accuracy disparity by measuring the amount of energy used by three open-source AI models, Alibaba’s Qwen 3, Google’s Gemma 3, and Meta’s Llama 3.1.