I don't know where else to write this, but I want to throw the idea out there. I have long wanted to take old broodwar televised matches, many of which are terrible quality 240p, and use machine learning to convert them to into perfect Broodwar Remastered frames. This seems tractable to me because you should be able to map the terrain sets to their remastered equivalents, and the game is just a series of sprites rendered at specific frames. Even if the source quality is terrible, I imagine this is able to be extracted at high quality since you can, eg, set up an automated pipeline which generates training data. Maps from original graphics to remastered, and then again for 240p -> tilemap positions for frame camera center + sprite positions / animation index.

Back in 2010, during the early days of bwapi, there was a Brood War AI tournament held by the Expressive Intelligence Studio at UC Santa Cruz. It's interesting to see how different the approaches were back then, vs this or Deepmind's SC2 work.

https://web.archive.org/web/20091124210529/http://eis.ucsc.e...

There's a great contemporary Ars Technica piece by a competitor:

https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...

As an undergrad I did a project using genetic programming. It was not very successful, but it was a lot of fun.

https://tomisin.space/archive/starcraft-genetic-programming/

Unrelated to the benchmark...

I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...

I miss those days so much.

Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.

I love this. Funnily enough StarCraft has influenced how I approach AI at a meta level

Protoss: powerful and expensive frontier coding agents you directly micromanage for the toughest tasks

Terran: versatile team comps of dedicated agent roles you can delegate well-defined tasks to

Zerg: massive swarms of specialist custom agents inside your apps that you evolve and optimise for speed and cost

Knowing every faction has its strengths and weaknesses helps me decide which tools to use for the job.

There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.

I thought that it was videos of the replays on the site. Was very impressed that it was a replay playing that you can scroll around in and select units.

A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on

[1] https://rolandgao.com/blog/gobench/

[2] https://rolandgao.com/gobench.pdf

This thing kept me sane through university. I had a shitty computer which could barely run this and no Internet so I just played against the computer, which was both frustrating and educational.

I will always love this and now I'm going to play it again. Remastered and on a fancy modern machine.

Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?

It sounds like it might have been actually played in real time, which would be very important to distinguish.

I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE

This is a very interesting benchmark, and I think it has a lot of potential to make the speed of a model quantifiable.

I'm often asking myself is it better to use higher or lower effort levels, or to maybe drop down to a "dumber" but faster model. And so using a real-time based competition as a benchmark could shed some light on this, I think.

In this vein, here are what I would love to see added in this benchmark:

- Include Google's Gemini models. I keep hearing Gemini being praised for its speed, and I would like to see whether that gives it a big enough edge over the bigger but slower models. - How does a Cerebras-accelerated open source model fare against a much larger but much slower frontier model?

I also feel like in general there is a lot of very low-hanging fruit to start benchmarking models across the spectrum of real-time vs batch-style workloads. Perhaps Brood War sits somewhere quite near the "real-time" end of the spectrum, but what about something like a game of speed chess, or a turn-based game with time limits?

I think what I would like to see the most is for someone to come up with a benchmark that supports tuning the "real-timeliness" of the benchmark, and then running a sweep of a model across the whole spectrum. That could get result in real nice graphs with multiple models on the pareto-frontier, varying based on the hosting provider and the model dimensions.

Check out pluto, RL trained sc bw bot. Somehow it is now going rogue on the Korean ladder and flattening pros. There are some yt games by it, search for ^333^

It was the late 90s, my very first day at a new school, I was asked to introduce myself at the front of the class, I mentioned I like computers, one guy at the back of the class blurts out “En Taro Adun” and without skipping a beat I replied “J’tokoh zohl”, and we instantly became best friends.

If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net/

Astra had an explicit medium/xhigh levels, Fable - just Fable. What reasoning level was used? Why not multiple were tested?

I’ve scrolled the article, but haven’t noticed any remarks about Fable’s levels.

No model will ever figure out how to make a ling tight wall.

Your minds will be blown when you realize just how much StarCraft is ingrained into South Korean culture. They literally had (or have) dedicated TV channels just for StarCraft.

Brood War was the first video game to be broadcast on TV in Korea. I'm pretty sure it's still going.

At last something that feels properly orthogonal to pelicans on bicycles.

“Older models tended to play the RTS as a turn-based game, leading them to get destroyed while they were thinking. ”

That’s me. I’ve always struggled with real-time games because I need to pause and think. While I excel at chess and board games, I’m just no good at real-time ones. At last I can only manage by sticking to a fixed set of tactics for a game, which minimizes the need for on-the-fly thinking. Seeing current models face the same difficulty leaves me with mixed feelings.

This is a great idea for a benchmark. Something all the benchmarks seem to be missing is strategy, tactical solutions in most of the benchmarks are all thats required but here requires actual long term thinking and tactical thinking, balancing and orchestration.

I'm trying to play but it says "another match is already active". Can only a single player play on your system at the same time?

EDIT: Nevermind, seems to work now. Watching a local qwen3.8-flash-next play this.

Interesting that it benches down to Haiku but doesn't bench any Chinese models (which are at least between Sonnet and Opus, when they aren't beyond Opus).

Openai 5 dots 2, and alpha Star, AI research used to be very fun

Would Jev be good for this?

+1 because StarCraft

I think we're on the early days of games you connect with your agent to. Human + AI units one versus the other. Like knights with their horses. Not sure which is the horse..

Any details about the harness the agents were given? I am curious what representation of the screen and world state was provided to the agents and what tools they had available.

Would be interesting if you could team a fast and slow agent together -- slow model can either act directly or maybe just communicate to the fast model.

+ Playable Agent driven Starcraft

I like this as a concept - taking a very real world task and checking whether it works.

Surprised the outcomes are so poor though. I recall years ago AI was capable of beating pro level DOTA teams.

I guess in one case it was specifically trained on the interface & game while here it was not?

Make the OpenAI models develop AI-scripts to play the game instead (like the good old AI-scripts, not AI as in LLMs). They are amazing at that.

Fable's effort level is a significant omission, given it was ranked 3rd behind Astra xhigh and Astra medium.

A ton of conversations about the game must be in the training set. I wonder, is there any way just from watching how they play, of telling if they tend to pick strategies that people complain or meme about online?

Why was Luna Low so good?!? Better than Terra XHigh. Can't even say it's all APM because Sol Medium beat Sol Low (both beat Sol XHigh).

Wondering what it would look like if you allowed them to write scripts. You could throttle the number of clicks to make it interesting.

Good example of how these things sometimes spend way too much time thinking to be useful

undefined

working on a similar project for street fighter currently https://www.youtube.com/watch?v=dJYTV3ZiT6o

I would love to create one of these benchmarks for age of empires 2, but I have no idea how to make the AIs play it. Maybe I can get Claude to do it anyway.

typesafe.ai would eat them all for lunch.

Game performance is one of those topics that makes it so abundantly clear how limited these systems still are. StarCraft is predominantly a mechanical game so the horizon of what you need to do is quite short and tactile, and even then without advantages no system has come close to beating a human.

I saw someone recently try to get an agentic system to play Final Fantasy and it did about as well as a Roomba.

undefined

Gemini wasn’t included, but I’m guessing its performance would have been similar to Grok’s performance despite having roots in DeepMind.

*wonders if running startcraft in wasmjs uses less CPU than anubis

"another match is already active"

Really looking forward to playing- are you all limiting boxes?

Hmmmmm, almost like these models aren't generally intelligent

I don't normally anthropomorphize the AIs but this is super cute, lol.

Somewhat related: In 2018, Google DeepMind had already created AIs that were capable of beating professional gamers in StarCraft 2 (the sequel to Brood War): https://www.youtube.com/watch?v=cUTMhmVh1qs