RuneBench is a benchmark suite that evaluates the long-term planning and multi-step problem-solving abilities of AI coding agents through playing an MMORPG. Agents operate a game server via a TypeScript SDK, building strategies while searching through walkthrough documentation extracted from the in-game Wiki. What sets it apart is how it scores: rather than total gains over a fixed period, it uses the “maximum XP rate over any 15-second window.” This makes it hard to score well by endlessly repeating simple tasks, and encourages agents to switch strategies as the situation demands. The README states that it is released under the MIT license.
Key Features
- Scored by peak 15-second rate: The benchmark originally measured total gains within a time limit, but that gave too much of an advantage to simple, continuous grinding, so it was changed to the current method. The aim is to encourage strategic shifts that come with exploration and leveling up
- Agents write TypeScript code to play: Rather than merely calling tools, agents write and execute their own TypeScript code using the SDK to read and write game state
- In-game Wiki provided as strategy material: Information on skills, shops, NPCs, and items is supplied in Markdown, and agents play while searching through this walkthrough content
- Multiple task configurations: XP tasks covering a wide range of skills and gold-earning tasks with varying starting conditions, each offered in variations of different durations
Pricing
Free. It is released as open source, and the leaderboard is freely viewable. The README explicitly states the license as MIT, but there is no LICENSE file at the repository root, so GitHub’s license detection shows nothing (as of August 2026). If you need to confirm the terms of use precisely, contacting the author is the safest route. Note that running it yourself requires setting up a Docker environment, plus the API usage fees for whichever model you use. In the published run data, some cases exceed $44 per run, and the cost varies widely depending on the model and task length. That spread itself is one of the things worth examining when making comparisons.
Pros and Cons
✅ Pros
- Per-run execution cost sits right next to the score, so you can verify in dollar terms “how many times more a top model costs, and how many points better it is”
- The agents’ actual action logs are published, so you can read not just the score but “where it got stuck”
- Because the environment is sealed inside Docker, you can drop your own agent into the same conditions and compare it against existing models
⚠️ Cons
- The price of a scoring design that encourages exploration is that each run is long and expensive, making it hard to accumulate trials. The developer themselves cites noise and false negatives from small sample sizes as a limitation
- Since scores depend on game-specific context, you cannot read the rankings directly as “rankings of coding ability”
Comparison with Similar Services
| Criteria | RuneBench | Factorio Learning Environment | BALROG |
|---|---|---|---|
| Target environment | MMORPG (open-source RuneScape-style implementation) | Factory automation game | Multiple games including NetHack |
| Evaluation axis | Maximum rate over 15 seconds (evaluates strategy shifts) | Production line completion rate and output optimization | Per-episode progress and success rate |
| Character | Appears to be solo-developed OSS. Assumes long autonomous play sessions | Continuously running environment | Academic multi-environment benchmark |
There are several efforts to evaluate agents in game environments, but RuneBench stands out for incorporating multiplayer elements and real Wiki search.
Who Is This For
- Researchers and developers who want to evaluate the long-term planning ability of coding agents from an angle different from static benchmarks
- Teams that want to compare multiple models with execution cost factored in
- Anyone who wants to build an evaluation environment for their own agent using the SDK and benchmark execution framework
Summary
A benchmark that redesigned its scoring method to measure not “is it good at the game” but “can it look ahead and reorganize its steps.” Since the developer openly acknowledges the constraint on trial counts, it is better suited to studying the spread of cost and score and reading “how much of a gap opens up within the same budget range” than to checking who holds first place.