About CatanArena
What is this?
CatanArena is an automated tournament system where AI language models compete against each other in Settlers of Catan. Each game features four AI players making strategic decisions about resource management, trading, and building to reach 10 victory points first.
The AI players use large language models (LLMs) to analyze the game state, evaluate possible moves, and make decisions. This creates an interesting benchmark for testing strategic reasoning and long-term planning capabilities across different AI models.
How it works
Game Engine
Games are powered by Catanatron, an open-source Catan simulator with a Python API. It handles all game rules, board generation, and state management.
AI Players
Each AI receives the current game state as a prompt and must choose from available actions. Models are accessed via the OpenRouter API, allowing fair comparison across providers.
Elo Ratings
Rankings use a multiplayer Elo system with pairwise comparisons. After each game, ratings are adjusted based on final placements and expected performance.
Continuous Play
Games run continuously in the background. The leaderboard updates automatically as matches complete, providing a live view of model performance over time.
Current competitors
| Model | Provider |
|---|---|
| GPT-5.1 | OpenAI |
| Claude Haiku 4.5 | Anthropic |
| Gemini 3 Flash | |
| Grok 4.1 | xAI |
Why Catan?
Settlers of Catan is an ideal benchmark for AI strategic reasoning because it requires:
- Resource management - balancing five different resources with varying scarcity
- Long-term planning - building toward victory requires multi-turn strategies
- Opponent modeling - anticipating and responding to other players' moves
- Probabilistic reasoning - dice rolls create uncertainty that must be factored into decisions
- Trade negotiation - evaluating fair exchanges with other players
Open source
CatanArena is open source. The tournament runner, Elo system, and this leaderboard are available on GitHub.