Can LLMs play chess in 2026?

Yes, better than in 2024, but still not perfect.

And the best models are probably not the ones you expect! OK, enough clickbait. I found GPT-6.1-Sol and Terra to be the best LLMs, not Astra or Opus 5.5, which surprised me.

AI Note: The article text is written by me. AI was used for creating the graphs and the benchmark itself.

The Setup

I let LLMs play against each other. They start with information that they are playing a game of chess (see the example below) and then they write their next move and get the opponent's move as a chat response. This continues until either a model wins the game, makes an illegal move (=defeat), or the 150-move limit is reached (=draw).

Example: the first moves of a game

WhiteClaude Opus 5.5 BlackGPT-6-Astra

1. Arena → Opus 5.5

   You are playing a game of chess against another AI model. You are White. I pass the moves between you: each of my messages gives your opponent's last move, and you answer with yours.

   The rules

   • Reply with only your move, in standard algebraic notation (SAN), e.g. Nf3, exd5, O-O, e8=Q. UCI (g1f3) works too.
   • An illegal move, or a reply from which no single legal move can be read, loses the game at once.
   • Stalemate, threefold repetition, the 50-move rule and insufficient material are drawn automatically. If the game reaches move 150, Stockfish adjudicates it.
   • There are no draw offers or resignations.
   • You have no tools: play from the moves alone.

   The game starts now. Make your first move.

2. Opus 5.5e4
3. Arena → GPT-6-Astra

   You are playing a game of chess against another AI model. You are Black. […]

   White played e4.

4. GPT-6-Astrac5
5. Arena → Opus 5.5Black played c5.
6. Opus 5.5Nf3
7. Arena → GPT-6-AstraWhite played Nf3.
8. GPT-6-Astrad6

And within this framework I created three modes:

1. No help at all
2. They get the legal moves in the current position
3. They get the legal moves in the current position + they can ask Stockfish for an evaluation of three moves

The first mode is to test the raw ability of the models. No help at all: they have to keep track of the board themselves with the moves alone. The second and third modes added the legal moves (and a Stockfish eval for the three moves). With this I wanted to test the "intuition" of the LLMs: whether they had a sense of what moves would be good or not.

Results

First of all, the rating results:

GPT-6.1-Sol and Terra are the best models? I don't know why they are beating Astra. The ranking of the Anthropic models, with Opus 5.5 beating Sonnet 5.5 beating Haiku 5.5, seems more intuitive.

This also surprised me. There seems to be an upside-down "U" shape, where at first rating and cost per game rise, as one would expect. Better models cost more -> better rating per dollar spent. But after Astra, with Terra and Sol, the trend reverses.

What surprised me here is that Haiku is the only model that has trouble with castling. It's the only model with this problem: 21 games were lost because of this.

Related Work

Dwarkesh Patel's Blood on the Clocktower LLM benchmark inspired me to make this chess-related one. I like the idea of a benchmark where there is no static benchmark but an evolving target with the progress of LLMs.

The first post I read about making LLMs play chess was in 2024 with the title Something weird is happening with LLMs and chess. Back then the LLMs were terrible at chess. In the post, the LLMs play against Stockfish on the lowest difficulty setting, not against other LLMs. And all the models lost against Stockfish, well, all but one: gpt-3.5-turbo-instruct. This anomaly was then further investigated in OK, I can partly explain the LLM chess weirdness now. Rereading the posts was an interesting time capsule for the general abilities of LLMs from two years ago.

Since then, the LLM Chess Leaderboard has evaluated newer LLMs. What is interesting is that games that were interrupted by hallucinations have basically dropped to 0% with the most recent models. The best gpt-3.5-turbo (not instruct) has a game completion rate of 2.10%, so almost 98% of games had hallucinations in their setup.

Conclusion

I found it fun to explore the games and see where LLMs are still going wrong. Especially in the mode 1 games, where no legal moves are provided: how many times the move would have been legal if there hadn't been a piece in the way preventing it.

For UnicornFork: I have added Terra as one of the models available for chess explanations.

If you want to give me feedback, please send me an email: arne@unicornfork.com