How well can LLMs solve chess puzzles?
How well can LLMs solve chess puzzles?
github.com GitHub - kagisearch/llm-chess-puzzles: Benchmark LLM reasoning capability by solving chess puzzles.
Benchmark LLM reasoning capability by solving chess puzzles. - kagisearch/llm-chess-puzzles
![GitHub - kagisearch/llm-chess-puzzles: Benchmark LLM reasoning capability by solving chess puzzles.](https://lemmy.world/pictrs/image/3305c223-0ec8-4d17-8bee-515d83e4032d.png?format=webp&thumbnail=256)
Each LLM is given the same 1000 chess puzzles to solve. See puzzles.csv
. Benchmarked on Mar 25, 2024.
Model | Solved | Solved % | Illegal Moves | Illegal Moves % | Adjusted Elo |
---|---|---|---|---|---|
gpt-4-turbo-preview | 229 | 22.9% | 163 | 16.3% | 1144 |
gpt-4 | 195 | 19.5% | 183 | 18.3% | 1047 |
claude-3-opus-20240229 | 72 | 7.2% | 464 | 46.4% | 521 |
claude-3-haiku-20240307 | 38 | 3.8% | 590 | 59.0% | 363 |
claude-3-sonnet-20240229 | 23 | 2.3% | 663 | 66.3% | 286 |
gpt-3.5-turbo | 23 | 2.3% | 683 | 68.3% | 269 |
claude-instant-1.2 | 10 | 1.0% | 707 | 66.3% | 245 |
mistral-large-latest | 4 | 0.4% | 813 | 81.3% | 149 |
mixtral-8x7b | 9 | 0.9% | 832 | 83.2% | 136 |
gemini-1.5-pro-latest* | FAIL | - | - | - | - |
Published by the CEO of Kagi!
You're viewing a single thread.
All comments
21
comments
Of course they are bad at solving problems. The I in LLM stands for intelligence.
(Credit: https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-for-intelligence/)
34 0 Reply
21
comments
Scroll to top