ChatGPT 'got absolutely wrecked' by Atari 2600 in beginner's chess match — OpenAI's newest model bamboozled by 1970s logic

2025 Mazda MX-5 Miata 'got absolutely wrecked' by Inflatable Boat in beginner's boat racing match — Mazda's newest model bamboozled by 1930s technology.

this is because an LLM is not made for playing chess

If you don't play chess, the Atari is probably going to beat you as well.

LLMs are only good at things to the extent that they have been well-trained in the relevant areas. Not just learning to predict text string sequences, but reinforcement learning after that, where a human or some other agent says "this answer is better than that one" enough times in enough of the right contexts. It mimics the way humans learn, which is through repeated and diverse exposure.

If they set up a system to train it against some chess program, or (much simpler) simply gave it a tool call, it would do much better. Tool calling already exists and would be by far the easiest way.

It could also be instructed to write a chess solver program and then run it, at which point it would be on par with the Atari, but it wouldn't compete well with a serious chess solver.

An LLM is a poor computational/predictive paradigm for playing chess.

Sometimes it seems like most of these AI articles are written by AIs with bad prompts.

Human journalists would hopefully do a little research. A quick search would reveal that researches have been publishing about this for over a year so there's no need to sensationalize it. Perhaps the human journalist could have spent a little time talking about why LLMs are bad at chess and how researchers are approaching the problem.

LLMs on the other hand, are very good at producing clickbait articles with low information content.

I swear every single article critical of current LLMs is like, "The square got BLASTED by the triangle shape when it completely FAILED to go through the triangle shaped hole."

Ah, you used logic. That's the issue. They don't do that.

ChatGPT has been, hands down, the worst AI coding assistant I've ever used.

It regularly suggests code that doesn't compile or isn't even for the language.

It generally suggests AC of code that is just a copy of the lines I just wrote.

Sometimes it likes to suggest setting the same property like 5 times.

It is absolute garbage and I do not recommend it to anyone.

Tbf, the article should probably mention the fact that machine learning programs designed to play chess blow everything else out of the water.

Using an LLM as a chess engine is like using a power tool as a table leg. Pretty funny honestly, but it's obviously not going to be good at it, at least not without scaffolding.

LLM are not built for logic.

Hardly surprising. Llms aren't -thinking- they're just shitting out the next token for any given input of tokens.

Can i fistfight ChatGPT next? I bet I could kick its ass, too :p

I suppose it's an interesting experiment, but it's not that surprising that a word prediction machine can't play chess.

Can ChatGPT actually play chess now? Last I checked, it couldn't remember more than 5 moves of history so it wouldn't be able to see the true board state and would make illegal moves, take it's own pieces, materialize pieces out of thin air, etc.

A strange game. How about a nice game of Global Thermonuclear War?

The Atari chess program can play chess better than the Boeing 747 too. And better than the North Pole. Amazing!

It's not that hard to beat dumb 6 year old who's only purpose is mine your privacy to sell you ads or product place some shit for you in future.

I mean, that 2600 Chess was built from the ground up to play a good game of chess with variable difficulty levels. I bet there's days or games when Fischer couldn't have beaten it. Just because a thing is old and less capable than the modern world does not mean it's bad.

You say you produce good oranges but my machine for testing apples gave your oranges a very low score.

This made my day

Is anyone actually surprised at that?

So, it fares as well as the average schmuck, proving it is human

/s

Okay, but could ChatGPT be used to vibe code a chess program that beats the Atari 2600?

They used ChatGPT 4o, instead of using o1 or o3.

Obviously it was going to fail.

Next, pit ChatGPT against 1K ZX Chess in a ZX81.

Isn't the Atari just a game console, not a chess engine?

Like, Wikipedia doesn't mention anything about the Atari 2600 having a built-in chess engine.

If they were willing to run a chess game on the Atari 2600, why did they not apply the same to ChatGPT? There are custom GPTs which claim to use a stockfish API or play at a similar level.

Like this, it's just unfair. Both platforms are not designed to deal with the task by themselves, but one of them is given the necessary tooling, the other one isn't. No matter what you think of ChatGPT, that's not a fair comparison.

Edit: Given the existing replies and downvotes, I think this comment is being misunderstood. I would like to try clarifying again what I meant here.

First of all, I'd like to ask if this article is satire. That's the only way I can understand the replies I've gotten that critized me on grounds of the marketing aspect of LLMs (when the article never brings up that topic itself, nor did I). Like, if this article is just some tongue in cheek type thing about holding LLMs to the standards they're advertised at, I can understand both the article and the replies I've gotten. But the article never suggests so itself. So my assumption when writing my comment was that this is not the case and it is serious.

The Atari is hardware. It can't play chess on its own. To be able to, you need a game for it which is inserted. Then the Atari can interface with the cartridge and play the game.

ChatGPT is an LLM. Guess what, it also can't play chess on its own. It also needs to interface with a third party tool that enables it to play chess.

Neither the Atari nor ChatGPT can directly, on their own, play chess. This was my core point.

I merely pointed out that it's unfair that one party in this comparison is given the tool it needs (the cartridge), but the other party isn't. Unless this is satire, I don't see how marketing plays a role here at all.