[Legal question] It is allowed to use actual video games for training AI that can play them?

Not a lawyer

Should be fair use under most circumstances.

However, with online multiplayer, you'd be at the mercy of whoever's running the server, and usually they'd hate bots in any way and form.

Only if you're big enough to pay your way out of it

Reinforcement learning research has been using Atari games as standard benchmarks for over a decade now and no one has faced legal issues yet.

(definitely not a lawyer) If you play multiplayer games (especially ranked stuff) you might be going against some "no botting" rule of that games tos.

One clear and dry example of this that comes to mind is that Code Bullet got banned from JStris(or some other online tetris server?) because he was botting and not in their dedicated bot queue
- it's jstris, and i really want to watch his bot get absolutely destroyed by one of the top players like firestorm or czsmall
  
  or another bot! tetris has had a pretty big following for the development of bots, and it's basically min-maxed at this point. i want to watch CB's bot get destroyed by cold clear running at 1.5pps.

(not a lawyer). If you bought the game copies that the AIs are playing, then it seems like you're not making a copy of the game just to have the AI play it.

That kind of assumes that your AI is playing the game through a mechanism like AutoHotKey, generating keyboard or controller inputs that pass through the operating system to the game.

If your AI hooks into or modifies the game code to "play", then it could run afoul of anti-reverse engineering clauses that are common in the click through license agreements. Those clauses may not be enforceable in your jurisdiction. Legal results on anti-reverse engineering clauses are kind of mixed in the United States.

Edit: for reference, there was a software called "Glider" that played World of Warcraft for you, so you don't have to grind to level up. Blizzard absolutely hated the makers of Glider, but it stuck around for a long time, before it was ultimately sued into oblivion.