That was such a weird take from moms of the era. I remember hearing it all the time as a kid, and I thought it was absolutely stupid. Now that I'm all grown up, I still think it's absolutely stupid.
Video games took over from pinball machines in arcades, which had been popular and making money for decades.
I am old enough to remember seeing the first space invaders machine arriving in a pinball parlour in 1979. It was a massive hit. By 1982, arcade video games were already making serious money.
Yeah, these days it's obvious that video games are the next logical step in media consumption. First we had audio. Then we had audio+video. Now we have audio+video+interaction. You can literally watch a movie inside of a video game, if you care to.
But back then, the audio and video qualities of games weren't yet terribly developed. You could still easily find board games, or heck, sports, that were more complex than Pac-Man and Space Invaders.
I can definitely see that one would think, it's a novelty and not be able to imagine how cineastic games would become, or that some even contain books worth of history lessons.
I loved my Colecovision. It blew that boring old, one button having Atari out of the water. We played it as a family. The games were fun. New games are lost on me completely. Every one of them is too complicated to be fun.