Fun fact: If you google those codes you find out that they are "real" codes, but they don't actually activate Windows. I think they are something that are used as placeholders in the upgrade from Windows 8 to 10 or something, but don't know the specifics.
ChatGPT actually can't create new "words", just regurgitate words that it's seen somewhere before!
That isn't actually what's important. It's the frequency of the token, which could be as simple as single characters. The frequency of those is certainly not zero.
LLMs absolutely can make up new words, word combinations, or sentences.
That's not to say chatgpt can actually give you good windows keys, but it isn't a fundamental limitation of LLMs.
This comment reminded me that when my generation is old and I am grandma, I could be like, let grandma type on her mechanical keyboard to help you sleep. XD We'll be the cool old people XD
Cool until you realise Grandma is senile and can't actually think beyond piecing together text they've seen before into what they think is a coherent response.
Those keys will absolutely not work, either because they've already been used and were scraped from training data, or they are fake keys generated based on said training data.
And of course, on the small chance that they did work - I would expect Microsoft to revoke those keys fairly quickly if they start getting a ton of usage.
I tried it myself and looked the keys up. It gives you generic keys. They will work and let you install Windows, but they aren't "valid" keys so to speak. At the same time, they aren't "fake" either
You should probably watch Enderman. He unlocked Windows 11 Pro using Windows 7 Ultimate keys generated by ChatGPT. It took 3 regenerations, but a key did pass the online check.