To be fair, the Aiel didn't initially grow up there (and too short for evolution, in my non-scientific brain), but rather migrated there to use it as "a shaping stone to make them, a testing ground to prove their worth, and a punishment for their sin." And it would be too easy on those three reasons if they were perfectly (or mostly) acclimated and suited for the desert genetically.
You're right, and it's only been a few thousand years. I'll still complain to anyone who will listen. If I don't complain how will you know I really enjoy the series?
In my homebrew world, elves have a different skin pigment that reflects UV light (whereas melanin absorbs it). Lower amounts of pigmentation mean darker skin, and more means whiter skin
Because of this, deep elves aren't as dark skinned as some humans, with the darkest being an ashy charcoal gray, but desert elves develop extremely white skin with almost no translucence