It stands to reason. The light at the beginning and end of the day has a lower color temperature, i.e. (confusingly) reddish rather than bluish. Blue midday-temperature light is therefore something that evolution has prepared us to see in the middle of the day rather than late at night.
"color temperature" makes more sense if you think of it as hot iron - red hot iron is cooler than white (blue) hot.
People usually think of red fire and blue ice though.