Synesthesia. I was about 20 before I learned it has a name and not everyone has strong colour associations for numbers and letters, or sees a visible map of time in their head, or has music take shape. It never occurred to me to question it because it's always been my norm.
I don't think the argument about color perception is about the actual color sensing organs (eyes). we can track where that differs i.e. colorblindness
I think it has to do with the way the brain perceives the input from the eyes. so I may see the world inverted from how you see it and I think that would remain true if we somehow swapped eyes successfully.