when I worked at walmart some people would get so offended if I suggested wearing the opposite genders clothes even if the item was basically the same for males and females.
a polo shirt for men is cheaper and you're probably just wearing it because of your workplace why does it matter?
I agree with the sentiment and even own some opposite-gender clothing that I love. However, some clothing is cut certain ways to flatter stereotypical shapes so you may want to be aware if a shirt if cut to accentuate breasts or cut for certain shape hips.
Just pay a bit more attention to the cut of the clothing and you'll be fine though; everybody is shaped a little different so it may work for you. My aunt wore mens jeans basically her whole life, she said they were cheaper and more durable.
"But dresses are naturally female!" (I've legit heard this exact statement multiple times.)
Naturally? They're naturally female huh? How? Did our ancestors the Neanderthals and Australopithecus africanus wear dresses? Are genetic female babies born in dresses? Do you see female non-human animals wearing them?