That is usually how males and females of a species are differentiated in general: males have the small gamete and females have the large one. (As you said, some individuals may not produce gametes so it only applies in general).
Of course humans are a lot more complicated. We have a concept of gender which doesn't necessarily align with biological sex, and many people modify their sex characteristics to match their gender, so applying generalizations blindly gets you nowhere.
Indeed, just as gender is a spectrum so is sex. I love when someone says "Its basic biology" because the best response is "and this is intermediate biology".