Love wearing a bra. The only time I don't is whilst in water. The pain of not wearing one is ridiculous. That being said, some bras are just awful and uncomfortable and you need the right one, or two, for the right occasion.
I hate wearing pants though and will happily wear nothing but a bra in my own house.
I can't relate to how men's underwear feels, but bras are super uncomfortable. I only wear sports bras and I still hate them. It's harder to breathe- there's a constant pressure around the rib cage.
I only wear them when I have to, and even then sometimes I'm like: "fuck it, I'll deal with the looks" (mostly from other women who are with men- and my tits are small!)
In my experience if I get one that fits right they're not that bad. Some styles are just uncomfortable regardless, but bodies can be shaped in a lot of different ways, so of course preferences will vary just as much.