I used to use "man" in a gender neutral way a lot. Like "hey man" or "what's up man". Probably a '90s kid thing. But here's the thing, it's not about how I feel about the word. It's about how the person getting called that word feels about it. So I've made a conscious effort to stop using it. It's really not difficult to not be a selfish asshole.
These days, the only person I still say "hey man" to is my weed guy.