Anytime I mention it to people IRL they look uncomfortable before I explain it's the federated universe of distributed social platforms.
That moment of 😬 is due to the name Fediverse sounding like 'Fetiverse.' These people aren't sex obsessed. Most are tech professionals who were on Fark, SA, Digg...etc.
I guess OP means it sounds like "fetish", but this looks like a retread of the "what if people think the fediverse logo looks like a pentagram?" hypothetical that didn't have any root in reality either.
In that case, sorry to break it to you: It's hard to swallow (no pun intended), but sometimes you just have to accept that people of your immediate acquaintance or relation are utterly unkinky.