Atheists might be annoying sometimes but they won't try to use their atheism to control your life. Not the same thing at all.
Someone might consider they're minding their own business, but they're actually harming/indoctrinating someone. That's why it's not so simple, and there is going to be conflict between people with different moral beliefs on how we should treat others.
And some religious people might think they are harming nobody with their beliefs but them arguing with faith-based arguments for harmless stuff and counting as members of a church allows others to argue with faith-based arguments for more harmful stuff and to claim that they have x church members behind them.
As long as religion is influencing laws, religious people are not minding their own business, because they are funding and backing this influence. The goal of religion (all religion) is gaining influence over everyone and everything, the core of religion is the opposite of minding your own business.
