I believe the original purpose of that phrase, was to differentiate between people who act on those attractions, and those who understand it's wrong and refrain from doing so.
The issue is using the same kind of language as like "Person of Color", as if pedophilia is an identity rather than an illness.
If we truly want a neutral clinical name, we'd say something like "Minor Attraction Disorder". It's something that's only permissible if you are actively seeking help and self control like any other disorder that's prone to harm.
You're conflating pedophilia with MAP, which is the very thing they're trying to combat.
The difference between MAD and MAP would be if you're referring to the person or the condition, obviously. And I'm sure you can see why nobody wants to be referred to as a MAD person.
That's the thing, you shouldn't be identifying as a person attracted to minors. You don't say "I'm a Person of Depression" you say "I have depression".
If you're serious about outlining the minor attraction as a problem you have to struggle with, don't declare it like an identity to be proud of.
And if you don't see it as a problem, you're just enabling pedophilia. The people inflicted with it need to admit they have an illness to get treated.