Yes, but you can boost your own comments to counteract the downvotes. The whole system is sketchy at best and shouldn't be relied on to determine anything about a user.
If you click on "more" you can also see everyone who up/downvoted and boosted a comment/post. So seeing that you upvoted/boosted yourself might be frowned upon by some. But who cares
The upvote button on kbin was originally mapped to boosts. People didn't care for they, and it was incongruent with how Lemmy does it. So, they changed the functionality, but didn't get around to updating the rest of the reputation system.
That said, kbin doesn't federated down votes, so the system is going to be weird no matter what.
I'm not sure if Lemmy federates downvotes. ernest has said that kbin doesn't, though. Maybe that's changed since he made that statement.
AFAIK, though, there's no need for changes to ActivityPub. It has everything it needs to propagate 'actions', which can be defined however a platform chooses. Other platforms just need to know how to interpret those actions in order to do anything with them, and I don't see sites that aren't aping Reddit's engagement model to care about downvotes in any way.
I hope the boost and upvote list don't get messed up. I've been using boost for things I want to comeback later on top of things I want more visibility on. I'd hate to lose that and have to manually sort them.