The post and comment votes do federate, but it's not particularly reliable. No clue why though. That's a bit outside my wheelhouse and there is nothing in the documentation about post voting.
Another user explained why I likely saw the discrepancy. Though, aside from that, federation generally is very much wonky and unreliable unfortunately.
If the community's server does not track downvotes, then indeed, everyone just sees downvotes by users local to their home instance. Otherwise, downvotes work the same as upvotes - that is, somewhat reliably - unless your frontend or user instance hides them but that's your problem.