Getting a large amount of downvotes on a regular basis might matter on the other site depending on various things like how automod is set up, but over here? My account could be net -1,000 on votes but its still only going to show how many submissions and comments I have on my profile.
Also it seems like some people have a bunch of bot accounts that vote alongside them, especially vegans, I mean, how many vegans are there really here...