The thing that bothered me was getting downvoted when it made absolutely no sense. When I made some book recommendations, for example.
And I'm pretty sure that in at least some cases, one or more people were following me around to downvote everything I wrote. That kind of petty malice is hard to understand.