The problem is that it's publicly shown. Let me pull an old comment:
When you have an upvote/downvote structure, there is an incentive to not go against the grain and just post stuff that’s considered “acceptable” in the local community.
Not everyone treats the votes the same way as well. Some people downvote because they don’t agree with the content, the poster, just because they sound disagreeable or simply because the post is too much to read. You can apply the same myriad of methods to upvotes. Not one motivator is the same.
If we’d ditch showing the numbers and just have them work under the hood, discussions might start looking a lot more genuine.
When my instance switched from 0.19.3 - which your lemmy.world is still running on (awaiting 0.19.6 iirc for reasons) - to 0.19.5, downvotes became hidden by default and I had to turn them on specifically.
I turned them on bc for me, more information is better - e.g. I like to distinguish between e.g. "1 upvote" that has not been seen by anyone yet (perhaps due to server sync issues, which just happened to me yesterday on startrek.website - e.g. my post has different numbers of comments and differences in voting numbers depending on which instance you view it from; such issues are still somewhat common on Lemmy, often due to keeping up with Lemmy.world that has 80% of the userbase, though 0.19.6 promises a reprieve iirc), vs. 1 net upvote that is made up of X upvotes + X downvotes.
A lot of my posts tend to be controversial for some reason - e.g. this video that points out bias in news media reporting (towards more "exciting" content that sells rather than actual facts) got only 6 upvotes (above the default +1) and 5 downvotes. So... it's actual information to know that (a) people actually did see it, rather than it still sitting at just 2, and (b) as many people didn't like it as liked it, though far more people simply ignored.
We cannot make people like things - even (literally) award-winning content that arguably they "should". All we can do is be sensitive to people's reactions. Like the man vs. bear debate - perhaps choosing the bear is "silly"? Then again, is it, really, truly, and anyway it's not our call to make, only others to choose as they wish. At which point, it's at least good to know what people's opinions are?