In the past 5 years of using Matrix, I have received exactly 2 direct spam messages, and seen maybe 5-10 in public rooms. (There have been none in my private chats, of course.) If you're seeing much more, I guess it must depend on how you use it and what rooms you join.
The problem we have is trying to get our friends to jump ship to a new alternative. Pretty much any barrier, even small, prevents people from doing this because they already have a working app that does what they need.