That can be true for self-contained command line tools, but not for complex programs with actively development dependencies (especially anything dealing with networking or encryption). For example hexchat uses GTK2 which is likely to be removed from mainstream distro repos in the coming years because it has been obsolete for a long time. Also openssl which is known to change its API occasionally which means that anything that uses it needs to be updated to stay compatible.
It's more akin to XMPP rather than IRC. From what I've seen, a Matrix server would be more resource-heavy than an XMPP one. Synapse one would probably not run on my weak machine at all, and Dendrite/Conduit are not feature-complete. And the primary reason I still haven't been on Matrix is that I have very limited disk space on my VPS, and Matrix saves media from every chat its servers are on, and I still haven't figured out how to opt out of that.