I'm on Signal (obviously not self hosted) and even if I really wanted to move to another platform be it self hosted or yet another privacy focussed one, I can't ask my friends and family to move to another platform again. I already asked them to move away from WhatsApp, can't do it again..
This is what I told most of my friend when they asked me to move to signal. Is is going to be a very shitty company managed by a shitty egocentric person and you are going to regret. But you will make people move and they won't do it again and won't understand the reasons
Matrix. With its bridges you can "wire-in" networks like WhatsApp, Slack, Signal, Telegram, Discord, iMessage, SMS, e-mail, ... and have a single app that interacts with them all. You can have a single group chat with users from all those networks participating and no one would be any the wiser.
From my experience (with Dendrite, not synapse, so keep that in mind), bridges create "fake" users to replicate your contacts on these platform as matrix users, and they are visible on the whole instance by all their users (but you might not be able to talk to them). Also, in puppeted mode (which is what you want to "replace" your app with matrix), only a single user can use the bridge at a time, so the other users cannot use it.
If you're already using Nextcloud, it has a chat w/ video chat as well.
Matrix / Synapse / Element.io is also pretty cool. The UX might not be on par with what some family expects though. I don't know if voice/video chat is built-in yet or not, but it was at least an option before.
Just for a family and friends I'd go for xmpp. Matrix is still an enormous greavy piece of software, hard to self host if you don't want to pay for a gigantic server just for it. Also the UI is more like gamer/company chat (discord, slack...), what may not be what your family expect, coming from whatsapp, telegram, or plain sms. In the contrary xmpp is very light and nowadays a lot of tutorial exists on how to configure it, even with voice/video. Plus mobile apps like conversation match the habbits of other messengers.
My family has been using Synapse since before Covid hit, including TURN server (coturn) for audio and video calls. No complaints about the UI so far, except for Element on iPhone reportedly "freezing" for a few moments when sending messages - and that was over a year ago.
I tried setting up coturn with docker but I can't remember offhand what I had issues with. I am running a reverse proxy (Traefik) with wildcard certs. I should dig into it again. Do you happen to have any pointers or any good guides I could try following?
Matrix works, but it's way harder and more expensive to selfhost than for example XMPP, which can be hosted even on cheapest VPS or first RPi. I would definitely take the cost and "how hard is it to maintain in the long run" into consideration.
Mattermost also works and is pretty easy to selfhost, but it doesn't have federation.
Another option is always an email with delta.chat - I don't think it offers voice calling, but email is one of the most basic services one can host, and many automated solutions to help with that exist.
Another Nextcloud user here. If you setup a Nextcloud server you will see many benefits, with the Talk app being just one of them. Install OnlyOffice and you can even collaboratively edit documents in the web browser similar to Google Docs or o365. I really can't recommend Nextcloud enough. Anyone who is into self hosting should at least give it a try.
The only downside to the OnlyOffice document server is that if you can't edit documents through the Nextcloud app (or mobile browser) on your phone with the community edition. However, you can set up your Nextcloud instance in the OnlyOffice documents app on your phone, and edit that way.
That's a great tip! I had no idea that you could do that with the OnlyOffice app now. I tried a few years ago and you could only connect to their hosted services. This is a game changer! I had been doing all my editing on the desktop.