What skills are needed to self host without too many headaches?
For this new year, I’d like to learn the skills necessary to self host. Specifically, I would like to eventually be able to self host Nextcloud, Jellyfin and possibly my email server too.
I've have a basic level understanding of Python and Kotlin. Now I'm in the process of learning Linux through a virtual machine because I know Linux is better suited for self hosting.
Should I stick with Python? Or is JavaScript (or maybe Ruby) better suited for that purpose? I'm more than happy to learn a new language, but I'm unsure on which is better suited.
And if you could start again in your self hosting journey, what would you do differently? :)
EDIT: I wasn't expecting all these wonderful replies. You're all very kind people to share so much with me :)
The consensus seems to be that hosting your own email server might be a lot, so I might leave that as future project. But for Nextcloud and Jellyfin I saw a lot of great tips! I forgot to mention that ideally I would like to have Nextcloud available for multiple users (ie. family memebers) so indeed learning some basic networking/firewalling seems the bare minimum.
I also promise that I will carefully read the manuals!
If you want to program something, the closest you're gonna get to programming is Ansible and Bash scripts.
You might want to get self hosting hardware like Synology or the like if you're not ready to dig.
Otherwise here's some things you need to know:
Docker
Easy, consistent deployment of services in their own environments. Think a VM but with almost no overhead.
Docker Compose
Run docker containers with consistent configuration in files.
Connect various containers to each other on the same or different networks.
Get multiple containers to start together and talk to each other.
Systemd
Manage any service on Linux. If anything needs to start on boot, restart when crashed, start on timer, you want Systemd.
You can manage your docker compose containers lifecycle via Systemd.
NGINX/Apache/Caddy
A web server for reverse proxy. You'd probably need one at some point, especially if you want HTTPS. Your services get hidden behind it.
ZFS
Reliable redundant storage. You'll need storage. Use ZFS with 2-disk redundancy.
Supports automatic snapshots for recovering from oopsies. E.g. deleted something or some software shat on your data.
Can use recertified disks from serverpartsdeals.
Can use USB disks or USB box with multiple disks. If you end up going the USB route, ask me for tested hardware.
Backup system
Something to do backup. There are many options.
Ansible
If you want to write code that describes your services and make them happen, you want Ansible. You write code (well YAML) and Ansible installs things, writes config files, sets up Systemd services, restarts things. It can be convenient especially if you have a lot of stuff and you want to be able to see all of your infrastructure in code in one place and be able to version it.
Prometheus
Monitoring your stuff. Is my backup service running? If not send me an email.
Ansible is nice but I'll repeat (as I said in another thread) it's kind of advanced and gives a much better return on investment if you manage several hosts, plan to switch hosts regularly, or plan to do regular rebuilds of the environment.
Why Debian or Ubuntu? (I have my own thoughts, but it would be useful to show even high-level reasons why they're preferred).
Re: Backup - Backblaze has a great writeup on backup approach today. I'm a fan of cloud being part of the mix (I use a combo of local replication and cloud, to mitigate different risks). Getting people to include backup from the start will help them long-term, so great you included it!
Predictable cadence, stable operation, timely updates, huge community and therefore documentation. You can get up to 5 years from an LTS release of Debian or Ubuntu. With Ubuntu LTS and Ubuntu Pro (free) you could theoretically run a machine without upgrading for 10 years. If you run workloads in containers, it doesn't matter how old the host OS is. As long as it's security patches, you can keep on trucking.
X570 is especially bad, though I've had some success with B350, when using the chipset ports. The CPU ports are all bad. Small form factor PCs often only expose CPU USB ports. They work with single disk per port but if you peg a port with a multi-disk box, they crap out regularly.
Post AM5
Have only tested USB4 on X870 and it's solid.
Client side
WD Elements / MyBook
If you get disconnects under load and you're not on a shit AMD USB host, the USB-SATA controller is overheating. Open them and ahere a heatsink on it. Drill a hole in the case above it for better ventilation. Disconnections will stop. If you don't want to deal with any of that buy the item below.
OWC Mercury Elite Pro Quad
Well built, solid controllers, no issues over a year of testing. I have 2, hosting an 8-disk RAIDz2 and 2 hosting a 5-disk RAIDz2.
Terramaster
A friend bought a 6-bay and tore it down for me. It has the same controllers as the OWC in a similar topology. If it's cheaper it might be OK. I can vouch for the OWC though.
Cables
Get name brand cables, ideally higher spec than what you'd need! They aren't important for a single USB disk but running a 4-disk box can max out the port bandwidth. If the cable can't handle it... errors. Casually transmitting 10Gbps via easily detachable cables and ports isn't trivial.
Much appreciated 🙏
Gnarly stuff with the WD's huh? Unfortunately I think that's what I'll end up having to put up with since I can't really find the other options for a decent price around here.
Funny enough I was half-considering just using a bunch of WD Elements. You think the MyBooks might fare any better?
I used a mix of Elements and MyBook for years. Upon opening to heatsink, I didn't see any significant differences between them. They use ASMedia or Jmicron, mostly ASMedia. The overheating issue depends on ambient temp and load. I've had one machine in a basement never experience them. Either way the solution is pretty straightforward and cheap. Once heatsinked, I haven't had a problem.