I've only ever used desktop Linux and don't have server admin experience (unless you count hosting Minecraft servers on my personal machine lol). Currently using Artix and Void for my desktop computers as I've grown fond of runit.
I'm going to get a VPS for some personal projects and am at the point of deciding what distro I want to use. While I imagine that systemd is generally the best for servers due to the far more widespread support (therefore it's better for the stability needs of a server), I have a somewhat high threat model compared to most people so I was wondering if maybe I should use something like runit instead which is much smaller and less vulnerable. Security needs are also the reason why I'm leaning away from using something like Debian, because how outdated the packages are would likely leave me open to vulnerabilities. Correct me if I'm misunderstanding any of that though.
Other than that I'm not sure what considerations there are to make for my server distro. Maybe a more mainstream distro would be more likely to have the software in its repos that I need to host my various projects. On the other hand, I don't have any experience with, say, Fedora, and it'd probably be a lot easier for me to stick to something I know.
In terms of what I want to do with the VPS, it'll be more general-purpose and hosting a few different projects. Currently thinking of hosting a Matrix instance, a Mastodon instance, a NextCloud instance, an SMTP server, and a light website, but I'm sure I'll want to stick more miscellaneous stuff on there too.
So what distro do you use for your server hosting? What things should I consider when picking a distro?
@bjoern_tantau@communism That 'support for years and years' means security support. So even if the nominal versions stay stable, security fixes are backported. Security scans that only check versions usually give false positives: they think fixes in newer versions are not present when in fact they are.
Many others distros do exactly the same. I only chose Debian because the amount of software already packaged in the distro itself is bigger than any other, barring 3rd party repos.
Always, always, always: Debian. It's not even a debate. Ubuntu is a mess for using as a server with their snaps bullshit. Leave that trash on the desktop, it's a mess on a server.
I run NixOS. It (or something like it, with a central declarative configuration for basically everything on the system) is imo the ideal server distro.
Very true haha. NixOS is great and the best I've got right now but I would lie if I said it has never been painful.
Especially for desktop use I want to build my own distro which takes a lot from NixOS, mostly in terms of the central configuration but not much else (I definitely want a more sane package installation situation where you don't need stuff like wrapper scripts which are incredibly awful imo), but also other distros, and also with some unconventional things (such as building it around GNUstep). But who knows if that ever gets off the ground, I have way too many projects with enormous scale...
I switched mine to NixOS a while ago. It's got a steep learning curve, but it's really nice having the entire server config exist in a handful of files.
Because Ubuntu is the worst of both worlds. Its packages are both old and unstable, offering zero benefit over always-up-to-date distros like Arch or the standard Debian.
Especially when you're running a containerised environment, there's just no reason to opt for anything other than a stable, boring base OS while your containers can be as bleeding edge, crazy, or even Ubuntu-based as you like.
How did you set up the intial system?
From what I've seen, FCOS needs an ignition file and has no Anaconda installer. I would like to set it up soon too, but it looked like a huge hazzle...
It’s not conventional wisdom, but I’m happiest with arch.
I’m familiar with it
can install basically any package without difficulty
also love that I never have a gigantic version upgrade to deal with. sure there might be some breaking change out of nowhere, but it’ll show up in my rss feeds and it hits all my computers at the same time so it’s not hard to deal with.
Arch never really surprises me because there’s nothing installed that didn’t choose to put there.
I agree and use Arch as well, but of course I wouldn't recommend it for everyone. For me, having the same distribution on both server and desktop makes it easier to maintain. I run almost everything using containers on the server and install minimal packages, minimizing my upgrade risk. I haven't had an issue yet, but if I did I have btrfs snapshots and backups to resolve.
same exact setup, I'm running arch for years on both server and desktop, btrfs and containers. It's beautiful and I click perfectly with it's maintenance workflow
arch is great if you don’t really care about your server being reliable (eg home lab) but their ethos isn’t really great for a server that has to be reliable… the constant update churn causes issues a lot more than i’d personally like for a server environment
I could not disagree more. Arch is unstable in the meaning that it pushes breaking changes all the time, (as opposed to something like Ubuntu where you get hit with them all at once), but that’s a very different thing from reliability.
There are no backported patches, no major version upgrades for the whole system, and you get package updates as soon as they are released. Arch packages are minimally modified from upstream, which also generally minimizes problems.
The result has been in my experience outstandingly reliable over many years. The few problems I do encounter are almost always my own fault, and always easily recovered from by rolling back a snapshot.
It's not insecure. Quite the contrary debian repositories only include packages that has been through extensive testing and had been found secure and stable. And of course it regularly introduce security updates.
I used to use Ubuntu, but nowadays I just go with Debian for servers (as well), but you said you wish to choose something else, so I can't give you any meaningful inputs...
I don't know how real the outdated packages threat, but I would assume, a server never really wants the bleeding edge software and Debian usually gets the critical security updates and patches.
But I'm no expert.
It is true that Bookworm is kinda old now, though.
Yeah I agree I don't want bleeding edge hence why I won't be using anything Arch-based (despite the fact that Arch-based systems are the ones I'm most familiar with, I'm typing this on an Artix system rn). But there is definitely a middle ground between bleeding edge and outdated, and I imagine a server should want to be somewhere between the middle and outdated, depending on how they balance stability and security.
I'm also not categorically opposed to using Debian. Ubuntu was my first Linux distro so I'm at least more familiar with Debian-based distros than most other popular server distros. I was just thinking probably not Debian because of how old its packages are and that I'm fairly concerned with security.
Debian runs on most cloud servers, it's pretty secure. The outdated packages refer mostly to apps, which is the reason why Debian is so stable. No frills and boring. Documentation is plenty on the internet and for server space it's probably the most compatible OS.
I'm running Debian 11, kernel 6.10 on Odroid. Arch on my desktop.
Anyway, I'm pretty sure almalinux-deploy allows migration from Centos Stream 8... it's your second chance to be done with fickle management decisions from RedHat/IBM: don't miss it this time :)
I have tons of experience with enterprise linux, so I tend to use Rocky linux. It’s similar to my Fedora daily driver, which is nice, and very close to the RHEL and Centos systems I used to own.
You are slightly mistaken with your assumption that debian is insecure because of the old packages. Old packages are fine, and not inherently insecure because of its age. I only become concerned about the security implications of a package if it is dual use/LOLBin, known to be vulnerable, or has been out of support for some time. The older packages Debian uses, at least things related to infrastructure and hosting, are the patched LTS release of a project.
My big concerns for picking a distro for hosting services would be reliability, level of support, and familiarity.
A more reliable distro is less likely to crash or break itself. Enterprise linux and Debian come to mind with this regard.
A distro that is well supported will mean quick access to security patches, updates, and more stable updates. It will have good, accurate documentation, and hopefully some good guides. Enterprise linux, Debian and Ubuntu have excellent support. Enterprise linux distros have incredible documentation, and often are similar enough that documentation for a different branch will work fine. Heck, I usually use rhel docs when troubleshooting my fedora install since it is close enough to get me to a point where the application docs will guide me through.
Familiarity is self explanatory. But it is important because you are more likely to accidentally compromise security in an unfamiliar environment, and it’s the driving force behind me sticking with enterprise linux over Nixos or a hardened OpenBSD.
As a fair word of warning, enterprise linux will be pretty different compared to any desktop distro, even fedora. It takes quite a bit of learning, to get comfortable (especially with SELinux), but once you do, things will go smoothly. you can also use a pirated rhel certification guide to learn enterprise linux
If anything, you can simply mess around in a local VM and try installing the tools and services needed before taking it to the cloud.
I have one server running arch and 3 running debian.
So far they are equally stable after running for about half a year.
Autoupdates are turned on on all of them. Which I am aware is against the arch wiki recommendations, but the server is not critical, easy to migrate and has nightly offsite backups anyway.
Servers are the one thing I've generally heard people agree that snaps are good for, so given its history it's a bit of a strange thing to hear of Ubuntu being a better server distro than desktop distro nowadays.
snaps are like poor man’s containers when it comes to servers… maybe better than having single-use VMs but if you’re wanting to build out real systems in a modern way, i literally haven’t worked with anyone using ubuntu in the last ~10 years
Debian backports security updates to most software, including popular server software. Stable also always uses an LTS kernel, which stays supported upstream. So long as you’re using latest Debian Stable (Bookworm as of this writing), run apt update often (in fact, ‘’’unattended-upgrades’’’ is probably not the worst idea in this case) and do common sense security practices like a firewall and (brain is not working), you should be good.
In brief, it’s totally fine to use Debian and in fact one of the best options in my opinion.
Unraid is amazing for getting into servers. It's just the right amount of WebUI and minimalism. Very safe and comfortable defaults, and the ability to start tweaking and adding more.
I use nixos, due to the incredible state management. You know exactly what versions of packages are on your machine, can build all packages from source yourself or download from a binary cache. 100% reproducible. Steep ass learning curve but tbh it's well worth it. Saves you configuration time and energy in the long run. I've stopped distro hopping the implementation is so good. If you are concerned about security you can definitely harden it. There's a lot more to security then package version. And even then nixos gives you the choice.
You don’t wanna use rolling release distros trust me, the whole point of server is automation and less maintenance. I got couple personal servers running, after things i need got setup and all of them running at a decent capacity, i just turn them on and never worry about them. Old package and software doesn’t necessarily mean less security, quite opposite actually, i suggest you take a look at how stable distros distribute their software, such as Debian. For a Debian package becomes stable, it has to go through several stages, experimental, unstable, testing, and finally stable, that’s why their packages are old, and because they are old, they are secure. It might be quite opposite than what you expect.
Mostly i use Debian for my personal servers, some of them are stable and some of them are testing, because of Podman’s new feature Quadlet. Honestly many features of Debian feel really old, like APT’s source list, preferences, and the way to deal with unattended upgrades. It’s kinda hard to get it at first and it’s easy to shoot yourself in the foot, especially many people tend to unintentionally mix and match packages from different suites for new software. But once you get comfortable with it things just work.
As my experience, no matter what distros i use, the worst distros are always those that i don’t understand and in a hurry to put them into production. Just pick one popular server distro and learn the ecosystem, you will find out what distros you like really soon.
Yeah, and key point in why old packages are secure is that versions with serious bugs and vulns don't get to the next stage, and if a package in stable is finally going to have one, they'll release a patch for it with just enough changes that fixes the serious issue.
There are some exceptions for very complex software, like Debian maintainers cannot be expected to be able to understand and see through something like Firefox. There they mitigate it by using ESR releases that are maintained by Mozilla.
I've been running arch for like 3 years now. Why arch? Because it just works (and its the only one i have esperience with). Maybe ill try nixos one day.
I currently use Ubuntu for all my machines (desktops, laptops, and servers), but I used to use Void Linux on my machines for about 6 years, including on a couple of VPSes. Since you are familiar with Void Linux, you could stick with that and just use Docker/Podman for the individual services such as Matrix, Mastodon, etc.
In regards to Debian, while the packages are somewhat frozen, they do get security updates and backports by the Debian security team:
I use Alpine Linux. It's exceptionally stable, great for pretty much any device and is best for small VPS with limited space/ram. Nice package manager too, but it is limited in packages.
It works great for me since I only use docker containers, but some things outside docker may require something like Debian instead.
Debian but mostly Ubuntu LTS with the free Ubuntu Pro that gives 10-year support. If I get hit by a bus, chances are the self-hosted systems I've setup would continue to work for years till my family can get someone to support or migrate the data. 😅
Personally, I use Rocky Linux on my servers. It’s stable, and has plenty of support since it’s RHEL-based. It’s supported until 2030 or so, and it doesn’t have any of the cloud-init or netplan stuff that Ubuntu Server has.
It’s also pretty simple to set up docker/podman containers, although you need the EPEL for podman-compose and for a lot of other packages, but once you get your setup the way you like it, it just keeps running and running.
I always use Rocky Linux or Alma Linux, since I have extensive experience with enterprise Linux and RPM packages. I have Fedora on my main desktop computer. Both Rocky Linux and Alma Linux are rock-solid and are ideal for any kind of workload.
Also, Debian is a good choice if you know how to manage DEB packages and you feel comfortable with APT.
Fedora is a good choice if you want fresh packages and are willing to upgrade your server every 6 months (following the Fedora release cycle).
Rocky Linux and Alma Linux follow a similar slow release cycle of RHEL, wherein you can install your server and not have to worry for years (as long as the packages are updated with dnf update) Debian is also a slow release distribution, which makes it good for servers.
If you are already familiar with one package manager, pick a distro that also uses that package manager.
When deciding on the release track, the harder it is to recover the system, the more stable the track should be. Stable does not imply secure.
As you move up through virtualization layers, the less stable the track needs to be, allowing access to more recent features.
Steer clear of distros that pride themselves on using musl. It's historically slow and incomplete. Don't buy into the marketing.
Think about IaC. Remote management is a lot more comfortable if you can consider your server ephemeral. You'll appreciate the work on the day you need to upgrade to a new major release of the distro.
openSUSE Leap - YaST is the greatest thing since sliced bread, and works great on command line over SSH. Yes, sometimes installing some software is difficult, but generally most stuff you would want is there and a lot of stuff runs on Docker anyway now. Very stable too, have had nearly zero issues.
Been running Ubuntu LTS releases on all my server VMs for 8 years and haven't had a single problem. Absolutely solid as a rock. Fantastic support, loads of guides to do anything. Plus you can get 10years of support as a home user with a free Ubuntu Pro subscription.
Ubuntu server, though I am thinking of using arch even though it is a rolling distro. It doesn't really matter. As long as docker is supported, I am fine using any.
I'll just wait a few days or even weeks before doing any big updates, read the news page of archlinux.org and maybe some forum stuff.
Nothing broke so far on my personal laptop, but I also don't tinker alot. All of the data of the containers are also stored in a storagebox from Hetzner so the system breaking wouldn't even mean that much, I'll just restore from a snapshot and everything will be fine.
I also might think of switching to NixOS instead. They say it's hard but pays off well and can be very stable.
Rocky and now moving too OpenSuse leap micro to move into immutable OS deployments.
Its all RKE2 (a k8s distro) on top anyways, so its very minor mods underneath, and base updates so I really want to maximize reproducibility and minim8ze attack surface.
If I were to do it again, though, I'd probably just use either fedora or the server equivalent to silverblue (I can't remember the name). I am so heavy on docker use at this point that I wouldn't mind going full immutable.
I guess you could use something like those new immutable distros to move away from state and related vulnerabilities. TBH there are plenty of hardening guides for Debian.
Or you could use any hardened version of Fedora which gets security fixes quicker, and then harden it some more yourself. The good part about Debian is that you are free to use SysVInit, I do not know if you could do that on Fedora. I do not think Systemd is a massive risk (if they have reached Systemd you have many other, bigger problems to think of).
I think I should study some more about Fedora. I run k3s on top and will go through their CISA hardening guide at some point to round things out.