Seeking advice for selfhosting critical data
Seeking advice for selfhosting critical data
Hello everybody,
my plan is to switch from Android to GrapheneOS. In this process, I want to get rid of my reliance on my google account as much as possible.
To this end, I'd like to selfhost some "critical" data, e.g.
- contacts
- calendar
- online drive for files (e.g. google drive alternative)
- some basic note-taking app (like google keep)
and so on.
I do some selfhosting already, though it is not that kind of "cannot lose this" data. So I'd like to share my thoughts and ask for your opinions and experience for the process.
More details for what I want
contacts
- have to be syncable to the phone
- if possible, some webinterface to edit / browse
calendar
- has to be syncable to the phone
- webinterface + sync to desktop / phone
- if possible, send invite-links to events to others
drive
- files of my choosing must be offline-available
- ever other file should not use storage on the phone
- if possible, able to share links to download files
- if possible, able to share links to view with online editor (see below)
document editor
- think google sheets / google docs
- if possible, able to share links to view documents online
smartphone photos
- auto-backup camera folder
There may be some things I'm not thinking about right now, but this seems to pretty much be it.
If possible, all of this should be accessible only via vpn.
What I already have
I have a pfSense physical appliance that's already managing my home network, got an OpenVPN already setup, dynamic DNS working properly for the lack of a static IP, etc.
I own 2 mini-PCs (some Intel NUC, some passive-cooled zotac with an intel with 4c/8t). One of them (zotac) is currently running as my Proxmox Virtual Environment Hypervisor, managing 3 VMs.
I also have a second PC which misses some critical parts, so it is not currently in working condition. I think there's an AM4 mainboard and 16 or 32GB of DDR4 RAM in there. I could make a NAS or a new hypervisor out of this, but the case (Fractal Design Define 7) is quite big and a full PC is probably worse for energy-efficiency than my 2 mini-PCs and is going to be more expensive.
Not much in terms of storage sadly
- 1x 6TB external USB HDD (used for backups)
- 1x 2TB external USB HDD (used for data)
What I plan to do
The kind of data I'm going to be hosting myself now is very import, so it cannot be lost or corrupted.
But the feature list doesn't seem to be overly complicated. This seems like something nextcloud could do.
This means, I will probably need to buy
- 2x 4 TB HDD for storage for data RAID
- 2x 8-10 TB HDD for backups
- 2x external RAID case
Then I could connect the data RAID to the already running zotac pc and spin up new VMs for nextcloud and whatever else I might need and start serving my data from home.
The Intel NUC will be used as a Proxmox Backup Server, connected to the backup RAID. Keeping some daily, weekly and monthly backups.
On the phone-side, I'd have the vpn always active. Whenever active, sync of contacts, calendar entries, photos etc. should be possible.
Questions
Is there anything I missed? Did any of you already try something like that? Does anybody here see a potential problem with any of the above?
Can anyone recommend a RAID-1 external enclosure without a fan and some quiet and energy-efficient HDDs?
You're viewing a single thread.
3-2-1?
Thank you for your input!
I also thought about the 3-2-1 backup rule, but am unsure if that is overkill.
My VM-backups and file-level-backups are proxmox backup server (pbs) backups. Meaning, to have them offsite, I'd need to rent a dedicated root server on which I am able to install pbs to act as an offsite sync-target. With TB of backups, this is gonna get very costly very fast.
I thought about regularly exporting encrypted calendar and contacts onto some free online storage, hoping I can automate this process.
With what I have layed out in my post, to lose contacts and calendar events, both my intel NUC and the zotac mini-PC have to be corrupted at the same time. Or both RAIDs simultaniously failing both drives. Am I not paranoid enough or is that an acceptable level of failure-safety?
The offsite rule is mainly in the scenario where your house burns down for example, or if someone steals your stuff. It can happen.
Maybe your electricity will have variations and will fuck up all your devices in a specific location..
Yes, you're right. As David From Space said in this comment, the real critical data is far less then all of the backed up data.
So I definitely can have an offsite-backup, it just depends on if I can single these things out in nextcloud, possibly via regular export to the filesystem.
Yea if you don’t need much then you can do with exporting not a lot of stuff.
Google is evil but I know that GDrive has pretty low prices on data storage
There are many cold storage services out there with good pricing. If you need a VPS with good storage (to automate sync, etc.. idk), I know I would use Interservers, based in the USA, priced at 3$/TB/month (HDD)
But if you only want to sync a small amount then you can do with free services, probably. Don’t forget to encrypt everything when uploading to these services! Don’t want them to be able to see the content of your files.
Google is evil but I know that GDrive has pretty low prices on data storage [...] Don’t forget to encrypt everything when uploading to these services!
That is what I am hoping for :) My free Google account grants me 15GB of online storage and my free Microsoft account provides me with another 5GB. The 15 GB should be enough for encrypted photo backups, while 5GB definitely is enough for encrypted calendar, contact and probably some document backups. I just need to find a way to automate backups to these.
based in the USA, priced at 3$/TB/month
If I am going to pay money for something and with how the world currently is, I'm going to use some EU based service. My only VPS resides at hetzner, if the need arises I will probably just add a storage volume to my VPS or upgrade it to the next tier.
Got it! Sadly, hetzner doesn’t have the best prices on storage
Yeah, hetzner's more about having your own servers than providing cheap storage.
Eh, you can have your own servers with many other providers...
A storage server is still a server. No need to be elitist
I didn't intend to be elitist about anything and I actually fail to see the elitism by saying "that hoster is not about providing cheap storage"? Maybe there's something in the english language I do not pickup on?
It's just that Interservers also provide VPS that aren't focused on cheap storage, allowing you to have, as you said, your own servers.
I understood the message as "interservers is only for storage, not for hosting anything serious on it", and thought you prioritized the brand image of Hetzner while saying competitors were shit without any argument
I also thought about the 3-2-1 backup rule, but am unsure if that is overkill.
Maybe you shouldn’t be home hosting critical data if you think this is overkill.
You've got a point, but now I gotta ask: Where do you store your original paperform documents? You know, the real-life critical things. Maybe I'm wrong, but I feel like most people store these things at home, possibly tucked away in a neat, little, sorted folder, for preservation. Which would be a nightmare for all the same reasons, but seems strangely accepted and widely practiced.
No data I own is life-or-death critical. Losing everything would be really bad, but many things can be restored in alternative ways, except the photos.
Also, I may be able to backup the most important stuff (which would only be a few GB at most) to an offsite server, as long as nextcloud (or an alternative) is able to export contacts, calendar and photos, or I can single these out in some other way. As long as this somehow works, I can rent a cheap hetzner server with a few GB of storage and have that be the backup target for the most critical stuff.
My critical documents that MUST be in print are stored in a fireproof safe. If it doesn’t need to be in print it’s scanned, shredded, and backed up.
Photos are scanned and stored. They’re also backed up.
Then I give praise to you, for you are more prepared than any other individual I personally know of and even some smaller companies I had worked with.
Thanks.
It comes from a combination of working a job where I saw data loss on a weekly basis and then working a job in a highly regulated industry.
What @AtariDump@lemmy.world said is correct, if it's critical data, 3-2-1 is necessary. I personally use BuyVM as my offsite as it's got pretty cheap storage (~$5USD/1TB/month), but if you've got family or friends with a decent internet connection, it's trivial to set up a remote sync job to any offsite Proxmox Backup Server, perhaps on a box stored at their house.
Now, just to throw it out there, my actual 'critical data' is way smaller than my total backed up data, including my media library, random ISOs, etc. - it can be worthwhile to determine if you really need to backup everything offsite or if you can sort out some less necessary data, and only upload some data to a remote server. Maybe the answer is yes, and you'll need to account for that!
I took a look at the BuyVM offer you mentioned since it sounds really good, but am I understanding correctly that to make use of the 1TB storage offer I would have to also order a dedicated VM with them to actually make use of it? (i.e. no mounting from a vps with a different provider)
That's correct, I also pay for their cheapest VPS, which is about $3, pretty good overall for my purposes!
Now, just to throw it out there, my actual ‘critical data’ is way smaller than my total backed up data
That's also the case for me. I'd probably count a few GB as critical. Contacts, Calendar, some photos, some documents.
If nextcloud (or some other alternative) has the ability to regularly export these things to an on-disk location, I could definitely backup that to some cheap hetzner server. This will not be a pbs backup, but I can get by with an offsite-backup done by something like restic or rsnapshot
Thank you for your advice!
Or a house fire, or flood, or lightning strike, or theft. Or just plain fat fingering something and deleting it all.
If you really mean life-or-death critical, yeah, 3-2-1 is the starting point.
If you really mean life-or-death critical
No data I own is "life-or-death" critical.
I can ask around for contact info again, same with calendar events I had planned. Some documents can be restored via the original service or by paying a fee to get a new original document, I still have folders full of originals in paper form. Some info can be restored by looking through my bank account or online buying activity. Losing my photos would be really sad, but nothing of that will kill me or destroy my life.
But I definitely can save the most critical stuff (probably a few GB only), if nextcloud (or some alternative) has the ability to regularly export these to an on-disk location. This way, some backup utility like restic or rsnapshot shoud be able to do the job.
Okay so not critical, just mildly inconvenient if lost.
I would just keep one copy in RAID, and for the most important stuff a second copy locally or in the cloud. Yes, RAID is not backup, but a disk failure is probably the most likely failure scenario. Corruption is the second most likely.
Okay so not critical, just mildly inconvenient if lost.
I wouldn't put it at "mildly inconvenient", as the photos I could lose can never be restored. Most of the other things can. I'd be really sad if I lost all the photos, but it wouldn't threaten my existence in any way.
I'm sorry, I should have specified in more detail what I meant by "critical".
It's not life-threatening, it's just critical to me. It's kinda like "my priciest possession" could mean a yacht or a half-dead car, depending on the context.
[EDIT]
a disk failure is probably the most likely failure scenario. Corruption is the second most likely
Yes, these are things that are 100% going to happen at some point. I cannot guarantee theft, floods, earthquakes or anything like that, but hardware degrades with time and use, so at some point things are going to fail.