The hidden cost of self-hosting
The hidden cost of self-hosting
Self-hosting services has been a life-changer. And I thank this community for helping me a lot recently. Not only did I learn a lot more about linux, network and docker, but it helped me understand better how platforms and advertising just f*cked up the internet I grew up with.
But I wonder: do any of you hate how self-hosting services like photo- or document-management systems, or even a simple rss tool, forces you to sort your stuff out, and put your decades old files in order?!
I'm in the process of migrating my web browser bookmarks to linkding because it's a GREAT tool. But I have like 2k websites to manualy check wether they're still there, wonder at how cool they still are, tag properly and archive with SingleFile!
And that's just ONE service...
Have you considered the possibility that, if you have 2k bookmarks, this isn't necessarily a self-hosting issue, but rather a bookmark hoarding issue :)
That’s why I don’t go back and reorganize old bookmarks. I just start fresh every time.
This is what I do. I keep the old ones around for a while, and every time I realize that I'm not missing anything, and delete them.
Worst case, I'll have to root around in my backups. But it has never happened wrt browser bookmarks.
I don't think I've even used bookmarks since Vista.
I know, right!? Do I have to let go? Yes! Am I defined as a person by the shit I accumulate? No!
Sometimes hitting delete is the best thing you can do. Especially bookmarks, how many of them is out of date, or not relevant to you any more. And if you needed some of it, you can find it again. Sure, there is a few things a bit harder to find, but it should take less time than sort through 2k bookmarks. 😀
I think that’s a feature 😅😅
Just kidding, I had no issue with my photos for example with Photoprism, but for streaming my music with gonic I need to make some modifications for all my album art to show up, and in some cases titles and album names…
It is indeed a great feature but how time-consuming haha :)
I inherited my music collection from my 20 years old limewire addicted self, so it's a complete mess. I'm in the process of completing albums and using Picard to properly tag everything... 20 years of music collection...it'll take me 20 more years!
Anyway, I guess it's a warning for anyone starting to accumulate data: think about metadata, formats and data-management. NOW!
I went through the exact same situation with my 20 year old music collection and MusicBrainz Picard and unfortunately this is just the state of music piracy even now as things are rarely tagged correctly or come from compilation albums, sample albums, or are just obscure and don't have the proper tags available on the service. This is why I don't mind paying for music streaming even though I selfhost most other media formats because you can put in days of work tagging songs and still have a jumbled mess at the end (Picard's tagging template is also a huge PITA to use with some obscure legacy language).
Same boat. I put everything in Picard and let it analyze everything. It turned out about 95% perfect. Haven’t touched it since, and I’m using the metadata it generated.
There is a hidden cost to every hobby and everybody is willing to tolerate a certain degree of shittyness.
I have a friends that has a rather old car and something on it is always broken. But he has no problem having 20 different apps for appliances, instead of deploying home assistants. Or having ads everywhere and even trying pihole or at least NextDNS.
On the other hand, I see my car as a transportation tool and when I need it I want to use it without worrying about some random part exploding. But I have no problem running Proxmox and hosting tons of services for my family.
That said, I would definitely not self-host something like NextCloud or any business critical component for my business and just paid somebody for the service.
You understand the value of risk management.
I do both - older vehicles always needing attention, and self-hosting shit
I just spun up Lube Logger so I feel this haha
I thought the hidden cost is my power bill by having a PC run 24/7....
You should see if you can get all of your services to work on a Raspberry Pi. They hardly use any power.
Some things would, but not everything, so at best case, I would need to run 2 computers instead of one. I have a beefy spare M1 MacBook Pro, uses almost the same as the Raspberry but it's horrible for selfhosting.
I condensed down from a power hungry tower server to a couple of thinkstations and a nas. Much nicer on the power.
I hate having to run my own backups. That's been a massively hidden cost behind self hosting that I did not originally account for. Anything sufficiently robust is expensive and anything cheap is unreliable (at least at the scales of data I have, 4k+ RAW videos and photos are massive).
Does it still count as "self hosting" if one of your backups uses something like restic to push to b2 or hetzner storage boxes? It's not consumer point and click.
I have one copy going there, and one going to a $50 thinkstation usff connected to a single external hard drive. It's not raid, but if it dies, it just gets quickly replaced while I rely on the hosted backup.
It's important to use services with a workflow that works for you; not every popular service is going to be a good fit for everyone. Find your balance between exhaustive categorization and meaningless pile of data, and make sure you're getting more out than you're putting in. If you do decide that an extensive amount of effort is worth it, make sure that the service in question is able to export your data in a data-rich format so that you won't have to do it all again if you decide to move to a different tool.
There's also the slightly-less-hidden cost.
Electricity to run your home server(s).
Just 2k in bookmarks? Pffft! Those are rookie numbers. Check back when you have 59k bookmarks. Currently there are 1.1k in the broken links category. The vast majority of the links are topics I research or have interest in, exterior of self-hosting. I do not consume TV data, but I do a ton of reading. I find that reading gives me better retention of the topic, and it's rather easy to highlight & search for cross comparisons, and further research. Ever since I was a wee lad, barely able to read, I have had an insatiable lust for knowing. It is this that drives the link counts. LOL
This is so foreign to me. I never bookmark anything ever. I leave a few tabs open until I complete that task, read that article or decide I don't care anymore.
I found that going back to bookmarking (and subscribing to RSS) is the best way to pull away from the algorithm-feed-trough of the social media websites and SEO bullshit. As I got more and more bookmarks of interesting sites, and found lots of feeds to subscribe too, I found I naturally gravitated away from the corporate web. It's a requirement now if you are interested at all in indie-web type stuff, forums for esoteric hobbies or software communities, or personal web pages of interesting people -those things just don't show up on search engines or social media anymore.
I digitally collect odd things, selfhosted in several apps depending on if it's for 'read later and decide' or preserve.. For one, I like the etymology of words or phrases and how they've evolved in meaning, and in some instances bastardized the meaning. For another, I collect political cartoons from any country. I am fascinated how some of the ones I've read about, have changed some people's minds. Things I find educational. Things that are totally polar opposite me. You'd be surprised what you learn even tho you may still remain opposed. So these are a back up of a backup which gets backed up, lol, It's the source files if you will, and I archive them in another app however I still keep the source as a backstop.
I'll end with this as an example since this might be misconstrued as not about selfhosting, As a wee lad, someone donated a set of Encyclopedia Britannica to us. I read those cover to cover many times. So, with the help of self hosting and dedicated devs around the globe, thank you so very much for being so generous with your skills and time, I can continue my quest to know.
TL:DR: I'm just a weird, old man.
Hate? Digital decluttering feels really good, for me anyway.
Pain feels good. It's like sport, is it? Is it sport? I'm healthy.
I guess the trick is to not look for stuff to host because you'll end up with all kinds of things you weren't doing in the first place.
And so little time!
Actually, that is a thing I like. Going through this stuff can be tedious, but it brings a lots of memories, things that I forgot about, things I once wanted to do. And also, after cleaning my digital life I feel similar as after cleaning in the physical world - good - I did something, I made my world a tad bit more organized and a tad less overwhelming. (I should note that I am lazy and I always must force myself to clean, but I never regret doing that after I start 😀) P.S. as I wrote in one comment below, maybe bookmarks is not a necessarily a thing that you want to go through and sort. Here I am more writing about my notes, or photos, etc..
Definitely second your feeling. I am similar in my relationship to cleaning. It feels like a lot of effort, but efforts feel good afterwards.
Karakeep. It will throw an error if a website is down and you won't get tags.
If you really access them that infrequently, are they actually worth keeping?
Plus if they're links, how many still work?
Actualy a lot more than I thought, and they bring joy, as souvenirs :)
do any of you hate how self-hosting services like photo- or document-management systems, or even a simple rss tool, forces you to sort your stuff out, and put your decades old files in order?!
What is this "sort" thing you speak of? I don't sort anything, I have NextCloud syncing my entire photos, videos and documents folders and they are just as messy as ever. Granted, I do go through my photos and videos once a year and dump them in a folder named for the year they were taken. Occasionally, I'll go hog wild and try to sort some of a year's photos/videos into folders named after events. Though, that hasn't happened in a number of years. I setup NextCloud so I could have everything synced to my own server and just forget, not have to deal with labeling my data.
As for bookmarks. I already keep those in folders; but, I don't sync those. I use my desktop far more than I use my phone for web browsing. And the types of things I use my phone for (mostly recipes), I just keep bookmarked there.
It looks like we found another person that's immune! Sample their blood
Make sure you check https://karakeep.app/, because it has, at least, automatic tagging and full text search on the bookmarks
Thanks for mentioning SingleFile. I’m not using it right now.
The workflow with linkding and the linkding injector is gold.
Nothing better than a properly formatted data file.
Self hosting teaches you thisIsn't that the goal? If you have an old drawer full of unorganized stuff, implementing a selfhosted management tool is getting an organizer and thinking about how to fill it, but you still have to sort your stuff in.
The only selfhosted thing where I really have to re-organize is my documents in paperless but I'm so glad to finally have it all organized and searchable instead of some hot mess of an inconsistent folder structure.
I'm in the long process of paperlessing. It's THE perfect example of that (not so) hidden cost. But there's no lying or trying to sell you magic. You put effort in a systematization that empowered by a great tool and a well thought out and tried model, and voila, winning.
Yeah I think of it the other way round: I couldn't get myself to organize them without combining it with a nice selfhosted tool. The goal is getting my stuff organized, the cost is doing work, which includes setting up a system. I can cheat on the cost a little by including a fun project in the cost part.
I do think there's a hidden cost in selfhosting though and it's maintenance. Fortunately, there's selfhosted tools that help with that too :-)
Thinking back on your rhetorical question, I think it's just it.
It's the goal. The goal was always to try and make me think that I am not just simply taking care of my stuff (and by extension myself). Because taking care (of yourself) isn't valorized in a capitalist society.
Fuck it all. I'm putting YEARS of work into just sorting myself out.
Simplify as much as you can.
And remember, if you're also self-hosting for family, someone will need to take over all that software and digital clutter when you're gone.
I've been trimming as much as I can on my NAS, including only keeping the most important self-hosted software and heavily purging old files and backups.
This. I'm not that old yet, but the realization hit me in the face pretty hard. And all the more reasons to sort it out. And definitely simplify. Or "make it usable" let's say.
Content management makes up very little of what I self host.
That stuff's just curated and is always been curated as I take new things that I need I curate them.
Then there's another class of data I deal with which is synchronization. Synchronization as the wheat and the chaff in it and if any of it goes away I don't really care because anything that was really worth keeping already got curated.
IMO that's not a hidden cost. That's a decision you made. The actual hidden cost is the electricity and time you spend by configuring and mantaining your services.
Are you kidding me? True, there is time involved. My biggest 'sin' right now is "home gallery" for it works on MY directory structure which I won't give up.
The geoguessing game that hides in it is superb ! I'm still amazed with the images I've been able to locate. Sometimes 40 years back.
I'm frustrated that Immich doesn't have a "back up new photos only" option.
All the photos on my phone are already in a huge external library with my backups from previous phones. I don't want to delete them from my phone just so immich doesn't freak out, and I don't wanna have them on my NAS twice
Immich seems great, but this seems like the bread and butter migration path that nearly everyone would take.When I got a new phone, immich was weird about the photos that came over when I transferred my phone data. After I confirmed they were all backed up (in like 3 places) I just nuked the copies pictures and videos from my phone.
I don't want to nuke them from my phone though. I want them on my phone, and I want them on my NAS 1 time.
Yes. Some services are not good at accepting existing naming conventions, insist on their own naming and sorting conventions, or require a lot of third party services that will unfortunately rename your movie to something foreign or otherwise completely wrong.
It takes quite a bit of time to clean up titles and metadata that you may be migrating or adding from a personal collection. Sure, it’s free…but it doesn’t mean it isn’t frustrating or time consuming.
Yes and no. A lot of sorting and optimizing processes can be done via scripts. For example, I had chatgpt generate one that finds audio streams in videos that are not in the language I need. Manual verification and then let another script remove the remaining lists streams that I don't need.
Yeah, I don't bother sorting and organizing old files/bookmarks/whatever. Automatic tagging and full-text search solve that need. I try to keep recent stuff organized nicely though.
Can't you use a script for that? There is a method to bulk import into linkding from Firefox and a REST API for linkding that allows you to remove all expired links.
2k bookmarks? i would just automate the process of saving each of them locally and just forget it lol. if it's somehow needed later search on the older archive
2000 Websites? WTF why?
I didn't move shit haha. Dumped OneDrive onto the Nas and mounted it for next cloud, I didn't even clean out the photos, which I copied into immich. I did move some ebooks, but that was very few things that I have
I just moved 20k bookmarks from Pocket to Readeck, and can sympathize lol. A lot of the links are dead. I found a cleanup script I'm going to run but it's still a huge curation challenge
There are web extension that check for bookmakers that responds with 404 and automatically deletes them.
Idk. My folders are always decently organized since I've been nutty about since I was a kid, but the specific file structures different services can demand is a headache. This is why I prefer more simplistic services without a database, but there's always trade-offs to be had with both options.
I'm a bit split on it, but I do agree that it can be annoying and when you mess up, services and links you've sent to other people don't work and it can be quite agonizing. It'll probably get better for me as time goes on, but man it can bite at times.