You're too kind! I didn't think I would ever actually need any of them, to be honest. Was preparing for a "grid is down" scenario, not a memory-hole situation.
It's unfortunate that we're entering a period where knowledge is actively suppressed.
So much the better than you've got this forum for us. I don't know when they new subreddits are going to catch the banhammer, but I sure would like to see people migrate off a corporate platform. No freedom in a company town.
Submissions are in git, but comments still have awhile to cook. Wanted to stay up for them tonight but I don't think they'll be done til morning.
I'll check in when they're uploaded.
Oh, that's awesome, thanks!
Grey Summit Gear has actually got a lot of this built out; I'm working on getting the last couple years of data ready and then he can finish rebuilding the fosscad sub at
https://forum.guncadindex.com/c/ark@fosscad.io
Just a heads up; I was able to finish pulling down the archives, but it's going to take awhile to parse; wasn't expecting to need this much storage touching my compute lol. I'm hoping I can have those ready for upload tonight or early tomorrow AM.
Hey, if you hadn't built this platform for us, it wouldn't be happening.
Seeing a bit of the old internet come together really warms my cold, dead heart.
Looks like the download speed dipped a bit while I was out; around halfway through now. So another 12 hours or so before the torrents are done, sooner if it picks back up, then I'll need to parse.
I'll ding you as soon as I've got the new zst's up.
And thank you! Would have taken me a week to put that scripting together.
Had to rearrange some things, but I'm pulling data from end of 2022 through 2024. It's a chonker. This is everything, so will need to parse through and find anything id'd as fosscad. Not sure how long it would take to iterate through all of that; it's over 1TB.
I'll be back this evening to update progress; download speed is pretty decent so if no big changes, should have the raw files tonight.
Edit Happy surprise; it's everything from 2023 to 06/2025. So losing the last handful of months of data (unless more gets added later). Still a pretty huge win.
Thanks to Grey Summit Gear for kicking the shit out of this, and the folks who pulled all these dumps!
There should be more; they may be split into several differential files. I'm going to work on getting the others right now, but I've got limited time before I have to leave for work. If I can't get them up in time, I've been using resources like https://academictorrents.com/details/ba051999301b109eab37d16f027b3f49ade2de13/tech&filelist=1
(if url's can't be posted, it's academictorrents dot com, posts by Watchful1)
I could have sworn I grabbed newer data for the fosscad subreddit specifically from a different site, but may have to go through the monthly diffs in this link and pull out anything under the /fosscad id.
These are much bigger, since they're the entirety of reddit (or top 20,000 subs, something like that), so download and parse is going to take a lot longer. Several hundred gigs to pull and parse. On review, no way I can do it all this morning but I can at least get some downloads cooking. Going to start pulling comments/subs for 2023, but won't be able to check on any of this until roughly 6pm eastern.
Any help is greatly appreciated. I'm not tied in to any social media, but if you're fluent in discord, matrix, element, any of the other popular IRC-likes, getting a room/discord set up would be super helpful. Or maybe if there is an existing discord for the community, point me in the direction and I'll try to join.
If you're not on social media, either, no worries (believe me, I understand). The encouragement helps, too.
Oh, that's awesome. If you've got a bot that can already parse and push to fosscad.io, we should definitely be able to tweak that.
I'm not active on discord (any social media really), but I imagine that's the place to organize an effort like this. If there is an alternative that others prefer (I've heard about matrix and element), I'm open to suggestion.
I've been fighting the flu the last couple of days, but on the upswing now. Dunno how much I'll be able to dig in today, but I'll get a github set up; I can dump the zst's there and some psuedocode and notes. If I can see what your bot is ingesting, I can try to match output from the zst's to it.
I think there will be a bit of work marrying comments to submissions; they're split up into two separate archives. Since the pictures are time sensitive (potentially), maybe the move is trying to focus on looping through the submissions and grabbing the pics from their urls, then rebuilding after the fact.
I've got plenty of local storage for pics or if we can dump straight to lemmy, that would be great. I'm completely ignorant to this platform as far as rate-limiting, storage, any of that fun stuff. I don't know how big a whole subreddit will end up being, but I imagine it's not inconsequential.
You're too kind! I didn't think I would ever actually need any of them, to be honest. Was preparing for a "grid is down" scenario, not a memory-hole situation.
It's unfortunate that we're entering a period where knowledge is actively suppressed.
So much the better than you've got this forum for us. I don't know when they new subreddits are going to catch the banhammer, but I sure would like to see people migrate off a corporate platform. No freedom in a company town.