New to Lemmy @lemmy.ca megane-kun @lemm.ee 1w ago

How to Archive One's Own Posts and Comments?

Not exactly new to Lemmy, but my search-fu has been noobish at best.

Lemm.ee would be shutting down at the end of this month, and while I've already moved to a new instance, I'm yet to do any archiving on my posts and comments. Is there an automated way to save my posts and comments.

Ideally, I'd want the archive to be:

a full copy of all the posts including
- the full OP text
  - any embedded images will be saved and included in the OP
  - outward links are left as is
- all of the comments, including any deleted ones (deleted by the user and deleted by mods/admins)
a full copy of the comments including
- the full OP text of the thread in which the comment is made
- the full tree up to the top-level comment
- optionally including any deleted comments in this tree

Is this already a thing? I don't think I have the skills and the time to make one before June 23 (one week before the instance shuts down on June 30), so that is not an option. I doubt anyone can make it upon short notice either.

Is there any other method I can do this without resorting to manual saving? And if I have no choice other than to save each and every post and comment manually? How should I be doing it?

If anyone can suggest anywhere else I can crosspost this for better visibility, that will be welcome as well.

18 comments

Just make a simple script (with bash or something) and go through your post/comment history via the API. Save the JSON of the posts/comments somewhere.
- ‘Just’…
  
  Rest of the fucking owl moment.
- My coding skill is worse than my search-fu. While I can make a "hello world" script with bash, that's about my level of coding skill.
  
  However, let's assume that I'm willing to still do it. I have this coming weekend to do the following:
  
  Study the Lemmy API with the aim of extracting the JSON of:
  
  all of my posts
  
  all my comments and the posts where they were made
  
  Create the logic:
  
  extracting the posts
  
  tracking the posts where the comments were made, and then extracting them
  
  Write the script.
  
  I think the worst of it would be studying the API, but programming using Bash (or Phyton--which I am totally ignorant of, but might be better for handling the data) might also give me trouble.
  
  I'm going to need a lot of luck if I'm going this route. I dunno if it's better than just doing it manually.
  
  You don't need to "study" the API. The only thing you need is the /user API endpoint: https://mv-gh.github.io/lemmy_openapi_spec/#tag/User
all of the comments, including any deleted ones (deleted by the user and deleted by mods/admins)

You probably will not have access to this.

Is there any other method I can do this without resorting to manual saving?

Only if you are an Admin. https://join-lemmy.org/docs/index.html

The data seems to be stored in a postgres database, and could technically be queried.

And if I have no choice other than to save each and every post and comment manually? How should I be doing it?

It depends on your need, if it is for legal purpose, it depends on your jurisdiction. Maybe web archive is enough for this case, https://archive.org/.

Otherwise, you can use the API as others mentioned. Or you can use Selenium/Scrapy/BeautifulSoup to scrape the website.

If you would rather not program something with code, look into browser add-ons that scrape websites, they are mostly visual, and you click on the things you want to save or navigate into. I am not familiar with them to recommend you something, but there are plenty of videos on how to use them.

That said, depending on the security of your instance, your ip/account might get flagged.

Talk with the Admins of your instance first and express your intentions, maybe they can help with what you need.
- I'd rather not bother my admins (they're already burnt-out), and with what you just said, maybe I'm better off doing it manually--if I would do it afterall. TBH, I'm scared off by the impression that saving my own posts and comments is somehow taboo.
  
  Thanks!
  
  I don't think it would be taboo to save something that is public available, as you can just as simple visit those pages and print them to pdf for example.
  
  The same goes to things you have access with your account, and it is not bound by a non-disclose agreement (you can still save, but not broadcast it - depending on the laws of your jurisdiction).
  
  I looked into the lemm.ee default profile page, and you might have success with the tools I mentioned.
  
  You just need to navigate these links, and make the tool open the post names.
  
  https://lemmy.ca/u/megane_kun@lemm.ee?page=1&sort=New&view=Posts https://lemmy.ca/u/megane_kun@lemm.ee?page=1&sort=New&view=Comments
  
  You will need to take in account navigation with "next" buttons, and some pages need to be scrolled down to load all the comments.
  
  If you don't want to contact the Admins, and you are not pressed for time, put a delay on the tool's web requests to not overload the servers. The browser add-on might avoid that because it mimics a more natural way of navigating a website.
Why? My comments at a contribution to a public discussion, saved just for me they do little for their original purpose.

given the nature of Lemmy, perhaps you could run your own instance that you could use to fetch every thread you've participated in, and back up that way?
- In my response to someone, I already expressed concern about their long-term visibility--which is the reason why I even thought of doing this. I can't do anything about "contribution to a public discussion" if those very discussions pretty much disappear. This might be showing my lack of knowledge about what federation actually is, and if it is, it's pretty much on me.
  
  For the second point, I already addressed it in the OP and in one of my responses. I'm pretty much not a programmer and my knowledge about it is pretty much writing a "hello world" program.
- Follow-up, I peeked at your profile and I think I've found what you're intent on preserving, posting here to keep the record alive:
  
  I remember when I first heard the rumours and I immediately thought of how sensitive one's anal linings would be to perceive Morse code via a butt plug. Then pondered upon the max possible bandwidth of buttplug-mediated information transfer. Finally, I thought about how to send back information via rythmic anal clenching.
  
  Only then did I conclude that it's probably easier to get better at chess.
  
  https://sh.itjust.works/comment/3662094
  
  PS, I could be wrong, but I think your comments should continue to exist through the federated record of other instances. I don't think they purge their history whenever a server goes offline.
  
  Got me!
  
  I am still into that research. Yes. Research. For science!

18 comments