Any post and community could be accessed through a theoretically limitless amount of instances, which also means a theoretically limitless amount of URLs.
Will this hinder Lemmy from ever coming into the mainstream? If I type any topic in Google, I will get a reddit thread that deals with that. Can something like that ever happen for Lemmy?
Would the canonical tag make any sense for Lemmy? The problem is, if you search for something your preferred site / URL is your instance. So the canonical would be different for every user?
Cool! How does it technically work? Does it fetch all titles (and maybe the body and comments) via the api from each instance or do you set up your own private instance and tap into the instance database?
Please make sure that you're only indexing Lemmy communities and Kbin magazines (i.e. not microblogs)
In the wider fediverse, there is an actual expectation of privacy beyond "well it's technically possible to scrape everything so we may as well give up". Several people (with reasons of innocent naivete & explicit and blatant malice alike) have tried making fediverse search engines, but all of them are either dead or blocked.
Lemmy/Kbin is in a unique position where global search does make some sense to have, due to it being a public forum focused on topics (and not people), but there is a very real chance that assholes could use an "unbounded" fediverse search engine to find vulnerable people (quite a few of them specifically fleeing to the fediverse to avoid that kind of problem) and harass them.
Currently it appears that a non-logged-in user (try an incognito window!) will only see posts on a particular server's local communities. So a search engine bot crawling multiple Lemmy servers will only see duplicates if they've been explicitly crossposted.
Yes I'm sure that it won't take too long for search engines to cotton on and start indexing Fediverse stuff - after all it's going to be getting linked from other sites, which will cause the spider to head to the source URL and start having a gander.
I think so, at least compared to more centralised sites, since there's no index or aggregator that concentrates that information in a place that can be easily accessible.
Unlike some place like Reddit, I can't just stick "Lemmy" or "site:lemmy.ml" on the end of my link, and expect that I would be able to get all the information across all the instances. At best, I would be able to search for what's contained within an individual instance, but that's about it, which makes it more trouble to search for things, since you have to know the server and community you're looking for first.