Are search engines indexing the fediverse?
Are search engines indexing the fediverse?
I was wondering if the search engines are indexing the fediverse. Because recently Google search became so bad so I was always putting Reddit at the end to get some not so generic results and I haven't even noticed to get any replies pointing to the fediverse ever. So I was wondering if search engines were actually indexing the content here?
Yes, but keep in mind top level domains like .world .ai .life and similar are heavily punished by the search algorithm.
So let's say you search for "best Android apps 2023" and there's a Lemmy.world post with this exact title and a great list of apps, versus a clickbait techblog written by ChatGPT and a .com domain... The techblog will very likely be ranked higher.
One good thing about Reddit is that one could add a "site:reddit.com" at the end of each Google search to improve the results.
This won't work as well with Lemmy (since it's all a bunch of different domains). Hope that in the future a feature like this is implemented to search across the fediverse
Yeah, perhaps another qualifier that's, say, specifies the protocol, e.g. Activity Pub? Would be handy
They certainly are!
Sure they are. But the amount of info in the fediverse right now is minuscule to say R*ddit. It will take time and growth before we start seeing fediverse stuff show up regularly in search.
Some platforms like Mastodon allow individual users to discourage search engines from indexing their profile. But, by default, as far as I know all platforms allow indexing. Lemmy seems to not provide any option to control this kind of thing so everything should get indexed.
The thing is, using "thingtosearch reddit" you're not using any search engine properly, that's kind of a hack. What you would do is actually "thingtosearch site:reddit.com" to limit searches to a specific site. This works with any site, of course, so you could for example do "thingtosearch site:feddit.it" (that's my instance), and you will get specific results (which actually might include results from other instances, due to how this indexing works, even though they will be displayed from the site of the instance you specified). (I just noticed btw that DuckDuckGo doesn't list anything for site:myinstance... well, that's strange, Google has no problem.)
DuckDuckGo uses their own web crawlers along with a few others (but not google's). Probably just haven't indexed it yet.