Cloudflare plans marketplace to sell permission to scrape websites

Cloudflare announced plans on Monday to launch a marketplace in the next year where website owners can sell AI model providers access to scrape their

2mo ago

9 comments

They're already ignoring robots.txt, so I'm not sure why anyone would think they won't just ignore this too. All they have to do is get a new IP and change their useragent.
- Cloudflare is protecting a lot of sites from scraping with their POW captchas. They could allow people who pay
How can I do this without Cloudflare?
- Put a page on your website saying that scrapping your website costs [insert amount] and block the bots otherwise.
  
  The hard part is reliably detecting the bots

9 comments