Bookmarks

Robots.txt API

Interesting API to provide robots.txt output to block (known) various web scrapers, including those from AI bots.