standardebooks/web
1
0
Fork 0
mirror of https://github.com/standardebooks/web.git synced 2025-07-13 10:02:02 -04:00
Code Issues Projects Releases Packages Wiki Activity Actions

Update robots.txt to disallow downloads by SEO bots

Browse source
This commit is contained in:
Alex Cabal 2025-05-22 13:24:57 -05:00
parent 475c437126
commit f6aef54767
1 changed files with 8 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

8
www/robots.txt
View file

@ -1,4 +1,12 @@
Sitemap: https://standardebooks.org/sitemap Sitemap: https://standardebooks.org/sitemap
# Badly-behaved bots
User-agent: * User-agent: *
Disallow: /honeypot Disallow: /honeypot
# SEO crawlers
User-agent: SemrushBot
User-agent: DotBot
User-agent: AhrefsBot
User-agent: SEOkicks
Disallow: /ebooks/*/downloads/*