mirror of
https://github.com/standardebooks/web.git
synced 2025-07-15 02:46:46 -04:00
12 lines
233 B
Text
12 lines
233 B
Text
Sitemap: https://standardebooks.org/sitemap
|
|
|
|
# Badly-behaved bots
|
|
User-agent: *
|
|
Disallow: /honeypot
|
|
|
|
# SEO crawlers
|
|
User-agent: SemrushBot
|
|
User-agent: DotBot
|
|
User-agent: AhrefsBot
|
|
User-agent: SEOkicks
|
|
Disallow: /ebooks/*/downloads/*
|