mirror of
https://github.com/standardebooks/web.git
synced 2025-07-14 10:31:59 -04:00
15 lines
281 B
Text
15 lines
281 B
Text
Sitemap: https://standardebooks.org/sitemap
|
|
|
|
# Badly-behaved bots
|
|
User-agent: *
|
|
Disallow: /honeypot
|
|
|
|
# SEO crawlers
|
|
User-agent: SemrushBot
|
|
User-agent: DotBot
|
|
User-agent: AhrefsBot
|
|
User-agent: SEOkicks
|
|
User-agent: DataForSeoBot
|
|
User-agent: proximic
|
|
|
|
Disallow: /ebooks/*/downloads/*
|