tonytins/barkpack-lists
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Added allowlist.txt and robots.txt

Browse source
This commit is contained in:
Tony Bark 2025-02-17 12:33:48 -05:00
parent eebee18ccc
commit 61921ae7fc
3 changed files with 73 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

28
robots.txt Normal file
View file

@ -0,0 +1,28 @@
User-Agent: GPTBot
User-Agent: ChatGPT-User
User-Agent: ClaudeBot
User-Agent: anthropic-ai
User-Agent: CCBot
User-agent: Bytespider
User-Agent: FacebookBot
User-Agent: Twitterbot
User-Agent: PerplexityBot
User-Agent: OmigiliBot
User-Agent: AISearchBot
User-Agent: mlbot
User-Agent: PetalBot
User-Agent: Amazonbot
User-Agent: DuckAssistBot
User-agent: Diffbot
User-Agent: OAI-SearchBot
User-Agent: OAI-SearchBot
User-Agent: omgili
User-Agent: omgilibot
User-Agent: PerplexityBot
User-Agent: Webzio-Extended
User-Agent: Meta-ExternalAgent
User-Agent: Meta-ExternalFetcher
User-Agent: cohere-ai
User-agent: Google-Extended
User-agent: PerplexityBot
Disallow: /