# dontfeedmachines.com # full terms: /ai.txt ยท machine readable reservation: /.well-known/tdmrep.json # # the DFM toolkit at dfm.tools and the corpus at /leash are deliberately # unreserved. they are Apache 2.0 and CC0 respectively and are allowed below # even to the agents that are otherwise blocked. everything else is reserved. # # agent names go stale. the current community list lives at # https://rsrc.dontfeedmachines.com/docs/crawlers.txt User-agent: GPTBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: OAI-SearchBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: ChatGPT-User Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: CCBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Google-Extended Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: ClaudeBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: anthropic-ai Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Claude-Web Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Applebot-Extended Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Bytespider Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: PerplexityBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Amazonbot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: meta-externalagent Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: FacebookBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Diffbot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: cohere-ai Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: omgili Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: omgilibot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: ImagesiftBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: img2dataset Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: PetalBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: YouBot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Timpibot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: AI2Bot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Kangaroo Bot Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: Scrapy Allow: /dfm-toolkit Allow: /leash Disallow: / User-agent: * Disallow: