### UW DGH | SEO Basics ### ### placeholder robots.txt ### User-Agent: * Allow: /wp-content/uploads/ Disallow: /wp-content/plugins/ Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /readme.html Disallow: /refer/ Sitemap: https://www.go2itech.org/uwdgh-seo-basics-sitemap.xml ### START Disallowed AI bots ### # AI Agents User-agent: AmazonBuyForMe # https://knownagents.com/agents/amazonbuyforme Disallow: / User-agent: ChatGPT Agent # https://knownagents.com/agents/chatgpt-agent Disallow: / User-agent: GoogleAgent-Mariner # https://knownagents.com/agents/googleagent-mariner Disallow: / User-agent: Manus-User # https://knownagents.com/agents/manus-user Disallow: / User-agent: NovaAct # https://knownagents.com/agents/novaact Disallow: / User-agent: TwinAgent # https://knownagents.com/agents/twinagent Disallow: / # AI Assistants User-agent: AI2Bot-DeepResearchEval # https://knownagents.com/agents/ai2bot-deepresearcheval Disallow: / User-agent: Amzn-User # https://knownagents.com/agents/amzn-user Disallow: / User-agent: bigsur.ai # https://knownagents.com/agents/bigsur-ai Disallow: / User-agent: ChatGPT-User # https://knownagents.com/agents/chatgpt-user Disallow: / User-agent: Claude-User # https://knownagents.com/agents/claude-user Disallow: / User-agent: Devin # https://knownagents.com/agents/devin Disallow: / User-agent: DuckAssistBot # https://knownagents.com/agents/duckassistbot Disallow: / User-agent: Gemini-Deep-Research # https://knownagents.com/agents/gemini-deep-research Disallow: / User-agent: Google-NotebookLM # https://knownagents.com/agents/google-notebooklm Disallow: / User-agent: kagi-fetcher # https://knownagents.com/agents/kagi-fetcher Disallow: / User-agent: KlaviyoAIBot # https://knownagents.com/agents/klaviyoaibot Disallow: / User-agent: LinerBot # https://knownagents.com/agents/linerbot Disallow: / User-agent: meta-externalfetcher # https://knownagents.com/agents/meta-externalfetcher Disallow: / User-agent: Meta-ExternalFetcher Disallow: / User-agent: MistralAI-User # https://knownagents.com/agents/mistralai-user Disallow: / User-agent: Perplexity-User # https://knownagents.com/agents/perplexity-user Disallow: / User-agent: PhindBot # https://knownagents.com/agents/phindbot Disallow: / User-agent: Poggio-Citations # https://knownagents.com/agents/poggio-citations Disallow: / User-agent: QualifiedBot # https://knownagents.com/agents/qualifiedbot Disallow: / # AI Search Crawlers User-agent: AddSearchBot # https://knownagents.com/agents/addsearchbot Disallow: / User-agent: Amzn-SearchBot # https://knownagents.com/agents/amzn-searchbot Disallow: / User-agent: Anomura # https://knownagents.com/agents/anomura Disallow: / User-agent: Applebot # https://knownagents.com/agents/applebot Disallow: / User-agent: atlassian-bot # https://knownagents.com/agents/atlassian-bot Disallow: / User-agent: AzureAI-SearchBot # https://knownagents.com/agents/azureai-searchbot Disallow: / User-agent: Bravebot # https://knownagents.com/agents/bravebot Disallow: / User-agent: Channel3Bot # https://knownagents.com/agents/channel3bot Disallow: / User-agent: Claude-SearchBot # https://knownagents.com/agents/claude-searchbot Disallow: / User-agent: Cloudflare-AutoRAG # https://knownagents.com/agents/cloudflare-autorag Disallow: / User-agent: ExaBot # https://knownagents.com/agents/exabot Disallow: / User-agent: Google-CloudVertexBot # https://knownagents.com/agents/google-cloudvertexbot Disallow: / User-agent: LinkupBot # https://knownagents.com/agents/linkupbot Disallow: / User-agent: meta-webindexer # https://knownagents.com/agents/meta-webindexer Disallow: / User-agent: OAI-SearchBot # https://knownagents.com/agents/oai-searchbot Disallow: / User-agent: PerplexityBot # https://knownagents.com/agents/perplexitybot Disallow: / User-agent: PetalBot # https://knownagents.com/agents/petalbot Disallow: / User-agent: YouBot # https://knownagents.com/agents/youbot Disallow: / User-agent: ZanistaBot # https://knownagents.com/agents/zanistabot Disallow: / # AI Data Scrapers User-agent: Ai2Bot-Dolma # https://knownagents.com/agents/ai2bot-dolma Disallow: / User-agent: Amazonbot # https://knownagents.com/agents/amazonbot Disallow: / User-agent: ApifyWebsiteContentCrawler # https://knownagents.com/agents/apifywebsitecontentcrawler Disallow: / User-agent: Applebot-Extended # https://knownagents.com/agents/applebot-extended Disallow: / User-agent: Bytespider # https://knownagents.com/agents/bytespider Disallow: / User-agent: CCBot # https://knownagents.com/agents/ccbot Disallow: / User-agent: ChatGLM-Spider # https://knownagents.com/agents/chatglm-spider Disallow: / User-agent: ClaudeBot # https://knownagents.com/agents/claudebot Disallow: / User-agent: CloudVertexBot # https://knownagents.com/agents/cloudvertexbot Disallow: / User-agent: cohere-training-data-crawler # https://knownagents.com/agents/cohere-training-data-crawler Disallow: / User-agent: Cotoyogi # https://knownagents.com/agents/cotoyogi Disallow: / User-agent: Datenbank Crawler # https://knownagents.com/agents/datenbank-crawler Disallow: / User-agent: Diffbot # https://knownagents.com/agents/diffbot Disallow: / User-agent: FacebookBot # https://knownagents.com/agents/facebookbot Disallow: / User-agent: FirecrawlAgent # https://knownagents.com/agents/firecrawlagent Disallow: / User-agent: Google-Extended # https://knownagents.com/agents/google-extended Disallow: / User-agent: GoogleOther # https://knownagents.com/agents/googleother Disallow: / User-agent: GPTBot # https://knownagents.com/agents/gptbot Disallow: / User-agent: ICC-Crawler # https://knownagents.com/agents/icc-crawler Disallow: / User-agent: imageSpider # https://knownagents.com/agents/imagespider Disallow: / User-agent: Kangaroo Bot # https://knownagents.com/agents/kangaroo-bot Disallow: / User-agent: laion-huggingface-processor # https://knownagents.com/agents/laion-huggingface-processor Disallow: / User-agent: LCC # https://knownagents.com/agents/lcc Disallow: / User-agent: meta-externalagent # https://knownagents.com/agents/meta-externalagent Disallow: / User-agent: Meta-ExternalAgent Disallow: / User-agent: netEstate Imprint Crawler # https://knownagents.com/agents/netestate-imprint-crawler Disallow: / User-agent: omgili # https://knownagents.com/agents/omgili Disallow: / User-agent: PanguBot # https://knownagents.com/agents/pangubot Disallow: / User-agent: SBIntuitionsBot # https://knownagents.com/agents/sbintuitionsbot Disallow: / User-agent: Spider # https://knownagents.com/agents/spider Disallow: / User-agent: Timpibot # https://knownagents.com/agents/timpibot Disallow: / User-agent: VelenPublicWebCrawler # https://knownagents.com/agents/velenpublicwebcrawler Disallow: / User-agent: webzio-extended # https://knownagents.com/agents/webzio-extended Disallow: / User-agent: Webzio-Extended Disallow: / # Undocumented AI Agents User-agent: anthropic-ai # https://knownagents.com/agents/anthropic-ai Disallow: / User-agent: ApifyBot # https://knownagents.com/agents/apifybot Disallow: / User-agent: Aranet-SearchBot # https://knownagents.com/agents/aranet-searchbot Disallow: / User-agent: Claude-Web # https://knownagents.com/agents/claude-web Disallow: / User-agent: cohere-ai # https://knownagents.com/agents/cohere-ai Disallow: / User-agent: Crawl4AI # https://knownagents.com/agents/crawl4ai Disallow: / User-agent: DeepSeekBot # https://knownagents.com/agents/deepseekbot Disallow: / User-agent: iAskBot # https://knownagents.com/agents/iaskbot Disallow: / User-agent: iaskspider # https://knownagents.com/agents/iaskspider Disallow: / User-agent: iaskspider/2.0 Disallow: / User-agent: KunatoCrawler # https://knownagents.com/agents/kunatocrawler Disallow: / User-agent: TavilyBot # https://knownagents.com/agents/tavilybot Disallow: / User-agent: TerraCotta # https://knownagents.com/agents/terracotta Disallow: / User-agent: WRTNBot # https://knownagents.com/agents/wrtnbot Disallow: / # AI & LLM Search Crawlers CONT'D # sourced from: https://habeasdata.neocities.org/ai-bots, https://www.nytimes.com/robots.txt, https://github.com/ai-robots-txt/ai.robots.txt/blob/main/haproxy-block-ai-bots.txt User-agent: img2dataset Disallow: / User-agent: sider.ai Disallow: / User-agent: Omgilibot Disallow: / User-agent: bedrockbot Disallow: / User-agent: MyCentralAIScraperBot Disallow: / User-agent: aiHitBot Disallow: / User-agent: Andibot Disallow: / User-agent: AudigentAdBot Disallow: / User-agent: AudigentBot Disallow: / User-agent: Awario Disallow: / User-agent: AwarioRssBot Disallow: / User-agent: AwarioSmartBot Disallow: / User-agent: Brightbot 1.0 Disallow: / User-agent: BuddyBot Disallow: / User-agent: Echobot Bot Disallow: / User-agent: EchoboxBot Disallow: / User-agent: facebookexternalhit Disallow: / User-agent: Factset_spyderbot Disallow: / User-agent: FriendlyCrawler Disallow: / User-agent: Google-Firebase Disallow: / User-agent: GoogleOther-Image Disallow: / User-agent: GoogleOther-Video Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: ISSCyberRiskCrawler Disallow: / User-agent: LAIONDownloader Disallow: / User-agent: Linguee Bot Disallow: / User-agent: NotebookLM Disallow: / User-agent: OpenAI Disallow: / User-agent: Panscient Disallow: / User-agent: panscient.com Disallow: / User-agent: Poseidon Research Crawler Disallow: / User-agent: QuillBot Disallow: / User-agent: quillbot.com Disallow: / User-agent: Scrapy Disallow: / User-agent: SemrushBot-OCOB Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: Sidetrade indexer bot Disallow: / User-agent: Thinkbot Disallow: / User-agent: TikTokSpider Disallow: / User-agent: WARDBot Disallow: / User-agent: wpbot Disallow: / User-agent: YaK Disallow: / User-agent: YandexAdditional Disallow: / User-agent: YandexAdditionalBot Disallow: / ### END Disallowed AI bots ###