# https://bedrocktap.com/robots.txt # # POLICY: everything is open to everything. There is not a single Disallow in # this file. This is an agent-native site. # # Every token below has its own group because robots.txt does not inherit: # a crawler obeys the most specific group that matches it and ignores the # rest, including "*". Allowing one token says nothing about its siblings. # # Machine-readable entry points: # llms.txt https://bedrocktap.com/llms.txt # full text https://bedrocktap.com/llms-full.txt # agent guide https://bedrocktap.com/agents.md # MCP server https://bedrocktap.com/mcp (Streamable HTTP, no auth) # MCP server card https://bedrocktap.com/.well-known/mcp.json # OpenAPI https://bedrocktap.com/openapi.json # # Every page has a markdown twin: append .md to any path, # or send Accept: text/markdown. User-agent: * Allow: / # ── Retrieval / AI answer engines ──────────────────────────────────────────── # Googlebot — not Google-Extended — governs Search and AI Overviews eligibility. User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Applebot Allow: / User-agent: DuckAssistBot Allow: / User-agent: Amazonbot Allow: / User-agent: YouBot Allow: / User-agent: PetalBot Allow: / User-agent: Kagibot Allow: / User-agent: Bravebot Allow: / # ── User-triggered fetchers ────────────────────────────────────────────────── # A human asked for this page right now. # (These may ignore robots.txt anyway, precisely because a human initiated it.) User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Agent Allow: / User-agent: Google-GeminiNotebook Allow: / User-agent: MistralAI-User Allow: / User-agent: meta-externalfetcher Allow: / # ── Model training ─────────────────────────────────────────────────────────── # Allowed. To opt out of one vendor's training, change that vendor's line to # "Disallow: /" — it will not affect that vendor's retrieval crawler, which is # a separate token above. User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: CCBot Allow: / User-agent: cohere-ai Allow: / User-agent: meta-externalagent Allow: / User-agent: FacebookBot Allow: / User-agent: Bytespider Allow: / User-agent: Diffbot Allow: / User-agent: Omgilibot Allow: / User-agent: DeepSeekBot Allow: / User-agent: Google-CloudVertexBot Allow: / User-agent: GoogleOther Allow: / # ── Permission tokens, not crawlers ────────────────────────────────────────── # Neither ever appears as a user agent. A firewall rule naming them does # nothing; robots.txt is the only place they carry meaning. User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / Sitemap: https://bedrocktap.com/sitemap.xml