# AgentBrush production robots policy - https://agentbrush.dev
#
# This single static file is served on every host (agentbrush.dev and the
# *.pages.dev hosts). Environment separation is enforced by _headers, which
# adds "X-Robots-Tag: noindex" on the *.pages.dev hosts, so staging and the
# Pages project subdomain are not indexed even though they serve this same
# permissive file. Keep this file production-oriented.
#
# AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User,
# PerplexityBot, Google-Extended, etc.) are intentionally NOT blocked: this is
# public marketing/docs content we want AI answer engines to read and cite.

User-agent: *
Allow: /

# Private app + internal surfaces: behind auth or not useful landing pages.
# Disallow keeps crawl budget on public content.
Disallow: /dashboard
Disallow: /billing
Disallow: /account
Disallow: /editor/
Disallow: /sso-callback
Disallow: /device
Disallow: /onboarding

# NOTE: /sign-in and /sign-up are deliberately NOT disallowed. They are publicly
# linked (header CTAs), so to keep them out of the index reliably we let crawlers
# fetch them and serve "<meta robots noindex>" in their prerendered HTML. Google
# never sees a noindex on a robots-disallowed URL, so disallowing them would risk
# bare-URL indexing instead of removing them.

Sitemap: https://agentbrush.dev/sitemap.xml