Robots monitor

AI crawler rules without guesswork

The local checker reads robots.txt, parses user-agent groups, evaluates public paths, and flags broad root blocks or contradictory allow/disallow rules.

Search and retrieval

OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, and PerplexityBot are treated as discovery-sensitive.

Training and policy

GPTBot, Google-Extended, ClaudeBot, and CCBot are reported as crawler-policy controls.

Sitemaps

Sitemap declarations in robots.txt and `/sitemap.xml` are checked together with crawler access rules.