OpenAI crawler robots.txt checker

Separate search, user-triggered, and training-control rules

AICrawlReady checks whether OpenAI-related crawler tokens are left to wildcard rules or handled explicitly, and whether important public paths are blocked from discovery-sensitive agents.

OAI-SearchBot

Reported as discovery-sensitive because site owners may want public pages available for search-oriented retrieval surfaces.

ChatGPT-User

Reported as user-triggered retrieval behavior so important public pages can be reviewed separately from broad wildcard blocks.

GPTBot

Reported as a policy/training-control token so allow or disallow behavior is visible without changing site settings.

What to look for

Common findings include no explicit AI crawler groups, broad `Disallow: /`, contradictory allow/disallow entries, missing sitemap declarations, and blocked docs or policy pages.