Google-Extended robots.txt checker

Make Google-Extended policy visible in the readiness report

AICrawlReady reports whether Google-Extended has an explicit robots.txt group, whether wildcard rules apply, and whether the rest of the public discovery stack is configured.

Explicit policy

Detects a `User-agent: Google-Extended` group and reports broad allow or broad disallow behavior.

Wildcard inheritance

Flags when Google-Extended inherits from `User-agent: *`, making the policy less obvious to future maintainers.

Discovery context

Pairs the policy check with sitemap, canonical, JSON-LD, noindex, and server-rendered content checks.

Monitoring value

The local snapshot mode can show when a robots.txt policy changes after a deploy, CDN rewrite, static export, or hosting migration.