AI crawler access observation

AI crawler access, directives, and visibility monitor

Check whether AI crawlers can access, understand, or are blocked by your site directives. Focused on robots.txt, GPTBot, OAI-SearchBot, ClaudeBot, AI crawler directives, and referral visibility risk.

87 average score across 6 portfolio sites
0 hard issues in the first portfolio run
19 readiness warnings found
View crawler access report View sample report

Start an access scan

The public page records intent only. The current checker runs controlled single-URL checks and does not expose an unrestricted crawler endpoint.

Sample AICrawlReady report preview with score, crawler matrix, llms.txt status, and next fixes.

First local run

The first portfolio scan found repeated crawler directive gaps.

  • Explicit AI crawler policywarning
  • AI referral visibility signalsmixed
  • Sitemap discoverypass
  • JSON-LD coveragemixed
  • Entry-page fetchpass

View the public sample report

Coverage

Crawler access checks that belong together

Open llms.txt

Robots and AI agents

GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, Claude, Perplexity, and CCBot rules are evaluated against public paths.

AI crawler directives

Checks explicit GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, and wildcard behavior before relying on default access.

Discovery and visibility

Sitemap, canonical, meta description, JSON-LD, noindex/nofollow conflicts, important links, and readable content are reported as visibility risk signals.

Readiness pages

Common checks for AI search surfaces

AI search readiness

Entry-page, sitemap, canonical, metadata, structured data, and readable-content checks in one local report.

Open page

OpenAI crawler rules

Separate checks for search/retrieval and training-control user-agent tokens in robots.txt.

Open page

Google-Extended

Report whether Google-Extended is left to wildcard behavior or handled by an explicit robots.txt group.

Open page

Monitoring intent

Recurring readiness is the paid hypothesis.

The kill metric is strict: pause before more spend if 300 targeted visits or 100 checker runs produce fewer than 5 monitor or report intents.

Support intake

Corrections and product questions

This intake is for product feedback, correction requests, and privacy questions. It is not a done-for-you SEO or crawler setup service.