Agent Analytics

When a user asks an answer engine a question, the engine needs to find information to build its answer. It sends out specialized bots, known as AI crawlers, to read web pages. These include GPTBot for ChatGPT, ClaudeBot for Claude, PerplexityBot, and Google-Extended for Google's AI models.

Your website's robots.txt file is the first gatekeeper. It tells these bots which parts of your site they are allowed to visit. An AI crawler robots.txt check is the process of auditing this file to see which bots you are welcoming in and which you are blocking. It is a foundational piece of technical SEO for AI search. If your key pages are blocked, they cannot be read, and they will not appear as citations in AI answers.

Traditional SEO focused on ranking in a list of ten blue links. AI search, or answer engine optimization, is different. The goal is to be one of the few sources cited in a single, generated paragraph. The engines are selective, and they only cite pages their crawlers have been allowed to read.

Allowing AI crawlers is the first, non-negotiable step. If GPTBot cannot access your blog, ChatGPT cannot learn from it. If Google-Extended is blocked, your content is invisible to Google AI Overviews. This check is not about traffic volume, it is about eligibility. You are checking your ticket into the arena where AI answers are written.

Agent Analytics, found at /dashboard/crawlers, gives you a technical readout focused on AI search. It works by reading your site from the outside, the same way an AI bot would. It checks your robots.txt and any llms.txt file you have set up, then reports what it finds.

The screen is organized into three areas. AI crawler visibility shows a list of known AI bots, like GPTBot and ClaudeBot, and states clearly whether your current rules allow or disallow them. Crawlability runs a simulated fetch of your pages, reporting on response times, blocked resources, and how the page renders without JavaScript, highlighting issues that could make your content unreadable. Content performance tracking shows which of your pages are being indexed by AI-relevant services and gathers data on anchor text pointing to your site from across the web.

For a marketer, this is a diagnostic and monitoring tool. A typical workflow starts with an audit. Open Agent Analytics and review the AI crawler visibility list. For a new product launch site, you would check that all major bots are allowed on your key landing and documentation pages.

Next, use the crawlability report. Filter to a high-priority page, like your flagship service description. Look for warnings about blocked CSS or JavaScript files that might prevent the page from rendering correctly for a bot. Fix those issues in your code. Finally, check content performance weekly. Look for new pages that have entered the 'indexed' list, indicating they are now being picked up by AI services, and review the anchor text report to see how other sites are describing your brand when they link to you.

A green 'Allowed' status for GPTBot means your robots.txt permits it. It does not mean GPTBot has visited, or that your content will be cited. It simply means the door is open. A slow response time in the crawlability report is a direct problem, as bots have time limits and may abandon slow pages.

The 'indexed coverage' metric shows which pages systems know about. An increase here is a positive signal of growing AI visibility. The anchor text report is useful for understanding brand perception. A shift from 'software tool' to 'AI marketing platform' in how others link to you reflects a changing market narrative. Remember, these are leading indicators, not promises of traffic or citations.

It is important to state the limitations clearly. Agent Analytics reads your site from the public internet. It does not ingest your server logs. Server-log ingestion would show you exactly which bots visited, how often, and what they fetched. That level of detail is not built.

The feature also cannot benchmark your site against others. To say your crawlability is in the 80th percentile requires a large population of comparable sites to measure against. We do not have that population, and we would rather show nothing than provide a misleading percentile drawn from a sample of one.

Free includes one saved snapshot of up to 3 questions and 3 pages. No card. No expiry. Lite $29/month · Growth $99/month · Agency $249/month (USD).

Growth: 50 questions · 40 shared draft and agent credits/month · $99/month (USD).