AI Crawlers
The AI Crawlers tab in Agent Analytics gives you a page-by-page view of how AI platforms are crawling and evaluating your site. For each indexed page it shows a readiness score, token counts for human and AI audiences, the last crawl time, and any optimization suggestions. Use it to identify which pages are best prepared for AI indexing and which ones need attention.
Getting to AI Crawlers
Agent Analytics lives under Insights in the left sidebar. To open the AI Crawlers tab:
In the left sidebar, click Insights. If the sidebar is collapsed to icons, click the toggle at the top of the sidebar to expand it, or hover over an icon to see its label.
Under Insights, click Agent Analytics.
Click the AI Crawlers tab.
Crawl analysis table
The table lists every page Sona has crawled for your domain, one row per page. Use the domain selector in the top-left to switch between domains, or type a path fragment into the Search bar and click Search to narrow the list. Click any column header to re-sort.
Page: the URL path of the crawled page. Click the path to open a detailed view for that page.
Suggestions: optimization issues Sona detected on the page, such as Missing Canonical URL. Click the tag or the ... overflow to see the full list.
Page Score: an overall readiness score from 0 to 100. A higher score means the page is better structured for AI indexing.
Human Experience Tokens: the number of tokens in the page's content as a human reader would encounter it. A high count indicates content-rich pages.
AI Experience Tokens: the number of tokens in the AI-optimized version of the page's content. Compare this with Human Experience Tokens to see how much content is being surfaced to AI crawlers.
Last Crawled: how recently Sona last crawled this page, for example Past week or about 1 month ago.
Status: the crawl status for this page, for example Completed.
Inclusion: controls whether this page is included in or excluded from AI crawler analysis. Use the Include or Exclude toggle on each row to change the setting.
Tip: click Rediscover URLs in the top-right to prompt Sona to re-scan your sitemap and pick up any new or updated pages.
Adding a domain
Click + Add Domain in the top-right corner to connect an additional website. A modal opens where you enter the domain URL and Sona automatically discovers its sitemap and prepares AI-optimized versions of its pages.
Click + Add Domain in the top-right corner of the AI Crawlers tab.
In the Domain URL field, enter your website address, for example https://acme.com.
Click + Add Domain to confirm.
Note: after adding a domain, Sona automatically discovers your sitemap and begins preparing AI-optimized versions of your pages. The crawl results appear in the table once the initial analysis is complete.
FAQs
What does the Page Score measure?
What does the Page Score measure?
Page Score is a 0–100 readiness rating that reflects how well a page is structured for AI indexing. It factors in elements such as canonical URLs, content clarity, and metadata completeness. Pages with a low score are shown with suggestions in the Suggestions column to help you improve them.
What is the difference between Human Experience Tokens and AI Experience Tokens?
What is the difference between Human Experience Tokens and AI Experience Tokens?
Human Experience Tokens count the tokens in the page as a human visitor would read it, including all standard content. AI Experience Tokens count the tokens in the version of the page that Sona surfaces to AI crawlers. A large gap between the two can indicate that AI crawlers are seeing significantly more or less content than human visitors, which is useful for diagnosing why a page may be over- or under-indexed.
How do I fix a suggestion like Missing Canonical URL?
How do I fix a suggestion like Missing Canonical URL?
Each suggestion in the Suggestions column points to a specific structural issue on that page. For Missing Canonical URL, add a <link rel="canonical"> tag to the page's HTML head pointing to the preferred URL. Click the suggestion tag or the ... overflow for a description of the issue and guidance on how to fix it.
When should I use Rediscover URLs?
When should I use Rediscover URLs?
Use Rediscover URLs after publishing new pages, restructuring your sitemap, or making significant content updates. It prompts Sona to re-scan your sitemap so that new or changed pages appear in the crawl table and are included in the AI optimization analysis.