S SCHEMA.BIZ

AI Search Tools

Free AI search tools for what AI assistants and answer engines can read on your site: which of their crawlers may fetch it, and the llms.txt file some of them look for.

How AI search reads a site

AI assistants and answer engines such as ChatGPT search, Claude, Perplexity and Google's AI features read the web in three ways. Some crawlers collect pages to train models. Some index pages so an assistant can cite them in an answer, the way a search engine does. And some fetchers load one page because a person asked about it. Each of these has its own user agent, and each company documents which robots.txt token controls which job, so a site can welcome AI search while keeping its pages out of model training, or the other way round.

Once a crawler can fetch a page, what it understands comes from the page itself: its text, its title and headings, and its structured data. Nothing here promises a citation or a ranking. No AI company publishes the rules it uses to choose sources, and what the tools check are the parts you control and that are documented: access, directives and markup.

Checking one page

The AI readiness checker takes a page's address and reports on everything documented that decides whether AI search can use it: whether the page loads, whether Google may index it and show text from it, whether Googlebot, Bingbot and each AI search crawler may fetch it, and whether its structured data is valid and says who's behind it. It scores those, lists the top fixes, and notes the site's training-crawler rules and llms.txt without scoring them.

AI crawlers and robots.txt

Robots.txt is where a site says which crawler may fetch what. The robots.txt checker tests a site's file against a list of AI crawler tokens, from OpenAI's GPTBot and OAI-SearchBot to Google-Extended and Applebot-Extended, each checked on its vendor's own documentation, and shows which are blocked and by which line. Some fetchers that act for a person say robots.txt may not apply to them; the list marks those.

llms.txt

llms.txt is a proposed convention, published at llmstxt.org in September 2024: a Markdown file at a site's root with the site's name, a one-line summary and lists of links to its most useful pages, written for language models rather than people. It's a proposal, not a standard. Support from the major AI providers is limited or unconfirmed: we haven't found one that documents reading it. It is cheap to add, so some sites publish one as an optional step; the guide to llms.txt explains the format and its limits. The llms.txt generator drafts one from a site's sitemap and pages for you to edit, and the llms.txt validator checks a file against the format, line by line.

Structured data for AI search

Structured data names what a page is about in a form a machine reads without guessing: the organization behind it, the author, the product and its price, the dates. The AI Search Optimizer scores a page's markup on those signals, and the Schema Markup Validator checks that it's valid.