SchemaBizBot
Last updated:
SchemaBizBot is the page fetcher of Schema.biz, run by Horizon Two Labs. It fetches a web page when someone asks one of our markup checkers to check it: the Schema Markup Validator, the Schema Audit and the AI Search Optimizer each have a "Check a URL" option. SchemaBizBot downloads the page's HTML and hands it to that person's browser, where the page's structured data is checked.
What it fetches
- Only the page a person asked to check, once per check, and the redirects on the way to it (at most 5).
- It doesn't follow links, crawl other pages, run JavaScript, or download images, scripts or stylesheets.
- It reads at most 5 MB of a page and gives up after 10 seconds.
- It fetches public websites only, over http or https on the standard ports. It refuses addresses on private networks and addresses that lead to private or reserved IP addresses, and checks every redirect the same way.
- If the same address is checked again within 10 minutes, we reuse the copy already fetched instead of asking your server again.
robots.txt
Each fetch is asked for by a person, for one page, so SchemaBizBot doesn't read robots.txt, like other fetchers that act on a user's request. If we ever run an automated crawl, for example for published research, it will honor robots.txt, and we'll describe it on this page before it starts.
How to recognize it
SchemaBizBot sends this user agent:
Mozilla/5.0 (compatible; SchemaBizBot/1.0; +https://schema.biz/bot/)Schema.biz runs on Cloudflare Workers, so requests come from Cloudflare's network rather than from a fixed list of IP addresses.
How to block it
To stop SchemaBizBot fetching your pages, block requests whose user agent contains SchemaBizBot, for example with a firewall rule on your server or CDN. A check of one of your pages will then report the status your server returns, such as 403 Forbidden.
What we keep
The pages SchemaBizBot fetches aren't logged or kept, apart from the 10-minute cache described above. We record the address that was checked, without its query string, along with what the check found, as described in the privacy policy.
Contact
Questions or problems with SchemaBizBot: hello@schema.biz.