MAIA crawlers

MAIA reads public information about the physical world to help people and agents answer questions about places.

MAIABot

MAIABot visits sites served by Cloudflare. Our deployed crawlers sign these requests with Web Bot Auth so sites can verify they came from MAIA.

User-Agent: MAIABot/1.0 (+https://maia-analytics.com/bot)
robots.txt token: MAIABot

MAIAFetch

MAIAFetch visits sites that are not served by Cloudflare. These requests are not signed.

User-Agent: MAIAFetch/1.0 (+https://maia-analytics.com/bot)
robots.txt token: MAIAFetch

Control crawler access

For automated crawling, both crawlers obey robots.txt Disallow rules and crawl-delay. To opt out of automated crawling, disallow both tokens as shown below. When a person directly asks MAIA to read a page, those user-initiated MAIAFetch requests do not apply robots.txt or crawl-delay. A refused request is never retried as the other crawler.

User-agent: MAIABot
Disallow: /

User-agent: MAIAFetch
Disallow: /

Verify MAIABot

Signed requests carry Signature, Signature-Input and Signature-Agent headers. The Signature-Agent is https://api.maia-analytics.com. Read our public keys from the signed key directory on that origin. A User-Agent alone is not proof that a request came from MAIA. Interactive agent-session tools use MAIABot without a signing key; those requests cannot be verified with Web Bot Auth.

MAIABot public key directory

Contact and opt out

For crawler questions or an opt-out request, email info@maia-analytics.com with your domain and the affected URLs.