What Amazonbot Collects Before It Feeds Amazon's AI Models

What is Amazonbot?

Amazon operates Amazonbot as a web crawler that gathers site content to improve the accuracy of Amazon's products and services and may use what it collects to train Amazon AI models, according to Amazon's own Amazonbot documentation at developer.amazon.com, checked 8 September 2026. The bot crawls pages the way any search or AI crawler does, requesting URLs and reading the response, and Amazon states plainly that this is a different purpose from indexing a page for a search result.

What is Amazonbot's user agent string?

Amazonbot identifies itself with the string "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1) Chrome/W.X.Y.Z Safari/537.36", per Amazon's published documentation. The compatible; Amazonbot/0.1 token inside that string is the part a server-side check should match, since the surrounding Mozilla and Chrome segments exist for compatibility with older user agent parsing and appear in many unrelated crawlers. Amazon also publishes the current list of IP addresses the bot crawls from at developer.amazon.com/amazonbot/ip-addresses, so a site can confirm a request claiming to be Amazonbot actually came from Amazon before trusting the user agent string alone.

What does Amazonbot collect, and why?

Amazonbot collects the content of the pages it crawls to improve the accuracy of Amazon's products and services, and Amazon's documentation states the collected data may also train Amazon AI models. Amazon draws a line between Amazonbot and a second, separate crawler named Amzn-SearchBot: Amazon's documentation ties Amzn-SearchBot to search experiences, while Amazonbot is the one Amazon names in connection with AI model training. Whether crawling a page at all is even permitted under copyright and site terms is a separate question covered in is web scraping legal, which applies to any bot pulling content from a site it does not own.

How do you block Amazonbot in robots.txt?

Add a dedicated user agent block naming Amazonbot and disallow the paths you want excluded.

User-agent: Amazonbot
Disallow: /

Amazon's documentation states that Amazonbot respects the Robots Exclusion Protocol, checks the host-level robots.txt file, and caches it for up to 30 days, so a rule change takes roughly 24 hours to take effect and can take longer to fully clear the cache. Amazonbot also honors a noindex, noarchive or none meta tag and rel=nofollow links on pages it does crawl, giving a site owner page-level control in addition to the robots.txt block. Read the full syntax options in the robots.txt guide before writing a rule for any bot by name.

How do you allow Amazonbot, and does it help your site?

Allowing Amazonbot means adding no disallow rule for its user agent, or an explicit allow rule if your robots.txt otherwise blocks crawlers by default. Amazon's documentation states that site owners who allow Amazonbot may qualify for the Amazon Content Partners program, which Amazon describes as offering an affiliate commission boost on eligible sales, hosting credits and AI traffic management tools; check the current program terms on Amazon's own site before assuming eligibility, since program terms change independently of the crawler's documentation.

What is Amazonbot's relationship to Alexa and Rufus?

Amazon's Amazonbot documentation does not name Rufus, Amazon's shopping assistant, so no confirmed link exists in Amazon's own material between the two. The documented connection to a shopping or search assistant runs through Amzn-SearchBot instead: Amazon's documentation states that content crawled by Amzn-SearchBot becomes eligible to appear in search experiences such as Alexa. Treat Amazonbot and Amzn-SearchBot as two separate crawlers with two separate robots.txt rules, since blocking one does not block the other. Applebot-Extended documents the equivalent split on Apple's side, where the crawler that powers search and the token that governs AI training use are also two separate controls.

How do you verify a crawler claiming to be Amazonbot?

Match the request's source IP address against Amazon's published Amazonbot IP address list at developer.amazon.com/amazonbot/ip-addresses before trusting a user agent string alone, since a user agent header is a claim a request makes about itself, not proof of origin. A visitor that sends the Amazonbot user agent string from an IP address outside Amazon's published range is not Amazonbot, regardless of what the header says. Pair that check with the same brand monitoring you would run for any AI system: AI search brand monitoring tracks how a brand is represented once a crawler's data reaches an AI answer, which is the outcome that matters once the crawl itself is done.

Frequently Asked Questions

Does blocking Amazonbot also block Amazon's shopping search results?

Amazon's documentation ties shopping and search experiences like Alexa to Amzn-SearchBot, a separate crawler from Amazonbot, so blocking Amazonbot by name does not touch a rule you have set for Amzn-SearchBot. Write a separate user agent block for each crawler you want to control.

How long does an Amazonbot robots.txt change take to apply?

Amazon's documentation states that Amazonbot checks the host-level robots.txt file and caches it for up to 30 days, and that changes take about 24 hours to apply. Expect a change to apply within a day, though the cache can hold an older version longer in some cases.

Does Amazonbot respect noindex and nofollow?

Yes. Amazon's documentation states Amazonbot respects noindex, noarchive and none meta tags along with rel=nofollow links on pages it crawls, in addition to standard robots.txt disallow rules.

Is Amazonbot the same crawler that powers Rufus?

Amazon's own Amazonbot documentation does not name Rufus, so there is no confirmed, published link between the two. Treat any claim connecting Amazonbot directly to Rufus as unverified until Amazon documents it.

Can I allow Amazonbot only on part of my site?

Yes. A robots.txt Disallow rule under the Amazonbot user agent block accepts specific paths, so you can block a directory like /private/ while leaving the rest of the site open to the same crawler.

Where does Amazon publish Amazonbot's current IP addresses?

At developer.amazon.com/amazonbot/ip-addresses, which Amazon's documentation names as the source for verifying that a request claiming to be Amazonbot actually originated from Amazon's published ranges.

What is the Amazon Content Partners program?

Amazon's documentation states that site owners who allow Amazonbot may qualify for the Amazon Content Partners program. Amazon describes the program as offering an affiliate commission boost on eligible sales, hosting credits, and AI traffic management tools. Program terms change independently of the crawler's documentation, so check Amazon's own site for current eligibility before assuming you qualify.

What happens if I don't add any robots.txt rule for Amazonbot?

No rule means no disallow, which amounts to allowing Amazonbot to crawl your site. Amazon's documentation ties that state to eligibility for the Amazon Content Partners program. The bot then collects page content under its stated purpose of improving the accuracy of Amazon's products and possibly training Amazon AI models.

How is Amazonbot different from a regular search crawler?

Amazon states plainly that Amazonbot's purpose, improving the accuracy of Amazon's products and training Amazon AI models, differs from indexing a page for a search result. Amazon runs a separate crawler, Amzn-SearchBot, for search experiences like Alexa. Blocking or allowing one crawler by name has no effect on the other.

Which Amazon crawler should I block to keep my content out of Amazon's AI training?

Block Amazonbot by name, since Amazon's documentation ties AI model training specifically to Amazonbot and not to Amzn-SearchBot. A dedicated User-agent: Amazonbot block with Disallow: / in robots.txt stops the crawl. Leave Amzn-SearchBot's rule separate if you still want your content in search experiences like Alexa.