AmazonAffiliateFinderBot

What the Amazon Affiliate Finder crawler does when it reaches your site, and how to keep it out.

User agent

Every request identifies itself with this string, so it can be found in your logs and matched by a rule:

AmazonAffiliateFinderBot/1.0 (+https://amazonaffiliatefinder.com/bot)

What it fetches

The crawler is driven by a search, not by a list of sites. Someone enters a product category, and that category goes to a public web search engine. The crawler opens the public pages the search returns, one page per site, and counts the product links to Amazon on each one. A page with no such link is dropped and that site is not read further.

When the search is on a plan that includes contacts, the crawler visits that same site for the addresses it publishes. It starts at the home page and follows the contact, about, team, author, press and legal pages the site links to on its own host, up to the page cap below. It never leaves the host it is reading.

It reads only public pages that our own search found. It does not search the web on its own, and it does not guess an address that no page carries.

How often it requests

• One request at a time per host, with at least 1 second between two requests to the same host.
• If your robots.txt sets a Crawl-delay, the crawler waits that long up to 10 seconds. A longer value is capped there, so one site cannot hold a crawl open forever.

How much it reads

• At most 25 pages on one site, counting the pages it opens for contact addresses.
• It reads no page larger than 2 MB, and stops waiting on a page that has not answered within 10 seconds.
• It follows at most 5 redirects, and every hop goes through the same safety check that keeps requests off private and loopback addresses.

robots.txt

The crawler obeys robots.txt for every host it visits. It reads the file before the first page, and applies it to every URL it considers on that host, so a page the file disallows is not fetched. A ranking page that robots.txt keeps the crawler off is not read, so that site does not become a row in the result.

How to block it

Add these two lines to your robots.txt:

User-agent: AmazonAffiliateFinderBot
Disallow: /

With those lines in place the crawler reads your robots.txt to find the rule, and fetches nothing else on your site. You can also refuse the user agent string at your server or firewall, and the opt-out list keeps a single address out of the results even when the page it sits on is public.

Bot walls and protected addresses

The crawler never works around protection. If a site answers with a bot challenge or a block page, the crawl stops and that site is left out. There is no headless trick, no different user agent and no retry with a changed address to get past that answer.

The same applies to an address a site has chosen to hide. When a page carries an address behind email protection, or writes it in a scrambled form that only a script can rebuild, the crawler leaves it alone: we do not decode it, and it does not appear in a result. Hiding an address is read as a decision and respected.

Contact

Questions about a crawl, or to tell us the crawler did something on your site that this page does not describe: [email protected].