Crawler

CompareBrandsBot

Last updated: October 8, 2026. How CompareBrands' web crawler identifies itself, behaves, and can be controlled.

At a glance

User-agent token

CompareBrandsBot

Full user-agent

CompareBrandsBot/1.0 (+https://comparebrands.com/bot)

Request method

GET only

Pace

One request at a time per site, at least 1.5 seconds apart

robots.txt

Always read first and obeyed (RFC 9309)

What CompareBrandsBot is

CompareBrandsBot is the automated fetcher operated by CompareBrands. It reads publicly available product pages and product sitemaps from manufacturer and retailer websites so that CompareBrands can list products and their published specifications accurately.

It identifies itself in every request. It is not used to train general-purpose AI models, and it does not collect personal data about site visitors.

What it does and does not do

• Requests public pages and sitemaps with plain HTTP GET; it never submits forms, signs in, or bypasses a login, paywall, or CAPTCHA.

• Fetches only HTML, XML, JSON, and plain-text responses; other content types are skipped.

• Stops reading any response that exceeds a fixed size limit.

• Follows a limited number of redirects and re-checks robots.txt and the destination address at every hop.

• Refuses to fetch private, loopback, or internal network addresses.

• Sends conditional requests (If-None-Match, If-Modified-Since) so unchanged pages are not downloaded again.

How it respects your site

• Reads your robots.txt before any other request and obeys the rules for the CompareBrandsBot user-agent, falling back to the * group.

• Honors Crawl-delay when you set one; otherwise waits at least 1.5 seconds between requests to the same site.

• Uses a single connection per site, so it never fetches in parallel from your server.

• Backs off automatically on HTTP 429, 502, 503, and 504, and honors Retry-After.

• Pauses a site entirely after repeated 403 or 429 responses.

• If your robots.txt cannot be retrieved, it treats the site as off-limits instead of guessing.

Control or block access

To keep CompareBrandsBot out of your whole site, or only part of it, add a rule to your robots.txt. Changes are picked up the next time the file is read, before the next page request.

User-agent: CompareBrandsBot

Disallow: /

Verifying the crawler

Genuine requests carry the user-agent shown above and a link back to this page. A user-agent string alone can be forged. We do not currently publish a fixed list of source IP addresses, so if you need to confirm that a request came from us, email support@comparebrands.com with the time, the path requested, and the source IP, and we will confirm.

Questions, removal requests, and problems

If the crawler is causing load, reaching pages it should not, or you want data removed, email support@comparebrands.com with your domain and a short description. We aim to respond within five business days and will pause crawling of your site while we investigate.

Crawler questions go to support@comparebrands.com.