Our Indexer.

01 — Who

Who we are

Follow The Market is a price-intelligence service for Australian retailers, operated by Follow The Market Pty Ltd. We index publicly available product pages so merchants can see how their own prices compare with the wider market — without having to babysit dozens of competitor websites by hand.

02 — Identification

Identifying our indexer

Our indexer always identifies itself via the HTTP User-Agent header:

FollowTheMarketIndexer/1.0 (+https://followthemarket.com.au/indexer)

If you operate a site and want to allow or disallow us explicitly, target this user-agent in your robots.txt. We read it on every crawl and honour what it says — see “What we do” below for the one recorded exception.

03 — Behaviour

What we do

• Read publicly reachable product detail pages.
• Observe price, sale price, and stock status.
• Capture a screenshot as evidence for price changes we report to our customers.
• Read every site’s robots.txt on every crawl and honour its pacing directives — including Crawl-delay — always.
• Honour Disallow directives as written, with one recorded exception: where a site’s robots.txt is the unmodified stock file its CMS shipped with — a blanket Disallow: / nobody at that business ever wrote — we may index that site under a per-site override. The exception is only available where a dated copy of the file itself, its Last-Modified header, and the reasoning are kept on record (and our software refuses the override where the recorded file doesn’t actually disallow us). A robots.txt anyone actually wrote — or edits — is followed.
• Rate-limit our requests to ≤ 1 per 3 seconds per domain by default, and back off automatically on 429 or 503 responses.

04 — Limits

What we don’t do

• We don’t log in or create accounts.
• We don’t bypass CAPTCHAs or TPMs (technological protection measures).
• We don’t harvest personal information, reviews, or seller identities.
• We don’t republish long product descriptions or images.
• We don’t resell raw competitor data to third parties. Each customer sees only the competitors they configured.

05 — Opt out

Opt out

If you operate a site and would prefer we didn’t index it, email bot@followthemarket.com.au with the domain you operate. We’ll remove our indexer within seven days and confirm by return email. You can also block our User-Agent in robots.txt — we read it on every crawl, and any Disallow a person actually wrote is honoured (the only exception, above, is for untouched CMS default files — and editing yours takes your site out of that exception).