Our indexer.
How our indexer behaves on the open web — what it identifies as, what it reads, and what it refuses to do.
01 — Who
Who we are
Follow The Market is a price-intelligence service for Australian retailers, operated by Follow The Market Pty Ltd. We index publicly available product pages so merchants can see how their own prices compare with the wider market — without having to babysit dozens of competitor websites by hand.
02 — Identification
Identifying our indexer
Our indexer always identifies itself via the HTTP User-Agent header:
FollowTheMarketIndexer/1.0 (+https://followthemarket.com.au/indexer)
If you operate a site and want to allow or disallow us explicitly, target this user-agent in your robots.txt. We honour the directives we read on every crawl.
03 — Behaviour
What we do
- Read publicly reachable product detail pages.
- Observe price, sale price, and stock status.
- Capture a screenshot as evidence for price changes we report to our customers.
- Respect
robots.txtdirectives, includingCrawl-delay. - Rate-limit our requests to ≤ 1 per 3 seconds per domain by default, and back off automatically on 429 or 503 responses.
04 — Limits
What we don’t do
- We don’t log in or create accounts.
- We don’t bypass CAPTCHAs or TPMs (technological protection measures).
- We don’t harvest personal information, reviews, or seller identities.
- We don’t republish long product descriptions or images.
- We don’t resell raw competitor data to third parties. Each customer sees only the competitors they configured.
05 — Opt out
Opt out
If you operate a site and would prefer we didn’t index it, email bot@followthemarket.com.au with the domain you operate. We’ll remove our indexer within seven days and confirm by return email. You can also block our User-Agent in robots.txt — we read it on every crawl.
Want the upside of all this on your catalogue?
Follow The Market watches every competitor on every line, every day — and turns that into clear, defensible repositions.
See how it works →