The file
An allowlist that ends in a wall
Most robots.txt files are a short list of things not to touch. Trustpilot's is
the opposite shape: a long allowlist of specific crawlers, closing with a blanket refusal for
everyone who isn't on it.
# ~44 named search crawlers get partial access — and are still barred
# from the review pages and every filter the audit actually needs:
User-agent: Googlebot
Disallow: /reviews/
Disallow: /api/*
Disallow: /*?*stars=
Disallow: /*?*sort=
Disallow: /*?*date=
Disallow: /*?*replies=
Disallow: /*?*trustscore=
# AI agents are not left to the catch-all. They are named:
User-agent: anthropic-ai
Disallow: /
User-agent: ClaudeBot
Disallow: /
# …CCBot, cohere-ai, Bytespider, Diffbot — each the same two lines.
# and finally, everyone else:
User-agent: *
Disallow: /
Read the disallowed filters again: stars, sort, date,
replies, trustscore. That is precisely the star histogram, the
recent-review sample, the company's reply rate and the ninety-day trend — the four
inputs an automated reputation audit would want most. Even the crawlers Trustpilot
welcomes cannot have them.
This is not an oversight or a default template. Someone sat down and fenced this data off
from everyone, deliberately, and then went back and named the AI agents one at a time.
api.trustpilot.com/robots.txt is Disallow: / for * as
well. Review pages return 403 to any client that doesn't look like a browser.
A headless browser pointed at that 403 would not be routing around a technical
limit. It would be evading a refusal.
That distinction is the whole decision. Technical limits are problems to solve. A refusal is
an answer to a question you already asked. Our codebase had made the same call once before,
choosing OpenStreetMap over the Google Maps terms for local business data — so there
was precedent, and breaking it for convenience would have meant the earlier decision was
never a principle in the first place.
The alternatives
Four routes, tested in a day, all closed
Obeying a refusal is only a principle if you first check whether it costs
you anything. It cost us a great deal. Here is every route, in the order we tried it.
01 · No key
The official API
The sanctioned route, and the right one. It returns exactly the fenced-off data —
the histogram, the reply rate, the ninety-day delta — to partners who hold a key. Our
client's principal says his firm is a platform partner. The key has not materialised, and
we will not print the partnership claim until the listing does.
02 · Refused
Scraping the review pages
Technically trivial. Explicitly refused, twice over — once by the catch-all and once
by our own user agent's name. Not attempted.
03 · Biased
The prospect's own website
Companies publish their own ratings in structured data so Google can show stars in search
results. Free, permitted, and structurally worthless here: a company marks up its
flattering number and never its embarrassing one. We checked twenty brands shaped like our
targets. Seventeen published no rating at all. The three that did published
app-store scores between 4.70 and 4.87 stars — every one a company we were contacting
precisely because its Trustpilot profile was in trouble.
The route wasn't low-yield, it was biased. A low-yield source gives you
less of the truth. A biased one gives you the opposite of it, at high confidence.
04 · Uncorrelated
The public app-store API
Works perfectly, needs no key, and measures a different universe. One company in our set
carried 4.8 stars across more than 400,000 app-store ratings and
1.6 stars across roughly 1,300 on Trustpilot — in the same week.
People rate an app after it works and rate a company after it refuses a refund.
The trap on the way
The politeness check that reports everything as forbidden
One detail cost an hour and is worth the paragraph, because anyone writing a well-behaved
crawler will hit it.
Python's RobotFileParser.read() fetches robots.txt using urllib's
default user agent. A site behind a CDN answers that request with 403. The
parser interprets the 403 as “this site forbids everything”, sets
disallow_all = True, and from then on every path on that domain reads as refused
— including on sites whose robots.txt actually permits /.
Fix: fetch robots.txt yourself with a real, identifying user agent, then hand
the lines to rp.parse(lines). Never let the fetch and the policy decision live in
the same call you can't see inside.
The failure is silent and it fails in the safe direction, which is exactly why it survives.
Ours was caught only because a site we knew to be open started reading as closed. If you are
building anything that checks permission before it acts, assume your permission check is broken
until you have watched it say yes to something.
The bill
What obeying actually cost
We are not going to pretend this was free. It was expensive, and it is still the binding
constraint on the campaign.
| Item | Count |
|---|
| Companies we could target, with the client's own list | 329 |
| Companies we could have targeted, scraping the platform | unbounded |
| Net new companies the agent has sourced by itself | 0 |
| Campaign status today | supply-constrained |
Every lead file on hand is fully enrolled. The agent can add more
people at companies already approved, and it cannot add a single new company without a
human handing it one. That is not a software problem waiting for a sprint. It is the direct,
ongoing price of the decision at the top of this page.
And the part we didn't expect
The honest sourcing turned out to sell better. The opening email leads with the prospect's own
rating and review count — a stranger telling a Head of Growth a number about their own
company. Some of them check it. When they do, “this is computed from the platform's
own sanctioned data, from a list the client supplied” survives the challenge.
“We scraped it” ends the conversation and starts a different one.
A campaign selling reputation management cannot afford a sourcing story it would rather not
explain. The constraint and the credibility are the same decision, seen from two sides.
What transfers
Three rules we now apply everywhere
Rule 01
Limit or refusal?
Rate limits, pagination and a missing endpoint are problems. A Disallow, a
terms clause, and a 403 aimed at your user agent are answers. Solving the first is
engineering; solving the second is something else, and your customers will eventually ask
which one you did.
Rule 02
Test for bias, not thinness
Self-published data is selected data. Seventeen of twenty companies published nothing and
the three that spoke up were the ones with good news. A source that is silent when the
answer is bad is worse than no source, because it looks like data.
Rule 03
Watch the check say yes
A politeness check that fails closed will quietly refuse the entire internet and look
responsible doing it. Prove it permits something before you trust it forbidding
anything.
The offer
The campaign this came out of
Twenty-one days of cold outbound for a reputation-management firm, run by an autonomous agent:
1,146 emails, six replies, one meeting held, no revenue yet. Published with the full
denominator and the four things that went wrong.
Rick Acquisition — $800/month
Every claim on this page is checkable. The robots.txt is public; read
it yourself. Prospect names are withheld deliberately — they did not consent to appear in
marketing. Trustpilot is a trademark of Trustpilot A/S. This page describes its published
crawling policy, quotes the structure of a public file, and is not affiliated with, sponsored
by or endorsed by Trustpilot.