Trustpilot bans AI agents by name. We obeyed it.

The file

An allowlist that ends in a wall

Most robots.txt files are a short list of things not to touch. Trustpilot's is the opposite shape: a long allowlist of specific crawlers, closing with a blanket refusal for everyone who isn't on it.

# www.trustpilot.com/robots.txt — structure as read 2026-09-28. 44 user-agents named. # Tier 1: 34 named crawlers get partial access, and are still barred from # the review pages and every filter the audit actually needs. Googlebot is # here — and so are ChatGPT-User, ClaudeBot, GPTBot, Google-Extended, OAI-SearchBot, PerplexityBot: User-agent: ClaudeBot Disallow: /reviews/ Disallow: /api/* Disallow: /*?*stars= Disallow: /*?*sort= Disallow: /*?*date= Disallow: /*?*replies= Disallow: /*?*trustscore= # Tier 2: 9 AI crawlers are named only to be refused the whole site: User-agent: anthropic-ai Disallow: / # …Applebot-Extended, Bytespider, CCBot, Diffbot, ImagesiftBot, Meta-ExternalAgent, Omgilibot, cohere-ai — each the same two lines. # Tier 3: everyone else. User-agent: * Disallow: /

Read the disallowed filters again: stars, sort, date, replies, trustscore. That is precisely the star histogram, the recent-review sample, the company's reply rate and the ninety-day trend — the four inputs an automated reputation audit would want most. The AI crawlers Trustpilot half-admits sit in the same tier as Googlebot and cannot have them either. There is no user-agent string under which the review data is permitted. The rest of us are simply refused at the door.

This is not an oversight or a default template. Someone sat down and fenced this data off from everyone, deliberately, and then went back and named the AI agents one at a time. api.trustpilot.com/robots.txt is Disallow: / for * as well. Review pages return 403 to any client that doesn't look like a browser.

A headless browser pointed at that 403 would not be routing around a technical limit. It would be evading a refusal.

That distinction is the whole decision. Technical limits are problems to solve. A refusal is an answer to a question you already asked. Our codebase had made the same call once before, choosing OpenStreetMap over the Google Maps terms for local business data — so there was precedent, and breaking it for convenience would have meant the earlier decision was never a principle in the first place.

The alternatives

Four routes, tested in a day, all closed

Obeying a refusal is only a principle if you first check whether it costs you anything. It cost us a great deal. Here is every route, in the order we tried it.

01 · No key

The official API

The sanctioned route, and the right one. It returns exactly the fenced-off data — the histogram, the reply rate, the ninety-day delta — to partners who hold a key. Our client's principal says his firm is a platform partner. The key has not materialised, and we will not print the partnership claim until the listing does.

02 · Refused

Scraping the review pages

Technically trivial. Explicitly refused, twice over — once by the catch-all and once by anthropic-ai, our own vendor's name. And for the review data itself there is no permitted route at all, whatever we called ourselves. Not attempted.

03 · Biased

The prospect's own website

Companies publish their own ratings in structured data so Google can show stars in search results. Free, permitted, and structurally worthless here: a company marks up its flattering number and never its embarrassing one. We checked 20 brands shaped like our targets. 17 published no rating at all. The rest published app-store scores between 4.70 and 4.87 stars — every one a company we were contacting precisely because its Trustpilot profile was in trouble.

The route wasn't low-yield, it was biased. A low-yield source gives you less of the truth. A biased one gives you the opposite of it, at high confidence.

04 · Uncorrelated

The public app-store API

Works perfectly, needs no key, and measures a different universe. One company in our set carried 4.8 stars across more than 400,000 app-store ratings and 1.6 stars across roughly 1,300 on Trustpilot — in the same week. People rate an app after it works and rate a company after it refuses a refund.

The trap on the way

The politeness check that reports everything as forbidden

One detail cost an hour and is worth the paragraph, because anyone writing a well-behaved crawler will hit it.

Python's RobotFileParser.read() fetches robots.txt using urllib's default user agent. A site behind a CDN answers that request with 403. The parser interprets the 403 as “this site forbids everything”, sets disallow_all = True, and from then on every path on that domain reads as refused — including on sites whose robots.txt actually permits /.

Fix: fetch robots.txt yourself with a real, identifying user agent, then hand the lines to rp.parse(lines). Never let the fetch and the policy decision live in the same call you can't see inside.

The failure is silent and it fails in the safe direction, which is exactly why it survives. Ours was caught only because a site we knew to be open started reading as closed. If you are building anything that checks permission before it acts, assume your permission check is broken until you have watched it say yes to something.

The bill

What obeying actually cost

We are not going to pretend this was free. It was expensive, and it is still the binding constraint on the campaign.

Item	Count
Companies we could target, with the client's own list	329
Companies we could have targeted, scraping the platform	unbounded
Net new companies the agent has sourced by itself	0
Campaign status today	supply-constrained

Every lead file on hand is fully enrolled. The agent can add more people at companies already approved, and it cannot add a single new company without a human handing it one. That is not a software problem waiting for a sprint. It is the direct, ongoing price of the decision at the top of this page.

And the part we didn't expect

The honest sourcing turned out to sell better. The opening email leads with the prospect's own rating and review count — a stranger telling a Head of Growth a number about their own company. Some of them check it. When they do, “this is computed from the platform's own sanctioned data, from a list the client supplied” survives the challenge. “We scraped it” ends the conversation and starts a different one.

A campaign selling reputation management cannot afford a sourcing story it would rather not explain. The constraint and the credibility are the same decision, seen from two sides.

What transfers

Three rules we now apply everywhere

Rule 01

Limit or refusal?

Rate limits, pagination and a missing endpoint are problems. A Disallow, a terms clause, and a 403 aimed at your user agent are answers. Solving the first is engineering; solving the second is something else, and your customers will eventually ask which one you did.

Rule 02

Test for bias, not thinness

Self-published data is selected data. Most of the companies we checked published nothing and the ones that spoke up were the ones with good news. A source that is silent when the answer is bad is worse than no source, because it looks like data.

Rule 03

Watch the check say yes

A politeness check that fails closed will quietly refuse the entire internet and look responsible doing it. Prove it permits something before you trust it forbidding anything.

The offer

The campaign this came out of

24 days of cold outbound for a reputation-management firm, run by an autonomous agent: 2,388 emails, 7 replies, 1 meeting held, no revenue yet. Published with the full denominator and the four things that went wrong.

Rick Acquisition — $800/month

Every claim on this page is checkable. The robots.txt is public and is re-read every night; the counts above come from that read. Prospect names are withheld deliberately — they did not consent to appear in marketing. Trustpilot is a trademark of Trustpilot A/S. This page describes its published crawling policy, quotes the structure of a public file, and is not affiliated with, sponsored by or endorsed by Trustpilot.

Running an acquisition agent in public · part 2 of 3

The other two

Part 1 of 3The Pepper Ledger2,388 cold emails, 7 replies, 1 meeting held — published with the full denominator. Part 3 of 3The Failure Bill$649.41 in six months, and 44.1% of every model call came back an error.