A blocked retrieval path can keep your sales message out of reach.
A client says ChatGPT never cites their site. Your first instinct is content: rewrite the product pages, add FAQ schema, publish more comparisons. That work matters later. First you need to know whether the engines can fetch the pages at all.
A page can load for a buyer while a bot receives an error or challenge. That access gap can reduce discovery opportunities before your sales message is ever read. Diagnose the actual request path before commissioning a rewrite.
Why do browsers get 200 while AI bots get 500?
Browsers and AI crawlers often take different paths through your stack. A page that returns 200 to Chrome can still return 500 to OAI-SearchBot or ChatGPT-User, or 403 to PerplexityBot behind Cloudflare. Investigate that mismatch before treating it as a copy problem.
Treat fetch failures as an infrastructure ticket, not a content ticket. Compare status codes, response bodies and redirects for the same URL when requested as a normal browser and as each AI bot. If the bot path errors, stop rewriting copy until it returns the same HTML humans see.
Are OAI-SearchBot and GPTBot the same thing?
No. OAI-SearchBot supports ChatGPT search discovery. GPTBot is for model training and improvement. Allowing one does not automatically allow the other, and blocking the wrong bot can cut ChatGPT Search while you think you only opted out of training data use.
When you audit robots.txt and CDN rules, list the bots that matter for citation separately from the bots that matter for training. OAI-SearchBot supports search discovery. ChatGPT-User handles user-triggered visits and is not the search crawler; test each role separately. GPTBot is a different decision about how your content may be used for model improvement.
How does Cloudflare block ChatGPT Search without looking broken?
Cloudflare bot controls or incorrect robots rules can restrict retrieval even when the site looks fine in a browser. A missing robots.txt alone is not proof of a block. The homepage loads, marketing is happy, and the bot never gets a usable response for search.
Practical checks:
- Confirm
robots.txtis served at the hostname root the bot actually hits, not only on a vanity domain or www redirect chain that bots do not follow the same way. - Review Cloudflare AI Crawl Control / bot fight settings for OAI-SearchBot, ChatGPT-User, PerplexityBot and related agents.
- Log bot requests separately so a silent 403 does not look like "no AI traffic."
Do all AI crawlers hit your server the same way?
No. Crawlers have different purposes and request patterns. Use your own logs to assess load before changing capacity or rate limits. Verify crawler identities with the provider's published guidance and IP ranges. A user-agent string alone is easy to imitate.
Review rules narrowly with your technical team. Keep authentication and security protections for private areas; do not disable a firewall globally to make one bot request succeed.
How is crawled different from cited or clicked?
Crawled, cited and clicked are three different AI-visibility events. A bot fetch proves retrievability. A citation is a source link in the observed answer; a brand name without a source link is a mention. A verified human referral visit indicates someone followed a link to your page. Measuring only one of them hides where the funnel broke.
Event | What it answers | Typical failure |
|---|---|---|
Crawled | Can the bot fetch and read the page? | 500s to OAI-SearchBot, Cloudflare 403s |
Cited | Did the engine link to your page as a source? | Thin pages, no criteria content, weak third-party mentions |
Clicked | Did a human leave the AI answer for your URL? | Unlinked mentions, tiny citation chips, poor snippet |
Fix the left column before you blame the middle one. Agencies that skip this table end up rewriting pages that bots never successfully retrieved.
What should you check before rewriting content for AEO?
Run a retrievability pass first: request key URLs as OAI-SearchBot, ChatGPT-User and PerplexityBot; compare status codes to the browser; confirm robots.txt at the hostname root; review Cloudflare AI Crawl Control; and confirm important facts live in raw HTML. Only then open the content backlog.
A sensible agency order of work:
- Compare status, body and redirects for the homepage, sales pages and important articles.
- Check robots rules at the correct hostname root. A missing file alone does not demonstrate blocking; distinguish a 404 from server errors or a challenge.
- Review CDN events for the specific request and rule.
- Corroborate test requests with verified crawler traffic. A spoofed user agent is only a diagnostic probe.
- Confirm the key facts are readable in HTML, then address useful answers and source evidence.
When the matrix is green, move to citation work: answer-first pages, comparison content and third-party mentions. When requests fail, resolve the access issue and repeat the observation.
Use the official OpenAI crawler guide, Perplexity crawler documentation and robots.txt guidance to distinguish bot purposes and verify requests. Successful retrieval creates an opportunity to be considered; it does not guarantee citations or revenue.
Connect AEO work to sales and revenue
Your next buyer could find a competitor first. Choose the question whose missing answer puts a qualified enquiry or a sales conversation at risk: price, fit, proof, alternatives or implementation. Give that buyer a useful answer and a clear next step.
Track the path from AI referral to landing page, enquiry, qualified opportunity and attributed revenue where your data supports it. Record unattributed enquiries separately. More citations can increase visibility without producing sales; review lead quality and buying intent before celebrating a higher count.
FAQ
Why would ChatGPT never cite a site that ranks in Google?
Access problems can affect retrieval even if Google indexed the page. A failed request is one possible cause, not proof of the reason an engine omitted the site; relevance and source selection also matter.
Should we allow GPTBot to appear in ChatGPT Search?
Not necessarily. GPTBot is for model improvement; OAI-SearchBot supports ChatGPT search. Decide training opt-in separately from search discovery, and test the search bots explicitly.
Is Cloudflare always the cause of AI bot blocks?
No. Check CDN rules, application errors, robots directives and request logs. A missing robots.txt alone does not establish that crawling is blocked.
Do Meta and Amazon AI crawlers matter for ChatGPT citations?
They are different clients with different load profiles. They matter for capacity and WAF design so you do not write blunt rules that also block OpenAI search bots you want to allow.
What is the first GEO checklist item for affiliates and publishers?
Allow AI robots. Start by checking access for the retrieval services relevant to your audience.
How do we prove a fetch failure to a client?
Show a side-by-side: same URL, browser 200 with full HTML, bot 500 or 403 with the response body or challenge page. Log the user agent, timestamp and CDN rule that fired.
When should we start rewriting pages for AEO?
After the crawled column is healthy for the URLs you care about. Then measure cited and clicked separately so content work has a clean before/after.
Fix fetch failures, then write for buyers
Turn buyer attention into the next sales conversation.
Use buyer question research and answer gap analysis to identify the questions and pages worth improving.
The free research page explains availability; free intake is currently paused. You can use the manual method in this article now.
Discuss your buyer questions and sales priorities
Research supports decisions; it does not guarantee citations, sales or revenue.
