Your next buyer may ask an AI assistant who to call before opening your website. Start by checking whether the service can reach and read your important pages. A clear answer about your services has little chance of helping when the request meets a block, a login screen or an empty page.
A plumber can publish a useful emergency repair page and still have a firewall challenge in front of it. A dental practice can display appointment information in a widget while leaving that information out of the initial HTML. These are different problems, so diagnose access before buying another round of content.
Why can't an AI assistant read my website?
Check three gates: the HTTP response, the bot instructions and the page text. A successful response can contain a challenge instead of your content. A permissive robots.txt can sit behind a firewall block. A page that looks complete in your browser can depend on scripts that a particular fetcher does not execute.
Pick the homepage and two pages that lead to enquiries. Record the exact URL, final destination, time and what you found. Keep a failed request separate from a successful fetch of a page that simply lacks the information. That record makes a useful handover to your developer.
What's the difference between OAI-SearchBot and GPTBot?
OpenAI documents OAI-SearchBot for search, GPTBot for potential training use and ChatGPT-User for certain user requests. Their controls are separate. A user-requested visit is not proof of automatic search inclusion. Perplexity also distinguishes its search crawler from user-requested fetching.
Choose access for each purpose deliberately. Public marketing pages, private customer records and paid content can require different decisions. Ask your host to verify genuine crawler traffic using the operator's documented verification method; a familiar name in a request header can be copied.
Check the three gates: Access, Bot rules, Readable text.
How do I check robots.txt?
Open /robots.txt at the root of the exact hostname you are testing. Ask your developer to review the rules for the relevant crawler as well as the general User-agent: * group. Named groups do not simply inherit every general restriction. The robots standard explains the matching rules.
Check the www hostname, subdomains and redirected destination separately when they differ. Preserve existing private-path rules. Avoid copying a blanket allow file over the one your site already uses. Robots instructions are voluntary guidance; they are not a substitute for authentication on private information.
Does Cloudflare block AI crawlers?
It can, depending on your site's settings. Review the actual policy and request evidence rather than assuming a default. Cloudflare AI Crawl Control provides crawler controls and activity information. Your host or security plugin may apply additional rules.
A 403 can indicate a refused request, while a 429 can indicate a limit. Neither identifies the cause by itself. Ask the site administrator to connect the response with the relevant firewall event. Keep normal security protections in place while correcting a specific access problem.
Can assistants read JavaScript?
Capabilities vary by service and fetch path. Make core service facts available as readable page text: what you do, where you work, opening hours, price factors and how to contact you. Google's JavaScript SEO guidance explains why server rendering can help crawlers that do not execute scripts.
Use View page source to look for a distinctive sentence from the service page. If it is missing, ask your developer whether the initial response contains it. A complete browser display and an empty initial response are useful evidence; neither should be replaced by a guess about every AI service.
How do I run a quick access check?
Use the WhatBuyersAsk homepage checker, or, with WBA CLI installed, enter wba https://example.com and replace the example with your public website. The default check tests three named request identities, reviews robots instructions and checks readable HTML. It requires no workspace login.
The current CLI also supports wba https://example.com --all-bots for the expanded audit. Its 20 identifiers include robots-only controls and clearly labelled experimental names. These requests simulate user agents; they do not verify actual crawler IP access, indexing or recommendations. Keep the result's limitations with any client report.
Then try two separate manual questions. First, provide a page URL and ask the assistant to identify a specific public fact. Second, ask a buyer question without naming the business and save the answer and sources. Passing the first test does not establish the second. If the service reports an access failure, record it rather than counting it as an ordinary omission.
Can I limit training while staying eligible for search?
Some operators provide separate controls. Google-Extended is a content-use control, not a separate HTTP crawler. Google now also documents a Search generative AI control in Search Console. Review that setting separately from Googlebot access and training choices. Do not rely on older advice that treats all these decisions as one switch.
What is GPTBot?
It is OpenAI's crawler for content that may be used in training. Review its purpose separately from the search and user-request identities, then document the site's choice.
Can ChatGPT read a link I paste?
It may retrieve it when the product and request support that action. Verify the quoted fact against the actual page. A confident summary alone is not evidence that the page was fetched successfully.
How long do access fixes take?
There is no universal deadline. Bot rules may be cached, and page recrawling and answer selection follow separate schedules. Retest the exact URL, keep the timestamp and check genuine server evidence when available.
Does a green check mean AI will recommend us?
No. It is an access observation. Whether the page answers the buyer's question and whether a service selects it are separate questions. Use the recommendation-check guide for that next step.
Once access is understood, focus on the questions that stand between an enquiry and a competitor. WhatBuyersAsk helps agencies select 40 critical buyer questions for the agreed client scope, inspect saved evidence and plan useful content. Read how the research works, then create your workspace. Existing customers can sign in.
