How to Check if ChatGPT Can See Your Website (and Fix It if It Can't)

The problem nobody told you about

You spent months on SEO. Then ChatGPT started answering the questions your customers used to Google — and you found out you weren't in the answer. Worse: you might be blocking the very crawler that decides whether you get cited.

OpenAI's current publisher guidance names OAI-SearchBot for discovery and inclusion in ChatGPT search summaries and snippets. GPTBot is a separate control for potential model training. Allowing search while opting out of training is a valid policy; neither setting guarantees a citation.

Step 1: Check your robots.txt

Open https://yourdomain.com/robots.txt and look for these agents:

# This blocks ChatGPT search summaries and snippets
User-agent: OAI-SearchBot
Disallow: /

A surprising number of Next.js and WordPress templates ship a blanket Disallow: / for "AI bots" by default. To explicitly allow the ones you care about:

User-agent: OAI-SearchBot
Allow: /

# Optional, separate policy: opt out of potential training
User-agent: GPTBot
Disallow: /

Do not collapse every OpenAI token into one “AI access” setting. Search discovery, potential training and user-initiated retrieval have different purposes. Start from the outcome your publishing policy intends, then use OpenAI's current documentation for that specific token.

Step 2: Simulate the fetch

robots.txt is advisory; the real test is whether the page returns crawlable HTML. Curl as the bot:

curl -A "OAI-SearchBot" \
  -s -o /dev/null -w "%{http_code}\n" https://yourdomain.com

A 200 is good. A 403 means your CDN or WAF (Cloudflare's "Block AI Bots" toggle is a common culprit) is rejecting the agent before it ever reaches your app.

Step 3: Make sure there's HTML to read

The safest publishing baseline is meaningful content in the initial HTML. If the answer appears only after hydration, non-JavaScript clients and extraction systems may receive an empty shell. Check the raw document:

curl -s https://yourdomain.com | grep -i "<h1\|<article\|<main"

If your headline isn't in there, neither answer engine can quote it.

Fixes for the empty-shell problem

• Use server-side rendering or static generation for content pages. In Next.js App Router, that's the default for Server Components — don't push your content behind a "use client" boundary.
• Render the meaningful text in the initial HTML response, not after a useEffect fetch.
• Avoid gating content behind a cookie wall, infinite-scroll, or "click to load."

A fast way to confirm what a non-JS client sees is to strip your own response down to text:

curl -s https://yourdomain.com | sed 's/<[^>]*>//g' | tr -s ' \n' | head -40

If your actual article copy shows up there, the answer engine can read it. If you get a near-empty result or just Loading..., your content is trapped behind JavaScript and you need server rendering.

Step 4: Give the answer engine something quotable

AI engines extract passages, not pages. Help them:

1. Put a direct, one-sentence answer in the first paragraph under each ## heading.
2. Use real semantic HTML — <article>, <h1>–<h3>, <ul> — not a soup of <div>s.
3. Add Article or FAQPage JSON-LD structured data so the parser knows what's what.

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "Can ChatGPT see my website?",
    "acceptedAnswer": { "@type": "Answer", "text": "A public page is technically accessible when OAI-SearchBot is allowed and the canonical URL returns useful indexable HTML." }
  }]
}

Quick checklist

• robots.txt allows OAI-SearchBot if ChatGPT search visibility is intended
• Training policy for GPTBot is decided separately
• CDN/WAF is not challenging the intended crawler
• Core content is in the server-rendered HTML
• Headings and structured data are present

Use the right OpenAI control for the outcome you want

OpenAI's current publisher guidance makes an important distinction. OAI-SearchBot controls whether page content can be included in summaries and snippets in ChatGPT search. GPTBot is a control for potential model training. A publisher can allow search discovery while disallowing training; treating those tokens as interchangeable produces the wrong robots policy. User-initiated fetch behavior is another path and should not be used as a substitute for search-crawler eligibility.

A minimal policy for a publisher that wants ChatGPT search visibility but opts out of training could look like this:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

That is only a crawl preference. The page must still be public, return a useful success response, expose indexable text and avoid a noindex directive if discovery is intended. Allowing a crawler does not guarantee selection, ranking, a summary or a citation.

Diagnose the full delivery path

Test four layers separately. First, fetch robots.txt from the canonical host and check the most-specific matching group. Second, request the canonical page and record redirects, status, content type and any WAF challenge. Third, inspect raw HTML for the title, primary answer, canonical URL and index directives. Fourth, validate that visible structured data matches the page; markup is descriptive evidence, not a visibility switch.

Do not spoof a crawler user agent and conclude that you reproduced the crawler. CDNs can identify clients by IP, TLS behavior and bot-verification mechanisms. A user-agent curl test is useful for catching a crude block, but server logs and verified crawler documentation provide stronger evidence. Also test a normal non-JavaScript request: if the answer is absent until hydration, extraction remains fragile even when access is allowed.

Measure the outcome with referrals and citations, not crawler hits alone. OpenAI says ChatGPT referral URLs include utm_source=chatgpt.com; preserve that parameter in analytics and review landing pages. Run a dated set of representative questions monthly and record whether your brand is cited, which URL is linked and whether the quoted claim remains accurate.

Google's AI features use ordinary Search eligibility and Googlebot controls; Google-Extended has a separate Gemini training/grounding purpose and does not affect Google Search inclusion. The SEO versus AEO guide covers that boundary, and What is AEO? supplies the broader measurement model.

Scan it with Troja

Troja's AEO scan checks published access and extraction signals, reports the evidence it observes and separates crawler policy from page structure. Use that report as a diagnostic—not a promise that an answer product will select or cite the page.