How ChatGPT Actually Finds and Reads Your Website

GPTBot doesn't start from scratch. It uses three discovery sources to build its crawl queue:

• Sitemaps — GPTBot reads your sitemap.xml via the Sitemap: directive in robots.txt. Every URL in your sitemap is a candidate for crawling. This is the highest-leverage discovery signal you control directly.
• Common Crawl — OpenAI ingests Common Crawl datasets, a public archive of billions of web pages crawled regularly. If Common Crawl has indexed you, OpenAI has probably seen you.
• Link discovery — GPTBot follows links from pages it's already crawling. If a credible site links to yours, GPTBot will eventually find you through that link.

When GPTBot fetches a page, it receives the same initial HTML your browser receives — but it stops there. No JavaScript execution. No waiting for React components to mount. No fetching of dynamic data. What's in the raw HTML document is what GPTBot reads.

• Title tag — read and heavily weighted. The single most important signal.
• Meta description — read in full. Your one-sentence identity for GPTBot.
• Static HTML text — everything visible in the raw document before JS runs.
• JSON-LD schema — parsed and used directly. `<script type="application/ld+json">` blocks are read even when body content is thin.
• Heading hierarchy — H1→H2→H3 structure tells GPTBot how your content is organized.
• Alt text on images — the only part of an image GPTBot can read.

• JavaScript-rendered content — invisible. If your hero text mounts after page load, GPTBot never sees it.
• CSS animations and transitions — zero signal.
• Video and audio — not processed.
• Images without alt text — black holes.
• Content behind authentication — GPTBot doesn't log in.

GPTBot operates in two distinct contexts that require different strategies to optimize for:

• Training data crawling — GPTBot indexes your site and the data feeds into OpenAI's next training run. This builds ChatGPT's parametric memory — what it knows about you without browsing. This is slow: training cycles happen over months. Being well-represented here requires being findable, readable, and substantive enough to pass OpenAI's quality filters.
• Real-time browsing — when a ChatGPT user has web browsing enabled and asks a question that needs current information, GPTBot fetches pages live and the model synthesizes from them directly. This is fast: a page you published yesterday can show up in a ChatGPT answer today. This mode rewards freshness and direct answer formatting.

• Allow GPTBot in robots.txt — add `User-agent: GPTBot` with `Allow: /`. Check you haven't inherited a blanket `Disallow: /` from a staging environment.
• Add Sitemap: directive — `Sitemap: https://yourdomain.com/sitemap.xml` in robots.txt. This is how GPTBot auto-discovers your pages without relying on Common Crawl.
• Deploy a static crawler profile page — a noindex HTML page with full JSON-LD schema that requires zero JavaScript. This is a purpose-built surface for GPTBot when your main site is JS-heavy.
• Make your title and meta description work alone — assume GPTBot reads only those two fields. Do they clearly communicate what you are, who you serve, and what problem you solve?
• Use server-side rendering on key pages — homepage, about, solutions. If these are client-side only, GPTBot sees a shell.
• Add JSON-LD schema — Organization, FAQPage, and SoftwareApplication schema in `<script type="application/ld+json">` are parsed by GPTBot regardless of JS rendering status.

• Count the words in the curl output — under 600 means AI crawlers are working with almost nothing
• Check for `<script type="application/ld+json">` blocks — these should be present even if body content is thin
• Verify your title tag and meta description are in the raw HTML (not injected by JS)
• Confirm `Disallow:` rules in robots.txt don't accidentally block GPTBot