A semantic HTML page, a sitemap entry, and a robots.txt snippet. Here's exactly why each one matters — and what happens without them.
Most websites are accidentally invisible to AI crawlers. Not because the content is bad, but because the infrastructure that tells crawlers where to look, what to read, and whether they're allowed in simply doesn't exist. Brandioz generates three files that fix this in minutes. This is what each one does, why it matters, and what AI systems actually do with them when they find your site.
The core of the package is a static HTML file — a purpose-built page designed to be read by machines, not humans. It carries a `noindex` tag so it never appears in search results, but every AI crawler that finds it gets a clean, fully rendered, semantically structured document. This matters because most modern websites are built on JavaScript frameworks that render content client-side. When GPTBot or ClaudeBot requests your homepage, they often receive a thin HTML shell with fewer than 600 words of actual content. The rest only appears after JavaScript executes — which most AI crawlers don't do. The generated HTML page sidesteps this entirely. It's static. No JavaScript execution required. Every word is present in the raw HTML response. Inside the file, three layers of structured data work together: **Semantic HTML structure** — `<main>`, `<article>`, `<section>`, and a proper `<h1>` → `<h2>` → `<h3>` heading hierarchy. AI systems interpret this structure as a signal of organized, reliable information. A page that flows logically from "what we are" to "what we do" to "who we serve" is far more parseable than a flat marketing page. **JSON-LD schema** — machine-readable structured data embedded in `<script type="application/ld+json">` tags. The package includes Organization schema (name, URL, description, contact), category schema (SoftwareApplication, Product, or WebSite depending on what you do), FAQPage schema with auto-generated questions and answers, and BreadcrumbList schema for structural context. AI answer engines actively extract JSON-LD and use it to populate their understanding of your brand — it's the highest-signal format you can provide. **Microdata attributes** — `itemscope` and `itemprop` attributes on HTML elements provide a redundant signal layer for crawlers that process microdata separately from JSON-LD. It's belt-and-suspenders structured data. The result is a page that tells any AI crawler exactly what your company is, what it does, who it serves, and what category it belongs to — in every machine-readable format that matters.
The semantic HTML page is useless if crawlers never find it. That's where the sitemap entry comes in. AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended — all follow the same discovery process. They request your `robots.txt` first. From there, they find your sitemap URL. From your sitemap, they build a list of pages to crawl. If your crawler profile page isn't in the sitemap, it won't be crawled. The generated `<url>` block includes four fields: `<loc>` with the exact URL where the file is hosted, `<lastmod>` set to the generation date so crawlers know when it was last updated, `<changefreq>weekly</changefreq>` as a re-check hint, and `<priority>0.5</priority>` — deliberately lower than your homepage so crawlers hit your main pages first. The `lastmod` field is particularly important for retrieval-first platforms like Perplexity. Research on 118,000+ AI-generated answers shows content updated within 30 days receives significantly more citations than older content. A visible, recent `lastmod` timestamp signals freshness — even on a page whose content hasn't changed.
Even if a crawler finds your sitemap entry, it checks `robots.txt` before fetching any page. If a broad `Disallow: /` rule exists — which is more common than you'd expect, especially on staging environments that were promoted to production — every AI crawler gets blocked before it can read a single word. The generated robots.txt snippet grants explicit `Allow` access to five crawlers: GPTBot (ChatGPT/OpenAI), PerplexityBot (Perplexity AI), ClaudeBot (Claude/Anthropic), Google-Extended (Google Gemini), and CCBot (Common Crawl, which feeds multiple training datasets). Critically, the snippet is scoped. It grants access to `/crawler-profile.html` specifically — not your entire site. Your existing `Disallow` rules remain completely untouched. If you've blocked certain directories for legitimate reasons, those blocks stay in place. The snippet also references your sitemap URL explicitly, using the `Sitemap:` directive. This is how AI crawlers auto-discover your sitemap without relying on Google Search Console submission — they read the `Sitemap:` line directly from `robots.txt` and fetch it immediately.
Once all three files are in place and your sitemap is submitted to Google Search Console, the discovery chain works automatically. An AI crawler requests your `robots.txt`. It finds the `Sitemap:` directive and fetches your sitemap. It finds the crawler profile URL in the sitemap with a recent `lastmod` date. It fetches the page, reads the static HTML, extracts the JSON-LD schema, processes the semantic structure, and updates its internal model of your brand. For retrieval-first platforms, this can happen within days of deployment — they actively re-crawl sitemaps and prioritize recently modified pages. For parametric-first platforms, the training data cycle is longer, but the structured data you provide feeds into future training runs. The practical test is simple: after deploying, run `curl -A "GPTBot" https://yourdomain.com/crawler-profile.html`. If you get your HTML page back, GPTBot can read it. If you get blocked or an empty response, something in the chain needs fixing. Most websites that deploy these three files go from near-invisible to fully parseable for AI crawlers within a week. Not because AI suddenly learned something new about them — but because they finally gave AI something to read.
Key takeaway
Three files — a semantic HTML page with JSON-LD schema, a sitemap entry, and a robots.txt snippet — form a complete AI crawler discovery chain. Most sites are missing all three. Deploying them takes under five minutes and moves your brand from invisible to fully indexed across every major AI crawler.
See how your site scores
Free AI visibility analysis — takes 10 seconds.