How Perplexity Finds Its Sources (And How to Be One of Them)

PerplexityBot is Perplexity's dedicated web crawler, identified by the user agent string `PerplexityBot`. Like Googlebot and GPTBot, it discovers pages through sitemaps and links. Unlike both, its crawl is heavily influenced by query demand — pages that are frequently retrieved in answer to user queries get crawled more aggressively.

• Discovery — sitemaps, links from already-crawled pages, and its own proprietary index built from prior crawls
• Freshness weighting — pages updated recently are re-crawled faster and prioritized in retrieval
• robots.txt — PerplexityBot respects robots.txt. A missing `Allow: /` for PerplexityBot, or a broad `Disallow: /`, blocks it completely
• No JavaScript execution — like GPTBot, PerplexityBot processes raw HTML. Client-side rendered content is invisible.

When a user submits a query, Perplexity runs a search against its index and retrieves candidate pages. It then re-ranks them for relevance to the specific query and uses the top results as sources for its generated answer. The selection process rewards several specific signals:

• Topical match — the page content must directly address the query. Perplexity doesn't infer from vague pages the way a parametric model might. If you don't clearly cover the topic on the page, you won't be retrieved.
• Freshness — content updated within 30 days is significantly more likely to be retrieved. Visible `lastmod` timestamps in sitemaps signal freshness directly to Perplexity's crawler.
• Direct answer formatting — pages that lead with the answer (rather than context-first, answer-later structure) are more extractable. Perplexity's synthesis engine pulls the answer paragraph and cites it — if the answer is buried, extraction is harder.
• Source authority — Perplexity weights sources it has retrieved successfully before and that users have engaged with positively. Brand recognition and domain credibility matter.
• Structural clarity — H1, H2, paragraph structure. Perplexity's extraction is easier on well-structured pages than on dense blocks of unbroken text.

• ChatGPT (no browsing) cites from training data. Being cited requires being in OpenAI's training dataset — which means being findable and substantive enough to survive quality filtering, months before the answer is given.
• Perplexity cites from live retrieval. Being cited requires being crawlable right now, having fresh content right now, and being topically relevant to the query right now.

This means Perplexity rewards a publishing cadence. Sites that consistently add specific, factual, answer-formatted content outperform sites with static pages — even if those static pages are better written. Freshness is a first-class signal.

Analysis of Perplexity citation patterns across 118,000+ generated answers reveals consistent source preferences:

• Original data and research — Perplexity actively cites sources with unique numbers, statistics, and benchmarks. If you publish data nobody else has, Perplexity has a specific reason to retrieve you.
• Technical community content — Reddit, Hacker News, Stack Overflow, and specialized forums appear frequently. Perplexity retrieves community-validated answers, not just brand content.
• Frequently updated reference pages — pages with visible "last updated" dates that are updated regularly (pricing pages, comparison pages, guide pages) get high retrieval rates.
• FAQ-structured content — question-answer format maps directly to how Perplexity synthesizes answers. A FAQ section is essentially pre-formatted for Perplexity extraction.
• Specific, factual claims — vague marketing language gets skipped. "We help businesses grow" is not a citable claim. "We analyzed 500 websites and found 71% have no AI-readable metadata" is.

• Allow PerplexityBot in robots.txt — explicit `User-agent: PerplexityBot` / `Allow: /` stanza
• Add lastmod to sitemap entries — set to the actual last modified date, update it when content changes
• Publish on a cadence — weekly or bi-weekly posts significantly outperform monthly for Perplexity citation rates
• Lead every article with the answer — first paragraph should directly answer the question the title poses, before any context
• Add FAQ sections — one FAQ section per article. Each question-answer pair is an independent extraction surface for Perplexity.
• Include specific numbers — data points, statistics, and benchmarks in every article. Even small-scale original data ("we analyzed 50 sites") gets cited.
• Use server-side rendering — PerplexityBot reads raw HTML. If your pages are client-side rendered, it sees almost nothing.
• Keep pages topically tight — one clear topic per page. Perplexity's retrieval is query-specific. A page that covers five things is less likely to surface for any one of them than a page that covers one thing deeply.