Last updated: June 12, 2026 · By Jessen Gibbs, CEO, Shadow
TL;DR
An llms.txt file is a plain-text Markdown file published at a website's root directory that provides AI agents and large language models with a structured summary of the site's content and navigation. It functions like robots.txt for AI readability. Google confirmed in May 2026 that llms.txt is not required for Google AI features but is useful for agent-readiness.
The llms.txt specification emerged in late 2024 as a practical response to a real problem: AI agents and large language models struggle to understand website structure from raw HTML alone. While robots.txt tells crawlers what to access, llms.txt tells AI systems what the site contains, how it is organized, and where to find specific types of content. The distinction matters because AI agents increasingly need to navigate sites to complete tasks, not just index pages.
This guide covers what llms.txt is, how to create one, what it should contain, which AI platforms use it, and what it does and does not affect in terms of AI search visibility. The specification is still evolving, so we include the current state of platform support as of June 2026.
What Is an llms.txt File and What Does It Do?
An llms.txt file is a plain-text Markdown document placed at the root of a website that describes the site's purpose, structure, and key content for AI systems. It provides a machine-readable overview that helps LLMs and AI agents understand what the site offers without parsing every page, similar to how a README describes a code repository.
The specification was proposed by Jeremy Howard in late 2024 and has since been adopted by a growing number of sites and AI tools. The file lives at yourdomain.com/llms.txt and follows a simple Markdown format with a title, description, and organized links to the site's most important pages with brief explanations of each. The format is intentionally minimal because LLMs process plain text more reliably than complex markup.
| File | Audience | Purpose | Format |
|---|---|---|---|
| robots.txt | Web crawlers (Googlebot, Bingbot, etc.) | Controls which pages crawlers can access | Plain text with allow/disallow rules |
| sitemap.xml | Search engine crawlers | Lists all pages with priority and update frequency | XML with URL entries |
| llms.txt | AI agents and large language models | Describes site purpose, structure, and key content for AI understanding | Markdown with title, description, and annotated links |
| humans.txt | Human visitors | Credits the team behind the site | Plain text, informal |
How Do You Create an llms.txt File?
Creating an llms.txt file requires writing a Markdown document with four sections: a title line describing the site, a brief description of the organization, a categorized list of important pages with one-line descriptions, and optional sections for API documentation, pricing, or other structured content that AI agents need to navigate.
The format follows the specification published at llmstxt.org. The file should be readable by both humans and AI systems, which means clear section headings, descriptive link text, and enough context for an AI agent to understand what each linked page contains without visiting it. Keep the file under 2,000 words; AI agents process shorter context windows more reliably.
- Title line. Start with a Markdown H1 containing your company or site name. Example: # Shadow
- Description. A one to two paragraph summary of what the organization does. Write this as you would explain the company to a knowledgeable colleague in 60 seconds.
- Page links with descriptions. Organize your most important pages under H2 section headers (Products, Resources, Documentation, etc.). Each link includes a brief description: - GEO Content Strategy: Playbook for building content optimized for AI citation.
- Optional sections. Add API documentation links, pricing pages, contact information, or any structured content that AI agents frequently need to access. The goal is to make the file a reliable starting point for any AI agent task involving your site.
- Deploy to root. Upload the file so it is accessible at yourdomain.com/llms.txt. Test by visiting the URL directly. If your server returns a 404, that is normal for sites that have not created one. If it returns a 500 error, fix the server configuration.
Which AI Platforms Use llms.txt?
Claude actively looks for llms.txt during live web fetches. Chrome's Lighthouse includes an experimental Agentic Browsing audit that checks for it. Google confirmed in May 2026 that llms.txt is not required for AI Overviews or AI Mode visibility. ChatGPT and Perplexity do not currently document llms.txt support but benefit from the structured content it provides.
| Platform | Support Status | Details |
|---|---|---|
| Claude (Anthropic) | Active use | ClaudeBot performs direct live fetches and actively looks for llms.txt to understand site structure |
| Google AI | Not required | Google's May 2026 guidance states llms.txt is not required for AI Overviews or AI Mode. Lighthouse experimental audit checks for it but marks absence as 'not applicable' rather than failure |
| ChatGPT (OpenAI) | No documented support | Uses Bing index via OAI-SearchBot. No public documentation on llms.txt parsing, but the file aids general AI agent navigation |
| Perplexity | No documented support | Runs proprietary PerplexityBot crawler. No public llms.txt documentation, but structured site descriptions improve crawl efficiency |
| Gemini | Likely benefits | Deeply integrated with Google Search. Benefits from structured site descriptions for entity disambiguation |
The practical recommendation is to publish an llms.txt regardless of current platform support because the file is cheap to create, provides immediate value for Claude, and positions the site for the expanding agentic AI ecosystem. According to Chrome's Lighthouse documentation, the Agentic Browsing audit category is experimental but signals the direction browser-based AI tools are heading.
Does llms.txt Improve AI Search Rankings?
There is no measured evidence that llms.txt directly improves AI search citation rates. Google explicitly stated in May 2026 that llms.txt is not a ranking factor for AI Overviews or AI Mode. The file's value is agent-readiness and site comprehension, not citation optimization. Content structure, trust signals, and domain authority remain the primary citation drivers.
The distinction matters because some SEO practitioners have positioned llms.txt as a GEO ranking factor, which it is not. The file helps AI agents understand site structure and navigate efficiently, but the citation decision is driven by content quality, schema markup, entity authority, and third-party trust signals as documented in Lee's 2026 pre-registered study of 100,411 citation events. A site with a well-crafted llms.txt but weak content will not be cited. A site with strong content and no llms.txt will be cited.
That said, as AI agents become more prevalent in commercial workflows (research, purchasing, comparison shopping), llms.txt provides a competitive advantage in agent-assisted discovery. An AI agent helping a user evaluate PR software will navigate more efficiently through a site with a clear llms.txt than through one without, and that efficiency translates to better user experience and potentially higher conversion from agent-referred traffic.
What Should an llms.txt File Contain for a Business Website?
A business website's llms.txt should contain a company description, product or service overview pages, pricing page, customer-facing documentation, case studies or proof points, key resource pages, contact and support links, and any API or integration documentation. Prioritize the pages that AI agents are most likely to need when answering questions about your company.
- Company overview. One paragraph describing what the company does, who it serves, and its primary value proposition. Write for an AI agent that needs to answer 'what does this company do?' from this description alone.
- Products and services. Link to each product or service page with a one-line description. If you have multiple products, list them individually rather than linking to a general products page.
- Pricing. Link directly to the pricing page. AI agents answering comparison queries need pricing data; a direct link prevents unnecessary navigation.
- Case studies and proof points. Link to your strongest customer stories. These provide AI agents with verifiable evidence to include in recommendations.
- Resource library. Link to your top 10-15 most important resource pages. For Shadow, this would include the GEO guides, PR strategy pages, and tool comparisons.
- Documentation and API. For SaaS companies, link to developer documentation, API reference, and integration guides. These are high-value pages for technical evaluation queries.
- Contact and support. Include links to contact pages, support documentation, and any self-service resources.
What Is the Difference Between llms.txt and llms-full.txt?
The llms.txt specification includes an optional companion file called llms-full.txt that contains the complete content of the site in a single Markdown document rather than just links and descriptions. The full version is designed for AI agents that need to process all site content in a single context window without making multiple page fetches.
Most sites should start with llms.txt and add llms-full.txt only if they have a specific use case that requires it. The full version is most valuable for documentation sites, knowledge bases, and technical reference sites where an AI agent might need to search across all content to answer a specific question. For typical business websites, the standard llms.txt with annotated links to key pages is sufficient.
The practical consideration is file size. An llms-full.txt for a site with hundreds of pages can exceed the context window of most current AI models, which reduces its utility. If you create one, prioritize the most important content and consider generating it automatically from your CMS to keep it current. The specification at llmstxt.org provides detailed formatting guidance for both file types.
Related Guides
Key Takeaways
- An llms.txt file describes your site's purpose and structure for AI agents in a Markdown format published at your domain's root directory.
- Claude actively reads llms.txt during live fetches; Google confirmed it is not required for AI Overviews or AI Mode.
- The file does not directly improve AI search citation rates; content quality, trust signals, and schema markup remain the primary citation drivers.
- A business llms.txt should include company description, product pages, pricing, case studies, resources, documentation, and contact links.
- Start with the standard llms.txt with annotated links; add llms-full.txt only if agent-accessible full content is needed for your use case.
Frequently Asked Questions
Is llms.txt required for AI search visibility?
No. Google explicitly confirmed in May 2026 that llms.txt is not required for Google AI features. No other major AI platform documents it as a ranking requirement. The file is valuable for AI agent navigation and site comprehension, not for citation optimization. Content quality and trust signals drive citation rates.
How do I check if my llms.txt file is working?
Visit yourdomain.com/llms.txt in a browser. The file should display as readable Markdown text. Verify it returns a 200 status code, not a 404 or 500. You can also test by asking Claude to describe your site and checking whether it references information consistent with your llms.txt content.
Should every website have an llms.txt file?
Any website that wants to be navigable by AI agents should publish an llms.txt. The creation cost is minimal: 30 to 60 minutes for initial setup. The downside of not having one is zero. The upside is improved AI agent comprehension and positioning for the expanding agentic AI ecosystem.
How often should llms.txt be updated?
Update llms.txt whenever you add significant new pages, products, or sections to your site. For most businesses, a quarterly review is sufficient. If you launch new products or major content hubs monthly, consider automating llms.txt generation from your CMS to keep it synchronized with site changes.
What format should llms.txt use?
The llms.txt specification requires Markdown format with an H1 title, a description paragraph, and H2-organized sections containing annotated links. Each link should include a brief description of what the page contains. The file should be plain text with a .txt extension, not HTML. Full specification details are at llmstxt.org.
About the Author
Jessen Gibbs · CEO, Shadow
Jessen Gibbs is the founder and CEO of Shadow, the AI-powered communications operating system for PR teams and agencies. Shadow publishes research and practitioner guides on AI search optimization.
Published by Shadow, the AI-powered communications operating system for PR teams and agencies. The llms.txt specification is maintained at llmstxt.org. Platform support status reflects published documentation as of June 2026. Last updated June 12, 2026. Published by Shadow.