Search engine optimization is no longer the only thing website owners think about when publishing content online. As AI assistants and autonomous agents become more common, website owners are exploring new ways to make their content easier for machines to discover, understand, and use.
One proposal attracting attention is llms.txt, a Markdown file designed to give AI systems a structured overview of a website and point them toward its most useful content.
Similar to how robots.txt communicates crawling instructions and XML sitemaps help search engines discover URLs, llms.txt aims to provide a curated guide to important website resources. However, these files serve different purposes, and llms.txt is not an officially ratified web standard.
So, what exactly is an llms.txt file? How do you create one, where should you place it, and does optimizing it actually improve AI visibility?
This guide explains the format, shows you how to create a file manually, covers WordPress implementation, and explores what website owners should realistically expect from it.
What Is llms.txt?
llms.txt is a proposed Markdown file that website owners can publish to help language models and AI agents understand a website’s content and find relevant resources.
The proposal was introduced by Jeremy Howard of Answer.AI in September 2024. Its purpose is to offer a concise, machine-readable overview of a website, with links to important pages and, where available, clean Markdown versions of those pages.
A typical file is published at the root of a website:
https://example.com/llms.txt
The file can contain:
- The name of the website or project
- A short description of what the website offers
- Background information that helps explain the content
- Links to important pages and resources
- Brief descriptions of the linked pages
- Links grouped into useful sections
For example, a WordPress website about SEO might use llms.txt to list its SEO tutorials, technical guides, WordPress plugins, and content discovery resources.
Instead of leaving an AI agent to navigate an entire website without guidance, the file offers a curated starting point.
The important distinction is that llms.txt is a proposal, not a requirement imposed on websites. AI systems may choose to read it, ignore it, or use other methods to discover information.
Why Do Websites Need an llms.txt File?
The idea behind llms.txt comes from a practical problem: webpages are primarily designed for people browsing with a web browser.
A typical webpage can contain navigation menus, advertisements, cookie notices, related posts, scripts, sidebars, and other elements alongside the main article. Extracting the useful information from that page may require additional processing.
A clean Markdown document can make the main content easier to process.
An llms.txt file takes this idea a step further by identifying which pages matter most and organizing them into a structured list.
1. Provide a curated overview of your website
A website may contain hundreds or thousands of pages. A concise index helps describe its main topics and directs an interested agent toward relevant resources.
For example, a website covering digital marketing could organize its content into sections such as SEO, social media, WordPress, analytics, and AI tools.
2. Help agents discover relevant documentation
Developers and businesses often publish detailed documentation across multiple pages. A curated index can help an AI coding assistant or another tool locate the right documentation more efficiently when it is configured to use the file.
This is one of the clearest use cases for llms.txt, particularly for documentation-heavy websites.
3. Point toward cleaner content formats
The proposal also discusses providing Markdown versions of useful pages. Markdown can make headings, lists, code examples, and other textual information easier to process without much of the surrounding webpage interface.
However, publishing Markdown does not guarantee that an AI service will retrieve it.
4. Make your content structure easier to understand
A well-organized index communicates the main subject areas of a website and the relationships between its resources.
This can also benefit human readers and developers who want a simple map of a site’s most useful content.
Keep expectations realistic: creating an llms.txt file does not automatically make your website appear in AI-generated answers, improve Google rankings, or guarantee that AI crawlers will read your content.
llms.txt vs. robots.txt vs. sitemap.xml
These three files are sometimes discussed together because they relate to website discovery, but they have different purposes.
| Feature | robots.txt | sitemap.xml | llms.txt |
|---|---|---|---|
| Main purpose | Communicates crawler access rules | Lists URLs for discovery | Provides a curated overview and links |
| Typical format | Plain text directives | XML | Markdown |
| Common location | /robots.txt | /sitemap.xml | /llms.txt |
| Helps communicate crawling restrictions | Yes, for compliant crawlers | No | No |
| Can list important content | Sometimes through sitemap references | Yes | Yes, selectively |
| Officially standardized for its stated purpose | Established robots exclusion protocol | Established sitemap protocol | Community proposal |
| Guarantees AI visibility | No | No | No |
A robots.txt file can tell compliant crawlers which paths they should not access, although it is not a security mechanism. A sitemap helps search engines discover URLs, but listing a URL does not guarantee indexing.
An llms.txt file is different: it is intended to guide a language model or agent toward selected resources.
You should not replace your existing robots.txt or XML sitemap with llms.txt. Maintain each file for its intended purpose.
What Should an llms.txt File Contain?
The original proposal defines a relatively simple Markdown structure. The version published at llmstxt.org was updated to version 2 in August 2026, so older guides may describe conventions that have since changed.
The core structure includes the following elements.
1. An H1 heading
Start with a single top-level heading that identifies the website or project.
# Example WebsiteThis tells a reader or processing system what the file describes.
2. A short summary
A blockquote can provide a concise description of the site.
> Practical guides covering SEO, WordPress,
> digital marketing, and website development.Keep the description factual. Explain what the website covers instead of filling it with promotional claims.
3. Additional context
You can add a short paragraph explaining the site’s audience, purpose, or content structure.
Example Website publishes tutorials, technical
guides, and resources for website owners,
publishers, and digital marketers.This section should help someone understand the content rather than repeat the same keywords.
4. Organized content sections
Use H2 headings to group related links.
## WordPress
- [WordPress Beginner Guide](https://example.com/wordpress-guide/):
Learn how WordPress themes and blogging work.
## SEO
- [SEO Fundamentals](https://example.com/seo-guide/):
Practical advice for improving website visibility.Each link should point to a real, useful page. Descriptions should accurately reflect the linked resource.
5. Optional resources
You can include additional sections for supporting resources, documentation, or secondary pages. The current proposal retains the ## Optional heading as a convention for secondary information; it no longer gives that section the special machine-processing behavior described in older versions.
The objective is to make the file useful and easy to navigate, not to list every URL on your website.
Example: A Complete llms.txt File
Here is a practical example for a fictional website. Replace the example URLs and descriptions with your own site’s real content before publishing.
# Example Website
> Practical resources for SEO, WordPress,
> website development, and digital marketing.
Example Website publishes tutorials, tools, and
technical guides for website owners and publishers.
## WordPress
- [WordPress Beginner Guide](https://example.com/wordpress-guide/):
Understand themes, blogging, and website setup.
- [WordPress Security Guide](https://example.com/security/):
Learn essential website security practices.
## SEO
- [SEO Fundamentals](https://example.com/seo/):
Learn the basics of search engine optimization.
- [Technical SEO Guide](https://example.com/technical-seo/):
Explore crawling, indexing, and site structure.
## Tools and Resources
- [Website Tools](https://example.com/tools/):
Explore useful tools for website owners.
- [Contact](https://example.com/contact/):
Find contact information and support options.This example uses descriptive headings, concise summaries, and Markdown links. There is no need to add complicated syntax, JSON objects, or unnecessary metadata to make a basic file useful.
The key is accuracy: every URL should exist, and every description should match the destination.
How to Create an llms.txt File for Your Website
You do not need a paid service to create an llms.txt file. You can write it in a text editor, generate it through a script, or use a content management system that supports the format.
Here is a simple manual process.
Step 1: Identify your website’s most important content
Start by listing the pages you want to include.
For a blog, these might be your main guides, category hubs, tools, documentation, and evergreen articles.
For a business website, you might include product documentation, service pages, technical resources, and support information.
Do not simply export every URL from your sitemap. A curated list is more useful when it prioritizes pages that explain your website and its main offerings.
Step 2: Create the file
Open a plain-text editor such as Notepad, Visual Studio Code, or another code editor.
Create a new file named:
llms.txt
Write your website’s name as an H1 heading, add a short summary, and organize relevant links under H2 headings.
Save the file as UTF-8 plain text with the .txt extension. Make sure your editor has not silently saved it as llms.txt.txt.
Step 3: Add accurate URLs
Use the canonical URLs of the pages you want to include. Prefer the final destination rather than a URL that redirects elsewhere.
For example:
- [Website Speed Guide](https://example.com/website-speed/):
Advice on improving page performance.Avoid broken links, duplicate entries, temporary URLs, and pages that should not be publicly exposed.
Do not include private dashboards, unpublished drafts, customer information, or confidential documentation. Anything you list in a public file should be treated as publicly discoverable.
Step 4: Upload it to your website root
For a typical website, the expected URL is:
https://example.com/llms.txt
On a self-hosted WordPress website, the document root is commonly the directory containing files such as wp-config.php, wp-admin, and wp-content.
You can upload the file using your hosting control panel’s file manager or SFTP, provided you have the appropriate permissions.
Do not upload it inside the WordPress media library if that would make it available only at a media URL rather than the expected root path.
If your server already uses a rewrite rule or plugin to generate llms.txt, check the existing implementation before creating a physical file. Two competing implementations can cause confusion about which version is served.
Step 5: Open the published URL
Visit your website’s /llms.txt URL in a browser.
For example:
https://example.com/llms.txt
Check that the content appears as readable text, the URL returns a successful HTTP response, and the links point to the intended destinations.
You can also test the response using a terminal:
curl -i https://example.com/llms.txtLook for a successful status such as 200 OK. A response of 404 Not Found, 403 Forbidden, or 500 Internal Server Error means you should investigate the server configuration or file permissions.
A valid file should be accessible without a login if it is intended as a public discovery resource.
How to Optimize an llms.txt File
Creating the file is only the first step. Its usefulness depends on the quality of the information and the resources it points to.
1. Write for clarity, not keyword density
Use a concise description of your website and its subject areas.
Avoid repeating the same keyword in every heading or link. The purpose is to communicate meaning clearly, not to create a page designed to rank for a phrase.
For example, a description such as “SEO tutorials, WordPress guides, and website tools for publishers” is more informative than a long string of repeated SEO keywords.
2. Prioritize your most useful pages
Start with the content that best represents your website.
Depending on the site, this might include:
- Core tutorials and reference guides
- Product or service documentation
- Important tools and resources
- Detailed technical explanations
- Pages that explain how a product works
A small, well-organized list may be easier to use than an enormous index of low-value URLs.
3. Use descriptive link text
Write link titles that explain what the destination contains.
Prefer:
- [WordPress Security Guide](https://example.com/security/):
Learn how to protect a WordPress installation.Instead of:
- [Click here](https://example.com/security/):
More information.Descriptive labels help both people and automated systems understand the link’s purpose.
4. Keep descriptions accurate
Do not claim that a page covers a topic if it only mentions it briefly. If a guide has been replaced or substantially updated, revise its description accordingly.
Accuracy matters more than promotional language.
5. Keep the file manageable
An llms.txt file should provide a useful overview, not reproduce your entire website unnecessarily.
For a large website, organize links into clear categories and focus on important resources. Your XML sitemap can continue to provide broader URL discovery.
There is no universal maximum file size established by the proposal for every use case. Choose a size that remains practical to read and process.
6. Maintain it as your website changes
Remove dead links and update descriptions when content changes.
If you publish a major new guide, launch a tool, or replace an important resource, review whether the file should be updated.
You do not need to rewrite the file every time you publish a minor article. Focus on changes that affect the site’s main content structure.
7. Do not confuse the file with permission to crawl
Listing a URL in llms.txt does not grant an AI service permission to ignore your access rules. Likewise, leaving a URL out does not prevent a crawler from discovering it elsewhere.
Use appropriate access controls, authentication, licensing terms, and crawler policies for the protections you need.
Should You Also Publish Markdown Versions of Your Pages?
The llms.txt proposal discusses serving clean Markdown versions of useful pages, in addition to the index itself.
Markdown is a lightweight format that represents headings, paragraphs, links, lists, and code examples without the visual layout of a normal HTML page.
For example, a webpage might have this HTML structure:
<h1>WordPress Hosting Guide</h1>
<p>Learn how WordPress hosting works.</p>Its Markdown equivalent could be:
# WordPress Hosting Guide
Learn how WordPress hosting works.The Markdown version is easier to inspect as plain text, particularly when a page contains a large amount of navigation or interface markup.
However, you should not assume that every AI assistant prefers Markdown or that publishing a Markdown version guarantees it will be used. Different systems have different retrieval and content-processing methods.
For WordPress publishers who want to explore this approach, SimplyIrfan Markdown is a relevant resource. It covers a WordPress plugin designed to serve Markdown content in response to requests that accept text/markdown.
The important implementation detail is to test content negotiation and caching carefully. Your ordinary HTML pages should continue working correctly for normal browser requests, and any Markdown response should contain the intended article content rather than a duplicate page layout.
How to Implement llms.txt on WordPress
WordPress users have several ways to publish the file.
Option 1: Upload a static file
Create llms.txt and upload it to the website root through your hosting file manager or SFTP.
This is simple and gives you direct control over the content. However, you must update the file manually whenever you want to change the index.
Option 2: Use a WordPress plugin
A plugin can generate or manage discovery files from your WordPress content. Depending on the implementation, it may let you select posts, pages, categories, or other public resources.
Before installing a plugin, check whether it creates a static file or serves the response dynamically, whether it handles caching properly, and whether it supports the structure you want.
If you are exploring broader AI discovery features for WordPress, see SimplyIrfan AI Content Discovery. The project focuses on helping websites expose structured information about their identity, content, and discovery resources. Check the plugin’s current feature list and documentation to confirm which endpoints and formats it supports.
Do not assume that a general AI content discovery plugin automatically generates every file or follows every version of the llms.txt proposal.
Option 3: Generate the file automatically
For websites with frequently changing content, a script or plugin can generate the file from selected WordPress posts and pages.
An automated generator should:
- Include only public, intended-for-discovery content.
- Use canonical URLs.
- Create useful descriptions rather than repeating generic text.
- Group links by meaningful topic.
- Exclude drafts, private content, and broken destinations.
- Update the output when relevant content changes.
- Avoid overwriting a manually maintained file without warning.
Automation can save time, but a list generated from every published URL is not necessarily a good curated index. Review the output to ensure that the most important resources are easy to find.
Does llms.txt Improve Google Rankings?
This is one of the most important questions to answer accurately.
Google says that llms.txt is not needed for Google Search and does not improve or harm search visibility or rankings. Its published guidance states that Google Search ignores these files for ranking and visibility purposes.
That means you should not create the file expecting it to boost organic traffic, improve indexing, or give your website a direct ranking advantage.
For conventional SEO, continue focusing on the fundamentals:
- Publish original, useful content.
- Ensure important pages can be crawled and indexed.
- Maintain a clear website structure.
- Use descriptive titles and internal links.
- Provide a good page experience.
- Keep your XML sitemap accurate.
- Monitor technical issues through Search Console.
These activities remain important regardless of whether you publish an llms.txt file.
Does llms.txt Help AI Search and Chatbots?
The answer depends on the particular system and how it retrieves information.
The proposal offers a format that an AI agent can use to find important pages. Some tools, particularly those designed to read documentation or retrieve information from specified websites, may find it useful.
But there is no universal rule requiring AI assistants to fetch or follow the file. Google’s documentation explicitly says it does not use llms.txt for Search visibility, including its generative AI search features. <Cite refs={[“turn917521search10”]}/>
For other AI systems, public documentation does not establish that every product uses the file as a routine part of its retrieval process.
This distinction matters because there are three different questions:
- Can an AI agent read the file? Yes, if it can access the URL and is configured to process it.
- Will a particular AI service read it? That depends on the service’s implementation.
- Will publishing it make the service cite your website more often? There is no reliable basis for promising that outcome.
The practical approach is to make your content accessible, useful, and easy to understand, then treat llms.txt as an optional discovery aid rather than a shortcut to AI visibility.
How to Test and Maintain Your llms.txt File
After publishing the file, perform a few straightforward checks.
Check the HTTP response
Open the file in your browser or use curl to confirm that it returns the expected content without errors.
Verify the links
Test important URLs to make sure they work, return the intended pages, and do not lead to login screens or redirects that obscure the actual content.
Check for accidental caching issues
If you generate the file dynamically, make sure your caching system serves the current version when the content changes. A stale cached file can leave removed pages in the index or omit new resources.
Review the Markdown structure
Check that the file starts with a suitable H1 heading, uses readable descriptions, and groups links under relevant H2 headings.
Monitor server logs where possible
If you want to know whether a bot has requested /llms.txt, your server logs may show the request. However, a user-agent string alone does not prove that a request came from a genuine AI service, and a request does not prove the file was used to generate an answer.
Likewise, the absence of requests during a short observation period does not establish that no tool will ever use the file.
Common llms.txt Mistakes to Avoid
Treating it as a replacement for robots.txt
These files serve different purposes. Keep your crawling rules in robots.txt and use an XML sitemap for broad URL discovery.
Expecting guaranteed AI citations
An llms.txt file is not a guaranteed route into AI-generated answers. Do not promise traffic or citation improvements based on the file alone.
Listing every page without prioritization
A huge list of unrelated URLs can make the file harder to navigate. Select important resources and organize them into sensible categories.
Including private or sensitive URLs
A public index can make listed URLs easier to discover. Do not publish confidential material or assume that omitting a URL secures it.
Forgetting to update the file
A discovery index that points to deleted pages or describes outdated resources becomes less useful over time.
Using it as a substitute for good content
No special file can compensate for inaccurate information, poor documentation, broken pages, or content that offers little value to readers.
Frequently Asked Questions
What is the correct location for llms.txt?
The commonly used location is the website root, such as https://example.com/llms.txt. The current proposal also supports files within subdirectories, where the file can cover content under that path.
Is llms.txt an official web standard?
No. It is a community proposal introduced by Jeremy Howard and published at llmstxt.org. It has implementations and adopters, but it should not be confused with a universally mandated web standard.
Is llms.txt the same as llms-full.txt?
No. llms.txt is intended to provide a concise overview and links to relevant resources. llms-full.txt is a convention used by some tools and platforms to provide a larger combined collection of content; it is not the same as the core index described by the proposal.
Should every website create an llms.txt file?
Not necessarily. It may be useful for documentation-heavy websites, software projects, and publishers who want to maintain a curated index for compatible agents. It is optional, and it is not required for Google Search visibility.
Can I create llms.txt without a plugin?
Yes. You can write a Markdown file manually and upload it to your website root. A plugin or script is useful when you want to generate and maintain the file automatically.
Does llms.txt replace an XML sitemap?
No. An XML sitemap provides a broader list of URLs for search engine discovery. llms.txt provides a selective, human- and machine-readable overview of resources.
Should I include every blog post in llms.txt?
Usually, you should prioritize important and representative pages rather than automatically listing every post. Your XML sitemap can continue to include the URLs intended for search engine discovery.
Can llms.txt improve AI visibility?
It can provide a structured entry point for an agent that chooses to read it, but it does not guarantee crawling, inclusion, or citations. Its value depends on how the relevant system retrieves and uses website content.
Conclusion
Creating an llms.txt file is a straightforward way to publish a curated, Markdown-based guide to your website’s most useful resources. It can describe what your site covers, group important pages by topic, and give compatible AI agents a clear starting point.
But it is important to separate the proposal’s intended purpose from proven search benefits. Publishing the file does not automatically improve Google rankings or guarantee that AI assistants will cite your content.
If you decide to implement it, keep the file concise, use accurate URLs, organize resources clearly, and update it when your website changes. Pair it with sound technical SEO, accessible content, an accurate sitemap, and sensible crawler policies.
The broader goal should be to make your website useful and understandable to both people and machines. llms.txt is one optional tool for that work, not a substitute for it.