# How to Create and Optimize an llms.txt File for Your Website

- Link: https://simplyirfan.com/this-is-howto-optimize-llms-txt-file/
- Published: 2025-08-12T13:17:38-04:00
- Author: Irfan

[  Stay connected via Google News ](https://www.google.com/preferences/source?q=simplyirfan.com)

Search engine optimization is no longer the only thing website owners think about
when publishing content online. As AI assistants and autonomous agents become more
common, website owners are exploring new ways to make their content easier for machines
to discover, understand, and use.

One proposal attracting attention is `llms.txt`, a Markdown file designed to give
AI systems a structured overview of a website and point them toward its most useful
content.

Similar to how `robots.txt` communicates crawling instructions and XML sitemaps 
help search engines discover URLs, `llms.txt` aims to provide a curated guide to
important website resources. However, these files serve different purposes, and `
llms.txt` is not an officially ratified web standard.

So, what exactly is an `llms.txt` file? How do you create one, where should you 
place it, and does optimizing it actually improve AI visibility?

This guide explains the format, shows you how to create a file manually, covers 
WordPress implementation, and explores what website owners should realistically 
expect from it.

## **What Is llms.txt?**

`llms.txt` is a proposed Markdown file that website owners can publish to help language
models and AI agents understand a website’s content and find relevant resources.

The proposal was introduced by Jeremy Howard of Answer.AI in September 2024. Its
purpose is to offer a concise, machine-readable overview of a website, with links
to important pages and, where available, clean Markdown versions of those pages.

A typical file is published at the root of a website:

`https://example.com/llms.txt`

The file can contain:

 * The name of the website or project
 * A short description of what the website offers
 * Background information that helps explain the content
 * Links to important pages and resources
 * Brief descriptions of the linked pages
 * Links grouped into useful sections

For example, a WordPress website about SEO might use `llms.txt` to list its SEO 
tutorials, technical guides, WordPress plugins, and content discovery resources.

Instead of leaving an [AI agent](https://simplyirfan.com/is-your-site-ai-agent-ready/)
to navigate an entire website without guidance, the file offers a curated starting
point.

The important distinction is that `llms.txt` is a proposal, not a requirement imposed
on websites. AI systems may choose to read it, ignore it, or use other methods to
discover information.

## **Why Do Websites Need an llms.txt File?**

The idea behind `llms.txt` comes from a practical problem: webpages are primarily
designed for people browsing with a web browser.

A typical webpage can contain navigation menus, advertisements, cookie notices, 
related posts, scripts, sidebars, and other elements alongside the main article.
Extracting the useful information from that page may require additional processing.

A clean Markdown document can make the main content easier to process.

An `llms.txt` file takes this idea a step further by identifying which pages matter
most and organizing them into a structured list.

### **1. Provide a curated overview of your website**

A website may contain hundreds or thousands of pages. A concise index helps describe
its main topics and directs an interested agent toward relevant resources.

For example, a website covering digital marketing could organize its content into
sections such as SEO, social media, WordPress, analytics, and AI tools.

### **2. Help agents discover relevant documentation**

Developers and businesses often publish detailed documentation across multiple pages.
A curated index can help an AI coding assistant or another tool locate the right
documentation more efficiently when it is configured to use the file.

This is one of the clearest use cases for `llms.txt`, particularly for documentation-
heavy websites.

### **3. Point toward cleaner content formats**

The proposal also discusses providing Markdown versions of useful pages. Markdown
can make headings, lists, code examples, and other textual information easier to
process without much of the surrounding webpage interface.

However, publishing Markdown does not guarantee that an AI service will retrieve
it.

### **4. Make your content structure easier to understand**

A well-organized index communicates the main subject areas of a website and the 
relationships between its resources.

This can also benefit human readers and developers who want a simple map of a site’s
most useful content.

**Keep expectations realistic:** creating an `llms.txt` file does not automatically
make your website appear in AI-generated answers, improve Google rankings, or guarantee
that AI crawlers will read your content.

## **llms.txt vs. robots.txt vs. sitemap.xml**

These three files are sometimes discussed together because they relate to website
discovery, but they have different purposes.

| Feature | `robots.txt` | `sitemap.xml` | `llms.txt` | 
| Main purpose | Communicates crawler access rules | Lists URLs for discovery | Provides a curated overview and links | 
| Typical format | Plain text directives | XML | Markdown | 
| Common location | `/robots.txt` | `/sitemap.xml` | `/llms.txt` | 
| Helps communicate crawling restrictions | Yes, for compliant crawlers | No | No | 
| Can list important content | Sometimes through sitemap references | Yes | Yes, selectively | 
| Officially standardized for its stated purpose | Established robots exclusion protocol | Established sitemap protocol | Community proposal | 
| Guarantees AI visibility | No | No | No |

A `robots.txt` file can tell compliant crawlers which paths they should not access,
although it is not a security mechanism. A sitemap helps search engines discover
URLs, but listing a URL does not guarantee indexing.

An `llms.txt` file is different: it is intended to guide a language model or agent
toward selected resources.

You should not replace your existing `robots.txt` or XML sitemap with `llms.txt`.
Maintain each file for its intended purpose.

## **What Should an llms.txt File Contain?**

The original proposal defines a relatively simple Markdown structure. The version
published at `llmstxt.org` was updated to version 2 in August 2026, so older guides
may describe conventions that have since changed.

The core structure includes the following elements.

### **1. An H1 heading**

Start with a single top-level heading that identifies the website or project.

    ```wp-block-code
    # Example Website
    ```

This tells a reader or processing system what the file describes.

### **2. A short summary**

A blockquote can provide a concise description of the site.

    ```wp-block-code
    > Practical guides covering SEO, WordPress,
    > digital marketing, and website development.
    ```

Keep the description factual. Explain what the website covers instead of filling
it with promotional claims.

### **3. Additional context**

You can add a short paragraph explaining the site’s audience, purpose, or content
structure.

    ```wp-block-code
    Example Website publishes tutorials, technical
    guides, and resources for website owners,
    publishers, and digital marketers.
    ```

This section should help someone understand the content rather than repeat the same
keywords.

### **4. Organized content sections**

Use H2 headings to group related links.

    ```wp-block-code
    ## WordPress

    - [WordPress Beginner Guide](https://example.com/wordpress-guide/):
      Learn how WordPress themes and blogging work.

    ## SEO

    - [SEO Fundamentals](https://example.com/seo-guide/):
      Practical advice for improving website visibility.
    ```

Each link should point to a real, useful page. Descriptions should accurately reflect
the linked resource.

### **5. Optional resources**

You can include additional sections for supporting resources, documentation, or 
secondary pages. The current proposal retains the `## Optional` heading as a convention
for secondary information; it no longer gives that section the special machine-processing
behavior described in older versions.

The objective is to make the file useful and easy to navigate, not to list every
URL on your website.

## **Example: A Complete llms.txt File**

Here is a practical example for a fictional website. Replace the example URLs and
descriptions with your own site’s real content before publishing.

    ```wp-block-code
    # Example Website

    > Practical resources for SEO, WordPress,
    > website development, and digital marketing.

    Example Website publishes tutorials, tools, and
    technical guides for website owners and publishers.

    ## WordPress

    - [WordPress Beginner Guide](https://example.com/wordpress-guide/):
      Understand themes, blogging, and website setup.
    - [WordPress Security Guide](https://example.com/security/):
      Learn essential website security practices.

    ## SEO

    - [SEO Fundamentals](https://example.com/seo/):
      Learn the basics of search engine optimization.
    - [Technical SEO Guide](https://example.com/technical-seo/):
      Explore crawling, indexing, and site structure.

    ## Tools and Resources

    - [Website Tools](https://example.com/tools/):
      Explore useful tools for website owners.
    - [Contact](https://example.com/contact/):
      Find contact information and support options.
    ```

This example uses descriptive headings, concise summaries, and Markdown links. There
is no need to add complicated syntax, JSON objects, or unnecessary metadata to make
a basic file useful.

The key is accuracy: every URL should exist, and every description should match 
the destination.

## **How to Create an llms.txt File for Your Website**

You do not need a paid service to create an `llms.txt` file. You can write it in
a text editor, generate it through a script, or use a content management system 
that supports the format.

Here is a simple manual process.

### **Step 1: Identify your website’s most important content**

Start by listing the pages you want to include.

For a blog, these might be your main guides, category hubs, tools, documentation,
and evergreen articles.

For a business website, you might include product documentation, service pages, 
technical resources, and support information.

Do not simply export every URL from your sitemap. A curated list is more useful 
when it prioritizes pages that explain your website and its main offerings.

### **Step 2: Create the file**

Open a plain-text editor such as Notepad, Visual Studio Code, or another code editor.

Create a new file named:

`llms.txt`

Write your website’s name as an H1 heading, add a short summary, and organize relevant
links under H2 headings.

Save the file as UTF-8 plain text with the `.txt` extension. Make sure your editor
has not silently saved it as `llms.txt.txt`.

### **Step 3: Add accurate URLs**

Use the canonical URLs of the pages you want to include. Prefer the final destination
rather than a URL that redirects elsewhere.

For example:

    ```wp-block-code
    - [Website Speed Guide](https://example.com/website-speed/):
      Advice on improving page performance.
    ```

Avoid broken links, duplicate entries, temporary URLs, and pages that should not
be publicly exposed.

Do not include private dashboards, unpublished drafts, customer information, or 
confidential documentation. Anything you list in a public file should be treated
as publicly discoverable.

### **Step 4: Upload it to your website root**

For a typical website, the expected URL is:

`https://example.com/llms.txt`

On a self-hosted WordPress website, the document root is commonly the directory 
containing files such as `wp-config.php`, `wp-admin`, and `wp-content`.

You can upload the file using your hosting control panel’s file manager or SFTP,
provided you have the appropriate permissions.

Do not upload it inside the WordPress media library if that would make it available
only at a media URL rather than the expected root path.

If your server already uses a rewrite rule or plugin to generate `llms.txt`, check
the existing implementation before creating a physical file. Two competing implementations
can cause confusion about which version is served.

### **Step 5: Open the published URL**

Visit your website’s `/llms.txt` URL in a browser.

For example:

`https://example.com/llms.txt`

Check that the content appears as readable text, the URL returns a successful HTTP
response, and the links point to the intended destinations.

You can also test the response using a terminal:

    ```wp-block-code
    curl -i https://example.com/llms.txt
    ```

Look for a successful status such as `200 OK`. A response of `404 Not Found`, `403
Forbidden`, or `500 Internal Server Error` means you should investigate the server
configuration or file permissions.

A valid file should be accessible without a login if it is intended as a public 
discovery resource.

## **How to Optimize an llms.txt File**

Creating the file is only the first step. Its usefulness depends on the quality 
of the information and the resources it points to.

### **1. Write for clarity, not keyword density**

Use a concise description of your website and its subject areas.

Avoid repeating the same keyword in every heading or link. The purpose is to communicate
meaning clearly, not to create a page designed to rank for a phrase.

For example, a description such as “SEO tutorials, WordPress guides, and website
tools for publishers” is more informative than a long string of repeated SEO keywords.

### **2. Prioritize your most useful pages**

Start with the content that best represents your website.

Depending on the site, this might include:

 * Core tutorials and reference guides
 * Product or service documentation
 * Important tools and resources
 * Detailed technical explanations
 * Pages that explain how a product works

A small, well-organized list may be easier to use than an enormous index of low-
value URLs.

### **3. Use descriptive link text**

Write link titles that explain what the destination contains.

Prefer:

    ```wp-block-code
    - [WordPress Security Guide](https://example.com/security/):
      Learn how to protect a WordPress installation.
    ```

Instead of:

    ```wp-block-code
    - [Click here](https://example.com/security/):
      More information.
    ```

Descriptive labels help both people and automated systems understand the link’s 
purpose.

### **4. Keep descriptions accurate**

Do not claim that a page covers a topic if it only mentions it briefly. If a guide
has been replaced or substantially updated, revise its description accordingly.

Accuracy matters more than promotional language.

### **5. Keep the file manageable**

An `llms.txt` file should provide a useful overview, not reproduce your entire website
unnecessarily.

For a large website, organize links into clear categories and focus on important
resources. Your XML sitemap can continue to provide broader URL discovery.

There is no universal maximum file size established by the proposal for every use
case. Choose a size that remains practical to read and process.

### **6. Maintain it as your website changes**

Remove dead links and update descriptions when content changes.

If you publish a major new guide, launch a tool, or replace an important resource,
review whether the file should be updated.

You do not need to rewrite the file every time you publish a minor article. Focus
on changes that affect the site’s main content structure.

### **7. Do not confuse the file with permission to crawl**

Listing a URL in `llms.txt` does not grant an AI service permission to ignore your
access rules. Likewise, leaving a URL out does not prevent a crawler from discovering
it elsewhere.

Use appropriate access controls, authentication, licensing terms, and crawler policies
for the protections you need.

## **Should You Also Publish Markdown Versions of Your Pages?**

The `llms.txt` proposal discusses serving clean Markdown versions of useful pages,
in addition to the index itself.

Markdown is a lightweight format that represents headings, paragraphs, links, lists,
and code examples without the visual layout of a normal HTML page.

For example, a webpage might have this HTML structure:

    ```wp-block-code
    <h1>WordPress Hosting Guide</h1>
    <p>Learn how WordPress hosting works.</p>
    ```

Its Markdown equivalent could be:

    ```wp-block-code
    # WordPress Hosting Guide

    Learn how WordPress hosting works.
    ```

The Markdown version is easier to inspect as plain text, particularly when a page
contains a large amount of navigation or interface markup.

However, you should not assume that every AI assistant prefers Markdown or that 
publishing a Markdown version guarantees it will be used. Different systems have
different retrieval and content-processing methods.

For WordPress publishers who want to explore this approach, [SimplyIrfan Markdown](https://simplyirfan.com/wp-markdown/)
is a relevant resource. It covers a WordPress plugin designed to serve Markdown 
content in response to requests that accept `text/markdown`.

The important implementation detail is to test content negotiation and caching carefully.
Your ordinary HTML pages should continue working correctly for normal browser requests,
and any Markdown response should contain the intended article content rather than
a duplicate page layout.

## **How to Implement llms.txt on WordPress**

WordPress users have several ways to publish the file.

### **Option 1: Upload a static file**

Create `llms.txt` and upload it to the website root through your hosting file manager
or SFTP.

This is simple and gives you direct control over the content. However, you must 
update the file manually whenever you want to change the index.

### **Option 2: Use a WordPress plugin**

A plugin can generate or manage discovery files from your WordPress content. Depending
on the implementation, it may let you select posts, pages, categories, or other 
public resources.

Before installing a plugin, check whether it creates a static file or serves the
response dynamically, whether it handles caching properly, and whether it supports
the structure you want.

If you are exploring broader AI discovery features for WordPress, see [SimplyIrfan AI Content Discovery](https://simplyirfan.com/wp-ai-content-discovery/).
The project focuses on helping websites expose structured information about their
identity, content, and discovery resources. Check the plugin’s current feature list
and documentation to confirm which endpoints and formats it supports.

Do not assume that a general AI content discovery plugin automatically generates
every file or follows every version of the `llms.txt` proposal.

### **Option 3: Generate the file automatically**

For websites with frequently changing content, a script or plugin can generate the
file from selected WordPress posts and pages.

An automated generator should:

 1. Include only public, intended-for-discovery content.
 2. Use canonical URLs.
 3. Create useful descriptions rather than repeating generic text.
 4. Group links by meaningful topic.
 5. Exclude drafts, private content, and broken destinations.
 6. Update the output when relevant content changes.
 7. Avoid overwriting a manually maintained file without warning.

Automation can save time, but a list generated from every published URL is not necessarily
a good curated index. Review the output to ensure that the most important resources
are easy to find.

## **Does llms.txt Improve Google Rankings?**

This is one of the most important questions to answer accurately.

**Google says that** `**llms.txt**` **is not needed for Google Search and does not
improve or harm search visibility or rankings.** Its published guidance states that
Google Search ignores these files for ranking and visibility purposes.

That means you should not create the file expecting it to boost organic traffic,
improve indexing, or give your website a direct ranking advantage.

For conventional SEO, continue focusing on the fundamentals:

 * Publish original, useful content.
 * Ensure important pages can be crawled and indexed.
 * Maintain a clear website structure.
 * Use descriptive titles and internal links.
 * Provide a good page experience.
 * Keep your XML sitemap accurate.
 * Monitor technical issues through Search Console.

These activities remain important regardless of whether you publish an `llms.txt`
file.

## **Does llms.txt Help AI Search and Chatbots?**

The answer depends on the particular system and how it retrieves information.

The proposal offers a format that an AI agent can use to find important pages. Some
tools, particularly those designed to read documentation or retrieve information
from specified websites, may find it useful.

But there is no universal rule requiring AI assistants to fetch or follow the file.
Google’s documentation explicitly says it does not use `llms.txt` for Search visibility,
including its generative AI search features. <Cite refs={[“turn917521search10”]}/
>

For other AI systems, public documentation does not establish that every product
uses the file as a routine part of its retrieval process.

This distinction matters because there are three different questions:

 1. **Can an AI agent read the file?** Yes, if it can access the URL and is configured
    to process it.
 2. **Will a particular AI service read it?** That depends on the service’s implementation.
 3. **Will publishing it make the service cite your website more often?** There is 
    no reliable basis for promising that outcome.

The practical approach is to make your content accessible, useful, and easy to understand,
then treat `llms.txt` as an optional discovery aid rather than a shortcut to AI 
visibility.

## **How to Test and Maintain Your llms.txt File**

After publishing the file, perform a few straightforward checks.

### **Check the HTTP response**

Open the file in your browser or use `curl` to confirm that it returns the expected
content without errors.

### **Verify the links**

Test important URLs to make sure they work, return the intended pages, and do not
lead to login screens or redirects that obscure the actual content.

### **Check for accidental caching issues**

If you generate the file dynamically, make sure your caching system serves the current
version when the content changes. A stale cached file can leave removed pages in
the index or omit new resources.

### **Review the Markdown structure**

Check that the file starts with a suitable H1 heading, uses readable descriptions,
and groups links under relevant H2 headings.

### **Monitor server logs where possible**

If you want to know whether a bot has requested `/llms.txt`, your server logs may
show the request. However, a user-agent string alone does not prove that a request
came from a genuine AI service, and a request does not prove the file was used to
generate an answer.

Likewise, the absence of requests during a short observation period does not establish
that no tool will ever use the file.

## **Common llms.txt Mistakes to Avoid**

**Treating it as a replacement for robots.txt**

These files serve different purposes. Keep your crawling rules in `robots.txt` and
use an XML sitemap for broad URL discovery.

**Expecting guaranteed AI citations**

An `llms.txt` file is not a guaranteed route into AI-generated answers. Do not promise
traffic or citation improvements based on the file alone.

**Listing every page without prioritization**

A huge list of unrelated URLs can make the file harder to navigate. Select important
resources and organize them into sensible categories.

**Including private or sensitive URLs**

A public index can make listed URLs easier to discover. Do not publish confidential
material or assume that omitting a URL secures it.

**Forgetting to update the file**

A discovery index that points to deleted pages or describes outdated resources becomes
less useful over time.

**Using it as a substitute for good content**

No special file can compensate for inaccurate information, poor documentation, broken
pages, or content that offers little value to readers.

## **Frequently Asked Questions**

### **What is the correct location for llms.txt?**

The commonly used location is the website root, such as `https://example.com/llms.
txt`. The current proposal also supports files within subdirectories, where the 
file can cover content under that path.

### **Is llms.txt an official web standard?**

No. It is a community proposal introduced by Jeremy Howard and published at llmstxt.
org. It has implementations and adopters, but it should not be confused with a universally
mandated web standard.

### **Is llms.txt the same as llms-full.txt?**

No. `llms.txt` is intended to provide a concise overview and links to relevant resources.`
llms-full.txt` is a convention used by some tools and platforms to provide a larger
combined collection of content; it is not the same as the core index described by
the proposal.

### **Should every website create an llms.txt file?**

Not necessarily. It may be useful for documentation-heavy websites, software projects,
and publishers who want to maintain a curated index for compatible agents. It is
optional, and it is not required for Google Search visibility.

### **Can I create llms.txt without a plugin?**

Yes. You can write a Markdown file manually and upload it to your website root. 
A plugin or script is useful when you want to generate and maintain the file automatically.

### **Does llms.txt replace an XML sitemap?**

No. An XML sitemap provides a broader list of URLs for search engine discovery. `
llms.txt` provides a selective, human- and machine-readable overview of resources.

### **Should I include every blog post in llms.txt?**

Usually, you should prioritize important and representative pages rather than automatically
listing every post. Your XML sitemap can continue to include the URLs intended for
search engine discovery.

### **Can llms.txt improve AI visibility?**

It can provide a structured entry point for an agent that chooses to read it, but
it does not guarantee crawling, inclusion, or citations. Its value depends on how
the relevant system retrieves and uses website content.

## **Conclusion**

Creating an `llms.txt` file is a straightforward way to publish a curated, Markdown-
based guide to your website’s most useful resources. It can describe what your site
covers, group important pages by topic, and give compatible AI agents a clear starting
point.

But it is important to separate the proposal’s intended purpose from proven search
benefits. Publishing the file does not automatically improve Google rankings or 
guarantee that AI assistants will cite your content.

If you decide to implement it, keep the file concise, use accurate URLs, organize
resources clearly, and update it when your website changes. Pair it with sound technical
SEO, accessible content, an accurate sitemap, and sensible crawler policies.

The broader goal should be to make your website useful and understandable to both
people and machines. `llms.txt` is one optional tool for that work, not a substitute
for it.

## Further Reading

 * [The llms.txt proposal and format](https://llmstxt.org/)
 * [Google’s guidance on AI features and search optimization](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)
 * [SimplyIrfan AI Content Discovery](https://simplyirfan.com/wp-ai-content-discovery/)
 * [SimplyIrfan Markdown](https://simplyirfan.com/wp-markdown/)

 [  Stay connected via Google News ](https://www.google.com/preferences/source?q=simplyirfan.com)
