{"id":19511,"date":"2026-11-05T05:45:00","date_gmt":"2026-11-05T05:45:00","guid":{"rendered":"https:\/\/eoxysit.com\/blogs\/?p=19511"},"modified":"2026-10-28T19:44:58","modified_gmt":"2026-10-28T19:44:58","slug":"llms-txt-ai-crawlers-wordpress","status":"publish","type":"post","link":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/","title":{"rendered":"llms.txt and AI Crawlers on WordPress: What to Allow in 2026"},"content":{"rendered":"\n<p><img loading=\"lazy\" decoding=\"async\" class=\"alignnone wp-image-19488 size-full\" src=\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg\" alt=\"llms.txt and AI crawlers illustration: AI search bots reaching a website through access rules on an isometric navy platform\" width=\"1600\" height=\"900\" srcset=\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg 1600w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search-300x169.jpg 300w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search-1024x576.jpg 1024w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search-768x432.jpg 768w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search-1536x864.jpg 1536w\" sizes=\"auto, (max-width: 1600px) 100vw, 1600px\" \/><\/p>\n\n\n\n<p>Many WordPress owners have heard they need an <strong>llms.txt<\/strong> file to show up in ChatGPT or Google&#8217;s AI answers. Others worry that AI bots are scraping their content and want to block them all. Both reactions can cost you: one wastes effort, the other can quietly remove you from AI search answers.<\/p>\n\n\n\n<p>This guide explains what <strong>llms.txt on WordPress<\/strong> can and cannot do, what Google has said about it, which AI crawlers do what, and how to set up robots.txt and llms.txt on WordPress without breaking anything. Everything here was checked against official documentation on 29 October 2026.<\/p>\n\n\n\n<p><strong>Key takeaways<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li>Google says llms.txt files aren&#8217;t needed for Google Search and &#8220;won&#8217;t negatively or positively impact your visibility or rankings&#8221; (<a href=\"https:\/\/developers.google.com\/search\/updates\" target=\"_blank\" rel=\"noopener\">Google Search Central, 15 June 2026<\/a>).<\/li>\n\n\n<li>llms.txt is a proposed standard that may help other AI tools and agents find your key pages. It is cheap to add but not a ranking lever.<\/li>\n\n\n<li>robots.txt matters more: most AI companies separate search crawlers from training crawlers, so you can block training and still be cited.<\/li>\n\n\n<li>On WordPress, generate llms.txt with your SEO plugin, edit robots.txt carefully, and check your CDN or firewall is not blocking the bots you allowed.<\/li>\n\n<\/ul>\n\n\n\n<p><strong>In this guide<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li><a href=\"#what\">What llms.txt is<\/a><\/li>\n\n\n<li><a href=\"#google\">What Google says<\/a><\/li>\n\n\n<li><a href=\"#crawlers\">AI crawlers: search vs training<\/a><\/li>\n\n\n<li><a href=\"#setup\">Setting it up on WordPress<\/a><\/li>\n\n\n<li><a href=\"#robots\">A sample robots.txt<\/a><\/li>\n\n\n<li><a href=\"#mistakes\">Common mistakes<\/a><\/li>\n\n\n<li><a href=\"#faq\">FAQ<\/a><\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"what\">What llms.txt is<\/h2>\n\n\n\n<p>llms.txt is a plain Markdown file at the root of a site (yoursite.com\/llms.txt). The <a href=\"https:\/\/llmstxt.org\/\" target=\"_blank\" rel=\"noopener\">llmstxt.org proposal<\/a>, first published by Jeremy Howard in September 2024 and updated in August 2026, describes it as a way &#8220;to provide information to help agents use a website&#8221;: a short summary plus links to your most useful pages, in a format that is easy for language models to read.<\/p>\n\n\n\n<p>It does not control access. robots.txt says which bots may crawl. llms.txt only suggests what to read. A bot that ignores robots.txt will ignore llms.txt too.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"google\">What Google says<\/h2>\n\n\n\n<p>On 15 June 2026 Google added a note to its <a href=\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\" target=\"_blank\" rel=\"noopener\">AI optimization guide<\/a>. The <a href=\"https:\/\/developers.google.com\/search\/updates\" target=\"_blank\" rel=\"noopener\">Search Central changelog<\/a> sums it up: these files &#8220;aren&#8217;t needed for Google Search (and won&#8217;t negatively or positively impact your visibility or rankings)&#8221;, but &#8220;it&#8217;s fine if you want to maintain these files for other services or systems that use them.&#8221;<\/p>\n\n\n\n<p>So for Google AI Overviews and AI Mode, the usual basics still apply: crawlable pages, clear answers and content people trust. Our guide to <a href=\"https:\/\/eoxysit.com\/blogs\/ai-overviews-and-google-search-what-businesses-need-to-know-in-2025\/\">AI Overviews and Google Search<\/a> covers those.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"crawlers\">AI crawlers: search vs training<\/h2>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"576\" src=\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/ai-crawlers-search-vs-training-1024x576.jpg\" alt=\"AI crawlers split into search and user fetchers versus training crawlers, with robots.txt tokens for each\" class=\"wp-image-19510\" srcset=\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/ai-crawlers-search-vs-training-1024x576.jpg 1024w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/ai-crawlers-search-vs-training-300x169.jpg 300w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/ai-crawlers-search-vs-training-768x432.jpg 768w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/ai-crawlers-search-vs-training-1536x864.jpg 1536w, https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/ai-crawlers-search-vs-training.jpg 1600w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption class=\"wp-element-caption\">Search and user fetchers affect whether AI answers can cite you; training tokens control model training.<\/figcaption><\/figure>\n\n\n\n<p>The key point is that most AI companies run separate bots for search, for user-requested fetches, and for training. Each can be allowed or blocked on its own.<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Company<\/th><th>Token<\/th><th>What it is for<\/th><th>Source<\/th><\/tr><\/thead><tbody><tr><td>OpenAI<\/td><td>OAI-SearchBot<\/td><td>Search results in ChatGPT<\/td><td><a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">OpenAI crawlers<\/a><\/td><\/tr><tr><td>OpenAI<\/td><td>GPTBot<\/td><td>Content that may be used for training foundation models<\/td><td><a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">OpenAI crawlers<\/a><\/td><\/tr><tr><td>Perplexity<\/td><td>PerplexityBot<\/td><td>Surfacing and linking sites in Perplexity search; not for training<\/td><td><a href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener\">Perplexity crawlers<\/a><\/td><\/tr><tr><td>Perplexity<\/td><td>Perplexity-User<\/td><td>Fetches a page when a user asks; generally ignores robots.txt<\/td><td><a href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener\">Perplexity crawlers<\/a><\/td><\/tr><tr><td>Anthropic<\/td><td>Claude-SearchBot<\/td><td>Indexing to improve Claude search results<\/td><td><a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">Anthropic help centre<\/a><\/td><\/tr><tr><td>Anthropic<\/td><td>ClaudeBot<\/td><td>Collecting content that could contribute to model training<\/td><td><a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">Anthropic help centre<\/a><\/td><\/tr><tr><td>Google<\/td><td>Google-Extended<\/td><td>Gemini training and grounding; does not affect Google Search<\/td><td><a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers\" target=\"_blank\" rel=\"noopener\">Google crawler docs<\/a><\/td><\/tr><\/tbody><\/table><figcaption class=\"wp-element-caption\">Checked 29 Oct 2026. OpenAI and Perplexity say robots.txt changes can take about 24 hours to apply.<\/figcaption><\/figure>\n\n\n\n<p>OpenAI&#8217;s docs give the exact example: a webmaster &#8220;can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot&#8221; to opt out of training. Google states that Google-Extended &#8220;does not impact a site&#8217;s inclusion in Google Search nor is it used as a ranking signal&#8221;.<\/p>\n\n\n\n<blockquote class=\"wp-block-quote is-layout-flow wp-block-quote-is-layout-flow\">\n<p><strong>Not sure which bots reach your site today?<\/strong> Eoxys checks robots.txt, CDN rules and server logs as part of an <a href=\"https:\/\/eoxysit.com\/blogs\/ai-visibility-audit\/\">AI visibility audit<\/a>. <a href=\"https:\/\/eoxysit.com\/contact\/\">Get a real project estimate within 24 hours<\/a>. NDA on request.<\/p>\n<\/blockquote>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"setup\">Setting it up on WordPress<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">1. Generate llms.txt with your SEO plugin<\/h3>\n\n\n\n<p><a href=\"https:\/\/yoast.com\/features\/llms-txt\/\" target=\"_blank\" rel=\"noopener\">Yoast SEO<\/a> can generate an llms.txt file &#8220;automatically with a click&#8221; in Yoast SEO and its premium editions. Other SEO plugins offer similar options. Check the output: it should list your service pages, key guides and contact page, not every tag archive.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">2. Review robots.txt<\/h3>\n\n\n\n<p>WordPress serves a virtual robots.txt unless a physical file exists. Edit it through your SEO plugin&#8217;s file editor or your host&#8217;s file manager, and keep a copy of the old version first.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">3. Check your CDN and firewall<\/h3>\n\n\n\n<p>Many CDNs and security plugins can block AI bots regardless of robots.txt. Cloudflare, for example, offers <a href=\"https:\/\/developers.cloudflare.com\/ai-crawl-control\/\" target=\"_blank\" rel=\"noopener\">AI Crawl Control<\/a> to analyse and manage AI crawler traffic. Make sure these rules match what you set in robots.txt.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">4. Verify<\/h3>\n\n\n\n<p>Open yoursite.com\/robots.txt and yoursite.com\/llms.txt in a private window, then check server or CDN logs over the following days for the user agents you allowed.<\/p>\n\n\n\n<p>Need help on a WordPress site? Our <a href=\"https:\/\/eoxysit.com\/wordpress\/\">WordPress development team<\/a> handles this kind of technical setup.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"robots\">A sample robots.txt<\/h2>\n\n\n\n<p>A common starting point for a business that wants AI search citations but not training use. Adjust it to your own policy:<\/p>\n\n\n<pre class=\"wp-block-code\"><code>User-agent: OAI-SearchBot\nAllow: \/\n\nUser-agent: PerplexityBot\nAllow: \/\n\nUser-agent: Claude-SearchBot\nAllow: \/\n\nUser-agent: GPTBot\nDisallow: \/\n\nUser-agent: ClaudeBot\nDisallow: \/\n\nUser-agent: Google-Extended\nDisallow: \/\n\nUser-agent: *\nDisallow: \/wp-admin\/\nAllow: \/wp-admin\/admin-ajax.php\n\nSitemap: https:\/\/example.com\/sitemap_index.xml<\/code><\/pre>\n\n\n<p>Blocking Google-Extended is a business decision. It does not affect Google Search, but it does stop your content being used to ground answers in Gemini Apps. If being cited in Gemini matters to you, leave it allowed.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"mistakes\">Common mistakes<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li>Blocking every AI bot with one rule, which also removes you from AI search answers.<\/li>\n\n\n<li>Expecting llms.txt to raise Google rankings.<\/li>\n\n\n<li>Allowing bots in robots.txt while a firewall blocks them.<\/li>\n\n\n<li>Listing thin or outdated pages in llms.txt.<\/li>\n\n\n<li>Editing robots.txt with no backup and accidentally blocking the whole site.<\/li>\n\n<\/ul>\n\n\n\n<p>Crawler access is one part of AEO. For the rest, see our <a href=\"https:\/\/eoxysit.com\/blogs\/aeo-services-ai-search-citations\/\">AEO services guide<\/a> and <a href=\"https:\/\/eoxysit.com\/blogs\/get-business-recommended-by-chatgpt\/\">how to get your business recommended by ChatGPT<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"faq\">FAQ<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Does llms.txt help SEO?<\/h3>\n\n\n\n<p>Not for Google. Google says llms.txt won&#8217;t positively or negatively affect visibility or rankings in Google Search. It may help other AI tools and agents that choose to read it, but no major search engine has said it uses the file for ranking. Treat it as an optional extra, after robots.txt, crawlable HTML and clear content.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Should I block GPTBot?<\/h3>\n\n\n\n<p>Blocking GPTBot asks OpenAI not to use your content for training. It does not remove you from ChatGPT search, which uses OAI-SearchBot, according to OpenAI&#8217;s crawler documentation. Each token is handled separately, so you can allow search while opting out of training. OpenAI says robots.txt changes can take about 24 hours to apply.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How do I add llms.txt to WordPress?<\/h3>\n\n\n\n<p>Use your SEO plugin&#8217;s llms.txt option (Yoast offers one) or upload a Markdown file to the site root. Then open yoursite.com\/llms.txt to confirm it loads. Check that a caching or security plugin, or your CDN, is not blocking the file or rewriting it.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Can Eoxys set this up for my site?<\/h3>\n\n\n\n<p>Yes. We review robots.txt, llms.txt, CDN rules and logs, then make the changes with a backup and confirm each AI crawler can reach the pages you want cited. <a href=\"https:\/\/eoxysit.com\/contact\/\">Get a real project estimate within 24 hours<\/a>, with an NDA on request.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"work-with-eoxys\">Work with Eoxys<\/h2>\n\n\n\n<p>Eoxys IT Solution LLP has built websites, apps and AI products since 2009, with 700+ projects delivered. See the work on our <a href=\"https:\/\/eoxysit.com\/portfolio\/\">portfolio<\/a>, or explore our <a href=\"https:\/\/eoxysit.com\/artificial-intelligence-and-machine-learning-solutions\/\">AI and machine learning solutions<\/a>.<\/p>\n\n\n\n<p>We set up AI crawler access, llms.txt and AEO basics on WordPress sites, with a backup of every change. <a href=\"https:\/\/eoxysit.com\/contact\/\">Get a real project estimate within 24 hours<\/a>. NDA on request. Prefer to talk? Call +91 96678 22661.<\/p>\n\n\n\n<p><strong>Related reading:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li><a href=\"https:\/\/eoxysit.com\/blogs\/aeo-services-ai-search-citations\/\">AEO Services: Get Cited in ChatGPT and Google AI Overviews<\/a><\/li>\n\n\n<li><a href=\"https:\/\/eoxysit.com\/blogs\/ai-overviews-and-google-search-what-businesses-need-to-know-in-2025\/\">Google AI Overviews: What Businesses Need to Know in 2026<\/a><\/li>\n\n\n<li><a href=\"https:\/\/eoxysit.com\/blogs\/ai-visibility-audit\/\">AI Visibility Audit: Check If ChatGPT and AI Overviews Cite You<\/a><\/li>\n\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"sources\">Sources<\/h2>\n\n\n\n<p>Checked on 29 Oct 2026 against Google Search Central, the llmstxt.org proposal, OpenAI, Perplexity, Anthropic, Yoast and Cloudflare documentation.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n\n<li><a href=\"https:\/\/developers.google.com\/search\/updates\" target=\"_blank\" rel=\"noopener\">Google Search Central: Latest updates (15 June 2026 llms.txt note)<\/a><\/li>\n\n\n<li><a href=\"https:\/\/developers.google.com\/search\/docs\/fundamentals\/ai-optimization-guide\" target=\"_blank\" rel=\"noopener\">Google: AI optimization guide<\/a><\/li>\n\n\n<li><a href=\"https:\/\/llmstxt.org\/\" target=\"_blank\" rel=\"noopener\">llmstxt.org: The \/llms.txt file<\/a><\/li>\n\n\n<li><a href=\"https:\/\/developers.openai.com\/api\/docs\/bots\" target=\"_blank\" rel=\"noopener\">OpenAI: Overview of OpenAI crawlers<\/a><\/li>\n\n\n<li><a href=\"https:\/\/docs.perplexity.ai\/guides\/bots\" target=\"_blank\" rel=\"noopener\">Perplexity: Crawlers<\/a><\/li>\n\n\n<li><a href=\"https:\/\/support.claude.com\/en\/articles\/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler\" target=\"_blank\" rel=\"noopener\">Anthropic: Does Anthropic crawl data from the web?<\/a><\/li>\n\n\n<li><a href=\"https:\/\/developers.google.com\/crawling\/docs\/crawlers-fetchers\/google-common-crawlers\" target=\"_blank\" rel=\"noopener\">Google: Common crawlers (Google-Extended)<\/a><\/li>\n\n\n<li><a href=\"https:\/\/yoast.com\/features\/llms-txt\/\" target=\"_blank\" rel=\"noopener\">Yoast: llms.txt feature<\/a><\/li>\n\n\n<li><a href=\"https:\/\/developers.cloudflare.com\/ai-crawl-control\/\" target=\"_blank\" rel=\"noopener\">Cloudflare: AI Crawl Control<\/a><\/li>\n\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>What llms.txt does and does not do, which AI crawlers to allow or block in robots.txt, and how to set both up safely on a WordPress site.<\/p>\n","protected":false},"author":4,"featured_media":19488,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_yoast_wpseo_title":"llms.txt and AI Crawlers on WordPress: What to Allow","_yoast_wpseo_metadesc":"llms.txt on WordPress: what Google says about it, which AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) to allow, and safe setup steps.","footnotes":""},"categories":[156],"tags":[153,174,155,173],"class_list":["post-19511","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-search-aeo","tag-aeo","tag-ai-crawlers","tag-ai-search-visibility","tag-llms-txt"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v26.8 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>llms.txt and AI Crawlers on WordPress: What to Allow<\/title>\n<meta name=\"description\" content=\"llms.txt on WordPress: what Google says about it, which AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) to allow, and safe setup steps.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"llms.txt and AI Crawlers on WordPress: What to Allow\" \/>\n<meta property=\"og:description\" content=\"llms.txt on WordPress: what Google says about it, which AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) to allow, and safe setup steps.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/\" \/>\n<meta property=\"og:site_name\" content=\"EoXys IT\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/EoxysIt\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-11-05T05:45:00+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"900\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Shafeeq Khan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Shafeeq Khan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"BlogPosting\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/\"},\"author\":{\"name\":\"Shafeeq Khan\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/#\/schema\/person\/cdbb21706e1f044261a3f6f1ed28074e\"},\"headline\":\"llms.txt and AI Crawlers on WordPress: What to Allow in 2026\",\"datePublished\":\"2026-11-05T05:45:00+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/\"},\"wordCount\":1392,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/#organization\"},\"image\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg\",\"keywords\":[\"AEO\",\"AI crawlers\",\"AI search visibility\",\"llms.txt\"],\"articleSection\":[\"AI Search & AEO\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#respond\"]}],\"dateModified\":\"2026-11-05T05:45:00+00:00\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/\",\"url\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/\",\"name\":\"llms.txt and AI Crawlers on WordPress: What to Allow\",\"isPartOf\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg\",\"datePublished\":\"2026-11-05T05:45:00+00:00\",\"description\":\"llms.txt on WordPress: what Google says about it, which AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) to allow, and safe setup steps.\",\"breadcrumb\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage\",\"url\":\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg\",\"contentUrl\":\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg\",\"width\":1600,\"height\":900,\"caption\":\"AEO vs SEO vs GEO: a search bar above an AI answer panel that cites three web sources\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/eoxysit.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"llms.txt and AI Crawlers on WordPress: What to Allow in 2026\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/#website\",\"url\":\"https:\/\/eoxysit.com\/blogs\/\",\"name\":\"EoXys IT\",\"description\":\"Digitise your business\",\"publisher\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/#organization\"},\"alternateName\":\"Eoxys It\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/eoxysit.com\/blogs\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/#organization\",\"name\":\"Eoxys IT Solution\",\"alternateName\":\"Eoxys IT Solution LLP\",\"url\":\"https:\/\/eoxysit.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/01\/cropped-favicon.png\",\"contentUrl\":\"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/01\/cropped-favicon.png\",\"width\":512,\"height\":512,\"caption\":\"Eoxys IT Solution\"},\"image\":{\"@id\":\"https:\/\/eoxysit.com\/blogs\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/EoxysIt\/\",\"https:\/\/www.linkedin.com\/company\/eoxysitsolution\",\"https:\/\/www.instagram.com\/eoxysitsolution\/\"],\"legalName\":\"Eoxys IT Solution LLP\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/#\/schema\/person\/cdbb21706e1f044261a3f6f1ed28074e\",\"name\":\"Shafeeq Khan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/eoxysit.com\/blogs\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/333febb8e01a61e722548ec5b1caa9d9?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/333febb8e01a61e722548ec5b1caa9d9?s=96&d=mm&r=g\",\"caption\":\"Shafeeq Khan\"},\"description\":\"Shafeeq Khan is a web developer who creates modern, responsive, and user-friendly websites with clean code and creative design\",\"url\":\"https:\/\/eoxysit.com\/blogs\/author\/shafeeq\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"llms.txt and AI Crawlers on WordPress: What to Allow","description":"llms.txt on WordPress: what Google says about it, which AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) to allow, and safe setup steps.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/","og_locale":"en_US","og_type":"article","og_title":"llms.txt and AI Crawlers on WordPress: What to Allow","og_description":"llms.txt on WordPress: what Google says about it, which AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) to allow, and safe setup steps.","og_url":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/","og_site_name":"EoXys IT","article_publisher":"https:\/\/www.facebook.com\/EoxysIt\/","article_published_time":"2026-11-05T05:45:00+00:00","og_image":[{"width":1600,"height":900,"url":"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg","type":"image\/jpeg"}],"author":"Shafeeq Khan","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Shafeeq Khan","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"BlogPosting","@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#article","isPartOf":{"@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/"},"author":{"name":"Shafeeq Khan","@id":"https:\/\/eoxysit.com\/blogs\/#\/schema\/person\/cdbb21706e1f044261a3f6f1ed28074e"},"headline":"llms.txt and AI Crawlers on WordPress: What to Allow in 2026","datePublished":"2026-11-05T05:45:00+00:00","mainEntityOfPage":{"@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/"},"wordCount":1392,"commentCount":0,"publisher":{"@id":"https:\/\/eoxysit.com\/blogs\/#organization"},"image":{"@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage"},"thumbnailUrl":"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg","keywords":["AEO","AI crawlers","AI search visibility","llms.txt"],"articleSection":["AI Search & AEO"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#respond"]}],"dateModified":"2026-11-05T05:45:00+00:00"},{"@type":"WebPage","@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/","url":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/","name":"llms.txt and AI Crawlers on WordPress: What to Allow","isPartOf":{"@id":"https:\/\/eoxysit.com\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage"},"image":{"@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage"},"thumbnailUrl":"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg","datePublished":"2026-11-05T05:45:00+00:00","description":"llms.txt on WordPress: what Google says about it, which AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot) to allow, and safe setup steps.","breadcrumb":{"@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#primaryimage","url":"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg","contentUrl":"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/10\/eoxys-06-aeo-ai-search.jpg","width":1600,"height":900,"caption":"AEO vs SEO vs GEO: a search bar above an AI answer panel that cites three web sources"},{"@type":"BreadcrumbList","@id":"https:\/\/eoxysit.com\/blogs\/llms-txt-ai-crawlers-wordpress\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/eoxysit.com\/"},{"@type":"ListItem","position":2,"name":"llms.txt and AI Crawlers on WordPress: What to Allow in 2026"}]},{"@type":"WebSite","@id":"https:\/\/eoxysit.com\/blogs\/#website","url":"https:\/\/eoxysit.com\/blogs\/","name":"EoXys IT","description":"Digitise your business","publisher":{"@id":"https:\/\/eoxysit.com\/blogs\/#organization"},"alternateName":"Eoxys It","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/eoxysit.com\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/eoxysit.com\/blogs\/#organization","name":"Eoxys IT Solution","alternateName":"Eoxys IT Solution LLP","url":"https:\/\/eoxysit.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/eoxysit.com\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/01\/cropped-favicon.png","contentUrl":"https:\/\/eoxysit.com\/blogs\/wp-content\/uploads\/2026\/01\/cropped-favicon.png","width":512,"height":512,"caption":"Eoxys IT Solution"},"image":{"@id":"https:\/\/eoxysit.com\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/EoxysIt\/","https:\/\/www.linkedin.com\/company\/eoxysitsolution","https:\/\/www.instagram.com\/eoxysitsolution\/"],"legalName":"Eoxys IT Solution LLP"},{"@type":"Person","@id":"https:\/\/eoxysit.com\/blogs\/#\/schema\/person\/cdbb21706e1f044261a3f6f1ed28074e","name":"Shafeeq Khan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/eoxysit.com\/blogs\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/333febb8e01a61e722548ec5b1caa9d9?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/333febb8e01a61e722548ec5b1caa9d9?s=96&d=mm&r=g","caption":"Shafeeq Khan"},"description":"Shafeeq Khan is a web developer who creates modern, responsive, and user-friendly websites with clean code and creative design","url":"https:\/\/eoxysit.com\/blogs\/author\/shafeeq\/"}]}},"_links":{"self":[{"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/posts\/19511","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/comments?post=19511"}],"version-history":[{"count":1,"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/posts\/19511\/revisions"}],"predecessor-version":[{"id":19524,"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/posts\/19511\/revisions\/19524"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/media\/19488"}],"wp:attachment":[{"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/media?parent=19511"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/categories?post=19511"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/eoxysit.com\/blogs\/wp-json\/wp\/v2\/tags?post=19511"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}