Site settings
AI crawlers, robots.txt and llms.txt
Choose separately whether AI search tools may read and cite your site and whether AI companies may train on it, control each bot, add your own robots.txt rules, and publish an llms.txt guide to your key pages.
Go to Kufi → SEO → Settings → AI and crawlers.
Two separate choices
Let AI search tools read and cite the site (on by default, recommended). These bots fetch your pages to answer people's questions and link back to you:
| Bot | Used for |
|---|---|
| OAI-SearchBot | ChatGPT search results |
| ChatGPT-User | Pages ChatGPT opens for a user |
| Claude-SearchBot | Claude search results |
| Claude-User | Pages Claude opens for a user |
| PerplexityBot | Perplexity answers |
| Perplexity-User | Pages Perplexity opens for a user |
Allow AI model training on the content (on by default). These collect content to train AI models, such as GPTBot, ClaudeBot, Google-Extended and Common Crawl. Turning this off doesn't affect Google or Bing search results.
Many sites keep AI search on, for the visits and citations, and decide about training separately.
Bot by bot
Choose crawler by crawler lists every known AI bot with its owner and purpose. For each one: Follow the switch above, Allow or Block.
Good to know: robots.txt is a request that reputable crawlers honor; it isn't access control. To truly block a bot, use a firewall at your host or CDN.
Your own robots.txt rules
Extra rules are added after the WordPress defaults. Only User-agent, Allow, Disallow, Crawl-delay and Sitemap lines are kept. Current file opens your live robots.txt.
Kufi SEO adds your sitemap address to robots.txt automatically.
llms.txt
Publish llms.txt (on by default) creates https://your-site.com/llms.txt: a plain list of your key pages with short descriptions, in a format proposed for AI tools.
- Introduction: two or three sentences on who you are, what you offer and who it's for.
- Which content types to list, and Items per type.
Some AI assistants and agents read llms.txt. The big search engines haven't said they use it, so treat it as a bonus rather than a ranking factor.
Make your pages easy to quote
The SEO box's AI answers checks help: start with a direct answer, phrase some headings as questions, and use lists and tables. See Focus keywords and the score.
More in Site settings
Still need help?
Tell us what you are trying to do and what you see. We read every message.