Why controlling AI use of your content matters
Your website is one of the main sources AI companies draw on to learn about your company and to answer questions about it. That cuts two ways. Some companies are uneasy about their writing, research, or product pages being used to train AI models they'll never be paid for. Those same companies usually want ChatGPT, Gemini, Perplexity, and Claude to recommend them when a buyer asks for a product like theirs.
Those two goals feel like they're in conflict, but they mostly aren't, because AI companies don't send a single crawler to your site. They send several, each with a different purpose:
- Training crawlers collect pages to train future AI models. OpenAI's GPTBot, Anthropic's ClaudeBot, Apple's Applebot-Extended, Meta's Meta-ExternalAgent, and Common Crawl's CCBot do this work. Common Crawl publishes an open archive of the web that many AI companies train on.
- Search crawlers build the index an assistant searches when it answers a question. OpenAI's OAI-SearchBot, Anthropic's Claude-SearchBot, and PerplexityBot do this work.
- On-demand fetchers visit a page when a person asks an assistant to read it. ChatGPT-User, Claude-User, and Perplexity-User do this work.
Google works a little differently. Google-Extended controls whether Gemini can train on your pages and use them to ground its answers, and Google says it has no effect on your Google Search ranking. Google's AI Overviews come from regular Google Search, so they follow your rules for Googlebot.
The work, then, is to decide what you want each kind of crawler to do, write that decision into your robots.txt file correctly for every crawler that matters, and check afterward that you blocked what you intended and nothing more. It sounds simple, and the file itself is short. The trouble is that crawler names change, new ones appear, and a single misplaced line can take your whole site out of AI search without anyone noticing for months. That makes it a good job for an AI agent that knows the current crawlers and can check the result.
Kite is an AI marketer. It researches your business, builds a growth strategy, and ships the work that brings you new customers. Controlling how AI systems use your site is part of the work it does to make your website readable by AI. Here's how it handles the job.
How Kite controls how AI companies use your content
Kite starts by asking what you want, in plain terms. Most teams fall into one of three positions: let AI use everything, block training but stay visible in AI answers, or block AI entirely. Kite explains what each choice costs you in AI visibility before it changes anything.
Next, Kite translates your choice into rules for each named crawler and, if you want, a Content Signals line that states your terms. It publishes the change where your site serves robots.txt, or gives you the finished rules and tells you where they go.
Once the change is live, Kite confirms the result. It checks that the file says what you intended, that the crawlers you kept open can still reach your pages, and, when you track AI answers with Kite, whether your company still appears in them.
- Can you block AI training but still show up in ChatGPT answers?
- What happens if you block all AI crawlers?
- How do you set terms for how AI can use your content?
- Put the rules where your site serves them
- Check that the rules work and see what they cost you
Can you block AI training but still show up in ChatGPT answers?
Yes. You block the training crawlers and leave the search crawlers and on-demand fetchers open. That's the policy Kite writes when you ask to stop AI training while staying visible in AI answers. It blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, and CCBot, and it keeps OAI-SearchBot, Claude-SearchBot, PerplexityBot, and the on-demand fetchers allowed. Here's a simplified version of the rules:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
Allow: /With these rules, ChatGPT can still find your pricing page when someone asks it to compare products like yours, and it can still cite that page. Meanwhile, your rules ask OpenAI not to use the page to train its next model.
The cost is small but real, and Kite tells you about it. Some AI answers don't come from a live search at all. They come from what the model learned in training, and when a model hasn't seen your site, it knows less about you. Blocking training mostly affects future models, because a block doesn't remove pages that were collected before it went up. For most companies that trade-off is worth it, and Kite lays it out so you can decide.
What happens if you block all AI crawlers?
If you block every AI crawler, the assistants that respect your robots.txt file stop reading your site. ChatGPT, Claude, and Perplexity can no longer find your pages when they search or cite them as sources. OpenAI and Perplexity do say their fetchers may still open a page when a user asks for it directly, because the user requested it. They may still mention your company based on older training data and on what other websites say about you, but they can't point to your own pages to back it up.
Because that's a large cost, Kite warns you before it makes the change. It explains that blocking all AI crawlers takes you out of AI answers and works directly against any effort to get your company cited. If you still want it, Kite blocks every crawler on its list and sets your Content Signals to say no to AI training and no to use in AI answers. It keeps the signal for ordinary search set to yes, so you stay in Google and Bing results. Kite only blocks regular search engines if you explicitly ask it to.
After the change, Kite tells you in one sentence exactly what is now blocked and what is still visible, so nobody on your team is surprised later.
How do you set terms for how AI can use your content?
You set terms by adding a Content Signals line to your robots.txt file. Cloudflare introduced Content Signals in 2025 as a way for a website to state, in a form machines can read, how its content may be used after it has been collected. There are three signals, and each one is a simple yes or no:
- Search covers building a search index and showing your pages in search results with links back to you.
- AI input covers feeding your pages into an AI model to answer a question in real time, the way an assistant does when it searches the web and summarizes what it finds.
- AI training covers using your pages to train or fine-tune an AI model.
A team that wants to stay visible everywhere but opt out of training would publish this line:
Content-Signal: search=yes, ai-input=yes, ai-train=noCrawler rules and Content Signals do different jobs, so Kite writes them together. The crawler rules say who may visit. The signals say what anyone who does visit may do with what they collect. The signals matter most for crawlers that serve several purposes at once, where a block-or-allow rule can't express what you want.
Kite adds Content Signals when you ask to declare your terms, and it sets each value to match the policy you chose.
Put the rules where your site serves them
The rules only work when they appear at yoursite.com/robots.txt, and how they get there depends on how your website is built. Kite checks your setup before it writes anything.
- A website Kite builds and hosts. Kite updates the robots.txt file itself. New sites Kite builds allow AI crawlers by default and name each major one in the file, so changing the policy means moving specific crawlers from the allowed group to the blocked group. Kite keeps your sitemap listed at the bottom of the file, because crawlers use that line to find all your pages.
- A site Kite works on through your GitHub repository. Kite makes the change in your repository and merges it into your main branch, or opens a pull request when that branch is protected. Your own deployment then ships it.
- A site on another platform, such as Webflow or WordPress. Kite writes the finished rules and tells you where they go on your platform, so you can paste them in.
On a site Kite hosts, the change waits as an unpublished change that you can preview. Nothing goes live until you publish it.
Check that the rules work and see what they cost you
Kite confirms that your site does what you decided, because a robots.txt file can look right and still behave differently. After the change, Kite reads the live file to make sure the rule groups and the sitemap line came through as intended.
Kite's readiness scan, which runs 24 checks on how AI systems read a website, then tests the practical side. It checks whether your robots.txt names AI crawlers explicitly, whether AI crawlers can actually reach your pages, and whether bot protection on your site is turning them away. That last check catches a common problem. A firewall or bot-blocking setting can quietly reject AI search crawlers even when your robots.txt welcomes them, and the effect is the same as blocking them on purpose. When the scan recommends opening access, Kite only reopens the search and answer crawlers the scan names. Your training blocks stay in place, because they're your decision. Check your website's readiness for AI agents explains the full scan.
If you track AI answers with Kite, it runs your buyers' questions through ChatGPT, Gemini, Perplexity, and Claude again after the change. That shows whether your company still appears and which of your pages still get cited, so you can see the real cost of your policy rather than guess at it. Improve your visibility in AI answers covers that measurement loop.
What you get
A plain explanation of your choices
Kite explains what each policy allows, what it blocks, and what it costs you in AI answers before anything changes.
Crawler rules for each AI company
Your robots.txt file names each major AI crawler and blocks or allows it according to the policy you chose.
Your terms in writing
When you want them, a Content Signals line states whether your content may be used for search, AI answers, and AI training.
The change in place
Kite publishes it on a site it hosts or through your GitHub repository, and hands you the finished rules for any other platform.
A one-sentence summary of the result
Kite tells you exactly what is now blocked and what is still visible.
A check that it worked
Kite scans the live site to confirm that the crawlers you kept open can reach your pages, and it can measure your AI visibility again to show the effect.
How to get started
Tell Kite what you want in plain language. You might write, "Stop AI companies from training on our site, but keep us in ChatGPT and Perplexity answers," or "What would happen if we blocked all AI crawlers?"
You can work with Kite in your browser or in Slack. Kite reads your current robots.txt file, explains what it allows today, and proposes the change that matches what you asked for. If your request would cost you AI visibility, Kite says so before it goes further. You approve the change with the Approve button or a reply, or tell Kite what to adjust.
If Kite built and hosts your website, you don't need to connect anything. If Kite works on your site through GitHub, connect the repository and Kite can make the change there. For a site on any other platform, Kite gives you the finished rules to paste in.
By default, Kite asks before it publishes anything to your website. Once you trust its work, you can give it more room to act on its own.
If you want to see how AI answers describe your company before you change anything, try Kite's free AI visibility checker. It asks ChatGPT, Gemini, Perplexity, and Claude the questions your customers ask, and in about 45 seconds it shows whether they recommend you and who they recommend instead.
Integrations used to control how AI companies use your content
Kite writes the rules, publishes them to the sites it hosts, and checks the result with its own built-in tools. A connection comes in when your website's code lives in your own repository.
Common questions
Does robots.txt actually stop AI companies from using my content?
Robots.txt is a request, not a lock. OpenAI, Anthropic, Google, Apple, Perplexity, Meta, and Common Crawl all publish the names of their crawlers and say they follow robots.txt, so for the companies most people care about, it works. Crawlers that ignore it can only be stopped at the network level, for example with a bot-blocking setting from your hosting provider. Kite's readiness scan reports whether bot protection is active on your site, so you can see whether it's affecting the crawlers you want to keep.
If I block AI training now, does that remove my content from existing AI models?
No. A block stops future crawls, so it keeps your pages out of training data collected from that point on. Pages that were collected before the block may already be part of current models, and robots.txt can't take them back. That's one reason to set your policy early, before you publish content you care most about protecting.
Does blocking Google-Extended keep my site out of Google's AI Overviews?
No. Google-Extended controls whether Gemini can train on your content and use it to ground its answers. AI Overviews are part of Google Search and follow your rules for Googlebot, the same crawler that decides whether you appear in Google at all. Blocking Googlebot would remove you from Google Search entirely, so for most companies, appearing in AI Overviews comes with appearing in Google.
Will blocking AI training hurt my Google rankings?
No. Google says Google-Extended has no effect on your Search ranking, and the other training crawlers don't belong to search engines. Kite leaves Googlebot, Bingbot, and the other ordinary search crawlers open unless you specifically ask it to block them.
Can I keep AI training crawlers out of just part of my site?
Yes. Robots.txt rules can apply to a specific section instead of your whole site. For example, you could block training crawlers from a paid research library or a members' area and leave your product and pricing pages open to everyone. Tell Kite which sections you want to protect, and it writes the rules for those paths.
If I block AI crawlers, should I still keep an llms.txt file?
It's worth keeping if you leave any AI search crawlers open. An llms.txt file is a short guide that tells AI systems what your company does and which pages answer which questions. It helps the assistants that are allowed to read your site, but it does nothing for a company whose crawlers you've blocked. Create an llms.txt file for your website explains how Kite writes and maintains one.