Why test your AI visibility changes one at a time?
Most teams working on AI visibility have already made changes. They added an FAQ, rewrote the homepage, published a comparison page, cleaned up their structured data, or got listed in a roundup. Then someone asks whether any of it worked, and nobody can say. The changes went out together, nobody wrote down the starting point, and the only evidence is a screenshot of ChatGPT mentioning the company once.
That uncertainty is expensive. If you can't tell which changes moved your visibility, you can't do more of what works, and you keep paying for work that doesn't. Testing gives you that answer. It treats AI visibility the way a good marketer treats any channel: measure where you are, change one thing, and measure again.
AI answers make this harder than testing a web page, for two reasons. First, the same question can produce a different answer each time it's asked, so one before-and-after pair can show a gain or a loss that is pure chance. Second, answer engines take time to pick up changes, often weeks, so checking the day after you publish tells you nothing. A useful test asks the same questions many times, waits long enough, and refuses to call a result until the numbers support it.
That patient, repetitive work suits an AI agent. It can ask the same 30 questions every day without getting bored, remember exactly what shipped and when, and come back on the right date without being reminded. Kite is an AI marketer. It researches your business, builds a growth strategy, and ships the work that brings you new customers. Here's how it runs an AI visibility experiment.
How Kite tests what improves your AI visibility
Kite runs each test as a loop with four parts. It agrees with you on a fixed set of questions your buyers ask AI assistants. It measures how often the answers name your company before anything changes. It ships a small number of changes and writes down exactly what they were and when they went live. Then, after the answer engines have had time to notice, it measures the same questions again and reports which questions gained, which lost, and which didn't move.
The result of each round shapes the next one. A change that worked suggests more of the same on other questions. A change that didn't move anything tells Kite to try a different approach, such as earning a mention on a third-party page instead of rewriting your own.
- Which questions should an AI visibility test measure?
- How do I measure AI visibility before a change?
- What should I change first to improve AI visibility?
- Ship the change
- How long should I wait to measure the result?
- Did the change actually improve my AI visibility?
- Plan the next round
Which questions should an AI visibility test measure?
An AI visibility test should measure a fixed set of questions that real buyers ask, and it should keep that set the same in every round. If the questions change between measurements, the before and after numbers describe different things and can't be compared.
Kite proposes the questions from what it already knows about your positioning, your customers, and your competitors. A typical set covers the questions where you'd want to be recommended:
- Category questions, such as "What's the best project management tool for agencies?"
- Problem questions, such as "How do I stop clients from missing deadlines?"
- Comparison questions, such as "Asana vs your company for small teams"
- Questions about the capabilities you want to be known for, such as "Which tools have built-in client approvals?"
You can add exact questions of your own, and Kite includes them word for word in every measurement. Once you agree on the set, Kite saves it and reuses it for every round, so the third test is still comparable with the first.
Related How-to pages
- How to choose which questions to track for AI visibility: Pick the buyer questions worth measuring and leave out the ones that only tell you what an AI already knows about your name.
- How to run an AI visibility audit for your brand: Take a full first reading of where you appear, who appears instead, and which sources the answers cite.
How do I measure AI visibility before a change?
You measure the starting point by asking each tracked question many times over a few weeks and counting how often the answers name your company. That count, such as "named in 7 of 40 answers," is your baseline. It has to be taken before the change ships, because a measurement taken afterward can't tell you what things looked like before.
Kite asks your tracked questions on a schedule, daily by default, or weekly or monthly if you'd rather spend less. For every answer, it records whether your company is named, which competitors are named, and which pages the answer cites. Over time this builds a series of answers rather than a single snapshot, and that series is what makes a fair comparison possible.
Kite reports visibility with its counts every time, so you can see how much evidence sits behind a number. It also tracks your share of voice, which is your company's portion of all the brand mentions across your questions, and how often answers cite your own website.
If you ask whether a change that already shipped worked, Kite looks for a measurement dated before that change. If there isn't one, it tells you plainly, reports where you stand now, and offers to make today's reading the baseline for the next test. It won't present an after-only reading as a before-and-after result.
Related How-to pages
- How to measure how often your brand appears in AI answers with DataForSEO: Run your tracked questions through the answer engines and record the mentions, citations, and competitors in each answer.
What should I change first to improve AI visibility?
Change the thing your baseline shows is missing, and change only one or two things per round. If you publish five pages and restructure your site in the same week, a rise in mentions can't be credited to any one of them.
Kite chooses the change by reading the answers from the baseline. For each question where you're missing, it opens the pages the answer cites and asks why those pages won. The reason usually points to one of three kinds of change:
- Answer the question directly on your site. When no page of yours answers the question plainly, Kite writes one, such as a comparison page, a "best tools for this job" page, or a clearer explanation of your pricing.
- Make your pages easier for AI systems to read. When the content exists but is hard for crawlers to find or understand, Kite fixes the structured data, the
llms.txtfile, or the crawler rules. - Get included where the answers already look. When the answers keep citing a review site, a directory, or a roundup that leaves you out, Kite finds who runs that page and drafts the outreach for you to send.
Kite writes down each change, the questions it was meant to win, and the date it went live. That record is what lets the next measurement say something about cause rather than coincidence.
Related How-to pages
- How to find out why competitors get cited in AI answers with Semrush: Compare the pages and search rankings behind a competitor's appearances, so you know which change is most likely to close the gap.
Ship the change
Kite builds and publishes the change itself when it can, and prepares it for you when it can't. On a website Kite hosts, it writes and publishes the page, updates the structured data, or edits llms.txt directly. If your site runs on Webflow, WordPress, Contentful, Sanity, or Storyblok, it publishes through your connected content system. For a third-party page, it drafts the message and you send it from your own account.
Kite asks for your approval before anything goes live. You can approve it with the Approve or Publish button in the conversation, or with a reply.
If your team already made the change, Kite skips this step. Tell it which page changed and the date, and it measures the change you made.
How long should I wait to measure the result?
Wait at least two to four weeks after the change goes live before you judge it. Answer engines revisit and re-read the web slowly, so a new page or a fixed crawler rule can take weeks to show up in answers. Checking too early mostly measures the time it takes them to notice.
Kite keeps asking your tracked questions on its normal schedule during the wait, so no extra effort is needed to gather the after period. For each change it ships against your tracked questions, it sets the judgment date in advance: the result covers the 28 days after the work finished, compared with the 28 days before. The window stays the same no matter how often your questions are measured, so a result can't be called early.
Did the change actually improve my AI visibility?
A change improved your AI visibility only if the questions it targeted were answered with your company named noticeably more often afterward, by more than the ordinary swing between runs. A rise from 3 of 40 answers to 5 of 40 can easily happen by chance. A rise from 3 of 40 to 18 of 40 almost never does.
Kite applies that standard for you. Every change it reports carries one of a small set of verdicts: up, down, gained, lost, no measurable change, or not enough data yet. When the difference falls within normal variation, Kite says "no measurable change," even if the raw number went up. When too few answers were collected, or too few questions were tracked, it says there isn't enough data yet rather than guessing.
Kite reports the movement question by question, with the before and after counts for each one. That detail helps separate your change from everything else. If the questions your new comparison page targeted rose while the rest of your questions stayed flat, the page is the likely reason. If every question rose, including ones the change never touched, something broader happened, such as an update to the answer engine itself, and Kite says so rather than giving your change the credit.
For pages Kite wrote and published, it can also check Google search results. With Search Console connected, it compares the page's click-through rate over the 28 days before and after publication, so you can see whether the change helped in search as well as in AI answers.
Related How-to pages
- How to compare AI mentions with Google search performance using Search Console: Put your AI visibility next to the searches, impressions, and clicks your pages earn, to see whether the two moved together.
Plan the next round
Each result tells Kite what to try next. A change that worked on one question suggests the same kind of change for similar questions. A change that showed no measurable effect tells Kite to try a different lever, such as third-party coverage instead of another page on your own site.
Kite saves every round in your team's knowledge base: the question set, the changes and their dates, the before and after counts, and the verdict. That history keeps the next test comparable with the last one and stops anyone from repeating an experiment that already has an answer. When you're ready, Kite brings the next round's proposed changes to you for approval. For the broader work of finding where you're missing and closing those gaps, see Improve your visibility in AI answers, and for ongoing monitoring without a specific change to test, see Track whether AI assistants recommend your brand.
What you get
A fixed set of buyer questions
The questions your test measures, saved and reused so every round compares like with like.
A baseline
How often answers to those questions named your company before the change, with the counts behind each number, your share of voice, and the competitors who appeared instead.
A record of what changed
Each change, the questions it was meant to win, and the date it went live.
The change itself
The pages, structured data, and llms.txt updates Kite publishes after your approval, and the outreach it drafts for pages someone else controls.
A verdict for each change
Before and after counts for the targeted questions over 28 days on each side, with a plain label: up, down, no measurable change, or not enough data yet.
A plan for the next round
The changes Kite thinks are worth testing next, based on what this round showed.
How to get started
Start by seeing where you stand today. Kite's free AI visibility checker asks ChatGPT, Gemini, Perplexity, and Claude the questions your customers ask, and in about 45 seconds it shows whether they recommend you and who they recommend instead. It's free and needs no account.
To run a test, tell Kite what you want to find out in plain language. You might say, "We're publishing a comparison page against our biggest competitor next week. Can you tell me whether it gets us recommended more?" or "We added FAQs to our product pages on October 1. Did that change anything in ChatGPT?"
Kite proposes the questions to track and the competitors to watch, and you approve or adjust the list before the first measurement. It works best when you ask before you ship, because Kite can capture the baseline while it still describes the old version of your site. You can work with Kite in your browser or in Slack.
You don't need to connect anything to measure your AI visibility, because Kite's measurement is built in. Connect Google Search Console if you want Kite to compare the result with your search clicks and impressions, and connect your content system if your website runs somewhere Kite doesn't host.
Integrations used to test your AI visibility
Kite's AI visibility measurement is built in, so you can run your first test without another account. Connected tools add your own search data and help Kite pick which change is most worth testing.
Measure your AI visibility before and after
Check the result against your search performance
Common questions
How do I run a controlled test for AI search?
You can't run a true split test on an AI answer, because every answer engine reads the same public version of your website, so there's no way to keep half of them on the old page. The closest practical design is a before-and-after test with a comparison group. Change pages that target some of your tracked questions, leave the rest alone, and compare how the two groups moved over the same weeks. If only the targeted questions rose, your change is the likely cause. Kite reports results question by question, which gives you that comparison automatically.
Why does ChatGPT give different answers to the same question?
AI assistants generate each answer fresh, so wording, recommended brands, and cited sources can vary from one run to the next, even for an identical question. Some assistants also search the web while answering, and the pages they find can change from day to day. That variation is why a single check can't tell you whether a change worked. Kite asks each question repeatedly and reports how often you're named across all the answers, not whether you appeared once.
How many questions do I need for an AI visibility test?
Somewhere between 10 and 30 questions is a practical range for most companies. Fewer than that, and one unusual question can swing the whole result. Kite won't report an overall change across fewer than five questions, because so small a set can't separate a real shift from chance. A larger set costs more to measure on a daily schedule, so choose questions that matter to your buyers rather than adding variations of the same one.
Can I test several changes at once?
You can, but you won't learn which one worked. If you publish a comparison page, fix your structured data, and get listed in a roundup in the same week, a rise in mentions can only be credited to the bundle. Kite recommends shipping one or two changes per round. When several have to go out together, Kite records them as one experiment and reports the result that way.
What is a good AI visibility score?
There's no universal good number, because visibility depends on the questions you track and how crowded your category is. Being named in 20 percent of answers might lead a busy category and trail a narrow one. The more useful comparison is with yourself and your competitors on the same questions over time. For a fuller explanation, see What is an AI visibility score, and what counts as a good one? Coming soon . If you're comparing measurement tools, see Best AI Visibility Tools for Small Teams.