Measure a website change without an A/B test

Kite chooses the measurement design based on two things: whether the change can be split between visitors, and whether your traffic can reach an answer in a reasonable time. Getting this choice right matters, because a before-and-after comparison run on a change that could have been tested properly throws away a better answer.

Some changes reach every visitor at once, so they can't be split no matter how much traffic you have. Kite recognizes these and says so plainly:

• A new heading font or brand color. The site's styles apply to every page and every visitor together.
• A redesigned navigation bar or footer. These appear on every page of the site, so there's no single page to attribute conversions to.
• A faster site or an upgraded software package. These changes happen when the site is built, so there's no version for some visitors and not others.
• A new page title or meta description. Search engines and browsers read these once per page, not per visitor.
• A redirect or a change to how pages are routed. The redirect happens before the page loads, so every visitor follows it.

For these, Kite suggests a before-and-after comparison. When a narrower version of the idea can be tested, it offers that too. If you want to know whether a new font helps, for example, Kite might suggest testing a new homepage headline as a split test and measuring the font change before and after.

On a site Kite hosts, it can run a real A/B test with a goal and a traffic split between versions. When a running test isn't reaching a clear answer, Kite tells you and suggests ending it as inconclusive.

Kite measures the baseline as the changed page's conversion rate over the two to four full weeks before the change. A comparison is only as good as its starting point, so this step comes before anything ships.

First, Kite identifies the one conversion the page exists for, such as a form submission, a signup, a purchase, or a click on a call to action. Then it checks that your analytics actually records that conversion on that page. If it doesn't, Kite recommends adding the tracking first and stops there, because a change can't be measured against a number that was never recorded.

Kite also looks at the page the way a visitor would and notes what it sees. That observation feeds the next step. If the change is already live, Kite works out the before period from the date you give it and can check archived copies of the page to confirm what it looked like before.

Kite writes the purpose of the change down as one sentence before it ships, so the result gets judged on the right number. The sentence names the evidence, the change, the expected effect, the audience, and the measurement. For example: "Because most visitors leave the pricing page before scrolling to the form, moving the demo form above the plan table will raise demo requests from pricing-page visitors, measured by demo request submissions."

Writing it down keeps the measurement honest. If the change was meant to raise demo requests, Kite judges it on demo requests, not on whichever metric happened to rise.

Kite also keeps the experiment to one change whenever it can. If you rewrite the headline, shorten the form, and swap the hero image in the same week, a rise in conversions can't be credited to any one of them. When a full redesign is the only practical option, Kite measures it as a single experiment and says so, so nobody mistakes it for three separate lessons.

Kite checks your team's knowledge base for earlier experiments before proposing anything, so it doesn't rerun a test that already has a clear answer.

Kite can build and publish the change itself on a site it hosts or a site you've imported from GitHub, and it can update pages through a supported content system such as Webflow. It asks for your approval before anything goes live. You can approve it with the Approve or Publish button in the conversation, in the Kite web app or in Slack.

If your team has already shipped the change, Kite skips this step and measures the change you made. All it needs is the page, what changed, and the date it went live.

Kite waits until the after period matches the before period in length and in weekdays, and until it holds enough conversions to read. In practice, that means at least two full weeks, and longer when the page gets few conversions. A comparison of six conversions against nine tells you almost nothing, so Kite keeps the window open rather than reporting a single-digit result.

Matching weekdays matters more than most people expect. Many sites convert differently on weekends than on weekdays, and a period with one extra Saturday can move the rate on its own.

You don't need to remember to come back. Kite schedules the readout for the date the window closes, and the result arrives in the conversation where you asked.

Kite checks the comparison for the outside events that most often break a before-and-after reading. This is the step that separates an honest result from a hopeful one. It looks at:

• Traffic sources. A paid campaign, a newsletter send, or a post that got shared can bring visitors who convert at a very different rate. When the mix shifted, Kite compares the conversion rate within each source, such as search visitors before against search visitors after.
• Devices. A change can help desktop visitors and hurt phone visitors, and the overall rate can hide that. Kite reads them separately.
• The rest of the site. Kite compares the changed page with pages you didn't change over the same dates. If the changed page rose 12 percent and the rest of the site rose 10 percent, most of that gain came from something sitewide, such as the season or a campaign.
• Tracking. A conversion rate that drops to zero the day of a change is usually a broken form event, not a disaster. Kite compares how often the conversion event fired in both periods, so a tracking problem doesn't pass for a result.

Kite separates what the data shows from what it infers. A rise that holds within each traffic source and beats the rest of the site is strong evidence that the change worked. A rise that disappears once you account for a new campaign isn't evidence, however good the overall number looks.

Kite reports one of three verdicts: won, lost, or inconclusive. A change wins when the page's conversion rate rose over the full window and the checks held up. It loses when the rate fell under the same conditions. It's inconclusive when the window was too short, the conversions were too few, or something else changed enough to explain the difference.

Every verdict comes with the baseline rate, the after rate, the dates of both periods, and the caveats that apply. When the findings need more room than a message, Kite publishes a findings page with charts and tables and links to it.

An honest inconclusive result is still useful. It tells you to keep measuring, to test the idea a different way, or to move on to a change with a clearer expected effect.

Related How-to pages

• How to measure a website change with Google Analytics when you can't A/B test: The full before-and-after comparison on a site measured in Google Analytics 4, from the baseline report to the verdict.

Kite records every experiment in your team's knowledge base: the change, the sentence that said what it should do, the baseline, the result, and the verdict. When the verdict is clean, it also adds the lesson to what your team knows works, such as "A shorter form raised demo requests on our pricing page."

That record shapes what Kite proposes next. It can suggest the next change to try, build it, and measure it the same way. On a site Kite hosts, the next change can run as a true A/B test when the change and your traffic allow it. For more on the wider work of finding and fixing what stops visitors from converting, see Improve website conversion.