Measure whether your ads actually drive sales

The test group is the set of regions where your ads keep running, and the control group is the set where they pause. The comparison only means something if the two groups would have sold about the same way without any change to your ads, so the choice of regions decides whether the test is worth running.

Kite builds the groups from your own sales history. It reads 6 to 12 months of orders from Shopify, using each order's shipping address, or Stripe payments, using each customer's billing address, and totals sales by state, metro area, or postal code for each week. It then pairs regions whose sales rose and fell together over that period and assigns one of each pair to each group. Two metro areas with similar sizes and seasonal patterns make a better pair than two neighboring states that happen to be the same size.

Kite also keeps unusual regions out of the test. It leaves out the city where you have a store or a sales team, a region where you're running a local promotion, and anywhere a big share of sales comes from a single wholesale account. Those regions keep running as normal and simply don't count toward the result.

Before it recommends the test, Kite checks whether you sell enough for it to work. If a region's weekly sales normally swing by 30 percent on their own, a 10 percent lift from your ads will be hard to see. When the history shows that, Kite proposes larger groups, a longer test, or a different measure of sales, such as new customers instead of total orders.

Kite writes the decision rules down before the test starts, so the result can't be reinterpreted once the numbers come in. It's the same discipline a good analyst would insist on, because it's easy to find a reason to believe the answer you were hoping for.

The plan names each of these in plain terms:

• the channel and campaigns under test, such as your Meta prospecting campaigns or your Google search campaigns that don't target your brand name
• the regions in each group and the regions left out
• the measure of sales, such as orders, revenue, or new paying customers
• how long the test runs, which is usually long enough to cover several full buying cycles
• what result would lead you to cut the spend, keep it, or increase it

Kite also lists what has to stay the same in both groups while the test runs. A sitewide sale, an email to your whole list, or a price change is fine, because it reaches both groups equally. A promotion that runs only in some regions would spoil the comparison.

The plan lives with the initiative in Kite, where your team can see it, and the final report is judged against it.

A holdout only works if the control regions truly stop seeing the ads. Kite gives your team the exact change to make in each campaign, then reads the ad accounts afterward to confirm it took effect.

For Meta, Kite names the location exclusions to add to each ad set using Meta's own names for those places, which it looks up in Meta's targeting options. For Google Ads, it lists the locations to exclude in each campaign. Your team makes those changes in Meta Ads Manager and Google Ads. Kite then reads each Meta ad set's targeting and each Google campaign's location settings to check that every campaign under test excludes the control regions.

Kite also checks the spend itself. In Meta Ads, it splits results by region, so a campaign that was missed shows up as spend in a region that should have none. Catching that in the first few days saves a test that would otherwise be quietly ruined.

While the test runs, Kite checks in every week, and it can do that on a schedule that delivers to your Slack channel or your inbox. Each check confirms that the control regions still show no ad spend, that the test regions are spending at the planned level, and that nothing has happened in only one group, such as a stockout in one warehouse or a local news story.

Kite doesn't declare a winner partway through. Weekly sales are noisy, and a test that looks decisive in week two often evens out by week five. If something goes wrong that the plan didn't anticipate, Kite tells you what happened and proposes whether to extend the test, drop the affected regions, or start again.

The result of a geo holdout test is the gap between the sales your test regions actually made and the sales they would have made without ads. The control regions supply that second number.

Kite works it out from your Shopify orders or Stripe payments. Before the test, it measured how the two groups' sales normally compare. During the test, it applies that same relationship to the control group's actual sales to estimate what the test group would have sold on its own. The difference is your incremental sales, and dividing your ad spend by that difference gives your true cost for each extra sale.

Here's how that plays out. Say your test regions usually sell 1.5 times as many orders as your control regions. Over six weeks, the control regions sell 1,000 orders, so the test regions would have sold about 1,500 without ads. They actually sell 1,740. Your ads caused about 240 extra orders. You spent $20,000 on Meta in those regions, so each extra order cost about $83. Over the same weeks, Meta Ads Manager reported 900 purchases, which would put the cost near $22 an order. The ads worked, but Meta claimed almost four times as many sales as they caused.

Kite reports the lift with a range rather than a single number, along with the dates, the regions, and anything that might have affected the result. If the range is wide enough to include zero, Kite says plainly that the test didn't find an effect it can rely on, and it explains what a longer or larger test would need.

A measured lift is only useful when it changes where your money goes. Kite compares your true cost for each extra sale with what a sale is worth to you, then recommends what to do with the budget based on the rules you agreed at the start.

If the ads earn back more than they cost, Kite can recommend increasing the spend and proposing a follow-up test at the higher level, because the next dollar rarely works as well as the first. If they don't, Kite can recommend cutting the campaigns that showed no lift and moving that money to the channel with the next-best case. Kite also saves the ratio between what the platform reported and what the test measured, so later reports can adjust Meta's or Google's numbers instead of taking them at face value.

The result goes into your team's knowledge base with the plan and the data behind it. The next time someone asks whether paid social is worth it, Kite answers from your own test rather than from the ad platform's dashboard.