Run a cohort analysis

A cohort analysis is only as good as two definitions: the event that puts someone in a cohort and the event that proves they're still around. Kite proposes both before it runs anything, because the wrong pair produces a chart that looks precise and answers the wrong question.

The starting event is usually a signup, a first purchase, or the start of a paid subscription. The "still here" event depends on what your product is for. For a design tool, opening a project is a better sign of staying than logging in. For a subscription business, still paying is the measure that matters. For an ecommerce store, a second order is usually the one to watch.

Kite picks these from what it can see in your data and what it knows about your product. In PostHog or Mixpanel, it reads the list of events you track. In Stripe, it works from customer creation dates and subscription statuses. In Supabase, it reads your tables to find the column that records when each user signed up. It then tells you which events it plans to use and why, so you can correct it before the analysis runs. For example, Kite might say it plans to count a customer as retained if they created at least one report in a given week, because logging in alone includes people who opened the app and left.

Kite groups customers into cohorts by the period they started in, and it picks the period size to match how quickly your customers make decisions. Weekly cohorts suit a product people use every day, where a change shows up within days. Monthly cohorts suit a subscription that bills monthly or a store where people buy every few weeks.

The period size also decides how many people land in each cohort. If you get 40 signups a week, weekly cohorts are too small to compare, because one or two customers leaving moves the percentage a lot. Kite checks the cohort sizes before drawing conclusions and moves to larger periods when the groups are too thin to trust.

Kite can also split cohorts by something other than date when your question calls for it. A cohort of everyone who signed up after a pricing change can be compared directly with everyone who signed up in the same number of weeks before it. Cohorts can also be split by plan, by the first product someone used, or by where they came from, as long as your tools record that detail.

Kite builds the standard cohort view: one row for each cohort, one column for each period since joining, and the share of each cohort still active in that period. Reading down a column compares every cohort at the same age, which is the comparison that matters.

Here is the shape of the table, using made-up numbers for a product that changed its onboarding in April:

Reading down the Month 1 column, the April and May cohorts kept roughly 11 more points of their customers than February and March did. The empty cells are months that haven't happened yet, which is normal: the newest cohorts always have the shortest history.

Where the data comes from decides how Kite builds the table:

• PostHog and Mixpanel. Kite runs a retention query directly in your account, using the starting and returning events it proposed. It can also save the cohort in the tool so your team can reuse it in their own charts.
• Amplitude. Kite pulls Amplitude's retention analysis and your existing cohorts. It can also add a note to your Amplitude charts on the date of the change you're testing, so anyone looking at those charts later sees why the line moves.
• Stripe. Kite lists customers by the date they were created and subscriptions by status and cancellation date. From those, it builds cohorts that show how many customers from each month are still paying.
• Supabase. Kite runs a read-only query against your product database to group users by signup date and count the ones who came back. This is the route to take when your best record of what customers do lives in your own tables.

In the Kite web app, the table and chart appear as an interactive chart and table in the conversation. In Slack, Kite shares the findings in the thread and can link to a published report.

Kite's write-up says which cohorts differ, by how much, at what age, and what probably explains it. A table of percentages doesn't answer "Did the new onboarding help?" until someone reads it carefully, so Kite does that reading and shows its reasoning.

Kite looks for the things that commonly mislead people reading a cohort chart:

• Comparing at different ages. A cohort that joined last month can't be compared with one that joined last year on three-month retention, because it hasn't reached three months. Kite only compares cohorts at ages they've all reached.
• Mixed causes. If you changed pricing and launched a new ad channel in the same month, a better cohort could be explained by either one. Kite names both and, where your data records the source, splits the cohort to separate them.
• Small cohorts. A swing of a few points in a cohort of 30 customers can be noise. Kite says when a difference is too small for the cohort size to support.
• Seasonal patterns. A December cohort may behave differently from a September cohort for reasons unrelated to anything you changed. Kite points this out when the cohorts span a season your business is sensitive to.

The write-up separates what the data shows from what Kite infers about the cause, and it states how confident Kite is. It also suggests what to do next. For example, Kite might suggest keeping the new onboarding and looking at the step where the remaining customers drop off. Or, if a cohort from one channel retains far worse than the rest, it might suggest moving budget away from that channel.

A cohort analysis gets more useful every month, because each new cohort adds a row and each older cohort adds a column. A one-time chart taken two weeks after a change only shows the first column, which is rarely enough to make a decision.

Kite can put the analysis on a schedule, such as the first Monday of each month, and refresh the same cohorts with the same definitions every time. Using the same definitions keeps each refresh comparable with the last one. Each run starts a fresh conversation in the Kite web app, or posts to the Slack channel you choose if your team uses Slack, and it calls out what moved since the last run.

If a report needs more room than a chat message, Kite can publish it as a report or dashboard at a stable link. A dashboard can refresh its data when someone opens it, so your team always sees the latest cohorts without asking Kite again.