Experimentation Agency Message Testing Training Pricing Fast Marketing Community Blog Resources Help

Test your user state models

Use case

Make important optimization decisions about long-term growth.

Upgrade for unlimited access to thousands of playbooks on increasing traffic, improving ROAS and more.

Start 7-day trial for $1

1. Choose a user metric that is relevant to your company's success.

Healthy vs unhealthy user state models are the most common. Healthy users are the people exhibiting behavior that indicates they are active and likely to keep using your product or service.

For example, common metrics that can define healthy user behavior include days since the last login, number of logins in the past 30 days, session length, interactions, and app screens per session.

Alternatively, you can choose other state models such as casual vs core users.

Join the discussion on how to complete this step.

2. Gather a cohort of users and input their data into a spreadsheet.

Include your healthy user metric and others, for example, downloads, and content consumed or created.

Join the discussion on how to complete this step.

3. Use cohort analysis to run correlations and see what tends to correlate with your healthy user metric.

Run a regression analysis using a tool like Amplitude to verify your hypothesis. For example, for an app that is a website uptime monitoring service, like Pingdom, by running regression analysis, you can see that receiving a text 6 or more times, is positively correlated with long-term retention.

The Positive and Negative Predictive Values in your analysis help you assess whether this behavior is something to push people towards. In the above case receiving 6 texts has a PPV of 26% and an NPV of 93%. A PPV of 26% means that receiving 6 texts leads to the person being retained 1 in 4 times, which makes for a pretty solid foundation for building further retention.

The Proportion Above Threshold number shows how many of your users overall are already exhibiting the behavior. In this case, a low number, such as the 4.3%, shows that there is still room to maneuver with this particular behavior, and you can drive more people towards it.

Join the discussion on how to complete this step.

4. Test your user state model with A/B testing, to find its limitations.

Prioritize the input variable and increase it, possibly at the expense of another variable, and see if those users are more successful as a result of your change.

Join the discussion on how to complete this step.

5. Identify counter metrics early.

Pair every metric with an appropriate counter metric. For example pair: signups with activation, activation with churn, and new paid customers with total revenue.

Determine counter metrics to take a holistic approach to growth, and help keep you on track with your user state models.

Join the discussion on how to complete this step.

6. Optimize to increase the metrics that correlate with your healthy user metric.

Optimize so that new users adopt healthy behaviors quickly and healthy user’s stay healthy. You can also optimize to push unhealthy users back to the healthy user state.

For example, Plenty of Fish spoke to over 1000 women who married someone they met on their dating site. They found that most women messaged their future partner first. Plenty of Fish’s primary focus was ensuring their users find long-lasting love, so they optimize their site in a way that encourages women to message men first more often.

Join the discussion on how to complete this step.

7. Perform resurrection experiments on at risk, and transitioning out users.

At-risk users are those who are not demonstrating healthy user behavior often as of late. For example, only 1-2 logins in the last 14 days when a healthy user might have logged in 7-10 times.

Transitioning out users are similar to at risk users, but they have been exhibiting the unhealthy behavior for a longer period of time. For example, only 1-2 logins in the last 30 days.

Users transitioning out are about to be churned users. Apply resurrection messaging and experiments to them before they leave while it is still much easier to reach them.

Join the discussion on how to complete this step.

Current Playbook:

Test your user state models

Mar 14, 2022

0 votes

Blog - Why (and How) Optimizers Should Take Advantage of User State Models

Request a playbook

Get unlimited access

Thousands of playbooks on increasing traffic, improving ROAS and more.

Sign up now