C.F. Kettering once said, “My interest is in the future because I am going to spend the rest of my life there.”
When we look at data and analytics, we’re focused on the past. How did we do last quarter? What happened H1 2019? And how does that compare to H1 2018? How well did landing pages X, Y, and Z convert last Monday at 1:03 p.m.? (I’m kidding, I’m kidding.)
Data becomes more valuable when we use it to predict the future instead of just analyzing the past. That’s where propensity modeling comes in.
What is propensity modeling?
Propensity modeling attempts to predict the likelihood that visitors, leads, and customers will perform certain actions. It’s a statistical approach that accounts for all the independent and confounding variables that affect said behavior.
So, for example, propensity modeling can help a marketing team predict the likelihood that a lead will convert to a customer. Or that a customer will churn. Or even that an email recipient will unsubscribe.
The propensity score, then, is the probability that the visitor, lead, or customer will perform a certain action.
Why optimizers should care about propensity modeling
Even if you’re not currently using or considering propensity modeling, understanding the mathematics behind the process is important. For example, do you know the difference between linear and logistic regression models?
The same way SEO experts need to understand a bit of content marketing, HTML, etc., to be competent, optimizers need a basic understanding of statistics and propensity models.
Still, propensity modeling and experimentation are not mutually exclusive. The two work best when combined—when a propensity model fuels an experimentation program and vice versa.
Even if you don’t face any of the experimentation challenges Royston-Webb mentions, propensity modeling can help you:
- Fill your pipeline;
- Save time on quantitative conversion research;
- Explore smarter segmentation options.
How to build a propensity model
Not all propensity models are created equal.
As Mojan Hamed, Data Scientist at Shopify, explains, there’s no shortage of options to choose from, and none are inherently superior:
For example, if you’re forecasting and notice some segments do well with the base models while others do not, you can dig deeper into those low-accuracy segments to identify the issue. With regression, the whole process won’t take more than a few minutes. With other models, that diagnosis is more time-consuming and complex.
Once you’ve selected the model that’s right for you (in this article, we’ll focus on regression), building it out has three steps:
- Selecting your features;
- Constructing your propensity model;
- Calculating your propensity scores.
Edwin Chen, Software Engineer at Google, summarizes the process in more detail:
Let’s explore each step further.
1. Selecting the features for your propensity model
First, you need to choose the features for your propensity model. For example, you might consider:
- Product milestones;
- App and theme downloads;
- Device usage;
- Buying history;
- Plan selection.
Your imagination is the only limit.
Selecting features is easier when you’re interested only in a prediction. You can just add all the features you’re aware of. The less relevant the feature, the closer the coefficient will be to 0. If you’re interested in understanding the factors of that prediction, it becomes more difficult.
As Hamed explains, there are a few checks and balances:
Whether you’re interested in interpreting the coefficients or not, one thing is certain: You’ll need to gather insight from internal experts. Despite popular belief, propensity modeling does not diminish the need for business and marketing know-how.
So, gather a room full of domain experts: email marketers, conversion optimizers, data scientists, finance experts, CRM specialists—anyone with relevant business acumen.
There are numerous mathematical ways to decide which features to select, but they can’t replace human knowledge and experience.
2. Constructing your propensity model
Regression has been mentioned a few times already. But what exactly is regression analysis? It’s a predictive modeling technique that examines the relationship between a dependent variable (e.g. lead-to-customer conversion) and independent variables (e.g. product milestones, app and theme downloads, etc.)
For the purposes of this article, you’ll want to be familiar with linear and logistic regression.
In linear regression, the outcome is continuous, meaning it can have an infinite number of potential values. It’s ideal for weight, number of hours, etc. In logistic regression, the outcome has a limited number of potential values. It’s ideal for yes/no, 1st/2nd/3rd, etc.
3. Calculating your propensity scores
After constructing your propensity model, train it using a data set before you calculate propensity scores. How you train the propensity model and calculate propensity scores depends on whether you chose linear or logistic regression.
It’s important that the propensity model works with your real-world data. This is a perfect example of how propensity modeling and experimentation go hand-in-hand. Experimentation can validate the accuracy of propensity scores.
No matter how confident you are about the accuracy, run an experiment. There could be factors you haven’t considered. Or, for example, the model may unexpectedly optimize for quantity (e.g. session-to-lead conversion rate) without considering the impact on quality (e.g. lead-to-customer conversion rate, retention, etc.)
The use of experimentation to validate propensity modeling is critical. It gives you peace of mind.
Again, propensity modeling is a tool at an optimizer’s disposal, not a replacement for a thorough understanding of experimentation and optimization. Take advantage of how open regression is—look under the hood and ensure the data you’re seeing makes sense before running wild with it.
How to use your propensity model for smarter experimentation
I know, I know. You don’t need another lecture on how correlation is not causation. But, with propensity models, it’s easy to see causation where it doesn’t exist.
In a regression model, you can’t assume features have a causal relationship with the variable you’re attempting to predict.
It’s easy to look at the model and see, for example, that downloading X apps during a trial is a good indication that the lead will convert into customer. But there’s absolutely no proof that driving more app downloads during a trial will make anyone more likely to convert into customer.
Another important word of caution: Don’t substitute propensity scores for your (very valuable) optimization knowledge.
Propensity modeling, like other tools, will not tell you how to optimize. When you open up Google Analytics or fire up an Adobe Analytics dashboard, the insights don’t fall off the screen and into your lap. You use your experience, knowledge, and intuition to dig for those insights.
For example, you might know that a customer is highly likely to churn thanks to your propensity model. But is the value of what you spend preventing that churn higher than the lifetime value of that customer? Your model can’t answer that question—it’s not a replacement for critical thinking.
Alright. As we gently step over all of that caution tape, let’s look at three valuable propensity models optimizers can leverage:
- Propensity to buy or convert. How likely are visitors, leads, and customers to make a purchase or convert to the next step of the funnel? Those who have a lower propensity score need more incentives than others (e.g. you might offer a higher discount if you’re an ecommerce store).
- Propensity to unsubscribe. How likely are recipients, leads, and customers to unsubscribe from your email lists? To those with a higher propensity score, you might try reducing the frequency of emails or sending a special offer to reinforce the value of remaining a subscriber.
- Propensity to churn. Who are your at-risk leads and customers? If they have a high propensity score, you might experiment with in-product win-back campaigns or assign account concierges to reconnect them with your core value proposition.
Propensity modeling is not prescriptive. Knowing that a group of leads has a higher propensity to convert alone is not particularly valuable. What’s valuable is combining that knowledge with optimization know-how to run smarter, more targeted experiments and extract transferable insights.
The future is not an exact science. (Arguably, exact science is not an exact science.) But you can predict the future with a reasonable degree of certainty with propensity modeling. All you need is a rigorous process and a data scientist.
Here’s the step-by-step process:
- Select your features with a group of domain experts. Carefully consider whether you want to interpret the coefficients or not.
- After choosing linear or logistic regression, construct your model.
- Train your model using a data set and calculate your propensity scores.
- Use experimentation to verify the accuracy of your propensity scores.
- Combine propensity modeling with your optimization expertise to run smarter, more targeted experiments that lead to more valuable, more transferable insights.
You’ll be able to push your data beyond what has already happened and toward what is probably going to happen in the future.