Experimentation is a cornerstone of modern digital strategy, supporting businesses in driving return on investment (ROI) and generative insights on user behavior.
However, when companies are focused solely on experimentation, they understand what people are doing and how many are doing it. Still, they don’t know why they are doing it, which limits the value and actionability of their experiments.
For many businesses, incorporating UX Research into their experimentation has been a game changer, allowing them to make decisions with more confidence, answer a broader range of questions, and deepen their understanding of their customers.
This blog will delve into the benefits of integrating UX research into your experimentation practices. But first, it’s essential to understand a little more about the ideas behind a mixed-methods approach.
Table of contents
What is mixed-methods experimentation?
Mixed methods experimentation is the combination of different evidence-generating activities.
These activities generate different types of insights about users. Some aim to understand user behavior (i.e., what people do), while others capture attitudes (i.e., what people say, think, and feel). Some are quantitative in nature, answering questions of how many or how much and aiming for statistical significance, while others are qualitative, aiming at understanding the deeper why and how.
Mixed methods experimentation integrates different research methods, combining elements of attitudinal to behavioral and quantitative to qualitative.
Behavioral methodologies help us understand what customers do, while attitudinal methodologies help us understand why they do it. Depending on the methodology applied, different types of insights will be generated from insights that are quantifiable in nature, answering questions of “How many/How much?”, along to those that are qualitative in nature, essentially answering the Whys and the Hows.
Behavioral vs. attitudinal insights
Behavioral insights focus on what users are doing, and they can typically be derived from data emerging from A/B tests and analytics. This highly quantitative approach relies on statistical significance to understand user actions.
On the other hand, attitudinal insights delve into what users are thinking, feeling, and saying. These insights are gathered through qualitative methods like interviews and diary studies, or quantitative methods like surveys.
Quantitative vs. qualitative methods
Quantitative methods answer questions about how many or how much, often involving large sample sizes to ensure statistical reliability. Qualitative methods, in contrast, explore deeper, more nuanced understandings from smaller groups, focusing on the why and how of user behavior.
By plotting where all these different methodologies land on this spectrum, we can identify what types of insights we are generating with each, and decide which methodologies are best suited to answer our business question.
However, if one focuses solely on methodologies that sit squarely in the top left of the spectrum, a company is limited in the knowledge of where to go next with the insights gathered.
Although they have a firm understanding of what their customers do, they’re less informed about the why, so that insight is less actionable than it might otherwise be.
Likewise, at the bottom right of this scale, a company lacks the hard data–the ‘what’–to put their understanding of their customer’s motivations and thought processes to good use.
Why mixed-methods experimentation matters
Moving on to the ‘why,’ there are several key benefits to consider. We will speak to three of the main benefits that organizations can expect when adopting a mixed methods approach: higher confidence in decision-making, the ability to answer a broader range of business decisions, and, most importantly, more business value.
Boosting confidence in decisions
Relying on a single data point or methodology can be limiting. Mixed-methods experimentation allows you to cross-validate insights across different types of data, providing a more robust basis for decision-making.
It also helps us prioritize insights and areas of focus. For example, if we hear insight from users and observe a similar behavioral insight in an AB test, this indicates that this is an insight to prioritize.
Addressing broader questions
A/B testing, for example, is excellent for answering specific, quantifiable questions about user behavior. However, UX research can address a broader range of questions, including generative and evaluative inquiries.
Generative questions explore user motivations and needs, while evaluative questions assess user experiences and identify areas for improvement.
For example:
‘What are they trying to do?’ It’s possible to start identifying an audience’s functional, emotional, and social jobs to be done.
‘How do they think about this’ Understanding how an audience actually thinks about a product/service category or behavior beyond how they directly engage with it.
‘How do they perceive this?’ Understanding how an audience ‘perceives’ the experience offered by a company or brand.
‘How could this be improved for them?’ Identifying potential barriers or areas of improvement for an audience using an experience.
‘How well does this work for them?’ Evaluating how well a new concept or experience works for an audience in conjunction with their aims.
Enhancing business value
Combining quantitative and qualitative insights helps to build a deeper understanding of user needs, enabling more strategic decision-making. By understanding not just what users do but why they do it, a company can take more informed risks and innovate more effectively.
This deeper insight can drive greater business value, going beyond mere conversion metrics to enhance overall user satisfaction and engagement.
Practical applications of UX research in experimentation
The ‘How?’ Of integrating UX research into your experimentation practice is clearly the vital next step, and it involves applying these insights in various contexts to drive meaningful outcomes.
It also requires a mindset shift. Like with all big changes in a company, it can be useful to think of it as a cultural reset. The question will no longer be ‘Should we conduct UX research?’ but ‘What UX research should we be doing?’
Another mindset shift is a step towards clarity of intention. If AB testing is the only methodology in your toolkit, people will likely be focused on asking, ‘How can we optimize?’ But taking a mixed methods approach allows people to ask the more impactful question, ‘What is our business question?’ Starting with the business question allows us to then pick the best methodology(ies) to answer this question.
There are various ways that UX research and A/B testing can be applied together, and how the activities should be sequenced. Below, we’ll highlight and illustrate four key use cases.
Explaining unexpected A/B test results
When A/B testing yields unexpected results, UX research can provide context and explanations.
For instance, Conversion worked with a global technology corporation that saw poor performance from lifestyle imagery on their product listing pages (PLP) during COVID.
Multiple rounds of A/B tests had shown a decrease in conversion whenever lifestyle imagery was used in place of product imagery. But we were at a loss on why. Was it because lifestyle imagery wasn’t valuable? Was it because of the specific examples of lifestyle being used?
UX Research helped us understand why these tests were losers and identified what we could do next. The research revealed that users wanted to get a look and feel of the products, particularly when they couldn’t go in-store. The lifestyle imagery also didn’t resonate with their home environments. These insights allowed the team to pivot and double down on product imagery to align with user needs.
[Example – Control vs Variant]
Informing test hypotheses with UX research
Before running A/B tests, UX research can help formulate and prioritize hypotheses by better understanding user needs, preferences, and pain points.
For example, if you’re testing different layouts on a product detail page (PDP), UX research can help identify which elements users find most valuable, guiding the design of test variations that are more likely to boost conversions.
Guiding strategic decisions
For significant strategic decisions, combining UX research with A/B testing can provide a comprehensive view. Conversion worked with a global kitchen appliance manufacturer to evaluate different search functionalities by running A/B tests and simultaneously conducting UX research.
This approach allowed them to understand both the quantitative performance and qualitative user feedback, leading to a more informed choice among different vendors.
Iterating design improvements early
Using UX research early in the design process can help iterate and refine concepts before they are fully developed.
For instance, when we were designing a new wizard for a storage solution company, we used UX research to test low-fidelity prototypes, ensuring that the final design was user-friendly and met actual user needs. This iterative approach helped avoid costly revisions later in the process.
Key Takeaways
By incorporating UX research into your experimentation practice, you discover key advantages such as improved decision-making, answering a wider array of business questions, and increasing overall business value. Begin by adapting your approach from simply optimizing experiences to exploring critical business questions and selecting appropriate methodologies.
A combination of qualitative and quantitative insights offers a greater understanding of user behavior and motivations, paving the way for more creative and impactful strategies. When UX research complements A/B testing, the insights gained are far more valuable, driving better results in your experimentation process.
