AI can speed up CRO research. But which parts should you actually trust it with?
That’s the question we set out to answer in one of our AI Native Marketer workshops, hosted by CXL President Hesh Fekry and CRO consultant Ruben de Boer, who has spent more than 15 years running experimentation programs.
The goal was to move beyond the theory and see what happens when you actually apply AI to a proven CRO research process. We built an AI-powered workflow that automates three parts of ResearchXL: heuristic review, thematic analysis, and test idea generation.
Some parts worked surprisingly well. Others showed exactly where human judgment still matters.
TL;DR
- AI can automate parts of CRO research, including heuristic reviews, thematic analysis, and test idea generation. The key is to map the existing process before building an agent.
- In CXL’s August 2026 workshop, an AI agent reviewed five complete user journeys in about 30 minutes, compared with the hours typically needed to organize and run a human heuristic workshop.
- Heuristic reviews are a sensible starting point because they’re subjective, lower-evidence assessments. AI can flag potential problems, but actual user behavior is still needed to validate them.
- AI-generated scores can create a false sense of precision. A 42% heuristic score isn’t a measured conversion rate, and human judgment is still needed to decide what matters.
- The workflow works best in stages: review user journeys, combine findings into evidence-backed themes, and generate test ideas. Human review between stages helps stop weak assumptions from contaminating the entire backlog.
Table of contents
- How do you decide which CRO research tasks to automate with AI?
- What did we build to automate CRO research?
- Why is heuristic review a good starting point for AI in CRO?
- Can you trust AI-generated CRO research scores?
- How can AI identify patterns across CRO research?
- How do you turn CRO research themes into test ideas?
- What should AI automate in CRO research, and what should humans still decide?
- What to do next
How do you decide which CRO research tasks to automate with AI?
The starting point is simple: don’t start with the AI. Start with the process.
We used a swimlane to map each step, the people involved, the tools they use, and roughly how long each task takes. From there, we separated the steps that require human judgment from the operational work AI can take over.
Once the workflow is visible, it becomes much easier to turn those individual steps into skills and commands for an agent.
What did we build to automate CRO research?
We built three chained pieces to automate parts of ResearchXL: heuristic review, thematic analysis, and test idea generation.
The screenshot shows the Claude project structure, including its commands and skills. This is worth making explicit because someone looking to build something similar can understand how the workflow was organized.
The build uses three Claude commands: heuristic-review.md, thematic-analysis.md, and generate-test-ideas.md. Each calls on more specific skills for things like auditing user journeys, creating scoring matrices, identifying themes, and generating test ideas. Instead of one giant prompt trying to do everything, the work is split into separate, reusable pieces.
- Heuristic review: Audits a user journey against UX heuristics (relevance, clarity, friction, motivation) and produces both an annotated audit and a scoring matrix.
- Thematic analysis: Takes research inputs from any method (heuristic review, surveys, analytics exports, session recordings) and groups the findings into themes, showing which sources back each theme.
- Generate test ideas: Turns those themes into Spearo’s five buckets: just-do-it, instrument, test, hypothesize, investigate, with a PXL-style priority score attached.
Why is heuristic review a good starting point for AI in CRO?
In ResearchXL, heuristic review sits at the bottom of the evidence hierarchy: it’s usually one or two people, it lacks data volume, and it carries obvious individual bias. That makes it the lowest-risk research method to hand to AI.
Synthesized interviews or anything that replaces real human feedback stays off the table for now.
The speed difference is real. A human heuristic workshop takes hours to set up and run. The agent went through five full journeys in about 30 minutes.
Can you trust AI-generated CRO research scores?
AI-generated scores are useful for spotting potential problems, but they aren’t actual measurements of conversion impact. A score like 42% or 53% may look precise, but it’s still based on subjective judgment.
The faster review is useful, but Ruben’s pushback points to a bigger lesson: we need to be careful about how much we trust AI when it scores or prioritizes research.
The AI gives very specific scores, like 42% or 53%, which makes the results look more scientific than they really are. Underneath, they’re still based on subjective judgment, not actual measurement.
AI also doesn’t really know what motivates a user. It can check for things like urgency, social proof, or authority, but it can’t know whether those things actually matter to a specific visitor. Something that looks like a problem according to a framework might have little impact on conversions.
The biggest weakness is prioritization. AI is useful for finding and organizing potential issues, but deciding what actually matters most still requires human judgment. Treat AI-generated scores as a starting point, not the final answer, and validate them against real user behavior and test results.
How can AI identify patterns across CRO research?
AI can group findings from different CRO research methods into recurring themes, showing which sources support each pattern.
The second part of the AI workflow moves beyond individual observations.
It takes inputs from different research methods, including heuristic reviews, surveys, analytics, copy testing, heatmaps, and more, and groups the findings into common themes.
Instead of ending up with hundreds of disconnected observations, you can start seeing patterns such as:
Theme → supporting research sources → underlying customer problem
And when the same theme is supported by several independent research methods, you have more evidence behind it.
This is also where the output becomes more strategically useful.
Instead of telling stakeholders “we should add social proof,” you can discuss the larger customer problem your experiments are trying to solve.
How do you turn CRO research themes into test ideas?
The final part of this workflow takes those themes and turns them into potential experiments and next actions.
But there’s another AI failure mode worth watching for here: research as decoration.
As Ruben described it, sometimes AI comes up with an answer or best practice first and then searches the research for evidence that supports it.
That’s backwards.
The research should lead to the insight, which should lead to the hypothesis, which should lead to the test idea, not the other way around.
Where should humans review AI-generated CRO research?
At every handoff. Review the heuristic findings, check whether the themes make sense, and challenge the final test ideas before anything gets prioritized. That’s how you stop a weak assumption in the first stage from becoming an entire backlog of bad experiments.
The goal isn’t to build one giant agent, give it all your research, and blindly accept whatever comes out.
A better workflow is:
- After the heuristic review, add your own research and context.
- After thematic analysis, review whether the themes actually make sense.
- Before generating test ideas, add anything the AI missed.
- Then review and prioritize the final backlog yourself.
Breaking the workflow into stages makes it easier to catch mistakes early rather than discovering at the end that the entire analysis was built on a bad assumption.
Ruben framed the AI workflow as something closer to another member of your brainstorming session. It can work quickly, surface ideas you hadn’t considered, and challenge your thinking, but it shouldn’t replace the diverse human perspectives around the table.
What should AI automate in CRO research, and what should humans still decide?
AI can dramatically speed up CRO research workflows. It can review more journeys, organize more research, surface patterns, and generate test ideas much faster than doing everything manually.
But faster analysis isn’t the same as better judgment.
The strongest workflow combines AI’s speed and scale with human context, expertise, and quality control.
The bigger lesson is that the better your process and inputs, the better AI can amplify them. If the underlying process is bad, you’re simply automating bad work faster.
Frequently asked questions
Can AI replace user interviews in CRO research?
No. AI can analyze transcripts, summarize interview findings, and identify repeated themes, but it shouldn’t replace real customer feedback with invented responses. In the CXL workshop, synthesized interviews were deliberately excluded from the automation. The point is to process actual evidence faster, not manufacture the evidence you’re supposed to be researching.
What’s the difference between AI heuristic analysis and usability testing?
Heuristic analysis evaluates a user journey against established UX principles, such as clarity, relevance, friction, and motivation. Usability testing observes real people completing tasks and reveals where they struggle. AI can speed up heuristic reviews, but the findings remain potential problems to investigate. Actual user behavior provides another layer of evidence.
What research inputs can AI use for thematic analysis?
AI can combine findings from heuristic reviews, surveys, analytics exports, copy testing, heatmaps, and session recordings. The important part is keeping the original sources attached to each observation. When multiple research methods point to the same customer problem, the theme becomes more useful for generating hypotheses and deciding what to investigate next.
Can AI automatically prioritize CRO experiments using PXL?
AI can generate PXL-style scores to help organize an experimentation backlog, but the final prioritization still needs human review. The team should check the supporting research, likely business impact, and implementation effort before deciding what to test. A high score means an idea may deserve attention. It doesn’t mean the experiment is guaranteed to win.
How do you keep AI from generating unsupported CRO test ideas?
Make the research the starting point. Each proposed test should connect to a specific customer problem, a supporting research theme, and the original observations behind it. If that evidence isn’t there, investigate further rather than inventing a rationale. The CXL workflow deliberately separates heuristic review, thematic analysis, and test generation so people can catch mistakes between stages.
What to do next
If you want to go deeper into the research process behind this build, CXL’s Strategic Research for Experimentation course covers heuristic reviews, thematic analysis, and turning research findings into test hypotheses. The User Research course focuses on gathering and interpreting real customer evidence, including how to use AI to make research more efficient. Both are useful foundations before automating more of your CRO workflow.
