Experimentation is changing. Fast.
What was once a human-led process—running A/B tests, tweaking User Interface (UI) elements, and optimizing conversion paths—will increasingly be handed over to AI.
With AI automating tests and variations, humans now have the opportunity to focus on big bets—experiments that don’t just tweak conversions but fundamentally impact business strategy and growth. This shift marks a new era in AI Conversion Rate Optimization (CRO), where automation enables teams to think strategically rather than tactically.
However, it does raise some critical questions: What happens to experimentation teams when AI runs more tests? How does AI-driven testing change the economics of experimentation?
Table of contents
Let AI Sweat the Small Stuff—Here’s Where CRO Teams Should Focus
Let’s face it. For years, CRO teams have focused on iterative optimization – testing headlines, personalization, form layouts, and other small but measurable changes.
But as technology continues to develop at lightning speed, AI-powered platforms will soon be capable of automating these small tweaks automatically—optimizing in real time, across thousands of variations, without human involvement.
Major players like Airbnb, Uber, and Netflix have already started leveraging AI-driven experimentation.
Netflix employs AI-driven A/B testing to improve user experience, including recommendation algorithms, adaptive streaming, noisy neighbor detection, and user interface designs.
Uber uses Michaelangelo, its in-house machine learning platform, to optimize day-to-day operations like dynamic pricing, ETA predictions and route optimization, fraud detection, as well as personalized rider and driver matching.
Airbnb uses AI to optimize certain operations like personalized recommendation based on user behavior analysis, enhanced guest screening through AI analysis of personality traits and social media activity and posted content, as well as AI-driven pricing optimization.
What’s more, Airbnb CEO Brian Chesky also announced that the platform is planning to make listing a new property even easier than before through AI-powered image recognition. AI will be able to generate an amenities list using the photos hosts upload to the platform, speeding up the listing process and improving efficiency.
So, what does this mean for human teams? Freedom.
With AI tackling low-value tasks, CRO experts can go big—testing new business models, exploring innovative customer acquisition strategies, and conducting deep user research.
This fundamentally changes the role of human teams. They will no longer be responsible for minor adjustments, leaving space to shift focus to high-impact, strategic experimentation and high-risk, high-reward initiatives that can truly transform a business.
The cost equation of experimentation is changing
Historically, experimentation has been expensive and, for many companies, has always been constrained by cost.
Running a CRO program required extensive resources, limiting how many tests businesses could realistically conduct. Every test requires:
- Research;
- Design;
- Development;
- Engineering;
- Data analysis.
And if the win rate is low (which it often is), the cost per successful experiment becomes prohibitive.
In a recent webinar, CXL President Hesh Fekry and experimentation Legend Ton Wesseling explored how AI has the potential to take over the small-scale tests that have traditionally occupied CRO teams.
“For some companies, experimentation simply does not make financial sense. The costs outweigh the potential wins,” said Wessling.
This is where AI will change everything.
- Automated experimentation lowers human labor costs.
- More tests can be run at near-zero incremental cost.
- Faster iteration means compounding learnings.
By automating routine tasks, AI significantly reduces experimentation costs, making CRO more scalable and cost-effective than ever before.
Wesseling put it bluntly:
“There is money to be made with these smaller experiments, all these tweaks. If we can automate the work, then the cost will go down, and the business case becomes positive.”
The fundamental economics of experimentation are shifting:
- When AI drives testing, cost is no longer a limiting factor.
- If the cost per experiment is significantly lower, companies should test everything.
With AI, businesses can now test more hypotheses with fewer resources, gaining valuable insights faster. The result? A CRO strategy that’s not just about optimizing web pages but about reshaping business models and driving substantial growth.
AI vs. human: Who owns the learnings?
There’s a hidden risk in AI-led experimentation: If AI does the testing, do humans lose the insights? Experimentation isn’t just about finding wins – it’s about understanding why users behave the way they do.
“Insights will be lost to the humans that used to work on that process. The insights will be within the system.”
With this comes some interesting considerations:
- AI can optimize but doesn’t explain its logic.
- AI can improve conversions but might not provide human teams with strategic takeaways.
- AI-driven learning can be a black box – without visibility; companies risk missing critical insights.
AI is a tool, not a black box—Why human oversight still matters
While AI is fantastic at automating tasks and uncovering patterns, it’s not infallible. A poorly monitored AI system can become a “black box,” making decisions that no one fully understands.
By now, most of us would’ve experienced machine learning hallucination, for example when ChatGPT simply makes up facts in response to queries. But in CRO, AI-driven systems could falsely detect patterns or AI-powered recommendations might “hallucinate” correlations between user behavior and conversion rates, recommending changes that don’t improve performance or may even harm it.
This is when human oversight would be crucial.
As mentioned, Airbnb uses AI-driven pricing optimization using various data points like location, seasonality, and competitor rates to suggest the best price points. AI-driven dynamic pricing models can optimize revenue, for sure, but could unintentionally alienate certain customer segments if left unchecked.
Similarly, Amazon and Shopify use AI-powered checkout optimization like one-click Shop Pay (which saves and autofills payments details) to simplify the purchasing process, or “one-click checkout” or “Buy Now ordering” to shorten the checkout process and increase conversions. But without human oversight, it might overlook a critical security step, leading to increased fraud risk.
How to avoid the AI black box
To ensure AI doesn’t hide the learnings, companies will need:
- Structured documentation of AI-driven experiments.
- AI-human hybrid models, where AI optimizes, but humans analyze insights.
- Feedback loops to ensure AI-driven tests translate into business-wide learnings.
Without this, teams risk letting AI optimize in isolation – improving conversion rates while missing the critical learnings. At the end of the day, AI should enhance strategy, not dictate it blindly. The best results come when AI and human expertise work in tandem.
What Happens When AI Starts Buying Instead of Just Testing?
Here’s a wild thought: what if AI starts making purchasing decisions on behalf of consumers?
We’re already seeing glimpses of this with AI-powered recommendation engines and automated shopping assistants.
Wesseling raised a critical question:
“At some point, the consumer will have their own AI. If my product feed is available to AI, why do I need a website at all?”
Think about what happens when AI-powered assistants:
- Scrape product feeds instead of visiting websites,
- Compare offers in real time,
- Make purchase decisions autonomously on behalf of the consumer.
This upends traditional CRO. If AI is doing the buying, then:
- What does a conversion funnel even look like now?
- Should companies optimize for AI instead of humans?
- Is CRO now about structuring data for AI search engines?
This shift is already starting.
Search Engine Optimization (SEO) is moving towards AI-friendly structured data and CRO may soon focus on optimizing for AI-driven purchases, not human browsing.
If AI begins handling more of the buying process—fundamentally changing how buying decisions are made—businesses will need to rethink their CRO strategies—not just for human behavior, but for AI-driven purchasing.
The traditional buyer’s journey, as we know it, won’t exist anymore.
Think bigger—AI frees you to go bold
AI isn’t just making experimentation easier – it’s changing the entire framework of what experimentation is.
If you’re in CRO, growth, or product, here’s what to do now:
- Embrace AI to automate micro-optimizations: AI can handle the small tweaks and it’s getting more advanced by the day. Your focus should shift to big bets—new business models, bold product experiments, and transformational ideas.
- Build a culture that rewards bold experimentation: With AI handling the low-hanging fruit, teams should focus on riskier, high-impact tests—ones that can redefine a company’s trajectory. Encourage your team to think bigger. Move beyond A/B testing small UI changes and start testing entirely new customer journeys, pricing structures, and acquisition models.
- Prevent AI from hoarding insights: AI will soon be able to run tests faster and cheaper than humans—but if companies don’t set up structured knowledge-sharing, they’ll lose valuable insights.
- Prepare for AI-driven buying: If AI assistants make purchasing decisions, companies must optimize for AI discoverability, not just human persuasion.
- Leverage AI for hypothesis generation: AI can analyze large datasets to uncover patterns and predict what types of experiments might yield the highest impact. Use these insights to prioritize high-value tests.
- Develop AI conversion rate optimization strategies: As AI-driven buying becomes more prevalent, optimize your funnels for both human and AI interactions. This could mean structuring content and pricing models in ways that AI-powered shopping assistants can interpret and act on.
- Invest in cross-functional collaboration: Work closely with data scientists, UX researchers, and product teams to design experiments that align with broader business objectives, not just conversion rate increases.
- Use AI to automate and scale testing: Implement AI-powered testing platforms to automate multivariate tests and personalization efforts, freeing up human teams to focus on strategic analysis and innovation.
- Ensure ethical AI use in CRO: Establish transparency in AI decision-making to avoid unintended biases. Make sure your AI-driven optimizations enhance the user experience rather than manipulating behavior in ways that could backfire.
- Educate and upskill your team: AI is evolving fast, and staying ahead means investing in training. Ensure your team understands how to work alongside AI, interpret AI-driven insights, and apply them strategically.
The biggest takeaway?
AI is not just changing experimentation—it could redefine the future of optimization itself. Companies that embrace AI conversion rate optimization will not only get ahead but stay ahead too.
Companies that don’t? They’ll still be tweaking landing pages while AI makes all the buying decisions.
For marketing leaders and professionals looking to deepen their expertise and stay ahead of the curve, check out these courses:
- Strategic Research for Experimentation
- Advanced Experimentation Masterclass
- Experimentation Program Management
The game is changing. Make sure you stay up to date.
