How to Hire a CRO Agency: A Process to Get It Right

Historically, CRO has taken a backseat to SEO, PPC, and other forms of digital marketing. But it’s on the rise—60.8% of businesses are making CRO a priority.

As more companies think about starting or expanding CRO programs, the “agency or in-house?” question is also earning more attention. Hiring an agency can make sense for companies that don’t have the time or resources to build an in-house team.

But to find the right agency, you need to do your homework. CRO isn’t cheap: Picking an agency is a five- or six-figure decision. (Some 44% of companies spend more than $10,000 a year on A/B testing products alone—many spend far more.)

Before you start contacting agencies, weigh the pros and cons of doing the CRO work in-house versus outsourcing.

Should you outsource CRO to an agency?

Both in-house and agency CRO teams have advantages and drawbacks. The right choice depends on your company’s size, goals, and budget.

Even then, some limitations apply to any CRO program. For example, if you don’t have enough traffic to your site, you won’t have enough data to run tests. The potential ROI of CRO also varies based on revenue. Peep Laja, CXL Founder, explains:

If you increase sales by 1% for a company that makes $100 million, that’s $1 million of added revenue. CRO makes sense. But if you get that same 1% increase for a site that makes $1 million a year, the ROI—$10,000—isn’t there. 

If your company and site clear those initial hurdles, here are other factors to consider.

Building an in-house team:

• Takes time. Finding seasoned CRO experts can take many months, and launching a CRO program can take even longer.
• Is prohibitively expensive for many companies. You’ll need at least $500,000 to build a skeletal team.
• Can be a bottleneck for launching multiple tests. An in-house team often can’t launch multiple tests because their plates are full.
• Doesn’t make sense if you have only one funnel to optimize. Once your in-house team maxes out conversions for that funnel, what will they do next?

On the other hand, hiring a CRO agency isn’t always the best fit either. This is the case if you have:

• No clear (or realistic) goals. You’re not aligned on what an experimentation program needs to achieve or can realistically achieve.
• No one to own the relationship. You need a point person who can clear roadblocks and hold the agency accountable.
• No one to implement changes. Test wins become revenue only after they’re deployed across the site.

The two options aren’t mutually exclusive: If you hire an agency, you can lean on them to help you build and run (or speed up) an in-house CRO program. They can also, periodically, serve as a second set of eyes on a program or come in to help resolve a particular challenge.

As Viljo Vabrit, Managing Director of Speero agency notes:

The best goal for every organization should be running a CRO program in-house. Hiring a CRO agency should create additional capacity when there’s a lack of resources.

Agencies are also used to teach processes to improve existing CRO programs. If a CRO program ends after an agency is done with their work, it’s not been a successful engagement

Optimization has to be continuous.

If your business goals and challenges push you away from the in-house option, here are some additional reasons that companies settle on agency services.

4 common reasons companies go with CRO agencies

1. Conversion rates are low—or unknown.

Plenty of sites have succeeded at acquisition—getting people to the site—but struggle to persuade those same visitors to take action.  

Codeacademy struggled with that exact issue. “We were just trying to get better at monetizing the business side of Codecademy, and CRO is a big part of that,” said Daniel Layfield, Product Manager of Growth at Codeacademy. “Codecademy has always had strong traffic, but we just didn’t have a lot of expertise in conversion.”

Other companies have trouble nailing down their conversion rate due to analytics challenges. That was the obstacle Priscilla Leake, previously the Conversion Optimization Manager for startup JungleScout, faced. Her team needed to untangle a misconfigured analytics setup:

I was hired to work on the marketing team and own analytics and CRO for the website and realized that our revenue-tracking in Google Analytics was misconfigured by 60–80%, not directly tied to our credit card processor, and didn’t have events or tracking to account for churn and returns—which had large revenue implications for the business model. 

Before scaling testing, we wanted to get foundational tracking implemented on the marketing side to better understand testing results relative to the health of the business.

2. You can test CRO before going all-in with a team.

Only 7.8% of companies made CRO less of a priority in 2018, and most optimizers (56.4%) reported better results compared to the previous year. Strong CRO programs get results. But how do you go from “nothing” to “strong”?

Outsourcing CRO to an agency lets companies try optimization before building their own team. Unlike SEO, you don’t have to wait six months to see whether the changes are delivering results. 

If you have a wealth of traffic and want to see if you can make it more efficient, a CRO agency is an option. The ROI from a short-term project can help gauge whether building an in-house team makes sense.

3. In-house teams are a big, long-term financial commitment.

“Running an effective CRO program is like running an Olympic cycling team,” Vabrit advises.

He continues: 

The coach (CRO agency) is not enough to get your champion to the pedestal; you need world-class equipment, physiotherapists, technicians, funding, etc. If some of that is missing, the whole program falls apart. A CRO program is never isolated from other marketing and development programs.

Experts are hard to source, and expensive—an average of $76,000 in the United States, according to Payscale. And you don’t want to hire below-average talent.

As Laja cautions, “CRO is 100% skill work—it’s not a lottery or game of luck. A mediocre person can get you results if your site is terrible. If it’s somewhat or highly optimized, hiring a cheap person won’t work.”

While hiring one or two CRO generalists can help a company get set up, it takes a team to implement the ongoing testing that makes a major impact on your bottom line. That team can set you back around $500,000 annually. 

Even if your team is able to run a few tests here and there, the reality is that CRO goes far beyond a single test. It’s a process that takes time, consistent effort, and ongoing resources.

“It is a process, not a solution,” cautions Ben Labay, Research Director of CXL Agency. “There aren’t any silver bullets in CRO—those days are gone.”

4. The rate of testing is too slow.

The slow rate of testing by in-house teams is one of the biggest reasons companies outsource CRO work to agencies.

Lars Lofgren, CEO of Quicksprout, said that, in his experience, building a bare-bones growth program typically takes six months to establish, and several months more to “run at full speed.” Agencies can get up and running more quickly and, quite often, run faster.

“The development resources are a common bottleneck,” explains Laja. “Agencies can come in and offer additional capacity for building tests.”

If the agency route is the right choice, here’s how to hire the right agency.

How to set an agency up for success (and get the most out of your engagement)

Before you jump on a call with prospective agencies:

• Figure out your budget and needs;
• Compile your own testing history and customer research. 

Figure out your budget and needs

“CRO services” can mean many things: ongoing experimentation, user research, or analytics audits.

For help with ongoing experimentation, retainer services make sense. A longer engagement means an agency can run multiple, iterative experiments over time.

Many CRO agencies also offer stand-alone research projects that include user surveys, mouse-tracking analysis, and user testing. Research can help your internal team better understand what customers want, what problems they’re encountering, and how they’re using your site.

You can also engage an agency, as Leake did, for an analytics health check. You can’t run an effective experimentation program—or, for that matter, marketing department—on bad data. 

Once you have a sense of your needs, you should be able to answer the following questions:

• How much can you realistically spend? What is your budget for CRO services plus the costs to run tests? One report shows that top-converting companies spend more than 5% of their budget on optimization alone. “It’s tough to find a great agency under $10,000 a month,” says Laja. “And $3,000 a month is a major red flag.”
• What are your objectives? Get as specific as possible. You won’t know the percentage improvements that are possible; you may not even know what needs optimizing. But any details beyond “increase conversions” (e.g., you’re planning UX changes, leads are down, your internal team lacks experience for a particular project, etc.), can help identify the right agency partner. 
• How will you measure success? This could include KPIs like customer lifetime value, average conversion rate, cart abandonment rates, and others.
• Which services fit for your needs? Are you looking for ongoing experimentation, customer research, analytics implementation, or all of the above?
• What tools are you currently using? Catalog your martech stack with all the log-in details. With this handy list, it’s easy for an agency to dive in and get started right away.
• Who’s the point person in your company for the agency?

Compile your own testing history and customer research

If your team or company has run experiments or done testing in the past, put together a comprehensive history. Include details like:

• A description of each test/experiment;
• The goal of each experiment;
• How and when the test was executed;
• The results and learnings from all testing.

The history offers much-needed context and avoids waste. “We can either learn from them and iterate on these ideas, or we will just know what has been already done. This will save us time,” says Gertrud Vahtra, an analyst for CXL Agency.

Customer data is equally vital. “Experimentation and testing is about being customer-centric,” Labay notes. “It’s about listening to customers, finding out about their problems, coming up with solutions and testing those. It is the opposite and doesn’t work very well with product-centric and siloed teams.” 

Collect all existing information about your ideal customers:

• Buyer personas;
• Customer surveys;
• Other customer research (e.g., demographic data).

You can share existing documents or set up a call to walk through the information. If an agency doesn’t request historical testing data or customer info, consider it a giant red flag.

Once you’ve gathered all your research and data, it’s time to start chatting with a few CRO agencies. 

How to vet CRO agencies

Word-of-mouth referrals are a great first step. It’s also a good idea to ask agencies for references. Request a list of companies they’ve worked with in the past, particularly any in your industry—and actually contact them. 

“Asking about an agency’s experience is essential, especially to assess if an agency has experience working with companies in your league/size—startups, SMEs, enterprise,” Vabrit notes. “A solid CRO process should be universal to whatever industry; experience is needed to know how to apply it effectively for organizations in different growth stages.”

You can also ask agencies about their experience solving specific problems. In Leake’s case, JungleScout looked for digital agencies that worked on analytics issues. “Our use-case was pretty specific since it was more of an analytics/development challenge than a traditional CRO ask,” Leake explained. 

To figure out whether an agency can help you achieve your specific goals, contact at least three of their previous clients and ask key questions:

• What problem did they hire the agency to solve?
• Did they have a process when researching and launching experiments? If so, what did that process look like?
• What were the tests and experiments that the agency launched?
• What were the results of those experiments?
• How was the agency’s response time and level of service? Did they always reply quickly and were they polite and helpful? Were they transparent throughout their engagement?
• Is the company still working with the agency? If not, why?

If an agency is hesitant to provide a list of clients or claims that it’s “confidential,” Vahtra says, that’s a red flag. While an agency might not be able to divulge info about all their clients due to non-disclosure agreements, they should be able to offer a few client references.

On top of referrals and references, you can also establish an agency’s credibility with case studies. Check their website (or ask your contact person) for case studies for customers within your industry, of a similar size, and/or with a similar problem. 

Case studies reveal agencies’ approach to CRO and give you an overview of the kinds of problems they’re great at solving. But be wary of case studies that tout big conversion gains without offering specifics on how they achieved those results. 

Almost every agency has case studies that showcase outsized wins. But how big were their sample sizes? Did the test results hold up when pushed live across the site? If it seems too good to be true, it probably is. 

Ask agencies about their process, costs, and structure

Once you’ve checked an agency’s references, it’s time to ask about their approach to CRO—the related elements of process, cost, and structure.

Process. “Ask about the process that’s being used to optimize websites,” Vabrit suggests. “This process should be data-driven. Clients should investigate what data will be gathered, how it will be analyzed and turned into a testing hypothesis, and how learning management is organized.”

Frameworks are particularly important because the CRO industry is rife with rogue consultants promising big lifts. I reality, those “experts” are just throwing ideas at the wall to see what sticks.

Asking about stopping rules for tests can be revealing. There are no magic numbers, like “100 conversions per test,” a response that would be a major red flag. Sample size calculators should determine how many conversions are necessary for statistical validity.   

“A good agency has already found the patterns of successful work,” said CRO expert Andre Morys, CEO and founder of konversionKRAFT. Morys continues:

And if you found these patterns, usually you build your own framework or methodology because success in CRO is not a result of trial and error. There are enough people who misuse A/B testing for trial and error or gambling. But what you want is a success that is based on a great methodology.

There are a variety of CRO systems and frameworks used throughout the industry. For some examples, check out this rundown of a few effective CRO systems, or read about CXL Agency’s ResearchXL framework and PXL prioritization model.

Some other ways to separate the amateurs from the experts:

• Big, upfront promises. If they predict big lifts in conversions without running any tests, those results are unlikely to materialize.
• No process details. A good agency will be transparent about all the testing details before launching. For example, they’ll work with you to decide how long specific tests should run.
• Too many clients per analyst. How many clients does each CRO analyst manage at an agency? It’s a revealing question. “They should have one analyst per 3–4 clients max,” suggests Laja. “Anything more and quality suffers.”

Cost. “Ask about all costs related to running a CRO program with a specific agency,” Vabrit says. “What’s included in the service, what’s not. Should they allocate additional budget to develop and QA tests? What are different tool costs needed for optimization?”

For example, one agency’s retainer may be half that of a competitor—but if that fee doesn’t include QA testing and development costs, you may end up paying more in the long run.

Ask whether an agency’s fee includes:

• QA testing;
• Multiple variations for each test (ask how many variations are included in a package);
• Optimization/analytics tools;
• Developer time;
• Designer time.

If one agency comes in vastly lower than others, ask why. As Laja notes, some young agencies are just hungry—everyone starts somewhere—and they may be willing to take on clients for less money.

However, others may try to close the revenue gap with client volume, meaning that analysts are managing 10 or 12 clients at a time.

Structure. Does the agency team function like a factory (i.e. many clients per analyst), or would you receive a personalized, consultant-like service? Will you have a single contact, and, if so, who? 

Some aspects of team structure aren’t necessarily better than others, but knowing how the team is structured can help you establish expectations. “The main piece of advice that I would have is making sure that all of the parts of the company that an agency will touch are aligned,” Codeacademy’s Layfield said. “Communicating everything is key.”

Also, ask about their human resources. Do they have developers, designers, customer researchers, and data specialists in-house, or do they outsource their projects to external experts? While outsourcing design or development isn’t inherently bad, this setup can affect timelines, so it’s important that you know what to expect.

“It’s one thing to have a customer research specialist and to have a web analytics or data specialist. A lot of agencies don’t have both,” Morys explained. “On a deeper look, if I have to drill down, I would ask also them: ‘How do you connect data and customer behavior?’ Because for me, this is the secret sauce.”

Additional checks before choosing a CRO agency

At this point, you’ve established a rapport with your top CRO agency candidates and asked some tough questions.

Here are a few final methods to help you narrow down your list further:

• Customer testimonials. Read any testimonials on their website (or ask for them, if none are published).
• Let your team ask questions. Leake had plenty of questions while vetting JungleScout’s CRO agency. She recommends letting relevant team members ask questions as well. This engages the entire team and helps them understand how the agency would affect their role.
• Ensure the agency specializes in CRO. Vahtra cautions against hiring an agency that focuses on SEO, for example, but does CRO as an add-on. Outsourcing your CRO as an add-on service often leads to slower testing, inaccurate data interpretations, and poor processes.
• Read their content. Is the agency publishing thought-leadership content that highlights new, innovative ideas? 
• Educational opportunities. CRO agencies will often help “teach a company to fish.” Do they educate your team so that you can eventually run your own in-house CRO program? 

Conclusion

More and more companies are trying conversion optimization, but it doesn’t always make sense to build an in-house CRO team—at least, not right away.

Still, concedes Laja, “hiring any agency is a risk. Even the best agencies in the world fail. Every CRO agency has failed. If you hire an agency and think they’ll magically solve everything, odds are, it’s not going to happen.”

Here are ways to give yourself the best possible chance at success:

• Know whether the potential ROI is there. Hiring a CRO agency won’t make sense for every company.
• For some, leaning on experts can help you run tests efficiently and transition the agency to a consulting role as your team matures.
• Put together a history of the tests you’ve run previously and any customer data to set your agency up for success and get the most out of your relationship.
• When vetting CRO agencies, ask tough questions. Talk about costs, processes, testing frameworks, team structures, and client referrals.