Moneyball Design: Why the Research and Data Analysis is Critical to Getting Your Prospects to Convert

Moneyball Design: Why the Research and Data Analysis is Critical to Getting Your Prospects to Convert

136 CMOs were recently asked “What do you see as your biggest opportunity for revenue generation in early 2013?” 46% of them said “improving conversions”. Excellent. Now that we know where the opportunity lies, how do we go about doing it?

We had two prospective clients this past week where their overall site design was good enough that they didn’t need a full redesign. Both of these were established and successful sites, profitable businesses. We were proposing to start with conversion research and usability testing, and then move on to conversion rate optimization based on the findings and do A/B testing.

One prospect said to me in an email, “I’d like to try and improve the actual conversion rates, instead of doing all the testing or gathering feedback.” The other prospect made a comment in a phone conversation that the research and usability testing wouldn’t be very helpful and was “mainly for our benefit”.

This reminded me of this great quote:

“Everybody wants to be a bodybuilder, but nobody wants to lift no heavy ass weights.”

– Ronnie Coleman,
bodybuilder and 8-time Mr. Olympia

The truth is that the research and data analysis we do at the beginning of every project is critical to success. Saying you want to improve conversions without testing or feedback is like trying to do blind archery. The research, the testing, and the data analysis is the heavy weightlifting that has to be done if you want optimal results in trying to improve conversions.

You may have a great product or service and great design on your existing site, but you know you could be doing better. The first step in figuring out how to improve conversions is by gathering feedback and analysis. This is taking the Moneyball strategy and applying it to design and conversion rate optimization.

“I’ve been amazed at how often those outside the discipline of design assume that what designers do is decoration—likely because so much bad design simply is decoration. Good design isn’t. Good design is problem solving.”

– The Art & Science of Web Design by Jeffrey Veen

What’s Moneyball?

Moneyball: The Art of Winning an Unfair Game is a book by Michael Lewis, published in 2003. It’s about the Oakland A’s and their General Manager Billy Beane.  The focus of Moneyball is the team’s analytical, evidence-based approach to fielding a competitive baseball team despite not being able to pay high dollar for star players like the Yankees and Red Sox can.

Moneyball, the strategy, is based on the central premise that the conventional and collected wisdom of so called baseball insiders over the past century is subjective, flawed and full of bias. A’s were able to win games cheaply by buying the qualities in a baseball player that the market undervalues, and selling the ones that the market overvalues.

Moneyball uses rigorous statistical data analysis in player selection and attempts to predict the value of the players by using metrics more granular then batting averages and RBIs.

It’s not just baseball. The book, “Scorecasting”, talks about studies done by economists David Romer. Romer analyzed every fourth down that occurred in every quarter of every NFL game from 1998 to 2000. What he found was that overwhelmingly NFL coaches make the wrong decisions in terms of maximizing their chances of winning. They do this because conventional wisdom says to punt or kick a field goal. They ignore the data available and make decisions based
on bias.

Hey – a “moneyball” statistician predicted the success of Jeremy Lin 2 years before anyone knew who he was.

Data doesn’t lie. (The interpretation might, though).

Moneyball, Obama team and elections

2012 was the Moneyball election.

This Moneyball idea of using data driven analysis for making better decisions and for winning is becoming more and more relevant in business, sports, and politics recently. Nate Silver of FiveThirtyEight Blog has become famous for predicting Obama’s wins in the last 2 elections based on his analysis of polling data. He was analyzing the data and saying the race wasn’t nearly as close as many pundits and news sources were saying the days leading up to the election.

Obama’s own campaign used Moneyball and Lean Startup strategies to win the ultra high stakes Presidential Election. “They measured every single thing” by developing one massive database of key metrics. Their analytics department was five times the size of the previous election. They did split testing of email messages for fundraising effectiveness and analyzed behavioral traits of their prospects to help reach more voters and get them to donate more money.

Some impressive numbers here:

  • The site had 17.8 million unique visitors who donated in 4.2 million events. This means they had a 23.6% visit-to-donation conversion rate. Booyah!
  • They conducted 240 A/B tests that resulted in a 49% increase in donation conversion. It’d be safe to estimate that this improvement helped bring in more than 80 million.

No gut feelings, pure data. Damn right the biggest opportunity lies with improving conversions.

VCs becoming moneyballers

A recent article on Venture Beat  talks about how Venture Capital firms are starting to take a Moneyball approach:

In Silicon Valley, new firms are going a step further by creating an entire investment thesis around data. Algorithms will be the heart and soul of due diligence — it’s not just a sanity- check mechanism,” said Matt Oguz, the managing partner of new investment firm Palo Alto Venture Science. ”It’s the only way to cut through human bias.”

One guy wanting to make startup investing more like moneyball even started a service for this. He’s not alone – 500 Startups and Google Ventures are also working on predictive systems to see the likelihood of a company succeeding. E.ventures uses its own data-driven approach to investment decisions

You can’t go one week without seeing an article on Forbes or Harvard Business Review or Venture Beat that’s talking about Big Data as the next big thing in business. More and more companies are looking for ways to compile and analyze data to help make better strategic decisions.

This is exactly how we handle the beginning of every design project or client we take on. Since a lot of our clients are small to mid size businesses, we’re not always working with “big data”, but we start every design project with research and data analysis.

How to conduct conversion research

There’s a ton of methods you can use for research and data analysis to get the answer to the question “what should be done to increase conversions”.

Customer surveys

One of the main methods we use is to survey our clients’ users and customers. Your existing users are a gold mine of information and data, and will tell you exactly how they buy if you ask them the right questions.

Excerpts from the book “Roadmap to Revenue, How to Sell the Way Your Customers Want to Buy“, by Kristin Zhivago of Cloud Potential:

“Your current customers have needs and perceptions in common. Not only with each other, but with your future customers. If you understand your current customers, you will know how to sell successfully to your future customers. The very act of coming to you to fill a need, then buying from you – creates a demographic, a cluster of human beings who have quirks in common. This is YOUR demographic.”

“Marketers who actually interview their current customers before they write copy will understand who their customers are, how smart they are, and what their experiences and reactions have been. The resulting copy aimed at prospects – whose issues are the same – will be in sync with the buyer’s reality, and will answer all but the most arcane questions.”

At CXL.agency we use qualitative surveys with long form answers to find out how our client’s users want to buy. We try to identify key benefits, objections, and we often use the exact words the clients use to describe how they interact with your product or service in benefits and USP copy.

Usability testing

Whenever we’re improving an existing website, usability testing is amazingly insightful. This is where we come up with tasks for test subjects (people matching the ideal customer profile) to do that require they interact with the client’s website. It may be to browse an ecommerce store and try to find a set of specific complementary products and place them into a shopping cart or try to find a particular product that you might buy and place it into the cart, and then checking out. We record the users screen and audio (they’re commenting everything they do out loud) and look for usability problems or friction in the conversion process.

From Conversion Rate Experts:

“Usability testing — You can carry out this testing on pretty much anyone you can get your hands on. These tests are gold dust—literally. If we could have just one testing tool, it would be usability testing. Web analytics tell you what visitors are doing, but usability testing tells you why. No other tool provides so many head-slapping, “I can’t believe I didn’t think of that” moments.

Usability testing can also find issues where the messaging is incomplete or hidden. We have a client where we did usability testing and we found that people interested in becoming an affiliate couldn’t find where to sign up. All of them looked for it in the footer, so we added a link there and problem got solved.

During the same test, we also found out that there were several questions prospects wanted an answer to, but couldn’t find on the website. The client was selling a “7 day course”, but it wasn’t clear how many hours were actually required to complete the course. We found this out through usability testing and then added the information to the sales copy and the FAQ. This helped to answer potential objections ahead of time and resulted in conversion increases.

Tools we recommend for remote usability testing: Open Hallway and YouEye. We’ve used UserTesting.com on many occasions, but I recommend you don’t use their testers. In almost all cases we have, it has rendered the test useless as the people were not the right people. Test subjects NEED to match your ideal customer profile. Recruit them yourself.

Digging through the analytics data

Analyzing Google Analytics or similar software data is another important data analysis method we use for getting insights to improve conversions.

So what can we discover from analytics data? A ton of stuff.

Often times what the client thinks is the most attractive thing on their site or where they think everyone clicks is not even in the top 30 visited pages. The analytics will indicate that either the link is not visible enough or not actually interesting to the users.

Analyzing bounce rate in analytics can tell us if there’s a disconnect between what the users want to see and what’s actually on the page. We have a recent client who runs an ecommerce store and we found out through the analytics goals and the survey data that about 85% of his clients were purchasing a specific category of his products. We changed the USP, sales copy, and design for the website to reflect this focus on a specific category.

Relevancy is super important: we look at conversion rates by traffic source and compare pre- and post-click messaging. Some traffic sources of course don’t convert for anyone – such as StumbleUpon. From a recent survey we did of CXL readers, only about 50% of survey respondents measure their conversions. I’m sorry, but that’s ridiculous – go to your Analytics right now and set up conversion tracking. It’s not difficult at all.

Lots of other ways

There are other research methods one can use like heatmap, clickmap or scroll tracking to know where people click and where they don’t. You can also set up exit surveys on key pages in the sales funnel to figure out why people don’t sign up or buy.

You have to test, there’s no way around it

Even the best data in the world can only give a hypothesis – and it’s extremely difficult to predict the impact of a single change. That’s why it’s critical that we test everything. Would behavioral targeting increase conversions? By how much? There’s only one way to find out.

Perhaps the most important method of data gathering and analysis is the one ignored by an even larger contingent of website owners. From the CXL survey : 84% of respondents rarely or never do any AB testing:

You gotta do better than this. Main reasons as to why they’re not doing testing were “don’t know what to do” or “too busy”. Two solutions for this:

  • Learn, and make time for it. Otherwise it’s like saying you’re too busy driving to actually pay attention to where you’re going.
  • Hire a conversion expert that will do it for you

For instance you couldn’t have figured out this stuff by just having a “gut feeling”. Without A/B testing you’re forced to mediocrity in a situation where you don’t have to be. A simple test might give a 5% or 10% lift in conversions. I don’t know about you, but I’d sure would like to increase my revenue by 10% (and more – which is why I keep testing).

A/B testing allows you to continually gather data and feedback about what’s working and what’s not. It allows you to make the incremental changes to design elements, copy, and your sales funnel that can lead to substantial increases in conversions, sales , and profits. Most test will either give small incremental improvements or insights as to what makes no difference – but you have to keep going. It’s a process.

Side note: You can’t split test by changing the layout each week

19% of the survey respondent said they don’t use a tool for a/b testing, but they run one version for a while and then switch to another version. And compare the results. No no no – don’t do that. The results will not be accurate. You have to rule out any changes in seasonality, day of the week, month, variations in traffic sources and many other factors. The only way to get accurate results is to split all the incoming traffic in the moment when it’s coming in.

Why most websites perform suboptimally

We think that most websites don’t sell as well as they should or produce the desired results because too often they’re created by designers or developers looking for something cool or flashy. Another reason is when the client describes what kind of website they want and the designer produces it.

This process is rife with assumption, conventional wisdom, and bias and it leads to poor conversions. We know that using a data heavy and analysis driven strategy in marketing, design, and conversion rate optimization is the key to getting more hits(conversions), scoring more runs(higher sales and profits), and winning more games (crushing your competition) the “Moneyball” way.

Evidence based marketing is here to stay

Running businesses and marketing by data is here to stay, and it’s going to be mainstream soon.

When I started writing I thought if I proved X was a stupid thing to do people would stop doing X. I was wrong.

Bill James in his 1984 Baseball Abstract

There are always people who stick with the “old ways”. Don’t be that guy that is going to continue with the “I think that…” or “it seems to me” method. Don’t get left behind or your competition will wipe the floor with you. Time to get serious about it. <

This post was written by Brooks Wiley of Markitekt. 

If you’re too busy doing your own data collection and analysis, or want to bring in outside expertise – we just released a new conversion research service you should look into.

Related Posts

Join the conversation Add your comment

  1. Fantastic article once again. I understand the importance of testing and UX and all that. What if someone, like me, has a website that is just starting out and has 25 visitors/day? Is this enough for testing? If not, how can I ensure that my site is optimized for conversion as well as it can be for a new site? And at what amount of traffic can I start to seriously test? Cheers.

    1. Avatar photo


      As Paul said, you can do testing – but it’s just going to take *much* longer to get statistically significant results.

      As for doing research – you can always perform usability testing (amount of traffic you get is irrelevant). If you’re still in the early phases with your business model, you need to do customer development instead (seek out Steve Blank and everything he’s written http://steveblank.com/)

  2. Joseph, you can test with any amount of traffic. Use Google Analytics > Content > Experiments to set up A/B testing on your site. The one caution I’ll offer is this: Test only ONE thing at a time, for at least 30 days, preferably 90 or more. Then switch all relevant pages to the better-performing content, and move on to another test.

  3. Hey Joseph, i know brooks will provide you a better answer but as for me I test even with low traffic, but I let the test run until i have 500 hits in total for that specific test. I just dont run multivariate tests. Awesooome article , thank u once again…. You are by far my favorite blog.

  4. Now you’ve got me wanting to read moneyball! lol

    As Conversion-Rate-Experts have said “A doctor wont prescribe you medicine before they diagnose you. The same goes for conversion, you need to dig into the data and diagnose the problems so you can prescribe a solution.”

    And by doing this you can get high ROE (yes ROE not ROI) by discovering a list of small tweaks and fixes that can lead to an improved conversion rate. We can only get these smart wins by doing the research as you’ve explained with “evidence based marketing”.

  5. Especially liked the usability testing discussion and examples of what you’ve learned from it. Another really interesting company that fits the Moneyball model is Relativity Media. They’re a film studio, and before they commit “to financing a particular movie — either through its slate deals with Sony and Universal or on its own — it’s fed into an elaborate Monte Carlo simulation, a risk-assessment algorithm normally used to evaluate financial instruments based on the past performance of similar products. Enough variables are included in the Monte Carlo for Wilson and his team to have reached the limits of their Excel’s sixty-five thousand rows of data.” (that quote’s from a great Esquire article about the company’s founder).

    I ran some quick numbers on the 12 movies they released this year. The total budget for all 12 movies was $622 million. The total box office those 12 pulled in was over $1.4 billion! And only 2 lost money (Wanderlust and That’s My Boy lost a total of $26 million).

  6. I think you’re exactly right that running user tests can give you great ideas about what to A/B test. It leads to a whole lot of “Phil-slapping-his-head-because-he-didn’t-think-of-that moments.” Those moments are great for our company, but not so great for my forehead :).

    What I also like to do (and this is a bit unusual) is run user tests on my A/B test after a winner has already been decided. The A/B test gives me the data I need to know which test won, but the user tests help me to know why the winner one and the loser lost. This instantly sets me up with ideas for my next test.

    Since I work at UserTesting.com I also want to quickly jump in and help clarify one small part of the post. With our service you can definitely recruit and use your own participants at no extra charge. There’s no need to use our testers if you don’t want to. Plus, for Enterprise members you can even recruit using intercepts, social media, your customer file, and 3rd party panels.

  7. Excellent read. Thanks for compiling and sharing.

  8. I am really interested in these kind of posts as they motivate me to learn online.

  9. Wow. So much information to digest. Rest assured I’d be getting my conversion analysis up and running!

Comments are closed.

Current article:

Moneyball Design: Why the Research and Data Analysis is Critical to Getting Your Prospects to Convert