The Advanced Guide to Holiday Testing

When you think about the holiday season, you think more traffic, right? More traffic means more conversions. More conversions mean more data. More data means more insights.

But is the data from the tests you run during major shopping holidays representative? We can’t throw sample pollution out the window, can we? Will your insights still be relevant in February when the rush dies down?

These are just a few of the most common questions optimizers have about testing in November and December. The answers are, unfortunately, far from black and white. But, let’s start at the beginning…

How Much of an Impact Does the Holiday Season Have, Anyway?

When the holiday season (i.e. from Black Friday through to Christmas) rolls around, online revenue for retailers spikes. In fact, they pull in 50-100% more revenue compared to shopping days throughout the rest of the year, according to RJMetrics.

Of course, some industries are more affected by the holiday season than others…

In 2015, eMarketer predicts that total retail sales will increase 5.6% while online retail sales will increase 13.9%. [Tweet It!]

Why You Should Test During the Holiday Season

A few years ago, Cyber Monday was the most popular shopping day of the holiday season. Retail traffic reached an impressive 13 million pageviews per minute, a 16% increase from the previous year.

Cyber Monday, a popular day for online shopping… shocking, right? Well, as it turns out, so is Black Friday, a traditionally offline shopping holiday.

Even in the holiday-insensitive industries, online revenue spikes around Black Friday.

Thanks largely to Christmas and Boxing Day, late December is also a peak time for eCommerce sales. In fact, December is the month where eCommerce revenue accounts for the largest percentage of total annual revenue.

With all of this extra traffic, it seems like a waste not to run a few tests, right?

What to Expect

When the holidays season rolls around, you can count on…

1. An increase in new visitors to your site.
2. An increase in visitors from new or traditionally less popular sources.
3. An increase in average order value and cart size.
4. An increase in comparison shoppers, who are driven by price and availability.

Why You Shouldn’t Test During the Holiday Season

So, you know eCommerce traffic spikes around the holiday season. Are there any negatives to this? Or is it all just more data, more insights, more money?

As it turns out, sometimes mo’ traffic means mo’ problems. There are, in fact, a few reasons you might hold off on testing during these (extra) busy months.

What to Expect

When the holidays season rolls around, you can also count on…

1. An increase in cart abandonment due to an increase in unqualified leads.
2. An increased bounce rate.
3. A decrease in profit margin due to discounts and promotions.
4. An increase in technical issues due to a dramatic increase in traffic.

How to Run Holiday Tests

If you choose to test during the holidays, there are some guidelines you can follow to get the most from your experiments.

1. Run Bandits Tests

If you’re reading this, you’re familiar with A/B tests. Bandit tests are similar, but move from the explore to the exploit phase much more quickly. Take a look…

Bandit tests are ideal for short promotions because they’re more efficient. Let’s say you have a high traffic site and are running a week-long test surrounding Cyber Monday. By the time you have valid results, the holiday is over and you can’t apply your insights until next year.

With a bandit test, your highest value variation will gradually be shown more often, meaning you’re reaping the rewards while you test.

When it comes to bandit holiday tests, a good place to start is…

• Your seasonal value proposition.
• Product names / descriptions.
• Promo / Sale / Discount names / descriptions.
• Calls to action in PPC ads, on landing pages, in emails.

We’ve written a much more in-depth article on bandit tests, which you should take 8-10 minutes to read if you’re considering testing during the holiday season.

2. Run A/B Tests Again

If you run an A/B test during a period of time where traffic is irregular (i.e. during the holiday season), you should run the test again following the period to confirm the validity of the results.

Your Christmas 2015 traffic will not yield the same results as your March 2016 traffic. Heck, your Christmas 2015 traffic might not even yield the same results as your Christmas 2016 traffic.

Always test it again before you apply the results blindly to your regular traffic.

3. Test Only What Matters

If you’re going to test during the holiday season, test elements that will have a significant impact. Do not go for micro-conversions or minor growth, focus on the high impact aspects of your site.

According to Optimizely, after a related #OptiChat, “Experts say that A/B tests with the biggest impact for the holiday rush are the ones that bust any fear, uncertainty or doubt a shopper might have. Types of tests to avoid during the holidays are any radical, structural changes that may break the site or slow performance.”

That means…

• Your seasonal value proposition.
• Urgency messaging.
• Your offered merchandise.
• Your promoted products.
• Your return policy.
• Your shipping guarantee (i.e. Christmas promise).

4. Be Aware of Sample Pollution

Sample pollution is any external factor, beyond your control, that influences your test and may lead to invalid data. For example, there is a lot of sample pollution around the holidays because you’re dealing with irregular, non-representative traffic.

John Ekman of Conversionista! had this to say when asked about his biggest A/B testing mistakes…

The problem is that John tried to apply data he gathered from holiday behavior to his regular traffic. They’re in different mindsets, they have different motivations.

However, if he had run a bandit test and applied the data he gathered from holiday behavior to his holiday traffic, he could have increased conversions.

Holiday traffic is essentially an anomaly. Their behavior cannot be generalized throughout the year. As long as you’re aware of sample pollution, you can limit it.

We’ve written a detailed post on sample pollution and A/B test validity threats. If you’re curious about representative samples and valid data (which you should be), take a few minutes to read them before continuing.

5. Optimize Your Site Pre-Season

If you’re really unsure about testing during the holiday season, the least you can do is some pre-season optimization. That is, cover the basics to get your site ready for all of the seasonal shoppers.

• Perfect your seasonal value proposition. To be blunt, no one cares that you are offering discounts around the holidays… that’s not unique. Why should anyone choose your discounts over your competitors’ discounts?
• Take inventory of your online reviews (on your site and on third-party sites). What are people saying about you? Can you help resolve their issues, encouraging them to remove negative reviews? Ask your best customers to leave a review. Why? Because online reviews are the leading holiday influencer.
• Set benchmarks. You’re about to see an influx of highly motivated consumers. Use a scroll and click mapping tool like Hotjar to see how your visitors behave pre-season. Do the same during the peak traffic days to compare behavior. Does it match up? Can this data be used to make smarter design / UX decisions year-round?
• Build your list. We all know email is a popular medium during the holiday seasons. According to Litmus, email volume increases each year and in 2013, sites sent 13% more emails than the year before. So the larger your list, the more potential revenue.

6. Conduct Quality Assurance… Twice!

When asked about the biggest mistakes sites make during the holiday season, eCommerce expert Linda Bustos had this to say (spelling and grammar edited, as it was originally published via Twitter)…

Quality assurance is always important, but it’s especially important during the holiday season when technical / capacity issues are almost expected. Even the big names aren’t exempt from it.

In 2014, Best Buy’s site famously went down on Black Friday. For over an hour, their site said: “We’re sorry. BestBuy.com is currently unavailable. Check back soon.”

According to Best Buy, the site was down due to a sudden increase in mobile traffic. Given Best Buy generated $11.5 billion during the 9-week holiday period the year before, it’s safe to say that hour on Black Friday was a costly mistake.

But don’t stop there. Ensure your site and your emails are mobile-friendly (and no, not just iPhone 6-friendly). Litmus reports that during the 2013 holiday season, 50% of email opens were on mobile and over 71% of subscribers delete emails that don’t look good on mobile.

This stuff is always important, but it’s even more important from November to December.

7. Segment Returning Visitors

One of the core issues of holiday testing is that you’re testing a non-representative audience, which means you can’t generalize the results. Right?

Instead, consider segmenting your traffic and running tests for returning visitors. People who have purchased from you before (and had a positive experience) are likely to surface again around the holiday season. So, while you’ll see an influx of new visitors, you should also see more returning visitors.

Note that their intent is still much stronger and their behavior may be different from the rest of the year. You should still run the tests again post-season, but segmenting should bring you closer to valid data.

8. Treat Promotions / Seasonal Offers as Experiments

When testing, consider all of your promotions and seasonal offers test hypotheses. That means, before you run them, you should be conducting conversion research.

• What is the biggest point of friction? Shipping cost? Shipping time?
• How much are consumers willing to pay for your products?
• Do your visitors view your products as higher quality? Can you charge extra for that?

Conduct the qualitative and quantitative research before you hypothesize (i.e. come up with promotions / seasonal offers).

It also means gathering as much data as you can during the holiday season.

• How did your visitors respond to each promotion?
• Were they more interested in a $20 coupon or 10% off?
• What were the best places to advertise these promotions on your site?

You can use the insights gained for other holiday seasons and promotional / seasonal offers.

3 Test Ideas to Consider

You’re ready to start testing, but where do you even begin? Here are some ideas that will have a real impact…

1. Extend Sales Beyond 24 Hours

Black Friday and Cyber Monday are purely commercial holidays. They were invented for retailers to make more money. So, if you want to have a Black Friday week sale, go for it.

The December holidays are more meaningful, but still highly commercialized. I’ve been seeing Santa displays since before I bought my Halloween candy.

The point is that you aren’t confined to certain timeframes. Your offers and promotions don’t need to run exclusively from December 19-26th. You can start your Cyber Monday promotion efforts before Thanksgiving, if you want to.

If you visit Amazon in early- to mid-November, you’ll see something like this…

December holiday CTA? Check. Black Friday CTA? Check. When you click through to the Black Friday Deals Week (note the word “week”) landing page, you find this…

In the left-hand column, you’ll notice “Cyber Monday”, “Year-End”, etc.

Don’t feel restricted to specific dates. November and December are essentially fair game as buyer intent and motivation increases across the board.

So…

• How many promotions should / can you run in approximately 9 weeks?
• What types of promotions are the most impactful (e.g. discounts, coupons, BOGO, etc.)?
• Should you create a separate landing page for deals and promotions?

2. Gift Ideas / Suggestions

There are a lot of options, especially during the holidays. There’s also a lot of pressure to buy gifts for people you may not know very well. The result? Indecision. The best way to combat this is to create gift guides.

Let’s continue using Amazon as an example. At the top of the Black Friday Deals Week landing page (just out of view in that last screenshot), there is a CTA for an Electronics Gift Guide. When you click through, you’re brought to this page…

Of course, Amazon isn’t the only one offering a free gift guide, based on interests and budget. Chapters does something similar…

Note the “Black Friday Pre-Sale”.

So does HBC (Hudson’s Bay Company)…

So…

• What products would make romantic gifts?
• What products would be perfect gifts for a boyfriend / husband?
• What products would be perfect gifts for a girlfriend / wife?
• What products would kids love? Teenagers? Young adults?
• What products would be perfect cheap (under $30) gifts for office friends and neighbors?
• Instead of a gift guide, how can you better optimize suggested products? Can you offer personalization based on what the visitor has looked at in the past? Can you suggest complementary products in the cart?

3. Countdown Central

We all know that urgency is important. The closer you get to major holidays, the more urgency there is. Consumers begin to worry about stock / availability and delivery dates.

As a result, they’re more motivated to buy. To help encourage this, you can use countdowns on your site to trigger the mental countdown in your visitors’ heads.

There are three options here: (a) a countdown to the end of the sale / promotion, (b) a countdown to the closest holiday, or (c) a countdown related to product availability.

Macy’s is known for their urgency-inducing promotions year-round, but especially during the holiday season…

A one day sale seems pretty urgent, no? This could be your only chance… no time to comparison shop.

Note that Macy’s also offers mini gift guides (if you will), based on demographic and interests, within the urgent promotion.

Amazon is also a fan of the countdown…

Here you see “Ends November _” style offers as well as a live countdown for a Prime Early Access Deal.

So…

• How many days are left until Christmas? Boxing Day?
• How many days are left before you no longer guarantee pre-holiday delivery?
• How many more minutes will you hold the product the visitor has selected?
• How many more days remain before the promotion ends and prices go back up?

Conclusion

Should you test during the holiday season? Yes, but try bandit tests since you’re working with a short timeframe or run A/B tests again post-season.

Can you generalize your test results from the holiday season? No, your traffic will not behave the same way the rest of the year.

Holiday testing can be tough, but if you commit to test validity, you’ll reap serious rewards. [Tweet It!]

If you choose to optimize during the holiday season, remember to test what matters most (i.e. factors that will eliminate fear, uncertainty and doubt) and…

• Conduct quality assurance on different devices and browsers – twice.
• Try segmenting your traffic and running tests exclusively to returning visitors.
• Treat all of your seasonal promotions and offers as individual hypotheses.