Marketers of all stripes are obsessed with tools.
This obsession has bred comprehensive lists of growth tools, SEO tools, and general online marketing tools. It is no different with us in conversion optimization. We nerd out on testing tools.
Though no optimization program has ever hinged on which tool you used, there are important distinctions between A/B testing tools—from the statistics they use, their price, and more.
One thing that is often either overlooked or misunderstood is the difference between client-side and server-side testing tools.
Table of contents
- Client-side, server-side: what’s the difference?
- Okay, but when should I use which?
- The flicker effect
- Synchronous vs. asynchronous scripts
- What client-side tools do right
- Server-side tools: when to use?
- Which type is your tool?
Client-side, server-side: what’s the difference?
Here’s the basic difference between client-side and server-side testing tools:
Server-side tools are different in that no modification occurs at the browser level. When the visitor lands on your page, a randomly picked version of your test is sent straight from your server. Yes, this means you have to involve your developers in testing, but it also gives you robustness and flexibility.
Tim Ash was quoted in a TrustRadius report, summing up the difference between the two types of tools very well:
“With server-side tools, the content management systems are building in testing, maintaining multiple versions of content and serving them up based on segmentation. It offers control over deployment regardless of browser problems. Client-side tools are usually easier for the marketer. They have WYSIWYG editors and make it easy to make surface level changes without needing IT, which is a huge pain point. It’s hard to use them to restructure the registration path or the checkout, though, because you’ll touch the CMS.”
There’s a third type of tool as well, closely related to server-side capabilities. There’s only one company I know that uses this method (SiteSpect) so it’s not super common. What they use is something called the proxy server or intercept method.
Basically, the tool acts as a proxy server between the visitor’s browser and your server. It controls the site content that is presented to the visitor, so it eliminates most of the problems that occur with client-side tools, while retaining the robustness of server-side tools.
Okay, but when should I use which?
As Paras Chopra from VWO said in a blog post, the two aren’t in opposition. Different tools for different needs and goals:
Client-side tools offer ease of use, faster execution, and a WYSIWYG editor. You don’t necessarily need to take up your development team’s time to get experiments running.
That said, server-side tools have benefits as well. An article from Google claimed that server-side implementations offer more flexibility to do things such as:
- Run experiments for websites with dynamic content.
- Test non-UI changes that still have an affect on your objective. For example, a database query result set that is returned to a user.
- Integrate Google Analytics experiments with your service (e.g., content management provider).
- Manage experiments using your own optimization platform.
In summary, they offer more flexibility. They also tend to be more robust, and they eliminate a few of the weaknesses of client-side tools. Some of these potential weaknesses include the flicker effect (which we’ll talk about below), load speed, and flexibility.
“They’ve solved the biggest problem, which is the technical barriers to testing. The downside is load time and site speed, and the lack of flexibility of what you can manipulate. There are cases where it will have a flicker or it loads more than it needs to. Other approaches might allow for more flexibility and speed. I don’t think we’ve hit the best solution yet as an industry.”
Pete Koomen, co-founder of Optimizely, summed it up well:
“The advantage of testing on the client side is speed and simplicity. You can test a lot of changes quickly without much initial investment. On the other hand, testing on the server side is both more work and generally more powerful.”
Though, as another way to look at it, Nate Weiss (CTO of Conductrics) says that the ‘skipping IT’ benefit of client-side tools could have a cost for some companies. He explains:
“For companies that have a rigorous QA and approval process for their web content in general (outside of testing), the “skipping IT” benefit of client-side tools has a cost: it becomes a “back door” where the usual approval processes and so on end up getting ignored, because the testing tool doesn’t know about the internal workflows that lead to normal content being published. It is then very difficult, if done at all, to keep these two “moving targets” (changes made normally, and changes made via the testing tool) in sync.”
In summary, use different tools for different uses and goals.
The flicker effect
As mentioned above, one of the weaknesses of client-side tools can be the flicker effect. What is it?
Basically, it’s when, for very small amount of time, your website visitor sees the default page before the variation loads. It’s also known as the “Flash of Original Content.”
This generally happens because of an asynchronous script with the testing tool. While it has the advantage of not slowing down the page, this type of script can cause a flicker effect once in a while.
This matters because the human eye detects, and we change our behavior when we know we’re part of an experiment. Though, there is the surprising (fluke) occasion that it helps a test. Take this example from an Unbounce blog post:
As David Hauser, co-founder of Grasshopper, said in the article:
“Our A/B testing tool had a bug that delayed the $25 activation fee from being crossed out until a few seconds after the page loaded.
This error ended up creating a much larger uplift than having it already crossed out on load, when the bug was fixed.
The result now is that the activation fee shows, and then is crossed out after a few seconds.”
Most of the time, though, flicker effect results in statistically significant differences in test results. That’s totally affecting the test, and can skew the outcome. The goal is to reduce the flicker to 0.0001 seconds so people won’t see anything.
So, I mentioned that flicker effect is really only a problem with client-side tools. What are some ways, if there are any, that we can mitigate this?
Synchronous vs. asynchronous scripts
Client-side tools can load scripts in two distinct ways: synchronously and asynchronously. What’s the difference?
Synchronous loading is when a testing tool’s script loads completely before the page starts to load. It calls back to the server and waits for an answer. Put another way, the scripts are loaded sequentially starting with the tag.
For example, Optimizely uses a synchronous snippet to prevent the flicker effect and a balanced CDN system to make sure the impact on the page is minimized.
Asynchronous is when some scripts can be loaded simultaneously. They load while the testing script runs, possibly causing a flicker effect if the original content loads a split section before the variation.
You can run client-side tools like VWO and Optimizely either way. In general, page loading speed is important, so asynchronous code is usually preferred. The batched scripts allow the pages to load faster, and you can usually prevent the flicker effect with a few measures.
As Paras Chopra said in a GrowthHackers thread:
“Generally with VWO’s asynchronous code, there should not be any flickering. In edge cases or in cases where you haven’t put asynchronous code in the head of the page, flickering might happen. We’ve made our asynchronous code in such a way that the original page is hidden until the settings are loaded. (The page gets loaded in parallel and that’s why it is async but it’s not displayed).
With regards to VWO’s (or any other tool’s) synchronous code, I wouldn’t recommend it generally unless there are specific cases that cannot be fixed. Synchronous slows down the page speed.”
Alhan Keser from WiderFunnel also expressed a similar sentiment. Unless you’re running a massive site with tons of visitors, page speed is of primary importance:
“Asynchronous loading addresses a separate issue: helping with overall site speed, not FOOC [flash of original content]. Given the speed of modern CDNs, snippets loading synchronously should be the least of your concerns. But, if you’re like our neighbors here in Vancouver, PlentyOfFish, with a bajillion users hitting their site at the same time, you may want to be considerate of what and how things load on your pages.”
Other tips to prevent flicker effect
One way to prevent the Flicker Effect that Optimizely outlined is to add a timeout on asynchronous scripts. Their default setting is one second. This means that if the A/B testing script isn’t loaded in that time, the A/B won’t be executed. If the visitor has a slow connection, that visitor won’t be included in the test.
Some other ways to prevent the flicker effect:
- Optimize your site for speed. Make sure everything, especially the images, load fast. Move your site onto a fast web host. Use compression, optimized images, and content delivery networks.
- Take your A/B testing tool script out of the tag manager (if you use one, like Google Tag Manager).
- If you can, do split URL testing instead.
- Pay attention to the HTML elements you change. If you made changes on the page itself (as opposed to doing split URL testing), the key thing to keep in mind is the amount of code that was changed. If you changed thousands of lines, you’re far more likely to see a flicker effect as opposed to changing button copy. So the scope of changes influences everything.
What client-side tools do right
Client-side tools let marketers run tests accurately without the help of developers.
That’s not a small accomplishment. Development resources are expensive, in time and cash, so enabling testing independent of product development is efficient. Optimizely put it well:
“When it’s time to take action and make changes to your site, there is no replacement for dedicated technical resources—in most organizations, these resources are hard to come by and typically demand waiting in line.”
Sean Ellis introduced the idea of high-tempo testing as a key mechanism for growth.
The basic premise is that the faster and more efficient you can run tests, the more of them you can run, and the faster you can grow. Though their bottlenecks were mostly at the ideation and organizational levels, many optimizers find bottlenecks at the development and implementation levels. Client-side tools using visual editors solve this.
In a previous piece, Andrew Anderson wrote about the importance of speed in testing tools. He gives a few examples of tools as well:
This means that items such as templatized rules, easy site interaction, easy interface, and easy navigation are vital. It also means that tools that require complex URL targeting rules, constant IT deployment, or require manual jQuery set-up should be lowered on your priority scale.
An example of a tool that for the most part does this right is Optimizely while an example of those that doesn’t is SiteSpect.”
Andrew also noted that, while lots of tools provide easy-to-use visual interfaces to create treatments, the automatically created jQuery code is often pretty bad. This results in cross-browser compatibility issues and slowness. Always have a developer check and improve the code—or learn jQuery. It will really help.
Server-side tools: when to use?
As I’ve mentioned, a huge benefit to client-side tools is the speed and ease-of-use for not-super technical people. So what, then, are the benefits of server-side tools? Why would companies use them?
As Matt Gershoff, the CEO of Conductrics explains, server-side tools have their advantages as well. Here’s what he had to say about them:
“The main advantages of server side will be:
- Greater flexibility and control. Use test and targeting for non-visual testing targeting (call centers, CMS systems), and tight integration into marketing tech stack for single workflow (CMS, database, web servers, and so on).
- Privacy. Keep all testing/targeting decisions internal.
- Security. Reduce attack surface.
- For web apps, reduce the need for ‘heavy’ pages.”
He continues to talk about the applicability to native mobile applications:
“Some verticals continue to shift toward native mobile apps for engagement. If your company has a native mobile app or plans to in the future, selecting a tool that supports server-side testing now means that you’ll be ready to start using the same tools for testing in your apps (with developer involvement).”
Another topic of fewer conversations that Nate brought up to me was that, depending on the nature of your site and the tests you plan to run, it may be more respectful for your visitors to use server-side tools.
As Nate said, “consider which is more important: your convenience when setting tests up, or your visitor’s user experience (and impact on their mobile data plans).”
In addition, server-side tools broaden your capabilities for optimization beyond web-based-visual presentation issues, in a way. What they do is allow you to think a little different about optimization, more towards optimizing any of your transactional marketing efforts.
With client-side tools, the main topic of discussion is around the flicker effect. This is an issue, but it isn’t the only, or the largest, issue to be discussed with tools. Things like flexibility, control, and privacy affect your optimization as much or more than flicker, especially if you take the precautions to mitigate that risk.
Which type is your tool?
Here’s a chart that quickly categorizes popular tools. Let us know if we’re missing any or getting anything wrong:
|Conductrics (both capabilities)||AB Tasty (both capabilities)|
|Adobe||Conductrics (both capabilities)|
|SiteSpect (both capabilities)||Convert|
|Optimizely (both capabilities)||Bound|
|AB Tasty (both capabilities)||Kameleoon|
|Optimizely (both capabilities)|
I’m simplifying things, but here’s the bird’s eye difference between server-side and client-side tools:
Client-side tools use either synchronous or asynchronous code, both of which have pros and cons regarding pagespeed, flicker effect, and loading issues.
Client-side tools are easier for marketers to use, and you can often get tests up and running faster without burning development resources. However, server-side tools offer greater flexibility and control. They also protect your privacy and security better, and for web apps, reduce the need for ‘heavy pages.’
Both tools have their use cases and don’t need to be used exclusively. However, far more important than the tool you choose is the process by which you optimize. Don’t overthink it with tools.
Join the conversation
Add your comment
In the client-side list, I would also suggest adding Kameleoon, which is already running in top players e-commerce in France.
Comments are closed.