No, You Shouldn’t Use Visual Editors of Testing Tools [Rant]

“Our A/B testing tool’s visual editor allows marketers to set up tests without needing developers!”

A version of this is communicated by pretty much every testing tool out there. Their websites show off screenshots or videos of visual editors. But it’s wrong. You shouldn’t use visual editors to run your testing program.

Saying that non-devs can set up AB tests now is misleading and wrong. Yes, you can change button color and make copy changes easily, but that’s not what one should do. You should test what actually matters and what actually needs to be tested (based on your conversion research).

Note: there might be cases where you need to test something quick and easy. If that’s really whats you think would be best, visual editors will help you do it.

If you are limited by what the visual editor allows you to test, you will start testing stuff that makes no difference. Instead of testing what matters, you test what your lacking skills enable you to test.  So the lack of your skills dictates what you test. Obviously, this approach is wasting everyone’s time and money.

Your tests should tackle real issues, your test variations should cover a wide range of ideas – and no idea for a treatment should be limited by the lack of coding skills.

The moment you try to set up a more complicated test via visual editors, there’s close to 90% chance that it messes up the code – and variations look wonky and won’t work properly on half the browsers and devices. QA is hard as it is, and now you let people with no coding skills – and most likely clueless about quality assurance testing – set up tests?

I see surprised folks every day in my training program. They ask me “what, I need to learn to code to set up tests?” Yes, yes you do. “But, but – …”.

And I’m not even going to rant about test targeting rules (regex anyone?), goal configuration and stuff that can also be done completely wrong.

Promoting visual editors is uneducating marketers. It’s a disservice to the community.

It’s fine to have them – but they should come with a disclaimer: “can’t be used to set up any real tests.”

Update. This sparked interesting Twitter conversations.

Should you use visual editors of testing tools?