Think you’re sitting on the next big idea for a product or a product feature? Before you spend dozens of hours and tens of thousands of dollars on the idea, you need to validate it.
In other words, you need to make sure your audience is just as excited about the idea as you are.
So, how can you validate the ideas sitting in your startup notebook or your product feature backlog without wasting resources? By running a smoke test.
Table of contents
- What is smoke testing?
- Why do marketers use smoke tests?
- Example: Pijon Box
- Example: Flowtown
- What should you smoke test first?
- The 6-Step Smoke Test Process
- Insights > Winning
What is smoke testing?
Smoke testing is a testing method that evaluates the critical and most important features and functionalities of a system.
So, using this definition, smoke testing addresses questions like…
- Does the program run?
- Does the UI open?
- Does clicking the key functionality do anything?
Essentially, does everything work as it’s intended to? Smoke tests are specifically designed to be run quickly and frequently during the software development stages.
In the marketing world, smoke test has a similar meaning. However, instead of being focused on “does it work properly” like programmers, marketers want to know “will it make me money”.
Trevor Lohrbeer of Teikametrics and Lab Escape defines a smoke test as…
So, if you put the time and money into creating the product, will it actually sell or will it fall flat? A smoke test is designed to help you answer that question.
Dominic Coryell of GIMME GROWTH explains further…
Why do marketers use smoke tests?
Consider these two situations…
- You have a new business idea that you’re excited about and want to develop.
- You have a new product feature idea that you’re excited about and want to develop.
In both cases, you’re about to spend a lot of time and money. You will also likely need help along the way, whether it comes in the form of a co-founder or a product team.
What if that idea, after months of development and tens of thousands of dollars invested, falls flat on its face? Crickets.
Marketers use smoke tests to avoid this exact scenario. Dan Martell, a 5x founder who now helps entrepreneurs build their tech products, explains…
Dan’s four step process might look something like this, when applied…
- How do you come up with or choose the right idea? Choose a problem you’ve experienced, choose something that’s a must-have (not a “nice to have”), choose something you’re passionate about.
- Find customers using your competitor’s product and conduct validation research. Your friends will tell you your idea is great; people who are already paying to solve the problem you want to solve are the best choice.
- Design 3 core features of the idea, not the entire product. What can you build in a 6-week product sprint vs. a 6-month development period?
- Get your target audience (see step two) to pre-buy the product. Work with them to develop the full product.
Now, this process doesn’t just apply to startups and entrepreneurs. It’s certainly the way that Dan positions it above, but it can easily be applied to bigger companies looking to test new product ideas, feature ideas, service ideas, etc.
This type of agile thinking is what matters.
Dan’s process proposes something very interesting: that there’s something wrong with the current minimum viable product (MVP) concept. If you’re a marketer who is interested in rapid experimentation and product efficiency, then there is.
Here’s how the product team at Spotify explains it…
The most common objection to this pre-sale (or smoke test) process is that customers will be upset that full functionality is not available. It’s never smart to fall short of customer expectation, right?
Eric Ries, CEO at Long-Term Stock Exchange and author of Lean Startup, explains why this isn’t as big of a deal as it seems for agile teams…
Example: Pijon Box
During a presentation on smoke testing, Dominic shared a case study from Pijon Box, a care package service for the parents of college kids. Pijon Box wanted to test add an “Add to Box” feature so that parents could add more products to their care packages, if they wanted to.
Here’s how rolling out that feature would look typically…
Of course, if the Pijon Box team had taken this route, they would’ve spent a month on development and missed the end of April deadline for the last box.
So, instead, they tried this smoke test approach…
They found out which products they had extra of in the inventory, put them into an email and sent the emails out to customers. If the customers clicked the link, the Pijon Box team would have to manually invoice them and notify the warehouse.
The result? 9.55% of active subscribers clicked the link and converted. Monthly gross profit grew by 45% because of the “Add to Box” feature. Marginal gross profit per Pijon Box (on those boxes) grew over 357%.
Not too bad for a day’s work. Now Pijon Box had validated the idea and knew they wouldn’t be wasting a month of resources developing it.
Another example comes from Dan, who used smoke testing frequently to valid Flowtown. Dan calls it a minimum economical viable offer (MEVO)…
Here’s what Flowtown looked like in February 2009…
That’s right, a service company. After three months of working with 15+ customers, they decided to use all of the feedback they had gathered to double down on creating a product. (The plan all along.)
That resulted in…
Which only resulted in one customer over three months. So, they went through many more iterations (smoke tests)…
Finally, they arrived here…
Within the first day, they had their first customer. Then, growth took a turn for Flowtown…
Two years later, they were acquired by Demandforce.
Something that they emphasize is what they call the “bullshit bar”. That is, having people actually pay for the idea…
Here’s another one…
It’s important to consider the transfer of funds when validating an idea for a product or feature. A lot of people will tell you the idea is great. “Oh, yeah, sure, I’d use it.” But would they really spend their money on it? Even something $5 or $9.99 a month can add an extra layer of validation.
What should you smoke test first?
Let’s assume you’re in one of these two scenarios…
- You have a dozen business ideas you’d like to smoke test.
- You have a dozen product features you’d like to smoke test.
How do you decide which to smoke test first?
Derek Sivers suggests you “test the riskiest assumptions first”.
Otherwise, consider the following…
- High impact ideas. If you’re prioritizing your experimentation, you likely have some measure for “anticipated impact”.
- Ideas that have been backlogged for too long. Smoke test it and move on.
The 6-Step Smoke Test Process
In his presentation, Dominic lays out the six steps of his smoke test process, which is arguably the most definitive and widely used.
Step 1: Define the necessary proof.
As with most tests, smoke tests begin with a hypothesis. If I do X, I expect Y.
If we improve monthly gross profit by X% with the “Add to Box” upsell email, I expect it’s worth spending the resources to automate the feature.
If we capture info from more than X% of visitors to the landing page in the first X weeks, I expect it’s worth building the actual product.
If we can pre-sell $X in X weeks, I expect it’s worth building the actual product.
Step 2: Design an experiment without engineers.
The idea of a smoke test is to limit the amount of time and money invested before validation. So, your smoke test should be able to run without the help of an engineer.
Manually invoicing customers and notifying the warehouse vs. having engineers automate the process.
Creating a landing page to capture zip code and email for an eventual release vs. having engineers create the entire product.
Step 3: Drive the right traffic to your experiment.
Once you have the smoke test set up, you’ll need to drive some traffic to the test. This shouldn’t be a spray and pray effort. You want to make sure you’re recruiting the right kind of traffic for your test.
Here’s how Dominic knows if he’s recruiting the right audience for his smoke tests…
Eric turns to AdWords when he’s looking to generate traffic for his smoke tests…
Now, if you’re testing a feature, you can simply send existing users to the smoke test. You may want to consider using segmentation, however. For example, perhaps you will only test the feature among active users who have signed in at least 3 times in the last 30 days.
Whatever route you go, be sure of two things…
- You have tapped into the right audience. The best product or feature in the world will fail miserably if it’s presented to the wrong audience.
- You have generated enough traffic to make a decision. If you’re a large enough company to be A/B testing, use a sample size calculator upfront as you normally would. If you’re a small company (or not yet a company at all), aim for those 100 clicks / day for two weeks that Eric mentions to start.
Step 4: Define your metrics and measure them.
This (almost) goes without saying, but be sure you have tracking in place and are measuring KPIs before starting your smoke test. You should know where people are coming from, how they’re behaving at each stage of the funnel, what device they’re using, etc.
Choose KPIs that relate back to the proof you established in step one.
Step 5: Analyze the results and iterate / optimize.
If you fall into the “testing a product idea from scratch” camp, you likely do not have the traffic (and cannot afford the traffic) to make your test statistically significant.
That doesn’t mean you can’t run a smoke test. It just means that your smoke test will not be perfect and there is a lot of room for error, so take results with a grain of salt. (Think of it the way you think of heuristic analysis, for example.)
Even if you can get a statistically significant result, chances are that result will be “maybe”.
That’s where A/B testing and informal iteration comes into play. Eric explains his iterative process…
Another example Dominic gave during his presentation was Suprizr, which promised to deliver a surprise meal based on your preferences within an hour.
Here’s what the landing page looked like before…
It had a <3% email capture rate. Here’s where the importance of testing comes into play again. This low conversion rate might, to some, indicate that the idea is not valuable.
Instead of abandoning it altogether, Suprizr changed their approach by first asking for zip code…
And then email, when they were told that Suprizr was not yet available in their area…
The result? 30% of visitors entered their zip code, 25% entered their email and 10% of them shared the site.
Sometimes, your first attempt at a smoke test will not pan out the way you thought. Continue iterating before giving up on the idea completely.
Step 6: Call it.
Some of you will have the luxury of being bigger businesses with statistically significant data to analyze while some of you will not. No matter your situation, eventually you will have to make a decision.
It can be a good idea to set a designated timeframe for step five as analysis and iteration can go on for ages, which would defeat the purpose of running a smoke test in the first place.
Work with the data you have, make a decision and move on to the next.
Insights > Winning
As with regular A/B testing, strive to create an insights > wins smoke test culture. After all, as Dominic explains, even if your idea is proven to be a loser, you still win…
Always consider what other insights you can draw out…
- Why might this idea have failed?
- Was it an execution error?
- Did some iterations work better than others?
Before you launch your next product or feature idea, run a smoke test to validate it. You’ll save yourself time, money and (potentially) embarrassment.
Just follow these simple smoke test steps…
- Define the necessary proof.
- Design an experiment without engineers.
- Drive the right traffic to your experiment.
- Define your metrics and measure them.
- Analyze the results and iterate / optimize.
- Call it.
Remember, smoke tests are often thought of strictly in the startup sense, but the process is easily applicable to big companies looking to launch product features or new products / services as well.
Working on something related to this? Post a comment in the CXL community!