The Concept Test – A Game Of Russian Roulette

The Concept Test is plagued with problems.

New products and services, line extensions, and repositioning efforts regularly fail because many fail to ignite consumer interest even though they achieved the required scores. Nevertheless, companies spend hundreds of thousands every year on concept tests.

Take the new product idea, describe it in one or two paragraphs, put a price on it, and show it to a hundred or more buyers in a product category. The winning products/ packages/ sizes are then selected. Two years and untold amounts of money later, the product fails to live up to expectations and its gone.

What’s going wrong? Often, it’s the testing…

1. Extrapolation and Over-Excitement

Almost every marketeer/ researcher has done one (if not hundreds) of these tests. Yet, such tests often raise as many questions as they answer.

Assumptions are then made about the true drivers and barriers of the idea. Or even worse, some ignore this level of diagnosis. Instead they make decisions purely based on a top box score, which often is uniqueness (or more to the point, novelty) driven in many cases.

This means that it isn’t solving a genuine problem in people’s lives, but piques their interest because it’s new and different. Most of these ideas sound great to people on a page, but when it comes to parting with hard earned cash, it doesn’t happen.

We often hear marketing folk asking questions like “Is 19% in the top box? Is there a 4th variation we should offer?” Or, “if we changed the price, how much will trial increase? Would it go to 30%?”

Is the dog wagging the tail, or the tail wagging the dog?

2. Concept Construction

An enthusiastic marketeer triggers the traditional concept test when they ask, “what’s the potential for this big new idea?” Let’s look at how many variables there might be for this so-called new idea.

Here’s a mind-blowing example of the number of component parts that could be combined to form a concept for a new food dressing.

So how do you narrow it down to the ones you want to actually put in front of consumers? Which concepts do you test or how do you describe the product exactly? What features and benefits do you stress and what price do you charge? If you think the answer is a focus group, read our opinion on why this isn’t the solution and why we strongly advocate that this is what you should be doing instead.

3. Zero Reality

So next, consumers rate the concept in terms of purchase probability. Like all self-reported measures of consumer buying though, the scales used tend to overstate the actual purchasing that takes place.

Researchers sometimes assume all prospective consumers will be aware of the product (which never happens) and those aware of it will be able to buy it (which also never happens).More often than not, people are more likely to say they “Definitely will buy” than in fact do buy. This is true of all product categories we have worked across.

Also, the consumer is evaluating the idea in a vacuum, and rarely given a real-life scenario – the sales environment and competitor products and noise to contend with. They are basing their judgements on an extremely hypothetical scenario.

4. The Test Audience

Marketing 101 is that a brand or product should be well targeted and that you can’t be all things to all people.

So, if and when the idea is built with consumers in qual, the marketeer and researcher are hell-bent on defining the consumer target and recruiting the right people to participate. But, when it comes to quant. concept testing, this line of thought goes out the window. Nationally representative audiences are often chosen. No wonder an idea which thrives in qual then bombs in quant.

So why does no one seem to be drilling into this single, yet mission-critical moment within the innovation process? The key decision about which innovative product ideas or concepts to fund? The concept test has remained largely unchanged for 30 years and is still taught by academics as being the ultimate innovation business decision making tool.


We aren’t the only people to be questioning, challenging this established practice and asking why?!

Here are some numbers about the state of innovation – please judge for yourself if there is a crisis…

In 1971, research determined that 40% of new FMCG products failed. Today the failure rates are between 80-90% (ProductScan Report 2015)

Also, the rate of truly new products among all new FMCG goods has declined from 18% in 1986 to barely 8% in 2015, according to ProductScan. The rest were mere “me-too’s” and line extensions.

No wonder that BCG’s most current executive survey finds that nearly 50% of executives are unhappy with the return on their innovation investment, the second most unsatisfactory sector being FMCG (Boston Consulting Group, 2018 Senior Executive Innovation Survey)


See Part 2 for our tips on how to change your approach to concept testing without abandoning it as a decision-making tool altogether.