How to Use Quantitative Research for Product Innovation

If you’re a product innovator or a marketer, especially in a new space, you need to have a passionate and sustained obsession with researching and understanding your target users.

Why and when do we need quantitative research

When you begin to validate your opportunity hypothesis, getting to know your users’ needs (including the latent ones), motivations, and pain points, is perhaps best initiated by performing qualitative research as the first step. 10-15 targeted users per segment of users is usually a good number to target (if doing 1-1 interviews) .

Let us assume you are conducting research for a new product innovation opportunity in CRM for financial planners to determine if there is a product-market fit.  You’ve discovered some unmet needs through user interviews because many financial planners you interviewed described that they feel frustrated trying to keep a track of their leads, accounts, and relationships when they’re on the go. You know you’re on to something here. Assume that you’ve also discovered a secondary need as part of the interviews which may be valuable enough to explore as a product pivot or extension.

However, you’re not quite sure if there is a clear product-market fit…. Yet.  Since determining this is your overall research objective, you should conduct quantitative research that builds on top of the early discovery and gives you richer insight by testing this opportunity with a broader audience. You may want to dive into more specific questions that produce “structured” data that you can quantify, enumerate, and analyze. 

What are some other benefits of quantitative research

The quantitative survey can also help you determine market sizing (i.e. how many users have the same unmet needs).

Another benefit of conducting a round of quantitative survey is that it can help identify clear segmentation in the respondents which can further aid the positioning and marketing of the solution. This can be achieved by highlighting differences in user’s responses based on demographic/economic/geographic attributes. E.g. you may discover/validate that a certain segment of users (say Gen X or millennials) is much more likely to do business with a new company based on the answers in the survey.

When structured correctly, the survey can also provide insight into the commercial viability of the intended solution by testing users on purchase intent.

Additionally, product concept, configuration, and positioning testing are more areas where quantitative research helps with your product definition, creation, and launch decisions

How do you perform quantitative research– Step by Step Approach

How do you collect structured insight that can be quantified to make the right decisions.

While the survey still has exploratory ambitions, you’re looking for further statistical and numerical validation of specifics.

Here are the steps involved in creating and conduct quantitative research

1. Revisit research strategy:  The overall intent of your research is going to largely be the same regardless of what approach you take (qualitative and/or quantitative). This is set upfront before undertaking any research.
2. Refine specific goals for the quant study: As mentioned above, it is possible that some of the research measures and specifics may have to be refined further.
3. Design a quantitative survey: You want the survey you create to provide further actionable insight and therefore constructing it methodically is crucial. Every question should be included for a reason and in line with the research objectives. For each question, ask yourself what you will do with that information. Ask questions that start with “how many”, “how frequently”, “how likely”… instead of “how”, why” (which are suitable for in-depth qualitative rounds). See the section below on the types of questions with some examples.
4. Conducting the research: Keeping the statistical confidence level and margin of error desired (a minimum of 95%, 5% respectively should be acceptable in many product research goals ), determine the number of respondents you need. Here’s a quick link to derive the number for your research. There are many options to find the respondents depending on your budget, time, and channel needs. You can work with providers such as surveymonkey, pollfish, qualtrics. Social media, local groups, and associations of respondent groups, personal networks, email campaigns, and other paid providers, directly targeting visitors on your business website are other ways of reaching the right respondents. The mediums to conduct the survey can also be multiple: E.g web surveys, telephone, text, print, mail, even in person.
5. Analyzing the data: Performing the right analysis of the data obtained from the surveys is vital. This can be done through various statistical methods (to be covered in a subsequent blog).

Types of questions to ask in the survey

• Screening:  Dichotomous questions: (e.g. “Yes/No”) can serve the purpose well for screening, but multiple questions can also be used for screening (E.g. “What describes your employment affiliation: Self employed/ Contractor / Full time”)
• Pick the right Scale: If the user interviews asked the question “How do you feel about your current service” during qualitative research, the quantitative survey can now ask “How satisfied are you with the current solution” on a pre-defined scale.We can use Likert scale: E.g. “Very dissatisfied .. Neutral…. Very Satisfied”   (5 or 7 options). Or Semantic differential scales to provide gather more “descriptive” product opinions. These are also widely used for customer satisfaction surveys and test users’ ideologies. See this post or this for a detailed explanation
• Predefine numerical choices: If you had earlier asked “How will a new service offering help you grow your business” in a user interview, you can now focus on asking “By how much can a new solution increase your revenue”. You can use predefine the choices through options like “<10%, 11-25%, 26%-50%,>51%))
• Ranking questions: If you want to test the repondents on their most important needs/pains, you can reate a question that presents them with a ranking ability for the provided choices.
• Constant sum: In some cases, a “Constant sum” type of question can prove valuable either in conjunction with the above or independently. E.g. “What % percentage of your daily time do you spend on each of the following activities” assuming those sum up to 100%
• Matrix type: If you’re trying to test the users on two factors (say significance and frequency) for their most pressing needs, you can have these two factors side by side in the question.

This is by no means a comprehensive guide for survey creation, but a good starting point to appreciate the approach and the value it brings.

Other quick tips

• Start with simple questions and build up to the more incisive ones.
• Frame your questions such that they are not leading. Reframe any that contain compound questions that cannot be answered directly.
• Stay consistent with the scales you use as much as possible and present the questions in the same order to every respondent.
• Try adding doesn’t apply/other/’prefer not to say’ as a choice in the relevant questions.
• My personal experience says under 5 minutes and under 15 questions is a good target.
• Some product researchers don’t mind throwing in a couple of simple open-ended questions to avoid “survey fatigue”.
• It is possible to actually perform the quantitative survey first with a broad audience and then go deeper with 1-1 qualitative user interviews. But doing it too early when the understanding of the space and the user journey is limited, makes product innovation ineffective.
• Quantitative user research should not be used as the sole determinant of your product or research strategy. (They can go wrong. For a non-product-related research example, that didn’t prove accurate, look at the 2016 US Presidential elections polls!)