How to Use Quantitative Research for Product Innovation

If you don’t know what problem you’re trying to solve and for whom, all solutions will look good.

If you’re a product innovator or a marketer, especially in a new space, you need to have a passionate and sustained obsession with researching and understanding your target users.

Why and when do we need quantitative research

When you begin to validate your opportunity hypothesis, getting to know your users’ needs (including the latent ones), motivations, and pain points, is perhaps best initiated by performing qualitative research as the first step. 10-15 targeted users per segment of users is usually a good number to target (if doing 1-1 interviews) .

After performing these interviews and synthesizing them, some of the user needs and findings surfaced may require further research with a larger cross-segment of users, especially if you’d like to quantify some of the findings.

Let us assume you are conducting research for a new product innovation opportunity in CRM for financial planners to determine if there is a product-market fit.  You’ve discovered some unmet needs through user interviews because many financial planners you interviewed described that they feel frustrated trying to keep a track of their leads, accounts, and relationships when they’re on the go. You know you’re on to something here. Assume that you’ve also discovered a secondary need as part of the interviews which may be valuable enough to explore as a product pivot or extension.

However, you’re not quite sure if there is a clear product-market fit…. Yet.  Since determining this is your overall research objective, you should conduct quantitative research that builds on top of the early discovery and gives you richer insight by testing this opportunity with a broader audience. You may want to dive into more specific questions that produce “structured” data that you can quantify, enumerate, and analyze. 

What are some other benefits of quantitative research

The quantitative survey can also help you determine market sizing (i.e. how many users have the same unmet needs).

Another benefit of conducting a round of quantitative survey is that it can help identify clear segmentation in the respondents which can further aid the positioning and marketing of the solution. This can be achieved by highlighting differences in user’s responses based on demographic/economic/geographic attributes. E.g. you may discover/validate that a certain segment of users (say Gen X or millennials) is much more likely to do business with a new company based on the answers in the survey.

When structured correctly, the survey can also provide insight into the commercial viability of the intended solution by testing users on purchase intent.

Additionally, product concept, configuration, and positioning testing are more areas where quantitative research helps with your product definition, creation, and launch decisions

How do you perform quantitative research– Step by Step Approach

How do you collect structured insight that can be quantified to make the right decisions.

While the survey still has exploratory ambitions, you’re looking for further statistical and numerical validation of specifics.

Here are the steps involved in creating and conduct quantitative research

  1. Revisit research strategy:  The overall intent of your research is going to largely be the same regardless of what approach you take (qualitative and/or quantitative). This is set upfront before undertaking any research.
  2. Refine specific goals for the quant study: As mentioned above, it is possible that some of the research measures and specifics may have to be refined further.
  3. Design a quantitative survey: You want the survey you create to provide further actionable insight and therefore constructing it methodically is crucial. Every question should be included for a reason and in line with the research objectives. For each question, ask yourself what you will do with that information. Ask questions that start with “how many”, “how frequently”, “how likely”… instead of “how”, why” (which are suitable for in-depth qualitative rounds). See the section below on the types of questions with some examples.
  4. Conducting the research: Keeping the statistical confidence level and margin of error desired (a minimum of 95%, 5% respectively should be acceptable in many product research goals ), determine the number of respondents you need. Here’s a quick link to derive the number for your research. There are many options to find the respondents depending on your budget, time, and channel needs. You can work with providers such as surveymonkey, pollfish, qualtrics. Social media, local groups, and associations of respondent groups, personal networks, email campaigns, and other paid providers, directly targeting visitors on your business website are other ways of reaching the right respondents. The mediums to conduct the survey can also be multiple: E.g web surveys, telephone, text, print, mail, even in person.
  5. Analyzing the data: Performing the right analysis of the data obtained from the surveys is vital. This can be done through various statistical methods (to be covered in a subsequent blog).

Types of questions to ask in the survey

Here are some of the question types you can consider incorporating into your quantitative research design.

  • Screening:  Dichotomous questions: (e.g. “Yes/No”) can serve the purpose well for screening, but multiple questions can also be used for screening (E.g. “What describes your employment affiliation: Self employed/ Contractor / Full time”)
  • Pick the right Scale: If the user interviews asked the question “How do you feel about your current service” during qualitative research, the quantitative survey can now ask “How satisfied are you with the current solution” on a pre-defined scale.We can use Likert scale: E.g. “Very dissatisfied .. Neutral…. Very Satisfied”   (5 or 7 options). Or Semantic differential scales to provide gather more “descriptive” product opinions. These are also widely used for customer satisfaction surveys and test users’ ideologies. See this post or this for a detailed explanation
  • Predefine numerical choices: If you had earlier asked “How will a new service offering help you grow your business” in a user interview, you can now focus on asking “By how much can a new solution increase your revenue”. You can use predefine the choices through options like “<10%, 11-25%, 26%-50%,>51%))
  • Ranking questions: If you want to test the repondents on their most important needs/pains, you can reate a question that presents them with a ranking ability for the provided choices.
  • Constant sum: In some cases, a “Constant sum” type of question can prove valuable either in conjunction with the above or independently. E.g. “What % percentage of your daily time do you spend on each of the following activities” assuming those sum up to 100%
  • Matrix type: If you’re trying to test the users on two factors (say significance and frequency) for their most pressing needs, you can have these two factors side by side in the question.

This is by no means a comprehensive guide for survey creation, but a good starting point to appreciate the approach and the value it brings.

Other quick tips

  • Start with simple questions and build up to the more incisive ones.
  • Frame your questions such that they are not leading. Reframe any that contain compound questions that cannot be answered directly.
  • Stay consistent with the scales you use as much as possible and present the questions in the same order to every respondent.
  • Try adding doesn’t apply/other/’prefer not to say’ as a choice in the relevant questions.
  • My personal experience says under 5 minutes and under 15 questions is a good target.
  • Some product researchers don’t mind throwing in a couple of simple open-ended questions to avoid “survey fatigue”.
  • It is possible to actually perform the quantitative survey first with a broad audience and then go deeper with 1-1 qualitative user interviews. But doing it too early when the understanding of the space and the user journey is limited, makes product innovation ineffective.
  • Quantitative user research should not be used as the sole determinant of your product or research strategy. (They can go wrong. For a non-product-related research example, that didn’t prove accurate, look at the 2016 US Presidential elections polls!)

What has been your experience designing a survey and conducting quantitative product research?

10 thoughts on “How to Use Quantitative Research for Product Innovation

  • Kim

    Perhaps you can try a focus group, small roundtable of agents. Ask a few targeted questions and offer a gift card or giveaway or something.

    • Vipin Makhija

      @k4dwin Using small focus groups, as you state is definitely a good method for qualitative interviews.
      When the need is to project to and test with a large audience (potentially 100s) and geography (say across the country), a quantifiable online survey (as part of the quantitative research) is the recommended approach. What other methods do you think can be used for testing out a product concept with 100s of users?

      • Kim

        Hi Vipin,

        Sorry, just now seeing this. While having a license to sell real estate in all states and the requirements are different, each individual has to go through professional development to keep up their credentials annually. Perhaps you can reach out to RE course providers that offer professional development (this is just something I googled as an example and tell them that you are working on a project and ask if they have a weekly or monthly newsletter or email they sent out to their mailing list, subscribers and ask if you can write a blurb with the survey link to get exposure and generate interest. Can also check with the state entity responsible for upkeep of licensing renewals and fees and ask to write a blurb with a survey link to get exposure and generate interest to their subscribers or on their website. You can also start posting on RE bloggers sites comments and eventually share the link on occasion. Same as I mentioned with agents on Twitter. Create A Landing Page using WP as well sharing something about the project with a CTA to start collecting emails of agents. All the best!

  • Varun Sharma

    Nice thoughts on quant survey. Just wanted to know how it should be used in product research if it is not the sole determinant

    • Vipin Makhija

      @varuns None of the research approaches should be used as the only method or criteria for product innovation. Even if the research points towards a good product market fit, feasibility, commerical viability are all important factors that need to be considered. Research however should absolutely aid decision making.

  • Rahul

    If you’re starting out to build your first survey – then my recommendation will be to test it with a few people to see the ‘quality’ of your questions. The responses to your survey questions will force you to rethink your questions to get the data you need. Qualtrics provides a tool that allows you to insert dummy data into your form which can help you do a quick analysis of your questions.

    What I’ve also found is that it’s easy to blur the lines between wanting qualitative information from a quantitative survey – the point of the survey is to see ‘how many people said X instead of Y’ and not to get individual subjective comments from all survey respondents.

    I’ve also found it useful to build multiple hypotheses based on the qualitative research that you want to validate in the survey.

    Good points overall and well-structured @vipinmakhijaipin. The link to test statistical significance was really cool. Thanks!

    • Vipin Makhija

      Great points @rsridhar. Especially around doing a “test run” of the survey in the process of refining it.
      I haven’t personally used Qualtrics for analysis but will take a look at its capabilities.

  • Rosanna

    The definition of a quantitive research gave me an important learning point: defining questions that allow to collect additional insight or trends rather than validate the data and information available with the market research already available.
    I am willing to see how to actually handle with the data received, how to cross them with the other data available for the following step of the process

    • Vipin Makhija

      @rosanna That’s true. That’s why it is so important to have a clearly defined research intent and strategy upfront. That enables constructing the questions so the most important aspects are surfaced when the results are available.

Leave a Reply

Looking for something?

Visit the knowledge repository.

Copyright © 2020 All rights reserved