Qual Research by the Numbers
By Elisabeth A. Sullivan, Staff Writer for Marketing News
Online Communities Can Provide Valuable Insights, But Keep Your Research Goals In Mind When Drafting Your Guest List
Qualitative research isn’t quantifiable, of course. It’s not a numbers game. But numbers do come into play when you’re building an online research community because the size of the community you choose may have a significant impact on the feedback you receive.
Online communities are a boon to marketers seeking qualitative research data to guide their increasingly customer-focused strategies. They are cost-efficient, accessible and effective forums in which marketers and market researchers can connect with hundreds or thousands of consumers at the click of a mouse”but they’re not without their issues. The dubious quality and integrity of respondents is an oft-discussed snag and once an online sample’s quality is assured, marketers often face yet another hurdle: determining the appropriate size of an online community to fit their research needs.
Some call it a “hot potato,” while others say it’s a “religious war.” The debate over which online community size is most appropriate for qualitative research has turned into a contested battle. “I view this whole thing as very religious right now, big
versus small,” says Brad Bortner, a principal analyst at Cambridge, Mass.-based Forrester Research Inc. Vendors and researchers take sides based on “what they’ve done,” he says, and marketers find themselves in a muddle, trying to decide which size best fits their research needs.
Smaller online communities, with just a few hundred members, can be more manageable than larger communities and can give their members a sense of exclusivity and privilege, which might encourage them to contribute more detailed feedback on customer service issues or new product ideas, for example. But smaller groups also might not give marketers enough fresh insight. Meanwhile, online communities with thousands of participants can help market researchers reach a highly segmented consumer base or unearth some insightful hidden gems to help guide marketers’ decisions on product development or innovation. But they might also lead to a wasted investment on inactive or unhelpful respondents who stand by as others contribute to the conversation.
The goal of qualitative research, of course, is to gather feedback and ideas from consumers that can help direct your marketing decisions, rather than support them.
There is no optimal solution, no one-size-fits-all approach, so roll up your sleeves and do your homework before you try a few online communities on for size.
Online community provider Communispace Corp., based in Watertown, Mass., has managed more than 300 private online communities for b-to-b and b-to-c clients including AARP, Frito-Lay, Hewlett-Packard and Reebok. Most of those communities top out at around 400 members. Not surprisingly, the company encourages clients that the size of a community does matter”and smaller is often better.
“This is such a hot potato and, obviously, I’m biased because of where I work,” says Manila Austin, Communispace’s director of research, but she and her colleagues argue that smaller online communities allow consumers to foster relationships with their peers and with the sponsoring corporations, thus engendering deeper, more insightful qualitative feedback. Participants feel like the chosen ones, speaking on behalf of their fellow consumers in a venue in which their voices are not only heard but respected.
The goal of qualitative research, of course, is to gather feedback and ideas from consumers that can help direct your marketing decisions, rather than support them, Austin notes, so a select group of highly engaged consumers provides more than adequate insight. “Strength in numbers definitely bears out when you want that sort of predictability and ‘projectability’ ” necessary for quantitative analysis, she says, “but people just can’t connect with 1,000 people or more. … It’s better to get to know a handful of the right people really well.”
“Once you get up to 500 or so, participation starts to go down; people start to feel anonymous,” Austin says. Communispace clients looking for a higher volume of feedback oft en create multiple communities, she says, rather than building one massive community in which consumers can feel insignificant. Larger communities are an option, she adds, but companies have to scale the intimacy by breaking participants into smaller groups.
On the other side of the equation, Justin Cooper, co-founder and chief innovation and marketing officer at Passenger, a Los Angeles-based online community provider, says that when it comes to bringing people together to glean actionable qualitative insight, the more the merrier. Cooper calls the communities Passenger manages “open conversations,” with “lots and lots and lots of people representing lots of perspectives” that can help companies innovate.
Passenger develops and manages private online communities with an average of 2,000 members (or as many as 5,000 or more) for clients such as Chrysler, the Coca-Cola Co. and Adidas. “We don’t really think about size as a limitation,” Cooper says. “We think about size as a brand’s ability to have a meaningful dialogue with their customers, and the customers’ ability to have a relationship with the brand.”
Passenger’s communities are invitation-only, and there are no incentives for membership. Consumers who agree to participate do so because they have the chance to interact directly with brand managers, marketers and other company representatives, Cooper says. It’s a privileged conversation on a larger scale. “What we do is more like a party, your ability as a host to move around and engage in conversations with all of these different people,” he says. It’s as if you’re granted the ability to have meaningful conversations with everyone there simultaneously because of Passenger’s technology. And rather than segmenting that audience, Passenger uses its technology platform to mine specific information from specific consumer groups without interrupting the communities’ conversational flow.
But not all party guests are good conversationalists, and some tend to drown out their peers. “More respondents do not automatically lead to better data,” says Kim Dedeker, vice president of external capability leadership in the consumer and market knowledge division at Procter & Gamble Co. “While greater numbers of respondents
have the potential to generate broader diversity or unexpected insights, [or] the survey equivalent of finding a needle in a haystack,” larger online communities or panels also pose the risk of bogging down marketers with disparate or off-target ideas. And, adds Lawrence Hamer, an associate professor of marketing at DePaul University in Chicago, “Even if you do find a needle in the haystack, it might be hard to know it’s a needle when you see it.” The optimal size for an online research community may very well be in the eye of the beholder. According to Forrester’s Bortner, the real key to success with online communities is to dedicate the resources necessary to both drive your community’s discussions and harvest the valuable insights they generate, which is a valid point. The aim of qualitative research, after all, is to access consumers’ feedback and insights that may help lead you to your next brilliant marketing decision, whether you find more useful ideas from talking to a select group of consumers or opening up the floor to a cacophony of voices. As long as you’re a deft virtual host or hostess, both you and your guests will consider your online interactions time well spent.
Online Communities Can Provide Valuable Insights, But Keep Your Research Goals In Mind When Drafting Your Guest List
Qualitative research isn’t quantifiable, of course. It’s not a numbers game. But numbers do come into play when you’re building an online research community because the size of the community you choose may have a significant impact on the feedback you receive.
Online communities are a boon to marketers seeking qualitative research data to guide their increasingly customer-focused strategies. They are cost-efficient, accessible and effective forums in which marketers and market researchers can connect with hundreds or thousands of consumers at the click of a mouse”but they’re not without their issues. The dubious quality and integrity of respondents is an oft-discussed snag and once an online sample’s quality is assured, marketers often face yet another hurdle: determining the appropriate size of an online community to fit their research needs.
Some call it a “hot potato,” while others say it’s a “religious war.” The debate over which online community size is most appropriate for qualitative research has turned into a contested battle. “I view this whole thing as very religious right now, big
Quality First
For a lot of market research, online communities and panels are the ultimate focus group, available to marketers 24/7 and offering companies the chance to connect with as many of their target customers or general consumers as they desire. But apart from sample sizes and participation rates, the word on every researcher’s lips is, of course, quality.
The Internet allows researchers to connect with very specific consumer segments, pinpointing target demographics and target markets quickly and affordably, but it’s a faceless medium in which proof of identity can be questionable.
Some online community members or survey participants might be in it for the incentives: “Suzy,” a 37-year-old mother from Ohio, may very well be Mitch, a 23-year-old bartender in Arizona looking to score a few free gift cards by participating in an online community or survey. And Mitch might be a serial survey taker, registered on several online panels at once.
“The key questions any market researcher needs to ask about panels are: ‘What is the quality of the panel?’ and ‘What is the quality of the data the respondents provide?’ ” says Kim Dedeker, vice president of external capability leadership in the consumer and market knowledge division at Procter & Gamble Co. “In choosing a panel, ensuring panel integrity”knowing that the people on the panel are who they say they are, and are providing their genuine thoughts and responses”is a much more important factor for researchers to consider than just the number of respondents on a panel.”
Community and panel providers are already taking steps to ensure the quality of their participants and responses by raising recruiting standards, downplaying incentives or carefully monitoring participation and engagement levels.
For example, MarketTools Inc., a San Francisco-based online research firm and community and panel provider, offers a solution called TrueSample that helps vet online survey respondents to confirm that they are both authentic and engaged by validating as much personal information as possible and monitoring responses for unnatural or inconsistent results.
Meanwhile, Quester Corp., a Des Moines, Iowa-based marketing research company and Internet software provider, has built its online survey system called Socrates to discourage bogus respondents from participating by asking a series of intuitive, in-depth follow-up questions that a less interested participant could not”or wouldn’t want to”answer.
“Marketers should worry much more about the quality of the samples that they get from online panels,” says Kusum Ailawadi, a marketing professor at Dartmouth College’s Tuck School of Business. “Larger sample sizes won’t make up for poor quality. They will just make their results more precise” more precisely wrong if the sample is not representative of the population they are interested in.”
- E.A.S.
versus small,” says Brad Bortner, a principal analyst at Cambridge, Mass.-based Forrester Research Inc. Vendors and researchers take sides based on “what they’ve done,” he says, and marketers find themselves in a muddle, trying to decide which size best fits their research needs.
Smaller online communities, with just a few hundred members, can be more manageable than larger communities and can give their members a sense of exclusivity and privilege, which might encourage them to contribute more detailed feedback on customer service issues or new product ideas, for example. But smaller groups also might not give marketers enough fresh insight. Meanwhile, online communities with thousands of participants can help market researchers reach a highly segmented consumer base or unearth some insightful hidden gems to help guide marketers’ decisions on product development or innovation. But they might also lead to a wasted investment on inactive or unhelpful respondents who stand by as others contribute to the conversation.
The goal of qualitative research, of course, is to gather feedback and ideas from consumers that can help direct your marketing decisions, rather than support them.
There is no optimal solution, no one-size-fits-all approach, so roll up your sleeves and do your homework before you try a few online communities on for size.
Online community provider Communispace Corp., based in Watertown, Mass., has managed more than 300 private online communities for b-to-b and b-to-c clients including AARP, Frito-Lay, Hewlett-Packard and Reebok. Most of those communities top out at around 400 members. Not surprisingly, the company encourages clients that the size of a community does matter”and smaller is often better.
“This is such a hot potato and, obviously, I’m biased because of where I work,” says Manila Austin, Communispace’s director of research, but she and her colleagues argue that smaller online communities allow consumers to foster relationships with their peers and with the sponsoring corporations, thus engendering deeper, more insightful qualitative feedback. Participants feel like the chosen ones, speaking on behalf of their fellow consumers in a venue in which their voices are not only heard but respected.
The goal of qualitative research, of course, is to gather feedback and ideas from consumers that can help direct your marketing decisions, rather than support them, Austin notes, so a select group of highly engaged consumers provides more than adequate insight. “Strength in numbers definitely bears out when you want that sort of predictability and ‘projectability’ ” necessary for quantitative analysis, she says, “but people just can’t connect with 1,000 people or more. … It’s better to get to know a handful of the right people really well.”
“Once you get up to 500 or so, participation starts to go down; people start to feel anonymous,” Austin says. Communispace clients looking for a higher volume of feedback oft en create multiple communities, she says, rather than building one massive community in which consumers can feel insignificant. Larger communities are an option, she adds, but companies have to scale the intimacy by breaking participants into smaller groups.
On the other side of the equation, Justin Cooper, co-founder and chief innovation and marketing officer at Passenger, a Los Angeles-based online community provider, says that when it comes to bringing people together to glean actionable qualitative insight, the more the merrier. Cooper calls the communities Passenger manages “open conversations,” with “lots and lots and lots of people representing lots of perspectives” that can help companies innovate.
Passenger develops and manages private online communities with an average of 2,000 members (or as many as 5,000 or more) for clients such as Chrysler, the Coca-Cola Co. and Adidas. “We don’t really think about size as a limitation,” Cooper says. “We think about size as a brand’s ability to have a meaningful dialogue with their customers, and the customers’ ability to have a relationship with the brand.”
Passenger’s communities are invitation-only, and there are no incentives for membership. Consumers who agree to participate do so because they have the chance to interact directly with brand managers, marketers and other company representatives, Cooper says. It’s a privileged conversation on a larger scale. “What we do is more like a party, your ability as a host to move around and engage in conversations with all of these different people,” he says. It’s as if you’re granted the ability to have meaningful conversations with everyone there simultaneously because of Passenger’s technology. And rather than segmenting that audience, Passenger uses its technology platform to mine specific information from specific consumer groups without interrupting the communities’ conversational flow.
But not all party guests are good conversationalists, and some tend to drown out their peers. “More respondents do not automatically lead to better data,” says Kim Dedeker, vice president of external capability leadership in the consumer and market knowledge division at Procter & Gamble Co. “While greater numbers of respondents
Reap What You Sow
Yes, online market research communities can yield actionable, directive insight, but that insight is useless unless it’s harvested. Online communities can’t function on their own. You need to have the tools and dedicated resources in place to both guide the conversation and gather the worthwhile tidbits.
Whether you’re building and hosting your own online research community or working with a community provider, make sure your team is on board and ready to participate in conversations with your customers. Marketers, product designers, brand managers, everyone should be ready and willing to participate in the give-and-take with customers to drive the conversation forward, to pose questions to the community and field any questions from it. The more your team is involved in guiding discussions or lifting the curtain a bit on your company’s objectives, the better the resulting insight can be, experts say.
Be wary of tiring your community members by asking too much of them - or overharvesting, as Forrester Research’s Brad Bortner says - but keep members engaged by supporting a regular conversational flow. “And what you can do, whether [your community] is small or large, is if someone says something you think is cool, you can pull him aside and have a conversation with him,” Bortner says.
If you’re working with a small community, you can catch that cool feedback by simply monitoring members’ conversations on a regular basis. Dedicate internal resources to keep tabs on the community. If you’re working with a large community, choose a provider that offers a software platform that can mine useful data by searching for keywords or other actionable cues.
Whatever your method, make sure that your online community members aren’t simply talking amongst themselves. Let them know you’re listening.
- E.A.S.
have the potential to generate broader diversity or unexpected insights, [or] the survey equivalent of finding a needle in a haystack,” larger online communities or panels also pose the risk of bogging down marketers with disparate or off-target ideas. And, adds Lawrence Hamer, an associate professor of marketing at DePaul University in Chicago, “Even if you do find a needle in the haystack, it might be hard to know it’s a needle when you see it.” The optimal size for an online research community may very well be in the eye of the beholder. According to Forrester’s Bortner, the real key to success with online communities is to dedicate the resources necessary to both drive your community’s discussions and harvest the valuable insights they generate, which is a valid point. The aim of qualitative research, after all, is to access consumers’ feedback and insights that may help lead you to your next brilliant marketing decision, whether you find more useful ideas from talking to a select group of consumers or opening up the floor to a cacophony of voices. As long as you’re a deft virtual host or hostess, both you and your guests will consider your online interactions time well spent.
