It is no secret the market research industry is under pressure to deliver sound and strategic insights within shrinking budgets and timeframes. In fact, ten years from now efficiency is predicted to be the deciding factor when commissioning research. This presents a serious challenge for qualitative research, with traditional methods being directly at odds with the need for speed. However, demand for qualitative insight is predicted to increase, driven by a need to convert big data into smart data and provide meaning, clarity and focus to the growing sea of numbers. Confronted with this scenario, SKIM was keen to investigate potential solutions to ensure qualitative research remains an accessible source of insight for our clients.
An Intelligent Solution?
From Uber’s self-driving cars to Amazon’s warehouse robots, artificial intelligence (AI) seems to be reaching human-level dexterity, with robots taking on the role of professionals across different industries, from law to healthcare to education. But does this apply to market research? SKIM was curious to explore the possibility, to find out whether machines can identify insights and speed up the qualitative research process. At the same time, we wanted to explore the potential trade-offs and whether robots will render human researchers unnecessary.
William Serfaty, global strategy and insights manager at Danone, turned to SKIM to understand the drivers of consumption for a new product to inform Danone’s communication strategy. Given the very tight timeline for this project, we saw an opportunity to put these robots to the test. With Danone on-board, we partnered with Voxpopme, a video research platform that captures in just a few days, in-context, consumer feedback at scale and provides a range of AI and automation tools to analyse the data. Together, we launched a head-to-head competition between human researchers and machines. Two teams were formed, each tasked with analysing 127 self-recorded consumer videos using different research methodologies. One team had access to AI and automated tools, while the other team relied on traditional human methods of analysis. In total three reports were produced and judged by William Serfarty at Danone:
• automated topline
• human-only full report
• automated + human analysis full report
Each team was required to log the amount of time spent analysing the videos and creating each report. By doing so we could compare and evaluate the process and insights produced by machines, humans and a combination of the two.
Much to our surprise the winner of this competition was the automated plus human team, based on the full report created by human and machine analysis. By using this collaborative approach we were able to produce a full report in half the time it took the human analysis team. However, most unexpectedly, William selected this as his favourite option even before learning it took less time to create. In fact, he deemed both full reports to be equally insightful and actionable and had real trouble telling them apart. This means that by collaborating with machines we can enhance efficiency without compromising the depth and quality of insights. However, it is important to emphasize that collaboration is key. The machine on its own could not produce meaningful insights and the topline output that relied the most on automated tools was not deemed sufficient as a final deliverable. Whilst only taking two days (versus five days) to produce and costing two-thirds of the price, efficiency savings incurred were too much to the detriment of the insights gained. In essence, the trade-off was too great.
The Need for Speed – and Quality
Our research therefore validates the use of automated tools during qualitative research analysis whilst also revealing the importance of humans in driving the analysis process. The role of automated tools being to facilitate, not replace researchers. At SKIM we believe these tools can help the qualitative industry adapt its processes and remain accessible to clients in the face of increasing efficiency pressures and achieve the seemingly impossible goal of delivering on time, money and quality expectations, without any trade-off. It is for this reason that William was so pleased by the outcome of this experiment and sees this methodology as a way to bridge his stakeholders’ need for both speed and quality. “The outcome was a nice surprise! Now we can get a report faster that provides the level of detail you’d get from a traditional report”, said William Serfaty.
Five Tips on how to Collaborate with Machines
Based on our experiment we have identified five tips on how to take advantage of this technology and collaborate with machines during qualitative analysis.
1: Initial scepticism of AI will be inevitable; push past it to reap the rewards of automation.
Originally we believed that opportunities to automate were more suited to structured, quantitative research processes. Given the human nature of qualitative research, we went into this experiment questioning whether automation is even possible. Although we’ve seen developments in natural language processing, at this point we are far from achieving total automation. With this in mind, our automated team was initially sceptical and resistant to using the AI tools. It was therefore necessary to overcome this hurdle in order to benefit from automation. By doing, so we discovered the opportunity to halve the time it takes to create an equally insightful full report.
2: Don’t expect machines to provide the answers. While there is much industry hype around AI and other next-gen automation technologies, we learned that machines do not offer a “magic bullet” solution to fulfil a brand’s insight needs. When conducting qualitative analysis, there is currently limited value in automated tools without human involvement. The outputs produced are words and charts that hold little meaning on their own and with mixed levels of accuracy. Machines can’t connect the dots, determine which of the insights are truly key or identify the drivers. Even to create an initial topline report, human analysis is required to review automated outputs, understand their meaning, and narrow down which information is relevant. While in time it is likely their intelligence will increase, for now at least, automation tools cannot provide stand-alone answers. As a result, it is important to understand how best to use them to our advantage.
3: Use AI outputs as the starting point for human analysis. While they don’t provide a magic bullet solution, we learned that automation tools can empower qualitative researchers to conduct analysis at speed. In contrast to our human analysis team, which had to spend a week reviewing all the video transcripts, the starting point for our automated team was the machine outputs. By analysing these, rather than the raw data, within just one day we were able to build up a picture of the overall story and identify key learnings.
4: Expect high-speed analysis to produce high-level findings. When turning to these tools it is important to have the right expectations. If internal time pressure means immediate answers are required, this technology can certainly help. However, the result of high-speed analysis is a bird’s-eye view, meaning very highlevel findings – not the deep dive insights and strategic recommendations clients have come to expect from qualitative studies. 5: Being strategic takes time; don’t cut this corner. Even once we revealed the time and cost involved in creating each of the outputs, Danone’s William Serfaty felt the collaborative (automated + human researcher) report delivered greater returnon-investment over the mostly automated and fully human-generated versions. Thus, reports that rely heavily on automated outputs may be quicker, but speed comes at the cost of strategic and actionable insights.
The Future Potential of AI
More time and deeper human analysis is therefore required to translate high-level information into clear guidelines and recommendations. Nevertheless, this process takes qualitative researchers half the time when armed with automated tools to help them. As a result, at SKIM we’re understandably optimistic about the future potential for automation and AI-enhanced qualitative research methodologies. Automated charts, thematic text analysis and filters are basic forms of automation compared to sentiment and object analysis that require machine intelligence. Yet, at this stage it is the basic tools that are most valuable in enabling qualitative researchers to efficiently analyse large quantities of data at speed.
Take Advantage of Automation
Yet even despite their limited sophistication, automated tools were able to halve the time it takes to create a full report without compromising the depth and quality of insights. Given this result, SKIM has introduced an automation solution to our qualitative research offerings, SKIM’s Smart Qual. We use this solution as a standalone methodology, pre/post task, or to replace traditional open-ends in surveys to capture in-context insights at speed. As technology advances and becomes more sophisticated, we believe the uptake and benefit of using these tools will only increase and help qualitative researchers to deliver strategic insights faster. At SKIM we are therefore keen to continue our investigation in this area, learning how to collaborate with machines and take advantage of AI and automation! ■