This research is inspired by the result of the works of Professor Tetlock on prediction science in the geopolitical and economics domains. He suggests that some non-experts are better than experts in predicting the future. This research attempts to identify if a group of individuals with high prediction skill exists in the general public by testing on ad testing surveys.
Modern businesses spend billions of dollars on branding and advertising of their products. Ad testing is commonly used as a tool to gauge the success and effectiveness of such campaigns. A problem faced by ad testing surveys is that the main campaign has to be kept on hold until the survey data is collected. Usually, larger the sample size the longer the delay. If a smaller group of forecasters are able to predict the opinion of a larger sample, the delay faced in ad testing surveys could be minimized.
Data from a prediction survey collected from 659 subjects living in the UK who predicted the best ads from set of 16 ad-pairs, were analyzed in this research. The analysis found that few individuals were able to predict more successfully and with greater confidence than others. Nonetheless, more research in the same domain with greater vigor is needed to fortify the claim.
Table of Contents
I. INTRODUCTION
A. Study area
B. Research Objective
II. LITERATURE REVIEW
A. Advertising Testing
B. Innovation in online surveys
C. Preference for online surveys
D. Prediction Markets
E. Gamification
F. System 1 and System 2 thinking
G. Lives based on predictions
H. Are expert forecasts always correct?
I. How do experts get away?
J. Superforecasters
III. METHODOLOGY
A. Introduction
B. Data Used
C. Sample
D. UI Design
E. Cleaning and preparing data for analysis
IV. DATA ANALYSIS
A. Inroduction
B. Analyzing the polls
C. Analyzing the prediction correctness
D. Aggregating prediction correctness
E. Analyzing prediction confidences
F. Brier Score
G. Aggregating prediction confidence
H. Ranking the Superforecasters
I. Discussion
V. RESULTS
A. Introduction
B. Score Distribution
C. Overall Accuracy
D. Are superforecasters always perfect?
E. Discussion
VI. DISCUSSION
A. Introduction
B. Discussion
VII. CONCLUSION
A. Conclusions
B. Recommendations
VIII. APPENDICES
Research Objective and Key Themes
The primary goal of this research is to determine whether individuals with high forecasting skills—termed "superforecasters"—exist within the general public and if their collective predictions regarding advertising effectiveness can reliably substitute for larger sample sizes in market research. The study investigates if these specific forecasters can consistently identify winning advertisements, thereby reducing the time and costs associated with traditional advertising testing surveys.
- Application of "Superforecasting" concepts to the domain of market research and advertising testing.
- Implementation of gamification techniques to improve respondent engagement and data quality in online surveys.
- Use of prediction polls to identify exceptional forecasters from a general survey population.
- Development of quantitative methods for aggregating predictions and assessing confidence levels using extremization functions.
Excerpt from the Book
I. INTRODUCTION
ONLINE market research is a collective term for several types of research such as social media research, web traffic measurement, market measurement, advertising effectiveness research, media research, new product analysis, opinion research and many others [1]. Market research firms are constantly looking at improving the quality of their services. Some of the major areas focused are improving the quality of the data collected, reducing the time taken to complete a project and reducing the overall costs incurred to complete a project.
In terms of improving the quality of the data collected, an innovative concept labelled gamification by market research experts was explored. The concept, though it has a broad and ambiguous definition, is the process of incorporating game components like role-playing, leaderboards, mini-challenges, instant gratification onto a traditional survey.
The end result for the survey taker would be more like playing a game than answering a survey. Researchers have found that gamified surveys increase respondent engagement and as a result improve the quality of the data collected.
This study is inspired by Professor Tetlock’s [2] findings on the prediction in the domains of geo politics and economics. He claims that some non-experts are better than experts at predicting future events. Integrating the prediction model for online market research is an area less explored by the research community. This exploratory study attempts to find if Tetlock’s claim holds true in market research domains. In order to restrict the scope to a manageable level this research will focus primarily on ad-testing.
Summary of Chapters
I. INTRODUCTION: Introduces the scope of online market research, the potential of gamification to enhance data quality, and the research objective of identifying superforecasters within the advertising testing domain.
II. LITERATURE REVIEW: Reviews the theoretical foundations of advertising testing, the evolution of online surveys, prediction markets, gamification, and the psychological principles of System 1 and System 2 thinking, alongside a critique of expert forecasting.
III. METHODOLOGY: Details the research design, including the use of an online survey to collect prediction and opinion data from 659 subjects across 16 ad-pairs, the UI design elements, and the data cleaning procedures.
IV. DATA ANALYSIS: Explains the quantitative processes used to analyze poll results, calculate prediction correctness, aggregate confidence levels using extremization functions, and the final ranking of superforecasters.
V. RESULTS: Presents the findings regarding the distribution of superforecasters, their overall accuracy compared to the general sample, and the analysis of outliers and prediction consistency.
VI. DISCUSSION: Reflects on the experiment's outcomes, potential limitations, the need for more diverse research designs, and the ongoing academic debate regarding mathematical scoring methods like the Brier score.
VII. CONCLUSION: Summarizes the study’s findings that superforecasters can indeed represent the opinions of larger samples, offering potential cost and time savings for market research, and provides recommendations for future study.
VIII. APPENDICES: Contains detailed supporting data, including pie charts of poll preferences, Python scripts for data processing and analysis, and descriptive statistics tables.
Keywords
Advertising testing, Forecasting, Gamification, Online surveys, Market Research, Superforecasters, Prediction markets, Data collection, Survey engagement, SPSS, Prediction accuracy, Confidence extremization, Brier score, Respondent behavior, Quantitative analysis.
Frequently Asked Questions
What is the primary focus of this research?
The research explores whether "superforecasters"—individuals exceptionally skilled at predicting outcomes—can be identified within the general public to improve the efficiency of online advertising testing.
What are the central themes of the study?
Key themes include the application of prediction science in market research, the use of gamification to increase survey engagement, and the comparative analysis of prediction accuracy between the general public and identified superforecasters.
What is the main research objective?
The goal is to determine if a small subset of superforecasters can provide prediction data that is as accurate as the aggregated opinion of a much larger sample, thereby allowing for reduced survey sample sizes and faster project turnaround.
Which scientific methods are employed?
The study uses a descriptive quantitative approach, employing an online survey with 16 ad-pairs, Python scripting for data manipulation, SPSS for statistical analysis, and extremization functions to calibrate and aggregate prediction confidence.
What does the main body of the paper cover?
The main body covers the theoretical background of market research and forecasting, the methodological steps for data collection and processing, detailed statistical analysis of the poll results, and the identification and performance evaluation of superforecasters.
Which keywords characterize this work?
The study is characterized by terms such as Advertising testing, Forecasting, Gamification, Market Research, Prediction markets, and Survey engagement.
How is "gamification" specifically used in this study?
Gamification is applied by incorporating interactive elements and game-like structures into the survey to trigger "System 2 thinking" (deliberate and purposeful reasoning) in respondents, which is hypothesized to lead to more thoughtful and accurate feedback.
Why was the "Brier score" discussed but not used as the primary metric?
While the Brier score is a standard mathematical formula for quantifying prediction accuracy, the researchers found it unsuitable for this specific design because it requires probabilistic percentages rather than single selections and lacks a clear mechanism to distinguish correct from incorrect predictions in the provided context.
What is the significance of the "extremization function"?
The extremization function is used to adjust and calibrate the confidence levels reported by respondents. It moves values away from the midpoint (0.5), effectively quantifying and amplifying extreme confidence in predictions to better align with actual accuracy.
What conclusion does the author reach regarding the sample size?
The author concludes that using a handful of superforecasters can effectively represent the aggregated opinion of a larger sample, potentially allowing companies to run market research with smaller groups, which leads to significant savings in time and operational costs.
- Quote paper
- Ruhaim Izmeth (Author), 2016, Identifying Superforecasters in Online Market Research via Advertisement Testing Surveys, Munich, GRIN Verlag, https://www.hausarbeiten.de/document/367922
