26 Forecasting Platforms
Forecasting platforms provide another approach to aggregating predictions about future events. Rather than using markets to establish prices that represent probabilities, as in prediction markets, these platforms collect probabilistic judgments from participants and combine them using statistical methods. This chapter examines how these platforms function, their potential applications in corporate settings, and considerations for implementation.
26.1 Introduction
Following the success of the Good Judgment Project (described in the previous chapter), several forecasting platforms have emerged that aim to crowd-source predictions on various topics. These platforms build on the techniques that made the Good Judgment Project successful — aggregating diverse judgments and applying sophisticated weighting algorithms. While prediction markets use financial mechanisms to aggregate forecasts, these platforms focus directly on collecting and combining probabilistic judgments.
Major platforms include:
Metaculus: Founded in 2015, Metaculus hosts thousands of forecasting questions across domains including science, technology, economics, and politics. Forecasters provide probabilistic predictions that are aggregated into a “Metaculus prediction” using algorithms that give more weight to forecasters with strong track records.
Good Judgment Open: A public forecasting platform run by Good Judgment Inc., which emerged from the Good Judgment Project. It hosts forecasting questions and identifies superforecasters through performance evaluation.
INFER: A platform operated by RAND that has hosted forecasting tournaments on geopolitical and economic events.
These platforms have demonstrated impressive results in a small number of academic studies. Mellers et al. (2014) documented how the Good Judgment Project teams outperformed intelligence analysts with access to classified information in the IARPA forecasting tournament. Atanasov et al. (2017) followed this with their study “Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls,” showing that properly designed forecasting platforms can match or exceed prediction market accuracy while avoiding many regulatory challenges. Similarly, Codi et al. (2022) found that aggregated human judgment from Metaculus and Good Judgment Open performed comparably to computational model ensembles, outperforming the COVID-19 Forecasthub in in 3 of 6 predictions for incident cases and 4 of 6 predictions for incident deaths.
26.2 How Forecasting Platforms Work
The core mechanics of forecasting platforms typically include:
Question generation: Questions are crafted to have clear resolution criteria and timeframes. For example: “Will company X’s market capitalisation exceed $100 billion by December 31, 2026?”
Forecaster participation: Individual forecasters submit probability estimates (for binary questions) or distributions (for numeric questions). They can update these estimates as new information becomes available.
Aggregation algorithms: Platforms use sophisticated methods to combine individual forecasts, often weighting contributors based on their track records. As illustrated in the Good Judgment Project, aggregation algorithms can outperform simple averaging. However, McAndrew et al.’s (2021) survey of aggregation techniques identified that there is limited experimental literature comparing the various options.
Scoring and feedback: Forecasters receive scores based on the accuracy of their predictions once events resolve, creating incentives for accuracy and helping identify the most skilled predictors.
Community discussion: Most platforms include comment sections where forecasters share reasoning, sources, and updates, creating a collaborative knowledge-sharing environment.
The success of these platforms rests on several key principles identified in the forecasting literature and demonstrated by the Good Judgment Project:
Diversity of perspectives: Aggregating predictions from forecasters with different backgrounds and information sources reduces collective bias, similar to how “foxes” outperformed “hedgehogs” in Tetlock’s (2005) original research.
Independent judgment: Platforms typically ask forecasters to make their own predictions before seeing others, reducing herding behaviour and preserving cognitive diversity.
Quantification: By requiring explicit probabilities rather than vague verbal estimates, platforms avoid the ambiguity problems. Research by Friedman et al. (2018) provides strong evidence for this approach, showing that precise numeric probabilities outperform verbal probability expressions in forecasting contexts.
Continuous updating: Unlike traditional forecasting, these platforms allow and encourage constant revision of estimates as new information emerges, supporting Bayesian updating practices that were characteristic of successful forecasters in the Good Judgment Project.
26.3 Differences from Prediction Markets
While prediction markets and forecasting platforms share the goal of aggregating distributed information to improve forecasts, they differ in several important ways:
Incentive Structure: Prediction markets rely on financial incentives — whether real or virtual — to motivate accurate forecasting. In contrast, forecasting platforms typically rely more on intrinsic motivation, reputation, and competition. Atanasov et al. (2017) found that these non-financial incentives can be sufficient to elicit thoughtful predictions.
Aggregation Methods: Prediction markets use price formation through trading to establish probabilities, while forecasting platforms employ statistical aggregation methods.
Barrier to Entry: Forecasting platforms typically have lower barriers to entry than prediction markets. There’s no need to understand trading mechanisms or manage a bankroll. Participants simply provide their forecasts directly. This may allow for broader participation from employees with relevant information.
Regulatory Constraints: Prediction markets face gambling-related regulatory hurdles in many jurisdictions. Forecasting platforms avoid these issues by not involving financial transactions.
Question Flexibility: Forecasting platforms can more easily handle complex questions, including those requiring numeric predictions (e.g., “What will revenue be in Q3?”) or conditional probabilities. Prediction markets typically require more structure to be tradable.
26.4 Applications in Corporate Decision Making
Forecasting platforms could potentially enhance corporate decision making in several ways. These include:
Strategic Planning: Companies could use forecasting platforms to improve estimates of future market conditions, technological developments, or regulatory changes that affect strategic planning. Rather than relying solely on executive judgment or consultant reports, a diverse set of forecasters might provide more accurate and well-calibrated probabilities. For example, a technology company might pose questions about adoption rates of new technologies, competitor product launches, or shifts in customer preferences to inform product roadmaps.
Estimating Market Opportunities: When considering new products or market entries, companies often struggle to estimate market sizes and growth rates accurately. Forecasting platforms could aggregate diverse judgments to improve these estimates.
26.5 Limitations and Critiques
Despite their promise, forecasting platforms face several limitations. This is perhaps best illustrated by the dearth of public case studies of their successes in corporate settings.
Misalignment of Questions: Forecasting tournaments work best with questions that have clearly defined resolution criteria and reasonable timeframes, but these may not align with the most strategically important questions for a company. They also don’t help companies determine which questions are most important to forecast.
Costs and implementation challenges: Forecasting platforms require significant resources to implement effectively. They need to develop questions, implement a mode of contribution (e.g. an online interface), advertise and encourage participation and provide incentives.
Limited participation: Some people don’t enjoy forecasting and may resist participating. While public forecasting sites such as Metaculus can gain a critical mass of participants, a corporation has a smaller potential participation pool.
Weighting consequences: What if a platform shows a low probability of a high-impact event? That in itself does not mean it should be ignored. The decision making process also needs to weight event consequences is determining the response.
For a perspective on the challenges with forecasting platforms, read Story (2022).