19  The outside view

The outside view is a simple modelling approach in which you develop an estimate based on a class of similar previous cases. The outside view can overcome problems such as overplacement, overestimation, availability, representativeness, and anchoring and adjustment.

The following story provides an illustration.

19.1 Designing a textbook

Daniel Kahneman (2011) was involved in a project to develop a curriculum and textbook for a course in judgement and decision making. At the conclusion of a working session on the curriculum, Kahneman asked each of the participants in that session to write down how long they thought it would take to submit a draft of the textbook to the Department of Education.

Kahneman collected the estimates and wrote them on the board. All of the responses were between 18 and 30 months, a narrow band suggesting a couple of years of work to come.

Kahneman then asked the curriculum expert in the room about his past experience.

The expert started: “You know, I never realized this before, but in fact not all the teams at a stage comparable to ours ever did complete their task. A substantial fraction of the teams ended up failing to finish the job.”

Kahneman asked how large he estimated that fraction was.

The expert: “About 40%”

Kahneman: “Those who finished, how long did it take them?”

The expert: “I cannot think of any group that finished in less than seven years,” he replied, “nor any that took more than ten.”

Kahneman again: “When you compare our skills and resources to those of the other groups, how good are we? How would you rank us in comparison with these teams?”

“We’re below average,” he said, “but not by much.”

All of this came from someone who only moments earlier had estimated a time of similar magnitude as the rest of the group.

The textbook was ultimately delivered in 8 years and was never used.

19.2 The inside versus the outside view

Contrast the two estimates of the time to complete the textbook.

First, we have an estimate of the time it would take from the perspective of those who know all of the details of the plan to develop the textbook. They have taken these details and turned them into an estimate. An estimate of this type - looking at the specifics of the case - is often called the inside view.

We also have a second estimate from the curriculum expert derived from other similar projects. It incorporated none of the specific details of this particular textbook. This is often called the outside view.

Contrasting the two, the inside view focuses on the specific circumstances and experiences, maybe with margin for caution. The outside view captures a bigger picture absent the detail. The inside view uses the specific information about the problem at hand. The outside view looks at whether there are similar situations - a reference class - that can provide a statistical basis for the judgement.

The problem with the inside view is that, while seemingly taking more into account, it effectively fails to account for unforeseen events that inevitably crop up during every project. There are many ways for plans to fail. Although most are improbable, the likelihood that something will go wrong in a big project is high. These problems are naturally contained in the outside view.

The result is that the outside view - ignoring the finer details of the project - can give us a better estimate.

19.3 Reference class forecasting

One way of using the outside view is reference class forecasting. When making a forecast, don’t just look at the specific circumstances of the case. Ask if there is a broader reference class of events that you can look at to see how they turned out. Base your forecast on the outcomes of the reference class rather than your own specific forecast. Or if you believe your case has some unique features, start with your reference class forecast, and cautiously adjust from there for any unique features of your case.

The power of reference case forecasting in corporate decision settings is well established. For example, Lovallo et al. (2012) examined private equity investment decisions and found that an outside view performed better than using a few analogies familiar to the decision maker.

Reference case forecasting can be particularly powerful in overcoming the planning fallacy. The planning fallacy is the tendency of people to underestimate the completion times and costs for difficult tasks (recall the circumstances where we are “overconfident”), even when they know that most similar tasks have run late or over-budget. An estimate based on a reference class can provide a check to the project estimate.

Flyvbjerg (2008) catalogued the planning mistakes associated with large projects, such as IT and infrastructure builds. These projects consistently run over budget and over time, and are often based on overoptimistic assumptions of their value. For example, he had found rail cost forecasts were typically out by 44% for rail projects, 33% for bridges and tunnels and 20% for roads. Demand for rail projects is typically overestimated by 51%.

Using a database of major projects he developed, Flyvberg has access to an effective reference class with which to adjust cost estimates for major projects. This has now been used in decision making for projects such as the Edinburgh Tram and London’s Crossrail project.

If we think of a reference-class forecast as a model, the variable that is input into the model is the reference class outcome, such as the mean time to completion of similar projects or the average cost overrun. In the simplest case, there is no mathematical transformation of that input: the output, such as the estimation of the time to completion of your project, simply equals the input.

For another perspective on the outside view, read Lovallo and Kahneman (2003) or listen to Russ Roberts (n.d.) interview Bent Flyvbjerg on Econtalk about the political economy of megaprojects.

19.4 Failure to use the outside view

The failure of the curriculum expert to use the outside view initially in making an estimate of the time to complete the forecast was not an anomaly. There is ample evidence that people will ignore the outside view even when it is right in front of them.

Freymuth and Ronan (2004) asked experimental participants to select treatment for a fictitious disease. Participants were told the efficacy of two different treatments, plus given an anecdote of a patient outcome for each.

When the participants were able to choose a treatment with a 90% success rate that was paired with a positive anecdote, they chose it 90% of the time (choosing a control treatment with 50% efficacy the remaining 10% of the time). But when paired with a negative anecdote, only 39% chose the 90% efficacy treatment. Similarly, a treatment with 30% efficacy paired with a negative anecdote was chosen only 7% of the time, but this increased to 78% when it was paired with a positive anecdote. The stories drowned out the base rate information.

19.5 Taking the outside view

Lovallo and Kahneman (2003) provide the following example of how to take the outside view.

Making a forecast using the outside view requires planners to identify a reference class of analogous past initiatives, determine the distribution of outcomes for those initiatives, and place the project at hand at an appropriate point along that distribution. This effort is best organized into five steps:

  1. Select a reference class. Identifying the right reference class involves both art and science. You usually have to weigh similarities and differences on many variables and determine which are the most meaningful in judging how your own initiative will play out. Sometimes that’s easy. If you’re a studio executive trying to forecast sales of a new film, you’ll formulate a reference class based on recent films in the same genre, starring similar actors, with comparable budgets, and so on. In other cases, it’s much trickier. If you’re a manager at a chemical company that is considering building an olefin plant incorporating a new processing technology, you may instinctively think that your reference class would include olefin plants now in operation. But you may actually get better results by looking at other chemical plants built with new processing technologies. The plant’s outcome, in other words, may be more influenced by the newness of its technology than by what it produces. In forecasting an outcome in a competitive situation, such as the market share for a new venture, you need to consider industrial structure and market factors in designing a reference class. The key is to choose a class that is broad enough to be statistically meaningful but narrow enough to be truly comparable to the project at hand.


  2. Assess the distribution of outcomes. Once the reference class is chosen, you have to document the outcomes of the prior projects and arrange them as a distribution, showing the extremes, the median, and any clusters. Sometimes you won’t be able to precisely document the outcomes of every member of the class. But you can still arrive at a rough distribution by calculating the average outcome as well as a measure of variability. In the film example, for instance, you may find that the reference-class movies sold $40 million worth of tickets on average, but that 10% sold less than $2 million worth of tickets and 5% sold more than $120 million worth.


  3. Make an intuitive prediction of your project’s position in the distribution. Based on your own understanding of the project at hand and how it compares with the projects in the reference class, predict where it would fall along the distribution. Because your intuitive estimate will likely be biased, the final two steps are intended to adjust the estimate in order to arrive at a more accurate forecast.


  4. Assess the reliability of your prediction. Some events are easier to foresee than others. A meteorologist’s forecast of temperatures two days from now, for example, will be more reliable than a sportscaster’s prediction of the score of next year’s Super Bowl. This step is intended to gauge the reliability of the forecast you made in Step 3. The goal is to estimate the correlation between the forecast and the actual outcome, expressed as a coefficient between 0 and 1, where 0 indicates no correlation and 1 indicates complete correlation. In the best case, information will be available on how well your past predictions matched the actual outcomes. You can then estimate the correlation based on historical precedent. In the absence of such information, assessments of predictability become more subjective. You may, for instance, be able to arrive at an estimate of predictability based on how the situation at hand compares with other forecasting situations. To return to the movie example, say that you are fairly confident that your ability to predict the sales of films exceeds the ability of sportscasters to predict point spreads in football games but is not as good as the ability of weather forecasters to predict temperatures two days out. Through a diligent statistical analysis, you could construct a rough scale of predictability based on computed correlations between predictions and outcomes for football scores and temperatures. You can then estimate where your ability to predict film scores lies on this scale. When the calculations are complex, it may help to bring in a skilled statistician.


  5. Correct the intuitive estimate. Due to bias, the intuitive estimate made in Step 3 will likely be optimistic—deviating too far from the average outcome of the reference class. In this final step, you adjust the estimate toward the average based on your analysis of predictability in Step 4. The less reliable the prediction, the more the estimate needs to be regressed toward the mean. Suppose that your intuitive prediction of a film’s sales is $95 million and that, on average, films in the reference class do $40 million worth of business. Suppose further that you have estimated the correlation coefficient to be 0.6. The regressed estimate of ticket sales would be:

$95M + [0.6 ($40M–$95M)] = $62M

As you see, the adjustment for optimism will often be substantial, particularly in highly uncertain situations where predictions are unreliable.