21  Expertise

It is a simple house fire in a one-story house in a residential neighborhood. The fire is in the back, in the kitchen area. The lieutenant leads his hose crew into the building, to the back, to spray water on the fire, but the fire just roars back at them.

“Odd,” he thinks. The water should have more of an impact. They try dousing it again, and get the same results. They retreat a few steps to regroup.

Then the lieutenant starts to feel as if something is not right. He doesn’t have any clues; he just doesn’t feel right about being in that house, so he orders his men out of the building—a perfectly standard building with nothing out of the ordinary.

As soon as his men leave the building, the floor where they had been standing collapses. Had they still been inside, they would have plunged into the fire below.

Gary Klein (1998), Sources of Power

Much of the content in this course is informed by the heuristics and biases approach to decision making. Pioneered by Kahneman and Tversky, the heuristics and biases approach often takes a sceptical lens to experts and expertise. Kahneman was inspired by the work of Paul Meehl, whose work on clinical versus statistical prediction we have just discussed. Kahnmen and Tversky’s work has generally shown that the faulty statistical intuitions they have uncovered in their work survives training and experience. These faults are exhibited by experts.

However, in some domains, such as chess and firefighting, there is considerable evidence that experts have a pattern identification ability that enables them to make good decisions. Naturalistic decision making is one of a number of alternative approaches that examines on the success of expert intuition. Its practitioners seek to understand how experts use intuition to make their judgements, even where that involves tacit knowledge that is difficult for the researchers to articulate.

Expertise is the development of deep domain knowledge that enables the expert to apply more effective strategies, form rich mental models, and have highly organised domain knowledge. Under certain conditions, this can lead to more effective decision making.

kahneman_conditions_2009 teased out how the heuristics and biases approach and naturalistic decision making could be reconciled. Under what conditions should we build and trust expertise? And where are experts likely to be poor decision makers?

21.1 Conditions for developing expertise

The first condition for the development of expertise is that the environment must provide valid cues about the situation. Experts draw on cues in the environment to make their judgements. Those cues provide access to information stored in memory, which then provides the answer.

A person making a judgement can only use and rely on cues in an environment of sufficient regularity. The relationship between the cues and outcomes or actions needs to be stable. Kahneman and Klein describe such an environment as having “validity”. The stock market is an environment of low validity. To the extent there is public information (cues) as to future price movements, these should be already reflected in the price (at least according to the efficient markets hypothesis). The absence of evidence of expertise in stock picking is indicative of this low validity. In contrast, chess and poker are high validity environments.

A high-validity environment can be uncertain in that the cues only provide probabilistic information, but can still be learned. Poker is an example of an uncertain high validity environment.

The ability to learn is the second condition for the development of expertise. The ability to learn requires both an adequate opportunity for learning the environment (prolonged practice) and feedback that is both rapid and unequivocal. For many professions, the feedback received is delayed, sparse and ambiguous, making it hard to learn any cues. Even if there were cues to identify an attractive merger opportunity, how many opportunities to learn with clear feedback is a manager likely to receive in their career?

Most of those domains where Paul Meehl and others’ simple algorithms outperformed “experts” lacked one of these elements. Recruiters and admissions officers typically don’t get to see how their choices perform. Judges aren’t provided with rapid feedback about the result of their parole decisions.

21.2 Discussion

Is corporate decision making an environment of high validity?

What of management decisions? MBA students spend much of their courses discussing business case studies. Can the case study approach lead to the development of expertise?

What of employee decisions? Which employees make their judgements in an environment of high validity?

Even if the conditions for expertise exist, it may not develop. In an experiment by Ball et al. (1991), MBA students were given 20 consecutive trials of the acquiring a company problem that you saw in Module 1 [Add link to 2.9]. For each trial, they made an offer, a random company value was generated, and they got to see the result of their offer. They also got to see the effect of the trial on their asset balance (generally a decline).

Only 5 of the 72 students learned that the optimal answer is $0 over the course of the trials. Despite the environment meeting the conditions for the development of expertise, they did not gain any.