It is important to reiterate a point from the first chapter. This book is NOT about practical skills honed by repetition – such as plumbing, bridge building or surgery – nor is it about judgements related to such skill-based tasks. Rather, it deals with technical estimates and predictions of unknown quantities or future events made by people whose standing is defined by their qualifications, experience and status among their peers.

When we think of the kind of expert we'd like to consult for a prediction or estimate, we often think of someone middle-aged (or certainly not too young), with excellent training and qualifications, and ample relevant experience. We imagine a thoughtful and attentive person who is held in high esteem by their peers. Often, the property of ‘being an expert’ is taken to be self-evident. It is someone whom everyone else agrees is an expert.

When governments and other institutions set up expert panels they look for many of the same attributes; they examine reputation, qualifications, track record, professional standing and experience. Similarly, courts and statutory bodies accept expert opinion because they believe experts have specialised knowledge not available to all. In their view, it is obtained through training and experience. It is proven by track records of efficient and effective application.

How should a manager or decision-maker decide which experts to consult? If we subscribe to the conventional definition of expertise, we just need a pool of qualified people. Literature reviews would be useful to define the expert pool because books and articles in scholarly journals reflect qualifications and experience. Frequently, experts suggest other experts who suggest others, and so on (termed ‘snowballing’). This approach assumes that experts are able to identify other similarly trained and experienced people who will provide reliable estimates of facts.

In general, the strategies above are effective for finding a skill-based specialist. When the task is learned by training and repetition, when the outcomes are tied unambiguously to the person who performed them, and when we can examine a record of performance, then reputation and track record are a useful guide to performance.

Unfortunately, when making difficult judgements in unpredictable environments, these strategies for identifying the best judge may not be helpful. In many situations, you would do just as well to draw names out of a hat.