Policy-makers are concerned with ensemble judgements. They are obliged to consider all potential sources of uncertainty, including those not examined formally by experts. Some aspects of a decision may be concrete, but almost always, other aspects will be political or intangible. Some may affect policy-makers personally. Decision-makers may be more interested in robust strategies that avoid the worst outcomes, than in trying to maximise expected benefits.

In this book, I treat only the relatively simple problems of how to ask experts about well-defined, unique facts (numbers, quantities, rates, outcomes of events). The facts may exist in the present or they may be realised in the future. Even in these relatively simple circumstances, we need to combat the pervasive weaknesses and unacknowledged myopia that is deeply buried in expert judgement. There are tractable ways to achieve this.

ADVICE FOR DECISION-MAKERS

To improve the judgements we elicit from experts, I recommend that decision-makers and analysts adopt the following rules:

1. Be clear about what you want from experts: estimates of simple facts, predictions of the outcomes of events, or advice on a best course of action.

2. Be clear about the domains of expertise that will help, and choose people whose skills, training or verified experience (where it exists) are squarely in those domains. This may include people who, on paper, don't fit the typical ‘expert’ mould. Where the opportunity exists, offer appropriate training.

3. Choose as many experts as possible; don't be concerned about their age, number of publications, peer status, technical qualifications or apparent impartiality.

4. If the matter at hand is politically sensitive or socially or emotionally charged, ensure that the experts have diverse relevant opinions or positions. Work to diversify the culture, gender, context and perspectives of the participants. Try to include people who are less self-assured and assertive, and who integrate information from diverse sources.

5. Compose questions to avoid arbitrary linguistic misunderstandings and psychological trip wires such as framing, anchoring, availability bias and so on.

6. Use structured question formats to counter tendencies towards overconfidence, and oblige participants to consider counter-factuals and alternative theories.

7. Use structured, facilitated group interactions to counter dominance effects, anchoring and other factors that lead to groupthink.