The experiences with experts and their frailties over the last 60 years have taught us an important lesson. People, including experts, have an overriding desire for a direct line to certainty. Because of this, we call on experts to make estimates outside the bounds of their expertise. The experts are willing to provide them. This chapter outlines some first steps towards shedding our naïve views of expertise, and embracing the philosophy that expert judgement should be treated with the same care and constructive scepticism that we afford to data.

Unreliable opinions can be uncovered, excluded or adjusted when they are tested against independent information. Strategies for doing so should be decided before the expert advice is received. If not, outcomes may be contaminated by the prejudices and context of the analyst or group that has control over the decision. Note that, when I refer to the ‘analyst’, I am referring to the person responsible for extracting and combining information from experts. They may be the facilitator and/or the decision-maker.

Post-hoc adjustment of expert estimates is one possibility for mitigating biases, for example, when we know that the biases are systematic because we have compared previous estimates against facts. But what else can be done to improve the quality of estimates?

There are several options. We can widen the set of experiences and skills of the pool of people we consult. We can test the knowledge of potential contributors and train experts to make good estimates. Finally, we can use procedures that minimise biases, and encourage honest opinions and critical examination of evidence. This chapter outlines some simple prescriptions for dealing with an individual expert to estimate a precise fact or predict the outcome of an event. The following chapter deals with group dynamics.

STRUCTURED QUESTIONS FOR QUANTITIES

My colleagues and I recommend a structured procedure for questioning experts about quantities that uses four questions, termed a four-point format (Figure 4.1). We use three questions, termed a three-point format, to elicit probabilities, which is outlined in the next section. In both cases, we elicit an interval to capture uncertainty.