One of the fascinating features of the history of science is how often new ways of seeing preceded new insights. The achromatic-lens microscope in the early 19th century paved the way for germ theory, and X-ray crystallography in the early 20th century played a vital role in the later discovery of the structure of DNA. In the same way flows of data – for example, about how people move around a city, or how blood cells change – can prompt new insights.

But how important is measurement to social change? Many people are attracted to metrics and indices of all kinds. But, as my colleague Mark Moore used to warn, ‘do you really think the leaders of the Civil Rights Movement were counting the placards or measuring the decibels of their cries for human rights?’ In social change, as in our own daily lives, measurement often feels inappropriate for the things that matter most.

This chapter examines some of the history of social observation as a tool for public policy, social innovation and social change, and I suggest where it might lead in the future. Without some means of measurement, it can be hard to know if a social innovation is good. It may feel good to the beneficiaries – but still be less effective than an alternative. Or it may work well for one group but not another. And, even if it may not be appropriate to measure the passions of movements, once these ideas become part of the mainstream, and are transformed into the cool logic of laws, regulations and programmes, measurements do start to matter a lot, as the Civil Rights Movement discovered.

A short history of measurement

For centuries, governments have sought to map and measure social phenomena in order to better exercise control over them. In the modern era these attempts can be traced back to figures like Sir William Petty in England and the cameralists in Prussia.