Increasing transparency – by which we mean reducing the monopolistic control over information by government and private organisations – is becoming a central concern for the information age. The combination of big data and increasingly intelligent computer algorithms is creating a world in which the rules around the control of information will be as important to a fair society as freedom of speech is to democracy. However, as more and more data is generated with all the information technologies at our collective disposal, we risk having less and less transparency.

The immediate impetus for writing this book has been an awareness that transparency policy is not keeping up with changes in the power of technology and data. Access to information legislation may have some relevance to understanding how decisions are made by bureaucrats and committees. It has less to offer when decisions are driven by big data, machine learning systems and artificial intelligence.

This book sets out a personal view regarding the most useful direction in which transparency policies could be developed. It comes from two entrepreneurs who have spent the past 15 years working in transparency and promoting it as a practical enabler in the improvement of public services.

We come from a background in the media, working both in the UK and abroad as correspondents for national newspapers and consumer advocacy groups. In 2000 we founded a transparency business in the UK which created a precedent. Dr Foster was set up as an organisation which worked with a university research department at Imperial College London. The research department was allowed access by the government to the identifiable data set covering all hospital treatment throughout the National Health Service in England under strict terms of confidentiality but with freedom to publish anonymised analysis. The only constraint was that it had to be in the public interest, a constraint overseen by an independent ethical oversight framework.

It was a unique and brave experiment. Over the 15 years that followed, Dr Foster published independent assessments of standards of care, supplied data monitoring systems to hospitals and put quality of care at the centre of the political debate about the NHS. The Dr Foster publications identified significant failings in healthcare that were leading to high excess death rates and poor standards of treatment for patients – information that, up until then, patients had been unaware of.