Introduction

Despite being ugly and uninformative, the term ‘big data’ has entered the language of science as well as that of the media. The meaning of ‘big’ data is explored in the introduction to this book. This chapter focuses on administrative data, meaning data related to services such as health and education, which are attached to individual people and are generated through the administration of services in the public or private sector. We are especially concerned with the linking together of such data files.

Requirements for sharing of administrative data

The legal requirements governing sharing of administrative data depend on whether the data contains information that can be used to identify individuals, known as personal data. The Data Protection Act (1998) allows government and other organisations to hold and use personal data without explicit consent if they have a legitimate interest in doing so (Data Protection Act, 1998). This can include the business of the organisation, evaluation of a service, or research. Similar principles are implemented under the European Union General Data Protection Regulation (GDPR), which came into force in May 2018 (EUGDPR, 2016), and is interpreted in the UK by the Information Commissioner's Office (ICO, 2017). Data will need to be processed fairly, with the public informed through a privacy notice that enables a limited right to opt out, and in many cases requires them to formally opt in to specific uses of their personal data. Personal data can directly identify an individual and needs to be handled in line with reasonable expectations under the common law duty of confidentiality. Under the Data Protection Act 1998, these requirements do not apply if individual data does not contain personal identifiers, for example, if personal identifiers have been encrypted or ‘hashed’ so that the data user could not deduce an individual's identity directly, although other techniques may be employed by someone wishing to ‘attack’ such data, as discussed below.

Beyond these generic legal requirements, researchers and other users of administrative data must satisfy requirements stipulated by the data providers in order to establish a data sharing agreement, although this can be costly (Gilbert et al, 2015). Reports by the Law Commission and the Administrative Data Taskforce highlighted the often slow, riskaverse and narrow interpretations of regulations, which typically limit data sharing (Law Commission, 2014; Administrative Data Taskforce, 2012).