Introduction

For militarily advanced countries, conflict is becoming increasingly information-centric. As a tool for supporting strategic and tactical decisions, designing weapons and increasingly as a weapon (or target) in itself, data and the infrastructure for gathering and processing it is critical to the success of military undertakings. Complex operational environments, large battlespaces and adversaries that follow unconventional strategies place heavy demands on commanders who must make, and justify, decisions which depend on huge numbers of variables. Accordingly, armed forces gather and process vast amounts of data from many sources, both military and civilian, to support those decisions. The task of gathering, storing and analysing that data increasingly falls within the scope of ‘big data’ projects.

Armed forces are aware of the stakes, as suggested by the vast resources allocated by American, Chinese and other armed forces to the development of big data technologies during the past ten years. Those technologies offer unprecedentedly effective tools for systematically retrieving information and establishing correlations within extremely large, heterogeneous and unstructured data sets. The unstructured data that are actively harvested and interpreted by mining technologies come not only from traditional data storage devices, but from virtually any online system. The advantage offered by the capillary presence and interoperability of these technologies enables an innovative concept of intelligently selective collection and analysis: much better than its precursors, big data technology identifies context-sensitive information, recognises and compares the occurrence of relevant patterns across many formats and media, reveals the presence of unobvious correlations among large numbers of variables, and generates predictions about their most likely effects.

Big data has great potential to support security through intelligence collection, in particular (Thuraisingham et al. 2017). Surveillance and reconnaissance operations benefit from big data's capability to scan very large environments, detecting specific individuals, facilities, weapons or other threats (Andrejevic and Gates 2014): pattern recognition routines are used to automate the analysis of footage recorded by UAVs patrolling seas to spot objects of interest (Costlow 2014). The Defense Advanced Research Projects Agency (DARPA) developed two data-mining programs to anticipate unapparent threats during the US -lead mission in Afghanistan.