Overview

In this chapter we study data flow programming, including flow composition and flow manipulation. Flow composition focuses on techniques used for creating the topology associated with the flow graph for an application, while flow manipulation covers the use of operators to perform transformations on these flows.

We start by introducing different forms of flow composition in Section 4.2. In Section 4.3, we discuss flow manipulation and the properties of stream processing operators, including their internal state, selectivity, and arity, as well as parameterization, output assignments and functions, punctuations, and windowing configuration used to perform such manipulation.

Flow composition

Flow composition patterns fall into three main categories: static, dynamic, and nested composition.

Static composition is used to create the parts of the application topology that are known at development time. For instance, consider an application that has a source operator consuming data from a specific Internet source, for example, a Twitter [1] feed. Let's assume that the stream generated by the source is to be processed by a specific operator that analyzes this data, for example, a sentiment analysis operator [2] that probes the messages for positive or negative tone. In this case, the connection between the source operator and the analysis operator is known at development time and thus can be explicitly created by connecting the output port of the source operator to the input port of the analysis operator.