Processing (also known as computational) graph is a sequence of operations which transforms input data in a desired way.

The most important advantage is the ability to separate the description of operations from their execution. It allows to describe the logic once and then launch it in different environments:

• On a user machine / remote machine.
• Using single thread / multiple threads.
• Using multiple machines.

Record is a group of key-value pairs. It represents a single unit of data - info about one person, one document, etc. In our project it is implemented by Record class.

A sequence of records is called Stream. Processing graph can accept stream of records as input, transform records one by one and then output them as a new stream.

Streams can represent one of the two different concepts:

• Infinite sequence of records, which are generated in real time. For example, clicks on items in the online store. You can never hope to see "all" such records, because they continue to arrive again and again.
• Finite sequence of records. For example, contents of all documents currently present in a library. We call such finite sequence Dataset.

In order to mark the end of a dataset, we can insert a so called terminal record.

Operators represent basic actions, which transform records in some way. They can be chained together in order to implement a graph with a desired logic.

Operator accepts records one at a time. Each time it can output either zero, one or more records (by passing them to collector object). Operator can store any info about previously seen rows inside itself.

Operators can be roughly divided into two types: mappers and reducers.

Mapper operator accepts a single record and outputs one or more new records. All mappers must implement Mapper interface.

One mapper is already implemented: IdentityMapper.

Reducer operator accepts sequence of records, which split into groups by key. All reducers must implement Reducer interface.

One reducer is already implemented: CountReducer, use it as example.

Each operator is contained inside a ProcNode object. Node handles all communication with the outside world:

• It accepts records from the previous nodes.
• Preprocesses records if needed.
• Passes records to operator.
• Collects its outputs.
• Passes these outputs to the next nodes.

Each node has several input gates, which allows it to accept different streams of records (this is used in join operations, for example).

Nodes can be divided into four types:

• MapperNode skips terminal records. When a terminal record arrives, it is simply redirected to the next node without any processing.
• ReducerNode does not skip terminal records. It accepts sorted sequence of records and groups them by specified keys.
• SorterNode does not uses operator. It accepts records and stores them inside itself until it receives a terminal record. Then it sorts all stored records and outputs them.
• JoinerNode does not uses operator. It accepts inputs from two gates (0 and 1) and outputs joined rows. You will have to implement it.

ProcGraph object represents the actual processing graph - a group of interconnected nodes. It exposes its input nodes and output nodes.

Since graph is a non-linear data structure, it is difficult to create a convenient API for assembling a new graph. In this project we solve this problem by using a GraphPartBuilder. Each builder instance tracks the first and the last node of some linear part of the graph. You can do three basic actions with a builder:

• Append a new node to the current linear part (by using map, reduce and so on).

• Merge with another linear part, which is represented by another builder (by using join).

• Split a new linear part, creating a new builder (by using branch).

  For the example of using builder see code in WordCount.

We have a single-module gradle project. It includes a lot of things:

• All basic classes in the root package.
• Different node types (one not implemented), in package nodes.
• Different operators (most of them not implemented), in package operators.
• Different graphs (only one implemented), in package graphs.
• Utilities for visualizing graphs in package ui.

Our project includes tools for visualizing processing graphs you have created. It uses a well-known graph library GraphStream. All code is located in ui package.

How to use it: simply pass your ProcGraph object to the method GraphVisualizer.visualizeGraph. It will open a Swing UI window which contains your graph. You may do it inside class VisualizationDemo (remove @Disabled annotation first). You will see something like this:

All nodes are colour coded:

• Inputs: green
• Outputs: red
• Mappers: white
• Reducer: blue
• Sorters: violet
• Joiners: orange

Your task can be divided into two parts:

• Develop a "toolbox" of operators and nodes.
• Use it to assemble processing graphs for solving actual practical problems.

You must implement the following operators (and pass tests for them):

• Mappers
  • AddColumnMapper
  • FilterMapper
  • LowerCaseMapper
  • RetainColumnsMapper
  • TokenizerMapper
• Reducers
  • FirstNReducer
  • SumReducer
  • WordFrequencyReducer
• Nodes
  • JoinerNode

• Task: Count how much words each author has written
• Input: stream of records like:

  {"Id": 1, "Author": "Caesar", "Text": "Et tu, Brute"}

• Output: stream of records like:

  {"Author": "Caesar", "TotalWords": 420}

  Records should be sorted by Author column, ascending.

Graph for this task is already implemented in a class WordCount, but some operators are not implemented yet. You must implement these operators (as part of task "Operators toolbox") and pass the tests in WordCountTest.