Overview of Nodes

All nodes in a VOR Stream process run independent of each other and are connected only through a queue. The queue passes protocol buffers for each observation.

There are currently 4 different kinds of nodes:

1. Input / Output (IO) Nodes
   • Read or write to disk
   • Are end points or sync points in the stream
   • Take data from CSV, Hadoop, database, or AWS S3
2. SQL Nodes
   • Process queue data using SQL syntax
   • Use a SQL SELECT statement to perform joins, filters, and GROUP BY clauses
   • Automatically generate a Golang node
3. SAS Nodes
   • Execute arbitrary SAS code
   • Partition and multi-thread the execution on an input and synchronize the input and output queues
   • Automatically generate a Golang node
4. Computational Nodes
   • Are generic, user-programmable nodes
   • Are created by the user from a custom template
   • Execute Python or Golang code

Node CLI

Nodes are created by using the *.strm file, as explained in the Process section.