Dynamic Substitutions

Introduction

Substitutions are a way to make processes more dynamic. They eliminate the need to recreate a process due to changes in filenames and constants. Substitutions are variables that refer to character variables defined outside the process itself. The substitution variables have the following form:

• var. name - These are variables defined in the system portion of joboptions.json file. These are best used for simple substitutions that are more static.
• opt. class1.class2.name - These are variables defined in the process portion of the joboptions.json file and are available to be changed in the UI. These are best used when there is a context for the variable and the user may want to change for each run. For example, IFRS9->Reporting->AuditName would be represented as the variable opt.IFRS9.Reporting.AuditName.
• dyn. name - These are variables that are dynamic facts. These are best used when deriving the variable value can be done programmatically or depends on computations.

Substitutions can also be used in INPUT statements for the input file name for CSV files and for the MS SQL and PostGres SELECT= statements. Substitution characters can be used in OUTPUT statements for the output CSV file. Lastly, substitution characters can be used in quoted strings in SQL nodes.

Examples

Substitutions using var.* form

To define var.* variables use the vars tag in the system options in the joboptions.json file.

{
   "system": {
    "vars":{"input":"input.csv", "output":"modtest.csv", "id":"price", "where":"class1 != 'Oysterhive'"}
   }
}

Then use these variables in input and output statements:

name subs
in var.input -> input 

out input -> var.output

At run time, the input file used will be input.csv and the output file will be modtest.csv. These files can be changed without re-creating the subs process.

Substitution variables can also be used inside quoted strings but need to be surrounded by a ${}:

name subs
in var.input -> input where="${var.where}"

out input -> var.output where="${var.where} and lower(class2) = 'florida'"

Substitutions using opt.class1.class1.* form

To define opt.class1.class1.* variables use the classification tags in the process options in the joboptions.json file.

{
   "processoptions": {
    "category1": {
         "subcat1": {
            "name": "output.csv",
            "input": "input.csv"
         }
   },
   "pg": {
         "select": {
            "hist": "select django.model_type.Id, django.model_type.Desc from django.model_type",
            "out": "filtered.csv"
         },
         "subcatbar": {
            "name": "value,",
            "name2": "value2"
         }
      }
}

Then use these variables in input and output statements:

name subs
in opt.category1.subcat1.input -> input 

out input -> opt.category1.subcat1.name

At run time, the input file used will be input.csv and the output file will be output.csv. These files can be changed without re-creating the subs process.

Substitution variables can also be used inside quoted strings but need to be surrounded by a ${}:

name subs
in django.input -> input db=mssql select="${opt.pg.select.hist}"

Substitutions using dyn.* form

Dynamic, dyn.*, variables are defined using dynamic facts. In addition to where the other substitution variables can be used, dynamic variables can also be used within programming nodes to pass information between nodes as long as the node has the setdyn= option set. Note the following example

name subs
in dyn.input -> input getdyn=input

node initdyn(input)(output) setdyn=input,output,date

sql select * from input where '${dyn.date}'d > '2021-05-25'd into output; getdyn=date name=sqldyn2

out input -> dyn.output getdyn=input

The initdynU.go file would have the following code in the _init() function:

u.DynFactSet("input", "input.csv")
u.DynFactSet("output", "output.csv")
u.DynFactSet("date", "2022-05-25")

The input, sql, and output nodes all wait until the _init() function executes and defines the dynamic facts.