Skip to content

Frameworks

Introduction

A framework is a set of instructions for how models should interact with each other. Examples of frameworks are a competing hazard scheme, a transition matrix system, just a single model, or no model at all.

Frameworks can be used for:

  1. Managing and changing models without changing the processes that employ them.
  2. Easily applying overlays. In most cases, overlays have to be documented, approved, and tracked. Also, overlays can require specific logic in their application.
  3. Compiling models on the fly. Since models are not connected to processes until run time, frameworks can ensure models are available and ready when needed.
  4. Providing logic between models, such as
    1. Defining model interdependency
    2. Passing parameters and static variables to and between models
    3. Creating risk factor transformations

Framework Attributes

A framework instance is made up of the following attributes:

name
name of the framework
description
description of the framework
type
this identifies what the framework is going to be used for. Common types include "PD", "LGD", and "STAGING"
code
the instructions on how to execute the models.
filter (optional)
this associates a framework and its encompassed models to a particular subset of a portfolio. For example, a transition matrix framework could be associated with 30-year mortgages.
language
the programming language used in the framework code. The frameworks can be written in Golang, Python, or SAS.
models
a list of models that are used in the framework (attributes detailed below)

Structured Models in Frameworks

Structured models (regression models defined declaratively through the UI) can be used within frameworks just like freeform models. The framework handles model selection via filters and determines which model to run for each observation. The structured definition determines how the selected model computes its output — the framework does not need to change based on whether a model is freeform or structured.

See Structured Models for details on creating structured model definitions.

Model Attributes

A model instance is made up of the following attributes:

name
name of the model
description
description of the model
code
the model code to execute
filter (optional)
filter to associate the model with a particular subset of a portfolio beyond the framework filter
coefficients (optional)
the parameters can be separate from the model or baked in to the model code
metadata (optional)
whether the model is a constant, "From" and "To" indicators for transition matrix models, and a space for user-specified model information.

Frameworks and Models in Playpens

Model development is best done within a playpen. To organize this work, create these three directories in the input directory of the playpen:

  • fwinstances - This is where framework instances live. The subdirectories under this playpen correspond to the framework types.

  • frameworks - this is where the framework code lives.

  • models - this is where the model code lives.

Framework Instances

Framework instances are defined using a JSON format.

{
    "name": "framework instance name",
    "desc": "description",
    "filter": "Optional SQL like filter",
    "framework": "framework template file name",
    "frameworktype": "framework type",
    "lang": "go | py | sas",
    "models": [
        {
            "filter": "Optional SQL like filter",
            "model": "model template file name",
            "modeltype": "model type",
            "name": "model instance name 1"
        },
        {
            "From": "d30",
            "To": "d60",
            "filter": "Optional SQL like filter",
            "isConstant": true,
            "modeltype": "model type",
            "name": "model instance name 2",
            "value": 5e-05,
            "valueName": "pd"
        },
        {
            "model": "model template file name",
            "modeltype": "model type",
            "name": "model instance name 3",
            "parmFile": "parms.csv",
            "parmValues": {
                "intercept": 0,
                "x1": -8345.2,
                "x2": 0.563
            }
        }
    ]
}

where the framework tags are defined as:

Tag Description
name A unique name for this framework instance.
desc Optional description for the framework instance.
frameworktype Should match the name of the directory the framework instance is placed in. Used for when the instance is uploaded to mid-tier.
filter Optional SQL like filter, used to determine when to apply the framework. If no filter is provided, the framework matches all instruments.
lang The language the framework and model code is written using.

and the model tag are defined as:

Tag Description
name A unique name for this model instance.
modeltype Identifies the type of model. Used when the model is uploaded to mid-tier and for categorizing models.
model The name of the model template file in the models directory. This isn't required for constant models
filter Optional SQL like filter, used to determine when to apply the model.
isConstant Optional. Specifies whether the model is a constant (true or false).
value Required only for constant models. This is the value of the constant.
valueName Required only for constant models. This is the name of the variable the constant is assigned to.
from Optionally describes the row of the transition matrix the model represents.
to Optionally describes the column of the transition matrix the model represents.
parmValues Optional name-value list of parameter values.
parmFile Optional CSV file name for the names and values of the parameters. This file is in the input/fwinstances/\<frameworkType\> directory. The format of the file is two comma separated rows. The first row is the parameter names and the second row is their corresponding values.
modelMeta Optional JSON describing user defined model information.

SQL Framework Functions

The filters for models and frameworks are expected to be in a SQL-like syntax. Specifically, the filters are evaluated using the format described in the Evaluating Run Time Expressions in SQL section. The filter should be a boolean expression.

  • Comparison operators: > >= < <= == != =~ !~
  • Logical operators: || && !
  • Date constants (single quotes, using any permutation of RFC3339, ISO8601, ruby date, or Unix date; date parsing is automatically tried with any string constant)
  • Boolean constants: true false

These filters are used by the three functions to create the IDs needed by the framework and model code. This allows the framework and model code to use filters that are agnostic to language used by the framework and model code. The getframeworklang() function allows users to change model languages without changing the process. The process would need to have a node of each language.

The filters should be evaluated using the SQL functions:

  • getframeworkID(<framework type>) - Returns the name of the matching framework for the current observation
  • getmodelID(<framework type>) - Returns the name of the matching model for the current observation
  • getframeworklang(<framework type>) - Returns the language used for the matching framework for the current observation

Here is an example node creating the necessary IDs:

sql select *,
           getframeworkID("PD") as frameworkid,
           getmodelID("PD") as modelid,
           getframeworklang("PD") as language
       from instrument into enrichedinstrument;

Using multiple Framework Instances in a single run

A process can employ multiple framework instances in a single run. For example, consider a situation where a firm's portfolio consists of two types of instruments: mortgages and auto loans. To calculate the firm's exposure to risk, we can create a single process to employ two framework instances--one for each instrument type. Note that although a process may use multiple framework instances, a single framework instance can have only one framework associated to it. The way the process associates records with a particular framework instance is through the use of the framework filter.

Below is an example of two framework instances that are used by the same process. Note that both framework instances are of the same type (LGD). Using multiple framework instances in a single process requires that all framework instances be of the same type.

{
    "name":"Mortgages",
    "desc":"15 and 30 yerar mortgages",
    "frameworktype": "LGD",
    "framework": "mortLGD.py",
    "filter" : "instrumenttype in ('mortgage15', 'mortgage30')",
    "lang": "py",
    "models": [
        {"name" : "mort30",
          "modeltype" :"LGD_Model",
          "model" :"mort30LGD.py",
          "filter": "instrumenttype == 'mortgage30'"
        },
        {
        "name" : "mort15",
        "modeltype" :"LGD_Model",
        "model" :"mort15LGD.py",
        "filter": "instrumenttype == 'mortgage15'"
        }
        ]
  }

{
    "name":"Auto",
    "desc":"Auto loans",
    "frameworktype": "LGD",
    "framework": "autoLGD.py",
    "filter" : "instrumenttype == 'auto'",
    "lang": "py",
    "models": [
        {"name" : "auto",
          "modeltype" :"LGD_Model",
          "model" :"autoLGD.py"
        }
        ]
  }

In the above example, the process sends records of type 'mortgage15' or 'mortgage30' through the mortLGD.py framework, and records of type 'auto' through the autoLGD.py framework. Then with the help of model level filters, mortgage instruments are further subdivided by mortgage duration and passed through two models mort15LGD.py and mort30LGD.py.

Risk Factor Transformations

Model regressors are often transformations of risk factors in a scenario. These risk factor transformations must be created in order to evaluate a model. Examples of transformations are lags of a risk factor and the log of a risk factor. Other transformations can be much more complex. The computation of the transformations is usually done once before the models are computed.

Models can list the risk factor transformations they require. This is done in the models.json file in the input/models directory. The format of this file is

{
   "modelName1" : {"rftransVars":["variable1", "variable1", "set3"]},
   "modelName2" : {"rftransVars":["variable3", "variable5", "set3"]},
   ...
}

This risk factor transformation metadata allows the framework system to provide a list of unique, required risk factor transformations for the selected set of framework instances. A single risk factor transformation can create multiple variables (set). The risk factor transformation list a global list for all the models in the specified language. The intention is for this list to be used to perform only the necessary transformations, one time, for all models. The actual code to perform the transformations should be provided as an import. For Golang and Python, the format should be a function for each risk factor variable or set so that they only the necessary transformations are performed.

Golang Framework Example

For Golang, frameworks have at least four components:

  • framework instance - Placed in the fwinstances/<frameworkType> directory.
  • framework code - Placed in the framework directory.
  • model code - Placed in the model directory. Note, constant models don't require a code attribute.
  • node code - This is code that calls u.ReadFramework("<frameworkType>") to get the framework - usually in the init section. And worker code to call the appropriate framework based on a filter.

Here is an example of a framework instance:

{
    "name": "Mortgages",
    "desc": "30 year conforming mortgages",
    "filter": "region == 'Southeast'",
    "framework": "hazard.go",
    "frameworktype": "PD",
    "lang": "go",
    "models": [
        {
            "model": "model.go",
            "modeltype": "PD_Model",
            "name": "ModelSeg2"
        }
    ]
}

Here is the 'hazard.go' framework referred to in the instance:

package main

import (
 "frg.com/streamr/sdk"
 log "github.com/sirupsen/logrus"
)

var Frame *sdk.FrameworkS // set by VOR Stream

func Framework(input map[string]interface{}, args ...interface{}) (ans map[string]float64, err error) {

 var models []sdk.ModelS
 models = Frame.Models

 // loop over models to see which matches
 found := false
 modelName := ""
 for _, mod := range models {
  if !mod.Filter || input["Modelid"] == mod.Name {
   if found { // found another model that matched
    log.Warnf("In framework %s, multiple models matched, %s and %s", Frame.Name, modelName, mod.Name)
    continue
   }
   found = true
   modelName = mod.Name

   // execute model
   ans, err = mod.Model(input, args...)
  }

 }
 if !found {
  log.Warnf("In framework %s, no models matched: %v", Frame.Name, input)
 }

 return ans, err
}

Here is the 'model.go' referred to in the model instance:

package main

func Model(input map[string]interface{}, args ...interface{}) (map[string]float64, error) {

 output := make(map[string]float64)

 if input["Segment"] == 1. {
  output["pd"] = 0.005
 } else {
  output["pd"] = 0.007
 }

 return output, nil
}

If parameters are provided external to the model, then the following code can be used to set the parameters for the current run:

parms := map[string]float64{ $parms$ }

And finally, here is the node code that runs the framework:

...

type User struct {
 hh        *frgutil.Handle                              // required - don't remove
 threadNum int                                          // required - don't remove
 options   map[string]map[string]map[string]interface{} // required - don't remove
 mem       *frgutil.FuncMem                             // required - don't remove
 Pds       *Pds.Pds
 frame     []*sdk.FrameworkS
}

// this function is called once for each thread
// when the node starts
func (u *User) _init() {

 // Get the code for framework
 var err error
 u.frame, err = u.ReadFramework("PD")

 if err != nil {
  log.Errorln(err)
  frgutil.EndJob(u.hh)
  return
 }
}

// This function is called for each observation on the input queue
// input is a struct with fields the names of the fields in the expected table.
// Capital first letter, lowercase the remaining
func (u *User) worker(input *Enrichedinstrument.Enrichedinstrument) {

 if input.Lang != "go" {
  return
 }

 // convert input struct to a map
 inMap, _ := frgutil.ToMap(input, "")

 // determine which framework instance to call
 found := false
 frameName := ""
 for _, frame := range u.frame {
  if !frame.Filter || frame.Name == input.Frameworkid {
   if found { // found another framework that matched
    log.Warnf("Multiple frameworks matched, %s and %s", frame.Name, frameName)
    continue
   }
   found = true
   frameName = frame.Name

   // Run the framework and models
   ans, err := frame.Framework(inMap)
   if err != nil {
    log.Println(ans, err)
    u.Pds.Pd = math.NaN()
   } else {
    u.Pds.Pd = ans["pd"]
   }
  }
 }

 Pds.Post(u.Pds)
}

// this function is called once for each thread
// when the node ends
func (u *User) term() {

}

Adding more Functions to a Model

By default, the framework system is only expecting a function called Model to exist in the model code and a function called Framework to exist in the framework code. Other functions can be added and called. For example, the model might be divided into a static portion and a dynamic portion. Or there could be by convention an overlay function. To access these functions, use the plugin object returned in the framework and model objects:

import "plugin"

 var err error
 u.frame, err = u.ReadFramework("PD")
 if err != nil {
  log.Errorln(err)
  frgutil.EndJob(u.hh)
  return
 }
 var over plugin.Symbol
 over, err = frm.P.Lookup("Overlay")
 if err != nil {
  log.Errorf("framework %s doesn't contain a Framework function Overlay: %s", frm.Name, err)
  return
 }
 u.over = over.(func(map[string]interface{}, ...interface{}) (map[string]float64, error))

The last statement is just a suggested function signature.

Python Framework Example

For Python, frameworks have at least four components:

  • framework instance - Placed in the fwinstances/<frameworkType> directory.
  • framework code - Placed in the frameworks directory.
  • model code - Placed in the models directory. Note, constant models don't need code.
  • node code - This is code that calls self.framework = framework.ReadFramework("<frameworkType>",<runID>) to get the framework - usually in the __init function. Then use worker code to call the appropriate framework based on a filter.

Here is an example of a Python framework instance:

{
    "name": "MortPy",
    "desc": "30 year conforming mortgages",
    "filter": "region == 'Southwest' || region == 'West'",
    "framework": "transmat.py",
    "frameworktype": "PD",
    "lang": "py",
    "models": [
        {
            "model": "logistic.py",
            "modeltype": "PD_Model",
            "name": "ModelName1",
            "parmFile": ""
        }
    ]
}

Here is the transmat.py framework referred to in the framework instance:

import logging

class Framework:
    output = dict()
    models = list()

    def __init__(self, models : list()):
        self.models = models

    def framework(self, input : dict(), *argsv ) :

        foundModel = False
        for m in self.models:
            if not m.filter or input["Modelid"] == m.name:
                if foundModel :
                    logging.warning("In default framework, multiple models matched, %s and %s", modelName, m.name)
                foundModel = True
                modelName = m.name
                m.Model.model( input, *argsv )
                self.output = m.output

        if not foundModel:
            logging.warning("In default framework, no models matched")

        return

Here is the logistic.py model referred to in the model instance:

import logging

class Model:
    var1 = 0.0
    var2 = 0.0
    output = dict()

    def __init__(self, output: dict()):
        self.output = output

    def model(self, input : dict(), *argsv ) :
        logging.info("In model " + str(input))

        self.output["Pd"] = 0.00005

        return

If parameters are provided external to the model, then the following code can be used to set the parameters for the current run:

parms = dict( {$parms$} )

And finally, the Python code to call the framework from a node is:

from sdk import framework
import logging

class pymod:
    options = dict()

    def __init__(self, handle, pds):
        self.Pds = pds

        self.hh = handle
        if 'processoptions' in handle.options['JobOptions']:
            self.options = handle.options['JobOptions']['processoptions']
        else:
            self.options = None
        # perform one time initializations if necessary

        self.framework = framework.ReadFramework("PD", 0)

    # worker is called for each observation sent on the input queue
    # input is a named tuple
    def worker(self, threadNum, input):

        if input.Lang != "py":
            return

        foundFrame = False
        for frame in self.framework:
            if not frame.filter or input["Frameworkid"] == frame.name:
                if foundFrame:
                    logging.warning(
                        "Multiple frameworks matched, %s and %s", frameName, frame.name)
                foundFrame = True
                frameName = frame.name
                frame.framework.framework(input.__dict__)
                self.Pds.Pd = frame.framework.output["Pd"]

        # You don't have to post to and/or all or output queues
        self.Pds.Post()

   def term(self):
      return

SAS Framework Example

For SAS, frameworks have at least five components:

  • framework instance - Placed in the fwinstances/<frameworkType> directory.
  • framework code - Placed in the frameworks directory.
  • model code - Placed in the models directory. Note, constant models don't need code.
  • node code - This is code in the .strm file that sets up to call SAS code.
  • SAS code - This is code that calls %<frameworkType>_FRAMEWORK; to run the framework--usually in a data step.

Here is an example of a SAS framework instance:

{
    "name": "MortSAS",
    "desc": "30 year conforming mortgages",
    "filter": "region == 'Northeast' || region == 'Midwest'",
    "framework": "hazard.sas",
    "frameworktype": "PD",
    "lang": "sas",
    "models": [
        {
            "filter": "segment == 1",
            "model": "logistic.sas",
            "modeltype": "PD_Model",
            "name": "ModelParmsSAS",
            "parmFile": "parms1.csv"
        }
    ]
}

Here is the hazard.sas framework referred to in the framework instance:

%macro $frameworkName$_FRAMEWORK( modelList, filterList);

  /* loop over the modelList and filter list and create a SAS program */
    %local i filter model;
    %do i=1 %to %sysfunc(countw(&filterList));
        %let filter = %unquote(%scan(&filterList, &i));
        %let model = %unquote(%scan(&modelList, &i));
        %if &i = 1 %then %do;
            if &&&filter then do;
                %&model;
            end;
        %end;
        %else %do;
            else if &filter then do;
                %&model ;
            end;
        %end;

    %end;

    else  do;
        put "ERROR: no data matched any model ";
        put (_all_)( = );
    end;

%mend $frameworkName$_FRAMEWORK;

The $frameworkName$ tag is substituted at runtime to allow you to reuse the framework for multiple instances.

Here is logistic.sas referred to in the model instance:

%macro $modelName$_MODEL;
$parms$
pd = &intercept + &slope * fixed_spread_rate;
%mend;

The $modelName$ tag is substituted at runtime to allow you to reuse the model for multiple instances. The $parms$ tag is used to substitute the current set of parameter value and are specified as macro variables. Note that both tags are case-sensitive. The $parms$ tag is optional.

Here is the code for creating the SAS node that uses a framework:

sas enrichedinstrument -> (ds=enrichedinstrument)
        (ds=pdout) -> pds
        sasFile= "pds.sas"
        framework = PD
        name=pds

Note, unlike Golang and Python, A SAS node can only use a single framework type.

Finally, this is the SAS code called in the SAS node, pds.sas:

data pdout; set enrichedinstrument;
where lang="sas";

%pd_FRAMEWORK;

run;