VOR keeps getting better, and we are excited to announce the latest features, enhancements, and changes:
VOR Workbench
Playpen Configuration
You can now make playpen configuration changes directly in the VOR Workbench.
To do so, navigate to User Settings => Options => Playpen Config to access the JSON editor.
For information on the configurable options, see the Playpen Configuration documentation.
Golden Copy
VOR now supports having multiple golden copies for the same study.
In addition, it is possible to unmark a golden copy. Note that unmarking a golden copy requires you to have permission to make changes to a golden copy of a study. Talk to your administrator if you need this permission set up.
Waterways
The loading of the list of runs on the Waterways page has been improved.
VOR CLI
Show Process
The line command vor show process now includes filtering capabilities and a sort flag. Also, the output from the command has better readability.
Use the link above or vor show process –help for more details.
Show Runs
The new line command vor show runs gives you a list of all runs, including their status, approval and other details.
You can filter them by:
- execution start time
- state (pending, finished, or specific state)
- name
- name-like
- approval
Delete Runs
Use the new line command vor delete runs to delete runs that are no longer needed. This command can be filtered in the same way as vor show runs.
For example, to delete all completed runs that are not approved (golden copy flag not set), enter
vor delete run –finished –not-approved
The command deletes both run metadata from the database and run outputs on the file system. A future release will add support for deletion of run results from reporting databases.
Test
The command vor test has been modified to execute tests concurrently by default. Projects that leverage this functionality will run significantly faster.
SQL Nodes
VOR now has a new and improved version (v2) of SQL nodes that is more SQL-compliant.
The new version includes:
- The syntax for several SQL statements has been improved to closely match the SQL standard.
- Results for multiple joins are now consistent with the SQL standard.
- Support for set operations. The following operations are now supported:
- UNION
- UNION ALL
- INTERSECT
- EXCEPT
- Better error messages for invalid SQL queries. The error messages are now more descriptive and provide hints on how to fix the issue.
It is highly encouraged to migrate your SQL nodes to the new version since SQL v1 syntax will be deprecated in a future release. For more information on the improvements and how to migrate, see the Migration Guide.
Deployment
You can now set up Sentry for error tracking in the VOR Stream mid-tier by setting the super_sentry_dsn variable in your inventory file.
Release Details
✨ Features
- (ansible) Support deploying into externally-managed Vault
- (waterways) Allow multiple golden copies per study
- (sql) Add function to log formatted string
- (cli) Add cleanup of file system outputs to vor delete run
- (engine) Support nanoseconds for timestamps in CSV reads and writes
- (engine) Add optional Sentry configuration to VOR Stream Midtier
- (modelperf) Accept model performance equations with no categorical variables + fix rendering issues
- (cli) Add vor delete run command
- (cli) Add vor show run command to list run details
- (engine) Make a poison pill multiplier a configuration parameter
- (ui) Add the ability to unmark a run as golden copy
- (ui) Add button to edit playpen configuration from UI
- (sql) Add option to choose parser version
🐛 Fixes
- (sql) Add support to order by int and bool types
- (django) Allow deletion of model runs
- (sql) Enforce AS clause for result expressions in SQL queries
- (django) Methodology import fails with addtional user_email and user_id fields
- (engine) Panic in CPU time retrieval
- (sql) Implement precedence order for set operations
- (sql) Improve type comparison to be case insensitive
- (sql) Disallow non-aggregated columns that are not in the group by clause
- (engine) Correct scale on wait in queueLimit code
- (engine) Ctrl+C is hanging some nodes
- (engine) Expected count is zero for the first run
- (cli) Race condition when deleting runs with reporting runs
- (sql) Store join AST as pointers
- (sql) Pass function memory to expression environment
- (django) Allow ReportRun instances to be deleted when deleting StreamRunDim instances
- (sql) Avoid using literal values in hash comparison
- (python) Add error handling for python nodes
- (sql) Exclude alias table to input queue list )
- (engine) Do not print early termination warnings for nodes that do not execute
- (sql) Fix hash join input validation for multiple joins
- (sql) Fix hash join with multiple conditions
- (filters) Display filter metadata on tooltip for archived filters
- (django) Increase patch notes version length
- (sql) Fix distinct and dedup queries
- (sql) Allow select all using alias table
- Close python nodes gracefully
- (sql) Rename sql interfaces method signature for uniqueness
- (sas) Race condition with SASUSER when multiple threads are used
- (ui) Make id and user_playpen fields uneditable in playpen config JSON
- (engine) Use quoted string when setting input path environment variable
- (deps) Update SciChart
- (ui) Scenario viewer does not list scenarios
- (engine) Panics in Go code hanging processes
- (ansible) Account for mounts under RabbitMQ installation directory when upgrading RabbitMQ
- (engine) Allow the –race flag to be used for Go models
- (cli) Vor show process panics
- (engine) Fix handling of mismatched columns when reading data from S3
- Do not quote environment variables when running vor update playpen
- (engine) Wrong VOR_STREAM_PLAY results in a panic
- (sql) Fix ambiguous column error message
- Ensure virtual environment is sourced inside a shell when creating documentation site
- (django) Use get_object_or_404 when retrieving a study when updating a run status
- (runstudy) Fix structure of engine options in runStudy request body for report process
- (engine) Reduce the size of the Job information stored
- Remove invalid table inheritance
- Fix benchmark errors
- (sql) Fix failing tests in v2 test suite
- (sql) Add support for implicit joins
- Fix inconsistent results in join operation with USING constraint
📖 Documentation
- (sql) Update documentation for V2
- (engine) Document the PARTITION option in a SAS node
⚡Performance
- (django) Add tags to prefetch_related for job status query
- (sql) Pass records in join operation using channels
- (sql) Improve performance for aggregations and sorting
- (sql) Add hash join implementation
- (sql) Use channels to pass observations between operations
- (sql) Improve performance for creating sql results
- (sql) Add optimization for expression evaluation
- Improve performance of sql operation
✏️ Refactor
- (sql) Revise logic for set operations
- Use batch query when retrieving associated reporting runs in vor delete run
- (engine) Use well-known and common Protobuf types for dates and timestamps
- (engine) Change termination of node message from error to warn level
- (engine) Remove skip-rabbitmq-connection option
- (engine) Move and simplify shared memory functions
- (cli) Add vor show process filtering by tags; refactor other process commands
- (engine) Move initial RabbitMQ connection from Super startup to when needed
- Improve error messages when creating and updating a playpen
- (sql) Move SQL functions to its own package
🧪 Testing
- Add ACCESSIBLE=true to process runs executing under vor test
- Add a timeout for processes running under vor test
- Add –renew-token=false to vor show stats when running tests in parallel
- Run end-to-end tests in parallel
- (sql) Fix benchmark tests for join
- (sql) Sort test output
- (sql) Update bench results
You can request a demo of the new VOR release here.