Profiling Go and Python computational nodes

VOR Stream profiling is intended to assist developers in improving code effeciency by helping them understand resource consumption in Go and Python code. That understanding can then be used guide the developer on code optimization.

Code optimzation will typically consist of iteration of a three step process. The first step will be to use VOR Stream's --go-profile and/or --python-profile options which will cause VOR Stream to collect profiling data from the execution of Go and/or Python computation node execution. The second step will be to use tools to analyze the collected profiling data. The third step will be to optimize code based on the information obtained from the profiling data analysis.

Go and Python computational node profiling is enabled by using the --go-profile and --python-profile options on the "vor run" command. Use of the --go-profile option requires an argument of either cpu or mem to indicate if CPU or memory profiling data is to be collected. The --python-profile option does not require an argument since the Python profiler captures CPU and memory data simultaneously. Profing data is written to a subdirectory of the playpen's output directory. That subdirectory will have the form "process name"/profiledata. The profiling data will be placed in files with names of the form "nodename"."extension" where "nodename" is the name of the computational node for which the data is collected and "extension" will be either gocpuprof, goheapprof or pyprof depending on whether the data is Go CPU profile data, Go memory profile data or Python profile data.

Analyzing Go computational node profile data

Analyzing both Go CPU and memory profiling data is easily done using the Go pprof tool. The tool can be installed using a command something like "go install github.com/google/pprof@latest".

The pprof tool can be run in a web server mode that allows the graphs to be viewed in a web browser. An example of running pprof in web server mode and providing access to Go profile data stored in a file named priceall.gocpuprof would be something like "pprof -http 192.168.100.241:4000 priceall.gocpuprof". The web server would then be accessed using the URL "http://192.168.100.241:4000".

Analyzing Python computational node profile data

Analyzing Python profiling data is easily done using the Python gprof2dot tool. The tool can be installed using a command something like "pip install gprof2dot --user".

The output from gprod2dot is in the dot graph format. That format can be converted to more common formats using the dot tool that's installed by default on many Linux systems. An example of generating a PNG format graph of Python profiling data stored in a file named priceall.pyprof would be something like "gprof2dot -f pstats priceall.pyprof | dot -Tpng -o output.png". The file output.png could then be viewed with any tool that supports PNG format.