A module to plot ITK performance benchmarking data.
Three dataset are provided for the purpose of demonstrating the visualization capabilities of the module:
- simplified_data.tar.gz: a reduced set of files, primarily intended as a restricted benchmarking dataset.
- scatter_os_data.tar.gz: a dataset that is intended to demonstrate the capabilities of the scatter plot visualization method, and provide a means to compare the performance of a module across different platforms and ITK versions.
- clay_data.tar.gz: original
claybenchmarking files hosted at data.kitware.com, a dataset intended to fully exercise the module.
Note that the JSON files in the simplified_data.tar.gz and the scatter_os_data.tar.gz were manually edited from the original clay benchmarking files hosted in the data.kitware.com to simulate benchmarks for ITK versions that were not present in the original clay dataset, as well as to give rise to some situations (e.g. modules not present in some ITK versions, different number of time probes across modules or versions, etc.) that could potentially arise when benchmarking ITK modules across time. Thus, they may not reflect actual ITK module performances.
The visualization scripts were tested on all three datasets using Python 3.6.5.
Please follow the steps described:
- Clone the repository.
- Install Python.
- Install the requirements using the requirements.txt file provided.
Run the example visualization program on the input dataset by providing the folder where the (uncompressed) benchmarking
JSONfiles are located, and provide a filename for theJSONsummary, e.g.:python ./examples/itk_performance_benchmarking.py ./example_data/clay ./output/summary.json
These example visualizations were created using the JSON files contained in the scatter_os_data.tar.gz example dataset.
Module error bar plot:
Module scatter plot:
Historical heatmap:
This software is distributed under the Apache 2.0 license. Please see the LICENSE file for details.