A script for regression testing of command line executables that output text (E.g. compilers).
All it does is run small command line files that end in .case.sh
, manage their outputs and detect changes. It's a
very simple Python program but has proven to be useful. It was first written to test a compiler built for a university course.
- Copy, submodule or otherwise get
crystalmountain.py
to be callable from a directory in your project - In such directory, create a
cases
subdirectory - Populate
cases
with*.case.sh
files that run the program you want to test
And you are al set. The suite is operational. You can run crystalmountain.py -g
to generate initial outputs. Which
will be written to corresponding *.spec
files.
Note that cases can also be in nested subdirectories
Assuming the above setup:
crystalmountain.py
: run the test suite. This will record if there are any changes in the outuptscrystalmountain.py cases/my/test.case1.sh cases/my/test.case2.sh
: run those two particular test casescrystalmountain.py -g
: generate new outputs, will not override existing onescrystalmountain.py -s
: show the diffs from the last runcrystalmountain.py -l -f
: list all the test cases that are failing
usage: crystalmountain.py [-h] [-p | -c | -s | -r | -g | -l] [-f | -n | -a] [CASE ...]
Run a regression test suite for liblsqecc
positional arguments:
CASE Specify one or more particular test cases
optional arguments:
-h, --help show this help message and exit
-p, --displayonly display the results of the execution without writing anything
-c, --copyover Instead of running, copy over the results from the previous run
-s, --showdiffs Show diffs after failed tests
-r, --removeoutput Removes outputs for this test
-g, --generate Generates tests if they don't exist yet, otherwise do nothing
-l, --list List available tests
-f, --failing Only consider cases that failed in the latest run
-n, --nospec Only consider cases that don't have a spec defined
-a, --passing Only consider cases that passed in the last run
The requirements are pretty minimal:
- A shell to run commands (E.g. bash or zsh)
- A Python >= 3.7 installation
- A
diff
command for rendering changes (Optional)
Only tested on Linux systems, but probably works on MAC WSL/GitBash.
Details of the license in the top of crystalmountain.py