Run regression tests

The Python script runs a suite of regression simulations and compares the output to the reference output for each simulation. It is set up to run the simulations on an HPC cluster; however, you can run modify the script to run tests locally. However, the script was written with remote execution in mind and running it locally can be time-consuming. Running the entire regression suite locally will take several hours on average.

The regression scenarios, script, configuration file, and other relevant files are all in the Regression directory. Be aware that many of these tests, due to abnormally high or low values, will produce output that should not be considered scientifically accurate.

  1. Modify the configuration file, regression_test.cfg, for your environment, setting the values for the location of the working directory, input data files, binary file, and cluster settings.

    • For local Windows simulations, set the values under [WINDOWS].
    • For local CentOS on Azure simulations, set the values under [POSIX]. Note that CentOS on Azure simulations are run locally by default and cannot be commissioned to an HPC cluster.
    • For simulations on IDM HPC clusters, no changes are necessary if your username and password are cached locally.
    • For simulations on your own HPC cluster, create [HPC-<cluster>] and [ENVIRONMENT-<cluster>] sections for your cluster that contain the same variables as shown for IDM HPC clusters.
  2. Select the suite of regression tests you want to run. This is indicated by a JSON file in the following format:

        "tests": [{
            "path": "Relative path to test directory."
        }, {
            "path": "Relative path to test directory."

    You can use one of the JSON files in the Regression directory or create your own. The sanity.json file is recommended for quickly testing a wide range of EMOD functionality.

  3. From the Regression directory, open a Command Prompt window and run the regression test script, It requires the name of the regression suite (without the .json extension) and the relative path to Eradication.exe. For example: sanity ..\Eradication\x64\Release\Eradication.exe

    In addition, you may need to include the following optional arguments depending on your testing environment or how Eradication.exe was built.

    Argument Default Description
    --perf False Measure Eradication.exe performance.
    --hidegraphs False Suppress pop-up graphs in case of validation failures.
    --debug False Use the debug path for EMODules.
    --label   Add a custom suffix for HPC job name.
    --config regression_test.cfg The regression test configuration file.
    --disable-schema-test True Include to suppress schema testing, which is on by default.
    --use-dlls False Use EMODules when running tests.
    --all-outputs False Use all output JSON files for validation, not just InsetChart.json.
    --dll-path   The path to the root directory of the EMODules to use.
    --skip-emodule-check False Skip checking if EMODules on the cluster are up-to-date, which can be slow.
    --scons False Indicate that this is a SCons build so custom DLLs are found in the build/64/Release directory.
    --local False Run all simulations locally.
  4. Review the output and examine any failures.

    EMOD will output the standard error and logging files, StdErr.txt and StdOut.txt, produced from any simulation (see Error and logging files). In addition, will output time.txt under the regression test working directory and report_xxxx.xml under Regression/reports. The time report contains the EMOD version and total run time of the simulation. The regression report is in JUnit standard form and captures run information, including pass/fail/complete and time to complete.

    If a simulation completes saying the run passed but the channel order was different than the reference output, this is considered a pass. However, if any output completes but does not match the reference output, this is considered a failure and a matplotlib chart of the output will appear in a pop-up window. The chart will appear immediately after the simulation, before the entire suite of regression tests completes. You can manipulate the output of the charts, such as adjusting the scale of the plots, zooming or panning, and so forth, through the icons at the bottom of the chart window.


If any of the regression tests fail and you have not made any changes to the EMOD source code, email