Test Tags, Pass and Fail Conditions

Test tags allow to influence the execution condition of the given test configuration, or to define a required condition for passing or failing the test. These expressions are simply placed in the configuration file of the corresponding tests, a tag at the beginning of the line indicates which test tag the line corresponds to. The following tags are available:

  • Passing a test: The expression marked with the tag #PASS has to be found in the output in order for the test to pass. If the expression is not found, the test fails.

  • Failing a test: If the expression tagged with #FAIL is found in the output, the test fails. If the expression is not found, the test passes.

  • Depending on another test: The tag #DEPENDS can be used to indicate dependencies between tests. For example, module test ROOTObjectReader/01-reading implements such a dependency as it uses the output of module test ROOTObjectWriter/01-write to read data from a previously produced Allpix Squared data file.

  • Defining a timeout: For performance tests the runtime of the application is monitored, and the test fails if it exceeds the number of seconds defined using the #TIMEOUT tag.

  • Adding additional CLI options: Additional module command line options can be specified for the allpix executable using the #OPTION tag, following the format found in Section 3.5. The -o flag will be added automatically. Multiple options can be supplied by repeating the #OPTION tag in the configuration file, only one option per tag is allowed. In exactly the same way options for the detectors can be set as well using the #DETOPION tag, where -g will be added automatically. For all other command line options to be passed to the executable, the #CLIOPTION can be used. Here, the complete flag and possible value needs to be passed, e.g. -j9.

  • Defining a test case label: Tests can be grouped and executed based on labels, e.g. for code coverage reports. Labels can be assigned to individual tests using the #LABEL tag.

  • Describing the test: Every test should bear a short description of its goal. This descriptive text can be provided via the #DESC tag and is required for every test. CMake will print warnings for every test missing this tag.

  • Running scripts: Some tests require additional input which needs to be generated by a script. For this propose, commands to be executed before the tests starts can be provided via the #BEFORE_SCRIPT tag, e.g.

    #BEFORE_SCRIPT python @PROJECT_SOURCE_DIR@/etc/scripts/create_deposition_file.py --type a --detector mydetector --events 2 --steps 1 --seed 0

    to run a Python script that generates an input file read by the test.

  • Requiring external data: Some tests require external data which needs to be downloaded before executing the test. For this purpose, the #DATA tag is available, which can contain file paths which will be set as required files for the test. If registered with CMake’s ExternalData module, they will be downloaded automatically.