Execute test cases and sets with a certain tag

Large test projects may consist out of several thousands of test cases. Often executing them all would need a considerable amount of time, even if multiple machines are used in order to replay the different tests.

In order to develop such large test projects, it is often useful to store the different test sets/cases in different files. For example we may create one file that contains all tests for feature A and another file that contains all tests for feature B. Nevertheless it may still be a good idea to annotate the different tests with tags.

Tags like “feature1”, “feature2”, “fastRunningTest”, “slowTest”, “importantTest”, “oftenFailingTest” and so on make it possible to run a dedicated subset of all test cases. For example this makes it possible to run only those test cases tagged with “oftenFailingTest” and “importantTest”.

The “Condition” field of a test case/test-set node can be used to assign tags to any test case/test-set. For example if you want to assign the three tags tag1, tag2 and tag3 to a test case / test-set you can use the following syntax:

"""${default:tagsToExecute:}""" == "" or len(set(["tag1", "tag2", "tag3"]).intersection([x.strip() for x in rc.lookup("tagsToExecute", expand=False).split(",")])) >= 1

This condition will advice QF-Test to execute the test case/set when one of the following conditions are met:

• The tagsToExecute variable is not set or empty.
• The test case/test-set is annotated with a tag that is also listed in the tagsToExecute variable (comma-separated list of tags).

This means in order to execute only those tests annotated with tag1, the variable tagsToExecute needs to be set to tag1. In order to execute those tags either annotated with tag1 or tag2 the variable  tagsToExecute needs to be set to tag1,tag2 and in order to execute all tests annotated with tag1, tag2 or tag3 the variable tagsToExecute needs to be set to tag1,tag2,tag3.

Remarks

• When calling QF-Test from command line, it is possible to set a  tagsToExecute variable via the -variable command line argument (e.g. -variable tagsToExecute=tag1,tag2), see
  Command line arguments and exit codes
• For better readability, it is allowed to add spaces before and after the comma. So in order to execute the tests annotated with tag1 or tag2 the variable  tagsToExecute may be set to
  tag1,tag2 or
  tag1, tag2 or even tag1 , tag2.