Re-execution of nodes (Rerun)

When a test run has finished, the run log or report is a good entry point for evaluating the results. In case of errors you may face various challenges. You might want to re-execute the failed test cases to investigate the reason for the error or because you want to perform an official re-test of this failing situation after removing the error condition. If the re-test results are to be shown in the test-report, you may want to replace the previous results or append them to the existing ones. Or you might just want to repeat the test case with the previous variable settings and keep the new run logs and reports separately.

To that end QF-Test offers the capability to trigger re-execution from the run log. You can trigger a rerun via selecting the run log node or any test set node and choose »Rerun test cases« from the »Edit« menu or from the context menu. Alternatively you can select the nodes to rerun it the error list and use the context menu entry »Rerun test cases of selected nodes«. The dialog then shown lets you select the test cases for the rerun and choose how to handle run logs via the selection box Mode for merging run logs with the following options:

Choice	Meaning
Replace test cases	Replace the test cases from the original run log with the results from the rerun, i.e. the previous results will get lost. The previous run log will be saved in a backup copy.
Merge run logs	The new test results will be merged into the existing structure.
Append run log	The new test results will be appended to the end of the run log. The test set structure will be ignored.
Keep run logs separated	The new test results will be written to a new run log, the original one remains unchanged.

Table 25.1: Choices for handling the run log of a rerun

For each test case the variable values are taken from the run log of the original test run. Since only the String value of the variables is stored in the run log, all values are of type "String" during rerun from long. During re-execution the variable ${qftest:isInRerunFromLog} is set to true, making it possible to distinguish between a normal test run and a rerun.

NoteMerging of run logs makes use of names of test cases and test sets. Therefore, those names should be unique. In case of data-driven testing you should take care to keep those names unique via the attributes Name for separate run log or Characteristic variables.

During your test automation project you can sometimes face situations where some test steps don't provide reliable results, failing sometimes but not always. Most of the time such cases depend on timing and can be stabilized using Wait for component to appear nodes, or checks for conditions, delays, scripts or other control structures. As an alternative or additional approach QF-Test offers the capability to repeat such steps whenever they fail.

This automated rerunning in case of error can be applied to every executable node using a certain doctag in the comment attribute. This doctag can look like this:

@rerun attempts=3;errorlevel>=ERROR;newerrorlevel=WARNING;
          handler=handlers.errorhandler

Example 25.18: Example for a rerun definition

In the example above a failed node will be repeated up to three times until an attempt succeeds. Failed attempts will be downgraded to "warning" in the run log. In case all attempts fail, the last attempt will be reported as an error or exception. After every failed attempt QF-Test will execute the procedure handlers.errorhandler.

If you are interested in the number of the current rerun attempt, you can use the variable reruncounter from the qftest variable group, see section 6.8.

The @rerun doctag offers the following parameters:

attempts: The maximum number of attempts.
errorlevel (optional): Defines the error states to be handled. Possible values are EXCEPTION, ERROR or WARNING with = for an exact match or > or >= for a range. Specifying errorlevel=ERROR means to rerun that node only in case of an error whereas errorlevel>=ERROR triggers the rerun in case of errors or exceptions. If this parameter isn't specified the value errorlevel>=ERROR will be taken as default.
newerrorlevel (optional): Specifies the error-level in the run log for the initial run and possible additional failed runs. You can again choose between EXCEPTION, ERROR or WARNING with the additional options NOLOG and KEEP. The level NOLOG stands for removing the failing results from the run log entirely. NOLOG should be used with extreme care. Using the level KEEP doesn't override the original error level and reports it unchanged. If this parameter isn't specified the value WARNING will be taken as default.
handler (optional): The name of the procedure which should be called in case a caught error state occurs. This procedure will be called after each failed attempt.
reusevariables (optional, default=true): Specifies whether to reuse the variable values from the beginning of the first run. When set to false the current variable values will be used.
logmessages (optional, default=true): If that parameter is set to true a message will be written into the run log, when an attempt begins and when the execution of that sequence terminates. In addition, every node gets an annotation in the run log with the current attempt.
logmessagesintoreport (optional, default=true): If this parameter and the parameter logmessages are set to true, all messages will be written to the report as well.
keepfirst (optional, default=false): If this value is set to true the first failing attempt will be logged with its original error level. In case of further failing attempts those errors will be logged with the newerrorlevel level.
exceptiontype (optional): In case you want to catch only one specific exception you can specify the exception type here, e.g. CheckFailedException or just ClientNotConnected for a ClientNotConnectedException. This parameter should only be used if you set Exception as value for the parameter errorlevel. Please see the Catch node for details about exceptions.
exceptionmessage (optional): In case you want to catch only one specific exception with one text, you can specify the text here. This parameter should only be used if you set Exception as error level. Please see the Catch node for details about exceptions.
exceptionregex (optional): If true, the value of exceptionmessage is a regular expression. This parameter should only be used if you set Exception as error level and an exception message. Please see the Catch node for details about exceptions.
exceptionlocalized (optional): If true, the value of exceptionmessage should be the localized exception message, e.g. mostly the full text. This parameter should only be used if you set Exception as error level and an exception message. Please see the Catch node for details about exceptions.

4.1+25.3 Re-execution of nodes (Rerun)

25.3.1 Triggering rerun from a run log

25.3.2 Rerunning failing nodes immediately