QF-Test and Tosca in comparison

For its customer, DVZ GmbH has compared the test automation tools QF-Test and Tosca and concluded that QF-Test is the better automation tool for desktop applications, both for individual and multiple projects. The customer is advised to use QF-Test.

About DVZ, the IT Service Provider for Public Administration

The IT service provider for the state administration of Mecklenburg-Vorpommern is a long-standing and strong partner of the public sector. With the expertise of approximately 700 employees and state-of-the-art technologies, they shape the future of digital administration together with their customers. Below we reproduce the detailed comparison report by Martin König with slight editorial changes. You can find the original report in the German PDF version.

Why Did DVZ Compare QF-Test and Tosca?

DVZ compared the two automation tools to provide its customer with a recommendation on which tool is better suited for their automated desktop application tests. The evaluation was based on a Java Swing application with a browser component for login.

Initial Situation

The comparison focused on usability, required training time, and various deployment possibilities. Both solutions are similar in terms of application but follow different approaches.

• Different applications were used for various tests, with a focus on desktop applications.
• Mobile applications for Android and iOS were not considered, as there were no internal use cases.
• Maintainability and expansion possibilities were also evaluated.
• The comparison was conducted on a laptop with an i7 CPU.
• Tests within virtual machines (VMs) were also performed, but they had minimal impact on the evaluation.

Prerequisites and Installation

Installing QF-Test is significantly easier and more intuitive compared to Tosca.

• For Tosca, infrastructure must be set up in advance for the on-premise solution. A server is required to install the “Tosca Server,” which manages agent distribution and repository uploads. Additional components include a database for repositories and a license server.
• Installing Tosca Commander, the actual automation application, is straightforward, but integrating it with various servers and parts of the Tosca infrastructure requires manual configuration.
• Tosca has higher hardware requirements due to its infrastructure.

Both tools benefit from basic knowledge of test automation and software development.

• According to Tricentis, this is not necessary for Tosca, but internal experience has shown that without prior knowledge, the process and procedure can be misunderstood. To explain this: 3-4 people without a programming background were introduced to the tool over the deployment period of Tosca. Unfortunately, none of the people involved were able to use Tosca completely and they refused to continue using it because Tosca is too complex.

Use Cases

Both tools support various technologies, facilitating their use in different projects. The primary project for comparison was a Java SWING application with a browser component for login.

Both tools were able to automate and implement the test cases.

• It should also be mentioned that Tosca cannot be integrated into a pipeline, at least not without major effort. The infrastructure of the tool is too complex and the hardware requirements of the client are too high to enable quick and easy integration. The automated execution takes place via a call to the server, which in turn instructs the agent to start the execution. Although this works, it is a duplication within the pipeline and not an agile solution in the modern Dev-Ops environment, as activations etc. are required.

First Launch, Usability, and Overview

• The first start with Tosca is very extensive, as many parameters have to be set, such as creating the repository and connecting it to the servers.

With QF-Test, on the other hand, you can start the tutorial directly in the tool and view the demo suites without much effort for further settings.

• The interface of Tosca Commander is relatively confusing and packed with several functions via an old ribbon, which makes the application itself appear very inaccessible. It should of course be noted that Tosca also provides other parts for software testing in Tosca Commander. For example, requirements can be created directly in the tool or more complex data tables can be realized via TestDesign.

The tutorials are also very different.

• The tutorial series from Tricentis cost around 20 to 40 hours in total. These are necessary in order to understand how the tool works. Unfortunately, the tutorials are only prepared in English and sometimes contain errors, for example in the punctuation. For the Tosca tutorial, an account must also be set up on the support site and the manual unfortunately does not provide a good basis for getting started with the tool.

The QF-Test tutorial takes considerably less time and no additional registration is required. In addition, the manual is very well prepared and even clearer in the newer version. You can also start with QF-Test directly by “Learning-By-Doing” and is supported by “Step-By-Step” instructions directly in the tool.

Maintainability

• Maintenance of the components in Tosca is very time-consuming, as they have to be completely re-recorded if they are not recognized. The hierarchy of recognition in Tosca is: IDs, properties (combination), path, pixels (positions X/Y). The recognition is about 80% correct, but requires a lot of manual effort.

In QF-Test, the elements can be distributed over several .qft files, which simplifies recognition and finding. Depending on the application, the manual effort can of course also increase here. However, component recognition is significantly higher than with Tosca.

For larger test scenarios, such as process chains, the complexity can increase enormously.

• With Tosca in particular, the test cases are then very difficult to read and understand, as the components cannot be accessed directly, but are usually nested in a “shell”. The automatic designation of the “component shell” usually has little relation to the specialist application and must be described manually.
• Tosca also offers an AI component recognition called “Ara”. This was tried out with a test account, but provided little to no added value. Furthermore, an additional login is required here and the “Ara” can then only be used by certain people. Extra costs are incurred for use.

The comparison

Conclusion by DVZ: QF-Test is the Better Choice for Test Automation

QF-Test provides a better overall experience in handling, test script creation, installation, and maintenance than Tricentis Tosca. The added flexibility through scripting in JavaScript or Jython makes QF-Test significantly more adaptable compared to Tosca’s rigid drag-and-drop system.

• Although Tosca covers a broader range with additional tools like qTest or NeoLoad, these are costly and offer minimal added value for agile projects. Moreover, Tosca’s total cost is approximately 5–6 times higher than QF-Test, even when purchasing five QF-Test technology licenses.

Component recognition in QF-Test is about 20-30% more accurate and easier to understand, which significantly improves maintainability.

! Overall, QF-Test is the superior automation tool for desktop applications, both for single and multiple projects. DVZ recommends QF-Test for automation.

The advantages

Very good = 5 points, good = 4, middle= 3, sufficient = 2, unsatisfactory = 1

Martin König, test automation engineer, DVZ Datenverarbeitungszentrum Mecklenburg-Vorpommern GmbH, Germany

Case studies

to the overview

Automated GUI tests

GUI test tool QF-Test

Customers about QF-Test

This is what testers, developers and decision makers say about QF-Test.