QF-Test - casestudies

End-to-end tests for automated testing of a Swing-based ERP solution

2026-02-03T00:00:00+01:00

About ABS

ABS is a leading software company for laundry management systems, specialized worldwide in the laundry and textile rental industry. Our software has evolved into a comprehensive and integrated ERP solution for industrial laundries—including route accounting and contract management.

Automated End-to-End Testing without programming skills

After conducting extensive research on multiple automated testing solutions, we selected QF-Test as the best fit for our Java Swing based laundry ERP system.

A key advantage was the ease of use, which enabled our team to quickly build business flow tests without requiring advanced programming knowledge.

This enabled us to

automate critical flows early on,
significantly reduce manual testing effort, and
accelerate delivery.

Expanding test coverage

Over time, the QF-Test tool has continued to mature, allowing us to extend our automated test coverage beyond Java Swing to web applications, mobile applications, and REST web services.

For unsuccessful test flows, QF-Test provides highly valuable diagnostic data, including

screenshots,
detailed step-by-step error information, and
error messages.

Additionally, by integrating Jython scripts into our dependencies, we are able to validate error log files, giving us deeper insights into failures and improving root-cause analysis.

Automated nightly testing saves time and effort

We implemented a fully automated nightly testing process using Jenkins that executes all QF-Test test runs automatically and provides comprehensive results.

These reports are extremely helpful for identifying code defects as well as opportunities to improve test flows.

The ability to run these automated tests on both master and feature branches has saved our team substantial amount of time and effort compared to manual end-to-end testing.

Outstanding support

Beyond the technology itself, our collaboration with the QF-Test team has been outstanding. Their support team is responsive and knowledgeable, consistently helping us solve challenges when needed. They also regularly offer webinars that provide valuable insights into new features, enhancements, and best practices, ensuring we continuously get more value from the tool. After more than 15 years of use, QF-Test remains a trusted partner in our automated testing strategy.

Conclusion

QF-Test, as the best solution for automated testing of our Swing-based ERP solution for the laundry industry, has exceeded our expectations for over 15 years. The intuitive approach enabled us to rapidly create end-to-end tests with minimal programming effort, drastically reducing manual testing needs. Thanks to the continuous innovation from the QF-Test team, we have since expanded our test coverage to include mobile applications, web applications, and web services, making QF-Test a trusted long-term partner.

Anca Stamatopol, QA Automation Team Lead, ABS Laundry Business Solutions

Mobile Regression Testing for iOS and Android

2026-02-02T00:00:00+01:00

Expanding the GKV Software Standard BITMARCK_21c|ng

About BITMARCK

As a leading digitalization partner in the statutory health insurance sector, BITMARCK is accelerating digital transformation within the industry through innovative products and services. The foundation for this is the GKV software standard BITMARCK_21c|ng, which is used by affiliated healthcare funds. Clients of the BITMARCK Group include company and guild health insurance funds, as well as DAK-Gesundheit and other substitute health insurers—over 30,000 employees and around 35 million insured individuals within statutory health insurance (GKV) benefit from BITMARCK’s IT services. More than 80 percent of German statutory health insurers are clients of the group. With more than 1,900 employees, BITMARCK generates annual revenues of more than half a billion euros.

The Challenge: Seamless Quality Assurance for Mobile Channels

After successfully implementing test automation for desktop clients (Web and Swing), BITMARCK GmbH faced the challenge of bringing the quality of its native iOS and Android apps within the GKV software standard BITMARCK_21c|ng up to the same high level.
This required a forward-looking and flexible extension of the existing test automation framework in order to:

Ensure a consistent user experience and functionality across both mobile operating systems (iOS and Android).
Quickly and efficiently create new mobile test cases and integrate them into the existing set of over 4,500 regression tests.
Integrate mobile test coverage into the Continuous Delivery (CI/CD) pipeline to ensure stability with every release cycle.

The Solution: Leveraging the QF-Test Platform for Mobile

The existing foundation, consisting of QF-Test and the jointly developed Multi-Client Recorder, served as the basis for the mobile expansion.

Mobile Regression Testing Strategy in Detail

BITMARCK focused on securing its native iOS and Android apps, which serve as important customer and employee portals, by taking the following steps:

Technology Selection and Integration: Thanks to QF-Test’s platform-agnostic nature, the framework could be directly extended to native mobile apps. Integration was achieved via QF-Test, enabling seamless use of both physical devices and emulators.
Mobile Test Object Recognition: Close collaboration with QFS enabled the early implementation and optimization of mobile test object recognition. This ensured robust test cases that are resilient to minor UI changes across both mobile platforms.
Specific Mobile Interactions: Specific mobile interactions (such as swiping and handling virtual keyboards) are captured by the recorder and incorporated into test cases.
Result: Mobile test coverage has been successfully integrated into the CI/CD pipeline and now ensures the quality of iOS and Android applications in every release cycle.

The Benefits

Cross-Platform Efficiency: The Multi-Client Recorder optimizes test automation for Web clients, Swing clients, and native iOS and Android apps—all under one unified framework.
Comprehensive Coverage: Integration allows management and execution of regression tests across all client architectures, including the new mobile coverage.
Accelerated Test Development: The recorder’s intuitive design carries over to mobile development, simplifying the creation and maintenance of test cases while significantly reducing maintenance effort.
Maximum Mobile Software Quality: By expanding its regression testing strategy, BITMARCK Software GmbH ensures the highest quality for its mobile solutions for statutory health insurance.
Continuous Delivery: Mobile coverage has become part of the CI/CD pipeline, guaranteeing the stability of native apps in every daily run.

Conclusion

The successful integration of mobile regression testing for iOS and Android using QF-Test and the Multi-Client Recorder highlights the flexibility and future-readiness of BITMARCK GmbH’s test automation strategy. This expansion safeguards the growing importance of mobile channels and reaffirms the commitment to the highest quality standards in software development.

Dirk Kruck, Test Engineer, BITMARCK Software GmbH

Successful migration of an ERP software solution thanks to efficient test automation with QF-Test

2025-01-01T00:00:00+01:00

QF-Test at remoso GmbH

For more than 30 years we have been a leading provider of ERP software solutions for mobility. We develop state-of-the-art fleet operator concepts for fleet management, vehicle rental and car sharing.

Our current ERP software originally consisted of a monolith that was more than 30 years old. In order to keep up with the market and meet the planned future requirements, we decided to discard this monolithic approach and focus on a scalable software solution. In doing so, we have chosen the following technology stack:

Multi-Module Maven Project
Java
Typescript
Spring Framework
Angular Framework
Kendo UI Components
Bootstrap CSS Framework & Responsive Design

No sooner said than done!

The way was prepared and the new software product was developed. But now we were faced with a whole new challenge: the regression tests would take more than 3 weeks in manually executed tests, due to the architectural changes! In our agile software development approach, more than 3 weeks lead time is not acceptable. In order to increase testing speed while maintaining quality, we decided to implement a test automation system.

Why QF-Test?

As part of a market analysis, we examined the available software solutions and compared them with each other. QF-Test emerged as the winner from the comparison. It meets our specific requirements for test automation very well. For example, QF-Test is able to recognise Angular components and thus saves us code adaptations, such as the assignment of IDs for each individual element. Competing products could not offer this added value. We have now been using QF-Test for 3 years. By automating the regression tests, we are able to run our complete test cycles every night. Through the Jenkins integration, the tests are executed automatically, time-controlled and the result is reported to our test automation team. By using QF-Test, we have been able to save a lot of time that was previously spent on manual testing. Our quality assurance team has been relieved and has more time to deal with the other sub-areas of software quality assurance. In the future, we plan to extend test automation to our mobile apps as well.

The decision to rely on test automation with QF-Test has paid off for us after a short time. We carry out more tests in a shorter period of time, relieve our employees and increase quality because human errors are excluded. The investment in QF-Test and the corresponding qualification of the employees is also seen as more than justified by the management.

Many thanks to QFS!

We would like to praise the cooperation with Quality First Software GmbH. No matter with which question or problem we turned to the support, we were always helped quickly, competently and purposefully. In addition, the numerous video tutorials and the free webinars are simply exemplary. At this point, a big thank you to the support department.

Patrick Veit and Marius Pramsoler, remoso GmbH, Ravensburg, Germany

Daily UI regression tests with QF-Test

2025-01-01T00:00:00+01:00

QF-Test at Swiss Life

Since 2012, we utilize QF-Test for daily UI regression testing of our new insurance quote system EVApro and are very satisfied.

For us, the use of QF-Test mainly has the following advantages:

Generic approach: With QF-Test it is possible to record actions elegantly for each occurring control type and then to abstract these, e.g. the actions ‘set value’ and ‘check value’ for text fields and combo boxes or the action ‘click’ for buttons.
Shared test case files for HTML and Java interfaces: The application under test exists in both an RAP and an RCP variant. Due to the generic approach it is possible to use the same test case files (*.xls) for the test of the two variants.
Simple recording within the application: Due to the generic approach the test case files contain only relatively simple data such as (set value, PersonDialog.BirthDate: DATE FIELD, 06/05/1978), through which a control with the specified ID obtains the specified date value. Such information is recorded automatically when a user operates the application; when he opens a special dialog of the application, a table will appear from which he can copy the information directly into his test case file.
Small, well structured, easily readable test suite file: We need for RAP and RCP only a single shared test suite file (*.qft), which is very small (approximately 150 KB) and well structured (XML). In addition, it is fully readable: In contrary to some QF-Test competitive products this file has no encrypted or proprietary sections. This also greatly facilitates the comparison of different development states of this file in our version management system.
Easy upgrade from RAP 1 to RAP 2: Due to the generic approach the switch from RAP 1 to RAP 2 was easy to accomplish: Only the test suite file had to be revised, and for most control types it was even sufficient to change just the component reference ‘class’, i.e. type of component, of the appropriate generic component.
Competent support: In several one-day workshops at our site an employee of Quality First Software GmbH (QFS) ably supported us to continuously develop the test suite file. Especially in case of more complex controls (such as tables), he helped us through profound product knowledge, in particular through the use of scripts. E-mail queries between the workshops were always quickly and competently answered by QFS.
Excellent documentation: Manual and Tutorial have often helped us. It is obvious that the authors do not view documentation as a chore, but are convinced of their product and want to describe it fully and understandably. They have succeeded there.
Very comfortable call options: Since variable values can be passed when calling QF-Test we are able to determine very comfortably via the parameters of a self-written batch file,
- what test cases are meant to be run (all 400 test cases of all 35 fares of our insurance quote system? Only the test cases of a given tariff? Only a particular test case?),
- which variant is to be tested (RAP or RCP ?)
- what level of development is to be tested (released version or local unstable?)
- whether to move the mouse during testing
- whether QF-Test is to be started interactively or in the background
- which browser to use in the version in the RAP case (Internet Explorer or Firefox?)
- which QF-Test version to use.Each call parameter is either a (tariff or test case) number or a single letter, thus very simple. This way our department can easily create and run test cases without having to know intricate details.
Simple workarounds for version changes: For UI regression tests several applications have to work together:
in the RAP variant: the browser in which the actual application is running (by default Internet Explorer)
the program in which format the test case files are present (Excel)
the UI test tool (QF-Test).

In these interactions problems can occur, of course, if the version of one of the involved application is changed. As a workaround, we therefore have, for example,

tested the RAP variant with a different version of QF-Test than the RCP variant
used instead of the current Internet Explorer version an older Firefox version in the RAP variant.

Such workarounds so far always helped us to bridge the time until the test suite file was revised in a subsequent workshop, a new QF-Test version was provided or an improved Internet Explorer version was available. Thanks to the very comfortable methods mentioned above we could implement and undo these workarounds always relatively easy.

We can therefore absolutely recommend the usage of QF-Test.

Author:

Dirk Stanke

Contact:
Max Gest
Application manager EVApro
Swiss Life AG, German Branch
Berliner Straße 85
80805 Munich

(Original German texts and citations are translated into English.)

UI Tests of SAP Manufacturing with QF-Test Integration in Microsoft Azure Pipelines

2025-01-01T00:00:00+01:00

About Systema GmbH

Integration of manufacturing systems, extensions, consulting and customized software solutions for vertical industries of discrete manufacturing are the core competence of Systema GmbH. Systema has over 200 employees at 8 locations worldwide. The good reputation of our brand is based on more than a quarter century of business based on an ethic of mutual respect and trust – among our team, our customers, partners and competitors.

Initial situation

We develop extensions for the SAP Manufacturing System for our customer, a well-known company in the precision mechanics and optics industry.

These plugins complement the existing functionality and provide users with individually customized interfaces depending on their specific requirements.

To ensure the quality and stability of our development, we were looking for a way to test these plugins directly in the SAP Manufacturing System.

Since we had already worked successfully with QF-Test in other projects, it was our first choice of testing tool.

Objective

Our goal was to recreate the actions of an actual user in the tests in order to reproduce the processes in production as close as possible.

The execution of the tests must be automated to reduce manual efforts.

Implementation

One requirement for the tests is that they replicate reality as closely as possible.

UI test tools such as QF-Test, which actually interact with the user interface, represent the best solution here.

In preparation, test cases were created based on the specified requirements, which were then grouped into scenarios. The result was a detailed test plan, which was implemented by the automation team using QF-Test.

Technical realization

The test creation with QF-Test was mostly done using the recording function.

Problems arose due to the use of generated component IDs, which prevented unambiguous identification and thus made the tests non-reproducible.

To solve this problem, generic components were created, which enabled reproducible resolution of the target components using regular expressions and an appropriately configured CustomWebResolver. The excellent support of QFS, who helped to develop this solution, should also be mentioned here.

During test development, another challenge became apparent. The SAP Manufacturing System consists of several components, which together provide complex data. Our target system is the ME as the front-end with the MII as an extension of the back-end. In addition, there is the ERP system.

Dependencies in the production system

To reduce complexity and get reproducible results, we decided to simulate the ERP (mockup).

Dependencies in the test system

Using the MockServer framework, we developed this mock, which provides predictable data.

MockServer is a separate software project and contains different templates, which are preconfigured for different functionalities. In order to implement new requirements, we just had to create a suitable template.

After completing the test development and the successful manual testing, the next step was to automate it. A Microsoft Azure pipeline was used for this.

The pipeline allows us to automatically prepare the test data, pass user data and variables to the test suite, evaluate the results, and then clean up all systems.

Azure pipeline flow

Since both MockServer and the QF-Test test suite are maintained in GIT, we can ensure that the latest version is always used at runtime.

Conclusion and outlook

By using MockServer and QF-Test, a combination of backend (logics) testing and UI testing was achieved. This allowed us to efficiently build a realistic and reliable test environment to automate testing plugins developed for our customer directly in the SAP Manufacturing system. The principle of continuous testing was implemented by integration into an Azure pipeline. This achieves fast test feedback and thus effectively prevents early regression, as well as minimizing the manual testing effort.

In the further course of the project, the tests will be continuously maintained, improved and expanded in order to ensure that new functionality is also secured.

Non-Regression Tests with QF-Test

2025-01-01T00:00:00+01:00

QF-Test at Thales Australia: Saving time with the tests

Let‘s say the automation of our tests is successful and this is a big step forward. I am not an expert on all the other available tools but from what I know, I don‘t see how I could have interfaced other tools than QF-Test with our system, moreover in such a short period of time.

The few points that make QF-Test the killer application:

A mature and complete HMI to design the Test Suite.
A synthetic report generated automatically in HMTL.
No need to hack into the JVM or the application under test to make things testable.
Jython scripting: You can do anything you need with this.
Extension API: This allows to make QF-Test aware of internal objects inside the UI application.

In terms of how far we went into the integration of QF-Test into our process: Our Non-Regression tests have been almost fully implemented under QF-Test. It takes roughly three hours to run and hence saves me heaps of time.

In terms of ROI, there are several aspects: Regarding the time saving aspect, it usually took me a full day to perform the Non-Regression Tests. It has now been reduced to three hours. The big difference is that I can do something else in the time saved! The direct benefit is that I can now run some tests on different versions of our System (i.e. different branches) in parallel, when it was impossible before (not enough time).

Another less visible aspect is that it improves the communication with the Software Development. There are no feelings or personal judgement in the appraisal of the results. It happened previously that depending on who was running the tests, the results were slightly different. Indeed, as I mentioned this is a real time System and depending on what action you were doing at what time, the behaviour of the System might not always be the same. Having automated the tests removes (or decreases) these timing issues.

Software Development cannot say anymore that you stuffed up or did not use the System properly during the tests (and asking you to redo them and hence gain some time …). It is always the same tests that are reproduced and if the results changes, it is because the System has changed. Sometimes good changes (improved functionality, bug fixes, faster response times ….) but the automation picks that up, which is the whole point of the NRG Testing: detecting changes.

There are some drawbacks though in automating the tests:

First, tests need to be maintained! The SUT is evolving and the components need to be updated. New functionalities are also introduced and they need to be added into the scope of the tests.

Then the tests are not perfect: they are not bullet proof and sometimes an issue can generate a domino effect and void the complete result. If you are not in front of the machine during the test, you will not see it and possibly loose three hours. So every time it happens, I fix the test and add more checks and protections. But once in a while I need to relaunch the test.

Then QF-Test will only detect issues that you taught it to recognize. In comparison, a human tester would be able to detect odd behaviours of the System. Again, it is real time and with a lot of algorithms. Not everything is predictable. That means that issues might not be detected by QF-Test (or more accurately the Test Suite I wrote). This is usually detected further down the test chain (IBB, V&V Testing …) but then it costs a bit more to correct. Every time it happens, I try to improve the Test Suite, which is a bit time consuming.

Well, you may wonder then if there is a real gain in automating tests.

My conclusion would still be yes, but the ROI is not on the time. The time you do not spend on the testing anymore needs to be spent on different topics: maintenance, improvements, analysis of what you want to test and how you want to test it (not always as straightforward as one could think). The gain you obtain is on the quality and consistency.

Denis Gauthier Software Integration, Thales Australia, Melbourne

QF-Test and Robot Framework – A Case Study from Scheidt & Bachmann

2024-01-01T00:00:00+01:00

When automating the execution of test cases, Scheidt & Bachmann has been using the test tool QF-Test for several years already, as it is ideally suited for testing Java UI applications.

Now, the challenge was to use the already existing Java UI test cases including all procedures created in QF-Test also within Robot Framework. This was necessary because Robot Framework is the tool of choice for testing field devices running on the Linux operating system with the Qt application under test.

In order to create complete end-to-end test cases between field device (Qt application) and back office application (Java application), we needed a way to connect QF-Test with Robot Framework. Here we quickly came across the interface offered by QF-Test, which allows Robot Framework to address and execute individual procedures via keyword (Robot Framework uses the approach of keyword-driven testing).

The following figure shows how Robot Framework uses and controls a QF-Test procedure:

This approach allowed us to create complete end-to-end test cases that greatly simplify and speed up the execution of release tests at field device and back office level.

A significant impact during the implementation of this new feature had the very cooperative and fast customer support of QFS, which allowed us to make noticeable progress in a very short time.

In the future, this approach will save us a lot of time in creating new automated test cases, as well as in executing them.

Author: Marvin Klerx
QS – Fare Collection
Scheidt & Bachmann Fare Collection Systems GmbH
Mönchengladbach, Germany

Testing Java Web Start with QF-Test

2024-01-01T00:00:00+01:00

QF-Test at Siemens: Testing Java Web Start

UI tests are a vital step in the software testing life-cycle because they are performed from the view of the end user of an application. Test automation of this phase is the final goal as it is already done with unit or integration tests.

When you search the Internet for such a task you will find a broad spectrum of functional test tools that are either „general purpose“ test tools or dedicated to a specific kind of UI. My job was to find a suitable test tool that is able to perform automated tests for a complicated Java Swing UI (provided as a Java Web Start application).

Of course, you can start evaluating all the „big players“ in this area but „big player“ also means „big money“ in terms of license fee and maintenance cost. The other possibility is to look for vendors that are specialized in Java Swing UI testing if this is all you want. Such a first preselection was my starting point and QF-Test was one of my first candidates for a deeper analysis.

Open source tools may be a choice too for every user that is looking for a low cost solution. However, in my opinion these tools are capable of handling simple UI tests but reach their limits with complex UIs (custom components, etc.) if you do not invest time and money to do a lot of „customization“ (it‘s open source, so you are free to extend them). But you can also decide to take the middle road and then you should definitely give QF-Test (for Java) a try.

For a tool comparison I have created more than 30 evaluation criteria (annotated with different priorities) like

record and playback possibility
smart UI component recognition
object mapping
image checks
test failure and error recovery
support for data-driven testing
use of popular scripting languages
base ground for keyword-driven testing (like QF-Test‘s parameterized procedures)
integration into popular build and/or continuous integration systems (like Ant/Maven or CruiseControl)
test execution/result reporting using XML and HTML
multi-screen support
drag&drop support
possibility to recognize complex custom components
license & maintenance costs
quality of support (availability of skilled resources, validity of responses from the help desk)
online resources (like active forums rather than ringing the support line)
speed of responses
quality of online documentation
…

Having these evaluation criteria as well as dedicated test scenarios for the real system under test I have to say that QF-Test succeeded in fulfilling all my requirements with a score of more than 90%. That result placed QF-Test above all other evaluated commercial and open source tools — at least for my specific requirements and target.

However, I think you will hardly find another Java Swing UI test tool that gives you a better value for your money.

Result: With QF-Test UI testing can change from an obligatory task into excitement.

Klaus Berg, Senior Engineer, Siemens Corporate Technology, Munich

(Original German texts and citations are translated into English.)

Test automation with QF-Test as central pillar in the fight against bugs: QF-Test and SIV.AG

2024-01-01T00:00:00+01:00

SIV.AG successfully uses proven tool of partner Quality First Software GmbH for its customers and in its own company

SIV.AG relies on test automation to ensure the quality of their software. Every week, several hundred process tests are carried out on the user interface of the complex ERP system kVASy using the tool QF-Test of Quality First Software GmbH. “Over time, we have reached a remarkable complexity in the test cases, which is why our regression tests achieve large-scale test coverage with QF-Test,” says Andrea Menda, Deputy Head of Department Management Operations at SIV.AG.

“In addition, with the help of QF-Test, we operate complex monitoring regarding availability and performance on the systems of one of our largest customers. Furthermore, several of our customers use the test automation tool independently. They have built up the know-how in close cooperation with us. This allows for more flexible testing of unique configurations with customer data sets.”

Excellent cooperation since 2014

The extraordinarily good long-term cooperation deserves special mention: “Ever since we started using test automation at SIV.AG, we have had an extremely reliable and internationally experienced partner in Quality First Software GmbH. With their help we have been able to increasingly professionalize ourselves. The new feature developments of the tool and the reliably quick support helped us achieve this,” Andrea Menda continues. A bidirectional knowledge transfer also strengthens the relationship. Over the years, SIV.AG has developed a comprehensive tool for static code analysis, which is planned to also be used by Quality First Software GmbH in the future.

Further information:

Product booklet German PDF only
SIV.AG page

(Original German texts and citations are translated into English.)

Seamless transition between testing Java software and web testing

2023-01-01T00:00:00+01:00

About pds GmbH

pds GmbH has been developing programs for the main and ancillary construction industry since 1974. With the pds software, it offers an ERP solution that maps all processes for companies from start-ups to established corporations through additions via app and web.

The way to QF-Test

At the beginning of the millennium, pds began developing the Java-based pds software and planning future quality assurance. At the time, the company was known for its slogan “Quality in software”, which was also to be a high priority for the new generation of software. Tests on the code were already standard at the time, while the user interface still had to be tested manually. With an ever-increasing number of components, the search for a more efficient solution began.

As a result, many programs were tested in 2007. QF-Test was one of the solutions tested, and this program was an immediate success. Testing could be started immediately and without a lot of settings or prior knowledge. All you had to do was enter the path to the software, then record, play and stop, just like a cassette recorder (which almost everyone knew back then). There are also many expansion options for integrating scripts and other calls as well as options for fulfilling special software requirements.

A whole department

After the order was placed, a full-time position for test automation was created very soon, the first scenarios were recorded and an internal administration for daily software installation and test execution was set up. The simple maintenance, evaluation and re-recording of tests in QF-Test have proven their worth.

More tests, requirements and results led to a second and now third position in the company. Around 180 tests are now run daily, providing valuable and timely results and freeing up manual testers for other tasks.

Current projects

This year, the pds quality assurance team has set itself the goal of starting to automate the testing of the web apps. First of all, a test license was requested from QFS in order to check functionality and compatibility. This gave us the opportunity to clarify minor teething problems and unanswered questions before the acquisition in a well-functioning exchange with QFS.

The result was an initial test scenario that completely convinced everyone involved. Our development team was particularly impressed by the seamless transition between testing the Java software and testing the Flutter website. The full license for the web engine is now in active operation and an extension to include Android compatibility is being planned internally.

Highlights

QF-Test of QFS convinced us:

Quick access to the tests
Good communication with QFS
Intuitive operation of the software
Good documentation with useful examples: find the right one super fast, directly on the web or as a pdf in the software
Workshops that still have an effectiveness-enhancing effect even after many years of using QF-Test

Many thanks to the QFS team for the good cooperation so far. We look forward to the future together.

Jo Meyer, pds GmbH, Rotenburg, Germany

Regression tests for Java Swing

2023-01-01T00:00:00+01:00

About Planon

Planon provides software and consultancy services to enable companies to manage their facilities and real estate assets efficiently and intelligently. A steady growth has made Planon one of the most successful and fastest growing IT companies in the world. We are recognized as the number one provider of Integrated Workplace Management Systems in Europe. We believe that superior technologies is one of the key ingredients for building an even stronger international market leadership.

QF-Test at Planon: daily regression tests of the Java swing client

What is QF-Test’s role in our software development process? We use the Agile method to design, build and test all our software products. This method is characterized among others by daily communication between all involved disciplines (product owners, designers, developers, testers, technical writers) and short monthly iterations. The main goal of each iteration is to deliver compilable software. QF-Test plays an important role in our daily regression tests of the Java Swing client. Every morning QF-Test presents us its latest results of the code that was compiled the evening before.

QF-Test is highly appreciated because of the following features:

The tool is very powerful in its recognition of components, making the tool suitable for regression testing of applications under development.
Minimal maintenance needed to update tests to handle changes in the client interface.
Test scripts are easy to execute and replay is fast.
QF-Test offers good test management features, which allowed us to have a well structured and documented set of test ware.
Log files are clear and complete, making analysis of results an easy job.
The ability to extend the tool with Jython.
The tool is very easy to learn by new testers.
Good, fast and competent support.

In short, QF-Test is a powerful automated testing tool and plays an important role in our method to create high quality software.

René Hegeman,Technical Test Engineer, Planon B.V., Nijmegen, Netherlands

Successful switch from Java Swing to Web UI Test Automation

2022-01-01T00:00:00+01:00

Atruvia, formerly called Fiducia & GAD IT AG, is a service provider for information technology within the cooperative financial services network. The German company with its administrative headquarters in Karlsruhe and Münster and branches in Munich, Frankfurt and Berlin employs almost 5,600 people that generate annual group sales of about 1.3 billion euros.

Clients of Atruvia include all of the approximately 1,000 cooperative banks (Volksbanken and Raiffeisenbanken) in Germany as well as several private banks and companies from other sectors, including the ADAC (This is a famous automobile club in Germany). Atruvia serves more than 164.000 banking workplaces, administrates nearly 79 million customer accounts and ensures a smooth cash supply nationwide with about 36.000 self-service devices.

Atruvia operates the two bank procedures “agree21” and “bank21” in its high security computer center.

QF-Test at Atruvia

QF-Test together with Atruvia framework

We have been working intensively with the company QFS for many years. We use QF-Test in combination with a self-developed framework which replaces the capture process of QF-Test. This framework abstracts the script operation as well as the object recognition from the tool specific code.

By using this framework it was possible for us to switch from the tool of another supplier to QF-Test with very little effort and without any interventions to the in this way abstracted test cases. The reason for the migration was the requirement of being able to execute tests also in a Linux environment. This wasn’t possible with the tool we used before. Because of the extreme flexibility of QF-Test (Integration of Jython and Groovy, customizability of object recognition etc.) the adaption of the framework code was quickly feasible.

Transition from Java swing to web application

Within the planned switchover from our Java Swing based main application “agree” to a web application we faced the challenge to implement the UI automation also for JX-browser components last year. JX-browser is being used as a swing embedded browser during the migration. Concerning this challenge it turned out that the cooperation with the company QFS was outstanding. We received individual responses to emerging issues as well as fast and competent custom implementations – great praise and many thanks to the employees of the company QFS.

(Original German texts and citations are translated into English.)

Robust test automation with QF-Test

2022-01-01T00:00:00+01:00

For the development and execution of automated test-cases for our customer (HP / HP OpenView Software) a test tool was to be selected which had to be well-suited to testing of Java UI applications. The requirements that had to be met are:

Support for the whole Software life-cycle process: Configuration management, test case management, refactoring, component Tests, system Tests
Support for the platforms MS Windows, HP-UX, Linux, Solaris, AIX
Development of robust test-cases: Easy to capture, easy to execute, robust recognition of graphical components even without ‘object.name’, capturing and test-case development on a single platform, execution of the test-cases on multiple platforms (MS Win, Linux, HP-UX, Solaris, …), execution of captured tests on a different platform, support for localized test-cases, easy to extend
Easy to use and / or well-known language for test-cases (Java, Perl, Python, …)
Batch Mode (unattended execution)
Sensible licensing (to be used in complex test environments with firewall, proxy ?)
Simple installation of the tool: Simple mechanism, little or no dependencies
Easy and complete evaluation of test results
Support: prompt & competent support for questions concerning development and execution of test-cases, short & reliable development cycles

QF-Test at hp / daemons point: low maintenance effort

Unfortunately, many of the well known tools of the more renowned vendors met only parts of the requirements. Especially remarkable during the evaluation was QF-Test’s ability to run tests on different platforms. The standard recognition of graphical components is already very powerful. By making use of the Name- and ItemResolver extension API it should be possible – even in intractable situations (none or inconsistent Object.Names) – to create a robust test-case that will run reliably even in localized environments.

One can see clearly that the features of QF-Test have been implemented very practically. Only thereby is it possible to very efficiently create test-cases, which – during the usual changes of software throughout its life-cycle (a number of years in the current case!) – can always be adapted to the latest version with low maintenance overhead. Additionally, QF-Test’s plugin interface opens the way for integration into the existing distributed test framework so that development of a lot more complex test-cases should be possible.

Therefore, QF-Test came off as the definite winner of the evaluation. The fact, that its license and support costs for development and execution were the lowest wasn’t even taken into account.

(Original German texts and citations are translated into English.)

QF-Test replaces Selenium at HPV

2022-01-01T00:00:00+01:00

About Hamburger Pensionsverwaltung

Hamburger Pensionsverwaltung eG (HPV) offers an extensive pension service. It advises its members in the area of company pension and creates new pensions plans and reports for balance and forecasts. HPV administrates for its member companies entitlements and pensions and also whole pension and provident funds. HBV oversees half a million pension promises assets of more than seven billion Euros.

About Selenium

About 10 years ago we used Selenium for testing our portal applications. The following advantages and disadvantages rested in mind:

Advantages: It was possible to record and replay tests via capture-replay and without scripting. Also complicated scenarios were testable. Tests could be copied and via editing new data sets could be included. In case of small adjustments within the program sequence, we could add the new partial sequences via clever copy/paste to our existing test.

Disadvantages: Unfortunately you can only repeat the tests with the same set of test data that was used for the recording. Generally die tests had to be recorded again. We had performance problems during test execution. We could only run the tests in slow mode (The question remained open whether this was depending on our infrastructure or our own application). After two mayor versions we abandoned Selenium due to this high maintenance effort.

About QF-Test

In 2010 one of our colleagues mentioned QF-Test. We then could test our portal applications comfortably. We test also our Java basted inventory management system with QF-Test. Since then we can also execute consistent tests between the inventory management system and the portal applications. Actually we are considering applying QF-Test also for our via typo3 updated website.

Editor and contact person: Attila Enez, Tester
Hamburger Pensionsverwaltung e.G.
Adenauerallee 21
20097 Hamburg
Germany

(Original German texts and citations are translated into English.)

Kolumbus Software automated with QF-Test

2022-01-01T00:00:00+01:00

About innovas GmbH

Since 1990, innovas GmbH develops individual and standard software solutions with high customer satisfaction and advises customers organizationally, technically and in concrete project environment. We understand that innovation as an envisaged long-term process requiring foresight and clarity in sensitive insurance environment. Therefore, our business model is based essentially on the integration of technology and professionalism which steadily is merging in an increasingly complex business environment. Our depth of services ranges from the design and restructuring of complete administration systems to medical accounting control up to the support of new approaches to care management.

QF-Test at innovas: Management of hospital cases

The standard software KOLUMBUS serves for examination and management of hospital cases and is used by over 100 cost bearers. The examination is performed according to formal and medical-content criteria.

We use QF-Test since the beginning of 2015 for daily UI regression testing of our software KOLUMBUS DRG. The employees in the Automation Team are belonging representatives both from the Test Center as well as the Product Development.

For us, the use of QF-Test has especially the following advantages:

Intuitive ergonomics
Readable HTML reports
Java infrastructure
Easy integration into our infrastructure
Continuous Delivery: nocturnal install and run of automated suites
Implicit test of DB scriptsand installation
Quick expansion in adapted versions via properties files
Creation of a Component Library
Expandability through custom scripts and libraries
Integration MS-Excel for test data management
Detection of manually not traceable problems (e.g. in the area of the Java event handler)
Automatic start of batches
FTP Upload
Integration of Web services
XML schema validation
Edifact generation
Quick and good support (solution always at the next day)

Also your great training, which was rated very positively by all employees, must be mentioned with praise.

So in total we are very satisfied and looking forward to more automation projects with QF-Test.

Author: Jörg Riebschläger, Test responsible KOLUMBUS

innovas GmbH
Theodor-Heuss-Ring 19-21
50668 Cologne

(Original German texts and citations are translated into English.)

Test scaling via Dockers/Jenkins

2022-01-01T00:00:00+01:00

QF-Runtime scaling via Jekins/Docker incl. license

Janitza electronics GmbH is an internationally operating, dynamic family business seated in Lahnau, Hesse. Founded in 1986 and having around 200 employees, it is one of the leading companies offering complete solutions relating to energy management, electric power quality monitoring and differential current measurement (RCM). Janitza focusses on the development and manufacturing of high-quality measurement devices with GridVis®, a software specifically adjusted for this, as well as an extensive range of services. Janitza is considered one of the market leaders in this branch of technology, with distribution channels on over 60 countries.

Janitza has been using QF-Test extensively over many years to live up to the constant growth and guarantee a high bar of quality at the same time.

Starting with a Swing application, the technology switches to JavaFX, JxBrowser and web applications were also covered effectively with QF-Test.

To guarantee quick responses despite increasing tests, we at Janitza rely on scaling via Docker.

The first requirement to parallel execution is as many little test suites as possible, sharing one common base. With the help of the JobDSL-Plugin a suitable Jenkins job is created automatically for every test suite. The Jenkins-Docker plugin then creates the Docker agents appropriate to the load and distributes them to the Docker hosts. This scales very easily with the addition of further Docker hosts.

The Docker image of the agents is an in-house development, using a regular Ubuntu image (Phusion/baseimage) plus necessary packages and SSH keys.

To ensure the surface tests working, the slave starts on a virtual screen (Xvfb).

Some test cases however are dependent on a Windows environment. These tests use a different label in order to run on stable Windows agents.

The agents all use the same QF license server.

Tristan Himmelreich
Diplom-Informatiker | Research & Development

(Original German texts and citations are translated into English.)

Keyword-driven Testing with QF-Test

2022-01-01T00:00:00+01:00

QF-Test at the Austrian insurance for the public sector

Initial situation

In 2013 the Österreichische Beamtenversicherung – ÖBV (Austrian insurance for the public sector) decided to introduce an IT system for the core processes of insurance inventory management and insurance benefit processing tailored to their needs. The Java Swing/AWT based application was implemented in cooperation with an external software service provider and had its go live in July 2016.

The plan was to have short release cycles after the go live to be able to react quickly to changes and new functionalities.

Short release cycles generate substantial test effort:

Approval tests for implemented tickets
Regression tests for quality assurance of existing functionality
Sanity & smoke tests after deployments in test and quality assurance environments as well as in production.

The specialist departments cannot permanently provide the necessary resources for full test coverage to take on all tests.

The IT department is responsible for operation of the test automation tool QF-Test.

Objective

Due to the highly insurance-specific knowledge required for the design of the automated tests, the question arose: which department was to implement the tests?

Should the specialist department acquire knowledge about the test tool or
Should the IT department involve employees with specialist insurance knowledge?

The ÖBV chose the following solution: The IT knowledge stays within the IT department and the specialist knowledge within the specialist department coordinated by the test automation coordinator. The handling of the interface has to be very easily on the specialists’ side.

Implementation

Due to scarce resources the test automation coordinator is supported by Q_PERIOR for test management and Erler Consulting International for test engineering.

Test automation coordinator between specialist department and IT

The specialist insurance department designs test cases including test parameters that are forwarded to the test automation coordinator. Based on these he creates the tests in Excel files that are processed by the test automation tool QF-Test executing the keywords with their parameters.

Technical realization

A technical solution was chosen that

enables non-technical users to implement test cases in Excel, to copy them etc.
covers, respectively handles, all screens of the insurance software
can switch easily between the test environments.

The tests are keyword driven. All user actions within the application (e.g. mouse clicks, text input, text checks) are mapped in QF-Test to procedures. Test engineering provides the test automation coordinator with keywords who then puts them together in test cases.

The single test cases are assembled in further Excel files to test scenarios. These test control files define which test case is executed in which test scenario.

The keyword specific parameters can be entered either directly with the keyword in the test case file or in a separate excel file fora whole dialog of the application.

For the result file, date, time and the test result of the each test case are added to the test control file. Additionally, QF-Test creates a HTML report with graphics of the overall result. Detailed QF-Test run logs can be used by test engineering for error analysis.

Conclusion

This solution helps every department fulfil its specialized tasks. In the first step all smoke tests were automated (tests for basic functionality coverage of the application), which up to then had been done manually by the specialist departments.

The advantages for ÖBV are:

Savings of 5-10 man-days per release compared to manual testing
Stable quality
Automated generation of test data

Step by step, further tests will be implemented to be runas regression tests after each software deployment.

Test management: Q_PERIOR AG
The IT consulting house Q_PERIOR supported concerning test management. This covered the deduction of the test strategy from the risk view, the creation of the test concept, all tasks such as design of test plans and their implementation, planning and control of all tests as well as coordination of the test case design, test execution and test resources.

Test engineering: Erler Consulting International GmbHErler Consulting International GmbH, represented by Ute Erler, was responsible for test engineering. This included design and implementation of the keyword processing, programming of the keywords and structuring of the test scenarios. In ongoing operation she is assisting with error analysis of the executed test cases.

(Original German texts and citations are translated into English.)

Migrating Java Swing automated tests to JavaFX

2021-01-01T00:00:00+01:00

About Centris AG

Centris AG is a service provider for Swiss health and accident insurances and supports them implementing new IT solutions. The software Syrius used at Centris AG is a Java application and is being migrated to JavaFX.

Initial situation: Swing – Setup / structure

The Centris AG QF-Test test framework is designed that every (specialist-) logical test procedure of an application can be mapped easily in a data driver. Thanks to a freely selectable distribution of the source data insertion in a navigation and/or data insertion section. The process logic with the appropriate insert values, written in Excel tables (data driver) correspond 1:1 the UI design respectively naming of the components within the application.

The prerequisities are:

Implementation of generic components for every relevant class (about 30: from genericTextField to genericMultiChoiceComboBox up to genericList).
Implementation of name resolvers to determine unique and UI appropriate component names for test fields, check boxes etc.

Swing to JavaFX migration

The central tasks for porting from Swing to FX was:

Adaption/Extension of existing components and implementation of new generic components
New name resolver for JavaFX for naming the components to be accessed and the realization of component recognition.

Conclusion

The adaption expenses of this transfer were relatively low. There were just a few changes within the basic structure, except the migration of certain limited Jython/Groovy scripts (utilities) to JavaFX. We could implement the resolver relatively smoothly with only a few challenges which we overcame with the assist of QFS, e.g. the subsequent integration of the missing “imagehashes” in the extra features.

Outlook

The migration (of about 80% so far) has been lasting about one “man-month”. The finetuning e.g. of timing is a permanent tasks.

The last 20% are being migrated permanently and we feel confident with our gained experience not facing greater difficulties or something unexpected.

The so far rather easy migration confirmed the test concept of Centris AG and the selection of the proper test automation tool.

This is the reason why we are very optimistic that we will soon conclude this migration also with the very competent, fast and friendly support of QFS and skip to maintenance mode.

Future

Java 9… we’ll be surprised!

(Original German texts and citations are translated into English.)

Transition from manual to automated testing

2021-01-01T00:00:00+01:00

About CertiCon

Since 1996, CertiCon has been improving the competitiveness of our clients around the globe. CertiCon’s B2B solutions comprise the full spectrum of services and solutions in software development and quality-assurance verification testing. Our experience in the fields of medical and mission-critical applications and high-tech industrial applications enables us to harness the best practices and solutions for our clients. CertiCon designs, develops, rapid-prototypes and deploys sophisticated software solutions. Our projects and solutions cross medical and industrial domains leveraging cutting edge state of-the-art methodologies and technologies.

QF-Test used by CertiCon

The project

The project is a sophisticated vehicle diagnostic solution which integrates existing applications, extended applications, and new applications and processes, into a common system. The potential clients are authorized dealerships and freelance workshops.

From the end-user’s perspective, the application eliminates multiple user inputs with the implementation of single takeover and single sign-on mechanisms. The initial goal of the project was to create a lean solution that delivers competent diagnostic results to its end-users.

The challenge

Release after release, the application grew with the addition of more features. Maintaining the defined quality assurance process was a demanding task.

With over 2,000 manual test cases (increased with every release), we faced the challenge of reducing the time needed for testing, and in the process, continue to add new test cases and improve the quality of the software overall.

Solution

With a stable testing environment, automating the testing process was the answer to our needs. The proverbial tool of choice was QF-Test by Quality First Software GmbH, a licensed professional tool for regression and load testing of Java and Web applications with a graphics user interface (GUI).

The automation process included the following stages:

Writing initial testing scripts, common for all test cases.

Reusability

Recording individual test cases as separate sequences.
Executing the testing scripts.
Amending testing scripts after the fixing of defects revealed in the application during script execution.
Preparing test suites for daily regression testing executed on the latest release.

Results: Reduced testing time by almost 50%!

In a matter of months we succeeded in reducing the test processing time by almost 50%, adding hundreds of additional test cases, with the test cycle and schedule remaining intact.

As depicted by the general figure, although the initial investment on additional resources was high, in the long run, the time and cost of software testing was reduced and the addition of test cases and increased regression testing brought with it improved software quality.

From Winrunner to QF-Test

2021-01-01T00:00:00+01:00

About CoreMedia

CoreMedia is the leader in innovative people-centric social software. The company, based in Hamburg, Germany, has successfully positioned itself in the highly-dynamic market for Internet technology. The company is characterized by its innovative and creative culture, and by the focus on collective learning. These values are also to be found in CoreMedia’s technology: CoreMedia develops software that enables staff, partners and customers to achieve their goals rapidly, creatively and productively.

Founded in 1996 and headquartered in Hamburg, Germany, CoreMedia has over 150 employees and maintains a global network of implementation partners that includes 500 professional consultants.

ÜOver 200 companies worldwide rely on CoreMedia’s content management (CMS) and digital rights management (DRM) solutions, including ARD, Bertelsmann, BenQ, BILD, Continental, DAK, DaimlerChrysler, debitel, Deutsche Telekom, GMX, IKK, NEC, MLP, Motorola, Nokia, O2, Panasonic, Plus, PREMIERE, Quelle, Qualcomm, Samsung, SEAT, Sony Ericsson, T-Mobile, T-Online and Vodafone. CoreMedia is also the basis for online services provided by over 80 public institutions and administrative bodies.

CoreMedia has received many international awards in recent years, praising its innovative culture and software, including the European Business Award, recognition as Best Innovator, and the inclusion in the Deloitte Technology Fast 50 list for fast-growing companies.

QF-Test at CoreMedia: Robustness of tests important

Excerpt from a facebook community article from Mark Michaelis, Software Engineer Quality Assurance, CoreMedia AG, Hamburg, Germany:

About the poor UI Automation Tools: I also think that UI test automation is perhaps one of the most challenging things to do in testing. It takes a long evaluation to get the right automation tool, especially to get one accepted by the whole team.

For Java (Swing Applications) we switched from WinRunner (used for years before) to QF-Test. WinRunner has not been accepted by the team because of its proprietary language TSL. In contrast QF-Test is developed in Java and uses Jython as scripting language. A great start for Java developers. There are many more benefits of QF-Test, but I will only state three of them here:

As QF-Test is written in Java it has no problems dealing with the latest Java releases. It’s been a piece of cake to switch from testing with Java 1.5 to testing with Java 1.6
Tests can be written in that way that also managers can easily read what the test actually does without going into details.
Great support for Data Driven Testing.Never got so many test cases done in such a short amount of time..

About the robustness of UI test automation: I think there are two things which need to be done to get the tests as robust as possible. One is already mentioned in the StickyMinds article you quoted: Involve developers. It’s for sure needed to add test-hooks into the application. The most simple one is to give UI elements a unique ID (in Java: using setName()).

The next thing is to have UI Maps (a term I learned from WinRunner but I also adopted it for QF-Test): Place the rules to identify UI components in an own database/file. This way you only have to touch this file for slight UI changes.

But the best thing to do is to create a testing framework. E. g. for every menu action (every in XP sense: every currently used one) I created a library function. Developers just need to call this library function and don’t need to care about if there are radio-buttons or a drop-down list behind. If the UI element changes its type only the framework needs to be adjusted.

Even more complex functions moved into the framework. E. g. for the CMS I created framework functions which automatically create documents (using the UI). So it’s very easy for developers to get some test fixtures to work with.

Finally, back to the code, I added a UI test facade to the application. This has two advantages:

I can access the very inner of the application.
Developers can easily see that I do this and how and thus can take care of it during refactoring of the code.

Having these three things: UI Maps, UI Automation Test Framework, UI Test Facade I finally got a very robust set of UI Tests.

Additional Evaluation report from CoreMedia

(Original German texts and citations are translated into English.)

Testing the eGECKO Business Software with QF‑Test

2021-01-01T00:00:00+01:00

First steps after training

I have created several universal procedures after the training that cover elementary processes in our software and you then can refer to them in other test cases. Also the webinar about Dependencies helped me to understand this topic and optimize a test.

At the beginning we focused on the accounting field of our software. We are now testing different processes e.g. interfaces, document creation and accounting transactions with different master data or creating invoices. We then also check the created PDF files. That works super!

Outlook

We still face challenges concerning the independency of the tests. Our tests star with an empty data base, the data is created by the tests during the test run and then imported. But, when there is an error in test #5, tests 1 to 4 have to run through first that we can find the error and debug. At this point this is working, but with an increasing number of tests this may become difficult due to lack of time. We should maybe save the data base after every test case, but this would have a deep impact in the performance.

We automated the tests via our Jenkins with pipelines (not the QF-Test plugin), since we have several setups and cleanups of our tests, this makes the tests better controllable. After the creation of a new branch of our software we just have to change the path to the new program version – We haven’t had other changings or bugs on QF-Test’s side when checking a new version! In the near future we are going to update eGECKO on Java 11, but I am optimistic that we won’t face any problems there.

eGECKO

eGECKO is a modular and very comfortable business software for very different economic processes.

Test results QF-Test eGECKO

Evaluation Report: Testing the business software eGECKO – March 2019, Leonhard Zentgraf, CSS AG.

(Original German texts and citations are translated into English.)

Testing an ERP-application

2021-01-01T00:00:00+01:00

QF-Test at eMundo

We have been using QF-Test in our company for a long period for testing an ERP-application with several hundred input masks. It is based on a Swing surface and was enlarged by diverse UI libraries: Jide, JGoodies, etc. We are very happy with the functionality of QF-Test for testing an extensive application.

The support of multiple UI elements, also of complex elements like tables, trees etc. is very good. For requirements that go beyond this QF-Test is offering the opportunity of extending the functionality individually by diverse interfaces (using Groovy and Jython as scripting language). These extensions integrate easily in the clear and well-conceived control concept of QF-Test.

QF-Test is robust, also local test runs aren’t disrupted by working at the same time. For the integration into a CI system, in our case Jenkins is used, QF-Test can be executed in batch mode.

QF-Test has an extensive documentation, questions are answered fast by the competent and very attentive support team!

Gerhard Katstaller, eMundo GmbH, Salzburg, Austria

(Original German texts and citations are translated into English.)

QF-Test secures software product for practice management

2021-01-01T00:00:00+01:00

About Epikur

Since 1999, we at Epikur Software have seen it as our task to make the work of psychotherapists and doctors easier. With our practice management system EPIKUR, we support individual and group practices, MVZs, psychotherapeutic training institutes and research outpatient clinics in the organization of their daily processes: We optimize appointment allocation, speed up documentation and improve digital communication between practitioners and patients. We also take care of IT security and the seamless connection to the telematics infrastructure.

Requirements

Our practice management system is based on Java with AWT, Swing and JavaFX for the UI and is designed for use in Windows, macOS and Linux. As a result, there is a need for test software that can map all these aspects at once.

During extensive research, various other products were compared. The criteria were

Price
Platform compatibility for Windows, macOS, Linux
Self-hosting or cloud-only
performance
support
Usability
Integration and version control (Git)
Programming skills necessary?

QF-Test was able to assert itself particularly in the aspect of platform compatibility and the question of the necessity of programming skills.

Implementation

We use QF-Test to map regression tests for the release testing of program updates. Not all manual tests are fully automated yet, but it is already foreseeable that the potential time savings in manual testing will be significant, which in turn will free up time for other types of testing.

For example, there is a use case that takes around ten minutes per operating system and product variant when tested manually and is executed after each build. In automated form, this only takes three minutes. The automation of our test cases also allows us to test the same test cases in two different product variants on three different operating systems with a shorter execution time, which saves a lot of time compared to manual execution.

One challenge in the implementation of our tests with QF-Test is the stability with regard to the retrieval of the display elements, which could be largely solved for the time being by changing the standard waiting times or by assigning additional, dedicated waiting times.

We use gitlab pipelines as test infrastructure, which execute the corresponding regression tests via chronologically triggered schedules. These gitlab pipelines are part of a release testing process that recurs two to three times a week. Due to the fact that we have to test the same test case in two different product variants as well as three different operating systems, even a small number of test cases results in an enormous number (approx. 120) of automatically executed test cases. We currently run these test cases at night after the program build and can evaluate the test results in the morning. This usually takes around five to six hours.

Since QF-Test allows a batch call, i.e. without the need to start from the editor mode, other integrations of the tests are also conceivable, especially in real test management systems.

Philipp Strauß, Epikur Software GmbH & Co. KG, Berlin, Germany

QF-Test at Airbus Defence and Space

2020-01-01T00:00:00+01:00

1 Cross-Platform UI Regression Testing in a Continuous Deployment Chain with QF-Test

Airbus Defence and Space uses the UI test tool QF-Test for daily automated regression testing. The “System under Test” is an application software (“SW”) used to process image, video and motion information in military environments. For this product and its variants, it is ensured that newly implemented changes do not affect existing functionalities. The architecture of the SW together with QF-Test allows a cross-operating-system approach of “write once, test anywhere” to specifically check the quality of both the standard product and its variants.

2 QF-Test at Airbus Defence and Space

2.1 Test Project

The application software developed by Airbus Defence and Space is based on a client-server architecture. The client is based on Eclipse RCP and can be started under RHEL in two different modes. The backend is an OSGi server solution and is supported for two operating systems: RHEL Server for the standard product and Oracle Solaris 11 for a product variant that is already in operational use.

Based on new customer requirements, the standard product is further developed, compiled in a daily build process and distributed to both platforms. As part of this continuous development chain, automated UI regression tests built with QF-Test are run to check the integrity of the client-server interfaces and their basic functionality. This ensures that any misbehavior following the implementation of changes can be detected early on and quickly remedied.

2.2 Test Architecture

The continuous deployment chain for the SW product and its variant is illustrated in the figure above. In the build and deployment process, both the SW to be tested and QF-Test are initially packaged and stored in a repository. In addition to QF-Test, the QF-Test package stored in the NEXUS repository consists of two other parts, a product library (Product Library.qft) and a corresponding test suite (Smoke-Test-Suite.qft) for each platform. The product library contains all records to access graphical elements of the Eclipse RCP based client: menus, parts, tables, buttons, sliders etc. The Windows and Components Tree is also stored here, which together with a one-to-one component identifier allows access to the graphical elements of the client. The dynamic flow of the tests varies between the two server solutions (RHEL and Oracle Solaris) and is adapted accordingly in the respective Smoke test suite for each platform. The test suites implemented here are started daily after installation and make use of the basic functions stored in the product library. First, test suites are executed to configure the data transfer between server and client (ports, IP addresses, protocols). Then, the correctness of the implemented interfaces between server and client is verified for each platform by simulating and automatically checking a simple data exchange (images, videos, motion information). The use of Data Driver/Tables facilitates the execution of the same test sequence with different configurations, while keeping the test code maintainabe. The individual procedures are encapsulated in simple try/catch blocks, so unforeseen errors can be easily caught and logged.

Running about 100 tests automatically and in parallel takes about one hour. Running the tests manually used to occupy one person for three hours per configuration. For the standard product and its configuration (two modes), this results in a saving of currently 8 hours per day, time which we happily invest in further automated tests or other activities.

2.3 Conclusion

Through this approach of ”write once, test anywhere”, we found a big amount of errors that were related to the configuration of the application on the respective operating system. Overall, test automation has a very positive effect on accelerating the development process, since test results are available quickly, and time-consuming manual tests can be eliminated. Furthermore, successful automated test results are considered a handover criterion for the installation and configuration of the SW at the next integration stage.

QF-Test as a tool for UI test automation is designed for cross-platform use and is very well suited for our technical needs. QF-Test is easy to learn and we were able to integrate it into our operating environment without any problems. Particularly noteworthy is the ability to add the included pixel-based image comparison algorithms for images and videos to the test flow. In addition, we took advantage of QF-Test’s exception handling in the flow control of the individual test suites. Finally, the very fast and reliable German support from the manufacturer also helped us out.

2.4 Outlook

For the targeted testing of the product variant already in operational use, further regression tests are planned, which are to check frequently executed parts of the system. This involves the simulation of long-running and asynchronous processes as well as investigations of the handling of large data volumes in order to draw conclusions about the performance (CPU load, memory requirements) of the system. Here, too, the already existing product library can be used, only a new regression test suite has to be created that contains the new use cases. With this planned approach we hope to further increase the quality of the SW and the minimization of the risk that errors are only found at a late stage and are therefore much more expensive to fix.

Author: Thomas Schöning
ISTQB Certified Test Manager
Airbus Defence and Space GmbH
Multi-INT Products & Projects Germany

(Original German texts and citations are translated into English.)

QF-Test at Amadeus Airline Software

2020-01-01T00:00:00+01:00

About Amadeus

Amadeus is a leading transaction processor for the global travel and tourism industry, providing transaction processing power and technology solutions to both travel providers and travel agencies. The company acts both as a worldwide network connecting travel providers and travel agencies through a highly effective processing platform for the distribution of travel products and services and as a provider of a comprehensive portfolio of IT solutions which automate certain mission-critical business processes.

QF-Test at Amadeus: UI Tests of special Swing components

As part of its historical offer, Amadeus proposes a suite of products for Airline IT named Altéa. Based on highly-customized Swing components, its UIs are deployed on hundreds of thousands of workstations around the world.

In order to ensure a sustainable high level of quality, we carried out in 2005 a study comparing seven UI test tools, including QF-Test, which eventually ranked first for the following main reasons:

The most powerful and reliable component recognition: based on Java Swing specific heuristics, QF-Test suite requires less maintenance than the other tested products.
The speed, at which QF-Test is able to play a scenario is quite impressive: it allows on-time delivery without compromising quality.
The graphical record-replay paradigm coupled with Jython (and now Groovy) procedures: grants user-friendliness for users without developer knowledge and the power of shared advanced custom features.

Our offer and our needs then evolved, and so did QF-Test:

The daemon mode – allowing workstation clustering and UI re-using – reduces drastically the amount of time needed for a full regression run.
QF-Test/web the cross-browser web module on top of QF-Test regression core engine, allows us to test our web applications without losing our QF-Test local expertise.

Only one thing did not change over time: the reactive and efficient level of support from the authors. The QF-Test team showed us many times they were able to react in a quick and accurate way. This is definitely what makes the difference between good and excellent products.

Our IT solutions are now used all over the world by the tourism and travel industry protagonists, and QF-Test never stopped helping us improving quality, even through drastic changes like:

Look&Feel migrations.
Java switches (from Java 4 to Java 6).
Architectural and technology-related changes (in-house Java Swing and web frameworks).

Able to cover important parts of both the Java and web worlds, easy to use and understand with its complete and up-to-date user manual, not forgetting the dedicated support and optional on-site training, QF-Test suite is now the offical Amadeus regression testing tool, used worldwide by our developers and functional analysts.

The APIs and modularity (daemon mode, custom reporting, shared libraries…) allow local initiatives which sometimes eventually end up as company-wide solutions or best practices, like the Windmill framework, dispatching test run commands 24/7 over a cloud of dozens of workstations.

The tool and the team behind QF-Test never disappointed me, so I wouldn’t hesitate in recommending QF-Test to anyone developing Java or web applications.

Laurent Dollé
Software Engineer, Java Front-end Middleware, Amadeus IT Group SA – amadeus.com

Critical banking software migrated from Java Swing to the web

2020-01-01T00:00:00+01:00

Migrating to WebSwing – Tests included!

B+S Banksysteme Aktiengesellschaft, listed on the Frankfurt Stock Exchange, is a software manufacturer for banks and financial service providers that was founded in Salzburg in 1982. The company’s headquarters were moved to Munich in 2008, with further locations in Austria and Switzerland.

With our team of over 100 experts, we meet the extensive and multi-layered challenges of the financial world.

The team structure with professional and technical specialists makes it possible to develop outstanding products for demanding customers and, if desired, to operate them in our own data center. The close cooperation with QFS has for many years enabled us to efficiently integrate all development technologies used in intensive and multi-layered automated test procedures.

QF-Test at B+S Bankensysteme

B+S has been using QF-Test since 2006. We started by setting up automated regression and integration tests for our banking software, its user interface having been developed in Java Swing. The easy development of test procedures and test cases convinced us very quickly and also ensured a very high level of acceptance among the employees, both those who develop the tests as well as those who carry them out. We especially want to emphasize the stable component recognition despite continued development of the interface elements and the Java versions. Through a merger with an e-banking manufacturer, we expanded our testing portfolio to include native web applications, which later also found their way into the classic banking software.

QF-Test is used throughout the entire group and is integrated in many stages of the development process:

Development and integration tests
Regression tests (continuous deployment, nightly builds)
Tests of customer-related software versions before delivery
Additional unit tests with integration in Jenkins

Deployment of Java Swing clients (to end users)

Many B+S customers use other software such as Citrix or similar tools for the deployment of Java clients in order to avoid installations on the workstations. After an increasing number of enquiries as to whether the application could also be made available to the end users in the browser, a preliminary in-house study of available options began. Since a development as a native web application was ruled out due to high costs and long development times, the solution WebSwing was examined more closely. WebSwing enables the presentation of Java Swing applications as websites via middleware and thus a very simple distribution to end users, supplemented by a central administration and installation option. During the proof of concept, the questions of the testing possibilities of the new interface were also analyzed. It quickly turned out that QFS was already working on the topic with WebSwing – the informative webinar Integrated testing of Webswing applications with QF-Test helped us in quickly understanding the new context and ways of doing things. Particularly important for B+S was the reusability of our extensive QF-Test test cases and procedures that had grown over the years.

As this need was also met, WebSwing ended up beeing included in our service portfolio as an additional deployment option and is in use in the B+S data center since 2022.

Tests of Java Swing with WebSwing

Two challenges related to testing with WebSwing needed to be solved:

Tests of development-related stages are carried out locally. The WebSwing server is started by the testers. Therefore, QF-Test needs to start the WebSwing instance.
The first challenge was quickly solved after briefly contacting Gregor Schmid from QFS. We received a pre-configured test file and were able to quickly incorporate our (minimal) changes into the procedures to start the WebSwing server and browser locally. We were pleasantly surprised that all existing tests continued to work seamlessly. The component recognition is extremely reliable and stable, all IDs and recognition features remained unchanged – the tests were immediately ready for use.
Further test stages run on dedicated test environments that are updated automatically. The complete client/server application is continuously run, the services of the servers are managed centrally. To make matters worse, the test servers are clustered and load balanced, several test environments running in parallel on the cluster. QF-Test is therefore not able to start the WebSwing instance itself.
A little bit more attention was needed to solve the second challenge. Here, too, the necessary changes to QFS were pointed out or files were transferred. For example, QF-Test (Linux) must be installed on the WebSwing server and must be kept updated to match QF-Test on the client. After the implementation of the specifications, however, it turned out that the components are recognised as web applications, which results in new IDs for the Swing components.

In a remote maintenance session, the problems were identified with Gregor Schmid and solved right away. For example, several ports had to be enabled on various firewalls. Ports must also be taken into account in the QF-Test start script. The integration of QF-Test with WebSwing on the server also had to be slightly modified in several places due to our server structure.

Thanks to the excellent support from QFS, for which we would like to express our sincere thanks, the second challenge was also solved in a stable manner. The existing tests are now running here without any significant need for adaptation, too.

Mag. FH Jürgen Macherhammer, Project Manager, B+S Banksysteme Salzburg GmbH, Austria

Comprehensive regression tests for complex client architectures with QF-Test

2020-01-01T00:00:00+01:00

About BITMARCK

As a managed service provider in the statutory health insurance IT market, BITMARCK is driving digitalization in the industry and for its customers with innovative products, solutions and services. The basis for this is the statutory health insurance software standard BITMARCK_21c|ng, which is used by the affiliated health insurance companies. The group’s customers are the company and guild health insurance funds as well as DAK-Gesundheit and other substitute health insurance funds – more than 30,000 employees and around 25 million SHI policyholders benefit from BITMARCK’s IT services, and more than 80 percent of German statutory health insurance funds are customers of the group. With around 1,800 employees, BITMARCK achieves an annual turnover of 393 million euros.

The challenge: Comprehensive regression tests for complex client architectures

BITMARCK Software GmbH was faced with the challenging task of ensuring the consistent quality and stability of the core system BITMARCK_21c|ng of its complex software solution for social insurance. In particular, this included comprehensive regression tests in the SHI software standard BITMARCK_21c|ng for various client architectures.
Both the web client and the swing client had to be secured with an extensive test set. The management and optimization of 4,500 currently running regression tests across these different client types, as well as the planned expansion to mobile platforms, required a future-oriented and flexible test automation solution.

The solution: QF-Test and the jointly developed multi-client recorder

To overcome these challenges, BITMARCK Software GmbH decided to use QF-Test, one of the leading tools for the automation of UI tests. In close partnership with the manufacturer of QF-Test BITMARCK Software GmbH developed an innovative multi-client recorder. This recorder was specially designed to optimally support test automation in the BITMARCK_21c|ng core system for both the web client and the swing client. The multi-client recorder enables highly efficient and precise recording and maintenance of the 4,500 regression tests currently running across all relevant client types. By combining QF-Test and this customized recorder, BITMARCK Software GmbH achieved a new dimension in test automation.

The advantages

Cross-platform efficiency in the SHI software standard BITMARCK_21c|ng
The jointly developed multi-client recorder optimizes test automation for web client and swing client in the core system BITMARCK_21c|ng.
Management of 4,500 regression tests for different clients
QF-Test and the recorder enable efficient management and execution of 4,500 running regression tests across web and swing clients.
Intuitive recorder for accelerated test development
The multi-client recorder simplifies the creation and customization of test cases for both client types and the native apps (iOS and Android) and significantly reduces maintenance efforts. In addition, the recorder can be used by specialists without the need for extensive training.
Maximum software quality across all clients
Thanks to the comprehensive regression test strategy with QF-Test and the recorder, BITMARCK Software GmbH ensures the highest quality of its software solutions for statutory health insurance across all client architectures as well as native apps.
Continuous Delivery
4,500 automated regression tests in the CI/CD pipeline ensure software quality with every daily run.

Outlook: Mobile test automation and end-to-end (E2E) tests

BITMARCK Software GmbH is continuously expanding test automation with QF-Test. Test coverage with the intuitive recorder is currently being extended to iOS and Android apps. QFS has already made great progress in the iOS area, which makes mobile test automation with QF-Test particularly promising. As a next step, BITMARCK is aiming to expand end-to-end testing between the native apps (iOS and Android) and the GKV software standard BITMARCK_21c|ng in order to comprehensively ensure the end-to-end functionality and interaction of all components.

The successful cooperation with QFS and the continuous further development of test automation with QF-Test underline the claim of BITMARCK Software GmbH to continue to set the highest quality standards in software development in the future and to actively shape the digital transformation of statutory health insurance.

The cooperation between BITMARCK Software GmbH and QFS is characterized by a close and solution-oriented partnership that enables BITMARCK Software GmbH to efficiently master even complex test automation challenges. When problems arise, BITMARCK Software GmbH benefits from the fast and competent support of QFS, which contributes significantly to finding solutions.

Dirk Kruck, Test Engineer, BITMARCK Software GmbH

Migration from Java into the Web at Experian

2020-01-01T00:00:00+01:00

About Experian

Experian is the world’s leading global information services company. During life’s big moments — from buying a home or a car to sending a child to college to growing a business by connecting with new customers — we empower consumers and our clients to manage their data with confidence. We help individuals to take financial control and access financial services, businesses to make smarter decisions and thrive, lenders to lend more responsibly, and organizations to prevent identity fraud and crime. Forbes has named us as one of the “World’s Most Innovative Companies”.

Desktop Java Swing client as a web application

Some time ago, we decided to transform one of our desktop Java Swing applications into a web application. We found nice and easy way to do that, by using already-available third-party software, and we decided to go with it. The difficult part was the testability of the new web solution. We explored several options to be able to create automated tests, but none of them was sufficiently robust or promising. Then we looked at it from a different angle – “It’s the same swing application we have underneath, and we have a great automation framework and tests suite using QF-Test. Can’t we test it through the web with the same tool?”. We knew it wouldn’t be possible with the available features of the tool. We also knew the great customer focus the QFS team has, the responsiveness and spot-on solutions of their support to some of our seemingly wild ideas or complex problems. It was just another wild idea, and we went with it to QFS, asking them for what seemed like an imaginary, magical solution to us back then…

…and it was worth it. Our requirements have been taken into account really quickly and one of the best technical experts in the QFS team started working on this personally. It required a couple of iterations on a proof of concept implementation that we tried out, and a great collaborative effort between our technical specialists and the QFS team, before we started to believe that this implementation could actually cover our needs.

Now we have 95% (10,000+ tests) of the existing suite of test cases for the Swing version of our software, reusable for the web version. We achieved this with very minimal rework, and it saved us a huge amount of effort for creating tests from scratch. These tests are giving us very high confidence that each new build of the web version is with the required quality.

Thank you QFS team!

P.S.: The QFS team would also like to thank you for the constructive cooperation since 2009, be it with the Swing applications or the web solutions through the tool integration of Webswing and QF-Test.