QF-Test - blog

Using GitHub Copilot CLI via Remote SSH from QF-Test

2026-04-01T00:00:00+02:00

With QF-Test 10, it is possible to integrate third-party AI models such as GitHub Copilot into QF-Test. In the case of Copilot, it is usually necessary to have the Copilot CLI installed on the same system as QF-Test.

However, there are situations where you cannot install the Copilot CLI on your test machines for technical or security reasons, or if you want to run multiple parallel QF-Test instances via Daemon mode.

For such cases, this guest article by long-time QF-Test user Irfan Yigit shows you how to set up Copilot on a single central machine and establish an SSH network connection to your test machines. You can then use your GitHub Copilot AI models with QF-Test via SSH.

Prerequisites

To set this up on a Windows system, OpenSSH is required. OpenSSH must be installed on both the client VM and the target VM.

The client VM additionally needs QF-Test (10.0 or newer) installed. The target VM must have a current GitHub Copilot CLI installation, and the GitHub login (/login) must have been completed within the SSH environment.

To use GitHub Copilot CLI via remote SSH reliably, an SSH key is required, because OpenSSH cannot handle SSH password prompts during automated remote calls (such as those triggered from QF-Test). SSH key authentication is needed solely to establish a passwordless, stable SSH connection.

Step-by-Step Guide

Preparation on the Client VM

1. Create the .ssh directory (if it doesn’t exist, e.g. C:\Users\Administrator\.ssh)

New-Item -ItemType Directory -Path "$env:USERPROFILE\.ssh" -Force | Out-Null

2. Generate an SSH key (ed25519) on the client VM (without a passphrase)

ssh-keygen -t ed25519 -C "qftest-remote" -f "$env:USERPROFILE\.ssh\id_ed25519"

Simply leave the passphrase empty (press Enter twice).
The generated public key (…/.ssh/id_ed25519.pub) will be needed on the target VM later.

Preparation on the Target VM

1. Set Access Control Lists (ACLs) correctly

Windows OpenSSH only accepts keys if the file is in the right location and the permissions are set with the correct ACLs. The file C:\ProgramData\ssh\administrators_authorized_keys is created and secured as follows:

icacls C:\ProgramData\ssh\administrators_authorized_keys /inheritance:r
icacls C:\ProgramData\ssh\administrators_authorized_keys /grant 'Administrators:F'
icacls C:\ProgramData\ssh\administrators_authorized_keys /grant 'SYSTEM:F'
Restart-Service sshd

2. Add the public key

Paste the contents of id_ed25519.pub from the client VM into the administrators_authorized_keys file on the target VM and save it (one key per line).

3. Log in to Copilot CLI in the SSH environment on the target VM (once)

Copilot stores its login credentials locally in the respective user profile. The login must be completed beforehand — either directly on the target VM or via SSH, depending on whether different Windows users are involved.

Once the Copilot login is saved in the SSH environment, Copilot can be fully scripted via remote SSH from QF-Test.

Run the following in PowerShell if all client VMs should interact with the same user and SSH environment:

SSH directly to the target VM: ssh -i ~/.ssh/id_ed25519 Administrator@<target-IP>
Start Copilot: copilot
Start login: /login
Confirm the device code in the browser.

Functional Test (Client VM → Target VM, headless)

Run the following command on the client VM in PowerShell:

ssh -i ~/.ssh/id_ed25519 Administrator@<target-IP> "copilot -p 'Say hello' --stream on"

Troubleshooting

Test SSH key login from the client VM

Verify that SSH works without a password using the SSH key (prerequisite for QF-Test):

ssh -i $env:USERPROFILE\.ssh\id_ed25519 Administrator@<target-IP> 'echo OK'

If OK appears → key authentication is working.
If there is an error: check the SSH keys or ACLs on the target VM.

Connect Copilot CLI to QF-Test via SSH

Once everything works, follow these instructions from step 2:

Tutorial: Using GitHub Copilot with the QF-Test AI Integration

Since QF-Test 10, AI-driven capabilities are integrated directly into your testing workflow. This post shows how to connect GitHub Copilot to QF-Test using a simple custom model and the Copilot CLI.

Max Melzer

09/02/2026

Make the following adjustments to the ai.addCustomModel script (insert the correct user, path and IP for your setup):

ai.addCustomModel("GitHub Copilot", (msg) -> {
    def remoteUser = "Administrator"
    def remoteHost = "192.168.0.2" // IP of target VM
    def remoteCmd = "cmd /c \"C:/Users/Administrator/AppData/Local/GitHubCopilotCLI/copilot.exe --silent --stream off --model claude-sonnet-4.6 --prompt \\\"${msg}\\\"\""
    def cmd = [
        "ssh",
        "-o",
        "StrictHostKeyChecking=no",
        "${remoteUser}@${remoteHost}",
        remoteCmd
    ]

    def proc = cmd.execute()
    proc.outputStream.close()
    proc.waitFor()

    return proc.in.text.replace("● ", "")
})

Learn more about the AI integrations in QF-Test 10:

AI-driven test automation with QF-Test

AI-driven testing and intelligent test automation for efficient software quality assurance

How to use the QF-Test AI integration with a free local model using Ollama

2026-03-09T00:00:00+01:00

QF-Test has always been about reliable, maintainable test automation for web, desktop, Java-based, and mobile applications. With the new AI integration in QF-Test 10, you can enhance your tests with AI-driven insights — and the best part is, you don’t need to rely on cloud services. In this post, we’ll show you how to connect QF-Test to a locally running, free AI model using Ollama, so you can start experimenting with AI-enhanced testing right on your own machine.

Step 1: Set up Ollama

To get started, you’ll need to install Ollama on the machine where your tests run. Ollama provides free, locally hosted AI models that powerful enough for the more basic parts of QF-Test’s AI integration.

Download and install Ollama from the official website: https://ollama.com/ (You don’t need any account).
Start the Ollama server by opening a terminal and running:

bash ollama serve
In another terminal, download a model, which typically is a couple of gigabytes:

bash ollama pull llama3.2

By default, Ollama will host the model at http://localhost:11434 as long as ollama serve is running. You can now use this endpoint in QF-Test to send AI requests locally — no internet connection required.

Step 2: Configure QF-Test to Use the Local AI

You can connect the Ollama server to QF-Test convieniently via the QF-Test Options dialog, accessible via QF-Test → Options → AI Integration. Click the “plus” icon to add a new row to the “AI Configurations” table and enter the model details:

Type: “Ollama”
Name: “Local AI” (or whatever you like)
URL: “http://localhost:11434”
API key: “” (leave blank)
Model name: “llama3.2” (the name of a model you added to Ollama via ollama pull)

QF-Test also makes it easy to integrate an AI model via the ai scripting module, if you prefer the scripting approach.

Here’s a minimal example of a server script that connects QF-Test to your locally running Ollama model:

// Configure the AI model
ai.setDefaultConfig("Ollama", "http://localhost:11434", "", "llama3.2", displayName: "Local AI")

// Make a request to the AI and print it to the terminal
print(ai.ask(
  "What is the answer to life, the universe and everything?",
  configName: "Local AI"
))

Let’s break down what’s happening:

ai.setDefaultConfig(...):
This line registers the model with QF-Test. You specify:
- The type of endpoint (“Provider”) to use (Ollama)
- The local endpoint url (http://localhost:11434)
- An empty string for the API key (not needed for local models)
- The model name (llama3.2 in our case)
- A friendly display name (Local AI) for use in the UI
- ai.ask(...):
  This line sends a prompt to the AI model. The configName ensures it uses the local configuration we just defined. The result is returned directly and printed in the QF-Test terminal.

Step 3: Use AI in your tests

Beyond this toy example, you can now leverage your local AI model directly in your test cases:

Check Text with AI
The AI integration can be used in a Check Text with AI node. This is perfect for situations where you want QF-Test to validate text in your application UI dynamically. For example, you could check if a text matches a natural language description or even check if it’s in the expected language. To learn more, see the documentation here: Check Text with AI node
AI Scripting Module
For advanced automation, the ai scripting module gives you full programmatic control over AI requests in QF-Test: AI scripting module documentation

The advantages of running locally

Running a local AI model has several advantages:

Privacy: Your test data never leaves your machine.
Performance: Local inference can be faster than cloud requests.
Cost-free: No API subscription needed for experimentation.

By combining QF-Test’s robust automation framework with AI insights, you can enhance your tests, automate language checks, or even generate test suggestions — all without leaving your enterprise environment.

What’s next

Once you have your local Ollama model running and integrated with QF-Test, try experimenting with:

Natural language validations in your test cases
Dynamic data generation for forms
Multilingual text checks

For more details on all AI integration options in QF-Test, check out our free special webinar “When tests become intelligent: AI-driven checks with QF-Test”:

[[qf.Preview(videos/2026-03-02-special-webinar-ai-checks)]]

With just a few steps, your tests can now become AI-aware while staying fully under your control. It’s time to explore the power of combining QF-Test with local AI models!

Tutorial: Using GitHub Copilot with the QF-Test AI Integration

2026-02-09T00:00:00+01:00

QF-Test is a comprehensive test automation solution built for Web, Desktop, Java-based, and Mobile applications, with a focus on long-term maintainability and enterprise-ready reliability. With the launch of QF-Test 10, AI-driven capabilities are now integrated directly into your testing workflow. This post shows how to connect GitHub Copilot to QF-Test using a simple custom model and the Copilot CLI.

Usually in QF-Test, AI providers are set up via Options → Artificial Intelligence → AI Configurations. GitHub Copilot is an exception because it doesn’t expose a standard API endpoint with a URL. Therefore, you cannot add Copilot in Options → Artificial Intelligence → AI Configurations. The good news: you can still integrate Copilot by registering your own custom model in QF-Test that calls the GitHub Copilot CLI.

What you’ll need

QF-Test 10 or later.
Access to GitHub Copilot and GitHub Copilot CLI. As of now, a free GitHub account on the “Free” Copilot plan is enough to experiment with the Copilot CLI.
Basic familiarity with scripting in QF-Test.

Step 1: Install and configure the GitHub Copilot CLI

Install the GitHub Copilot CLI
Follow the official instructions
Log in from your Terminal
Run copilot login
Verify it works
Try a quick prompt: copilot --prompt "Hello World"

If that prints a response from Copilot, you’re ready to connect it to QF-Test.

Step 2: Wire Copilot into QF-Test via a custom model

We’ll use ai.addCustomModel in a Server Script so QF-Test can talk to Copilot by launching the CLI as a subprocess. In QF-Test:

Create a new Server Script node (language: Groovy).
Paste the code below into the script.
Adjust the path to the copilot executable. /opt/homebrew/bin/copilot will only work if Copilot CLI was installed via Homebrew on macOS.
Run the script node.

ai.addCustomModel("GitHub Copilot", (msg) -> {
    def cmd = [
        "/opt/homebrew/bin/copilot",
        "--silent",
        "--stream", "off",
        "--model", "claude-sonnet-4.5",
        "--prompt", msg
    ]
    def proc = cmd.execute()
    proc.outputStream.close()
    proc.waitFor()

    return proc.in.text.replace("● ", "")
})

println(ai.ask("What is the answer to life, the universe and everything?", configName: "GitHub Copilot"))

What this does:

Registers a LLM model named “GitHub Copilot” with the QF-Test AI integration.
For each prompt (msg), QF-Test starts the Copilot CLI and returns the CLI’s text output back to QF-Test.
Prints the result of a sample ai.ask call in the QF-Test terminal.

Tip: You can change the model name passed to --model to your preferred Copilot model.

If you get an error saying Cannot run program "/path/to/copilot": error=2, No such file or directory, you must update the path in the first item in the cmd list to where the Copilot CLI is installed on your system.

If everything works, you can remove the println line at the end.

Step 3: Use Copilot with the QF-Test `ai` scripting module

Once the custom model is registered, you can use it from any script with ai.ask, for example:

def answer = ai.ask(
    "Generate a JSON string of test data in the following format: ...",
    configName: "GitHub Copilot"
)
rc.setLocal("data", JSON.parse(answer))

For more on the ai scripting module in QF-Test, see the QF-Test manual.

Step 4: Validate UI text with “Check text with AI”

You can also bring Copilot into your test checks. The “Check text with AI” node lets you have the AI verify UI elements in your application using natural language. In the node’s “AI configuration” field, “GitHub Copilot” should automatically appear in the dropdown.

That’s it! Even though Copilot doesn’t expose a standard URL+API key endpoint, you can still use it inside QF-Test by registering a custom model that calls the Copilot CLI. From there, ai.ask helps you validate and generate data right in your tests, and “Check text with AI” lets you validate UI content in plain language. Happy testing!

For more details on all AI integration options in QF-Test, see our free special webinar “When tests become intelligent: AI-driven checks with QF-Test”:

When tests become intelligent: AI-driven checks with QF-Test

In this special webinar, we’ll show you how to get the most out of the new AI integrations in QF-Test. With QF-Test 10, you can use AI to test non-deterministic UIs, validate UI components based on semantic criteria, generate test data, and much more.

Using GitHub Copilot CLI via Remote SSH from QF-Test

This quick guide explains how to use the GitHub Copilot CLI via a remote SSH connection directly from QF-Test — including SSH key setup and troubleshooting.

Irfan Yigit, ASC Technologies

01/04/2026

More about AI-driven testautomation.

Testing ZK Applications with QF-Test in 2026

2026-02-05T00:00:00+01:00

This text was originally published as a ZK Small Talk. ZK is a framework for building web applications with Java and a long-standing partner of QF-Test.

The great thing about ZK has always been that you can build web application without having to deal with the hassle of writing and maintaining HTML and JavaScript code. You define your application UI in concise ZUML markup and little richlets written in Java, you can write unit tests for your business logic and even use ZATS to test your UI.

QF-Test Featured in mgm technology partners’ Blog on Quality and Efficiency Strategy

2026-01-29T00:00:00+01:00

In the latest post on the mgm technology partners blog, “How Quality Engineering Makes Complex Systems Manageable”, Lilia Gargouri presents their holistic approach to quality engineering – with a focus on sustainable, scalable quality assurance in complex software projects. Unlike traditional testing, this approach emphasizes a systematic interplay of architecture, processes, data, models, and tools, making quality throughout the entire software lifecycle both predictable and controllable.

A particularly interesting aspect

The article highlights the critical role automation plays in this approach—and this is where QF-Test comes in. As a powerful tool for automated end-to-end and UI testing, QF-Test delivers exactly the scalability and repeatability that complex applications require, helping teams use resources efficiently while ensuring high quality.

Why QF-Test matters for quality engineering

Automated tests that go beyond classic manual methods to ensure consistent, reproducible quality.
Support for different types of applications (web, desktop, Android, iOS applications, etc.), making it easier to integrate into diverse projects.
Contribution to a shift-left strategy, helping to prevent defects early in the development process and thereby reducing effort and cost.

Conclusion

Overall, the article demonstrates how test automation—and especially the use of QF-Test as part of a test and quality engineering strategy—helps manage the challenges of complex software environments and ensures the highest quality standards.

How Quality Engineering Makes Complex Systems Manageable by Lilia Gargouri, Head of Quality Engineering at mgm technology partners

Scrolling on the Web with QF-Test – The complete guide

2025-07-16T00:00:00+02:00

A question that comes up time and again during web application test automation with QF-Test is: How do I scroll? How do I trigger a scroll event? Can I tell QF-Test to turn the mouse wheel? Can I simulate scrolling with the mouse in QF-Test? I just want to scroll down a little!

In general, our first response to these kinds of questions is: You shouldn’t have to. QF-Test handles scrolling by itself and will fully automatically scroll elements into the viewport as soon as it needs to interact with them. That’s why QF-Test does not record scroll events, or offer a ‘Scroll’ node, because in 99% of cases, QF-Test will do the right thing automatically.

But you’re probably here because you’re part of that 1% of cases where that mechanism is not enough. So in this blog post, I will tell you all about how QF-Test scrolls your web application and how to control that scrolling yourself.

Scroll manually using keyboard events

If you just want to scroll a set amount, you should try to scroll by recording keyboard events.

In most web applications, the arrow keys can be used to scroll around. Space and Shift+Space will scroll roughly one window height, and the Home and End keys should scroll to the top and bottom of the document, respectively. That way you might be able to skip the more complicated methods described below.

Fine-tune the QF-Test scrolling algorithm

Let’s say QF-Test is already scrolling for you, but it’s not quite doing it the way you need. QF-Test offers several SUT options you can use to configure its scrolling behavior.

You can set these options in a SUT script node or using the @option doctag in QF-Test 9.

`Options.OPT_WEB_SCROLL_VISIBLE`

This option controls how web nodes are scrolled visible. Possible values are:

Options.VAL_WEB_SCROLL_VISIBLE_DEFAULT (use the QF-Test algorithm),
Options.VAL_WEB_SCROLL_VISIBLE_NEVER (prohibits scrolling),
Options.VAL_WEB_SCROLL_VISIBLE_TOP
Options.VAL_WEB_SCROLL_VISIBLE_BOTTOM
Options.VAL_WEB_SCROLL_VISIBLE_MINIMAL
a custom JSON string matching the options parameter of the Element:scrollIntoView JavaScript API

Options other than “default” and “never” will call node.scrollIntoView() directly in the browser’s JavaScript context. scrollIntoView is a pretty powerful part of the JavaScript Element API. You can learn all about it’s advanced options in the mdn web docs.

You can also override this option on an individual component by setting the node property qfs:scrollvisible:

def node = rc.getComponent("myElement") // replace with a component's QF-Test ID
node.setProperty("qfs:scrollvisible", Options.VAL_WEB_SCROLL_VISIBLE_NEVER)

`Options.OPT_WEB_MIN_DELAY_AFTER_AUTO_SCROLL`

With this option, you can introduce some additional delay after every automatic scroll action, in ms. This is useful if your application uses animations that are triggered by scrolling or if your application just needs a little more time to reload data after scrolling. The default value is 150.

`Options.OPT_PLAY_SCROLL_ITEM`

Set to False to stop QF-Test from automatically scrolling to sub-items of lists, tables or trees. You can use this if QF-Test scrolls to weird places when navigating a tree, for example.

`Options.OPT_PLAY_SCROLL_PADDING`

Extra padding for automatic scrolling, in px. Use this to make QF-Test increase or decrease the calculated minimal required scroll amount so that more of the surroundings of the target element remain visible. The default value is 5.

`Options.OPT_PLAY_WEB_AUTO_SCROLL`

When set to False, QF-Test will not perform any auto-scrolling at all. Use this to completely opt-out of automatic scrolling and handle it manually.

Scroll components into view

You can trigger the QF-Test scrolling algorithm manually in an SUT script like this:

node = rc.getComponent("myElement") // replace with a component's QF-Test ID
// Boolean scrollPointVisible(int targetX, int targetY, boolean propagate)
node.scrollPointVisible(0, 0, true)

This will try to scroll the element so that it becomes visible. If the element cannot be made completely visible, it will scroll it so that at least the requested target point becomes visible.

If the QF-Test algorithm is not helpful in your situation, maybe because you want to trigger a scroll action by itself, you can use a small SUT script which calls any JavaScript scrolling function like this:

node = rc.getComponent("myElement") // replace with a component's QF-Test ID
node.evalJS("_qf_node.scrollIntoView()") // arbitrary JavaScript

Scroll by set amounts

But what if you need even more control over the scrolling or don’t have a particular component you want to scroll to?

You can also any scroll container by a precise amount using JavaScript.

For example, this SUT script would scroll the window down by 25 pixels:

node = rc.getComponent("genericDocument")
node.evalJS("window.scrollBy(0,25)")

The first parameter of the scrollBy method controls the x-axis, so you can even scroll sideways using this method.

And that’s it. That was all you need to know about scrolling web content with QF-Test to get you out of even the trickiest situations.

If you are still stuck, you can get in touch with our support team, who are always happy to help!

Maximizing SUT windows – a good idea?

2025-06-03T00:00:00+02:00

In QF-Test product support we are often asked how to set a QF-Test controlled browser or other SUT windows to fullscreen mode.

It’s better to work with fixed window sizes

In fact, we advise against maximizing windows for the following reason: The maximized window size depends on the system, a fixed window size is the same on every system instead.

QF-Test is comparatively resistant to changes in component coordinates. However, web applications in particular can display completely different page layouts depending on the window size. In the worst case, this can lead to individual components no longer being found.

If the browser is always started with a fixed window size instead, the test behaves uniformly on all computers.

In the preparation sequence that opens the browser, you will find an “Open browser window” node below the “Start SUT if necessary” node, in which you can specify a uniform window size.

How to use fullscreen anyway

If you still want to maximize your browser window, you can use the following Jython server script in QF-Test to maximize all windows containing e.g. “Edge” in the window title under Windows:

import autowin, re

windows = autowin.getAllWindows()
for win in windows:
    title = autowin.getWindowText(win)
    if re.match(".*Edge.*", title):
        autowin.maximizeWindow(win)

You can customize the regular expression ".*Edge.*" to address exactly the window you are interested in with its window title.

How can I create screenshots of my client application with QF-Test?

2025-05-20T00:00:00+02:00

QF-Test only provides on-board means to write screenshots of a client application to the test-run log. This can be done via the “Message” node or the procedure qfs.run-log.screenshots.logScreenshot from the qfs.qft standard library.

Save screenshots to a file

Sometimes you don’t want to store a screenshot in the test-run log but use it for other things and store it in a directory first. This is of course also possible with QF-Test, but it requires a script – but just a very small one:

from imagewrapper import ImageWrapper
from java.io import File
screenshot = ImageWrapper(rc).grabImage(rc.getComponent("#Window:"))
screenshot.writeToFile(File(str(rc.groups.qftest.suite.dir) + "/screenshot.png"))

This Jython SUT script uses the exact same mechanism to create screenshots that is used for the test run log.

rc.groups.qftest.suite.dir is a useful shortcut to get the directory of the current test suite. You can of course also call writeToFile() with any other path. "#Window:" is a SmartID and stands for the first window of your application. You can also specify a more specific component here to take a screenshot of a specific part of the application UI.

You can read more about the ImageWrapper class in the QF-Test manual: The ImageWrapper class.

It’s not magic: How the CustomWebResolver makes your web application UI testable

2025-04-04T00:00:00+02:00

With QF-Test, our goal is to make testing web application UIs as easy as testing a native application. This presents some challenges for which we developed a unique solution we call the CustomWebResolver. In this blog post, I will explain why QF-Test is so much better at testing web applications than other tools and what this “CustomWebResolver” has to do with it.

The problem with CSS and XPath selectors

Most browser-based UI testing tools rely on CSS selectors or XPath to select an element to interact with or validate. This approach has several downsides:

Very verbose: A typical CSS selector looks like this: body > main > div > ul > li:nth-of-type(3) > button.my-action, the same as an XPath would be: .//body/main/div/ul/li[3]/button[@class="my-action"]. Not very readable.
Not very stable: These selectors rely on the underlying HTML structure of the application to remain the same, otherwise they break.
Not easy to auto-generate: This becomes important when recording an interaction. Generating a CSS or XPath selector for an HTML element programmatically is easy. Generating a stable, concise and reusable selector, however, must usually be done by hand.
No Shadow DOM access: It is not possible to use CSS or XPath to select an element encapsulated in a “shadow root”. This is a problem with applications built on the Web Components standard. Instead, JavaScript and a deep understanding of web technologies are required.
Technology-specific: These selectors are only applicable to HTML, they cannot be used in a cross-platform context.
No checking of state: With these selectors, you only found the element. Checking if the element has the required state to pass a test needs to be handled separately.

There is a better way: The QF-Test CustomWebResolver

Although you can also work with selectors here, QF-Test offers a better way: A technology we call the CustomWebResolver, or CWR for short, resolves the raw HTML of your application into a semantic tree of well-defined UI components. This offers several advantages:

No knowledge of HTML required: Test authors are not required to understand the intricacies of the DOM structure of the application to address components.
Semantic tests: UI components can be addressed based on their semantic function as presented to the user, instead their structural position in the DOM or the HTML code.
Unified handling of similar components: UI components that have the same semantic function but are implemented differently on the HTML level are still addressed in the same way.
Checks included: The state of a UI component can automatically be checked according to common criteria like “is it visible”, “what’s the text content?”, “what’s the input value?”, “is it enabled or disabled?”, “is it checked or selected?”, and more.
Don’t worry about the Shadow DOM: With a CustomWebResolver you never have to think about things like shadow roots – QF-Test just takes care of everything.
Robust against DOM changes: The CustomWebResolver is much more resilient against changes to the DOM structure. And if changes are necessary, they can be made in a central place instead of everywhere a component is used.
Cross-platform: Because it is not bound to HTML, the QF-Test component structure can be reused between applications built with Java UI libraries, Native Windows UIs and even iOS and Android apps with little adjustment needed.
Selection via SmartID: In QF-Test you can select a component via the very compact SmartID syntax like this: #List:&2@#Button:

Okay, but how does this magical CustomWebResolver technology work?

The basis of the CustomWebResolver is the translation of the DOM structure into a tree of components of certain generic classes based on a set of mappings. Let’s untangle that statement.

Generic classes

Any application UI consists of a number of widgets, or components, that serve some purpose. There are usually windows with different panels which contain labels, tables, trees, lists, checkboxes, text inputs, sliders, drop-downs, and so on.

QF-Test comes with a big list of generic classes which can be used to describe almost every component imaginable in an application UI. QF-Test understands their semantics, i.e. how they work, and can automatically offer the right actions and checks per component.

For example, QF-Test knows to offer a “‘checked’ state” check for the generic class CheckBox, but not for e.g. the generic class TextField, and it understands that you can click a CheckBox component to toggle it’s ‘checked’ state. It also understands that a Table or List can contain child elements and gives you easy access to them.

CWR mappings

The job of the CustomWebResolver is to look at the DOM of your application and figure out which HTML elements belong to which generic class. To do this, it uses mappings which are usually defined in a YAML document. The simplest mapping possible looks like this:

genericClasses:
- Button: btn

This piece of YAML, when interpreted by the CustomWebResolver, will tell QF-Test to assign every HTML element with the CSS class btn the generic class Button and therefore understand that it’s a button that can be activated as long as it’s not disabled or invisible.

With the CustomWebResolver configured like this, you can now select “OK” buttons using the SmartID #Button:OK, regardless of where this button is or what HTML tag it actually uses, as long as it has the CSS class btn and a label “OK”.

Here is another example of a CustomWebResolver mapping, this time a more complex one:

genericClasses:
- ComboBox:
    attribute: role
    attributeValue: combobox
- List:ComboBoxList:
    tag: ul
    parent: ComboBox
- Item:ComboBoxListItem:
    tag: li
    parent: List:ComboBoxList

The mapping above is for a ComboBox, i.e. some kind of drop-down select input. It actually consists of multiple sub-components:

The drop-down itself,
the list that appears when the drop-down is opened and
the individual selectable options.

Thanks to these mappings, the CustomWebResolver can tell QF-Test to handle this kind of drop-down as a single semantic component: QF-Test can automatically select a given option in the list, it can fetch the current selection or check that a given option is the current selection, or it can return the entire list of available options – all through the QF-Test UI without writing any code or messing with the DOM.

The component tree

Finally, once all relevant components are mapped to generic classes, QF-Test can discard the rest of the HTML and you are left with a beautifully simple tree of semantic components, on which you can build your tests.

And how do I configure this CustomWebResolver for my application?

The only thing users of QF-Test have to do is to provide the initial configuration for the CustomWebResolver. We have implemented three ways to make this as easy as possible:

For a growing list of web component frameworks, we bundle comprehensive CWR configurations with QF-Test. QF-Test will automatically recognize and translate any components from these frameworks into generic classes, ready for you to use in your tests and checks. We support popular frameworks like Vaadin, Angular Material UI, and Google Web Toolkit. Check out the full list to see if your framework is already supported.
If you use a different framework, but your application adheres to the WAI-ARIA standards for web accessibility by marking components with role and aria-* attributes, QF-Test will use these attributes to automatically configure the CustomWebResolver. The better the accessibility of your UI, the better QF-Test will work with it – a double win!
Otherwise, you’ll need to write your own CWR mappings. This is not super-difficult, but it requires some knowledge of the HTML structure of your application. Don’t worry, QF-Test does a lot to help you with this. QF-Test comes with a comprehensive user interface which helps you at every step when editing and verifying your configuration. And with the QF-Test UI Inspector, you can see the effect of your CWR configuration in real-time. And if you’re still having trouble, our support team is ready to assist you.

We believe that semantic components and the CustomWebResolver are the best way to build reliable, robust and comprehensive UI tests for web applications. And our customers think so, too.

If you have been using QF-Test for the web already but have not made full use of the power of the CustomWebResolver yet, you should give it a try. And if you are totally new to QF-Test, you can get started with a free trial today!

The CustomWebResolver in the QF-Test manual: Improving component recognition with a CustomWebResolver
Tutorial for configuring your first CustomWebResolver: Example for “CarConfigurator Web” demo
List of frameworks with built-in CWR: Special support for various web frameworks
Detailed documentation of everything the CWR can do: The Install CustomWebResolver node

Objects in QF-Test variables – an overview

2025-02-20T00:00:00+01:00

When working with variables in QF-Test, the biggest challenge so far has been to understand which variable is currently active with which value. Important here are the variable stacks – the manual explains their function in detail.

The status quo up to QF-Test 9

The values stored in variables used to be strings always. Other objects (e.g. numbers) were converted to a string when stored in a variable with rc.setLocal or rc.setGlobal and stored in the QF-Test main process. When read with rc.getInt or rc.getBool the value was then translated back.

Using this mechanism it is also possible to set a variable in an SUT script and then use its value in a server script or an If node (and vice versa). If you wanted to cache complex objects, you always had to work with the “global” script variables of a scripting language, and across process boundaries you had to use rc.fromSUTand rc.toSUT or rc.fromServer and rc.toServer.

After more than 20 years this simple understanding of values has reached its limits and has been extended with QF-Test 9: it is now possible to store arbitrary objects in QF-Test variables and thus return complex objects from a procedure, for example.

Ensuring backwards compatibility

First things first: We have invested a lot of effort to ensure backwards compatibility. The preexisting access methods in the run context return the same types, values can be stored in the same way and when things get complicated (for example when a non-serializable or very large object from an SUT script wants to be stored in a QF-Test variable), QF-Test falls back to the previously used string representation.

Object variables in variable tables

In QF-Test you can tell right away in any variables table (e.g. in the options dialog or in the debugger window) whether a variable value is a string or has a different type: In the first case, everything looks the same as before, i.e. only the string appears in the table. In the second case, the variable value is preceded by the type of the object in parentheses, e.g. (Boolean) true.

The type of a stored variable can also be changed here via the context menu.

Object variables in “Set variable”, “Return” and parameter tables

In case of the “Set variable” node and the “Return” node in a procedure, the type of the set or returned variable can now also be explicitly specified. This is done at runtime, i.e. when the node is executed, the character string specified as the value in the node is converted into the selected type.

In the same way, the type of the parameter can also be defined in parameter tables, for example in sequences – here too, the conversion takes place when the node is executed. The value of the parameter then appears in the table again with a type reference, for example: (as Number) 3.141. The “as” indicates that the conversion takes place at runtime.

Such a conversion can also be carried out everywhere a variable expansion takes place with the help of the special group “as”, the syntax looks like ${as:fromjson:[1,2,3,4]}.

Object variables via `rc.getObj` and `rc.vars`

The rc.getObj method can now be used to read the object value of a variable in a script or script expression. Analogous to the existing methods such as rc.getStr, rc.getNum or rc.getBool, the variable (with one parameter) or a group variable (property, with two parameters) is evaluated here and its value returned – in this case without conversion.

An expansion of nested variable expressions in the value only takes place automatically if the value is a string, otherwise it must be explicitly requested via the parameter expand=true.

The run context now also offers a practical shortcut to any variable object: instead of rc.getObj("variablename") you can now simply write rc.vars.variablename, and rc.groups.qftest.suite.file returns the File object of the current test suite.

Object variables and `$(name)`

When you use the familiar $ notation for variables, the object value of the variable is used for as long as possible: if the variable average has the value (double) 2.3, for example, the new variable would also have the type double for a “Set variable” node with the default value $(average). However, if you combine the variables with further text, e.g. My average: $(average), the string representation of the variable value is automatically used and the new variable becomes a string, as well.

As before, you cannot simply write $(name) == "John" as a check expression in an If node – the correct syntax would be "$(name)" == "John". But script expressions are preferably used without the $ notation, so it’s better to use rc.getStr("name") == "John" – or now rc.vars.name == "John".

Various small improvements

These changes to the QF-Test variable system also allowed us to integrate other long-awaited features into QF-Test 9: The standard procedures and the special “qftest”-group now return their values as objects, as seen above.

It is now also possible in many places to hide values in variables using the context menu and only evaluate them again at runtime with ${decrypt:...} without the true values appearing in the run log.

Also the “Start process” node now accepts a list object as a parameter, which makes it possible to implement start nodes with a variable number of parameters much more easily.

These are small things, but they all contribute to making working and testing with QF-Test even more comfortable.

Outlook

If you have made it this far, you will have noticed that the topic is complex and certainly cannot be covered exhaustively here.

The developers behind QF-Test will provide details in an upcoming special webinar on the topic of object variables – and the newly redesigned QF-Test manual also explains variables in detail.

From a simple email to a memorable internship

2025-02-10T00:00:00+01:00

In my 21 years, I never left France before. And suddenly, I am driving a thousand kilometers to live in Germany for three months. How did this happen?

Soon after I began working with QF-Test at Thales, and since I am the only person on my team working with this software, I decided that it might be fun to learn more about QF-Test directly next to the people who built it. So, I sent an email, and a few days later, I got an interview with QFS.

It only took a week or two to get a definitive answer, and I left for Germany on the 30th of July. Of course – because I like stupid challenges – I decided to drive to Leipzig, from Paris, on my motorcycle. 14 hours of riding later, through rain and sunshine, I finally arrived in Leipzig. Two days later, I started working.

Working at QFS was a really enriching and weird (in a good way) experience. The work culture is completely different from what I am used to, whether it’d be at Thales or other smaller companies. Coffee break? 3 minutes. 3 minutes!? Fastest I ever managed was 10! Maybe we are a bit too much into coffee in France. Or is it just an excuse not to work? I’ll let you decide. Same goes for the lunch break. If we go out in France, at a restaurant, it’ll be for at least an hour and a half. In Germany, 30 minutes is all you need. These are only small things, but it shocked me how much we love not to work during working hours in France.

About the actual work: I have never seen such a good cohesion in a team as at QFS. Communication was smooth, not too many meeting, so we actually had time to work. And the team is awesome. Found a bug? Have an issue? A question? Just ask or talk about it, and you’ll get the answer, or the bug fixed. And even as an intern, a new person working at the company, the onboarding was pretty smooth. Everything is done logically and is easily understandable. It didn’t take me more than a week to understand most of my job and start working. This feeling of being ready and useful very quickly was extremely satisfying and gave me a lot of motivation to perform my tasks.

The tasks were satisfying as well. To be given a task, some time, and some liberties to do it, without having a strict deadline, is really comfortable. And when it is done, and everybody is happy about it, it feels even better. I don’t have the words to explain it better, but trust me, it was a pleasure working at QFS.

This internship also allowed me to visit and live in a foreign country. As a first timer, this was a bit difficult. Going to Aldi, the barber, or the bakery for the first time felt like I had never done it before. But in the end, it is just like everywhere else. It is not that hard. I also got to visit the countryside, and I even made a rider friend along the way. I got to ride very fast, to see incredible landscapes (The Zugspitze with Gregor and Tina!), towns, to feel the german weather (not the best…) and to meet new people.

Even though I loved my stay in Germany, going back to France was a relief. I got my habits back, my birds, my cars, my gym, everything. But I would go back to Germany if given the opportunity, no doubt.

In conclusion: QFS is an amazing company, leaded by amazing people, and I understand why it is considered such a great place to work. I learned a lot, and am ready to get QF-Test operational at Thales very soon, as well as making it more widely used across our different teams.

Extending the QF-Test Assertion API – A practical introduction

2025-01-14T00:00:00+01:00

Since QF-Test 9 there is a new and comfortable way to check conditions in scripts: The QF-Test Assertion API, which is loosely based on the chai.JS library known from Javascript.

In a Groovy script, for example, we can now write:

def foo = 'bar'  
def beverages = [ tea: [ 'chai', 'matcha', 'oolong' ] ]  

expect(foo).to.be.a('String')  
foo.should.be.equal('bar')  
expect(foo).to.have.lengthOf(3)  
expect(beverages).to.have.property('tea').with.lengthOf(3)

The API of the fluid check expressions is well equipped so that most checks can be written very easily. In some special cases, however, you may wish to extend the API, as is possible with chai.JS plugins. For example, it would be nice if you could use the following expression to check whether a given string represents a valid JSON object:

def validJson = '[1,2,3,"s",{}"key":"value"}]'
def invalidJson = '[{]'

validJson.should.be.json()
invalidJson.should.not.be.json()

QF-Test offers an interface for such extensions of the Assertions API as well, which we would like to demonstrate in this blog post.

To enable a dynamic extension of the API, a proxy object is created under the hood for each assertion whenever a test expression is executed. In addition to the standard methods of each assertion, this proxy also supports any methods that have been registered as extensions in the meantime. An interface definition in the form of an interface is first required for these extensions:

package de.qfs.lib.assertions.external;

import de.qfs.lib.assertions.Assertion;

public interface JsonAssertion extends Assertion {
    Assertion json();
}

Please note that

the interface is normally placed in the en.qfs.lib.assertions.external package,
it extends the existing Assertion interface,
and that each new API method in turn returns the assertion object itself.

This makes it possible to chain the method calls. The code to be executed can now be defined in a separate class that implements the interface and derives it from en.qfs.lib.assertions.AssertionExtension. However, it is easier to do this directly in the interface definition as a default implementation:

default Assertion json() {  
    return self();  
}

The implementation of the method does not yet do very much, it merely returns a reference to its own assertion object, as required. To perform or report the actual check, the inherited method doAssert(condition, message, negatedMessage, expected, actual) must be called:

condition is a boolean value or a function with a boolean return value that contains or returns the result of the check.
message is a string that can be used for an error message or for the check performed in QF-Test – the placeholder #{this} is replaced by the checked object, #{exp} by the expected value and #{act} by the actual value (usually identical to the checked object).
negatedMessage contains the message that is used for checks whose call chain contains a not.
expected contains the expected value.
actual contains the actual value.

With the exception of condition, all parameters are optional and can be omitted at the end of the parameter list.

Our assertion method might now look like this:

default Assertion json() {  
    doAssert(this::isObjectJson,  
            "expected #{this} to be a json string",  
            "expected #{this} not to be json string");  

    return self();  
}

We have outsourced the actual check to a separate method in which we try to read the input as JSON and return true if successful and false if an error occurs:

default boolean isObjectJson() {  
    try {  
        final Object s = _obj();
        if (s instanceof String) {  
            Json.parse((String) s);  
            return true;  
        }  
        return false;  
    } catch (final ParseException ex) {  
        return false;  
    }  
}

Note that the expression _obj() was used to access the object to be checked.

In order to be able to use our new method in check expressions, it must be registered once with the AssertionFactory. This factory is a singleton and provides the instance method registerExtension(extension, overwrite) for this purpose:

extension is the new assertion interface of the extension. Either a class that implements the interface or an object of such a class can be specified here.
overwrite determines how to proceed with already registered methods of the same name. If true, the method of the newly registered extension is preferred, with false the previously registered one is.

The easiest way to register an extension is to do so when initializing a QF-Test plugin. A general description of how to build QF-Test plugins can be found in the previous blog article. For the plugin that provides our Assertion API extension, the corresponding plugin class could look like this:

import de.qfs.apps.qftest.shared.QFTestPlugin;  
import de.qfs.lib.assertions.AssertionFactory;  
import de.qfs.lib.assertions.external.JsonAssertion;  

public class JsonAssertionPlugin implements QFTestPlugin {  

    @Override  
    public void init()  
    {  
        AssertionFactory.instance()
                    .registerExtension(JsonAssertion.class, false);  
    }  
}

If you want to add “properties” to the call chain, please note that the associated methods must each be preceded by a get – the script engines Jython or Groovy then automatically convert the properties in the check expression into the corresponding method call.

As an example, we want to extend our new API to include the check for “primitive” JSON objects:

def primitiveObj = '"String"'
def nonPrimitiveObj = '[1,2,3]'

primitiveObj.should.be.primitive.json()
nonPrimitiveObj.should.not.be.primitive.json()

Then we add the corresponding getPrimitive method to our JsonAssertion interface:

default Assertion getPrimitive() {  
    self().flag("primitive", true);  
    return self();  
}

and add the test method:

default boolean isObjectJson() {    
    final boolean primitive = AssertionUtil.flagAsBoolean(self(), "primitive");

    try {  
        final Object s = _obj();
        if (s instanceof String) {
            final JsonValue value = Json.parse((String) s);  

            if (primitive) {  
                final boolean valueIsObjectOrArray = 
                                  value.isObject() || value.isArray();  
                return ! valueIsObjectOrArray;  
            }  

            return true;  
        }  
        return false;  
    } catch (final ParseException ex) {  
        return false;  
    }  
}

A typical Assertion API scheme is used here: in the property method, a status value is set in the assertion object with self().flag(name, value), which is then retrieved during evaluation. To achieve this, the self().flag(name) method returns the corresponding status value. To simplify matters, the class en.qfs.lib.assertions.AssertionUtil offers the static helper methods flagAsBoolean(assertion, name) and flagAsString(assertion, name).

The message for the error case should also be adjusted to reflect the status:

default Assertion json() {  

    final boolean primitive = AssertionUtil.flagAsBoolean(self(), "primitive");  
    final String descriptor = primitive ? "primitive " : "";  

    doAssert(this::isObjectJson,  
            "expected #{this} to be a " + descriptor + "json string",  
            "expected #{this} not to be " + descriptor + "json string");  

    return self();  
}

Finally, in order to hide the internal details in the stack trace of the AssertionError thrown in the event of an error, the test method can define itself as an entry point (ssfi stands for start stack function indicator)):

default Assertion json() { 
    _ssfi(JsonAssertion.class,"json");

    (...)
}

Note: Due to the class loader used, extensions are currently restricted to Server scripts and Groovy SUT scripts.

The code of the entire extension is available on GitLab.

Introduction to QF-Test Plugin Development

2025-01-09T00:00:00+01:00

Some people may have stumbled over the plugin directory in QF-Test and wondered why it is there and how to use it. In fact, the directory (with its subdirectories qftest and sut) serves multiple purposes:

Upfront, if you place any .jar file containing Java classes in that directory, they will be automatically added to the QF-Test and/or SUT classpath and therefore available for use in server or SUT script nodes, in data driver nodes (e.g. JDBC drivers) etc. If the library is placed in the qftest subdirectory, the contained classes are only available within the main QF-Test process and server scripts, but not within the classpath of the tested application. If you place the library in sut, you can use it within SUT scripts only. A library placed directly in plugin is available in both context.

In addition, it is possible to enhance the QF-Test test execution itself, or to add special functionality. Unforgotten is the holistic plugin, which caused QF-Test to play “characteristic” sounds during test execution so users could listen to their tests running. While this example may be rather playful, we will see how to use this mechanism for more serious tasks.

With QF-Test 8 and the QF-Test Grade plugin 2.1.0 we made the development of QF-Test plugins very comfortable – it’s time to take a deep dive into the topic.

In this example, we will build add a plugin which extends QF-Test with an embedded web server that displays the current execution state. The final result can be found at gitlab.com/qfs/qftest-plugins/webstatus-plugin.

Technically, QF-Test plugins are also .jar files which are placed in the plugin folder. But they contain a specific pointer file in META-INF/services which allows QF-Test to load the plugin class at runtime.

For development, any IDE which supports Gradle builds can be used, for example IntelliJ. To follow along, create a new project, name it webstatus, select Gradle as build system and make sure that at least Java 17 is selected as JDK.

As a first step, we add the QF-Test Gradle plugin to the project by adding it to the plugins section of the build.gradle file:

plugins {  
    id 'java'  
    id 'de.qfs.qftest' version '2.1.0'
}

If we now reload the project (e.g. using the reload button in the “Gradle” IntelliJ panel), Gradle will automatically download the latest QF-Test and add it to the project’s testImplementation dependencies.

You can instruct the plugin to use the QF-Test installation you already have on your system by adding the version = 'local' property in a new qftest section of the build.gradle file (If you did not install QF-Test at the default location, you can use the versionDir property instead, as described in the documentation of the QF-Test Gradle Plugin):

qftest {  
    version = 'local'  
}

Until now, the instructions have been similar to any project which uses the QF-Test Gradle plugin for test execution. To put the project into “QF-Test plugin development mode”, we need to add another property to the qftest section of build.gradle:

qftest {  
    version = 'local'  
    pluginClass = 'de.qfs.apps.qftest.plugins.WebStatusPlugin'  
}

In our example de.qfs.apps.qftest.plugins.WebStatusPlugin is the fully qualified class name of the plugin class we’re about to develop. Since the QF-Test plugin will only be available in the QF-Test main process, we can tell the Gradle plugin to focus on this development environment:

qftest {  
    version = 'local'  
    pluginClass = 'de.qfs.apps.qftest.plugins.WebStatusPlugin'
    devScopes = ['qftest']  // default: ['qftest','sut']
}

After having setup the build system, we can start developing the plugin itself: Let’s create a new class and let it implement de.qfs.apps.qftest.shared.QFTestPlugin:

Since the Gradle plugin automatically extended the implementation dependencies, the required QF-Test classes are directly available using code completion. As a first step, we simply add a “Hello World” output:

@Override
public void init() {  
    System.out.println("Hello plugin world!");  
}

To start up QF-Test including the plugin, we can use the startQFTest task of Gradle – the Gradle Plugin will automatically take care of creating the required services file and make the new class available to QF-Test:

 % ./gradlew startQFTest
Starting a Gradle Daemon

> Task :startQFTest
Starting QF-Test (local)

BUILD SUCCESSFUL in 7s
3 actionable tasks: 3 executed

Our plugin class is compiled, QF-Test starts up – but no output! What went wrong?

By default, the startQFTest task is used to start QF-Test interactively in a “fire-and-forget” manner – no outputs are attached and the Gradle task does not wait until QF-Test is closed. To see the debug output, we must change the fork property of the task, by adding this new definition to the build.gradle:

task startQFTestAndWait(type: StartQFTestTask) {  
    fork = false  
}

Now, we use this new task to start QF-Test, and our output becomes visible:

% ./gradlew startQFTestAndWait

> Task :startQFTestAndWait
Starting QF-Test (local)

Hello plugin world!
<=========----> 75% EXECUTING [17s]
> :startQFTestAndWait

The init() method is called very early during the startup of QF-Test or the SUT, as soon as the class has been loaded. You can distinguish different execution contexts by calling the static methods isRunningInSUT() (i.e. SUT (true) or QF-Test (false)) and isRunningInteractively() (interactive (true) or batch mode (false)) of the class de.qfs.apps.qftest.shared.Util.

If you want to delay execution until QF-Test has loaded or the SUT has been connected, you can listen for the "qftest-loaded" or "sut-loaded" notifications respectively:

import de.qfs.lib.notifications.DefaultNotificationCenter;  
import de.qfs.lib.notifications.Notification;  

[...]

@Override  
public void init() {  
    System.out.println("Plugin init");  
    DefaultNotificationCenter.instance().addObserver(this::qftestLoaded,"qftest-loaded");  
    DefaultNotificationCenter.instance().addObserver(this::sutLoaded,"sut-loaded");  
}  

private void qftestLoaded(Notification n) {  
    System.out.println("QF-Test loaded");  
}  

private void sutLoaded(Notification n) {  
    System.out.println("SUT loaded");  
}

Notifications are a powerful mechanism in QF-Test to send synchronous messages (= well defined Strings with optional additional data) in a decoupled way. The concept is well known from the Apple Foundation framework.

You can also use notifications in your QF-Test scripts – just type “notifications.” in a script and press Ctrl+Space to get a documentation about the available UserNotifications and simplified sender and observer methods. We can also use such a UserNotifications with the notification observer technique – e.g. to monitor test execution state in a basic fashion (If compilation fails due to a class file version issue, make sure you’re using at least JDK 17 for your project):

import de.qfs.apps.qftest.extensions.qftest.TestRunEvent;  
import de.qfs.apps.qftest.extensions.qftest.TestSuiteNode;
import de.qfs.apps.qftest.shared.script.modules.UserNotifications;

[...]

public class WebStatusPlugin implements QFTestPlugin {  

    public final Queue<TestSuiteNode> nodeStack = new LinkedList<>();  
    public boolean running = false;  

    @Override  
    public void init() {  
        DefaultNotificationCenter.instance().addObserver(this::runStarted,
            UserNotifications.RUN_STARTED);  
        DefaultNotificationCenter.instance().addObserver(this::runStopped, 
            UserNotifications.RUN_STOPPED);  

        DefaultNotificationCenter.instance().addObserver(this::nodeEntered, 
            UserNotifications.NODE_ENTERED);  
        DefaultNotificationCenter.instance().addObserver(this::nodeExited, 
            UserNotifications.NODE_EXITED);  
    }  

    private void runStarted(Notification notification) {  
        nodeStack.clear();  
        running = true;  
    }  

    private void runStopped(Notification notification) {  
        running = false;  
    }  

    private void nodeEntered(Notification notification) {  
        TestRunEvent event = 
            (TestRunEvent) notification.getUserInfoValue("event");  
        nodeStack.add(event.getNode());  
    }  

    private void nodeExited(Notification notification) {  
        nodeStack.poll();  
    }  

}

In order to present the current test execution state in the browser, we need an embedded web server delivering this information as a web page. Unfortunately, Java 17 (which is used to execute QF-Test) does not include a simple HTTP server module yet (this has only been introduced with Java 18). But a look in the lib library of QF-Test reveals that the undertow library is bundled, which includes a simple HTTP server class. When we add 'undertow' to the devScopes property, the corresponding library bundled with QF-Test is automatically added to our implementation dependencies:

qftest {  
    version = 'local'  
    pluginClass = 'de.qfs.apps.qftest.plugins.WebStatusPlugin'
    devScopes = ['qftest', 'undertow']
}

With this library, we sketch out a simple WebStatusServer class which returns the current execution state upon request:

package de.qfs.apps.qftest.plugins;  

import io.undertow.Undertow;  
import io.undertow.server.HttpServerExchange;    
import io.undertow.util.Headers;

public class WebStatusServer {  
    public WebStatusServer(WebStatusPlugin status) {  
        Undertow server = Undertow.builder()  
                .addHttpListener(9000,"localhost")  
                .setHandler(e -> {  
                    if (status.running) {  
                        var node = status.nodeStack.peek();  
                        if (node != null) {  
                            output(e, node.getType() + ": " + node.getName());  
                        } else {  
                            output(e,"Test execution running");  
                        }  
                    } else {  
                        output(e, "Test execution stopped");  
                    }}).build();  
        server.start();

        Runtime.getRuntime().addShutdownHook(new Thread(server::stop));

        final Undertow.ListenerInfo listenerInfo = server.getListenerInfo().iterator().next();
        System.out.println("QF-Test status available at " +
                listenerInfo.getProtcol() + ":/" + listenerInfo.getAddress());
    }  

    protected void output(HttpServerExchange exchange,  
                          String response) {    
        exchange.getResponseHeaders().add(Headers.REFRESH,"1");
        exchange.getResponseSender().send(response);  
        exchange.endExchange();  
    }  
}

We load this web server in the WebStatusPlugin class as soon as QF-Test has loaded:

private void qftestLoaded(Notification notification) {  
    new WebStatusServer(this);  
}

But unfortunately, the start of our plugin now fails:

% ./gradlew startQFTestAndWait

> Task :startQFTestAndWait
Starting QF-Test (local)
1 (...) main de.qfs.lib.notifications.DefaultNotification.getDefaultExceptionHandler().ExceptionHandler.handleException(Throwable,Notification,Observer): (#147) exception: java.lang.NoClassDefFoundError: io/undertow/Undertow (...)

The undertow library appears to be bundled by QF-Test, but is not normally available in the default classpath, so we have to do some reflection calls to add the library to the classpath and instantiate it – luckily the QF-Test de.qfs.lib package has helper methods for these tasks:

import de.qfs.apps.qftest.shared.system.Native;
import de.qfs.lib.util.DynamicClassLoader;  
import de.qfs.lib.util.Reflector;

[...]

private void qftestLoaded(Notification notification) {  
    ClassLoader cl = this.getClass().getClassLoader();  
    try {  
        DynamicClassLoader.loadJarIntoClassLoader(cl, 
                Native.getVersionDir() + "/lib/undertow.jar");
        Class<?> serverClass =  
                cl.loadClass("de.qfs.apps.qftest.plugins.WebStatusServer");  
        Reflector.createInstance(serverClass, WebStatusPlugin.class, this);  
    } catch (Exception ex) {  
        System.err.println("Could not start WebStatus server: ");  
        ex.printStackTrace();  
    }  
}

Now start QF-Test, head over to [http://localhost:9000] and watch our test progress on every automatic reload!

Finally, to ship this great new QF-Test plugin, we build an all-containing .jar file using ./gradlew jar and copy the resulting file from the build/libs folder to the plugin/qftest folder of QF-Test, so it is available in all our test runs.

In an upcoming blog post, we’ll see how easy it is to use the QF-Test plugin mechanism to extend the new QF-Test 8 Assertion API. Until then, let this post inspire your creativity!

My 4-Year Experience as a working student at QFS – A Retrospective

2024-12-04T00:00:00+01:00

Over the last four years, I had the opportunity to work as a working student at Quality First Software GmbH, where I gained invaluable experience. Looking back, this time was not only educational but also extremely exciting. It allowed me to grow professionally and as a person.

When I started my role at QFS, I already had some theoretical knowledge from my studies, but working on real projects in real life was a completely different experience. From the very beginning, I was involved in ongoing projects and given the chance to actively contribute and bring in my own ideas. This allowed me to deepen my knowledge of various programming languages, investigate new technologies, and continually develop my skills. Thanks to this long-term position, I felt like I was truly contributing to the success of the team and the project.

One of the most important factors that made my time as a working student so special was the team. My colleagues were not only highly skilled and professional but also incredibly helpful and open. Whether it were technical questions I couldn’t solve on my own or bringing in new ideas, I always felt supported and appreciated. Regular team events like table football at the office, team sports, and game nights after work strengthened the team spirit and made the work environment very enjoyable.

Another key aspect was the flexible work-life balance, which was especially helpful during exam periods. In these intense phases, I was able to reduce or rearrange my hours as needed. This consideration was crucial in allowing me to perform consistently well in both my studies and my work.

In conclusion, I would like to express my heartfelt thanks to everyone for the wonderful time that gave me this valuable experience and the opportunity for personal and professional growth.

AI is a Big Topic: 11.14. is World Quality Day

2024-11-11T00:00:00+01:00

Did you know that November 14th is World Quality Day? This tradition was started by the United Nations in 1990. It deals with the question of what quality is and what quality standards mean for our work.

The task of the World Quality Report by Capgemini and Sogeti is to answer this question specifically for software quality. The 15th edition of the report, which has just been published, surveys managers from various Fortune 500 companies in detailed interviews. This year, unsurprisingly, the focus is on generative AI. Read the current report in full here.

Of course, “quality” has always been a core concept in our work. But what about QF-Test and AI, especially Large Language Models (LLMs)? In our view, AI can be used sensibly to help achieve goals, but it is not an end in itself. For our customers, QF-Test is always part of a toolchain, and it is not a problem for QF-Test if AI-generated data plays a role in such a chain.

Useful scenarios for QF-Test can therefore arise when QF-Test is used as part of a GenAI-based toolchain where structured data must be read from program interfaces or can only be entered via such interfaces, i.e. in cases where there is only a UI and no API available. QF-Test can fill such gaps much faster and more reliably than current AI systems can.

Where there is currently no need for AI support in QF-Test is the critical point of object recognition in an application UI, as QF-Test already recognizes components reliably and plausibly from the user’s point of view, thanks to its flexible and mature algorithms.

World Quality Report 2024/25: Recommendations

Here is a brief summary of the key recommendations from this year’s World Quality Report for those with responsibilities in today’s test automation industry:

Develop an organization-wide QA automation strategy to achieve consistency and increase cost efficiency.
Leverage the potential of GenAI to improve and accelerate test automation.
Future-proof QA automation tools to optimize integration with new technologies.
Understand that GenAI will not replace your quality engineers, but will significantly increase their productivity.
Evaluate the contribution of quality engineering to business goals, such as customer satisfaction, impact on revenue and overall product quality.

A little journey through time

Back in 2019, the World Quality Report reported that especially in Germany there was a high demand for end-to-end testing, including a high degree of automation. The difficulty is that the faster pace of the transition to agile and DevOps development means that the nature, form and timing of testing needs to evolve. It needs people who are experienced testers on the one hand, but who are also able to express test routines within development phases in program code and who also know how to build end-to-end test automation frameworks.

In 2021, an increase in the importance of quality assurance was again noted. The expectations of testing and quality assurance are becoming more realistic, the report concluded. Important focal points within the IT strategy were the improvement of the customer experience, greater security, faster responsiveness to business requirements and the high quality of software solutions. Artificial intelligence (AI), agile practices and DevOps, and in particular the emerging application area of the smart industry, have also had a significant impact on quality assurance and testing.

Foosball, Coffee and Code – My three-month internship at QFS

2024-09-17T00:00:00+02:00

3 to 3. Match ball. The players stand in their positions, tense, their hands sweaty. Throw-in. Team Blue has the ball. A targeted pass forward, the forward takes it. A pass to the boards, over the top, shot! The moment in which the ball finds its way towards the goal line stretches out, feels like an eternity. At the last moment, the red goalkeeper jumps in and blocks the ball. There it is already back in midfield. The ball is precisely parried by both teams again and again – until Blue makes a mistake. The ball is now at the feet of the red striker. And there it lies, safe, dangerous. A breath later, a treacherous press ball is on its way towards the goal. The ball winds its way around the defense, but the blue goalkeeper is standing firm, too firm. At the last second, the ball takes a small right turn due to the spin. You can hear it clack. GOOOOOAL! The red team wins this match 4 to 3.

If there was something I hadn’t expected from my three months at QFS, that would probably have been it. But to only talk about the twice-weekly table football matches in the this post would probably be a gross understatement of what I actually experienced and learned during this time. But first a bit about me:

I started to take an interest in computer science back in 2020, when I started working on bigger and bigger hobby projects with friends. I heard about QFS for the first time through friends and my computer science teacher, whose former students I eventually got to know at QFS.

In February 2024, four months before my own A-level exams, I did a one-week internship at QFS, where I was able to gain my first experience with the company and was also introduced to the project which would accompany me throughout the months between my A-levels and my university studies: Video streaming the screen content of a (virtual) Android device via Android Accessibility over the network to a Swing application.

If you didn’t quite get that last part, dear reader, don’t worry: I didn’t either. But once I understood what this all meant (with a little help, because Android can be so annoying), I was able to show off a working prototype, which ran on my computer separately from QF-Test, after my first week at QFS.

A bit of school and A-levels later, I thought to myself – in view of my upcoming computer science studies – that it might make sense to gain a bit more than one week of experience as a “professional” developer. Before I knew it I had a three-month temp job at QFS with the task to actually integrate my prototype into QF-Test and migrate it from Android Accessibility to the Android Media Projection API.

I know, I know: fancy!

In addition to the actual work as a developer, where I learned, for example, how to find my way around a large project, or that the debugger is quite a useful invention, there were some internal trainings in which, for example, cybersecurity or proper testing was discussed. I was also part of dev meetings that gave me a deeper insight into problems that went completely unnoticed from the outside. There I was able to learn how to approach such tasks in a structured way.

There was a clear schedule on Monday and Thursday, which gave the day a rhythm and allowed me to talk to my colleagues about topics other than the previous Jira ticket. 9:30 a.m., for example, was the coffee break (other drinks were also available, of course). 15 minutes later (sometimes after a small workout), the stand-up meeting started, during which upcoming tasks and the past weekend were discussed and sometimes a colleague’s birthday was serenaded. Then came the lunch break. On Mondays, we ate together in the meeting room. On Thursdays, we went over to the Greek or Italian restaurant or – if both were closed – to Lieferando. And between 2 and 3 pm, the long-awaited foosball game took place – a welcome distraction from that line of code that won’t run.

This created a very pleasant working atmosphere, where you don’t have to be afraid of a dull office, but can ride your bike full of anticipation from Geretsried to Gelting. I never felt alone with my project and this was one of the reasons why I was able to present another finished prototype at the end of August, which I was definitely proud of.

So in conclusion, I want to again thank everyone at QFS: thank you very much for this great experience and the nice gifts for both my birthday and my farewell to Passau. The way you do it, IT is really fun. My time with QFS has given me an insight into a functioning, pleasant and productive working life. And who knows, maybe I’ll be back in the future…

P.S.: But only if I survive math at university, I think that will be a challenge of its own.

Webswing and QFS: A Powerful Alliance for Java Application Migration

2024-05-12T00:00:00+02:00

We have been working closely with Webswing for several years to bring Swing applications to the web and do it in a safe, well-tested way. Now, Webswing have written a short blog post to talk about the benefits of this partnership:

Recognizing the synergy between Webswing and QFS, QFS integrated extensive support for Webswing applications. This integration allows comprehensive testing of Java applications in their native form, as web applications via Webswing, or both concurrently. This makes QF-Test an ideal tool for companies migrating their Java applications to the web.

We are excited about the continuous improvements coming to Webswing and will make sure that QF-Test remains the best way to accompany a migration to Webswing with reliable and easy-to-use end-to-end testing.

Learn more on the Webswing blog or on our “Integrated Webswing testing” page.

JavaFX – More alive than ever

2024-04-02T00:00:00+02:00

After the JFX Adopters Meeting in late 2021, I asked the provocative question “Is FX Even Still Alive?”. How should the question be answered today? “JavaFX – more alive than ever” is my current answer. After the Covid years, when meetings of the FX community could only take place online, this is not a matter of course.

It was an impressive demonstration of the diverse activities of the FX world: organized with a lot of passion by Christian Heilmann, we came together at the beginning of March 2024 for the JFX Adopters Meeting at ZEISS in Munich. Meeting in-person felt really good after the Covid years – and I say that as someone who usually tends to avoid meetings and crowds in general.

JFX Adopters Meeting at ZEISS

The schedule was tightly packed. I want to take a moment to thank Christian for his impressive commitment to providing the community with a platform for professional exchange. The presentations ranged from live coding and practical examples to integration projects such as embedding Excel tables.

What else stuck with me? JavaFX impresses with its outstanding security features. I am curious about the latest FX features for version 17-22, rendering in FX as a great technology for 2D and 3D, subtitles for TV using AI… and much more.

A good overview of the FX community beyond the presentations can be found on the JFX-Central website (nicely set up with JavaFX and JPro). New projects are added there and in the JFX-Ensemble on an ongoing basis.

JavaFX and QF-Test

QF-Test has supported testing of JavaFX applications since 2014 (10 year anniversary, I just realized!), at that time inspired by Wolfgang Weigend from Oracle.

“Innovations in QF-Test and development of the FX engine compared to others” was the title of my presentation this year. In short: Of the engines supported by QF-Test (Swing, SWT, Web, Windows, PDF and Android), JavaFX is by far the one with the least support effort. Web at the other end of the list requires much more support. FX is very stable and has been in use at around 150 customers of ours – often in combination with other engines – for many years.

The be-all, end-all of test automation is stable recognition. I was able to demonstrate the latest QF-Test mechanisms live and in detail – in particular SmartID. Using the example of a car configurator running in parallel on the three platforms Swing, Web and FX, I was able to demonstrate that recording and recognition of components across the technologies works out-of-the-box thanks to SmartID.

The future of JavaFX – and the JFX Adopters Meeting

Finally, here are a few other voices with their personal impressions of the JFX Adopters Meeting:

Christian Heilmann (ZEISS, organizer of the JFX Adopters Meeting):

The JFX Adopters Meeting is the only user event in the world that enables the JavaFX community to meet in person in a safe space. We deliberately refrain from recordings in order to encourage intensive and open discussions during the event.

The JFX Adopters Meeting has helped us to get in touch with many smart people, to gather the key players of the JavaFX eco-system, to get new inspiration and to learn from each other.

Having an interface toolkit where we are able to fix bugs ourselves and a stable roadmap with a long lifespan are the pillars for long-lasting projects.

JavaFX is a proven solution for this requirement. Java developers have extensive know-how with patterns that enable them to write long-lasting software with a lifespan of 20 years or more.

The people behind such technologies and how they deal with them are much more important than the technology itself. Ultimately, it’s all based on dedicated people keeping the technology alive and solid financial support from the companies. The JFX Adopters Meeting has created long-lasting friendships and a solid community.

And I ask myself: Who will you bring along to the next JFX Adopters Meeting?

Wolfgang Weigend (ORACLE Global Services Germany GmbH, Master Principal Solution Engineer):

JavaFX has demonstrated its reliability and stability in mission-critical Java desktop applications for more than 10 years.

The Java SE Support Roadmap applies to JavaFX technology, taking into account “Web Deployment Technology and JavaFX” for earlier versions. Current JavaFX releases can be found on the Java homepage, as well as the JavaFX 23 early access builds.

Remarkable is the seamless programming of graphical user interfaces with a single language.

The next JFX Adopters Meeting is planned for fall 2025, ideally the scope should be further enlarged. I will certainly be there – please include nice quiz interludes again like this year. 😉

Performance-Comparison of VMs and Docker containers

2024-01-15T00:00:00+01:00

As part of my studies in computer science at the Staatliche Berufsakademie Leipzig together with my “Praxispartner” Quality First Software, I wrote a scientific thesis on the topic of “Testing user interfaces in virtualised environments”. The thesis deals with the performance analysis of virtual machines and Docker containers for web tests.

My focus was on comparing the runtimes of tests in QF-Test in different test environments.

Schmied, Marcel: Testen von Benutzeroberflächen in virtualisierten Umgebungen (PDF in german)

Test results

The results have confirmed the assumptions that the Docker container is faster overall than the virtual machine, and that Headless Chrome is faster than the interactive Chrome with UI.

The first diagram shows that QF-Test executes test steps with low overhead more than twice as fast in a dockerised environment. The performance differences are even more pronounced in interactive sessions than in “headless” browsers.

The second diagram shows that the measured QF-Test test runs in a dockerised environment are about twice as fast as in a full VM.

The third diagram shows that the more time-consuming QF-Test nodes “DocumentWaiter”, “InstallCWRStep” and “BrowserClientStarter” are executed significantly faster in a dockerised environment than in a full VM. However, the differences are not always as significant as for the simpler nodes.

Workflow

A colleague from QFS was at my side during the entire time I was working on the project, supporting me with any questions I had and helping me to find topics, structure and organise the project.

In order to be able to compare the two test environments virtual machine and container environment, I first did extensive research on both, as well as on the topics of test automation and web testing. The detailed results of my research are recorded in chapters 2 and 3.

As an introduction to the practical part of my work, I started by setting up VMs and Docker containers with the same hardware configurations so that I could obtain a reliable test result afterwards. A Linux OS was installed on the VM and then QF-Test. For the Docker container, I only needed to download the matching Docker image, which is provided by QFS.

The Web CarConfigurator (the demo application supplied with QF-Test) was then run several times on each. This resulted in four data sets: A web test with Chrome and a web test with headless Chrome, each for VM and Docker containers. These data sets were later analysed using the methods of statistical result analysis.

Conclusion: Save time with Docker and Headless Browser

If you look at the individual node execution times, you can see that in some nodes, such as ClientWaiter or InstallCWRStep, the differences are relatively small, but in other nodes – those that interact with the UI – the differences are greater: This applies to MouseEventStep, SelectionEventStep and TextInputStep with up to 300% performance gain!

For the fastest possible test execution with the same available resources, you should therefore use a container environment with a headless browser for web tests. This saves up to two thirds of the time required for a test run compared to a virtual machine in combination with a normal interactive browser.

I personally found it a little more difficult to get started with Docker containers than with VMs. However, it is very easy with the QF-Test containers already prepared by QFS and is definitely worth it when you compare the runtime results.

How Webswing is revolutionizing the Migration of Java desktop applications to the web

2024-01-08T00:00:00+01:00

At QFS, we are in contact with many different companies using and testing Java desktop applications. These are often business critical legacy applications. Companies would love to replace these old programs with something more modern and easier to maintain, like a native web app. But completely rewriting old software from scratch takes a lot of time and money. So they stick with what they have.

A Java application in a web browser?

One company that has totally disrupted this cycle is Webswing. Webswing is a software solution for migrating existing Java software into the web browser. At its most basic, it’s a web server that mirrors the application’s Java UI into the user’s browser window. This makes it possible to translate an existing Java application into the web with very few steps.

Because there is a considerable overlap between Webswing customers and potential QF-Test customers, we have built first-class support for testing Webswing applications into QF-Test. With QF-Test you can test your Java application as it is, you can test your application running through Webswing in the browser as a pure web application, and even both at the same time! That means that QF-Test is perfectly suited to accompany companies (no pun intended) on their migration journey from Java to the web.

QF-Test + Webswing = Seamless migration testing.

The Webswing Migration Framework

But of course, many companies don’t just want to run their clunky legacy UI in a browser, they want to replace it with something more modern, a native web app.

That’s why Webswing went one step further and built their Webswing Migration Framework. Companies can use Webswing technology to migrate their Java Application UIs into real web apps piece by piece. Thanks to some ingenious technical wizardry from their developers, Webswing is able to replace individual UI components in an application with real, native Web Components – without even touching the Java application code!

An infographic illustrating how a Swing app UI is replaced piece by piece with native web components until only a tiny bit of Swing UI remains. Source: Webswing.

This means that companies can migrate old Java application UIs to the web at their own pace, and without having to rebuild the entire business logic.

And you guessed it: QF-Test is along for the ride. Because it can seamlessly interface with both Webswing and web applications, with QF-Test you are able to secure the correct functionality of your application at every step of the migration process.

Watch a YouTube video in which Webswing explains the approach of the Webswing Migration Framework.

How mgm Advances Test Automation with QF-Test

2023-11-02T00:00:00+01:00

Our parent company mgm technology partners has published a blog post about how they have been using QF-Test to supercharge their test automation efforts over the last 10 years:

“At mgm, QF-Test is employed in projects with test automation in a modular and structured manner. Automation of software testing is an integral part of software development, including quality assurance tests like QF-Test. […] We’ve achieved efficiency in our processes by adopting QF-Test on a broad scale.”

The company utilized QF-Test in a clever way to automate testing of their model-driven Enterprise Low Code Framework A12 by building “mgm A12-ATA”, a tool that automatically creates end-to-end test cases from A12 model definitions in QF-Test:

“ATA generates the code from the A12 models for QF-Test automation of the A12 forms, including scopes and SmartIDs. Moreover, it features an interpreter with a library that can control all A12 widgets on the user interface using QF-Test. With ATA, automatic filling or testing of A12 forms with minimal QF-Test implementation is feasible.”

You can read the whole post over on the mgm insights blog. And to learn more about the powerful tools for testing web applications in QF-Test, you can take a look at our product page.

Revolutionizing Test Automation with AI: Insights from TACON 2023

2023-10-16T00:00:00+02:00

The Test Automation Conference (TACON) 2023, hosted in Leipzig, Germany on September 26 and 27, was an exploration of AI’s transformative potential in the domain of software test automation. My attendance at TACON was driven by three core objectives:

to comprehend the extent of AI adoption in other organizations,
to assess our mgm standing in test automation, and
to foster connections with industry peers and test automation experts.

The TACON 2023 introduction

The role of AI in test case creation

One of the prominent topics discussed at TACON was AI’s role in test case creation.

Imagine AI generating test cases based on ticket descriptions or User Stories. The potential is undeniable, but challenges emerge. AI can produce an avalanche of test cases, often more than a human tester can handle. The critical questions arise: Who will ensure the quality of these generated test cases? Who will condense them into manageable sets? And, most importantly, who will execute and maintain this vast set of test cases?

AI for defect prediction

Another fascinating application is AI’s ability to predict defects, helping to mitigate potential issues before they reach production. By analyzing historical test data and code changes, AI becomes a proactive guardian, alerting development teams to potential pitfalls and enabling them to take corrective action early in the development cycle.

AI as a test result evaluator

Furthermore, AI assists in error analysis by swiftly identifying the source of issues through the analysis of test results and log files, enabling rapid and effective remediation.

Gravity by Smartesting

Gravity by Smartesting garnered attention for its innovative approach to test automation, leveraging AI-driven insights.

By tracking user activities in production and generating reports on the most visited actions, workflows, and pages, Gravity empowers QA teams to enhance test coverage effectively. It even generates test automation code on demand, making end-to-end tests more realistic and easy.

Sustainability in software development

Lastly, TACON reminded us of the pressing need for sustainability in software development. The digital footprint of our industry is undeniable, contributing to environmental challenges. Embracing the triple bottom line approach—evaluating software development from social, economic, and ecological perspectives—becomes a necessity.

Tools like the Digital Carbon Footprint Calculator are essential in measuring and reducing our ecological impact.

Explore your carbon footprint using digitalcarbonfootprint.eu

The fonts table shocked me! (Source: bit.ly/fontsImpactOnCarbonFootprint)

mgm’s position in comparison

Our mgm A12 low-code platform and associated A12 test automation catalog (QF-Test, Smart ID, Resolver, ATA, TDG) are and remain outstanding. The mgm model-driven Quality tools (A12 Test Data Generator, A12 Automated Test Automation, A12 Test Data Suite) distinguish us from other companies and make us very special and also very efficient!

The QF-Test approach to component detection, based on a logical UI derived from the physical UI, remains exceptional and contributes to very stable component detection.

Conclusion

TACON 2023 offered a glimpse into the future of test automation with AI at its core. From test case creation to predictive defect prevention, AI is reshaping the landscape.

As we are heading for AI transformation, it’s imperative that we do so responsibly, considering the environmental impact of our software development efforts. Let’s harness the power of AI while promoting sustainability for a better, more efficient future of software development and testing.

Mitigation of the WebP 0-day vulnerability CVE-2023-4863 in QF-Test

2023-09-28T00:00:00+02:00

Last updated 10/11/2023, 10:30 AM.

We are aware of the recently disclosed critical vulnerability in the libwebp library (CVE-2023-4863), potentially enabling remote code execution through a specially crafted WebP image file.

If QF-Test is used for opening files from untrusted sources, QF-Test versions from 4.5.0 to (including) 7.0.5 are vulnerable to this exploit through maliciously modified run logs or test suites.

Today we released QF-Test 7.0.6 which fixes this vulnerability. We advise all our users to update to the latest version.

If you are unable to update to QF-Test 7.0.6 and need to open untrusted run logs or test suites with QF-Test 7.0.5 or older, you can secure that installation of QF-Test against this vulnerability with the following steps:

Open the QF-Test system directory of the QF-Test installation.

To do this, start QF-Test, select “Help” – “About” from the QF-Test menu bar (on macOS “QF-Test” – “About QF-Test”), switch to the “System Info” tab and click the link next to dir.version. 2. Quit all running instances of QF-Test. 3. Navigate to the subdirectory bin of the QF-Test system directory. 4. Delete the directory webp from the bin subdirectory. 5. Download the updated WebP library and extract the included webp directory: Updated WebP library. 6. Copy the extracted webp directory to the bin directory.

You may need administrator privileges to perform this update.

Update 10/11/2023:

In the meantime, the embedded Chrome browser for QF-Driver on Windows has also been updated with QF-Test 7.0.7.

Besides, the Electron demos have been updated. These are downloaded automatically by the Electron demo test suites. If you want to be on the safe side, delete possibly existing old demos from the directory electron in the cache directory of QF-Test. This can be found similar to point 1 via the link dir.cache.

Annotating screenshots with QF-Test

2023-08-14T00:00:00+02:00

It is well known that, for many people, a picture is worth a thousand words. Accordingly, screenshots are an important part of QF-Test.

In my last blog article “Creating Screenshots with QF-Test” I went into how exactly to create screenshots manually.

Today I would like to demonstrate how to edit such images with QF-Test, for example to insert simple geometric figures. This can be very useful to highlight certain regions in the image – for example a click position or a certain component.

Let’s start by getting an image of a window component in a Jython SUT script:

from imagewrapper import ImageWrapper

iw = ImageWrapper(rc)

com = rc.getComponent("#Window:")
img = iw.grabImage(com)

Next, we get the necessary data for image processing, namely the width of the image and the ARGB array to be processed:

width = img.getWidth()
argb = img.getARGB()

The ARGB array contains the color values of all pixels of the image, line by line, starting from the upper left corner.

For example, to address the fourth pixel in the third row of the image, the value with index 4 + 3 * width must be addressed in the array.

The color value first specifies the alpha value, then the red value, the green value and finally the blue value of the pixel, where each value can be anything between 0 and 255.

So now we can modify the array to add a 15 × 15 pixel red cross at any position. The following code can be used to do this. For this example, it is important to maintain at least a 15 pixel gap from the edge of the image, otherwise the cross will appear messed-up.

# Get coordinates from QF-Test variables
x = rc.getInt("x")
y = rc.getInt("y")

for i in range(-15, 15+1):
    pos = x + y * width + i + i * width
    argb[pos] = 0xFFF0000
    pos = x + y * width + i - i * width
    argb[pos] = 0xFFF0000

Finally, we need to pass this array back to the image object and then save the image object in the run log, for example, to make it available to another user:

img.setARGB(argb)
rc.logImage(img)

Once everything is said and done, we get the following complete script:

from imagewrapper import ImageWrapper

iw = ImageWrapper(rc)

com = rc.getComponent("#Window:")
img = iw.grabImage(com)

width = img.getWidth()
argb = img.getARGB()

# Get coordinates from QF-Test variables
x = rc.getInt("x")
y = rc.getInt("y")

for i in range(-5, 5+1):
    pos = x + y * width + i + i * width
    argb[pos] = 0xFFF0000
    pos = x + y * width + i - i * width
    argb[pos] = 0xFFF0000

img.setARGB(argb)
rc.logImage(img)

“X” marks the spot.

In the same way you can also draw circles, rectangles or any number of similar geometric figures on the drawing surface using the corresponding mathematical functions.

I received a license upgrade. What now?

2023-06-12T00:00:00+02:00

I have a file called “license.new”. Why am I getting the error message “Invalid license”?

The license file ‘license.new’ for company YourCompany LLC is invalid.

If this problem occurs after a license update, please contact your local license administrator.

If you see this message on your screen, you have a license update file instead of a regular license file. You use this file (ideally once) in combination with a regular license file to create a new one. However, your computer does not have the previous QF-Test license to which the update could be applied.

The following instructions are therefore not so much intended for you as for the person responsible for distributing QF-Test licenses at your company.

I have received a maintenance upgrade from QFS. Should I just forward this email to the affected colleagues?

No, please don’t.

Instead, forward it only to the person responsible for distributing QF-Test licenses at your company. If you are that person, this blog post will help you.

Okay, so how do I get to the actual license file?

Well, the recipient of the maintenance upgrade will also receive the following information in their email:

You can activate or update licenses directly within QF-Test via “Help” > “Update license…”.

Please distribute the resulting new license file to the affected workstations afterwards and import it into your backup system.

You can find more information in the manual or in this YouTube video.

Therefore, the recipient’s next steps should be:

perform the license update process as described in the email
save the new license file thus generated locally
and then distribute it to the affected workstations

Backing up the locally generated license file

We have explained the dialog that can be accessed via “Help” > “Info…” in QF-Test in our previous blog post QF-Test license information explained.

At the bottom of said dialog you will not only find the specific location of the license file, but you can also navigate there directly by clicking on the file path.

Distributing the license

Every company has different mechanisms for distributing software (including associated configuration and license files). For this reason, it is difficult for us to describe the appropriate process in your environment. Your administrators would be the better contacts.

One possibility would be to use a file and/or folder synchronizing software. Or even storing them on a central file server in conjunction with the QF-Test parameter -license along with the appropriate path. For Windows domains, it could also be distributed via a Group Policy Object (GPO).

As you can see, there are any number of ways to implement this. We recommend to use the same mechanism you already use to distribute QF-Test installations and their configuration.

Okay, but why do “license.new” files and this update process exist at all?

Well, customers who use QF-Test in more than one department or location usually have one central purchasing department, but multiple license files in use. However, that department does not necessarily know which of the various license files to distribute to which locations. In the past, this has resulted in the customer ultimately using only one license file in multiple physical locations. Apart from the fact that this was against licensing conditions, the customer no longer had their full license feature set at their disposal.

Hence the “license.new” files. Because they only ever fit one license file, the purchasing department can simply distribute all license updates to all locations without having to worry about which ones belong where.

Integrating QF-Test with TeamCity in three easy steps

2023-05-11T00:00:00+02:00

QF-Test can be easily integrated with common CI tools like Jenkins and TeamCity.
In fact, a QF-Test plug-in for Jenkins is available out-of-the-box which makes things pretty easy but even with other CI tools, an integration is possible with just a few steps.

In this blog post, I will demonstrate these steps for TeamCity CI.

Important note: CI tools often suggest using a headless service for the agent process. However, for UI testing it is crucial to use a real user session to ensure that the desktop environment behaves as expected. This may not be the case if a CI agent is running as a service. That’s why we recommend starting the agent in a real user session. Refer to the Integration with Development Tools chapter of our user manual for additional information.

Step 1: Command Line Execution Build Step

Note: You need to have a project with a build configuration ready to go.

Central element for integrating QF-Test with TeamCity is the “execution” build step. QF-Test provides an extensive command line interface which is documented in the reference manual.

In TeamCity, add a new build step …

… and select a “Command Line” build step.

There,

define an appropriate step name
select “Executable with parameter”
set “qftestc.exe” as “Command executable” (qftestc.exe is a version of qftest.exe which is optimized for command line usage)
define the following parameters:

-batch -report.html report
-report.junit junitreport
-exitcode-ignore-exception -exitcode-ignore-error -exitcode-ignore-warning
-nomessagewindow
"<Full path to the test suite to be executed>"

Here is a quick explanation of the used command line parameters:

-batch: we want to run QF-Test in batch mode without QF-Test UI
-runlogdir runlog: run logs should be put into a dedicated sub-directory
-report.html report: we want to create an HTML report which we will integrate into TeamCity later
-report.junit junitreport: an additional report in JUnit format allows TeamCity to parse the test results properly
-exitcode-ignore-exception -exitcode-ignore-error -exitcode-ignore-warning: don’t return a non-zero exit code on test errors, since they would be interpreted as hard crashes by TeamCity
-nomessagewindow: Don’t display any error message windows, print everything to standard output instead

So, now we have completed the first step and can start a first test run before creating the next configuration.

Note: You need to have at least one TeamCity agent available. As a prerequisite, QF-Test must be installed on the host of the agent.

We can switch to the “build configuration page” to see the build result.

Here we should see the results of our first test run, hopefully with status “Success”.

That’s great but “Success” just means that the QF-Test process has returned a 0 exit code on the command line, but there is no information about actually successful of failed test cases.

Step 2: Test result reporting

To achieve this, we will use the JUnit reports which have been generated by QF-Test. Those contain the information about successful and failed test cases that can be interpreted by TeamCity.

Within TeamCity we need to

switch to the “Build Features” area
add a new build feature
choose the “XML report processing” with report type “Ant Junit”
create a Monitoring rule %teamcity.build.workingDir%/junitreport/report_junit.xml, which is where the JUnit report is created by QF-Test as defined in the command line build step
check “Verbose output” (optional)

Saving and then triggering another run on the build configuration page gets us a much more detailed result. Notice that there was one passed and one failed test in our run, which is expected for the sample suite I used.

So far, so good. But one important feature is still missing. For test result analysis QF-Test provides run logs that should be available from within TeamCity, as well as the QF-Test HTML reports that serve as a great overview.

Step 3: Run log and HTML Custom Report Integration

First, both the run logs and HTML report need to be defined as build artifacts so they are stored by TeamCity.
This can be done in the “General Settings” area:

report => report.zip: The report directory will be zipped up and stored
runlog => runlog The runlog directory will also get stored. Since .qrz run logs are already compressed, an additional zip step would not make sense.

Reminder: Run logs provide detailed information about the test run. HTML reports provide an overview.

Triggering another run will deliver us the respective artifacts, containing the runlog directory and report.zip, ready for downloading.

Now, only one final integration detail remains: displaying the QF-Test HTML report in a separate build tab.

To do this:

navigate to the administration page of your project
and from there to the “Report Tabs” page
to “Create a new build report tab”
with the title “QF-Test Report”
and a “Start page” of report.zip!report.html, which leads to the report.html page inside the report.zip build artifact.

After heading back to the project and running another build, open the build details by clicking the link below “Status” …

… and then the familiar QF-Test HTML report can be found right inside the new “QF-Test Report” tab.

This is just a first taste of the integrations possible between TeamCity and QF-Test.

You may want to take a look at the other QF-Test command line options documented in our reference manual. Especially the parameter -suitesfile <file> can be handy for customizing the test suites from outside the build step. Also, the TeamCity documentation includes even more advanced possibilities for integration, such as the concept of Service Messages, which communicate progress via standard output during test execution.

In case you have additional suggestions or questions about integrating QF-Test with TeamCity CI, feel free to let us know by contacting our support team.

Switching languages in QF-Test

2023-02-07T00:00:00+01:00

Did you know that QF-Test is not only available in English, but is also completely localized into German?

Admittedly, setting the language of the QF-Test interface is something most of our users never have to care about, since the system setting is used by default.

However, it does happen that you have to change the language of QF-Test for a presentation, for example.

Thats why I want to take the opportunity to explain exactly all of the ways you can set the QF-Test interface language on the different platforms supported by QF-Test.

Usually, you don’t have to worry about setting the QF-Test interface language. QF-Test will use the same language as your operating system, and will fall back to English. You can switch your OS interface language, log out and back in, and QF-Test will appear in the matching language.

However, for customers who need just a little more control, there are additional methods for adjusting this behavior.

On Windows

On Windows, QF-Test comes bundled with the configuration utility qfconfig.exe. You can launch this tool from the same directory as your qftest.exe executable, or just search for “qfconfig” in the Start menu.

Screenshot of the QF-Test Java Configuration utility.

Here, you can override the default “Language” setting with the language of your choice.

On Linux

On other platforms, the configuration utility is not available. On Linux, the preferred method of switching QF-Test language is by launching with command line parameters for the qftest start script:

qftest -J-Duser.language=de

This parameter may seem familiar. Because QF-Test is a Java application, it uses the Java APIs for managing its interface language. The language of the JVM is controlled with the command line parameter -Duser.language. By convention, to pass a command line parameter from a Java-based application to the JVM, the prefix -J is used, resulting in -J-Duser.language.

This method of course also works on other platforms.

You can learn more about all of the available parameters in the QF-Test manual chapter Command line arguments.

On macOS

On computers running macOS, starting QF-Test with command line parameters is relatively cumbersome, because the actual binary is hidden within the “QF-Test.app” application bundle.

Thats why, in QF-Test for macOS, we have added an additional section in the QF-Test application settings at “General” → “Startup”.

Screenshot of the QF-Test settings window section “Startup”.

Simply paste the following into “Startup Arguments” to always use the German localization:

-J-Duser.language=de

Of course, you can set all of the available command line arguments here. And if you ever accidentally mess these settings up in a way that QF-Test will no longer launch at all, you can find recovery instructions in the manual.

And that’s it! These are all the ways to change the interface language of QF-Test across all operating systems.

SmartID – The next generation of component recognition

2022-12-16T00:00:00+01:00

Many QF-Test users are familiar with the problem: As your tests grow, so does the component tree under “Windows and components”. It eventually grows so huge that you are forced to give up on keeping on top of individual components.

For less complex test scenarios, this is not a problem. QF-Test is designed in a way that ideally you don’t have to pay attention to the component tree at all; thanks to intelligent “record & replay”, QF-Test takes care of creating and reusing components automatically.

However, more than a few of our customers broke through that limit where maintenance of the component tree could not be left to QF-Test alone. Together with them we worked on different solutions. A frequently used pattern were generic components. QF-Test version 6.0 then introduced the superior SmartID feature.

A detour through generic components

A generic component is a component where the deciding properties are filled with a variable, so the component can be resolved to different UI elements depending on the current variable value. In the right hands, generic components are a powerful tool to keep the component tree clear and concise, thus increasing overall readability of a test suite.

But generic components repeatedly reached their limits as well. The use of the QF-Test recording function for test development was severely limited. To completely rely on generic components, individual components for every single class and generic property and all combinations of both had to be created and maintained. This quickly became tedious. It got especially unmanageable when changes in the application under test required modifications to component recognition. Then, variable values and procedure parameters had to be adjusted across all test suites until the generic components were again correctly resolved.

So, we continued working on this problem, and finally arrived at a completely new paradigm: SmartID.

SmartID – Component recognition “smart”

Different from generic components, SmartID is not just a usage pattern, but a completely new feature for addressing UI elements. During development of SmartID, our guiding principle was to keep simple things simple.

One of the strenghts of the classic component tree is QF-Test mostly taking care of maintaining the concrete components (thanks to the recording feature). The decicisive advantage of generic components is avoiding the complexity of the component tree by focusing only on relevant element properties.

SmartID in turn combines these two advantages: Taking care of the component tree is no longer neccessary, and the deciding recognition criteria can still be precisely targeted.

But how does it work? The component definition is moved entirely into the “QF-Test component ID” reference:

In a SmartID, one or a combination of recognition criteria is explicitly selected through a powerful syntax. The SmartID is declared directly in place of the QF-Test ID of the component. So, no component step needs to be referenced or generated at all. And: QF-Test is intelligent enough to even create SmartIDs instead of component steps during recording of clicks and checks, if desired.

All the while, SmartID has access to all the same recognition criteria as recorded components.

Areas of SmartID application

Thanks to SmartID it is now possible to comfortably do test-driven development (TDD) or keyword-driven testing (KDT) with QF-Test. SmartIDs do not have to be recorded, but can be derived from the specification of an even not yet developed application.

This independence from the concrete state of the SUT also enables easier integration with other test tools like Robot Framework, or technology-agnostic tests, for example of a Web and an Android client of the same application.

Available since QF-Test 6.0

Since QF-Test version 6.0, SmartID is fully available as a preview feature. Automatic recording can be activated through the “Recording” menu.

But what does “preview” mean? We want to allow ourselves to add fixes and small changes in the details of SmartID, because we have not jet implemented our full vision for the feature. However, we encourage our customers to use SmartID without fear of backwards compatibility issues. More than a few companies (ourselves included) already have SmartID in productive use.

Is this the end of the component tree?

We are conviced that SmartID will be the best way to target UI components in QF-Test going forward. We want to especially discourage the ongoing use of generic components, since SmartID outperforms them in almost all areas of application.

Still, the component tree in QF-Test will not be deprecated in the forseeable future. There will always be situations where a classic recorded component will be easier to manage or faster than a SmartID. SmartID and classic components can be seamlessly combined, so there is no need to for rivalry.

Do you want to learn more about SmartID? You can read about the nitty-gritty details in the QF-Test manual or watch the video recording of our special webinar “SmartID”.

My internship at QFS: Sorting and comparing XML files (and other things)

2022-10-24T00:00:00+02:00

“Geodata systems and optimization: Evaluation and optimization of the testability of geodata systems, in particular using QF-Test” was the topic of my internship semester at QFS. It was part of my studies in “Geoinformatics and Navigation” at the University of Applied Sciences Munich and lasted from September 2021 until the beginning of February 2022.

Geodata and also geoinformation systems have often been part of lectures during my studies and so I had an idea about how complex these applications are. They are used to manage, store and/or display geodata (i.e. information that can somehow be drawn on a map). Often such systems have an interface where various georeferenced geometries (such as estates, roads, or boundaries) are displayed on maps. Since the functionality can be very diverse and far-reaching, the topic of testability is very important and exciting.

Beyond the internship semester, I have been working as a student trainee at QFS for quite some time and was able to continue and complete my projects even after the official end of the internship.

In search of a problem

In order to get a deeper insight into the use of QF-Test at geo-related companies, I conducted several interviews with customers. On the one hand, the questions were aimed at general topics, such as whether there are special requirements for the geospatial industry in the area of quality assurance. On the other hand, I asked what daily work with QF-Test looks like and which processes are tested.

Another question was about future use: Is there room for optimization? Which areas might perhaps not yet have a working solution? With these questions I also hoped to find a main project for my practical semester.

The result of the interviews was that in most cases QF-Test already works great and is used in many ways. There were only a few difficulties concerning QF-Test, for example with the image comparison (as this can be unstable and time-consuming to maintain).

The decisive idea for my project came from Uwe Päsler from the “Staatsbetrieb Geobasisinformation und Vermessung Sachsen” (GeoSN). He suggested among other things to improve the procedure for the comparison of XML files which is part of QF-Test.

Since the data exchange of the applications used by GeoSN mostly takes place via a NAS interface, corresponding tests have been carried out using XML files for some time. In addition to the informational and geometric data of the individual objects, these XML files also contain references of the objects to each other, which leads to very time-consuming tests with correspondingly large files (10 MB and up).

An improved procedure for comparing large XML files

For a long time it was possible to compare XML files via the procedure qfs.utils.xml.compareXMLFiles of the standard library qfs.qft.

The procedure provides a variety of ways to customize the comparison, such as ignoring nodes or attributes.

The previous comparison worked via a Jython script that parses the files using the minidom parser and then compares them. The problem with minidom is that before the files can be compared, they are processed fully and represented in a DOM tree. This leads to a very high memory requirement for large files, which also explains the long runtime of it’s previous comparison.

With my new script, the speed problem is now solved. The comparison is no longer implemented using the minidom parser, but is based on the SAX parser. The abbreviation “SAX” stands for “The Simple Application Programming Interface for XML” and it’s a Python library just like minidom. In contrast to the minidom parser, the SAX parser reads the file line by line and attribute by attribute. This and the structure of the script prevents the entire file from getting loaded into memory and thus speeds up the comparison enormously, so that it only takes about one minute for a 10 MB file!

The comparison with minidom quickly reached a CPU utilization of over 90 percent for large files, or exhausted the RAM available to QF-Test. This could cause the program to crash, or to abort the comparison with an error message.

In my new version, each node of a file is read by a thread and stored in a special queue. There is a separate thread for each of the two files, which takes care of the parsing. A third parallel thread compares the contents of the two queues simultaneously and clears them afterwards. This ensures that the amount of memory required is kept within reasonable limits.

The new SAX-based XML comparison function as seen in the QF-Test standard library.

Efficient sorting

All functionalities offered by the previous procedure are also offered in the new procedure.

Sorting before comparing is also possible. However, the content of the file is stored differently, namely in a custom class derived from TreeMap. This TreeMap consists again of at least two TreeMaps. In these, the nodes of a given level are stored. As soon as the original node contains a child node, a new instance of this TreeMap is created and stored in the child node TreeMap of the original node. In the end you get a single TreeMap which holds the entire file. Now you can iterate through it and compare the individual contents.

The reason why a TreeMap is used here is that the contents can already be stored sorted by a comparator at runtime. The TreeMap also makes the process of input and output very fast.

Since the entire file must still be kept in memory, the previous sorting increases the duration of the comparison, although it is of course still faster than the same action with the previous minidom process.

Conclusion of my practical semester

Besides this big project of my internship semester, I did many smaller tasks and activities. These made the period highly varied and interesting.

During the entire internship semester, I also had support from my supervisor, who helped me a lot with the implementation of this project.

Thanks to this internship, I was able to apply, deepen and expand the skills I had acquired during my previous studies. Especially the cooperation with QFS customers was exciting and I was able to gain many new impressions.

PS from a colleague: Sarah’s work is very valuable and useful for many who work with XML data. Therefore, her solution has been built into QF-Test and released as part of the standard library for all with version 6.0.2.

QF-Test license information explained

2022-10-10T00:00:00+02:00

Let’s imagine that a new QF-Test version has just been released. And you were now wondering if your current license is valid for this version.

To get the answer to this question, you would just have to open the “QF-Test information” dialog. You can do this via “Help” > “Info…” (in macOS “QF-Test” > “About”). There you will be presented with the contents of your current license file:

That’s a whole lot of information! Who is supposed to understand this?

Don’t panic! Let’s go through the highlighted sections together and explain.

“The license expires: Never”

The most common case is where it says “Never”, meaning the license is valid for an unlimited period of time.

However, there are also subscription licenses that are only valid until a certain date.

We also ship licenses before payment is received, so that the customer can (continue to) work immediately. These interim licenses can also only be used for a limited period of time and will be replaced by the unlimited licenses after payment has been received.

“It is valid for QF-Test versions –6.0”

QF-Test licenses are generally valid for an unlimited period of time (except for subscription, interim and test licenses).

The license in the example can therefore be used with QF-Test versions that are older than 6.0 or belong to the “6.0 series” (i.e. versions that were released or will be released before version 6.1.0).

The version series specified here is always the most recent one at the time of ordering the license. You do not need a maintenance contract to use these versions.

“… and all QF-Test versions, which are released up to 12/31/2023.”

Now, there is also the maintenance contract, which not only includes the entitlement for timely and comprehensive support, but also the entitlement to additionally use all version series of QF-Test that are going to be released during the maintenance period without any further costs.

The date specified here is the end of your current maintenance period. All QF-Test versions released within this period (or before) may be used with this license.

License file: …\licenses\license

Here you can find out where exactly your license file is stored, for example in case you need it again for a new computer or want to distribute the newly created file after a renewal of the maintenance contract.

Note: In the screenshot you can see a different location than the default. So you will have a different path.

But what happens when my maintenance contract expires?

A maintenance contract is valid for a limited period of time (usually one year) and, if you agree, is renewed before it expires.

Since the license file cannot “know” whether you have renewed your contract, we send appropriate license update files (license.new)automatically after the maintenance period has been extended.

You should import this update file – even if you are using an older QF-Test version. The new license file created in this way then only needs to be saved and distributed to all affected workstations (you now know how you would find the location of this license file).

All right, all right. But why is there a version number in the license at all?

The version information within the license file is still included for backwards compatibility with QF-Test versions prior to 4.1.

Take a look at the contents of our example license file:

product = qftest
name = 
company = Quality First Software GmbH, Geretsried, Deutschland
type = Commercial
versions = -6.0
expires = 
concurrent = 5
awt = 5
win = 5
android = 5
runtime = 2
rtawt = 2
rtwin = 2
rtandroid = 2
ext9 = {"maxUpdate":"2023-12-31"}
replaces = 
key = 8B9F68262556A671
signature = ...

Older QF-Test versions are not yet able to evaluate the line starting with ext9 =, and thus neither the information about the maintenance contract contained therein. And indeed, some customers still use such old QF-Test versions for various reasons (for example, because they still have to use an outdated Java version).

Therefore, we continue to include the current version of QF-Test at the time of license creation in versions, so that even very old QF-Test versions can be operated with a current license.

From mgm’s coffee kitchen to test automation with QF-Test

2022-09-22T00:00:00+02:00

Lilia Gargouri is a project manager on the mgm Quality Team. In this interview, she describes how she found her way to QF-Test, what’s important to her in test automation, and her hopes for the closer cooperation between our companies in the future.

QFS: Lilia, we’ve been working together across companies for many years. How and when exactly did you find your way to QF-Test?

Over nine years ago now, during one of the testing phases of my project at the time, I overheard in the coffee kitchen that our QA department was being faulted for a single persistent yellow job in our automated testing pipeline. I was a senior software developer at the time and had little to do with quality assurance. So I spontaneously and voluntarily offered my help to QA. And so, on June 27, 2013, I received a link with installation instructions for QF-Test.

That day was the beginning of a success story for me with QF-Test at mgm.

QFS: What spontaneously comes to mind when you think of test automation with QF-Test?

Ease of use. It’s very visual. Self-explanatory. Very pleasant. I quickly found my way around the tool. I didn’t need any training for beginners.

QFS: What is your philosophy when it comes to test automation?

It was immediately clear to me that test automation is also software development. Test automation projects need to be set up in a structured way so that they remain readable, maintainable, and extensible over the years. If the package structure mirrors the UI structure, and the parameter names of procedures follow the order and names in the UI, then no one needs to be trained in the code. For this, names of procedures and test cases must be meaningful and instantly indicate their primary function.

Each test case must be seen as a unit and must be executable at any time, independent of the test cases before and after it.

Redundant code kills a test project faster than you think. Centralizing code is a very important key to success. It greatly reduces maintenance efforts.

Here are a few key questions I ask myself during development:

What is the focus of this test case?
What about implicit and explicit testing? Do I really need this step if an upcoming step covers the same implicitly anyway.
How do I keep track of the quality of the whole project? Do I have good test coverage?
How do I implement test cases that cover a lot, are easily maintainable and don’t take too long to execute?

Lilia from mgm and Thomas from QFS together in one of their many video call sessions

QFS: What features of QF-Test do you find most helpful, especially in terms of test development?

The introduction of Scopes and SmartIDs was a critical leap for QF-Test. It has opened the gates wide for generic solutions. Scope and SmartIDs are very easy to explain, understand and use. They are also super powerful, easy to maintain, parameterize, centralize, and bring a lot of accuracy and stability.

I switched all the test automation projects I was managing at the time to SmartIDs three months after my introduction to Scopes and SmartIDs. I also implemented a model-based software solution “automated test automation tool” that generates test code for model-based applications and uses QF-Test to automatically populate and validate the UI. This significantly reduced the implementation and maintenance efforts in the projects. After this, QF-Test and me became obviously unstoppable at mgm.

Analogous to the scopes and SmartIDs, I find the mappings in the CustomWebResolver very powerful. The accuracy to recognize a project specific component on the UI and render it in a stable manner can be controlled there. I discovered #Class\:MyOwnType: as well as #MyOwnClass: there and can address many things more precisely without my extensions affecting other types of the same class. Utterly powerful and very exciting!

QFS: How have you experienced the team at QFS behind QF-Test over the years?

Totally nice people. Very friendly, patient and knowledgeable. The exchange with the QFS team was always fruitful. I have learned a lot over the years. I made the solutions I developed with the team at QFS available internally to the mgm projects in a central test automation FAQ. QFS reminds me a lot of mgm when I joined 22 years ago. Maybe that’s why I felt very comfortable with the QFS team from the beginning.

QFS: A few weeks ago we celebrated a “wedding” between mgm Technology Partners and Quality First Software, in the future we’ll work together even more closely. What do you think about this?

I knew about this years ago when it was just an idea. To me, it was overdue.

Now that QFS is part of the mgm family, more new doors are opening for innovations, visions and mega-exciting features.

I’m very much looking forward to a promising future together. Test automation, now more than ever!

Why Can’t Developers and Testers Just Get Along?

2022-05-31T00:00:00+02:00

The relationship between software testers and software developers is fraught with tension. More often than not, it’s defined by condescension and grudging dependence on both sides.

On one hand, testers sometimes suffer from the widely-held idea that testing software is generally a less difficult or demanding task than developing software. This is exacerbated by many companies making the “rise” from QA engineer to “real” software engineer a big step on their internal career ladder.

Testers also frequently have to petition development for “extra” features such as backwards compatibility and accessibility descriptors, which are at times perceived by developers as superfluous and tedious to implement.

Developers on the other hand often only hear from QA if there is a perceived problem with their work. Nobody likes to hear they made a mistake, and not everyone can always handle criticism gracefully.

But developers actually depend on the diligence and knowledge of the QA department. I have personally witnessed how developers can build “technically correct” software without actually understanding the underlying use case. This results in idiosyncrasies in the software that are hard to spot without intimate knowledge of the business domain.

The question is: What can software businesses do to ease this tension?

Can we make building software so easy everyone can do this?

One idea might be to try and make traditional “software development” skills unnecessary for building new applications.

This was the aim of the “Low Code” movement. The idea, sometimes also called “visual programming” was to build “meta programs”, toolkits for building software for a specific domain out of visual blocks, templates, and maybe simple scripts.

This was motivated by the ongoing shortage of software engineers. This shortage leads to companies being unable to progress the software they use or sell at a sufficient pace to keep up with demand and competition.

Instead of building each piece of software in code from scratch, software testers (or even managers and end-users) would be able to use a “no code” or “low code” tool to piece together new software for their needs from existing building blocks, reducing the job of software engineers to the programming of the initial tool.

This resulting software would still need to be thoroughly tested, of course, to make sure it meets the initial specifications. But the skill set of the software tester, with their intimate knowledge of what the software should do, is suited well to both building and testing low-code software.

But the dream of making building software as easy as editing a spreadsheet has largely failed.

Most software requirements are, in the details, too complex and specific to fit into any modular system. In the best case, it’s possible to build software that kind of meets the original specs, but is filled with compromises, dumbed down and clunky to use. Software from a modular system will always require its users to adapt to it, instead of serving its users.

Software built with low-code tools is cheap and simple, but usually not very powerful or flexible. Photo: Michał Bożek, https://unsplash.com/photos/RcxR1aLw8X0

Some of these “low code” ideas have stuck around in modern website building services and some process automation tools, and found success in these limited scopes.

But they are by far not the magic bullet to enable testers to get by without relying on developers.

Can we make testing software so convenient every developer can do this?

So, what about resolving our tester-developer-conundrum in the other direction? Can we make testing software so convenient for developers that they can take care of it by themselves?

Most software developers today know about the importance of testing. Many even use testing as a core part of their development workflow, as in test-driven development.

But developers prefer the kinds of tests that they can write, run, and maintain with their accustomed tools. Developers write tests in code; unit tests that run quickly and headlessly.

And developers usually only test the code they are working on and are responsible for. Rarely does it naturally fit into a developer’s workflow to write integration tests between systems from different developers or teams, to say nothing of end-to-end tests.

Test-driven development also usually does not care about behavior on other hardware or operating systems, or on legacy hardware or software. “Works on my machine” is the mantra for the types of tests developers write to aid in their development.

If they are solely responsible for testing, developers will also tend to avoid writing code that is difficult to test with their preferred testing methods. Most importantly, this means they will write as little UI code as possible and avoid statefulness wherever they can.

The end result is software like a command line interface: technically correct, but hard to use and understand.

Command line tools are a typical developer-driven style of application.

Can we meet in the middle?

What we have just explored are not two solutions, but more like two software quality nightmare scenarios. What we actually should do is try to combine the best qualities of both approaches.

We need two things for this: Communication and respect between testers and developers.

The problem between testers and developers are not technical, they are organizational. And so they need an organizational solution.

Companies should foster a positive relationship between testers and developers. QA and Dev teams should be encouraged to work together and go a mile in each other’s shoes. Developers must be made aware of the tools and workflows QA uses for their work. Testers should gain at least a cursory understanding of the way the software is built. And companies should be structured in a way as to not encourage competition between teams, but communicate that everyone is working towards a common goal.

Can’t developers and testers just get along?

I know this is much easier said than done. But the divide between testers and developers is not as big as it may seem at times. If each is free to bring their skills and their focus to the table, together, we can build great software.

Advantages of QF-Test for developers and advantages for testers

Our kicker – once again, one of the most important “employees”

2022-05-05T00:00:00+02:00

On Wednesday the 27th of April 2022 at the career fair in Bad Tölz. QFS was represented as an exhibitor with Yann Spöri, Martina Schmid and the aforementioned company kicker. Young people from different schools visited us – admittedly mostly attracted by the table soccer. But also serious conversations. Interest in computer science is high, but QFS does not offer However, QFS does not offer the often sought-after IT specialist training.
Our offers are aimed at students of MINT/Informatics courses of study - The first application in this direction from a visitor reached us just one day later.
reached us just one day later. It may also be due to the young age and that hardly anyone with programming knowledge found their way to us. The chosen day of the fair was somewhat unfortunate for both QFS and a handful of a handful of exhibiting colleges: High school graduates did not come, the school-leaving exams in Bavaria started on that day.

In addition to the young people, however, the kicker was also for ourselves and some other exhibitors a welcome occasion for a sports session. The temperatures at the trade fair venue, the freshly thawed ice rink, were somewhat low. Heated battles kept spirits and body temperatures high and the QFS team also put up a good fight in a narrow 9:10 loss against the neighboring team from the customs, which competed with a former student world champion.

Testing Content Security Policy (CSP) protected web pages with Firefox

2022-05-03T00:00:00+02:00

The Content Security Policies are a powerful tool to make cross site scripting attacks on web sites and web applications difficult. The web server sends response headers with one or more Content-Security-Policy header rows to the browser, defining explicitly for example which Javascript code may be executed in the context of the web page.

When you test a web application with QF-Test, it has to interact closely with the application in order to recognize the components or to execute checks or to be able to record a test. QF-Test executes for example Javascript code in the context of the web page via the WebDriver interface of QF-Test – and here a strict CSP rule may interfere. Therefore, QF-Test starts the browser in a mode which ignores the content security policy header rows.

Unfortunately, this is no longer possible as easily with Firefox since version 99 – with the result that QF-Test will come up with a DocumentNotLoaded exception for applications thus protected.

In order to avoid the problem, basically, you have three options:

Execute the web application in a “test mode” without CSP headers.
Remove the CSP headers via a proxy.
Remove the CSP headers via a browser add-on.

Execute the application in “test mode”

The “test mode” is the best solution when you have influence on the web application – starting the application on a test server in a special, “CSP free” mode. Then, the problems will not occur anymore when testing with QF-Test.

Remove the CSP headers via proxy

At the start of a web test you can specify a proxy via which to load the web pages in the parameters proxyAddress and proxyPort of the procedure call to qfs.web.browser.settings.doStartupSettings. In the proxy you can then remove the impeding response headers, for example with nginx, squid, mitmproxy or with the Java proxy browsermob-proxy which can even be configured via SUT scripts.

Remove the CSP headers via browser add-on

The third option is to remove the header in the browser via an add-on, for example with ModHeader or with simple-modify-headers. This is the approach recommended by Mozilla developers.

Installation:

Create a new setup sequence with the quickstart assistant of QF-Test opening the test page https://content-security-policy.com/browser-test/. When all the boxes on the page are shown in red everything is all right (for the test), if they are green CSP is not being ignored.
Navigate in that browser to the installation site of the add-on, for example https://addons.mozilla.org/de/firefox/addon/simple-modify-header/. Since QF-Test uses a dedicated profile for the tests it is important the installation will be done with QF-Test profile.
Install the add-on via the button “Add to Firefox”.
Now, an icon for configuration will appear next to the URL on the right. Click it to open the configuration page (if the browser starts to flicker you can close the first tab).
Define a rule valid for all URLs (“Url Patterns*: *”), with the action “Delete”, the Header Field Name Content-Security-Policy and an empty Header Field Value being applied on server responses (“Apply on Response”). Click on “Save”.
With “simple modify headers” you then have to activate the add-on via the “START” icon.
Now, when you close the browser and run the setup sequence again, the boxes on the test page should all be red.
And now, even with CSP protected applications you should not encounter any CSP problems anymore with testing. Screenshot for Step 5.

The Blockchain Icon Mode – QF-Test out of blocks

2022-04-01T00:00:00+02:00

Because some results of theoretical informatics (e.g. the halting problem) indicate that certain software properties are not calculatable – and because AI-approaches like Lisp-machines, perceptrons etc. do have their limits (Vapnik–Chervonenkis Dimension) – we managed to achieve a breakthrough in the redesign of our icons. A bunch of monkeys led by the monkey Mally have designed new icons for today’s first of April, whereby the majority finding was guaranteed by a blockchain algorithm:

In case you prefer the traditional view, you can disable the icon block mode in the view menu:

Welcome to JavaLand 2022

2022-03-29T00:00:00+02:00

The conference keynote “Common Misunderstandings in Software Architecture” was held by by Dr. Carola Lilienthal Here I am, sitting in a cinema-style seat inside a huge auditorium, in front of a stage styled like an ancient library. Around me are almost a thousand enthusiastic people from all over the world. I’m nervous, but excited: I have not been around this many people in a long, long time. But it feels good. The lights dim and the conference begins.

The Conference

The JavaLand conference is one of the largest Java conferences in Europe. Here, over a thousand professionals from all over the world come together to talk about all kinds of Java topcis.

We at QFS were very excited to attend a JavaLand conference in person again this March, since that had not been possible since 2019 because of – well, you know why. Up until the last minute it was unclear if the conference would be allowed to happen as an in-person event, but in the end, a couple of developer colleagues and me took the wonky German train system across the country all the way to Cologne to learn about the latest trends in Java development and spend some quality time together and with other like-minded people. Also, there was the promise of riding roller-coasters… But more on that later.

For some on the team, this conference was the first time we ever met in person For us, the biggest win of the conference was the social aspect: Colleagues from Geretsried and Leipzig finally got to meet live and in color, some even for the first time. Together, we had a ton of fun, but also learned a thing or the other about topics relevant to our daily work.

The Talks

In “REST in Peace. Long live gRPC”, Mario-Leander Reimer explained why REST is great, but not as great as gRPC when it comes to communication between micro services In terms of speaker sessions, there was a lot of talk about Microservice architectures and the no-longer-quite-new Java 17, also some talks about applications for the buzzword technologies AI and deep learning.

In “JDK 17 – das neue LTS-Release ist da!”, the history of Java was recapitulated by Wolfgang Weigend and Hendrik Ebbers Most interesting for our daily work at QFS were talks about Green Coding (assessing the environmental impact of our software) and a talk about the “State of the Java Metrics Libraries” (because we’re all about making sure code works well).

“Balancing Cognitive Load in Teams” with Sabine Wojcieszak However, there were also some interesting talks about non-technical topics, such as a primer on the history of feminism, about balancing mental load in teams, and about the things we don’t talk about at work but maybe should.

The Park

Looking down the main street of the theme park hosting JavaLand In case of JavaLand, having the conference in person is especially important because (besides the fact that it was nice to be able to interact with speakers directly) the whole conference takes place in a huge theme park. So when strolling from one session to the next, you walk through a lovely recreation of Belle Époque France, or a Mexican canyon, or you are surrounded by a huge western-themed rollercoaster. There is also a Chinese garden. And a huge medieval castle tower permantently looms in the distance.

The entire theme park is filled with beautifully detailed worlds to explore This makes just moving from talk to talk very entertaining.

Fine dining at JavaLand 2022 And almost all the restaurants and cafés were filled with free snacks, drinks, and food (which was pretty great by conference food standards).

The famous Black Mamba rollercoaster (sadly not open for business this time) And of course, the cherry on top of the conference was the evening when many of the rides all around the park opened their gates. All attendees were free to ride rollercoasters, explore fun houses, go on the merry-go-round, or have a beer at the open bar. The technically impressive “Crazy Bats” VR rollercoaster and the dizzyingly fast spinning coaster “Winja’s Fear” were some favorites among colleagues.

Many rides and venues were open free of charge for attendees on Tuesday evening

The Takeaways

The QFS team is on the go

Personally, this was my first time attending a Java conference. I was impressed by the generally high quality of the talks and the absolutely seamless, professional organization of the event. Also, despite COVID-19 being on everyone’s mind, the conference felt pretty safe, with lots of opportunities for fresh air and a laudable mask discipline among the attendees.

JavaLand was hosted here this year. It really was a great mix of social get-together, learning new things, and getting a feeling for what’s “hot” and “cool” right now in the Java community. I think I speak for all of us when I say that I’m looking forward to being back at JavaLand next year, when it’s time for another round of “JATUMBA”!

4 Quick Tips from the QF-Test Support Team

2022-02-28T00:00:00+01:00

Day in, day out, our diligent support team reads and answers the questions of our customers coming in by mail. To pass on some of this collected support wisdom, here are some of the questions we were asked in the last weeks, as well as our answers.

How to “Check text” of a data driver item instead of a component

One of our customers had a data driver set up to query their database, and they wanted to use a “Check text” node to check the queried data instead of checking a component.

Unfortunately, the “Check text” step always relates to a UI component as source. But you can easily achieve the same thing with a short SUT script:

rc.checkEqual(
    rc.lookup("variableWithDataFromDataDriver"),
    rc.lookup("variableWithExpectedValue"),
    "Check if data driver returns expected value"
)

How to compare two data sources to each other

This one also has to do with data drivers.

A customer wanted to check database values against another dataset (in a data table). They noticed that nesting data drivers or placing two sources in one data driver did not have the desired effect. (Placing two sources in a data driver is defined as iterating over the cross product of both data sources, which is cool but not what we’re looking for here).

So, is there a way to iterate through the rows of a database using the iterator from the data table? Yes! To compare the elements in a data driver against elements from a database query, you can proceed like this:

Before iterating over the actual data, use the procedure qfs.database.executeSelectStatement from the standard library to load the reference data into a property group “expectedValues”.

Then, inside the data driver loop you can perform the checks like this:

rc.checkEqual(
    rc.lookup("variableWithDataFromDataDriver"),
    rc.lookup("expectedValues", "dataColumn:" + rc.lookup("counter")),
    "Check if data driver returns expected value"
)

How to filter error messages from the QF-Test terminal

Another customer had a question about an error message (caused by their SUT) filling up the QF-Test terminal and making them miss other important log information.

Even though QF-Test can not stop errors in an SUT it is possible to instruct QF-Test to filter certain messages from the terminal output via an output filter.

This can be done by adding the following doctag to the comment field of your “Start Web Engine” node:

@outputFilter drop [regular expression matching part of the message to filter]

How to check if a variable is set at runtime

The last customer question for today is about variables. A customer wondered how to check if a QF-Test variable has previously been assigned a value during a test run.

In general, accessing such an “unbound” variable will trigger an exception of type “UnboundVariableException”. You could always use a “Try” and “Catch” node to handle this exception. But there is a more elegant way for most use cases.

Instead of accessing the variable with $(myvariable) you can use the special default group to define a fallback value in case the variable is not set: ${default:myvariable:MyFallbackValue}.

That’s it for today! If you have your own question about using QF-Test you need answered, feel free to drop a line to our support team.

Running Website Tests in Parallel with QF-Test

2022-02-15T00:00:00+01:00

From time to time, customers approach us asking if QF-Test can be used to run tests in parallel. The advantages of this are obvious: Running multiple tests at once could improve test execution time a lot (if raw system performance is not a bottleneck) while saving lots of infrastructure overhead because everything can run on one machine.

Our answer for years has been that this “is possible to a limited degree”. Which is still true. But in this blog post, I want to dive a little deeper and see how and to which degree exactly this can be done.

The hardest problem of running anything in parallel on one machine is the I/O interfaces. Your computer usually only has one mouse and keyboard attached for inputting things, and also only one desktop running at a time to display stuff.

This means your test suites must stay within the following boundaries to allow them to run in parallel:

Tests should not rely on hard system events for simulating keyboard or mouse input.
Tests must not rely on an exclusive desktop or user session.
Tests must be able to run in arbitrary order and must not depend on another.

Running a Test Suite in Multiple Threads

Launching a QF-Test test suite multiple times in parallel is the easy part: You can use the batch mode on the command line:

qftest -batch -threads 3 -run /Path/to/test/suite.qft

This will kind of work, but the results will most likely be a jumbled mess, since each thread will try to launch and control the same client, which will eventually fail in some way. We still have more work to do.

Starting Separate Browser Clients per Thread

First we need to be able to launch separate browser clients for each thread. QF-Test separates clients by their name, so you can use the built-in ${qftest:thread} variable when naming your client: In the “Variable Definitions” section of your test suite root step, set “client” to something like web${qftest:thread}. (Storing the client name in a variable called $(client) is only a convention, so you have to make sure your test suite actually follows it.)

Then, you should use something like the following Jython server script to configure every client browser with it’s own separate profile for settings, cookies, and caches:

import tempfile

client = rc.lookup("client")
profilepath = "%s/%s-profile" % (tempfile.gettempdir(), client)
rc.setLocal("profilepath", profilepath)
print "Benutze Profilpfad %s für Client %s" % (profilepath, client)

Dont forget to use the new $(profilepath) variable in the “Start Web Engine” node to have each browser use a different temporary directory for profile data.

Browsers without User Session in Headless Mode

If you now launch your test suite in batch mode, you will see multiple browser windows showing up and all running through your tests. This is nice, but you can still see a big problem. The browser windows all “steal” the window focus from each other on every interaction, which will very likely screw with test stability. To work around this, you’ll need to use a headless browser, a browser which runs invisibly in the background, without requiring an actual window or desktop.

To use a headless browser in your tests, set the browser type to “headless” (or “headless-firefox”, “headless-chrome”, or “headless-edge), in the “Browser type” option of your “Start Web Engine” step. Be mindful that headless browsers don’t support all features that normal browsers do.

Alternatively, you can experiment with setting rc.setOption(Options.OPT_WEB_ASSUME_HEADLESS, true) in a SUT script between “Start Web Engine” and “Open Browser Window” to tell QF-Test to treat a normal browser window as headless.

If you run your test suite again, you will see … nothing (which is a good thing in this case). You should probably add some debug logging to your suite in strategic places to make sure everything runs as expected. You can write to stdout at any time via good old println (or print() in Jython).

Dividing Your Tests Between Threads

Until now, we have only been running the same tests in multiple clients. This is not the kind of parallel testing we’re after here. We’d like each of our threads to handle a subset of tests, not have all tests run on each thread.

For this you will need to adjust your test structure itself. It is completely up to you how you want or can split up your tests. One very simple way is to use the modulo operator % to split your test cases evenly between your number of threads. You can use the special variables ${qftest:thread} and ${qftest:threads} to figure out in which thread you are and how many threads there are in total and enclose each test case with an “if” conditional with a statement like this:

$(test_case_index) % ${qftest:threads} == ${qftest:thread}

Or, more elegantly, you could implement a Dispatcher via a TestRunListener at the beginning of your test suite with the following Groovy server script:

synchronized(binding) {
    if (! binding.hasVariable("handledSteps")) {
        binding.setVariable("handledSteps",new java.util.concurrent.ConcurrentHashMap())
        binding.setVariable("localStepIndices", ThreadLocal.withInitial({[:]}))
        // Registriere TestRunListener
        notifications.addObserver(notifications.NODE_ENTERED, { args ->
            def step = args.event.getNode()
            if (step.getType() == "TestCase") {
                def thread = Thread.currentThread()
                def handled = binding.getVariable("handledSteps")
                def id = step.getUid()
                def localStepIndices = binding.getVariable("localStepIndices")
                def localStepIndex = localStepIndices.get().get(id)
                localStepIndex = localStepIndex == null ? 0 : localStepIndex+1
                localStepIndices.get().put(id,localStepIndex)
                id = "${id}#${localStepIndex}"

                def existing = handled.putIfAbsent(id,thread)
                if (existing !== null) {
                    // Überspringe Testschritt ${id} in ${thread}, wird bereits durch ${existing} ausgeführt
                    rc.skipTestCase()
                } else {
                    // Führe Testschritt ${id} in ${thread} aus
                }
            }
        })
    }
}

We Did It! Or Did We?

If you added some print statements to your test suite, you should now see that QF-Test will run your test batches out of order. Congratulations, you did it!

Still, there remain some caveats.

Firstly, when editing or adding tests in your suite, you will have to be very careful to not break any of the boundaries established at the start of this post.

Also, the algorithm for splitting your tests between threads is far from optimal from a performance perspective. Different test batches could take very different amounts of time, resulting in idle threads. A more complete solution would include some dispatch scheduler to assign test batches to free threads on-demand.

Working around all of these caveats can easily get very complicated. So complicated that it may in fact be faster, cheaper, and more stable to run multiple instances of QF-Test on separate machines or VMs instead, and use some existing dispatch infrastructure.

So in the end, we come back to were we started. Parallel test execution inside of QF-Test really is possible to a limited degree. But you should investigate if it’s a good fit for your test suites and your System Under Test before diving in head-first and investing a lot of time and effort.

Addendum 1: Licensing

If you want to run multiple instances of QF-Test in parallel, you’ll need an individual license for every instance, just like when QF-Test is running in a CI/CD environment.

When launching in batch mode, QF-Test will let you know you if your licenses are insufficient for the number of threads.

Addendum 2: Launching Parallel Tests in Daemon Mode

Depending on your environment, you may also want to try QF-Test’s daemon mode to run multi-threaded tests remotely.

To start the daemon process, run the following command on the host:

qftest -daemon -daemonport=3544

Then, you could trigger parallel tests on the host via the following Jython server script:

from de.qfs.apps.qftest.daemon import DaemonRunContext
from de.qfs.apps.qftest.daemon import DaemonLocator
from java.util import Properties

host = "localhost"
port = 3544
testcase = "%s" % (rc.lookup("qftest","suite.path"))
timeout = 60 * 1000

def calldaemon(host, port, testcase, timeout=0, count=1):
    daemon = DaemonLocator.instance().locateDaemon(host, port)
    trd = daemon.createTestRunDaemon()

    contexts = trd.createContexts(count)
    if not contexts:
        raise UserException("Could not create %d run contexts. Not enough licenses available?" % count)

    # start tests
    for i in range(0,count):
        contexts[i].runTest("%s#Run web tests.Test %d" % (testcase, i % 2), bindings)

    # wait for end
    for i in range(0,count):
        if not contexts[i].waitForRunState(DaemonRunContext.STATE_FINISHED, timeout):
            # Run did not finish, terminate it
            contexts[i].stopRun()
            contexts[i].rollbackDependencies()
            if not contexts[i].waitForRunState(DaemonRunContext.STATE_FINISHED, 5000):
                # Context is deadlocked
                rc.logError("No reply from daemon RunContext.")
            rc.logError("Daemon call did not terminate and had to be stopped.")
        result = contexts[i].getResult()
        log = contexts[i].getRunLog()
        rc.addDaemonLog(log)
        contexts[i].release()

    return result

result = calldaemon(host, port, testcase, timeout, 4)
rc.logMessage("Result from daemon: %d" % result)

For more tips and information about parallel load testing, see the user manual chapter “Performing UI-based load tests”.

Working as a Team with QF-Test – Dos and Don’ts

2022-02-02T00:00:00+01:00

“Teamwork: A few harmless flakes working together can unleash an avalanche of destruction.” — Justin Sewell

Sure, a team of testers can handle extensive test projects faster and better than a single person. But some things that may seem simple at first actually can become much more difficult when working in a team.

Probably every team knows the dreaded merge conflict that arises when several people have made changes to a file at the same time that contradict each other.

Resolving such conflicts is time-consuming and error-prone, so it pays to do what you can to avoid them in the first place.

And even if you don’t run into a conflict, two commits may still do the same thing in different ways, which is another way to unnecessarily complicate your project.

During the development of QF-Test, we too racked our brains on how to avoid such conflicts and built some useful tools for that directly into QF-Test. So here come some of the Dos and Don’ts of teamwork with QF-Test.

1. Don’t Reinvent the Wheel

When you first start implementing your test suites, you will come across many small tasks that need to be done. But you don’t have to reinvent all these things! QF-Test comes with the standard library qfs.qft, which bundles many frequently needed functions. You need to compare two numbers? qfs.check.compareTwoNumbers is your friend! Need to interact with an external database? qfs.database.executeStatement takes the boilerplate code off your hands! Send an email? qfs.utils.email.sendEmail! Reset browser cookies? qfs.browser.settings.deleteCookies! We could list many more examples, but you should just browse our library yourself.

And if you do need to invent something that’s missing from qfs.qft: Start your own standard library! Collect your basic procedures in a common file and keep your team informed about new developments.

2. Don’t Put Everything in One File

The number one cause of hairy merge conflicts is simultaneous editing of a file by multiple people. You will also experience this during test development with QF-Test when working in a team on a large .qft file.

Our recommendation is therefore to split up larger test suites. In our years of working with clients with test projects of varying sizes, the following basic split has proven effective:

One file with all basic procedures
One file with all windows and components
Individual files with test cases and procedures as needed

In order for a test suite to access components or procedures from another file, that file must be listed in the details of the “Test suite” node under “Include Files” (ideally as a relative path).

3. Import Instead of Copy

When you split your test suites apart, you may be tempted to simply drag and drop or Ctrl+X, Ctrl+V your procedures and components from one suite to another. The better way is to use the import functionality from the target suite under “File” › “Import…”. Now QF-Test will take care of updating all internal references correctly.

Ideally, you also create new tests this way: Start developing the new test in a completely independent temporary test suite. After finalizing the test, import the individual parts (test cases, components, and procedures) into the corresponding existing test suites. This way, you can conveniently use, for example, the recording functionality of QF-Test without worrying about messing up your global component tree.

4. Use QF-Test Projects

When you start splitting up your suites, you will notice that QF-Test always tries to keep references to components or procedures in your suites up to date, even if the referenced object has been moved. For this to work smoothly, you should organize your files into a QF-Test Project.

To do this, select “File” › “New Project” and navigate to the root directory of your project. The .qpj file thus created represents your project, to which all files in the same directory and subdirectories belong. QF-Test will monitor the references in all files and automatically update them as needed.

As a bonus you will get a convenient overview of the files in your project in an additional sidebar in QF-Test whenever you open the associated .qpj file.

5. Talk to Each Other

Despite all efforts, no project management method or technical solution can replace the most important component of teamwork: Communication. For joint work to succeed, you need to be in regular conversation with your team members, be it at a regular standup or a random encounter at the coffee machine.

Clarify responsibilities and plan tasks so that no one gets in another’s way.

Only if everyone knows approximately what the other members of the team are doing and planning can they adjust to each other and ensure, for example, that two people are not working on the same test case or even that one team member is reworking part of your test architecture while others are actively working with this architecture. Even the most sophisticated project strategy cannot protect you from such problems.

Is teamwork still giving you a headache? We recently did a video webinar about teamwork with QF-Test. You can find the full-length video recording of it here:

Jan. 2022 – Dos and Don’ts when Working with QF-Test as a Team e.g. Version Management (GIT, SVN), Import, Projects.

Presentation and Testsuite

20 years Quality First Software GmbH – Happy Birthday

2021-12-28T00:00:00+01:00

Today 20 years ago, on December 28, 2021, Quality First Software GmbH was founded by Gregor Schmid and Karlheinz Kellerer to replace QFS as a GbR. First, also international customers existed already at that time. When asked about the GmbH foundation date – why so close to the turn of the year – Gregor Schmid remembers: “This was a date request of the notary’s office and unfortunately resulted in having to prepare a whole GmbH balance sheet 2001 for the 4 days in 2001.” Such small start-up difficulties provide amusement 20 years later. There was no plan to lead the company QFS in 10 or 20 years to this or that point regarding turnover, number of employees or technology. All the more pleasing that the high flexibility in all areas, also disparagingly called “business by accident” by some, has led to a permanent but slow growth with a great dedicated team of 25 people and more than 1,400 customers worldwide.

Quotes from colleagues and their first QFS memory

Wow K. is leaving Siemens and doing Quality First with G., sounds exciting.
They are doing something with Android and smartphones, I don’t know if that fits you.
We use QF-Test for testing from the EAI tool. Check it out and work your way in.
QFS has home offices.
Hello, would any of you be interested in editing videos for my work as a student/student job?
I have a very nice employee oriented client in Geretsried.
There might be a company nearby, take a look at the site – got it from badminton training. These are the ones from the colorful house.
QFS, a company with a broader horizon
If you don’t want to tinker with computers anymore, but want to program something, there’s a company where I’m in sales, they make software to test software. It’s a fun bunch and there’s kicker table!
My boss allows us 2h per week free development time where you can also just try something out. Actually he would like to allow more, but we are too small a company and there is too much to do…
…is a young company that develops software and is quite successful with it; they just bought an (office) house…
QFS, the small nice company with super sympathetic people, always worth a visit. The first company that really takes quality seriously and doesn’t just write it on the website.
Very sympathetic company, I think I could feel really comfortable there.
I waited outside – because I was too early – “you can ring the bell”.
I have discovered a new TA tool, from a German company, with German-speaking support, and even already got an answer that it is possible to play long clicks!
I got the most heartwarmingly friendly rejection ever from T. at the time.
Do you really think that QF-Test has hardly any bugs?
We all have kids and we know how important family life is, that’s why 2 days a week are home office.
What a “cool” idea, that would have definitely saved me a lot of complex JUnit tests.

“Is FX Even Still Alive?” – JavaFX Adopters Meeting 2021

2021-12-22T00:00:00+01:00

Is FX even still alive? From QFS’ point of view, we can answer this question with a clear “yes”: JavaFX is alive and well and has been in use via QF-Test/FX in multiple large projects since 2014.

The lovely thing is that QF-Test/FX requires little maintenance, and tests of JavaFX are very stable.

Originally developed by Oracle and made an official part of the JRE with Java 8, JavaFX was removed again with Java 11 and continued as the open source project OpenJFX - still mainly maintained by Oracle, but in cooperation with Gluon and other active participants. On a technical level this move was quite reasonable and OpenJFX is very much alive, as also proved by the JavaFX Adopters Meeting of November 16.

Invited by Zeiss, the aforementioned players and many other people from Europe interested in JavaFX came together online to share their experiences and ideas around JavaFX. Among them, for example, was JPro which brings FX into the web browser – and of course can be easily tested with QF-Test. For QFS, our own Gregor Schmid and Michael Höber attended.

Uncertainty and Criticism

So why does the question “Is JavaFX still alive?” come up again and again?

If you search for JavaFX on the internet, you will find some amount of uncertainty and criticism from voices marked by the disastrous marketing behavior on the part of Oracle at that time. The ‘net does not forget. At least the tanker Oracle is now slowly moving in a different direction. Nevertheless, JavaFX’s notoriety caused by this history does not do justice to its outstanding technical quality.

We at QFS do not experience JavaFX as the long awaited successor of Java Swing. Yes, a part of our customers who migrate their applications away from Swing rely on JavaFX. But a much larger portion is moving toward Web technologies instead. The biggest point in favour of JavaFX is the “perceived quality” of projects built with it. Where we accompany JavaFX projects more intensively in testing we generally see high application design quality and consistently built test architectures.

What can we do?

So the participants of the JavaFX Adopters Meeting came back to the core question: How can the awareness and popularity of JavaFX be improved? A patent remedy is not emerging. Maybe Oracle will increase its visible efforts again in the foreseeable future – that would be helpful at any rate, especially since JavaFX (and the associated logo!) is a trademark of Oracle and may not be used freely. Therefore there was also a serious discussion whether the – less burdened – name OpenJFX should be emphasized instead. A nice, custom logo would be the urgent first step for this.

In any case, we would need companies whose core applications are based on JavaFX to actively support this project. They could, for example, commit themselves to communicating their framework preferences more clearly. Or they could contribute to the further development of JavaFX by transferring their own improvements to the OpenJFX project, enter them at JFX-Central or JFX Ensemble or support Gluon by signing a maintenance agreement. After all, Gluon manage a significant amount of the development efforts and they are the central point of contact for the binary packages.

So we do have a chance over the next few years to make JavaFX known to the broader Java developer community for what it could be: A high-quality, modern, mature replacement for Java Swing.

No Log4j Vulnerability in QF-Test

2021-12-13T00:00:00+01:00

Last update: 2022-05-19

QF-Test and the QF-Test license server are not vulnerable to Log4shell attack CVE-2021-44228

In the last days, a vulnerability in the popular open source library log4j has been reported, nick-named Log4shell. QF-Test is not (and has not) been vulnerable to this attack, for a number of reasons:

QF-Test does not directly use log4j, but has always used its own logging system which does not allow any string interpretation or code evaluation.
The JREs bundled with QF-Test – Oracle JDK and OpenJDK – have shipped with a default setting that should prevent exploitation since Oracle JRE 8u121 in 2019, the variable com.sun.jndi.rmi.object.trustURLCodebase is set to false or null by default, disallowing access to remote resources, and it is not changed by QF-Test. Our software currently ships with JRE 8u292.
Normally QF-Test is not executed in a server mode, so intruders cannot inject any string with malicious content. But also in daemon or license server mode QF-Test does not require any log4j classes.
None of the third-party libraries shipped with QF-Test contain log4j in any version vulnerable to the log4shell attack*. This has been verified by us as described below for all current and past official QF-Test versions back to QF-Test 3.4.11 which predates the first vulnerable log4j version 2.0beta9.

*) The log4j classes, which were embedded in the bundled version of the external ui-automation library before QF-Test 6.0.0 are known to not be vulnerable to the CVE-2021-44228 attack.

How to verify yourself that no vulnerable version of log4j is in use

To verify that no vulnerable version of log4j is bundled with QF-Test, you can use the Log4j detector tool. First, download the library to your system, e.g. to C:\TEMP. Open a command shell and navigate to the directory where you placed the file in. Then you can use the Java bundled with QF-Test to run the detection tool like this:

# On Windows:
> "C:\Program Files\QFS\qftest\qftest-5.3.4jre\win64\bin\java.exe" -jar log4j-detector-2021.12.13.jar "C:\Program Files\QFS\qftest\"
-- Analyzing paths (could take a long time).
-- Note: specify the '--verbose' flag to have every file examined printed to STDERR.
-- No vulnerable Log4J 2.x samples found in supplied paths: [C:\Program Files\QFS\qftest\]
-- Congratulations, the supplied paths are not vulnerable to CVE-2021-44228 !  :-)

# On Linux:
> /data/install/qftest/qftest-5.3.4/jre/linux64/bin/java -jar log4j-detector-2021.12.13.jar /data/install/qftest/
-- Analyzing paths (could take a long time).
-- Note: specify the '--verbose' flag to have every file examined printed to STDERR.
-- No vulnerable Log4J 2.x samples found in supplied paths: [/data/ins tall/qftest/qftest-branch]
-- Congratulations, the supplied paths are not vulnerable to CVE-2021-44228 !  :-)

# On macOS:
% /Applications/QF-Test.app/Contents/PlugIns/*.j*/Contents/Home/jre/bin/java -jar log4j-detector-2021.12.13.jar /Applications/QF-Test.app/Contents/Resources
-- Analyzing paths (could take a long time).
-- Note: specify the '--verbose' flag to have every file examined printed to STDERR.
-- No vulnerable Log4J 2.x samples found in supplied paths: [/Applications/QF-Test.app/Contents/Resources]
-- Congratulations, the supplied paths are not vulnerable to CVE-2021-44228 !  :-)

(In this example, default paths are used. Please adapt them to the requirements of your system.)

Verifying your own plug-ins

If your tests rely on external plugins not bundled and shipped with QF-Test you may want to make sure that these plugins are clean, too. From the QF-Test menu, open “Help->Info…” (on macOS “QF-Test->About QF-Test”), select the “System info” tab and follow the link to “dir.plugin”. In your shell, execute the command from above, but replace path in the last argument with the path for “dir.plugin”, for example:

# On macOS:
% /Applications/QF-Test.app/Contents/PlugIns/*.j*/Contents/Home/jre/bin/java -jar log4j-detector-2021.12.13.jar "/Users/pascal/Library/Application Support/de.qfs.apps.qftest/plugin"

Testing Excel files

2021-11-30T00:00:00+01:00

Free Trial

The procedure call creates a property variable which contains the different values of the Excel file. Hereby, the name of the property group is passed via the resultGroupName parameter. so, in the above example ${resultGroup:A1} will expand to the value located in the cell A1, ${resultGroup:A2} will expand to the value located in the cell A2 and so on.

Afterwards, it’s possible to check the different values of the excel file, for example with the help of the different procedures in the check-Package (e.g. checkValueToBeEqual). However, especially if you do have a large Excel file, creating so many procedure calls can be cumbersome. In this case it makes sense to create a reference Excel file. Then QF-Test can read both the Excel file and the reference Excel file into different property groups. The following Jython server script can then be used to compare the property groups with one another:

p1 = rc.lookup("groupName1")
p2 = rc.lookup("groupName2")
props1 = rc.getProperties(p1)
props2 = rc.getProperties(p2)

errors = []

for p in props1.keys():
    if props2.containsKey(p):
        v1 = props1.get(p)
        v2 = props2.get(p)
        if v1 != v2:
            errors.append("%s has value %s in %s but %s in %s" % (p, v1, p1, v2, p2))
    else:
        errors.append("%s in %s but not in %s" % (p, p1, p2))

for p in props2.keys():
    if not props1.containsKey(p):
        errors.append("%s in %s but not in %s" % (p, p2, p1))

if errors:
    rc.logError("\n".join(errors))

Excel files can be read into QF-Test using a simple procedure call where the procedure qfs.utils.files.readExcelFile must be called:

How to test browsers like Brave, Vivaldi, Yandex, Iron

2021-10-19T00:00:00+02:00

Out of the Box, QF-Test supports testing Websites using Chrome, Firefox, Edge, Opera, some others, and even good ol’ Internet Explorer.

But there are a lot of other, lesser known browsers that you may want to test with anyway, such as (takes deep breath)

the open source Chromium,
the crypto-enabled Brave
the privacy-focused Iron,
the incredibly customizable Vivaldi,
or the Russian Firefox-offshoot Yandex,
(phew, did we miss any?).

Luckily, all of the above (and a lot of even smaller browsers) are based on the Chromium engine that is behind Google Chrome.

Because QF-Test understands the CDP protocol for controlling Google Chrome, it can also work with a lot of the browsers above, but we’ll need to trick QF-Test a little for it to agree to run those unsupported browsers.

To run any Chromium-based browser, we need to make QF-Test believe it’s simply running Chrome. This only takes four edits in our “Start web engine” node:

We keep “chrome” as our “Browser type” of choice to make QF-Test think what we’re doing is business as usual.
Under “Directory of browser installation”, we sneakily insert the path to our Browser, for example “C:\Program Files\BraveSoftware\Brave-Browser\Application” (or “/Applications/Brave.app” on macOS).
To tell QF-Test how the .exe file of the Browser is called (Certainly not “GoogleChrome.exe”!), add the following to the “Executable parameters” section:
-Dqftest.web.webdriver.browserFileName=brave.exe (Or whatever your browser of choice is called). This is not necessary on macOS.
To make sure your custom browser does not share any settings or caches with your Google Chrome installation, you should override the profile path with a second parameter: -Dqftest.web.profilepath=C:\tmp\brave-profile.

QF-Test controlling the officially unsupported Brave browser.

Trick QF-Test into running a Chromium-based browser it does not officially support Of course, you can still use $(variables) in pretty much any field of the “Start web engine” node. You can easily customize the browser to use and even run the same test suite in a bunch of different browsers using data drivers.

This whole thing was an interesting experiment, but in the end you should keep the following in mind: Because these browsers all use the same Chromium engine under the hood, they will generally perform very similarly. In most cases, you will be fine just sticking to the default “chrome” Browser type for all your Chromium-based-browser-testing needs.

Another clear advantage: If something doesn’t work properly during web testing with one of the officially supported browsers, the QF-Test support team at support@qftest.com will be happy to help.

TestRunListeners in QF-Test

2021-08-18T00:00:00+02:00

The TestRunListener interface can be used to execute additional actions before or after the execution of each node or in the case of any exception / error.

This actions can (for example) be used for testdocumentation or error analysis.

These actions can for example be used for test documentation or error analysis. In the following some TestRunListeners are introduced (Jython server scripts):

Variable monitoring

The following TestRunListener always logs an error when the value of a certain variable (in the example offset) changes. This is especially helpful when debugging a test suite if it is not clear where exactly the variable is assigned a certain value:

varname = "offset"

from de.qfs.apps.qftest.extensions.qftest import AbstractTestRunListener

def checkVar(state, var):
    try:
        val = rc.lookup(var)
        if(val != state):
            state = val
            rc.logError("Variable '%s': %s" % (var, state))
    except:
        if state != "":
            rc.logError("Variable '%s' nicht verfügbar" % var)
            state = ""
    return state

class VarChanger (AbstractTestRunListener):
    def __init__(self):
        self.state = None
    def nodeEntered(self, event):
        self.state = checkVar(self.state, varname)
    def nodeExited(self, event):
        self.state = checkVar(self.state, varname)

global varChanger

try:
    rc.removeTestRunListener(varChanger)
except:
    pass
varChanger = VarChanger()
rc.addTestRunListener(varChanger)

Alternatively, it’s easy to adapt this script so that the TestRunListener always logs an error when a certain value is assigned to the variable.

Error sound

It can happen that a user starts several tests on several computers at the same time. The following TestRunListener helps if it can happen that the test execution on a machine – for whatever reason – can come to a standstill; The TestRunListener always plays a tone if the execution of a node takes longer than a certain specified time limit. It is possible to redefine this time limit for a node using a @informUserTimeOut doctag. (Note: The sound output probably only works on Windows)

# A TestRunListener that informs the user when the current testrun has stopped for more then a certain period of time.

# some imports we need later
from de.qfs.apps.qftest.extensions.qftest import AbstractTestRunListener
from java.lang import System, Thread
from java.awt import Toolkit
import time, re

# The default time in ms after which to inform
# the user.
TIME_IN_MS_AFTER_WHICH_TO_INFORM_THE_USER = 2 * 60 * 1000

# The function that is used in order to inform the user
def informUserAction(lastActionTimestamp):
    """ This function is called whenever the test-
        runlistener detected the timeout. In oder words,
        this function should play a sound or do something
        else in order to inform the user.

        :lastActionTimestamp: The timestamp of the last action.
    """
    print "=== HEY %s ms passed, since we entered the last node! ===" % (System.currentTimeMillis() - lastActionTimestamp)
    # output a sound on windows
    Toolkit.getDefaultToolkit().getDesktopProperty("win.sound.exclamation").run()

# the thread that will inform you once the timeout is reached ...
class InformingThread(Thread):
    def __init__(self):
        self.updateTimeout(TIME_IN_MS_AFTER_WHICH_TO_INFORM_THE_USER)
        self.stop = False
    def updateTimeout(self, timeout):
        self.lastAction = System.currentTimeMillis()
        self.errorAfterTimestamp = self.lastAction + timeout
    def run(self):
        while not self.stop:
            time.sleep(1)
            if System.currentTimeMillis() > self.errorAfterTimestamp and not self.stop:
                try: informUserAction(self.lastAction)
                except: pass

# the testrunlistener that keeps track of when a new node get's enetered ...
class InformUserWhenHaveBeenIdleTestRunListener(AbstractTestRunListener):
    def __init__(self):
        self.thread = InformingThread()
        self.thread.start()
        self.myRegex = re.compile("@informUserTimeOut\\s+(\\d+)",  re.DOTALL)
    def nodeEntered(self, event):
        timeout = 0
        comment = event.getNode().comment
        if comment != None and comment.find("@informUserTimeOut") != -1:
            match = self.myRegex.search(comment)
            if match:
                timeout = int(match.group(1))
        self.thread.updateTimeout(TIME_IN_MS_AFTER_WHICH_TO_INFORM_THE_USER if timeout <= 0 else timeout)
    def runStopped(self, event):
        self.thread.stop = True

# register the testrun listener
global informUserWhenHaveBeenIdleTestRunListener
try:
    informUserWhenHaveBeenIdleTestRunListener.thread.stop = True
    rc.removeTestRunListener(informUserWhenHaveBeenIdleTestRunListener)
except:
    pass
informUserWhenHaveBeenIdleTestRunListener = InformUserWhenHaveBeenIdleTestRunListener()
rc.addTestRunListener(informUserWhenHaveBeenIdleTestRunListener)

Counting check nodes

The following TestRunListener counts how many check nodes where executed during test execution:

from de.qfs.apps.qftest.extensions.qftest import AbstractTestRunListener

class StatisticTestRunner (AbstractTestRunListener):
    def __init__(self):
        self.steps, self.checkSteps = 0, 0
    def runStopped(self, event):
        print "steps: %s\ncheckSteps: %s" % (self.steps, self.checkSteps)
        self.steps, self.checkSteps = 0, 0 # reset counts
    def nodeEntered(self, event):
        self.steps += 1
        if (event.getNode().getType().startswith("Check")):
           self.checkSteps += 1

global statisticTestRunner

try:
    rc.removeTestRunListener(statisticTestRunner)
except:
    pass
statisticTestRunner = StatisticTestRunner()
rc.addTestRunListener(statisticTestRunner)

General remarks

Similar to SUT Scripts you should avoid calling rc.callProcedure from within a TestRunListener. Although it’s often possible to call rc.callProcedure from within a TestRunListener, as this can lead to unexpected problems.
Instead of derivating from TestRunListener you should always derivate from AbstractTestRunListener.

Creating screenshots with QF-Test

2021-07-29T00:00:00+02:00

After the execution you can find the screenshot in the corresponding run log.

Run log Screenshot in QF-Test

It is also possible to save the screenshot on the hard drive. A Jython Server / SUT script with the following content helps here:

from imagewrapper import ImageWrapper  
iw = ImageWrapper(rc)  
screenshot = iw.grabScreenshot()  
iw.savePng(r"C:/temp/foo.png", screenshot)

It is relatively easy to take a screenshot of the complete picture with the help of QF-Test. For example, to create a screenshot of the current desktop, only the procedure logScreenshot must be called. (The following blog article explains how to insert a procedure call.)

It should be noted that screenshots can consume a lot of storage space. In this respect, a screenshot should not necessarily be created after each step – even if this is relatively easy to do. (With the help of a Testrunlistener)

In practice the whole thing looks like this:

JIRA Examples with QF-Test

2021-06-17T00:00:00+02:00

On a popular demand we have created examples on how QF-Test can be integrated with JIRA. This is done via the JIRA REST API The examples are built with the help of the “HTTP Server request” node which is the foundation of the Web Service Testing capabilities in QF-Test.

The test suite can be found in our public Git repository https://gitlab.com/qfs/demo-suites/-/tree/master/jira.

You are free to use it and create push requests. In this way you can contribute your generic procedures and help others form the community.

Useful info

Use a proxy to sniff the traffic while you are building your integration via the “HTTP Server Request Node” in QF-Test. More examples for Web Services testing can be found in the demo test suite webservices_testing.qft shipped with QF-Test

Test automation of KeePass with QF-Test – Seminar paper

2021-04-29T00:00:00+02:00

“Why do my automated UI tests break?” that was the seminar question of a student. With this, he asked the central question for test automation. Because the success of this, as well as the daily work of QA, stands and falls with the recognition of the objects – or their failure/breakage of the tests.

His research object was KeePass for Windows in the versions 2.0 to 2.18 (=SUT, System under test). With QF‑Test version he created KeePass2.00 UI tests, which he ran in the following 17 versions.

To the seminar paper

The immediate results of the KeePass UI tests with QF-Test If tests failed, what was the reason? He found three main reasons (besides single reasons like disappearance of a component) and from all of them he identified the category “Trivial renaming” as the main cause.

How representative is his result for the experience of the support team of QFS?

There it was confirmed to me that – when tests are finished and running – equally trivial renaming actions in the further development of the SUT make up the main focus of the requests in the support team. That is, UI tests break quite often because the recognition characteristics of the component have changed.

So dear developers:

Please set unique names or IDs and never ever change them anymore.

As written in the seminar paper, changing “Lock Windows” to “Lock Window” is clearly recognizable as “identical” for a developer and human, but a test automaton stumbles over it.

QF-Test is constantly developing its component detection concepts and optimizing the algorithm to make the generated tests as robust as possible against changes. It offers various possibilities to influence the detection of UI components to make them as stable as possible.

Currently a new approach is developed, which will make it possible to limit the search area for a component still more dedicated, in order to make so apart from a more stable recognition by exclusion of multiple possible components the test still more stable and beyond that also faster. In addition the maintainability is increased, since one must maintain no more complex component trees.

First QF-Test Meetup Switzerland

2021-04-23T00:00:00+02:00

Choo choo! Hop on board on the train to Switzerland!

Again and again customers are reporting back to us about their interesting solutions with QF-Test – in support, in training or as an advisor on location. We often get the question: “How do other customers do it? What problems have they encountered? What solutions have they found?” This resulted in the desire on our part to be able to show something, and in a desire for our customers to be able to see alternate solutions.

For a while we’ve had the idea to bring together the growing clientele from Switzerland. It was supposed to happen last year, but the COVID-pandemic had us looking for an alternative.

The idea just hasn’t left us alone – we wanted to go to the mountains so we planned the hike and invited the other travelers to an online meeting.

The time has finally come last Wednesday, April 14th 2021. Backpacks packed, we could start the online session five minutes before start. As punctual as a Swiss watch, all the participants were there at 5:30 – 13 QF-Test users and project managers from various companies. Three of us were there to represent QFS: Helena Aschenbrenner (sales) – thank you for organizing, Thomas Max (development) moderated the meetup, and me (testing) has the pleasure to write this blog post.

After a long, but interesting introduction round we got to see two presentations and get a look at development and testing processes, infrastructure, and testing architecture. Incredible how many possibilities of application there are for QF-Test – like a Swiss army knife! How many testing cases are executed daily? How are tests started? Both presentations focused more on the testing strategies within the projects, as well as the place and role of test automations, rather than the technical implementation of the individual test cases. It was interesting to hear that test automation is being accepted more and more, and what lead to this. We have contributed to the evening as well, with Thomas presenting QF-Test’s newest feature, which is still in the development phase.

The meetup ended with an open Q&A and discussion round. We are planning to do meetings like this regularly in the future – several participants volunteered to do a short presentation next time. With many of our customers working in the Syrius environment and implementing complex frameworks for this, there is going to be a detailed presentation about that topic as well. This meetup will also serve as an exchange of experiences between all participants and can help optimizing their specific solutions.

And this is how three QFS team members from three locations in Germany traveled to Switzerland at the same time and came back home less than two hours later. I’m already looking forward to our next travel! Who wants to come along?

Against the Corona-Blues

2021-03-02T00:00:00+01:00

For many, especially those in IT, the past year was probably similar to ours in many ways:

Since March 2020 our offices have been closed. Except for the occasional visit to take care of the plants or paperwork, all of us are working from home. We were lucky – thanks to our regular 2 days per week home-office there were no technical problems and the infrastructure providers did their job remarkably well. Basically everything works, some online meetings are even more efficient and personal than before and the market for our niche is stable enough for us to get by well.

Still, it’s not always easy to maintain high spirits and the longer the lock-down continues the more deeply the Corona-Blues takes hold. My own content and physical well-being were also on the decline due to lack of opportunity and motivation for exercise and sports. But then a stroke of luck changed things and gave me a push in a direction that is promising to thoroughly change my work-life: In a video-meeting a colleague was happily tottering away. He’d gotten a treadmill and placed it under his adjustable desk.

Printed on the shirt: So that your software runs well.

Of course! I’d toyed with that idea years before but unfortunately not gone through with it at that time, partly due to lack of suitable choices. But now I understood immediately: This is it! I already had a suitable desk so I did a quick market research, read and watched a lot of user stories, compared prices, ordered a treadmill, got it delivered, set it up and started walking.

Today’s generation of treadmills for walking, not running, is ideally suited for that purpose. The maximum speed is limited to 6 km/h – not enough for running but more than enough at a desk. With prices below € 400.- the devices are smaller, lighter and less noisy than their large running counterparts in fitness studios and energy consumption is far less.

Familiarization didn’t take long. At a relaxed pace of 4 km/h (5 with a bit of practice), typing and mouse operation are not an issue as long as I don’t need to hit exact pixels. My concentration at work is so high that I barely notice walking. The light tension resulting from this contrast even helps focusing and the increased blood flow stimulates thought. Sitting has its well-known drawbacks and standing is not much better – I can stand comfortably for maybe half an hour at most. But walking feels like there’s no limit. After a month the magic still isn’t gone and I’m walking about 80-100 km per week without much effort. And that’s on top, while doing my work, without having to cut out the time and not in competition with going for a run outside in this wonderfully warm spring. And so far without wear and tear.

Though this approach is not for everyone and not every workplace is suited, the health potential is tremendous. One colleague also started walking, another mounted his bike below his desk. The others are getting used to the waggling in video meetings and, thanks to headsets, acoustics are OK though even the better treadmills aren’t really silent and there’s the occasional heavy breath coming through the microphone.

I’m curious to find out what the next months have in store for us. How quickly or slowly “normalcy” returns and what that’s going to mean for social and work interactions, what new balances we’re going to find. But what I’m definitely going to retain after Corona is working while walking (or is it walking while working?) and I’m totally thankful for that.

Automated website audits with Lighthouse and QF-Test

2021-02-18T00:00:00+01:00

What is Lighthouse?

Lighthouse, an open-source tool which was developed by Google, makes it possible to analyze your website in the four categories performance, accessibility, best practices and SEO. To analyze each category, Lighthouse runs a barrage of tests called audits. Depending on the results of the audits, the respective category will get a score between 0 and 100 (see Figure 1).

Figure 1: Lighthouse Scores The performance category analyzes how quickly a website loads and how quickly users can access or view the content. The corresponding score will be influenced by metrics like First Contentful Paint or Time to Interactive. First Contentful Paint indicates the time before the first text or image becomes visible to users while Time to Interactive measures how long it takes a page to become fully interactive.

Accessibility ensures that people with disabilities can access content and navigate the site effectively. One audit used to determine this category’s score is checking whether images have been assigned an alt attribute. They allow reader programs to describe the image’s content for visually impaired people.

The third category makes sure that your site follows standard best practices for the web. For example, does it avoid deprecated APIs and does it only embed resources from secure sources?

Finally, SEO checks how well your page is optimized for search engine result rankings.

Integrate Lighthouse audits in QF-Test

The procedure allows you to define a warning and/or error limit between 0 and 100 for each of the four categories. For example, if the performance score is below the error limit you specify, QF-Test will alert you (see Figure 2). By setting the value of a limit parameter to -, you can also disable its checking.

Figure 2: Performance score fell below your error limit The attached test suite lighthouse.qft contains a procedure “checkLighthouseScores” to start lighthouse audits and check its results. With the help of the procedure’s parameters, you can define a warning and/or error limit for each of the four categories (Value must be between 0 and 100). If, let’s say, the performance score falls below your specified error limit, QF-Test will inform you with the respective error message (See Figure 2). To deactivate the check of a limit parameter, you just need to set its value to -.

By integrating Lighthouse audits into your automated testing, your site can be continuously checked for unexpected changes. Besides a summary that will be saved in the run log, the procedure also provides several optional parameters for further configuration. For example, the parameter “mobileEmulation” allows you to analyze your website not only in a desktop but also a smartphone environment.

How to install Lighthouse

The first step is to install the Google Chrome Browser, as Lighthouse analyzes your website using its headless mode.

Please install Google Chrome. The latest version can be downloaded from: https://www.google.com/chrome

To be able to execute the procedure, Lighthouse needs to be installed as a Node.js module.

Install the current long-term support version (LTS) of Node.js. The latest version can be downloaded from: https://nodejs.org/de
After Node.js is installed, please open the command line of your operating system (Windows: command prompt).
Use the opened command line to execute the command shown in figure 3. If Lighthouse should not be installed as a global module, do not use the -g flag. Figure 3: Install Lighthouse
To ensure correct installation, execute the command lighthouse in the already opened command line. Following output should appear: Figure 4: Lighthouse Output For further information regarding Lighthouse please visit: Lighthouse

Final hints

In case the lighthouse command cannot be found anyway, a missing entry in your path environment variable may be missing. On Windows you may add e.g. %appdata%\npm by hand.
After installing npm and lighthouse, QF-Test need to be restarted in order to get aware of changes in the envrionment settings.
Some lighthouse results are dependent on the network connection and machine performance, the audits are executed on. There should be no constraints or your need to take them into account respectively.

Change language in QF-Test

2021-02-01T00:00:00+01:00

Notice: An improved version of this post can be found here: Switching languages in QF-Test.

QF-Test exists in German and English. You can switch the language in QF-Test. It is easy you just need to know how to change the language settings.

1. Find qfconfig.exe

a. Click your way to your QF-Test installation up to \bin

b. Or right click on the the QF-Test file association on your desktop and then click on “Open file location”

e.g. C:\Program Files\QFS\QF-Test\qftest-5.2.1\bin

c. Or search for QF-Test in the Windows menu and click directly on “QF-Test Java configuration” (Continue with 3.)

2. Open qfconfig.exe

3. Select your required language

4. Click “OK” and then you’re done.

Logging the Chrome development console

2021-01-15T00:00:00+01:00

Loading a web page in the browser and working with it may produce various errors. To log those errors you may use the linked test suite. The suite is created to be used with Chrome via the WebDriver Connection Mode of QF-Test: Chrome_Logging.qft

If you implement a similar logging for other browsers, please let me know at support@qftest.com, I will gladly publish it here.

Public GIT repository for test suites

2021-01-11T00:00:00+01:00

Dear readers and users of the QF-Test Blog,

We have a New Year present for you. We are happy to announce the public GIT repository for test suites:

https://gitlab.com/qfs/demo-suites

Here you will find test suites containing generic procedures for different use-cases which we gather from our users in different occasions. As an example, there are two suites already JIRA.qft and web.qft. Feel free to dive in and bravely explore them!

Most importantly :

YOU can contribute to the repository, push new test suites, procedures, and comment. All changes will be carefully reviewed, but keep in mind not to commit sensitive information.

Parsing JSON

2020-12-07T00:00:00+01:00

Especially when testing websites, you are now and then faced with the problem of having to parse JSON (=JavaScript Object Notation) strings.

Often, for this, a JSON library is downloaded and placed into the corresponding plugin directory. After a restart it is then possible to e.g. use the Python JSON library in Jython scripts.

In this blog article, however, the JSON library provided by QF-Test should be described. This library is based on the minimal-json-library and has the advantage that:

it is already supplied with QF-Test (so not a plugin first must be installed)
It is available in both Jython and Groovy scripts

The following chapters describe how this library can be used for JSON parsing and JSON generation.

JSON parsing

The general code to parse a JSON string using this library looks like this:

from de.qfs.lib.json import Json
jsonStr = """{"a" : 1, "b" : "zzz", "c" : [1,2,3]}"""
json = Json.parse(jsonStr)

(Jython Code to parse a Jython String)

import de.qfs.lib.json.Json
def jsonStr = """{"a" : 1, "b" : "zzz", "c" : [1,2,3]}"""
def json = Json.parse(jsonStr)

(Groovy Code to parse a Jython String)

Afterwards, you can easily access the different values defined in the JSON string:

qf.println(json["a"])           # Output: 1
qf.println(json["b"])           # Output: zzz
qf.println(json["c"])           # Output: [1,2,3]
qf.println(json["c"][0])        # Output: 1
qf.println(json["c"][1])        # Output: 2
qf.println(json["c"][2])        # Output: 3

(Jython Code in order to access the various values / output them on the terminal)

qf.println(json["a"])           // Output: 1
qf.println(json["b"])           // Output: zzz
qf.println(json["c"])           // Output: [1,2,3]
qf.println(json["c"][0])        // Output: 1
qf.println(json["c"][1])        // Output: 2
qf.println(json["c"][2])        // Output: 3

(Groovy Code in order to access the various values / output them on the terminal)

In Groovy, the following notation is also possible for JSON Maps:

qf.println(json.a) // Output: 1

(Groovy Code in order to access the various values / output them on the terminal)

which makes it possible to “save” further characters in the script. In this context let me also point to Groovy’s ?-Notation which is helpful when a certain value can be present in the JSON string but does not have to be present.

Generating JSON

The general code to generate a JSON string using this library looks like this:

from de.qfs.lib.json import JsonBuilder
obj = { "a" : [1,2,3,4] }
jsonStr = JsonBuilder.toJsonValue(obj)

(Jython code to generate a JSON string from an object)

import de.qfs.lib.json.JsonBuilder
def obj = [ "a" : [1,2,3,4] ]
def jsonStr = JsonBuilder.toJsonValue(obj)

(Groovy code to generate a JSON string from an object)

Covid-19 and home office at QFS

2020-11-12T00:00:00+01:00

Covid-19 and home office for three quarters of a year. How has this influenced my (working) life?

When we were asked at the beginning of March 2020 to work from home every 5 days during the coming week instead of the usual 2 days, nobody had guessed what would happen after that.

The idea was only to avoid a possible risk of infection. Then the first exit restriction of the government came. How the further rules and recommendations of the government and the health ministry were appearing … Well, we all know this already.

Technical no issue. We all had (and have) a working place at home anyway, at which we always (at least in the more than 10 years I am working at QFS) were working at two days a week. This changed to five days, but the technique is the same.

In the beginning it was quite pleasant to simply save some time and fuel as the work way was eliminated. But I quickly have realized what I was missing. The social contact to the colleagues could now no longer take place face-to-face in the hallway. Just simply to go to the office next door and exchange some information … that was no longer possible.

Also, the common table soccer game towards the end of the working days in the office was dropped. Surprisingly, I miss that much less than the sometimes-accidental encounters in the hallway. I would not have thought so if someone had asked me beforehand.

In retrospect, QFS reacted great, both from the management and from the colleagues. Without the need for any agreements, the stand-up meetings were used more for private exchange (via video conferencing) than for their actual purpose. Even though we talk more about our work tasks of the day or week at the standup meetings nowadays, the personal and human aspect has remained ingrained and has become indispensable.

In addition, we also have now, mainly driven by our espresso machine expert, simple meetings / videoconferences where we meet on various topics that have nothing to do with QFS. It is often about making coffee, which is why the colleague puts his notebook in his private kitchen in front of his espresso machine. But not only. We simply want to have a human exchange and compensate for the lack of personal contacts.

In the meantime, we have already arrived in November 2020 and working from home has become part of our daily life. But not a day goes by without talking to at least one colleague. Of course, it is all about work. But we also try to do it mostly via video chat. A phone call would also be sufficient in most cases. But to see the other person and to be seen yourself … That keeps up the friendly relationship to the colleagues and helps you (at least me) not to succumb to a “Corona-tantrum”, because of only sitting at home and seeing your own four walls / your flat from inside only.

This situation helps me a lot to recognize what is important to me and what is less. Also, the experience that I am employed by a company where working together plays a big role is once again confirmed in the current situation.

Thus, I use the situation to learn a lot about myself and in this respect, I see it quite positively. In general, I have become more modest, but also more aware of my wishes.

Nevertheless:
Of course, I am slowly getting fed up with the inevitable phase of reflection and I would love to have social contacts again without a screen in between. Or just visit my father again.

But that is also a positive insight, I think.

How do you feel about it? Have you had similar or maybe completely different experiences?

Just write it in the comments. I am curious…

Node Transformations in QF-Test

2020-11-03T00:00:00+01:00

In QF-Test it is possible to transform a node into another node.

This feature can be useful when

a certain action cannot be recorded, for example “Wait for component”
you want to insert actions similar to a recorded one: copy and convert can then be easier as performing a new recording.

To this end, right click on the node to be converted

then select “Transform node into”

and choose the desired target node:

The comment node in QF-Test

2020-10-28T00:00:00+01:00

This node can be inserted everywhere and has (normally) no influence on test execution but can be used exclusively for documentation:

Highlighting

In order to ensure better visibility and legibility of comments nodes, it is possible to highlight a comment (or parts of a comment) by using emphasis (i.e. bold, italic, …) or by using a specific text color:

The corresponding HTML tags can be used to achieve such a highlight. Doing this, a starting tag, indicates that from here on a certain font and / or color should get used. This font and / or color is then retained until a corresponding, so-called end tag appears.

Example:

Normal text bold text normal text again

Hereby indicates that from here on the text should be marked in bold. The following cancel the bold emphasis, so that only the middle part of the text is displayed in bold.

Support for the following tags has been implemented:

to indicate that the text should be shown in italics from here on, as end-tag
to indicate that the text should get underlined, as end-tag
to indicate that the text should be shown in bold from here on, as end-tag
~~that the text should be crossed out,~~ as end-tag

For colored markings it is possible to add a color=”colorname”-attribute to these tags. That means for a bold, blue text the whole thing has to be changed as follows:

bold, blue text

Whereby you shouldn’t make the mistake of repeating the color attribute with the end tag (this means to write ), even though will also cancel that later text will be displayed in blue.

In order to get get a colored text without emphasis, the -tag can be used. So:

green text

Notes

Comment nodes are written to the run log and are visible for TestRunListener, too.
Even if it wasn’t explicitly forbidden – It only makes limited sense to insert a comment node starting with ” → “. With the help of ” → ” QF-Test is indicating the step that just gets executed. Thus a comment step starting with ” → ” may confuse users as they may think that this is the step that currently gets executed. Since QF-Test 5.0.0 a new node was added, the comment node:

Image checks in a world of Font Antialiasing and ClearType

2020-08-03T00:00:00+02:00

Image comparison across browsers

Everyone knows the phenomenon that websites are displayed differently depending on the browser. These differences are particularly noticeable between Internet Explorer and Microsoft Edge. You can also see this quite clearly on our homepage, as the following pictures illustrate:

The following text was displayed with Microsoft Edge:

This text was shown with Microsoft Internet Explorer:

Very clearly different – right?

However, this has primarily nothing to do with Antialiasing or ClearType but is in the nature of the rendering engines of the respective browsers. Antialiasing and ClearType of course still play a role, but this would be masked due to the completely different display.

It is immediately apparent that image comparisons are unsuitable here, as the display difference is significant and is visible to the naked eye.

Image comparisons are obviously only suitable to a limited extent here, perhaps with an If/Else construct. However, a Check Text is clearly preferable.

Are image comparisons always useless then?

It is left to the operating system (or additionally to the browser in the case of web tests) to render and display the components (such as buttons, checkboxes, etc.) and text. Image comparisons of “whole” components or components containing text are therefore always difficult and can lead to check errors when tests are executed in different environments.

Generally, every operating system displays components differently. Linux displays buttons, check boxes, and of course also fonts differently than Windows. But even Windows 7 renders differently than Windows 10.

Similarly, versions of Java as well as the graphics card driver used also play a role (especially in the case of font rendering): The browser and its respective version is also crucial for the display of web pages, as shown earlier.

In summary, this means that such image comparisons may fail with any Windows update/upgrade, Java update, graphics card driver update, browser update, or changed user settings.

And not only hardware and software are crucial, but also what respective users will set for themselves personally….

But you have already seen all of this in the introduction above. Topic exhausted? No, we have not even started yet!

Wouldn’t it be “bad” enough that operating systems, graphic card drivers and Java versions play a role in the display? No, we are just getting to the tricky part: Antialiasing and ClearType!

Excursion into theory: Font Antialiasing and ClearType

The topics of Font Antialiasing and ClearType are much more subtle – different settings are often not visible to the naked eye, even though the images (especially around the edges) have virtually no identical color. Most users only notice in a direct comparison whether a text has been modified at all.

These features don’t only affect web applications, but every application.

If you are interested in learning more about this, I recommend the following Wikipedia entries as an introduction:

https://en.wikipedia.org/wiki/Anti-aliasing

https://en.wikipedia.org/wiki/Subpixel_rendering

Back to the practical application

Find all the differences in the following images:

They are identical… or are they?

At a first glance, sure, maybe even at the second glance… but have you looked at the details more closely? Tip: if you touch the monitor with your nose (not on a touchscreen), you’ll notice subtle differences, like with the first “A” for example.

Here it is enhanced:

Now you should be able to see the individual pixels (every square corresponds to one logical monitor pixel) and how the edges of the letters are displayed differently. Do you remember the excursion into theory with the Wikipedia entries? If you have skipped over the theory, now is a good time to catch up.

By the way: the only difference that leads to these two results is a minimally changed ClearType setting. Everything else is absolutely identical!

Human eyes first identify the images as identical – QF-Test however immediately recognizes these images as different and thus first protocols an error at this failed image comparison.

In the run log you can also look at an XOR of the two images and will very quickly see that the images actually differ significantly from each other (XOR highlights color differences depending on the colors; black means no difference).

Antialiasing and ClearType in the user’s hand

With Font Antialiasing and Cleartype, the following settings are of significance, the first four of these depending on user sessions and their respective settings:

“ClearType”: is adjusted via Windows control panel (cross-application for all applications)
graphics card driver settings: setting optings depending on the graphics card manufacturer and the driver used (cross-application for all applications)
Java settings in the Java configuration (found via Windows control panel) (Java applications only)
Browser settings (browser applications only)
Java settings via system property or directly as part of the source code of the application (Java applications only)

QF-Test solution

Usually we never recommend verifying text in functional tests by image, we instead recommend an explicit check text node. Also, “whole” components should not be verified by image but by content checks, such as Check Text, Check boolean and so on, see also https://www.qftest.com/doc/manual/en/checks.html#sec_checks and especially: https://www.qftest.com/doc/manual/en/checks.html#step_CheckBooleanStep

In case of tests that only differ in relation to Antialiasing and ClearType (i.e. not like the introductory example!) you can also choose to use an image comparison algorithm for the respective check image nodes, for example:

identity;expected=0.91

This means that more than 91% of all pixels need to be absolutely identical, but 9% are allowed to be different – you can intensify or soften this as you wish.

Alternatively, even better for the specific example:

similarity;expected=0.98

Here, a similarity between expected and got color value is calculated for every pixel, and the average of all color values must be 98% identical, which the QF-Test procotol also tells us in the result of the checks:

The images are within tolerance: Expected probability: 98.0 % Got probability: 98.56 %

You can find more about the algorithms here:

https://www.qftest.com/doc/manual/en/tech_imagealgorithmdetails.html#sec_imagealgorithmdetails

Important to know:

If you have calculated a suitable algorithm, make sure you also check that this algorithm can differentiate clearly between “good enough” and “not good enough”, so that you don’t get any false positive results!

Tip:

You need to input the algorithm manually in the field ‘algorithm for image comparison’. But you can also implement your own checkers and maybe even have different check images in the check input menu, which all differ in the algorithms used – but this is a topic for a different blog post.

Addressing multiple browser windows

2020-06-09T00:00:00+02:00

Take the demo from here: Chrome_popup_window_name_blog.qft

With multiple opened browser windows, how can I ensure that the correct is used?

In order to distinguish between browser windows QF-Test requires additional information.

This information specifies which window should be used for replaying events. The attribute ‘Name of the browser window’ has to be set in both ‘Wait for document to load’ and the recorded ‘Web page’. In the ‘Web page’ Component set it to {default:windowname:}.

Important

Do not forget to set the variable windowname before replying the required test actions.

Manual FAQ

WebSaga | Wake up!

2020-05-28T00:00:00+02:00

Wake up! The daily tickle units are beginning! Let’s see if you can still dance.

This is how it starts every night. I barely have a moment of rest until I’m clicked and checked again. But it’s fun, I’m the website www.qftest.com!

First they used the quickstart assistent to prepare the basis for a dependency. It tests if I’m already awake and ready to dance the next steps – or do I still need to be woken up?

Then I need to quickly jump through a hurdle – the cookie banner. Every time I am woken up new it stands before me and asks in capital letters if I’ll accept it. This only happens in the automation with QF-Test because the parameter “deleteCookies” is set on “true”.

This hurdle is like a warm-up exercise. As soon as I’m awake and the next dance begins, I don’t need to do this exercise anymore.

If I make a big mistake and can’t go on at all (we call it “Exception”) I’m allowed to take a short rest and continue from the next dance. “Go sleep!”, they tell me, even though I’ll be awoken again immediately. This is done by the dependency. Cleaning up and preparing until all the dances are completed. Only then can I truly rest.

There’s a special card being used for the choreography – this way all my trainers will know where to tickle me, so that I jump to the right place. The secret to this is the CustomWebResolver(). I will talk about that more next time.

Until next time – happy clicking and checking!

Analyzing ComponentNotFoundExceptions

2020-05-26T00:00:00+02:00

In the above image, you can see a mouse click targeting a component (named mFile) in the application. Now whenever an action (like a mouse click) or a verification gets replayed QF-Test first needs to find the corresponding component in the application.

In case QF-Test is unable to identify the corresponding component in the application a ComponentNotFoundException gets thrown.

The following blog article delineates the different actions needed in order to analyse the root cause of such a ComponentNotFoundException.

Check that the component was present in the moment the action / verification got replayed

Although the above step may sound naive, users often miss to identify whether the target component is actually present in the application. In the moment QF-Test is replaying the action / the verification of course, not later on!

Whenever an exception occurs QF-Test (by default) logs a screenshot. Press the “Jump to run log” button (see above error dialog) or press Ctrl+L (in case you already closed that dialog) in order to open the run log. In the run log press the “Find next error” icon (see screenshot below) in order to jump to the exception. Now have a look at the screenshots in that run log. Is the target component visible there? If yes, then continue by reading the next chapter “Check why QF-Test was unable to find the component although it is present in my application”.

If the component is not visible, please check

if the application needs more time till it displays the component

Here, it is likely that QF-Test was too impatient. QF-Test tried to find the target component although the application is not yet showing the component. Instead the application is for example showing a “I’m busy”-animation (like a progressbar).

Often users tend to work around this issue by specifing a waiting time in the ‘Delay before’ or the ‘Delay after’ attributes. However this is – most often – not recommended. In order to increase test execution speed, it is preferable to use a ‘Wait for component’, a ‘Wait for document to load’ or a ‘Wait for component absence’ node (with the latter it is possible to wait till the processbar disappears).

In order to insert a ‘Wait for component’: Copy/Paste the action / verification step that throwed the ComponentNotFoundException while replaying. (You now have this step two times). Then right click the FIRST step → Transform node into → Wait for component.

if the action/verification is targeting a component that only appears sometimes

Here, the application is sometimes coming up with something that needs to get handled. This could for example be a dialog with a message like “Could not connect to server” that sporadically occurs after a given action.

Well, if you have an action targeting a component that sometimes appears (like a click on the OK-Button of such a dialog) and that sometimes does not appear (and that doesn’t matter for the rest of the test) – you can use a try/catch construct (This construct tells QF-Test to ignore any exception for that particular step). The fastest way to get such a construct, is to right click the failing action node → Pack nodes → Try.

Alternatively, an ‘If’ node can be used in order to verify whether the target component is present before the action/verification node is getting executed.

if the action/verification is targeting a component that only appears when certain conditions are met

For example the target component of your click/verification step is only available after the login step.

Although try/catch constructs as well as ‘If’ nodes (see above point “if the action/verification is targeting a component that only appears sometimes”) often help here, too, it should be mentioned, that ‘Dependency’ nodes may be preferable here.

if the component needs to get scrolled visible

Well, here scolling logic needs to get added. (Actually scrolling in QF-Test is an interesting topic for another blog article.)

if this is a bug in your application

That means that the component is not available although it should be available. For example a component indicating an error message that should be there – but somehow that error message is not present.

Well this is something you need to discuss with your dev or bugfixing team ;)

if the target component was removed by development team

This is also something that you need to discuss internally. Maybe the test case should get removed or rewritten?

if the target component is not shown because a previous action failed

This is rarely the case. But sometimes a previous action fails and QF-Test does not recognize that the previous action failed (e.g. a click that then opens a dialog with the desired component). Here the task is to analyze why the previous action failed.

Check why QF-Test was unable to find the component although it is present in my application

In order to avoid any misunderstandings, let me briefly discuss some fundamental ideas of the component recognition algorithm that is used by QF-Test.

The “QF-Test component ID” attribute of a an action/verification is not describing the target component by itself. Instead the string is referring to a node in the “Windows and Components” subtree of QF-Test. E.g. if we take the above mouse click:

Then this mouse click is referring to the mFile-node (right-click the node → Locate component):

It is the information in that referred “component definition node” that is used by the component recognition algorithm. In fact the string “mFile” is just a reference (Feel free to change the QF-Test ID of any component to make your tests more readable).

The reason why the recognition information is kept in one place is the following: It can be that this component in your application changes and then QF-Test may not be able to re-recognize that component anymore (e.g. the text of a button changes from OK to Continue). However you can have an arbitrary number of actions/verification nodes in your test suite that target this component (e.g. 500 clicks on that button). Now instead of going through your test suite and check where a mouse click is targeting that component, you only have to fix one node …

As you may already know, QF-Test is using a probability algorithm for component recognition. And that algorithm determines each component in the component tree one after another. So in the image above, QF-Test will first search for the window, then for the MenuBar and then for the MenuItem named mFile.

Because QF-Test first searches for the window component etc. before it is trying to locate the target component, we need to find out which component QF-Test is not able to find. Is it the window itself (e.g. because the title of the window changed from “MyApplication – version 1.0” to “MyApplication – version 2.0”)? Or is it any intermediate component?

If you have the possibility, simply re-record a click or check on the target component (And don’t forget to delete the recording after analysis – otherwise your test suite gets littered with recording / component definition nodes). Then by comparing the recorded versus the existing “Window and components” subtree, you can see where the tree differs – and thus which component definition node QF-Test was unable to find in your application.

For example let’s re-record the click on the File-menu:

Here we can see that the window component is still the same (marked black) so QF-Test is still able to find / re-recongize the window component. Next is the MenuBar (marked red). And here we have two different component nodes … This means, that QF-Test is unable to re-recognize the MenuBar component.

Another way is to have a look at the run log. Whenever QF-Test is not able to find a component it is logging a message indicating the components present in the application and why QF-Test excluded which component:

After we have identified the component definition node that causes the problem check which node attribute changed. Then have a look at the following table:

if the class attribute changed

The class attribute is indicating the component type QF-Test should look for. E.g. whether QF-Test should search for a Button, a List, a TextArea …

It rarely happens, that the component class changes (e.g. that a button get’s replaced by a label with the same functionality). If you nevertheless have that case, just change the class attribute accordingly.

if the name attribute changed

The name attribute is the most dedicated attribute for component recognition. It should be a unique identifier of the given component in your application.

Developers can add such an identifier to each component in your application. However because these identifier are not visible to the user of your application, this is not often done. Furthermore these identifier are sometimes misused for other purposes – e.g. to store additional component related data.

If only one component is affected by a name attribute change, you can try whether clearing this attribute helps. Otherwise a NameResolvers may help in order to change whether and what QF-Test records as component name.

if the feature attribute changed

The feature attribute gives the main feature QF-Test should use in order to recognize the component. This may for example be the the text of a component (e.g. Ok in case of an Ok-Button).

If you have any variable parts in the feature (e.g. the title of your window may either be “MyApplication version 1.0” or “MyApplication version 2.0” and so on), you can click the “As regex” checkbox and specify a regular expression, that then should match the feature.

Via an FeatureResolver it is possible to change what QF-Test takes/records as feature.

if any extra feature attribute changed

Additional features which QF-Test may use in order to recognizes your component (similar to the feature attribute).

You can use an ExtraFeatureResolver to change (add/remove) extra features. Similar to a NameResolver / FeatureResolver an ExtraFeatureResolver can also affect which extra features QF-Test records (records at all / records by default).

if any structure or geometry attribute changed

The structure / geometry is not often taken into account by QF-Test. In fact, this can only be an issue if QF-Test does not have a good name / feature or extra feature. Then QF-Test may recognize your component based on structure and/or geometry – e.g. if you have multiple OK-Buttons in your application and all buttons have the same name / feature / extra feature (or even no name / feature / extra feature).

So as long as you do have good name / feature or extra feature attributes … ignore it (or correct / remove these geometry / structure attributes).

However when you do not have any good name / feature or extra feature attributes. Take the time to write some of the above mentioned resolver scripts for your component. This is often better then to rely on structure and geometry. (There are some problems with geometry based component recognition. E.g. replaying your tests on different machines often fails, as smaller / larger screens tend to change the geometry where a component gets rendered …)

Video on this subject

Find the keys – UI test automation needs robust component recognition

Component recognition in QF-Test is quite flexible. In addition to standard controls like buttons or checkboxes it also supports complex components (dynamic trees, tables etc.). This webinar shows the basics of this algorithm and configuration options for Java, Web and Windows applications. We will explain how to achieve robust component …

Remarks

Be sure to run your tests in an unlocked, active desktop. Otherwise you may get sporadic ComponentNotFoundExceptions everywhere (and if you rerun your test, other action/verification steps will/may be affected).

WebSaga | Click click – it tickles

2020-05-14T00:00:00+02:00

Click click – it tickles! I’m jumping back and forth, it’s almost like a dance. Always the same steps, I show a different form with every jump.
My educators (I call them colleagues) have been training me for a long time, until I learned to do it perfectly. Quality first, as they always say. So that I always jump to the right spot, always show my best form.
My predecessor was pretty good too, but they’re old now and deserved their retirement. I learned a lot from them, too.

Now it’s time! Now I can introduce myself to the world. Here I am! Can you see me? Hi! I’m the new website, www.qftest.com!

My colleagues say, it’s not enough to learn the steps, I can’t forget them either, or make wrong steps. They call it “bugs”. (Why not sidesteps?)

Everyday they want to click, click, tickle me! And check, check, verify me!
They talk about test automation, regression tests, and that I will also need a resolver! Because of my components. I also heard them talk about ProcBuilder!
Are you excited, too? Soon I will report here how my daily tickle and check units are built, developed and protocoled, or as the colleagues say: how I am tested automatically.

See you soon and happy clicking and checking!

Fantastic Electron apps – and how to test them

2020-05-07T00:00:00+02:00

Last update: 2022-01-14

Even after Electron’s own test framework “Spectron” is no longer being developed, it is still possible to professionally test your Electron apps with QF-Test – since version 5.4.0 even better than ever before thanks to CDP connection mode!

Many developers – 72% according to a survey from stackoverflow (german version) – are specialized in web development. Web technologies are widely spread and supported by basically all operating systems. They provide a huge range of frameworks to the developers. This doesn’t make desktop applications obsolete. Why not use existing resources from the web for local applications? As the headline states there is already a solution for this: Electron.

Testing Electron apps with QF-Test

Video version

Electron is a framework for executing cross-platform desktop applications using the web browser Chromium and the Node.js framework.

Further information

Almost every user knows or uses Electron apps. Popular apps like Skype, Microsoft Visual Studio Code or Discord are built using the Electron framework. Electron origins from GitHubs well known editor Atom which gave it its initial name “Atom Shell”. The idea of GitHub was to combine the browserless “Node.js” framework with the Chromium rendering engine. This provides an easy and fast way to create desktop apps out of existing web apps. Except for the initial download there is no need for an network connection.

Apps can now be developed using web technologies like HTML, CSS and JavaScript. This enables apps to be developed OS independent which makes them easier to maintain. Even on the code base it is quite easy to handle native functions like menus, notifications, access to the file system and also some low-level informations about the hardware.

The best summary for the framework comes from the official site: “It takes care of the hard parts so you can focus on the core of your application.”

But I don’t write this to sing the praises of Electron – of course I want to mention that QF-Test supports testing of Electron apps since version 4.5 and I want to give a quick introduction.

How to test Electron Applications with QF-Test

The basic concept is still the same. The best way to connect to an Electron app is by using our Quickstart Wizard (Menu: Extras → Quickstart Wizard). Then select “An Electron application” and on the next page you select the app you want to test plus option command line parameters.

The next step is to specify the ChromeDriver. You can either choose the correct Electron version or let QF-Test detect it so you know what ChromeDriver version you need. QF-Test can then automatically download the ChromeDriver.

Electron provides all features already known from the web. Additionally native menus can be controlled by a Selection step (Insert->Event nodes->Selection). The syntax looks like this:

To be able to control this functionality QF-Test has to get access to the Electron API. This means that the following values (currently default) inside the BrowserWindow should not be changed.

mainWindow = new BrowserWindow({
    webPreferences: {
        enableRemoteModule: true,
        contextIsolation: false,
        nodeIntegration: true
    }
})

Please try it yourself and test your own app! If you only have an older version of QF-Test (<4.5) you can get the latest version for free from this site and evaluate it: Free trial

The proof of the pudding is in the eating – Canned Procedures

2020-04-08T00:00:00+02:00

Packet soup or baking mixture, cook by yourself or bake? Concerning food opinions divide. Pros and cons.

QF-Test doesn’t ship convenience food, you have to cook it yourself. But you will be given a variety of spices that will ease your life: the standard library qfs.qft. From a collection of fine salt and pepper to complex pizza spices, the genuine for every meal.

What’s inside the standard library?

procedures (spices)
generic components (vitamins)

Fish, meat or vegetarian? Some spices go together with any meal, other just with specific dishes. The standard library contains drawers – the packages – where you can find the procedures waiting for their service.

For example: What should I take to cook pasta? Let’s take a look inside the “pasta” drawer. With mushrooms? With cheese? Pesto?

In qfs.qft you have those packages for engines such as “swing”, “web” or “win” as well as universal procedures in “utils” drawer.

Do I only take salt and pepper (mouse click)? You have the refined version called general.doClick() for every engine. Or italian spices for my salad? Then I take swing.combobox.getItemCount(). Or something similar for pizza? Then fx.list.getItemCount(), etc.

What are the advantages? You are finished. Mastered by famed chefs, tested by gourmets. It saves my time and guarantees constant quality.

And if it’s not my flavor? Then I can still write my own procedures.

So why not give it a shot? Unlike eating a pudding, procedures won’t disappear if you try them… :)

Enjoy your meal!

QF-Test 5.0 and macOS Catalina (10.15)

2020-02-10T00:00:00+01:00

In their recent version of macOS, named “Catalina” (10.15), Apple introduced a feature protecting users from executing malicious software downloaded from the Internet. To separate “good” from “bad” software, Apple implemented a so called “notarization” process, and software which has not been “notarized” cannot be executed on macOS Catalina without further ado.

Unfortunately, Apple tightened the criterias for notarization by February 2020 and excludes within others all software from notarization which build upon Java 8 – this also affects QF-Test in the latest version 5.0. Therefor, after downloading and moving QF-Test to the application folder, another step is required to execute QF-Test on those systems:

After starting QF-Test a message is shown that Apple cannot check QF-Test for malicious software:

Close the dialog by clicking “OK”.

Then open the system preferences and there the section “Security & Privacy”:

In the tab “General” you can now find the remark that the start of QF-Test has been blocked.

Confirm the execution of QF-Test by clicking “Open Anyway”. In the following dialog, you can finally allow the execution of QF-Test:

Now you can use QF-Test like usual, the operating system remembers the exception.

How to debug Jython Scripts in QF-Test

2020-01-30T00:00:00+01:00

Motivation

By the usage of script nodes QF-Test allows embedding of special custom functionality into your test cases. Typically, these scripts are supposed to be short and relatively simple containing just dedicated functionality, whereas the majority of test logic is done via QF-Test visual nodes.

Verifying the correctness of the functionality of those scripts can be done mostly by inspection or inserting simple print statements for generating debug output into the QF-Test terminal (see also blog about print). Jython also comes with an integrated post mortem debugger to analyze the cause of exceptions within a script execution (see post mortem manual chapter).

Nevertheless there may be situations of more complex scripts that demand for enhanced ways of debugging, e.g. by setting break points, executing step by step and analyzing variable values. To allow such debugging features you can make use of the PyDev, an IDE for Python development, available standalone or as an Eclipse Plugin.

In this article I explain how to prepare an Eclipse or standalone Liclipse IDE to connect to the Jython interpreter of QF-Test and allow enhanced debugging.

IDE configuration

If you already use Eclipse IDE the following step 1 shows how to install PyDev as a plugin.

Another option is to directly download the PyDev IDE called Liclipse that already contains the PyDev Plugin (https://www.liclipse.com/). You can skip the first step then.

Step 1: Eclipse Pydev Plugin installation (not required for Liclipse)

Open menu Help → Install New Software
Provide Pydev URL https://www.pydev.org/updates/
Select PyDev Package (for further Information see also https://www.pydev.org/download.html)

Step 2: Configure Jython Interpreter in Eclipse / Liclipse

Open Menu Window → Preferences
Choose PyDev → Interpreters → Jython Interpreter
Browse for Jython jar
Navigate to lib\jython.jar inside your QF-Test version specific installation directory

An error message occurs, but please proceed anyways

Add the QF-Test jython\Lib directory
New folder
Select the jython\Lib directory from inside your QF-Test version specific directory to the Python path

By this the basic Eclipse / Liclipse configuration is completed.

QF-Test configuration

Step1: Add PyDev to the QF-Test python path

In order to add the necessary pydev modules to the python path, you can start QF-Test with the additional command line parameter

-J-Dpython.path=<path to relevant pydev modules>

The relevant pydev modules are located in the pysrc directory inside the pydev.core plugin. The pydev.core plugin resides in the eclipse/liclipse plugins folder, starting with org.python.pydev.core.

e.g. for Liclipse is may look like:

"C:\Program Files\Brainwy\LiClipse 5.0.3\plugins\org.python.pydev.core_6.5.0.201809011413\pysrc"

one possible location in Eclipse is:

"C:\Program Files\Eclipse\eclipse-4.11-64\plugins\org.python.pydev.core_7.2..201904261721\pysrc"

The QF-Test command line or shortcut for Liclipse, should look similar to this:

"C:\Program Files\QFS\QF-Test\qftest-4.6.0-dev\bin\qftest.exe" -reuse -J-Dpython.path="C:\Program Files\Brainwy\LiClipse 5.0.3\plugins\org.python.pydev.core_6.5.0.201809011413\pysrc"

And for Eclipse respectively:

"C:\Program Files\QFS\QF-Test\qftest-4.6.1-dev\bin\qftest.exe" -reuse -J-Dpython.path= "T:\common\eclipse\eclipse-4.11-64\plugins\org.python.pydev.core_7.2.1.201904261721\pysrc"

Step 2: Configure QF-Test to always save Jython temp scripts

QF-Test writes the Jython script nodes into temporary files (called e.g. _script3452345823457.py, located in the QF-Test user directory), but typically only once or when an error occurs. Because those temporary files are now used by the PyDev Debugger, it is very recommended to configure QF-Test to update these files whenever the QF-Test Jython script node is changed. This can be achieved by setting a respective QF-Test option by use of a server script node with the content

rc.setOption(Options.OPT_PLAY_SCRIPT_JYTHON_SAVE, Options.VAL_PLAY_SCRIPT_JYTHON_SAVE_IF_CHANGED)

which needs to be executed once before starting with debugging.

Step 3: Set Eclipse/Liclipse as Command for external editor (optional)

When having Eclipse/Liclipse around, it may prove useful to be configured as default external editor for QF-Test Jython scripts. This can be achieved by following steps in QF-Test:

Open the QF-Test Options via Edit → Options
Select option: Generel → External Applications
Set the external editor command to point to the Eclipse/Liclipse executable with the parameter –launcher.openFile
Press OK

Now script nodes are directly opened in Eclipse/Liclipse when the Button “edit in external editor” is pressed.

Step 4: Prepare script for remote debugging

In order to prepare your Jython script to be debugged, you need to add some line of codes

import pydevd
pydevd.stoptrace()
pydevd.settrace('127.0.0.1', port=5678, suspend=True, stdoutToServer=True, stderrToServer=True, overwrite_prev_trace=True)

→ here execution will be interrupted and control switches to pydev debugger

Debug Jython scripts

Step 1: Start PyDev Debug Server

In order to allow PyDev to interrupt and control the Jython script execution, the pydev debug server needs to be started in Eclipse/Liclipse

Move to Debug perspective
Menu: Pydev → Start Debug Server

Step 2: Debug Jython Server Scripts

Start the test execution of a Jython Server Script in QF-Test

Once the script containing the pydevd.settrace statement gets executed, the run gets interrupted
Further control is possible in Eclipse/Liclipse

The debug server comes into action
Execution stack is shown in the left part
The script node is opened (it is a temporary generated file _script…..) – 1) execution pauses right after the pydevd.settrace call – 2) debugger actions can be used to control further execution.
Variables can be inspected in the right window part

NOTE: PyDev debugger warnings may appear in the terminal, looking like

pydev debugger: Unable to find real location for: C:\Users\YourName\AppData\Roaming\qfs\QF-Test\edit_ms.py.

but they are not supposed to do any harm, just warnings.

Step 3: Debug Jython SUT Scripts

Especially SUT Scripts are meant to be small and simple in order to influence the SUT as little as possible. All complex or time-consuming task shall be done within server scripts. If you pause an SUT Script in the Debugger this effectively puts your whole SUT-Appliction on hold, which may change its behavior or even crash.

If there is really the need to debug SUT scripts as well with remote debugging, the pysrc directory need to be added to python.path of the SUT than as well.
You will need to adapt the start command of your SUT respectively.

As an example you can adapt the start command of the Swing CarConfig in the CarConfig.cmd file in the demo/carconfig directory in your Qf-Test distribution and add a respective python path parameter:

%JAVA_BIN% -cp "../../qflib/qfdemo.jar;../../qflib/qflib.jar;../../lib/looks.jar;../../lib/jgraphx.jar" -Dpython.path=" T:\common\eclipse\eclipse-4.11-64\plugins\org.python.pydev.core_7.2.1.201904261721\pysrc " "de.qfs.apps.qftest.demo.multi.CarConfigurator" %*

After having started the CarConfig demo, you should be able to Debug Jython SUT scripts for this client.

Conclusion

The usage of Eclipse/Liclipse IDE for editing and debugging Jython code in QF-Test can be a very powerful and helpful enhancement for handling more complex scripts.

Groovy or JavaScript code cannot be debugged via PyDev or similar, therefore you are unfortunately still limited to verification by inspection and print statements.

The license server was shut down and still uses only the old license

2019-11-05T00:00:00+01:00

Introduction

OK, this is a really special case, but with the increasing use of the QF-Test license server this case is happening more often.

Before I start, I have to mention a few things.

What does a license server make special?

QF-Test is able to manage itself normally without any license server. QF-Test is communicating with the other active QF-Test instances when it is started to check, if there is at least one license left for the new starting QF-Test instance.

This only works within a subnet.

Now the license server comes into play. It just has to be reachable by all QF-Test instances, which use a compatible client license for the server license.
When starting, they ask explicitly just this license server if the requested license is still available.

In this way you can use the requested licenses beyond the borders of the subnet and the location effectively and dynamically. Moreover, the license upgrade has only to be installed on the server. The QF-Test installations with the client license stay as they are. And this also reduces the effort of course.

How do I update the license of the license server with a license upgrade?

For this there are two possibilities:

Simply start QF-Test at a machine with the client license and perform the license upgrade for the server by the use of the user interface and “Extras” → “License server administration…” like shown in the screenshot below.

Alternatively there is also the possibility to shut down the license server, to change the license file and start it up again

When does the title’s case occur now?

If you choose the alternative way (method 2) and you run the license server on Windows you can fall into a trap.

If you want to shut down the license server with the command "qftest -batch -licenseserver.shutdown", this is formally correct. But if there is an error message you do not receive it in this case. In contrast to Linux, you have to use "qftestc.exe" instead of "qftest.exe" if Windows is used, when messages should be shown within the console. Due to Microsoft Windows’s implementation this is technically not solvable in a different way.

So instead of "qftest -batch -licenseserver.shutdown" just run it with the command "qftestc".

If this is done, you receive the message that the password was not accepted.

Yes, this is usually the problem… The server is saved with a password, which also has to be inserted additionally when opening it.

So just use the additional parameter "-licenseserver -password" with the appropriate password and then the license server is shut down.

Example

enter

qftestc -batch -licenseserver.shutdown -licenseserver.password secret

in the command prompt respectively

& qftestc -batch "-licenseserver.shutdown" "-licenseserver.password" secret

in the PowerShell console.

For the sake of completeness

When it is started again there should be no message that the port of the license server is occupied. If the message comes up, the shut down did not work. In this very rare case, you should use the task-manager of Windows to end the associated QF-Test as well as its Java process.

Really a “Great Place to Work”? – An internship at QFS

2019-10-28T00:00:00+01:00

You are interested in QF-Test, you already have a trial version and/or want to know more about this company? Most of the time you can only catch a superficial glimpse of the companies on their websites. How the employees are and what it’s like to work in this company is not really known. You may find an article or an award. Fantastic. But what is behind this appearance? I spent three weeks at QFS as an intern and would like to give you a little insight into the company behind the product.

A friendly salutation with “Hello” should not be missing. It is sad that today lots of people forget to do this, but at QFS it is still present that everyone greets everyone. Incidentally, this promotes the good working atmosphere and makes most people happy immediately.

I was aware that I would like to collect more experience in the field of business computer science before I start the studies. By prior circumstances I applied short-term at QFS and at the beginning of September I was happy to be invited to an application conversation. The conversation with Dr. Martina Schmid, Karlheinz Kellerer and Michael Höber was positive. They have spent lots of time to get to know me and discussed the further procedure with me. Because of that, the next day was on an unexpected way my first day at work at QFS and each spoke to the other by their first name.

At the first day of work I was really excited that I got the task to edit the subtitles of a YouTube video. Of course my work was checked and corrected and I have got constructive feedback from which I could learn well. Most of all surprised me, that I was allowed to attend a meeting. This is a very exciting experience for an intern. After the meeting they immediately asked if I had any questions to the technical terms or at all.

Next day was home-office-day. It was awesome that they gave me a task that filled the day and was controlled on the next day. Because there was a bit wrong, I was glad that I had the opportunity to correct it myself and to learn from it actively.

I was happy to see that everyone was included in the meal and kicker right from the beginning.

After the first few days of getting to know each other, the atmosphere thawed.

In general, it was noticeable that the working atmosphere is relaxed and laid back, yet constructive and everyone enjoys working there.

Market research are very interesting especially for intern to know which differences on websites of diverse companies are existing on the internet. This is not only of interest if you are purely working with websites. I guess lots of you know that – even when you are working in a different part of the company, where you do not work with websites that much – you want to know how your company presents itself on the internet and also how the presentation is in relation to other companies.

The first week was over too fast and I noticed with pleasure, while sitting on my workplace on Monday, that in between there were private conversations about the last weekend. After this short break, the work was even easier and faster done.

During the week a translation had to be done and after mentioning that I also enjoy working with languages, I soon had a second one to do. Yes! ;-)

I have never really been without work and was never bored. The various tasks were given to me at a comfortable interval. Every time when I had finished something, I have been already asked if I have something to do, I was given a new work to do or I just asked somebody.

From time to time I had chances to see what the different areas of responsibilities of QFS are doing and was not tied to one department.

There was also the opportunity of working with their test tool QF-Test.

Do know what the product of this company exactly does, but also to know what work is behind the product, is a great experience. I am sure that I can definitely benefit from it in the future.

My wish of having QFS as my training partner during the dual study course, could not be realized for reasons of capacity, but I am so grateful that I had the opportunity to be a part of this friendly, family and very human company for this three weeks.

Are there no negative experiences? Of course there are discussions between colleagues sometimes, but this is just human. There is no way of being always on one opinion. So the answer is: No. I have no negative experience with QFS. And everyone is always at your side in case of problems, be it professional or private, and support you.

Yes, you have read right. I was warmly received and QFS absolutely is a “Great Place to Work” without any doubt.

I wish you a wonderful day and hope that you are also able to work in such a wonderful setting.

Read and write Excel files from QF-Test

2019-10-08T00:00:00+02:00

If you want to use data from an MS Excel file in your QF-Test tests or want to write test results to MS Excel files then go on reading.

Attached to the article you will find a test suite, QFSExcel.qft, containing examples and an accompanying Excel file, ‘Data.xlsx’. To run the examples, copy both files into the same directory and open and run the test suite with QF-Test.

QFSExcel.qft

Data.xslx

Reading Excel data

1. The ‘Data driver’ node

The easiest way to read Excel data is via the ‘Data driver’ node. The Excel file contains a number of data sets (one per row or also per column). The data driver reads the data sets and generates as many loops as there are data sets. In each loop all test cases of the test-set will be executed.

The effect of this structure is that the test case(s) will be run once for each data set.

The ‘Data driver’ node can be used within ‘Test set’ and ‘Test-step’ nodes.

2. The ‘Excel file’ node alone

If you want to use MS Excel data in a different way you can use the ‘Excel file’ node on its own to read the data into a group variable.

In the above example the Excel data are read into a variable group called ‘data’.

You can list all variables of the group using the procedure ‘listAllMembersOfPropertyGroup’ from the package qfs.utils.variables of the QF-Test standard procedure library qfs.qft.

The syntax for accessing a single variable in a group is ${groupname:variable}.
The syntax of the variable for referencing an Excel cell is ‘columnname.index’, respectively ‘rowname.index’. This said, the complete variable syntax for a cell, when the data sets are organized in rows, would be ${groupname:columnname.index}.
The names have to be in the first row, resp. column, of the data sheet. The index starts with the following row resp. column. All indexes are zero-based.

So, the variable referencing cell B4 would be ${data:Test-case.2}, where ‘data’ is the group name.

The ‘Excel file’ node also generates a variable for the number of data sets read, ${group:size}, and the number of rows, resp. columns, read, ${group:totalsize}. The two may differ if the file contains empty rows or columns between the data sets.

Writing Excel data

1. Using the sample procedures

The third example in the attached test suite copies an existing Excel file to a new one and adds a result value to each data set.

The procedure ‘createWorkBookFromFile’ reads an existing MS Excel file into a Jython variable using the Excel API. The next procedure assigns an Excel sheet to another global Jython variable. One procedure shows how to use the Excel API in order to read a cell value. Another procedure writes a value to a cell. The last procedure writes the work book to a new Excel file. All procedures make use of global Jython variables in order to pass the complex data structures for the work book and the data sheet between the procedures.

2. In-depth details and scripting possibilities

I used the Excel API of Apache. There are far more methods to the Excel API than used here. So feel free and use the sample scripts as a basis and adapt them to your needs. I found https://poi.apache.org/components/spreadsheet/quick-guide quite helpful. You can use the API methods in QF-Test Server Scripts.

Explanation to the script in the first procedure:

Lines 1 and 2 import the required modules for the Excel API (org.apache.poi.ss.usermodel) and the Java file handling (java.io).

Line 4 defines ‘wrkbook’ as a global Jython variable. Thus the ‘wrkbook’ variable can also be used by the other Jython scripts.

Lines 6 reads the Excel file parameter passed to the QF-Test procedure using the method ‘lookup’ of the QF-Test module ‘rc’.

Lines 7 to 10 read the Excel file. If it does not exist an exception is thrown.

Line 12 assigns the work book to the variable using a method of the Excel API.

Line 14 uses a Java method for closing the file.

Execute test cases and sets with a certain tag

2019-10-01T00:00:00+02:00

Large test projects may consist out of several thousands of test cases. Often executing them all would need a considerable amount of time, even if multiple machines are used in order to replay the different tests.

In order to develop such large test projects, it is often useful to store the different test sets/cases in different files. For example we may create one file that contains all tests for feature A and another file that contains all tests for feature B. Nevertheless it may still be a good idea to annotate the different tests with tags.

Tags like “feature1”, “feature2”, “fastRunningTest”, “slowTest”, “importantTest”, “oftenFailingTest” and so on make it possible to run a dedicated subset of all test cases. For example this makes it possible to run only those test cases tagged with “oftenFailingTest” and “importantTest”.

The “Condition” field of a test case/test-set node can be used to assign tags to any test case/test-set. For example if you want to assign the three tags tag1, tag2 and tag3 to a test case / test-set you can use the following syntax:

"""${default:tagsToExecute:}""" == "" or len(set(["tag1", "tag2", "tag3"]).intersection([x.strip() for x in rc.lookup("tagsToExecute", expand=False).split(",")])) >= 1

This condition will advice QF-Test to execute the test case/set when one of the following conditions are met:

The tagsToExecute variable is not set or empty.
The test case/test-set is annotated with a tag that is also listed in the tagsToExecute variable (comma-separated list of tags).

This means in order to execute only those tests annotated with tag1, the variable tagsToExecute needs to be set to tag1. In order to execute those tags either annotated with tag1 or tag2 the variable tagsToExecute needs to be set to tag1,tag2 and in order to execute all tests annotated with tag1, tag2 or tag3 the variable tagsToExecute needs to be set to tag1,tag2,tag3.

Remarks

When calling QF-Test from command line, it is possible to set a tagsToExecute variable via the -variable command line argument (e.g. -variable tagsToExecute=tag1,tag2), see
Command line arguments and exit codes
For better readability, it is allowed to add spaces before and after the comma. So in order to execute the tests annotated with tag1 or tag2 the variable tagsToExecute may be set to
tag1,tag2 or
tag1, tag2 or even tag1 , tag2.

The impact of geometry on component recognition

2019-09-19T00:00:00+02:00

…and what to do about it

This is a short article about why QF-Test records geometry information for components, what impact these values have and how to change them either after or during recording .

For the interested, the whole component recognition algorithm is explained in the technical details of the QF-Test manual, section “Component recognition”:

Here’s a short summary:

Based on the component information stored in the test suite QF-Test determines a match probability for each visible component in the SUT. The component with the highest probability wins, provided that probability is a above a configurable threshold. Otherwise a ComponentNotFoundException is thrown. Though the algorithm has been fine-tuned many times, it’s basic concept hasn’t changed since QF-Test’s early days and the default settings have proven useful in most cases.

Geometry is the least relevant part of component recognition, but it may have a significant impact in case the other characteristics are not conclusive.

The probability for a component starts with a 99.9% match. Then geometry is applied in one of three ways:

Each of the values – if specified and ignoring X and Y for top-level windows – is compared to the actual values and a deviation reduces the probability.
Consequently – and this is important – if geometry information in the test suite is removed, all components start out equal, but with a perfect geometry match.
If any of the geometry values is set to ‘-‘ the probability is immediately reduced to zero, so again all components start out equal but with zero probability so that a match of any of the other characteristics is required.

Hands-on example

Let’s work on an example to see first-hand what this means:

Note: In case you have a commercial license that does not include the Swing engine the CarConfig demo will not connect. To work through the example you can start QF-Test in demo mode with the command qftest -license=

Open the Help → “Explore sample test suites” menu item and select the first example, “Swing CarConfig Suite”.
To start the SUT expand the Test-set, select the Dependency reference “dependencies.sutStartedAndReset” and press [Return] or click the Run toolbar button. The QF-Test CarConfig demo should start up and the record button should become enabled.
Open a new, empty test suite and record three mouse clicks on the value fields for “Base price”, “Specials price” and “Discount”.

The recorded components should look like in the following screenshot

Naturally – this is a demo application after all – the recorded component information is totally overqualified. We have a perfect name, explicitly associated label as feature, implicitly determined qfs:label as extra feature plus structure and geometry information that is still fresh and valid.
Select the mouse click on JCarConfigurator.DiscountValue, activate ‘Replay as “hard” event’ and run it. The click should be spot-on.

In a real-world situation, name and feature information may be missing, the label might have change and structure and geometry information deteriorate as the layout of the components in the SUT changes or components get added or removed.

Playing with component information to learn about the effects

Let’s modify our recorded information in several steps to learn about their effect on component recognition:

1. No name

Remove the name “DiscountValue” from the component information, then replay the mouse click.

→ Still good. To see how good, open the options dialog via the Edit->Options menu item, select Run logs->Content in the tree and set the option “Log level for the SUT” to “All messages”. Don’t forget to reset this option at the end so that your run log don’t grow needlessly.

Then replay the mouse click and open the run log via Ctrl+L.

Expand the nodes via Alt+Right Arrow and select “Looking for matching top-level components…”. The information shown there is not user-friendly. Its main goal is to provide information for QFS support when helping to resolve component recognition issues.

Scroll down to the bottom, go back a little and you’ll find something close to the following:

...  
 .2 Calculating probability for


javax.swing.JTextField[DiscountValue,239,0,48x30,layout=javax.swing.plaf.basic.BasicTextUI$UpdateHandler,alignmentX=0.0,alignmentY=0.0,border=,flags=296,maximumSize=,minimumSize=,preferredSize=,caretColor=javax.swing.plaf.ColorUIResource[r=0,g=0,b=0],disabledTextColor=javax.swing.plaf.ColorUIResource[r=170,g=170,b=170],editable=true,margin=javax.swing.plaf.InsetsUIResource[top=1,left=2,bottom=2,right=2],selectedTextColor=javax.swing.plaf.ColorUIResource[r=0,g=0,b=0],selectionColor=javax.swing.plaf.ColorUIResource[r=172,g=210,b=248],columns=4,columnWidth=12,command=,horizontalAlignment=RIGHT]
 
Geometry probability: 99.9%Structure matchStructure probability: 100%, penalty: 100%Extra feature matchExtra feature probability: 100%, penalty: 100%Feature matchFeature probability: 100%, penalty: 100%Combined probability: 100%, acceptableFinal probability: 99%

Calculation starts with the “Geometry probability” as base value, then structure, extra feature, feature and name are applied – if available – as explained in the manual leading to the final probability.

2. No feature

Remove the feature “Label: Discount” from the component information, then replay the mouse click and check the run log as before.

→ Still good, almost unchanged, just no Feature match:

Geometry probability: 99.9%  
 Structure match  
 Structure probability: 100%, penalty: 100%  
 Extra feature match  
 Extra feature probability: 100%, penalty: 100%  
 Combined probability: 100%, acceptable  
 Final probability: 99%

3. Changed feature

Let’s make it more interesting. Undo the last change to restore the label, then edit the value, e.g. by appending “bla”, then run again and check the log.

→ The click is still on target, but the log shows a feature mismatch and the final probability is barely above the minimum probability of 50%:

Geometry probability: 99.9%  
 Structure match  
 Structure probability: 100%, penalty: 100%  
 Extra feature match  
 Extra feature probability: 100%, penalty: 100%  
 Feature mismatch  
 Feature probability: 100%, penalty: 55%  
 Combined probability: 55%, acceptable  
 Final probability: 54.4%

4. Changed feature, extra feature and structure

Let’s give it all we’ve got: Change the class count to 5 to simulate that a previously existing TextField has been removed in the meantime and modify the extra feature qfs:label, e.g. by also appending “bla” to the value “Discount” as shown in the

→ Surprisingly the click is still on target and despite 3 mismatch warnings the resulting probability has barely changed:

Geometry probability: 99.9%  
 Structure mismatch index: 2, count: 4  
 Structure probability: 99.9%, penalty: 60%  
 Extra feature mismatch  
 Extra feature probability: 99.9%, penalty: 55%  
 Feature mismatch  
 Feature probability: 99.9%, penalty: 55%  
 Combined probability: 54.9%, acceptable  
 Final probability: 54.4%

So in this case geometry saves the day, keeps the test running and enables you to recover current, valid component information by right-clicking the component node and selecting “Update component(s)”. Go ahead and try it, but then use “Undo” to restore the deliberately broken state from the last screenshot.

5. Empty geometry – match everything

Now lets finally start messing with the geometry values. As we have a perfect geometry match anyway, things shouldn’t change if we remove the geometry values X, Y, Width and Height entirely (cf. summary at the beginning).

→ Oops! Now the click goes to the Base price value. Looking at the run log we see that all four TextFields now end up with the same final probability of 54.4% which makes sense as all start with the same geometry probability and have the same mismatches. For lack of further options QF-Test simply chooses the first one.

This shows that removing geometry values entirely is inviting disaster in the form of false positive matches. The ability to leave individual geometry fields empty is still useful though. For example you can target the individual TextFields based on geometry only by setting the Y field to 30, 60, 90 or 120 respectively.

6. Geometry “-” – match nothing

False positive matches are difficult to analyze so we may want to avoid a geometry based match altogether. Set one or all of the X, Y, Width and Height attribute to just ‘-‘.

→ OK, we’ve now broken things to the point where we get a ComponentNotFoundException. The details in the run log show that all candidates now have zero probability as expected.

7. Restore structure match

Let’s restore the structure information by changing Class count back to 4.

→ No luck

The run log shows that the structure match would have brought us up to an acceptable 70% match, but the feature and extra feature mismatch bring it down again to 38.5% which is below the threshold.

8. Restore feature

Undo to break structure again and fix the feature instead, setting it back to Label: Discount

→ The click is back on target

From the run log we see that (in the absence of names) the feature has the highest relevance and is sufficient to raise the probability above the threshold in spite of the other mismatches.

9. Roll your own

Go ahead and experiment with restoring structure and/or extra feature information while also removing the higher relevance attributes feature and/or extra feature. You’ll see that correct structure info alone is sufficient when not hindered by some other mismatch.

Generic components

Note: The manual chapter “How to achieve robust component recognition” is generally worth reading. Section Avoiding recording every component or using generic components () explains the concept of generic components which we are going to need for the following.

So let’s apply the knowledge gained to creating a generic component based on just the qfs:label extra feature:

Select TextField JCarConfigurator.DiscountValue , copy, select Window JCarConfigurator and paste. Select “Make QF-Test IDs unique” to prevent ID conflicts, then change the QF-Test ID of the copied node to genericTextField.
Set geometry to ‘-‘, remove feature, structure and all extra features except qfs:label and set the value of the qfs:label extra feature to $(label). The result should look as shown below:

To make use of that component, create a Test-step under Extras with a variable named “label”, value “Discount”. Copy the mouse click node on JCarConfigurator.DiscountValue into the Test-step and change the QF-Test component ID to genericTextField. It should work right away.
And as for why and where this can be useful: Add a data driver node to the Test-step and inside that a Data table, name “label”, column “label”, four rows with values “Base price”, “Specials price”, “Discount” and “Final price” as shown below, then run the whole test step

Summary

Geometry values in recorded components are useful in combination with other characteristics. If you tend to work mostly with recorded components, just keep it as is. In case most of the recorded components have no reliable names or other features you’ll get best result by ensuring fixed window sizes upon SUT startup.
Removing all geometry values is a bad idea because that can lead to false-positive matches when features change. To take geometry out of the picture, set the values to ‘-‘ instead.
Generic components should always use geometry ‘-‘ unless they are explicitly based on coordinates.

If you feel that disabling geometry via ‘-‘ could improve your tests you may wonder how to implement that efficiently. So I’ll finish this article by looking at how to mass-replace geometry values with ‘-‘ and provide an as-yet-undocumented tool to make sure new components get recorded that way.

Mass-changing recorded geometry values

Note: Before performing any kind of mass operations, always make sure that you test suites are properly saved and backed up, ideally in a version control system like git!

Mass-replacement in a single test suite is simple: Select the ‘Windows and components’ node, press Ctrl+H to open the replace dialog (menu Edit->Replace…), make sure to use the advanced dialog, clicking the ‘>>’ icon if necessary and set the values as shown:

Be sure to check “Match whole attribute” and “Use regular expressions”, then click “Replace all”.

For aesthetic reasons you may want to repeat the same for attributes X, Y and Height, but it is not necessary to achieve the wanted effect.

You can do the same for all test suites in a project by opening the replace dialog from the project view and setting the scope of the operation to “Project”. There’s the small problem that X and Y attributes also apply to mouse event nodes, but for those ‘-‘ is not a legal value so no harm should be done.

In case you don’t want to drop geometry entirely but change it only based on some condition, e.g. only for components of class TextField, please see https://www.qftest.com/doc/manual/en/user_gui.html#usec_tunesearch about how to combine searching, setting marks and limiting the replace operation to those marks.

Change the way geometry gets recorded

Finally, here’s the promised resolver that can modify geometry settings while recording. It is based on an internal API that is sometimes used by our support for fine-tuning highly individual needs. It is too complex for general use but serves well for this case. To enable this resolver, add the following Groovy SUT script node after the ‘Wait for client to connect’ node for your SUT:

def getElementInfo(com, info) {
    info.setX(Integer.MIN_VALUE)
    info.setY(Integer.MIN_VALUE)
    info.setWidth(Integer.MIN_VALUE)
    info.setHeight(Integer.MIN_VALUE)
    return info
}

resolvers.addResolver("drop geometry", this.&getElementInfo)

To limit the effects of the resolver to components of one or more specific classes, change the addResolver method accordingly, e.g.

resolvers.addResolver("drop geometry", this.&getElementInfo, "TextField", "MenuItem")

You can also filter by info.getClazz(), info.getName() or info.getFeature() in the resolver method. As indicated above, for highly individual needs please contact our support.

Software Testing Thesis

2019-08-06T00:00:00+02:00

We want to help students in writing their software testing thesis. We were also once students and we know how valuable any help is. That’s why we have decided to create a list of references list for relevant testing topics. We provide here mainly literature from books, because web links are hard to maintain, since they often lead to 404 Errors after some years.

We offer paid software testing internships. This includes of course assistance from our developers and testers. If you are interested, just contact us.

You may find thesis which include QF-Test on our website in the section evaluation reports.

For your thesis we will gladly provide you with a trial license of QF-Test.

Feel free to post further good sources for these topics here to help future students in writing their software testing theses.

Software Testing

Assassa, Gassy; Mathkour, Hassan; Al-Ghafees, Bander: Automated Software Testing in Educational Environment: A Design of Testing Framework for Extreme Programming, Department of Computer Science, College of Computer and Information Sciences, King Saud University, Saudi Arabia (2006)

Belorustes, Vladimir: Efficient Configuration Test Automation using Virtual Machines, Plaxo, Inc.

Berner, Stefan; Weber, Roland; Keller, Rudolf: Observations and Lessons Learned from Automated Testing, Zuelke Engineering, AG, Zurich Switzerland (2005)

Candea, George; Bucur, Stefan; Zamfir, Cristian: Automated Software Testing as a Service, School of Computer and Communication Services, EPFL, Lausanne, Switzerland (2010)

Dustin, Elfriede; Rashka, Jeff; Paul, John: Automated Software Testing, Introduction, Management, and Performance, Addison Wesley (1999)

Fewster, Mark; Graham, Dorothy: Software Test Automation, Addison-Wesley (1999)

Haley, Allen: Development and application of a white box approach to integration testing (1984)

Hauptmann, Benedikt Dr.: Reducing System Testing Effort by Focusing on Commonalities in Test Procedures. PhD thesis, Technische Universität München (2019)

Khan, Mohammed Ehmer; Khan, Farmeena: A Comparative Study of White Box, Black Box and Grey Box Testing Techniques (2012)

Littlewood, Bev: How Good Are Software Reliability Predictions? Software Reliability Achievement and Assessment, Oxford: Blackwell Scientific Publications (1987)

Memon, Atif M.; Stoffa, Mary Lou: Regression Testing of GUIs (2003)

McMaster, Scott; Memon; Atif M.: ” An Extensible Heuristic-Based Framework for UI Test Case Maintenance”. In: Proceedings of the IEEE International Conference on Software Maintenance and Evolution (ICSME). Edmonton, Alberta (Canada) (2009)

Microsoft Corporation, Designing a Test Environment https://learn.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2003/cc778654(v=ws.10) (2003)

Myers, Glenford; Sandler, Corey; Badgett, Tom: The art of software testing. New Jersey: John Wiley & Sons, Inc., Hoboken (2012)

Nedyalkova, Stanislava; Bernardino, Jorge: “Open Source Capture and Replay Tools Comparison”. In: Proceedings of the 6th International C* Conference on Computer Science and Software Engineering. Porto (Portugal) (2013)

Sun, Yahong; Jones, Edward: Specification-Driven Automated Testing of UI-based Java Programs, Meditronic, Inc, and Florida A&M University (2004)

Paradkar, Amit; M.A. Vouk, K.C. Tai: Specification-based testing using cause-effect graphs (1997)

Patton, Ron: Software Testing (2nd Edition) (2005)

Oliveira, Gustavo de; Duarte, Alexandre: A framework for Automated Software Testing on the Cloud, Federal University of Paraiba, Paraiba, Brazil (2010)

Rafi, Dudekula Mohammed; Moses, Katam Reddy Kiran; Petersen, Kai: Benefits and Limitations of Automated Software Testing: Systematic Literature Review and Practitioner Survey, Blekinge Institute of Technology/Erikson AB School of Computing, Karlskrona, Sweden (2012)

Wang, Shuang; Offutt, Jeff: Comparison of Unit-Level Automated Test Generation Tools, George Mason University, Fairfax, VA (2009)

UI Testing

Adamoli, Andrea et al. ” Automated GUI Performance Testing”. In: Software Quality Journal 19.4 (Dez. 2011)

Huang, Si; Cohen, Myra; Memon, Atif M.: “Repairing UI Test Suites Using a Genetic Algorithm”. In: Proceedings of the 3rd IEEE International Conference on Software Testing, Verification and Validation (ICST). Washington, DC (USA) (2010)

Fachrul Pralienka Bani, Muhamad et al.: Visual GUI testing in continuous integration environment (2016)

Memon, Atif M.: “Automatically Repairing Event Sequence-Based UI Test Suites for Regression Testing”. In: ACM Transactions on Software Engineering and Methodology 18.2 (Nov. 2008)

Nguyen, Bao N. et al.: “GUITAR: An Innovative Tool for Automated Testing of GUI-driven Software”. In: Automated Software Engineering 21.1 (März 2014).

Fachrul Pralienka Bani Muhamad, et al.: Visual GUI testing in continuous integration environment (2016)

Manager’s view on UI Tests

Kaner, Cem: Avoiding Shelfware: A Managers’ View of Automated UI Testing, https://www.kaner.com/pdfs/shelfwar.pdf, 2002

Ramler, Rudolf; Wolfmaier, Klaus: Economic Perspectives in Test Automation: Balancing Automated and Manual Testing with Opportunity Cost, Software Competence Center Hagenberg GmbH, Hagenberg, Austria (2006)

Mueller, Matthias; Padberg, Frank: About the Return on Investment of Test-Driven Development, University of Karlsruhe, Germany (2017)

Software Engineering in general

IEEE Standard Glossary of Software Engineering Technology ANSI/IEEE 610.12, IEEE Press (1990)

ISO/IEC/IEEE. Standard 29119-1: Software and systems engineering – software testing – part 1: Concepts and definitions (2013)

Martin, Robert C.: Clean Code, Prentice Hall (2009)

McConnell, Steve: Code Complete: A Practical Handbook of Software Construction, Second Edition (2004)

Using QF-Test in Continuous Integration Systems

2019-07-11T00:00:00+02:00

Last update: 2022-03-30

Continuous Integration Systems (CI-Systems) are essential for modern software development. They support the teams at their steady work on the various sub-steps of the software development process. The main objective being the improvement of software quality. As an example, with CI systems, you can automate and regularly run the process of compiling and linking the software modules whenever the source code they are based on changes. However, you can trigger any other process by the CI system like setting up metrics, generating documentation or even executing automated tests checking certain functionalities of the software modules. For these automated tests you can use Unit tests on the source code level or QF-Test for functional UI tests.

The automatic execution of operations within a CI System can either be scheduled for certain times or triggered by certain changes in the software modules. Thus reducing the so-called feedback cycles – a huge gain with CI systems.

The system informs you without delay about errors which may have occurred within the single CI operations. For example, when a developer checked in non-compilable source code into the version management system or a UI test reveals an hitherto unknown error in the regression tests started automatically.

It saves a lot of time and energy if things like that pop up right after applying the change rather than much later during the full nightly build and test.

There are various types of Continuous Integration Systems with different functionality and ranges of use. Some of the more known systems are

Jenkins
TeamCity
Travis CI
Bamboo
Gitlab CI/CD
CircleCI
among many others.

QF-Test can be run in batch mode and therefore be easily be integrated into those systems.

In order to run a test in batch mode with QF-Test you just open a command line shell and execute the following command:

qftestc.exe -batch -run c:\mysuites\suiteA.qft

This call will start QF-Test in batch mode and run all the test of test suite suiteA.qft. Subsequently QF-Test will generate the respective QF-Test log and save it to the file system.

With QF-Test batch mode you can use a great number of parameters. It would be beyond the scope of this blog to explain them all.

For further information please have a look at:

Almost all CI systems offer the option to execute command lines as part of the continuous integration process. Just insert the command line mentioned above to start the tests.

Jenkins Test automation

If you are using Jenkins, life is even easier. QFS provides a plugin you can directly install into the Jenkins environment. It only take two clicks to install the plugin, then you have a UI to configure the QF-Test command line. Additionally to the classic CI model Jenkins supports pipelines.

Find further information on the plugin here: https://plugins.jenkins.io/qftest/

Case study

Nice practical example of a CI chain – besides faster feedback, better SW quality also time saving. Thank you Mr. Schöning for the exemplary implementation!
go to the detailed case study

Veni, vidi, variavi – Using Variables

2019-07-10T00:00:00+02:00

There‘s no way around using variables if a test suite has to to be flexible, reusable, and elegant. But where are variables set? Which value of the variable is used if I defined multiple ones at different positions? And how can I find out? This article aims to answer these questions.

Which value is used?

Variables are saved on two stacks: the primary stack and the secondary stack, also called fallback stack. The second-mentioned is used for default-values, that are only called if there is no value found on the primary stack. Those stacks are not hidden, but can be viewed anytime – e.g. by pausing the test execution or debugging. Also the stack trace is attached to the run log in case of an error/exception (Fig. 1).

Figure 1: Error ⇒ Stack trace can be found in the run log

When the value of a variable is needed, QF-Test will try to find it in the stack. It will start at the “top” of the primary stack, going one layer “lower” if it does not find a binding for the variable. The default values on the secondary stack are only taken into account if direct binding is found in the primary stack.

To illustrate this, take a look at (Fig. 2). Here, “Variable1” was bound to a default value in “ExampleProcedure”. However, when this procedure is called, the value is explicitly set to “not default”. This binding ranks higher and overwrites the default value on the fallback stack.

Figure 2: The default value of “Variable1” is overwritten by the procedure call

How are variables set?

There are many ways to set variables. At Edit → Options → Variables latter can be set, some even before a test suite exists. The stack trace shows which binding (System, Suite Command line, Globals) ranks the highest.

Every node that has the attribute “Parameter default values” allows to set default values on the fallback stack.

The most obvious choice is the “Set variable”-node, of course. But there are also multiple ways to read values from our system under test, like the text in a table cell. The simplest way to achieve this is to record a mouse click on the element that contains the value, then transform the node into a “Fetch text”-node. The result of the fetch can be bound to a variable and used later. (Fig.3) shows such a node and its attributes. But what does “Local variable” mean?

Figure 3: Text of a table cell is fetched and saved as a local variable

To keep it short and simple: Global variables exist from their creation to the end of time (that being the end of the QF-Test process or until they are explicitly deleted). Also, they are not just bound to your current test suite. Thats an easy to understand concept, but it also can lead to easy to avoid mistakes, where a variable is unintentionally changed somewhere else or not reset after an earlier test execution.

In the stack, local variables can be found bound to the node on which they are valid. The respective node can be found by debugging, as shown later. If the variable is set in a test case, for example, its value exists in the scope of this test case and all test steps, sequences etc. within it. After this node is done and disappears from the stack trace, all of the local variables that belonged to it cease to exist. This way side effects in huge and complex test suites can be avoided.

At last there are scripts. A variable can easily be set in a Server- or SUT-script, as you can see in (Fig. 4).

Figure 4: Setting and reading variables in a script Up to version 9.0, all QF-Test variables were stored as strings. To process them in scripts or similar, they sometimes had to be converted. This changes with object variables – see the blog post Objects in QF-Test variables – an overview for more information.

Which value is used for the variable?

It‘s easy to lose track of which variable is assigned to which value when working with default values, multiple nested procedure calls and global variables. Luckily there are multiple possibilities to find out the current value of a variable, two of which are presented here:

1. Everyones favorite: “print debugging”

Simply insert a script with qf.println(rc.lookup("name_of_your_variable")) to print the value of the variable in the terminal. qf.print got introduced with version 4.6 and works in every scripting language – for older version use the language-specific print method. You can log the value by using rc.logMessage or rc.logError. Just be a good boy scout and clean up those print statements after you are done.

2. Normal debugging

Ctrl+F8 (Mac: ⌘+⇧+B) enables breakpoints in your suite. QF-Test will stop the test execution at those breakpoints and switch to debugging mode. Now, these helpful buttons (Fig. 5) enable you to execute nodes step-by-step and a tab for “Variables” showing the current stack trace appears next to the terminal (Fig. 6). As mentioned before, the highest binding of a variable is its currently used value.

Figure 5:step in (F7), step over (F8), step-out (Ctrl+F7)

Figure 6: The highest binding of “Variable1” is the currently used one

News about the print(…) and println(…) module

2019-07-04T00:00:00+02:00

If you are working with QF-Test scripting nodes, no matter whether you prefer Jython, Groovy or JavaScript, you will come across the QF module sooner or later. This module, accessible in all script nodes and script terminals via the variable qf, holds a bunch of helper methods designed to make the life of a script programmer easier.

From time to time the module will be enhanced by new methods – in QF-Test 4.6.0 by the methods print and println. A good opportunity to have a look at the module and its new methods.

qf.print(ln)

The method qf.print(…) prints a string (resp. the string value of an object) to the QF-Test terminal (and with SUT scripts also to the client terminal). qf.println(…) adds a line break to the output to ensure the next output will be printed to a new line. But why not just use a simple print (without qf. in front)?

Theoretically yes, but… The classic print writes a string to the so-called Standard Output of the program. QF-Test intercepts the string and redirects it to the terminal. Usually, this works all right, however, especially with SUT scripts, you might run into the problem that the client application itself redirects the Standard Output thus preventing QF-Test ever to ‘see’ the output. This can become really annoying when you need to resort to the classic ‘print debugging’ when writing resolvers.

qf.print(…) skirts the problem, not transmitting the string via the normal output channels, but sending it ‘directly’ via the internal RMI interface (i.e. via the same ‘line’ QF-Test uses to control the SUT and to send values for checks and events). Now, client specific adaptions will not matter anymore and on top it is slightly more efficient.

The small print

I would like to go into detail with some side effects:

1. qf.print(…) does not only take one parameter but as many as you like and joins them by space characters. This little JavaScript program:

const x = 1  
const y = 1  
const z = x + y  
qf.println(x,y,z)

will produce the terminal output ‘1 1 2'.

2. qf.print(…)calls the Java string method toString() to generate the output. So, if you have been trying in vain to check whether the Jython array module really produces a Java array:

import array  
from java.lang import String  
   
print array.array(String,("a","b","c")) # array(java.lang.String, [u'a', u'b', u'c'])

qf.println(…) will now be a solution:

import array  
from java.lang import String  
   
qf.println(array.array(String,("a","b","c"))) # [Ljava.lang.String;@353191ee]

The first script will print a Jython string representation of the array (“When an array object is printed or converted to a string, it is represented as array(typecode, initializer). “), however not representing the view of Java on the object. The second script will print the output of toString() for Java arrays, i.e. the type information and its hash code (just for once this output will prove helpful in order to check it is a real Java array. In the Java world you would rather prefer Arrays.deepToString(a) to see the content). If you want to print the Jython string representation just add str(…): qf.println(str(array.array(String,("a","b","c")))).

3. Like the other ‘standard’ variables (rc, out, resolvers, notifications, …) with Groovy scripts qf principally only works on top level:

class InnerClass {  
    def printSomething() {  
        println("Something")  
    }  
       
    def printSomethingElse() {  
        qf.println("SomethingElse")  
    }  
}  
   
def c = new InnerClass()  
   
c.printSomething() // => Does not reach the terminal in a Server-Script nicht im Termial an, because "println" directly uses the standard output.  
c.printSomethingElse() // => "No such property: qf for class: InnerClass"

To get around this, you can transfer the bindings via the contructor to the inner class:

class InnerClass {  
      
    def out,qf  
       
    def printSomething() {  
        out.println("Something")  
    }  
       
    def printSomethingElse() {  
        qf.println("SomethingElse")  
    }  
}  
   
def c = new InnerClass(out:out,qf:qf)  
   
c.printSomething()  
c.printSomethingElse()

Simple and effective, especially when writing your own checker or when updating older resolvers.

Something doesn’t work in QF-Test

2019-06-04T00:00:00+02:00

You are working with QF-Test and something something doesn’t work as expected? Then you can (as usually) search the Internet or do research at StackOverflow… A faster way is going on our channels …

General FAQ, Technical FAQ and Videos
context-sensitive help (click right mouse button on arbitrary tree node or attribute (in the details pane) → select “What’s this?” → reference in the manual)
Tutorial, Manual, Mailing list, the Blog here
full-text search (onsite search in the upper right corner) including manual, tutorial, standard lilbrary, QF-Test website, mailing list archive, blog
All information for a successful start can be found on Get started
Free starter webinar every Monday 4 p.m. CET/CEST
Ask our support team directly:
- via mail to support@qftest.com (please include your run log – see manual Chapter 8.1 – and test suites)
- via support form (in QF-Test’s “help” menu “Contact support team”)
- phone our support hotline +49 8171 – 38648 20

Which help was especially valuable for you? We looking forward to your comments – thanks.

Slightly different topics to classic support are bug reports or feature requests. They are followed by individual blog posts.

No coordinates based Events and no Image based checks! QF-Test is smarter than that! Part I

2019-05-29T00:00:00+02:00

Hello fellow QA Experts!

Some days ago I had the pleasure to visit a Meetup Venue here in Leipzig, Germany. The topic was very interesting – “Experience with the introduction of Test Automation”.

I was (not) surprised that many of the participants consider Web UI testing very fragile and cumbersome to keep in “Tests OK” status.

Especially in the early stages of the web development when Tests are already created the unexpected changes in the Web Page structure, Components’ IDs or Names can lead to overwhelming tests tweaking. Furthermore lots people think that because of this limitation of automated web tests only Image Based Checks and Absolute Mouse-Events are used by testing-tool.

For those that do not know QF-Test very well and for those that have not evaluated QF-Test I have decided to write a few post Blog series with the topic:

No coordinates based Events and no Image based checks! QF-Test is smarter then that!

P.S. Many thanks for the great place and hosting of the Meetup at Saxonia Systems AG, Leipzig.

Stay tuned,

Plamen Vesselinov

Mark nodes via colors

2019-05-14T00:00:00+02:00

In order to simplify test suite editing it is possible to assign colored marks to nodes. This feature is especially (but not only) helpful in the context of complex find and replace operations – in this case, the color markers can for example be used to indicate that a node still needs revision or that a node was edited successfully. It may even make sense to use another color mark in order to indicate that you may want to re-look at a certain edited node later on.

There are several ways to assign color markers to nodes. Manually this is often done via the Edit → Marks menu. In this menu first activate the desired color mark. Then via Edit → Marks → Toggle mark it is possible to toggle the desired color mark of the currently selected node. The following image shows different mouse-click nodes. While the first mouse-click node does not have any color marker, the second click has a blue marker, the third a red marker, and so on:

As can be seen in above picture, QF-Test is also providing the parent-nodes with corresponding smaller color marks. Like this users can quickly see whether any node in the tree (still) has any color marker and in which subtree the node with the corresponding color mark can be found.

Via the search dialog (Edit → Search) it is also possible to assign color marks to nodes. To do this, select the button “Show result list” in the search dialog. The dialog that opens lists all nodes that fulfill the previously entered search criterion. With the keyboard shortcut Ctrl+A it is now possible to select all nodes. You can then assign the selected color to the selected node:

A search based on these colors is possible. Just click on the yellow >> symbol in the search dialog to reach the extended search. Then you can choose to search only nodes having a certain color mark.

Remarks

Please note that QF-Test is not saving color marks in .qft files. This means that after saving, closing and re-opening a test suite, all previously set color marks disappear. In order to save a color mark, you need to enter @green, @red, @blue or @yellow in the comment field of a node. When QF-Test opens a test suite, it will interpret these doctags and set the appropriate markers.

Mouse Events Explained – Web

2019-05-06T00:00:00+02:00

Here comes one more short blog article, which will shed some light on the mouse events and specifically semi-hard events. QF-Test uses semi-hard events as default (based on web framework, browser and browser version, OS etc), but not all cases.

There are three event types available

Soft: A soft event is an artificial event which is inserted directly into the event queue. They are inserted directly at DOM/JavaScript level.
Semi hard: QF-Test own terminology. This is an artificial system event which gets sent to the browser. The browser has to interpret it and create the relative (soft) events for the web application.
Hard: a real movement of the mouse cursor and a simulation of a mouse down and mouse up which triggers a system/os event.

Set semi hard events via SUT Script before the mouse click:

rc.setOption(Options.OPT_WEB_SEMI_HARD_EVENTS, True)

reset the option via:

rc.unsetOption(Options.OPT_WEB_SEMI_HARD_EVENTS)

Disabled node

2019-05-03T00:00:00+02:00

In some cases it is helpful to disable nodes temporarily. Disabled nodes are skipped during test execution. To deactivate or reactivate a node, either the keyboard shortcut Ctrl+Shift+D or the menu item Edit → Toggle disabled state can be used.

In the tree, deactivated nodes are displayed in gray so that it is easy to distinguish between active and inactive nodes:

“license”, “license.new”, “license.txt”, “license.dat”, “license.old”, “license.server” … Who’s gonna know about that?

2019-04-24T00:00:00+02:00

Indeed, it is quite confusing if you would take a look how the supposed license files attached to emails are arriving. Although it is actually quite simple …

“license”

The name of a full QF-Test license file is “license”. As QF-Test was developed under Linux originally there was no apparently reason to add an extension to the filename.
Why we should? Linux doesn’t interpret the extensions.

Looking at Windows we have a totally different behavior. Windows is using the fie extensions to determine the type of the file (Linux is cleverer in that case). For that reason, each administrator in a Windows environment is configuring the computer in the company that the file explorer will always show the file extensions (so that you don’t open attachments like “order.pdf” and then it turns out that the attachment was named “order.pdf.exe” instead and was a malware).
But I’m digressing…

So under Windows files without an extension are not allowed at all (in fact they are possible despite of that).
And there is now the e-mail client, which in its emergency with a file “license” seals in its opinion a suitable extension. Outlook likes to attach a neutral “.dat”. Thunderbird looks into the content and finds that it is an ASCII text file, so it appends a “.txt” to the file. Other email clients can make other decisions here, but will never attach a “.new” or “.server”.

So you only need to know that all QF-Test license files attached, which are not named “license.new” or “license.server”, are full QF-Test license files which should be saved as “license”.

“license.new”. What’s about that?

This is a license update or license upgrade file. Such files need a fitting full license file to create a file “license” locally.
The old file will be renamed to “license.old” by the way and may be deleted. If you wouldn’t do this, next time a file “license.old2” will appear and so on.

Thus this file never can be used for QF-Test at a new installed machine. Instead the, ideally, once created file “license” must be used.

Therefore you should provide the resultant file “license” after an update or upgrade process always to your affected colleagues.

Special case “license.server”

This file is in this sense a special case as it is a full license file, but only for the QF-Test license server (in case you would use it instead of the standard license mechanism). But the QF-Test installations which are communicating with that license server are using a file “license”, too. The file “license” contains the information how to address the license server in that case.

My license file still seems not to be valid

In such rare cases that your license file still seems no to be valid although you have checked to possible reasons explained above we will help you, of course. In such a case please let us know what you already have tried and what happened in detail, please.

Writing your own checkers

2019-04-16T00:00:00+02:00

While testing an application, one of the most important step is to verify that the actual state of the application is correct. For example, when we test a calculator application, then at a certain moment we need to verify that the application has calculated the right result. This means, we need some kind of action (called check) to verify that some kind of UI component, e.g. a textfield is showing the right piece of information.

The most easiest way to record a check action is to enter the check recording mode, e.g. by clicking on the corresponding item in the toolbar.

Then right click the corresponding component and select the wanted check.

Wait a minute, what should we do, if the wanted check is not in that list? Well, either we use a SUT script in order to verify the component state or we simply implement our own check.

Please note, both ways to verify the component state are similar and both require programming knowledge and – most often – some knowledge about the used UI technology and/or how the component itself was implemented. Thus it may be a good idea to ask one of the developers for help.

Let’s give an example. As shown in the above image, the list of possible checks does include a “text color” check. I will use Jython here, but you can do the same thing with Groovy or Java Script (or any other scripting language that may be added in future).

1. Analyze the UI component

When we don’t know, how exactly the text color is implemented, we need to analyze the properties / methods of the component in order to find a method that returns the wanted information. The easiest way to do so in Jython is the dir and the type command:

com = rc.getComponent("<QF-Test component id>") print type(com) print dir(com)

You can get a QF-Test component id for your component by simply recording a click or any other action on the desired component. Instead of the print statements you can also use the rc.logMessage() method. Then the output will be written into the run log.

In my example

type(com)

will return that the component is a standard javax.swing.JTextField. So we can have a look at the corresponding documentation to find out that it is possible to implement such a thing via the setForeground() method. This would mean that we can get the corresponding data via the getForeground() method:

com = rc.getComponent("<QF-Test component id>") print com.getForeground().getRGB()

will return the desired color as integer and bingo, this will print the color as integer.

2. Step, add a SUT script

Now let’s use our knowledge that the getForeground() method will return the desired color as integer and simply wrap this code into a checker script:

from de.qfs.apps.qftest.extensions.checks import CheckerRegistry, Checker, DefaultCheckType, CheckDataType  
from de.qfs.apps.qftest.shared.data.check import StringCheckData  
from de.qfs.lib.util import Pair  
import jarray  

componentClass = "javax.swing.JTextField"  
foregroundColorCheckerType = DefaultCheckType("Color", CheckDataType.STRING, "Foreground color")  

class ForegroundColorChecker(Checker):  
    def getSupportedCheckTypes(self, com, item):  
        return jarray.array([foregroundColorCheckerType], DefaultCheckType)  

    def getCheckData(self, com, item, checkType):  
        if foregroundColorCheckerType.getIdentifier() == checkType.getIdentifier():  
            color = com.getForeground().getRGB()  
            return StringCheckData(checkType.getIdentifier(), str(color))  
        return None  

    def getCheckDataAndItem(self, com, item, checkType):  
        data = self.getCheckData(com, item, checkType)  
        if data == None: return None  
        return Pair(data, None)  

global foregroundColorChecker  

# unregister a maybe already present instance of this checker  
try: CheckerRegistry.instance().unregisterChecker(componentClass, foregroundColorChecker)  
except: pass  

# after unregistering, register a new instance of this checker  
foregroundColorChecker = ForegroundColorChecker()  
CheckerRegistry.instance().registerChecker(componentClass, foregroundColorChecker)

Now after we have executed this script, the corresponding checker will be available in the checker list and the corresponding checker can get recorded and replayed.

Remarks

This is a basic example and in real life you may want to convert the integer representation of the color into a more readable color representation.

You may use your own checker to implement checks for custom components.

If you want to work in Germany, come to us

2019-04-09T00:00:00+02:00

QFS (Quality First Software GmbH) is one of the best employers of the ITC industry in Germany and also in Bavaria. We are placed in Geretsried and Leipzig and have won the Great Place to Work award already two times now. Geretsried is between Munich and the Alps at the Isar river.

In 2018 we took first place as newcomer in the category of firms with less than 50 employees. This result was presented in the Great Place to Work Institute in Munich. This year we won as well in this voluntary audit. The anonymous employees’ opinions count twice in comparison to the company’s report about fairness, credibility, respect, pride and team spirit. All of our 16 employees took part in the survey.

Here is a short overview of what QFS offers: Besides two home office days for the whole company, we go and have lunch together paid by the company three times per month. Who wants to have a short nap afterwards can sleep whenever there is the need except in the afternoon when we are playing table football.

Accessing sub-items of UI components

2019-04-01T00:00:00+02:00

In most cases we are dealing with more or less straight-forward UI components, such as buttons and text fields. Nevertheless we often have to deal with more complex UI components like trees, tables and lists too. These UI components have sub-items. For example the UI component “Table” consists out of different table cells (=the sub-items of the table):

QF-Test provides multiple ways to access sub-items of UI components:

Using the “QF-Test component ID” to specify the wanted sub-item

In the “QF-Test component ID” attribute it is possible to specify that an action should be replayed on the sub-item of a UI component. In order to do so an identifier is added to the component id. This identifier may either be numerical (represented via the character &), textual (represented via the character @) or the identifier may be a regular expression (represented via the character %).

Examples

List or comboboxes

You only need one identifier to describe a sub-item in a list or a combobox:

As can be seen, the above combobox has 4 different sub-items. The sub-item “- no special model -” (with the numerical index 0), “Gomera” (with the numerical index 1), “Jazz” (with the numerical index 2) and the sub-item “Luxus” (with the numerical index 3). So either qftestIdOfComboBox@Jazz or qftestIdOfComboBox&2 will access the sub-item Jazz in this combobox. Alternatively we can also use qftestIdOfComboBox%J.* – as J.* is a regular expression that is matching the sub-item “Jazz”.

And in order to access the sub-item “Gomera” we can use:

Tables

In the case of a table two identifiers are needed. The first identifier is specifying the column and the second identifier the row of the wanted cell. So if we have the following table:

An action on the QF-Test component ID qftestIdOfTable@Price%.*12.* will target the cell that contains the text $12,300.00. The same cell gets targeted by the QF-Test component ID qftestIdOfTable@Price&0.

Trees

In order to access sub-items in a tree, we need an identifier that is taking into account the hierarchical framework of this UI component. So if we have the following tree:

The sub-item “Mats” can be accessed via:

qftestIdOfTree@/Information/Accessories/Mats
qftestIdOfTree&/0/0/2
qftestIdOfTree%/Inf.*/Acc.*/M.*
qftestIdOfTree%/.*/.*/Mats

(and via other regular expressions). However it will not work to access the sub-item Mats via the QF-Test component ID qftestIdOfTree%/.*/Mats. This is because with %/.*/Mats we are advising QF-Test to search a sub-item matching the regular expression .* on the first level of the tree. In this example there is only one first level sub-item (“Information”). The regular expression provided to QF-Test (.*) is matching this sub-item. QF-Test will then go on and use the next regular expression (“Mats”) in order to search through the child nodes of any matching node (“Accessories”, “Description”). As can be seen, both strings are not matching the desired regular expression, so %/.*/Mats does not work.

Using item nodes to specify the wanted sub-item

Item nodes provide the same functionality, namely to specify the wanted sub-item by index, name or regular expression. However these item nodes are used less commonly:

You can insert these nodes over the menubar (Insert → Component nodes → Item) as child nodes of any list, table or tree component node (or more general every node representing a component that supports sub-items):

The text specified in the “QF-Test ID” attribute of this node (GomeraItemInComboBox in the given example, but you can use nearly any other text as well) can then be used in the “QF-Test component ID” attribute of a mouse event node (or any other event replaying node). When such an event node gets executed, QF-Test will first determine the wanted UI component via the component recognition information provided by the parent node of the inserted item node. Then QF-Test will determine the wanted sub-item by the information provided in the inserted item node. Once this is done, the desired action will be replayed on that sub-item.

QPath

Sometimes we need to access sub-components that are located within sub-items of any UI component. This may (for example) be a button or a checkbox located inside a table cell. In order to access these subcomponents, we can use the @:ClassName-syntax (the so called QPath syntax). For example, if we have a button located inside a table cell, we can access it via: qfcomponentIdOfTheTable@columnname@rowname@:Button<0>.

XPath and CSS Selectors

When you are testing a web-application it is also possible to use an XPath or CSS Selector

in order to access the wanted UI component and/or sub-item. As XPath and CSS selectors have been proven to be relatively difficult to write they are normally not my first choice. However this feature is nevertheless quite useful when it comes to rewriting already existing Selenium web tests into QF-Test.

The general syntax in order to use an XPath/CSS selector inside a “QF-Test component ID” attribute of an event node is:

genericHtml@:xpath=${quoteitem:yourXPathGoesHere}
genericHtml@:css=${quoteitem:yourCSSSelectorGoesHere}

Alternatively it is possible to specify an xpath and/or css selector in the extra features of a component node (or in SUT scripts). This can be done via a “Must match” extra feature with the name qfs:item and the value @:xpath=${quoteitem:yourXPathGoesHere}. (In case you want to use a css selector use @:css=${quoteitem:yourCSSSelectorGoesHere} as value.)

Notes

Out of obvious reasons the @:xpath/@:css-syntax assumes that the used XPath/CSS-Syntax returns a single component. The usage of a XPath-syntax that does not return a single component, but for example an integer (Example: count(.//input[@id!='Google'])) or a boolean (Example: nilled($in-xml//child[1])), may lead to unexpected behavior.
If you know what you are doing, you can nest multiple xpath/css selector statements. For example: genericHtml@:xpath=${quoteitem:firstXPath}@:xpath=${quoteitem:secondXPath}
For those of you who are wondering what the genericHtml part in the string genericHtml@:xpath=${quoteitem:yourXPathGoesHere} means. This substring is referring to the genericHtml component node that is specified in the qfs.qft.
The used XPath is searching for the wanted component from this UI component onwards.

Examples

Let’s assume you have a single text input field on your website that is implemented via:

<input type="text" ... additional attributes ... />

XPath within a QF-Test component ID

In event nodes it is possible to address the text input field (in the above sample website) via the QF-Test component ID genericHtml@:xpath=${quoteitem:.//input}.

CSS Selector within a QF-Test component ID

In event nodes it is possible to address the text input field (in the above sample website) via the QF-Test component ID genericHtml@:css=${quoteitem:input}.

XPath within a QF-Test component node

The following image show an example component node that is specifying an XPath in its extra features:

Once this component node got inserted as a sub-children of the “Windows and components” node, it is then possible to use the string “myXPathEle” (or whatever got specified in the “QF-Test ID” of this component node) in the “QF-Test component ID” attribute of event nodes.

XPath and/or CSS Selectors within SUT Scripts

In order to use an xpath in a SUT script, the following statements can be used:

com = rc.getComponent("genericHtml") # or rc.getComponent("genericDocument")  
res = com.getByXPath(rc.lookup("xpath"))    # find subcomponent via xpath  
res = com.getByCSS(rc.lookup("css"))        # find subcomponent via css  
res = com.getAllByXPath(rc.lookup("xpath")) # find all subcomponent via xpath  
res = com.getAllByCSS(rc.lookup("css"))     # find all subcomponent via css

Password encryption

2019-03-05T00:00:00+01:00

Sensitive information like passwords should be treated with care. However often tests need to fill out a login form and thus these tests require to know the password.

In order to keep passwords save, it is possible to tell QF-Test to encrypt them. This may be done by finding the “Text input” node that is inputting the password into the password field. In this node right-click the text attribute, then chose “Crypt password”:

While executing this input text step, QF-Test will then decrypt the encrypted password in order to input the password in the corresponding login password field. The decrypted password will not be mentioned in the run log or other reports created by QF-Test.

Remarks

It is a good idea to set a salt in order to improve the password encryption. In order to do so open the options dialog (menu: Edit → Options…). In this dialog open the replay options and set a random string as password encryption salt:

As the salt plays an important role in the password en-/decrypting algorithm, the salt has to be set before the password gets encrypted.

The salt is saved in the system.cfg. So by ensuring that all QF-Test instances in your network are using the same system.cfg configuration file, you can ensure that all QF-Test instances can encrypt/decrypt the password. The ”-systemcfg ” commandline argument may be used in order to ensure this.

Pay attention: Everyone who has both, the password salt and the encrypted password has the possibly to decrypt your password.

Creating custom HTML/XML/Junit reports

2019-02-28T00:00:00+01:00

Via the run log QF-Test provides a detailed log about the actions it executed during a test run. The overall results of this run log can be summarized in other reports like the HTML/XML/Junit report.

A not well documented – but gladly used – feature is that it is possible to adapt/change the style/content of these reports to the own needs. Nevertheless, in order to do so, basic programming knowledges are inevitably needed.

The HTML/XML/Junit reports are customizable. So the content of the “name” and “package” attribute can get changed.

The HTML/XML/JUnit reports are generated via an XSLT-Transformation. The therefore needed XSLT sheets are located in the directory <Path to the QF-Test installation directory>\qftest\qftest-$(version)\report.

If you are changing these Sheets, you may want to copy/move them to <Path to the QF-Test installation directory>\qftest\report.

As the XSL transformation sheets in this directory are preferred over the XSLT sheets in the <Path to the QF-Test installation directory>\qftest\qftest-$(version)\report directory. Thus, your modified XSLT sheets will then also be used by later QF-Test versions. However – as an disadvantage – whenever a new QF-Test version is coming out with any XSLT-Sheets improvements, you will not notice these improvements anymore (as QF-Test will keep using the Sheets in the <Path to the QF-Test installation directory>\qftest\report directory).

Notes

The XSLT Identity Transformation may be used in order to get the original XML document out of which the HTML/XML/JUnit-Reports are getting generated:

<xsl:stylesheet version="1.0"  
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform">  
  <xsl:template match="@*|node()">  
    <xsl:copy>  
      <xsl:apply-templates select="@*|node()"/>  
    </xsl:copy>  
  </xsl:template>  
 </xsl:stylesheet>

When generating an HTML report, you may want that QF-Test copies additional files (like additional JavaScript or stylesheet files) into the generated target directory. It is possible to inform QF-Test about such files. Therefore one simply creates a file with the name ‘files-to-copy.txt’. In this text file the additional JavaScript, stylesheet or image files are specified. (One file by line.)

How to create procedure calls

2019-02-22T00:00:00+01:00

Procedures (also called function or subroutine) may be used in order to solve often (re-)occurring challenges. Like this, we may specify that certain successive actions are needed in order to solve a challenge (=programming a procedure). When we then encounter this challenge, we can simply call the procedure (and let the procedure solve this challenge for us).

How to insert a procedure call within QF-Test

Go to the point in your test suite where you want to call a certain procedure.
Insert a procedure call. In order to do so you may use the shortcut Ctrl+A or the menu Insert →Procedure nodes → procedure call.
A dialog will pop up:
1. Click the button next to the label ‘Procedure name’.
2. Another dialog will pop up. In this dialog choose the suite in which your procedure is located and select the wanted procedure in the tree.
3. After clicking on the OK-button the wanted procedure is specified in the ‘Procedure name’ textfield of the (first) pop up. If the procedure has some parameters, then these parameters are defined in the “Variable definition” table of the procedure call.
Depending on the exact challenge, you may need to change the predefined parameters in the “Variable definition” table or specify a text in the ‘Variable for return value’ textfield.

E.g. at a certain point in our test suite, we want to get the current date in the format “year-month-day”. In order to solve this challenge, we can use the procedure named qfs.utils.getDate which is already predefined in the standard library (suite: qfs.qft). So:

Step 3.2: We need the procedure qfs.utils.getDate in the qfs.qft. So we select the tab qfs.qft and then the procedure qfs.utils.getDate in the tree.
Step 4: By default this procedure is returning the current date in the format “day.month.year”. Because we want to get the current date in the format “year-month-day” we replace the text “dd.MM.yyyy” in the “Variable definition” table by “yyyy-MM-dd” (see ‘format’ row).
Step 4: We need to specify the name of the variable (e.g. currentDate) in which the current date should get saved. Therefore we simply specify currentDate in the ‘Variable for return value’ textfield.
After executing this procedure call we can access the content of this variable (and thus the string describing the current date). This can be done via the syntax $(nameOfTheVariable) (e.g. $(currentDate)) in the attributes of the different nodes or via rc.lookup in scripts.

The complete procedure call is then looking like so:

Remarks

The qfs.qft is public. This means everyone is allowed to see, analyze and learn from the different procedures in this suite. (You can open this suite via QF-Test. It is located in the folder <path to the installation folder of qf-test>/qftest-$(version)/include/)
It may also be a good idea to place suites with procedures you are use in multiple testing projects in an own, independent, directory. You can then tell QF-Test about this directory via the -libpath command line parameter.

Capture & Replay

2019-02-18T00:00:00+01:00

A significant part of all QF-Test tests have been – at least partly – created via the QF-Test capture and replay feature. Furthermore this feature has proven to be quite helpful – especially for beginners. The opposite of capture-replay is test creation via scripting.

The main goal of the capture & replay feature is to show/teach QF-Test the actions it should perform on the application it is testing. In the end QF-Test should be able to reproduce the shown steps (and thus to automatically perform the wanted test).

After QF-Test established a connection to the application it should test, the record button in the taskbar will become enabled.

Once the capture and replay feature (black square) got activated we can simply perform (green arrow) the wanted actions (e.g. click a certain component of the application). Once the stop recording button gets pressed QF-Test will add the recorded actions into the currently opened test suite.

Instead of long explanations, just take a look at QF-Test’s operation.

Remarks

Although recording a “complete test case” in one go is perfectly legal, it is often a good idea to implement the different actions step by step (modularization). This way it is easier to gasp a recording problem (in case one is occurring). Apart from the different recording settings, it is possible to influence the recording via resolver scripts, too.

Request quotation

2019-02-12T00:00:00+01:00

You have two possibilities to request a quote.

Send request via the following form
Or contact our sales team va email to sales@qftest.com or telephone +49 8171 386 48-10 and mention the following information:

Your data

Name
Telephone number
Company name + invoicing address
Country
Email address
VAT-ID for EU-Orders (outside Germany) or company-proof for orders from outside the EU
Technical contact + his/her email address (When you send the request as a reseller, please tell us also the name of the client’s company)

QF-Test

Number and type of licenses (Developer or Runtime)
Desired technologies (Java Swing, SWT, JavaFX, Web, Android)
Find an overview of all QF-Test offers here.

Training or Consulting

Desired number of training days, face to face or online
Desired number of consulting days, face to face or online
Find an overview of all training/consulting offers here.

Using QF-Test as Calculator

2019-02-05T00:00:00+01:00

Sometimes it is necessary to perform mathematical calculations in QF-Test, e.g. in order to calculate the expected height of a UI element in dependence of the complete window size or the exact clicking position for a mouse click.

In order to do so we can use a Scripting node. Alternatively it is possible to use the $[some mathematical expression] expression within node attributes. Like this the expression $[3 + 3] will be replaced by 6. When we have stored the width of a current element in a QF-Test variable named “canvasWidth”, we can use $[$(canvasWidth) – 1] in order to click somewhere at the right border of a UI element:

In fact the mathematical expression in the brackets is passed to a Jython eval statement. So it is also possible to perform more sophisticated operations within such an “$[…]” expression:

$["Hello World"[1:5]] will be replaced by “ello”.
$[rc.getLanguageName()] will be replaced by “jython”.
$[re.split("\\d*", "foo123bar4foobar")[1]] will be replaced by “bar”. Please note, this statement will only work if you have previously executed a Jython Server Script that imports the re-module.
$[__import__('re').split("\\d*", "foo123bar4foobar")[1]] will be replaced by “bar” (this statement will work whether a Jython Server Script that imports the re-module has been executed or not).

QF-Test and Java client code coverage analysis

2019-01-30T00:00:00+01:00

Code coverage analysis can be used in order to analyze to which degree the source code of a program gets executed during test execution. A high percentage of executed code suggests a lower chance of undetected software bugs.

QF-Test alone does not provide a possibility to generate code coverage reports. However it is possible to combine QF-Test with Open Source Tools like JaCoCo. The integration of such tools make it possible to generate such code coverage reports for the teststeps performed by QF-Test.

How to integrate JaCoCo into a QF‑Test test run

In order to integrate JaCoCo into a QF-Test test run, you first need to extract the ‘jacocoagent.jar’ file from the lib folder of the ‘jacoco-$(version).zip’ file. This zip file can be downloaded from here.

It is easiest to integrate JaCoCo, if the Java application you are testing is a .jar or a .class file. In this case you simply need to locate the ‘Start Java SUT client’ node that is starting your Java application. Then, specify

-javaagent:=destfile=

in the “parameter table” of this node.

In case you have a .exe or a .bat Java application, the above way will not work. In this case two Jython Server Scripts are needed. The first script with the content

jacocojar = "<path to jacocoagent.jar>"  
destfile = "<path+filename to where jacoco should write its statistics>"  
rc.setProperty("env", "JAVA_TOOL_OPTIONS", "-javaagent:%s=destfile=%s" % (jacocojar, destfile))`

needs to get inserted before the ‘Start SUT Client’ node and another more optional script with

rc.setProperty("env", "JAVA_TOOL_OPTIONS", None)

can be inserted after the ‘Wait for client to connect’ node.

Remarks

Please note, that JaCoCo is not directly writing its report into the result file. The result file is written, when the application you are testing gets terminated!
The JaCoCo result file is quite unreadable when opened with most text editors. So in order to analyze the statistics you probably want to use a plugin for your IDE.
In order to gain control over the application you are testing, QF-Test is injecting some of its own classes into the application. As a result the result file does not only contain code coverage statistics for the classes of your application, but also for some QF-Test classes. Because you probably do not want to test QF-Test but your application, you can simply ignore the statistics created for QF-Test classes.
Please be sure that the process that is executing your application has the rights to write the JaCoCo result file.
When specifying a (Windows) path in a Jython/Groovy Script, please replace all “" characters in the path by “/”.

Where do I get a trial license?

2019-01-25T00:00:00+01:00

Simply trying a tool is always the best approach of testing it. So jut get your QF‑Test trial license.

It is valid for four weeks after transmission and can be used by up to three users at the same time.

You can try QF-Test directly with your own application, get started!

We gladly support you with your POC.
You can create first test cases and suites for your further use. They then can be transferred into the commercial license.

Why should you test your software?

2019-01-16T00:00:00+01:00

Software bugs are annoying – for users and software manufacturers alike. The latter, however, should use bugs to continually improve the software, preferably from the beginning of development. Nevertheless, there will probably never really be flawless, right? Rather a philosophical question.

The history of software programming has also been shaped by software bugs. Of course software bugs are also a cause for trouble – especially if any bug is harmful, costly or has other unpredictable consequences.

But that does not mean we cannot learn from each bug. Each analyzed and fixed bug gives us the opportunity to learn about what went wrong. This learning process can help us to avoid bugs in the feature. The learning process helps to find other similar undiscovered bugs in other softwares too.

Here are two general ways to find software bugs:

The first one is to look at the source code of a program, analyze each step and check if we can find source code patterns, that may be problematic. (Either by hand (software review) or to some extend also automatically.) For example we can check if we find any source code patterns in the code that are similar to source code patterns that have proven to be problematic. To some extend finding and analyzing such patterns is already done by the compiler. This is why we should take compiler warnings seriously.
The other way to find software bugs is to simply execute the program/program-parts and test whether the program (under certain circumstances) crashes or shows an unexpected behavior. Once an unexpected behavior gets noticed, the bug needs to get analyzed and fixed. An observation here is that while fixing a bug it is also possible to introduce new bugs into the code. This wired connection between the debugging/fixing effort and the number of bugs in the code is also represented in the ‘Programmer’s Drinking Song’:

99 little bugs in the code.

Take one down, patch it around,

127 little bug in the code…

So we are not only testing in order to find new bugs but also in order to ensure that once fixed bugs stay closed. But this means that – whenever the source code gets changed – every test case/test step needs to get redone. When this is done by hand, this may be a pity, boring and time consuming step. With a specialized software – like QF-Test – we can automatize our test steps.

Welcome to the QF-Test Blog

2019-01-08T00:00:00+01:00

We want to inform you about technical topics for beginners, advanced users or even specialists with this blog. The blog should replace the a bit out dated mailing list, answers technical questions as well as general topics like (license and purchasing questions, QFS…).

You have new topics in mind? Please share them with us.

Happy reading!

QF-Test - blog

Using GitHub Copilot CLI via Remote SSH from QF-Test

Prerequisites

Step-by-Step Guide

Preparation on the Client VM

Preparation on the Target VM

Functional Test (Client VM → Target VM, headless)

Troubleshooting

Connect Copilot CLI to QF-Test via SSH

How to use the QF-Test AI integration with a free local model using Ollama

Step 1: Set up Ollama

Step 2: Configure QF-Test to Use the Local AI

Step 3: Use AI in your tests

The advantages of running locally

What’s next

Tutorial: Using GitHub Copilot with the QF-Test AI Integration

What you’ll need

Step 1: Install and configure the GitHub Copilot CLI

Step 2: Wire Copilot into QF-Test via a custom model

Step 3: Use Copilot with the QF-Test ai scripting module

Step 4: Validate UI text with “Check text with AI”

When tests become intelligent: AI-driven checks with QF-Test

Testing ZK Applications with QF-Test in 2026

QF-Test Featured in mgm technology partners’ Blog on Quality and Efficiency Strategy

A particularly interesting aspect

Why QF-Test matters for quality engineering

Conclusion

Scrolling on the Web with QF-Test – The complete guide

Scroll manually using keyboard events

Fine-tune the QF-Test scrolling algorithm

Options.OPT_WEB_SCROLL_VISIBLE

Options.OPT_WEB_MIN_DELAY_AFTER_AUTO_SCROLL

Options.OPT_PLAY_SCROLL_ITEM

Options.OPT_PLAY_SCROLL_PADDING

Options.OPT_PLAY_WEB_AUTO_SCROLL

Scroll components into view

Scroll by set amounts

Maximizing SUT windows – a good idea?

It’s better to work with fixed window sizes

How to use fullscreen anyway

How can I create screenshots of my client application with QF-Test?

Save screenshots to a file

It’s not magic: How the CustomWebResolver makes your web application UI testable

The problem with CSS and XPath selectors

There is a better way: The QF-Test CustomWebResolver

Okay, but how does this magical CustomWebResolver technology work?

Generic classes

CWR mappings

The component tree

And how do I configure this CustomWebResolver for my application?

Read more

Objects in QF-Test variables – an overview

The status quo up to QF-Test 9

Ensuring backwards compatibility

Object variables in variable tables

Object variables in “Set variable”, “Return” and parameter tables

Object variables via rc.getObj and rc.vars

Object variables and $(name)

Various small improvements

Outlook

From a simple email to a memorable internship

Extending the QF-Test Assertion API – A practical introduction

Introduction to QF-Test Plugin Development

My 4-Year Experience as a working student at QFS – A Retrospective

AI is a Big Topic: 11.14. is World Quality Day

World Quality Report 2024/25: Recommendations

A little journey through time

Foosball, Coffee and Code – My three-month internship at QFS

Webswing and QFS: A Powerful Alliance for Java Application Migration

JavaFX – More alive than ever

JFX Adopters Meeting at ZEISS

JavaFX and QF-Test

The future of JavaFX – and the JFX Adopters Meeting

Performance-Comparison of VMs and Docker containers

Test results

Workflow

Conclusion: Save time with Docker and Headless Browser

How Webswing is revolutionizing the Migration of Java desktop applications to the web

A Java application in a web browser?

The Webswing Migration Framework

Step 3: Use Copilot with the QF-Test `ai` scripting module

`Options.OPT_WEB_SCROLL_VISIBLE`

`Options.OPT_WEB_MIN_DELAY_AFTER_AUTO_SCROLL`

`Options.OPT_PLAY_SCROLL_ITEM`

`Options.OPT_PLAY_SCROLL_PADDING`

`Options.OPT_PLAY_WEB_AUTO_SCROLL`

Object variables via `rc.getObj` and `rc.vars`

Object variables and `$(name)`