As part of my studies in computer science at the Staatliche Berufsakademie Leipzig together with my « Praxispartner » Quality First Software, I wrote a scientific thesis on the topic of « Testing user interfaces in virtualised environments ». The thesis deals with the performance analysis of virtual machines and Docker containers for web tests.
My focus was on comparing the runtimes of tests in QF-Test in different test environments.
Schmied, Marcel: Testen von Benutzeroberflächen in virtualisierten Umgebungen (PDF in german)
Test results
The results have confirmed the assumptions that the Docker container is faster overall than the virtual machine, and that Headless Chrome is faster than the interactive Chrome with GUI.
The first diagram shows that QF-Test executes test steps with low overhead more than twice as fast in a dockerised environment. The performance differences are even more pronounced in interactive sessions than in « headless » browsers.
The second diagram shows that the measured QF-Test test runs in a dockerised environment are about twice as fast as in a full VM.
The third diagram shows that the more time-consuming QF-Test nodes « DocumentWaiter », « InstallCWRStep » and « BrowserClientStarter » are executed significantly faster in a dockerised environment than in a full VM. However, the differences are not always as significant as for the simpler nodes.
Workflow
A colleague from QFS was at my side during the entire time I was working on the project, supporting me with any questions I had and helping me to find topics, structure and organise the project.
In order to be able to compare the two test environments virtual machine and container environment, I first did extensive research on both, as well as on the topics of test automation and web testing. The detailed results of my research are recorded in chapters 2 and 3.
As an introduction to the practical part of my work, I started by setting up VMs and Docker containers with the same hardware configurations so that I could obtain a reliable test result afterwards. A Linux OS was installed on the VM and then QF-Test. For the Docker container, I only needed to download the matching Docker image, which is provided by QFS.
The Web CarConfigurator (the demo application supplied with QF-Test) was then run several times on each. This resulted in four data sets: A web test with Chrome and a web test with headless Chrome, each for VM and Docker containers. These data sets were later analysed using the methods of statistical result analysis.
Conclusion: Save time with Docker and Headless Browser
If you look at the individual node execution times, you can see that in some nodes, such as ClientWaiter or InstallCWRStep, the differences are relatively small, but in other nodes – those that interact with the UI – the differences are greater: This applies to MouseEventStep, SelectionEventStep and TextInputStep with up to 300% performance gain!
For the fastest possible test execution with the same available resources, you should therefore use a container environment with a headless browser for web tests. This saves up to two thirds of the time required for a test run compared to a virtual machine in combination with a normal interactive browser.
I personally found it a little more difficult to get started with Docker containers than with VMs. However, it is very easy with the QF-Test containers already prepared by QFS and is definitely worth it when you compare the runtime results.