“Why do my automated GUI tests break?” that was the seminar question of a student. With this, he asked the central question for test automation. Because the success of this, as well as the daily work of QA, stands and falls with the recognition of the objects – or their failure/breakage of the tests.
His research object was KeePass for Windows in the versions 2.0 to 2.18 (=SUT, System under test). With QF‑Test version he created KeePass2.00 GUI tests, which he ran in the following 17 versions.
The immediate results of the KeePass UI Tests with QF-Test
If tests failed, what was the reason? He found three main reasons (besides single reasons like disappearance of a component) and from all of them he identified the category “Trivial renaming” as the main cause.
How representative is his result for the experience of the support team of QFS?
There it was confirmed to me that – when tests are finished and running – equally trivial renaming actions in the further development of the SUT make up the main focus of the requests in the support team. That is, GUI tests break quite often because the recognition characteristics of the component have changed.
So dear developers:
Please set unique names or IDs and never ever change them anymore.
As written in the seminar paper, changing “Lock Windows” to “Lock Window” is clearly recognizable as “identical” for a developer and human, but a test automaton stumbles over it.
QF-Test is constantly developing its component detection concepts and optimizing the algorithm to make the generated tests as robust as possible against changes. It offers various possibilities to influence the detection of GUI components to make them as stable as possible.
Currently a new approach is developed, which will make it possible to limit the search area for a component still more dedicated, in order to make so apart from a more stable recognition by exclusion of multiple possible components the test still more stable and beyond that also faster. In addition the maintainability is increased, since one must maintain no more complex component trees.