QF-Test vs. SilkTest vs. Squish vs. Marathon
Evaluation results
Realization
Olav Krapp and Matthias Besemer, SBE network solutions GmbH, Heilbronn Products tested: Borland SilkTest, Froglogic Squish, QFS QF-Test, Marathon
Conclusion
The decision fell in favour of „QF-Test“ by QFS.
Justification
The main reason for this decision is the fact that QF-Test is much less developer-oriented than the other products. However, QF-Test still grants the necessary and also sufficient flexibility regarding changes in the GUI in test. Another decisive advantage of QFTest to be mentioned is its clearly arranged view and its – after some (although slightly longer) familiarization – also intuitive interface. The display of the test results and the available reports completed the overall impression positively.
Comparison matrix
Product |
SilkTest |
Squish |
QF-Test |
Marathon |
Advantages
|
- smooth support of obfuscating
- web testing already integrated
- screenshots can easily be recorded
- comprehensive integration in requirements-based testing / test management
- extensive tutorials, documentation and support
|
- component picker for e.g. verification
- unlimited Linux support
- user has not to be logged in for testing on remote computers
- utilization of common script languages
- three runtime licenses included in a user license
- extensive tutorials, documentation and support
- web testing available (though as a separate edition with additional cost)
|
- non-developer-oriented solution* (though very flexible)
- modern looking and basically user-friendly user interface (very clearstructured display of test suites, tests and other items)
- unlimited Linux support
- dependencies to avoid redundancies
- component picker for e.g. verification
- smooth support of obfuscating
- connection to SBE application without problems
- screenshots can easily be recorded
- reports for the test results
- floating license included
- extensive tutorials, documentation and support
|
- available at no charge
- open source software
- component picker for e.g. verification
- connection to SBE application without problems
|
Disadvantages
|
- developer-oriented solution*
- outdated appearing user interface, handling not very intuitive
- only limited Linux support (RedHat Enterprise)
- connection to SBE application faulty
- utilization of a proprietary script language
- user must be logged in for testing on remote computers
- expensive runtime license
- floating licenses for additional charge only
|
- developer-oriented solution*
- outdated appearing user interface, handling partially laborious (e.g. forced naming of test cases)
- connection to SBE application faulty
- no simple support of obfuscating
- no easy provision of screenshots in case of errors
- cumbersome concept of „verification points“ (no renaming possible, redundancy)
- renaming of symbolic component names is very laborious, manual interaction necessary
- limited integration in test management
- floating licenses for additional charge only
|
- slightly raised familiarization effort regarding handling, some icons aren’t very intuitive
- web testing not yet available (announced as separate edition at additional costs)
- user must be logged in for testing on remote computers
- limited integration in test management
|
- developer-oriented solution*
- product obviously not mature
- web testing not available
- documentation and support not on sufficient level
|
*Remark: A developer-oriented solution (mainly script-based) is not necessarily a disadvantage, but we have been looking for a product usable without advanced programming skills if possible.
Evaluation of GUI-test tools for Java Swing: QFS QF-Test, Borland SilkTest, Froglogic Squish and Marathon- November 2007, O. Krapp, M.Besemer, SBE network solutions GmbH, Heilbronn, Germany.
(Original German texts and citations are translated into English.)
