The Assertions module: Natural Language Assertions

Inspired by Chai.js we have implemented our own assertion API for scripting. It can be used from Groovy, JavaScript and Jython scripts.

The idea is to make the checks implemented in the scripts in QF-Test more readable and closer to the human language. Verifying and validating data when working in the Server or SUT Scripts is usually done via rc.check() or the Java keyword assert. They are fine when working with basic data types like strings. However, it can become tedious when you have to check complex data types like structured objects, e.g. created from a JSON string. This is where the QF-Test assertions API makes live a lot easier.

Here are two Groovy script examples where you can see the difference between the natural language assertions and the traditional rc.check() and assert().

def foo = 'bar'
              def beverages = [ tea: [ 'chai', 'matcha', 'oolong' ] ]

              expect(foo).to.be.a('String')
              foo.should.be.equal('bar')
              expect(foo).to.have.lengthOf(3)
              expect(beverages).to.have.property('tea').with.lengthOf(3)

Example 50.9: Groovy script with natural language assertions

def foo = 'bar'
              def beverages = [ tea: [ 'chai', 'matcha', 'oolong' ] ]

              rc.check(foo instanceof String,"")
              rc.checkEqual(foo,'bar',"")
              rc.checkEqual(foo.length(),3,"")
              assert(beverages.tea!=null)
              assert(beverages.tea.size()==3)

Example 50.10: Assertions with QF-Test check method and Java assert

The QF-Test assertions API has the interfaces Assert and expect. In Groovy scripts, also a direct chaining with should is available. expect and assert support language chains, the Assert syntax is more traditional.

The result of an assertion can be either be written to the run log as failed or successful check, or optionally as an exception. Additionally, the result of the last assertion can be retrieved as a boolean value, which can be assigned to a variable. For details please see Result handling.

The API documentation is provided in doc/javadoc/qfaa.zip. The documentation lists all the methods available for Assert. They can also be used with expect and should (in Groovy), where they are part of the language chains. Since the QF-Test assertions API is very similar to Chai.js, many of the examples on https://www.chaijs.com/api/ will also work with QF-Test. For methods available in Chai.js but not yet implemented for QF-Test please refer to "Unavailable assertions".

When working with Assert you can use autocompletion and display the documentation of the available methods by typing Assert. and then pressing Ctrl⁠+⁠Space.

For Regular expressions use the module java.util.regex.Pattern.

Note For an introduction to extending the assertion API with your own assertions, check out our blog post Extending the QF-Test Assertion API – A practical introduction.

The biggest advantage comes via the language chains. They can be used with expect() and should(). The following chainable getters are available: .to .be .been .is .that .which .and .has .have .with .at .of .same .but .does .still .also

def testObj = [
              "name": "test",
              "sub": [
              "name": 'test sub'
              ],
              "numbers": [1, 2, 3, 4],
              "hasNumbers" : true
              ];

              expect(testObj).to.be.an('Object').and.is.ok
              expect(testObj).to.have.property('sub').that.is.an('Object').and.is.ok
              expect(testObj.sub).to.have.property('name').that.is.a('String').and.to.equal('test sub')
              expect(testObj).to.have.property('numbers').that.deep.equals([1, 2, 3, 4])
              expect(testObj).to.have.property('hasNumbers', true)

Example 50.11: Language chains with expect

rc.setLocal("jsonData", """
              {
              "Actors": [
              {
              "name": "Tom Cruise",
              "age": 56,
              "Born At": "Syracuse, NY",
              "Birthdate": "July 3, 1962",
              "photo": "https://jsonformatter.org/img/tom-cruise.jpg",
              "wife": null,
              "weight": 67.5,
              "hasChildren": true,
              "hasGreyHair": false,
              "children": [
              "Suri",
              "Isabella Jane",
              "Connor"
              ]
              },
              {
              "name": "Robert Downey Jr.",
              "age": 53,
              "Born At": "New York City, NY",
              "Birthdate": "April 4, 1965",
              "photo": "https://jsonformatter.org/img/Robert-Downey-Jr.jpg",
              "wife": "Susan Downey",
              "weight": 77.1,
              "hasChildren": true,
              "hasGreyHair": false,
              "children": [
              "Indio Falconer",
              "Avri Roel",
              "Exton Elias"
              ]
              }
              ]
              }""")

              def data = rc.getJson("jsonData")
              data.Actors.should.be.a("ArrayList")
              expect(data.Actors[0]).to.be.a("LinkedHashMap")
              Assert.instanceOf(data.Actors[0], "LinkedHashMap", "Bla")
              data.Actors[0].name.should.be.a("String")
              data.Actors[0].age.should.be.a("Long")
              data.Actors[0].weight.should.be.a("Double")
              data.Actors[0].hasChildren.should.be.a("Boolean")
              rc.setGlobalJson("gData",data)

Example 50.12: Language chains with should

For the documentation of the chainable getters please refer to: https://www.chaijs.com/api/bdd.

Due to the Java implementation, some syntax in QF-Test differs from Chai.js.

As assert is a reserved word in Java and Groovy, the QF-Test Assert is spelled differently, which means the first letter is a capital "A". The same applies to the assertions TRUE, FALSE, and NULL, which have to be written all-caps.
All methods with strict* prefix use == for comparison, otherwise Java’s Object equal() is used. For a list of all strict* methods type Assert.strict in the script editor and then press Ctrl⁠+⁠Space.

Assert.test replaces assert:

Assert.test('foo' !== 'bar', 'foo is not bar')
                  Assert.test({true}, 'Closures can return true')

Example 50.13: Assert.test(...)

Some of the assertions implemented by Chai.JS can not be directly translated from Javascript to Java, and some assertions are not implemented, yet. Among these are:

Assert

isAbove(), isAtLeast(), isBelow(), isAtMost()
isNaN(), isNotNan()
isUndefined()
isFinite()
throws(), doesNotThrow()
operator()
closeTo()

Expect/Should

.to.be.above(), .to.be.least(), .to.be.below(), .to.be.most()
.to.be.NaN, .not.to.be.NaN
.to.be.undefined
.to.be.finite
.to.throw(), .to.not.throw()
.to.be.closeTo()

Result handling with Assert, expect(), should() (System)

Server (automatically forwarded to SUT) script name: OPT_PLAY_HANDLE_ASSERTION
Possible Values: VAL_PLAY_HANDLE_ASSERTION_AS_CHECK, VAL_PLAY_HANDLE_ASSERTION_WITH_EXCEPTION, VAL_PLAY_HANDLE_ASSERTION_SILENTLY

The option is used to configure return value and logging.

"Handle as check" - VAL_PLAY_HANDLE_ASSERTION_AS_CHECK
In case of failure, an error, otherwise a successful check is logged to the run log
"As Exception" - VAL_PLAY_HANDLE_ASSERTION_WITH_EXCEPTION
An assertion exception is thrown in case of a failure. Can be caught in scripts as Throwable and in Try-Catch nodes as ScriptException
"As return value" - VAL_PLAY_HANDLE_ASSERTION_SILENTLY
The assertion check will be executed, but no error will be logged or exception will be thrown. Nevertheless, the assertion will be executed and returns a result value (true or false). When using expect/should, the result can be accessed with a chained .getResult().
"Handle automatically" - VAL_PLAY_HANDLE_ASSERTION_AUTOMATICALLY (Default value)
Same as VAL_PLAY_HANDLE_ASSERTION_AS_CHECK, except within a Unit test node, where VAL_PLAY_HANDLE_ASSERTION_WITH_EXCEPTION is used to fulfill the JUnit contract.

Use the option at the beginning of the script:

rc.setOption(Options.OPT_PLAY_HANDLE_ASSERTION, Options.VAL_PLAY_HANDLE_ASSERTION_SILENTLY)
              def a = 54
              def b = 55
              def isEqual = Assert.test(a==b, "")
              if (isEqual) {
              ...
              }

Example 50.14: Silent assertion in a Groovy script

If you used fluent assertions, you have to call .getResult() to query the result:

rc.setOption(Options.OPT_PLAY_HANDLE_ASSERTION, Options.VAL_PLAY_HANDLE_ASSERTION_SILENTLY)
              a = 54
              b = 55
              isEqual = expect(a).to.equal(b).getResult()
              if isEqual.getResult():
              ...

Example 50.15: Silent assertion in a Jython script

50.8 The `Assertions` module: Natural Language Assertions

50.8.1 Motivation

50.8.2 API documentation

50.8.2.1 Language chains

50.8.2.2 Differences between the QF-Test assertions API and Chai.js

50.8.2.3 Unavailable assertions

50.8.3 Result handling

50.8 The Assertions module: Natural Language Assertions