reportStatus() – How we tie Test Level to Assertions

This gets to the heart of all our assert functions. As indicated in the first code snippet (the assertTrue function), all decisions regarding how to report an event are left up to reportStatus.

Public Function assertTrue( bExpression, aOptions )
    Set oOptions = GetOpts( ARRAY ( _
        “sEvent”, “AssertTrue”, _
        “sLogMessage”, “Expected the given expression to return TRUE” _
        ), aOptions)
    ‘Return true or false
    assertTrue = reportStatus(bExpression, oOptions(“sEvent”), _
                                           oOptions(“sLogMessage”))
End Function

Note that the return value is the same “true” or “false” that the expression evaluates to. This is so you can invoke the function via the following:

If assertTrue(userMarcus.exists(NULL)) Then
    userMarcus.edit(ARRAY(“Role”, “Agile SCRUM Master”))
End If

The first lines use Will’s ultra-handy named argument parsing to set default messages for the the result reporter. You are strongly encouraged to override these with custom messages, but you don’t have to.

After that everything is very simple: reportStatus doesn’t want to know anything other than “do you consider this a pass or a fail?” So, assertTrue just passes along the expression it received, which theoretically already contains “true” or “false”. Then, it’s reportStatus’ job to decide how to report the pass/fail event to Reporter.ReportEvent.

This one function represents all the reasons we’ve stopped using QTP’s built-in checkpoints: if the test is very strict by nature, the test will fail and exit. If the test requires flexibility (because it is being tested in an unfamiliar environment or database), problems will be ignored, and the test will do everything it can to move on and get to the end. If that doesn’t make sense, go read our approach to strictness to get an idea of why we do things this way. It’s worked very well for us so far.

The Test Level definitions live in a separate Constants.vbs file, containing the following (thanks to cytoe for pointing out that I forgot to define auNoReport):

‘These are the Test Levels themselves
auReportNothing = 0
auReportNoWarnings = 1
auReportWarnings = 2
auReportFailures = 3
auFailuresAreFatal = 4‘This is used to determine whether ReportEvent should do anything
auNoReport = -1

And they’re meant to be self-explanatory. Going through the branches below, it’s a very simple decision table, based on the current Test Level Environment variable, which then uses Mercury’s built-in reporting constants.

One thing it also does is ensure that the ReporterFilter is fully enabled, then reset at the end of the function. We did this because QTP reports an awful lot of things we don’t care about in its results files, and this allows us to turn it off in other places, but guarantee that it’ll work like we expect here.

‘This handles all test results, including reporting the
‘ events (or not) to the ReportEvent method
Public Function reportStatus( bStatus, sEvent, sDetails )
    iStatus = auNoReport
sOldFilter = Reporter.Filter
Reporter.Filter = rfEnableAll
    If StringIsNotEmpty(Environment(“TestLevel”)) Then
        Select Case CInt(Environment(“TestLevel”))
            Case auReportNothing
                iStatus = auNoReport
            Case auReportNoWarnings
                iStatus = micDone
            Case auReportWarnings
                If bStatus Then
                    iStatus = micDone
                Else
                    iStatus = micWarning
                End If
            Case auReportFailures
                If bStatus Then
                    iStatus = micPass
                Else
                    iStatus = micFail
                End If
            Case auFailuresAreFatal
                If bStatus Then
                    iStatus = micPass
                Else
                    iStatus = micFail
                End If
        End Select
        If iStatus <> auNoReport Then
            Reporter.ReportEvent iStatus, sEvent, sDetails
        End If
        reportStatus = bStatus
    End If
    Reporter.Filter = rfEnableErrorsAndWarnings
End Function