Automatic and manual tests

Whether we like it or not, some tests will not be automated.

Instead of separating manual from automatic test specifications - the latter in a well specified machine readable format, the former in specification documents - let all tests be specified in the machine readable form (as talked about here).

Scenarios for manual tests can be thought as checklists, where the tester is prompted by the tool to perform an action and record it’s result. In 99% of the cases a question answered with a yes/no suffices to evaluate test success.

In our contrived example (first talked about here), say that a test asks for a measurement using an oscilloscope, and that it has been deemed impractical (or too expensive) to have a programmable oscilloscope. Instead, a snapshot of the expected curve is taken and the tester should perform the measurement and make sure it matches. It could go something like:

<scenario attended="true">
  <echo text="Please perform oscilloscope measurement."/>
  <display img="expected.jpg"/>
  <yesno>Does the measurement match the expected result?</yesno>
</scenario>

The advantages of such an approach is that the tester focuses only on the relevant test action that cannot be automated. The rest (test preparation etc.) is done automatically by the tool. The tool also takes care of validating the tester’s input, logging etc. This consistency reduces the possibility of errors.

Ofcourse it also simplifies management of test specifications using the same facilities used for automated tests.

Test management, test execution

Up to now we have been discussing the need for machine readable test specifications with a sample idea using XML and including manual as well as automated tests.

So there is a pile of test specifications in the desired format. From these specifications information can be extracted to:

• Create overview documents and lists with information about the test cases (test management)
• Execute tests and store the results (test execution)

The core problem is the existence (or lack of) a system that automates test execution in the sense that it integrates various tools that automate tests. Something like this would be very useful in a CI environment.

So lets present a system to automate test execution using a text (XML) based database of test specifications. You just have to wait for the next installment!