Understanding properties can enable us to assess the efficacy of scenarios/cases and also yield higher efficiency.
Properties of a good scenario/test case are,
- A given scenario and associated test cases should be clear on what it is validating i.e. what entity it is testing.
- It should be clear as what type of defect it has the power to uncover. This makes the scenarios/cases purposeful.
- That the number of scenarios/cases shall be proven to complete. This can be arrived only if the intended behaviour is modeled and then scenarios generated. The number of scenarios can be arrived by combining the conditions logically and the corresponding test cases generated by combining the data values corresponding to the conditions
- That the test scenarios/cases should be staged into quality levels rather than being one to uncover a variety of defects so that these are small and purposeful.
- That in addition to staging by quality levels, it is good to group scenarios/cases by test types to enable better clarity and purposefulness.
- The number of scenarios/cases as we progress from the lowest quality level to the highest is shaped like a frustum of a pyramid. i.e. number of test scenarios/cases at the earliest quality level is much higher compared to those at higher levels.
- That the distribution of positive and negative test scenarios/cases is indeed good enough. Hmm – how do we know this? Extending point (3), negative scenarios are generated when conditions that are violated are combined while negative test cases are generated when values of data that do not satisfy the conditions are combined. Given this more conditions implies more scenarios (and also negative scenarios) while more data values implies more test cases. As one proceeds from lowest to highest quality level the distribution of -ve:-+ve skews on lower side. i.e. at higher levels, the scenarios/cases are more conformance oriented.
- That the scenarios/cases are ranked in terms of priority of the entities they test and the types of defects that they defect, to enable intelligent choosing of which to execute when faced with crunch time.
- That the scenarios when automated, run unattended and as failure of one does not not stop the execution run. It would therefore be useful to design the execution order of scenarios. i.e which scenario to execute in case the current one fails. This can be factored into automation to allow for “long runs” intelligently.