We all design scenarios/cases to test system. What are some properties that a good set of scenarios/cases should possess? This post outlines the properties that a good test scenario/case should satisfy.
A test scenario and associated test cases prime objective is to uncover potential issues in the entity under test. In the process we would like to ascertain correctness. The properties outline this line of thinking.
- A given scenario and associated test cases should be clear on what it is validating i.e. what entity it is testing. This is common parlance is understood as “Requirements traceability”.
- It should be clear as what type of defect it has the power to uncover. This in HBT (Hypothesis Based Testing) is called “Fault traceability”. This makes the scenarios/cases purposeful.
- That the number of scenarios/cases shall be proven to complete. This is a “controversial” statement. In HBT, this property is called as “Countability” i.e. that the number of scenarios/cases are no more or no less. This can be arrived only if the intended behavior is modeled and then scenarios generated. The number of scenarios can be arrived by combining the conditions logically and the corresponding test cases generated by combining the values of the data satisfied by the conditions
- That the test scenarios/cases should be staged into quality levels rather than being one to uncover a variety of defects so that these are small and purposeful.
- That in addition to staging by quality levels, it is good to group scenarios/cases by types of tests to enable better clarity and purposefulness.
- The number of scenarios/cases as we progress from the lowest quality level to the highest is shaped like a frustum of a pyramid. i.e. number of test scenarios/cases are lowest level of quality is indeed a lot compared to the numbers at higher levels.
- That the distribution of positive and negative test scenarios/cases is indeed good enough. Hmmm – how do we know this? Extending point (3), negative scenarios are generated when conditions that are violated are combined while negative test cases are generated when values of data that do not satisfy the conditions are combined. Given this more conditions implies more scenarios (and also negative scenarios) while more data values implies more test cases. As one proceeds from lowest to highest quality level the distribution of -ve:-+ve skews on lower side. i.e. at higher levels, the scenarios/cases are more conformance oriented.
- That the scenarios/cases are ranked in terms of priority guided the entities that they test and the types of defects that they defect, to enable intelligent choosing of which to execute when faced with crunch time.
- To aid in ensuring scenarios when automated run as unattended and as long as they, it would be useful to design the execution order of scenarios. i.e which scenario to execute in case the current one fails. This can be factored into automated to allow for “long runs” intelligently.
Understanding properties can enable us to assess the efficacy of scenarios/cases and also yield higher efficiencies. In HBT this is captured in the HBT test case architecture whether nine properties allow segregation of scenarios/cases in a beautiful onion peel shell. The form and structure of this architecture enables better clarity and thus improves effectiveness & efficiency.
We would love to have your comments and if you like it, please share it with your friends.
Have a great day!