We know that the objective of good testing is to uncover those faults that have the potential to cause severe failures. The big question is “how good are the test cases to detect these faults?” i.e. How “potent” are they? Traditionally, this is seen as requiring deep domain knowledge. Further, Requirements trace-ability, code coverage are used as ‘crutches’ for assessment, sadly these are weak.
Potency is the “least number of test cases with the ability to target the specific types of defects in specific areas to ensure clean software”. We could assess potency by examining fours aspects:
- Strength of test case
- Area targeted
- Target bug – potential defect type
- Immunity
Strength of test case
Are the test cases sufficient? Assess the property of count-ability. What is this? If the behavior of an element under test is described as a set of conditions (i.e. a behavioral model) then an optimal combination of these conditions would result in ?countable scenarios i.e. irrespective of who designs, the number of scenarios is indeed the same. Instantiating each scenario with optimal combinations of inputs would result in test cases. The key element to note is that the number of test cases can be logically assessed for sufficiency. The second measure is the distribution of test cases that are conformance/robustness (+/-). This is a function of #inputs and values for each input. Note that inputs can be discerned from the conditions in the behavioral model and the values for each input mechanically generated based on the input specification. Hence this measure can allow us to ascertain if our distribution is indeed good enough. Finally segregation of test cases into quality levels enables test cases to be purposeful and enables a clear targeting of what type of defect a test case is targeting.
Area targeted
This is about examining if we are indeed covering the entire area well enough. Measuring how much external area is ?Requirements coverage/traceability whilst measuring the internal area covered is what ?Code coverage? focuses on. Note that these are the typical measures that we use. These measures describe the area that we cover, but not the types of defects we are looking for in that area.
Target bug – potential defect type
Tracing the test cases to the potential types of defects that they can uncover enables the test cases to be purposeful i.e. what issues am I interested in uncovering? So examining the properties/measures of Countability, Conformance: Robustness, Test case distribution of test cases by quality levels and Requirements/Code/Fault coverage can allow a scientific assessment of potency of test cases.
Immunity
Examining the yield of test case over time will give an indication of the ?resistance or immunity. Hence tracking the outcome of each test case ?defect yield(i.e. did they result in a defect) every time we run it is key to understanding this property.