Aesthetics in Software Testing

Software testing is typically seen as yet another job to be done in the software development life cycle. It is typically seen as a clichéd activity consisting of planning, design/update of test cases, scripting and execution. Is there an element of beauty in software testing? Can we see outputs of this activity as works of art?
Any activity that we do can be seen from the viewpoints of science, engineering and art. An engineering activity typically produces utilitarian artifacts, whereas an activity done with passion and creativity produces works of art, and this goes beyond the utility value. it takes a craftsman to produce objects-de-art, while it takes a good engineer to  produce  objects with high utility value.
An object of beauty satisfies the five senses (sight, hearing, touch, smell and taste) and touches the heart whereas an object of utility satisfies the rational mind. So what are the elements of software testing that touch our heart?

Beauty in test cases
The typical view of test cases is one of utility– the ability to uncover defects, Is there beauty in test cases? Yes I believe so. The element of beauty in test cases in its architecture – “the form and structure”.
If the test cases were organized by Quality levels, sub-ordered by items (features/modules) then segregated by types of test, ranked by importance/priority, sub-divided into conformance(+) and robustness(-),  then classified by early (smoke)/late-stage evaluation, then tagged by evaluation frequency, linked by optimal execution order and finally classified by execution mode (manual/automated), we get a beautiful form and structure that not only does the job well (utility) but appeals to the sense of sight via a beautiful visualization of test cases. This is the architecture of test cases suggested by Hypothesis Based Testing (HBT).

Beauty in understanding
One of the major prerequisites and for effective testing is the understanding of the product and the end user’s expectations. Viewed from a typical utility perspective, this typically translates into understanding of various features and intended attributes. To me the aesthetics of understanding is the ability to visualize the software in terms of the internal structure, its environment and the way end users use the software. It is about ultimately distilling the complexity into a simple singularity–to get the WOW moment where suddenly everything becomes very clear. It is about building a clear and simple map of the various types of users,  the corresponding use cases and technical features, usage profile, the underlying architecture and behavior flows, the myriad internal connections and the nuances  of the deployment environment. It is about building a beautiful mental mind map of the element to be tested.

Beauty in the act of evaluation
Typically testing is seen as stimulating the software externally and making inferences of correctness from the observations. Are there possibly beautiful ways to assess correctness? Is it possible to instrument probes that will self assess the correctness? Can we create observation points that allow us to take better into the system? Viewing the act of evaluation from the aesthetic viewpoint, can possibly result in  more  creative ways to assess the correctness of behavior.

Beauty in team composition
Is there aesthetics in the team structure/composition? Viewing the team collection of interesting people – specialists, architects, problem solvers, sloggers,  firefighters, sticklers to discipline and geeks etc. allows us to see the beauty in the power of the team. It is not just about a team to get the job done,  it is about the “RUSH”  that we get about the structure that makes us feel ‘gung-ho’, ‘can-do anything’.

Beauty in reporting/metrics
As professionals, we collect various metrics  to aid in rational decision-making. This can indeed be a fairly mundane activity. What is aesthetics in this? If we can get extreme clarity on the aspects that want to observe and this allows us to make good decisions quickly, then I think this is beautiful. This involves two aspects–what we collect and how we present these. Creative visualization metaphors can make the presentation of the aspects of quality beautiful. Look at the two pictures below,  both of them represent the growth of a baby.

The one on the left shows the growth of a baby using the dreary engineering  graph,  whereas the one on the right shows the growing baby over time. Can we  similarly show to growth of our baby (the software) using creative visualization metaphors?

Beauty in test artifacts
We generate various test artifacts –  test plan, test cases, reports etc. What would make reading of these a pleasure?  Aesthetics here relates to the layout/organization, formatting, grammar, spelling, clarity, terseness. These aesthetic aspects are probably expected by the consumers of these artifacts today.

Beauty in the process
The test process is the most clinical and the boring aspect. Beauty is the last thing that comes to mind with respect to process. The aesthetic aspects as I see here is about being disciplined and creative, being detailed yet nimble. To me it is about devising a process that flexes, evolves in complete harmony with external natural environment. It is in the hard to describe these in words, it can only be seen in the mind’s eye!

Beauty in automation and test data
Finally on the aspect of test tooling,  it is about the beautiful code that we produce to test other code. The beauty here is in the simplicity of the code, ease of understanding, modifiability, architecture and cute workarounds to overcome tools/technology limitations.
Last but not the least, aesthetics in test data is about having meaningful and real-life data sets rather than gibberish.
Beauty they say, lies in the eyes of the beholder. It takes a penchant for craftsmanship driven by passion, to not just do a job, but to produce object-de-art that appeals to the senses. As in any other discipline, this is very personal. As a community, let us  go beyond the utilitarian aspects of our job and produce beautiful things.

10x reduction in post release defects – Finest experience of applying STEM

This is about one of the finest experiences that STAG had with a global chip major where the early implementation of STEM yielded significant results. Our engagement with them was to setup an effective validation practice for their porting level API for video decoders. The customer had a great technical team and were involved in both development and QA. The challenge that they faced with their complex product that involved both hardware and software and later system integration on multiple real time OS on various platforms, was that of high defect escapes i.e. post-release field defects.

We spent about a month understanding their domain and the associated technologies. Post this, a detailed analysis yielded interesting data – test cases were primarily conformance oriented, coverage of test cases was suspect, escaped defects seem to propagate from early stages and finally the process of validation was loose.

Having understood the types of defects that were being found and the post-release defects, we figured out the various types of probable defects and the various combinatorial aspects that need to be considered to form a test case. We then staged the validation as consisting of three major levels, the first one at API level, then the next one at a system level, and the last level made up of a customer-centric level that involved using reference applications.

Applying the STEM approach to test design, the test cases were developed, yielding about 6000 test cases at level one and about 800 at the subsequent levels. Also, whereas the ratio of +ve vs –ve test cases was earlier towards the +ve side, after our re-design, the ratio shifted to 60%:40% at the lower level and about 85% :15% at the higher levels. Moreover, the test cases increased in number significantly by a factor of 1000%, allowing for a larger and deeper net to catch many more serious defects. Over the next 9 months, the rate and number of defects detected increased dramatically, resulting in post-release issues reducing by a jaw-dropping 10x times.

Once we solved the test effectiveness problem and increased the yield of defects, the focus shifted to streamlining the process by setting up proper gating in the test process and creating a centralized web based test repository, and finally setting a strong defect analysis system based on Orthogonal Defect Classification (ODC) method. This enabled a strong feedback system, resulting in shifting the defect finding process to earlier stages of SDLC and thereby lowering cycle time. Complementing this, we focused on setting up a custom tooling framework for automating this non-UI based software resulting in a significant cycle time reduction – an entire cycle of tests on a platform took less than 15 hours of time!

This has been one of the finest experiences that we had with STEM, and was a clear winner for STEM implementation. This was only possible because of the very mature engineering management staff of the customer, who were focused on systemic improvement and had systematic improvement goals.

What is to be noted is that our test team was NOT a team with significant depth of experience on the particular product domain. Applying STEM at a personal level, the team was able to understand what was necessary and sufficient for effective validation and complemented the strong technical team with mature defect-oriented thinking. This was an early case study for us to establish that a STEM based approach provided us with the right thinking skills for defect finding, rather than resort to a domain centric approach to defect finding.