Problem solving approaches for smart QA

Is testing bounded?

Testing seems like an activity whose outcome is definitive i.e., PASS or FAIL. Seems right? Hmmm, if expectations, specifications, intentions were clear then definitive outcomes i.e., met/not-met seems fair. What if that is not true i.e. specifications/expectations are vague/fuzzy, not stated, is not complete, or discovered/fine-tuned on the fly?

Systems are complex & therefore testing too

A software system is not a just collection of discrete parts/features but an interesting bundle of interactions with itself with other systems and environment, users in different situations/context. Is it possible to clearly list out intentions/expectations in these myriad interactions? Also, a system is continuously evolving with new features added, existing ones enhanced, issues fixed and code refactored for betterment. Hence system entropy increases continuously (2nd Law of Thermodynamics and SW engineers attempt to manage this complexity. This is what smart test folks focus on, to lower risk of failure, and in the process find binary answers(met/not-met). Testing is no more a simple problem of judging (aka checking), it is a complex problem-solving activity of analysing, understanding, questioning, hypothesising, proving and also evaluating/judging.

Problem solving approaches

What does it take to solve a complex problem, especially in the case of testing that is really unbounded? How does one do contextual analysis, review, design, evaluation, automation and assess systems built rapidly and constantly evolving? Is it by using techniques, following a process, technology?

Well, It takes a multi-faceted approach to solve complex problems:

  1. Technique/formula to solve precisely
  2. Principle(s) to make good choices
  3. Guidelines to steer in right direction
  4. Frameworks/Models to abstract well
  5. Process to do activities consistently
  6. Using prior experience or heuristics
  7. Explore, learn, course correct constantly
  8. Creativity – thinking out-of-box
  9. Luck/chance via random/ad hoc doing

Detailing these approaches

#1 TECHNIQUE/FORMULA

A technique is a particular method of doing an activity, usually a method that involves practical skills. [Collins dictionary] whilst a formula is generally a fixed pattern that is used to achieve consistent results, a recipe. [Vocabulary.com]

Bet you can relate this easily to estimate effort/time, design scenarios/test cases, measure code complexity and so on. Technique/formula is great as they help you solve any problem easily and clearly without ambiguity, all one has to do is to choose the right technique/formula.

#2 PRINCIPLE

A principle is a proposition or value that is a guide for behavior or evaluation. [Wikipedia]
Here are some of them that we use

  1. It is not testing later, but getting involved early
  2. Quality is everyone’s job
  3. Decompose to understand well
  4. Focus on the critical few (Pareto principle) and so on.
    Principles are not as precise as that technique/formula, they help you make choices.

#3 GUIDELINE(S)

A guideline is general rule, principle, or piece of advice, it gives you a sense of direction [Oxford Dictionary]. Test with real customer data before release, do validate interesting corner cases, use coverage to assess complex code, use code smells are some guidelines.

#4 EXPERIENCE/HEURISTICS

Experience is knowledge or skill gained by doing a job or activity for a long time.
[Collins English Dictionary] whilst Heuristics are strategies derived from previous experiences with similar problems. [Wikipedia]. These are deep personal knowledge gleaned from experiences and codified as patterns, anti-patterns.

#5 EXPLORATION

This is about trying to discover; learn about. [Cambridge dictionary]. This is not cookie cutter stuff, this is about walking around, exploring the software system in terms of how it works, how it is constructed, how data flows around. how users interact, how it interacts with environment, how I can use/abuse it and so on. It is staying curious, to question, to experiment to understand, to know what you still do not know. This is very useful in all the phases of test lifecycle commencing from understanding to evaluation.

#6 CREATIVITY

This is approaching a problem or challenge from a new perspective, alternative angle, or with an atypical mindset. [sessionlab.com]. This approach by the very word is unbounded, not done by following a strict method/process. We all have done “out-of-box” thinking, which is really creative thinking. Of not doing anything strictly logical or from first principles, but seemingly inspired by solutions from other domains.

#7 RANDOM/AD-HOC

Random is choosing or happening without a method or pattern.[Macmillan Dictionary] whist Ad hoc is not planned in advance, done only because a particular situation has made it necessary. [Collins English Dictionary]. This approach is the other extreme, of relying on lady luck, or chance to solve problems. Chaos, randomness is a powerful tool especially in evaluating software as we tend to be too organised in how we do this, what data sets we use and so on. Stimulating variant usage is also involving more people to evaluate software which we know is very useful for alpha/beta test.

In which test lifecycle activities are they applied in?

The test lifecycle consists of activities commencing from understanding intents/expectations to finally assessing the system.

UnderstandUnderstand needs & expectations – who, what (features, requirements, flows, components), what-for (criteria), how much, linkages, environment, stage (new, fix, enhancement), where
Strategy/planApproach – Test what (entities) & what-for (test types), how (test techniques & tools), how well (metrics for progress, quality)
DesignScenarios, cases and data sets
AutomationScripts
EvaluationEvaluating, enhancing scenarios/cases, learning, exploring
AssessmentJudging progress, quality, fitness

So how we perform each of these activities in the lifecycle? We use a judicious mix of techniques, principles, guidelines, models, heuristics, woven together by a process that would also encompass exploration, creativity and doing random/ad-hoc in each of these test lifecycle activities.

In conclusion…

It takes Smart QA to tackle the unbounded, lower the entropy and contain the risk. One of the facets of Smart QA is about understanding various problem-solving approaches and judiciously using a meaningful mix for the context.

In closing here are two questions for you:
Q1 – How do YOU test complex systems rapidly?
Q2 – What problem solving approaches do YOU USE in your testing life cycle?
Mull over it and relate it to the seven approaches to problem solving outlined above.

Blog4Pic-LandingPage-300x300_cae48d93120602fed5bae9c7bd64da81

Smart checklists make you think

The power of checklists in other disciplines

In other mature disciplines like medicine, aviation, construction where the impact of simple defects is enormous, it is smart checklists that have come to the rescue and saved millions of dollars and saved many lives.

The Checklist Manifesto  by Atul Gawande, a surgeon, extols the power of checklist. He states that any problem can be categorised into simple, complex and complicated, and how the smart checklists can solve these.

‘Simple’ problems are those that are individualistic in nature with a set of stuff to be done, while ‘Complicated’ problems implies multiple teams/people coordination/timing issues, and ‘Complex’ problem is where outcomes are different despite same application.

Checklists are not mindless compliance

Smart checklists are not about mindless compliance, not about ticking boxes, it is really about tickling the brain to think better and ensure fault-proofing rapidly. In our industry, checklists have been seen a cheap brainless activity that is about ticking the checkboxes, and therefore presumed to be useless.

Smart checklists make you think

To solve complex problems, “push the power of decision making from a central authority to the periphery, allowing people to make decision and take responsibility”  (Atul Gawande).

The checklist needs to allow for judgement to be used in the tasks rather than enforce compliance, so that actions may be taken responsibly. Hence, a smart checklist is about enabling your thinking process by having:

  • Set of checks to ensure stupid but critical stuff is not overlooked
  • Set of checks to ensure coordination
  • To enable responsible actions to be taken without having to ask for authority.

Good checklists are precise and easy to use in most difficult situations, it does not spell out everything, but provides reminders of critical & important steps that even highly skilled professionals would miss.

Checklists seem to defend everyone, a kind cognitive net designed to catch flaws of memory and attention, a well designed checklist enables one to DO well, SYNC with others well, and take appropriate ACTions as needed. [5]

“The power of checklists is limited, they help experts remember how to manage a complex process or machine, make priorities clearer and prompt people to function well as a team.  By themselves, however, checklists cannot make anyone follow them.” [Boorman, Boeing]

We use checklists too, but are we effective?

Do we use checklists for early tests like DevTest? Yes, we do. But from what I have seen of numerous checklists like code review/UI checklists , it is more often is used like a compliance assessment, of ticking away a long list of items-to-check for. So this turns to be a mindless job and suffers poor implementation, as it is most ill suited for smart validation.

Smart checklist enables developing clean code

So should we really do dev test? Should we not become sensitive and write cleaner code? Well lean thinking (aka Agile) is of producing less bugs in the first place, not about testing more.

A ‘Smart DevChecklist’ enables one  to precisely accomplish this, to become more sensitive and written code that certainly does L1 through L4 issues. Remember, this is the most cost effective method to good quality. Well, a Smart DevChecklist is an efficient complement to dev test enabling an easy and efficient method to producing great code.



If you are keen to try out a SmartDevChecklist to rapidly do DevTest without the pain of dev test, it is available as part of the e-book listed alongside.


Blog3Pic-LandingPage-300x300_4198bb26033f2a065209727863a822f7

LEAN: It is not doing more, it is about doing less

So, what should be really happening in Agile context?

Lean thinking is what inspired the Agile movement. Lean is about not producing waste in the first place. It is doing things ‘clean’ at at the first place,  so that waste is ideally not there. Waste in software context are bugs. And in the early stages there are ‘unit bugs’. Since our focus in Agile is to find these earlier and to ensure that they are never there whenever we modify, we resort to a high degree of automation. Therefore we have a large body of test cases at the lower levels automated to ensure that we can continually execute them. This is great, but should we not focus on adopting a practice that in essence prevents these issues and lessen the need for large number of unit tests to uncover these?

It is not about doing more, it is about doing less

When we find issues in the product/app especially those that can be caught earlier, we focus on more rigorous dev test with extreme focus on automation. Yes, that seems logical. But what a minute, for a developer already busy writing code, is this the right approach? Given that dev test is largely about issues in L1 thru L4, could we not focus on getting this right or statically assess these via smart checklist?

Great quality early stage code is not about doing more testing, it really is doing about doing less test, by enabling sharper focus on ‘what-can-go-wrong’, ‘have-you-considered-this’.

The ebook outlines in detail outlines how to purposefully do DevTest in the “LEANest” by clearly outlining what issues a dev has to go after and outlines SmartDevChecklist to do this the most LEAN way.

SaveSave

Blog2Pic-LandingPage-300x300_9fcc8a072405cb52be657682f9abeb86

What is expected of code quality from developers?

Unit testing? Dev testing is a better term

The definition of what an unit is most often unclear and therefore unit testing more misunderstood. Martin Fowler in his blog Unit Test  states “It’s very ill-defined, and I see confusion can often occur when people think that it’s more tightly defined than it actually is “.

So let us use the term “Dev test” to state the early validation by developers during the development of code. What is expected of the code quality from developers? What should dev test uncover?  To setup a clear objective of what dev test should accomplish (objective), let us take a goal focused approach of as to what types of defects should be  uncovered at dev test.

“Quality Levels” – Defect types matter

Purposeful testing is about hypothesising potential type of defects and going after them.

Of course, as we one engages in this activity in the scientific manner, we revise and fine tune what to go after and how to go after.

Characterising the system under test as containing a “mixture” of various defect types, and inspired by fractional distillation as an efficient method of separation, Hypothesis-Based Immersive Session Testing (HyBIST)  sets up NINE quality levels of which the first FOUR are certain candidates for dev testing.

So what are the quality levels and  what is the purpose of tests at those levels?

Dev testing – What is the objective?

The objective of dev testing is that the building blocks of a system i.e. structural components are indeed clean enough to deliver the basic feature and worthy of integration to form the larger system. This means that a building block is internally   robust (structurally clean) and behaviourally correct (externally clean).

HBT Quality Levels

L1: Ensure bad inputs are rejected, about data types, values, boundaries
L2: Ensure that input interface is clean, about syntax(format), order, messages
L3: Ensure that internal structure is robust, about exception, resource handling, timing/sync, loops/iteration
L4: Ensure behaviour of feature is fine, about the business logic of the base technical feature
L5: Ensure behaviour of requirement/flow is fine, about the business logic of the requirement/flow that is a collection of technical features
L6: Ensure system works well is all different external environments and does not affect the environment
L7: Ensure that  key requirement/flow satisfies the expected attributes (e.g performance, volume…)
L8: Ensure that  final system deploys well in the real environment. Installation, configuration, migration are of interest here.
L9: Done by end users, the intention is ensure that it satisfies the actual end user’s intended needs and expectations.

So, what levels matter to the developer?

Quality levels denote a progression of quality as software is built. So as a developer, it is necessary to validate that basic behaviour of the building blocks that we build/modify. This means that a developer performs tests to ensure Levels 1-4 are satisfied, implying that the component is clean enough to integrate into the larger system. Additionally, if the component is critical to certain system attributes, then higher level tests beyond L5 could be useful to do.

DevTest need not be hard or painful, we have made it so. Oh, just automating unit tests does not help, well it is additional work, is it always needed? . The e-book below outlines a practical implementation of this and gives you a SmartDevChecklist as an aid to very rapidly perform DevTest. 

SaveSave

Poor quality code is due to compromised unit testing

Poor quality code is due to compromised “Unit Testing”

The problem at large

The quality of early stage code is felt to be a concern by many engineering managers that I talk to. In my numerous consulting assignments, I have noticed that many issues found by QA folks seem to be ones that do not require the expertise of a specialist tester, compromising the effectiveness of ‘system test’  resulting in avoidable  customer issues.

Great quality code is not the result of intense system testing, it is result of well structured filtration of issues from the early stages. A compromised ‘unit test’ puts unnecessary strain on the QA folks who seem to be compelled to go after these issues at the expense of system test.

Developers on the other hand do not deliberately write bad code, it is just that accidents happen. Accidents seem to happen due to brute force push of unit testing without it being simple and practical, and developers already short of time are unable to  adhere to a heavyweight process. The other fallacy seems to the over dependence on automated unit tests as the saviour without paying attention to test cases. Also the incorrect notion of unit testing as being only white-box oriented with a skew towards code coverage results in ineffective tests that are introverted. Lastly the sheer emphasis of dynamic testing as the only method to uncover defects is possibly overwhelming, when easier static methods of uncovering issues could have been employed.

Business/economic impact

The impact of leakage of issues from early stage is not irritating, but serious enough. Issues reported by customers that are early stage simple issue like poor validation of inputs, results in significant drop of confidence in the customer. The QA folks focus on these issues result in their job of system validation being poor, resulting in field issues related to end-to-end flows  and sometimes attributes  being compromised.

Also the incorrect focus of a specialist QA results in insufficient time for doing things that can make system test more effective and efficient like automation of end-to-end flows, focus on non-functional requirements, revising the test strategy/approach and sharpening with knowledge gained every cycle.

Yes, this age old problem is boring. Don’t force your developers to more unit testing to solve the problem. Ask them to do it smartly by doing less. If you are keen to know how, check out the e-book listed below.

SaveSave

Regression

Is regression hindering your progression?

“It took a few hours to incorporate the change but took a few days to test, upsetting the customer. Why? Well, we found out that our QA team was doing too much regression. Wish we could be smarter” – Engineering Manager of a mid-sized IT company.

Have you ever felt this way? Have you wished if you do less regression to release faster?

In the current world of rapid development, software is constantly updated with new features, incremental additions and bug fixes. While features (new & incremental) are the focus for revenue generation and market expansion, bug fixes are necessary to ensure that customers stay.

While on the path of progression towards revenue enhancement, the challenge is “Did I break any existing features that are working well”? That may necessitate a regression test.

Note that as the product grows, so does regression, increasing cost and slowing down releases.

Regress means ‘go backwards’, and in this context, it means ‘checkout prior quality risks to ensure that they still are under control’. This implies that the product is retested from both functionalities and attributes aspects to ensure that functionalities and attributes like performance, security etc. are not compromised.

So, how can one regress smartly?

* Figure out how much not to regress by doing a smarter impact analysis using a scientific approach to understand fault propagation due to change.
* Figure out how much not to regress by analysing defect yields over time to understand what parts of the system have been hardened
* Well, automation is an obvious choice, ensure that the scenarios are “fit enough for automation” so that you don’t end spend much effort maintaining the scripts to be in sync with every change.

Change as we all know is very imminent, and does cause a domino effect. The smartness lies in, validating only those that have the potential for domino effect thereby doing less and exploiting automation to do faster.

Here is the link to TWO aids that can enable your QA team to regress smartly. Oh, ask your QA team to read this article before they use the tool.

tools for smart regression

Ideas to regress smartly

Ideas to regress smartly

The context

In the current world of rapid development, a software is constantly updated with new features, incremental additions and bug fixes. While features (new & incremental) are the focus for revenue generation and market expansion, bug fixes are necessary to ensure that customers stay.

While on the path of progression towards revenue enhancement, the challenge that is “Did I break any existing features that are working well”? That may necessitate a regression test.

Note that as the product grows, so does regression, increasing cost and slowing down releases.

Regression

Regress means ‘go backwards’, in this context this means ‘checkout prior quality risks to ensure that they are still under control’.  The product is retested from both functionalities and attributes aspects to ensure that functionalities and attributes like performance, security etc. are not compromised.

But, how do we tackle this?
Given the necessity of ensuring that the functionality and attributes are not compromised, we have to retest the functional/non-functional aspects constantly resulting in repetitive testing.

To do this well, we typically adopt:
1. Massive regression test automation to re-test thoroughly.
2. Deep product knowledge to assess the potential impact of changes and do focused regression.

So, what is the challenge?

1. Well, automation is great, but it requires continual investment to build and maintain.
2. In-depth product knowledge is also limited to a few people, and there are always in high demand!

Hmmm, how can we do better?
Instead of focusing only on how to do more and faster, could we do it less in a  smarter way? Let us ask some questions to figure this out:

1. Are you doing too much regression?

Could we do a smarter impact analysis? Could there be a logical approach to analysing change impacts without only relying on deep product knowledge? Yes, one of HyBIST’s technique “Fault propagation analysis” could be useful here. The technique, in a nutshell, states “Given that an entity has been modified and is linked to other entities, what types of defects can indeed propagate and affect the linked entities?”

2. Is your defect yield from regression good enough?

Software with time hardens, i.e. becomes fit. This implies that the same test cases executed yield less defects later, i.e. test case yield drops. So the lingering question is “should be executing these at all?”. Just like living beings who develop resistance to certain diseases over time, the software also can be thought to become ‘resistant to test cases’ with time. In HyBIST, we call this ‘Test case immunity’  and use this to logically ascertain test cases that may be dropped and therefore do less.

3. Are your test scenarios fit enough for automation?

If the software is volatile, automation is more volatile! Changes to software necessitate automation to be in sync. So to ensure rapid modification, frameworks are used. That is great, but do you know that structure, i.e. architecture of test cases also matter? It is not just about frameworks and great code. But it’s about how well the test cases are organised. In HyBIST this is done by using a technique “Levelisation analysis” that ascertains if the test cases are organised into well-formed levels enabling rapid automation with rapid modifiability.

In closing : SMART REGRESSION

In summary, the three questions were all about “How can we do less to do more”?  Do less regression. Do less automation maintenance. And therefore perform smart regression to progress further.

Smart regression complements the act of doing faster via automation by enabling one to do lesser.

SaveSave

SaveSave

SaveSave

design checklist-fi

Design checklists to “Do, Sync & Act”

This article is the second one on checklists, the first one being “The power of checklist

Checklists seem to defend everyone, a kind cognitive net designed to catch flaws of memory and attention.

Atul Gawande in his book “ The Checklist Manifesto” states that there are three kinds of problems in the world:

  1. SIMPLE: Individualistic in nature, solvable by application of simple techniques.e.g. Bake a cake.
  2. COMPLICATED: Can be broken into a set of simple problems, requires multiple people, often multiple specialized teams. Timing, coordination becomes a serious concern. e.g. Launching a rocket.
  3. COMPLEX: These are problems where the solution applied to two similar problems may not result in the same outcomes. e.g. Raising a child. Expertise is valuable, but most certainly not sufficient.

He continues on to say that checklists can provide protection against elementary errors in the case of simple problems that we are designed with. This can be accomplished by simple activity task checklist.

In the case of complex problems that require multiple specialists to coordinate and be in sync, a simple activity task checklist won’t do, what is needed is a checklist with communication tasks to ensure that experts discuss on the matter jointly and take appropriate action.

“Man is fallible, but maybe men are less so” Belief in the wisdom of the group, the wisdom that multiple parts of eyes are on the problem and letting watchers decide what to do.

So how can checklists help in solving simple/complex problems? Using simple activity task checklists to ensure simple steps are not missed or skipped and checklist with communication tasks to ensure that everyone talks though and resolves hard and unexpected problems.

Building tall buildings is a complex problem and the success rate of the construction industry’s checklist process has been astonishing. Building failure is less than 0.00002%, where building failure is defined as the partial or full collapse of a functioning structure. (from a sample size of a few million buildings!).

Now let us turn our attention to the complex problem. How does one deal with complex, non-routine problems that are fundamentally difficult, potentially dangerous and unanticipated? In these situations, the knowledge required exceeds that of any individual and unpredictability reigns. To solve these it is necessary to push the power of decision making from a central authority to the periphery, allowing people to make the decision and take responsibility.

So the checklist needs to allow for judgment to be used in the tasks rather than enforce compliance, so that actions may be taken responsibly. This needs to have set of checks to ensure stupid but critical stuff is not overlooked, a set of checks to ensure coordination and enable responsible actions to be taken without having to ask for authority. There must be room for judgment, but judgment aided and even enhanced by a procedure. Note that in COMPLEX situations, checklists not only help, they are *required* for success.

So, how does one make checklists that work? Aviation industry thrives on checklists in normal times and also to tackle emergencies. The advice from Boorman from the “The Checklist Factory” of Boeing for making checklists that work are:

  1. Good checklists are precise, easy to use in the most difficult situations. They do not spell out everything, they only provide reminders of critical and important steps, the ones that even the highly skilled professionals would miss.
  2. Bad checklists are too long, hard to use; they are impractical. They treat people as dumb and try to spell out every step. So they turn people’s brain off rather than turning them on.

“The power of checklists is limited, they help experts remember how to manage a complex process or machine, make priorities clearer and prompt people to function well as a team. By themselves, however, checklists cannot make anyone follow them.” (Boorman, Boeing)

So how should a good checklist look like?

  1. Keep the length of checklist between five to nine items, the limit of human memory.
  2. Decide on whether you need a DO-CONFIRM checklist or READ-DO checklist. With a DO-CONFIRM checklist, individuals perform jobs from memory and experience and pause to CONFIRM that everything that was supposed to be done was done. With the READ-DO checklist, individuals carry out the tasks as they tick them off, it is like a recipe. So choose the right type of checklist for the situation. DO-CONFIRM gives a people a greater flexibility in performing the tasks while nonetheless having to stop and confirm at key points.
  3. Define clear pause points at which a checklist should be used
  4. The look of checklist matters, they should be free of clutter and fit into a page

In summary

  • Ticking boxes is not the ultimate goal
  • Checklist is not a formula, it enables one to be smart as possible
  • It improves outcomes with no increase in skill
  • Checklists aid efficient working

As smart individuals, we don’t like checklists. It somehow feels beneath us to use a checklist, an embarrassment. The fear is that checklists are about mindless adherence to protocol.

Hey, the checklist should get the dumb stuff out the way so as to let you focus on the hard stuff. We are all plagued by failures – by missed subtleties, overlooked knowledge, and outright errors. Just working harder won’t cut. Accept the fallibilities. Recognise the simplicity and power of the checklist.

Try a checklist. It works.


If you find the article interesting, please ‘like’, ‘share’ or leave a comment below.

poc

The power of checklist

Recently I read the book “The Checklist Manifesto” by Atul Gawande. 

“An essential primer on complexity in medicine” is what New York Times states about his book whilst The Hindu states this as “An unusual exploration of the power of to-do list”.

As an individual committed to perfection, in constant search of scientific and smart ways to test/prevent and as an architect of Hypothesis Based Testing, I was spellbound reading this brilliantly written book that made the lowly checklist the kingpin, to tackle complexity and establish a standard for higher baseline performance.

The problem of extreme complexity The field of medicine has become the art of managing extreme complexity. It is a test whether such complexity can be humanly mastered: 13000+ diseases, syndromes and types of injury (13000 ways a body can fail), 6000 drugs, 4000 medicines and surgical procedures each with different requirements, risks and considerations. Phew, a lot to get right.

So what has been done to handle this? Split up knowledge into various specializations, in fact, we have super specialization today. But it is not just the breadth and quantity of knowledge that has made medicine complicated, it is also the execution of these. In an ICU, an average patient required 178 individual interactions per day!

So to save a desperately sick patient it is necessary to: (1) Get the knowledge right (2) Do the 178 daily tasks right.

Let us look at some facts: 50M operations/year, 150K deaths following surgery/year (this is 3x #road-fatalities), at least half of these avoidable. Knowledge exists in supremely specialized doctors, but mistakes occur.

So what do you when specialists fail? Well, the answer to this comes from an unexpected source, nothing to do with medicine.

The answer is: THE CHECKLIST

On Oct 30, 1985, a massive plane that carries 5x more bombs roared and lifted off from the airport in Dayton, Ohio and then crashed. The reason cited was “Pilot error”. A newspaper reported, “this was too much airplane for one man to fly”. Boeing the maker of this plane nearly went bankrupt.

So, how did they fix this issue? By creating a pilot’s checklist, as flying a new plane was too complicated to be left to the memory of any one person, however expert. The result: 1.8 million miles without one accident!

In a complex environment, experts are against two main difficulties: (1) Fallibility of human memory, especially when it comes to mundane/routine matters which are easily overlooked when you are strained to look at other pressing matters of hand (2) Skipping steps even when you remember them, as we know that certain steps in a complex process don’t matter.

Checklists seem to provide against such failures and instill a kind of discipline of higher performance.

Peter Provonost in 2001 decided to give a doctor’s checklist a try to tackle central line infections in ICU. So what was the result after one year of usage? Checklist prevented 43 infections and 8 deaths and saved USD 2M! In another experiment, it was noticed that patients not receiving recommended care dipped from 70% to 4% and pneumonia fell by a quarter and 21 fewer parents died.

In a bigger implementation titled the “Keystone Initiative” (2006) involving more hospitals of 18-month duration, the results were stunning- USD 17M saved, 1500+ lives saved!

ALL BECAUSE OF A STUPID CHECKLIST!

So where am I heading? As a Test Practitioner, I am always amazed at how we behave like cowboys and miss simple issues causing great consternation to the customer and users. Here again, it is not about lack of knowledge, it is more often about carelessness. Some of the issues are so silly, that they can be prevented by the developer while coding, and certainly does not demand to test by a professional. This is where a checklist turns out to be very useful.

In an engagement with a product company, I noticed that one of the products has a product backlog of ~1000 issues found both internally and by the customer. Doing HyBIST level-wise analysis, we found that ~50% of the issues could have been caught/prevented by the developer preventing the vicious cycle of fix and re-test. A simple checklist used in a disciplined manner can fix this.

So how did the checklists help in the field of medicine or aviation? They helped in memory recall of clearly set out minimum necessary steps of the process. They established a standard for higher baseline performance.

Yes! HIGHER BASELINE PERFORMANCE. Yes, this is what a STUPID CHECKLIST CAN DO!

So how can test practitioners become smarter to deliver more with less? One way is to instill discipline and deliver baseline performance. I am sure we all use some checklist or other but still find results a little short.

So how can I make an effective checklist and see a higher performance ? Especially in this rapid Agile Software world?

This will be the focus of my second part of this article to follow. Checklists can be used in many areas of software testing, I will focus in my next article on ‘How to prevent simple issues that plague developers making the tester a sacrificial goat for customer ire by using a simple “shall we say unit testing checklist”.

Related article: Design checklists to “Do, Sync & Act”


If you find the article interesting, please ‘like’, ‘share’ or leave a comment below.

as

Frictionless development testing

Very often in discussions with senior technical folks, the topic of developer testing and early stage quality pops up. And it is always about ‘we do not do good enough developer testing’ and how it has increased post release support. And they are keen on knowing ‘how to make developers test better and diligently’ and outlining their solution approach via automation and stricter process. The philosophy is always about “more early testing” which typically has been harder to implement.

Should we really test more? Well it is necessary to dig into basics now. Let me share my view as to what they probably mean by testing. My understanding is that they see testing as dynamic evaluation to ascertain correctness. To come up with test cases that will be executed using a tool or a human and checking correctness by examining the results. And therefore good developer testing is always about designing test cases and executing them.

And that is where the problem is. Already under immense time pressure, the developer faces serious time crunch to design test cases, execute (possible after automating them). In the case when it does happen, they all pass ! (Not that you would know if they fail!). And the reason that I have observed for the ‘high pass rate’ is that test cases are most often conformance oriented. When non-conforming data hits the system, Oops happens!

So should we continue to test harder? What if we changed our views? (1) That testing need not be limited to dynamic evaluation, but could also done by via static proving. That is, ascertaining correctness not only via execution of test cases but by thinking through what can happen with the data sets. (2) That instead of commencing evaluation with conformance test cases, we start in the reverse with non-conforming data sets first. Prove that the system rejects bad inputs before we evaluate for conformance correctness. (3)That instead of designing test cases for every entity, we use a potential defect type (PDT) catalog as a base to check for non-conformances first. Using PDT catalog as the base for non-conformance check preferably via static proving and devising entity specific positive data sets for conformance correctness.

So how do these views shift us to do better developer testing at an early stage? Well, the biggest shift is about doing less by being friction-less. To enable smooth evaluation by using PDT catalog to reduce design effort, applying static proving to think better and reduce/prevent defects rather that executing rotely, and finally focusing on issues (i.e PDTs) first complementing the typical ‘constructive mentality’ that we as developers have. Rather than do more with stricter process, let us loosen and simplify, to enable ‘friction-less evaluation’.

Think & prove vs Execute & evaluate

Picking up a PDT from the catalog and applying a mental model of the entity’s behaviour can enable us to rapidly find potential holes in implementation. To enable easy implementation of this idea, let us group the PDTs in into three levels. The first one deals with incorrect inputs only, while the second one deals with incorrect ways to accept these inputs while the last set deals with potential incorrect internal aspects related to code structure and external environment. Let the act of proving robustness to deal with non-conformances proceed from level 1 though 3 commencing by thinking through (1) what may happen when incorrect inputs are injected (2) how does interface handle incorrect order/relationship of these inputs and finally (3) how entity handles (incorrect)internal aspects of structure like resource allocation, exception handling, multi-way exits, timing/synchronisation or misconfigured/starved external environment.

Non-conformance first

Recently a senior executive was stating that his organisation’s policy for developer testing was based on ‘minimal acceptance’ i.e. ascertain if the entity worked with right inputs. As a result the test cases were more ‘positive’ and would pass. Post release was a pain, as failure due to basic non-conforming inputs would make the customer very irritated. And the reason cited for the ‘minimal acceptance criteria’ was the lack of time to test the corner cases. Here the evaluation was primarily done dynamically i.e executing test cases. When we get into the ‘Think & Prove’ mode, it makes far better sense to commence with thinking how the entity will handle non-conformance by looking at each error injection and potential fault propagation. As a developer, we are familiar with the code implementation and therefore running the mental model with a PDT is far easier. This provides a good balance to code construction.

PDTs instead of test cases

Commencing with non-conformance is best done by using patterns of non-conformance and this is what a PDT is all about. It is not an exact instantiation of incorrect values be it at any of the levels (1-3), it is rather a set of values satisfying a condition violation. This kind of thinking lends to generalisation and therefore simplifies test design reducing friction and optimising time.

To summarise, the goal was to enable to build high quality early stage entity code and we approached this by being ‘friction-less’. By changing our views and doing less. By static evaluation rather than resort to only dynamic evaluation. By focusing on robustness first and then conformance. By using PDT catalog rather than specific test cases.

Once the entity under development has gone through levels 1-3 quickly, it is necessary to come up specific conformance test cases and then dynamically evaluate them if the entity is non-trivial. If the entity under development is not a new one, but one that is being modified, then think through the interactions with other entities and how this may enable propagation of PDTs first before regressing.

So if you want to improve early stage quality, smoothen the surface for developer testing. Make it friction-less. Do less and let the entities shine. It is not doing more testing, it is about being more sensitised and doing less. Let the evaluation by a developer weave naturally and not be another burdensome task.

What are your views on this?