Smart checklists make you think

The power of checklists in other disciplines
In other mature disciplines like medicine, aviation, construction where the impact of simple defects is enormous, it is smart checklists that have come to the rescue and saved millions of dollars and saved many lives.

The Checklist Manifesto  by Atul Gawande, a surgeon, extols the power of checklist. He states that any problem can be categorised into simple, complex and complicated, and how the smart checklists can solve these.

‘Simple’ problems are those that are individualistic in nature with a set of stuff to be done, while ‘Complicated’ problems implies multiple teams/people coordination/timing issues, and ‘Complex’ problem is where outcomes are different despite same application.

Checklists are not mindless compliance
Smart checklists are not about mindless compliance, not about ticking boxes, it is really about tickling the brain to think better and ensure fault-proofing rapidly. In our industry, checklists have been seen a cheap brainless activity that is about ticking the checkboxes, and therefore presumed to be useless.

Smart checklists make you think
To solve complex problems, “push the power of decision making from a central authority to the periphery, allowing people to make decision and take responsibility”  (Atul Gawande).

The checklist needs to allow for judgement to be used in the tasks rather than enforce compliance, so that actions may be taken responsibly. Hence, a smart checklist is about enabling your thinking process by having:

  • Set of checks to ensure stupid but critical stuff is not overlooked
  • Set of checks to ensure coordination
  • To enable responsible actions to be taken without having to ask for authority.

Good checklists are precise and easy to use in most difficult situations, it does not spell out everything, but provides reminders of critical & important steps that even highly skilled professionals would miss.

Checklists seem to defend everyone, a kind cognitive net designed to catch flaws of memory and attention, a well designed checklist enables one to DO well, SYNC with others well, and take appropriate ACTions as needed. [5]

“The power of checklists is limited, they help experts remember how to manage a complex process or machine, make priorities clearer and prompt people to function well as a team.  By themselves, however, checklists cannot make anyone follow them.” [Boorman, Boeing]

We use checklists too, but are we effective?
Do we use checklists for early tests like DevTest? Yes, we do. But from what I have seen of numerous checklists like code review/UI checklists , it is more often is used like a compliance assessment, of ticking away a long list of items-to-check for. So this turns to be a mindless job and suffers poor implementation, as it is most ill suited for smart validation.

Smart checklist enables developing clean code
So should we really do dev test? Should we not become sensitive and write cleaner code? Well lean thinking (aka Agile) is of producing less bugs in the first place, not about testing more.

A ‘Smart DevChecklist’ enables one  to precisely accomplish this, to become more sensitive and written code that certainly does L1 through L4 issues. Remember, this is the most cost effective method to good quality. Well, a Smart DevChecklist is an efficient complement to dev test enabling an easy and efficient method to producing great code.

If you are keen to try out a SmartDevChecklist to rapidly do DevTest without the pain of dev test, it is available as part of the e-book listed alongside.


What is expected of code quality from developers?

Unit testing? Dev testing is a better term
The definition of what an unit is most often unclear and therefore unit testing more misunderstood. Martin Fowler in his blog Unit Test  states “It’s very ill-defined, and I see confusion can often occur when people think that it’s more tightly defined than it actually is “.

So let us use the term “Dev test” to state the early validation by developers during the development of code. What is expected of the code quality from developers? What should dev test uncover?  To setup a clear objective of what dev test should accomplish (objective), let us take a goal focused approach of as to what types of defects should be  uncovered at dev test.

“Quality Levels” – Defect types matter

Purposeful testing is about hypothesising potential type of defects and going after them.

Of course, as we one engages in this activity in the scientific manner, we revise and fine tune what to go after and how to go after.

Characterising the system under test as containing a “mixture” of various defect types, and inspired by fractional distillation as an efficient method of separation, Hypothesis Based Testing (HBT)  sets up NINE quality levels of which the first FOUR are certain candidates for dev testing.

So what are the quality levels and  what is the purpose of tests at those levels?

Dev testing – What is the objective?
The objective of dev testing is that the building blocks of a system i.e. structural components are indeed clean enough to deliver the basic feature and worthy of integration to form the larger system. This means that a building block is internally   robust (structurally clean) and behaviourally correct (externally clean).

HBT Quality Levels

L1: Ensure bad inputs are rejected, about data types, values, boundaries
L2: Ensure that input interface is clean, about syntax(format), order, messages
L3: Ensure that internal structure is robust, about exception, resource handling, timing/sync, loops/iteration
L4: Ensure behaviour of feature is fine, about the business logic of the base technical feature
L5: Ensure behaviour of requirement/flow is fine, about the business logic of the requirement/flow that is a collection of technical features
L6: Ensure system works well is all different external environments and does not affect the environment
L7: Ensure that  key requirement/flow satisfies the expected attributes (e.g performance, volume…)
L8: Ensure that  final system deploys well in the real environment. Installation, configuration, migration are of interest here.
L9: Done by end users, the intention is ensure that it satisfies the actual end user’s intended needs and expectations.

So, what levels matter to the developer?
Quality levels denote a progression of quality as software is built. So as a developer, it is necessary to validate that basic behaviour of the building blocks that we build/modify. This means that a developer performs tests to ensure Levels 1-4 are satisfied, implying that the component is clean enough to integrate into the larger system. Additionally, if the component is critical to certain system attributes, then higher level tests beyond L5 could be useful to do.

DevTest need not be hard or painful, we have made it so. Oh, just automating unit tests does not help, well it is additional work, is it always needed? . The e-book below outlines a practical implementation of this and gives you a SmartDevChecklist as an aid to very rapidly perform DevTest. 


Smart regression testing

FOUR key inferences from regression test survey

We did a quick survey on regression testing as a runup to our webinar “Is regression hindering your progression” consisting of FOUR questions listed below.

1.What fraction of total testing effort does regression tests consume ?
2.What fraction of regression test cases are automated?
3.What is your regression test yield?   RegressionTest yield= (#DefectsFound InRegression #RegressionTestCasesExecuted)
4.I think that a smarter assessment of how much to regress could reduce regression test effort by:

We had a total of forty four (44) anonymous responses, the four key inferences are:

  1. Regression is non-trivial, consuming possibly close to half the test effort.
  2. A long way to go for automation, only a quarter of folks seem to have half their test cases automated.
  3. Low yielding human effort is expensive, only fifth of regression test cases find issues, a low defect yield.
  4. Smart regression is business beneficial, at least 25% effort reduction is what practitioners feel.

Here it is, as an interesting infographic. What do you think? How may $$/time will you save with 25% reduction?

RT Infographic

RT Infographic







joesthe athlete

The promising athlete

“What is the objective of load test?” I asked the participants attending my workshop. “It is to find out if the system can handle the load” replied one of them. A circular answer, this does not help!

Another person chimed in “Having multiple users use the system at the same time and check if the system functions properly”.

“Does that mean that load test is only applicable for multi-user systems?” I asked. Silence followed.

I have observed that the clarity of understanding of “what some of the common non-functional tests like load, stress, performance, scalability, volume are” is typically lacking. Possibly, because the tests are inter-related. Possibly, because the participants did not go beyond of these jargons!

Using “Joe the promising athlete” as an example I have found it is very easy to explain the objectives of these tests and then delve into the test design techniques for these.Let me share this with you now.

Joe is a young athlete, his coach seeing in him a future world champion. Every day Joe spends significant time at the gym building muscles/strength. He lifts weights and is comfortable lifting up to 40kg snatch a few times.

The coach observing that Joe is comfortable with 40 kgs hands him 50 kg weights. Joe is successful but finds it progressively difficult. After a few minutes, he pauses to catch his breath. The coach hands him 60 kg!

“Coach” Joe whines. “Go on Son,” says the coach in a baritone voice. Joe succeeds, his body glistening with beads of perspiration.

After stating this short snippet, I asked the participants in the workshop “So why is Joe whining?”

“Well he is tired,” says a participant

“Why is he tired? I ask.

“It is pretty obvious, he is run out of energy”, says another.

I continue with my next question “So why has he run out energy?” knowing fully well that my audience will think that I am pretty dumb.

“Well he has done a lot of work and that is the reason Joe has run out of energy”.

“Great. So Joe has expended energy doing work. When his energy has run out, he is unable to continue further” I summarized.

“Now, can you tell me how this story relates to load/stress test?”

I see the participants eyes brighten and one of them chips in “Oh Yes, Load is about subjecting the system to do “work” using the energy i.e system resources. 60 kg is the maximum Joe could lift as he was stressed out, and therefore stress test is finding out the maximum load that the system can “lift” before the energy i.e. resources give out”.

“Wonderful guys. Let us summarize. A load test is about understanding if the system can do the typical real-life work with the given resources while the Stress test is about understanding the maximum amount of work that can be done by consuming ‘all’ the resources”.

Now the coach seeing Joe is tired gives him an energy drink. After a few minutes, Joe is ready to go and the coach hands him 75 kg. Joe now energized, lifts it with aplomb!

“Now what can we learn from this?” I ask. The participants think for a while and then “Well I think Joe got an additional shot of energy from the drink and he was able to lift more” said the guy in the last row.

“Wonderful correlation. Yes, he could do more work with the additional energy. Now can you connect this to scalability test?”

After a momentary pause, the person on the far left says “Yes, once stressed, more energy allows us to do continue to do more work and hence scalability test is about checking if the system when stressed out can indeed do more work if given more resources.”

“Brilliant. Now would one of you summarize this please” I ask.

The short guy with glasses in the first row says “Resources (i.e. energy) enables us to do the various System Operations (i.e. work). The three tests Load, Stress & Scalability” are about connecting these in different ways. I will illustrate this on the board”.

He walks to the board and writes:







“Great summary. Indeed simple. Now how does this relate to performance test?”

“That is easy. Performance does with time, the time taken to do the work (operation). Performance test is about measuring if the time taken to perform the operations is indeed acceptable (with the given resources)”

“Does this mean that the load and performance test can be done at the same time? The objective however of these two tests are indeed different, is it not?” asks the backbencher.

“Yes, you are right, they may be done together, however, their objectives are different,” I say.

“Before we conclude, one last question – What is volume test?” I ask, walking to the end of the room.

After a few seconds, a quiet but firm voice behind me says “System Operations is really about processing data and volume test is about checking if the system can indeed process large amounts of data. By the way, in some systems, a large volume can choke and stress the system”.

“Guys, guess Joe and his coach helped us understand Load, Stress, Scalability, Performance and Volume test. I hope you understand that any kind of system (single/multi-user) can be subjected to these tests. Real life load is about subjecting the system to various types of operations (concurrently if multi-user) and not just subjecting the system to a blast of a specific operation” concluding the class.












“Yes” chimed in the participants with a big smile.

Thank you, guys. I am done with the class.

This is one of the lessons in the online course “Load testing using JMeter” in CleanSoft Academy Online at Cleansoft Academy Online











If you find the article interesting, please ‘like’, ‘share’ or leave a comment below.


Behave yourself.

One of the common statements in software testing that I come across is “Domain knowledge is critical to testing the functionality”. The belief is that knowledge of the domain acquired from prior experience is very necessary to perform good functional testing. Not that I disagree, because a prior experience is always handy. But the key question is ‘What if I am encountering a functionality for the first time? Will I be effective?’. Wouldn’t it be nice to be able to do this using a set of techniques in a scientific manner?

Let us first examine the basics… The objective of functional testing is to ascertain if the behavior of the entity under test (EUT) is as intended. So what is behavior? When we tell an unruly kid “Behave yourself”, what we mean is ’please comply with the acceptable norms i.e. rules’. Hence behavior of an EUT is about ’obeying some rules’. These ’rules’ are really the specification. So to test a functionality of any EUT, we need to first understand the intended rules and then create scenarios that are really various behaviors by combining the rules in different ways and then check the outcome(s).

So how do we figure out the rules? Hmmm… If I had domain knowledge then I would know how this should work, and therefore the rules. Note that this can be dangerous too as this sets up a bias, and we may assume certain rules that may be incorrect. Anyways, the issue is, how otherwise? To understand the intended behavior, first, describe what the EUT is supposed to do. Describe the behavior as a series of steps. Each step accepts/uses data and processes the data according to some condition(s). Examine each step and extract the condition(s) hidden in the step. Voila… We now have the various conditions aka rules.

This technique of extracting conditions (aka rules) is what I call “Descriptive to Prescriptive”, that is converting the descriptive specification to a prescriptive specification. “Describing” enables us to understand the behavior from first principles while “Prescribing” helps us to validate the behavior. We all know that clear and comprehensive description consists of the 5W+1H – Who, What, When, Where, Why & How. So when you encounter a functionality for the first time, apply 5W1H and describe the behavior as a series of simple steps.

Now examine each step and identify conditions implicitly or explicitly present. Once the conditions are extracted from each step, re-write each step as a series of rules and what we have now is prescriptive behavior. From this, it is easier to create the test scenarios and then identify the various data sets for each scenario and subsequently generate test cases and therefore for the entire EUT.

This behavior-based approach to functional test design is implemented by two techniques ’Box Model’ and ’Behaviour-Stimuli Approach’ in HBT (Hypothesis Based Testing). This lessens the need for domain knowledge as a prerequisite by enabling you to question better to understand behavior, and then extracting conditions to test effectively. Click here to read some articles on this from

So whether you test a technical feature or a business flow try this approach. Be the doctor – “Ask the patient to describe so that can you can prescribe”. Enjoy the happiness that comes out of the cure.


May you contribute to healthy software and a cleaner world.

If you find the article interesting, please ‘like’, ‘share’ or leave a comment below.


The tale of two doctors

“This is a long story of two doctors who by their different approaches to diagnosis helped Joe a professional tester sort out the typical confusions about black, white and unit, system testing. “

{Be warned this story may take about 7-9 minutes to read!}

In a big city lived Joe, a typical urban yuppie. He was always focused on a great tomorrow. He worked very hard, partied furiously and lived a fast life.

Life was a blast until his body decided to act up. On this Sunday morning, he woke up panting, unable to breathe, body drenched in sweat, with a dull pain in his chest. The previous evening was a blast, a celebration party thrown for his best buddy getting engaged. After an evening spent at bowling, they hit the pubs, closing each one, until they could not find one open.

A typical Sunday morning would commence at noon; today, as he was rudely jolted out of his reverie, the bright LED clock showed 7:00. could not move his arms, it seemed to take a tremendous effort to reach out for the bottle of Evian on the table near his bed. He had read about old age diseases getting younger in these modern times, dismissed them brashly, a reflection of his supreme yuppie confidence. For once he faltered, worried seriously if he could become one of the stories. All these years, he had thought of God, as a fashion statement but today he genuinely wished to believe that God exists. For the next few seconds, which seemed like an eternity, his mind rapidly flew back over the past years on the constant abuse he had heaped on his body. For once he prayed dearly that he would do the right things if he was excused this time. The clock glowed 7:02, and he realized it had been the longest two minutes of his life.

At 8:55 a.m he was at the reception of GoodLife hospital for a 9 AM appointment with Dr. Robert Black Sr., a senior and very experienced cardiologist. He was soon ushered in and was face to face with a severe yet friendly gentleman, a few orderly strands of golden hair on his shiny head with a piercing pair of eyes. “Mr. Joe, would you please tell me your problem?” said Dr. Black. Joe described in detail his travails upon his rude awakening. An old school doctor, he believed in detailed physical examination rather than the fancy modern equipment. “Lie down on the bed and relax Mr. Joe”. He took his stethoscope, placed it on his chest and listened carefully. “Breathe in and out deeply now,” said the doctor as he continued to hover his steth on various parts of the chest. His sharp eyes showed no emotion, as he went about his job confidently. The young nurse standing next to the doctor was petite and beautiful. She dispassionately took out the sphygmomanometer, wrapped the elastic band around Joe’s arm, pumped it up and watched the mercury bobbing up and down, while her other hand measured the pulse. After a minute, she looked at Dr. Black and said 140/120 and 93, in a husky voice.

“Mr. Joe, have you been feeling very tired at the end of day lately?”

“Yes,” he said and added, “It is the busy time of the year at work, a string of late nights.”

“What kind of work do you, Mr. Joe?”

“I work in the software industry. We are in midst of building a cool application for mobile phones”

“Oh I see, you are the software guy. My nephew is in the software business too and is always racing against time.”

“I guess you must be tied to the desk most of the time. Do you exercise?”

“Well doctor, the days are long and busy, I catch up on my sleep over the weekend. I try to workout in the gym over the weekends, but it is challenging.”

“I see that you are a smoker, do you drink? And you a vegetarian?”

“Well doctor, I do drink and I am not a vegetarian.”

Dr. Black was one of the most famous cardiologists in the country. He was a master at diagnosis, believed in the scientific and systematic study of symptoms and their connections. He placed his gold-rimmed glasses on the table, rubbed his finger on his chin, leaned back on his cozy leather chair and his piercing eyes looked straight into Joe and said “Mr. Joe, you have an issue with the blood supply to the heart muscles, there seems an advanced arterial block. It is necessary that you undergo angioplasty, a procedure that relieves the constriction as quickly as possible, say within a fortnight.”

Joe sat still, staring at the statement “The most amazing non-stop machine. Take care.” written on a poster containing the picture of a heart, which prominently on the wall behind the doctor.

Dr. Black who was used to these reactions jolted Joe out of the reverie “Mr. Joe, you need to quit smoking and go vegetarian. You are young, the angioplasty procedure has a high success rate and you should be back on to active life quickly”. Dr. Black believed in conveying news straight and expected patients to face reality and act on the problem. “Mr. Joe, you would need to be in the hospital for 3-4 days, do decide on the date quickly. As I mentioned before, it is important that you act on this within a fortnight. It is a painless procedure and should be fairly straightforward in your case. Do you have any questions?”

“No doctor, I have none” replied Joe mechanically, his ability to think, numbed by the turn of events. As he exited the consulting room, the receptionist, a cheerful and bubbly woman in twenties whispered: “Have a good day Mr. Joe” with a beautiful smile, a genuine one. It had the effect, and for a moment he felt cheerful and returned the smile, a little weakly though.

As soon as he was outside the hospital, his hand went mechanically to his shirt pocket containing the cigarette packet. “Smoking kills,” said the packet loudly and he threw the packet on the wayside.

David, his roommate had just woken up as Joe returned to his apartment. “Hi, have you had breakfast? Got some muffins and bagels, want some?” asked David. “No thanks” mumbled Joe.

After a few minutes, David understood the reason for the strange behavior of his best friend. “Come on man, let us get a second opinion right away”. David was one who never lost his cool and his level-headed thinking in tough situations was one that helped his friends many a time.

At 10:50 they were at ValleyTech hospital for a consultation with Dr. James White. He had been referred by David’s boss, who had undergone a heart bypass a few months at ValleyTech.

At 11:05 Joe was called in. “Good morning Joe. Please sit down. I have read your case sheet and have a few questions. Do you have any recent ECGs?

“No doctor,” said Joe.

“I know your company has a yearly health check plan for all, as our hospital administers it. Have you not taken it this year?”

“No doctor, the last few months have been very hectic and I have not had my yearly checkup yet,” said Joe.

Dr. White was a modern doctor who relied on technology in diagnosis and treatment. A young cardiologist, he believed in seeing the ‘internals’ before the scalpel touched the body. He was amazed at the advancements in radiology and made it a point to recommend a few pictures to be taken before he touched a patient.

Unlike Dr. Black who believed in the power of external examination, Dr. White believed in the looking at internals for diagnosis and treatment. Dr. White had immense faith in scans, lab tests and preferred analyzing reports to spending time on and performing examinations on patients.

“Please get the ECG done now. I would like to see the report first. Thank you.”

Joe went to the diagnostic lab in the adjoining building. When his turn came, he went inside, removed his shirt, the technician smeared jelly on his chest and proceeded to stick colorful leads at various points. In a few seconds, the needle was dancing, drawing patterns on the strip of paper. The technician looked at the squiggles on the paper with a bored expression, and after a few minutes decided the machine has had its share of fun and switched it off. He tore the roll of paper, scanned it intently, and then proceeded to fold it and inserted into a cover. Joe was curious to know what the squiggles meant and asked: “Is it normal?”.

“It seems fine except for a small spike here” replied the technician. He had seen hundreds of such squiggles and knew exactly as to what was normal, but he was no doctor to interpret any abnormalities.

Joe stared at the report, the squiggles held a secret that Joe was scared about. “Hey, let’s go meet to the doc now,” said David breaking Joe’s train of negative thoughts.

“Show me the report Joe,” said Dr. White.

The doc held the strip of paper and rapidly scrolled it forward and then backward.

“Were you treated for any heart-related issue when you were young?” asked Dr. White.

“No,” said Joe, scared to ask questions.

“Joe, there is a slight aberration in the ECG, it may be nothing to worry about. To confirm this, I recommend that you get the 64 slice beating heart scan. This the most advanced technology for diagnosis of heart-related ailments that is available in the world and we are the only hospital in this city to have this. This gives a clear picture of the beating heart and enables clear diagnosis. And also take a chest X-ray too. I will be available until 1:00 PM, please get it done right away and then see me”. He wrote down the lab request and handed it over to Joe.

“Hello, I need to get ‘64 slice beating heart scan’ done. How much does it cost? Joe asked the grim looking gentlemen on the cash counter.

He was shocked at the cost of the hi-tech scan, it seemed to have enough digits to max his credit card. Joe looked at David conveying in his look “They are milking us”.

Joe realized that the second opinion was going to be expensive and needed to think about this before he went on a diagnostic spree.

Joe was a professional software tester who diagnosed software for defects. He decided to chat with David over a cup of coffee to decide whether to go ahead with the expensive scan. David went to get the coffee while Joe sat down at the corner table, gazing at the birds fluttering over the little pond outside.

During this mindless gaze, staring at the chirpy birds, it suddenly flashed on him, the parallels between his profession and the doctor. In his job, he used black box techniques that required him to examine the system externally and find defects. He relied on the deep understanding of the intended behavior, observation of behavior (“symptoms”) to design & refine his test cases. He had at times looked at internal information like architecture, technology, code structure to design test cases that were adept at catching issues related to structure.

His colleagues always used terms like “Black box testing” and “White box testing” and associated these to with system and unit levels respectively. Now he realized the general misconception that unit testing was white box testing and system box testing was black box testing. His train of thought was interrupted by hot coffee spilling on his shoulder followed by the shrill sound of glass breaking. “I am really sorry, hope you are ok,” said the elderly woman who had tripped over the protruding leg of the gentleman at the neighboring table spilling a hot cup of espresso that she was carrying. “I am fine, let me help you,” said Joe as he helped pick up the glass shreds.

He realized that certain types of defects were better caught via “internal examination”(white box test techniques) while some are most suited to be caught via “external examination” (black box test techniques). He now understood that both of these test techniques were required at all levels to uncover effectively and efficiently.

Suddenly the diagnosis approach followed by Dr. Robert Black and Dr. James White became clear. As soon as David laid the steaming cup of Cafe Latte on the table, Joe had made the decision. He was not withdrawn or worried. The confidence was back and he would not let the ECG scroll spoil his fun.

If you find the article interesting, please ‘like’, ‘share’ or leave a comment below.

design checklist-fi

Design checklists to “Do, Sync & Act”

This article is the second one on checklists, the first one being “The power of checklist

Checklists seem to defend everyone, a kind cognitive net designed to catch flaws of memory and attention.

Atul Gawande in his book “ The Checklist Manifesto” states that there are three kinds of problems in the world:

  1. SIMPLE: Individualistic in nature, solvable by application of simple techniques.e.g. Bake a cake.
  2. COMPLICATED: Can be broken into a set of simple problems, requires multiple people, often multiple specialized teams. Timing, coordination becomes a serious concern. e.g. Launching a rocket.
  3. COMPLEX: These are problems where the solution applied to two similar problems may not result in the same outcomes. e.g. Raising a child. Expertise is valuable, but most certainly not sufficient.

He continues on to say that checklists can provide protection against elementary errors in the case of simple problems that we are designed with. This can be accomplished by simple activity task checklist.

In the case of complex problems that require multiple specialists to coordinate and be in sync, a simple activity task checklist won’t do, what is needed is a checklist with communication tasks to ensure that experts discuss on the matter jointly and take appropriate action.

“Man is fallible, but maybe men are less so” Belief in the wisdom of the group, the wisdom that multiple parts of eyes are on the problem and letting watchers decide what to do.

So how can checklists help in solving simple/complex problems? Using simple activity task checklists to ensure simple steps are not missed or skipped and checklist with communication tasks to ensure that everyone talks though and resolves hard and unexpected problems.

Building tall buildings is a complex problem and the success rate of the construction industry’s checklist process has been astonishing. Building failure is less than 0.00002%, where building failure is defined as the partial or full collapse of a functioning structure. (from a sample size of a few million buildings!).

Now let us turn our attention to the complex problem. How does one deal with complex, non-routine problems that are fundamentally difficult, potentially dangerous and unanticipated? In these situations, the knowledge required exceeds that of any individual and unpredictability reigns. To solve these it is necessary to push the power of decision making from a central authority to the periphery, allowing people to make the decision and take responsibility.

So the checklist needs to allow for judgment to be used in the tasks rather than enforce compliance, so that actions may be taken responsibly. This needs to have set of checks to ensure stupid but critical stuff is not overlooked, a set of checks to ensure coordination and enable responsible actions to be taken without having to ask for authority. There must be room for judgment, but judgment aided and even enhanced by a procedure. Note that in COMPLEX situations, checklists not only help, they are *required* for success.

So, how does one make checklists that work? Aviation industry thrives on checklists in normal times and also to tackle emergencies. The advice from Boorman from the “The Checklist Factory” of Boeing for making checklists that work are:

  1. Good checklists are precise, easy to use in the most difficult situations. They do not spell out everything, they only provide reminders of critical and important steps, the ones that even the highly skilled professionals would miss.
  2. Bad checklists are too long, hard to use; they are impractical. They treat people as dumb and try to spell out every step. So they turn people’s brain off rather than turning them on.

“The power of checklists is limited, they help experts remember how to manage a complex process or machine, make priorities clearer and prompt people to function well as a team. By themselves, however, checklists cannot make anyone follow them.” (Boorman, Boeing)

So how should a good checklist look like?

  1. Keep the length of checklist between five to nine items, the limit of human memory.
  2. Decide on whether you need a DO-CONFIRM checklist or READ-DO checklist. With a DO-CONFIRM checklist, individuals perform jobs from memory and experience and pause to CONFIRM that everything that was supposed to be done was done. With the READ-DO checklist, individuals carry out the tasks as they tick them off, it is like a recipe. So choose the right type of checklist for the situation. DO-CONFIRM gives a people a greater flexibility in performing the tasks while nonetheless having to stop and confirm at key points.
  3. Define clear pause points at which a checklist should be used
  4. The look of checklist matters, they should be free of clutter and fit into a page

In summary

  • Ticking boxes is not the ultimate goal
  • Checklist is not a formula, it enables one to be smart as possible
  • It improves outcomes with no increase in skill
  • Checklists aid efficient working

As smart individuals, we don’t like checklists. It somehow feels beneath us to use a checklist, an embarrassment. The fear is that checklists are about mindless adherence to protocol.

Hey, the checklist should get the dumb stuff out the way so as to let you focus on the hard stuff. We are all plagued by failures – by missed subtleties, overlooked knowledge, and outright errors. Just working harder won’t cut. Accept the fallibilities. Recognise the simplicity and power of the checklist.

Try a checklist. It works.

If you find the article interesting, please ‘like’, ‘share’ or leave a comment below.


The power of checklist

Recently I read the book “The Checklist Manifesto” by Atul Gawande. 

“An essential primer on complexity in medicine” is what New York Times states about his book whilst The Hindu states this as “An unusual exploration of the power of to-do list”.

As an individual committed to perfection, in constant search of scientific and smart ways to test/prevent and as an architect of Hypothesis Based Testing, I was spellbound reading this brilliantly written book that made the lowly checklist the kingpin, to tackle complexity and establish a standard for higher baseline performance.

The problem of extreme complexity The field of medicine has become the art of managing extreme complexity. It is a test whether such complexity can be humanly mastered: 13000+ diseases, syndromes and types of injury (13000 ways a body can fail), 6000 drugs, 4000 medicines and surgical procedures each with different requirements, risks and considerations. Phew, a lot to get right.

So what has been done to handle this? Split up knowledge into various specializations, in fact, we have super specialization today. But it is not just the breadth and quantity of knowledge that has made medicine complicated, it is also the execution of these. In an ICU, an average patient required 178 individual interactions per day!

So to save a desperately sick patient it is necessary to: (1) Get the knowledge right (2) Do the 178 daily tasks right.

Let us look at some facts: 50M operations/year, 150K deaths following surgery/year (this is 3x #road-fatalities), at least half of these avoidable. Knowledge exists in supremely specialized doctors, but mistakes occur.

So what do you when specialists fail? Well, the answer to this comes from an unexpected source, nothing to do with medicine.

The answer is: THE CHECKLIST

On Oct 30, 1985, a massive plane that carries 5x more bombs roared and lifted off from the airport in Dayton, Ohio and then crashed. The reason cited was “Pilot error”. A newspaper reported, “this was too much airplane for one man to fly”. Boeing the maker of this plane nearly went bankrupt.

So, how did they fix this issue? By creating a pilot’s checklist, as flying a new plane was too complicated to be left to the memory of any one person, however expert. The result: 1.8 million miles without one accident!

In a complex environment, experts are against two main difficulties: (1) Fallibility of human memory, especially when it comes to mundane/routine matters which are easily overlooked when you are strained to look at other pressing matters of hand (2) Skipping steps even when you remember them, as we know that certain steps in a complex process don’t matter.

Checklists seem to provide against such failures and instill a kind of discipline of higher performance.

Peter Provonost in 2001 decided to give a doctor’s checklist a try to tackle central line infections in ICU. So what was the result after one year of usage? Checklist prevented 43 infections and 8 deaths and saved USD 2M! In another experiment, it was noticed that patients not receiving recommended care dipped from 70% to 4% and pneumonia fell by a quarter and 21 fewer parents died.

In a bigger implementation titled the “Keystone Initiative” (2006) involving more hospitals of 18-month duration, the results were stunning- USD 17M saved, 1500+ lives saved!


So where am I heading? As a Test Practitioner, I am always amazed at how we behave like cowboys and miss simple issues causing great consternation to the customer and users. Here again, it is not about lack of knowledge, it is more often about carelessness. Some of the issues are so silly, that they can be prevented by the developer while coding, and certainly does not demand to test by a professional. This is where a checklist turns out to be very useful.

In an engagement with a product company, I noticed that one of the products has a product backlog of ~1000 issues found both internally and by the customer. Doing HBT level-wise analysis, we found that ~50% of the issues could have been caught/prevented by the developer preventing the vicious cycle of fix and re-test. A simple checklist used in a disciplined manner can fix this.

So how did the checklists help in the field of medicine or aviation? They helped in memory recall of clearly set out minimum necessary steps of the process. They established a standard for higher baseline performance.


So how can test practitioners become smarter to deliver more with less? One way is to instill discipline and deliver baseline performance. I am sure we all use some checklist or other but still find results a little short.

So how can I make an effective checklist and see a higher performance ? Especially in this rapid Agile Software world?

This will be the focus of my second part of this article to follow. Checklists can be used in many areas of software testing, I will focus in my next article on ‘How to prevent simple issues that plague developers making the tester a sacrificial goat for customer ire by using a simple “shall we say unit testing checklist”.

Related article: Design checklists to “Do, Sync & Act”

If you find the article interesting, please ‘like’, ‘share’ or leave a comment below.




“Load testing is not bombarding the system with concurrent users”

“We want you to load test our application. We want you to test with a load of 50k concurrent users. Can you do this?” asked the gentlemen from a company developing an innovative HR application during the sales call.

And my first reaction was ‘Whoah, hold your horses, load is not just about mere number of concurrent users’.  And then I started my explanation as to what the objective of load testing is and how they should look at it. After half an hour, they understood. They understood that this was not a simple web site where you bombard it with bunch of hits from users, but a web application that does various operations. And evaluating the load handling capability of the application requires subjecting it to a mix of various operations with some of them being concurrent.

Allow me to share this with you.
Let us start with a simple definition of ’Load’. Load is the quantum of work to be performed. And load testing is about assessing if the given quantum of work can be done well in the stipulated calendar time with the given resources. Now what is work? It is various operations that is done by the system.  And different end users perform different types of operations.

Consider the analogy of traffic system, of assessing the load handling capacity of a road in a city. Is it merely the number of vehicles it is able to handle at any time?  Well it depends on the types of vehicles. The vehicles may be trailers, trucks, cars or bikes. The carrying capacity of the street would be much higher if the vehicle is a bike instead of trailer.

In reality, traffic is a mix of the various types of vehicles – some trucks, cars, bikes and maybe trailers. And there could be one or more vehicles of a given type (car/truck …) that may pass through the road at a point in time. So evaluating the load handling capacity of the road requires us to understand the traffic situation at different parts of the day/week.

Therefore the kinds of vehicles that pass through this road depends on the what areas this road connects and therefore the types of users it services. Trucks and trailers from the highway may use this road to go to the industrial complex during the late evening hours/night, whereas students on bikes may use the road to go/from university campus to residential complex /shopping district during the day, while working professionals may clog the road with cars during the morning/evening hours.

So coming back to our web application, load testing is about subjecting it to a mix of the various operations from a various types of users with some of them being concurrent and then assessing if they can indeed be processed with the given resources. That is, can the road allow the various users to go their destination safely and on time? So just bombarding the application with a bunch of users is not meaningful and can give you false alarm that the system is incapable of handling load.
Remember that a web site has only two major operations – GET and PUT and subjecting a web site to concurrent hits(a simple mix of GET and PUT) is most often good enough. And this is not true with a web application.

Now understand that a single user application can be also subjected to load and concurrency is not a pre-requisite for load testing. After all load testing is about assessing if a system can process the requisite number of operations in a given time with the given resources. And each operation may have different demand of resources and may take different times to complete. And simulating this mix and creating such a operational (or workload) profile representing different times of day/week/month/year is key to meaningful real life evaluation of load handling. Designing load test scenarios in HBT (Hypothesis Based Testing) is based on this technique “Operation profiling”.

So when somebody tells you perform load testing by merely bombarding the system with users, remember “The road analogy” and “Operational profiling”.

When you are stuck in a traffic jam, it may be good fun to estimate the load then by applying operational profiling! On a lighter note, I do not care about the road nor traffic jam as I cycle. I squeeze through the stationary vehicles and smile at the fretting humans inside the powered vehicles. Wicked eh!