Should I automate all my test cases? What criteria could I apply to select test cases for automation?
Well you automate test scenarios, not test cases. As we have understood, test scenarios in HyBIST represent a flow and test cases are combinations of data for each scenario. So we automate a scenario producing a data-driven script where the test case data are externalised.
The answer to the first question Should I automate all my test cases ? is “Not necessarily”. We need to ensure that we derive value from automating the scenarios. The scenarios that cannot be executed by a human needs to be automated, the typical examples of these are non-functional tests like load/perfomance etc. Most often test at Levels 6/7 require to be automated. As for the tests at Levels 1-5, we have choices to automate or not. Well some structural tests at L3 require tools and therefore automated evaluation especially leak detection, static analysis of complexity, automated code review etc. So now the question narrows down to quality levels L1, L2, L4, L5 . Early level test at L1, L2 and parts of L4 can be easily automated and can form part of build system to ensure that code can be automatically validate and then checked in. Parts of L4 and L5 is what we have to make choices.
So what are the criteria to select these test cases?
1. Test scenarios that form part of regression suite.
2. Test scenarios enable a basic health check.
3. Test scenarios that require deep domain to execute and evaluate.
4. Test scenarios that are time consuming and possibly need to run very many times.
In a nutshell, when you want to decide what to automate, think in terms of tests at the various Quality levels. Since the test cases at L4 & L5 tend to be a lot more, the decision making is more often focussed on ‘what to choose in L4/L5′.