We have been talking and advocating on various platforms, how scientific approach of HyBIST and the method, STEM, delivers key business value propositions. This time we thought it would be prudent to share our experience of implementing this in projects and convey the results as well as interesting benefits that it entails while delivering clean software to our customers.
HyBIST was applied on various projects that were executed on diverse technologies in variety of domains across different phases of product life cycle. The people involved where of mix from no experience to 5 years of experience.
We observed that HyBIST can be plugged into any stage or situation of the project for a specific need and one can quickly see the results and get desired benefits as required at that stage.
Our experience and analysis showed that there were varied benefits like rapid reduction in ramp up time, creation of assets for learning, consistent way of delivering quality results even with less experienced team members, increased test coverage, scientific estimation of test effort, optimization of regression test effort/time/cost, selection of right and minimal test cases for automation and hence reduction in its development effort/time/cost.
Following are the key metrics and the results/benefits achieved in some of the projects were HyBIST was implemented –
Project 1:
Domain: SAAS / Bidding Software
Technology – Java, Apache (Web server), Jboss (App server), Oracle 9i with cluster server on Linux
Lines of Code – 308786
Project Duration – 4 months
D1, D2 and D4 were done almost in parallel due to time constraints for this complete application that was developed from scratch.
- D1 – Business Value Understanding (Total effort of 180 person hours)
- 3 persons with 3+yrs experience were involved (had no prior experience in this particular domain)
- 4 main modules with 25 features listed.
- Landscaping, Viewpoints, Use cases, Interaction matrix (IM) were done.
- D1 evolved and developed by asking lot of questions to the design/dev team.
- D2 – Defect Hypothesis (Total effort of 48 person hours)
- 3 persons with 3+yrs experience were involved.
- 255 potential defects were listed.
- D4 – Test Design (Total effort of 1080 person hours)
- 3 persons with 3+yrs experience were involved.
- Applied Decision tables (DT) for designing test scenarios.
- Totally 10750 test cases were designed and documented.
- Out of which 7468 (69%) are positive test cases and 3282 (31%) are negative test cases.
- Requirement Traceability Matrix (RTM) and Fault Traceability Matrix (FTM) were prepared.
- D8 – Test Execution (Total effort of 3240 person hours)
- 9 persons were involved in test execution and bug reporting/bug fixes verification (3 persons with 3+ yrs experience and 6 persons with 2+ yrs experience).
- 12 builds were tested in 3 iterations and 4 cycles.
- Totally 2500 bugs were logged, out of which 500 bugs were of high severity.
Key benefits:
- No bugs were found in UAT.
- All change requests raised by QA team was accepted by Customer & Dev Team.
- Interaction matrix was very useful for selecting test cases for regression testing and also for selecting right and minimal test cases for automating sanity testing.
- Regression testing was for shorter periods like 2 to 3 days, interaction matrix was quite useful to do optimal and effective regression testing.
- The method, structure, templates (IM, DT, RTM, FTM, Test case, Reporting) used and developed in this project is being used as reference model for other projects at this customer place.
Project 2:
A web service with 5 features that had frequent enhancements and bug fixes (Maintenance)
Technology – Java, Apache Web Server
Project Duration – 4 weeks
- D1 – Business Value Understanding (Effort of 6 hours)
Mind mapping of the product and also the impact of other services & usage on this service
- D2 – Defect Hypothesis (Effort of 5 hours)
Listed 118 potential defects
Key Benefits:
- Preparation of D1 document enabled ramp up time for new members (Developers/Testers) to understand the product, to come down from earlier 16 hours to 4 hours now.
- Any member added to this team was productive from day one and could start testing for any regression testing cycles for enhancements and bug fixes.
- Listing of potential defects enabled adding missing test cases from the existing test case set.
Project 3:
Domain – E-learning
Technology – ASP.Net, IIS, SQL Server, Windows
Validation of a new feature added to the product
Duration – 2 weeks
- D1 – Business Value Understanding (Effort of 5 hours)
Understood the feature by asking questions and interacting with development team over emails/conf calls
- D2 – Defect Hypothesis (Effort of 2 hours)
Listed 130 Potential defects by thinking from various perspectives
- D4 – Test Design (Effort of 16 hours)
Designed and documented 129 test cases
- D8 – Test Execution (Effort for Test Execution – 626 Person hours,
Effort for bugs reporting/bug fixes Verification – 144 Person hours)
Executed test cases by performing 2 cycles of testing and 2 regression cycles
8 new test cases were added while executing the test cases
31 bugs were found in test execution of which 23 bugs were of high severity. 29 of the bugs can be linked to potential defects visualized and listed before. 2 of the bugs found were not linked to any documented test cases.
Key Benefits:
- Arrived at a consistent way of understanding the feature and designing test cases for new features irrespective of the experience of the team member involved
Project 4:
Domain – Video Streaming
Technology – C++, PHP, Apache, MySQL, Linux
An evolving new product in very initial cycles of development/testing
Duration – 4 weeks
People Involved – 2 Fresh test engineers (No previous work experience but trained in HyBIST/STEM)
- D1 – Business Value Understanding (Effort of 32 hours)
The understanding of the product in the form of listing features/sub features, Landscaping, Critical quality attributes, Usage environment/Use cases, by questioning
- D2 – Defect Hypothesis (Effort of 40 hours)
Listing of over 150 potential defects
Key Benefits:
- 2 fresh engineers could understand and comprehend the product features, the business flow and its usage in a scientific manner and also document it. They could also think and visualize possible defects to enable them to come up with needed test cases to identify and eliminate defects.
- The process of doing D1 and D2 by the fresh engineers generated lot of useful questions that enabled better thinking, understanding and different perspectives of the product behavior to the senior engineers. This helped them to design more interesting test cases to capture the defects during test execution.
- The assets created as D1 and D2 is helping the other members of the team to quickly ramp up on the product features and get the detailed understanding in 50 % lesser time.
Project 5:
Domain – Telecom protocol
3GPP TS 25.322 V9.1.0 Standards
Estimate effort for complete test design of RLC protocol by going through existing very high level test specifications and designing test cases for 2 sample functions
Duration – 3 weeks
None of the persons involved had any previous experience in testing protocol stack.
- D1 – Business Value Understanding (Effort of 40 person hours)
Went through the generic RLC standards and in specific understood the 2 functions, Sequence number check and Single side re-establishment in AM mode
Prepared flow charts with data/message flows between different layers
Prepared box model illustrating various inputs, actions, outputs and external parameters
- D2 – Defect Hypothesis (Effort of 16 person hours)
Listed 28 generic potential defects and the defect types
- D4 – Test Design (Effort of 48 person hours)
Prepared 2 input tables and 2 decision tables
Designed 26 test scenarios (6 positive, 20 negative) and 44 test cases (6 positive, 38 negative)
- Performed gap analysis of missing test cases in the customer’s test specification document for the 2 functions (Effort of 12 person hours)
- Estimated time and effort for the complete RLC test case design, based on the above data (Effort of 4 person hours)
Key benefits:
- Performed gap analysis in the existing high level test specs
- 20 times more test cases designed for 2 functions covered
- 86% of the test cases added were negative type
- Test cases developed were detailed and covered various combinations of inputs, parameters and intended/unintended behaviors
- Test cases developed were suitable in order to help easy conversion to test scripts using any tool
- Performed estimation for RLC test design covering 22 functions
- 256 Test scenarios and 1056 test cases to be designed with effort of 446 person hours for RLC
Project 6:
Domain – Retail
Validate railway booking software on point of sale device
Technology – Java
Duration – 4 weeks
- D1 – Business value understanding (Effort of 16 person hours)
Documented software overview, features list, use cases list, features interaction matrix, value prioritization and cleanliness criteria
- D2 – Defect Hypothesis (Effort of 18 person hours)
Listed 20 potential defects by applying negative thinking, 54 potential defects by applying Error Fault Failure model. The potential defects were categorized into 46 defect types and were mapped to the features listed.
- D3 – Test Strategy (Effort of 6 person hours)
Based on the listed potential defects types, arrived at test types, levels of quality and test design techniques needed as part of test strategy/planning. Quality level 1(Input validation and GUI validation), Quality level 2(Feature correctness), Quality level 3(Stated quality attributes) and Quality level 4(Use case correctness).
- D4 – Test Design (Effort of 24 person hours)
Designed and documented 30 test scenarios (15 positive, 15 negative) and 268 test cases (197 positive, 71 negative) for quality level 1, 70 test scenarios (21 positive, 49 negative) and 123 test cases (55 positive, 68 negative) for quality level 2 and 8 test scenarios for quality level 4. Created box models and decision tables to arrive at test scenarios
Prepared requirement traceability and fault traceability matrices
- D8 – Test Execution (Effort of 32 person hours)
Out of 293 test cases designed, 271 were executed and 22 were unable to be executed.
52 defects (27 high, 12 medium, 13 low) and 8 suggestions were logged.
Quality level 1 – 23 defects (2 high, 11 medium, 10 low) and 2 suggestions
Quality level 2 – 27 defects (25 high, 2 low) and 6 suggestions
Quality level 4 – 2 defects (2 medium)
Key Benefits:
- Complete validation of the product was performed successfully by one senior engineer guiding 2 fresh test engineers without any previous work experience and none of them having experience in this particular domain
- All the suggestions logged were accepted and valued
Project 7:
Domain – Mobile gaming
Technology – Java, Symbian
Duration – 3 weeks
- D1 – Business Value Understanding (Effort of 16 person hours)
Achieved product understanding by documenting software overview, technology, environment of usage, features list, use cases list, mapping of use cases to features, features interaction matrix, value prioritization and cleanliness criteria.
- D2 – Defect Hypothesis (Effort of 16 person hours)
Listed 96 potential defects by categorizing issues related to installation, download, invoking application, connectivity, input validation, Search, subscription, authorization, configuration, control, dependency, pause/resumption, performance, memory.
Mapped the features to the potential defects
- D3 – Test Strategy (Effort of 4 hours)
Based on the listed potential defects types, arrived at 4 levels of quality (Game initialization & invoking correctness, Game subscription correctness, Game download correctness, Dependency correctness). The different test types were mapped to the 4 quality levels.
- D4 – Test Design (Effort of 20 person hours)
Box models and decision tables were created
Designed and documented 37 test scenarios and 66 test cases
Key Benefits:
- Complete validation of the product was performed successfully by one senior engineer guiding 2 fresh test engineers without any previous work experience and none of them having experience in this particular domain
- The assets created in this became a useful reference for other projects in mobile gaming software for its understanding and validation