Analytical Risk-based and Specification-based Testing - Bui Duy Tam
5 likes2,195 views
The document presents a comprehensive overview of testing strategies in software development, emphasizing analytical, risk-based, and specification-based testing techniques. It outlines the importance of creating a test strategy to achieve objectives, including defect detection and stakeholder satisfaction, while defining key concepts like software risks and their management. Additionally, it highlights the benefits of employing analytical strategies and describes methodologies for prioritizing specifications and testing activities.
Analytical Risk-based and Specification-based Testing - Bui Duy Tam
- 1. Analytical: Risk-based and Specification- based Testing SPEAKER: TAM BUI
- 2. About me • B. A in English in 1996 – University of Hue. • B. Eng in Computer Science in 2005 – Hanoi University of Science and Technology. • ISTQB – Advanced Level – Test Manager in USA in 2013. • Over 14 year experiences in software testing. • Strong knowledge and experiences in automation testing, performance testing and security testing. • Great passion in software testing: learning and sharing knowledge and experiences. 2
- 3. Objectives • What is a Test Strategy? • What is about Analytical Test Strategy? • Risk-based Testing. • Specification-based Testing. • Benefits of Analytical Test Strategy. 3
- 4. What are Test Objectives? • Finding Defects. • Preventing Defects. • Providing Quality Information. • Gaining confidence about the level of quality. • Satisfying the needs of stakeholders: Scope, Time, and Budget. • Satisfying the Business/System Requirement Specification. 4
- 5. What is a Test Strategy? A Test Strategy is a document to be developed to inform project managers, testers, developers how to achieve test objectives. 5
- 6. Types of Test Strategies Analytical Model- based Methodical Standard- compliant Dynamic Consultative Regression- averse 6
- 7. Test objectives in real projects Project 1 You are a Test Manager. You are assigned to work with a new project with some difficult objectives: large scope with short time and resource shortage. Project 2 You are a Test Manager. You are assigned to work with a new project to develop a safety critical system which will be used in the surgery field. Which Test Strategy is selected? Analytical 7
- 8. What is about Analytical Strategy? All test planning activities are based on data and analysis of data Risk-based Testing Specification- based Testing 8
- 9. Risk-based Testing • What is Software Risk? • Risk Management. 9
- 10. What is Software Risk? A risk is something that has not happened yet and it may never happen. It is a potential problem. Risk is the possibility of a negative or undesirable outcome. A risk has some likelihood between 0% and 100%. 10
- 11. Software Risk Areas Functional complexity Performance Safety Data Selections Recoverability New Technology Scalability and Reliability Scope of Use Environment Security Reliability Usability Interface Complexity Technical Complexity 11
- 12. Risk Management Risk Identification Risk Control, Mitigation Risk Analysis 12
- 13. Risk Identification Expert interviews Independent assessments Use of risk templates Lessons learned • Metrics and data from past projects. • Develop a risk database or repository 13
- 14. Risk Identification (Cont.) Formal analysis methods • Failure modes and effects analysis (FMEA) • Hazard analysis, Cost of failure, Other Risk workshops Brainstorming Checklists Calling on past experience 14
- 15. Risk Analysis – Likelihood (Technical Factors) Complexity of technology Personnel issues Trainings Intra-team and inter-team issues Supplier and Vendor contractual problems Geographical distribution of the development organization New Technologies and designs Bad quality of tools and technologies used Bad management and technical leadership Time, resource, and management pressure Late testing and bad quality assurance Requirements, designs, code changes High defect rate The frequency of use of the affected feature 15
- 16. Risk Analysis – Impact (Business Factors) Potential damage to the company image Loss of Customers and Business Financial and social losses or liability Civil or criminal legal sanction Loss of licenses or permits The lack of reasonable workarounds The visibility of failure and the associated negative publicity 16
- 17. Risk Analysis – Risk Priority Number • Apply 5-point-scale for Likelihood and Impact for each risk: – 1 = Very High – 2 = High – 3 = Medium – 4 = Low – 5 = Very Low Risk Priority Number = Likelihood * Impact 17
- 18. Risk Analysis – Extent of Testing 1-5: Extensive: run large number of tests, both broad and deep, combine and vary interesting conditions, use all relevant techniques with strong coverage criteria. 6-10: Broad: run medium number of tests, exercise many different interesting conditions use most relevant different interesting conditions, use most relevant techniques with medium coverage criteria. 11-15: Cursory: run small number of tests, sample most interesting conditions, use efficient techniques with weak interesting conditions, use efficient techniques with weak coverage criteria 16-20: Opportunity: leverage other tests or activities to test 1-2 interesting conditions, investing very little time and effort, using reactive techniques especially. 21-25: Report bugs only: allocate only a small amount of extra time to report and manage these accidental bugs. 18
- 19. Example RISK LIST WITH EXTENT OF TESTING No Quality Risk Likehood Impact Risk Pri. # Extent of Testing Tracing 1 Functional Risks 1.1 Function 1 2 5 10 Broad 100-100-100 1.2 Function 2 1 1 1 Extensive 100-100-101 1.3 Function 3 3 5 15 Cursory 100-100-102 1.4 Function 4 4 5 20 Opportunity 100-100-103 1.5 Function 5 5 5 25 Report Bugs 100-100-104 2 Performance Risks 2.1 Function 6 1 1 1 Extensive 100-100-105 2.2 Function 7 2 4 8 Broad 100-100-106 2.3 Function 8 3 3 9 Broad 100-100-107 2.4 Function 9 2 2 4 Extensive 100-100-108 2.5 Function 10 5 5 25 Report Bugs 100-100-109 3 Usabilty Risks 3.1 Function 11 4 4 16 Opportunity 100-100-110 3.2 Function 12 2 3 6 Broad 100-100-111 3.3 Function 13 3 1 3 Extensive 100-100-112 3.4 Function 14 1 1 1 Extensive 100-100-113 3.5 Function 15 5 5 25 Report Bugs 100-100-114 19
- 20. Risk Control Mitigate: Take steps in advance to reduce the possibility and impact of the risk. Contingency: Have a plan in place to reduce the possibility of the risk to become an outcome. Transfer: Convince some other member of the team or project stakeholder to reduce the probability or accept the impact of the risk. Ignore: Ignore the risk, which is usually a good option only when there is little that can be done or when the possibility and impact of that risk are low in the project. 20
- 21. Bottom line of Risk-based Testing Plan Testing Activities such as design tests, implement tests, execute tests, report test results according to Risk Priority Number 21
- 22. Specification-based Testing • What is a Specification? • What are implicit Specifications? • How to learn from the Specification. • Clarify and fix ambiguities – Review techniques. • Prioritize Software Requirement specification. • Bottom line of Specification-based Testing. 22
- 23. What is a Specification? A detailed description of work to be done or materials to be used in a project - Marriam-Webster A software requirements specification (SRS) is a description of a software system to be developed. It lays out functional and non- functional requirements and may include a set of use cases that describe user interactions that the software must provide. - Wikipedia 23
- 24. What kinds of Specification can be tested? Software Requirement Specification Software Design Specification User Interface Description User Manual 24
- 25. What are implicit Specifications? Characteristics of the product are not mentioned in the specification. 25
- 26. What are implicit Specifications (Cont.) Business Culture Technical Norms Design/Code Standards Guidelines Regulations Marketing/Sale Presentations Memos (Software changes) 26
- 27. What are implicit Specifications (Cont.) Competing products Related products Email discussions within the project Customer comments Bug Reports Test Results Prototypes 27
- 28. How to learn from the Specification? SQ3R Reading Strategy Skim Question Read Respond Review 28
- 29. Clarify and Fix Ambiguities – Review Techniques Walkthrough Technical ReviewPeer Review Inspection 29
- 30. Prioritize Requirement Specifications MoSCoW MUST Mandatory SHOULD High Priority COULD Desired but not necessary WOULD Can be delayed and proposed for future releases 30
- 31. Example SOFTWARE REQUIREMENT LIST Req. ID Requirement Requirement Description Priority 1 Funtionality Requirement 1.1 Requirement 1 Description of Requirement 1 Must 1.2 Requirement 2 Description of Requirement 2 Could 1.3 Requirement 3 Description of Requirement 3 Would 1.4 Requirement 4 Description of Requirement 4 Must 1.5 Requirement 5 Description of Requirement 5 Should 1.6 Requirement 6 Description of Requirement 6 Must 1.7 Requirement 7 Description of Requirement 7 Could 1.8 Requirement 8 Description of Requirement 8 Must 1.9 Requirement 9 Description of Requirement 9 Would 1.10 Requirement 10 Description of Requirement 10 Must 2 Performance Requirements 2.1 Requirement 11 Description of Requirement 11 Should 2.2 Requirement 12 Description of Requirement 12 Could 2.3 Requirement 13 Description of Requirement 13 Should 2.4 Requirement 14 Description of Requirement 14 Must 2.5 Requirement 15 Description of Requirement 15 Should 31
- 32. Bottom line of Specification- based Testing Study Specifications which include implicit Specifications Review Specifications and update them if necessary Specifications are prioritized Plan testing activities such as design tests, implement tests, execute tests and report test results based on prioritized Specifications 32
- 33. Benefits of Analytical Strategy Risk-base Testing Specification-based Testing • Exhausted testing is impossible. Important features/functions will be addressed early. Important problems can be discovered early. • Defects/bugs can be prevented by running static testing. • Test Managers properly distribute test efforts, schedule time, use budget to mitigate risks and make contingency plan when risks occur. • Prioritized features/functions will be addressed early. Quality will be improved early. • Defects/bugs can be prevented by running static testing. • Tests can be derived easily. • Customer’s needs and expectations can be satisfied correctly. • Training new team members is easily. 33
- 34. Summary • Understand common test strategies and select appropriate test strategy for each project. • Comprehend risk-based testing and how to plan risk-based testing activities. • Comprehend specification-based testing and how to plan specification-based testing activities. 34
- 35. Reference • Lessons Learned in Software Testing – Cem Kaner, James Bach, Bret Pettichord. • Specification-based Testing – Testing Education and BBST – Cem Kaner. 35
- 36. 36
- 37. 37