Software Engineering Lec 10 -software testing--
- 1. 10- Verification and Validation
- 2. • Assuring that a software system meets a user’s needs • Verification: • “Are we building the product right” – The software should conform to its specification. • Validation: • “Are we building the right product” – The software should do what the user really requires. Verification vs Validation
- 3. • Is a whole life-cycle process - V & V must be applied at each stage in the software process. • Has two principal objectives: – The discovery of defects in a system. – The assessment of whether or not the system is usable in an operational situation. The V & V Process
- 4. • Dynamic V & V Concerned with exercising and observing product behavior (testing). • Static verification Concerned with analysis of the static system representation to discover problems. Dynamic and Static Verification
- 5. Static and Dynamic V&V Formal specification High-level design Requirements specification Detailed design Program Prototype Dynamic validation Static verification
- 7. Example
- 8. Testing – Terminology • Test case – Individual test • Test suite – Collection of test cases • Test harness – Program that executes a series of test cases • Test framework – Software that facilitates writing & running tests – Example – JUnit
- 9. Testing – Terminology • Test driver – Program to create environment for running tests – Declares variables, creates objects, assigns values – Executes code and displays results of tests • Tester (Quality Assurance) – Person devising and / or performing tests – More effective if 2nd person writes tests • Walkthrough – Programmer explains code to 2nd person
- 10. Types of Testing • Program testing can be used to show the presence of bugs, but never their absence.” • Static: Done without actually executing program – Code inspections – Walkthroughs • Dynamic: Done by executing program or its parts – Module or unit testing – Integration testing – System testing
- 11. Who Tests the Software? developer independent tester Understands the system but, will test "gently" and, is driven by "delivery" Must learn about the system, but, will attempt to break it and, is driven by quality
- 12. Testing Principles • All tests should be traceable to customer requirements. • Tests should be planned before testing begins. • 80% of all errors are in 20% of the code. • Testing should begin in the small and progress to the large. • Exhaustive testing is not possible. • Testing should be conducted by an independent third party if possible.
- 13. Software Defect Causes • Specification may be wrong. • Specification may be a physical impossibility. • Faulty program design. • Program may be incorrect.
- 14. Types of Errors • Algorithmic error. • Computation & precision error. • Documentation error. • Capacity error or boundary error. • Timing and coordination error. • Performance error. • Recovery error. • Hardware & system software error. • Standards & procedure errors.
- 15. Test Strategies • Black-box or behavioral testing – knowing the specified function a product is to perform and demonstrating correct operation based solely on its specification without regard for its internal logic • White-box or glass-box testing – knowing the internal workings of a product, tests are performed to check the workings of all possible logic paths
- 16. Black-box Testing • Focus: I/O behavior. If for any given input, we can predict the output, then the module passes the test. – Almost always impossible to generate all possible inputs ("test cases") • Goal: Reduce number of test cases by equivalence partitioning: – Divide input conditions into equivalence classes – Choose test cases for each equivalence class. (Example: If an object is supposed to accept a negative number, testing one negative number is enough)
- 17. Black-box Testing (Continued) • Selection of equivalence classes (No rules, only guidelines): – Input is valid across range of values. Select test cases from 3 equivalence classes: • Below the range • Within the range • Above the range – Input is valid if it is from a discrete set. Select test cases from 2 equivalence classes: • Valid discrete value • Invalid discrete value • Another solution to select only a limited amount of test cases: – Get knowledge about the inner workings of the unit being tested => white-box testing
- 18. White-box Testing • Focus: Thoroughness (Coverage). Every statement in the component is executed at least once. • Four types of white-box testing – Statement Testing – Loop Testing – Path Testing – Branch Testing
- 19. White-box Testing (Continued) • Statement Testing (Algebraic Testing): Test single statements (Choice of operators in polynomials, etc) • Loop Testing: – Cause execution of the loop to be skipped completely. (Exception: Repeat loops) – Loop to be executed exactly once – Loop to be executed more than once • Path testing: – Make sure all paths in the program are executed • Branch Testing (Conditional Testing): Make sure that each possible outcome from a condition is tested at least once
- 20. • Unit testing – testing of individual components • Module testing – testing of collections of dependent components • Sub-system testing – testing collections of modules integrated into sub-systems • System testing – testing the complete system prior to delivery • Acceptance testing – testing by users to check that the system satisfies requirements. Sometimes called alpha testing Testing Stages
- 21. The Testing Process Sub-system testing Module testing Unit testing System testing Acceptance testing Component testing Integration testing User testing
- 22. Unit Testing • Program reviews. • Formal verification. • Testing the program itself. – black box and white box testing.
- 23. Types of Testing • Unit Testing: – Individual subsystem – Carried out by developers – Goal: Confirm that subsystems is correctly coded and carries out the intended functionality • Integration Testing: – Groups of subsystems (collection of classes) and eventually the entire system – Carried out by developers – Goal: Test the interface among the subsystem
- 24. System Testing • System Testing: – The entire system – Carried out by developers – Goal: Determine if the system meets the requirements (functional and global) • Acceptance Testing: – Evaluates the system delivered by developers – Carried out by the client. May involve executing typical transactions on site on a trial basis – Goal: Demonstrate that the system meets customer requirements and is ready to use • Implementation (Coding) and testing go hand in hand
- 25. Eaxample:Test Cases • Test case 1 : ? (To execute loop exactly once) • Test case 2 : ? (To skip loop body) • Test case 3: ?,? (to execute loop more than once) These 3 test cases cover all control flow paths
- 26. Generating Test Data • Ideally want to test every permutation of valid and invalid inputs • Equivalence partitioning it often required to reduce to infinite test case sets – Every possible input belongs to one of the equivalence classes. – No input belongs to more than one class. – Each point is representative of class.
- 27. Regression Testing • Check for defects propagated to other modules by changes made to existing program – Representative sample of existing test cases is used to exercise all software functions. – Additional test cases focusing software functions likely to be affected by the change. – Tests cases that focus on the changed software components.
- 28. Integration Testing • Bottom - up testing (test harness). • Top - down testing (stubs). • Modified top - down testing - test levels independently. • Big Bang. • Sandwich testing.
- 29. Top-Down Integration Testing • Main program used as a test driver and stubs are substitutes for components directly subordinate to it. • Subordinate stubs are replaced one at a time with real components (following the depth-first or breadth-first approach). • Tests are conducted as each component is integrated. • On completion of each set of tests and other stub is replaced with a real component. • Regression testing may be used to ensure that new errors not introduced.
- 30. Bottom-Up Integration Testing • Low level components are combined in clusters that perform a specific software function. • A driver (control program) is written to coordinate test case input and output. • The cluster is tested. • Drivers are removed and clusters are combined moving upward in the program structure.
- 31. Acceptance Testing • Making sure the software works correctly for intended user in his or her normal work environment. • Alpha test – version of the complete software is tested by customer under the supervision of the developer at the developer’s site • Beta test – version of the complete software is tested by customer at his or her own site without the developer being present
- 32. System Testing • Recovery testing – checks system’s ability to recover from failures • Security testing – verifies that system protection mechanism prevents improper penetration or data alteration • Stress testing – program is checked to see how well it deals with abnormal resource demands • Performance testing – tests the run-time performance of software
- 33. Performance Testing • Stress test. • Volume test. • Configuration test (hardware & software). • Compatibility. • Regression tests. • Security tests. • Timing tests. • Environmental tests. • Quality tests. • Recovery tests. • Maintenance tests. • Documentation tests. • Human factors tests.
- 34. Testing Life Cycle • Establish test objectives. • Design criteria (review criteria). – Correct. – Feasible. – Coverage. – Demonstrate functionality . • Writing test cases. • Testing test cases. • Execute test cases. • Evaluate test results.
- 35. Testing Tools • Simulators. • Monitors. • Analyzers. • Test data generators.
- 36. Test Team Members • Professional testers. • Analysts. • System designers. • Configuration management specialists. • Users.
- 37. Test Documentation Needed • Requirement being tested. • Design verification methodology. • Code verification methodology.
- 38. Document Each Test Case • Requirement tested. • Facet / feature / path tested. • Person & date. • Tools & code needed. • Test data & instructions. • Expected results. • Actual test results & analysis • Correction, schedule, and signoff.
- 39. Test Case Design "Bugs lurk in corners and congregate at boundaries ..." Boris Beizer OBJECTIVE CRITERIA CONSTRAINT to uncover errors in a complete manner with a minimum of effort and time
- 40. Debugging • Debugging (removal of a defect) occurs as a consequence of successful testing. • Some people better at debugging than others. • Is the cause of the bug reproduced in another part of the program? • What “next bug” might be introduced by the fix that is being proposed? • What could have been done to prevent this bug in the first place?
- 41. The Debugging Process Locate error Design error repair Repair error Re-test program
- 42. Debugging Approaches • Brute force – memory dumps and run-time traces are examined for clues to error causes • Backtracking – source code is examined by looking backwards from symptom to potential causes of errors • Cause elimination – uses binary partitioning to reduce the number of locations potential where errors can exist
- 43. End of lecture