Software testing lecture software engineering
- 2. 2 • Software testing needs to be planned in advance and conducted systematically • A number of software testing strategies have been proposed in the literature • All of them have these four generic characteristics 2
- 3. 3 Characteristics of Testing Strategies - 1 • (1) Testing begins at the component level and works “outward” toward the integration of the entire computer-based system • (2) Different testing techniques are appropriate at different points in time 3
- 4. 4 Characteristics of Testing Strategies - 2 • (3) Testing is conducted by the developer of the software and (for large projects) an independent test group • (4) Testing and debugging are different activities, but debugging must be accommodated in any testing strategy 4
- 5. 5 Testing Strategy - 1 • Unit testing • Integration testing • Validation testing • System testing 5
- 6. 6 Testing Strategy unit test integration test validation test system test
- 7. 7 Testing Strategy - 2 • Unit testing concentrates on each unit (i.e., component) of the software as implemented in source code • Integration testing is next, where the focus is on design and the construction of the software architecture 7
- 8. 8 Testing Strategy - 3 • Next is the validation testing, where requirements established as part of software requirements analysis are validated against the software that has been constructed • Finally, we have system testing, where the software and other system elements are tested as a whole 8
- 9. 9 Testing Strategy: Unit Testing • Unit testing makes heavy use of white-box testing techniques, exercising specific paths in a module’s control structure to ensure complete coverage and maximum error detection 9
- 10. 10 Testing Strategy: Integration Testing • Next, components must be assembled or integrated to form the complete software package • Integration testing addresses the issues associated with the dual problems of verification and program construction • BB test case design techniques are the most prevalent during integration 10
- 11. 11 Testing Strategy: Validation Testing • After software has been integrated, a set of high-order tests are conducted • Validation criteria (established during requirements analysis) must be tested • Validation testing provides final assurance that software meets all functional, behavioral, and performance requirements • BBT techniques are used exclusively here 11
- 12. 12 Testing Strategy: System Testing • System testing is the last high-order testing and it falls outside the boundaries of software engineering and into the broader context of computer system engineering • System testing verifies that all elements (hardware, people, databases) mesh properly and that overall system function/performance is achieved 12
- 13. 13 The “V” Model of Software Testing Requirements Design Coding Unit Integration Validation
- 14. 14 Strategic Issues - 1 • Specify product requirements in a quantifiable manner long before testing commences – NFRs • State testing objectives explicitly • Understand the users of the software and develop a profile for each user category – Actors/users of use-cases 14
- 15. 15 Strategic Issues - 2 • Develop a testing plan that emphasizes “rapid cycle testing” • Build “robust” software that is designed to test itself – Software should be capable of diagnosing certain classes of errors • Use effective formal technical reviews as a filter prior to testing 15
- 16. 16 Strategic Issues - 3 • Conduct formal technical reviews to assess the test strategy and test cases themselves • Develop a continuous improvement approach for the testing process 16
- 17. 17 Unit Testing - 1 • Unit testing focuses verification effort on the smallest unit of software design – the software component or module • The relative complexity of tests and uncovered errors is limited by the constrained scope established for unit testing • Different units can be tested in parallel 17
- 18. 18 Unit Testing interface local data structures boundary conditions independent paths error handling paths module to be tested test cases
- 19. 19 Unit Testing - 2 • The module interface is tested to ensure that information properly flows into and out of the program unit under test • The local data structure is examined to ensure that data stored temporarily maintains its integrity during all steps in an algorithm’s execution 19
- 20. 20 Unit Testing - 3 • Boundary conditions are tested to ensure that the module operates properly at boundaries established to limit or restrict processing • All independent paths through the control structure are exercised to ensure that all statements in a module have been executed • All error handling paths are tested 20
- 21. 21 Unit Testing - 4 • Tests of data flow across a module interface are required before any other test is initiated 21
- 22. 22 Unit Testing: Common Errors Found • Among the more common errors in computation are – Misunderstood or incorrect arithmetic precedence – Mixed mode operations – Incorrect initialization – Precision inaccuracy – Incorrect symbolic representation of expression 22
- 23. 23 Unit Testing: Test Cases • Test cases should uncover errors such as – Comparison of different data types – Incorrect logical operators or precedence – Expectation of equality when precision error makes equality unlikely – Incorrect comparison of variables – Improper or non-existent loop termination – Failure to exit when divergent iteration is encountered – Improperly modified loop variables 23
- 24. 24 Unit Testing: Error Handling • Among the potential errors that should be tested when error handling is evaluated are – Error description is unintelligible – Error noted does not correspond to error encountered – Error condition causes system intervention prior to error handling – Exception-condition processing is incorrect – Error description does not provide enough information to assist in the location of the cause of the error 24
- 25. 25 Unit Testing: Boundary Testing • Boundary testing is the last and probably the most important task of unit testing step • Software often fails at its boundaries • Test cases that exercise data structure, control flow, and data values just below, at, and just above maxima and minima are very likely to uncover errors 25
- 26. 26 • Unit testing is simplified when a component with high cohesion is designed • When only one function is addressed by a component, the number of test cases is reduced and errors can be more easily predicted and uncovered • Let’s see a unit test environment 26
- 27. 27 Unit Test Environment Module stub stub driver RESULTS interface local data structures boundary conditions independent paths error handling paths test cases
- 28. 28 Integration Testing - 1 • “If they all work individually, why do you doubt that they’ll work when we put them together?” • The problem, of course, is “putting them together” – interfacing 28
- 29. 29 Integration Testing - 2 • Data can be lost across an interface • One module can have an inadvertent, adverse affect on another • Sub-functions, when combined, may not produce the desired major function • Individually acceptable imprecision may be magnified to unacceptable levels • Global data structures can present problems 29
- 31. 31 Integration Testing - 3 • Integration testing is a systematic technique for constructing the program structure while at the same time conducting tests to uncover errors associated with interfacing • Individual components have already been unit tested and the structure is dictated by design 31
- 32. 32 Integration Testing - 4 • There is tendency, sometimes, to attempt non- incremental integration – a sort of “big bang” approach • All components are combined in advance • The entire software is tested as a whole • And chaos usually results! • We should rather use incremental integration 32
- 33. 33 Integration Testing Approaches • Top-down integration • Bottom-up integration • Sandwich testing and integration 33
- 34. 34 Top Down Integration top module is tested with stubs stubs are replaced one at a time, "depth first" as new modules are integrated, some subset of tests is re-run A B C D E F G
- 35. 35 Bottom-Up Integration drivers are replaced one at a time, "depth first" worker modules are grouped into builds and integrated A B C D E F G cluster
- 36. 36 Sandwich Testing and Integration Top modules are tested with stubs Worker modules are grouped into builds and integrated A B C D E F G cluster
- 37. Test Case Design - 1 • Any engineered product can be tested in one of two ways – Knowing the specified function that a product has been designed to perform – Knowing the internal workings of a product 37
- 38. Test Case Design - 2 • In the first case, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function • In the second case, tests can be conducted to ensure that internal operations are performed according to the specifications and all internal components have been adequately exercised 38
- 39. Test Case Design - 3 • In the first case, testing is focused on the external behavior of a software system or its various components, and we cannot see inside the components • While in the second case, testing is focused on the internal implementation, and we must see inside the component 39
- 40. 40 Testing Techniques • Black-box testing (BBT) – aka functional/behavioral testing • White-box testing (WBT) – aka structural/glass-box testing
- 41. Black-Box Testing - 1 • Black-box testing alludes to tests that are conducted at the software interface • Although they are designed to uncover errors, they are also used to demonstrate that software functions are operational, that input is properly accepted and output is correctly produced, and that the integrity of external information is maintained 41
- 42. Black-Box Testing - 2 • A block-box test examines some fundamental aspect of a system with little regard for the internal logical structure of the software 42
- 43. 43 Black-Box Testing - 3 • The inner structure or control flow of the application is not known or viewed as irrelevant for constructing test cases. The application is tested against external specifications and/or requirements in order to ensure that a specific set of input parameters will in fact yield the correct set of output values
- 44. 44 Black-Box Testing - 4 • It is useful for ensuring that the software more or less is in concordance with the written specifications and written requirements • The simplest form of BBT is to start running the software and make observations in the hope that it is easy to distinguish between expected and unexpected behavior
- 45. Black-Box Testing - 5 • This is ad-hoc testing and it is easy to identify some unexpected behavior, like system crash • With repeated executions, we can determine the cause to be related to software and then pass that information to the people responsible for repairs 45
- 46. 46 Black-Box Testing Approaches • System testing of full application • New function testing • Lab testing • Customer acceptance testing • Field (Beta) testing
- 47. White-Box Testing - 1 • White-box testing of software is predicated on close examination of procedural detail • Logical paths through the software are tested by providing test cases that exercise specific sets of conditions and/or loops • The “status of the programs” may be examined at various points to determine if the expected/asserted status corresponds to the actual status 47
- 48. 48 White-Box Testing - 2 • The test developer is privy to inner structure of the application and knows the control flow through the application, or at least knows the control if the software works correctly • It is useful for ensuring that all or at least most paths through the applications have been executed in the course of testing
- 49. White-Box Testing - 3 • Using white-box testing methods, software engineers can derive test cases that – Guarantee that all independent paths within a module have been exercised at least once – Exercise all logical decisions on their true and false sides – Execute all loops at their boundaries and within their operational bounds – Exercise internal data structures to ensure their validity 49
- 50. White-Box Testing - 4 • The simplest form of WBT is statement coverage testing through the use of various debugging tools, or debuggers, which help us in tracing through program executions • By doing so, the tester can see if a specific statement has been executed, and if the result or behavior is expected 50
- 51. White-Box Testing - 5 • One of the advantages is that once a problem is detected, it is also located • However, problems of omission or design problems cannot be easily detected through white-box testing, because only what is present in the code is tested • Another important point is that the tester needs to be very familiar with the code under testing to trace through it executions 51
- 52. White-Box Testing - 6 • Therefore, typically white-box testing is performed by the programmers themselves 52