John Fodeh - Adventures in Test Automation-Breaking the Boundaries of Regression Testing - EuroSTAR 2012
0 likes734 views
The document discusses advancements in test automation, emphasizing the shift from traditional regression testing to innovative approaches like monkey testing and behavior-driven development. It highlights the benefits of automation, such as improved efficiency and early defect detection, while also noting limitations like the potential to miss obvious defects. The importance of adapting automation tools and processes to specific team needs and project scenarios is also stressed.
![Traditional Test Automation
| ©2012, Cognizant
5
Typically
Automation of regression tests
Purpose
Testing of a previously tested program following
modification to ensure that defects have not been
introduced or uncovered in unchanged areas of the
software, as a result of the changes made. It is performed
when the software or its environment is changed.
[ISTQB Glossary, v2.1]](https://image.slidesharecdn.com/w1adventuresintestautomationjf-141113103855-conversion-gate01/85/John-Fodeh-Adventures-in-Test-Automation-Breaking-the-Boundaries-of-Regression-Testing-EuroSTAR-2012-5-320.jpg)
John Fodeh - Adventures in Test Automation-Breaking the Boundaries of Regression Testing - EuroSTAR 2012
- 1. Adventures in Test Automation – Breaking the Boundaries of Regression Testing John Fodeh, Cognizant Technology Solutions www.eurostarconferences.com @esconfs #esconfs Insert speaker picture here, no more than 150x150 pixels
- 2. Adventures in Test Automation Breaking the Boundaries of Regression Testing John Fodeh Cognizant Technology Solutions [email protected]
- 3. Outline • Innovation and testing • The promise of automation • Going beyond regression testing • Automated Test Monkeys • Model-Based and Behaviour-Driven approaches • The human factor in automation | ©2012, Cognizant 3
- 4. The Promise of Test Automation • Replacing repetitive and tedious manual tasks • Ensuring the consistency and repeatability of tests • Performing tests that are difficult to run manually • Accelerating test execution | ©2012, Cognizant 4 * Image source: Rosemont Engineering
- 5. Traditional Test Automation | ©2012, Cognizant 5 Typically Automation of regression tests Purpose Testing of a previously tested program following modification to ensure that defects have not been introduced or uncovered in unchanged areas of the software, as a result of the changes made. It is performed when the software or its environment is changed. [ISTQB Glossary, v2.1]
- 6. Characteristics of Traditional Test Automation • Static • Simple • Synchronized • Vulnerable | ©2012, Cognizant 6
- 7. Monkey Testing | ©2012, Cognizant 7 Monkey Testing refers to the process of randomly exercising a software program by means of an automated test tool
- 8. Test Monkey Action List | ©2012, Cognizant 8 #script file: Test4.asc CP_KEY FOCUS_UP; TRACKBALL 0, 64; KEY L; KEY 5, LONG; KEY 8; CP_KEY FOCUS_UP; Random Test Tool #script file: Test4.asc KEY L; KEY 5, LONG; CP_KEY FOCUS_UP; #KEY BOX_SIZE_UP KEY 8; TRACKBALL 0, 64; KEY SHIFT, R; Log file Application Under Test KEY FOCUS_UP; Utilities
- 9. Basic Features A test monkey should: • Select randomly from input range • Enter input to AUT through the user interface • Detect “life signs” of AUT • Have robust logging facility • (Re-) start and initialize AUT • Have limited application knowledge but general environment awareness | ©2012, Cognizant 9
- 10. Using Test Monkeys • Indicator of application reliability • Incorporated into daily “smoke” test • Establish confidence in application • System test tool • Evaluation of release candidates • Previously failed scripts are run again as part of regression tests • Focus tests on specific areas of interest • Combinations of sequential and random scripts • Random scripts with weight on specific actions | ©2012, Cognizant 10
- 11. Metrics – Application Reliability | ©2012, Cognizant Stability Trend for 4101 120000 100000 80000 60000 40000 20000 0 Release Criterion 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 Build no. Mean number of random operations between Failures Random operations Start Criterion Release Stability Criterion 11 Entry Criterion
- 12. Added Value (1/2) • Early testing • Cost effective • Negative testing • Long and complex test runs • Stability indicator | ©2012, Cognizant 12
- 13. Limitations • Misses obvious defects • Does not emulate real use situations • Long tests runs can be difficult to debug • For a reliable statistical basis, long and multiple test runs are needed | ©2012, Cognizant 13 Monkey Testing is a supplement to your manual and automated testing - not a replacement!
- 14. Advanced Test Monkeys • Wide application knowledge • Application modeled in state table • Illegal input to test error handling and recovery • Emulate real and complex use scenarios • Effective in finding defects • Higher development and maintenance cost | ©2012, Cognizant 14
- 15. Test Monkey | ©2012, Cognizant 15 Random Test Tool State table Log file Application Under Test Utilities Start State Action End State Script Start state Action Probability Input range End state Scanning Adjust frequency 45 3 20 Scanning Adjust brightness 20 1 100 Scanning Abort 5 Displaying (abort) Edit preferences 10 Editing preferences Increment depth 20 Scanning
- 16. Intelligent Randomness • Probability Tables • Focus on selected areas • Weight on specific use scenarios • Continuous expansion and improvement • Possibility to exclude certain parts | ©2012, Cognizant 16 Enter Text Select Text Delete Copy Cut … Task n Action 25% 10% 5% 5% 5% * source: Whitmill, Kelly
- 17. Model-Based Test Monkeys MStoadteel table Script | ©2012, Cognizant 17 Random Test Tool Log file Application Under Test Utilities Start State Action End State
- 18. Next Step - Behavior-Driven approach • Based on Behavioral specifications • User Stories defined using ubiquitous language • Principles of Test Driven Development • Collaborative approach for business analysts, Software development & testing • Shared tools and process | ©2012, Cognizant 18
- 19. People Issues • Applying test automation involves people • Your team members are not interchangeable machine parts • Take that into account when implementing test automation | ©2012, Cognizant 19
- 20. Summary • No “one size fits all” solution • The tools, processes and skills are there • Maturity is about analyzing the current practice and finding ways to do evolve • Are you ready for your innovation journey? | ©2012, Cognizant 20