DevOps 2017 Conf: evolving from automated to continuous
4 likes338 views
Arthur Hicken discusses evolving from automated to continuous testing for agile and DevOps. Continuous testing requires a mature infrastructure and process with highly automated testing. It defines quality gates that produce binary pass/fail results. Automation and static analysis are critical to eliminate defects early and ensure software can pass through quality gates. Continuous testing supports continuous delivery by providing feedback to refine processes and tradeoffs between release scope, timeline, and quality.
DevOps 2017 Conf: evolving from automated to continuous
- 1. Evolving from Automated to Continuous Testing for Agile and DevOps DevOps Remote Conference 2017 Arthur “Code Curmudgeon” Hicken
- 2. Intro Arthur Hicken is Chief Evangelist at Parasoft where he has been involved in automating various software development and testing practices for over 20 years. He has worked on projects including cybersecurity, database development, the software development lifecycle, web publishing and monitoring, and integration with legacy systems and maintains the IoT Hall-of-Shame http://bit.ly/iotshame Follow him @codecurmudgeon Blog: http://codecurmudgeon.com Web: http://parasoft.com 2© 2017 Parasoft Corporation Perfecting Software
- 3. Agenda Today’s challenges What IS Continuous? Barriers Best practices Role of automation Role of static analysis!? 3© 2017 Parasoft Corporation Perfecting Software
- 4. Normal Situation • Pressure to deliver software quickly • Pressure to reduce defect rate • Late stage “bug-hunting” • No time for error prevention • Quality can be rushed • Security is forgotten 4© 2017 Parasoft Corporation Perfecting Software
- 5. How to you know a build is successful? • When it compiles? • When all unit-tests have run? • When the right coverage goal is achieved? • When the right % of test failures occurs? • When it’s deployed? 5© 2017 Parasoft Corporation Perfecting Software
- 6. Continuous • Build, Integration, Testing, Release, Delivery, Deployment • Builds on and enables DevOps • High dependency on automation: • Processes • Assessment • Decisions 6© 2017 Parasoft Corporation Perfecting Software
- 7. Feature Release Business Stakeholder Developer Customer Support Software Feature Continuous measurements mean continuous refinement of the process Real-time feedback from Objective assessment and go/no go Defects are eliminated at the point that they are easiest to fix Quality gates: Organizations can automatically promote software through quality gates when business expectations have been met 7© 2017 Parasoft Corporation Perfecting Software Continuous Testing is Beyond Automation
- 8. Challenges of Continuous • Tests must produce binary decision go/no-go • Reuse unit test and functional tests from dev to QA • High level of automation • Requires disciplined mature process • Testing must automatically answer • Is it stable • Will it do what it’s supposed to do 8© 2017 Parasoft Corporation Perfecting Software
- 9. Importance of Testing • How do you know when you're done? • How do you know if a fix for a minor bug broke a major function of the system? • How can the system evolve into something more than is currently envisioned? • Testing, both unit and functional, needs to be an integrated part of the development process. 9© 2017 Parasoft Corporation Perfecting Software
- 10. Continuous Testing re-positions the question from “are you done testing?” to “is the level of risk understood and accepted?” Continuous Testing Risk Assessment Policy Analysis Requirements Traceability Test Environment Access Test Optimization Advanced Analysis Ensure access to complete test environments Automate defect prevention and Policy measurement Expand test coverage and measure test effectiveness Define actionable practices Connect functional with non-functional requirements Process improvement opportunities 10© 2017 Parasoft Corporation Perfecting Software Elements of Continuous Testing
- 11. Prerequisites Tools • Version control • Build server • Deployment server • CI tools • Automation tools (test, etc) Process • Commit/update often (each change) • Always create tests (pass and fail) • Test regularly • Run regularly 11© 2017 Parasoft Corporation Perfecting Software
- 12. Continuous requires: Proper infrastructure A binary definition of “done” Reliable quality gates Extreme automation 12© 2017 Parasoft Corporation Perfecting Software
- 13. Infrastructure • Real requirements system • Flexible developer level project/scrum management system • Automated build/testing system • Data collection that can answer • Is it done? • When will it BE done? • Is it good enough to release 13© 2017 Parasoft Corporation Perfecting Software
- 14. Defining Done • Are requirements coded? • Do tests exist for the requirements and code? • Are the tests passing • To your satisfaction (often not 100%) • Will the tests be enough? 14© 2017 Parasoft Corporation Perfecting Software
- 15. Defining Quality • Consistent • Repeatable • Meaningful Automated objective measure • Unit tests • Functional tests • Regressions tests • Static analysis Made up of 15© 2017 Parasoft Corporation Perfecting Software
- 16. Effective Continuous Testing • Avoid manual end-to-end testing • Invest more into automated tests • Follow testing pyramid rules • Measure: • test results • test effectiveness • Determine: • Risk 16© 2017 Parasoft Corporation Perfecting Software
- 17. How to do it better ? • Give absolute priority to automated tests • Invest time into designing interfaces (API) • Use API (service) testing tools to cover interfaces • Measure the quality of the test 17© 2017 Parasoft Corporation Perfecting Software
- 18. How Can Static Analysis Help? Coding standards Quality gate (bug detection) Prevention 18© 2017 Parasoft Corporation Perfecting Software
- 19. Bug Gates • Binary Decisions • No “bugs” • No “critical” static analysis findings 19© 2017 Parasoft Corporation Perfecting Software
- 20. Quotable Quotes • An ounce of prevention is worth a pound of cure. – Benjamin Franklin • Cease dependence on inspection to achieve quality. Eliminate the need for inspection on a mass basis by building quality into the product in the first place. – Deming • Simply finding bugs with static analysis isn’t enough. • Avoid risky behavior 20© 2017 Parasoft Corporation Perfecting Software
- 21. Static Analysis Continuous Feedback Loop Identify Error Isolate Root Cause Fix New Rule to Prevent Monitor § Code review § Regression § QA § Field bugs § Post-mortem 21© 2017 Parasoft Corporation Perfecting Software
- 22. Main Points for Today • We must define “all” quality expectations upfront • We must change the primary goal of testing • We must accept that testing is only a task in the quality process • We must be able to measure trust of the process • We must automate breaking the application 22© 2017 Parasoft Corporation Perfecting Software
- 23. Speeding up the conveyor belt does not yield better results… Modern DevOps must embrace systematic process improvement— Focused on testing DevOps is Continuous Improvement 23© 2017 Parasoft Corporation Perfecting Software
- 24. How to Achieve Quality @ Speed • Business risks drive quality activities • Teams collaborate on risk definition • All team members trained on risks • Acceptance criteria visible and measured • DevTest team activities prioritized per risk definition • Results of quality activities are visible and translated for all levels of business 24© 2017 Parasoft Corporation Perfecting Software
- 25. Why Continuous Testing? Systematic, centralized decision making to factor business risk into SDLC Safety net to allow developers to bring innovations to market faster …and automation to break the application A feedback system for better trade-off decisions between release scope, time and quality Continuous Testing Mitigates Business Risk 25© 2017 Parasoft Corporation Perfecting Software
- 26. Enabling Technologies Stubs Service Virtualization API testing Test data management Environment management Self-service test environments 26© 2017 Parasoft Corporation Perfecting Software
- 27. How to do it better? • Use service virtualization to improve testing automation • Isolate at the message layer • Simulate functional scenarios and performance conditions 27© 2017 Parasoft Corporation Perfecting Software
- 28. The Test Environment Challenge Test environment access is outside the control of development and test leaving gaps in the process Parallel development delays… Need simple, realistic access to dependent components… Too much time waiting for access…Need reliable test data Need a realistic test environment easy to maintain Scheduling Configuration Access Limits Dependent Applications Staged Assets 3rd Party Assets Virtual Environments Hyper Visor App App App IT Operations 28© 2017 Parasoft Corporation Perfecting Software
- 29. • The Challenge • Multiple teams using the same test database • Teams not respecting data integrity & others test data records • Regression tests consistently failing. Takes >1 hour to determine that it was due to “data changes”. • “Real problems” were getting lost in the noise Shared Database Eliminated 83% of configuration time for a major telecom company Test data management for complex transactions ✔ ✖ ✖ 29© 2017 Parasoft Corporation Perfecting Software
- 30. • The Solution • Setup Virtual Assets to model the SQL queries and use API testing tool to manage automated nightly regressions against both virtual assets and live systems • The Business Benefit • Test teams able to focus on ‘real regressions’ and separate out data integrity issues from functional test failure Virtual Asset Virtual Asset Virtual Asset ✔ ✔ ✔ 30© 2017 Parasoft Corporation Perfecting Software Eliminated 83% of configuration time for a major telecom company
- 31. Summary • Continuous * requires mature infrastructure and process • Quality gates must produce a binary answer • Automation is everything • Policy is critical, because automation is critical • Static analysis gets you to the gate 31© 2017 Parasoft Corporation Perfecting Software
- 32. Thank you! 32 Continuous Testing Book: http://alm.parasoft.com/continuoustestingbook © 2017 Parasoft Corporation Perfecting Software