Best Practices In Load And Stress Testing Cmg Seminar[1]
Download as PPT, PDF2 likes1,358 views
The document discusses best practices for performance testing. It provides an overview of the typical performance testing process, including defining goals, planning tests, scripting tests, executing tests, analyzing results, and delivering findings. It also discusses considerations for choosing testing tools and resources as well as common pitfalls to avoid, such as not testing, poor planning, relying on customers to find issues, using the wrong tools, and failing to properly isolate variables.
Best Practices In Load And Stress Testing Cmg Seminar[1]
- 1. Best Practices in Performance Testing Jennifer Turnquist Storage Service Line Director Lionbridge Technologies
- 2. Lionbridge Profile Public Company (Nasdaq: LIOX) Nearly $400M+ in revenues Profitable Deep expertise across the application life cycle Application Development & Maintenance Testing (Independent V&V) Content Development, Conversion & Enhancement Globalization Worldwide scale and capability Over 4000 employees operating in 25 countries (Scale) SEI CMM Level 5 certified process model (Quality) 8 of the world’s 10 most valuable companies are Lionbridge customers BusinessWeek Global 1000, July 2004
- 3. Services Designed around our Client’s Need Lionbridge A Trusted Partner Around the World Global Development & Testing Solutions Global Language & Content Solutions Interps Software Development Lifecycle Application Development Testing & Certification Maintenance & Support Full Content Lifecycle Localization/Translation Technical Publications eLearning Courseware Off-shore platforms leverage more than staff in China, India, and Eastern Europe Global footprint enables local interaction and facilitates worldwide release and support Trusted, US-based public company protects against IP loss Localization services spanning more than 80 languages Proprietary web-architected TM and terminology solution accelerates production and improves consistency Authoring and eLearning development services integrate seamlessly with localization to address global demand
- 4. VeriTest: Setting the Standard in Testing Since 1987 World’s largest independent testing company Over 400 test architects, engineers, and analysts in 11 labs across US, Europe, Asia Rapid expansion in VeriTest India From PDAs and PCs to 32-way servers Data center class storage lab Industry leader Exclusive provider and architect of industry- leading certification programs Developer of PC Magazine benchmarks Test and publish industry standard ISP benchmarks Operate globally-networked onsite to offshore model
- 5. The Lionbridge Team Local Connections, Global Efficiency 4,000+ Worldwide Staff Experience and Efficiency
- 6. Today’s Agenda Why Test Performance? Different Types of Performance Testing Performance Testing Roadmap Choosing the Right Testing Tools Top 10 performance testing pitfalls
- 7. "The standard philosophy of 'test to destruction'... will probably give you an idea of roughly how many users your site can handle at once, but it won't always tell you why the site fails to function properly. And without knowing why, you're not likely to be able to do much about it..." -- Extreme Tech
- 8. Why Test Performance? The internet and IT infrastructure crucial to business Users—employees, business partners, customers—rely on portals, applications, and data to do their jobs Cost of failure can be devastating Performance testing in the enterprise is intermittent, cyclical, often prompted by upgrades Testing is highly specialized
- 9. The high cost of not conducting performance testing Performance testing overlooked until disaster strikes Lost and abandoned sales - most visible result of poor performance testing but… Efficiency of mission-critical systems directly impacts business productivity Preventing problems—lost productivity, lost business, lost reputation, and even injury or death—is a major incentive Knowing the vital performance metrics = ammunition to IT departments when planning and justify purchasing decisions Provides the ability to demonstrate to investors and other critical stakeholders that the company’s infrastructure is adequate
- 10. Events that trigger performance testing Build vs. buy Evolving requirements Technology due diligence Consolidating servers Deploying a SAN Deploying or upgrading enterprise application Migrating to a new platform Addition of features Response to public critique Enhancements due to buying trends Acquiring or merging a business Launching new product Enhancing web application Promoting an offering Doing any of the above globally
- 11. Performance testing is not a one time event.
- 12. Conduct the right test to get the right results Load Testing Determines the response time and throughput during typical user load Stress Testing Determines the peak user load Volume Testing Determines the problems that occur during long-term user activity Component Testing Determines the performance and behavior of a specific component Benchmark Testing Measures the performance of a system or component relative to a standard Transaction Cost Analysis Determines the system resources consumed by a single transaction
- 13. Performance Testing Roadmap Define Communicate Test Script Plan Identify stakeholders Agree on goals of testing Determine budget Determine schedule constraints Agree to promotion strategy Outline resources available Determine staffing plan Engage test lab (if needed) Verify basic functionality Generate use cases Capture user activity logging information Analyze user activity profile Model user activity Choose the tool(s) Identify re-usable script components Assign resources needed for scripting, testing Create test environment Design scripts Create scripts Validate scripts Build script library Execute tests Collect data Analyze test results Run possible iterations Troubleshoot bottlenecks Tune system Retest Log non-performance failures Outline context Draft results Provide feedback to stakeholders Deliver action items Finalize report(s) Promote results
- 14. An overview of the performance testing process After initiating the test, the load generator systems to begin accessing the system under test using the designed usage patterns. Depending on whether the test is a global, local, or isolated configuration, the load generators may be located worldwide or completely contained within a test lab. The one critical configuration requirement for the load-generating systems is that they have adequate network bandwidth throughput capability to access the system under test in a realistic manner without bandwidth constraints. If bandwidth constraints become a problem, adding additional load generators to the pool of load generators will typically fix this problem. If the test is global or local, the Internet will be an important factor in the configuration. For an isolated configuration, the Internet is not a factor.
- 15. An overview of the performance testing process Once a performance test is initiated, it can run for several minutes to several days, depending on the test goal. During the test time, the test tool monitors and collects performance data from all of the components within the system under test, such as the Web server, application server, or database server. All of the monitor data along with the performance test data collected at the generating client end to determine the overall performance as well as the potential system bottlenecks. In a typical performance test cycle, the performance bottlenecks are located, fixed, and iteratively retested to ensure that they are fixed as designed.
- 16. High Level Picture of the Process Overcome resource limitations • Replace testers with “Virtual Users” • Run many Virtual Users on few machines • Controller manages Virtual Users • Run repeatable tests with scripted actions • Get meaningful results with analysis tools System under Test Load Generation
- 17. You don’t have to go it alone Define Communicate Test Script Plan Build and train internal resources Hire contractors Utilize service offerings from test tool vendors Rely on application provider Engage with a consulting firm or SI Partner with an independent testing company
- 18. Important considerations for choosing the resources Deadlines Testing skills and experience Technology and/or application expertise Frequency and scale of testing requirements Infrastructure requirements Risk assessment Market factors
- 19. The vast number of performance testing tools can be overwhelming
- 20. Important considerations for choosing the right tool Do you already own the license? Do you have the internal resources to script and execute? Will it meet the test objectives? Is it compatible with your technology objectives? Does it fit within your budget constraints? Do you have the training and expertise to analyze the results? Does it match the frequency of your testing needs?
- 21. Leading performance tools Expensive hardware to purchase Uses SST TracePlus to provide record and playback feature No additional hardware required Spirent Avalanche/Reflector Expensive license Requires a unique license for each protocol type Compatible with numerous protocols Excellent data analysis tools WAN emulation Web transaction breakdown monitor Mercury Interactive LoadRunner Windows only Expensive license Offer “Lite” version for reduced price Excellent data analysis tools Root cause analysis tools included Segue SilkPerformer Covers few protocols; primarily Web-based automation Inexpensive license Good data analysis tools RadView WebLoad Cons Pros Benchmark Tool
- 22. Popular Benchmark Tools block level data transfer, OLTP (database) traffic block level data transfer terminal services traffic IOMeter IOZone TSScaling Other Industry Standard Tools SPECjbb SPECsfs SPEC Java Business Benchmark WCAT WMLS Microsoft LOADSIM DBHammer VeriTest NetBench VeriTest WebBench Benchmark Tool SPEC benchmarks Microsoft Tools VeriTest Tools Exercises the CPU NFS File / Network Traffic HTTP(S) traffic streaming media Exchange email traffic SQL database traffic CIFS file / network traffic HTTP / Web traffic Workload Simulated
- 23. Manual testing may be your best tool Frequency of testing requirements Rate of change Limitations of available tools
- 24. Top 10 performance testing pitfalls
- 25. Top 10 Not Testing. Lack of clearly defined test objectives and poor planning. Relying exclusively on beta customers to find performance issues. Using the wrong tool for the job. Introducing too many variables simultaneously into a test. Failing to test how your product or system is actually used. Conducting load testing in a vacuum. Treating performance testing like a one-time event. Assume that the scripting effort will be short and simple. Finding functional bugs during performance runs.
- 26. Conclusion Companies rely on systems to conduct business efficiently and effectively Performance testing ensures that your users are getting reliable and timely access to the resources they need Performance testing mitigates the risk of lost time and money due to poor performance A fully integrated performance testing program is the preventative medicine that keeps your system from becoming an inaccessible and costly resource. Though it may seem counterintuitive at first to slow your deployment for performance test planning and execution, the payoff in time, money, and quality will be big and will come soon.
- 27. Thank You