Cloud testing: challenges and opportunities, TaaS, Integration Testing
17 likes7,579 views
The document discusses test challenges and methodologies for cloud computing. It outlines various dimensions that need to be tested for cloud platforms and applications, including elasticity, security, performance, multi-tenancy, and integration. Testing in the cloud provides advantages over traditional testing such as improved scalability, asset utilization, and reduced costs and environmental impact. Testing as a service (TaaS) is also discussed as a shared services delivery model for software testing on demand.
Cloud testing: challenges and opportunities, TaaS, Integration Testing
- 1. Test challenges and methodologies with Cloud Dr Ganesh Iyer, Progress Software Ref: Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13),
- 2. Introduction Cloud Computing and Web 2.0 technologies • Web-driven applications • Driven by browser and do not need any installation • On-demand resource availability • Faster time to market • Reduced capital and operational expenses 2 © 2013 Progress Software Corporation. All rights reserved.
- 3. Introduction Paradigm shift Seamless upgrades Sharing of resources: Multitenancy Traditionally, applications are designed first, then developed, tested and distributed Traditionally, software upgrade needs to get new version of the software, down the system and upgrade Traditionally, software development & deployment is mostly in dedicated resources 3 But with Cloud, upgrade should happen live with minimal or no down time E.g. Gmail © 2013 Progress Software Corporation. All rights reserved. With Cloud, these resources are often being shared among multiple customers Auto-elasticity: Resources ondemand Performance Disaster recovery SaaS Applications; Browser based configuration and usage New test dimensions Changes in existing dimensions Compatibility Availability and Business Continuity Security: Multitenant penetration, Identity federation management
- 4. Common Cloud Testing Dimensions Elasticity Testing Security Testing Performance Testing Resource acquisition/ Release Time Traversal vulnerability Compatibility Testing Time to deploy Accessibility testing User access/ Roles Multi-tenancy Time to Genesis Provisionin g on-the-go Load Testing for ELB 4 Identity federation management Communicati on latency over SSL Multi-tenant penetration Connectivity and reliability with 3rd parties Reliability and Availability Latency © 2013 Progress Software Corporation. All rights reserved. Globalization and Localization testing API Integration Testing Live Testing Connectivity and invocation testing Disaster recovery API load testing Live upgrade testing API security Compatibility under different situations Self-healing ability Multitenancy Availability and business continuity Multitenancy Testing Multi-tenant penetration Rigid failure containment Availability and Business continuity Risk of correlated behaviors Service transition activity analysis
- 5. Elasticity Testing Based on subscription plans, check the maximum vertical/horizontal limit Auto provisioning/freeing on-the-go Testing for Load Balancing Performance 5 Test for the impact while auto scaling Response time/Release time for provisioning of resources Load Testing of Different subscriptions © 2013 Progress Software Corporation. All rights reserved.
- 6. Elasticity and Scalability • Limitations on max objects/applications at a time • Number of applications that can be developed per platform instance Load requirements Cloud App Development Platform Load requirements Elasticity Load Testing for different usage scenarios • 100’s of administrators accessing the management Time Time 6 Load requirements Load requirements console Time Unpredictable/Predictable burst: Some tenants have a specific pattern in their usage and test for sudden expected/unexpected variations in the usage © 2013 Progress Software Corporation. All rights reserved. Time
- 7. Security Testing Implications Outsourced + Insourced External (Data Storage) Sensitive data management in the Cloud S3 Storage or any other cloud vendor storage DBaaS (OE Database) Third party services to be tested only for connectivity with over services Functionality testing for our services Single sign on account for all the services Application Security – using both the ProPaaS and third party services Multi-tenant penetration testing Proprietary (API’s) 7 De-perimeterised Traversal vulnerability Communication latency over SSL API Level and Application Security User access/Roles testing Access from different Clients to the ProPaaS platform testing From VPN, Firewall settings, Antivirus software Authentication/Authorization Identity federation management testing © 2013 Progress Software Corporation. All rights reserved. https://collaboration.opengroup.org/jericho/cloud_cube_model_v1.0.pdf
- 8. Security User access/roles Web UI Vulnerabilities and Attacks • Authentication and authorization • Identity federation management Single Sign-On • Access from different clients to the platform • VPN, firewall settings, anti-virus • User privileges • SQL Injection • URL Manipulation • Cross-site scripting • Password cracking • Hidden-field manipulation • Multi-tenancy penetration testing • Traversal vulnerability • DDoS attacks Multi-tenant Database • Data management at DB (Encryption security) • Data retention and destruction for DB: Erase and sanitize when space is reallocated 8 Other Security Concerns • Fault-injection-based testing for web-driven services (Including verification for all input fields, network interface, environment variables etc) • Fuzzy testing for web-driven services (Injecting random data into application to determine whether it can run normally under the jumbled input) • Data privacy: Custom SLA capabilities © 2013 Progress Software Corporation. All rights reserved.
- 9. Performance Testing Time to deploy Density (multi-tenancy) Reliability Testing Availability Testing To facilitate Follow the Sun advantage model Connectivity and reliability with 3rd party components from our Cloud • Reliability of the data sent between two systems 9 © 2013 Progress Software Corporation. All rights reserved.
- 10. Latency under different conditions Network • The round-trip time between the browser and the server • The number of round-trip times it takes to completely load a web page • The protocol’s flow & congestion control properties, and • Competing traffic, unreliable network Processing Client side 10 • The time it takes to the server to prepare the content that will be sent to the user. • Resource sharing introduces contention risks, increased recovery latencies • The time the web browser needs to prepare the received content to be presented • Latency when accessed from different unreliable sources (e.g. handheld devices, PDAs etc..) © 2013 Progress Software Corporation. All rights reserved.
- 11. Live Testing and Failover Testing Robustness of infrastructure against failures Live Upgrade Testing - Managing customer applications for maintenance/upgrade Recovery time in case of product/platform crash Self healing ability in case of product/platform Availability and Business continuity in multi-tenant environment 11 © 2013 Progress Software Corporation. All rights reserved.
- 12. Multi-Tenancy Testing Verifiable resource accounting for Billing • In case, multiple tenant’s data need to be captured simultaneously by the billing agent Multi-tenancy penetration testing • Tenant level access and their boundaries based on authentication and authorization Rigid failure containment between tenants • Failure because of one tenant’s action shouldn’t stall other tenant’s activities or bring down the complete system down Testing for the risk of correlated behaviors • Example: multiple application instances execute the same recovery action or periodic maintenance actions simultaneously Service transition activity analysis • 12 No service impact on other tenant instances when each and every tenant-specific configuration parameter is changed. © 2013 Progress Software Corporation. All rights reserved.
- 13. Compatibility Testing Languages Browsers Platforms Devices 13 © 2013 Progress Software Corporation. All rights reserved.
- 14. PCTF: Progress Cloud Test Framework PCTF Injection strings Test Logger Input Crawler Error parameters Injector patterns Error Pattern Manager Security Testing Test manager Injection strings & Results Pages/hyperlinks Client shell Output Repository • Parameter Test Repository: • Test suite Plug-ins configuration • Libraries: SQL string library, Error • Test suite selection Result Analyzer • Result/Log collection API Integration Testing SLA patterns library etc System Under Test (Cloud Platform/Application) Test DB (MySQL) Synthetic Load Generator • SLA mapping table • SLA metrics database SLA information Stubs for 3rd parties SLA Monitor Testing 14 Ganesh Neelakanta Iyer, Jayakhanna Pasimutu and Ramesh Loganathan "PCTF: An Integrated, Extensible Cloud Test Framework for Testing Cloud Platforms and Applications" The 13th International Conference on Quality Software 2013, (QSIC,'13), © 2013 Progress Software Corporation. All rights reserved.
- 15. Integration Testing 15 © 2013 Progress Software Corporation. All rights reserved.
- 16. What is Integration Testing? Often, many Cloud-based systems will be integrated to each other for delivering a cloud-based offering. For example, for a SaaS application, the SSO (single-sign-on) mechanism may be handled by a 3rd party system and payment mechanisms might be handled by another 3rd party system such as PayPal. Hence, we need to get sandbox environments for such payment gateways which are identitical to its production environment in order to have the systems behave same in both test as well as live environments. 16 © 2013 Progress Software Corporation. All rights reserved.
- 17. Typical issues Multiple systems in an integrated Cloud-based product behave differently and verification process for different systems differ each other. Some of the systems do not allow automated deletion of data created for testing. So it imposes a unique requirement to have unique users created every time we perform such test automation Unpredictable delays in updating various systems. Different types of environments for testing. For example, presence of web UI testing and runtime testing in one test scenario poses its own unique challenges 17 © 2013 Progress Software Corporation. All rights reserved.
- 18. Integration test automation challenges Unique requirements: • UI components • Runtime components Need to either develop a test framework that can efficiently test both run time and UI components or use a combination of two frameworks for runtime/UI testing Needs to have mechanisms to initiate the test written in one framework from the other one and to generate a combined test results Using the exposed APIs provided by other third party systems integrated as part of the product 18 © 2013 Progress Software Corporation. All rights reserved.
- 19. Integration test automation with Cloud 19 © 2013 Progress Software Corporation. All rights reserved.
- 20. TaaS 20 © 2013 Progress Software Corporation. All rights reserved.
- 21. Overview of Testing as a Service TaaS Wikipedia - Testing as a Service (TaaS, typically pronounced 'tass') is a model of software testing whereby a provider undertakes the activity of software testing applications/solutions for customers as a service on demand. …involves the ondemand test execution of well-defined suites of test material, generally on an outsourced basis. Shared Services delivery model Pay per by drink and not by Glass Standardised, Repeatable services 21 © 2013 Progress Software Corporation. All rights reserved. Courtesy: http://tinyurl.com/taasmphasis
- 22. TaaS: Conceptual Model Customer 1 Customer …n Customer 2 Fixed price per product Commercial Models SLAs Customer Service Management Pre-defined Automation Offering Service Products Move towards standardised services Service Catalogue for business Web Performance Offering Perf .Test Web/ERP Functional Test Offering Customer Interface Activities and deliverables Predefined Manual Testing Demand Management Service Catalogue for Operations Owned by service provider. Continuous Improvement Service n… Service 3 Test Assets Test Assets Service 2 Test Assets Global Delivery Model Testing Framewor k Test Assets TAAS Engine Internal Service Management Owned by service provider 22 HAAS/ Cloud In house Tools / External Tools Test Analyst Test Process High sharing of resources Courtesy: http://tinyurl.com/taasmphasis © 2013 Progress Software Corporation. All rights reserved.
- 23. Advantages of testing in the Cloud Traditional Testing Testing in Cloud Low asset utilization Scalability: Long time to increase capacity Less time (instantaneous) increase and reduction in capacity Long time to build datacenters Purchased as a service from cloud providers Difficult to manage Better management and increased productivity Duplicate test systems Aggregated system Creates unnecessary wastes 23 Improved asset utilization Cleaner, greener testing, saving in CO2 emissions © 2013 Progress Software Corporation. All rights reserved.
- 24. Conclusions Various Cloud test dimensions • Elasticity, Multi-tenancy, Security, Live Upgrade, Performance Integration Testing: A unique requirement with Cloud Automation challenges and possible approaches TaaS 24 © 2013 Progress Software Corporation. All rights reserved.