SlideShare a Scribd company logo
D I S C I P L I N E D I N N O VAT I O N
TEST PERFORMANCE USING CLOUD
Abhinav Maheshwari (Chief Scientist & Director, 3Pillar Labs)
Sayantam Dey (Architect, 3Pillar Labs)
OBJECTIVES
1. What are the benefits of using the cloud for
testing performance over traditional means?
2. What are the various tools available for
cloud based performance testing?
3. How can we use open source load testing
frameworks in the cloud?
4. How to design a custom framework for rapid
testing with large number of concurrent
users?
TRADITIONAL PERFORMANCE TESTING
Load Generation
Server (High end or
multiple servers)
Application Server
Workstation
1. Deploy replica of the
production environment
in the LAN
2. Deploy a high-end
server for generating
load (on the same LAN)
3. Run agents of a load
test tool like JMeter,
Grinder or WAPT
4. Connect remotely to
load generation server
5. Load generation server
collects metrics and
generates reports
CLOUD BASED PERFORMANCE TESTING
Load Generation
Server (High end or
multiple servers)
Application Server
Workstation
Cloud machines
in different
geographies
Application Server
Workstation
BENEFITS OF USING THE CLOUD
Traditional Approach Cloud based Approach
Provisioning physical machines is costly,
specially for medium sized companies
Large number of small, virtual machines
running in bursts, ideal usage of cloud
from a cost perspective
Manual provisioning of load generation
server requires time and effort
Simpler provisioning, can even be
automated using API
Application needs to be deployed in the
same LAN
Application may be hosted at any public IP,
also more realistic from latency point of
view
Not dynamic to address any significant
increase in the performance requirement
Distribute load generation over multiple
instances which can be increased
dynamically
Extremely difficult to simulate
geographically distributed load
Cloud clusters in different geographic
locations can simulate geographically
distributed load





CLOUD APPROACH – CHOICES ?
Commercial Products
1 Record or upload scripts
for usage scenarios using
the web based interface
2 Configure testing
parameters like number of
virtual users, time for test,
etc. through interface
3 Start the test through web
based interface
4 Download reports and
analyze
Open source frameworks
1 Create the required number of VM
instances on the cloud, deploy the chosen
load testing framework
2 Configure all the parameters for the
framework and upload scripts written for
the chosen framework
3 Run agents on the system under test and
then start the test
4 Download reports and analyze
CLOUD APPROACH – CHOICES ?
Commercial Products
1 Record or upload scripts for usage
scenarios using the web based interface
provided by the vendor
2 Configure testing parameters like number
of virtual users, time for test, % allocation
to usage scenarios etc. through interface
3 Start the test through web based interface
4 Download reports and analyze
Open source frameworks
• Very popular, large user base
• Very little programming
knowledge required
• Handles many protocols
• Plugin architecture for
extension
• Requires programming
• Scripting in Jython or Clojure
• Distributed framework
CLOUD APPROACH – CHOICES ?
Commercial Products
• JMeter compatible
• Real time reports
• 10K virtual users x 25 hours
@ $1000
Web Performance
• Selenium IDE scripts
• Real browser or simple
request playback (virtual
user)
• 10K virtual users x 25 hours
@ $3750
• Custom scripts
• Real time analytics
• Custom pricing
Open source frameworks
Only few frameworks are mature enough
to work over the cloud
Only the cost of cloud infrastructure, which
is much less expensive
Completely flexible as scripting is done
through open APIs
Can only be used through command line,
user has to care about calculating VM
requirement, creation and closure
Reports are basic and sometimes not
helpful
A CUSTOM FRAMEWORK?
Commercial Products
Not many choices since there are only a
few products in the market
May prove to be expensive for repeated
load testing
May use recording or proprietary scripting,
which may not be fully flexible
Provide web-based interface to directly run
tests without worrying about virtual
machines
Provide easy to use reports


 X

Sizable investment - Suitable for organizations needing repeated performance tests
CASE STUDY - HAILSTORM
Configuration
 JMeter
 Amazon
 SUT Provisioning Manager
JMeter
Scripts
Script Manager
JMeter Controller
Data Collector
Report Engine
- JFreeChart
AMI
- JMeter
- JRE
Agent
System Under Test (SUT)
Agent
Agent
H
T
T
P
CASE STUDY - HAILSTORM
Observations
Sharp rise in response time
for over 50 users
“Detail Page” degrades
significantly for 75 users and
above
System is stable but peaks
out at 75 concurrent users
and starts degrading at 100
users
Maximum throughput of 3.2
transactions per second at 75
users is low
© Copyright 2012 3PILLAR GLOBAL All rights reserved 11
Sample Report Summary
CASE STUDY - HAILSTORM
© Copyright 2012 3PILLAR GLOBAL All rights reserved 12
Page Response
time Avg.
(ms)
Response
time 90tile
(ms)
% errors
Home Page 1384 2293 0.0
Login Page 559 764 0.0
Login 1050 1740 0.0
Browse 1996 2898 0.25
Category
Browsing
2553 3601 0.0
Detail Page 5368 7231 0.27
Search 2054 2948 0.0
Sample Report for one iteration
CASE STUDY - HAILSTORM
• Application
– A leading provider of online classroom and instruction
solutions
• Key Challenge
– Required simulation of 40,000 concurrent users
– Rapid and cost effective solution to capture load
metrics for large number of concurrent users
• The Solution
– Different JMeter scripts/plans seamlessly integrated
– Engaging reports pointing out bottlenecks generated

More Related Content

PDF
Using JMeter for Performance Testing Live Streaming Applications
PPTX
Prometheus (Prometheus London, 2016)
PPTX
Apache JMeter - A brief introduction
PDF
AWS Cloud Practitioner Tutorial | Edureka
PPTX
App Modernization with Microsoft Azure
PPTX
Reinventing Application Performance Testing with Service Virtualization
PDF
Cloud-based Performance & Web Load Testing
PPTX
QSpiders - Introduction to HP Load Runner
Using JMeter for Performance Testing Live Streaming Applications
Prometheus (Prometheus London, 2016)
Apache JMeter - A brief introduction
AWS Cloud Practitioner Tutorial | Edureka
App Modernization with Microsoft Azure
Reinventing Application Performance Testing with Service Virtualization
Cloud-based Performance & Web Load Testing
QSpiders - Introduction to HP Load Runner

What's hot (20)

PDF
Monitoring with Prometheus
PPTX
Migrating from IBM API Connect v5 to v2018
PDF
DevOps Open House III - Kubernetes using YAML
PDF
Testing Microservices
PPTX
Azure integration services from the IT Professional perspective
PPTX
Toronto Virtual Meetup #7 - Anypoint VPC, VPN and DLB Architecture
PPTX
Load Testing and JMeter Presentation
PPTX
Azure functions
PPTX
Azure kubernetes service (aks)
PDF
Running distributed tests with k6.pdf
PDF
Loadrunner vs Jmeter
PDF
What is Performance Testing?
PDF
Infographic: Importance of Performance Testing
PPTX
IBM Cloud Pak for Integration 2020.2.1 installation
PPTX
Introduction to Azure monitor
PPTX
Introduction to Kubernetes
PDF
Cloudhub 2.0
PPTX
Application Performance Monitoring (APM)
PPTX
Load and performance testing
PPT
Taking Testing to the Cloud
Monitoring with Prometheus
Migrating from IBM API Connect v5 to v2018
DevOps Open House III - Kubernetes using YAML
Testing Microservices
Azure integration services from the IT Professional perspective
Toronto Virtual Meetup #7 - Anypoint VPC, VPN and DLB Architecture
Load Testing and JMeter Presentation
Azure functions
Azure kubernetes service (aks)
Running distributed tests with k6.pdf
Loadrunner vs Jmeter
What is Performance Testing?
Infographic: Importance of Performance Testing
IBM Cloud Pak for Integration 2020.2.1 installation
Introduction to Azure monitor
Introduction to Kubernetes
Cloudhub 2.0
Application Performance Monitoring (APM)
Load and performance testing
Taking Testing to the Cloud
Ad

Viewers also liked (9)

PPT
Performance Testing Strategy for Cloud-Based System using Open Source Testing...
PPTX
Cloud based Performance Testing
PDF
Testing Framework on AWS Cloud - Solution Set
PPTX
Cloud based testing
PPTX
Cloud Testing
PDF
Agile Testing in the Cloud
PPTX
Cloud Strategy
PPTX
Cloud Migration Strategy Framework
PDF
Cloud Migration Cookbook: A Guide To Moving Your Apps To The Cloud
Performance Testing Strategy for Cloud-Based System using Open Source Testing...
Cloud based Performance Testing
Testing Framework on AWS Cloud - Solution Set
Cloud based testing
Cloud Testing
Agile Testing in the Cloud
Cloud Strategy
Cloud Migration Strategy Framework
Cloud Migration Cookbook: A Guide To Moving Your Apps To The Cloud
Ad

Similar to Cloud-based performance testing (20)

PPT
Testing using load runner performance testing
PDF
Modernizing Testing as Apps Re-Architect
PPTX
Cloud Performance Testing with LoadRunner
PDF
VMware End-User-Computing Best Practices Poster
PDF
LRITC introduction
PPTX
Silk Performer Presentation v1
PDF
TechTalk_Cloud Performance Testing_0.6
PDF
Build cloud native solution using open source
PPTX
Madrid meetup #7 deployment models
PPTX
Performance testingfromthecloud_usingBlazemeter
PPTX
V mware v fabric 5 - what's new technical sales training presentation
PDF
Pure Systems Patterns of Expertise - John Kaemmerer and Gerry Kovan, 11th Sep...
PDF
Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...
PPTX
Copy of Silk performer - KT.pptx
PDF
LoadRunner Performance Testing
PPTX
Load Runner
PPT
Advanced Load Runner
PPTX
Harbour IT & VMware - vForum 2010 Wrap
PDF
Top Performance Testing Tools of 2025: Ensure Speed, Stability, and Scale
PDF
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver
Testing using load runner performance testing
Modernizing Testing as Apps Re-Architect
Cloud Performance Testing with LoadRunner
VMware End-User-Computing Best Practices Poster
LRITC introduction
Silk Performer Presentation v1
TechTalk_Cloud Performance Testing_0.6
Build cloud native solution using open source
Madrid meetup #7 deployment models
Performance testingfromthecloud_usingBlazemeter
V mware v fabric 5 - what's new technical sales training presentation
Pure Systems Patterns of Expertise - John Kaemmerer and Gerry Kovan, 11th Sep...
Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...
Copy of Silk performer - KT.pptx
LoadRunner Performance Testing
Load Runner
Advanced Load Runner
Harbour IT & VMware - vForum 2010 Wrap
Top Performance Testing Tools of 2025: Ensure Speed, Stability, and Scale
VMworld 2013: How to Replace Websphere Application Server (WAS) with TCserver

Recently uploaded (20)

PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
PDF
Encapsulation theory and applications.pdf
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
PDF
Mushroom cultivation and it's methods.pdf
PDF
Diabetes mellitus diagnosis method based random forest with bat algorithm
PPTX
SOPHOS-XG Firewall Administrator PPT.pptx
PDF
Encapsulation_ Review paper, used for researhc scholars
PDF
August Patch Tuesday
PPTX
1. Introduction to Computer Programming.pptx
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
PDF
A comparative analysis of optical character recognition models for extracting...
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
PDF
Assigned Numbers - 2025 - Bluetooth® Document
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
PDF
Heart disease approach using modified random forest and particle swarm optimi...
PPTX
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Encapsulation theory and applications.pdf
Agricultural_Statistics_at_a_Glance_2022_0.pdf
Mushroom cultivation and it's methods.pdf
Diabetes mellitus diagnosis method based random forest with bat algorithm
SOPHOS-XG Firewall Administrator PPT.pptx
Encapsulation_ Review paper, used for researhc scholars
August Patch Tuesday
1. Introduction to Computer Programming.pptx
Advanced methodologies resolving dimensionality complications for autism neur...
A comparative analysis of optical character recognition models for extracting...
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Assigned Numbers - 2025 - Bluetooth® Document
Per capita expenditure prediction using model stacking based on satellite ima...
Heart disease approach using modified random forest and particle swarm optimi...
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...

Cloud-based performance testing

  • 1. D I S C I P L I N E D I N N O VAT I O N TEST PERFORMANCE USING CLOUD Abhinav Maheshwari (Chief Scientist & Director, 3Pillar Labs) Sayantam Dey (Architect, 3Pillar Labs)
  • 2. OBJECTIVES 1. What are the benefits of using the cloud for testing performance over traditional means? 2. What are the various tools available for cloud based performance testing? 3. How can we use open source load testing frameworks in the cloud? 4. How to design a custom framework for rapid testing with large number of concurrent users?
  • 3. TRADITIONAL PERFORMANCE TESTING Load Generation Server (High end or multiple servers) Application Server Workstation 1. Deploy replica of the production environment in the LAN 2. Deploy a high-end server for generating load (on the same LAN) 3. Run agents of a load test tool like JMeter, Grinder or WAPT 4. Connect remotely to load generation server 5. Load generation server collects metrics and generates reports
  • 4. CLOUD BASED PERFORMANCE TESTING Load Generation Server (High end or multiple servers) Application Server Workstation Cloud machines in different geographies Application Server Workstation
  • 5. BENEFITS OF USING THE CLOUD Traditional Approach Cloud based Approach Provisioning physical machines is costly, specially for medium sized companies Large number of small, virtual machines running in bursts, ideal usage of cloud from a cost perspective Manual provisioning of load generation server requires time and effort Simpler provisioning, can even be automated using API Application needs to be deployed in the same LAN Application may be hosted at any public IP, also more realistic from latency point of view Not dynamic to address any significant increase in the performance requirement Distribute load generation over multiple instances which can be increased dynamically Extremely difficult to simulate geographically distributed load Cloud clusters in different geographic locations can simulate geographically distributed load     
  • 6. CLOUD APPROACH – CHOICES ? Commercial Products 1 Record or upload scripts for usage scenarios using the web based interface 2 Configure testing parameters like number of virtual users, time for test, etc. through interface 3 Start the test through web based interface 4 Download reports and analyze
  • 7. Open source frameworks 1 Create the required number of VM instances on the cloud, deploy the chosen load testing framework 2 Configure all the parameters for the framework and upload scripts written for the chosen framework 3 Run agents on the system under test and then start the test 4 Download reports and analyze CLOUD APPROACH – CHOICES ? Commercial Products 1 Record or upload scripts for usage scenarios using the web based interface provided by the vendor 2 Configure testing parameters like number of virtual users, time for test, % allocation to usage scenarios etc. through interface 3 Start the test through web based interface 4 Download reports and analyze
  • 8. Open source frameworks • Very popular, large user base • Very little programming knowledge required • Handles many protocols • Plugin architecture for extension • Requires programming • Scripting in Jython or Clojure • Distributed framework CLOUD APPROACH – CHOICES ? Commercial Products • JMeter compatible • Real time reports • 10K virtual users x 25 hours @ $1000 Web Performance • Selenium IDE scripts • Real browser or simple request playback (virtual user) • 10K virtual users x 25 hours @ $3750 • Custom scripts • Real time analytics • Custom pricing
  • 9. Open source frameworks Only few frameworks are mature enough to work over the cloud Only the cost of cloud infrastructure, which is much less expensive Completely flexible as scripting is done through open APIs Can only be used through command line, user has to care about calculating VM requirement, creation and closure Reports are basic and sometimes not helpful A CUSTOM FRAMEWORK? Commercial Products Not many choices since there are only a few products in the market May prove to be expensive for repeated load testing May use recording or proprietary scripting, which may not be fully flexible Provide web-based interface to directly run tests without worrying about virtual machines Provide easy to use reports    X  Sizable investment - Suitable for organizations needing repeated performance tests
  • 10. CASE STUDY - HAILSTORM Configuration  JMeter  Amazon  SUT Provisioning Manager JMeter Scripts Script Manager JMeter Controller Data Collector Report Engine - JFreeChart AMI - JMeter - JRE Agent System Under Test (SUT) Agent Agent H T T P
  • 11. CASE STUDY - HAILSTORM Observations Sharp rise in response time for over 50 users “Detail Page” degrades significantly for 75 users and above System is stable but peaks out at 75 concurrent users and starts degrading at 100 users Maximum throughput of 3.2 transactions per second at 75 users is low © Copyright 2012 3PILLAR GLOBAL All rights reserved 11 Sample Report Summary
  • 12. CASE STUDY - HAILSTORM © Copyright 2012 3PILLAR GLOBAL All rights reserved 12 Page Response time Avg. (ms) Response time 90tile (ms) % errors Home Page 1384 2293 0.0 Login Page 559 764 0.0 Login 1050 1740 0.0 Browse 1996 2898 0.25 Category Browsing 2553 3601 0.0 Detail Page 5368 7231 0.27 Search 2054 2948 0.0 Sample Report for one iteration
  • 13. CASE STUDY - HAILSTORM • Application – A leading provider of online classroom and instruction solutions • Key Challenge – Required simulation of 40,000 concurrent users – Rapid and cost effective solution to capture load metrics for large number of concurrent users • The Solution – Different JMeter scripts/plans seamlessly integrated – Engaging reports pointing out bottlenecks generated

Editor's Notes

  • #4: Load DistributionLoad GenerationMerge ResultsGenerate Reports
  • #5: Load DistributionLoad GenerationMerge ResultsGenerate Reports
  • #13: Separate slide for subscription
  • #14: DemoReportsSuccess stories