Embedding Quality Engineering in DevOps pipeline

Embedding Quality
Engineering in DevOps Pipeline

About Aspire
Global technology services firm with
core DNA of Software Engineering
Specific areas of expertise around Software
Engineering, Digital Services, Testing and
Infrastructure & Application Support
Global presence across North America,
Europe, APAC and Middle East
Recognized 9 consecutive times as “Best
Place to Work for” by GPW Institute
Vertical focus on Independent
Software Vendors, Retail, BFSI,
Education
2600+ employees; 150+ active
customers
CMMI Maturity Level 3, ISO 9001:2008
and ISO 27001 : 2013 certified
A Great Place to Work- 9
Glorious Wins in a Row
Quality Maturity
Level
Technology Awards
Forrester Rated Aspire one of
the Top Continuous Testing
Service Providers (2017)
UK
Benelux
India
Middle East
Singapore
US
Netherlands
Poland

About Sauce Labs – The Continuous Testing Experts
Legacy of innovation: 10 years
Comprehensive integrations with CI/CD platforms servers – Jenkins, Bamboo, Microsoft VSTS and others
3,000 customers run over 2 million tests everyday on our platform
500Enterprise Customers
10Years of Innovation
2.0MTests Run / Day
Co-founder, Jason Huggins, invented Selenium and co-developed Appium

Agenda
o Importance of Quality Engineering in DevOps
o Different Dimensions of Quality Engineering
o How Continuous Testing can Make/Break a DevOps Environment
o Implementing Continuous Testing in a DevOps Environment – Use Case
o Tools for Continuous Testing

IMPORTANCE OF QUALITY ENGINEERING IN DEVOPS

How we Develop Software has Changed
New Processes & Tools Drive Faster Releases
Desktop/Packaged Apps
Thick-Client Apps
Desktop Web
Responsive Web
Web Apps
Mobile Apps
Annually Quarterly Monthly Weekly Daily On Demand
Waterfall
Fast
Waterfall
Continuous
Integration
Continuous
Delivery
Proprietary
Single Stack
On-Premise
Open Source
Best of Breed
Cloud-based
Speed of
Releases
Apps
Process
Tools

Transformational Changes In Quality
App works in few environments;
lots of back end code
One team runs tests
Test at one stage at end of dev process
Few long tests, run infrequently
Expensive proprietary test frameworks
App works in in 100s
of OS/Browser/device combos;
lots of front end code
Many teams run tests
Test at many stages throughout
dev process
Many short tests, run frequently
Free Open Source frameworks
o Comprehensive Web &
Mobile Coverage
o UI/Functional Testing
o Training
o Access for distributed teams
o Visibility across teams
o Speed
o Reliability
o Automated testing
o Parallel testing
o Training

The Need for Quality Engineering
Quality Engineering - One of the key levers catalyzing the DevOps mission
o Functional, Non-functional and
User experience
Always tested and certified before
release to production
o Strategized Test Automation
Ensures automation wherever possible
o Shift Left Approach
Starts testing at an earlier stage instead
of waiting for a stable code base
o Automation and certification of the
developed functionalities
Enables faster testing cycle

Shift Left From QA to QE
Project Lifetime
No.ofDefects
QA LifecycleQE Lifecycle
High occurrence of defects at
the later part of the project life
cycle
High occurrence of defects at
the early part of the project
life cycle
Shift Left Approach
Shift Left Approach
Quality Built-in
Review Req,
Architecture and Design
Complete Test
Automation
Continuous Integration /
Continuous Testing
Be a voice of the
customer
Provide Actionable Data
Development Testing
Quality
Engineering
Principles

DIFFERENT DIMENSIONS OF QUALITY ENGINEERING

3 Dimensions of Quality Engineering
Various facets of Quality Engineering
Non-
Functional
Testing
Tools and
ProcessesFunctional
o Cloud-based Test Automation
o Test Automation Framework
o CI/CD
o DevOps
o Application
o Data
o API Automation
o Resolution Tests
o Mobile/Device testing
o Performance
o Security
o Load
o Multi-platforms

Cloud-based Test Automation
Enables automatic testing and result management
Provisions infrastructure specific to testing
Enables testing combinations of modules and
products
Manages failures and trigger appropriate action items
Provides intelligence for the systems to scale and
distribute
Builds pipelines that span across all functions of
development
Cloud-based solution facilitates seamless validation of the automated test cases
Sample Cloud Environments: Proprietary VM, TFS, Sauce Labs, etc.

Tools for Continuous Testing
Functional & Regression Usability Performance Testing Mobile Test Automation
Security Continuous Integration API Cloud Testing
o JMeter
o LoadRunner
o Demonster
o ANTS
o NeoLoad
o QALoad
o WebLoad
o Silk Performer
o Selenium
o Lean FT
o Test Complete
o Silk Test
o Coded UI
o Squish
o UserFly
o UserVue
o Appium
o Sauce Labs
o M-eux
o Test Complete
o Xamarin
o Device Anywhere
o Monkey Talk
o Soap UI
o Soap UI Pro
o Fitnesse
o TestComplete
o Rest-Assured
o OWASP ZAP
o Veracode
o Fortify
o QAInspect
o Jenkins
o Bamboo
o VSTS
o TFS
o Sauce Labs

HOW CONTINUOUS TESTING CAN MAKE/BREAK A DEVOPS
ENVIRONMENT

Importance of Continuous Testing in DevOps
Test Early & Often Throughout Your CI/CD Pipeline
Test
Operate
OPSDEV
Continuous Testing

Continuous Testing – Paves the Way for Continuous Delivery
o Automate your testing as much as possible
o Automate more effective and efficient test cases which provides a wider test
coverage
o Start automating early in the lifecycle
o Continuous Feedback
o Early Defect Detection
o Pipeline Visibility
o Reliable and Scalable Resources
o Go/No Go Decision for Builds
o Continuous Code Check-Ins
o Never check-In a
broken/uncompiled Code
o Dev-Tester Collaboration
Continuous IntegrationContinuous Delivery
Continuous Testing

Continuous Testing – Best Practices
Automated
deployment
Centralized
code
repository
Automated
build
process
Automated unit
and integration
tests
Code pushes
should be built
on a CI server
Implement
faster build
mechanism
Testing in a clone of
the production
environment
CD
Practices

IMPLEMENTING CONTINUOUS TESTING IN A DEVOPS ENVIRONMENT–
USE CASE

Four Types of Testing in Pipeline
Continuous Testing Requires a Comprehensive Solution
Fast Feedback
Small Tests
Few Platforms
Isolated
GOAL
Stable Build Deploy Ready Monitor / Debug
APPROACH
End to End Testing
Few Platforms
Integrated
Full Suite of Tests
All Platforms
In Parallel
Business Critical Tests
Relevant Platforms
Targeted
DEVELOPMENT INTEGRATION PRE-RELEASE FEEDBACK

Four Types of Testing in Pipeline
Continuous Testing Requires a Comprehensive Solution
Fast Feedback
Small Tests
Few Platforms
Isolated
GOAL
Stable Build Deploy Ready Monitor / Debug
APPROACH
End to End Testing
Few Platforms
Integrated
Full Suite of Tests
All Platforms
In Parallel
Business Critical Tests
Relevant Platforms
Targeted
DEVELOPMENT INTEGRATION PRE-RELEASE FEEDBACK
Test on every
Pull Request
SAUCE
LABS
Automated Tests
In Parallel
Automated Tests
In Parallel
”Live” Tests
Bug Reproduction
Any Platform

How To Apply To Your Pipeline
CODEPLAN
CODEPLAN
CODEPLAN
Team 1
Shared Delivery Pipeline Production
BUILD
o Integration
test
TEST
o API testing
o Regression
testing
o Performance
testing
DEPLOY
o Smoke test
o Security
testing
o User
acceptance
testing
o Visual testing
OPERATE
o Production
testing
o Monitoring
o Debugging
M E R G E R E L E A S E
Team 2
Team 3
Team 4
Local Dev Environments
PLAN
o Feature
definition
o Test authoring
o Test driven
development
CODE
o Unit test
o Component
test
Fast Feedback Stable Build Deploy Ready FeedbackVerify Functionality

How To Apply To Your Pipeline: Mobile
CODEPLAN
CODEPLAN
CODEPLAN
Team 1
Shared Delivery Pipeline Production
M E R G E R E L E A S E
Team 2
Team 3
Team 4
PLAN
o Feature
definition
o Test authoring
o Test driven
development
CODE
o Unit test
o Component
test
Desktops
Mobile Emulators & Simulators
Mobile Real Devices
BUILD
o Integration
test
TEST
o API testing
o Regression
testing
o Performance
testing
DEPLOY
o Smoke test
o Security
testing
o User
acceptance
testing
o Visual testing
OPERATE
o Production
testing
o Monitoring
o Debugging
Local Dev Environments

Quality Engineering & Continuous Testing Implementation
`
Use Case Scenario: Cloud-based automation testing for an online assessment solution
Test Automation
Manual Testing

CI Best Practices Employed
o 8 hours
o 7 browser in parallel
o Proceed to validate
Integration suite on
completion
o 3 browsers in parallel
- Test report from Sauce Labs - verified during every PR review
Deployment/Release testing done in phased manner using Sauce Labs
Sauce Labs - A part of Acceptance Process
Smoke suite
Regression suite
Integration suite
o 15 minutes or abort
o 7 browser + 7 devices
in parallel
o Raise Showstopper if
any test fails, else
continue to regression

CI Components
Sauce Labs
WebDriver tests
matrix
Sauce on-demand
plugin
Sauce Labs test
publisher
Automation Repo Sauce Connect

CI Workflow
DEV Version Control Build & Unit Test QA UAT PROD
Commit
Test Results
Feedback
Commit
Full Regression
Test Results
Trigger
Feedback
Commit Trigger
Test Results
Feedback
Trigger
Commit Trigger
UAT Testing
Feedback
Commit Trigger
Test Results
Feedback
TriggerCommit
CIPipeline

ROI Realized
AutomationToolStack
Selenium Web Driver
Java (Scripting)
Jenkins (CI)
Sauce Labs (Cloud
Execution)
Rest Assured (API)
TechnologyStack
Web Application
Responsive Design
iOS, Android &
Windows Mobile
Browser Support
Rest APIs
50
2400
300
50
1920
300
0 500 1000 1500 2000 2500 3000
Smoke
Regression
Integration
Smoke Regression Integration
Automated 50 1920 300
Tests# 50 2400 300
Automation Coverage
Modules /
Coverage
Tests
#
# of Browsers
Before Automation After Automation
Test Duration Team Size Test Duration Team Size
Smoke (100%) 50
14 (7 desktop + 7
devices)
12 hours 12 15 mins 10
Regression (80%) 2400 7 desktop combos 3 weeks 10 10 hours 4
Integration (90%) 300 3 desktop combos 2 days 4 3 hours 2

ROI Realized
Befor
Automation
Smoke Regression Integration
Team Size 12 10 4 1
Befor
Automation, 12
Smoke, 10
Regression, 4
Integration, 1
0
2
4
6
8
10
12
14
Team Size (Post Automation)
Time Duration to Automate
Module Resources Duration
Smoke 3 20 days
Regression 10 8 months
Integration 6 1 month
Total Savings – Last 1 Year
Module # of runs Time saved (Hrs)
Smoke 48 856.8
Regression 24 22752
Integration 12 1500
Maintenance Effort
Module Hours/Month
Smoke 12
Regression 80
Integration 25
Duration:Pre-Automation(in
Hrs)
Duration:Post-Automation(in
Hrs)
Smoke 12 0.24
Regression 120 10
Integration 64 3
0
20
40
60
80
100
120
140
Turnaround Time(Pre/Post Automation)
e
e

ROI Realized
Case CI Execution
Test Suite Time Infrastructure VM/Physical Machines
Smoke Suite 15 mins 7 Browser combinations in Parallel 7 Devices in Parallel 30
Regression Suite 8 Hours 7 Browser combinations in Parallel 70
To accomplish the task of completing the Smoke Test in 15 mins and Regression Test in 8 hours, a minimum of 100 VM/Physical machines
would be needed. Setting up the CI for this combination using Physical machines or VMs is a tremendous task and the cost involved in setting up
and running this infrastructure is represented below :
3 Yr. Total Cost of Ownership
On-Premises Virtual Machines Cloud
Server $3,078,750 $3,278,175 $1,446,685
Storage $ - $ - $ -
Network $590,286 $429,304 $ -
IT-Labor $ - $ - $ -
Total $3,669,037 $3,707,479 $1,446,685
$ 5,000,000
$ 4,000,000
$ 3,000,000
$ 2,000,000
$ 1,000,000
$ 0
On-Premise Virtual Machines Cloud
Server Storage Network IT Labor

Q & A
maria.josephine@aspiresys.com

Thank You
maria.josephine@aspiresys.com

Embedding Quality Engineering in DevOps pipeline

More Related Content

What's hot (20)

Similar to Embedding Quality Engineering in DevOps pipeline (20)

More from Aspire Systems (20)

Recently uploaded (20)

Embedding Quality Engineering in DevOps pipeline

Editor's Notes