SlideShare a Scribd company logo

Technical meeting automated testing with vs2010

Clemens Reijnen, profile picture
Clemens Reijnen

The document discusses automated testing using Microsoft Visual Studio 2010 and its application lifecycle management tools, detailing the test lifecycle and management strategies. It emphasizes collaboration on test plans and test suites, automation benefits, and configurations for different testing environments. Additionally, it covers the integration of test results, bug tracking, and execution methods for enhancing testing efficiency across various setups.

TechnologyBusiness
Related topics:
Automated Testing
1 of 152
Downloaded 192 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
Technical Meeting Automated Testing with VS2010
Suite of Application Lifecycle Management tools http://msdn.microsoft.com/en-us/library/fda2bad5.aspx
application lifecycle management http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx
work together __with work packages __on alm artifacts
Testing Lifecycle See also on www.tmap.net: - The Test Lifecycle Model - The 4 essentials
http://msdn.microsoft.com/en-us/library/bb385832.aspx
Microsoft Test Manager 2010
Test Case Management See also on www.tmap.net: Control Phase
Test Plans Test Suites Test Configurations Organize the testing effort in a Test Execution Plan Create dynamic test suites for changing selection criteria Define various test configurations and share automation across those
Test (Execution) Plans
Test (Execution) Plans
Test Plan Organization Naming of test plans for agile environments: Iteration number and name and one extra test plan for end to end tests.
Test Configurations
Test Configurations
Test Suites
Test Suites Organization Naming of test suites for agile environments: Suite per user story (requirement based) and one additional test suite for regression tests.(query based?) (test cases are copied to the regression test suite)
Proposed organization
TCM.exe
TCM plan /list
TCM suites /lists planid:16
Share MTM Artifacts mtm://clemenes-pc:8080/tfs/defaultcollection/p:MSF/Testing/testplan/connect?id=19
mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center group>/<group specific> Open Test results Open Test Cases (or any work item) Open Test Runs Open Test Plans Connect to test plans mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67 http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft- test-manager.aspx
Test Case Specification See also on www.tmap.net: Specification Phase
Test Case Shared Steps Test Data Iterations Link test cases and requirements to provide requirement traceability Data drive test cases with different parameters Refactor common shared steps in test cases
Test Case
Test step types
Test steps Make them descriptive Not every step is a validation step Use the step attachment
Test steps copy-past
Test Data Iterations Data drive test cases with different parameters
Test Data Iterations
Test data copy-past
Shared Steps Refactor common shared steps in test cases Reduce test maintenance by sharing test steps across test cases
Shared Steps Shared Step candidates? • REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work. • EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases which need this step are updated automatically, also action recording. • PRE- CONDITIONS – bring the system under test to a certain state and execute the different verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case • BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test cases.
Test Execution Test Runner A test that you will want to run multiple times. Different manual tests that contain common steps. Verifying a bug is fixed by using a manual test.
Test Case Execution See also on www.tmap.net: Execution Phase
Technical meeting automated testing with vs2010
Test Case Execution
Technical meeting automated testing with vs2010
Action recording
TCM run /create
From TCM run /export id:239 /resultsfile:”c:runresult”
Share with mtm:// mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292
Controlling the testing effort BUG’s
File a BUG
Diagnostic Data Adapters File a Bug File an Exploratory Bug Verify a Bug
Diagnostic Data Adapters
File a Bug Submit a Bug using Test Runner
Ways to file a Bug Submit a Bug from the Test Runner Submit a Bug from the Verify Bugs View Submit a Bug from Test Results for a Test
Verify a Bug Verify a Bug Fix Using Microsoft Test Manager
The Bug Flow Link build 2 test plan. T Execute File Bug Verify fix Close or Configure Data E Manual by using reopen collectors. S Tests playback Bug T Build definition is the intergration Setup build and deploy D Resolve Checkin E Bug Build V Deploy
Test Impact
Test Impact Assign build 2 test plan. T Execute Assign Evaluated and reset Execute Execute Configure Data E ‘passed’ ‘NEW’ build tests impacted by ‘passed’ ‘passed’ collectors. S Tests 2 Test Plan code churn Tests Tests T Build definition is the intergration Setup build and deploy D Make Checkin E code Build V changes Deploy
Create test from Bug
Technical meeting automated testing with vs2010
File an Exploratory Bug Submit an Exploratory Bug using Test Runner
PRO: - Reuse scenario knowledge business user - No typing detailed test steps - Easy to parameterize 1 Record scenario by business user with - Writing standards for test steps MTM on application under test. Action - MTM capabilities in step 3. (bug, codedUI, recommended test etc …) recording is captured. Most benefit during step 2 (test case writing), benefit in step 1: run once in common situation. Benefit in step 3: common MTM benefits (record playback, fast forward, parameters etc) and benefit of well written steps. 2 Create test steps (with parameters) in MTM by using the action recording. CON: - Manual actions and tests. - Test steps text are technical names - Action recording (step 1) must be clean. - Unknown what is validated, during step 2, need additional knowledge. - The business user must think of the important scenarios 3 Save test case and execute it by a tester The action recording during step 1 can have on application under test, by using MTM. unnecessary steps, these can be removed during the execution (1) or during the specification (2).
RECAP Test Activities Planning…
Test Case Management RECAP
RECAP Test Case Specification
RECAP Test Case Execution
Test Case Management RECAP
Intellitrace
F5
1 Open the employee application Set the intellitrace options Set a breakpoint at …
2 When on the breakpoint, save the trace file on your desktop
3 Close Visual Studio and double click the intellitrace file
4 In the intellitrace summary, open the main threat …
5 … and search for The values that where entered during the previous run.
6 And now from Microsoft Test Manager
6 Set the diagnostic adapter
7 Configure the build… it needs a drop location for the symbols (pdb files)
8 Run the test case and see it crash… file a bug.
9 See the itrace file in the bug report...
10 Close MTM, open VS (with or without the employee application). Got to the bug and click the itrace file… See the exception and start debugging…
11 Go back in time
11 Go back in time … and find the entered values
Regression Test Candidates See also on www.tmap.net: Completion Phase
Test Automation met Visual Studio 2010 Test automation Where? When? How? Who? Why? What?
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
Where? When? How? Who? Why? What?
Where? When? How? Who? Why? What?
Where? When? How? Who? Why? What?
VS2010 Test Tools UI Test Automation Capabilities Fast Forward CodedUI Test Infrastructure
Test Automation Investment Levels 0 No investment in any kind of automation. Just click and test. 1 Clean up of the Action log maybe re-execute, for more smooth Fast Forward. 2 Basic CodedUI. Only use the default generation, add your own assertion. 3 Advanced CodedUI. Customize the CodedUI and UIMap for optimization.
1 Clean up of the Action
2 Basic CodedUI VS2010 ALM MTLM usages patterns for Windows Azure hosted application development
3 Advanced CodedUI
Fast Forward
Cleaning the action recording When: executing the same test case on multiple environments or when using test data iteration or … and don’t forget the shared steps
CodedUI
CodedUI from existing action recording from Manual Tester
CodedUI from existing action recording from Manual Tester
CodedUI from existing action recording from Manual Tester 2 1 4 3
CodedUI Create a new action recording by Developer
UIMap
UIMap
Technical meeting automated testing with vs2010
Data-Driven Coded UI Test
Test Infrastructure Where and how to run the automated test?
Test Environments See also on www.TMap.net: - Checklist "Intake Test Environment“ - Checklist "Test Environment“ - Infrastructure Phase
Test lab Visual Studio 2010 Lab Management
Where and how to run the automated test? physical test environments virtualized test environments
physical test virtualized test environments environments MTLM Client MTLM Client Physical Test Client TFS 2010 System Center Virtual TFS 2010 Machine Manager (SCVMM) Hyper-V Library Physical Host Shares Test Client VM VM share share
Build application physical test environments versus virtualized test environments Provision Run tests ‘clean’ env Deploy application
Lab for Physical Environments, the different flavors… Visual Studio Team Foundation Server Test Manager Build Server Test Agent Test Agent Test Agent
Configurations for automated test execution: A Flavor A: Execution from VS2010… Real developer test / test automation dry run (Triggered by: right mouse click – run tests, in test project) http://msdn.microsoft.com/en-us/library/dd286580.aspx No additional configuration needed* * Data Diagnostic Settings in the Local.Testsettings file are configurable. Pro: Con: Easy setup No collection of test results in TFS Debug-able test automation Run on developer environment
Configurations for automated test execution: B Flavor B: Execution during build with Build controller… not recommended, strange to run UI tests on build Part of Build Verification Tests (triggered by: build) Set the Build Service to run in interactive mode http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType * Data Diagnostic Settings in the *.Testsettings file are configurable. Pro: Con: Easy setup No collection of test results in TFS Test results in build result Build Controller needs to run in interactive mode Tests are executed on build environment Run on build environment
Configurations for automated test execution: C Flavor C: Execution during build with Test controller… B Preferred configuration above flavor Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B) Pro: Con: Test run on test environments No collection of test results in TFS Tests run on multiple environments Harder to configure Test Results in Build result Need for specific test client environments Test Settings from VS
Configurations for automated test execution: D Flavor D: Execution from Microsoft Test Manager…than BVT Other type of test Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html Pro: Con: Test run on test environments Manually triggered by Tester Tests run on multiple environments Hard to configure Test Result in MTM and TFS Need for specific test client (same as MTM?) Test Settings from MTM
Configurations for automated test execution: E Flavor E: Execution from MTM duringPreferred configuration above flavor C Build… Part of BVT Flavor D and E can be configured together (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx Pro: Con: Test run on test environments Hard to configure Tests run on multiple environments Need for specific test client (same as MTM?) Test Result in MTM and TFS Triggered by build
Test Controller
Test Agent
New physical environment
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
Execute CodedUI During the build with Visual Studio 2010
Configurations for automated test execution: C Flavor C: Execution during build with Test controller… B Preferred configuration above flavor Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B) Pro: Con: Test run on test environments No collection of test results in TFS Tests run on multiple environments Harder to configure Test Results in Build result Need for specific test client environments Test Settings from VS
1 Configure Test Controller.
2 Add new Test Settings file Set remote execution and test controller.
3 Create a new build definition Set the Automated Test Properties
4 Queue a new Build Watch the Test Agent and Test Environment
Execute CodedUI Within a test case as an associated automation manualy
Configurations for automated test execution: D Flavor D: Execution from Microsoft Test Manager…than BVT Other type of test Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html Pro: Con: Test run on test environments Manually triggered by Tester Tests run on multiple environments Hard to configure Test Result in MTM and TFS Need for specific test client (same as MTM?) Test Settings from MTM
1 Configure Test Controller.
2 Associate the CodedUI with a Test Case.
2 Create Test Environment
3 Set test Settings
4 Associate the CodedUI with a Test Case.
5 Run Automated Tests from Test Manager
Execute CodedUI Within a test case as an associated automation from the build
Configurations for automated test execution: E Flavor E: Execution from MTM duringPreferred configuration above flavor C Build… Part of BVT Flavor D and E can be configured together (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run MTC or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx Pro: Con: Test run on test environments Hard to configure Tests run on multiple environments Need for specific test client (same as MTM?) Test Result in MTM and TFS Triggered by build Test Settings from MTM
Execute automated test cases on different environments
0 Same steps as previous.
1 Configure additional environments in Test Plan properties. Win 7 IE9 configuration Win 7 IE8 configuration TC MTLM Client Win 7 IE8 configuration Physical Test Client TFS 2010 Win 7 IE9 configuration Physical Test Client
1 Get Test Plan ID Get Test Suite Get Test Configurations with TCM.exe
2 Create command line for Test Run for the different Test Configurations "run /create /title:""Run on Host for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=13"" /settingsname:""Automated Test on Windows 7 Environment IE9"" /testenvironment:""Win 7 IE9"" /collection:http://clemenes- pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication" "run /create /title:""Run on test env for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=12"" /settingsname:""Automated Test on Windows 7 Environment IE8"" /testenvironment:""Win 7 IE8"" /collection:http://clemenes- pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"
3 Add build commands
6 Queue a new Build Watch the Test Agent and Test Environment
Build application Provision Run tests ‘clean’ env Deploy application
VS2010 Lab Management • Reproduce the exact conditions of a bug or other development issue. • Build, deploy, and test applications automatically in a clean environment. • Reduce the time required to create and configure machines for testing an application. • Run multiple copies of a test or development at the same time. • Enable members of a team to create and manage virtual environments without requiring system administrator privileges.
MTLM Client TFS 2010 System Center Virtual Machine Manager (SCVMM) Hyper-V Library Host Shares VM share VM share
VM Lab Agent
TFS 2010
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
Technical meeting automated testing with vs2010
staat voor resultaat

More Related Content

PPTX
Test Tooling in Visual Studio 2012 an overview
Clemens Reijnen
 
PPTX
Coded UI - Test automation Practices from the Field
Clemens Reijnen
 
PPTX
Coded ui - lesson 1 - overview
Omer Karpas
 
PPTX
Advanced Coded UI Testing
Shai Raiten
 
PPTX
Coded ui in a nutshell
Omer Karpas
 
PPT
UI testing frameworks and the Coded UI testing paradigm
Arnold Stellio
 
PPTX
Testing the User Interface - Coded UI Tests with Visual Studio 2010
Eric D. Boyd
 
PPTX
Coded ui test
Abhimanyu Singhal
 
Test Tooling in Visual Studio 2012 an overview
Clemens Reijnen
 
Coded UI - Test automation Practices from the Field
Clemens Reijnen
 
Coded ui - lesson 1 - overview
Omer Karpas
 
Advanced Coded UI Testing
Shai Raiten
 
Coded ui in a nutshell
Omer Karpas
 
UI testing frameworks and the Coded UI testing paradigm
Arnold Stellio
 
Testing the User Interface - Coded UI Tests with Visual Studio 2010
Eric D. Boyd
 
Coded ui test
Abhimanyu Singhal
 

What's hot (20)

PPTX
Unit testing with visual studio 2012
Abhimanyu Singhal
 
PPSX
QA with Microsoft Test Manager and Lab Management
Rofiqi Setiawan
 
PPT
MTM Test Management System
Minu Mishra
 
PPTX
Coded ui - lesson 9 - test windows
Omer Karpas
 
PPTX
Intro to Microsoft Test Manager
Esteban Garcia
 
PPTX
Deep Dive Modern Apps Lifecycle with Visual Studio 2012: How to create cross ...
Microsoft Developer Network (MSDN) - Belgium and Luxembourg
 
PPTX
OPASS - Microsoft Test Manager
Esteban Garcia
 
PPT
Automation testing by Durgasoft in Hyderabad
Durga Prasad
 
PPT
Automation testing material by Durgasoft,hyderabad
Durga Prasad
 
PPTX
Full Testing Experience - Visual Studio and TFS 2010
Ed Blankenship
 
PPTX
SE2018_Lec 20_ Test-Driven Development (TDD)
Amr E. Mohamed
 
DOCX
Testing in TFS
Sampath kumar Mohan
 
PPTX
Visual Studio 2010 Testing Overview
Steve Lange
 
PPTX
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.
Wolfgang Grieskamp
 
PPTX
STARWEST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
Anna Russo
 
PDF
Automated Testing
John Paulett
 
PDF
Tutorial ranorex
radikalzen
 
PDF
Testing Tools
Ted Husted
 
PPT
Automation testing IBM RFT - Rational Functional Tester
VijayChowthri Nagaprakasham
 
PPTX
Testing tools in visual studio
Mahdi Esmailoghli
 
Unit testing with visual studio 2012
Abhimanyu Singhal
 
QA with Microsoft Test Manager and Lab Management
Rofiqi Setiawan
 
MTM Test Management System
Minu Mishra
 
Coded ui - lesson 9 - test windows
Omer Karpas
 
Intro to Microsoft Test Manager
Esteban Garcia
 
Deep Dive Modern Apps Lifecycle with Visual Studio 2012: How to create cross ...
Microsoft Developer Network (MSDN) - Belgium and Luxembourg
 
OPASS - Microsoft Test Manager
Esteban Garcia
 
Automation testing by Durgasoft in Hyderabad
Durga Prasad
 
Automation testing material by Durgasoft,hyderabad
Durga Prasad
 
Full Testing Experience - Visual Studio and TFS 2010
Ed Blankenship
 
SE2018_Lec 20_ Test-Driven Development (TDD)
Amr E. Mohamed
 
Testing in TFS
Sampath kumar Mohan
 
Visual Studio 2010 Testing Overview
Steve Lange
 
Model-Based Testing: Theory and Practice. Keynote @ MoTiP (ISSRE) 2012.
Wolfgang Grieskamp
 
STARWEST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
Anna Russo
 
Automated Testing
John Paulett
 
Tutorial ranorex
radikalzen
 
Testing Tools
Ted Husted
 
Automation testing IBM RFT - Rational Functional Tester
VijayChowthri Nagaprakasham
 
Testing tools in visual studio
Mahdi Esmailoghli
 
Ad

Similar to Technical meeting automated testing with vs2010 (20)

PPT
1.microsoft visual studio 2010 test manager
Ashwin Jujgar
 
PDF
Visual Studio 2010: A Perspective - David Chappell
Spiffy
 
PPTX
Test Case Management with MTM 2013
Raluca Suditu
 
PPTX
Test case management with MTM 2013
Raluca Suditu
 
PPTX
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
Anna Russo
 
PDF
MTLM Visual Studio 2010 ALM workshop - day1
Clemens Reijnen
 
PPTX
Improved software testing using Visual Studio and TFS 2010
Alexander Vanwynsberghe
 
PDF
SAP-Solution-Manager
Douglas Bernardini
 
PDF
Implementing Test Automation in Agile Projects
Dominik Dary
 
PPTX
Implementing Test Automation in Agile Projects
Michael Palotas
 
PDF
Test and Behaviour Driven Development (TDD/BDD)
Lars Thorup
 
PPTX
Working Software Over Comprehensive Documentation
Andrii Dzynia
 
PPTX
Lab Management with TFS 2010
Ed Blankenship
 
PPTX
Quality Coding with Visual Studio 2012
Imaginet
 
PPTX
Quality Coding: What's New with Visual Studio 2012
Imaginet
 
PPTX
Quality Coding: What’s New with Visual Studio 2012
Imaginet
 
PPTX
DaKiRY_BAQ2016_QADay_Marta Firlej "Microsoft Test Manager tool – how can we u...
Dakiry
 
PDF
MTLM Visual Studio 2010 ALM - day2
Clemens Reijnen
 
PDF
Testing Tools Online Training.pdf
SpiritsoftsTraining
 
PDF
Gateway test labs
gatewaytestlabs
 
1.microsoft visual studio 2010 test manager
Ashwin Jujgar
 
Visual Studio 2010: A Perspective - David Chappell
Spiffy
 
Test Case Management with MTM 2013
Raluca Suditu
 
Test case management with MTM 2013
Raluca Suditu
 
STAREAST 2011 - 7 Steps To Improving Software Quality using Microsoft Test Ma...
Anna Russo
 
MTLM Visual Studio 2010 ALM workshop - day1
Clemens Reijnen
 
Improved software testing using Visual Studio and TFS 2010
Alexander Vanwynsberghe
 
SAP-Solution-Manager
Douglas Bernardini
 
Implementing Test Automation in Agile Projects
Dominik Dary
 
Implementing Test Automation in Agile Projects
Michael Palotas
 
Test and Behaviour Driven Development (TDD/BDD)
Lars Thorup
 
Working Software Over Comprehensive Documentation
Andrii Dzynia
 
Lab Management with TFS 2010
Ed Blankenship
 
Quality Coding with Visual Studio 2012
Imaginet
 
Quality Coding: What's New with Visual Studio 2012
Imaginet
 
Quality Coding: What’s New with Visual Studio 2012
Imaginet
 
DaKiRY_BAQ2016_QADay_Marta Firlej "Microsoft Test Manager tool – how can we u...
Dakiry
 
MTLM Visual Studio 2010 ALM - day2
Clemens Reijnen
 
Testing Tools Online Training.pdf
SpiritsoftsTraining
 
Gateway test labs
gatewaytestlabs
 
Ad

More from Clemens Reijnen (18)

PPTX
Azure for software development teams
Clemens Reijnen
 
PDF
06 operations and feedback
Clemens Reijnen
 
PDF
05 test infrastructure
Clemens Reijnen
 
PDF
04 test controlling and tracking
Clemens Reijnen
 
PDF
03 test specification and execution
Clemens Reijnen
 
PDF
02 test planning
Clemens Reijnen
 
PDF
01 the value of quality
Clemens Reijnen
 
PPTX
A Collaborative culture
Clemens Reijnen
 
PPTX
ALM - Getting Testing done in a Sprint
Clemens Reijnen
 
PDF
TFS11 on Azure advanced
Clemens Reijnen
 
PDF
TFS11 on Azure the basics
Clemens Reijnen
 
PDF
MTLM Visual Studio 2010 ALM workshop
Clemens Reijnen
 
PDF
Scrum with VS2010
Clemens Reijnen
 
PDF
Alm 4 Azure with screenshots
Clemens Reijnen
 
PDF
Talk Through Sogeti ALM 4 Azure
Clemens Reijnen
 
PDF
Alm 4 Azure
Clemens Reijnen
 
PPTX
TMap for VS2010 - SlideShare.
Clemens Reijnen
 
PPTX
Work agile with VS2010
Clemens Reijnen
 
Azure for software development teams
Clemens Reijnen
 
06 operations and feedback
Clemens Reijnen
 
05 test infrastructure
Clemens Reijnen
 
04 test controlling and tracking
Clemens Reijnen
 
03 test specification and execution
Clemens Reijnen
 
02 test planning
Clemens Reijnen
 
01 the value of quality
Clemens Reijnen
 
A Collaborative culture
Clemens Reijnen
 
ALM - Getting Testing done in a Sprint
Clemens Reijnen
 
TFS11 on Azure advanced
Clemens Reijnen
 
TFS11 on Azure the basics
Clemens Reijnen
 
MTLM Visual Studio 2010 ALM workshop
Clemens Reijnen
 
Scrum with VS2010
Clemens Reijnen
 
Alm 4 Azure with screenshots
Clemens Reijnen
 
Talk Through Sogeti ALM 4 Azure
Clemens Reijnen
 
Alm 4 Azure
Clemens Reijnen
 
TMap for VS2010 - SlideShare.
Clemens Reijnen
 
Work agile with VS2010
Clemens Reijnen
 

Recently uploaded (20)

PPT
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
PDF
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PDF
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
PDF
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
PDF
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
PDF
Accuracy of neural networks in brain wave diagnosis of schizophrenia
IAESIJAI
 
PPTX
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
PDF
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PPTX
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
PDF
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
PDF
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
PPTX
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 
“AI and Expert System Decision Support & Business Intelligence Systems”
ZubinRadhakrishnan
 
Mobile App Security Testing_ A Comprehensive Guide.pdf
flufftailshop
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
Reach Out and Touch Someone: Haptics and Empathic Computing
Mark Billinghurst
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
A comparative analysis of optical character recognition models for extracting...
IAESIJAI
 
NewMind AI Weekly Chronicles - August'25-Week II
NewMind AI
 
Network Security Unit 5.pdf for BCA BBA.
Serpent6
 
Accuracy of neural networks in brain wave diagnosis of schizophrenia
IAESIJAI
 
SOPHOS-XG Firewall Administrator PPT.pptx
AgnesMunisi
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
KOM of Painting work and Equipment Insulation REV00 update 25-dec.pptx
muhammadmuneebnaeem7
 
Spectral efficient network and resource selection model in 5G networks
IAESIJAI
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
Blue Purple Modern Animated Computer Science Presentation.pdf.pdf
Akar16
 
Build a system with the filesystem maintained by OSTree @ COSCUP 2025
Jian-Hong Pan
 
Digital-Transformation-Roadmap-for-Companies.pptx
David Malick Dieng
 

Technical meeting automated testing with vs2010

  • 2. Suite of Application Lifecycle Management tools http://msdn.microsoft.com/en-us/library/fda2bad5.aspx
  • 4. work together __with work packages __on alm artifacts
  • 5. Testing Lifecycle See also on www.tmap.net: - The Test Lifecycle Model - The 4 essentials
  • 8. Test Case Management See also on www.tmap.net: Control Phase
  • 9. Test Plans Test Suites Test Configurations Organize the testing effort in a Test Execution Plan Create dynamic test suites for changing selection criteria Define various test configurations and share automation across those
  • 12. Test Plan Organization Naming of test plans for agile environments: Iteration number and name and one extra test plan for end to end tests.
  • 16. Test Suites Organization Naming of test suites for agile environments: Suite per user story (requirement based) and one additional test suite for regression tests.(query based?) (test cases are copied to the regression test suite)
  • 20. TCM suites /lists planid:16
  • 22. mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center group>/<group specific> Open Test results Open Test Cases (or any work item) Open Test Runs Open Test Plans Connect to test plans mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67 http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft- test-manager.aspx
  • 23. Test Case Specification See also on www.tmap.net: Specification Phase
  • 24. Test Case Shared Steps Test Data Iterations Link test cases and requirements to provide requirement traceability Data drive test cases with different parameters Refactor common shared steps in test cases
  • 27. Test steps Make them descriptive Not every step is a validation step Use the step attachment
  • 29. Test Data Iterations Data drive test cases with different parameters
  • 32. Shared Steps Refactor common shared steps in test cases Reduce test maintenance by sharing test steps across test cases
  • 33. Shared Steps Shared Step candidates? • REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work. • EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases which need this step are updated automatically, also action recording. • PRE- CONDITIONS – bring the system under test to a certain state and execute the different verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case • BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test cases.
  • 34. Test Execution Test Runner A test that you will want to run multiple times. Different manual tests that contain common steps. Verifying a bug is fixed by using a manual test.
  • 35. Test Case Execution See also on www.tmap.net: Execution Phase
  • 41. From TCM run /export id:239 /resultsfile:”c:runresult”
  • 42. Share with mtm:// mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292
  • 43. Controlling the testing effort BUG’s
  • 45. Diagnostic Data Adapters File a Bug File an Exploratory Bug Verify a Bug
  • 47. File a Bug Submit a Bug using Test Runner
  • 48. Ways to file a Bug Submit a Bug from the Test Runner Submit a Bug from the Verify Bugs View Submit a Bug from Test Results for a Test
  • 49. Verify a Bug Verify a Bug Fix Using Microsoft Test Manager
  • 50. The Bug Flow Link build 2 test plan. T Execute File Bug Verify fix Close or Configure Data E Manual by using reopen collectors. S Tests playback Bug T Build definition is the intergration Setup build and deploy D Resolve Checkin E Bug Build V Deploy
  • 52. Test Impact Assign build 2 test plan. T Execute Assign Evaluated and reset Execute Execute Configure Data E ‘passed’ ‘NEW’ build tests impacted by ‘passed’ ‘passed’ collectors. S Tests 2 Test Plan code churn Tests Tests T Build definition is the intergration Setup build and deploy D Make Checkin E code Build V changes Deploy
  • 55. File an Exploratory Bug Submit an Exploratory Bug using Test Runner
  • 56. PRO: - Reuse scenario knowledge business user - No typing detailed test steps - Easy to parameterize 1 Record scenario by business user with - Writing standards for test steps MTM on application under test. Action - MTM capabilities in step 3. (bug, codedUI, recommended test etc …) recording is captured. Most benefit during step 2 (test case writing), benefit in step 1: run once in common situation. Benefit in step 3: common MTM benefits (record playback, fast forward, parameters etc) and benefit of well written steps. 2 Create test steps (with parameters) in MTM by using the action recording. CON: - Manual actions and tests. - Test steps text are technical names - Action recording (step 1) must be clean. - Unknown what is validated, during step 2, need additional knowledge. - The business user must think of the important scenarios 3 Save test case and execute it by a tester The action recording during step 1 can have on application under test, by using MTM. unnecessary steps, these can be removed during the execution (1) or during the specification (2).
  • 63. F5
  • 64. 1 Open the employee application Set the intellitrace options Set a breakpoint at …
  • 65. 2 When on the breakpoint, save the trace file on your desktop
  • 66. 3 Close Visual Studio and double click the intellitrace file
  • 67. 4 In the intellitrace summary, open the main threat …
  • 68. 5 … and search for The values that where entered during the previous run.
  • 69. 6 And now from Microsoft Test Manager
  • 71. 7 Configure the build… it needs a drop location for the symbols (pdb files)
  • 72. 8 Run the test case and see it crash… file a bug.
  • 73. 9 See the itrace file in the bug report...
  • 74. 10 Close MTM, open VS (with or without the employee application). Got to the bug and click the itrace file… See the exception and start debugging…
  • 76. 11 Go back in time … and find the entered values
  • 77. Regression Test Candidates See also on www.tmap.net: Completion Phase
  • 78. Test Automation met Visual Studio 2010 Test automation Where? When? How? Who? Why? What?
  • 81. Where? When? How? Who? Why? What?
  • 82. Where? When? How? Who? Why? What?
  • 83. Where? When? How? Who? Why? What?
  • 84. VS2010 Test Tools UI Test Automation Capabilities Fast Forward CodedUI Test Infrastructure
  • 85. Test Automation Investment Levels 0 No investment in any kind of automation. Just click and test. 1 Clean up of the Action log maybe re-execute, for more smooth Fast Forward. 2 Basic CodedUI. Only use the default generation, add your own assertion. 3 Advanced CodedUI. Customize the CodedUI and UIMap for optimization.
  • 86. 1 Clean up of the Action
  • 87. 2 Basic CodedUI VS2010 ALM MTLM usages patterns for Windows Azure hosted application development
  • 88. 3 Advanced CodedUI
  • 90. Cleaning the action recording When: executing the same test case on multiple environments or when using test data iteration or … and don’t forget the shared steps
  • 92. CodedUI from existing action recording from Manual Tester
  • 93. CodedUI from existing action recording from Manual Tester
  • 94. CodedUI from existing action recording from Manual Tester 2 1 4 3
  • 95. CodedUI Create a new action recording by Developer
  • 96. UIMap
  • 97. UIMap
  • 100. Test Infrastructure Where and how to run the automated test?
  • 101. Test Environments See also on www.TMap.net: - Checklist "Intake Test Environment“ - Checklist "Test Environment“ - Infrastructure Phase
  • 102. Test lab Visual Studio 2010 Lab Management
  • 103. Where and how to run the automated test? physical test environments virtualized test environments
  • 104. physical test virtualized test environments environments MTLM Client MTLM Client Physical Test Client TFS 2010 System Center Virtual TFS 2010 Machine Manager (SCVMM) Hyper-V Library Physical Host Shares Test Client VM VM share share
  • 105. Build application physical test environments versus virtualized test environments Provision Run tests ‘clean’ env Deploy application
  • 106. Lab for Physical Environments, the different flavors… Visual Studio Team Foundation Server Test Manager Build Server Test Agent Test Agent Test Agent
  • 107. Configurations for automated test execution: A Flavor A: Execution from VS2010… Real developer test / test automation dry run (Triggered by: right mouse click – run tests, in test project) http://msdn.microsoft.com/en-us/library/dd286580.aspx No additional configuration needed* * Data Diagnostic Settings in the Local.Testsettings file are configurable. Pro: Con: Easy setup No collection of test results in TFS Debug-able test automation Run on developer environment
  • 108. Configurations for automated test execution: B Flavor B: Execution during build with Build controller… not recommended, strange to run UI tests on build Part of Build Verification Tests (triggered by: build) Set the Build Service to run in interactive mode http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx Configure the build to run the tests http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType * Data Diagnostic Settings in the *.Testsettings file are configurable. Pro: Con: Easy setup No collection of test results in TFS Test results in build result Build Controller needs to run in interactive mode Tests are executed on build environment Run on build environment
  • 109. Configurations for automated test execution: C Flavor C: Execution during build with Test controller… B Preferred configuration above flavor Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B) Pro: Con: Test run on test environments No collection of test results in TFS Tests run on multiple environments Harder to configure Test Results in Build result Need for specific test client environments Test Settings from VS
  • 110. Configurations for automated test execution: D Flavor D: Execution from Microsoft Test Manager…than BVT Other type of test Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html Pro: Con: Test run on test environments Manually triggered by Tester Tests run on multiple environments Hard to configure Test Result in MTM and TFS Need for specific test client (same as MTM?) Test Settings from MTM
  • 111. Configurations for automated test execution: E Flavor E: Execution from MTM duringPreferred configuration above flavor C Build… Part of BVT Flavor D and E can be configured together (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run TCM or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx Pro: Con: Test run on test environments Hard to configure Tests run on multiple environments Need for specific test client (same as MTM?) Test Result in MTM and TFS Triggered by build
  • 120. Execute CodedUI During the build with Visual Studio 2010
  • 121. Configurations for automated test execution: C Flavor C: Execution during build with Test controller… B Preferred configuration above flavor Part of Build Verification Tests on multiple test agents (triggered by: build) Configure Test Controller (don’t register it with a project collection ) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers Configure Test Agents on clients (interactive mode) http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents Configure *.Testsettings file in solution to use Test Controller and Test Agents Configure the build to run the tests (see B) Pro: Con: Test run on test environments No collection of test results in TFS Tests run on multiple environments Harder to configure Test Results in Build result Need for specific test client environments Test Settings from VS
  • 123. 2 Add new Test Settings file Set remote execution and test controller.
  • 124. 3 Create a new build definition Set the Automated Test Properties
  • 125. 4 Queue a new Build Watch the Test Agent and Test Environment
  • 126. Execute CodedUI Within a test case as an associated automation manualy
  • 127. Configurations for automated test execution: D Flavor D: Execution from Microsoft Test Manager…than BVT Other type of test Part of Regression Tests (triggered by: MTM user, right mouse click on test case, run) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. http://msdn.microsoft.com/en-us/library/ee390842.aspx http://msdn.microsoft.com/en-us/library/dd293551.aspx Associate CodedUI test with WI Test Case from VS. http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html Pro: Con: Test run on test environments Manually triggered by Tester Tests run on multiple environments Hard to configure Test Result in MTM and TFS Need for specific test client (same as MTM?) Test Settings from MTM
  • 133. 5 Run Automated Tests from Test Manager
  • 134. Execute CodedUI Within a test case as an associated automation from the build
  • 135. Configurations for automated test execution: E Flavor E: Execution from MTM duringPreferred configuration above flavor C Build… Part of BVT Flavor D and E can be configured together (triggered by: Build) Configure Test Controller (register it with a project collection ) Configure Test Agents on clients (interactive mode, can be the same as MTM) Configure Lab Center in MTM to use test controller and create test ‘agent’ environment. Associate CodedUI test with WI Test Case from VS. Create Build task to run MTC or MSTEST task for Test Plan http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx Pro: Con: Test run on test environments Hard to configure Tests run on multiple environments Need for specific test client (same as MTM?) Test Result in MTM and TFS Triggered by build Test Settings from MTM
  • 136. Execute automated test cases on different environments
  • 137. 0 Same steps as previous.
  • 138. 1 Configure additional environments in Test Plan properties. Win 7 IE9 configuration Win 7 IE8 configuration TC MTLM Client Win 7 IE8 configuration Physical Test Client TFS 2010 Win 7 IE9 configuration Physical Test Client
  • 139. 1 Get Test Plan ID Get Test Suite Get Test Configurations with TCM.exe
  • 140. 2 Create command line for Test Run for the different Test Configurations "run /create /title:""Run on Host for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=13"" /settingsname:""Automated Test on Windows 7 Environment IE9"" /testenvironment:""Win 7 IE9"" /collection:http://clemenes- pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication" "run /create /title:""Run on test env for specific Test Point"" /planid:16 /querytext:""SELECT * FROM TestPoint WHERE ConfigurationId=12"" /settingsname:""Automated Test on Windows 7 Environment IE8"" /testenvironment:""Win 7 IE8"" /collection:http://clemenes- pc:8080/tfs/defaultcollection /teamproject:EmployeeApplication"
  • 142. 6 Queue a new Build Watch the Test Agent and Test Environment
  • 143. Build application Provision Run tests ‘clean’ env Deploy application
  • 144. VS2010 Lab Management • Reproduce the exact conditions of a bug or other development issue. • Build, deploy, and test applications automatically in a clean environment. • Reduce the time required to create and configure machines for testing an application. • Run multiple copies of a test or development at the same time. • Enable members of a team to create and manage virtual environments without requiring system administrator privileges.
  • 145. MTLM Client TFS 2010 System Center Virtual Machine Manager (SCVMM) Hyper-V Library Host Shares VM share VM share