ICSE17Tech Briefing - Automated GUI Testing of Android Apps: From Research to Practice
3 likes1,299 views
The document discusses the current state of automated GUI testing for Android applications, highlighting various challenges, tools, and methods used in the industry. It emphasizes the importance of GUI testing for software quality and the need for automation to improve efficiency amidst continuous app development and updates. Additionally, it outlines the future research vision for fully automated mobile testing, addressing critical challenges such as test oracles, fragmentation, and mobile-specific fault models.
![Key Challenges: Fragmentation
http://iossupportmatrix.com/
BY
iOS 10
Code names; Whitetail
iOS 9
Code names; Monarch, Boulder,
Castlerock, Eagle
iOS 8
Code names; Okemo, OkemoTaos,
OkemoZurs, Stowe, Copper
iOS 7
Code names; Innsbruck, Sochi
iOS 6
Code names; Sundance, Brighton
iOS 5
Code names; Telluride, Hoodoo
iPhone SDK 4.0
Code names; Apex, Baker,
Jasper, Phoenix, Durango
iPhone SDK 3.0
Code names; Kirkwood,
Northstar, Wildcat
iPhone SDK 2.0
Code names; Big Bear,
Sugarbowl, Timberline
iPhone OS 1.0
Code names; Alpine, Heavenly
Little Bear, Snobird, Oktoberfest
3.0
5.0
6.0 6.0 6.0 6.0
7.0 7.0 7.0 7.0
1.0 1.1
3.1.3
▼
3.1.3
▼
4.2.1
▼
4.2.1
▼
2.0 2.1.1
4.3.5
3.1.1
3.2
iPad SDK
4.0 (GSM)
4.2.6 (CDMA) 4.2.1 4.3.5
5.1
8.1 8.0 8.0 8.1 8.4
9.0 9.0 9.0 9.1 9.3 9.3
6.1.6
▼
5.1.1
▼
5.1.1
▼
7.1.2
▼
6.1.6
▼
9.3.3
▼
9.3.3
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
10.0
▼
9.3.3
▼
9.3.3
▼
9.3.3
▼
iPadPro9.7”
March2016
iPhoneSE
March2016
iPadPro12.9”
November2015
iPhone6sPlus
September2015
iPhone6s
September2015
iPadmini4
September2015
iPodtouch
(6thgeneration)July2015
iPadAir2
October2014
iPhone6Plus
September2014
iPhone6
September2014
iPadmini3
October2014
iPadmini2
October2013
iPadAir
October2013
iPhone5s
September2013
iPhone5c
September2013
iPadOctober2012
iPhone5
September2012
iPadmini
October2012
iPodtouch
(5thgeneration)September2012
[new]iPad
March2012
iPhone4S
October2011
iPad2
March2011
iPodtouch
(4thgeneration)September2010
iPhone4
June2010
iPadApril2010
iPodtouch
(3rdgeneration)September2009
iPhone3GS
June2009
iPodtouch
(2ndgeneration)September2008
iPodtouch
September2007
iPhone3G
July2008
iPhone
June2007
32bit/Swift/ARMv7-A
A6 64bit/Cyclone/ARMv8-A
A7 64bit/Typhoon/ARMv8-A
A8 64bit/Twister/ARMv8-A
A932bit/Cortex A8/ARMv7-A
A4 32bit/Cortex A9/ARMv7-A
A5
128MB 256MB 256MB 256MB512MB 512MB 512MB1GB 1GB 1GB 1GB 2GB 2GB 4GB2GB 2GB1GB
@isupportmatrix
iOSSupportMatrix.com
Summer 2016 — v4.0b6
B E T A
6](https://image.slidesharecdn.com/icse17-android-testing-tech-briefing-170523161504/85/ICSE17Tech-Briefing-Automated-GUI-Testing-of-Android-Apps-From-Research-to-Practice-172-320.jpg)
ICSE17Tech Briefing - Automated GUI Testing of Android Apps: From Research to Practice
- 1. AUTOMATED GUI TESTING OF ANDROID APPS: FROM RESEARCH TO PRACTICE Kevin Moran Mario Linares-Vásquez Denys Poshyvanyk
- 2. Mario Linares-Vásquez Assistant Professor Universidad de los Andes Bogotá, Colombia [email protected] http://sistemas.uniandes.edu.co/~mlinaresv Kevin Moran Ph.D. candidate College of William and Mary Williamsburg, VA, USA [email protected] http://www.cs.wm.edu/~kpmoran Denys Poshyvanyk Associate Professor College of William and Mary Williamsburg, VA, USA [email protected] http://www.cs.wm.edu/~denys
- 3. PART 1: CURRENT STATE OF RESEARCH & PRACTICE PART 2: MOBILE TESTING CHALLENGES PART 0: BACKGROUND AND CORE CONCEPTS PART 3: PRELIMINARY SOLUTIONS & RESEARCH VISION
- 4. PART 0: BACKGROUND & CORE CONCEPTS
- 5. The Importance of GUI Testing • Several different types of testing are important for ensuring software quality:
- 6. The Importance of GUI Testing Unit Testing Performance Testing Regression Testing Integration Testing Compatibility Testing • Several different types of testing are important for ensuring software quality:
- 7. The Importance of GUI Testing Unit Testing Performance Testing Regression Testing Integration Testing Compatibility Testing • Several different types of testing are important for ensuring software quality: • For Mobile, GUI-Based Testing subsumes many other types of testing • GUI-Testing is typically expensive, and test scripts are difficult to maintain • There is a clear opportunity for automation to Improve development workflows
- 8. GUI Testing: The Main Idea UI Events
- 9. GUI Testing: The Main Idea UI Events Oracle
- 10. GUI Testing: The Main Idea Output, layout, exceptions, presentation logic, quality attributes, … UI Events Oracle
- 11. Test result GUI Testing: Core Concepts Oracle
- 12. GUI Testing: Core Concepts Oracle Test result
- 13. GUI Testing: Example Detecting and Localizing Internationalization Presentation Failures in Web Applications. Abdulmajeed Alameer, Sonal Mahajan, William G.J. Halfond. In Proceeding of the 9th IEEE International Conference on Software Testing, Verification, and Validation (ICST). April 2016.
- 14. GUI Testing: Example Detecting and Localizing Internationalization Presentation Failures in Web Applications. Abdulmajeed Alameer, Sonal Mahajan, William G.J. Halfond. In Proceeding of the 9th IEEE International Conference on Software Testing, Verification, and Validation (ICST). April 2016.
- 16. Inputs: combinatorial explosion GUI Testing (Challenges)
- 17. Inputs: combinatorial explosion GUI Testing (Challenges)
- 18. Inputs: combinatorial explosion Internationalization GUI Testing (Challenges)
- 19. Inputs: combinatorial explosion Internationalization GUI Testing (Challenges)
- 20. Inputs: combinatorial explosion Internationalization GUI Testing (Challenges) Responsive design
- 21. Inputs: combinatorial explosion Internationalization GUI Testing (Challenges) Responsive design
- 22. Inputs: combinatorial explosion Internationalization GUI Testing (Challenges) Responsive design Unexpected usage scenarios
- 23. Inputs: combinatorial explosion Internationalization GUI Testing (Challenges) Responsive design Unexpected usage scenarios
- 25. MONKEY TESTING !! AUTOMATED TESTING !!
- 26. Automated GUI Testing Output, layout, exceptions, presentation logic, quality attributes, … UI Events Monkey
- 28. Unique Challenges in Mobile Development
- 29. Thousands of apps are released and updated every day on the online store
- 30. apps - Google Play 2.8M downloads - Google Play65B releases (Android) since 2008 25
- 31. Large volume of crowdsourced requirements and ratings
- 32. Fragmentation at device and OS level
- 33. Pressure for continuous delivery
- 34. Manual testing is still preferred
- 35. Mobile-specific quality attributes, inputs, and scenarios
- 36. PART 1: STATE OF RESEARCH & PRACTICE FOR MOBILE TESTING
- 37. Overview of Tools & Services
- 38. Overview of Tools & Services • Automation Frameworks & APIs
- 39. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools
- 40. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools • Automated Input Generation Tools
- 41. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools • Automated Input Generation Tools • Bug & Error Reporting
- 42. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools • Automated Input Generation Tools • Bug & Error Reporting • Crowdsourced Testing
- 43. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools • Automated Input Generation Tools • Bug & Error Reporting • Crowdsourced Testing • Cloud Testing Services
- 44. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools • Automated Input Generation Tools • Bug & Error Reporting • Crowdsourced Testing • Cloud Testing Services • Device Streaming Tools
- 45. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools • Automated Input Generation Tools • Bug & Error Reporting • Crowdsourced Testing • Cloud Testing Services • Device Streaming Tools }Traditional Android Testing Tools and Approaches
- 46. Overview of Tools & Services • Automation Frameworks & APIs • Record & Replay Tools • Automated Input Generation Tools • Bug & Error Reporting • Crowdsourced Testing • Cloud Testing Services • Device Streaming Tools } } Traditional Android Testing Tools and Approaches Bug Reporting, Crowdsourcing and Services
- 47. ANDROID TESTING TOOLS & APPROACHES
- 48. Automation Frameworks/APIs (AF/A) TESTS JUnit, Espresso, UI Automator, Robotium Monkey
- 49. Testing Automation Frameworks/APIs UI Automator
- 60. Pros and Cons Automation Frameworks ✓ Easy reproduction ✓ High level syntax ✓ Black box testing - Learning curve - User-defined oracles - Expensive maintenance
- 61. Record and Replay (R&R) AUT/SUT UI Events
- 62. Record and Replay (R&R) AUT/SUT UI Events Recorder Script
- 63. Record and Replay (R&R) AUT/SUT UI Events Recorder Script Scripts
- 64. Record and Replay (R&R) AUT/SUT UI Events Recorder Script Scripts UI Events Monkey AUT/SUT UI Events
- 70. Pros and Cons Automation Frameworks ✓ Easy reproduction ✓ High level syntax ✓ Black box testing - Learning curve - User-defined oracles - Expensive maintenance Record & Replay ✓ Easy reproduction - Expensive collection and maintenance - Coupled to locations
- 71. Automated Input Generation (AIG) Techniques
- 72. Automated Input Generation (AIG) Techniques • Differing Goals:
- 73. Automated Input Generation (AIG) Techniques • Differing Goals: • Code Coverage
- 74. Automated Input Generation (AIG) Techniques • Differing Goals: • Code Coverage • Crashes
- 75. Automated Input Generation (AIG) Techniques • Differing Goals: • Code Coverage • Crashes • Mimic Real Usages
- 76. Automated Input Generation (AIG) Techniques • Differing Goals: • Code Coverage • Crashes • Mimic Real Usages • Three Main Types:
- 77. Automated Input Generation (AIG) Techniques • Differing Goals: • Code Coverage • Crashes • Mimic Real Usages • Three Main Types: • Random-Based
- 78. Automated Input Generation (AIG) Techniques • Differing Goals: • Code Coverage • Crashes • Mimic Real Usages • Three Main Types: • Random-Based • Systematic
- 79. Automated Input Generation (AIG) Techniques • Differing Goals: • Code Coverage • Crashes • Mimic Real Usages • Three Main Types: • Random-Based • Systematic • Model-Based
- 80. Random/Fuzz Testing (R/FT) Monkey X or Y ? AUT/SUT
- 81. Random/Fuzz Testing (R/FT) Monkey X or Y ? AUT/SUT Event x Invalid
- 82. Random/Fuzz Testing (R/FT) Monkey X or Y ? AUT/SUT Event x Invalid Event Y Valid
- 83. Random/Fuzz Testing (R/FT) ./adb shell monkey -p com.evancharlton.mileage 10000
- 84. Random/Fuzz Testing (R/FT) ./adb shell monkey -p com.evancharlton.mileage 10000
- 85. Pros and Cons Automation Frameworks ✓ Easy reproduction ✓ High level syntax ✓ Black box testing - Learning curve - User-defined oracles - Expensive maintenance Record & Replay ✓ Easy reproduction - Expensive collection and maintenance - Coupled to locations AIG: Random Based ✓ Fast execution ✓ Good at finding crashes - Invalid events - Lack of expressiveness
- 86. Aside: GUI Ripping Ripper/Extractor ModelAUT/SUT Monkey
- 87. Aside: GUI Ripping Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1
- 88. Aside: GUI Ripping Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1
- 89. Aside: GUI Ripping Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1
- 90. Aside: GUI Ripping Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1 A or B ?
- 91. Aside: GUI Ripping Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1 Event 1 A or B ?
- 92. Aside: GUI Ripping Ripper/Extractor ModelAUT/SUT Monkey Snapshot 1 Snapshot 1 GUI State 1 Event 1 Snapshot 2 Snapshot 2 GUI State 2 A or B ?
- 93. GUI State extraction Ripper/Extractor Computer/Mobile device OS - Framework - API - Utilities GUI State Events Monkey
- 95. Monkey A or B ? Systematic Exploration
- 96. Monkey A or B ? Breadth-First (BF)Depth-First (DF) Systematic Exploration
- 97. Monkey A or B ? Breadth-First (BF)Depth-First (DF) Random (Uniform) Random (A-priori distr.) Other options (online decision) Systematic Exploration
- 98. Tools: Google Robo Test https://firebase.google.com/docs/test-lab/robo-ux-test
- 99. Pros and Cons Automation Frameworks ✓ Easy reproduction ✓ High level syntax ✓ Black box testing - Learning curve - User-defined oracles - Expensive maintenance Record & Replay ✓ Easy reproduction - Expensive collection and maintenance - Coupled to locations AIG: Random Based ✓ Fast execution ✓ Good at finding crashes - Invalid events - Lack of expressiveness AIG: Systematic ✓ Achieves Reasonable Coverage ✓ May miss crashes - Can be time consuming - Typically cannot exercise complex features
- 100. Model-Based Testing (MBT) UI Events Model Monkey AUT/SUT
- 101. Model-Based Testing (MBT) UI Events Model Monkey AUT/SUT - Manually generated - Automatically generated (source code) - Ripped at runtime (upfront) - Ripped at runtime (interactive)
- 102. Pros and Cons Automation Frameworks ✓ Easy reproduction ✓ High level syntax ✓ Black box testing - Learning curve - User-defined oracles - Expensive maintenance Record & Replay ✓ Easy reproduction - Expensive collection and maintenance - Coupled to locations AIG: Random Based ✓ Fast execution ✓ Good at finding crashes - Invalid events - Lack of expressiveness AIG: Systematic ✓ Achieves Reasonable Coverage ✓ May miss crashes - Can be time consuming - Typically cannot exercise complex features AIG: Model Based ✓ Event sequences ✓ Automatic exploration - Some Invalid sequences - State Explosion - Incomplete models
- 103. Other Types of AIG Approaches • Recently New Approaches have been introduced for AIG: • Search-Based Approaches1 • Symbolic/Concolic Execution2 • NLP for Text Input Generation - @ICSE17 1Ke Mao, Mark Harman, and Yue Jia. 2016. Sapienz: multi-objective automated testing for Android applications. In Proceedings of the 25th International Symposium on Software Testing and Analysis (ISSTA 2016) 2Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of android applications. In Proceedings of the 38th International Conference on Software Engineering (ICSE '16)
- 104. BUG REPORTING, CROWDSOURCING, & SERVICES
- 105. Bug Reporting Services AUT/SUTThird Party Library Web Service
- 107. Bug Reporting Services • Features of Bug Reporting Services:
- 108. Bug Reporting Services • Features of Bug Reporting Services: • Video Recording
- 109. Bug Reporting Services • Features of Bug Reporting Services: • Video Recording • App Analytics
- 110. Bug Reporting Services • Features of Bug Reporting Services: • Video Recording • App Analytics • Automated Crash Reporting
- 111. Bug Reporting Services • Features of Bug Reporting Services: • Video Recording • App Analytics • Automated Crash Reporting
- 112. Pros and Cons Bug Reporting Services ✓ Allows for more details about field failures ✓ App Analytics can help with UI/UX design - Can be expensive - Requires integrating library - Typically do report GUI- traces
- 113. Crowdsourced Testing AUT/SUTThird Party Service
- 114. Crowdsourced Testing
- 115. Crowdsourced Testing • Types of Crowdsourced Testing Services:
- 116. Crowdsourced Testing • Types of Crowdsourced Testing Services: • Expert Testing
- 117. Crowdsourced Testing • Types of Crowdsourced Testing Services: • Expert Testing • Functional Testing
- 118. Crowdsourced Testing • Types of Crowdsourced Testing Services: • Expert Testing • Functional Testing • UX Testing
- 119. Crowdsourced Testing • Types of Crowdsourced Testing Services: • Expert Testing • Functional Testing • UX Testing • Security Testing
- 120. Crowdsourced Testing • Types of Crowdsourced Testing Services: • Expert Testing • Functional Testing • UX Testing • Security Testing • Localization Testing
- 124. Crowdsourced Testing: Examples https://www.pay4bugs.com Expert Testers Functional Testing UX Testing Security Testing Localization Testing
- 125. http://testarmy.com/crowd-testing Expert Testers Functional Testing UX Testing Security Testing Localization Testing Crowdsourced Testing: Examples
- 126. https://www.applause.com/testing/ Expert Testers Functional Testing UX Testing Security Testing Localization Testing Crowdsourced Testing: Examples
- 127. Pros and Cons Crowdsourcing Services ✓ Low effort required from developers ✓ Expert tester might uncover unexpected bugs - Can be expensive - May not fit within Agile workflows - Quality of Reports can vary Bug Reporting Services ✓ Allows for more details about field failures ✓ App Analytics can help with UI/UX design - Can be expensive - Requires integrating library - Typically do report GUI- traces
- 128. Cloud Testing & Device Streaming (Devices farm) Third Party Service Third Party Service
- 129. Tools: Xamarin Test Cloud https://www.xamarin.com/test-cloud
- 130. Tools: AWS device farm https://aws.amazon.com/device-farm/
- 131. Tools: Google Cloud Test Lab https://developers.google.com/cloud-test-lab/
- 135. Tools: STF http://openstf.io and SEMERU extension
- 136. Tools: STF http://openstf.io and SEMERU extension
- 137. Pros and Cons Crowdsourcing Services ✓ Low effort required from developers ✓ Expert tester might uncover unexpected bugs - Can be expensive - May not fit within Agile workflows - Quality of Reports can vary Bug Reporting Services ✓ Allows for more details about field failures ✓ App Analytics can help with UI/UX design - Can be expensive - Requires integrating library - Typically do report GUI- traces Device Streaming ✓ Allows remote users access to controlled devices ✓ Allows for collection of detailed user information - Can be difficult to configure - Relies on strong network connection - Cannot simulate mobile specific contexts like sensors
- 138. PART 2: CURRENT CHALLENGES IN MOBILE TESTING
- 139. Accidental Challenges in Mobile GUI-Testing
- 140. Accidental Challenges in Mobile GUI-Testing Accidental Challenges
- 141. Accidental Challenges in Mobile GUI-Testing Accidental Challenges Application Data and Cold Starts
- 142. Accidental Challenges in Mobile GUI-Testing Accidental Challenges Application Data and Cold Starts Bugs in Framework Utilities
- 143. Accidental Challenges in Mobile GUI-Testing Accidental Challenges Application Data and Cold Starts Bugs in Framework Utilities Bugs and Limitations in the SDK Tools
- 144. Accidental Challenges in Mobile GUI-Testing Accidental Challenges Application Data and Cold Starts Bugs in Framework Utilities Bugs and Limitations in the SDK Tools Challenges Scaling Concurrent Virtual Devices
- 145. Essential Challenges in Mobile GUI-Testing
- 146. Essential Challenges in Mobile GUI-Testing Essential Challenges
- 147. Essential Challenges in Mobile GUI-Testing Essential Challenges Test Oracles
- 148. Essential Challenges in Mobile GUI-Testing Essential Challenges Test Oracles Event Coordination
- 149. Essential Challenges in Mobile GUI-Testing Essential Challenges Test Oracles Event Coordination External Features
- 150. Essential Challenges in Mobile GUI-Testing Essential Challenges Test Oracles Event Coordination External Features External Dependencies
- 151. Accidental Challenges: Application Data and Cold Starts
- 152. Accidental Challenges: Application Data and Cold Starts adb shell rm <app-data-folder> adb shell pm clear <app-package-name>
- 153. Accidental Challenges: Bugs in Framework Utilities
- 154. Accidental Challenges: Bugs in Framework Utilities
- 155. Accidental Challenges: Bugs in Framework Utilities adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml
- 156. Accidental Challenges: Bugs in Framework Utilities adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml
- 157. Accidental Challenges: Bugs in Framework Utilities adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml Solution: Modify Framework Components for the AOSP
- 158. Accidental Challenges: Bugs and Limitations in the SDK Tools adb Server 1 adb Server 2
- 159. Accidental Challenges: Bugs and Limitations in the SDK Tools adb Server 1 adb Server 2 No more than 15 devices per adb server instance!
- 160. Accidental Challenges: Scaling Concurrent Virtual Devices Potential Solution: Use android-x86 virtual machines
- 161. Accidental Challenges: Scaling Concurrent Virtual Devices Potential Solution: Use android-x86 virtual machines
- 162. Essential Challenges: Test Oracles Potential Solution: Use a hash of the current screen hierarchy Drawbacks: May miss some fined grained information about the GUI (e.g. colors, images)
- 163. Essential Challenges: Test Oracles Potential Solution: Use a hash of the current screen hierarchy adb shell shell /system/bin/uiautomator dump /sdcard/ui_dump.xml Drawbacks: May miss some fined grained information about the GUI (e.g. colors, images)
- 164. Essential Challenges: Event Coordination
- 165. Essential Challenges: Event Coordination adb shell dumpsys window -a mAppTransitionState APP_STATE_READY APP_STATE_IDLE APP_STATE_TIMEOUT APP_STATE_RUNNING
- 166. Essential Challenges: External Features & Dependencies Potential Solution: Use real usage data from the field?
- 167. PART 3: FUTURE RESEARCH VISION
- 168. Fully automated mobile testing: Are we there yet? APP - Oracles - Models - Scripts - Strategies - Fuzzers - Rippers - MBT - Automation APIs
- 169. Key Challenges 1. Open problems without solutions (e.g., oracles generation) 2. Facets of mobile testing not investigated yet (e.g., mobile-specific fault models) 3. Specific “characteristics” in mobile testing that drastically impact the testing process (e.g., fragmentation)
- 171. Key Challenges: Fragmentation https://developer.android.com/about/dashboards/index.html 25 API versions 7 API versions 97% OS Marketshare
- 172. Key Challenges: Fragmentation http://iossupportmatrix.com/ BY iOS 10 Code names; Whitetail iOS 9 Code names; Monarch, Boulder, Castlerock, Eagle iOS 8 Code names; Okemo, OkemoTaos, OkemoZurs, Stowe, Copper iOS 7 Code names; Innsbruck, Sochi iOS 6 Code names; Sundance, Brighton iOS 5 Code names; Telluride, Hoodoo iPhone SDK 4.0 Code names; Apex, Baker, Jasper, Phoenix, Durango iPhone SDK 3.0 Code names; Kirkwood, Northstar, Wildcat iPhone SDK 2.0 Code names; Big Bear, Sugarbowl, Timberline iPhone OS 1.0 Code names; Alpine, Heavenly Little Bear, Snobird, Oktoberfest 3.0 5.0 6.0 6.0 6.0 6.0 7.0 7.0 7.0 7.0 1.0 1.1 3.1.3 ▼ 3.1.3 ▼ 4.2.1 ▼ 4.2.1 ▼ 2.0 2.1.1 4.3.5 3.1.1 3.2 iPad SDK 4.0 (GSM) 4.2.6 (CDMA) 4.2.1 4.3.5 5.1 8.1 8.0 8.0 8.1 8.4 9.0 9.0 9.0 9.1 9.3 9.3 6.1.6 ▼ 5.1.1 ▼ 5.1.1 ▼ 7.1.2 ▼ 6.1.6 ▼ 9.3.3 ▼ 9.3.3 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 10.0 ▼ 9.3.3 ▼ 9.3.3 ▼ 9.3.3 ▼ iPadPro9.7” March2016 iPhoneSE March2016 iPadPro12.9” November2015 iPhone6sPlus September2015 iPhone6s September2015 iPadmini4 September2015 iPodtouch (6thgeneration)July2015 iPadAir2 October2014 iPhone6Plus September2014 iPhone6 September2014 iPadmini3 October2014 iPadmini2 October2013 iPadAir October2013 iPhone5s September2013 iPhone5c September2013 iPadOctober2012 iPhone5 September2012 iPadmini October2012 iPodtouch (5thgeneration)September2012 [new]iPad March2012 iPhone4S October2011 iPad2 March2011 iPodtouch (4thgeneration)September2010 iPhone4 June2010 iPadApril2010 iPodtouch (3rdgeneration)September2009 iPhone3GS June2009 iPodtouch (2ndgeneration)September2008 iPodtouch September2007 iPhone3G July2008 iPhone June2007 32bit/Swift/ARMv7-A A6 64bit/Cyclone/ARMv8-A A7 64bit/Typhoon/ARMv8-A A8 64bit/Twister/ARMv8-A A932bit/Cortex A8/ARMv7-A A4 32bit/Cortex A9/ARMv7-A A5 128MB 256MB 256MB 256MB512MB 512MB 512MB1GB 1GB 1GB 1GB 2GB 2GB 4GB2GB 2GB1GB @isupportmatrix iOSSupportMatrix.com Summer 2016 — v4.0b6 B E T A 6
- 173. Key Challenges: Lack of fault models
- 174. Key Challenges: Lack of fault models Fault model (a.k.a., fault profile): Catalog describing recurrent (or uncommon) bugs associated with software developed for a particular platform or domain Fault Model Potential locations/features in source code to be tested
- 175. Key Challenges: History awareness in test cases
- 176. Key Challenges: History awareness in test cases Current automated techniques do not support system/ acceptance testing because they are not history aware. - Fuzzers are totally random - Rippers follow heuristics for GUI traversal - Test scripts are hard-coded
- 177. Key Challenges: Evolution/maintenance of test scripts
- 178. Key Challenges: Evolution/maintenance of test scripts Location-coupled scripts Need different versions for each device and are highly sensitive to changes in the GUI Device/Location-agnostic scripts Are resilient to device fragmentation but need to be updated when the GUI changes in terms of added/ deleted components
- 179. Key Challenges: Oracles generation
- 180. Key Challenges: Oracles generation Manually Codified Oracles (MCO) Usually implemented as assertions or expected exceptions Exceptions-As-Oracle (EAO) Crashes and errors reported during the execution of a test case Gui-state-As-Oracle (GAO) Expected transitions between windows, and GUI screenshots
- 181. Key Challenges: Multi-goal testing
- 182. Key Challenges Challenge Solution Fragmentation Fault models History-awareness Multi-Goal testing Test scripts evolution Oracles generation Crowd/cloud based testing Research is needed MBT is a promise Open problem Open problem Open problem
- 183. The CEL Testing Principles A “fully" automated solution for testing mobile apps should continually generate test cases, underlying models, oracles, and actionable reports of detected errors, without human intervention.
- 184. The CEL Testing Principles Automated testing of mobile apps should help developers increase software quality within the following constraints: (i) restricted time/budget for testing, (ii) needs for diverse types of testing, and (iv) pressure from users for continuous delivery
- 185. The CEL Testing Principles Continuous: - Continuous multi-goal testing under different environmental conditions - Any change to the source code or environment should trigger — automatically – a testing iteration on the current version of the app - Test cases executed during the iteration should cover only the impact set of the changes that triggered the iteration
- 186. The CEL Testing Principles Evolutionary: - App source code and testing artifacts (i.e., the models, the test cases, and the oracles) should not evolve independently of one another - The testing artifacts should adapt automatically to changes in (i) the app, (ii) the usage patterns, and (iii) the available devices/OSes
- 187. The CEL Testing Principles Large-scale: - CEL testing should be supported on infrastructures for parallel execution of test cases on physical or virtual devices. - The large-scale engine should be accessible in the context of both cloud and on- premise hardware -This engine should enable execution of test cases that simulate real conditions in-the-wild.
- 188. CEL Testing: Proposed Architecture APIs evolution monitor Changes Monitoring Subsystem Developers User reviews APIs Users On-device usages monitor Source code changes monitor Markets monitor Source code Impact analyzer Execution traces+ logs analyzer API changes analyzer User reviews + ratings analyzer Models repository Models generator Test cases generator Large Execution Engine Containers Manager Reports repository Reports generator Test cases runner Testing Artifacts Generation Artifacts repository Multi-model generator Multi-model repository Test cases + oracles Domain, GUI, Usage, Contextual, Faults models Multi-model
- 189. Multi-Model GUI model Usage model Domain model Context model Fault model Windows, transitions, components Single use cases and combinations; (un)common usages Entities, data types, values, enumerations External events, sensors, connectivity (Un)Common programming errors
- 190. Multi-Model + GUI model Usage model Domain model Context model Fault model
- 191. Multi-Model + GUI model Usage model Domain model Context model Fault model Multi-model
- 192. Multi-Model + GUI model Usage model Domain model Context model Fault model Multi-model - Multi-goal testing - Oracles generation - On-demand test cases generation
- 193. SEMERU-Tools - MonkeyLab: New method for Modeling Actions - CrashScope: Practical Automated Testing - Fusion: Enhanced Bug Reporting - CrashDroid: Automated Crash Reproduction - GEMMA: Multi-Objective Energy Optimization - ODBR: On-Device Bug Reporting
- 195. Discussion Questions • Potential solutions to challenges we covered? • Other future research directions? • How does the rise of native web-programming frameworks for mobile apps impact testing?