TMPA-2017: Unity Application Testing Automation with Appium and Image Recognition
2 likes1,833 views
The document discusses the introduction of automated smoke testing for Unity applications using Appium and image recognition, focusing on its integration into continuous integration pipelines and the challenges of testing mobile games on various devices. Key issues include designing GUI tests, maintaining a mobile device farm, and the complexities of testing on iOS devices due to specific system requirements. Despite the challenges, the authors conclude that automated testing has significantly improved their development process by identifying bugs and providing a safety net.
TMPA-2017: Unity Application Testing Automation with Appium and Image Recognition
- 1. Unity Application Testing Automation with Appium and Image Recognition Maxim Mozgovoy Evgeny Pyshkin University of Aizu, Japan
- 2. Introducing Automated Smoke Testing • Aimed at performing basic checkups: • Program runs at all. • Program is able to react to user input. • Program is able to connect to perform certain basic functions. • Can be a part of a continuous integration pipeline: • Run automated smoke tests for each new build. • (Already integrated into the Google Play Developer Console). 2
- 3. The Case of a Mobile Game • Resource-intensive application, may crash or run slowly. • May behave very differently on different devices. • Contains no native GUI controls. Contains elements that are easy to overlook with manual testing. • Stress tests may be beneficial (such as running the app for 5 hours or with poor internet connection). • Real mobile devices required (not emulators / simulators). 3
- 4. Problem 1: Engineering the Tests How to design and run a GUI / smoke test on a mobile device? De facto standard approach: • Connect mobile devices to a “test server” running some special software. • Execute testing scenarios on a remote machine via the server. 4 Client (runs tests) Server (operates mobile devices)
- 5. Server Software • Appium: • Can be installed on Mac / Windows / Linux. • Supports test scripts written in Java / JavaScript / Python / C# / etc. • Can test Android, iOS and Windows applications. • Calabash: • Supports Android and iOS applications via different server versions. • Supports Ruby tests scripts only. • Note: these systems work because target platforms explicitly support automation (via adb & UIAutomator on Android, UIAutomation & XCUITest on iOS, etc.) 5
- 6. Test Scripts • Logically, a test script is a sequence of statements like this: wait 10 sec tap location (100, 50) assert that OK button appears press OK button • In native GUIs Appium can directly access individual controls: 6 e = appium.find_element_by_class_name('android.widget.EditText') e.send_keys("hello") ok = appium.find_element_by_class_name('android.widget.Button') ok.click()
- 7. Testing Apps with Custom GUI (Unity-made Games) 7 similarity = 0.78 similarity = 0.96 Take screenshots with Appium and use OpenCV to recognize areas via matchTemplate() function. Note: 100% similarity cannot be achieved due to screen rendering differences. In both cases tests are quite slow (inherent limitations)
- 8. Problem 2: Maintaining a Mobile Farm • Ideally, each new build should be tested on a variety of mobile devices. • Solution: use a cloud mobile farm provider! (AWS, Bitbar, etc.) • Advantages: hassle-free setup, hundreds (!) of mobile devices. • Drawbacks: still not all devices, high costs: 14-17 US¢ per minute per device, while our tests take > 1hr on each device per build (+ stress tests). So, e.g., 5 devices, 1hr ≈ 45 USD. 8 Client Server New build Reports
- 9. 9 Building Your Own Mobile Farm + Appium + test scripts + Test Runner Software (in-house)
- 10. Test Runner • Gets fresh builds from TeamCity (build machine) • Runs all tests on all devices. • Generates HTML reports with game action logs and screenshots. • Still under heavy development. 10
- 11. The Adventure Begins (and Never Ends…) 11 First, you might want to use a small Windows or Linux server It will work perfect with any Android device …but not with iPhone or iPad
- 12. The Adventure Begins (and Never Ends…) 12 So you’ll have to ditch it and use a Mac
- 13. The Adventure Begins (and Never Ends…) 13 Then you’ll realize that long tests drain the battery and search for powered USB hubs …will find lots of them…
- 14. The Adventure Begins (and Never Ends…) 14 …and realize that “powered” means that they simply have some dedicated charging ports and get excited with their specs
- 15. The Adventure Begins (and Never Ends…) 15 …And finally find the hub that does the job (not without quirks, but nobody’s perfect) …then study its compatibility table:
- 16. The Adventure Begins (and Never Ends…) 16 Then realize that not all devices are created equal. For example: On Nexus 7 sometimes you need to make 2-3 attempts before an application is installed, it is a norm to get an INSTALL_FAILED_INVALID_URI error with no reason.
- 17. The Adventure Begins (and Never Ends…) 17 Then realize that not all devices are created equal. For example: On Xiaomi Redmi Note 3 Pro after a number of install / uninstall cycles you’ll get a “no storage memory left” error (a reboot fixes it till the next time)
- 18. The Adventure Begins (and Never Ends…) 18 Then realize that not all devices are created equal. For example: Kindle Fire (5th Gen) will sometimes refuse to unlock the screen to run tests (fixed by installing CyanogenMod)
- 19. The Adventure Begins (and Never Ends…) 19 Then realize that not all devices are created equal. For example: and Samsung Galaxy Tab E will not charge at all from anything but a wall charger. iPad 3 will not charge fast enough and will end up with an empty battery,
- 20. The Adventure Begins (and Never Ends…) 20 Running tests on iOS devices requires: • Specific version of Appium. • Specific version of Xcode and Mac OS. • Specific version of iOS (in practice it can’t be iOS 10, and once installed, no way back) • Ability to block all iOS on-the-air updates, otherwise the “Please update” popup will prevent the tests from running. • Understanding of numerous MacOS-specific tools. • Arcane operations with certificates and permissions. • Certain phase of the Moon. • Appium will still crash once in a while.
- 21. Conclusion 21 • Automated smoke tests worked really well for us. • Despite complications of initial setup and maintenance, we now consider them an integral part of our pipeline. • We found many bugs • More importantly, they provide a “safety net” feeling. • Appium has quirks, but there aren’t many choices around. • Image processing in general works reasonably well. • Most problems we had with tests are not related to image processing (they are basically logical errors). • These errors are often caused with changes that happen between the screenshot time and the test response time (e.g., appearance of a modal dialog window).