Automated Visual Testing That Does Not Suck
Download as PPTX, PDF15 likes6,649 views
The document discusses the importance of automated visual testing to enhance quality assurance of graphical user interfaces by comparing screenshots with baseline images. It covers various tools and technologies, such as Applitools Eyes, PhantomCSS, and Wraith, that assist in visual testing and address challenges like false positives and image comparison. The workflow includes driving the application, capturing screenshots, and reporting differences, making it essential for developers, designers, and QA teams.
Automated Visual Testing That Does Not Suck
- 1. AUTOMATED VISUAL TESTING THAT DOES NOT SUCK Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools [email protected]
- 2. YOU CAN AND SHOULD AUTOMATE YOUR VISUAL TESTS!
- 4. AGENDA Why automated visual testing? Tools & Technology Where does it fit?
- 5. WHAT IS VISUAL TESTING? A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
- 6. A VISUAL BUG
- 7. WHY SHOULD IT BE AUTOMATED? THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY Web browsers Operating systems Screen resolutions Responsive designs L10n
- 8. WHY SHOULD IT BE AUTOMATED?
- 9. WHY SHOULD IT BE AUTOMATED? MANUAL VISUAL TESTING IS ERROR PRONE
- 10. WHY SHOULD IT BE AUTOMATED? Many are already doing it (and sharing)… PhantomCSS Fighting Layout Bugs CSSCritique Wraith Needle Grunt PhotoBox dpdxt WebdriverCSS Eyes Huxley FBSnapshotTestCase Selenium Visual Diff Gemini VisualCeption Specter Snap And Compare
- 11. AGENDA Why automated visual testing? Tools & Technology Where does it fit?
- 12. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 14. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 15. DRIVING AND CAPTURING OTHER PhantomCSS (CasperJS) Fighting Layout Bugs (Java) Wraith (URLs) CSSCritique (URLs) Needle (Python) dpdxt (URLs) Grunt PhotoBox (URLs) WebdriverCSS (JS) Eyes (All) Huxley (RP) FBSnapshotTestCase Selenium Visual Diff (Java) VisualCeption (PHP) Specter (JS) Wraith-Selenium (Ruby) Snap And Compare (URLs) Eyes (All) Gemini (JS) Gemini (JS)
- 16. SCREENSHOTS Real browsers? Full page? Frames? Regions? Viewport size? Page stabilization? Page preparation?
- 17. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 19. WHY NOT? FALSE POSITIVES
- 20. ANTI-ALIASING?
- 23. BRIGHTNESS 1/2
- 24. BRIGHTNESS 2/2
- 25. SUB PIXEL SHIFTS 1/2
- 26. SUB PIXEL SHIFTS 2/2
- 27. SUB PIXEL SHIFTS 1/2
- 28. SUB PIXEL SHIFTS 2/2
- 29. ANDMORE… 1 pixel offsets in element positioning Dynamic content Moving elements Images of different size
- 32. ImageMagick A powerful command line tool for image processing. APIs are available for most programming languages. Fuzzing is used to eliminate slight color differences An error ratio is usually used to determine a match $ compare –metric AE –fuzz 5% img1.png img2.png diff.png 2246
- 33. Resemble.js An image comparison tool implemented in Javascript Used by PhantomCSS and other tools. Good de-antialiasing An error ratio is usually used to determine a match http://huddle.github.io/Resemble.js/
- 34. Applitools Eyes A specialized image processing stack designed to compare computer generated UI images Anti-aliasing Partial and full pixel offsets Images of different size Dynamic content Moving elements Layout matching Available as a cloud service
- 35. DEMO
- 36. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 37. REPORT DIFFERENCES As files on the file system (combined with source control)
- 38. REPORT DIFFERENCES As a Gallery (example from Selenium Visual Diff)
- 39. THE WORKFLOW Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 40. UPDATE THE BASELINE Rename or commit individual image files Overwrite mode
- 41. UPDATE THE BASELINE GUI (Gemini GUI) Automatic maintenance (demo)
- 42. AGENDA Why automated visual testing? Tools & Technology Where does it fit?
- 43. WHERE DOES IT FIT? • Component s • Code review • Developers • Designers • QA
- 44. WHERE DOES IT FIT? • Pages • Page sections • Developers • Designers • QA • Others
- 45. WHERE DOES IT FIT? • Staging vs. Production • Ops • QA
- 46. WHERE DOES IT FIT? • Monitoring • Ops • QA
- 47. QUESTIONS?
- 48. THANK YOU Adam Carmi (@carmiadam) Co-Founder and VP R&D at Applitools [email protected]
Editor's Notes
- #3: What is Visual Testing? Sikuli??? It's too flaky and unstable for practical use…
- #11: 100% coverage No locators to find and maintain Cover functionality exposed through the UI
- #13: Most tools are for web apps except for FBSnapshotTestCase which is for IOS and Eyes which is platform agnostic.
- #15: * Does not apply to fighting layout bugs.
- #19: * Does not apply to fighting layout bugs.
- #20: Like Boromir said…
- #36: * Does not apply to fighting layout bugs.
- #39: * Does not apply to fighting layout bugs.
- #43: CSS files and HTML falling out of sync CSS changes with unexpected global implications. Frontend refactoring.
- #44: Project / Product managers (collaboration)
- #46: Data from external sources Not enough time to test pre-production Missing components in deployment.