Php com con-2011
4 likes1,070 views
1) The document discusses the evolution of test automation and continuous deployment practices at a company over time. 2) Early on, deployments were weekly and error-prone, but tools like Deployinator automated deployments and increased the frequency to several per day. 3) Continuous integration tools like Buildbot and Jenkins were implemented to run tests on code commits and provide feedback, helping eliminate bugs and speeding development cycles. 4) Over time, the company refined their testing practices, moving more tests to PHPUnit for consistency, classifying test types, and implementing features like database mocking to make tests more reliable and maintainable.
Php com con-2011
- 1. Is It Handmade Code If You Use Power Tools? Photo by matsbeads on Etsy Laura Beth Denker (@elblinkin)
- 2. Is It Handmade Code If You Use Power Tools? In the beginning... •Deployments happened weekly •Development happened on branches •Release Manager role rotations efo re B
- 3. Is It Handmade Code If You Use Power Tools? Deployinator •One button deployment •Eliminated painful weekly deploys • Deployments went up to a handful a day •Eliminated release manager role rotations • If you committed code, it was your responsibility to push the button efo re B
- 4. Is It Handmade Code If You Use Power Tools? Voila! Continuous Deployment •Buy-in •One-button deployment process (NO fear!*) •Monitoring and graphs * NO fear in the deploy process, but still some healthy fear in the code changes
- 5. Is It Handmade Code If You Use Power Tools? “Confidence is contagious; so is lack of confidence.” Vince Lombardi
- 6. Is It Handmade Code If You Use Power Tools? Buildbot •Developer written and maintained CI System • Resulted in Test Infrastructure being convoluted with site architecture • Test infrastructure was untested •Tests written in Python • But we’re a PHP shop •Slow and resource intensive • Could not run on a developer VM •Non-deterministic failures • Flaky tests • Assertions in helpers • EC2 connectivity issues
- 7. Is It Handmade Code If You Use Power Tools? (Hudson) Jenkins CI •Open source •High community involvement •Highly configurable •Numerous plugins •Eliminated the pitfalls of developer written and maintained CI 21 J u ne
- 8. Is It Handmade Code If You Use Power Tools? PyUnit to PHPUnit •~1,500 tests moved to our lingua franca •~3 minutes to run new PHPUnit tests •Eliminated context-switching between languages ly 19 Ju
- 9. Is It Handmade Code If You Use Power Tools? Commit Mutex 1. Commit code to trunk (maybe merge a development branch into trunk) 2. Wait for Buildbot (and Jenkins) to tests a. If tests PASS, continue to Deployinator and begin deployment b. If tests FAIL, kick Buildbot (and Jenkins) because it might be a flake, then do any of the following i. Repeat step (b) until it passes ii. Debug failures and begin fixing trunk iii. Rollback Note: While there are committed changes in trunk that have not been deployed, DO NOT COMMIT
- 10. Is It Handmade Code If You Use Power Tools? Why is there a Commit Mutex? 1.Branch development 2.Tests cannot run in a timely fashion on the developer VMs
- 11. Is It Handmade Code If You Use Power Tools? Try Server v.1 •Jenkins parameterized build + shell script •Parameters • username • patch file •Shell script • Create a patch file • Create the JSON • Send a request to Jenkins via cURL •Branches NOT SUPPORTED •Eliminated the Commit Mutex
- 12. Is It Handmade Code If You Use Power Tools? g us t 19 Au
- 13. Is It Handmade Code If You Use Power Tools? Try Server v.1.1 •Jenkins parameterized build + PHP script •Parameters • username • patch file •PHP script • Support for creating git-svn patches • Support for dry runs • Support for uploading manually created patches
- 14. Is It Handmade Code If You Use Power Tools? Test Classification •Unit •Integration •Functional (System)
- 15. Is It Handmade Code If You Use Power Tools? Test Classification •Functional (System) • Provides confidence that a particular use case works • Unit • Integration
- 16. Is It Handmade Code If You Use Power Tools? Legacy CI System is Dead! er 15 pte mb Se
- 17. Is It Handmade Code If You Use Power Tools? Test Classification • Functional (System) •Integration • Tests interaction between a discrete number of components • A set of interacting classes • Interaction with an external service • Unit
- 18. Is It Handmade Code If You Use Power Tools? Test Classification • Functional (System) • Integration •Unit • Fast • Deterministic • NO database • NO file system • NO network • NO threading, forking processes • NO sleep • NO time dependencies • Tests a single object • Best way to obtain line coverage • Reduces debug time
- 19. Is It Handmade Code If You Use Power Tools? PHPUnit @group •@group to declare external dependencies •PHPUnit XML Configuration •Jenkins Project for each configuration er 21 Oc to b
- 20. Is It Handmade Code If You Use Power Tools? PHPUnit @group @group cache for tests that use memcache or another cache er 21 Oc to b
- 21. Is It Handmade Code If You Use Power Tools? PHPUnit @group @group database for tests that use a database like postgres or mysql er 21 Oc to b
- 22. Is It Handmade Code If You Use Power Tools? PHPUnit @group @group network for tests that talk to external services (in-house or 3rd party) er 21 Oc to b
- 23. Is It Handmade Code If You Use Power Tools? PHPUnit @group @group sleep for tests that call a sleep() function er 21 Oc to b
- 24. Is It Handmade Code If You Use Power Tools? PHPUnit @group @group time for tests that depend on the time of day or day of the year, etc. er 21 Oc to b
- 25. Is It Handmade Code If You Use Power Tools? PHPUnit @group @group smoke for tests that execute cURL and regex based tests against an instance of the web service under test er 21 Oc to b
- 26. Is It Handmade Code If You Use Power Tools? PHPUnit @group @group flaky for tests that fail without code changes er 21 Oc to b
- 27. Is It Handmade Code If You Use Power Tools? PHPUnit @group There is NO @group for unit tests. er 21 Oc to b
- 28. Is It Handmade Code If You Use Power Tools? PHPUnit @group •@group to declare external dependencies •PHPUnit XML Configuration •Jenkins Project for each configuration er 21 Oc to b
- 29. Is It Handmade Code If You Use Power Tools? 40 Deploys in a Day •November 18th, 2010 • <5 months after first dedicated Test & Automation engineer • <3 months after releasing Try Server • 1 week before Thanksgiving freeze •34 deploys the day after ber No ve m ! !! 18
- 30. Is It Handmade Code If You Use Power Tools? How do we keep the momentum? •The tests ported to PHPUnit are flaky • Re-building intermittent failures is wasting time in the push queue •We are hiring more engineers • We need to deploy as fast, if not faster •We are writing more tests • More tests can take more time, so we need to write faster tests • Started at 1,500 tests, now running >7,000 tests ber ! Cy y ! Mo n da
- 31. Is It Handmade Code If You Use Power Tools? Random Data BAD •Fixtures Framework •Written and maintained by developers •Original intent • Generate data to run test instance of the website •Unexpected usage • Flaky and hard to debug integration tests posing as unit tests
- 32. Is It Handmade Code If You Use Power Tools? DBUnit for Multiple Databases •Shard-ed database architecture •Our ORM talks to multiple databases • At least the index and a shard •PHPUnit_Extensions_Database_MultipleDatab ases •Increased clarity in old tests ported to PHPUnit • Data explicitly specified in YAML, not at random • Developer needs to know what databases 8test will touch ra em be De c
- 33. Is It Handmade Code If You Use Power Tools? Worth Mentioning: Try Server v.2 •Master try project in Jenkins • PHP Script executed in Jenkins shell builder • Started all sub-projects over cURL • Pinged all sub-projects for status • Used Jenkins Text Finder plugin to toggle project state • Sent a unified result e-mail •Eliminated problems finding test results be r8 De cem
- 34. Is It Handmade Code If You Use Power Tools? Worth Mentioning: Try Server v.2 be r8 De cem
- 35. Is It Handmade Code If You Use Power Tools? Why to mock? unit tests test the functionality of a single method in a single object with the expectations of the collaborating objects test case as a suitcase contains suits, a test case contains tests
- 36. Is It Handmade Code If You Use Power Tools? PHPUnit Mock Helpers Making it easier to create mocks, should support writing more unit tests • Tweaking prod code • Writing short, tested helpers for internal types • PHPUnit_Extensions_MockObject_Stub_ReturnMapping • UPDATE: Now in PHPUnit as $this->returnValueMap() • Still more to do... ry 31 Ja nua
- 37. Is It Handmade Code If You Use Power Tools? PHPUnit Extensions Open-sourced on Github ry 27 Fe b r ua
- 38. Is It Handmade Code If You Use Power Tools? Try Server v.3 Jenkins Master Project Plugin Try’d out on Try Server ch 24 M ar
- 39. Is It Handmade Code If You Use Power Tools? Pake Now test locally, just like Jenkins! Ap ril 5
- 40. Is It Handmade Code If You Use Power Tools? Deployinator Info in Jenkins! Jenkins Master Project Plugin Just like Try Jenkins Deployinator Plugin Special CLI Command ONE Jenkins job per deployment stage Previously, ~15 Jobs that were difficult to correlate with a push pr il 9 A
- 41. Is It Handmade Code If You Use Power Tools? On the Big Monitors!
- 42. Is It Handmade Code If You Use Power Tools? Still More to Come... •Flaky test resolution •More focused DBUnit Tests •Make pure unit testing even easier •Try to solve whatever comes next... on d !! Bey
- 43. Is It Handmade Code If You Use Power Tools? References Code As Craft http://codeascraft.etsy.com/ Deployinator http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/ Divide and Concur http://codeascraft.etsy.com/2011/04/20/divide-and-concur/ Etsy PHPUnit Extensions https://github.com/etsy/phpunit-extensions Jenkins CI http://jenkins-ci.org/ PHPUnit https://github.com/sebastianbergmann/phpunit/
- 44. Is It Handmade Code If You Use Power Tools? Wardrobe Furnished By Spring Daisies Short Patchwork Dress http://www.etsy.com/shop/pineapplepop of Nashville, TN Asymmetric Flower and Beads Necklace (Silver) Created by Michelle D’Netto, Software Engineer @ Etsy Knitted DROPS Bolero with Lace Pattern http://www.garnstudio.com/lang/en/visoppskrift.php?d_nr=127&d_id=24 Design by DROPS Design Kni t bu t te Knitted by Laura Beth Denker a l as d at t h , e f o r i t wa s C o n f, the t dre s o o d a r k s :-(
- 45. Is It Handmade Code If You Use Power Tools? Our software is artisanally crafted using the ancient techniques from the 1960s. All releases are aged in a handmade filled cave for at least 20 minutes before decanting to the Web. --Noah Sussman
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: \n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n
- #29: Reprise\n
- #30: \n
- #31: \n
- #32: \n
- #33: \n
- #34: Moved all logic except for the patch creation to Jenkins\n diffutils to filter appropriate diffs for each sub-project\n
- #35: \n
- #36: \n
- #37: \n
- #38: \n
- #39: \n
- #40: \n
- #41: \n
- #42: \n
- #43: \n
- #44: \n
- #45: \n
- #46: \n