The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale
1 like552 views
The document discusses the evolution of continuous experimentation in software product development, emphasizing the journey from having data to becoming a data-driven organization at scale. It outlines challenges and stages of experimentation evolution—crawl, walk, run, and fly—while highlighting the importance of trustworthiness in data collection and analysis. Key insights include the need for technical, organizational, and business practices to adapt as scale increases.
The evolution of continuous experimentation in software product development: from data to a data-driven organization at scale
- 2. From Data to a Data-driven Organization at Scale
- 4. Key Learnings 1. The journey from a company with data to a data-driven company at scale is an evolution. 2. The experimentation does not need to be complex (at first)! 3. Trustworthiness enables scalability, and not the other way around! 4
- 5. Agenda • Background & Motivation • Research Method • The Experimentation Evolution Model • Crawl stage • Walk stage • Run stage • Fly stage • Conclusions and Q&A 5
- 6. Background & Motivation • Software companies are increasingly aiming to become data-driven. • Getting data is easy. Getting data that you can trust? Not that much... • What customers say they want/do differs from what they actually want/do. • Online connectivity of products opens new opportunities to collect and use data that reveals what the customers actually want/do, • Online Controlled Experiments (e.g. A/B tests) can enable Software Companies to more accurately identify what delivers value to their customers. 6
- 7. Twist… • Problem: Experimentation in large software companies is challenging. • Running a few A/B tests is simple. Scaling Experimentation not so much! • Challenges include instrumentation, data loss, data pipelines, assumption violations of classical statistical methods, finding the right metrics, etc. 7
- 8. Problem/Solution • Problem: Experimentation in large software companies is challenging. • Running a few A/B tests is simple. Scaling Experimentation not so much! • Challenges include instrumentation, data loss, data pipelines, assumption violations of classical statistical methods, finding the right metrics, etc. • Solution: We provide step-by-step guidance on how to develop and evolve the experimentation practices (technical, organizational and business). 8
- 9. Research Method • Inductive case study conducted in collaboration with the Analysis and Experimentation team. • Data Collection: • The study is based on historical data points (past experiments), and • Complemented with a series of semi-structured interviews, observations, and meeting participations. • Data Analysis: • We grouped the collected data in four buckets, and performed iterative category development to emerge with the three levels of evolution. 9
- 10. The Experimentation Evolution Model • Our model provides guidance on how to become data-driven at scale through the evolution of online controlled experimentation. • We identify four stages of experimentation evolution: • We identify the most important R&D activities to focus on in each of the stages in order to advance. 10
- 11. The Experimentation Evolution Model 11
- 12. The Experimentation Evolution Model 12
- 13. Crawl stage (1x experiments yearly) • Technical focus: • Logging of signals (clicks, dwell times, swipes, etc.) should be implemented, • Trustworthiness of collected data should be considered (data quality), • Analysis of the experiment results can be done manually. • Team Focus: • Product teams gain management support with the first experiments. • Business focus: • The Overall Evaluation Criteria consists of a few key signals. 13
- 14. Contextual Bar Experiment • Experiment Goal: • Identify whether the contextual command bar improves editing efficiency. • Value Hypotheses: • (1) increased commonality and frequency of edits, • (2) increased 2-week retention. • Outcome: • The initial experiment was unsuccessful due to logging misconfiguration. 14
- 15. Walk stage (10x experiments yearly) • Technical focus: • Starting to develop/integrate an experimentation platform. • Defining success metrics, debug metrics, guardrail metrics, and data quality metrics, • Team focus: • Product team designs and executes experiments related to their features. • Business focus: • The Overall Evaluation Criteria in this stage evolves from signals to a structured set of metrics (guardrail, success, and data-quality metrics) 15
- 16. The “Xbox deals” experiment 16 • Experiment Goal: • Identify the impact of showing the discount in the weekly deals stripe. • Value Hypotheses: • (1) increased engagement with the stripe • (2) no decrease in purchases. • Outcome: • Treatment C increased both engagement with the stripe and purchases made.
- 17. Run stage (100x experiments yearly) • Technical focus: • Features: alerting, control of carry over effects, experiment iteration, etc. • Learning experiments: Create comprehensive metrics. • Team Focus: • Experimentation expands to other feature teams (and other products), • Teams create, execute and monitor experiments. • Business focus: • The Overall Evaluation Criteria is tailored using the learning experiments. 17
- 18. The “MSN.com personalization” experiment 18 • Experiment Goal: • Identify the impact of ML sorting of articles in comparison to editor curated articles. • Value Hypotheses: • (1) ML curated articles increase engagement • Outcome: • At first, ML articles performed worse than editor curating. After a few iterations things changed!
- 19. Fly stage (1000+ experiments yearly) • Technical focus: • Features: interaction between experiments, autonomous shutdown, and an accumulation of institutional memory. • Experiment results should become intuitive (e.g. green for go / red for no-go) • Team Focus: • Product Teams experiment with every minor change in the portfolio, even the smallest bug fixes. • Business focus: • The Overall Evaluation Criteria is stable. • OEC can become used to set the performance goals for teams and a measure of their success. 19
- 20. Bing Bot Detection Experiment 20 • Experiment Goal: • Evaluate the improved detection of bots. • Value Hypotheses: • (1) No change to real user experience, • (2) Fewer resources used to compute search results. • Outcome: • ~10% saving on infrastructure resources.
- 21. 21
- 22. Conclusions 1. The journey from a company with data to a data-driven company at scale is an evolution: • (1) culture, (2) business, and the (3) technical capabilities. 2. The experimentation starts ‘easy’ and becomes challenging! • The need for a more detailed training arises when many experiments are being executed by many product teams • The need for a more sophisticated platform arises when many teams experiment and interfere with each other 1. Trustworthiness enables scalability, and not the other way around! 22
- 23. 23 Thank you!