DOES15 - Mark Michaelis - Metrics that Matter
Download as PPTX, PDF0 likes1,101 views
This document discusses metrics for measuring DevOps transformations. It begins by listing symptoms that can occur without proper measurement, such as downtime, customer dissatisfaction, and high employee turnover. It then discusses challenges with measurement, such as measuring too many things or prioritizing individual performance over team productivity. The document categorizes metrics by dimensions like operations, business, culture, and lifecycle stage. It provides examples of metrics for different stages like development and production. Principles of measurement discussed include automating metrics, using metrics to drive excellence, and ensuring metrics show trends over time. The document advocates measuring efficiency, effectiveness, and culture to optimize DevOps transformations.
DOES15 - Mark Michaelis - Metrics that Matter
- 1. Metrics that Matter A Myriad of Choices Efficiency Effectiveness Culture
- 3. • Down time outside of SLA • Customer dissatisfaction • Long mean time to repair/recover (MTTR) • Unacceptable Cost of change • Missed deadlines • High employee turnover • Large releases that still lack expected functionality and quality • … Symptoms Leading to Measurement
- 4. ● Complexity - Measuring too many things ● Gamed - Disconnected business and engineering measures ● Misleading - Emphasizing activities (process) rather than outcomes (products) ● Rejected - Prioritizing individual performance over team productivity ● Simplistic - Emphasizing values over trends ● Dishonest - Treating current values and targets as ground truth Have You Ever Experienced These?
- 6. A Myriad of Choices List see Itl.tc/DevOpsMetrics
- 7. A Categorization of choices
- 8. Category Dimensions CXO R&D HR Operations Business Development Cultural Operations Business Process People Operations Business Technology Discovery & Requirements Analysis Development & Test Integration/UAT Deployment Production Operations Business Technology Process People
- 9. Lead Time = Process Time + Wait Time Lead Time (LT): The elapsed time from receiving a customer request to delivering on that request. Process Time (PT): Process time begins when the work has been pulled into a doing state and ends when the work is delivered to the next downstream customer. Wait Time (WT): The time that work sits idle not being worked.
- 10. A Lifecycle of Metrics that Mater Discovery & Requirements Analysis Research & Development Integration/UAT Deployment Production Time to detect, time to discover/describe work item, time to validate, time to identify acceptance criteria Defects raised during UAT per release, Time per story point, Lead time to understand requirement per story point, Time to integrate, Time to implement per story point, Process time, Time to complete build, # experiments per release, Test code coverage and effectivity, Defects per, Release/Sprint, Area, Engineer, Time to complete automated tests, Unplanned work per, Release, Change per release, Work items, Story Points, Lines of code, Work in progress per, Release/Sprint, Engineer Time to integrate, defects raised during UAT, Total automated time, Total manual time, Deployment time, Total manual time, Total automated time, Frequency of failed deployments per Release, Time to rollback, Similarity of production to test environments Frequency of failed deployments per, Release, Time to rollback, Similarity of production to test environments, Time to escalate, Time to detect, Cost impact per, Outage, Down time, Outlier performance, Alerting efficiency, Time to mitigate, Time to detect, Time outside SLA, Frequency of outages, System response time Lead time, Unexpected expense per release, wait time per work items, Average wait time per work item per release/time-period, Unexpected expense per release, cost per release, frequency of customer tickets, total customers per time, cost to acquire customer, frequency of release/change, Employee retention/turnover, Skills per team/project, Satisfaction, Attitude towards continuous improvement, Technology experimentation, Team autonomy, Ownership vs. blame, #Individuals per skill (measuring cross-skilling) DevOps metric importance may be relative to lifecycle stage
- 11. Perspective Cost of new customer acquisition is more expensive than average profit per customer Total customers are increasing
- 12. Principles of Measurement • Don’t create cost for stuff that isn’t relevantRelevant • Automate measurement where possible, removing subjectivity and overheadAutomated • Use metrics as a driver for creating excellenceTransparent • Don’t’ look in the mirror and ignore what you see (Learn)Examined • Metrics are relevant with respect to a trend (its not the raw numbers).Comparative
- 13. Measure Efficiency, Effectiveness and Culture to Optimize DevOps Transformations White Paper
- 14. Moving to a Software-As-Service culture Seven habits of effective DevOps Dealing with high incident volumes Measuring signal/noise ratio is the single best measure Reducing complexity for integration environments Simplification and modernization, the only two paths to reducing technical debt Scenarios In Metrics Whitepaper
- 15. ● Chivas Nambiar, Director DevOps Platform Engineering, Verizon ● Dominica DeGrandis, Director, Learning & Development, LeanKit ● Eric Passmore, CTO MSN, Microsoft ● Gene Kim, co-author of upcoming "DevOps Cookbook“ ● Jeff Weber, Managing Director, Protiviti ● Mark Michaelis, Chief Technical Architect, IntelliTect ● Mirco Hering, DevOps Lead APAC, Accenture ● Nicole Forsgren, PhD, Director Organizational Performance & Analytics, Chef ● Sam Guckenheimer, Product Owner, Visual Studio Cloud Services, Microsoft ● Walker Royce, Software Economist ● You? Contributors
- 16. Takeaways •Create predictability •Focus your strategy •Transparent metrics trend towards excellence •Are different for each team/person/organization •Must be comparative Metrics
- 17. IntelliTect.com | [email protected] Thank you for your time. Questions?
- 18. About Us Mark Michaelis |
Editor's Notes
- #2: Placeholder during transition
- #7: If you have these symptoms you may be experiencing chaos Measurements and metrics are very important, yet we often face unexpected challenges
- #18: Interesting in how to succeed, come talk with us
- #19: Measurements and metrics are very important, yet we often face unexpected challenges
- #20: Thanks to our great list of contributors