Optimizing Dev Portals with Analytics and Feedback
Download as PPTX, PDF0 likes103 views
The document discusses optimizing developer portals through analytics and feedback, highlighting the unique challenges and opportunities involved. It emphasizes the importance of modern product development approaches, such as experimentation, to prioritize outcomes over outputs, reduce uncertainty, and improve user engagement. Recommendations include maintaining an experimental mindset, focusing on learning from experiments, and using multiple key measures for success.
Optimizing Dev Portals with Analytics and Feedback
- 1. Optimizing Dev Portals with Analytics and Feedback James Noes - PM
- 2. 2 2 To optimize a product with analytics and feedback, we must follow practices that emphasis the importance of analytics and feedback.
- 3. 3 Overview • Unique challenge of opportunity for Dev Portals • Side effects of having too many opportunities • How might we use experiments to help us maintain good practices and better optimize the experience
- 4. 4 Many Opportunities Partner Integrations Accounts Blog Catalog Newsletter Documentation FAQ Analytics What’s New Try it Now Security Community Notifications Support Events Search Feedback API Management Forums Reporting Use Cases Success Stories Terms of Service Payments RBAC Governance Health & Status About Us Standards SDKs Tutorials Policies Getting Started Dev Tools Social Media Release Notes Roadmap Pricing Administration Videos Scalability Reliability & Performance Developer Experience Product Pages API Access Test Automation Frameworks Dependencies Database Management Tech Debt Insights Best Practices Design Open Source Monitoring & Alerting Infrastructure Accessibility API Reference Careers Marketing News & Press Testing Monetization
- 5. 5 Prioritization Partner Integrations Accounts Blog Catalog Newsletter Documentation FAQ Analytics What’s New Try it Now Security Community Notifications Support Events Search Feedback API Management Forums Reporting Use Cases Success Stories Terms of Service Payments RBAC Governance Health & Status About Us Standards SDKs Tutorials Policies Getting Started Dev Tools Social Media Release Notes Roadmap Pricing Administration Videos Scalability Reliability & Performance Developer Experience Product Pages API Access Test Automation Frameworks Dependencies Database Management Tech Debt Insights Best Practices Design Open Source Monitoring & Alerting Infrastructure Accessibility API Reference Careers Marketing News & Press Now Next & Later Testing Monetization
- 6. 6 Side effects of many opportunities • Creating a detailed long-term roadmap • The habit of “checking things off the list” • Not questioning return on investment • Building to meet stakeholder requirements • Focusing on outputs, not outcomes • Less emphasis on analytics and feedback
- 7. 7 Traditional vs. Modern Product Development Product Development (Modern) Feature Factory (Traditional)
- 8. 8 How might we avoid it?
- 9. 9 What's the difference? Foundation & Features Experiments • Minimal uncertainty • Starts with “We know by doing.. We will..” • May take weeks, months, or even years • Typically checked off “the list” because they have less uncertainty • Only revisited when impacted by another feature • Moderate-high uncertainty • Starts with “We believe by doing.. We expect.. • Timeboxed to emphasis return on investment • Helps us refine “the list” with data • Structured to emphasis assessing outcomes, learning, and iteration
- 10. 10 Example Experiment Hypothesis We believe that by providing use cases for our API Products, that we will increase in active apps (consumer). Scope/Requirements • Limit development and testing to 1 sprint • Experiment will last 2 months (no changes to use cases) • No automated testing required • Accessible in top 20% of homepage • Use cases must have CTAs Key Measures • Number of active apps • Unique Views • Time on Page • CTA Engagement Success Criteria • 6% increase in active apps over previous 2 months • 25% of new users engage with use cases • Time on page = at least 70% of read time • 30% of users click CTA
- 11. 11 Traditional vs. Modern Product Development Product Development (Modern) Feature Factory (Traditional)
- 12. 12 Assessing Results Success Criteria • 6% increase in active apps over previous 2 months • 25% of new users engage with use cases • Time on page = at least 70% of read time • 30% of users click CTA Scenario 1 • 8% increase in active apps • 60% of new users engage with use cases • Time on page = 90% • 50% of users click CTA Experiment was successful – it is obvious that use cases contributed to outcome. Scenario 2 • 1% increase in active apps • 8% of new users engage with use cases • Time on page = 20% • 5% of users click CTA Experiment was unsuccessful – unlikely that will have an impact. Should discuss removing use from application to avoid a cluttered experience. Scenario 3 • 3% increase in active apps • 10% of new users engage with use cases • Time on page = 100% • 80% of users click CTA Experiment did not achieve expected results - that use case could be effective. Consider experiment with placement.
- 13. 13 Recommendations • Determine how much of your time you can afford to spend on experimentation (20% or less to start) • Bring the experimental mindset back into new feature development • Always have multiple key measures (3-4 is typically best) • Focus on experiments as a learning opportunity • Embrace learning – avoid output as a measure of success • Unsuccessful experiments are a success if you minimize investment and learn
- 14. Thank you!