Best Practices & Lessons Learned From Over 223,628 A/B Tests

1. Best Practices and Lessons Learned from Dan Siroker Co-Founder & CEO A/B Tests223,628 Wednesday, June 5, 13

2. ‘05 ‘06 ‘07 ‘08 ’09 ’10 Wednesday, June 5, 13

3. Wednesday, June 5, 13

5. Lessons Learned Wednesday, June 5, 13

6. 1. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Seek the global maximum. 5. Fail fast. 6. Start today. Lessons Learned Deﬁne quantiﬁable success metrics. Wednesday, June 5, 13

7. Lesson #1: Deﬁne quantiﬁable success metrics. Wednesday, June 5, 13

8. Visit visitors per day Wednesday, June 5, 13

9. Visit visitors per day Email sign up rate Wednesday, June 5, 13

10. Visit visitors per day Email sign up rate $ $ per recipient Wednesday, June 5, 13

11. Visit Email $ sign up rate $ per recipient visitors per day Wednesday, June 5, 13

12. Media Button Wednesday, June 5, 13

13. Button: 1. Sign Up 2. Learn More 3. Join Us Now 4. Sign Up Now Media: 1. Get Involved Image 2. Family Image 3. Change Image 4. Barack’s Video 5. Springﬁeld Video 6. Sam’s Video Splash page experiment Variations: Wednesday, June 5, 13

14. Button: “Sign Up” Wednesday, June 5, 13

15. Button: “Learn More” Wednesday, June 5, 13

16. Button: “Join Us Now” Wednesday, June 5, 13

17. Button: “Sign Up Now” Wednesday, June 5, 13

18. Button: 1. Sign Up 2. Learn More 3. Join Us Now 4. Sign Up Now Media: 1. Get Involved Image 2. Family Image 3. Change Image 4. Barack’s Video 5. Springﬁeld Video 6. Sam’s Video Variations: Splash page experiment Wednesday, June 5, 13

19. Media: “Get Involved” Wednesday, June 5, 13

20. Media: “Family” Wednesday, June 5, 13

21. Media: “Change” Wednesday, June 5, 13

22. Media: “Barack’s Video” Wednesday, June 5, 13

23. Media: “Springﬁeld Video” Wednesday, June 5, 13

24. Media: “Sam’s Video” Wednesday, June 5, 13

31. Email Subscriptions Volunteers Amount Raised Original: 7,120,000 712,000 $143,000,000 +40.6% +2,880,000 +288,000 +$57,000,000 New: 10,000,000 1,000,000 $200,000,000 Splash page experiment Wednesday, June 5, 13

34. Lesson #1: Deﬁne quantiﬁable success metrics. Wednesday, June 5, 13

35. Lesson #2: Less is more. Reduce choices. Wednesday, June 5, 13

36. vs. Wednesday, June 5, 13

37. vs. +15% conversion rate +10% $ per visitor Wednesday, June 5, 13

39. vs. +8% engagement Wednesday, June 5, 13

41. vs. +16.8% $ per visitor Wednesday, June 5, 13

42. v. Wednesday, June 5, 13

43. v. +14.8% conversion rate Wednesday, June 5, 13

46. Lesson #2: Less is more. Reduce choices. Wednesday, June 5, 13

47. Lesson #3: Words matter. Focus on your call to action. Wednesday, June 5, 13

48. Donation button experiment Variations Wednesday, June 5, 13

49. Never Signed Up Signed Up, Never Donated Previously Donated 0.0% 0.0% 0.0% +2.3% +27.8% +16.3% -27.8% N/A N/A +15.2% -24.6% +11.9% +8.5% +2.9% +18.4% Donation button experiment Variations Wednesday, June 5, 13

51. vs. +14.6% clicks Wednesday, June 5, 13

53. vs. +15.75% $ per visitor Wednesday, June 5, 13

54. Lesson #3: Words matter. Focus on your call to action. Wednesday, June 5, 13

55. Lesson #4: Seek the global maximum. Wednesday, June 5, 13

56. Source: http://blog.intercom.io/criticism-and-two-way-streets/ Wednesday, June 5, 13

58. vs. Comments: http://blog.netﬂix.com/2011/06/new-look-and-feel-for-netﬂix-website.html Wednesday, June 5, 13

59. vs. +0.5% retention +1.4% engagement Comments: http://blog.netﬂix.com/2011/06/new-look-and-feel-for-netﬂix-website.html Wednesday, June 5, 13

61. vs. +600% engagement Wednesday, June 5, 13

63. vs. +49% conversion rate from 240 A/B tests Wednesday, June 5, 13

64. Lesson #4: Seek the global maximum. Wednesday, June 5, 13

65. Lesson #5: Fail fast. Wednesday, June 5, 13

67. vs. -92.3% video clicks Wednesday, June 5, 13

69. vs. -10% conversions Wednesday, June 5, 13

71. vs. fewer conversions Wednesday, June 5, 13

72. Lesson #5: Fail fast. Wednesday, June 5, 13

73. Lesson #6: Start today. Wednesday, June 5, 13

76. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/ Wednesday, June 5, 13

77. Source: http://techcrunch.com/2013/04/10/optimizely-series-a/ Wednesday, June 5, 13

78. 4,000+ Happy Customers! Wednesday, June 5, 13

79. Lesson #6: Start today. Wednesday, June 5, 13

80. 1. Deﬁne quantiﬁable success metrics. 2. Less is more. Reduce choices. 3. Words matter. Focus on your call to action. 4. Seek the global maximum. 5. Fail fast. 6. Start today. Lessons Learned Wednesday, June 5, 13

81. The Future of Optimization Wednesday, June 5, 13

82. The Future of Optimization What’s the best experience for the average user? One to Many Wednesday, June 5, 13

83. The Future of Optimization What’s the best experience for the average user? One to Many One to Few What’s the best experience for segments of users? Wednesday, June 5, 13

84. The Future of Optimization What’s the best experience for the average user? One to Many One to Few What’s the best experience for segments of users? One to One What’s the best experience for each user? Wednesday, June 5, 13

85. One to Many Wednesday, June 5, 13

86. One to Few Wednesday, June 5, 13

87. One to One Wednesday, June 5, 13

94. Best Practices and Lessons Learned from Dan Siroker Co-Founder & CEO A/B Tests223,628 Wednesday, June 5, 13

95. Appendix Wednesday, June 5, 13

Best Practices & Lessons Learned From Over 223,628 A/B Tests

Recommended

More Related Content

Similar to Best Practices & Lessons Learned From Over 223,628 A/B Tests (20)

More from Optimizely (20)

Recently uploaded (20)

Best Practices & Lessons Learned From Over 223,628 A/B Tests

Editor's Notes