Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
90 likes122,137 views
The document discusses A/B testing strategies at Netflix, highlighting the execution of numerous tests to drive data-driven product development. It emphasizes the challenges of conditional dependencies and the benefits of templating and packaging in their development process. The final thoughts encourage rapid learning and innovation through iterative testing.
Netflix JavaScript Talks - Scaling A/B Testing on Netflix.com with Node.js
- 1. Scaling A/B testing on Netflix.com with
- 2. welcome Alex Liu @stinkydofu @mjr578 Micah Ransdell
- 9. Test 1 A B
- 11. Test 1 A B C D E F G Test 2 A B C D E F G Test 3 A B C D E F G Test 4 A B C D E F G Test 5 A B C D E F G Test 6 A B C D E F G Test 7 A B C D E F G
- 12. Test 1 A B C D E F G Test 2 A B C D E F G Test 3 A B C D E F G Test 4 A B C D E F G Test 5 A B C D E F G Test 6 A B C D E F G Test 7 A B C D E F G
- 14. hundreds of new A/B tests per year
- 16. 2,105 566 685 CSS JSTemplates
- 17. 2.5 million unique css/js packages per push cycle
- 21. ➔ Iterate Quickly ➔ Complex Concepts, Minimal Code ➔ Give the Power to the People Why Node?
- 23. Templating
- 25. Payment Test Testing the addition of Direct Debit
- 26. Payment Test Testing the addition of Direct Debit
- 27. Payment Test Testing the addition of Direct Debit
- 28. Payment Test Testing the addition of Direct Debit
- 29. Payment Test Testing the addition of Direct Debit
- 31. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if
- 32. payment templatepayment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if
- 34. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if payment_method_cc payment_method_dd
- 35. payment templatepayment template Control Cell 2 Cell 3 Cell 4 Cell 5 if if ifif if payment_method_cc payment_method_dd
- 36. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd if if if if if
- 37. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd if if if if if
- 39. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd ?
- 40. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd payment_method.json
- 41. payment template Control Cell 2 Cell 3 Cell 4 Cell 5 payment_method_cc payment_method_dd payment_method.json
- 49. ➔ Developers can combine rules ➔ Improves template legibility ➔ Increases template reuse Benefits
- 51. Packaging
- 55. app.js oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
- 56. app.js oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep
- 57. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 58. app.js
- 59. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 60. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 61. 685 files...?
- 62. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 63. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 64. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 65. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 66. oldSearch newSearch dep1 dep2 dep3 dep4 dep5 dep6 sub-dep sub-dep sub-dep sub-dep sub-dep sub-dep sub-depsub-dep app.js
- 69. RequestBuild
- 71. oldSearch.js
- 72. newSearch.js
- 73. anatomy of a rule
- 75. app.js
- 79. nf-include-when
- 82. Step 1: Get the full dependency tree for the requested package from the registry.
- 83. Step 2: Run the rules.
- 84. Step 3: Filter out all deps that resolved false.
- 85. Step 3: Filter out all deps that resolved false.
- 86. Step 3: Filter out all deps that resolved false. ✓
- 87. ✓ Step 4: Filter out all extraneous sub-deps.
- 88. Step 5: Concatenate the files.
- 91. ➔ Leverage build time tools ➔ Leverage the server when you can ➔ Divide and conquer with modules Take aways
- 93. Bonus Round
- 94. be creative with the registry
- 97. CSS analysis
- 98. and now, the coolest part...
- 102. UI BundleUI Bundle UI Bundle UI Bundle UI BundleUI Bundle UI BundleUI Bundle UI Bundle
- 105. we never touch the file system
- 106. < 5ms average response time
- 107. ➔ Static analysis FTW ➔ Independent UI deployments ➔ Requests never touch the file system ➔ Fast package response times Wins
- 108. final thoughts
- 109. learn by doing
- 111. I have not failed. I’ve just found 10,000 ways that won’t work.” Thomas Edison “
- 112. simplify
- 113. thank you
- 114. questions? Alex Liu @stinkydofu @mjr578 Micah Ransdell