Building a Distributed & Automated Open Source Program at Netflix
2 likes382 views
Netflix has been open sourcing since 2010 to enhance engineering practices, attract talent, and align with the industry. They maintain a variety of open source projects, including tools for big data, microservices, and security, supported by cross-functional teams that ensure project health. Key contributors are managed through a structured system on GitHub, focusing on security, automation, and community engagement.
Building a Distributed & Automated Open Source Program at Netflix
- 1. Netflix Open Source Andrew Spyker (@aspyker) - Engineering Manager Building a distributed and automated open source program
- 2. About Netflix ● 86.7M members ● A few thousand employees ● 190+ countries ● > ⅓ NA internet download traffic ● 500+ Microservices ● Many 10’s of thousands VM’s ● 3 regions across the world
- 4. Trivia Netflix been open sourcing, since? a) Around the start of streaming service - 2007 b) Around when we went international - 2010 c) Around House of Cards release time - 2013
- 5. Answer 2010
- 6. Why does Netflix Open Source? Improve Engineering ● Great feedback from wider community ● Collaborate through open code Recruit new and retain engineering talent ● Hard problems are openly worked on
- 7. Industry Alignment Why does Netflix Open Source? Netflix moves to cloud 2008 2013 2016
- 9. Open Source Functional Areas ● Contribute to Hadoop, Hive, Pig, Parquet, Presto, Spark ● Genie - RESTful API’s for Big Data Jobs ● Lipstick - Graphical depiction of executing Pig jobs ● Aegisthus - Data pipeline from Cassandra to Big Data
- 10. Open Source Functional Areas ● Nebula - Plugins for gradle to simplify builds ● Animator - Bakes AMI’s from OS installation packages ● Spinnaker - New continuous delivery platform
- 11. Open Source Functional Areas ● Eureka, Ribbon, Hystrix - Cloud native, resilient IPC ● Karyon, Prana, Archius - Microservice App Frameworks ● Fenzo - Mesos advanced scheduling library
- 12. Open Source Functional Areas ● Photon - Java Interoperable File Format implementation ● VMAF - Perceptual quality metric algorithm and test toolkit
- 13. Open Source Functional Areas ● Raigad/Priam - Management/ops sidecars for ES and C* ● EVCache - Distributed, replicated memcache++ ● Dynomite - Dynamo layer on top of non-dynamo data stores
- 14. Open Source Functional Areas ● Spectator/Atlas - Monitoring and Telemetry client and server ● Vector - Fine grained per instance performance monitoring ● Vizceral - Worldwide traffic to microservice graph visualization
- 15. Open Source Functional Areas ● Security Monkey - Automated cloud security monitoring ● Scumblr/Sketchy - Internet intelligence gathering ● FIDO - Security event orchestration (analysis/response) ● Lemur - Simplified x.509 cert management ● Sleepy Puppy - Delayed cross site scripting framework
- 16. Open Source Functional Areas ● Work across front end technologies including Restify ● Falcor - Virtual JSON graph & optimized query to backends ● RxJS - Simplify Javascript async event based programming
- 17. Netflix’s approach to open source Form a small cross-functional team working group that centralizes OSS competence, assisting decentralized teams working with OSS spend less time focusing on the administrative aspects (legal, tooling, branding, monitoring, and community promotion).
- 18. Open source enabler - OSS Interest Group ● Internal mailing list ● Meets once per month ● Topics from developers ● Help each other with common problems
- 19. Trivia How many OSS projects does Netflix have? a) 59 b) 102 c) 176
- 20. Answer 176 Netflix (119) Spinnaker (17) nebula-plugins (40)
- 21. Open Source Shepherds ● Management with business context ● Consistency across related projects ● Document how area fits together ● Focus on OSS health of each area
- 22. Common tools accelerate developers ● Security ● Backup ● Github user/group repo management ● Project tracking ● Build systems ● CI systems
- 23. Security tools ● We scan code for ○ Access keys, credentials, email addresses, hostnames ● Provide tools and automation to ○ Scan before initial release ○ Scan repeatedly on github
- 24. Source code management ● Backup and archival ○ Github down != Netflix down ● Internal mirrors we could build from
- 25. Project Ownership All projects have ● Development lead, Management lead ● Shepherd from OSS function area Only projects with active leads stay active!
- 26. Github management ● Has to be easy ○ Otherwise, teams will go it alone ● Has to be automated ○ Self service - chat ops ○ Following secure best practices
- 27. Github user management Support bring github id ● User links to internal id ● All tools then can associate identity Two Factor Auth Enforcement ● Automation to boot users who don’t ● Be careful - education on recovery!
- 28. Github group management ● Owners ○ Limited group - due to power ○ Automate via chatops all owner actions ● Netflixer group ○ Full write permissions on all repos ● Outside contributors ○ Added by netflixers, validated over time
- 29. Github automated through chat ops
- 30. Overall Org Health Tracking Metrics we track ● Issues ○ open, closed, TTC ● Pull Requests ○ open, closed, TTC ● Last commit timing ● Stars/forks ● Num contributors
- 32. ● Repeatable builds ● deb/rpm files for OS package baking ● Reduces boilerplate for common best practices ● Standards for release/version mgmt Common Build For Gradle/Java nebula-plugins.github.io
- 33. Common CI Systems ● Travis CI ○ Populate .travis.yml and sh files ○ Standard targets for snapshots, candidates, and releases ○ Binary upload credentials handled ○ Consistency across projects ● Cloudbees ○ Job-dsl to create release jobs
- 34. Using Docker to make projects easier ● A running image is worth a thousand wiki documents ● Started with ZeroToDocker ○ Monolithic solution ○ Leveraged Dockerhub trusted builds
- 35. Introducing TravisCI Docker builds Function Dockerhub trusted builds TravisCI Docker support Github commit traceable builds ✔ ✔ Trusted build servers ✔ ✔ Full build control (labels, etc.) ✖ ✔ Easy to integrate with artifact releases ✖ ✔ ● Experimenting: OSSTracker & Genie ● Docker compose used across images
- 36. TODO Group ● Joined 2015 ● Collaborate on how to better collaborate ● Leverage TODO group’s work ○ Github focus ○ Automation innovations ● Good group for helping OSS companies
- 37. Trivia Which of the following does Hystrix lead in? a) Most PR’s closed d) Most Forks b) Most Issues closed e) Most contributors c) Most Stars
- 38. Answer All of the above
- 39. Recent NetflixOSS Releases CI at Netflix scale Multi-region deployment control Advanced CI/CD pipelines
- 40. Recent NetflixOSS Releases Chaos Monkey 2.0 ● Integrated with Spinnaker ● Termination scheduling better ● Termination event tracking Photon ● Java IMF implementation ● Parsing, Interpretation, Validation
- 41. Recent NetflixOSS Releases Vizceral ● React and Web Component ● Graph data to visualize traffic Dynomite ● Dynamo layer on top of data stores ● Redis and memcache ● Manager (config, multi-region, backup)
- 42. Questions? Andrew Spyker (@aspyker) - Engineering Manager