Architectures That Scale Deep - Regaining Control in Deep Systems
1 like342 views
The document discusses the challenges of scaling deep systems in software architecture, focusing on observability and controllability as critical elements for managing complex, multi-layered services. It highlights the significance of effective service level objectives (SLOs) and the role of tracing in reducing cognitive load to improve system management. Furthermore, it emphasizes that deep systems require a different approach to observability, moving beyond traditional methods like metrics and logs by integrating traces for better context and understanding.
Architectures That Scale Deep - Regaining Control in Deep Systems
- 1. Ben Sigelman (@el_bhs, [email protected]) Co-founder & CEO: LightStep Co-creator: OpenTracing, OpenTelemetry, Google Dapper, Google Monarch Architectures that Scale Deep: Regaining Control in Deep Systems QCon SF, November 2019
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ properties-deep-systems/
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. Part I Scaling, and Deep Systems
- 5. What is scale, anyway?
- 6. Scaling wide
- 7. Scaling wide
- 8. Scaling wide
- 9. Scaling wide
- 10. Scaling wide
- 11. Scaling deep
- 12. Scaling deep
- 13. Scaling deep
- 14. Scaling deep
- 15. Scaling deep
- 16. How does this look for software?
- 19. How do real-world systems look?
- 20. Microservices at scale aren’t just wide systems, they’re deep systems
- 21. Deep Systems Architectures with ≥ 4 layers of independently operated services (including external/cloud dependencies) Deep Systems Architectures with ≥ 4 layers of independently operated services (including external/cloud dependencies)
- 22. What do deep systems sound like?
- 23. “Don’t deploy on Fridays” What do deep systems sound like?
- 24. “Where’s Chris?! I’m dealing with a P0 and they’re the only one who knows how to debug this.” What do deep systems sound like?
- 25. “It can’t be our fault, our dashboard says we’re healthy” What do deep systems sound like?
- 26. “Kafka is on fire” What do deep systems sound like?
- 27. “I need 100% availability from your team. One hundred percent.” What do deep systems sound like?
- 28. “I didn’t know I depended on that region” What do deep systems sound like?
- 29. “That was on a dashboard but I can’t find it” What do deep systems sound like?
- 30. Lots of challenges: - People-management - Security - Multi-tenancy - “Big-customer” success - Performance - Observability What do deep systems sound like?
- 31. Part II Control Theory: TL;DR Edition
- 32. Why do we care so much about observability, anyway?
- 34. A System Inputs Outputs … and its state vector,
- 35. Inputs A System Outputs … and its state vector, Observability How well can you infer internal state using only the outputs?
- 36. Outputs A System Inputs … and its state vector, Controllability How well can you control internal state using only the inputs?
- 37. Controllability is the dual of Observability
- 38. Controllability is the dual of Observability
- 39. Part III What Deep Systems Mean for Observability
- 41. Stress (n): responsibility without controlStress what you can control what you are responsible for
- 45. Managing Deep Systems Services must have SLOs (“Service Level Objectives”: latency, errors, etc) For effective service management, only three things matter: 0. Releasing service functionality 1. Gradually improving SLOs 2. Rapidly restoring SLOs In a deep system, we must control the entire “triangle” to maintain our SLOs
- 46. Controllability == ObservabilityControllability == Observability There’s that word again…
- 47. Observability: “The Conventional Wisdom” Observing microservices is hard Google and Facebook solved this (right???) They used Metrics, Logging, and Distributed Tracing… … So we should, too.
- 48. 3 Pillars, 3 Experiences Metrics Logs Traces
- 50. Three Pillars?Three Pillars? Two giant pipes… Logs Metrics Without Traces: Cognitive Load ≈ O(depth2 )
- 51. Three Pillars?Three Pillars? Two giant pipes… Logs Metrics
- 53. Two giant pipes… Logs Metrics Without Traces: Cognitive Load ≈ O(depth2 )
- 55. Traces
- 57. Traces provide Context And context rules out invalid hypotheses
- 58. Two giant pipes and a filter Logs Metrics Context (from traces)
- 59. Context (from traces) Context reduces cognitive load With Traces: Cognitive Load ≈ O(depth) Relevant Metrics Relevant Logs
- 62. Let’s Review
- 63. Microservices don’t just scale wide, they scale deep Recognize deep systems
- 64. Stress (n): responsibility without controlStress what you can control what you are responsible for
- 65. “Controllability” (of SLOs) depends on observability
- 66. … and traces are not sprinkles “The Three Pillars of Observability” is a lousy metaphor
- 67. Tracing can reduce cognitive load from O(depth2 ) to O(depth)
- 68. Tracing is the backbone of simple observability in deep systems
- 69. Thank You Feedback always welcome: twitter → @el_bhs the emails → [email protected] Play with LightStep, for free, anytime: (no email address required!) lightstep.com/play
- 70. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ properties-deep-systems/