Distributed Logging Architecture in Container Era
9 likes6,912 views
Distributed Logging Architecture in Container Era The document discusses distributed logging architecture in the container era. It covers: 1) The difficulties of logging with microservices and containers due to their ephemeral and distributed nature, 2) The need to redesign logging to push logs from containers to destinations quickly without fixed addresses or mappings; 3) Common patterns for distributed logging architectures including source aggregation, destination aggregation, and scaling; and 4) A case study using Docker and Fluentd to implement source aggregation and scaling for logging. Open source solutions are important to keep the logging layer transparent, interoperable, and able to scale independently of applications and infrastructure.
Distributed Logging Architecture in Container Era
- 1. Distributed Logging Architecture in Container Era LinuxCon Japan 2016 at Jun 13 2016 Satoshi "Moris" Tagomori (@tagomoris)
- 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, MessagePack-Ruby, Norikra, ... Treasure Data, Inc.
- 5. Topics • Microservices and logging in various industries • Difficulties of logging with containers • Distributed logging architecture • Patterns of distributed logging architecture • Case Study: Docker and Fluentd
- 6. Logging
- 7. Logging in Various Industries • Web access logs • Views/visitors on media • Views/clicks on Ads • Commercial transactions (EC, Game, ...) • Data from devices • Operation logs on Apps of phones • Various sensor data
- 8. Microservices and Logging • Monolithic service • a service produces all data about an user's behavior • Microservices • many services produce data about an user's access • it's needed to collect logs from many services to know what is happening Users Service (Application) Logs Users Logs
- 10. Containers: "a must" for microservices • Dividing a service into services • a service requires less computing resources (VM -> containers) • Making services independent from each other • but it is very difficult :( • some dependency must be solved even in development environment (containers on desktop)
- 11. Redesign Logging: Why? • No permanent storages • No fixed physical/network address • No fixed mapping between servers and roles • We should parse/label logs at the source, ship these logs by pushing to destination ASAP
- 12. Containers: immutable & disposable • No permanent storages • Where to write logs? • files in the container → gone w/ container instance 😞 • directories shared from hosts → hosts are shared by many containers/services ☹ • TODO: ship logs from container to anywhere ASAP
- 13. Containers: unfixed addresses • No fixed physical / network address • Where should we go to fetch logs? • Service discovery (e.g., consul) → one more component 😞 • rsync? ssh+tail? or ..? Is it installed in containers? → one more tool to depend on ☹ • TODO: push logs to anywhere from containers
- 14. Containers: instances per roles • No fixed mapping between servers and roles • How can we parse / store these logs? • Central repository about log syntax → very hard to maintain 😞 • Label logs by source address → many containers/roles in a host ☹ • TODO: label & parse logs at source of logs
- 16. Core Architecture • Collector nodes • Aggregator nodes • Destinations Collector nodes (Docker containers + agent) Destinations (Storage, Database, ...) Aggregator nodes
- 17. • Parse/Label (collector) • Raw logs are not good for processing • Convert logs to structured data (key-value pairs) • Split/Sort (aggregator) • Mixed logs are not good for searching • Split whole data stream into streams per services • Store (destination) • Format logs(records) as destination expects Collecting and Storing Data
- 18. Scaling Logging • Network traffic • CPU load to parse / format • Parse logs on each collector (distributed) • Format logs on aggregator (to be distributed) • Capability • Make aggregators redundant • Controlling delay • to make sure when we can know what's happening in our systems
- 19. Patterns
- 20. source aggregation NO source aggregation YES destination aggregation NO destination aggregation YES Aggregation Patterns
- 21. Source Side Aggregation Patterns w/o source aggregation w/ source aggregation collector aggregator / destination aggregate container
- 22. Without Source Aggregation • Pros: • Simple configuration • Cons: • fixed aggregator (endpoint) address • many network connections • high load in aggregator collector aggregator
- 23. With Source Aggregation • Pros: • less connections • lower load in aggregator • less configuration in containers (by specifying localhost) • highly flexible configuration (by deployment only of aggregate containers) • Cons: • a bit much resource (+1 container per host) aggregate container aggregator
- 24. Destination Side Aggregation Patterns w/o destination aggregation w/ destination aggregation aggregator collector destination
- 25. Without Destination Aggregation • Pros: • Less nodes • Simpler configuration • Cons: • Storage side change affects collector side • Worse performance: many small write requests on storage
- 26. With Destination Aggregation • Pros: • Collector side configuration is free from storage side changes • Better performance with fine tune on destination side aggregator • Cons: • More nodes • A bit complex configuration aggregator
- 27. Scaling Patterns Scaling Up Endpoints HTTP/TCP load balancer Huge queue + workers Scaling Out Endpoints Round-robin clients Load balancer Backend nodes Collector nodes Aggregator nodes
- 28. Scaling Up Endpoints • Pros: • Simple configuration in collector nodes • Cons: • Limits about scaling up Load balancer Backend nodes
- 29. Scaling Out Endpoints • Pros: • Unlimited scaling by adding aggregator nodes • Cons: • Complex configuration • Client features for round-robin
- 30. Without Destination Aggregation With Destination Aggregation Scaling Up Endpoints Systems in early stages Collecting logs over Internet or Using queues Scaling Out Endpoints Impossible :( Collector nodes must know all endpoints ↓ Uncontrollable Collecting logs in datacenter
- 31. Case Studies
- 32. Case Study: Docker+Fluentd • Destination aggregation + scaling up • Fluent logger + Fluentd • Source aggregation + scaling up • Docker json logger + Fluentd + Elasticsearch • Docker fluentd logger + Fluentd + Kafka • Source/Destination aggregation + scaling out • Docker fluentd logger + Fluentd
- 33. Why Fluentd? • Docker Fluentd logging driver • Docker containers can send logs to Fluentd directly - less overhead • Pluggable architecture • Various destination systems • Small memory footprint • Source aggregation requires +1 container per host • Less additional resource usage ( < 100MB )
- 34. Destination aggregation + scaling up • Sending logs directly over TCP by Fluentd logger library in application code • Same with patterns of New Relic • Easy to implement - good for startups Application code
- 35. Source aggregation + scaling up • Kubernetes: Json logger + Fluentd + Elasticsearch • Applications write logs to STDOUT • Docker writes logs as JSON in files • Fluentd reads logs from file parse JSON objects writes logs to Elasticsearch • EFK stack (like ELK stack) http://kubernetes.io/docs/getting-started-guides/logging-elasticsearch/ Elasticsearch Application code Files (JSON)
- 36. Source aggregation + scaling up/out • Docker fluentd logging driver + Fluentd + Kafka • Applications write logs to STDOUT • Docker sends logs to localhost Fluentd • Fluentd gets logs over TCP pushes logs into Kafka • Highly scalable & less overhead - very good for huge deployment Kafka Application code
- 37. Application code Source/Destination aggregation + scaling out • Docker fluentd logging driver + Fluentd • Applications write logs to STDOUT • Docker sends logs to localhost Fluentd • Fluentd gets logs over TCP sends logs into Aggregator Fluentd w/ round-robin load balance • Highly flexible - good for complex data processing requirements Any other storages
- 38. What's the Best? • Writing logs from containers: Some way to do it • Docker logging driver • Write logs on files + read/parse it • Send logs from apps directly • Make the platform scalable! • Source aggregation: Fluentd on localhost • Scalable storage: (Kafka, external services, ...) • No destination aggregation + Scaling up • Non-scalable storage: (Filesystems, RDBMSs, ...) • Destination aggregation + Scaling out
- 39. Why OSS Are Important For Logging?
- 40. Why OSS? • Logging layer is interface • transparency • interoperability • Keep the platform scalable • number of nodes • number of types of source/destination
- 41. Use OSS, Make Logging Scalable Thank you!