Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise at Otto
2 likes3,705 views
The document discusses the implementation of a distributed logging architecture at OTTO, emphasizing the shift from a dedicated monolith to microservices and cloud readiness. Key aspects include the use of Elastic Cloud Enterprise for efficient provisioning, as well as challenges related to cross-account actions and autoscaling of logging infrastructure. Lessons learned highlight the importance of knowledge sharing, automation, and operational overhead reduction in managing the elastic stack.
Divide & Conquer - Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise at Otto
- 1. Christian Herweg Thomas Klinger Lukas Kucharski DIVIDE & CONQUER Logging Architecture in Distributed Ecosystems with Elastic Cloud Enterprise
- 2. Elastic{ON} Tour: Frankfurt 2018 Facts & Figures 07.11.18 2 • Location: Hamburg (head office) • Employees: 4,700 (all locations) • Sales 2016/2017: 2.9 billion euros OTTO campus in Hamburg
- 3. Elastic{ON} Tour: Frankfurt 2018 OTTO – Number 1 in Fashion & Lifestyle* 07.11.18 3 *B2C mail order, GfK figures 2014 6.800 brands (in-house and many premium third-party brands) Over 2,8 million items online Huge product portfolio from fashion and lifestyle to household appliances and multimedia, DIY, kitchens, furniture and toys 120 specialist catalogues The only ‘big book’ company to make the jump into the digital world 6 specialist online shops
- 4. Elastic{ON} Tour: Frankfurt 2018 Business Domains Mirror the System Architecture Page Assembly Tesla ShopOffice AfterSales Search P13N Order User Authentication Tracking u.v.m. Code ownership results in high code quality ! Continuous Delivery permits more than 800 deployments per week. ! Verticals develop and test features fast and independently ! schematic presentation Business phases of the order process frame the distributed and parallel development. The technical mirror to this phase model allows highest possible flexibility in business concept development. ! 07.11.18 4
- 5. Elastic{ON} Tour: Frankfurt 2018 07.11.18 5 Step by Step Fragmentation of otto.de Dedicated Monolith Vertical Architecture Micro Services & „Cloud Readiness“ I n d e xI n d e x I n d e x I n d e x I n d e x I n d e x Serverless & Cloud 2011 df /var/log 2013 ~3TB data 2015 ~17TB data 2018 up to 42TB data & splunk >
- 6. Elastic{ON} Tour: Frankfurt 2018 DISTRIBUTED LOGGING @ AWS a.k.a. „How to enable tenants to log data“ 07.11.18 6
- 7. Elastic{ON} Tour: Frankfurt 2018 Requirements • Security • Encryption (at rest, in transit) • Authentication & Authorization • Isolation of resources („multi-tenancy“) • Accessing other vertical‘s logs => Rethink classic operations model, become a service provider 07.11.18 7
- 8. Elastic{ON} Tour: Frankfurt 2018 Core Principles • Multi tenancy • Shared responsibility • Security by design • Automation (Goal: Provision the logging platform during lunch break) 07.11.18 8
- 9. Elastic{ON} Tour: Frankfurt 2018 Elasticsearch Deployment at Scale • AWS Elasticsearch (Cost, missing encryption) • Elasticsearch on EC2 (Technical overhead) • Elastic Cloud Enterprise (fits requirements) 07.11.18 9
- 10. Elastic{ON} Tour: Frankfurt 2018 Challenges • AWS Cross Account actions • (Near-)Realtime processing of logs • Processing multiple data formats (JSON, Syslog ...) • Queueing input data for failure scenarios • Autoscaling Logstash • Automation of Elasticsearch cluster management • Keeping up with new features in the Elastic stack 07.11.18 10
- 11. Elastic{ON} Tour: Frankfurt 2018 ELASTIC CLOUD ENTERPRISE a.k.a. „How to provision tons of Elasticsearch and Kibana clusters“ 07.11.18 11
- 12. Elastic{ON} Tour: Frankfurt 2018 How Elastic Cloud Enterprise Helped Us • No need to build custom provisioning service for Elasticsearch and Kibana clusters • Provides security features via Elastic features (authentication, authorization, integration with LDAP) • Supports multiple Elastic stack versions • Easy to set up a basic installation • Customizable (stack packs, underlying EC2 instance) • Extensive API • Updates do not require downtime • Multi-tenancy 07.11.18 12
- 13. Elastic{ON} Tour: Frankfurt 2018 Architecture 07.11.18 13 Key facts: • 2 Loadbalancers • 3 ECE Proxies • 3 ECE Coordinators • 21 ECE Allocators (i3.4xlarge) • 20+ Clusters • Dedicated Monitoring Cluster
- 14. Elastic{ON} Tour: Frankfurt 2018 Challenges • Automated deployment • Stateful components in a cloud environment (allocators) • Troubleshooting ECE • No fine-grained authentication and authorization (LDAP, OAuth2) except readonly/root users => Custom tooling required 07.11.18 14
- 15. Elastic{ON} Tour: Frankfurt 2018 LOGINGEST a.k.a. „How to move logs from one account to another“ 07.11.18 15
- 16. Elastic{ON} Tour: Frankfurt 2018 Challenges • Cross account log ingestion • Pipeline performance & autoscaling • Finding a suitable input • S3, CloudWatch Logs (Logstash state management) • Kinesis, Kafka (too expensive) • S3 & SNS & SQS (logstash-input-s3-sns-sqs) • Pipeline management (in Kibana) 07.11.18 16
- 17. Elastic{ON} Tour: Frankfurt 2018 Architecture Logingest 07.11.18 17
- 18. Elastic{ON} Tour: Frankfurt 2018 Autoscaling logstash 07.11.18 18 • Autoscale between 30 and 110 Logstash containers in total • Scaling based on demand and cluster size • Manual intervention possible • consume unexpected peaks • stopping ingest in failure scenarios
- 19. Elastic{ON} Tour: Frankfurt 2018 HOUSEKEEPING a.k.a. „How to keep your clusters clean“ 07.11.18 19
- 20. Elastic{ON} Tour: Frankfurt 2018 Curator as a Service • Verticals provide configuration in Git • Synchronized to S3 • Master-worker architecture based on AWS Lambda, CloudWatch and SQS 07.11.18 20
- 21. Elastic{ON} Tour: Frankfurt 2018 Challenges • Verticals are responsible for curator configuration • Lack of knowledge (both Curator and Elasticsearch) • Monitoring Curator • Tracking origins of failed runs (misconfiguration, internal failure) • Transparency for verticals • AWS Lambda limits (max. 3 minutes runtime) 07.11.18 21
- 22. Elastic{ON} Tour: Frankfurt 2018 LESSONS LEARNED a.k.a. „Learning through failure“ 07.11.18 22
- 23. Elastic{ON} Tour: Frankfurt 2018 Learnings • Spread knowledge about Elasticsearch with teams • Create a sustainable knowledgebase • Automation is essential • Reduce operational overhead • Have time to develop and introduce new features • Know your I/O limits and requirements • Scaling Logstash is not a trivial task 07.11.18 23
- 24. Elastic{ON} Tour: Frankfurt 2018 BACKLOG a.k.a. „Still some work to do“ 07.11.18 24
- 25. Elastic{ON} Tour: Frankfurt 2018 Next steps • Develop custom Logstash pipeline management solution • Move Housekeeping workers from AWS Lambda to AWS Fargate • Evaluate Index Lifecyle Management via Elasticsearch • Upgrade to ECE 2.0 • Leverage potential of ECE & Elastic features • Tenants use and know about Machine Learning & APM • ECE 2.0 features • When available: cross-cluster search 07.11.18 25
- 26. Elastic{ON} Tour: Frankfurt 2018 WE ARE HIRING! https://www.otto.de/jobs 07.11.18 26
- 27. Elastic{ON} Tour: Frankfurt 2018 THANK YOU FOR YOUR ATTENTION 07.11.18 27
- 28. Elastic{ON} Tour: Frankfurt 2018 BACKUP 07.11.18 28
- 29. Elastic{ON} Tour: Frankfurt 2018 History – The Beginning... 07.11.18 29
- 30. K2 - Cloud Readiness 08.05.2018 30 Autonomous and team specific utilization of technology. Decoupling of interfaces and system components reduces architecture complexity and interdependences especially with service components. Perimeter protection is replaced by an integrated security concept, reducing vulnerability and allowing customised implementations. Cloud migration allows new sourcing and scaling models