Log aggregation: using Elasticsearch, Fluentd/Fluentbit and Kibana (EFK)
- 1. Log aggregation using Elasticsearch, Fluentd/Fluentbit and Kibana (EFK)
- 2. Who is this guy? . © 2018 Think Stack Limited thinkstack.io 2
- 3. Who is this guy and why is he talking to me? • I've worked in the IT industry for over 18 years • I started with VB6 & ASP and then .NET development • I progressed into deployment/release management and ultimately DevOps in 2012 • I've never presented a talk before - eek © 2018 Think Stack Limited thinkstack.io 3
- 4. What I'm covering • Introduction to log aggregation and why you want it • Why I'm using Fluentd/Fluentbit as opposed to Logstash • What's so great this setup? • Demo: Collect and parse logs from a MySQL container © 2018 Think Stack Limited thinkstack.io 4
- 5. What's not being covered • Elasticsearch or Kibana and their features • Detailed technical information that can be read online • Any of the supporting tech being used e.g. • Feel free to reach out to me if you have questions © 2018 Think Stack Limited thinkstack.io 5
- 6. What is log aggregation? © 2018 Think Stack Limited thinkstack.io 6
- 7. © 2018 Think Stack Limited thinkstack.io 7
- 8. Why am I using Fluentd? • Well supported pluggable architecture • Easy to understand configuration • Lightweight • Out-of-the-box compatibility in Kubernetes via Fluentbit • Buffering capabilities © 2018 Think Stack Limited thinkstack.io 8
- 9. Available Fluentd Plugin Types . © 2018 Think Stack Limited thinkstack.io 9
- 10. Fluentd Event Structure • tag: From where an event originated; used for message routing • time: The epoch time at which an event occurred • record: The event log content as a JSON object © 2018 Think Stack Limited thinkstack.io 10
- 11. What's so great about this setup? (1/2) • I wanted something that used Fluentd • I wanted to easily ingest logs from a variety of sources: • via the Docker logging driver • via the fluentd gem installed in a Ruby environment on Centos • via the td-agent apt package installed on Ubuntu • via the td-agent-bit yum package installed on Centos © 2018 Think Stack Limited thinkstack.io 11
- 12. What's so great about this setup? (2/2) • I wanted to play with non-trivial configurations, for example: • TLS encryption • Parsing of multi-line Java logs • Using the ReadonlyREST plugin for security • The use of Fluentd configuration include directives © 2018 Think Stack Limited thinkstack.io 12
- 13. Fluentd Plugins Used • in_forward: capture logs securely on port 24224 and unsecurely on port 24223 • parser_multi_format: parse logs where the log stream has more than one format e.g Redis • filter_record_transformer: used to add a 'source' key value pair • out_elasticsearch: forward logs to Elasticsearch targetting different indices as appropriate • out_copy: copies logs to more than one output source e.g. Elasticsearch AND stdout • out_rewrite_tag_filter: used to rewrite the tags from k8s and re-emit logs to process © 2018 Think Stack Limited thinkstack.io 13
- 14. Local dev environment architecture . © 2018 Think Stack Limited thinkstack.io 14
- 15. Demo Time © 2018 Think Stack Limited thinkstack.io 15
- 16. Sorry, I was drinking , eating or What did I miss? That's ok, during this demo we... • launched a new MySQL container into our Minikube node • captured MySQL logs from Kubernetes Fluentbit daemonset • forwarded the logs to our Fluentd aggregator • augmented the logs with a source key-value pair • parsed the logs using the standard MySQL format • wrote the logs to a new index mysql-* in Elasticsearch • viewed the logs in Kibana's UI © 2018 Think Stack Limited thinkstack.io 16
- 17. Any Questions? Project is available publicly on GitHub github.com/deploymentking/efk Please feel free to contact me via... [email protected] twitter.com/thinkstackio linkedin.com/in/leemyring