Open Source Monitoring Tools
Download as PPTX, PDF80 likes29,483 views
This document summarizes and compares several open source monitoring tools: Nagios, Graphite, StatsD, Logstash, and Sensu. Nagios is introduced as a commonly used tool that some love and some find frustrating. Graphite is described as a tool for storing and graphing time-series data. StatsD aggregates counters and timers and sends them to backend services like Graphite. Logstash is an tool for managing logs and events that can input, filter, and output data. Sensu is a monitoring router that connects check scripts to handler scripts to alert or process monitoring data. Examples are given for each tool and what types of metrics to collect.
![Graphite
•
•
No need to pre-define metric end-points
Determine granularity of data upfront.
/opt/graphite/conf/storage-schemas.conf
[stats]
pattern = ^stats.*
retentions = 10:2160,60:10080,600:262974
[catchall]
priority = 0
pattern = ^.*
retentions = 30:86400,300:525600](https://image.slidesharecdn.com/monitoring-130314084442-phpapp02/85/Open-Source-Monitoring-Tools-20-320.jpg)
Open Source Monitoring Tools
- 1. The State of Open Source Monitoring Tools Michael Richardson (@m_richo) Energized Work
- 2. What tools are we currently using to monitor and troubleshoot our systems?
- 3. What tools are we currently using to monitor and troubleshoot our systems? • • • • Nagios ssh + grep <something_bad> /some/random/log/file.log tail –f /some/random/log/file.log Others?
- 4. Nagios
- 5. Nagios – The lovers
- 6. Nagios – The lovers
- 7. Nagios – The lovers
- 8. Nagios – The lovers
- 10. Nagios Love-meter Where are you on the Scale? 0 10
- 11. Nagios Love-meter Where are you on the Scale? 0 Nagios shits me to tears 10 Sign me up to Nagios World Conference 2013!!!!
- 12. Alternatives ?
- 13. Alternatives ? Yep, there’s lots
- 14. Alternatives ? Yep, there’s lots some are better and some are worse
- 15. Today let’s check out • Graphite • Statsd • Logstash • Sensu
- 16. Graphite
- 17. Graphite • • • • • Metric storage Complex graph creation http://graphite.wikidot.com Apache 2.0 license Send time-series data that you are interested in graphing
- 18. Graphite Components 1. Web 2. Whisper 3. Carbon
- 19. Graphite • Everything stored in graphite has a path with components delimited by dots. Eg servers.HOSTNAME.METRIC applications.APPNAME.METRIC servers.database01.memfree applications.trading.loginattempts
- 20. Graphite • • No need to pre-define metric end-points Determine granularity of data upfront. /opt/graphite/conf/storage-schemas.conf [stats] pattern = ^stats.* retentions = 10:2160,60:10080,600:262974 [catchall] priority = 0 pattern = ^.* retentions = 30:86400,300:525600
- 21. Graphite What should I graph/trend? 1. Application Profiling Data 2. Operational Profiling Data 3. Regression Testing (releases) Why should I Graph/trend? 1. Trends can tell you when something is about to break. 2. …instead of hearing from your customers that it’s broken 3. Data can tell you when something is already broken but you don’t yet know it (regression). Source: Jason Dixon (@obfuscurity)
- 22. Graphite Demo Image source - http://joemiller.me/2011/11/05/correlating-puppet-changes-to-events-in-your-infrastructure/
- 23. StatsD
- 24. StatsD • • • • Measure Anything, Measure Everything Created and released by Etsy Aggregate counters and timers http://github.com/etsy/statsd
- 25. StatsD • Written in node.js • ~400 lines of javascript • Listens to statistics (counters & timers), and sends aggregates to backend services (like graphite). • simple
- 26. StatsD Don’t like Javascript or Node.js??
- 27. StatsD Don’t like Javascript or Node.js?? Google “statsd alternatives”…..
- 28. StatsD Don’t like Javascript or Node.js?? Google “statsd alternatives”….. 20+ rewrites/clones for you including.. Ruby, python, scala, python+twisted, erlang, clojure, C, groovy
- 29. StatsD Concepts • Buckets (a name that translates to graphite end-point) • Values • Flush (default 10 seconds) Counter metrics successfullogins:1|c|@0.1 Timing metrics apitimer:320|ms
- 30. StatsD Counter examples • Successful customer login attempts • Failed customer login attempts • Register a new customer • Hit 3rd party API
- 31. StatsD Timer examples • How fast is our function blah() • How fast is a database query • How fast is our 3rd party API service • How fast is our internet access • How fast are our page response times.
- 32. StatsD demo
- 33. LogStash
- 34. LogStash • • • • • Tool for managing Events and logs http://logstash.net https://github.com/logstash/logstash Apache 2.0 license Created by Jordan Sissel (@jordansissel)
- 35. LogStash • Written in ruby. • Built with jruby and ships as a jar file.
- 36. LogStash LogStash agent is an Event pipeline with 3 parts. 1. Inputs 2. Filters 3. Outputs
- 37. LogStash 1. Inputs – generate events 1. Filters – modify them 1. Outputs – ship them somewhere
- 38. LogStash Inputs include : amqp, drupal_dblog, eventlog, exec, file, ganglia, gelf, gemfire, generator, heroku, irc, log4j, lumberjack, pipe, redis, relp, sqs, stdin, stomp, syslog, tcp, twitter, udp, xmpp, zenoss, zeromq
- 39. LogStash Filters include : alter, anonymize, checksum, csv, date, dns, environment, gelfify, geoip, grep, grok, grokdiscovery, json, kv, metrics, multiline, mutate, noop, split, syslog_pri, urldecode, xml, zeromq
- 40. LogStash Outputs include : amqp, boundary, circonus, cloudwatch, datadog, elasticsearch, elasticsearch_http, elasticsearch_river, email, exec, file, ganglia, gelf, gemfire, graphite, graphtastic, http, internal, irc, juggernaut, librato, loggly, lumberjack, metriccatcher, mongodb, nagios, nagios_nsca, null, opentsdb, pagerduty, pipe, redis, riak, riemann, sns, sqs, statsd, stdout, stomp, syslog, tcp, websocket, xmpp, zabbix, zeromq
- 43. LogStash Shipper alternatives? • Syslog (rsyslog, syslog-ng,) • Lumberjack https://github.com/jordansissel/lumberjack • Beaver https://github.com/josegonzalez/beaver • Woodchuck https://github.com/danryan/woodchuck
- 44. LogStash Kibana • Web interface for viewing logstash records stored in elastic search • http://kibana.org/ • http://github.com/rashidkpc/Kibana • Search for records • Stream records (near realtime) • Create RSS feeds based on search results • Score, trend data
- 45. LogStash Kibana – search data Image source - http://kibana.org/
- 46. LogStash Kibana – trend data Image source - http://kibana.org/
- 47. LogStash Demo (Syslog & Apache access logs)
- 48. LogStash TIP – Go buy the Logstash Book – http://logstashbook.com/ James Turnbull (@kartar) It’s a great introduction to how to use Logstash.
- 50. Sensu • • • • • https://github.com/sensu/sensu Creator – Sean Porter (@portertech) Ruby, RabbitMQ, Redis <1200 lines of code Omnibus installation packages
- 51. Sensu Components • Sensu-server • Sensu-client • Sensu-api • Sensu-dashboard
- 52. Sensu • Message oriented architecture (messages are JSON objects) • Described as a monitoring router • Connects “check” scripts on Sensu Clients to “handler” scripts on Sensu Servers
- 53. Sensu Checks can • Determine if a service like apache up and running? (check exit code) • Collect metrics like page views or database cache usage.
- 54. Sensu Output of checks are router to 1 or more handlers who determine what to do.
- 55. Sensu Output of checks are router to 1 or more handlers who determine what to do. • Send alerts via email, pagerduty, IRC, twitter, basecamp, xmpp, hipchat, campfire, etc, etc
- 56. Sensu Output of checks are router to 1 or more handlers who determine what to do. • Send alerts via email, pagerduty, IRC, twitter, basecamp, xmpp, hipchat, campfire, etc, etc • Feed metrics to backend services like graphite, librato, opentsdb, etc, etc
- 57. Sensu demo
- 58. Questions??
- 59. Thank you
Editor's Notes
- #5: Anyone want a quick rundown of how it works?Fault detection, notifictations, escalations, acknowledgements, adding new nodes, no ajax
- #17: Graphite is a highly scalable real-time graphing systemwritten in pythonapache 2.0 license
- #18: Graphite is a highly scalable real-time graphing systemwritten in pythonapache 2.0 license
- #19: Web – djangoWhisper – metrics database format (similar to RRDTool). Accepts out-of-order data and supports pipelining of data in a single operation.Carbon – storage engine (agent + cache + persister)
- #20: Web – djangoWhisper – database for storing time series dataCarbon – listening service for capturing data
- #21: Web – djangoWhisper – database for storing time series dataCarbon – listening service for capturing data
- #22: Why Graphing and trendingApplication profiling dataOperational profiling data
- #23: Why Graphing and trendingApplication profiling dataOperational profiling data
- #30: Counter example add 1 to the particular bucket. Count is sent at flush interval and reset to 0tells statsd that counter is sampled every 1/10th of the time.Timing exampleAPI service took 320ms to completeStatsd determines percentiles, average (mean), standard deviation, sum, lower and upper bounds for the flush intervalCan support storing histogram of values too (not default)
- #32: Mean, upper, lower, stddev, upper 90, lower 90, count
- #42: Embedded web server and embedded elastic searchLead in shipper alternatives
- #51: Designed with CM in mind
- #52: Designed with CM in mind
- #53: Designed with CM in mindDescribe how client registers with server.
- #54: Reuse nagios plugins