Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
4 likes3,662 views
The document discusses building real-time analytics and visualization systems using various technologies like Elasticsearch, StatsD, and Logstash to enhance supply chain monitoring. It emphasizes the importance of complex event processing and suggests alternatives for managing query logs to optimize performance. Additionally, it introduces a creative way to visualize MySQL query performance through musical notes based on the queries processed.
Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates
- 1. Build your own Real Time Analytics and Visualization, Enable Complex Event Processing, Event Patterns and Aggregates Ramesh / Vishnu Supply Chain - Platform Team
- 2. Tom admiring his handywork !
- 4. Database Application Server
- 5. Elastic Search Graylog2 Logstash Database Application Server
- 6. Elastic StatsD Search Graylog2 Logstash Database Application Server
- 7. Elastic StatsD Search graphite Graylog2 Logstash Database Application Server
- 8. Search Elastic StatsD Search graphite Graylog2 Logstash Database Application Server
- 9. Search CEP Elastic StatsD Search graphite Graylog2 Logstash Database Application Server
- 10. Complex Event Processing ● ElasticSearch as a Storage or Alternate DB ○ Faster on Lookup Queries than RDBMS ○ Can do simple predicate queries ○ Does not need multiple indexes (full text indexing) ○ Create fields out of interesting values ● Statsd layer is a sliding window counter ○ Within a sliding window we can do regex patterns ○ Aggregates ○ Deviations ○ This is a Key aspect of the SOA Monitoring System (Complex patterns which need action) Push the complex pattern back to ES or as a trigger for action
- 11. Use cases ● Every PO has a matching SO? ● Has a shelf in the warehouse just gone empty? ● Where is the current pile up happening? ● Is the SLA being breached?
- 21. Search CEP Elastic StatsD Search graphite Graylog2 Logstash Database Application Server
- 22. Are logs the only source of events? ● No - The database can be used as well. ● Events can be generated by capturing the Updates/Inserts/Deletes being made to the tables. ● These events can be published to an MQ to speed up replication (batch processing) or sent to the CEP engine.
- 23. Search CEP Elastic StatsD Search graphite Graylog2 Change Data Capture Logstash Database Application Server
- 24. Distribute Replication Search CEP General Events Query Log Elastic MQ StatsD Search graphite Graylog2 Change Data Capture Logstash log.cc Database Application Server
- 25. Elasticsearch
- 27. Time to Sing Mood of Mysql Note:image is from http://www.technocation.org
- 28. Mood of Mysql ● Music is the best way to express how one feels. ● Well, Mysql has a soul too, it has a mood :) ● Mysql can sing through each query(good/bad) it gets. ● Every query, Mysql gets, is intercepted in log.cc and sent acrross to an MQ Server. Subscribers to the queue ,on receiving a message play a musical note depending on the query they get.
- 29. Use case: Divide & Conquer General query log ● Alternative to enabling general query log, which grows very fast in size and disk space becomes a concern on the master database. ● The queries are sent out to a queue on an MQ Server and an army of subscribers who listen to the queue , log the query on receiving a message. ● The general query log can now be distributed (among the subscribers). ● More number of subscribers => smaller the log & easy to rotate.