Webinar Monitoring in era of cloud computing
5 likes1,116 views
Create-Net is a research center that offers cloud computing research, consulting, training, and webinars. This webinar discusses monitoring in the cloud computing era, beginning with introductions to Ceilometer and Monasca. Ceilometer is OpenStack's metering framework that collects data from OpenStack services through agents and notifications. It stores data in a database and provides an API. Monasca is a monitoring as a service platform that processes metrics and events at scale through microservices and stores data for querying and visualization. The webinar concludes with a discussion of trends in cloud monitoring.
![2.
Introduction to Ceilometer
“..quando un nuvol vada sovr’essa..”
[Inf. XXXI ]](https://image.slidesharecdn.com/webinarmonitoringineraofcloudcomputing-151202135519-lva1-app6891/85/Webinar-Monitoring-in-era-of-cloud-computing-8-320.jpg)
![3.
Introduction to Monasca
“intrammo a ritornar nel chiaro mondo..”
[Inf. XXXIV ]](https://image.slidesharecdn.com/webinarmonitoringineraofcloudcomputing-151202135519-lva1-app6891/85/Webinar-Monitoring-in-era-of-cloud-computing-19-320.jpg)
![4.
Conclusions
“E quindi uscimmo a riveder le stelle..”
[Inf. XXXIV ]](https://image.slidesharecdn.com/webinarmonitoringineraofcloudcomputing-151202135519-lva1-app6891/85/Webinar-Monitoring-in-era-of-cloud-computing-30-320.jpg)
Webinar Monitoring in era of cloud computing
- 1. Monitoring in the era of Cloud Computing Speakers Attilio Broglio - [email protected] Silvio Cretti - [email protected] Webinar on November 26, 2015 - 15:00 CET
- 2. Who are we and what is our offer ? ■ Create-Net is a research centre active in the ICT field and located in Trento (Italy) ■ Our Cloud Computing offer comprises: Research and Consultancy in the Cloud Computing field cloud architectures, cloud deployment, cloud management, distributed cloud, energy efficiency in the cloud Training on Cloud Computing/Openstack (authorized Mirantis training partner)
- 3. OpenStack free webinar list offered by Create-Net Webinar Url OpenStack fundamentals https://goo.gl/VRg5Nr Platform as a Service (PaaS) and OpenStack https://goo.gl/VRg5Nr OpenStack Networking https://goo.gl/Htcs2s Monitoring in the era of Cloud Computing
- 4. Agenda 1. Introduction to Monitoring 2. Introduction to Ceilometer ■ Architecture ■ Agents (Compute/Central) ■ Storage & API ■ Quick Demo 3. Introduction to Monasca ■ Architecture ■ Events/Messages ■ Storage & API ■ Quick Demo 4. Conclusions and Trends 5. QA session
- 5. 1. Introduction to Monitoring ...in the Cloud Computing era
- 6. Introduction to the Monitoring ■ Monitoring a cloud requires a unified view of physical and virtual resources and of cloud services focusing on the customer SLA ■ Two complementary topics: ➪monitoring a cloud computing infrastructure (ref. Ceilometer) ➪Monitoring as a Service - xaaS - not limited to a specific cloud and offering higher level functionalities above the simple data collection (ref. Monasca)
- 7. Tools of the trade ■How to monitor a (federated) cloud? ■What is on the market now? ■Some examples: ➪ OpenNMS ➪Nagios ➪Zabbix ➪IBM Tivoli - Netcool ➪Zenoss ➪Openstack Ceilometer ➪OpenStack Monasca Modular (micro services) Distributed and scalable Fault tolerant Open Source & Open API Support for Big Data analytics (trend)
- 8. 2. Introduction to Ceilometer “..quando un nuvol vada sovr’essa..” [Inf. XXXI ]
- 9. Introduction to Ceilometer (1/2) OpenStack provides infrastructure as a service (IaaS) to end clients, it’s necessary to be able to meter its performance and utilization for: ■ billing ■ benchmarking ■ scalability ■ statistics Ceilometer aims to become the infrastructure to collect measurements within OpenStack. Main goals: ■ efficient collection of metering (CPU, network costs, ..) ■ collecting data by monitoring notifications (from services or by polling the HW) ■ accessing and inserting the metering data through the REST API ■ expanding the framework to collect custom usage data (by plugins)
- 10. Introduction to Ceilometer (2/2) What is a meter? Three type of meters are defined in ceilometer: ■ Cumulative: Increasing over time (i.e. instance hours) ■ Gauge: Discrete items (i.e. floating IPs ...) and fluctuating values (such as disk I/O) ■ Delta: Changing over time (i.e. bandwidth) Each meter is collected from one or more samples, which are represented by counter objects: ■ counter_name: meter id (i.e. disk.ephemeral.size) ■ counter_type: counter types (cumulative, gauge, delta). ■ counter_volume: measured data (CPU, #cores)
- 11. Ceilometer-Architecture (1/2) ■ Agents:Rest API central Agent compute Agent collectors data Stores AMQP glance libvirt nova-compute RPC RPC hypervisor APIAPI ■ Rest API: access to metering data ➪ central: Retrieve info not linked to instances or compute node ➪ compute: polls metering data and instances statistics from the compute node AMQP: message broker (RabbitMQ) data stores: mongo (read/write) collectors: message queues for metrics
- 12. Ceilometer-Architecture (2/2) Ceilometer offers 5 core services divide in 3 main groups: ■ data agents ■ collection ■ alarming The 5 blocks “daemons” are: 1. polling agent: polls OpenStack services and build Meters. 2. notification agent: ➪ listens to notifications on message queue ➪ converts to Events and Samples ➪ apply pipeline actions 3. collector daemon: gathers and records event and metering data 4. api service: to query and view data recorded by collector service 5. alarming: alarming system based on rules Data Agent Collection AlarmingPolling Notification Collector API Alarming
- 13. Ceilometer-Agents (1/3) How can we collect meter/sample? Two methods available to collect data: ■ Bus listener agent: ➪ events on the notification bus ➪ transforms them into Ceilometer samples ➪ community guideline (Oslo library) ➪ compliant with Notification Agent ■ Polling agents, ➪ polls API or other tool to collect information (Δt) ➪ local or remote hypervisor API
- 14. Ceilometer-Agents (2/3) Notification Agent? ..it listens for data ■ consumes messages from services (Nova, Glance, Cinder, Neutron, ..) ■ loads one or more listener plugins that listen to any topics (by namespace) ■ redirects to the proper endpoint (pipeline) ■ Event matching → wildcards e.g. compute.instance.*. ■ Notifications can be converted into Events
- 15. Ceilometer-Agents (3/3) Polling Agent? .. it asks for data ■ compute node: Polling for compute resources is handled by a polling agent running on the compute node (where hypervisor is) ■ central node: Polling via service APIs for non-compute resources. ■ The polling agent daemon is configured to run one or more pollsters ✓ Δt for sample from pollsters (set on pipeline) ✓ passes the samples to the pipeline for processing
- 16. Ceilometer-Storage and API (1/2) Pipeline? ..manipulates data Ceilometer uses pipeline for: ■ manipulate and publish samples it in various combinations ■ combined by historical or temporal context ■ transform by a scaling factor ■ publishes using 3 different transports: 1. notifier: a notification based that pushes to a message queue: (i.e. collector, external system) 2. udp: publishes samples using UDP packets; 3. kafka: publishes to a Kafka message queue
- 17. Ceilometer-Storage and API (2/2) Collector? ■ gathers the processed event and metering data (notification and polling agents) ■ validates data ■ writes the messages to: database, file, or http From Juno, database is divided into 3 connections: ■ alarm ■ event ■ metering API? RESTFUL API (read/write data) ■ raw ■ statistics/filter/group
- 18. Ceilometer-Demo CC License - Photo by Amy Loves Yah
- 19. 3. Introduction to Monasca “intrammo a ritornar nel chiaro mondo..” [Inf. XXXIV ]
- 20. Monasca - Introduction (1/2) MONitoring As a Service sCAlable: ■ Highly-performant, scalable ■ Fault-tolerant, capable of big data retention ➪ process 100s of thousands metrics/sec ➪ data stored for ~ 1 year ■ Based on REST API (http not standard CollectID, NSCA ”Nagios”) ➪ storing ➪ querying (metrics/historical data) ■ Multi-tenant (by keystone) ➪ metrics submitted and authenticated by Keystone ➪ metrics stored with tenants reference
- 21. Monasca - Introduction (2/2) MONitoring As a Service sCAlable: ■ Extensible based on micro-services message bus architecture ■ Real-time events processing ➪ threshold/alarm ->Metrics ➪ threshold/alarm ->storage/retrieval/statistics ■ Notification system ■ Real data: ➪ Performances: ~ 25K metrics/sec on three HP Proliant SL390s G7 ➪ Apache 2.0 license ➪ Who is working on it: HP, IBM, Rackspace
- 22. Monasca - Architecture (1/4) Monasca API (REST) Horizon Dashboard Monasca Client (CLI)System Monasca Agent Config DB mySQL Metrics Events DB Alarms Notification Engine Threshold Engine Anomaly and Prediction Engine Events Engine Transform Engine Persister Message Queue Kafka
- 23. Monasca - Architecture (2/4) Monasca API (REST) Monasca Client (CLI) Notification Engine Threshold Engine Anomaly & Prediction Engine Events Engine Transform Engine Persister Message Queue Kafka Metrics Events DB Alarms System Monasca Agent Horizon Dashboard Monasca Agent Monasca Agent: ■ different subcomponents ■ different metrics ➪ statsd, nagios plugin, ceilometer Monasca API: ■ REST API post/get metrics and events (JSON) ■ query and get statistics and metrics, alarms, alarm definitions, notifications Monasca API (REST) Message Queue “Kafka”: ■ receives ➪ metrics from API ➪ alarms from Threshold engine ➪ Publishes consumes events ■ RabbitMQ-> Kafka Message Queue Kafka POST metrics Publishes metrics/events
- 24. Transform Engine Monasca - Architecture (3/4) Monasca API (REST) Monasca Client (CLI) Notification Engine Threshold Engine Anomaly & Prediction Engine Events Engine Transform Engine Persister Message Queue Kafka Metrics Events DB Alarms System Monasca Agent Horizon Dashboard Monasca Agent Notification Engine: ➪ consumes alarms from kafka and notifies (email, sms,.. ) Threshold engine: ➪ stream data processing (Apache Storm) ➪ computes threshold on metrics and publish on kafka Monasca API (REST) Anomaly & Prediction engine: ➪ evaluates predictions and anomalies and publish predicted metrics (or anomalies) Message Queue Kafka Events engine: ➪ manages events Transform engine: ➪ transforms metrics into delta or time-based derivative ➪ generates new metrics (Kafka) Notification Engine Threshold Engine Anomaly & Prediction Engine Events Engine
- 25. Monasca - Architecture (4/4) Monasca API (REST) Monasca Client (CLI) Notification Engine Threshold Engine Anomaly & Prediction Engine Events Engine Transform Engine Persister Message Queue Kafka Metrics Events DB Alarms System Monasca Agent Horizon Dashboard Monasca Agent Persister: ■ Consumes alarms and metrics from Kafka and stores into nonRelational DB ➪ InfluxDB ➪ Vertica Horizon Dashboard: ■ Actions on alarms: ➪ CRUD alarm definitions ➪ read/delete alarm history ■ Actions on notifications: ➪ CRUD notifications methods ■ Actions on metrics ➪ visualizations of metrics “Grafana” Monasca API (REST) Message Queue Kafka Notification Engine Threshold Engine Anomaly & Prediction Engine Events Engine Persister Horizon Dashboard Create alarms definitions Query/Delete alarms Create notifications method Query Metrics
- 26. Monasca - Metrics flow (1/4) Monasca Agent Metrics Events DB Alarms Notification Engine Threshold Engine Transform Engine Persister Message Queue Kafka POST metric 1.0 publish metric 1.1.0 consume metrics 1.1.1 store metric 1.2.0 consume metric 1.2.2 publish metric 1.2.1 aggregate metric 1.3.0 consume metric1.3.3 publish alarm 1.3.2 theshold metric 2.1.0 consume alarm 2.1.1 send notification 2.2.0 consume alarms 2.2.1 store alarm 1.0 API authenticates and validates the request and publishes the metric to Kafka 1.1 Persister consumes the metric from Kafka and stores in the Metrics Store 1.2 The Transform Engine consumes the metrics from kafka, performs transform and aggregation operations on metrics, and publishes metrics that it creates back to Message Queue 1.3 Threshold Engine consumes metrics from Kafka and evaluates alarms. If a state change occurs in an alarm, an alarm is published into kafka 2.1 The Notification Engine consumes "alarm" from kafka, evaluates whether they have a Notification Method associated with it, and notifies 2.2 Persister consumes the "alarm" from Kafka and stores it in the Alarm State History
- 27. Monasca - Events flow (1/4) Monasca Agent Metrics Events DB Alarms Event Engine Transform Engine Persister Message Queue Kafka POST event 1.0 publish event 1.1.0 consume raw event 1.1.1 store event 1.2.2 publish tranformed event 1.2.1 transform raw event into monasca event 1.3.0 consume transformed event 2.0.0 fire event 1.0 API authenticates and validates the request and publishes the event to Kafka 1.1 Persister consumes the event from Kafka and stores in the Events Store 1.2 The Transform Engine consumes the event from kafka, performs transform and aggregation operations on events, and publishes events that it creates back to Message Queue 1.3 Event Engine consumes compliant events from Kafka. It converts the event into stream a “YAML” 2.0 Streams are processed by the Stream Engine Winchester. If needed it publishes Metric Stack Distiller StackTach Winchester 1.3.0 add event 2.0.1 publish metric 1.2.0 consume raw event
- 28. Monasca - API Keystone ● Metrics ● Transformations ● Alarms ● Notifications ● Streams ● Events Service Catalog provided by keystone http://my_monasca http://my_stacktach /metrics/ /transform/ /alarm-definitions/ or /alarms/ /notification-methods/ /streams/ /events/ Monasca API StackTach API Your API * Anomaly API Monasca client StackTach client Your client* Anomaly client
- 29. Monasca-Demo "Back to the Future with Michael J. Fox as Marty McFly" by Source. Licensed under Fair use via Wikipedia - here
- 30. 4. Conclusions “E quindi uscimmo a riveder le stelle..” [Inf. XXXIV ]
- 31. OpenStack Bootcamp The main topics covered are: • Overview on OpenStack and its architecture, • OpenStack networking; • Swift; • Ceilometer and its architecture, • Heat Overview; • OpenStack deployment. At the end of the Bootcamp each student will be able to: • Describe the architecture of an OpenStack deployment; • Discuss the main functionalities of OpenStack; • Deploy, configure and use the Openstack services; • Create and manage VMs and Virtual Networks; • Create and manage, suers, roles, and quotas; • Use the OpenStack CLI and Dashboard. In partnership with Mirantis For further information: http://openstack.create-net.org; [email protected]
- 32. Attilio Broglio Software architect @ Create-net [email protected] Tanks for your attention!! Silvio Cretti Deputy Area Head @ Create-net [email protected] Webinar on November 26th, 2015 - 15:00 CET