Improving monitoring systems Interoperability with OpenMetrics
3 likes462 views
The document discusses monitoring system interoperability using OpenMetrics and Prometheus, detailing their architectures and metric formats. It highlights Prometheus's evolution and its standardization efforts through OpenMetrics, aiming for vendor neutrality and improved integration. Key features include metric types, structured metrics integration, and potential improvements in the monitoring landscape.
![Metric Name
[a-zA-Z_:][a-zA-Z0-9_:]*.
▶Prefix: application, namespace or domain
• prometheus_notifications_total
• process_cpu_seconds_total
• http_request_duration_seconds
▶Suffix: single unit
• node_memory_usage_bytes
• http_request_duration_seconds
• process_cpu_seconds_total
• http_requests_total
• prometheus_build_info (provide metadata)](https://image.slidesharecdn.com/improvingmonitoringsystemsinteroperabilitywithopenmetrics-191210040108/85/Improving-monitoring-systems-Interoperability-with-OpenMetrics-10-320.jpg)
![Labels
▶Key-Value pairs associated with time series
▶Label name
• [a-zA-Z_][a-zA-Z0-9_]*.
▶http_requests_total
• http_requests_total{path="/login"}
• http_requests_total{path="/logout"}
• http_requests_total{path="/adduser"}
• http_requests_total{path="/comment"}
• http_requests_total{path="/view"}
▶Do not use labels to store dimensions with high cardinality
• IDs, email addresses, …](https://image.slidesharecdn.com/improvingmonitoringsystemsinteroperabilitywithopenmetrics-191210040108/85/Improving-monitoring-systems-Interoperability-with-OpenMetrics-11-320.jpg)
![Integrations: Prometheus
…
scrape_configs:
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
- job_name: 'openmetrics'
static_configs:
- targets: ['172.26.2.74:5050']](https://image.slidesharecdn.com/improvingmonitoringsystemsinteroperabilitywithopenmetrics-191210040108/85/Improving-monitoring-systems-Interoperability-with-OpenMetrics-24-320.jpg)
Improving monitoring systems Interoperability with OpenMetrics
- 2. Monitoring System Interoperability with OpenMetrics Chanshik Lim @ NexCloud
- 3. Agenda ▶Prometheus • Prometheus Architecture • Prometheus Metrics • Prometheus Exposition Format ▶OpenMetrics • Standardization • Improvements ▶Interoperability with OpenMetrics • Demo Architecture • Integrations ▶References
- 4. Who am I Chanshik Lim • Lead software engineer @NexCloud • Maintainer of NexClipper project https://github.com/NexClipper/NexClipper • Go, Python • Linux, Kubernetes, Prometheus • Hiring software engineer for #Kubernetes #Prometheus #Metrics #Monitoring
- 5. Prometheus
- 6. Prometheus https://prometheus.io/ • Started in 2012, Version 1.0 released in 2016 • Contributed to CNCF in May 2016 • Graduated in August 2018 • A multi-dimensional data model with time series data • Pulling model over HTTP • PromQL: Flexible query language • Alertmanager: handle alerts
- 9. Prometheus Exposition Format ▶Human-readable, Readable line by line ▶Metric Syntax ▶Supported metric primitives • Counter • Gauge • Histogram • Summary • Untyped
- 10. Metric Name [a-zA-Z_:][a-zA-Z0-9_:]*. ▶Prefix: application, namespace or domain • prometheus_notifications_total • process_cpu_seconds_total • http_request_duration_seconds ▶Suffix: single unit • node_memory_usage_bytes • http_request_duration_seconds • process_cpu_seconds_total • http_requests_total • prometheus_build_info (provide metadata)
- 11. Labels ▶Key-Value pairs associated with time series ▶Label name • [a-zA-Z_][a-zA-Z0-9_]*. ▶http_requests_total • http_requests_total{path="/login"} • http_requests_total{path="/logout"} • http_requests_total{path="/adduser"} • http_requests_total{path="/comment"} • http_requests_total{path="/view"} ▶Do not use labels to store dimensions with high cardinality • IDs, email addresses, …
- 12. Samples ▶Actual time series data ▶Float64 value ▶Millisecond-precision timestamp (optional) ▶Values • node_cpu_seconds_total{cpu="0",mode="idle"} 48649.88 • node_cpu_seconds_total{cpu="0",mode="iowait"} 169.99 • node_cpu_seconds_total{cpu="0",mode="irq"} 0 • node_cpu_seconds_total{cpu="0",mode="nice"} 57.5
- 13. OpenMetrics
- 14. Before Prometheus ▶SNMP (closest to metric standard) ▶Highly fragmented monitoring landscape ▶Proprietary formats (with missing documents) ▶Difficult to implement ▶Various monitoring system specific formats
- 15. After Prometheus ▶Prometheus exposition format became a de-facto standard ▶Re-use Prometheus exporters to integration ▶Hundreds of exporters ▶Main monitoring players support Prometheus exposition format
- 16. Standardization https://openmetrics.io/ ▶Prometheus exposition format is still the Prometheus format ▶Traditional vendors wants to avoid lock-in ▶Prefer to use official standards ▶Vendor neutral ▶CNCF sandbox project ▶Official standard (in progress) • IETF RFC • Registered content-type/MIME-type : application/openmetrics-text • (maybe) IANA port assignment
- 17. Improvements # UNIT : new directive to represent the unit of a metric # EOF : to detect interrupted metrics ▶Timestamps are in seconds ▶Counter requires _total on the time series ▶Considerations for pull and push ▶Optional support for protobufs ▶Info / Enum : First class features
- 18. Prometheus & OpenMetrics Prometheus OpenMetrics # TYPE foo_seconds_total counter foo_seconds_total 1.0 # TYPE foo_seconds counter # UNIT foo_seconds seconds foo_seconds_total 1.0 foo_seconds_created 1572688096.0 # EOF
- 19. Exemplars Attach an ID off a trace to directly link to certain metrics # TYPE foo histogram foo_bucket{le=“0.01”} 0 foo_bucket{le=“0.1”} 8 # {id=“abc”} 0.043 foo_bucket{le=“1”} 10 # {id=“def”} 0.29 foo_bucket{le=“10”} 17 # {id=“ghi”} 7.73 foo_bucket{le=“+Inf”} 18 foo_count 18 foo_sum 324789.3 foo_created 1520430000.123
- 21. Demo Architecture IoT Application Internal Network Cloud Platform Datadog Metric Explorer NexAgent IoT Application NexAgent
- 22. Scrape Device Metrics IoT Application NexAgent Filtering: iot_* metrics … process_open_fds 6.0 # HELP process_max_fds Maximum number of open file descriptors. # TYPE process_max_fds gauge process_max_fds 1024.0 # HELP iot_temperature_celsius Temperature of device # TYPE iot_temperature_celsius gauge iot_temperature_celsius{host="device-1"} 43.87955931016437 # HELP iot_voltage_volts Voltage of device # TYPE iot_voltage_volts gauge iot_voltage_volts{host="device-1"} 0.32506781422008024 # HELP iot_current_amperes Electric current of device # TYPE iot_current_amperes gauge iot_current_amperes{host="device-1"} 0.5505769171108466 # HELP iot_memory_free_bytes Free memory of device # TYPE iot_memory_free_bytes gauge iot_memory_free_bytes{host="device-1"} 1.499344896e+09 … # HELP iot_temperature_celsius Temperature of device # TYPE iot_temperature_celsius gauge iot_temperature_celsius{host="device-1"} 43.87955931016437 # HELP iot_voltage_volts Voltage of device # TYPE iot_voltage_volts gauge iot_voltage_volts{host="device-1"} 0.32506781422008024 # HELP iot_current_amperes Electric current of device # TYPE iot_current_amperes gauge iot_current_amperes{host="device-1"} 0.5505769171108466 # HELP iot_memory_free_bytes Free memory of device # TYPE iot_memory_free_bytes gauge iot_memory_free_bytes{host="device-1"} 1.499344896e+09
- 23. Metrics Integration with OpenMetrics NexAgent #1 NexAgent #2 # HELP iot_temperature_celsius IoT metric iot_temperature_celsius # TYPE iot_temperature_celsius gauge iot_temperature_celsius{host="device-1"} 12.571817488302889 # HELP iot_voltage_volts IoT metric iot_voltage_volts # TYPE iot_voltage_volts gauge iot_voltage_volts{host="device-1"} 7.845377222730105 # HELP iot_current_amperes IoT metric iot_current_amperes # TYPE iot_current_amperes gauge iot_current_amperes{host="device-1"} 7.629632512400462 … # HELP iot_temperature_celsius IoT metric iot_temperature_celsius # TYPE iot_temperature_celsius gauge iot_temperature_celsius{host="device-2"} 17.218040315106897 # HELP iot_voltage_volts IoT metric iot_voltage_volts # TYPE iot_voltage_volts gauge iot_voltage_volts{host="device-2"} 8.962618429932972 # HELP iot_current_amperes IoT metric iot_current_amperes # TYPE iot_current_amperes gauge iot_current_amperes{host="device-2"} 8.602160171006599 … Filtering: iot_* metrics
- 24. Integrations: Prometheus … scrape_configs: - job_name: 'prometheus' static_configs: - targets: ['localhost:9090'] - job_name: 'openmetrics' static_configs: - targets: ['172.26.2.74:5050']
- 26. Integrations: DataDog (2) init_config: instances: - prometheus_url: http://127.0.0.1:5050/metrics namespace: "openmetrics" metrics: - iot_*
- 27. Integrations: NexClipper Project Metric Pipeline Global View Exporter ManagerNexServer Multi-Custer Multi-Cloud …… Long term data Exporters NexAgent NexAgentNexAgentNexAgent Applications Exporters Applications Node cAdvisor …. Exporters Applications Cluster Node Workload Service Prometheus servers/ Clusters Legacy datacenter/ HostKubernetes Clusters IoT Clusters ‘Metrics Pipeline’ for interoperability and Enterprise Prometheus (https://github.com/NexClipper/NexClipper)
- 28. Questions ?
- 30. References OpenMetrics: What Does It Mean for You by Brian Brazil, PromCon 2019 https://promcon.io/2019-munich/talks/openmetrics-what-does-it-mean-for-you/ OpenMetrics: Is Prometheus unbound? https://medium.com/@leodido/openmetrics-20d40eb0b302 OpenMetrics - Transforming the Prometheus Exposition Format into a Global Standard by Richard Hartmann, PromCon 2018 https://promcon.io/2018-munich/talks/openmetrics-transforming-the-prometheus-exposition-format-into-a-global- standard/ The Prometheus Time Series Database by Björn Rabenstein, PromCon 2016 https://promcon.io/2016-berlin/talks/the-prometheus-time-series-database/