Introduction to Prometheus and Cortex (WOUG)
0 likes625 views
The document provides an overview of monitoring systems using Prometheus and Cortex, emphasizing the importance of metrics for tracking system performance and alerting on issues. It explains Prometheus's functionality, including metric naming, querying with PromQL, and automated alerting capabilities. Cortex is introduced as a scalable, long-term storage solution for Prometheus metrics, highlighting its role in enhancing monitoring strategies.
![Filter:
http_requests_total{service="webserver"}
Aggregate:
sum(http_requests_total)
Differentiate:
rate(http_requests_total[1m])
PromQL - Prometheus Query Language
9https://prometheus.io/docs/](https://image.slidesharecdn.com/20190702wougintro-prometheus-cortex-promqlbryanb-190702215736/85/Introduction-to-Prometheus-and-Cortex-WOUG-9-320.jpg)
![We don’t want to watch dashboards all day in case something goes
wrong - Prometheus can do that for us!
- alert: LowRequests
expr: rate(http_requests_total{service="webserver"}[1m]) < 0.1
for: 5m
Prometheus will execute this rule regularly.
AlertManager will send a notification via email, webhook, etc.
Automated Alerting
11](https://image.slidesharecdn.com/20190702wougintro-prometheus-cortex-promqlbryanb-190702215736/85/Introduction-to-Prometheus-and-Cortex-WOUG-11-320.jpg)
Introduction to Prometheus and Cortex (WOUG)
- 1. Introduction to Prometheus and Cortex Bryan Boreham – @bboreham July 2019 1
- 2. ● See whether your system is operating as expected ○ Get alerted when it isn’t ● Drill down for troubleshooting and incident investigation ● View trends over time, to drive strategy ● Input into continuous improvement Why do we Monitor? 2
- 3. Any quantity that tells you something about how your system is operating. - How much memory it is using - How long the last operation took - How many requests served today What do we mean by “metrics” ? 3
- 8. Prometheus metrics have a name and some labels: http_requests_total{service="webserver",region="EU"} 136 ● name is http_requests_total ● two labels, service and region, with values "webserver" and "EU" ● value is 136 Since this webserver program started, it has received 136 HTTP requests. Understanding metrics 8
- 9. Filter: http_requests_total{service="webserver"} Aggregate: sum(http_requests_total) Differentiate: rate(http_requests_total[1m]) PromQL - Prometheus Query Language 9https://prometheus.io/docs/
- 10. Demo! 10
- 11. We don’t want to watch dashboards all day in case something goes wrong - Prometheus can do that for us! - alert: LowRequests expr: rate(http_requests_total{service="webserver"}[1m]) < 0.1 for: 5m Prometheus will execute this rule regularly. AlertManager will send a notification via email, webhook, etc. Automated Alerting 11
- 12. Cortex 12
- 13. Cortex is a time-series store built on Prometheus - Horizontally scalable - Highly Available - Long-term storage - Multi-tenant Cortex is a CNCF Sandbox project https://github.com/cortexproject/cortex What is Cortex ? 13
- 17. Cortex high-level architecture 17 Sample Distribute Compress Store Query
- 18. Request rate Error rate Duration RED is for Services What to monitor: The RED model 18
- 19. Utilisation Saturation Errors USE is for Resources USE method 19
- 20. Questions? 20
- 21. ● How Google Runs Production Systems ● SREs: ○ Have the skillset necessary to automate tasks ○ Do the same work as an operations team, but with automation instead of manual labor ● SRE team responsible for latency, performance, efficiency, change management, monitoring, emergency response, and capacity planning Site Reliability Engineering 21