Intro to Telegraf
0 likes1,105 views
The document provides an overview of Telegraf, a powerful, open-source tool for collecting metrics and events, optimized for use with InfluxDB. It discusses its capabilities, including a plugin ecosystem with over 200 plugins, and highlights its ease of use with features like low-code customization, robust scheduling, and high-speed ingestion. Additionally, it emphasizes community involvement and resources for getting started with Telegraf, as well as its potential for integration with various systems and data sources.
![[[processors.execd]]
command = ["python", "./processors/forecasting.py"]
Sample ExecD Config: Input, Output, Processor](https://image.slidesharecdn.com/intrototelegraf-20220509-220511182617-696b97ef/85/Intro-to-Telegraf-38-320.jpg)
Intro to Telegraf
- 1. I N F L U X D B U N I V E R S I T Y Intro to Telegraf Getting Started Training Series Samantha Wang Product Manager, InfluxData
- 2. Brought to you by InfluxDB University InfluxDB University offers free live and self-paced training on: • InfluxDB • Telegraf • Flux • Kapacitor • and more influxdbu.com
- 3. Agenda • What is Telegraf • Plugin Ecosystem • Getting Started with Telegraf • Extending the Ecosystem
- 5. Characteristics of the data • Time-stamped • Generated in regular (metric) and irregular (event) time periods • Huge volumes • Real time and time sensitive
- 6. InfluxDB is 3 things API & Toolset POWERFUL for real-time apps HIGH PERFORMANCE MASSIVE for real-time data workloads of cloud & open source developers 1 2 3 Time Series Engine Community & Ecosystem
- 7. Reference Architecture Data Sources Application Workflows Infrastructure Insights Telegraf Client Libraries HTTP Syslog Kubernetes Apache Kafka Python Arduino Node.js JavaScript Go Data Systems Mobile apps Web apps Cloud Services Devices Sensors Databases Networks Message Queues APIs IoT Platforms CRMs InfluxDB Platform IoT Actions InfluxDB Purpose-Built Time Series Database Visualization, Query & Task Engine Collect Downsample Trigger Alert Transform … 200+ Plugins 20+ Languages … New Square Native Ecosystems JMeter NiFi AWS Kinesis Azure Event Hubs GCP PubSub Java .NET/C# PHP Ruby Vector Fluentd
- 9. Current Data Acquisition - All Methods Agent-based Push (aka Telegraf) Agentless Pull (aka Scrapers) Client Libraries Native/Ecosystem • 200+ Telegraf plugins • Regular cadence of releases • Why use it? ○ No code ○ Large community ○ Lightweight but powerful ○ Customizable • Prometheus scraper (OSS only) • Flux prometheus.from • Flux csv.from(url) • Why use them? ○ Get data in quickly ○ Doesn’t require agent downloads on monitoring device • 7 Libraries: Python, C#, Java, GO, Javascript/Node.js, Ruby, PHP • Handles batching, chunking, setting right headers, etc. • Why use them? ○ Easy way to get started ○ Need libraries when building custom applications • Source system speaks line protocol • Examples: JMeter, NiFi, Vector, Fluentd • Influx CLI CSV Import • Why use them? ○ Know what you want to monitor, quick and easy integration
- 10. Telegraf provides the benefits of… • Low/No code • Robust scheduler • High-speed ingestion • Full-streaming support • Metric routing • Flexible parsing, formatting, serializing • Customizable and Extensible (ExecD plugins, Starlark) Instead of… • Writing long data scraping scripts • Worrying about unreliable data collection • Trouble scaling your data collection • Resulting in messy data • Having a lot of unnecessary data in your database
- 11. Telegraf: Agent for Collecting Metrics & Events Plugin-driven server agent for collecting and reporting metrics • Written in Go • Single Binary, No external dependencies • Minimal memory footprint • Optimized for writing to InfluxDB • Optimized for streaming data Telegraf HTTP Syslog Kubernetes Apache Kafka InfluxDB Purpose-Built Time Series Database Visualization, Query & Task Engine Collect Downsample Trigger Alert Transform 200+ Plugins AWS Kinesis Azure Event Hubs GCP PubSub
- 12. Telegraf: Agent for Collecting Metrics & Events • Input plugins to source a variety of metrics • directly from the system it’s running on • pull metrics from third party APIs • listen for metrics via a statsd and Kafka consumer services • Output plugins to send metrics to a variety of other datastores, services, and message queues • InfluxDB, Graphite, OpenTSDB, Datadog, and many others • Writes to multiple outputs at the same time • Metrics routing and conversion • Aggregation stage for InfluxDB
- 13. Core Telegraf functionality • Robust scheduler • Adjustments for clock-drift • Adjustments for job scheduling issues that may occur • In-memory metric buffers • Metric tracking with flow back-pressure in plugins like Kafka • Full-streaming support • Metric routing: name & field pass & drop • Flexible parsing, formatting, serializing
- 15. Telegraf Plugin Ecosystem Input Plugins • collect metrics from system, services, or third-party APIs Output Plugins • write data to various destinations Processors • transform, decorate, and/or filter metrics Aggregators • create aggregate metrics (e.g. mean, min, max, quantiles, etc.) Supports:
- 16. Telegraf Plugin Types Input Output Processors Aggregators 200+ 45+ 25+ 5+ Covering: Applications / Build and Deploy / Logging / Messaging / Networking / IoT/ Systems
- 23. One Telegraf, Multiple Plugins InfluxDB File Kafka CloudWatch CPU Mem Disk Docker Kubernetes /metrics Kafka MySQL CloudWatch InfluxDB Purpose-Built Time Series Database Visualization, Query & Task Engine Collect Downsample Trigger Alert Transform Input Process Aggregate Output - mean - min,max - count - variance - stddev - transform - decorate - filter Data Systems Data Sources
- 24. Can’t find the plugin you need? Telegraf is 100% open source with a strong community of contributors It’s easy to write your own Telegraf plugin! 1. Follow the contribution guide for Go 2. Write your plugin in any language and run it externally with ExecD
- 25. Popular Plugins Out-of-the-box Custom Kubernetes (kubelet) HTTP/socket listener Kube_inventory (apiserver) HTTP (formatted endpoints) Kafka (consumer) Prometheus (/metrics) SNMP Exec/Execd AMQP (mq metadata) StatsD Redis Nginx HAproxy Jolokia2
- 27. Getting Started with Telegraf
- 30. Telegraf Configuration Snippets Include many plugins per Telegraf • multiple inputs, outputs, processors, aggregators
- 31. What’s in a configuration file? • The Telegraf config file needs to be specified for Telegraf agent to operate properly. • It contains setup for the agent, global tags, and enabled outputs (through commenting out or removing unnecessary lines)
- 32. Set and Test Your Configuration $ telegraf --input-filter cpu:system --output-filter influxdb --processor-filter starlark --aggregator-filter valuecounter config > telegraf.conf WRITE DESIRED CONFIG FILE RUN CONFIG FILE TEST $ telegraf --config telegraf.conf --test VERIFY YOUR CONFIG WORKS
- 33. Telegraf Demo
- 35. You can extend Telegraf by: • Work with the open-source community or submit upgrades or enhancements for existing plugins • Use ExecD to write a plugin in Go or a language of your choice • Starlark processor: calls a Starlark function for each matched metric, allowing for custom programmatic metric processing • Math operations • String operations • Renaming tags • Logic operations
- 36. External plugins via ExecD • Plugin runs in its own process • Requires line protocol • Avoid the need for review by Telegraf team • Supports the same API as an internal plugin • Can use for non-GO plugins • Can use for licensed software plugins • Can use for any type of plugin (input, output, processor, aggregator)
- 38. [[processors.execd]] command = ["python", "./processors/forecasting.py"] Sample ExecD Config: Input, Output, Processor
- 40. Customer Quotes “Telegraf is like a swiss army knife for connecting various MQTT sources and OPC UA sources.” — Fr. Ant. Niedermayr “Our next-generation pipeline takes advantage of Kafka and the Telegraf streaming service to create a more robust data topology. Essentially this allows us to explicitly implement the four R’s: routability, retention, resilience, and redundancy.” — Wayfair
- 41. Get involved with Telegraf Telegraf GitHub: github.com/influxdata/telegraf Community Slack: influxdata.com/slack • #telegraf • #telegraf-dev Community Website: community.influxdata.com
- 42. Changes to Telegraf releases • Discontinuing release candidates • Telegraf users will be directed to test the following: • nightly builds • pull request artifacts • If you wish to be actively tagged on PRs specific to technology (SNMP vs Windows vs logging) please fill out this form
- 43. Telegraf Office Hours / Clinic A place for users and contributors to bring their problems, pull requests, or even show-and-tell how they’re using Telegraf. • Debugging problems / diagnose issues with deploying / configuring / using Telegraf • Reviewing pull requests • Brainstorm ideas for improving Telegraf
- 44. References • Getting started with Telegraf • Telegraf plugins • Telegraf GitHub Page • External Plugins Guide
- 45. www.influxdbu.com