Update on OpenTSDB and AsyncHBase
2 likes822 views
This document summarizes an update on OpenTSDB, an open source time series database. It discusses OpenTSDB's ability to store trillions of data points at scale using HBase, Cassandra, or Bigtable as backends. Use cases mentioned include systems monitoring, sensor data, and financial data. The document outlines writing and querying functionality and describes the data model and table schema. It also discusses new features in OpenTSDB 2.2 and 2.3 like downsampling, expressions, and data stores. Community projects using OpenTSDB are highlighted and the future of OpenTSDB is discussed.
![Fuzzy Row Filter
Use a byte mask!
● Use the bloom filter to skip-scan
to the next candidate row.
● Combine with regex (after fuzzy
filter) to filter further.
FuzzyFilter{[FuzzyFilterPair{row_key=[18, 68,
-3, -82, 120, 87, 56, -15, 96, 0, 0, 0, 1, 0,
0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0],
mask=[0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0,
1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1]}]}
Now it takes 0.239 seconds
KEY
Match -> m t1 tagk tagv1
Skip -> m t1 tagk tagv2
m t1 tagk tagv3
m t1 tagk tagv4
m t1 tagk tagv5
m t1 tagk tagv6
Match -> m t2 tagk tagv1
Skip -> m t2 tagk tagv2](https://image.slidesharecdn.com/opssession3yahoo-160802230131/85/Update-on-OpenTSDB-and-AsyncHBase-19-320.jpg)
Update on OpenTSDB and AsyncHBase
- 1. OpenTSDB Update Distributed, Scalable Time Series Database Chris Larsen [email protected]
- 2. Who Am I? (no really, who am I?) Chris Larsen Current lead for OpenTSDB Software Engineer @ Yahoo! Monitoring Team
- 3. What Is OpenTSDB? Open Source Time Series Database Store trillions of data points Sucks up all data and keeps going Never lose precision Scales using HBase, Cassandra Or Bigtable
- 4. What good is it? Systems Monitoring & Measurement Servers Networks Sensor Data The Internet of Things SCADA Financial Data Scientific Experiment Results
- 5. Use Cases Backing store for Argus: Open source monitoring and alerting system 15 HBase Servers 6 month retention 10M writes per minute 95p query latency < 30 days = 200ms Moving to 200 node cluster writing at 100M/m
- 6. Use Cases ●Monitoring system, network and application performance and statistics 110 region servers, 10M writes/s ~ 2PB Multi-tenant and Kerberos secure HBase ~200k writes per second per TSD Central monitoring for all Yahoo properties Over 2 billion time series served
- 8. What Are Time Series? Time Series: data points for an identity over time Typical Identity: Dotted string: web01.sys.cpu.user.0 OpenTSDB Identity: Metric: sys.cpu.user Tags (name/value pairs): host=web01 cpu=0
- 9. What Are Time Series? Data Point: Metric + Tags + Value: 42 + Timestamp: 1234567890 sys.cpu.user 1234567890 42 host=web01 cpu=0 ^ a data point ^
- 10. How it Works
- 11. Writing Data 1) Open Telnet style socket, write: put sys.cpu.user 1234567890 42 host=web01 cpu=0 2) ..or, post JSON to: http://<host>:<port>/api/put 3) .. or import big files with CLI No schema definition No RRD file creation Just write!
- 12. Querying Data Graph with the GUI CLI tools HTTP API Aggregate multiple series Simple query language To average all CPUs on host: start=1h-ago avg sys.cpu.user host=web01
- 13. HBase Data Tables tsdb - Data point table. Massive tsdb-uid - Name to UID and UID to name mappings tsdb-meta - Time series index and meta-data tsdb-tree - Config and index for hierarchical naming schema
- 14. Data Table Schema Row key is a concatenation of UIDs and time: metric + timestamp + tagk1 + tagv1… + tagkN + tagvN sys.cpu.user 1234567890 42 host=web01 cpu=0 x00x00x01x49x95xFBx70x00x00x01x00x00x01x00x00x02x00x00x02 Timestamp normalized on 1 hour boundaries All data points for an hour are stored in one row Enables fast scans of all time series for a metric …or pass a row key filter for specific time series with particular tags
- 15. New for OpenTSDB 2.2 ● Append writes (no more need for TSD Compactions) ● Row salting and random metric IDs ● Downsampling Fill Policies ● Query filters (wildcard, regex, group by or not) ● Storage Exception plugin for retrying writes ● Released February 2016
- 16. New for OpenTSDB 2.3 ● Graphite style expressions ● Cross-metric expressions ● Calendar based downsampling ● New data stores ● UID assignment plugin interface ● Datapoint write filter plugin interface ● RC1 released May 2016
- 17. Fuzzy Row Filter How do you find a single time series out of 1 million? For a day? For a month?
- 18. Fuzzy Row Filter Instead of running a regex string comparator over each byte array formatted key… (?s)^.{9}(?:.{8})*Qx00x00x00x02 E(?:Q)x00x0F‡x42x2BE)(?:.{8})*$ TSDB query takes 1.6 seconds for 89,726 rows KEY Match -> m t1 tagk tagv1 No Match -> m t1 tagk tagv2 No Match -> m t1 tagk tagv3 No Match -> m t1 tagk tagv4 No Match -> m t1 tagk tagv5 No Match -> m t1 tagk tagv6 Match -> m t2 tagk tagv1 No Match -> m t2 tagk tagv2
- 19. Fuzzy Row Filter Use a byte mask! ● Use the bloom filter to skip-scan to the next candidate row. ● Combine with regex (after fuzzy filter) to filter further. FuzzyFilter{[FuzzyFilterPair{row_key=[18, 68, -3, -82, 120, 87, 56, -15, 96, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0], mask=[0, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1]}]} Now it takes 0.239 seconds KEY Match -> m t1 tagk tagv1 Skip -> m t1 tagk tagv2 m t1 tagk tagv3 m t1 tagk tagv4 m t1 tagk tagv5 m t1 tagk tagv6 Match -> m t2 tagk tagv1 Skip -> m t2 tagk tagv2
- 20. Fuzzy Row Filter Pros: ● Can improve scan latency by orders of magnitude ● Combines nicely with other filters Cons: ● All row keys for the match have to be a fixed length ● Doesn’t help much when matching the majority of a set ● Doesn’t support bitmasks, only byte masks
- 21. AsyncHBase AsyncHBase is a fully asynchronous, multi- threaded HBase client Supports HBase 0.90 to 1.x Faster and less resource intensive than the native HBase client Support for scanner filters, META prefetch, “fail-fast” RPCs
- 24. Upcoming in 1.8 ●Reverse Scanning ●New Yahoo! Cloud Serving Benchmark (YCSB) module for testing ●Lots of bug fixes
- 25. OpenTSDB on Bigtable ● Bigtable ○Hosted Google Service ○Client uses HTTP2 and GRPC for communication ● OpenTSDB heads home ○Based on a time series store on Bigtable at Google ○Identical schema as HBase ○Same filter support (fuzzy filters are coming)
- 26. OpenTSDB on Bigtable ● AsyncBigtable ○Implementation of AsyncHBase’s API for drop-in use ○https://github.com/OpenTSDB/asyncbigtable ○Uses HTable API ○Moving to native Bigtable API ● Thanks to Christos of Pythian, Solomon, Carter, Misha, and the rest of the Google Bigtable team ● https://www.pythian.com/blog/run-opentsdb-google- bigtable/#
- 27. OpenTSDB on Cassandra ● AsyncCassandra - Implementation of AsyncHBase’s API for drop-in use ● Wraps Netflix’s Astyanax for asynchronous calls ● Requires the ByteOrderedPartitioner and legacy API ● Same schema as HBase/Bigtable ● Scan filtering performed client side ● May not work with future Cassandra versions if they drop the API
- 28. Community Salesforce Argus ●Time series monitoring and alerting ●Multi-series annotations ●Dashboards Thanks to Tom Valine and the Salesforce engineers https://medium.com/salesforce-open-source/argus-time-series-monitoring-and- alerting-d2941f67864#.ez7mbo3ek https://github.com/SalesforceEng/Argus
- 29. Community Turn Splicer ●API to shard TSDB queries ●Locality advantage hosting TSDs on region servers ●Query caching Thanks to Jonathan Creasy and the Turn engineers https://github.com/turn/splicer
- 30. The Future of OpenTSDB
- 31. The Future Reworked query pipeline for selective ordering of operations Histogram support Flexible query caching framework Distributed queries Greater data store abstraction
- 32. More Information Thank you to everyone who has helped test, debug and add to OpenTSDB 2.3 including, but not limited to: TODO Contribute at github.com/OpenTSDB/opentsdb Website: opentsdb.net Documentation: opentsdb.net/docs/build/html Mailing List: groups.google.com/group/opentsdb Images http://photos.jdhancock.com/photo/2013-06-04-212438-the-lonely-vacuum-of-space.html http://en.wikipedia.org/wiki/File:Semi-automated-external-monitor-defibrillator.jpg http://upload.wikimedia.org/wikipedia/commons/1/17/Dining_table_for_two.jpg http://upload.wikimedia.org/wikipedia/commons/9/92/Easy_button.JPG https://www.flickr.com/photos/verbeeldingskr8/15563333617 http://www.flickr.com/photos/ladydragonflyherworld/4845314274/ http://lego.cuusoo.com/ideas/view/96