Presto Fast SQL on Anything
3 likes1,378 views
This document discusses Presto, an open source distributed SQL query engine. It is used by many large companies like Facebook, Uber, and Netflix for querying large datasets across various data sources. Presto provides high performance through its columnar processing, runtime compilation, and new cost-based optimizer. The document also describes how Presto can be run on AWS and Azure cloud platforms through partnerships with Starburst, who contributed many features to Presto and provides commercial support for enterprises.
![Beyond ANSI SQL
Presto offers a wide variety of built-in functions including:
● regular expression functions
● lambda expressions and functions
● geospatial functions
Complex data types:
● JSON
● ARRAY
● MAP
● ROW / STRUCT
SELECT regexp_extract_all('1a 2b 14m', 'd+'); -- [1, 2, 14]
SELECT filter(ARRAY [5, -6, NULL, 7], x -> x > 0); -- [5, 7]
SELECT transform(ARRAY [5, 6], x -> x + 1); -- [6, 7]
SELECT c.city_id, count(*) as trip_count
FROM trips_table as t
JOIN city_table as c
ON st_contains(c.geo_shape,
st_point(t.dest_lng, t.dest_lat))
WHERE t.trip_date = ‘2018-05-01’
GROUP BY 1;](https://image.slidesharecdn.com/prestofastsql2018alluxio-181206015340/85/Presto-Fast-SQL-on-Anything-8-320.jpg)
Presto Fast SQL on Anything
- 1. Kamil Bajda-Pawlikowski Co-founder and CTO www.starburstdata.com Fast SQL-on-Anything Alluxio meetup 2018 @ CA
- 2. Presto is SQL on anything Query anything, anywhere © 2018 2
- 4. Presto in production Facebook: 1000s of nodes, HDFS (ORC, RCFile), sharded MySQL, 1000s of users Uber: 800+ nodes (2 clusters on premises) with 200K+ queries daily over HDFS (Parquet/ORC) Twitter: 800+ nodes (several clusters on premises) for HDFS (Parquet) LinkedIn: 350+ nodes (2 clusters on premises), 40K+ queries daily over HDFS (ORC), 600+ users Netflix: 250+ nodes in AWS, 40+ PB in S3 (Parquet) Lyft: 200+ nodes in AWS, 20K+ queries daily, 20+ PBs in Parquet Yahoo! Japan: 200+ nodes (4 clusters on premises) for HDFS (ORC), ObjectStore, and Cassandra FINRA: 120+ nodes in AWS, 4PB in S3 (ORC), 200+ users
- 5. 5
- 6. Why Presto? © 2018 6
- 7. Why Presto? Community-driven open source project High performance ANSI SQL engine • New Cost-Based Query Optimizer • Proven scalability • High concurrency Separation of compute and storage • Scale storage and compute independently • No ETL or data integration necessary to get to insights • SQL-on-anything No vendor lock-in • No Hadoop distro vendor lock-in • No storage engine vendor lock-in • No cloud vendor lock-in 7
- 8. Beyond ANSI SQL Presto offers a wide variety of built-in functions including: ● regular expression functions ● lambda expressions and functions ● geospatial functions Complex data types: ● JSON ● ARRAY ● MAP ● ROW / STRUCT SELECT regexp_extract_all('1a 2b 14m', 'd+'); -- [1, 2, 14] SELECT filter(ARRAY [5, -6, NULL, 7], x -> x > 0); -- [5, 7] SELECT transform(ARRAY [5, 6], x -> x + 1); -- [6, 7] SELECT c.city_id, count(*) as trip_count FROM trips_table as t JOIN city_table as c ON st_contains(c.geo_shape, st_point(t.dest_lng, t.dest_lat)) WHERE t.trip_date = ‘2018-05-01’ GROUP BY 1;
- 9. JDBC / ODBC drivers for BI/SQL tools C/C++, Go, Java, Node.js, Python, PHP, R and Ruby on Rails UDFs, UDAFs, Connector SPI Tools, bindings, extensibility
- 10. https://www.starburstdata.com/presto-aws-cloud/ https://www.starburstdata.com/technical-blog/presto-available-on-aws-marketplace/ Presto on AWS Fully integrated with AWS: ● Amazon S3 ● AWS Glue Catalog ● Autoscaling ● AWS Marketplace
- 11. https://www.starburstdata.com/presto-azure/ https://azure.microsoft.com/en-us/blog/azure-hdinsight-and-starburst-brings-presto-to-micr osoft-azure-customers/ Presto on Azure Fully integrated with Azure HDInsight: ● Azure Blob Storage ● Azure Data Lake Storage ● External Hive Metastore ● Microsoft PowerBI
- 13. Built for Performance Query Execution Engine: • MPP-style pipelined in-memory execution • Columnar and vectorized data processing • Runtime query bytecode compilation • Memory efficient data structures • Multi-threaded multi-core execution • Optimized readers for columnar formats (ORC and Parquet) • Now also Cost-Based Optimizer 13© 2018
- 14. CBO in a nutshell Cost-Based Optimizer v1 includes: • support for statistics stored in Hive Metastore • join reordering based on selectivity estimates and cost • automatic join type selection (repartitioned vs broadcast) • automatic left/right side selection for joined tables https://www.starburstdata.com/technical-blog/ 14© 2018
- 15. Presto CBO Speedup Duration of TPC-DS queries (lower is better) © 2018 15 https://www.starburstdata.com/presto-benchmarks/
- 16. Cloud cost reduction ● on average 7x improvement vs EMR Presto ● EMR Presto cannot execute many TPC-DS queries ● All TPC-DS queries pass on Starburst Presto 16© 2018 https://www.starburstdata.com/presto-aws/
- 18. Why Starburst? © 2018 18
- 19. Starburst Data © 2018 19 Founded by Presto committers: ● Over 3 years of contributions to Presto ● Presto distro for on-prem and cloud env ● Supporting customers in production ● Enterprise subscription add-ons Notable features contributed: ● ANSI SQL syntax enhancements ● Execution engine improvements ● Security integrations ● Spill to disk ● Cost-Based Optimizer https://www.starburstdata.com/presto-enterprise/
- 20. Thank You! 20 Twitter: @starburstdata @prestodb Blog: www.starburstdata.com/technical-blog/ Newsletter: www.starburstdata.com/newsletter © 2018