Apache Superset - open source data exploration and visualization (Conclusion Code Café, march 2023)
Download as PPTX, PDF0 likes1,655 views
Apache Superset is an open-source data exploration and visualization platform that allows users to connect to any SQL data source and create visualizations and dashboards. Originally developed at Airbnb, it has evolved with a robust technology stack and is widely used in various companies. Its features include customizable visualizations, dashboard publishing, user-friendly interfaces, and fine-grained security controls.
Apache Superset - open source data exploration and visualization (Conclusion Code Café, march 2023)
- 1. Classificatie: vertrouwelijk Apache Superset Data Exploration, Visualization & Analysis co-star: Steampipe & Trino Conclusion Code Café – 20 maart 2023 Lucas Jellema, CTO & Architect AMIS | Conclusion SQL
- 2. Classificatie: vertrouwelijk Apache Superset • Data Visualization – ready to use product • browser based UI & web server backend • any SQL data source • quick table-to-visualization & dashboard • open source, end user friendly/self service • Design principles: • single or multi-user, • no data is stored in Superset (except meta-data) and ephemeral cache, • light weight & optional semantic layer • row based access control applied Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 2
- 3. Classificatie: vertrouwelijk Apache Superset • Typical workflow • connect data source and on boards “tables” • explore / filter/ aggregate / slide & dice data • create visualizations and annotate findings • compose and publish dashboards Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 3
- 4. Classificatie: vertrouwelijk History • Originated at Airbnb in 2015 as the frontend for Apache Druid • Druid: open source multi-dimensional in memory distributed real time timeseries database • Earlier names: Panoramix & Caravel • Under Apache Software Foundation since 2017 • July 2022 – Release 2.0 • Tech Debt resolved, better Databricks, Pinot & Trino support, much improved UI experience • Technology stack • JavaScript: React/Redux, D3.js, webpack • Python, Flask, Pandas, SQLAlchemy • Thriving open source project • Used in many companies. Example: Airbnb – 600+ daily users, 100K+ charts • Offered as SaaS: Preset Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 4
- 5. Classificatie: vertrouwelijk How to work with Apache Superset? • Install, Configure & Run => for example Kubernetes, Docker Compose, Gitpod • Configure Database Connection(s) • or upload CSV data files • Define Data Set(s) based on “Tables” • explore/refine in SQL Lab • define “Ninja Templates” – custom filters for specific SQL or context data Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 5
- 6. Classificatie: vertrouwelijk How to work with Apache Superset? • Install, Configure & Run => for example Kubernetes, Docker Compose, Gitpod • Configure Database Connection(s) • or upload CSV data files • Define Data Set(s) based on “Tables” • explore/refine in SQL Lab • define “Ninja Templates” – custom filters for specific SQL or context data • Create a Chart on that Data Set • select type of visualization • map data to visualization (x, y, series, time, ..) • configure chart: color-scheme, titles, legend • annotate chart – provide commentary • publish chart – image, CSV/Excel/JSON, email, add to dashboard • define alerts (on SQL condition) and schedule reports – Slack or Email • Compose and Expose dashboard Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 6
- 7. Classificatie: vertrouwelijk How to work with Apache Superset? • Install, Configure & Run Configure Database Connection(s) • Define Data Set(s) based on “Tables” • Create a Chart on that Data Set • Compose and Expose dashboard Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 7
- 8. Classificatie: vertrouwelijk Demo Apache Superset • Create Data Set from SQL Query • Explore Data • Create Visualization • Demonstrate Dashboard Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 8 Rob Laughter on Unsplash.com
- 9. Classificatie: vertrouwelijk Define Data Set for SQL Table or View Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 9
- 10. Classificatie: vertrouwelijk Explore Data Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 10
- 11. Classificatie: vertrouwelijk Visualize Data Set Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 11
- 12. Classificatie: vertrouwelijk End of Demo Apache Superset Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 12 Rob Laughter on Unsplash.com
- 13. Classificatie: vertrouwelijk Predictive Analytics • Predict values into the future • extrapolate from past • take seasonality into consideration • Based on Prophet • open source Python Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 13
- 14. Classificatie: vertrouwelijk Notifications - Alarms and Scheduled Reports • When? • condition • schedule • What? • To whom? • Channel/Method? Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 14
- 15. Classificatie: vertrouwelijk Annotation – Multiple Layers – Label Time Intervals and Timestamps Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 15
- 16. Classificatie: vertrouwelijk Security • Users identified through OAuth2 providers such as GitHub, Twitter, LinkedIn, Google, Azure, and custom OAuth2 providers • Users are associated with roles • Roles are authorized on data sources, views, dashboards • Row level access / Group level data filters • Define a security filter for a table and associate the filter with a specific group • Any data access on that table by someone in the group will have the filter applied “transparently” • Multiple filters will be combined Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 16 table Group Security Filter SQL
- 17. Classificatie: vertrouwelijk Custom Visualization Plugins • Adding custom data visualizations to Superset is well supported • Steps: • Generate Skeleton for custom plugin (CLI, Yeoman) • Register plugin (JSON) • Configure plugin – valid input, labels, hooks (JSON) • Implement/link React component that actually renders data (Typescript/JS) • At runtime: Superset • exposes custom plugin in gallery • allows users to set relevant configuration for plugin • passes data – query result set – to plugin • embeds the rendered outcome appropriately in the webpage Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 17
- 18. Classificatie: vertrouwelijk Data Source Reach of Apache Superset • Superset can process data in SQL enabled sources Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 18
- 19. Classificatie: vertrouwelijk Trino (pka Presto SQL) Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 19 • Distributed Federated Query Engine • OLAP system, enables data mesh • does not store data itself • MPP architecture • Trino processes SQL queries against multiple data engines • SQL and NoSQL • database and other (queue, event broker, file system, cache) • combines results across sources: join, union, group by / aggregate • Started in 2012 at Facebook as Presto • to replace Hive • Offered as SaaS by Galaxy
- 20. Classificatie: vertrouwelijk Superset can access data via Trino using SQL Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 20
- 21. Classificatie: vertrouwelijk Superset can access & combine non-SQL and SQL sources via Trino Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 21
- 22. Classificatie: vertrouwelijk Steampipe Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 22 102 plugins for various data sources data can be joined, filtered, union- ed/minussed, aggregated
- 23. Classificatie: vertrouwelijk Via Steampipe – Superset has access to 100 more sources Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 23
- 24. Classificatie: vertrouwelijk Extending Source Reach of Apache Superset – across platforms, data formats, protocols and query languages Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 24 Shillelagh Google Sheets HTTP => JSON, CSV GitHub GraphQL Datasette HTML Table S3 Weather API Socrata
- 25. Classificatie: vertrouwelijk workspace Superset web app at port 8088 docker-compose – running 6 containers database connection <> plugin Gitpod Workspace for Trying Out Apache Superset and Steampipe
- 26. Classificatie: vertrouwelijk Summary Data Visualization Across virtually any data source (also leveraging Trino, Steampipe etc) User friendly Appealing, insightful visualizations Data exploration (slice & dice) Customizable (custom visualizations) Open source, open architecture Fine grained security Free (and better?) alternative to Tableau, Qlik, PowerBI Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 26
- 27. Classificatie: vertrouwelijk Gitpod Workspace for Trying Out Apache Superset Conclusion Code Café - Apache Superset - Data Exploration, Visualization & Analysis 27 workspace Redis superset_app web app at port 8088 superset_worker superset_init superset_worker_beat superset_cache superset_db