VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger Menezes

9 likes4,348 views

Vegas is a visualization library for Scala, designed to fill the gap in the Scala/Spark ecosystem similar to Matplotlib in Python. It allows for declarative statistical visualizations with built-in operations like filtering and aggregation, enhancing productivity in machine learning experimentation. The library also supports various plot types, interactive selections, and is designed for seamless integration within notebooks.

Data & Analytics

Vegas
The Missing Matplotlib for
Scala/Spark
DB Tsai
Roger Menezes

Homepage Kids Page Downloads Page
Netflix Recommendations
Every aspect
of the
Experience is
Machine
Learned

Genres: 23 rows/page average
Sims: 10 rows/page average

My List:
Continue Watching:
Popular on Netflix:
Trending Now:
Watch It Again:
Top Picks:
Because You Watched:
Genres:
New Releases:
Recently Added:
Originals RowBillboard:

Machine Learning at Netflix
● Optimize the Experimentation usecase vs Productionization
● Experimentation
○ Opportunity sizing, Data Exploration
○ Feature Identification and Selection
○ Tweaks to ML algos
○ Model Evaluation

Experimenter’s loop
Problem
Explore
Data
Identify
Features
Produce
Model
Evaluate
Model
Share
Findings

Notebooks
● Optimal for Experimentation
● Sharing reproducible research
○ Facilitates feedback loop with Product Managers
● End to end ML experiment.
○ Interactivity drives productivity

Python Notebooks
● Seamless Experience - ML experimentation
● Well known Scientific computing libraries
● Huge catalog of Visualization plotting libraries
○ Matplotlib, Seaborn, Bokeh, BQPlot, Lightning, etc.

Scala Notebooks
● Zeppelin, Jupyter, Databricks, Spark-Notebooks, ...
● Computing library gap filling up
● Lack of Visualization Libraries
○ Main friction point in adoption
○ End to End ML use case not convincing

Introducing Vegas
● Visualization Library in Scala
● Mainly built for the notebook use case
● Scala wrapper around Vega-Lite
○ Missing MatPlotLib for the Scala/Spark world.

DECLARATIVE
STATISTICAL
VISUALIZATION
GRAMMAR
IN SCALA
You tell it WHAT should be done with the data, and it knows
HOW to do it!
Operations such as filtering, aggregation, faceting are built
into the visualization, rather than putting the burden on the
user to massage the data into shape.
Complex visualizations can be built with a few high level
abstractions:
DATA
TRANS-
FORMS
SCALES
GUIDES MARKS
cf : Altair Talk by Brian Granger in PyData 2016 https://youtu.be/v5mrwq7yJc4

Added Bonus of Declarative
Visualizations:
INTERACTIVITY!
D3JS
VEGAS
VEGAS CODE EXPANDS OUT TO D3JS CODE!

Anatomy of a plot: Channels
X/Y channel
Shape Channel
Size Channel
Color Channel

VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger Menezes

5. Built-in spark support
Vegas
.withDataFrame(myDataFrame)
.encodeX(“population”)
.encodeY(“age”)
Mapped Columns
Pass In DF.

6. Visual statistics
● Advanced Binning
● Sorting
● Scaling
● Custom Transforms
● Time Series
● Aggregation
● Filtering
● Math functions (log, etc)
● Descriptive Statistics

1. Specify in Scala
2. Embed HTML
(iFrame)
3. Render within
iFrame using JS

VEGA
D3JS
VEGA-LITE*
VEGAS
MOREABSTRACTION SCALA DSL EMITS TYPE-CHECKED
VEGA-LITE JSON
VEGA-LITE CONVERTS INTERNALLY
TO VEGA JSON SPEC
VEGA TRANSLATES JSON TO D3JS
CODE THAT CAN BE VERY VERBOSE
A SCALA DSL FOR VEGA-LITE
* Vega-Lite

Contributors
Aish DB Roger
Sudeep Jeremy

@NetflixResearch
@rogermenezes @dbtsai
The missing MatPlotLib
for Scala/Spark
http://vegas-viz.org

The document introduces Vegas, a visualization library for Scala, designed to fill the gap of visualization tools for Scala and Spark, similar to matplotlib for Python. It emphasizes the importance of a seamless machine learning experimentation experience, promoting interactivity and ease of use through high-level abstractions and integrated data manipulation features. Key functionalities include support for various plot types, statistical visualizations, and built-in support for Spark dataframes.

A Developer’s View into Spark's Memory Model with Wenchen FanDatabricks

Using Pluggable Apache Spark SQL Filters to Help GridPocket Users Keep Up wit...Spark Summit

The document discusses the use of pluggable Apache Spark SQL filters by GridPocket to enhance energy management and consumer engagement in smart grid applications. It emphasizes the importance of peer comparison in influencing energy-saving behaviors and details a dataset management solution involving object storage and metadata indexing. The proposed methods aim to improve query performance and reduce data transfer costs in Spark SQL operations, specifically for geospatial and other types of data.

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...Spark Summit

The document discusses MMLSpark, a machine learning library designed for Apache Spark, detailing its goals, architecture, and lessons learned during development. Key aspects include enhancing the Spark ecosystem through integration, improving model management, and reducing overhead from user-defined functions while ensuring composability. An example use case highlights its application in automatic image classification for snow leopard conservation, saving time and increasing accuracy.

State of Spark in the cloud (Spark Summit EU 2017)Nicolas Poggi

The document evaluates the performance of Spark and Hive in various Platform-as-a-Service (PaaS) cloud environments using the BigBench benchmark, focusing on scalability and efficiency. It compares different cloud providers, charts performance metrics, and concludes that while Spark shows improved performance at high data scales, Hive remains competitive at lower scales. Ultimately, the results highlight the tuning capabilities of cloud clusters and the potential for optimized SQL workloads in Hadoop environments.

Optimal Strategies for Large Scale Batch ETL Jobs with Emma TangDatabricks

The document discusses optimal strategies for managing large-scale batch ETL jobs at Neustar, emphasizing efficient resource use and addressing issues such as data skew and memory management. It provides insights into handling large datasets, improving job performance, and using tools like Ganglia for monitoring. Key recommendations include increasing partitioning, filtering data, and configuring Spark settings for performance optimization.

Building a Business Logic Translation Engine with Spark Streaming for Communi...Spark Summit

Patrick Bamba presents on building a translation engine with Spark Streaming to enable communication between legacy code and microservices. The translation engine acts as an anti-corruption layer, translating data and requests between the different systems. Spark is well-suited for this purpose due to its streaming capabilities and built-in connectors for various data sources and sinks. The presentation provides an example implementation using structured streaming to interface legacy systems with microservices through sources, transformations, and sinks.

Scaling out Tensorflow-as-a-Service on Spark and Commodity GPUsJim Dowling

The document discusses advancements in distributed deep learning, particularly the integration of TensorFlow with Spark to optimize training times and resources through frameworks like Horovod and Hopsworks. It highlights the AI hierarchy of needs, the technical requirements for deep learning systems, and the importance of efficient data pipeline management and GPU utilization. Additionally, it includes examples of practical applications and setups for achieving scalable performance in machine learning tasks.

SSR: Structured Streaming for R and Machine Learningfelixcss

The document outlines an overview of structured streaming in Apache Spark, specifically integrating machine learning with the R programming language. It discusses streaming concepts, the benefits of using R for statistical computing, and practical implementation challenges when combining streaming data and machine learning. Additionally, it highlights the use of user-defined functions in R to facilitate data processing and model updates.

Hardware Acceleration of Apache Spark on Energy-Efficient FPGAs with Christof...Spark Summit

The document discusses the use of FPGA accelerators to enhance the performance of Spark applications on the cloud, particularly in machine learning tasks such as logistic regression and digit recognition. It highlights the advantages of combining CPUs with FPGAs for improved efficiency and speedup compared to traditional setups, showcasing significant performance gains. Inaccel's hardware accelerators facilitate seamless integration with Apache Spark and Amazon AWS, allowing users to enhance their applications without the need for extensive code modifications.

Challenging Web-Scale Graph Analytics with Apache Spark with Xiangrui MengDatabricks

The document discusses challenging web-scale graph analytics using Apache Spark, particularly focusing on the GraphFrames package introduced in 2016 which enhances graph processing capabilities. It highlights the transition from RDDs to DataFrames for improved performance, along with supported graph algorithms like PageRank and connected components. The presentation also outlines lessons learned from implementing these algorithms, including strategies for handling data skewness and optimizing performance in large-scale graph processing.

Case study- Real-time OLAP Cubes Ziemowit Jankowski

This case study describes an approach to building quasi real-time OLAP cubes in Microsoft SQL Server Analysis Services to enable daily comparisons of production forecasts and outcomes. The cubes are partitioned by time to allow independent and frequent updates. Initial attempts failed due to deadlocks from simultaneous partition updates. The working solution takes advantage of a 6 day work week by switching partition dates on Saturdays only and reprocessing partitions then. This allows real-time and historical partition updates without gaps or overlaps in the data.

Art of Feature Engineering for Data Science with Nabeel SarwarSpark Summit

This document discusses feature engineering in data science, particularly in the context of machine learning at Comcast. It covers various processes involved in feature engineering such as selecting, cleaning, transforming, and enriching data, as well as methodologies for handling categorical and numerical variables. Additionally, it highlights the importance of domain knowledge, statistical correlation, and the challenges of dimensionality in optimizing machine learning models.

Spark Summit EU talk by Patrick Baier and Stanimir DragievSpark Summit

This document discusses Zalando's transition from using Scikit-learn and Python for machine learning tasks to using Spark and Scala on AWS. Zalando is a large European fashion platform that processes over 55 million orders per year. Their Payment Analytics team uses machine learning to estimate fraud risk for orders. Previously, their setup had issues with scalability, maintenance, and coupling of data and models. By moving to Spark on AWS using Scala, they were able to gain horizontal scaling, increase learning and prediction speeds by 2x, and unify their data sources. There were initial steep learning curves for Scala and debugging distributed systems, but overall Spark and AWS eliminated all of their previous pain points around scalability.

Lessons from the Field: Applying Best Practices to Your Apache Spark Applicat...Databricks

This document discusses best practices for optimizing Apache Spark applications. It covers techniques for speeding up file loading, optimizing file storage and layout, identifying bottlenecks in queries, dealing with many partitions, using datasource tables, managing schema inference, file types and compression, partitioning and bucketing files, managing shuffle partitions with adaptive execution, optimizing unions, using the cost-based optimizer, and leveraging the data skipping index. The presentation aims to help Spark developers apply these techniques to improve performance.

A Journey into Databricks' Pipelines: Journey and Lessons LearnedDatabricks

The document discusses Databricks' development of a next-generation data pipeline utilizing Apache Spark, highlighting challenges like fault tolerance and scalability. It outlines the architecture of their data pipeline, including real-time and batch processing capabilities, and shares lessons learned regarding efficiency and cost management. The conclusion emphasizes the benefits of Databricks and Apache Spark as a unified platform for ETL, data warehousing, and analytics.

Spark Summit EU 2015: Lessons from 300+ production usersDatabricks

The document outlines key lessons learned from over 100 production users of Spark at Databricks, focusing on performance challenges, optimizations, and common pitfalls in using Spark with Python and R. It emphasizes the importance of DataFrames for performance improvements and managing data efficiently, while addressing issues related to data processing and storage, particularly with S3. Additionally, it presents best practices for avoiding common mistakes, such as inappropriate use of RDD operations and the significance of join conditions in SQL.

Storage Engine Considerations for Your Apache Spark Applications with Mladen ...Spark Summit

The document discusses various storage solutions for Apache Spark applications, including HDFS, HBase, Kudu, and Solr, outlining their unique capabilities and ideal use cases. It emphasizes the importance of understanding ingestion and consumption requirements, asking the right questions, and carefully choosing a storage system based on specific use-case needs. Additionally, it provides design patterns and implementation strategies for integrating these storage systems with Spark.

Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and ...Spark Summit

The document discusses the challenges and best practices for hosting multitenant Spark notebook workloads, particularly focusing on improvements made at IBM. Key strategies include utilizing kernel remoting, efficient resource scheduling, and establishing smaller, isolated clusters to enhance performance and security. Additionally, the document emphasizes the importance of proactive resource management to prevent issues like starvation and improve overall analytics capabilities.

Spark Summit EU talk by Ruben Pulido Behar VeliqiSpark Summit

The document discusses IBM's transition from a single-tenant Hadoop architecture to a multi-tenant Apache Spark architecture for their Watson Analytics for Social Media product. The new architecture aggregates social media data from thousands of tenants into a single stream and uses Spark, Kafka and Zookeeper to provide robust real-time analytics with low latency switching between tenants. Key aspects of the new architecture include separating analytics into tenant-specific and language-specific components, and removing state from processing components.

What's new in pandas and the SciPy stack for financial usersWes McKinney

Wes McKinney discusses updates and planned improvements to Python packages for financial analysis, including pandas, NumPy, IPython, Cython, matplotlib, and statsmodels. Major changes include a redesign of pandas' DataFrame internals, hierarchical indexing, time series functionality in statsmodels, and performance optimizations. McKinney aims to make pandas the foundation for rich statistical computing and leverage the best of other languages in Python.

Apache Spark Usage in the Open Source EcosystemDatabricks

The document discusses the usage statistics and library integration of Apache Spark among users of Databricks, highlighting user demographics and language preferences. Users have extensively adopted external libraries for ETL, visualization, and advanced analytics across Python, Scala, and R. The summary indicates a strong trend towards mixing languages and packages, showcasing the flexibility of Apache Spark in an open-source ecosystem.

Spark Summit San Francisco 2016 - Ali Ghodsi KeynoteDatabricks

This document discusses Apache Spark and analytics in the cloud. It summarizes that there is a gap between the growth of data and ability to perform real-time analytics. It introduces Databricks as a cloud-hosted platform that can democratize big data by providing an integrated workspace with automated Apache Spark management and production-ready applications. Databricks also provides the first end-to-end security solution for Apache Spark to address challenges in securing analytics.

Fabian Hueske – Juggling with Bits and BytesFlink Forward

This document discusses how Apache Flink operates on binary data. Flink adopts a database management system approach by serializing data objects into fixed memory segments for efficient in-memory and out-of-memory processing. This approach improves memory safety, reduces garbage collection overhead, and allows for efficient algorithms to operate directly on the binary data representations. It requires significant implementation effort compared to using generic Java collections, but provides benefits like predictable performance and resource usage.

Migrating Complex Data Aggregation from Hadoop to Spark-(Ashish Singh andPune...Spark Summit

This document discusses migrating complex data aggregations from Hadoop to Spark. It outlines PubMatic's use cases involving large scale data and complex data flows. PubMatic developed an industry-first real-time analytics solution dealing with ever-increasing data scale and complexity. They faced challenges with hardware costs, complex data flows, and cardinality estimation for billions of users. Three use cases are presented showing Spark is faster than Hive for multi-stage workflows by 85%, cardinality estimation by 25-30%, and grouping sets by 150%. Tuning Spark configuration and challenges faced are also discussed.

Tagging and Processing Data in Real Time-(Hari Shreedharan and Siddhartha Jai...Spark Summit

This document discusses using Spark Streaming to process and normalize log streams in real time from 100k events per second to over 1 million per second. It proposes using RSyslog to collect logs from multiple sources into Kafka, then using Spark Streaming to apply regex matching and extract fields to normalize the data into a structured JSON format and write it to additional Kafka topics for storage and further processing. The solution was able to process 3 billion events per day with less than 20 seconds of end-to-end delay at peak throughput.

Espresso: LinkedIn's Distributed Data Serving Platform (Talk)Amy W. Tang

The document discusses LinkedIn's distributed data serving platform, Espresso, detailing its architecture, design principles, and functionalities such as fault tolerance and secondary indexing. It highlights the platform's capabilities in handling high read/write ratios, modularity, and integration with existing data ecosystems. Espresso, in production since June 2012, showcases a timeline consistent and document-oriented approach, serving various key use cases for LinkedIn's extensive user base.

Spark Summit EU talk by Ram Sriharsha and Vlad FeinbergSpark Summit

This document summarizes an online machine learning framework called Structured Streaming that is being developed for Apache Spark. Some key points: - It allows machine learning algorithms to be applied continuously to streaming data and update models incrementally in an online fashion. - Models are updated every time interval (e.g. every second) based on new data within that interval. This provides an approximation of processing all data to date. - It uses a stateful aggregation approach to allow models to be updated and merged across distributed partitions in a way that is deterministic but not necessarily commutative. - APIs are provided for common online learning algorithms like online logistic regression and gradient descent to interface with streaming data sources and sinks.

Spark Summit EU talk by Sudeep Das and Aish FaentonSpark Summit

The researcher wants to plot popular Netflix titles by country using Spark DataFrames. Doing this manually would be painful, requiring filtering, aggregation, faceting and other operations on the DataFrame before plotting. Vegas provides a declarative grammar for statistical visualization in Scala, allowing the user to specify what to do with the data without worrying about how. It works by compiling a Scala DSL to Vega-Lite JSON, then to Vega JSON and D3JS code to generate the visualization.

Data Processing and Visualisation Frameworks - Lecture 6 - Information Visual...Beat Signer

The document discusses various data visualization frameworks and tools used in programming languages such as R and Python for creating visual representations of large datasets. It highlights specific libraries like ggplot2, D3.js, and Dash, along with business intelligence tools like Tableau and Microsoft Power BI. Additionally, it references tutorials and resources for learning these technologies, aiming to assist designers and developers in effective data visualization.

More Related Content

What's hot (20)