VEGAS: The Missing Matplotlib for Scala/Apache Spark with Roger Menezes and DB Tsai

Oct 31, 20174 likes1,520 views

The document introduces Vegas, a visualization library for Scala, designed to fill the gap of visualization tools for Scala and Spark, similar to matplotlib for Python. It emphasizes the importance of a seamless machine learning experimentation experience, promoting interactivity and ease of use through high-level abstractions and integrated data manipulation features. Key functionalities include support for various plot types, statistical visualizations, and built-in support for Spark dataframes.

Data & Analytics

Vegas
The Missing Matplotlib for
Scala/Spark
DB Tsai
Roger Menezes

Homepage Kids Page Downloads Page
Netflix Recommendations
Every aspect
of the
Experience is
Machine
Learned

Genres: 23 rows/page average
Sims: 10 rows/page average

My List:
Continue Watching:
Popular on Netflix:
Trending Now:
Watch It Again:
Top Picks:
Because You Watched:
Genres:
New Releases:
Recently Added:
Originals RowBillboard:

ML at Netflix
● Optimize the Experimentation usecase vs Productionization
● Experimentation
○ Opportunity sizing, Data Exploration
○ Tweaks to ML algos
○ Feature Selection
○ Model Evaluation

Notebooks
● Optimal for Experimentation
● Sharing reproducible research
○ Facilitates feedback loop with PMs
● End to end ML experiment.
○ Interactivity drives productivity

Python Notebooks
● Seamless Experience - ML experimentation
● Well known Scientific computing libraries
● Huge catalog of Visualization plotting libraries
○ Matplotlib, Seaborn, Bokeh, BQPlot, Lightning, etc.

Scala Notebooks
● Zeppelin, Jupyter, Databricks, Spark-Notebooks, ...
● Computing library gap filling up
● Lack of Visualization Libraries
○ Main friction point in adoption
○ End to End ML use case not convincing

Introducing Vegas
● Visualization Library in Scala
● Mainly built for the notebook use case
● Scala wrapper around Vega-Lite
● Missing MatPlotLib for the Scala and Spark world.

VegaLite
● Statistical Visualization
● Design considerations for vega-lite
○ Imperative vs Declarative API

DECLARATIVE
STATISTICAL
VISUALIZATION
GRAMMAR
IN SCALA
You tell it WHAT should be done with the data, and it knows
HOW to do it!
Operations such as filtering, aggregation, faceting are built
into the visualization, rather than putting the burden on the
user to massage the data into shape.
Complex visualizations can be built with a few high level
abstractions:
DATA
TRANS-
FORMS
SCALES
GUIDES MARKS
cf : Altair Talk by Brian Granger in PyData 2016 https://youtu.be/v5mrwq7yJc4

Added Bonus of Declarative
Visualizations:
INTERACTIVITY!
D3JS
VEGAS
VEGAS CODE EXPANDS OUT TO D3JS CODE!

Anatomy of a plot: Channels
X/Y channel
Shape Channel
Size Channel
Color Channel

5. Built-in spark support
Vegas
.withDataFrame(myDataFrame)
.encodeX(“population”)
.encodeY(“age”)
Mapped Columns
Pass In DF.

6. Visual statistics
● Advanced Binning
● Sorting
● Scaling
● Custom Transforms
● Time Series
● Aggregation
● Filtering
● Math functions (log, etc)
● Missing data support
● Descriptive Statistics

1. Specify in Scala
2. Embed HTML
(iFrame)
3. Render within
iFrame using JS

VEGA
D3JS
VEGA-LITE*
VEGAS
MOREABSTRACTION SCALA DSL EMITS TYPE-CHECKED
VEGA-LITE JSON
VEGA-LITE CONVERTS INTERNALLY
TO VEGA JSON SPEC
VEGA TRANSLATES JSON TO D3JS
CODE THAT CAN BE VERY VERBOSE
A SCALA DSL FOR VEGA-LITE
* Vega-Lite

VEGAS: The Missing Matplotlib for Scala/Apache Spark with Roger Menezes and DB Tsai

@NetflixResearch
@rogermenezes @dbtsai
The missing MatPlotLib
for Scala/Spark
http://vegas-viz.org

Vegas is a visualization library for Scala, designed to fill the gap in the Scala/Spark ecosystem similar to Matplotlib in Python. It allows for declarative statistical visualizations with built-in operations like filtering and aggregation, enhancing productivity in machine learning experimentation. The library also supports various plot types, interactive selections, and is designed for seamless integration within notebooks.

Building a Business Logic Translation Engine with Spark Streaming for Communi...Spark Summit

Patrick Bamba presents on building a translation engine with Spark Streaming to enable communication between legacy code and microservices. The translation engine acts as an anti-corruption layer, translating data and requests between the different systems. Spark is well-suited for this purpose due to its streaming capabilities and built-in connectors for various data sources and sinks. The presentation provides an example implementation using structured streaming to interface legacy systems with microservices through sources, transformations, and sinks.

State of Spark in the cloud (Spark Summit EU 2017)Nicolas Poggi

The document evaluates the performance of Spark and Hive in various Platform-as-a-Service (PaaS) cloud environments using the BigBench benchmark, focusing on scalability and efficiency. It compares different cloud providers, charts performance metrics, and concludes that while Spark shows improved performance at high data scales, Hive remains competitive at lower scales. Ultimately, the results highlight the tuning capabilities of cloud clusters and the potential for optimized SQL workloads in Hadoop environments.

SSR: Structured Streaming for R and Machine Learningfelixcss

The document outlines an overview of structured streaming in Apache Spark, specifically integrating machine learning with the R programming language. It discusses streaming concepts, the benefits of using R for statistical computing, and practical implementation challenges when combining streaming data and machine learning. Additionally, it highlights the use of user-defined functions in R to facilitate data processing and model updates.

How to Share State Across Multiple Apache Spark Jobs using Apache Ignite with...Spark Summit

The document presents a comprehensive overview of Apache Ignite and its integration with Apache Spark, focusing on sharing state across Spark jobs. It details the functionalities of Ignite, including its memory-centric design, deployment modes, and the use of IgniteContext and IgniteRDD for real-time data sharing. Additionally, the document covers installation procedures and provides resources for further exploration of Ignite for Spark.

MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...Spark Summit

The document discusses MMLSpark, a machine learning library designed for Apache Spark, detailing its goals, architecture, and lessons learned during development. Key aspects include enhancing the Spark ecosystem through integration, improving model management, and reducing overhead from user-defined functions while ensuring composability. An example use case highlights its application in automatic image classification for snow leopard conservation, saving time and increasing accuracy.

High Performance Enterprise Data Processing with Apache Spark with Sandeep Va...Spark Summit

ZS Associates, with over 4,500 professionals globally, specializes in high-performance data solutions for life sciences, aiming to streamline complex systems and enhance operation efficiency. The document outlines their use of Apache Spark and cloud technologies to process vast data inputs while emphasizing the importance of domain expertise and effective DevOps practices. Key learnings include performance optimization strategies in Spark, methods for data management adaptation, and the use of visual orchestration to meet business needs.

High Performance Python on Apache SparkWes McKinney

This document contains the slides from a presentation given by Wes McKinney on high performance Python on Apache Spark. The presentation discusses why Python is an important and productive language, defines what is meant by "high performance Python", and explores techniques for building fast Python software such as embracing limitations of the Python interpreter and using native data structures and compiled extensions where needed. Specific examples are provided around control flow, reading CSV files, and the importance of efficient in-memory data structures.

Spark Summit 2016: Connecting Python to the Spark EcosystemDaniel Rodriguez

This document discusses connecting Python to the Spark ecosystem. It covers the PyData and Spark ecosystems, package management for Python libraries in a cluster, leveraging Spark with tools like Sparkonda and Conda, and using Python with Spark including with NLTK, scikit-learn, TensorFlow, and Dask. Future directions include Apache Arrow for efficient data structures and leveraging alternative computing frameworks like TensorFlow.

What's New in Apache Spark 2.3 & Why Should You CareDatabricks

The document discusses the major features and enhancements in Apache Spark 2.3, including improvements in continuous processing, stream-stream joins, and PySpark performance. Key elements include the new structured streaming execution mode that offers low latency, support for various data formats, and integration with Kubernetes. The emphasis is placed on building robust streaming applications and the advantages of using Spark's unified analytics platform.

[Spark Summit EU 2017] Apache spark streaming + kafka 0.10 an integration storyJoan Viladrosa Riera

The document discusses the integration of Apache Spark Streaming with Kafka, detailing its architecture, terminology, and various operational semantics like at-most-once, at-least-once, and exactly-once message handling. It highlights key features of both technologies, emphasizes Kafka's role as a message broker, and explains various strategies for consumer configuration and data processing. Additionally, it reflects on use cases and considerations for avoiding data loss or duplication in practical applications.

Deep Learning and Streaming in Apache Spark 2.x with Matei ZahariaJen Aman

The document discusses advancements in Apache Spark 2.x, particularly focusing on structured streaming and deep learning capabilities. It highlights the introduction of a high-level API for streaming that simplifies processes and allows for end-to-end applications, as well as the integration of deep learning frameworks to enhance usability. Key features include improved performance, cost efficiency, and the goal of making deep learning accessible to a broader range of users.

Hardware Acceleration of Apache Spark on Energy-Efficient FPGAs with Christof...Spark Summit

The document discusses the use of FPGA accelerators to enhance the performance of Spark applications on the cloud, particularly in machine learning tasks such as logistic regression and digit recognition. It highlights the advantages of combining CPUs with FPGAs for improved efficiency and speedup compared to traditional setups, showcasing significant performance gains. Inaccel's hardware accelerators facilitate seamless integration with Apache Spark and Amazon AWS, allowing users to enhance their applications without the need for extensive code modifications.

Scaling out Tensorflow-as-a-Service on Spark and Commodity GPUsJim Dowling

The document discusses advancements in distributed deep learning, particularly the integration of TensorFlow with Spark to optimize training times and resources through frameworks like Horovod and Hopsworks. It highlights the AI hierarchy of needs, the technical requirements for deep learning systems, and the importance of efficient data pipeline management and GPU utilization. Additionally, it includes examples of practical applications and setups for achieving scalable performance in machine learning tasks.

Supporting Highly Multitenant Spark Notebook Workloads with Craig Ingram and ...Spark Summit

The document discusses the challenges and best practices for hosting multitenant Spark notebook workloads, particularly focusing on improvements made at IBM. Key strategies include utilizing kernel remoting, efficient resource scheduling, and establishing smaller, isolated clusters to enhance performance and security. Additionally, the document emphasizes the importance of proactive resource management to prevent issues like starvation and improve overall analytics capabilities.

Build, Scale, and Deploy Deep Learning Pipelines with Ease Using Apache SparkDatabricks

The document presents a talk on building, scaling, and deploying deep learning pipelines using Apache Spark, highlighting its advantages over traditional methods. It covers topics such as the workflow of deep learning, the integration of Spark with deep learning libraries, and examples of applying pre-trained models. The presentation emphasizes the simplicity and efficiency of Spark for deep learning tasks, along with future developments in this area.

Extreme Apache Spark: how in 3 months we created a pipeline that can process ...Josef A. Habdank

The document outlines the process of creating a high-capacity data processing pipeline using Apache Spark, capable of handling up to 8 billion records per day. It details a successful implementation achieved in three months, using low-cost cloud servers, while discussing Spark's core functionalities and best practices for high throughput coding. The presentation also addresses the challenges faced and solutions found while scaling data operations and optimizing performance.

Spark Summit EU talk by Patrick Baier and Stanimir DragievSpark Summit

This document discusses Zalando's transition from using Scikit-learn and Python for machine learning tasks to using Spark and Scala on AWS. Zalando is a large European fashion platform that processes over 55 million orders per year. Their Payment Analytics team uses machine learning to estimate fraud risk for orders. Previously, their setup had issues with scalability, maintenance, and coupling of data and models. By moving to Spark on AWS using Scala, they were able to gain horizontal scaling, increase learning and prediction speeds by 2x, and unify their data sources. There were initial steep learning curves for Scala and debugging distributed systems, but overall Spark and AWS eliminated all of their previous pain points around scalability.

Deep Learning with Apache Spark and GPUs with Pierce SpitlerDatabricks

The document discusses various deep learning frameworks that can be integrated with Spark and GPUs, highlighting popular options like TensorFlow, Caffe, and Apache SystemML. It dives into performance comparisons for different configurations of single and multi-GPU setups, as well as distributed implementations with a focus on throughput and optimization strategies. Key takeaways emphasize the importance of hardware and network configurations for effective deep learning execution, alongside future work suggestions for optimizing performance with existing infrastructure.

Dynamic DDL: Adding Structure to Streaming Data on the Fly with David Winters...Databricks

The document outlines GoPro's transition from a file-based data pipeline architecture to a dynamic DDL architecture for real-time streaming data processing. Key aspects include handling diverse data formats, the implementation of a centralized Hive metastore, and the use of Amazon S3 for storage, highlighting the importance of dynamic DDL for rapid schema adaptation and efficient data ingestion. It also discusses challenges faced with the old system and improvements achieved with the new architecture, including reduced operational overhead and enhanced elasticity.

Apache Spark Performance: Past, Future and PresentDatabricks

This document discusses improving performance instrumentation in Apache Spark. It summarizes that existing instrumentation focuses on blocked times in the main task thread, but opportunities exist to better instrument read/write times and machine-level resource utilization. The author proposes combining per-task I/O metrics with machine utilization data to provide complete metrics about time spent using each resource on a per-task basis, improving performance clarity for users. More details are available at the listed website.

Deep Learning on Apache® Spark™ : Workflows and Best PracticesJen Aman

The document presents an overview of deep learning workflows and best practices using Apache Spark, highlighting its integration with various deep learning frameworks such as TensorFlow and Caffe. It discusses the architecture of Databricks, which simplifies data processing and machine learning for users, as well as monitoring techniques and solutions for managing distributed deep learning tasks. The presentation emphasizes the importance of experimentation and provides resources for further learning.

Scaling Apache Spark on Kubernetes at LyftDatabricks

The document discusses Lyft's data landscape and the implementation of Apache Spark on Kubernetes to improve batch data compute processes. It highlights challenges faced in data infrastructure, such as vendor dependency and versioning issues, while suggesting solutions through Kubernetes for enhanced scalability and resource management. Key takeaways emphasize the potential of Spark and Kubernetes in unifying batch data compute use cases despite existing challenges.

Apache Spark Usage in the Open Source EcosystemDatabricks

The document discusses the usage statistics and library integration of Apache Spark among users of Databricks, highlighting user demographics and language preferences. Users have extensively adopted external libraries for ETL, visualization, and advanced analytics across Python, Scala, and R. The summary indicates a strong trend towards mixing languages and packages, showcasing the flexibility of Apache Spark in an open-source ecosystem.

Analyzing IOT Data in Apache Spark Across Data Centers and Cloud with NetApp ...Databricks

The document discusses the challenges of managing IoT data and explores NetApp's Data Fabric architecture for big data, which helps organizations use hybrid cloud solutions effectively. It presents various customer use cases illustrating how IoT data can be analyzed in real-time and stored across different platforms, emphasizing the importance of data protection and cost management. Additionally, it highlights the benefits of using Apache Spark for analytics and the integration of on-premises and cloud storage solutions.

Spark Summit EU talk by Ruben Pulido Behar VeliqiSpark Summit

The document discusses IBM's transition from a single-tenant Hadoop architecture to a multi-tenant Apache Spark architecture for their Watson Analytics for Social Media product. The new architecture aggregates social media data from thousands of tenants into a single stream and uses Spark, Kafka and Zookeeper to provide robust real-time analytics with low latency switching between tenants. Key aspects of the new architecture include separating analytics into tenant-specific and language-specific components, and removing state from processing components.

Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...Jen Aman

This document discusses optimizations made to Apache Spark MLlib algorithms to better support sparse data at large scale. It describes how KMeans, linear methods, and other ML algorithms were modified to use sparse vector representations to reduce memory usage and improve performance when working with sparse data, including optimizations made for clustering large, high-dimensional datasets. The optimizations allow these algorithms to be applied to much larger sparse datasets and high-dimensional problems than was previously possible with MLlib.

Spark Summit EU talk by Heiko KorndorfSpark Summit

Heiko Korndorf presented on scaling SparkR in production and lessons from real-world projects. The presentation covered classifying SparkR as both a data science and data engineering tool. It discussed SparkR's architecture in versions 1.x and 2.x, approaches to parallelizing R code with Spark 1.5/1.6 and YARN, and dynamic R deployment including dependencies. Advanced techniques like 2-level parallelization using GPGPU were also presented. The talk concluded with an outlook on further integrating data engineering and data science and new technical approaches like simplifying data pipelines and moving calculations to GPUs.

Ehtsham Elahi, Senior Research Engineer, Personalization Science and Engineer...MLconf

The document discusses the implementation of Spark and GraphX in Netflix's recommender system, aiming to enhance content discovery for over 62 million members. It details the challenges of scaling machine learning algorithms across various devices and countries, and presents solutions using distributed algorithms to perform tasks like graph diffusion and clustering. Findings indicate that GraphX significantly improves performance for large datasets and highlights lessons learned regarding iterative machine learning and state management.

.NET per la Data Science e oltreMarco Parenzan

The document discusses the evolution of the C# programming language and its adaptation for data science, focusing on the integration of .NET with Apache Spark. It highlights the advancements in C# versions, the introduction of data processing capabilities, and the functionalities provided by Azure Synapse Analytics for big data and machine learning. The conclusion emphasizes that Spark is now accessible to .NET developers, positioning Azure Synapse Analytics as a bridge between traditional and modern analytics practices.

More Related Content

What's hot (20)