Build, Scale, and Deploy Deep Learning Pipelines Using Apache Spark
2 likes1,393 views
The document discusses the development and deployment of deep learning pipelines using Apache Spark, highlighting the current state of deep learning and its challenges in industry. It emphasizes the use of Databricks' open-source library to simplify deep learning workflows, integrate with common model architectures, and leverage pre-trained models. It also outlines the typical deep learning workflow and future directions, including non-image data domains and enhanced model support.
1 of 55
Downloaded 78 times
![Transfer Learning as a Pipeline
35put your #assignedhashtag here by setting the
featurizer = DeepImageFeaturizer(inputCol="image",
outputCol="features",
modelName="InceptionV3")
lr = LogisticRegression(labelCol="label")
p = Pipeline(stages=[featurizer, lr])
p_model = p.fit(train_df)](https://image.slidesharecdn.com/20171108basmdlpipelines-171124174511/85/Build-Scale-and-Deploy-Deep-Learning-Pipelines-Using-Apache-Spark-35-320.jpg)
Ad
Recommended
Build, Scale, and Deploy Deep Learning Pipelines Using Apache Spark
Build, Scale, and Deploy Deep Learning Pipelines Using Apache SparkDatabricks The presentation by Tim Hunter at the Databricks Spark Meetup discusses deep learning at scale using Apache Spark, highlighting its potential and current limitations in industry adoption. It introduces Deep Learning Pipelines, an open-source library designed for ease of use and integration with Spark, catering primarily to Python users while aiming to simplify the deep learning workflow from data loading to model evaluation. Future developments include support for more backends and improved functionalities such as distributed training and text featurization.
Build, Scale, and Deploy Deep Learning Pipelines with Ease Using Apache Spark
Build, Scale, and Deploy Deep Learning Pipelines with Ease Using Apache SparkDatabricks The document presents a talk on building, scaling, and deploying deep learning pipelines using Apache Spark, highlighting its advantages over traditional methods. It covers topics such as the workflow of deep learning, the integration of Spark with deep learning libraries, and examples of applying pre-trained models. The presentation emphasizes the simplicity and efficiency of Spark for deep learning tasks, along with future developments in this area.
Integrating Deep Learning Libraries with Apache Spark
Integrating Deep Learning Libraries with Apache SparkDatabricks The document discusses integrating deep learning libraries with Apache Spark, highlighting various frameworks such as TensorFlow, Keras, and MXNet that can be utilized within the Spark environment. It emphasizes the development of a deep learning pipeline API to streamline processes like featurization and model training, aiming for easier integration and scalability. The talk outlines challenges pertaining to communication patterns, flexible cluster deployments, and the importance of proper resource management in deep learning applications.
Jump Start with Apache Spark 2.0 on Databricks
Jump Start with Apache Spark 2.0 on DatabricksAnyscale This document provides an agenda for a 3+ hour workshop on Apache Spark 2.x on Databricks. It includes introductions to Databricks, Spark fundamentals and architecture, new features in Spark 2.0 like unified APIs, and workshops on DataFrames/Datasets, Spark SQL, and structured streaming concepts. The agenda covers lunch and breaks and is divided into hour and half hour segments.
Build, Scale, and Deploy Deep Learning Pipelines with Ease
Build, Scale, and Deploy Deep Learning Pipelines with EaseDatabricks The document discusses the process of building, scaling, and deploying deep learning pipelines using Databricks and Apache Spark, emphasizing ease of integration and performance. It covers topics such as the definition of deep learning, successful applications, and a typical workflow, while highlighting the advantages of using Spark for distributed computation. Additionally, it provides insights into the development of deep learning models with transfer learning and model deployment in SQL.
Deep Learning on Apache® Spark™ : Workflows and Best Practices
Deep Learning on Apache® Spark™ : Workflows and Best PracticesJen Aman The document presents an overview of deep learning workflows and best practices using Apache Spark, highlighting its integration with various deep learning frameworks such as TensorFlow and Caffe. It discusses the architecture of Databricks, which simplifies data processing and machine learning for users, as well as monitoring techniques and solutions for managing distributed deep learning tasks. The presentation emphasizes the importance of experimentation and provides resources for further learning.
Apache Spark MLlib's Past Trajectory and New Directions with Joseph Bradley
Apache Spark MLlib's Past Trajectory and New Directions with Joseph BradleyDatabricks - MLlib has rapidly developed over the past 5 years, growing from a few algorithms to over 50 algorithms and featurizers for classification, regression, clustering, recommendation, and more.
- This growth has shifted from just adding algorithms to improving algorithms, infrastructure, and integrating ML workflows with Spark's broader capabilities like SQL, DataFrames, and streaming.
- Going forward, areas of focus include continued scalability improvements, enhancing core algorithms, extensible APIs, and making MLlib a more comprehensive standard library.
What's New in Apache Spark 2.3 & Why Should You Care
What's New in Apache Spark 2.3 & Why Should You CareDatabricks The document discusses the major features and enhancements in Apache Spark 2.3, including improvements in continuous processing, stream-stream joins, and PySpark performance. Key elements include the new structured streaming execution mode that offers low latency, support for various data formats, and integration with Kubernetes. The emphasis is placed on building robust streaming applications and the advantages of using Spark's unified analytics platform.
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark with Ma...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark with Ma...Databricks The document discusses the development of automatic predictive modeling using Apache Spark by researchers Masato Asahara and Ryohei Fujimaki from NEC Corporation. It highlights the transition from traditional manual modeling to automated systems that generate accurate predictive models in hours, showcasing various applications and the technological challenges faced. Future work includes improving speed through advanced hardware and extending the modeling capabilities to deep learning.
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, & Deep Learning ...
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, & Deep Learning ...Databricks The document discusses three deep learning frameworks: TensorFlow, Keras, and Deep Learning Pipelines, highlighting their unique features, advantages, and use cases. It emphasizes the challenges posed by big data to machine learning systems and the importance of frameworks like Apache Spark for data processing. Additionally, the content provides insights on transfer learning, ease of integration, and the potential for deploying deep learning applications.
A Tale of Three Tools: Kubernetes, Jsonnet, and Bazel
A Tale of Three Tools: Kubernetes, Jsonnet, and BazelDatabricks The document discusses the use of Kubernetes, Jsonnet, and Bazel at Databricks, highlighting the challenges of managing multiple Kubernetes clusters and deployments. It emphasizes the benefits of using Jsonnet to simplify configuration management and improve deployment processes, as well as Bazel for customizable builds. The presentation concludes with an invitation for potential hires to join the team.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit The document outlines a presentation on using Redis for real-time machine learning, focusing on Redis-ML integrated with Apache Spark. It covers the capabilities of Redis as a key-value store, the functionalities of Redis-ML for model serving, and a case study involving a recommendation system. The concluding remarks emphasize the advantages of combining Spark for training with Redis for efficient model serving, highlighting reduced resource costs and simplified machine learning lifecycle.
Advanced Natural Language Processing with Apache Spark NLP
Advanced Natural Language Processing with Apache Spark NLPDatabricks The document presents an overview of Spark NLP, focusing on its accuracy, scalability, and speed for various natural language processing tasks. It highlights the library's capabilities in named entity recognition, document classification, sentiment analysis, and spell checking, along with benchmark performance and enterprise applications. Additionally, it provides insights into using pre-trained pipelines, available resources, and upcoming advancements in the field.
Building a Unified Data Pipeline with Apache Spark and XGBoost with Nan Zhu
Building a Unified Data Pipeline with Apache Spark and XGBoost with Nan ZhuDatabricks Nan Zhu discusses the integration of XGBoost and Apache Spark to create a unified machine learning pipeline, providing insights into the functionality and advantages of XGBoost as a gradient boost decision tree framework. The document outlines the collaborative efforts of the Distributed Machine Learning Community (DMLC) and highlights the design aspects of XGBoost-Spark, aimed at simplifying the machine learning process within the Spark ecosystem. Additionally, it emphasizes the importance of efficient communication between XGBoost and Spark for enhanced performance in distributed training scenarios.
Composable Parallel Processing in Apache Spark and Weld
Composable Parallel Processing in Apache Spark and WeldDatabricks The document discusses composable parallel processing in Apache Spark and introduces the Weld runtime, emphasizing the need for efficient composition of libraries in big data processing. It highlights Spark's goals to provide a unified engine and API for batch, interactive, and streaming applications, as well as the benefits of structured APIs in Spark 2.0 for performance and programmability. Additionally, it addresses challenges regarding data representation and the inefficiencies of traditional library composition, proposing new composition interfaces like Weld to optimize data movement and execution across various workloads.
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...Databricks The document discusses the three Apache Spark APIs: RDDs, DataFrames, and Datasets, detailing their features, use cases, and performance characteristics. RDDs offer low-level control and flexibility for working with distributed data, while DataFrames and Datasets provide higher-level abstractions that optimize performance and ease of use. The presentation also highlights the evolution of these APIs and their integration within Spark for structured data processing.
From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's Data
From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's DataDatabricks The document outlines a presentation by Databricks and Virgin Hyperloop on using a unified data analytics platform combining machine learning and big data analytics, specifically highlighting the Koalas library which allows seamless scaling from pandas to Apache Spark. It includes insights from speakers on the applications and challenges faced at Virgin Hyperloop, emphasizing the need for scalable data processing solutions. Additionally, it covers various features of MLflow for tracking machine learning experiments and data analytics within the Databricks ecosystem.
Jump Start on Apache® Spark™ 2.x with Databricks
Jump Start on Apache® Spark™ 2.x with Databricks Databricks The document is a workshop overview introducing Apache Spark 2.x and its features, focusing on DataFrames, Datasets, and Spark SQL. It covers the architecture of Spark, its deployment modes, major improvements in version 2.x, and structured streaming capabilities for handling real-time data processing. Additionally, it includes links to resources for hands-on practice with notebooks on Spark SQL and DataFrames.
Spark r under the hood with Hossein Falaki
Spark r under the hood with Hossein FalakiDatabricks The document is a presentation by Hossein Falaki on debugging SparkR code, covering the architecture, implementation, limitations, and common errors associated with the SparkR API. It highlights the communication between R and Spark, including serialization and deserialization issues, as well as practical debugging tips for common problems encountered when using User Defined Functions (UDFs). The document also emphasizes utilizing Java stack traces for error resolution and provides guidance on avoiding typical pitfalls in SparkR coding.
Performance Optimization Case Study: Shattering Hadoop's Sort Record with Spa...
Performance Optimization Case Study: Shattering Hadoop's Sort Record with Spa...Databricks The performance optimization case study showcases how Spark and Scala shattered Hadoop's sorting records by utilizing efficient in-memory processing and advanced engineering techniques. Key advancements included a sort-based shuffle, native network transport with Netty, and clever application-level optimizations to enhance cache and garbage collection performance. This effort culminated in achieving a new sorting record for 100TB and 1PB datasets, marking a significant milestone in big data processing.
Large-Scale Data Science in Apache Spark 2.0
Large-Scale Data Science in Apache Spark 2.0Databricks This document discusses the enhancements in Apache Spark 2.0, focusing on its scalability for large-scale data science and AI through improved hardware and user scalability. It emphasizes the use of structured APIs for efficient data manipulation and introduces new features for deep learning and integration with existing Python and R libraries. The content highlights Spark's capabilities in parallelizing computations and the ease of building complex data science models with high-level APIs.
Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)Databricks The document discusses the adoption and integration of Apache Spark within major Hadoop distributions, highlighting the need for simplified big data solutions. It outlines the Databricks platform, which offers a unified workspace for managing Spark applications and enhances user experience through training programs and on-demand cluster management. The goal is to make big data processing more accessible and efficient for organizations seeking to leverage their data effectively.
Accelerating Data Science with Better Data Engineering on Databricks
Accelerating Data Science with Better Data Engineering on DatabricksDatabricks Mediamath is a demand-side media buying platform that enhances ad performance using data analytics, facilitated by Databricks for efficient data processing. The document outlines the challenges faced with Hive due to the massive data volume and highlights the successful transition to Spark, resulting in significant performance improvements. The new workflow enables rapid development of data pipelines, increased productivity, and easy integration of data insights for reporting purposes.
Clipper: A Low-Latency Online Prediction Serving System
Clipper: A Low-Latency Online Prediction Serving SystemDatabricks Clipper is a low-latency online prediction serving system designed to support various machine learning models and frameworks, addressing challenges such as high throughput and low-latency serving. It aims to decouple applications from models, using container-based deployment for better resource isolation and functionality. Clipper supports easy integration and deployment of models, particularly from Spark and other frameworks, through a unified interface and adaptive batching strategies.
What's New in Upcoming Apache Spark 2.3
What's New in Upcoming Apache Spark 2.3Databricks Apache Spark 2.3 introduces several major features including continuous processing capabilities, enhancements to the data source API, improved PySpark performance, and robust machine learning operations over streaming data. It emphasizes the use of Databricks Delta for reliable data management and transactions, tackling issues of data corruption and complexity in data pipelines. The updates also include advanced SQL functionalities, support for Kubernetes, and improved stability for user-defined functions.
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...Databricks This document summarizes a presentation on extending Spark SQL Data Sources APIs with join push down. The presentation discusses how join push down can significantly improve query performance by reducing data transfer and exploiting data source capabilities like indexes. It provides examples of join push down in enterprise data pipelines and SQL acceleration use cases. The presentation also outlines the challenges of network speeds and exploiting data source capabilities, and how join push down addresses these challenges. Future work discussed includes building a cost model for global optimization across data sources.
Apache Spark Performance: Past, Future and Present
Apache Spark Performance: Past, Future and PresentDatabricks This document discusses improving performance instrumentation in Apache Spark. It summarizes that existing instrumentation focuses on blocked times in the main task thread, but opportunities exist to better instrument read/write times and machine-level resource utilization. The author proposes combining per-task I/O metrics with machine utilization data to provide complete metrics about time spent using each resource on a per-task basis, improving performance clarity for users. More details are available at the listed website.
Apache Spark: The Next Gen toolset for Big Data Processing
Apache Spark: The Next Gen toolset for Big Data Processingprajods The document presents an overview of Apache Spark as a next-generation toolset for big data processing, highlighting its capabilities compared to the Hadoop stack. It discusses the limitations of Hadoop and how Spark's in-memory processing, resilient distributed data sets, and features such as streaming and machine learning make it a powerful alternative. The document emphasizes Spark's performance advantages, including faster data processing speeds and a more streamlined architecture for handling both batch and real-time data.
Deep-Dive into Deep Learning Pipelines with Sue Ann Hong and Tim Hunter
Deep-Dive into Deep Learning Pipelines with Sue Ann Hong and Tim HunterDatabricks This document discusses the use of Apache Spark for building, scaling, and deploying deep learning pipelines, emphasizing the integration of various libraries such as TensorFlow, Keras, and others. It outlines the typical workflow of deep learning, covers the benefits of using Spark for distributed training and efficient data handling, and highlights features of Databricks' deep learning pipelines which simplify the process without sacrificing performance. Additionally, it addresses future developments and current practices in the industry for enhancing deep learning accessibility.
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, and Deep Learnin...
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, and Deep Learnin...Databricks The document discusses three deep learning frameworks: TensorFlow, Keras, and Deep Learning Pipelines, highlighting their features, benefits, and use cases. It emphasizes the importance of Apache Spark for big data processing and introduces concepts like transfer learning and model serving. The presentation includes a comparison of the frameworks along with the recommended scenarios for their usage.
More Related Content
What's hot (20)
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark with Ma...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark with Ma...Databricks The document discusses the development of automatic predictive modeling using Apache Spark by researchers Masato Asahara and Ryohei Fujimaki from NEC Corporation. It highlights the transition from traditional manual modeling to automated systems that generate accurate predictive models in hours, showcasing various applications and the technological challenges faced. Future work includes improving speed through advanced hardware and extending the modeling capabilities to deep learning.
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, & Deep Learning ...
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, & Deep Learning ...Databricks The document discusses three deep learning frameworks: TensorFlow, Keras, and Deep Learning Pipelines, highlighting their unique features, advantages, and use cases. It emphasizes the challenges posed by big data to machine learning systems and the importance of frameworks like Apache Spark for data processing. Additionally, the content provides insights on transfer learning, ease of integration, and the potential for deploying deep learning applications.
A Tale of Three Tools: Kubernetes, Jsonnet, and Bazel
A Tale of Three Tools: Kubernetes, Jsonnet, and BazelDatabricks The document discusses the use of Kubernetes, Jsonnet, and Bazel at Databricks, highlighting the challenges of managing multiple Kubernetes clusters and deployments. It emphasizes the benefits of using Jsonnet to simplify configuration management and improve deployment processes, as well as Bazel for customizable builds. The presentation concludes with an invitation for potential hires to join the team.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit The document outlines a presentation on using Redis for real-time machine learning, focusing on Redis-ML integrated with Apache Spark. It covers the capabilities of Redis as a key-value store, the functionalities of Redis-ML for model serving, and a case study involving a recommendation system. The concluding remarks emphasize the advantages of combining Spark for training with Redis for efficient model serving, highlighting reduced resource costs and simplified machine learning lifecycle.
Advanced Natural Language Processing with Apache Spark NLP
Advanced Natural Language Processing with Apache Spark NLPDatabricks The document presents an overview of Spark NLP, focusing on its accuracy, scalability, and speed for various natural language processing tasks. It highlights the library's capabilities in named entity recognition, document classification, sentiment analysis, and spell checking, along with benchmark performance and enterprise applications. Additionally, it provides insights into using pre-trained pipelines, available resources, and upcoming advancements in the field.
Building a Unified Data Pipeline with Apache Spark and XGBoost with Nan Zhu
Building a Unified Data Pipeline with Apache Spark and XGBoost with Nan ZhuDatabricks Nan Zhu discusses the integration of XGBoost and Apache Spark to create a unified machine learning pipeline, providing insights into the functionality and advantages of XGBoost as a gradient boost decision tree framework. The document outlines the collaborative efforts of the Distributed Machine Learning Community (DMLC) and highlights the design aspects of XGBoost-Spark, aimed at simplifying the machine learning process within the Spark ecosystem. Additionally, it emphasizes the importance of efficient communication between XGBoost and Spark for enhanced performance in distributed training scenarios.
Composable Parallel Processing in Apache Spark and Weld
Composable Parallel Processing in Apache Spark and WeldDatabricks The document discusses composable parallel processing in Apache Spark and introduces the Weld runtime, emphasizing the need for efficient composition of libraries in big data processing. It highlights Spark's goals to provide a unified engine and API for batch, interactive, and streaming applications, as well as the benefits of structured APIs in Spark 2.0 for performance and programmability. Additionally, it addresses challenges regarding data representation and the inefficiencies of traditional library composition, proposing new composition interfaces like Weld to optimize data movement and execution across various workloads.
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...Databricks The document discusses the three Apache Spark APIs: RDDs, DataFrames, and Datasets, detailing their features, use cases, and performance characteristics. RDDs offer low-level control and flexibility for working with distributed data, while DataFrames and Datasets provide higher-level abstractions that optimize performance and ease of use. The presentation also highlights the evolution of these APIs and their integration within Spark for structured data processing.
From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's Data
From Pandas to Koalas: Reducing Time-To-Insight for Virgin Hyperloop's DataDatabricks The document outlines a presentation by Databricks and Virgin Hyperloop on using a unified data analytics platform combining machine learning and big data analytics, specifically highlighting the Koalas library which allows seamless scaling from pandas to Apache Spark. It includes insights from speakers on the applications and challenges faced at Virgin Hyperloop, emphasizing the need for scalable data processing solutions. Additionally, it covers various features of MLflow for tracking machine learning experiments and data analytics within the Databricks ecosystem.
Jump Start on Apache® Spark™ 2.x with Databricks
Jump Start on Apache® Spark™ 2.x with Databricks Databricks The document is a workshop overview introducing Apache Spark 2.x and its features, focusing on DataFrames, Datasets, and Spark SQL. It covers the architecture of Spark, its deployment modes, major improvements in version 2.x, and structured streaming capabilities for handling real-time data processing. Additionally, it includes links to resources for hands-on practice with notebooks on Spark SQL and DataFrames.
Spark r under the hood with Hossein Falaki
Spark r under the hood with Hossein FalakiDatabricks The document is a presentation by Hossein Falaki on debugging SparkR code, covering the architecture, implementation, limitations, and common errors associated with the SparkR API. It highlights the communication between R and Spark, including serialization and deserialization issues, as well as practical debugging tips for common problems encountered when using User Defined Functions (UDFs). The document also emphasizes utilizing Java stack traces for error resolution and provides guidance on avoiding typical pitfalls in SparkR coding.
Performance Optimization Case Study: Shattering Hadoop's Sort Record with Spa...
Performance Optimization Case Study: Shattering Hadoop's Sort Record with Spa...Databricks The performance optimization case study showcases how Spark and Scala shattered Hadoop's sorting records by utilizing efficient in-memory processing and advanced engineering techniques. Key advancements included a sort-based shuffle, native network transport with Netty, and clever application-level optimizations to enhance cache and garbage collection performance. This effort culminated in achieving a new sorting record for 100TB and 1PB datasets, marking a significant milestone in big data processing.
Large-Scale Data Science in Apache Spark 2.0
Large-Scale Data Science in Apache Spark 2.0Databricks This document discusses the enhancements in Apache Spark 2.0, focusing on its scalability for large-scale data science and AI through improved hardware and user scalability. It emphasizes the use of structured APIs for efficient data manipulation and introduces new features for deep learning and integration with existing Python and R libraries. The content highlights Spark's capabilities in parallelizing computations and the ease of building complex data science models with high-level APIs.
Announcing Databricks Cloud (Spark Summit 2014)
Announcing Databricks Cloud (Spark Summit 2014)Databricks The document discusses the adoption and integration of Apache Spark within major Hadoop distributions, highlighting the need for simplified big data solutions. It outlines the Databricks platform, which offers a unified workspace for managing Spark applications and enhances user experience through training programs and on-demand cluster management. The goal is to make big data processing more accessible and efficient for organizations seeking to leverage their data effectively.
Accelerating Data Science with Better Data Engineering on Databricks
Accelerating Data Science with Better Data Engineering on DatabricksDatabricks Mediamath is a demand-side media buying platform that enhances ad performance using data analytics, facilitated by Databricks for efficient data processing. The document outlines the challenges faced with Hive due to the massive data volume and highlights the successful transition to Spark, resulting in significant performance improvements. The new workflow enables rapid development of data pipelines, increased productivity, and easy integration of data insights for reporting purposes.
Clipper: A Low-Latency Online Prediction Serving System
Clipper: A Low-Latency Online Prediction Serving SystemDatabricks Clipper is a low-latency online prediction serving system designed to support various machine learning models and frameworks, addressing challenges such as high throughput and low-latency serving. It aims to decouple applications from models, using container-based deployment for better resource isolation and functionality. Clipper supports easy integration and deployment of models, particularly from Spark and other frameworks, through a unified interface and adaptive batching strategies.
What's New in Upcoming Apache Spark 2.3
What's New in Upcoming Apache Spark 2.3Databricks Apache Spark 2.3 introduces several major features including continuous processing capabilities, enhancements to the data source API, improved PySpark performance, and robust machine learning operations over streaming data. It emphasizes the use of Databricks Delta for reliable data management and transactions, tackling issues of data corruption and complexity in data pipelines. The updates also include advanced SQL functionalities, support for Kubernetes, and improved stability for user-defined functions.
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...Databricks This document summarizes a presentation on extending Spark SQL Data Sources APIs with join push down. The presentation discusses how join push down can significantly improve query performance by reducing data transfer and exploiting data source capabilities like indexes. It provides examples of join push down in enterprise data pipelines and SQL acceleration use cases. The presentation also outlines the challenges of network speeds and exploiting data source capabilities, and how join push down addresses these challenges. Future work discussed includes building a cost model for global optimization across data sources.
Apache Spark Performance: Past, Future and Present
Apache Spark Performance: Past, Future and PresentDatabricks This document discusses improving performance instrumentation in Apache Spark. It summarizes that existing instrumentation focuses on blocked times in the main task thread, but opportunities exist to better instrument read/write times and machine-level resource utilization. The author proposes combining per-task I/O metrics with machine utilization data to provide complete metrics about time spent using each resource on a per-task basis, improving performance clarity for users. More details are available at the listed website.
Apache Spark: The Next Gen toolset for Big Data Processing
Apache Spark: The Next Gen toolset for Big Data Processingprajods The document presents an overview of Apache Spark as a next-generation toolset for big data processing, highlighting its capabilities compared to the Hadoop stack. It discusses the limitations of Hadoop and how Spark's in-memory processing, resilient distributed data sets, and features such as streaming and machine learning make it a powerful alternative. The document emphasizes Spark's performance advantages, including faster data processing speeds and a more streamlined architecture for handling both batch and real-time data.
Similar to Build, Scale, and Deploy Deep Learning Pipelines Using Apache Spark (20)
Deep-Dive into Deep Learning Pipelines with Sue Ann Hong and Tim Hunter
Deep-Dive into Deep Learning Pipelines with Sue Ann Hong and Tim HunterDatabricks This document discusses the use of Apache Spark for building, scaling, and deploying deep learning pipelines, emphasizing the integration of various libraries such as TensorFlow, Keras, and others. It outlines the typical workflow of deep learning, covers the benefits of using Spark for distributed training and efficient data handling, and highlights features of Databricks' deep learning pipelines which simplify the process without sacrificing performance. Additionally, it addresses future developments and current practices in the industry for enhancing deep learning accessibility.
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, and Deep Learnin...
A Tale of Three Deep Learning Frameworks: TensorFlow, Keras, and Deep Learnin...Databricks The document discusses three deep learning frameworks: TensorFlow, Keras, and Deep Learning Pipelines, highlighting their features, benefits, and use cases. It emphasizes the importance of Apache Spark for big data processing and introduces concepts like transfer learning and model serving. The presentation includes a comparison of the frameworks along with the recommended scenarios for their usage.
Data Con LA 2018 - A Tale of DL Frameworks: TensorFlow, Keras, & Deep Learnin...
Data Con LA 2018 - A Tale of DL Frameworks: TensorFlow, Keras, & Deep Learnin...Data Con LA The document discusses the impacts of big data and offers an overview of three deep learning frameworks: TensorFlow, Keras, and Deep Learning Pipelines. It highlights the features, advantages, and use-cases for each framework while emphasizing the challenges in machine learning related to data handling and infrastructure. The presentation also covers concepts like transfer learning and provides resources for further exploration.
Combining Machine Learning frameworks with Apache Spark
Combining Machine Learning frameworks with Apache SparkDataWorks Summit/Hadoop Summit This document discusses combining machine learning frameworks like TensorFlow with Apache Spark. It describes how Spark can be used to schedule and distribute machine learning tasks across a cluster in order to speed up model training. Specific examples are provided of using TensorFlow for neural network training on image data and distributing those computations using Spark. The document also outlines Apache Spark MLlib and its DataFrame-based APIs for building machine learning pipelines that can be trained and deployed at scale.
Combining Machine Learning Frameworks with Apache Spark
Combining Machine Learning Frameworks with Apache SparkDatabricks This document discusses combining machine learning frameworks with Apache Spark. It provides an overview of Apache Spark and MLlib, describes how to distribute TensorFlow computations using Spark, and discusses managing machine learning workflows with Spark through features like cross validation, persistence, and distributed data sources. The goal is to make machine learning easy, scalable, and integrate with existing workflows.
Machine learning at scale - Webinar By zekeLabs
Machine learning at scale - Webinar By zekeLabszekeLabs Technologies Zekelabs offers a comprehensive training program on machine learning, covering topics from understanding the ecosystem and data challenges to deploying models at scale using technologies like Spark and Kafka. The curriculum emphasizes practical applications through user stories and aims to equip organizations and their employees with the skills needed to navigate the evolving IT landscape. Additionally, the document discusses the importance of choosing the right tools and infrastructure for effective machine learning implementation.
Emiliano Martinez | Deep learning in Spark Slides | Codemotion Madrid 2018
Emiliano Martinez | Deep learning in Spark Slides | Codemotion Madrid 2018Codemotion The document discusses the integration of deep learning with Apache Spark, highlighting its capabilities in processing large datasets and supporting scalable machine learning algorithms. It emphasizes tools like BigDL, DL4J, and MMLSpark for enhancing machine learning workflows on existing Spark clusters. Additionally, it covers the training, model evaluation, and inference processes involved in utilizing deep learning techniques on Spark environments.
Data Science and Deep Learning on Spark with 1/10th of the Code with Roope As...
Data Science and Deep Learning on Spark with 1/10th of the Code with Roope As...Databricks The document discusses the use of Microsoft Machine Learning Library for Apache Spark (MMLSpark) to enhance data science and deep learning capabilities, specifically in relation to snow leopard conservation efforts. It emphasizes the benefits of automated image classification and advanced machine learning algorithms to improve resource allocation and data accuracy for population monitoring. MMLSpark aims to streamline machine learning processes on large datasets, simplifying workflows and enhancing productivity for data scientists.
Deep Learning on Apache® Spark™: Workflows and Best Practices
Deep Learning on Apache® Spark™: Workflows and Best PracticesDatabricks The document discusses deep learning integration with Apache Spark, emphasizing workflows and best practices. It highlights the various deep learning frameworks supported by Spark, including TensorFlow and Keras, and provides insights into data pipeline patterns, monitoring, and the use of GPUs. The talk aims to guide users in optimizing their deep learning applications within the Spark ecosystem.
Deep Learning on Apache® Spark™: Workflows and Best Practices
Deep Learning on Apache® Spark™: Workflows and Best PracticesJen Aman This document summarizes a presentation about deep learning workflows and best practices on Apache Spark. It discusses how deep learning fits within broader data pipelines for tasks like training and transformation. It also outlines recurring patterns for integrating Spark and deep learning frameworks, including using Spark for data parallelism and embedding deep learning transforms. The presentation provides tips for developers on topics like using GPUs with PySpark and monitoring deep learning jobs. It concludes by discussing challenges in the areas of distributed deep learning and Spark integration.
Build a deep learning pipeline on apache spark for ads optimization
Build a deep learning pipeline on apache spark for ads optimizationCraig Chao This document discusses building deep learning pipelines on Apache Spark for ad optimization. It begins by discussing how data has become a new form of colonialism. It then explains why deep learning should be done on Apache Spark rather than just TensorFlow. The remainder of the document discusses machine learning pipelines on Apache Spark, how machine learning and deep learning can be used for ad optimization, and various approaches to deep learning on Apache Spark using tools like MMLSpark, Databricks, DL4J, BigDL, and SystemML.
Building Machine Learning Inference Pipelines at Scale (July 2019)
Building Machine Learning Inference Pipelines at Scale (July 2019)Julien SIMON The document outlines the process of building scalable machine learning inference pipelines using open source tools and AWS services. It discusses the challenges involved in machine learning applications, such as exporting models, decoupling infrastructure, and achieving low-latency predictions. Additionally, it highlights the capabilities of AWS SageMaker for training and deploying models efficiently.
Scalable Data Science in Python and R on Apache Spark
Scalable Data Science in Python and R on Apache Sparkfelixcss The document outlines the use of Apache Spark for scalable data science applications in Python and R, covering various aspects such as machine learning, data ingestion, and ETL processes. It highlights the architecture and tools available for integrating Spark with popular ML libraries and emphasizes the Spark ML pipeline for efficient model training and evaluation. Additionally, it discusses advanced functionalities such as vectorized user-defined functions (UDFs) and options for implementing deep learning with libraries like BigDL and TensorFrames.
Machine Learning Pipelines - Joseph Bradley - Databricks
Machine Learning Pipelines - Joseph Bradley - DatabricksSpark Summit This document discusses Spark MLlib's goal of making practical machine learning easy and scalable. It introduces key concepts in Spark ML Pipelines including DataFrames, transformers, estimators, and evaluators. DataFrames provide a structured dataset with named columns. Transformers, estimators, and evaluators are abstractions that define different steps in a machine learning workflow. Pipelines allow assembling these abstractions into reusable workflows. The document demonstrates an example text classification pipeline and discusses ongoing work to improve parameter tuning and feature engineering support.
Deeplearning for industries | Data to Production
Deeplearning for industries | Data to ProductionRahul Kumar The document outlines the journey of Rahul Kumar, an AI scientist, deep learning practitioner, and author, who shares insights into deep learning mechanics and end-to-end pipeline processes, including data sources, pre-processing, feature creation, model training, hyperparameter optimization, and model deployment. It emphasizes the importance of building connectors, understanding data distribution, and post-training optimizations. Furthermore, it provides resources for learning and coding related to deep learning mechanisms.
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...
Ml pipelines with Apache spark and Apache beam - Ottawa Reactive meetup Augus...Holden Karau The document presents insights from a talk on machine learning pipelines using Apache Spark and Apache Beam, focusing on their benefits for handling big data and model training. It introduces foundational concepts such as RDDs, dataframes, and pipelines, providing practical examples of using Spark for model training. Additionally, it covers potential challenges in model serving and validation, highlighting the importance of effective tools and practices within the machine learning workflow.
Notes on Deploying Machine-learning Models at Scale
Notes on Deploying Machine-learning Models at ScaleDeep Kayal The document discusses the complexities of deploying machine learning models at scale, emphasizing the importance of understanding hidden technical debt and the ongoing maintenance costs associated with real-world ML systems. It outlines a process for validating machine learning models through expert reviews and introduces a blueprint for deploying scalable APIs for both streaming and batch data processing. The author advocates for treating ML software as conventional software, highlighting the challenges and best practices for effective deployment and versioning.
Deep Learning with Python EMERSON EDUARDO RODRIGUES.pdf
Deep Learning with Python EMERSON EDUARDO RODRIGUES.pdfEMERSON EDUARDO RODRIGUES The document is the introduction and contents of the book 'Deep Learning with Python' by François Chollet, published by Manning Publications, aimed at making deep learning accessible to a broad audience. It covers fundamental concepts, practical applications, and advanced techniques in deep learning, with a focus on the Keras framework. The book is structured into parts that discuss the principles of deep learning, its practical applications, and best practices.
Bringing Deep Learning into production
Bringing Deep Learning into production Paolo Platter - The document discusses deep learning frameworks and how to choose one for a given environment. It summarizes the strengths, weaknesses, opportunities and threats of popular frameworks like TensorFlow, Theano, Torch, Caffe, DeepLearning4J and H2O.
- It recommends H2O as a good choice for enterprise environments due to its ease of use, scalability on big data, integration with Spark, Java/Scala support and commercial support. DeepLearning4J is also recommended for more advanced deep neural networks and multi-dimensional arrays.
- The document proposes using Spark as a middleware to leverage multiple frameworks and avoid vendor lock-in, and describes Agile Lab's recommended stack for enterprises which combines H
No BS Guide to Deep Learning in the Enterprise
No BS Guide to Deep Learning in the EnterpriseJesus Rodriguez The document provides a comprehensive overview of deep learning in the enterprise, highlighting its challenges, methodologies, and current frameworks. It discusses various deep learning stack categories, including tiered frameworks and cloud solutions, emphasizing the complexity of implementation and the required expertise. The summary underscores that despite the crowded market, successful deep learning applications should start small and iterate quickly.
Ad
More from Databricks (20)
DW Migration Webinar-March 2022.pptx
DW Migration Webinar-March 2022.pptxDatabricks The document discusses migrating a data warehouse to the Databricks Lakehouse Platform. It outlines why legacy data warehouses are struggling, how the Databricks Platform addresses these issues, and key considerations for modern analytics and data warehousing. The document then provides an overview of the migration methodology, approach, strategies, and key takeaways for moving to a lakehouse on Databricks.
Data Lakehouse Symposium | Day 1 | Part 1
Data Lakehouse Symposium | Day 1 | Part 1Databricks The document discusses the concept of a data lakehouse, highlighting the integration of structured, textual, and analog/IOT data. It emphasizes the importance of common identifiers and universal connectors for meaningful analytics across different data types, ultimately aiming to improve healthcare and manufacturing outcomes through effective data analysis. The presentation outlines the challenges of managing diverse data formats and the potential for data-driven insights to enhance quality of life.
Data Lakehouse Symposium | Day 1 | Part 2
Data Lakehouse Symposium | Day 1 | Part 2Databricks The document compares data lakehouses and data warehouses, outlining their similarities and differences. Both serve analytical processing and contain vetted, historical data, but the data lakehouse handles a much larger volume of machine-generated data and features fundamentally different structures from transaction-based data warehouses. Ultimately, they are presented as related yet distinct entities in the realm of data management.
Data Lakehouse Symposium | Day 2
Data Lakehouse Symposium | Day 2Databricks The Data Lakehouse Symposium held in February 2022 discussed the evolution of data management from data warehouses to lakehouses, emphasizing the integration of governance and metadata. It highlighted the challenges companies face in utilizing various types of data, particularly unstructured textual data, and the importance of adding context for effective analysis. The presentation also examined strategies for transforming unstructured data into structured formats to enable better decision-making and analytical processes.
Data Lakehouse Symposium | Day 4
Data Lakehouse Symposium | Day 4Databricks The document discusses the challenges of modern data, analytics, and AI workloads. Most enterprises struggle with siloed data systems that make integration and productivity difficult. The future of data lies with a data lakehouse platform that can unify data engineering, analytics, data warehousing, and machine learning workloads on a single open platform. The Databricks Lakehouse platform aims to address these challenges with its open data lake approach and capabilities for data engineering, SQL analytics, governance, and machine learning.
5 Critical Steps to Clean Your Data Swamp When Migrating Off of Hadoop
5 Critical Steps to Clean Your Data Swamp When Migrating Off of HadoopDatabricks The document outlines the challenges and considerations for migrating from Hadoop to Databricks, emphasizing the complexities of the Hadoop ecosystem and the advantages of a modern cloud-based data architecture. It provides a comprehensive migration plan that includes internal assessments, technical planning, and execution while addressing key topics such as data migration, security, and SQL integration. Specific tools and methodologies for effective transition and enhanced performance in data analytics are also discussed.
Democratizing Data Quality Through a Centralized Platform
Democratizing Data Quality Through a Centralized PlatformDatabricks Zillow's Data Governance Platform team addresses data quality challenges by creating a centralized platform that enhances visibility and standardizes data quality rules. The platform includes self-service capabilities and integrates with data lineage, allowing for built-in alerting and scalable onboarding. Key takeaways emphasize the importance of early alerting, collaboration, and the shared responsibility for maintaining high-quality data to improve decision-making.
Learn to Use Databricks for Data Science
Learn to Use Databricks for Data ScienceDatabricks The document outlines the challenges and workflows involved in data science, emphasizing the need for proper setup and resource management. It highlights the importance of sharing results with stakeholders and describes how Databricks' lakehouse platform simplifies these processes by integrating data sources and providing essential tools for data analysis. Overall, the goal is to help data scientists focus on their core analytical work rather than dealing with setup complexities.
Why APM Is Not the Same As ML Monitoring
Why APM Is Not the Same As ML MonitoringDatabricks The document discusses the distinctions between application performance monitoring (APM) and machine learning (ML) monitoring, emphasizing the unique challenges of ML monitoring, such as the need for intelligent detection and alerting. It outlines the essential components of ML monitoring, including statistical summarization, distribution comparison, and actionable alerts based on model performance. Additionally, it introduces Verta's end-to-end MLOps platform designed to meet the specialized needs of ML monitoring throughout the entire model lifecycle.
The Function, the Context, and the Data—Enabling ML Ops at Stitch Fix
The Function, the Context, and the Data—Enabling ML Ops at Stitch FixDatabricks Elijah Ben Izzy, a Data Platform Engineer at Stitch Fix, discusses building abstractions for machine learning operations to optimize workflows and enhance the separation of concerns between data science and platform engineering. The presentation highlights the importance of a custom-built model envelope for seamless integration and management of ML models, as well as advancements in deployment and inference processes. Future directions include enhanced production monitoring and sophisticated feature integration to further streamline data science workflows.
Stage Level Scheduling Improving Big Data and AI Integration
Stage Level Scheduling Improving Big Data and AI IntegrationDatabricks The document discusses stage-level scheduling and resource allocation in Apache Spark to enhance big data and AI integration. It outlines various resource requirements such as executors, memory, and accelerators, while presenting benefits like improved hardware utilization and simplified application pipelines. Additionally, it introduces the RAPIDS Accelerator for Spark and distributed deep learning with Horovod, emphasizing performance optimizations and future enhancements.
Simplify Data Conversion from Spark to TensorFlow and PyTorch
Simplify Data Conversion from Spark to TensorFlow and PyTorchDatabricks The document discusses the importance of data conversion between Spark and deep learning frameworks like TensorFlow and PyTorch. It highlights key pain points, such as challenges in migrating from single-node to distributed training and the complexity of saving and loading data. Additionally, it introduces the Spark Dataset Converter, which simplifies data handling while training deep learning models and offers best practices for efficient usage.
Scaling your Data Pipelines with Apache Spark on Kubernetes
Scaling your Data Pipelines with Apache Spark on KubernetesDatabricks This document discusses the integration of Apache Spark with Kubernetes on Google Cloud, highlighting its advantages for running data engineering and machine learning workloads within existing infrastructure. It outlines benefits such as improved cost optimization, faster scaling, and enhanced resource management through Google Kubernetes Engine (GKE) and Dataproc, while detailing implementation steps and monitoring options. Additionally, it covers the compatibility with big data ecosystem tools, job execution, and enterprise security features.
Scaling and Unifying SciKit Learn and Apache Spark Pipelines
Scaling and Unifying SciKit Learn and Apache Spark PipelinesDatabricks The document discusses the integration and scaling of AI/ML pipelines using Ray, aiming to unify Scikit-learn and Spark pipelines. Key features include Python functions as computation units, data exchange capabilities, and support for advanced execution strategies. It concludes with contact information for collaboration and emphasizes the importance of feedback from the community.
Sawtooth Windows for Feature Aggregations
Sawtooth Windows for Feature AggregationsDatabricks The document discusses the Sawtooth Windows Zipline, a feature engineering framework focusing on machine learning with structured data. It emphasizes the importance of real-time, stable, and consistent features for model training and serving, while highlighting the challenges of data sources and the intricacies of aggregations. Key topics include model complexity, data quality, and various types of windowed aggregations for efficient data processing.
Redis + Apache Spark = Swiss Army Knife Meets Kitchen Sink
Redis + Apache Spark = Swiss Army Knife Meets Kitchen SinkDatabricks The document discusses the integration of Redis with Apache Spark for managing long-running batch jobs and distributed counters. It outlines the challenges faced in submitting queries and the inefficiencies of existing solutions, proposing a system that utilizes Redis for queuing and job status communication. Key workflows and code views are provided to demonstrate the proposed solutions for efficient query handling and data processing.
Re-imagine Data Monitoring with whylogs and Spark
Re-imagine Data Monitoring with whylogs and SparkDatabricks The document discusses the challenges of monitoring machine learning data, emphasizing how traditional data analysis techniques fall short in addressing issues in ML data pipelines. It introduces the open-source library Whylogs for data logging, highlighting its lightweight profiling methods suitable for large datasets and integration with Apache Spark. Key topics include data quality problems, the need for scalable monitoring, and approaches for logging and analyzing ML data effectively.
Raven: End-to-end Optimization of ML Prediction Queries
Raven: End-to-end Optimization of ML Prediction QueriesDatabricks The document discusses Raven, an optimizer for machine learning prediction queries at Microsoft, focusing on its ability to improve the performance of SQL-based ML operations. It details how Raven integrates with Azure data engines, utilizing techniques like model projection pushdown and model-to-SQL translation to enhance query efficiency. Performance evaluations indicate that Raven significantly outperforms existing ML runtimes in various scenarios, achieving speed increases of up to 44 times compared to traditional approaches.
Processing Large Datasets for ADAS Applications using Apache Spark
Processing Large Datasets for ADAS Applications using Apache SparkDatabricks The document outlines the use of Spark for processing large datasets in automated driving applications, focusing on semantic segmentation and the challenges of moving from prototype to production. It presents the architecture of the system, covering ETL processes, model training, and inference, while addressing design considerations like scaling, security, and governance. Key takeaways emphasize the importance of leveraging cloud-based solutions and effective workflow management to enhance the development of perception software for autonomous vehicles.
Massive Data Processing in Adobe Using Delta Lake
Massive Data Processing in Adobe Using Delta LakeDatabricks The document discusses massive data processing at Adobe using Delta Lake, highlighting various aspects such as data representation, schema evolution, and challenges in data ingestion. It emphasizes the performance benefits of utilizing Delta Lake for handling large-scale data efficiently, while considering issues like schema management and replication lag. Key features like ACID transactions and lazy schema on-read approaches are also outlined to address the complexities of multi-tenant data architecture.
Ad
Recently uploaded (20)
Step by step guide to install Flutter and Dart
Step by step guide to install Flutter and DartS Pranav (Deepu) Flutter is basically Google’s portable user
interface (UI) toolkit, used to build and
develop eye-catching, natively-built
applications for mobile, desktop, and web,
from a single codebase. Flutter is free, open-
sourced, and compatible with existing code. It
is utilized by companies and developers
around the world, due to its user-friendly
interface and fairly simple, yet to-the-point
commands.
FME as an Orchestration Tool - Peak of Data & AI 2025
FME as an Orchestration Tool - Peak of Data & AI 2025Safe Software Processing huge amounts of data through FME can have performance consequences, but as an orchestration tool, FME is brilliant! We'll take a look at the principles of data gravity, best practices, pros, cons, tips and tricks. And of course all spiced up with relevant examples!
Application Modernization with Choreo - The AI-Native Internal Developer Plat...
Application Modernization with Choreo - The AI-Native Internal Developer Plat...WSO2 In this slide deck, we explore the challenges and best practices of application modernization. We also deep dive how an internal developer platform as a service like Choreo can fast track your modernization journey with AI capabilities and end-to-end workflow automation.
Software Testing & it’s types (DevOps)
Software Testing & it’s types (DevOps)S Pranav (Deepu) NTRODUCTION TO SOFTWARE TESTING
• Definition:
• Software testing is the process of evaluating and
verifying that a software application or system meets
specified requirements and functions correctly.
• Purpose:
• Identify defects and bugs in the software.
• Ensure the software meets quality standards.
• Validate that the software performs as intended in
various scenarios.
• Importance:
• Reduces risks associated with software failures.
• Improves user satisfaction and trust in the product.
• Enhances the overall reliability and performance of
the software
Migrating to Azure Cosmos DB the Right Way
Migrating to Azure Cosmos DB the Right WayAlexander (Alex) Komyagin In this session we cover the benefits of a migration to Cosmos DB, migration paths, common pain points and best practices. We share our firsthand experiences and customer stories. Adiom is the trusted partner for migration solutions that enable seamless online database migrations from MongoDB to Cosmos DB vCore, and DynamoDB to Cosmos DB for NoSQL.
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...
Async-ronizing Success at Wix - Patterns for Seamless Microservices - Devoxx ...Natan Silnitsky In a world where speed, resilience, and fault tolerance define success, Wix leverages Kafka to power asynchronous programming across 4,000 microservices. This talk explores four key patterns that boost developer velocity while solving common challenges with scalable, efficient, and reliable solutions:
1. Integration Events: Shift from synchronous calls to pre-fetching to reduce query latency and improve user experience.
2. Task Queue: Offload non-critical tasks like notifications to streamline request flows.
3. Task Scheduler: Enable precise, fault-tolerant delayed or recurring workflows with robust scheduling.
4. Iterator for Long-running Jobs: Process extensive workloads via chunked execution, optimizing scalability and resilience.
For each pattern, we’ll discuss benefits, challenges, and how we mitigate drawbacks to create practical solutions
This session offers actionable insights for developers and architects tackling distributed systems, helping refine microservices and adopting Kafka-driven async excellence.
Who will create the languages of the future?
Who will create the languages of the future?Jordi Cabot Will future languages be created by language engineers?
Can you "vibe" a DSL?
In this talk, we will explore the changing landscape of language engineering and discuss how Artificial Intelligence and low-code/no-code techniques can play a role in this future by helping in the definition, use, execution, and testing of new languages. Even empowering non-tech users to create their own language infrastructure. Maybe without them even realizing.
Transmission Media. (Computer Networks)
Transmission Media. (Computer Networks)S Pranav (Deepu) INTRODUCTION:TRANSMISSION MEDIA
• A transmission media in data communication is a physical path between the sender and
the receiver and it is the channel through which data can be sent from one location to
another. Data can be represented through signals by computers and other sorts of
telecommunication devices. These are transmitted from one device to another in the
form of electromagnetic signals. These Electromagnetic signals can move from one
sender to another receiver through a vacuum, air, or other transmission media.
Electromagnetic energy mainly includes radio waves, visible light, UV light, and gamma
ra
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...
Milwaukee Marketo User Group June 2025 - Optimize and Enhance Efficiency - Sm...BradBedford3 Inspired by the Adobe Summit hands-on lab, Optimize Your Marketo Instance Performance, review the recording from June 5th to learn best practices that can optimize your smart campaign and smart list processing time, inefficient practices to try to avoid, and tips and tricks for keeping your instance running smooth!
You will learn:
How smart campaign queueing works, how flow steps are prioritized, and configurations that slow down smart campaign processing.
Best practices for smart list and smart campaign configurations that yield greater reliability and processing efficiencies.
Generally recommended timelines for reviewing instance performance: walk away from this session with a guideline of what to review in Marketo and how often to review it.
This session will be helpful for any Marketo administrator looking for opportunities to improve and streamline their instance performance. Be sure to watch to learn best practices and connect with your local Marketo peers!
OpenTelemetry 101 Cloud Native Barcelona
OpenTelemetry 101 Cloud Native BarcelonaImma Valls Bernaus A brief introduction to OpenTelemetry, with a practical example of auto-instrumenting a Java web application with the Grafana stack (Loki, Grafana, Tempo, and Mimir).
About Certivo | Intelligent Compliance Solutions for Global Regulatory Needs
About Certivo | Intelligent Compliance Solutions for Global Regulatory Needscertivoai Certivo delivers intelligent compliance solutions designed to simplify and automate regulatory management for modern businesses in the USA, UK, and EU. Our AI-driven compliance platform helps enterprises navigate complex requirements with ease, offering real-time automated compliance monitoring and powerful product compliance software. At Certivo, we’re driven by a mission to transform how companies handle compliance, reducing risk and boosting operational efficiency. Discover our core values, vision, and innovation behind our trusted compliance management solutions. Whether you're in life sciences, automotive, or tech, Certivo helps you simplify regulatory compliance and scale faster with confidence.
How to Choose the Right Web Development Agency.pdf
How to Choose the Right Web Development Agency.pdfCreative Fosters Choosing the right web development agency is key to building a powerful online presence. This detailed guide from Creative Fosters helps you evaluate agencies based on experience, portfolio, technical skills, communication, and post-launch support. Whether you're launching a new site or revamping an old one, these expert tips will ensure you find a reliable team that understands your vision and delivers results. Avoid common mistakes and partner with an agency that aligns with your goals and budget.
Artificial Intelligence Workloads and Data Center Management
Artificial Intelligence Workloads and Data Center ManagementSandeepKS52 Data centers play a crucial role in the modern digital landscape, serving as the backbone for data storage, processing, and management. Understanding the structure and function of these facilities is essential, as they house the technology that supports various applications and services. The use of Kubernetes and container orchestration has transformed how software is deployed and managed, allowing for greater efficiency and scalability in handling applications. Additionally, the management of AI workloads presents unique challenges and opportunities, as organizations seek to optimize resources and performance for complex algorithms and data processing tasks. Together, these topics provide a comprehensive overview of the technologies and strategies that drive today’s information systems.
Integrating Survey123 and R&H Data Using FME
Integrating Survey123 and R&H Data Using FMESafe Software West Virginia Department of Transportation (WVDOT) actively engages in several field data collection initiatives using Collector and Survey 123. A critical component for effective asset management and enhanced analytical capabilities is the integration of Geographic Information System (GIS) data with Linear Referencing System (LRS) data. Currently, RouteID and Measures are not captured in Survey 123. However, we can bridge this gap through FME Flow automation. When a survey is submitted through Survey 123 for ArcGIS Portal (10.8.1), it triggers FME Flow automation. This process uses a customized workbench that interacts with a modified version of Esri's Geometry to Measure API. The result is a JSON response that includes RouteID and Measures, which are then applied to the feature service record.
Making significant Software Architecture decisions
Making significant Software Architecture decisionsBert Jan Schrijver Presented at the iSAQB Software Architecture Community NL meetup on 12-6-2025
Open Source Software Development Methods
Open Source Software Development MethodsVICTOR MAESTRE RAMIREZ Open Source Software Development Methods
Reimagining Software Development and DevOps with Agentic AI
Reimagining Software Development and DevOps with Agentic AIMaxim Salnikov Key announcements about Developer Productivity from Microsoft Build 2025
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?
On-Device AI: Is It Time to Go All-In, or Do We Still Need the Cloud?Hassan Abid As mobile hardware becomes more powerful, the promise of running advanced AI directly on-device is closer than ever. With Google’s latest on-device model Gemini Nano, accessible through the new ML Kit GenAI APIs and AI Edge SDK, alongside the open-source Gemma-3n models, developers can now integrate lightweight, multimodal intelligence that works even without an internet connection. But does this mean we no longer need cloud-based AI? This session explores the practical trade-offs between on-device and cloud AI for mobile apps.
Advanced Token Development - Decentralized Innovation
Advanced Token Development - Decentralized Innovationarohisinghas720 The world of blockchain is evolving at a fast pace, and at the heart of this transformation lies advanced token development. No longer limited to simple digital assets, today’s tokens are programmable, dynamic, and play a crucial role in driving decentralized applications across finance, governance, gaming, and beyond.
Build, Scale, and Deploy Deep Learning Pipelines Using Apache Spark
- 1. Build, Scale, and Deploy Deep Learning Pipelines Using Apache Spark Bay Area Spark Meetup Nov8, 2017 Sue Ann Hong, Databricks
- 2. This talk • Deep Learning at scale: current state • Deep Learning Pipelines: the philosophy • End-to-end workflow with DL Pipelines • Future
- 3. Deep Learning at Scale : current state 3put your #assignedhashtag here by setting the
- 4. What is Deep Learning? • A set of machine learning techniques that use layers that transform numerical inputs • Classification • Regression • Arbitrary mapping • Popular in the 80’s as Neural Networks • Recently came back thanks to advances in data collection, computation techniques, and hardware.
- 5. Success of Deep Learning Tremendous success for applications with complex data • AlphaGo • Image interpretation • Automatictranslation • Speech recognition
- 6. Deep Learning is often challenging Labeled Data Compute Resources & Time Engineer hours • Tedious or difficult to distribute computations • No exact science around deep learning à lots of tweaking • Low level APIs with steep learning curve • Complex models à need a lot of data
- 7. Deep Learning in industry • Currently limited adoption • Huge potential beyond the industrial giants • How do we accelerate the road to massive availability?
- 9. Deep Learning Pipelines: Deep Learning with Simplicity • Open-source Databricks library • Focuses on easeof useand integration • without sacrificing performance
- 10. Deep Learning is often challenging Compute Resources & Time Engineer hours Labeled Data • Tedious or difficult to distribute computations • No exact science around deep learning à lots of tweaking • Low level APIs with steep learning curve • Complex models à need a lot of data
- 11. Deep Learning is often challenging • Tedious or difficult to distribute computations • No exact science around deep learning à lots of tweaking • Low level APIs with steep learning curve • Complex models à need a lot of data
- 12. • Tedious or difficult to distribute computations • No exact science around deep learning à lots of tweaking • Low level APIs with steep learning curve • Complex models à need a lot of data Instead • Be easy to scale • Require little tweaking • Be easy to write • Require little or no data Common workflows should
- 13. How • Be easy to scale • Require little tweaking • Be easy to write • Require little or no data Common workflows should • Use Apache Spark for scaling out common tasks • Leverage well-knownmodel architectures • Integrate with MLlib Pipelines API to capture ML workflowconcisely • Leverage pre-trained models for common tasks Apache Spark for scaling out MLlib Pipelines API pre-trained models
- 14. Demo: Build a visual recommendation AI 10minutes 7lines of code Elastic Scale-out using Apache Spark MLlib Pipelines API Leverage pre-trained models 0labels
- 16. To the notebook!
- 18. End-to-EndWorkflow with Deep Learning Pipelines 18put your #assignedhashtag here by setting the
- 19. A typical Deep Learning workflow • Load data (images, text, time series, …) • Interactive work • Train • Select an architecture for a neural network • Optimize the weights of the NN • Evaluateresults, potentially re-train • Apply: • Pass the data through the NN to produce new features or output Load data Interactive work Train Evaluate Apply
- 20. A typical Deep Learning workflow Load data Interactive work Train Evaluate Apply • Image loading in Spark • Distributed batch prediction • Deploying models in SQL • Transfer learning • Distributed tuning • Pre-trained models
- 21. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply
- 22. Adds support for images in Spark • ImageSchema, reader, conversion functions to/from numpy arrays • Most of the tools we’ll describe work on ImageSchema columns from sparkdl import readImages image_df = readImages(sample_img_dir)
- 23. Upcoming: built-in support in Spark • Spark-21866 • Contributing image format & reading to Spark • Targeted for Spark 2.3 • Joint work with Microsoft 23put your #assignedhashtag here by setting the
- 24. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply
- 25. Applying popular models • Popular pre-trained models accessible through MLlib Transformers predictor = DeepImagePredictor(inputCol="image", outputCol="predicted_labels", modelName="InceptionV3") predictions_df = predictor.transform(image_df)
- 26. Applying popular models predictor = DeepImagePredictor(inputCol="image", outputCol="predicted_labels", modelName="InceptionV3") predictions_df = predictor.transform(image_df)
- 27. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply Hyperparameter tuning Transfer learning
- 28. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply Hyperparameter tuning Transfer learning
- 29. Transfer learning • Pre-trained models may not be directly applicable • New domain, e.g. shoes • Training from scratch requires • Enormous amounts of data • A lot of compute resources & time • Idea: intermediate representations learned for one task may be useful for other related tasks
- 30. Transfer Learning SoftMax GIANT PANDA 0.9 RACCOON 0.05 RED PANDA 0.01 …
- 34. Transfer Learning as a Pipeline DeepImageFeaturizer Image Loading Preprocessing Logistic Regression MLlib Pipeline
- 35. Transfer Learning as a Pipeline 35put your #assignedhashtag here by setting the featurizer = DeepImageFeaturizer(inputCol="image", outputCol="features", modelName="InceptionV3") lr = LogisticRegression(labelCol="label") p = Pipeline(stages=[featurizer, lr]) p_model = p.fit(train_df)
- 36. Transfer Learning • Usually for classification tasks • Similar task, new domain • But other forms of learning leveraging learned representations can be loosely considered transfer learning
- 38. Featurization for similarity-based ML DeepImageFeaturizer Image Loading Preprocessing Logistic Regression
- 39. Featurization for similarity-based ML DeepImageFeaturizer Image Loading Preprocessing Clustering KMeans GaussianMixture Nearest Neighbor KNN LSH Distance computation
- 40. Featurization for similarity-based ML DeepImageFeaturizer Image Loading Preprocessing Clustering KMeans GaussianMixture Nearest Neighbor KNN LSH Distance computation Duplicate Detection Recommendation Anomaly Detection Search result diversification
- 41. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply Hyperparameter tuning Transfer learning
- 42. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply
- 43. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply Spark SQL Batch prediction s
- 44. Deep Learning Pipelines • Load data • Interactive work • Train • Evaluate model • Apply Spark SQL Batch prediction
- 45. Batch prediction as an MLlib Transformer • A model is a Transformer in MLlib • DataFrame goes in, DataFrame comes out with output columns predictor = XXTransformer(inputCol="image", outputCol="prediction", modelSpecification={…}) predictions = predictor.transform(test_df)
- 46. Hierarchy of DL transformers for images 46 TFImageTransformer KerasImageTransformer DeepImageFeaturizerDeepImagePredictor NamedImageTransformer(model_name) (keras.Model) (tf.Graph) Input (Image) Output (vector) model_namemodel_name Keras.Model tf.Graph
- 47. Hierarchy of DL transformers for images 47 TFImageTransformer KerasImageTransformer DeepImageFeaturizerDeepImagePredictor NamedImageTransformer(model_name) (keras.Model) (tf.Graph) Input (Image) Output (vector) model_namemodel_name Keras.Model tf.Graph
- 48. Hierarchy of DL transformers 48 TFImageTransformer KerasImageTransformer DeepImageF eaturizer DeepImageP redictor NamedImageTransformer ) Input (Image ) Output (vector ) model_namemodel_name Keras.Model tf.Graph TFTextTransformer KerasTextTransformer DeepTextFeat urizer DeepTextPre dictor NamedTextTransformerInput (Text) Output (vector) model_namemodel_name Keras.Model tf.Graph TFTransformer
- 49. Hierarchy of DL transformers 49 TFImageTransformer KerasImageTransformer DeepImageFeaturizerDeepImagePredictor NamedImageTransformer TFTextTransformer KerasTextTransformer DeepTextFeaturizerDeepTextPredictor NamedTextTransformer TFTransformer
- 50. Hierarchy of DL transformers 50 TFImageTransformer KerasImageTransformer DeepImageFeaturizerDeepImagePredictor NamedImageTransformer TFTextTransformer KerasTextTransformer DeepTextFeaturizerDeepTextPredictor NamedTextTransformer TFTransformer
- 51. CommonTasks Easy Advanced Ones Possible 51 TFImageTransformer KerasImageTransformer DeepImageFeaturizerDeepImagePredictor NamedImageTransformer TFTextTransformer KerasTextTransformer DeepTextFeaturizerDeepTextPredictor NamedTextTransformer TFTransformer
- 52. CommonTasks Easy Advanced Ones Possible 52 TFImageTransformer KerasImageTransformer DeepImageFeaturizer DeepImagePredictor NamedImageTransformer TFTextTransformer KerasTextTransformer DeepTextFeaturizer DeepTextPredictor NamedTextTransformer TFTransformer Pre-built Solutions 80%-built Solutions Self-built Solutions
- 53. Deep Learning Pipelines : Future In progress • Text featurization (embeddings) • TFTransformer for arbitrary vectors Future directions • Non-image data domains: video, text, speech, … • Distributed training • Support for more backends, e.g. MXNet, PyTorch, BigDL
- 55. Resources DL Pipelines GitHub Repo, Spark Summit Europe 2017 Deep Dive Blog posts & webinars (http://databricks.com/blog) • Deep Learning Pipelines • GPU acceleration in Databricks • BigDL on Databricks • Deep Learning and Apache Spark Docs for Deep Learningon Databricks (http://docs.databricks.com) • Getting started • Deep Learning Pipelines Example • Spark integration