Elasticsearch And Apache Lucene For Apache Spark And MLlib
7 likes3,512 views
This document summarizes a presentation about using Elasticsearch and Lucene for text processing and machine learning pipelines in Apache Spark. Some key points: - Elasticsearch provides text analysis capabilities through Lucene and can be used to clean, tokenize, and vectorize text for machine learning tasks. - Elasticsearch integrates natively with Spark through Java/Scala APIs and allows indexing and querying data from Spark. - A typical machine learning pipeline for text classification in Spark involves tokenization, feature extraction (e.g. hashing), and a classifier like logistic regression. - The presentation proposes preparing text analysis specifications in Elasticsearch once and reusing them across multiple Spark pipelines to simplify the workflows and avoid data movement between systems
1 of 58
Downloaded 252 times
![Elasticsearch as a DataFrame
val df = sql.read.format(“es").load("buckethead/albums")
df.filter(df("category").equalTo("pikes").and(df("year").geq(2015)))
{ "query" :
{ "bool" : { "must" : [
"match" : { "category" : "pikes" }
],
"filter" : [
{ "range" : { "year" : {"gte" : "2015" }}}
]
}}}](https://image.slidesharecdn.com/3icostinleau-160614190230/85/Elasticsearch-And-Apache-Lucene-For-Apache-Spark-And-MLlib-38-320.jpg)
![Work once – reuse multiple times
// prepare the spec for vectorize – fast and lightweight
val spec = s"""{ "features" : [{
| "field": "text",
| "type" : "string",
| "tokens" : "all_terms",
| "number" : "occurrence",
| "min_doc_freq" : 2000
| }],
| "sparse" : "true"}""".stripMargin
ML.prepareSpec(spec, “my-spec”)](https://image.slidesharecdn.com/3icostinleau-160614190230/85/Elasticsearch-And-Apache-Lucene-For-Apache-Spark-And-MLlib-51-320.jpg)
![Need to adjust the model? Change the spec
val spec = s"""{ "features" : [{
| "field": "text",
| "type" : "string",
| "tokens" : "given",
| "number" : "tf",
| "terms": ["term1", "term2", ...]
| }],
| "sparse" : "true"}""".stripMargin
ML.prepareSpec(spec)](https://image.slidesharecdn.com/3icostinleau-160614190230/85/Elasticsearch-And-Apache-Lucene-For-Apache-Spark-And-MLlib-55-320.jpg)
Ad
Recommended
Writing Continuous Applications with Structured Streaming Python APIs in Apac...
Writing Continuous Applications with Structured Streaming Python APIs in Apac...Databricks The document discusses the use of Apache Spark for building continuous streaming applications using structured streaming in PySpark, highlighting its benefits, challenges, and integration capabilities. It explains how Spark unifies various data processing needs and provides a detailed overview of constructing a streaming application with example code snippets. The session also covers topics such as data ingestion from sources like Kafka, processing data transformations, and writing results to output sinks while ensuring fault tolerance and efficient querying.
Apache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and SmarterDatabricks Apache Spark 2.0 enhances performance and usability with key features such as Project Tungsten for speed improvements of 5-20x, a unified API integrating datasets, dataframes, and SQL, and structured streaming for real-time analytics. The update focuses on maintaining backward compatibility while introducing richer semantics and optimizations. Users can explore Spark 2.0 through the Databricks Community Edition, with extensive improvements set to address around 2000 issues.
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...
Spark DataFrames: Simple and Fast Analytics on Structured Data at Spark Summi...Databricks The document presents insights on Spark SQL and DataFrames, highlighting its capabilities in processing structured data efficiently and effectively. It details the evolution of Spark SQL since its inception in April 2014, emphasizing features like multi-version support, various bindings, and a unified interface for reading and writing data in multiple formats. Additionally, it explores the optimization of data processing pipelines, integration with BI tools, and high-level operations for analytics while showcasing performance improvements with DataFrames compared to traditional RDDs.
LuceneRDD for (Geospatial) Search and Entity Linkage
LuceneRDD for (Geospatial) Search and Entity Linkagezouzias The document presents LuceneRDD, an open-source project that integrates Apache Spark with Apache Lucene for full-text and spatial search, enabling scalable entity linkage without external clusters. It discusses its motivation, functionality, and examples of implementation for processing large datasets efficiently. Future work includes improving performance tests and expanding use cases for the technology.
Robust and Scalable ETL over Cloud Storage with Apache Spark
Robust and Scalable ETL over Cloud Storage with Apache SparkDatabricks The document discusses the implementation of ETL (Extract, Transform, Load) processes on cloud storage using Databricks and highlights the challenges of reliability and performance with cloud-native storage compared to HDFS. It examines the trade-offs regarding costs, availability, durability, and metadata performance, proposing solutions such as the Databricks commit service to ensure atomic commits and manage eventual consistency. Overall, the document advocates for leveraging cloud storage while addressing its limitations to achieve efficient and reliable ETL operations.
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...Databricks This document summarizes key aspects of structuring computation and data in Apache Spark using SQL, DataFrames, and Datasets. It discusses how structuring computation and data through these APIs enables optimizations like predicate pushdown and efficient joins. It also describes how data is encoded efficiently in Spark's internal format and how encoders translate between domain objects and Spark's internal representations. Finally, it introduces structured streaming as a high-level streaming API built on top of Spark SQL that allows running the same queries continuously on streaming data.
Combining Machine Learning Frameworks with Apache Spark
Combining Machine Learning Frameworks with Apache SparkDatabricks This document discusses combining machine learning frameworks with Apache Spark. It provides an overview of Apache Spark and MLlib, describes how to distribute TensorFlow computations using Spark, and discusses managing machine learning workflows with Spark through features like cross validation, persistence, and distributed data sources. The goal is to make machine learning easy, scalable, and integrate with existing workflows.
Deep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDatabricks The document discusses memory management challenges in Apache Spark, focusing on efficient use of execution and storage memory. It outlines strategies for handling memory contention across various scenarios, including static and dynamic memory assignment for tasks and operators. Additionally, it introduces features from Spark 1.6 onwards, emphasizing unified memory management and the benefits of tungsten's optimized data representation.
Introduction to Spark with Scala
Introduction to Spark with ScalaHimanshu Gupta The document is a presentation on Apache Spark using Scala, outlining its significance as a fast and general engine for large-scale data processing. It introduces core concepts such as RDDs (Resilient Distributed Datasets) and Spark Streaming, providing installation steps and examples of code usage. The document also highlights Spark's evolution and its popularity in the field of data analytics.
Large-Scale Data Science in Apache Spark 2.0
Large-Scale Data Science in Apache Spark 2.0Databricks This document discusses the enhancements in Apache Spark 2.0, focusing on its scalability for large-scale data science and AI through improved hardware and user scalability. It emphasizes the use of structured APIs for efficient data manipulation and introduces new features for deep learning and integration with existing Python and R libraries. The content highlights Spark's capabilities in parallelizing computations and the ease of building complex data science models with high-level APIs.
Scalable Data Science with SparkR: Spark Summit East talk by Felix Cheung
Scalable Data Science with SparkR: Spark Summit East talk by Felix CheungSpark Summit The document provides an overview of SparkR, an R programming interface for Apache Spark, detailing its features, architectural components, and machine learning capabilities. It discusses the integration of R with Spark's distributed computing framework, focusing on user-defined functions (UDFs), the machine learning pipeline, and various model types available in SparkR. It also outlines challenges faced by SparkR users and upcoming improvements planned for future versions.
Spark Summit EU talk by Nimbus Goehausen
Spark Summit EU talk by Nimbus GoehausenSpark Summit This document discusses automatic checkpointing in Spark. It describes the current Spark development cycle and issues with it. It then presents an approach called Distributed Collections (DC) that allows defining data pipelines in a lazy manner and generating logical signatures to enable automatic checkpointing without rerunning previous stages. This is achieved by introducing the concepts of Deferred Results (DR) and following dependencies recursively to hash functions and classes. DC builds upon but does not modify Spark, aiming to improve the developer workflow while avoiding complexity. Sample code is provided and the approach is available as an open source project.
Introduction to Apache Spark
Introduction to Apache SparkMohamed hedi Abidi The document is an introduction to Apache Spark, covering its installation, core components like SparkContext, Resilient Distributed Datasets (RDDs), and different applications such as Spark SQL, GraphX, and MLlib. It highlights Spark's speed and flexibility for handling batch, interactive, and real-time data with a high-level abstraction for easier development. Additionally, it discusses Spark's persistent caching capabilities, cluster manager types, and provides examples of simple Spark applications.
Apache Spark Tutorial
Apache Spark TutorialAhmet Bulut The document is a tutorial on scalable data analytics using Apache Spark, highlighting its capability as a cluster computing platform that supports real-time data processing and SQL queries. It covers the architecture of Spark, including its core components like the Spark Core, Spark SQL, and Machine Learning libraries, and provides insights into the roles and workflows of data scientists. Various practical instructions are provided for downloading Spark, initializing applications, and performing data operations using Resilient Distributed Datasets (RDDs) and pair RDDs.
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Sparkfelixcss The document discusses using Spark for interactive analysis and visualization, detailing setup procedures, tools, and their functionalities such as PySpark, Jupyter, and Apache Zeppelin. It covers topics like k-means clustering, anomaly detection, and the streaming k-means algorithm, along with visualization techniques using libraries like Matplotlib and Lightning. Additionally, it highlights challenges related to package management and offers resources for implementation and further reading.
Frustration-Reduced PySpark: Data engineering with DataFrames
Frustration-Reduced PySpark: Data engineering with DataFramesIlya Ganelin The document discusses utilizing Spark and DataFrames for efficient data engineering, emphasizing ease of use in batch ETL processes while highlighting potential pitfalls such as scaling issues and the need for proper configuration. It presents detailed instructions on DataFrame creation, column and row manipulation, and error handling, alongside recommendations for model building and feature extraction. The document also addresses Spark's architecture, performance tuning, and provides links to additional resources for further learning.
Spark stream - Kafka
Spark stream - Kafka Dori Waldman Spark Streaming can be used to process streaming data from Kafka in real-time. There are two main approaches - the receiver-based approach where Spark receives data from Kafka receivers, and the direct approach where Spark directly reads data from Kafka. The document discusses using Spark Streaming to process tens of millions of transactions per minute from Kafka for an ad exchange system. It describes architectures where Spark Streaming is used to perform real-time aggregations and update databases, as well as save raw data to object storage for analytics and recovery. Stateful processing with mapWithState transformations is also demonstrated to update Cassandra in real-time.
Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...
Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...Spark Summit The document discusses the monitoring of resource usage for Scala and Python Spark jobs in a Yarn environment, highlighting the importance of a robust execution framework capable of testing at scale due to various challenges like performance issues and memory requirements. It describes a centralized monitoring and reporting system, alongside case studies demonstrating the resolution of OutOfMemory (OOM) exceptions related to the interaction between Python and Java through Py4j. Recommendations include understanding Spark's architecture and computing demands, best practices for coding, and the necessity of investing in monitoring tools.
Spark on YARN
Spark on YARNAdarsh Pannu Spark supports four cluster managers: Local, Standalone, YARN, and Mesos. YARN is highly recommended for production use. When running Spark on YARN, careful tuning of configuration settings like the number of executors, executor memory and cores, and dynamic allocation is important to optimize performance and resource utilization. Configuring queues also allows separating different applications by priority and resource needs.
Keeping Spark on Track: Productionizing Spark for ETL
Keeping Spark on Track: Productionizing Spark for ETLDatabricks The document discusses best practices for productionizing Apache Spark for ETL processes, focusing on metadata management, performance optimization, and schema handling. Key points include transforming raw files into efficient binary formats, strategies for error handling, and optimizing data partitioning for better efficiency. Examples of JSON schema handling and optimization techniques for Spark SQL joins are also provided.
Tuning and Monitoring Deep Learning on Apache Spark
Tuning and Monitoring Deep Learning on Apache SparkDatabricks The document discusses the integration of deep learning with Apache Spark, highlighting various frameworks and methods for tuning, monitoring, and utilizing GPU instances in Spark. It emphasizes the ongoing challenges in achieving a cohesive deep learning framework within Spark and presents strategies for effective data management and job scheduling. The talk also addresses the importance of monitoring for deep learning tasks and the specific requirements for successful deployment in distributed environments.
A Journey into Databricks' Pipelines: Journey and Lessons Learned
A Journey into Databricks' Pipelines: Journey and Lessons LearnedDatabricks The document discusses Databricks' development of a next-generation data pipeline utilizing Apache Spark, highlighting challenges like fault tolerance and scalability. It outlines the architecture of their data pipeline, including real-time and batch processing capabilities, and shares lessons learned regarding efficiency and cost management. The conclusion emphasizes the benefits of Databricks and Apache Spark as a unified platform for ETL, data warehousing, and analytics.
Beyond shuffling global big data tech conference 2015 sj
Beyond shuffling global big data tech conference 2015 sjHolden Karau This document provides tips and tricks for scaling Apache Spark jobs. It discusses techniques for reusing RDDs through caching and checkpointing. It explains best practices for working with key-value data, including how to avoid problems from key skew with groupByKey. The document also covers using Spark accumulators for validation and when Spark SQL can improve performance. Additional resources on Spark are provided at the end.
Easy, scalable, fault tolerant stream processing with structured streaming - ...
Easy, scalable, fault tolerant stream processing with structured streaming - ...Databricks The document discusses structured streaming with Apache Spark, focusing on its ease of use, scalability, and fault tolerance for stream processing applications. It highlights key features such as data integration from various sources, checkpointing for fault tolerance, and advanced transformations to handle complex workloads and data types. The presentation also covers practical examples of implementing streaming queries, event-time aggregations, and stateful processing with benefits for real-time analytics and decision-making.
Extending Spark With Java Agent (handout)
Extending Spark With Java Agent (handout)Jaroslav Bachorik This document discusses using Java agents to extend Spark functionality. Java agents allow instrumentation of JVM applications like Spark at runtime. Specifically, agents could be used to monitor block caching in Spark and make recommendations to improve caching decisions. The document provides examples of how to transform Spark classes at runtime using agents to track block usage and cache hits. Overall, agents provide a powerful way to enhance Spark without requiring code changes.
Apache Spark Introduction - CloudxLab
Apache Spark Introduction - CloudxLabAbhinav Singh This document outlines a hands-on session on big data processing using Apache Spark, covering topics such as RDD operations, transformations, and actions. Participants will learn to compute the word frequency of a text file stored in HDFS, using a cloud-based lab environment offered by CloudxLab. The session also provides a brief introduction to CloudxLab's features and the instructor's background in big data technologies.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit The document outlines a presentation on using Redis for real-time machine learning, focusing on Redis-ML integrated with Apache Spark. It covers the capabilities of Redis as a key-value store, the functionalities of Redis-ML for model serving, and a case study involving a recommendation system. The concluding remarks emphasize the advantages of combining Spark for training with Redis for efficient model serving, highlighting reduced resource costs and simplified machine learning lifecycle.
Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...
Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...Spark Summit This document describes Opaque, a secure distributed data analytics framework that allows complex analytics to run on sensitive data while preserving data privacy and functionality. Opaque utilizes hardware enclaves to protect computation and data within enclaves from a malicious operating system or cloud provider. It implements various oblivious primitives and oblivious operators to execute Spark SQL queries without leaking data access patterns. This allows sensitive data to be analyzed using existing Spark SQL queries while preventing privacy leaks from memory, network or computation access patterns.
Writing Continuous Applications with Structured Streaming PySpark API
Writing Continuous Applications with Structured Streaming PySpark APIDatabricks The document discusses the development of continuous applications using Structured Streaming in Apache Spark, highlighting its benefits, complexities, and capabilities in handling diverse data. It explains the architecture of Structured Streaming, including the process of reading, transforming, and writing data streams, as well as the importance of features like checkpointing for fault tolerance. The presentation includes tutorials and emphasizes the unification of batch and streaming processing to enable real-time data handling and analytics.
Intro to Spark and Spark SQL
Intro to Spark and Spark SQLjeykottalam Apache Spark is a fast and general cluster computing system that improves efficiency through in-memory computing and usability through rich APIs. Spark SQL provides a way to work with structured data and transform RDDs using SQL. It can read data from sources like Parquet and JSON files, Hive, and write query results to Parquet for efficient querying. Spark SQL also allows machine learning pipelines to be built by connecting SQL queries to MLlib algorithms.
More Related Content
What's hot (20)
Introduction to Spark with Scala
Introduction to Spark with ScalaHimanshu Gupta The document is a presentation on Apache Spark using Scala, outlining its significance as a fast and general engine for large-scale data processing. It introduces core concepts such as RDDs (Resilient Distributed Datasets) and Spark Streaming, providing installation steps and examples of code usage. The document also highlights Spark's evolution and its popularity in the field of data analytics.
Large-Scale Data Science in Apache Spark 2.0
Large-Scale Data Science in Apache Spark 2.0Databricks This document discusses the enhancements in Apache Spark 2.0, focusing on its scalability for large-scale data science and AI through improved hardware and user scalability. It emphasizes the use of structured APIs for efficient data manipulation and introduces new features for deep learning and integration with existing Python and R libraries. The content highlights Spark's capabilities in parallelizing computations and the ease of building complex data science models with high-level APIs.
Scalable Data Science with SparkR: Spark Summit East talk by Felix Cheung
Scalable Data Science with SparkR: Spark Summit East talk by Felix CheungSpark Summit The document provides an overview of SparkR, an R programming interface for Apache Spark, detailing its features, architectural components, and machine learning capabilities. It discusses the integration of R with Spark's distributed computing framework, focusing on user-defined functions (UDFs), the machine learning pipeline, and various model types available in SparkR. It also outlines challenges faced by SparkR users and upcoming improvements planned for future versions.
Spark Summit EU talk by Nimbus Goehausen
Spark Summit EU talk by Nimbus GoehausenSpark Summit This document discusses automatic checkpointing in Spark. It describes the current Spark development cycle and issues with it. It then presents an approach called Distributed Collections (DC) that allows defining data pipelines in a lazy manner and generating logical signatures to enable automatic checkpointing without rerunning previous stages. This is achieved by introducing the concepts of Deferred Results (DR) and following dependencies recursively to hash functions and classes. DC builds upon but does not modify Spark, aiming to improve the developer workflow while avoiding complexity. Sample code is provided and the approach is available as an open source project.
Introduction to Apache Spark
Introduction to Apache SparkMohamed hedi Abidi The document is an introduction to Apache Spark, covering its installation, core components like SparkContext, Resilient Distributed Datasets (RDDs), and different applications such as Spark SQL, GraphX, and MLlib. It highlights Spark's speed and flexibility for handling batch, interactive, and real-time data with a high-level abstraction for easier development. Additionally, it discusses Spark's persistent caching capabilities, cluster manager types, and provides examples of simple Spark applications.
Apache Spark Tutorial
Apache Spark TutorialAhmet Bulut The document is a tutorial on scalable data analytics using Apache Spark, highlighting its capability as a cluster computing platform that supports real-time data processing and SQL queries. It covers the architecture of Spark, including its core components like the Spark Core, Spark SQL, and Machine Learning libraries, and provides insights into the roles and workflows of data scientists. Various practical instructions are provided for downloading Spark, initializing applications, and performing data operations using Resilient Distributed Datasets (RDDs) and pair RDDs.
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Sparkfelixcss The document discusses using Spark for interactive analysis and visualization, detailing setup procedures, tools, and their functionalities such as PySpark, Jupyter, and Apache Zeppelin. It covers topics like k-means clustering, anomaly detection, and the streaming k-means algorithm, along with visualization techniques using libraries like Matplotlib and Lightning. Additionally, it highlights challenges related to package management and offers resources for implementation and further reading.
Frustration-Reduced PySpark: Data engineering with DataFrames
Frustration-Reduced PySpark: Data engineering with DataFramesIlya Ganelin The document discusses utilizing Spark and DataFrames for efficient data engineering, emphasizing ease of use in batch ETL processes while highlighting potential pitfalls such as scaling issues and the need for proper configuration. It presents detailed instructions on DataFrame creation, column and row manipulation, and error handling, alongside recommendations for model building and feature extraction. The document also addresses Spark's architecture, performance tuning, and provides links to additional resources for further learning.
Spark stream - Kafka
Spark stream - Kafka Dori Waldman Spark Streaming can be used to process streaming data from Kafka in real-time. There are two main approaches - the receiver-based approach where Spark receives data from Kafka receivers, and the direct approach where Spark directly reads data from Kafka. The document discusses using Spark Streaming to process tens of millions of transactions per minute from Kafka for an ad exchange system. It describes architectures where Spark Streaming is used to perform real-time aggregations and update databases, as well as save raw data to object storage for analytics and recovery. Stateful processing with mapWithState transformations is also demonstrated to update Cassandra in real-time.
Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...
Monitoring the Dynamic Resource Usage of Scala and Python Spark Jobs in Yarn:...Spark Summit The document discusses the monitoring of resource usage for Scala and Python Spark jobs in a Yarn environment, highlighting the importance of a robust execution framework capable of testing at scale due to various challenges like performance issues and memory requirements. It describes a centralized monitoring and reporting system, alongside case studies demonstrating the resolution of OutOfMemory (OOM) exceptions related to the interaction between Python and Java through Py4j. Recommendations include understanding Spark's architecture and computing demands, best practices for coding, and the necessity of investing in monitoring tools.
Spark on YARN
Spark on YARNAdarsh Pannu Spark supports four cluster managers: Local, Standalone, YARN, and Mesos. YARN is highly recommended for production use. When running Spark on YARN, careful tuning of configuration settings like the number of executors, executor memory and cores, and dynamic allocation is important to optimize performance and resource utilization. Configuring queues also allows separating different applications by priority and resource needs.
Keeping Spark on Track: Productionizing Spark for ETL
Keeping Spark on Track: Productionizing Spark for ETLDatabricks The document discusses best practices for productionizing Apache Spark for ETL processes, focusing on metadata management, performance optimization, and schema handling. Key points include transforming raw files into efficient binary formats, strategies for error handling, and optimizing data partitioning for better efficiency. Examples of JSON schema handling and optimization techniques for Spark SQL joins are also provided.
Tuning and Monitoring Deep Learning on Apache Spark
Tuning and Monitoring Deep Learning on Apache SparkDatabricks The document discusses the integration of deep learning with Apache Spark, highlighting various frameworks and methods for tuning, monitoring, and utilizing GPU instances in Spark. It emphasizes the ongoing challenges in achieving a cohesive deep learning framework within Spark and presents strategies for effective data management and job scheduling. The talk also addresses the importance of monitoring for deep learning tasks and the specific requirements for successful deployment in distributed environments.
A Journey into Databricks' Pipelines: Journey and Lessons Learned
A Journey into Databricks' Pipelines: Journey and Lessons LearnedDatabricks The document discusses Databricks' development of a next-generation data pipeline utilizing Apache Spark, highlighting challenges like fault tolerance and scalability. It outlines the architecture of their data pipeline, including real-time and batch processing capabilities, and shares lessons learned regarding efficiency and cost management. The conclusion emphasizes the benefits of Databricks and Apache Spark as a unified platform for ETL, data warehousing, and analytics.
Beyond shuffling global big data tech conference 2015 sj
Beyond shuffling global big data tech conference 2015 sjHolden Karau This document provides tips and tricks for scaling Apache Spark jobs. It discusses techniques for reusing RDDs through caching and checkpointing. It explains best practices for working with key-value data, including how to avoid problems from key skew with groupByKey. The document also covers using Spark accumulators for validation and when Spark SQL can improve performance. Additional resources on Spark are provided at the end.
Easy, scalable, fault tolerant stream processing with structured streaming - ...
Easy, scalable, fault tolerant stream processing with structured streaming - ...Databricks The document discusses structured streaming with Apache Spark, focusing on its ease of use, scalability, and fault tolerance for stream processing applications. It highlights key features such as data integration from various sources, checkpointing for fault tolerance, and advanced transformations to handle complex workloads and data types. The presentation also covers practical examples of implementing streaming queries, event-time aggregations, and stateful processing with benefits for real-time analytics and decision-making.
Extending Spark With Java Agent (handout)
Extending Spark With Java Agent (handout)Jaroslav Bachorik This document discusses using Java agents to extend Spark functionality. Java agents allow instrumentation of JVM applications like Spark at runtime. Specifically, agents could be used to monitor block caching in Spark and make recommendations to improve caching decisions. The document provides examples of how to transform Spark classes at runtime using agents to track block usage and cache hits. Overall, agents provide a powerful way to enhance Spark without requiring code changes.
Apache Spark Introduction - CloudxLab
Apache Spark Introduction - CloudxLabAbhinav Singh This document outlines a hands-on session on big data processing using Apache Spark, covering topics such as RDD operations, transformations, and actions. Participants will learn to compute the word frequency of a text file stored in HDFS, using a cloud-based lab environment offered by CloudxLab. The session also provides a brief introduction to CloudxLab's features and the instructor's background in big data technologies.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit The document outlines a presentation on using Redis for real-time machine learning, focusing on Redis-ML integrated with Apache Spark. It covers the capabilities of Redis as a key-value store, the functionalities of Redis-ML for model serving, and a case study involving a recommendation system. The concluding remarks emphasize the advantages of combining Spark for training with Redis for efficient model serving, highlighting reduced resource costs and simplified machine learning lifecycle.
Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...
Opaque: A Data Analytics Platform with Strong Security: Spark Summit East tal...Spark Summit This document describes Opaque, a secure distributed data analytics framework that allows complex analytics to run on sensitive data while preserving data privacy and functionality. Opaque utilizes hardware enclaves to protect computation and data within enclaves from a malicious operating system or cloud provider. It implements various oblivious primitives and oblivious operators to execute Spark SQL queries without leaking data access patterns. This allows sensitive data to be analyzed using existing Spark SQL queries while preventing privacy leaks from memory, network or computation access patterns.
Similar to Elasticsearch And Apache Lucene For Apache Spark And MLlib (20)
Writing Continuous Applications with Structured Streaming PySpark API
Writing Continuous Applications with Structured Streaming PySpark APIDatabricks The document discusses the development of continuous applications using Structured Streaming in Apache Spark, highlighting its benefits, complexities, and capabilities in handling diverse data. It explains the architecture of Structured Streaming, including the process of reading, transforming, and writing data streams, as well as the importance of features like checkpointing for fault tolerance. The presentation includes tutorials and emphasizes the unification of batch and streaming processing to enable real-time data handling and analytics.
Intro to Spark and Spark SQL
Intro to Spark and Spark SQLjeykottalam Apache Spark is a fast and general cluster computing system that improves efficiency through in-memory computing and usability through rich APIs. Spark SQL provides a way to work with structured data and transform RDDs using SQL. It can read data from sources like Parquet and JSON files, Hive, and write query results to Parquet for efficient querying. Spark SQL also allows machine learning pipelines to be built by connecting SQL queries to MLlib algorithms.
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...Microsoft Tech Community The document outlines a collaborative analytics platform based on Apache Spark optimized for Azure, specifically Databricks, featuring streamlined workflows for data scientists and analysts. It emphasizes integration with various Azure services and capabilities for real-time data processing, including event-time processing and exactly-once guarantees. The platform's advanced functionalities include support for complex queries, stateful processing, and built-in support for diverse data formats and sources.
Writing Continuous Applications with Structured Streaming in PySpark
Writing Continuous Applications with Structured Streaming in PySparkDatabricks The document discusses the use of Apache Spark for building continuous applications with structured streaming, emphasizing the challenges of stream processing and advantages of using Spark. It covers various components such as data integration, transformation, and real-time analytics, showcasing the power of structured streaming in dealing with complex data. The talk includes a demonstration of continuous applications and resources for further exploration of Spark's capabilities.
Easy, scalable, fault tolerant stream processing with structured streaming - ...
Easy, scalable, fault tolerant stream processing with structured streaming - ...Databricks The document discusses the complexities of building scalable and fault-tolerant stream processing applications using Structured Streaming with Apache Spark. It highlights key features such as handling diverse data formats and storage systems, ensuring exactly-once processing semantics, and leveraging Spark SQL for incremental execution. The presentation covers practical examples and concepts like triggers, output modes, and watermarking to efficiently manage state and late data in streaming queries.
Introduction to Apache Spark
Introduction to Apache SparkAnastasios Skarlatidis The document is an introduction to Apache Spark, detailing its features, architecture, and programming capabilities across multiple sections. It discusses Spark's advantages as a distributed in-memory computational framework, its applications for data scientists and software engineers, and how resilient distributed datasets (RDDs) function within Spark's ecosystem. Additionally, it covers basic programming with RDDs, transformations and actions, as well as advanced features like persistence and shared variables.
20170126 big data processing
20170126 big data processingVienna Data Science Group The document discusses big data processing techniques and technologies, focusing on architectures like Apache Spark and Apache Flink. It details the evolution of data processing engines, data processing patterns, components, and their advantages over previous frameworks. Additionally, it covers implementation details, programming interfaces, and performance metrics relevant to data processing tasks across varied industries.
Terrastore - A document database for developers
Terrastore - A document database for developersSergio Bossa Sergio Bossa is a software architect and engineer who has worked on online gambling and casino software. He is an open source enthusiast who has contributed to projects like Spring, Terracotta, and Terrastore. Terrastore is a document database for developers that is document-based, consistent, distributed, scalable, and written in Java using Terracotta. It allows for easy installation, no complex configuration, and simple basic operations like putting and getting documents from buckets. It also supports features like range queries, predicate queries, server-side updates, and easy scalability. Terrastore is best suited for data hot spots, computational data, complex or variable data, and throw-away data.
IoT Applications and Patterns using Apache Spark & Apache Bahir
IoT Applications and Patterns using Apache Spark & Apache BahirLuciano Resende The document discusses IoT applications and patterns using Apache Spark and Apache Bahir, focusing on integrating various data sources and streaming capabilities for real-time analytics. It highlights the core functionalities of Apache Spark, including Spark SQL, Spark Streaming, and the extensions provided by Apache Bahir for IoT connectivity such as MQTT and Cloudant. Additionally, it presents a live demo simulating an IoT environment, outlining the benefits of leveraging these technologies for processing IoT data.
Just one-shade-of-openstack
Just one-shade-of-openstackRoberto Polli This document discusses using the Shade library to simplify working with OpenStack. It begins with an introduction to OpenStack and its architecture. It then demonstrates how to quickly get started with Shade by connecting to clouds and listing servers. The document shows how to access OpenStack resources as objects and dictionaries, and generate reports with Pandas. It also discusses implementing Ansible modules with Shade to provision and manage projects across multiple OpenStack clouds. The document encourages contributing to Shade by implementing missing methods and writing functional tests.
ElasticSearch for .NET Developers
ElasticSearch for .NET DevelopersBen van Mol This document provides an overview of Elasticsearch and how to use it with .NET. It discusses what Elasticsearch is, how to install it, how Elasticsearch provides scalability through its architecture of clusters, nodes, shards and replicas. It also covers topics like indexing and querying data through the REST API or NEST client for .NET, performing searches, aggregations, highlighting hits, handling human language through analyzers, and using suggesters.
Big Data Processing with .NET and Spark (SQLBits 2020)
Big Data Processing with .NET and Spark (SQLBits 2020)Michael Rys .NET for Apache Spark is aimed at providing .NET developers with a first-class experience for big data processing, allowing them to utilize existing business logic and libraries efficiently. The open-source project integrates with Azure services and offers features such as batch processing, streaming, and support for various programming paradigms while optimizing performance through Apache Arrow. The recent release of version 1.0 introduces comprehensive C# and F# bindings, enhancing usability for .NET developers in big data analytics.
Spark what's new what's coming
Spark what's new what's comingDatabricks The document discusses the advancements and features of Apache Spark, highlighting its capabilities for big data processing with APIs in Python, Java, and Scala. New features in versions 1.4 and 1.5 include enhanced machine learning pipelines, dataframes for optimized data manipulation, and improved SQL support. The importance of Databricks as a key contributor to Spark and the ongoing commitment to an open-source platform is emphasized.
Real-Time Spark: From Interactive Queries to Streaming
Real-Time Spark: From Interactive Queries to StreamingDatabricks This document summarizes Michael Armbrust's presentation on real-time Spark. It discusses:
1. The goals of real-time analytics including having the freshest answers as fast as possible while keeping the answers up to date.
2. Spark 2.0 introduces unified APIs for SQL, DataFrames and Datasets to make developing real-time analytics simpler with powerful yet simple APIs.
3. Structured streaming allows running the same SQL queries on streaming data to continuously aggregate data and update outputs, unifying batch, interactive, and streaming queries into a single API.
Designing Structured Streaming Pipelines—How to Architect Things Right
Designing Structured Streaming Pipelines—How to Architect Things RightDatabricks The document outlines the architecture and processes of structured streaming using Spark, focusing on its capabilities for distributed stream processing, high throughput, and fault tolerance. It details various streaming query patterns, including ETL processes, stateful aggregations, and change data capture, with examples of handling data from sources like Kafka. The document emphasizes the design patterns for efficient data pipeline development and the importance of latency and throughput considerations in those designs.
Building a Unified Data Pipline in Spark / Apache Sparkを用いたBig Dataパイプラインの統一
Building a Unified Data Pipline in Spark / Apache Sparkを用いたBig Dataパイプラインの統一scalaconfjp The document provides an overview of Apache Spark, a fast and general cluster computing system that enhances big data analytics through in-memory computing and rich APIs. It discusses the evolution of Spark since its inception, its ability to handle complex analytics, interactive queries, and real-time stream processing, while emphasizing its unified architecture for various processing types. Additionally, it highlights the components of the Spark ecosystem, including Spark SQL, Spark Streaming, and MLlib for machine learning.
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...
From HelloWorld to Configurable and Reusable Apache Spark Applications in Sca...Databricks The document presents a session by Oliver Țupran on creating configurable and reusable Apache Spark applications using Scala, focusing on common problems and solutions related to application logic and configuration. Key solutions include separating business logic from configuration, utilizing a robust configurator framework, and applying testing methodologies. The session aims to enhance productivity while minimizing complexity in Spark application development.
Apache Spark Workshop, Apr. 2016, Euangelos Linardos
Apache Spark Workshop, Apr. 2016, Euangelos LinardosEuangelos Linardos This document outlines a workshop on Apache Spark, detailing the setup environment across different operating systems and providing hands-on training examples. It covers Spark's advantages, architecture, resilient distributed datasets (RDDs), and includes sample codes for operations like word count and data transformation. Additionally, it discusses Spark's performance benefits over MapReduce and the importance of persistence in RDDs for efficient computation.
Spark (Structured) Streaming vs. Kafka Streams
Spark (Structured) Streaming vs. Kafka StreamsGuido Schmutz This document compares two stream processing platforms, Spark Structured Streaming and Kafka Streams, detailing their architectures, features, and use cases. It highlights the differences in processing models, language support, and operational capabilities such as event time processing and state management. The summary concludes with a comparison of their advantages and limitations within stream processing ecosystems.
A Tale of Two APIs: Using Spark Streaming In Production
A Tale of Two APIs: Using Spark Streaming In ProductionLightbend The document presents a detailed overview of streaming data processing using Apache Spark, focusing on structured streaming and its various components such as Kafka and machine learning. It highlights practical use cases, code examples, and the integration of different data sources and operations. The author, Gerard Maas, is an experienced software engineer and an active contributor to the Scala and Spark communities.
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Leveraging Azure Databricks to minimize time to insight by combining Batch an...Microsoft Tech Community
Ad
More from Jen Aman (20)
Deep Learning and Streaming in Apache Spark 2.x with Matei Zaharia
Deep Learning and Streaming in Apache Spark 2.x with Matei ZahariaJen Aman The document discusses advancements in Apache Spark 2.x, particularly focusing on structured streaming and deep learning capabilities. It highlights the introduction of a high-level API for streaming that simplifies processes and allows for end-to-end applications, as well as the integration of deep learning frameworks to enhance usability. Key features include improved performance, cost efficiency, and the goal of making deep learning accessible to a broader range of users.
Snorkel: Dark Data and Machine Learning with Christopher Ré
Snorkel: Dark Data and Machine Learning with Christopher RéJen Aman The document discusses Snorkel, a project aimed at simplifying machine learning by enabling users to build systems for tasks like data extraction without the need for large hand-labeled training sets. It highlights how Snorkel leverages a method known as data programming, where users create labeling functions to generate noisy training data, enhancing the efficiency of model training. The document emphasizes the potential of machine learning to make programming significantly easier and showcases the effectiveness of Snorkel in achieving competitive results compared to traditional supervised approaches.
Deep Learning on Apache® Spark™: Workflows and Best Practices
Deep Learning on Apache® Spark™: Workflows and Best PracticesJen Aman This document summarizes a presentation about deep learning workflows and best practices on Apache Spark. It discusses how deep learning fits within broader data pipelines for tasks like training and transformation. It also outlines recurring patterns for integrating Spark and deep learning frameworks, including using Spark for data parallelism and embedding deep learning transforms. The presentation provides tips for developers on topics like using GPUs with PySpark and monitoring deep learning jobs. It concludes by discussing challenges in the areas of distributed deep learning and Spark integration.
Deep Learning on Apache® Spark™ : Workflows and Best Practices
Deep Learning on Apache® Spark™ : Workflows and Best PracticesJen Aman The document presents an overview of deep learning workflows and best practices using Apache Spark, highlighting its integration with various deep learning frameworks such as TensorFlow and Caffe. It discusses the architecture of Databricks, which simplifies data processing and machine learning for users, as well as monitoring techniques and solutions for managing distributed deep learning tasks. The presentation emphasizes the importance of experimentation and provides resources for further learning.
RISELab:Enabling Intelligent Real-Time Decisions
RISELab:Enabling Intelligent Real-Time DecisionsJen Aman The RISE Lab at Berkeley aims to develop an open-source data analytics stack for real-time intelligent decision-making using live data, with a focus on speed, security, and personalization. The initiative involves research in algorithms, machine learning, and systems to create frameworks that improve the responsiveness and robustness of decision systems. The project's ongoing efforts include enhancing existing tools like Apache Spark and exploring secure reinforcement learning applications.
Spatial Analysis On Histological Images Using Spark
Spatial Analysis On Histological Images Using SparkJen Aman This document describes using Spark for spatial analysis of histological images to characterize the tumor microenvironment. The goal is to provide actionable data on the location and density of immune cells and blood vessels. Over 100,000 objects are annotated in each whole slide image. Spark is used to efficiently calculate over 5 trillion pairwise distances between objects within a neighborhood window. This enables profiling of co-localization and spatial clustering of objects. Initial results show the runtime scales linearly with the number of objects. Future work includes integrating clinical and genomic data to characterize variation between tumor types and patients.
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...Jen Aman Kyle Foreman presented on using Spark for large-scale global health simulations. The talk discussed (1) the motivation for simulations to model disease burden forecasts and alternative scenarios, (2) SimBuilder for constructing modular simulation workflows as directed acyclic graphs, and (3) benchmarks showing Spark backends can efficiently distribute simulations across a cluster. Future work aims to optimize Spark DataFrame joins and take better advantage of Numpy's vectorization for panel data simulations.
A Graph-Based Method For Cross-Entity Threat Detection
A Graph-Based Method For Cross-Entity Threat DetectionJen Aman This document proposes a graph-based method for cross-entity threat detection. It models entity relationships as a multigraph and detects anomalies by identifying unexpected new connections between entities over time. It introduces two algorithms: a naive detector that identifies edges only in the detection graph, and a 2nd-order detector that identifies edges between entity clusters. An experiment on a real dataset found around 700 1st-order and 200 2nd-order anomalies in under 5 minutes, demonstrating the method's ability to efficiently detect threats across unrelated accounts.
Yggdrasil: Faster Decision Trees Using Column Partitioning In Spark
Yggdrasil: Faster Decision Trees Using Column Partitioning In SparkJen Aman This document introduces Yggdrasil, a new approach for training decision trees in Spark that partitions data by column instead of by row. Column partitioning significantly reduces communication costs for deep trees with many features. Evaluation on real-world datasets with millions of rows and thousands of features shows Yggdrasil achieves up to 24x speedup over the existing row-partitioning approach in Spark MLlib. The authors propose merging Yggdrasil into Spark MLlib to provide both row and column partitioning options for optimal performance on different problem sizes and depths.
Time-Evolving Graph Processing On Commodity Clusters
Time-Evolving Graph Processing On Commodity ClustersJen Aman Tegra is a system for efficiently processing time-evolving graphs on commodity clusters. It uses a distributed graph snapshot index to represent and retrieve multiple snapshots of evolving graphs. It introduces a timelapse abstraction to perform temporal analytics on windows of snapshots, avoiding redundant computation. Tegra supports both bulk and incremental graph computations using this representation, allowing results to be reused when graphs are updated. An evaluation on real-world graphs shows Tegra can store more snapshots in memory and reduce computation time compared to baseline approaches.
Deploying Accelerators At Datacenter Scale Using Spark
Deploying Accelerators At Datacenter Scale Using SparkJen Aman The document discusses the deployment of accelerators, specifically FPGA and GPU, at datacenter scale to enhance performance and energy efficiency as CPU core scaling nears its limits. It introduces the Blaze system, which provides an accelerator-as-a-service model, optimizing JVM-to-accelerator communication to streamline programming and enhance big data application integration. The paper highlights the challenges in accelerator management and programming, while showcasing improvements in job latency and energy consumption through optimized hardware utilization.
Re-Architecting Spark For Performance Understandability
Re-Architecting Spark For Performance UnderstandabilityJen Aman The document discusses a new architecture called "monotasks" that is designed to make it easier to reason about the performance of Apache Spark jobs. The key aspects are:
1) Each task in a Spark job is dedicated to using a single resource (e.g. network, CPU, disk).
2) Dedicated schedulers control contention between tasks for resources.
3) The timing of individual "monotasks" can be used to model an ideal performance and understand bottlenecks.
Re-Architecting Spark For Performance Understandability
Re-Architecting Spark For Performance UnderstandabilityJen Aman The document describes a new architecture called "monotasks" for Apache Spark that aims to make reasoning about Spark job performance easier. The monotasks architecture decomposes Spark tasks so that each task uses only one resource (e.g. CPU, disk, network). This avoids issues where Spark tasks bottleneck on different resources over time or experience resource contention. With monotasks, dedicated schedulers control resource contention and monotask timing data can be used to model ideal performance. Results show monotasks match Spark's performance and provide clearer insight into bottlenecks.
Low Latency Execution For Apache Spark
Low Latency Execution For Apache SparkJen Aman This document describes Drizzle, a low latency execution engine for Apache Spark. It addresses the high overheads of Spark's centralized scheduling model by decoupling execution from scheduling through batch scheduling and pre-scheduling of shuffles. Microbenchmarks show Drizzle achieves milliseconds latency for iterative workloads compared to hundreds of milliseconds for Spark. End-to-end experiments show Drizzle improves latency for streaming and machine learning workloads like logistic regression. The authors are working on automatic batch tuning and an open source release of Drizzle.
Efficient State Management With Spark 2.0 And Scale-Out Databases
Efficient State Management With Spark 2.0 And Scale-Out DatabasesJen Aman This document discusses efficient state management with Spark 2.0 and scale-out databases. It introduces SnappyData, an open source project that provides a unified in-memory database for streams, transactions, and OLAP queries to enable real-time operational analytics. SnappyData extends Spark by localizing state management and processing to avoid shuffles, supports approximate query processing for interactive queries, and provides a unified cluster architecture for OLTP, OLAP and streaming workloads.
Livy: A REST Web Service For Apache Spark
Livy: A REST Web Service For Apache SparkJen Aman Livy is an open source REST service for interacting with and managing Spark contexts and jobs. It allows clients to submit Spark jobs via REST, monitor their status, and retrieve results. Livy manages long-running Spark contexts in a cluster and supports running multiple independent contexts simultaneously from different clients. It provides client APIs in Java, Scala, and soon Python to interface with the Livy REST endpoints for submitting, monitoring, and retrieving results of Spark jobs.
GPU Computing With Apache Spark And Python
GPU Computing With Apache Spark And PythonJen Aman GPU Computing With Apache Spark And Python
- Python is a popular language for data science and analytics due to its large ecosystem of libraries and ease of use, but it is slow for number crunching tasks. GPU computing is a way to accelerate Python workloads.
- This presentation demonstrates using GPUs with Apache Spark and Python through libraries like Accelerate, which provides drop-in GPU-accelerated functions, and Numba, which can compile Python functions to run on GPUs.
- As an example, the task of image registration, which involves computationally expensive 2D FFTs, is accelerated using these GPU libraries within a PySpark job, achieving a 2-4x speedup over CPU-only versions
Spark And Cassandra: 2 Fast, 2 Furious
Spark And Cassandra: 2 Fast, 2 FuriousJen Aman The document discusses the Datastax Spark Cassandra Connector. It provides an overview of how the connector allows Spark to interact with Cassandra data, including performing full table scans, pushing down filters and projections to Cassandra, distributed joins using Cassandra's partitioning, and writing data back to Cassandra in a distributed way. It also highlights some recent features of the connector like support for Cassandra 3.0, materialized views, and performance improvements from the Java Wildcard Cassandra Tester project.
Building Custom Machine Learning Algorithms With Apache SystemML
Building Custom Machine Learning Algorithms With Apache SystemMLJen Aman This document discusses Apache SystemML, which is a machine learning framework for building custom machine learning algorithms on Apache Spark. It originated from research projects at IBM involving machine learning on Hadoop. SystemML aims to allow data scientists to build ML solutions using languages like R and Python, while executing algorithms on big data platforms like Spark. It provides a high-level language for expressing algorithms and performs automatic parallelization and optimization. The document demonstrates SystemML through a matrix factorization example for a targeted advertising problem. It shows how to wrangle data, build a custom algorithm, and get results. In conclusion, it recommends that readers try out SystemML through its website.
Spark on Mesos
Spark on MesosJen Aman This document summarizes a talk on Spark on Mesos given by Dean Wampler from Lightbend and Timothy Chen from Mirantis. It discusses why Spark on Mesos is useful by allowing one cluster system to run multiple tools. It then covers recent updates like a new integration test suite, coarse-grained scheduler improvements, and Mesos framework authentication. Finally, it outlines future plans such as GPU support on Mesos and making "production" use of Spark on Mesos easier.
Ad
Recently uploaded (20)
MICROSOFT POWERPOINT AND USES(BEST)..pdf
MICROSOFT POWERPOINT AND USES(BEST)..pdfbathyates Talks about presentation packages and their uses to man and how it functions being the best presentation package. Learn about presentation packages here with me
SAP_S4HANA_PPM_IT_Corporate_Services_Presentation.pptx
SAP_S4HANA_PPM_IT_Corporate_Services_Presentation.pptxvemulavenu484 The Corporate and IT Services industry is undergoing rapid transformation with growing demand for agility, innovation, and strategic project delivery. Enterprises are challenged to manage a wide variety of service-based projects, such as IT infrastructure upgrades, software development, digital transformation, regulatory compliance, and internal operational improvements.
SAP S/4HANA Portfolio and Project Management (PPM) is designed to help organizations align IT and corporate project portfolios with business goals, improve project visibility, enhance resource utilization, and ensure financial control. This document outlines key features of SAP PPM and its relevance for the Corporate and IT Services industry, emphasizing value realization and strategic alignment.
SAP_S4HANA_EWM_Food_Processing_Industry.pptx
SAP_S4HANA_EWM_Food_Processing_Industry.pptxvemulavenu484 SAP Extended Warehouse Management (EWM) is a part of SAP S/4HANA offering advanced warehouse and logistics capabilities. It enables efficient handling of goods movement, storage, and inventory in real-time.
apidays New York 2025 - The Future of Small Business Lending with Open Bankin...
apidays New York 2025 - The Future of Small Business Lending with Open Bankin...apidays The Future of Small Business Lending with Open Banking – Bridging the $750 Billion Funding Gap
Charles Groome, Vice President of Growth Strategy at Biz2Credit
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...apidays Beyond Webhooks: The Future of Scalable API Event Delivery
Phil Leggetter, Head of Developer Experience at Hookdeck
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Veilig en vlot fietsen in Oost-Vlaanderen: Fietssnelwegen geoptimaliseerd met...
Veilig en vlot fietsen in Oost-Vlaanderen: Fietssnelwegen geoptimaliseerd met...jacoba18 Dit project, in samenwerking met de provincie Oost-Vlaanderen, heeft als doel de veiligheid op de F45 tussen Oudenaarde en Gent te verbeteren door netwerkdata aan te vullen met snelheidslimieten, toegangsbeperkingen en voorrangsregels. Met mobile mapping en FME werden verkeersborden en kruispunten geanalyseerd, waarna de resultaten in ArcGIS inzicht boden in toegankelijke wegsegmenten en verkeersregels.
apidays Singapore 2025 - Building Finance Innovation Ecosystems by Umang Moon...
apidays Singapore 2025 - Building Finance Innovation Ecosystems by Umang Moon...apidays Building Finance Innovation Ecosystems
Umang Moondra, CEO at APIX
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Unifying OpenAPI & AsyncAPI by Naresh Jain & Hari Kri...
apidays New York 2025 - Unifying OpenAPI & AsyncAPI by Naresh Jain & Hari Kri...apidays Unifying OpenAPI & AsyncAPI: Designing JSON Schemas+Examples for Reuse
Naresh Jain, Co-founder & CEO at Specmatic
Hari Krishnan, Co-founder & CTO at Specmatic
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Boost API Development Velocity with Practical AI Tool...
apidays New York 2025 - Boost API Development Velocity with Practical AI Tool...apidays Boost API Development Velocity with Practical AI Tooling
Sumit Amar, VP of Engineering at WEX
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - The FINOS Common Domain Model for Capital Markets by ...
apidays New York 2025 - The FINOS Common Domain Model for Capital Markets by ...apidays The FINOS Common Domain Model for Capital Markets
Tom Healey, Founder & Director at FINXIS LLC
Daniel Schwartz, Managing Partner at FT Advisory LLC
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
THE FRIEDMAN TEST ( Biostatics B. Pharm)
THE FRIEDMAN TEST ( Biostatics B. Pharm)JishuHaldar The Friedman Test is a valuable non-parametric alternative to the
Repeated Measures ANOVA, allowing for the comparison of three or
more related groups when data is ordinal or not normally distributed.
By ranking data instead of using raw values, the test overcomes the
limitations of parametric tests, making it ideal for small sample sizes and
real-world applications in medicine, psychology, pharmaceutical
sciences, and education. However, while it effectively detects differences
among groups, it does not indicate which specific groups differ, requiring
further post-hoc analysis.
KLIP2Data voor de herinrichting van R4 West en Oost
KLIP2Data voor de herinrichting van R4 West en Oostjacoba18 Ontwikkeling van een centrale GIS-databank voor de periodieke import van KLIP-data, BIM- en CAD-modellen van het R4-project en bijhorende field app om de databank zowel online als offline te raadplegen en bij te werken.
apidays New York 2025 - Life is But a (Data) Stream by Sandon Jacobs (Confluent)
apidays New York 2025 - Life is But a (Data) Stream by Sandon Jacobs (Confluent)apidays Life is But a (Data) Stream: Building Quality Data Pipelines
Sandon Jacobs, Senior Developer Advocate at Confluent
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
SQL-Demystified-A-Beginners-Guide-to-Database-Mastery.pptx
SQL-Demystified-A-Beginners-Guide-to-Database-Mastery.pptxbhavaniteacher99 Unlock the power of data with “SQL Demystified” — your perfect beginner-friendly guide to mastering Structured Query Language (SQL)! Whether you're a student, aspiring analyst, or a professional stepping into the world of data, this detailed and practical PowerPoint presentation simplifies the fundamentals and builds your SQL skills step-by-step.
📘 What’s Inside?
Introduction to SQL and its real-world importance
Understanding database structures: tables, columns & relationships
Core SQL queries: SELECT, FROM, WHERE, ORDER BY
Hands-on examples and practice exercises
Advanced SQL: JOIN, subqueries, and aggregate functions
Data manipulation: INSERT, UPDATE, DELETE
Performance tips and indexing basics
SQL security best practices and learning paths for future growth
🧠 Who is it for?
Beginners eager to learn SQL
Data science and computer students
Business professionals and analysts
Anyone curious about how databases work
✅ Features:
Clear visuals and examples
Easy-to-follow explanations
Real-life applications in banking, healthcare, e-commerce, and more
Reusable and editable presentation file (PPTX format)
Take your first step toward data mastery with confidence. Learn SQL the smart way — with structure, clarity, and purpose.
METHODS OF DATA COLLECTION (Research methodology)
METHODS OF DATA COLLECTION (Research methodology)anwesha248 The important methods of data collection along with advantages and disadvantages are published in this power point presentation.
published by- Prateeti Anwesha Bordoloi, Research Scholar
Mahapurusha Srimanta Sankaradeva Vishwavidyalaya
June 2025
2.5-DESPATCH-ORDINARY MAILS.pptxlminub7b7t6f7h7t6f6g7g6fg
2.5-DESPATCH-ORDINARY MAILS.pptxlminub7b7t6f7h7t6f6g7g6fgmk1227103 8hunubybub6v6vubug7gyvtvtvyb7g7h7hih8h8hij8jihih
Elasticsearch And Apache Lucene For Apache Spark And MLlib
- 1. Elasticsearch & Lucene for Apache Spark and MLlib Costin Leau (@costinl)
- 2. Mirror, mirror on the wall, what’s the happiest team of us all ? Briita Weber - Rough translation from German by yours truly -
- 3. Purpose of the talk Improve ML pipelines through IR Text processing • Analysis • Featurize/Vectorize * * In research / poc / WIP / Experimental phase
- 4. Technical Debt Machine Learning: The High Interest Credit Card of Technical Debt”, Sculley et al http://research.google.com/pubs/pub43146.html
- 5. Technical Debt Machine Learning: The High Interest Credit Card of Technical Debt”, Sculley et al http://research.google.com/pubs/pub43146.html
- 7. Challenge: What team at Elastic is most happy? Data: Hipchat messages Training / Test data: http://www.sentiment140.com Result: Kibana dashboard
- 8. ML Pipeline Chat data Sentiment Model Production Data Apply the rule Predict the ‘class’ J / L
- 9. Data is King
- 10. Example: Word2Vec Input snippet http://spark.apache.org/docs/latest/mllib-feature-extraction.html#example it was introduced into mathematics in the book disquisitiones arithmeticae by carl friedrich gauss in one eight zero one ever since however modulo has gained many meanings some exact and some imprecise
- 11. Real data is messy originally looked like this: https://en.wikipedia.org/wiki/Modulo_(jargon) It was introduced into <a href="https://en.wikipedia.org/wiki/Mathematics" title="Mathematics">mathematics</a> in the book <i><a href="https://en.wikipedia.org/wiki/Disquisitiones_Arithmeticae" title="Disquisitiones Arithmeticae">Disquisitiones Arithmeticae</a></i> by <a href="https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss" title="Carl Friedrich Gauss">Carl Friedrich Gauss</a> in 1801. Ever since, however, "modulo" has gained many meanings, some exact and some imprecise.
- 12. Feature extraction Cleaning up data "huuuuuuunnnnnnngrrryyy", "aaaaaamaaazinggggg", "aaaaaamazing", "aaaaaammm", "aaaaaammmazzzingggg", "aaaaaamy", "aaaaaan", "aaaaaand", "aaaaaannnnnnddd", "aaaaaanyways" Does it help to clean that up? see “Twitter Sentiment Classification using Distant Supervision”, Go et al. http://www-cs.stanford.edu/people/alecmgo/papers/TwitterDistantSupervision09.pdf
- 14. Lucene to the rescue! High-performance, full-featured text search library 15 years of experience Widely recognized for its utility • It’s a primary test bed for new JVM versions
- 15. Text processing Character Filter Tokenizer Token FilterToken FilterToken Filter Do <b>Johnny Depp</b> a favor and forget you… Do Pos: 1 Johnny Pos: 2 do Pos: 1 johnny Pos: 2
- 16. Lucene for text analysis state of the art text processing many extensions available for different languages, use cases,… however…
- 17. … import org.apache.lucene.analysis… … Analyzer a = new Analyzer() { @Override protected TokenStreamComponents createComponents(String fieldName) { Tokenizer tokenizer = new StandardTokenizer(); return new TokenStreamComponents(tokenizer, tokenizer); } @Override protected Reader initReader(String fieldName, Reader reader) { return new HTMLStripCharFilter(reader); } }; TokenStream stream = a.tokenStream(null, "<a href=...>some text</a>"); CharTermAttribute term = stream.addAttribute(CharTermAttribute.class); PositionIncrementAttribute posIncrement = stream.addAttribute(PositionIncrementAttribute.class); stream.reset(); int pos = 0; while (stream.incrementToken()) { pos += posIncrement.getPositionIncrement(); System.out.println(term.toString() + " " + pos); } > some 1 > text 2
- 18. … import org.apache.lucene.analysis… … Analyzer a = new Analyzer() { @Override protected TokenStreamComponents createComponents(String fieldName) { Tokenizer tokenizer = new StandardTokenizer(); return new TokenStreamComponents(tokenizer, tokenizer); } @Override protected Reader initReader(String fieldName, Reader reader) { return new HTMLStripCharFilter(reader); } }; TokenStream stream = a.tokenStream(null, "<a href=...>some text</a>"); CharTermAttribute term = stream.addAttribute(CharTermAttribute.class); PositionIncrementAttribute posIncrement = stream.addAttribute(PositionIncrementAttribute.class); stream.reset(); int pos = 0; while (stream.incrementToken()) { pos += posIncrement.getPositionIncrement(); System.out.println(term.toString() + " " + pos); } > some 1 > text 2 How about a declarative approach?
- 20. Very quick intro to Elasticsearch
- 21. Elasticsearch in 5 3’ Scalable, real-time search and analytics engine Data distribution, cluster management REST APIs JVM based, uses Apache Lucene internally Open-source (on Github, Apache 2 License)
- 23. Elasticsearch in 3’ Sorting / Scoring
- 29. Machine Learning and Elasticsearch
- 30. Machine Learning and Elasticsearch Term Analysis (tf, idf, bm25) Graph Analysis Co-occurrence of Terms (significant terms) • ChiSquare Pearson correlation (#16817) Regression (#17154) What about classification/clustering/ etc… ?
- 31. 31 It’s not the matching data, but the meta that lead to it
- 32. How to use Elasticsearch from Spark ? Somebody on Stackoverflow
- 33. Elasticsearch for Apache Hadoop ™
- 34. Elasticsearch for Apache Hadoop ™
- 35. Elasticsearch for Apache Hadoop ™
- 36. Elasticsearch Spark – Native integration Scala & Java API Understands Scala & Java types – Case classes – Java Beans Available as Spark package Supports Spark Core & SQL all 1.x version (1.0-1.6) Available for Scala 2.10 and 2.11
- 37. Elasticsearch as RDD / Dataset* import org.elasticsearch.spark._ val sc = new SparkContext(new SparkConf()) val rdd = sc.esRDD(“buckethead/albums", "?q=pikes") import org.elasticsearch.spark._ case class Artist(name: String, albums: Int) val u2 = Artist("U2", 13) val bh = Map("name"->"Buckethead","albums" -> 255, "age" -> 46) sc.makeRDD(Seq(u2, bh)).saveToEs("radio/artists")
- 38. Elasticsearch as a DataFrame val df = sql.read.format(“es").load("buckethead/albums") df.filter(df("category").equalTo("pikes").and(df("year").geq(2015))) { "query" : { "bool" : { "must" : [ "match" : { "category" : "pikes" } ], "filter" : [ { "range" : { "year" : {"gte" : "2015" }}} ] }}}
- 39. Partition to Partition Architecture
- 40. Putting the pieces together
- 41. Typical ML pipeline for text
- 42. Typical ML pipeline for text Actual ML code
- 43. Typical ML pipeline for text
- 44. Pure Spark MLlib val training = movieReviewsDataTrainingData val tokenizer = new Tokenizer() .setInputCol("text") .setOutputCol("words") val hashingTF = new HashingTF() .setNumFeatures(1000) .setInputCol(tokenizer.getOutputCol) .setOutputCol("features") val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.001) val pipeline = new Pipeline() .setStages(Array(tokenizer, hashingTF, lr)) val model = pipeline.fit(training)
- 45. Pure Spark MLlib val tokenizer = new Tokenizer() .setInputCol("text") .setOutputCol("words") val hashingTF = new HashingTF() .setNumFeatures(1000) .setInputCol(tokenizer.getOutputCol) .setOutputCol("features") val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.001)
- 46. Pure Spark MLlib val tokenizer = new Tokenizer() .setInputCol("text") .setOutputCol("words") val hashingTF = new HashingTF() .setNumFeatures(1000) .setInputCol(tokenizer.getOutputCol) .setOutputCol("features") val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.001)
- 47. Pure Spark MLlib val analyzer = new ESAnalyzer() .setInputCol("text") .setOutputCol("words") val hashingTF = new HashingTF() .setNumFeatures(1000) .setInputCol(tokenizer.getOutputCol) .setOutputCol("features") val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.001)
- 48. Pure Spark MLlib val analyzer = new ESAnalyzer() .setInputCol("text") .setOutputCol("words") val hashingTF = new HashingTF() .setNumFeatures(1000) .setInputCol(tokenizer.getOutputCol) .setOutputCol("features") val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.001)
- 49. Data movement
- 50. Work once – reuse multiple times // index / analyze the data training.saveToEs("movies/reviews")
- 51. Work once – reuse multiple times // prepare the spec for vectorize – fast and lightweight val spec = s"""{ "features" : [{ | "field": "text", | "type" : "string", | "tokens" : "all_terms", | "number" : "occurrence", | "min_doc_freq" : 2000 | }], | "sparse" : "true"}""".stripMargin ML.prepareSpec(spec, “my-spec”)
- 52. Access the vector directly // get the features – just another query val payload = s"""{"script_fields" : { "vector" : | { "script" : { "id" : “my-spec","lang" : “doc_to_vector" } } | }}""".stripMargin // index the data vectorRDD = sparkCtx.esRDD("ml/data", payload) // feed the vector to the pipeline val vectorized = vectorRDD.map ( x => // get indices, the vector and length (if (x._1 == "negative") 0.0d else 1.0d, ML.getVectorFrom(x._2)) ).toDF("label", "features")
- 53. Revised ML pipeline val vectorized = vectorRDD.map... val lr = new LogisticRegression() .setMaxIter(10) .setRegParam(0.001) val model = lr.fit(vectorized)
- 54. Simplify ML pipeline Once per dataset, regardless of # of pipelines Raw data is not required any more
- 55. Need to adjust the model? Change the spec val spec = s"""{ "features" : [{ | "field": "text", | "type" : "string", | "tokens" : "given", | "number" : "tf", | "terms": ["term1", "term2", ...] | }], | "sparse" : "true"}""".stripMargin ML.prepareSpec(spec)
- 57. All this is WIP Not all features available (currently dictionary, vectors) Works with data outside or inside Elasticsearch (latter is much faster) Bind vectors to queries Other topics WIP: Focused on document / text classification – numeric support is next Model importing / exporting – Spark 2.0 ML persistence Feedback highly sought - Is this useful?
- 58. THANK YOU. j.mp/spark-summit-west-16 elastic.co/hadoop github.com/elastic | costin | brwe discuss.elastic.co @costinl