MatFast: In-Memory Distributed Matrix Computation Processing and Optimization Based on Spark SQL Yanbo Liang and Mingie Tang
4 likes1,675 views
The document presents MatFast, an in-memory distributed matrix computation platform optimized for processing queries over big matrix data, focused on applications like recommender systems and social network analysis. It discusses its implementation, optimization strategies, and comparison with existing systems in the Spark ecosystem, including data partitioning techniques for efficient execution. Future plans for MatFast include enhancing user APIs and developing additional optimization features.
1 of 34
Downloaded 56 times
Ad
Recommended
Near Data Computing Architectures: Opportunities and Challenges for Apache Spark
Near Data Computing Architectures: Opportunities and Challenges for Apache SparkAhsan Javed Awan The document discusses opportunities for improving Apache Spark performance using near data computing architectures. It proposes exploiting in-storage processing and 2D integrated processing-in-memory to reduce data movement between CPUs and memory. Certain Spark workloads like joins and aggregations that are I/O bound would benefit from in-storage processing, while iterative workloads are more suited for 2D integrated processing-in-memory. The document outlines a system design using FPGAs to emulate these architectures for evaluating Spark machine learning workloads like k-means clustering.
Powering a Startup with Apache Spark with Kevin Kim
Powering a Startup with Apache Spark with Kevin KimSpark Summit The document discusses the evolution of a startup called 'Between', led by Kevin Kim, detailing its growth from 100 beta users in 2011 to 20 million downloads by 2016. It highlights the team's use of Apache Spark for big data processing, the infrastructure they have built, and their current hiring needs for data analysts and machine learning experts. The document also emphasizes the importance of data analysis for product development and marketing strategies, along with the ongoing challenges and future plans for expanding operations.
Hardware Acceleration of Apache Spark on Energy-Efficient FPGAs with Christof...
Hardware Acceleration of Apache Spark on Energy-Efficient FPGAs with Christof...Spark Summit The document discusses the use of FPGA accelerators to enhance the performance of Spark applications on the cloud, particularly in machine learning tasks such as logistic regression and digit recognition. It highlights the advantages of combining CPUs with FPGAs for improved efficiency and speedup compared to traditional setups, showcasing significant performance gains. Inaccel's hardware accelerators facilitate seamless integration with Apache Spark and Amazon AWS, allowing users to enhance their applications without the need for extensive code modifications.
Improving Traffic Prediction Using Weather Datawith Ramya Raghavendra
Improving Traffic Prediction Using Weather Datawith Ramya RaghavendraSpark Summit The document discusses the development of a cognitive model by IBM Research that improves traffic prediction by integrating weather data, utilizing machine learning techniques like ARIMA, random forest, and LSTM. It highlights the significant impact of weather on traffic flow, emphasizes data collection challenges, and presents results from analyzing extensive traffic and weather records across multiple cities. The model aims to enhance commuter experiences by providing timely and precise traffic predictions, ultimately improving efficiency and reducing congestion.
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...
Extending Apache Spark SQL Data Source APIs with Join Push Down with Ioana De...Databricks This document summarizes a presentation on extending Spark SQL Data Sources APIs with join push down. The presentation discusses how join push down can significantly improve query performance by reducing data transfer and exploiting data source capabilities like indexes. It provides examples of join push down in enterprise data pipelines and SQL acceleration use cases. The presentation also outlines the challenges of network speeds and exploiting data source capabilities, and how join push down addresses these challenges. Future work discussed includes building a cost model for global optimization across data sources.
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang Wu
Apache Spark Structured Streaming Helps Smart Manufacturing with Xiaochang WuSpark Summit The document outlines the use of Spark Structured Streaming to build a smart manufacturing IoT data platform, focusing on enhancing efficiency in PCB manufacturing through predictive fault repair and material tracking. It details architectural features, data processing capabilities, and handling of stateless and stateful operations, along with improvement proposals for more complex event processing. Key components include real-time analytics, device management, data visualization, and the support for various data types.
A machine learning and data science pipeline for real companies
A machine learning and data science pipeline for real companiesDataWorks Summit Comcast is implementing a data science pipeline that integrates multi-datacenter, cloud, and on-premise machine learning to enhance customer experience and operational efficiency. The approach focuses on a standardized system for managing features and models, leveraging frameworks like Spark and Databricks, while promoting accessibility and self-service capabilities for users. Key components include a feature store, model store, job scheduler, and workflow management to enable streamlined data transformations and machine learning operations.
Building a Business Logic Translation Engine with Spark Streaming for Communi...
Building a Business Logic Translation Engine with Spark Streaming for Communi...Spark Summit Patrick Bamba presents on building a translation engine with Spark Streaming to enable communication between legacy code and microservices. The translation engine acts as an anti-corruption layer, translating data and requests between the different systems. Spark is well-suited for this purpose due to its streaming capabilities and built-in connectors for various data sources and sinks. The presentation provides an example implementation using structured streaming to interface legacy systems with microservices through sources, transformations, and sinks.
Art of Feature Engineering for Data Science with Nabeel Sarwar
Art of Feature Engineering for Data Science with Nabeel SarwarSpark Summit This document discusses feature engineering in data science, particularly in the context of machine learning at Comcast. It covers various processes involved in feature engineering such as selecting, cleaning, transforming, and enriching data, as well as methodologies for handling categorical and numerical variables. Additionally, it highlights the importance of domain knowledge, statistical correlation, and the challenges of dimensionality in optimizing machine learning models.
Parallelizing Large Simulations with Apache SparkR with Daniel Jeavons and Wa...
Parallelizing Large Simulations with Apache SparkR with Daniel Jeavons and Wa...Spark Summit The document outlines Shell's advancements in optimizing spare part inventory through advanced analytics, highlighting challenges in materials management, user confidence, and algorithm development. A pilot project in the Gulf of Mexico demonstrated significant cost savings and efficiency improvements, including reducing simulation times dramatically via upgraded computing methods. The initiative is set to scale globally, with potential for extensive benefits across Shell's operations.
Scaling Machine Learning with Apache Spark
Scaling Machine Learning with Apache SparkDatabricks The document discusses the use of Apache Spark for scaling machine learning, covering Spark's architecture, various ML training paradigms, and the application of different libraries like Spark MLlib and pandas. It emphasizes the importance of understanding when to utilize Spark for machine learning in a fast-paced environment and explores techniques for model training, tuning, and inference. Additionally, it introduces advanced topics such as hyperparameter optimization and the use of pandas UDFs for efficient data processing.
Feature Hashing for Scalable Machine Learning with Nick Pentreath
Feature Hashing for Scalable Machine Learning with Nick PentreathSpark Summit Nick Pentreath's presentation discusses feature hashing as a method for scalable machine learning using Apache Spark. It highlights the pros and cons of feature hashing, its application in text classification and ad click prediction, and includes performance results from experiments. Future work aims to improve the feature hashing method and integrate it into Spark ML for better handling of high-dimensional sparse data.
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...Spark Summit The document discusses the Everismoriarty platform, which simplifies the development of data analytics models using Apache Spark, facilitating collaboration between developers and data scientists. It outlines the platform's capabilities, including machine learning and data streaming, highlighting various work items and use cases such as dynamic supply networks and urban mobility. Additionally, it emphasizes the integration of Spark for efficient data processing and prototyping in developement environments.
Apache Arrow Flight: A New Gold Standard for Data Transport
Apache Arrow Flight: A New Gold Standard for Data TransportWes McKinney This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.
Spark Summit EU talk by Oscar Castaneda
Spark Summit EU talk by Oscar CastanedaSpark Summit The document discusses transforming Excel spreadsheets into Spark DataFrames by automatically translating Excel formulas into Spark code. It presents a program transformation pipeline that takes Excel formulas, parses them using a grammar and parser to generate a parse tree, and then generates Spark code from the parse tree. Key aspects covered include using an existing grammar and parser called XLParser to parse Excel formulas, treating Excel as a domain-specific language, and generating code by writing a pretty printer for the target Spark language. The talk concludes with a demonstration of the code generation approach.
Apache Pulsar: The Next Generation Messaging and Queuing System
Apache Pulsar: The Next Generation Messaging and Queuing SystemDatabricks This document discusses Apache Pulsar, an open-source distributed messaging and streaming platform. It provides concise summaries of Pulsar's key capabilities:
1) Pulsar provides messaging and streaming capabilities with a unified model and supports high throughput, low latency, availability, and durability.
2) It uses Apache BookKeeper for durable log storage and supports features like geo-replication, multi-tenancy, and lightweight compute functions.
3) Pulsar has been widely adopted with over 100 customers and has a large open-source community around it.
Apache Arrow at DataEngConf Barcelona 2018
Apache Arrow at DataEngConf Barcelona 2018Wes McKinney Wes McKinney is a leading open source developer who created Python's pandas library and now leads the Apache Arrow project. Apache Arrow is an open standard for in-memory analytics that aims to improve data sharing and reuse across systems by defining a common columnar data format and memory layout. It allows data to be accessed and algorithms to be reused across different programming languages with near-zero data copying. Arrow is being integrated into various data systems and is working to expand its computational libraries and language support.
End-to-End Data Pipelines with Apache Spark
End-to-End Data Pipelines with Apache SparkBurak Yavuz The document discusses building end-to-end data pipelines using Apache Spark, detailing the steps from understanding data to creating services and monitoring them. It highlights the Spark ecosystem, its contributions, and the tools available, including Spark SQL, MLlib, and Spark Streaming. The presentation also illustrates a scenario of implementing a recommendation engine for an e-commerce application using matrix factorization techniques.
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...Spark Summit The document discusses crucial insights into writing streaming applications, particularly focusing on monitoring, management, and preventing data loss. Key topics include the importance of orchestration, driver management, checkpointing, and using Spark Streaming for efficient data processing. It emphasizes strategies for graceful shutdowns and achieving exactly-once semantics in streaming applications.
Accelerate Your Apache Spark with Intel Optane DC Persistent Memory
Accelerate Your Apache Spark with Intel Optane DC Persistent MemoryDatabricks The document outlines Intel's offerings related to analytics solutions and persistent memory technologies, highlighting features like Intel Optane DC persistent memory and its compatibility with big data frameworks such as Spark. It discusses the performance of different memory configurations, emphasizing benchmarks and scenarios where persistent memory can significantly enhance workloads in data-intensive environments. Furthermore, it provides a technical overview of cache design, optimization strategies for Spark, and operational modes to improve data handling efficiency in various applications.
Spark Summit EU talk by Bas Geerdink
Spark Summit EU talk by Bas GeerdinkSpark Summit Get rid of traditional ETL processes and move to using Spark instead. Spark allows for parallel processing of data and can run on Hadoop systems where data is typically stored. It treats batch and streaming data equally and supports continuous processing without waiting for phases to complete. Spark code can extract, transform, and load data as well as perform machine learning tasks for data enrichment and prediction. This provides more flexibility than traditional ETL tools.
Simplifying Big Data Applications with Apache Spark 2.0
Simplifying Big Data Applications with Apache Spark 2.0Spark Summit Apache Spark 2.0 is a major new release that simplifies the Spark API and improves performance. Some key points:
1) It remains highly compatible with Spark 1.x while building on lessons learned to simplify the API with over 2000 patches from 280 contributors.
2) It introduces structured APIs like DataFrames that allow Spark to optimize queries via whole-stage code generation, providing up to 10x performance gains.
3) It launches a new higher-level streaming API called Structured Streaming that allows developers to write streaming jobs that behave like batch jobs and integrate easily with static data and batch jobs.
Apache Arrow Flight Overview
Apache Arrow Flight OverviewJacques Nadeau Arrow Flight is a proposed RPC layer for Apache Arrow that allows for efficient transfer of Arrow record batches between systems. It uses GRPC as the foundation to define streams of Arrow data that can be consumed in parallel across locations. Arrow Flight supports custom actions that can be used to build services on top of the generic API. By extending GRPC, Arrow Flight aims to simplify the creation of data applications while enabling high performance data transfer and locality awareness.
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...Databricks The document discusses the development of a hybrid online and semi-supervised learning framework using Apache Spark for real-time data processing, particularly in applications like malware detection and stock prediction. It outlines the challenges and solutions related to model retraining, accuracy maintenance, and low latency requirements in the context of real-time learning. The framework leverages unlabeled data to enhance predictions and includes components for feature extraction, model management, and various retraining policies.
Cooperative Task Execution for Apache Spark
Cooperative Task Execution for Apache SparkDatabricks The document discusses an innovative execution framework called Neptune for stream/batch applications that emphasizes cooperative task execution in Apache Spark. It highlights the advantages of unified applications, such as consistency and efficiency, while addressing challenges in scheduling diverse job requirements. Key features include suspendable tasks using coroutines and pluggable scheduling policies to minimize latency for sensitive jobs and optimize resource utilization.
Spark summit 2019 infrastructure for deep learning in apache spark 0425
Spark summit 2019 infrastructure for deep learning in apache spark 0425Wee Hyong Tok The document discusses Microsoft's infrastructure for deep learning using Apache Spark, covering topics such as data preparation, model training, and the challenges of distributed deep learning. It highlights performance improvements in model training times with advanced GPU setups and outlines design considerations for machine learning workflows at scale. The presentation also addresses job execution, resource management, and collaboration among data scientists and engineers in the context of machine learning pipelines.
Spark Summit San Francisco 2016 - Ali Ghodsi Keynote
Spark Summit San Francisco 2016 - Ali Ghodsi KeynoteDatabricks This document discusses Apache Spark and analytics in the cloud. It summarizes that there is a gap between the growth of data and ability to perform real-time analytics. It introduces Databricks as a cloud-hosted platform that can democratize big data by providing an integrated workspace with automated Apache Spark management and production-ready applications. Databricks also provides the first end-to-end security solution for Apache Spark to address challenges in securing analytics.
Spark Summit EU talk by Berni Schiefer
Spark Summit EU talk by Berni SchieferSpark Summit This document summarizes experiences using the TPC-DS benchmark with Spark SQL 2.0 and 2.1 on a large cluster designed for Spark. It describes the configuration of the "F1" cluster including its hardware, operating system, Spark, and network settings. Initial results show that Spark SQL 2.0 provides significant improvements over earlier versions. While most queries completed successfully, some queries failed or ran very slowly, indicating areas for further optimization.
Feature Hashing for Scalable Machine Learning with Nick Pentreath
Feature Hashing for Scalable Machine Learning with Nick PentreathSpark Summit The document discusses feature hashing as a method for scalable machine learning, highlighting its ability to efficiently encode high-dimensional sparse features without significant loss of accuracy. It covers the advantages and disadvantages of feature hashing, including its implementation in Apache Spark ML through tools like HashingTF and FeatureHasher. Future developments are anticipated to enhance its functionality within Spark ML, addressing challenges related to encoding and performance.
Partner Ecosystem Showcase for Apache Ranger and Apache Atlas
Partner Ecosystem Showcase for Apache Ranger and Apache AtlasDataWorks Summit This document provides information about Apache Ranger and Apache Atlas partner ecosystems and integration partnerships. It discusses Hortonworks' partner certification programs for SEC Ready and GOV Ready, and showcases partner technologies that have been integrated and certified with Apache Ranger and Apache Atlas, including from Talend, Arcadia Data, and Protegrity. The document also provides timelines and release information for Apache Ranger and Apache Atlas community development and integration with Hortonworks Data Platform (HDP) releases.
More Related Content
What's hot (20)
Art of Feature Engineering for Data Science with Nabeel Sarwar
Art of Feature Engineering for Data Science with Nabeel SarwarSpark Summit This document discusses feature engineering in data science, particularly in the context of machine learning at Comcast. It covers various processes involved in feature engineering such as selecting, cleaning, transforming, and enriching data, as well as methodologies for handling categorical and numerical variables. Additionally, it highlights the importance of domain knowledge, statistical correlation, and the challenges of dimensionality in optimizing machine learning models.
Parallelizing Large Simulations with Apache SparkR with Daniel Jeavons and Wa...
Parallelizing Large Simulations with Apache SparkR with Daniel Jeavons and Wa...Spark Summit The document outlines Shell's advancements in optimizing spare part inventory through advanced analytics, highlighting challenges in materials management, user confidence, and algorithm development. A pilot project in the Gulf of Mexico demonstrated significant cost savings and efficiency improvements, including reducing simulation times dramatically via upgraded computing methods. The initiative is set to scale globally, with potential for extensive benefits across Shell's operations.
Scaling Machine Learning with Apache Spark
Scaling Machine Learning with Apache SparkDatabricks The document discusses the use of Apache Spark for scaling machine learning, covering Spark's architecture, various ML training paradigms, and the application of different libraries like Spark MLlib and pandas. It emphasizes the importance of understanding when to utilize Spark for machine learning in a fast-paced environment and explores techniques for model training, tuning, and inference. Additionally, it introduces advanced topics such as hyperparameter optimization and the use of pandas UDFs for efficient data processing.
Feature Hashing for Scalable Machine Learning with Nick Pentreath
Feature Hashing for Scalable Machine Learning with Nick PentreathSpark Summit Nick Pentreath's presentation discusses feature hashing as a method for scalable machine learning using Apache Spark. It highlights the pros and cons of feature hashing, its application in text classification and ad click prediction, and includes performance results from experiments. Future work aims to improve the feature hashing method and integrate it into Spark ML for better handling of high-dimensional sparse data.
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...
Hiding Apache Spark Complexity for Fast Prototyping of Big Data Applications—...Spark Summit The document discusses the Everismoriarty platform, which simplifies the development of data analytics models using Apache Spark, facilitating collaboration between developers and data scientists. It outlines the platform's capabilities, including machine learning and data streaming, highlighting various work items and use cases such as dynamic supply networks and urban mobility. Additionally, it emphasizes the integration of Spark for efficient data processing and prototyping in developement environments.
Apache Arrow Flight: A New Gold Standard for Data Transport
Apache Arrow Flight: A New Gold Standard for Data TransportWes McKinney This document discusses how structured data is often moved inefficiently between systems, causing waste. It introduces Apache Arrow, an open standard for in-memory data, and how Arrow can help make data movement more efficient. Systems like Snowflake and BigQuery are now using Arrow to help speed up query result fetching by enabling zero-copy data transfers and sharing file formats between query processing and storage.
Spark Summit EU talk by Oscar Castaneda
Spark Summit EU talk by Oscar CastanedaSpark Summit The document discusses transforming Excel spreadsheets into Spark DataFrames by automatically translating Excel formulas into Spark code. It presents a program transformation pipeline that takes Excel formulas, parses them using a grammar and parser to generate a parse tree, and then generates Spark code from the parse tree. Key aspects covered include using an existing grammar and parser called XLParser to parse Excel formulas, treating Excel as a domain-specific language, and generating code by writing a pretty printer for the target Spark language. The talk concludes with a demonstration of the code generation approach.
Apache Pulsar: The Next Generation Messaging and Queuing System
Apache Pulsar: The Next Generation Messaging and Queuing SystemDatabricks This document discusses Apache Pulsar, an open-source distributed messaging and streaming platform. It provides concise summaries of Pulsar's key capabilities:
1) Pulsar provides messaging and streaming capabilities with a unified model and supports high throughput, low latency, availability, and durability.
2) It uses Apache BookKeeper for durable log storage and supports features like geo-replication, multi-tenancy, and lightweight compute functions.
3) Pulsar has been widely adopted with over 100 customers and has a large open-source community around it.
Apache Arrow at DataEngConf Barcelona 2018
Apache Arrow at DataEngConf Barcelona 2018Wes McKinney Wes McKinney is a leading open source developer who created Python's pandas library and now leads the Apache Arrow project. Apache Arrow is an open standard for in-memory analytics that aims to improve data sharing and reuse across systems by defining a common columnar data format and memory layout. It allows data to be accessed and algorithms to be reused across different programming languages with near-zero data copying. Arrow is being integrated into various data systems and is working to expand its computational libraries and language support.
End-to-End Data Pipelines with Apache Spark
End-to-End Data Pipelines with Apache SparkBurak Yavuz The document discusses building end-to-end data pipelines using Apache Spark, detailing the steps from understanding data to creating services and monitoring them. It highlights the Spark ecosystem, its contributions, and the tools available, including Spark SQL, MLlib, and Spark Streaming. The presentation also illustrates a scenario of implementing a recommendation engine for an e-commerce application using matrix factorization techniques.
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...Spark Summit The document discusses crucial insights into writing streaming applications, particularly focusing on monitoring, management, and preventing data loss. Key topics include the importance of orchestration, driver management, checkpointing, and using Spark Streaming for efficient data processing. It emphasizes strategies for graceful shutdowns and achieving exactly-once semantics in streaming applications.
Accelerate Your Apache Spark with Intel Optane DC Persistent Memory
Accelerate Your Apache Spark with Intel Optane DC Persistent MemoryDatabricks The document outlines Intel's offerings related to analytics solutions and persistent memory technologies, highlighting features like Intel Optane DC persistent memory and its compatibility with big data frameworks such as Spark. It discusses the performance of different memory configurations, emphasizing benchmarks and scenarios where persistent memory can significantly enhance workloads in data-intensive environments. Furthermore, it provides a technical overview of cache design, optimization strategies for Spark, and operational modes to improve data handling efficiency in various applications.
Spark Summit EU talk by Bas Geerdink
Spark Summit EU talk by Bas GeerdinkSpark Summit Get rid of traditional ETL processes and move to using Spark instead. Spark allows for parallel processing of data and can run on Hadoop systems where data is typically stored. It treats batch and streaming data equally and supports continuous processing without waiting for phases to complete. Spark code can extract, transform, and load data as well as perform machine learning tasks for data enrichment and prediction. This provides more flexibility than traditional ETL tools.
Simplifying Big Data Applications with Apache Spark 2.0
Simplifying Big Data Applications with Apache Spark 2.0Spark Summit Apache Spark 2.0 is a major new release that simplifies the Spark API and improves performance. Some key points:
1) It remains highly compatible with Spark 1.x while building on lessons learned to simplify the API with over 2000 patches from 280 contributors.
2) It introduces structured APIs like DataFrames that allow Spark to optimize queries via whole-stage code generation, providing up to 10x performance gains.
3) It launches a new higher-level streaming API called Structured Streaming that allows developers to write streaming jobs that behave like batch jobs and integrate easily with static data and batch jobs.
Apache Arrow Flight Overview
Apache Arrow Flight OverviewJacques Nadeau Arrow Flight is a proposed RPC layer for Apache Arrow that allows for efficient transfer of Arrow record batches between systems. It uses GRPC as the foundation to define streams of Arrow data that can be consumed in parallel across locations. Arrow Flight supports custom actions that can be used to build services on top of the generic API. By extending GRPC, Arrow Flight aims to simplify the creation of data applications while enabling high performance data transfer and locality awareness.
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...
An Online Spark Pipeline: Semi-Supervised Learning and Automatic Retraining w...Databricks The document discusses the development of a hybrid online and semi-supervised learning framework using Apache Spark for real-time data processing, particularly in applications like malware detection and stock prediction. It outlines the challenges and solutions related to model retraining, accuracy maintenance, and low latency requirements in the context of real-time learning. The framework leverages unlabeled data to enhance predictions and includes components for feature extraction, model management, and various retraining policies.
Cooperative Task Execution for Apache Spark
Cooperative Task Execution for Apache SparkDatabricks The document discusses an innovative execution framework called Neptune for stream/batch applications that emphasizes cooperative task execution in Apache Spark. It highlights the advantages of unified applications, such as consistency and efficiency, while addressing challenges in scheduling diverse job requirements. Key features include suspendable tasks using coroutines and pluggable scheduling policies to minimize latency for sensitive jobs and optimize resource utilization.
Spark summit 2019 infrastructure for deep learning in apache spark 0425
Spark summit 2019 infrastructure for deep learning in apache spark 0425Wee Hyong Tok The document discusses Microsoft's infrastructure for deep learning using Apache Spark, covering topics such as data preparation, model training, and the challenges of distributed deep learning. It highlights performance improvements in model training times with advanced GPU setups and outlines design considerations for machine learning workflows at scale. The presentation also addresses job execution, resource management, and collaboration among data scientists and engineers in the context of machine learning pipelines.
Spark Summit San Francisco 2016 - Ali Ghodsi Keynote
Spark Summit San Francisco 2016 - Ali Ghodsi KeynoteDatabricks This document discusses Apache Spark and analytics in the cloud. It summarizes that there is a gap between the growth of data and ability to perform real-time analytics. It introduces Databricks as a cloud-hosted platform that can democratize big data by providing an integrated workspace with automated Apache Spark management and production-ready applications. Databricks also provides the first end-to-end security solution for Apache Spark to address challenges in securing analytics.
Spark Summit EU talk by Berni Schiefer
Spark Summit EU talk by Berni SchieferSpark Summit This document summarizes experiences using the TPC-DS benchmark with Spark SQL 2.0 and 2.1 on a large cluster designed for Spark. It describes the configuration of the "F1" cluster including its hardware, operating system, Spark, and network settings. Initial results show that Spark SQL 2.0 provides significant improvements over earlier versions. While most queries completed successfully, some queries failed or ran very slowly, indicating areas for further optimization.
Viewers also liked (20)
Feature Hashing for Scalable Machine Learning with Nick Pentreath
Feature Hashing for Scalable Machine Learning with Nick PentreathSpark Summit The document discusses feature hashing as a method for scalable machine learning, highlighting its ability to efficiently encode high-dimensional sparse features without significant loss of accuracy. It covers the advantages and disadvantages of feature hashing, including its implementation in Apache Spark ML through tools like HashingTF and FeatureHasher. Future developments are anticipated to enhance its functionality within Spark ML, addressing challenges related to encoding and performance.
Partner Ecosystem Showcase for Apache Ranger and Apache Atlas
Partner Ecosystem Showcase for Apache Ranger and Apache AtlasDataWorks Summit This document provides information about Apache Ranger and Apache Atlas partner ecosystems and integration partnerships. It discusses Hortonworks' partner certification programs for SEC Ready and GOV Ready, and showcases partner technologies that have been integrated and certified with Apache Ranger and Apache Atlas, including from Talend, Arcadia Data, and Protegrity. The document also provides timelines and release information for Apache Ranger and Apache Atlas community development and integration with Hortonworks Data Platform (HDP) releases.
Ibm watson
Ibm watsonVivek Mohan IBM Watson has several advantages for data analysis: it allows for easy sharing and collaboration; provides descriptive, diagnostic, predictive, and prescriptive analysis; and uses pattern discovery technology. However, it also has some limitations: it can only take datasets with 12 or fewer columns; does not understand relational databases well; and has limited functions compared to some other tools. Datawatch Monarch is a self-service data preparation tool that allows business users to work directly with data on-premise without needing scripting skills. It can extract and prepare data from various sources for analysis in tools like IBM Watson Analytics. However, its ability to handle large datasets may be limited in the trial version.
빅데이터윈윈 컨퍼런스_데이터시각화자료
빅데이터윈윈 컨퍼런스_데이터시각화자료ABRC_DATA This document discusses data visualization and visual analytics. It begins by explaining how visualization can help make sense of large amounts of data by transforming data into information. Visualization includes techniques from scientific visualization, information visualization, and visual analytics. The goals of visualization are analysis, presentation, and aiding decision making. Visual analytics combines interactive visual interfaces with algorithms to help people analyze complex big data.
Softnix Messaging Server
Softnix Messaging ServerSoftnix Technology Softnix Messaging Server Enterprise offers a range of collaboration features and mobile support, including management, workflow, and custom email functionalities. It integrates with cloud systems and provides extensive email management capabilities such as quota management, aliasing, filtering, and multi-domain support. The system also includes various collaboration tools like calendars, task management, and project management, as well as enhanced security options like email encryption.
Using Big Data to Transform Your Customer’s Experience - Part 1
Using Big Data to Transform Your Customer’s Experience - Part 1
Cloudera, Inc. Cloudera's panel discussion highlights the importance of leveraging big data for enhancing customer experience through analytics and insights. Key speakers emphasize the need for real-time analysis, data sharing, and self-service business intelligence to improve customer retention and satisfaction. Solutions presented include transitioning to more efficient data systems that enable better insights and reduced costs.
Webinar - Sehr empfehlenswert: wie man aus Daten durch maschinelles Lernen We...
Webinar - Sehr empfehlenswert: wie man aus Daten durch maschinelles Lernen We...Cloudera, Inc. Das Dokument behandelt die Schaffung von Wert aus Daten durch maschinelles Lernen und die Rolle von Open Data Science im Unternehmenskontext. Es wird erläutert, wie Unternehmen wie Cloudera eine Plattform für effektives Data Engineering und Datenanalyse bieten, um Datenwissenschaftler zu unterstützen und maschinelles Lernen in großem Maßstab zu ermöglichen. Zudem werden Herausforderungen und Lösungsansätze im Umgang mit Daten sowie konkrete Anwendungsbeispiele wie Retail-Clustering und Empfehlungssysteme vorgestellt.
Apache Spark—Apache HBase Connector: Feature Rich and Efficient Access to HBa...
Apache Spark—Apache HBase Connector: Feature Rich and Efficient Access to HBa...Spark Summit The document presents the Apache Spark - Apache HBase connector (SHC), highlighting its efficient integration with Spark for enhanced access to HBase through Spark SQL. It details the architecture, implementation, and high-performance features such as partition pruning and data locality. Additionally, it provides usage instructions and examples, along with references for further information.
Put Alternative Data to Use in Capital Markets
Put Alternative Data to Use in Capital Markets
Cloudera, Inc. This document discusses alternative data in capital markets. It provides an overview of alternative data sources like social media, satellite imagery, and location data. It also describes how firms are using alternative data to enhance traditional analysis and develop new investment strategies. The document notes that most alternative data users have seen returns from using this data. However, accessing and analyzing large alternative data sets remains a challenge. It promotes the use of data platforms and visual analytics to more effectively ingest, store, and operationalize alternative data.
Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData, An...
Real-Time Analytics Visualized w/ Kafka + Streamliner + MemSQL + ZoomData, An...confluent The document summarizes a presentation about using Kafka, Streamliner, MemSQL and ZoomData for real-time analytics visualization. It shows an initial setup with one producer and queue feeding into Kafka, then adding a sink to an in-memory SQL database and real-time visualization consumer. It asks questions about ensuring the system is resilient, handles bad data and schema evolution, maintains consistency across visualization layers, and ability to scale throughput, concurrency and size.
Softnix Security Data Lake
Softnix Security Data Lake Softnix Technology The document discusses using a security data lake built on a big data platform to improve cyber defense visibility and analytics capabilities. It notes that traditional SIEM tools have limitations in scalability and openness that can be addressed using Hadoop and related big data technologies. Examples are given of how security data lakes can optimize SIEM systems by enabling greater data volumes, processing power, and analytics flexibility in a more cost effective and deployment flexible manner. The presentation concludes by describing Softnix's security data lake product and its capabilities for data collection, enrichment, analysis, search, and visualization.
The Fast Path to Building Operational Applications with Spark
The Fast Path to Building Operational Applications with SparkSingleStore Nikita Shamgunov gave a presentation about using MemSQL and Spark together. MemSQL is a scalable operational database that can handle petabytes of data with high concurrency. It offers real-time capabilities and compatibility with tools like Spark, Kafka, and ETL/BI tools. The MemSQL Spark Connector allows bidirectional transfer of data between Spark and MemSQL tables for use cases like operationalizing models in Spark, stream/event processing, and live dashboards. Case studies showed customers gaining 10x faster data refresh times and performing entity resolution at scale for fraud detection.
Building the Ideal Stack for Real-Time Analytics
Building the Ideal Stack for Real-Time AnalyticsSingleStore This document discusses building an ideal real-time analytics stack. It promotes the MemSQL database platform as being ranked #1 for operational data warehousing and being able to handle real-time analytics at massive scales. The ideal stack presented includes components for message queues, a transformation tier, data persistence with MemSQL, and real-time visualization. Use cases discussed include processing web logs, mobile apps, IoT data, and an energy company's drilling sensor data.
The Evolution of Data Architecture
The Evolution of Data ArchitectureWei-Chiu Chuang The document discusses the evolution of data architecture, emphasizing the rise of distributed systems and big data technologies due to stagnant hardware improvements and accelerating data generation. It explores the development of platforms like Hadoop, open source software's significance in the data economy, and various applications of big data across industries, including predictive maintenance and smart city management. Key takeaways include the importance of treating data as a vital resource and the shift towards open source solutions in technology infrastructure.
CWIN17 Frankfurt / Cloudera
CWIN17 Frankfurt / ClouderaCapgemini The document discusses Cloudera and its platform for machine learning and analytics. It provides an overview of Cloudera's customers, partners, products, and services. Specifically, it discusses how Cloudera helps customers drive insights, connect products and services through IoT, and protect businesses. Examples are given of how Cloudera has helped customers in manufacturing, telematics, and connected vehicles. The platform is described as running anywhere, being scalable and enterprise grade.
Spark meetup - Zoomdata Streaming
Spark meetup - Zoomdata StreamingZoomdata This document discusses interactive data visualization powered by Spark streaming. It describes how Zoomdata allows users to visualize streaming data in real-time as new data is delivered. The key challenges of streaming data like time, frequency, retention and synchronization are addressed. Zoomdata receives streaming data via Kafka or JMS, processes it using Spark Streaming in a single JVM, and stores the data in buffers like MongoDB. This allows for interactive data visualizations that update in real-time as new streaming data is processed. The document also outlines technologies used, how the system scales out, benefits, and includes a demo of streaming data from Twitter to MemSQL and Solr sinks using Spark Streaming.
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice Machine
Spark as part of a Hybrid RDBMS Architecture-John Leach Cofounder Splice MachineData Con LA Splice Machine offers a real-time database solution that integrates OLTP and OLAP capabilities, powered by Hadoop and Spark, to enable immediate decision-making using up-to-date data. Their platform eliminates costly ETL processes and allows faster analytics without the need for extensive system rewrites, promoting efficiency across various workloads. With advanced features like in-memory computing and high scalability, Splice Machine targets applications in sectors such as life sciences, digital marketing, and supply chain optimization.
Zoomdata
ZoomdataVivek Mohan Zoom Data is a data visualization tool that provides the following key capabilities:
1) It allows users to create near real-time interactive dashboards using live data at scale and stream data to real-time reports.
2) It supports exporting data to PDF and Excel formats and has options for multi-tenancy, one-click deployment on AWS, and is designed for large touchscreens.
3) While it is architecturally strong and can scale out well, it also has some limitations like limited front-end functionality, lack of offline analysis and mobile apps, and limited audit functionality.
Security implementation on hadoop
Security implementation on hadoopWei-Chiu Chuang The document provides a detailed overview of security implementation on Hadoop, highlighting essential guidelines for data protection measures such as user authentication, access control, data encryption, and auditing. It emphasizes the need for regulation compliance, especially concerning GDPR, and outlines practices for ensuring secure data management in both production and disaster recovery scenarios. Additionally, the document discusses the role of Cloudera's professional services in tailoring security implementations for specific organizational needs.
Cloudera and Qlik: Big Data Analytics for Business
Cloudera and Qlik: Big Data Analytics for BusinessData IQ Argentina This document discusses how Qlik and Cloudera can work together to enable big data analytics for businesses. It describes how QlikView integrates with Cloudera's Enterprise Data Hub to allow users to analyze both big data from the data hub as well as smaller datasets. Users can connect to big data sources via ODBC and Cloudera's driver. This gives business users the ability to apply QlikView's analytics and visualization tools to vast amounts of data stored in Cloudera. The combination provides businesses with speed, agility, and the ability to access and analyze all of their data to drive better decisions.
Ad
Similar to MatFast: In-Memory Distributed Matrix Computation Processing and Optimization Based on Spark SQL Yanbo Liang and Mingie Tang (20)
Enterprise data science at scale
Enterprise data science at scaleCarolyn Duby The document outlines a presentation about enterprise data science at scale. The agenda includes networking, announcements, a main presentation on introducing data science at scale, building and deploying models collaboratively, training models with all data, and putting models to work in streaming applications, followed by Q&A. The main presentation will discuss challenges of data science like data in multiple locations, too many tools, difficulty sharing insights and operationalizing models, and limitations of desktop. It will introduce Apache Spark as a distributed processing platform, Jupyter and Zeppelin notebooks, and deploying models as a virtual service. A demo will use customer churn data to train a random forest model to predict churn, and deploy it to production to deliver insights.
Unlocking insights in streaming data
Unlocking insights in streaming dataCarolyn Duby The document outlines a presentation on streaming analytics, focusing on the value of real-time data insights and the challenges of managing variable streams. It introduces various components and tools, including a schema registry for managing event schemas, and the Streaming Analytics Manager (SAM) for building and managing streaming applications with ease. Additionally, it discusses the Druid analytics engine for real-time data ingestion and querying, emphasizing its scalability and unique capabilities.
SparkR Best Practices for R Data Scientists
SparkR Best Practices for R Data ScientistsDataWorks Summit The document outlines best practices for using SparkR to enhance data science workflows, highlighting the interoperability between R and Apache Spark. It discusses the advantages and limitations of R, and details specific operations like data wrangling, sampling algorithms, and user-defined functions that enable efficient data processing in large datasets. Future improvements for SparkR include enhancing performance for data collection and UDFs, along with expanded machine learning capabilities.
SparkR best practices for R data scientist
SparkR best practices for R data scientistDataWorks Summit The document discusses the integration of R with Apache Spark through SparkR, emphasizing its applications in data science workflows for large datasets and machine learning. It highlights the advantages of using R, such as its open-source nature and extensive package ecosystem, while addressing limitations when handling big data. Future directions for improvement in SparkR are also outlined, including enhanced performance and scalability of machine learning algorithms.
Apache Spark Crash Course
Apache Spark Crash CourseDataWorks Summit The document provides an overview of Apache Spark, including its role as a unified data processing engine suitable for big data workloads and its capabilities for data manipulation through Spark SQL, Spark Streaming, and machine learning with Spark MLlib. It discusses the architecture and features of Spark, highlighting its efficiency in processing large data sets and real-time analytics, as well as the integration with tools like Apache Zeppelin for interactive data exploration. Additionally, the document touches on the evolution of Spark features and the Hortonworks Data Platform.
Integrate SparkR with existing R packages to accelerate data science workflows
Integrate SparkR with existing R packages to accelerate data science workflowsArtem Ervits This document discusses integrating SparkR with existing R packages to accelerate data science workflows. It provides an introduction to R and SparkR, describes typical data science workflows, and gives examples of how SparkR can be used with R for tasks like distributed data wrangling, partitioned aggregation, and large-scale machine learning. The goal is to leverage both Spark's distributed processing capabilities and R's rich ecosystem of packages.
Apache Phoenix and Apache HBase: An Enterprise Grade Data Warehouse
Apache Phoenix and Apache HBase: An Enterprise Grade Data WarehouseJosh Elser The document discusses the integration of Apache Phoenix and Apache HBase as an enterprise-grade data warehouse solution. It covers use cases, optimization strategies, architecture, and the performance features of this combination, including scalability, reliability, and manageability. Additionally, it highlights the capabilities of the Apache Phoenix Query Server and provides insights for effective schema design and query optimization.
Apache Phoenix + Apache HBase
Apache Phoenix + Apache HBaseDataWorks Summit/Hadoop Summit The document summarizes Apache Phoenix and HBase as an enterprise data warehouse solution. It discusses how Phoenix provides OLTP and analytics capabilities over HBase. It then covers various use cases where companies are using Phoenix and HBase, including for web analytics and time series data. Finally, it discusses optimizations that can be made to the schema design, queries, and writes in Phoenix to improve performance.
Crash Course HS16Melb - Hands on Intro to Spark & Zeppelin
Crash Course HS16Melb - Hands on Intro to Spark & Zeppelin DataWorks Summit/Hadoop Summit This document provides an agenda and overview for a hands-on introductory course on Spark and Zeppelin. The agenda includes a quick demo, overview of Spark and Zeppelin, a 1 hour lab, discussion of Spark 2.0 features, and a Q&A session. The overview sections explain key Spark concepts like RDDs, DataFrames, and MLlib as well as how Spark SQL, Streaming, and GraphX work. It also introduces the Apache Zeppelin notebook platform and Hortonworks Data Platform sandbox for experimenting with Spark and Hadoop technologies.
Machine Learning With Spark
Machine Learning With SparkShivaji Dutta This document provides an overview of machine learning concepts and techniques using Apache Spark. It begins with introducing machine learning and describing supervised and unsupervised learning. Then it discusses Spark and how it can be used for large-scale machine learning tasks through its MLlib library and GraphX API. Several examples of machine learning applications are presented, such as classification, regression, clustering, and graph analytics. The document concludes with demonstrating machine learning algorithms in Spark.
Data Science Crash Course
Data Science Crash CourseDataWorks Summit The document is a comprehensive overview of data science, machine learning, and deep learning, outlining the key drivers behind the recent AI explosion, application areas across various industries, and important machine learning concepts and algorithms. It emphasizes the significance of data in model training and the various phases involved in data preparation, model training, and evaluation. Additionally, it discusses the limitations of deep learning, addresses common challenges, and provides insights on leveraging AI to drive business value.
An Overview on Optimization in Apache Hive: Past, Present Future
An Overview on Optimization in Apache Hive: Past, Present FutureDataWorks Summit/Hadoop Summit The document outlines the evolution of optimization in Apache Hive, highlighting its journey from batch processing to supporting interactive queries and SQL standards. Key advancements include the integration of Apache Calcite for logical optimization, the implementation of various execution engines like Apache Tez and Spark, and the introduction of features such as cost-based join ordering and dynamic runtime filtering. Future developments focus on refining cost models for more effective query planning and leveraging materialized views to enhance query performance.
Spark and Hadoop Perfect Togeher by Arun Murthy
Spark and Hadoop Perfect Togeher by Arun MurthySpark Summit Spark and Hadoop are perfectly together. Spark is a key tool in Hadoop's toolbox that provides elegant developer APIs and accelerates data science and machine learning. It can process streaming data in real-time for applications like web analytics and insurance claims processing. The future of Spark and Hadoop includes innovating the core technologies, providing seamless data access across data platforms, and further accelerating data science tools and libraries.
Spark Summit EMEA - Arun Murthy's Keynote
Spark Summit EMEA - Arun Murthy's KeynoteHortonworks Spark and Hadoop are perfectly together. Spark is a key tool in Hadoop's toolbox that provides elegant developer APIs and accelerates data science and machine learning. It can process streaming data in real-time for applications like web analytics and insurance claims processing. The future of Spark and Hadoop includes innovating the core technologies, providing seamless data access across data platforms, and further accelerating data science tools and libraries.
Streamline - Stream Analytics for Everyone
Streamline - Stream Analytics for EveryoneDataWorks Summit/Hadoop Summit The document outlines the Streaming Analytics Manager (SAM), a platform for designing and managing streaming analytics applications using a user-friendly graphical interface. SAM addresses challenges in building streaming applications, such as the need for specialized skills and the complexity of programming, by enabling users to focus on business logic while abstracting underlying streaming technologies. The document also highlights SAM's key capabilities, integration with schema registry, and extensibility options, positioning it as a solution to streamline the development and operational efficiencies of streaming applications.
Schema Registry & Stream Analytics Manager
Schema Registry & Stream Analytics ManagerSriharsha Chintalapani The document discusses Hortonworks' Streaming Analytics Manager (SAM) and Schema Registry, crucial components for managing real-time data processing and analytics applications. It details SAM's capabilities in simplifying the design and deployment of streaming applications, while the Schema Registry provides a centralized repository for schema management, enhancing interoperability and efficiency. Key features include support for various streaming engines, modular services, and extensibility through custom processors and notification systems.
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source ToolsEsther Vasiete The document outlines pivotal data science's approach to data science at scale using MPP databases, emphasizing real-world applications such as predictive maintenance for connected vehicles, insurance claim predictions, and customer churn analysis. It discusses the use of an open source data science toolkit and includes technical details about machine learning, data processing, and inference models. Furthermore, it highlights the integration of structured and unstructured data analytics along with the development of user-defined functions for in-database analytics.
Apache Spark Crash Course
Apache Spark Crash CourseDataWorks Summit/Hadoop Summit This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.
#HSTokyo16 Apache Spark Crash Course
#HSTokyo16 Apache Spark Crash Course DataWorks Summit/Hadoop Summit This document provides an overview and crash course on Apache Spark and related big data technologies. It discusses the history and components of Spark including Spark Core, SQL, Streaming, and MLlib. It also discusses data sources, challenges of big data, and how Spark addresses them through its in-memory computation model. Finally, it introduces Apache Zeppelin for interactive notebooks and the Hortonworks Data Platform sandbox for experimenting with these technologies.
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies DataWorks Summit/Hadoop Summit Apache Atlas provides metadata services and a centralized metadata repository for Hadoop platforms. It aims to enable data governance across structured and unstructured data through hierarchical taxonomies. Upcoming features include expanded dataset lineage tracking and integration with Apache Kafka and Ranger for dynamic access policy management. Challenges of big data management include scaling traditional tools to handle large volumes of entities and metadata, and Atlas addresses this through its decentralized and metadata-driven approach.
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies
Apache Atlas: Why Big Data Management Requires Hierarchical Taxonomies DataWorks Summit/Hadoop Summit
Ad
More from Spark Summit (20)
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang
FPGA-Based Acceleration Architecture for Spark SQL Qi Xie and Quanfu Wang Spark Summit The document discusses an FPGA-based acceleration architecture for Spark SQL, detailing its components and advantages such as reduced execution times and CPU utilization improvements. It presents a performance comparison showing a significant speedup in query execution from 86 seconds to 44 seconds with FPGA acceleration. Additionally, it outlines future work, including expanding SQL expression support and configuring the FPGA SQL acceleration engine for more versatile application.
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...Spark Summit Vegas is a visualization library for Scala, designed to fill the gap in the Scala/Spark ecosystem similar to Matplotlib in Python. It allows for declarative statistical visualizations with built-in operations like filtering and aggregation, enhancing productivity in machine learning experimentation. The library also supports various plot types, interactive selections, and is designed for seamless integration within notebooks.
Improving Traffic Prediction Using Weather Data with Ramya Raghavendra
Improving Traffic Prediction Using Weather Data with Ramya RaghavendraSpark Summit The document outlines a research project by IBM to improve traffic prediction using weather data, highlighting the challenges of data collection and modeling. It details the machine learning processes used, including various algorithms like ARIMA and LSTM, to predict traffic flows based on historical traffic and weather data across multiple cities. The study demonstrates significant reductions in prediction errors, emphasizing the potential of predictive traffic systems to enhance commuting efficiency.
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...
A Tale of Two Graph Frameworks on Spark: GraphFrames and Tinkerpop OLAP Artem...Spark Summit The document compares two graph analytics frameworks, Tinkerpop and GraphFrames, used with Apache Spark, highlighting their distinct approaches. Tinkerpop is a flexible OLAP framework that operates on a node-centric model, while GraphFrames offers a relational structure and integrates closely with Spark's DataFrame API for optimization. The choice of framework depends on the complexity of traversals and the need for interoperability with external data sources.
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...
No More Cumbersomeness: Automatic Predictive Modeling on Apache Spark Marcin ...Spark Summit The document discusses automatic predictive modeling on Apache Spark, emphasizing the challenges and solutions associated with predictive modeling in industry. It highlights the advantages of the authors' technology that combines Spark with native machine learning engines to automate the process, providing accurate results in a fraction of the time typically required. The evaluation of their approach demonstrates competitive accuracy and efficiency across various datasets.
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim DowlingSpark Summit The document discusses advancements in deep learning, particularly the integration of TensorFlow and Spark for distributed training, highlighting the need for robust data pipelines and hardware configurations. It outlines various models and frameworks, such as Horovod and TensorFlow on Spark, to enhance efficiency in training using multiple GPUs and optimization techniques. Additionally, it presents the AI hierarchy of needs and the role of Hopsworks in supporting machine learning workflows.
Apache Spark and Tensorflow as a Service with Jim Dowling
Apache Spark and Tensorflow as a Service with Jim DowlingSpark Summit The document discusses advances in distributed deep learning (DDL), highlighting the use of technologies such as TensorFlow and Spark for training large models efficiently. It details the AI hierarchy of needs, the integration of GPU resources, and various methods like allreduce for optimizing training times. Additionally, it presents the capabilities of Hopsworks, a platform for managing AI workflows and resources.
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...
MMLSpark: Lessons from Building a SparkML-Compatible Machine Learning Library...Spark Summit The document discusses MMLSpark, a machine learning library designed for Apache Spark, detailing its goals, architecture, and lessons learned during development. Key aspects include enhancing the Spark ecosystem through integration, improving model management, and reducing overhead from user-defined functions while ensuring composability. An example use case highlights its application in automatic image classification for snow leopard conservation, saving time and increasing accuracy.
Next CERN Accelerator Logging Service with Jakub Wozniak
Next CERN Accelerator Logging Service with Jakub WozniakSpark Summit The document outlines the architecture and functionality of the Next CERN Accelerator Logging Service (nxcals), which improves upon the previous logging system by addressing scalability issues and enabling big data capabilities. It discusses the technologies used, such as Kafka, Hadoop, and Spark, along with the structured way to handle metadata and changing record schemas over time. Additionally, it describes various APIs for data ingestion and extraction to facilitate efficient data access and analysis.
How Nielsen Utilized Databricks for Large-Scale Research and Development with...
How Nielsen Utilized Databricks for Large-Scale Research and Development with...Spark Summit Nielsen used Databricks to test new digital advertising rating methodologies on a large scale. Databricks allowed Nielsen to run analyses on thousands of advertising campaigns using both small panel data and large production data. This identified edge cases and performance gains faster than traditional methods. Using Databricks reduced the time required to test and deploy improved rating methodologies to benefit Nielsen's clients.
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...
Spline: Apache Spark Lineage not Only for the Banking Industry with Marek Nov...Spark Summit The document discusses the usage of Apache Spark lineage in Barclays Africa Group Limited to meet regulatory compliance and enhance data clarity. It introduces the Spline tool that enables visualization of Spark execution plans, emphasizing its functionality and future development plans. The presentation includes a demonstration of a use case analyzing beer consumption per capita in correlation with GDP across various countries.
Goal Based Data Production with Sim Simeonov
Goal Based Data Production with Sim SimeonovSpark Summit Simeon Simeonov discusses the concept of goal-based data production within the context of a data-powered culture, emphasizing the need for agile data production methods utilizing machine learning and AI. He highlights the complexity and inflexibility of traditional SQL abstractions and the advantages of a generative approach that enhances metadata queries and allows for real-time context resolution. The document advocates for a self-service model for data scientists and business users, aiming to improve productivity, flexibility, and collaboration through open-source solutions.
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...
Preventing Revenue Leakage and Monitoring Distributed Systems with Machine Le...Spark Summit Movile implemented machine learning to monitor their subscription and billing platform, utilizing data analysis to predict and prevent revenue leakage effectively. The project involved extensive data handling with millions of daily billing requests and various machine learning models, achieving significant accuracy improvements. Key outcomes include saving over $3 million and reducing recovery time from 6 hours to 1 hour, demonstrating a successful application of machine learning in operational efficiency.
Getting Ready to Use Redis with Apache Spark with Dvir Volk
Getting Ready to Use Redis with Apache Spark with Dvir VolkSpark Summit The document outlines a presentation on using Redis for real-time machine learning, focusing on Redis-ML integrated with Apache Spark. It covers the capabilities of Redis as a key-value store, the functionalities of Redis-ML for model serving, and a case study involving a recommendation system. The concluding remarks emphasize the advantages of combining Spark for training with Redis for efficient model serving, highlighting reduced resource costs and simplified machine learning lifecycle.
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...Spark Summit The document discusses advancements in deduplication and author disambiguation of scholarly records using supervised models and content encoders within the Elsevier platform, emphasizing the importance of efficient data processing techniques. It outlines the challenges and solutions for managing peer-reviewed publications in Scopus, focusing on machine learning and cloud scalability for real-time document analysis. The implementation of algorithms like word2vec for document encoding aims to improve recommendation systems and personalized user experiences across various academic entities.
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...
Indicium: Interactive Querying at Scale Using Apache Spark, Zeppelin, and Spa...Spark Summit This document discusses Indicium, a system for enabling interactive querying at scale on Spark. It describes a unified data platform using Spark for storing and processing data, as well as enabling scheduled jobs, BI reports, and real-time lookups. It then discusses two parts of Indicium: (1) a managed context pool using Apache Zeppelin and Spark Job Server for multi-user SQL queries, and (2) a smart query scheduler for dynamic scheduling, load balancing, and high availability of queries. The smart scheduler aims to improve on the managed context pool by addressing issues like FIFO scheduling and lack of failure handling.
Apache Spark-Bench: Simulate, Test, Compare, Exercise, and Yes, Benchmark wit...
Apache Spark-Bench: Simulate, Test, Compare, Exercise, and Yes, Benchmark wit...Spark Summit The document presents an overview of spark-bench, a comprehensive benchmarking suite designed for the Spark data analytics platform, detailing its history, functionality, and how it facilitates performance testing using different data formats like CSV and Parquet. It highlights use cases for various user types, structural details of the project, and showcases examples of benchmarking tasks, including workload configurations and time management for performance comparisons. The content also includes insights into custom workloads and configurations to enhance benchmarking results.
Variant-Apache Spark for Bioinformatics with Piotr Szul
Variant-Apache Spark for Bioinformatics with Piotr SzulSpark Summit VariantSpark is a tool for bioinformatics that utilizes Apache Spark to analyze genomics data and identify disease genes in large cohorts. It addresses the challenges posed by extensive data dimensions through innovative approaches in machine learning. The presentation also emphasizes the importance of collaborative research and efficient data processing in genomic studies.
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed
Running Spark Inside Containers with Haohai Ma and Khalid Ahmed Spark Summit The document discusses IBM Spectrum Computing's approach to running Spark within containers, focusing on the benefits of using containers for resource management, isolation, and ease of deployment. It outlines the architecture for deploying Spark workloads on Kubernetes, detailing the structure of Spark instance groups and the dynamic scalability based on workload demands. Future work includes integrating Kubernetes batch workload scheduling and comparing its performance with other Spark on Kubernetes solutions.
Best Practices for Using Alluxio with Apache Spark with Gene Pang
Best Practices for Using Alluxio with Apache Spark with Gene PangSpark Summit The document discusses best practices for using Alluxio with Spark, highlighting how Alluxio streamlines data access across various storage systems without requiring application changes. It provides case studies showing improved performance and operational efficiency through Alluxio, including data query enhancements resulting in up to 300x speed improvements. Additionally, the document outlines the Alluxio architecture and how to deploy it effectively with Spark for optimal I/O performance.
Recently uploaded (20)
reporting monthly for genset & Air Compressor.pptx
reporting monthly for genset & Air Compressor.pptxdacripapanjaitan Genset data
spm monthly by sap
- GENSET DATA MONTHLY
- Analyst data for genset
- preventive genset
- warming up genset
Compressor Data
- Material genset
- PM Compressor
- logsheet compressor
- Data compressor
Pause Travail 22 Hostiou Girard 12 juin 2025.pdf
Pause Travail 22 Hostiou Girard 12 juin 2025.pdfInstitut de l'Elevage - Idele Diaporama de la Pause travail 22 sur les rizières au Ghana - 12 juin 2025 - Intervenants Nathalie Hostiou et Pierre Girard
Boost Business Efficiency with Professional Data Entry Services
Boost Business Efficiency with Professional Data Entry Serviceseloiacs eloiacs Boost Business Efficiency with Professional Data Entry Services
In today’s digital-first world, businesses generate and handle massive amounts of data every day — customer records, sales data, inventory logs, survey results, and much more. But raw data has no value unless it is well-organized, accurate, and easily accessible. That’s where professional data entry services come in.
By outsourcing data entry tasks to experts, businesses can streamline operations, reduce manual errors, and improve overall efficiency — while focusing their internal resources on core activities like growth and customer engagement.
What Are Data Entry Services?
Data entry services refer to the process of converting information from various formats (handwritten, scanned, PDF, image, or audio) into structured, digital formats such as Excel sheets, CRM databases, or cloud storage systems. This work may be done online or offline, manually or using automation tools, depending on the client’s requirements.
Popular Data Entry Services Include:
Manual data entry from paper documents or scanned files
Online data entry directly into websites, forms, or portals
Offline data entry in formats like MS Word, Excel, or custom software
Product data entry for eCommerce platforms like Shopify, Amazon, and Flipkart
Document and image conversion into editable formats
Data cleansing and validation to remove duplicates and fix errors
Remote data entry support for real-time operations
CRM and ERP data management
These services are essential for organizing business data and making it usable for analysis, reporting, and decision-making.
Who Can Benefit from Data Entry Services?
Outsourcing data entry is not limited to any one industry — it's a universal need for businesses of all types and sizes. Here are some examples:
eCommerce Businesses – For managing product catalogs, inventory updates, pricing, and customer orders.
Healthcare Providers – For digitizing patient records, prescriptions, and insurance documents.
Education Institutes – To maintain student records, exam results, and staff data.
Financial Services – For processing invoices, bank statements, transaction records.
Real Estate Companies – To handle property listings, legal paperwork, client records.
Research & Marketing Firms – To compile survey data, leads, and analytics reports.
Even startups and freelancers often require virtual data entry services to stay organized and competitive.
Top Benefits of Outsourcing Data Entry Services
Outsourcing data entry work to a professional company or virtual assistant offers multiple benefits — whether you're running a small business or managing a large enterprise.
1. Reduced Costs
Maintaining an in-house data entry team means salaries, hardware, training, and software expenses. Outsourcing eliminates these costs and provides flexible, pay-as-you-go solutions.
B.Tech Business Plan mena countries and europe
B.Tech Business Plan mena countries and europeAhmedSelim238929 B.Tech Business Plan mena countries and europe
最新版美国亚利桑那大学毕业证（UA毕业证书）原版定制
最新版美国亚利桑那大学毕业证（UA毕业证书）原版定制Taqyea 鉴于此，定制亚利桑那大学学位证书提升履历【q薇1954292140】原版高仿亚利桑那大学毕业证(UA毕业证书)可先看成品样本【q薇1954292140】帮您解决在美国亚利桑那大学未毕业难题，美国毕业证购买，美国文凭购买，【q微1954292140】美国文凭购买，美国文凭定制，美国文凭补办。专业在线定制美国大学文凭，定做美国本科文凭，【q微1954292140】复制美国The University of Arizona completion letter。在线快速补办美国本科毕业证、硕士文凭证书，购买美国学位证、亚利桑那大学Offer，美国大学文凭在线购买。
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
【复刻一套亚利桑那大学毕业证成绩单信封等材料最强攻略,Buy The University of Arizona Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
亚利桑那大学成绩单能够体现您的的学习能力，包括亚利桑那大学课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
定制OCAD学生卡加拿大安大略艺术与设计大学成绩单范本,OCAD成绩单复刻
定制OCAD学生卡加拿大安大略艺术与设计大学成绩单范本,OCAD成绩单复刻taqyed 2025年极速办安大略艺术与设计大学毕业证【q薇1954292140】学历认证流程安大略艺术与设计大学毕业证加拿大本科成绩单制作【q薇1954292140】海外各大学Diploma版本，因为疫情学校推迟发放证书、证书原件丢失补办、没有正常毕业未能认证学历面临就业提供解决办法。当遭遇挂科、旷课导致无法修满学分，或者直接被学校退学，最后无法毕业拿不到毕业证。此时的你一定手足无措，因为留学一场，没有获得毕业证以及学历证明肯定是无法给自己和父母一个交代的。
【复刻安大略艺术与设计大学成绩单信封,Buy OCAD University Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
安大略艺术与设计大学成绩单能够体现您的的学习能力，包括安大略艺术与设计大学课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
我们承诺采用的是学校原版纸张（原版纸质、底色、纹路）我们工厂拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有成品以及工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！
【主营项目】
一.安大略艺术与设计大学毕业证【q微1954292140】安大略艺术与设计大学成绩单、留信认证、使馆认证、教育部认证、雅思托福成绩单、学生卡等！
二.真实使馆公证(即留学回国人员证明,不成功不收费)
三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）
四.办理国外各大学文凭(一对一专业服务,可全程监控跟踪进度)
Measurecamp Copenhagen - Consent Context
Measurecamp Copenhagen - Consent ContextHuman37 Measurecamp Copenhagen 2025 presentation by Glenn Vanderlinden, Human37
最新版美国威斯康星大学拉克罗斯分校毕业证（UW–L毕业证书）原版定制
最新版美国威斯康星大学拉克罗斯分校毕业证（UW–L毕业证书）原版定制Taqyea 鉴于此，定制威斯康星大学拉克罗斯分校学位证书提升履历【q薇1954292140】原版高仿威斯康星大学拉克罗斯分校毕业证(UW–L毕业证书)可先看成品样本【q薇1954292140】帮您解决在美国威斯康星大学拉克罗斯分校未毕业难题，美国毕业证购买，美国文凭购买，【q微1954292140】美国文凭购买，美国文凭定制，美国文凭补办。专业在线定制美国大学文凭，定做美国本科文凭，【q微1954292140】复制美国University of Wisconsin-La Crosse completion letter。在线快速补办美国本科毕业证、硕士文凭证书，购买美国学位证、威斯康星大学拉克罗斯分校Offer，美国大学文凭在线购买。
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
【复刻一套威斯康星大学拉克罗斯分校毕业证成绩单信封等材料最强攻略,Buy University of Wisconsin-La Crosse Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
威斯康星大学拉克罗斯分校成绩单能够体现您的的学习能力，包括威斯康星大学拉克罗斯分校课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELSAmeya Patekar QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
Data Visualisation in data science for students
Data Visualisation in data science for studentsconfidenceascend Data visualisation is explained in a simple manner.
KLIP2Data voor de herinrichting van R4 West en Oost
KLIP2Data voor de herinrichting van R4 West en Oostjacoba18 Ontwikkeling van een centrale GIS-databank voor de periodieke import van KLIP-data, BIM- en CAD-modellen van het R4-project en bijhorende field app om de databank zowel online als offline te raadplegen en bij te werken.
Module 1Integrity_and_Ethics_PPT-2025.pptx
Module 1Integrity_and_Ethics_PPT-2025.pptxKarikalcholan Mayavan Module 1 describes about the value system and its comment are important
Veilig en vlot fietsen in Oost-Vlaanderen: Fietssnelwegen geoptimaliseerd met...
Veilig en vlot fietsen in Oost-Vlaanderen: Fietssnelwegen geoptimaliseerd met...jacoba18 Dit project, in samenwerking met de provincie Oost-Vlaanderen, heeft als doel de veiligheid op de F45 tussen Oudenaarde en Gent te verbeteren door netwerkdata aan te vullen met snelheidslimieten, toegangsbeperkingen en voorrangsregels. Met mobile mapping en FME werden verkeersborden en kruispunten geanalyseerd, waarna de resultaten in ArcGIS inzicht boden in toegankelijke wegsegmenten en verkeersregels.
FME Beyond Data Processing: Creating a Dartboard Accuracy App
FME Beyond Data Processing: Creating a Dartboard Accuracy Appjacoba18 Bij Nordend vroegen we ons af of FME ons kan helpen bepalen waar we moeten mikken voor de hoogste score, gebaseerd op onze nauwkeurigheid. We tonen hoe we met FME Flow, een zelfgemaakte WMS-dartbordlaag en geanalyseerde gooiresultaten optimale mikpunten berekenden.
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints: A...
Prescriptive Process Monitoring Under Uncertainty and Resource Constraints: A...Mahmoud Shoush We introduced Black-Box Prescriptive Process Monitoring (BB-PrPM) – a reinforcement learning approach that learns when, whether, and how to intervene in business processes to boost performance under real-world constraints.
This work is presented at the International Conference on Advanced Information Systems Engineering CAiSE Conference #CAiSE2025
MatFast: In-Memory Distributed Matrix Computation Processing and Optimization Based on Spark SQL Yanbo Liang and Mingie Tang
- 1. 1 © Hortonworks Inc. 2011 – 2017. All Rights Reserved MatFast IN-MEMORY DISTRIBUTED MATRIX COMPUTATION PROCESSING AND OPTIMIZATION BASED ON SPARK SQL Mingjie Tang Yanbo Liang Oct, 2017
- 2. 2 © Hortonworks Inc. 2011 – 2017. All Rights Reserved About Authors à Yongyang Yu • Machine learning, Database system, Computation Algebra • PhD students at Purdue University à Mingjie Tang • Spark SQL, Spark ML, Database, Machine Learning • Software Engineer at Hortonworks à Yanbo Liang • Apache Spark committer, Spark MLlib • Staff Software Engineer at Hortonworks à … All Other Contributors
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Agenda Motivation Overview of MatFast Implementation and optimization Use cases
- 4. 4 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Motivation à Many applications rely on efficient processing of queries over big matrix data: – Recommender systems – Social network analysis – Predict traffic data flow – Anti-fraud and spam detection – Bioinformatics
- 5. 5 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Motivation à Recommender Systems Netflix’s user-movie rating table (sample) Problem: Predict the missing entries in the table Input: User-movie rating table with missing entries Output: Complete user-movie rating table with predictions For Netflix, #users = 80 million, #movies = 2 million Batma n begins Alice in Wonde rland Doctor Strange Trolls Ironma n Alice 4 ? 3 5 4 Bob ? 5 4 ? ? Cindy 3 ? ? ? 2 movies users
- 6. 6 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Motivation à Gaussian Non-negative Matrix Factorization (GNMF) – Assumption: 𝑉"×$ ≈ 𝑊"×' × 𝐻'×$ V ≈ W H×users (80M) movies (2M) users (80M) modeling dims (e.g., topics, age, language, etc.) modeling dims (500) movies (2M) 𝐻 = 𝐻 ∗ 𝑊, × 𝑉 / (𝑊, × 𝑊 × 𝐻) 𝑊 = 𝑊 ∗ 𝑉 × 𝐻, / (𝑊 × 𝐻 × 𝐻, ) fori = 1 to nIter do end Initialize W and H Matrix operation for GNMF Algorithm huge volume dense/sparse storage iterative execution
- 7. 7 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Motivation à User-Movie Rating Prediction with GNMF val p = 200 // number of topics val V = loadMatrix(“in/V”) // read matrix val max_niter = 10 // max number of iteration W = RandomMatrix(V.nrows, p) H = RandomMatrix(p, V.ncols) for (i <- 0 until max_niter) { H = H * (W.t %*% V) / (W.t %*% W %*% H) W = W * (V %*% H.t) / (W %*% H %*% H.t) } (H %*% W).saveToHive()
- 8. 8 © Hortonworks Inc. 2011 – 2017. All Rights Reserved State of the art solution in Spark ecosystem à Alternative Least Square approach in Spark (ALS) – Experiment on Spotify data – 50+ million users x 30+ million songs – 50 billion ratings For rank 10 with 10 iterations – ~1 hour running time à How to extend ALS to other matrix computation? – SVD – PCA – QR
- 9. 9 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Observation H W transpose V mat-mat mat-mat mat-mat mat-elem mat-elem loop 𝐻 = 𝐻 ∗ 𝑊, × 𝑉 / (𝑊, × 𝑊 × 𝐻) 𝑊 = ( ) ( ) for i = 1 to nIter do end Initialize W and H Matrix computation evaluation pipeline 𝑊 ∗ 𝑉 × 𝐻, / 𝑊 × 𝐻 × intermediate result cost: 8 X 1016( )
- 10. 10 © Hortonworks Inc. 2011 – 2017. All Rights Reserved 𝑊 × 𝐻 Observation H W transpose V mat-mat mat-mat mat-mat mat-elem mat-elem intermediate result cost: 5 X 1011 loop materialization of result 𝐻 = 𝐻 ∗ 𝑊, × 𝑉 / (𝑊, × 𝑊 × 𝐻) 𝑊 = ( ) ( ) for i = 1 to nIter do end Initialize W and H Matrix computation evaluation pipeline 𝑊 ∗ 𝑉 × 𝐻, / 𝐻, )×(
- 11. 11 © Hortonworks Inc. 2011 – 2017. All Rights Reserved 𝑊 × 𝐻 Observation H W transpose V mat-mat mat-mat mat-mat chained mat-elem intermediate result size: 5 X 1011 loop 𝐻 = 𝐻 ∗ 𝑊, × 𝑉 / (𝑊, × 𝑊 × 𝐻) 𝑊 = ( ) ( ) for i = 1 to nIter do end Initialize W and H Matrix computation evaluation pipeline 𝑊 ∗ 𝑉 × 𝐻, / 𝐻,×( )
- 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Overview of MatFast
- 13. 13 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Matrix operators à Unary operator – Transpose: B = AT à Binary operators – B = A + 𝛽; B = A * 𝛽; – C = A ★ B, ★∈{+, *, /}; – C = A B (A %*% B) à Others – return a matrix: abs(A), pow(A, p) – return a vector: rowSum(A), colSum(A) – return a scalar: max(A), min(A) matrix-matrix multiplication
- 14. 14 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Optimization targets à MATFAST generates a computation- and communication- efficient execution plan: – Optimize a single matrix operator in an expression – Optimize multiple operators in an expression – Exploit data dependency between different expressions
- 15. 15 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Comparison with other systems Single Distributed w. multiple nodes R ScaLAPACK SciDB SystemML MLlib DMac huge volume. ✔ ✔ ✔ ✔ ✔ sparse comp. ✔ 〜 ✔ 〜 〜 multiple operators ✔ ✔ ✔ ✔ ✔ ✔ partition w. dependency ✔ opt. exec. plan ✔ interface R script C/Fortran SQL-like R-like Java/Scala Scala fault tolerance ✔ ✔ ✔ ✔ open source ✔ ✔ 〜 ✔ ✔
- 16. 16 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Compare with Spark SQL Matrix operators SQL relational query Data type matrix relational table Operators transpose, mat-mat, mat-scalar, mat-elem join, select, group by, aggregate Execution scheme iterative acyclic
- 17. 17 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Spark SQL System framework MATFAST ML algorithms: SVD, PCA, NMF, PageRank, QR, etc Spark RDD Applications: Image processing, Text processing, Collaborative filtering, Spatial computation, etc.
- 18. 18 © Hortonworks Inc. 2011 – 2017. All Rights Reserved System framework MATFAST Components Architecture
- 19. 19 © Hortonworks Inc. 2011 – 2017. All Rights Reserved MatFast within Spark Catalyst à Extend Spark Catalyst Rule based optimization (single matrix operators, multiple matrix operators) Cost based optimization (optimizing data partitioning)
- 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Implementation and optimization
- 21. 21 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Optimization 1: a Single Operator - Cost Based Optimization plan 1 plan 2 plan 3 plan 4 plan 5 MatFast AverageExecutionTime(s) 102 10 3 10 4 ((A1 × A2 ) × A3 ) × A4 (A1 × (A2 × A3 )) × A4 A1 × ((A2 × A3 ) × A4 ) A1 × (A2 × (A3 × A4 )) (A1 × A2 ) × (A3 × A4 )
- 22. 22 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Optimization 2: optimizing data partitioning in pipeline à Distribute matrix data over a set of workers à How to determine the data partitioning scheme for a matrix such that minimum shuffle cost is introduced for the entire pipeline? à Partitioning schemes – Row scheme (“r”) – Column scheme (“c”) – Block-Cyclic scheme (“b-c”) – Broadcast scheme (“b”)
- 23. 23 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Optimization 2: optimizing data partitioning in pipeline à How to determine the data partitioning scheme for a matrix such that minimum shuffle cost is introduced for the entire pipeline? 𝐻 = 𝐻 ∗ 𝑊, × 𝑉 / (𝑊, × 𝑊 × 𝐻) 𝑊 = 𝑊 ∗ 𝑉 × 𝐻, / (𝑊 × 𝐻 × 𝐻, ) compute firstWT 00 WT 10 WT 20 WT 30 WT 01 WT 11 WT 21 WT 31 E0 E1 E2 E3 W W00 W01 W10 W11 W20 W21 W30 W31 WT Hash-based partition, (i+ j) % N 3 block shuffles 7 block shuffles Total: 20 block shuffles Executors
- 24. 24 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Optimization 2: optimizing data partitioning in pipeline à How to determine the data partitioning scheme for a matrix such that minimum shuffle cost is introduced for the entire pipeline? 𝐻 = 𝐻 ∗ 𝑊, × 𝑉 / (𝑊, × 𝑊 × 𝐻) 𝑊 = 𝑊 ∗ 𝑉 × 𝐻, / (𝑊 × 𝐻 × 𝐻, ) compute firstWT 00 WT 10 WT 20 WT 30 WT 01 WT 11 WT 21 WT 31 E0 E1 E2 E3 W W00 W01 W10 W11 W20 W21 W30 W31 WT Row-based partition Total: 12 block shuffles Executors 3 block shuffles 3 block shuffles
- 25. 25 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Optimization 2: optimizing data partitioning in pipeline … …… … …… …… … W T V W T V WT W H WTW WTWH … H … H … V … VHT … W … WH … WHH T … W W … H stage 1 stage 2 stage 3 stage 4 stage 5 stage 6 𝐻 = 𝐻 ∗ 𝑊, × 𝑉 / (𝑊, × 𝑊 × 𝐻) 𝑊 = 𝑊 ∗ 𝑉 × 𝐻, / (𝑊 × 𝐻 × 𝐻, ) H Ã We need an optimized plan to determine an optimized data partitioning scheme for each matrix such that minimum shuffle overhead is introduced for the entire pipeline. Ã For example, with hash-based data partitioning, the computation pipeline involves multiple shuffles for aligning the data blocks.
- 26. 26 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Optimization 2: optimizing data partitioning in pipeline à MATFAST determines the partitioning scheme for an input matrix with min shuffle cost according to the cost model. à Greedily optimizes each operator 𝑠23(24) ⟵ argmin <=>(=?) 𝐶AB$$(𝑜𝑝, 𝑠23 , 𝑠24 , 𝑠B) à Physical execution plan with optimized data partitioning …… … WT V W T V … …… W T W … WTW … W T WH … H … H … V … VHT … HT … HHT … … W WHH T … W W stage 1 stage 2 stage 3 stage 4 … Row scheme for W Row scheme for V
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Case studies
- 28. 28 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Experiments à Dataset APIs – Code examples link à Compare with state-of-the-art systems – Spark MLlib (provided matrix operation) – SystemML (Spark) – ScaLAPACK – SciDB à Netflix data – 100,480,507 ratings – 17,770 movies from 480,189 customers à Social network data
- 29. 29 © Hortonworks Inc. 2011 – 2017. All Rights Reserved PageRank on different datasets MATFAST
- 30. 30 © Hortonworks Inc. 2011 – 2017. All Rights Reserved GNMF on the Netflix dataset MATFAST
- 31. 31 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Future plan à More user friend APIs à Advanced plan optimizer à Python and R interface à Vertical applications
- 32. 32 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Conclusion à Proposed and realized MATFAST, an in-memory distributed platform that optimizes query pipelines of matrix operations à Take advantage of dynamic cost-based analysis and rule-based heuristics to generate a query execution plan à Communication-efficient data partitioning scheme assignment
- 33. 33 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Reference – Yongyang Yu, MingJie Tang, Walid G. Aref, Qutaibah M. Malluhi, Mostafa M. Abbas, Mourad Ouzzani: In-Memory Distributed Matrix Computation Processing and Optimization. ICDE 2017: 1047-1058
- 34. 34 © Hortonworks Inc. 2011 – 2017. All Rights Reserved Thanks Q & A [email protected]