Build a Time Series Application with Apache Spark and Apache HBase
8 likes2,783 views
This document discusses using Apache Spark and Apache HBase to build a time series application. It provides an overview of time series data and requirements for ingesting, storing, and analyzing high volumes of time series data. The document then describes using Spark Streaming to process real-time data streams from sensors and storing the data in HBase. It outlines the steps in the lab exercise, which involves reading sensor data from files, converting it to objects, creating a Spark Streaming DStream, processing the DStream, and saving the data to HBase.
1 of 51
Downloaded 88 times
Ad
Recommended
Apache Spark Overview
Apache Spark OverviewCarol McDonald The document provides an overview of Apache Spark, including its components like Spark SQL, Spark Streaming, and machine learning capabilities. It contrasts Spark with Hadoop's MapReduce, emphasizing Spark's advantages such as efficiency in memory processing and ease of use with less code. Additionally, it covers use cases, the structure of Spark's Resilient Distributed Datasets (RDDs), and a simple word count example in Scala.
Introduction to Spark on Hadoop
Introduction to Spark on HadoopCarol McDonald This document provides an overview of Apache Spark, including:
- A refresher on MapReduce and its processing model
- An introduction to Spark, describing how it differs from MapReduce in addressing some of MapReduce's limitations
- Examples of how Spark can be used, including for iterative algorithms and interactive queries
- Resources for free online training in Hadoop, MapReduce, Hive and using HBase with MapReduce and Hive
Apache Spark streaming and HBase
Apache Spark streaming and HBaseCarol McDonald This document provides an overview of Apache Spark Streaming. It discusses why Spark Streaming is useful for processing time series data in near-real time. It then explains key concepts of Spark Streaming like data sources, transformations, and output operations. Finally, it provides an example of using Spark Streaming to process sensor data in real-time and save results to HBase.
NoSQL Application Development with JSON and MapR-DB
NoSQL Application Development with JSON and MapR-DBMapR Technologies The document discusses the limitations of relational databases for big data applications and introduces NoSQL data models, particularly focusing on JSON for its flexibility in handling various data types. It provides examples of JSON usage, advantages of NoSQL databases, and highlights the MapR platform's capabilities in integrating analytics and data management. Additionally, it outlines architectural benefits, optimized performance, and offers resources for learning more about MapR-DB.
Introduction to Spark
Introduction to SparkCarol McDonald This document provides an overview of Apache Spark, including:
- What Spark is and how it differs from MapReduce by running computations in memory for improved performance on iterative algorithms.
- Examples of Spark's core APIs like RDDs (Resilient Distributed Datasets) and transformations like map, filter, reduceByKey.
- How Spark programs are executed through a DAG (Directed Acyclic Graph) and translated to physical execution plans with stages and tasks.
Getting Started with HBase
Getting Started with HBaseCarol McDonald This document provides an overview of HBase and why NoSQL databases like HBase were developed. It discusses how relational databases do not scale horizontally well with large amounts of data. HBase was created by Google to address these scaling issues and was inspired by their BigTable database. The document explains the HBase data model with rows, columns, and versions. It describes how data is stored physically in HFiles and served from memory and disks. Basic operations like put, get, and scan are also covered.
Free Code Friday - Machine Learning with Apache Spark
Free Code Friday - Machine Learning with Apache SparkMapR Technologies The document provides an overview of machine learning techniques using Apache Spark, focusing on classification, clustering, and collaborative filtering. It discusses various algorithms available in Spark MLlib, such as decision trees, k-means clustering, and collaborative filtering methods along with examples and use cases, specifically predicting flight delays. It details the process of building and deploying a classification model, including feature extraction, data preparation, model training, and evaluation.
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...
Anomaly Detection in Telecom with Spark - Tugdual Grall - Codemotion Amsterda...Codemotion The document discusses anomaly detection in telecommunications using Spark, detailing its importance, methodologies, and application use cases. It outlines the process of defining 'normal' behavior, setting adaptive thresholds for alerts, and constructing a streaming application to analyze real-time data. Key technologies mentioned include Apache Kafka, Spark Streaming, and various machine learning techniques to effectively detect anomalies.
Advanced Threat Detection on Streaming Data
Advanced Threat Detection on Streaming DataCarol McDonald The document discusses using a stream processing architecture to enable real-time detection of advanced threats from large volumes of streaming data. The solution ingests data using fast distributed messaging like Kafka or MapR Streams. Complex event processing with Storm and Esper is used to detect patterns. Data is stored in scalable NoSQL databases like HBase and analyzed using machine learning. The parallelized, partitioned architecture allows for high performance and scalability.
Fast Cars, Big Data How Streaming can help Formula 1
Fast Cars, Big Data How Streaming can help Formula 1Carol McDonald This document discusses how streaming data and analytics can help Formula 1 racing teams. It provides examples of the large volume of sensor data collected from Formula 1 cars during races. The document demonstrates how streaming this data using Apache Kafka and analyzing it in real-time with tools like Apache Spark and Apache Flink can help teams with tasks like predictive maintenance, race strategy optimization, and driver coaching. It also discusses storing the streaming data in databases like Apache Drill and MapR-DB for ad-hoc querying and analysis.
Dealing with an Upside Down Internet
Dealing with an Upside Down InternetMapR Technologies The document discusses advancements in anomaly detection and time series data management by Ted Dunning of MapR Technologies. It highlights the challenges of internet infrastructure, emphasizes the need for efficient data handling strategies, and outlines the limitations of existing open-source systems while proposing improvements through modern NoSQL solutions. The summary concludes that the evolving internet landscape presents opportunities for broader usage of time series analytics solutions.
Deep Learning vs. Cheap Learning
Deep Learning vs. Cheap LearningMapR Technologies The document discusses how big data has enabled new opportunities by changing scaling laws and problem landscapes. Specifically, linearly scaling costs with big data now make it feasible to process large amounts of data, opening up many problems that were previously impossible or too difficult. This has created many "green field" opportunities where simple approaches can solve important problems. Two examples discussed are using log analysis to detect security threats and using transaction histories to find a common point of compromise for a data breach.
When Streaming Becomes Strategic
When Streaming Becomes StrategicMapR Technologies The document discusses the importance of streaming data processing in big data architecture, highlighting the limitations of traditional batch processing and the growing need for real-time event processing. It introduces the event-insight-outcome framework to improve business analytics, emphasizing the role of MapR Streams as a converged data platform that integrates storage and data streaming solutions. Various use cases and benefits of this architecture, including improved data management and analytics capabilities, are also presented.
MapR-DB – The First In-Hadoop Document Database
MapR-DB – The First In-Hadoop Document DatabaseMapR Technologies MapR-DB offers a NoSQL solution designed for big data, addressing issues prevalent in traditional RDBMS systems and other NoSQL databases. It emphasizes tighter Hadoop integration, high performance with low latency, and minimal administration for global deployments. The platform supports real-time analytics, automatic data management, and granular security, catering to diverse applications across complex datasets.
Fast, Scalable, Streaming Applications with Spark Streaming, the Kafka API an...
Fast, Scalable, Streaming Applications with Spark Streaming, the Kafka API an...Carol McDonald The document discusses the importance of stream processing in real-time data analysis, emphasizing how timely processing can optimize responses to events, such as temperature changes. It outlines the architecture for processing data using Spark Streaming and MapR technologies, detailing the collection, processing, and storage of event-based data. Additionally, it provides code examples and use cases for real-time monitoring of time-stamped sensor data.
Streaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APICarol McDonald The document discusses streaming data architecture, emphasizing the components such as streams, topics, and partitions. It highlights the advantages of event sourcing and real-time analytics for processing high volumes of data across various applications, including fraud detection and healthcare. Additionally, it emphasizes fault tolerance, scalability, and the integration of multiple data sources for effective data management.
NoSQL HBase schema design and SQL with Apache Drill
NoSQL HBase schema design and SQL with Apache Drill Carol McDonald The document provides an overview of HBase, including:
- HBase is a column-oriented NoSQL database modeled after Google's Bigtable. It is designed to handle large volumes of sparse data across clusters in a distributed fashion.
- Data in HBase is stored in tables containing rows, column families, columns, and versions. Tables are partitioned into regions distributed across region servers. The HMaster manages the cluster and Zookeeper coordinates operations.
- Common operations on HBase include put (insert/update), get, scan, and delete. The meta table stored in Zookeeper maps rows to their regions. This allows clients to efficiently access data in HBase's distributed architecture.
MapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR Technologies The document outlines key updates and features of the MapR Converged Data Platform with version 5.2, including improvements to cluster monitoring through the Spyglass initiative, support for real-time streaming via MapR Streams, and development capabilities with MapR-DB and OJAI API. It emphasizes enhanced data security through Access Control Expressions (ACEs) and the integration of SQL with JSON data via Apache Drill. The presentation provides an overview of system capabilities, ecosystem updates, and tools to streamline big data management.
Apache Spark & Hadoop
Apache Spark & HadoopMapR Technologies The document provides an overview of Apache Spark and its comparison with MapReduce, highlighting its advantages such as ease of use, performance, and in-memory data processing capabilities. It introduces key concepts like Resilient Distributed Datasets (RDDs), fault tolerance, and the platform's support for multiple programming languages. Examples of practical applications, such as word counting and log mining, demonstrate Spark's functionality and efficiency in handling big data tasks.
Applying Machine Learning to Live Patient Data
Applying Machine Learning to Live Patient DataCarol McDonald This document discusses applying machine learning to live patient data for real-time anomaly detection. It describes using streaming data from medical devices like EKGs to build a machine learning model for identifying anomalies. The streaming data is processed using Spark Streaming and enriched with cluster assignments from a pre-trained K-means model before being sent to a dashboard for real-time monitoring of patient vitals.
Apache Spark Machine Learning Decision Trees
Apache Spark Machine Learning Decision TreesCarol McDonald The document discusses machine learning techniques including classification, clustering, and collaborative filtering. It provides examples of algorithms used for each technique, such as Naive Bayes, k-means clustering, and alternating least squares for collaborative filtering. The document then focuses on using Spark for machine learning, describing MLlib and how it can be used to build classification and regression models on Spark, including examples predicting flight delays using decision trees. Key steps discussed are feature extraction, splitting data into training and test sets, training a model, and evaluating performance on test data.
Hadoop2 new and noteworthy SNIA conf
Hadoop2 new and noteworthy SNIA confSujee Maniyam The document is a presentation on new features in Hadoop 2. Some key highlights include:
- Hadoop 2 introduces NameNode high availability to address single point of failure through an active-passive setup using shared storage.
- Federation allows spreading metadata over multiple NameNodes for very large clusters.
- Snapshots provide point-in-time copies of data for backup and recovery from deletes or disasters.
- YARN separates processing from resource management, allowing various types of applications beyond batch processing.
Apache Eagle - Monitor Hadoop in Real Time
Apache Eagle - Monitor Hadoop in Real TimeDataWorks Summit/Hadoop Summit Apache Eagle is a distributed real-time monitoring and alerting engine for Hadoop created by eBay to address limitations of existing tools in handling large volumes of metrics and logs from Hadoop clusters. It provides data activity monitoring, job performance monitoring, and unified monitoring. Eagle detects anomalies using machine learning algorithms and notifies users through alerts. It has been deployed across multiple eBay clusters with over 10,000 nodes and processes hundreds of thousands of events per day.
MapR & Skytree:
MapR & Skytree: MapR Technologies The document discusses the integration of Hadoop in machine learning, emphasizing its advantages for big data processing and analytics. It outlines use cases like fraud detection, customer retention, and predictive maintenance, as well as highlights specific tools like Apache Spark and Mahout for machine learning tasks. The document also showcases Skytree's contributions to enhancing machine learning capabilities in various industries through improved model accuracy and efficiency.
What's new in Hadoop Common and HDFS
What's new in Hadoop Common and HDFS DataWorks Summit/Hadoop Summit The document discusses new features in Apache Hadoop Common and HDFS for version 3.0. Key updates include upgrading the minimum Java version to Java 8, improving dependency management, adding a new Azure Data Lake Storage connector, and introducing erasure coding in HDFS to improve storage efficiency. Erasure coding in HDFS phase 1 allows for striping of small blocks and parallel writes/reads while trading off higher network usage compared to replication.
How Big Data is Reducing Costs and Improving Outcomes in Health Care
How Big Data is Reducing Costs and Improving Outcomes in Health CareCarol McDonald The document discusses the application of big data in healthcare, highlighting the need for improved healthcare outcomes, value-based care, and data-driven decision-making. It emphasizes the potential of big data for reducing costs, detecting fraud, and enhancing patient management through analysis of various data sources and predictive analytics. Additionally, it outlines technological solutions, including a data lake architecture and the integration of machine learning and IoT for real-time patient monitoring and treatment optimization.
The Future of Hadoop: MapR VP of Product Management, Tomer Shiran
The Future of Hadoop: MapR VP of Product Management, Tomer ShiranMapR Technologies (1) The amount of data in the world is growing exponentially, with unstructured data making up over 80% of collected data by 2020. (2) Apache Drill provides data agility for Hadoop by enabling self-service data exploration through a flexible data model and schema discovery. (3) Drill allows business users to rapidly query diverse data sources like files, HBase tables, and Hive without requiring IT, through a simple SQL interface.
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...Mathieu Dumoulin The document discusses the challenges of machine learning in enterprises and presents the MapR Converged Data Platform as a solution. It highlights key features that support machine learning processes, such as efficient data handling, integration capabilities, and real-time processing. The platform is designed to simplify machine learning workflows while maintaining security and robustness for enterprise applications.
Time-Series Apache HBase
Time-Series Apache HBaseHBaseCon The document discusses the requirements and ideal characteristics of a time series database (TSDB) using HBase, emphasizing the importance of preserving temporal locality, efficient compression, and automatic time-based rollup aggregations. It critiques existing solutions like OpenTSDB for failing to meet these requirements and proposes various compaction policies such as FIFO and date-tiered compaction to optimize data handling. Recommendations are provided for configuring HBase to support time series data efficiently, including disabling major compactions and managing caching strategies.
A 3 dimensional data model in hbase for large time-series dataset-20120915
A 3 dimensional data model in hbase for large time-series dataset-20120915Dan Han This document outlines a study on migrating relational database content to NoSQL storage systems like HBase. It discusses challenges in migration and the need for design patterns for HBase schemas. A 3-dimensional data model in HBase is proposed and evaluated using cosmology and bike rental datasets. Experiment results show the 3D model improves performance for queries that use HBase's version dimension. Future work includes further evaluation of the model's scalability and designing models for other dataset types.
More Related Content
What's hot (20)
Advanced Threat Detection on Streaming Data
Advanced Threat Detection on Streaming DataCarol McDonald The document discusses using a stream processing architecture to enable real-time detection of advanced threats from large volumes of streaming data. The solution ingests data using fast distributed messaging like Kafka or MapR Streams. Complex event processing with Storm and Esper is used to detect patterns. Data is stored in scalable NoSQL databases like HBase and analyzed using machine learning. The parallelized, partitioned architecture allows for high performance and scalability.
Fast Cars, Big Data How Streaming can help Formula 1
Fast Cars, Big Data How Streaming can help Formula 1Carol McDonald This document discusses how streaming data and analytics can help Formula 1 racing teams. It provides examples of the large volume of sensor data collected from Formula 1 cars during races. The document demonstrates how streaming this data using Apache Kafka and analyzing it in real-time with tools like Apache Spark and Apache Flink can help teams with tasks like predictive maintenance, race strategy optimization, and driver coaching. It also discusses storing the streaming data in databases like Apache Drill and MapR-DB for ad-hoc querying and analysis.
Dealing with an Upside Down Internet
Dealing with an Upside Down InternetMapR Technologies The document discusses advancements in anomaly detection and time series data management by Ted Dunning of MapR Technologies. It highlights the challenges of internet infrastructure, emphasizes the need for efficient data handling strategies, and outlines the limitations of existing open-source systems while proposing improvements through modern NoSQL solutions. The summary concludes that the evolving internet landscape presents opportunities for broader usage of time series analytics solutions.
Deep Learning vs. Cheap Learning
Deep Learning vs. Cheap LearningMapR Technologies The document discusses how big data has enabled new opportunities by changing scaling laws and problem landscapes. Specifically, linearly scaling costs with big data now make it feasible to process large amounts of data, opening up many problems that were previously impossible or too difficult. This has created many "green field" opportunities where simple approaches can solve important problems. Two examples discussed are using log analysis to detect security threats and using transaction histories to find a common point of compromise for a data breach.
When Streaming Becomes Strategic
When Streaming Becomes StrategicMapR Technologies The document discusses the importance of streaming data processing in big data architecture, highlighting the limitations of traditional batch processing and the growing need for real-time event processing. It introduces the event-insight-outcome framework to improve business analytics, emphasizing the role of MapR Streams as a converged data platform that integrates storage and data streaming solutions. Various use cases and benefits of this architecture, including improved data management and analytics capabilities, are also presented.
MapR-DB – The First In-Hadoop Document Database
MapR-DB – The First In-Hadoop Document DatabaseMapR Technologies MapR-DB offers a NoSQL solution designed for big data, addressing issues prevalent in traditional RDBMS systems and other NoSQL databases. It emphasizes tighter Hadoop integration, high performance with low latency, and minimal administration for global deployments. The platform supports real-time analytics, automatic data management, and granular security, catering to diverse applications across complex datasets.
Fast, Scalable, Streaming Applications with Spark Streaming, the Kafka API an...
Fast, Scalable, Streaming Applications with Spark Streaming, the Kafka API an...Carol McDonald The document discusses the importance of stream processing in real-time data analysis, emphasizing how timely processing can optimize responses to events, such as temperature changes. It outlines the architecture for processing data using Spark Streaming and MapR technologies, detailing the collection, processing, and storage of event-based data. Additionally, it provides code examples and use cases for real-time monitoring of time-stamped sensor data.
Streaming Patterns Revolutionary Architectures with the Kafka API
Streaming Patterns Revolutionary Architectures with the Kafka APICarol McDonald The document discusses streaming data architecture, emphasizing the components such as streams, topics, and partitions. It highlights the advantages of event sourcing and real-time analytics for processing high volumes of data across various applications, including fraud detection and healthcare. Additionally, it emphasizes fault tolerance, scalability, and the integration of multiple data sources for effective data management.
NoSQL HBase schema design and SQL with Apache Drill
NoSQL HBase schema design and SQL with Apache Drill Carol McDonald The document provides an overview of HBase, including:
- HBase is a column-oriented NoSQL database modeled after Google's Bigtable. It is designed to handle large volumes of sparse data across clusters in a distributed fashion.
- Data in HBase is stored in tables containing rows, column families, columns, and versions. Tables are partitioned into regions distributed across region servers. The HMaster manages the cluster and Zookeeper coordinates operations.
- Common operations on HBase include put (insert/update), get, scan, and delete. The meta table stored in Zookeeper maps rows to their regions. This allows clients to efficiently access data in HBase's distributed architecture.
MapR 5.2: Getting More Value from the MapR Converged Community Edition
MapR 5.2: Getting More Value from the MapR Converged Community EditionMapR Technologies The document outlines key updates and features of the MapR Converged Data Platform with version 5.2, including improvements to cluster monitoring through the Spyglass initiative, support for real-time streaming via MapR Streams, and development capabilities with MapR-DB and OJAI API. It emphasizes enhanced data security through Access Control Expressions (ACEs) and the integration of SQL with JSON data via Apache Drill. The presentation provides an overview of system capabilities, ecosystem updates, and tools to streamline big data management.
Apache Spark & Hadoop
Apache Spark & HadoopMapR Technologies The document provides an overview of Apache Spark and its comparison with MapReduce, highlighting its advantages such as ease of use, performance, and in-memory data processing capabilities. It introduces key concepts like Resilient Distributed Datasets (RDDs), fault tolerance, and the platform's support for multiple programming languages. Examples of practical applications, such as word counting and log mining, demonstrate Spark's functionality and efficiency in handling big data tasks.
Applying Machine Learning to Live Patient Data
Applying Machine Learning to Live Patient DataCarol McDonald This document discusses applying machine learning to live patient data for real-time anomaly detection. It describes using streaming data from medical devices like EKGs to build a machine learning model for identifying anomalies. The streaming data is processed using Spark Streaming and enriched with cluster assignments from a pre-trained K-means model before being sent to a dashboard for real-time monitoring of patient vitals.
Apache Spark Machine Learning Decision Trees
Apache Spark Machine Learning Decision TreesCarol McDonald The document discusses machine learning techniques including classification, clustering, and collaborative filtering. It provides examples of algorithms used for each technique, such as Naive Bayes, k-means clustering, and alternating least squares for collaborative filtering. The document then focuses on using Spark for machine learning, describing MLlib and how it can be used to build classification and regression models on Spark, including examples predicting flight delays using decision trees. Key steps discussed are feature extraction, splitting data into training and test sets, training a model, and evaluating performance on test data.
Hadoop2 new and noteworthy SNIA conf
Hadoop2 new and noteworthy SNIA confSujee Maniyam The document is a presentation on new features in Hadoop 2. Some key highlights include:
- Hadoop 2 introduces NameNode high availability to address single point of failure through an active-passive setup using shared storage.
- Federation allows spreading metadata over multiple NameNodes for very large clusters.
- Snapshots provide point-in-time copies of data for backup and recovery from deletes or disasters.
- YARN separates processing from resource management, allowing various types of applications beyond batch processing.
Apache Eagle - Monitor Hadoop in Real Time
Apache Eagle - Monitor Hadoop in Real TimeDataWorks Summit/Hadoop Summit Apache Eagle is a distributed real-time monitoring and alerting engine for Hadoop created by eBay to address limitations of existing tools in handling large volumes of metrics and logs from Hadoop clusters. It provides data activity monitoring, job performance monitoring, and unified monitoring. Eagle detects anomalies using machine learning algorithms and notifies users through alerts. It has been deployed across multiple eBay clusters with over 10,000 nodes and processes hundreds of thousands of events per day.
MapR & Skytree:
MapR & Skytree: MapR Technologies The document discusses the integration of Hadoop in machine learning, emphasizing its advantages for big data processing and analytics. It outlines use cases like fraud detection, customer retention, and predictive maintenance, as well as highlights specific tools like Apache Spark and Mahout for machine learning tasks. The document also showcases Skytree's contributions to enhancing machine learning capabilities in various industries through improved model accuracy and efficiency.
What's new in Hadoop Common and HDFS
What's new in Hadoop Common and HDFS DataWorks Summit/Hadoop Summit The document discusses new features in Apache Hadoop Common and HDFS for version 3.0. Key updates include upgrading the minimum Java version to Java 8, improving dependency management, adding a new Azure Data Lake Storage connector, and introducing erasure coding in HDFS to improve storage efficiency. Erasure coding in HDFS phase 1 allows for striping of small blocks and parallel writes/reads while trading off higher network usage compared to replication.
How Big Data is Reducing Costs and Improving Outcomes in Health Care
How Big Data is Reducing Costs and Improving Outcomes in Health CareCarol McDonald The document discusses the application of big data in healthcare, highlighting the need for improved healthcare outcomes, value-based care, and data-driven decision-making. It emphasizes the potential of big data for reducing costs, detecting fraud, and enhancing patient management through analysis of various data sources and predictive analytics. Additionally, it outlines technological solutions, including a data lake architecture and the integration of machine learning and IoT for real-time patient monitoring and treatment optimization.
The Future of Hadoop: MapR VP of Product Management, Tomer Shiran
The Future of Hadoop: MapR VP of Product Management, Tomer ShiranMapR Technologies (1) The amount of data in the world is growing exponentially, with unstructured data making up over 80% of collected data by 2020. (2) Apache Drill provides data agility for Hadoop by enabling self-service data exploration through a flexible data model and schema discovery. (3) Drill allows business users to rapidly query diverse data sources like files, HBase tables, and Hive without requiring IT, through a simple SQL interface.
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...
Real-World Machine Learning - Leverage the Features of MapR Converged Data Pl...Mathieu Dumoulin The document discusses the challenges of machine learning in enterprises and presents the MapR Converged Data Platform as a solution. It highlights key features that support machine learning processes, such as efficient data handling, integration capabilities, and real-time processing. The platform is designed to simplify machine learning workflows while maintaining security and robustness for enterprise applications.
Viewers also liked (20)
Time-Series Apache HBase
Time-Series Apache HBaseHBaseCon The document discusses the requirements and ideal characteristics of a time series database (TSDB) using HBase, emphasizing the importance of preserving temporal locality, efficient compression, and automatic time-based rollup aggregations. It critiques existing solutions like OpenTSDB for failing to meet these requirements and proposes various compaction policies such as FIFO and date-tiered compaction to optimize data handling. Recommendations are provided for configuring HBase to support time series data efficiently, including disabling major compactions and managing caching strategies.
A 3 dimensional data model in hbase for large time-series dataset-20120915
A 3 dimensional data model in hbase for large time-series dataset-20120915Dan Han This document outlines a study on migrating relational database content to NoSQL storage systems like HBase. It discusses challenges in migration and the need for design patterns for HBase schemas. A 3-dimensional data model in HBase is proposed and evaluated using cosmology and bike rental datasets. Experiment results show the 3D model improves performance for queries that use HBase's version dimension. Future work includes further evaluation of the model's scalability and designing models for other dataset types.
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...DataWorks Summit/Hadoop Summit This document discusses leveraging Apache HBase as a non-relational datastore in Apache Spark batch and streaming applications. It outlines integration patterns for reading from and writing to HBase using Spark, provides examples of API usage, and discusses future work including using HBase edits as a streaming source.
Spark + HBase
Spark + HBase DataWorks Summit/Hadoop Summit Zhan Zhang presents improvements made to bring HBase data efficiently into Spark with DataFrame support. The improvements include high performance by moving computation to data and reducing network overhead through partition pruning and column pruning. Full DataFrame support is provided, allowing Spark SQL and integrated language queries to run on existing HBase tables with Java primitive type support.
SE2016 Java Valerii Moisieienko "Apache HBase Workshop"
SE2016 Java Valerii Moisieienko "Apache HBase Workshop"Inhacking This document provides an overview and agenda for an Apache HBase workshop. It introduces HBase as an open-source NoSQL database built on Hadoop that uses a column-family data model. The agenda covers what HBase is, its data model including rows, columns, cells and versions, CRUD operations, architecture including regions and masters, schema design best practices, and the Java API. Performance tips are given for client reads and writes such as using batches, caching, and tuning durability.
C* Summit 2013: High Throughput Analytics with Cassandra by Aaron Stannard
C* Summit 2013: High Throughput Analytics with Cassandra by Aaron StannardDataStax Academy The document discusses real-time analytics using Cassandra, Hive, and Solr, emphasizing their significance in monitoring app health and user data to drive sales. It outlines strategies for setting up Cassandra on EC2, incorporating Hive for processing large datasets, and utilizing Solr for indexing and searching efficiently. The document also provides insights on best practices and schema considerations for optimizing performance in real-time analytics applications.
Spark!
Spark!Przemek Maciolek This document discusses Spark, an open-source cluster computing framework. It notes that while Hadoop is useful for batch processing, it has limitations for interactive and iterative algorithms. Spark addresses these issues through its resilient distributed datasets (RDDs) which can be operated on in parallel and rebuilt if lost. RDDs support transformations like map and filter as well as actions that return values. The document provides examples of using Spark from Scala and discusses its architecture involving a DAG scheduler and task scheduler.
Apache spark core
Apache spark coreThành Nguyễn This document provides an overview of Apache Spark, including:
- The problems of big data that Spark addresses like large volumes of data from various sources.
- A comparison of Spark to existing techniques like Hadoop, noting Spark allows for better developer productivity and performance.
- An overview of the Spark ecosystem and how Spark can integrate with an existing enterprise.
- Details about Spark's programming model including its RDD abstraction and use of transformations and actions.
- A discussion of Spark's execution model involving stages and tasks.
SparkSQL et Cassandra - Tool In Action Devoxx 2015
SparkSQL et Cassandra - Tool In Action Devoxx 2015Alexander DEJANOVSKI This document discusses the integration and use of Spark SQL with Cassandra for data analysis, highlighting key concepts like Resilient Distributed Datasets (RDDs) and various stages of data processing. It includes practical code snippets for SQL queries, data transformations in Scala and Python, and the use of DataStax connectors. The overall goal is to analyze and manage conference talk data from Devoxx using efficient techniques that leverage both Spark and Cassandra capabilities.
The SparkSQL things you maybe confuse
The SparkSQL things you maybe confusevito jeng SparkSQL, SchemaRDD, DataFrame, and Dataset are Apache Spark APIs for structured data processing. SparkSQL is a high-level module introduced in Spark 1.0. SchemaRDD was introduced in Spark 1.0 from the Shark project and was later renamed to DataFrame in Spark 1.3. Dataset, introduced experimentally in Spark 1.6, allows SparkSQL optimizations while working with RDDs. DataFrame and Dataset were unified under a single API in Spark 2.0.
Getting started with SparkSQL - Desert Code Camp 2016
Getting started with SparkSQL - Desert Code Camp 2016clairvoyantllc The document discusses Spark SQL, an Apache Spark module for structured data processing. It provides an agenda that covers Spark concepts, Spark SQL, the Catalyst optimizer, Project Tungsten, and a demo. Spark SQL allows users to perform SQL queries and use the DataFrame and Dataset APIs to interact with structured data in a Spark cluster.
Making Scrum Work Inside Small Businesses
Making Scrum Work Inside Small Businesses Laszlo Szalvay This document discusses how entrepreneurs can benefit from using Scrum. It recommends that entrepreneurs incorporate a culture of learning and questioning, use both qualitative and quantitative metrics like cash on hand and number of happy customers and employees, and do Scrum at the organizational level across functions like marketing, sales, and executives. Adopting these agile practices can help entrepreneurs build learning organizations that continuously solve problems and adapt.
Streaming map reduce
Streaming map reducedanirayan - The document discusses using Flume and HBase for real-time analytics of streaming big data with service level agreements (SLAs).
- Flume is used to ingest streaming data into HBase for storage and querying. HBase's column families allow storing different types of data with different time to live settings.
- Maintaining SLAs for bucketing millions of product impressions within seconds across many nodes is challenging due to the complexity of the systems involved.
- Monalytics can help monitor end-to-end performance and identify issues in meeting SLAs, like garbage collection pauses, across the distributed Flume and HBase deployment.
Musings on Secondary Indexing in HBase
Musings on Secondary Indexing in HBaseJesse Yates The document discusses secondary indexing in HBase, addressing challenges in efficiently performing lookups deeper than row keys and proposing various implementations like Omid and Percolator. It highlights the need for flexible indexing strategies tailored to specific data use cases, pointing out considerations for built-in vs. external libraries, index cardinality, and client-side management of indexes. The author emphasizes the importance of allowing users to choose their indexing methods while maintaining ease of use and performance.
MongoDB and Apache HBase: Benchmarking
MongoDB and Apache HBase: BenchmarkingOlga Lavrentieva This document compares MongoDB and HBase on technical features and performance. MongoDB is a document-oriented database using JSON-like documents with dynamic schemas, ad-hoc queries, and indexing. HBase is a column-oriented NoSQL database modeled after Google Bigtable and stored on HDFS. The document benchmarks the two databases on loading performance using YCSB and shows MongoDB generally performs better, especially for read-heavy workloads. Popular use cases of each are also listed.
Galvanise NYC - Scaling R with Hadoop & Spark. V1.0
Galvanise NYC - Scaling R with Hadoop & Spark. V1.0vithakur The document discusses scaling R using Hadoop and Spark. It provides an overview of IBM's approach to big data, which leverages open source technologies like Hadoop, Spark, and R. It then summarizes IBM's investments in Spark and the Open Data Platform initiative. The rest of the document focuses on describing Big R, IBM's tool for scaling R to big data using Hadoop. Big R allows users to run R scripts on large datasets in Hadoop and provides functions for machine learning algorithms and accessing Hadoop data from within R.
Lspe
LspeArpit Tak Azkaban is a batch workflow job scheduler created by LinkedIn to run Hadoop jobs. It allows running a series of MapReduce, Pig, Java, and script actions as a single workflow job. It provides an easy to use web UI to define jobs and their dependencies, upload workflows, and track job executions. The Azkaban system consists of a web server to manage projects and workflows and executor servers to run jobs on Hadoop clusters. It supports features like retrying failed jobs, email alerts, and scheduling recurring workflow executions.
HBase Consistency and Performance Improvements
HBase Consistency and Performance ImprovementsDataWorks Summit The document discusses improvements to consistency and performance in HBase. It describes how HBase maintains consistency using timestamps and multi-version concurrency control. It identifies issues that could cause inconsistent reads and explains how storing timestamps in HFiles resolves it. The document also outlines several performance improvements in HBase 0.94 like checksum caching, deleting expired files, HLog compression and lazy-seeking.
Apache HBase 0.98
Apache HBase 0.98AndrewPurtell This document summarizes the new features and improvements in Apache HBase 0.98, including:
1) Performance enhancements such as reverse scans, improved write throughput, and stripe compactions.
2) New security features like endpoint access control, transparent encryption, per-cell ACLs, and visibility labels.
3) Additional features like MapReduce over snapshots and REST streaming scans.
4) The release manager ensures compatibility, performance, and other criteria are met before the 0.98 release.
Search Analytics with Flume and HBase
Search Analytics with Flume and HBaseSematext Group, Inc. This document summarizes an architecture for collecting and analyzing search analytics data using Flume and HBase. It describes:
1) Collecting search log data from applications using Flume agents and sending it to a Flume collector.
2) The Flume collector processes the log messages and sends them to an "raw logs" table in HBase using a Flume HBase sink.
3) The data in HBase is later processed by MapReduce jobs to generate search analytics reports and metrics that are displayed on a reporting web application.
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...DataWorks Summit/Hadoop Summit
Ad
Similar to Build a Time Series Application with Apache Spark and Apache HBase (20)
ETL with SPARK - First Spark London meetup
ETL with SPARK - First Spark London meetupRafal Kwasny The document discusses how Spark can be used to supercharge ETL workflows by running them faster and with less code compared to traditional Hadoop approaches. It provides examples of using Spark for tasks like sessionization of user clickstream data. Best practices are covered like optimizing for JVM issues, avoiding full GC pauses, and tips for deployment on EC2. Future improvements to Spark like SQL support and Java 8 are also mentioned.
Big Data on azure
Big Data on azureDavid Giard The document discusses Big Data on Azure and provides an overview of HDInsight, Microsoft's Apache Hadoop-based data platform on Azure. It describes HDInsight cluster types for Hadoop, HBase, Storm and Spark and how clusters can be automatically provisioned on Azure. Example applications and demos of Storm, HBase, Hive and Spark are also presented. The document highlights key aspects of using HDInsight including storage integration and tools for interactive analysis.
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in Production
Tugdual Grall - Real World Use Cases: Hadoop and NoSQL in ProductionCodemotion The document discusses the readiness of Hadoop for production environments, emphasizing its capabilities and the integration with NoSQL technologies. It outlines various use cases, including data warehousing and real-time analytics, along with examples of big data applications. The document also highlights the importance of choosing the right data connectors and tools for efficient data processing and storage.
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...
Global Big Data Conference Sept 2014 AWS Kinesis Spark Streaming Approximatio...Chris Fregly The document provides an overview of Spark and Spark Streaming, including their architecture, execution model, and use cases in big data processing. It covers various aspects such as API libraries, fault tolerance, cluster deployment, and monitoring strategies. Additionally, it touches on the integration with Kinesis for streaming applications and addresses approximation techniques for improved performance.
In Memory Data Pipeline And Warehouse At Scale - BerlinBuzzwords 2015
In Memory Data Pipeline And Warehouse At Scale - BerlinBuzzwords 2015Iulia Emanuela Iancuta The document describes an in-memory data pipeline and warehouse using Spark, Spark SQL, Tachyon and Parquet. It involves ingesting financial transaction data from S3, transforming the data through cleaning and joining steps, and building a data warehouse using Spark SQL and Parquet for querying. Key aspects covered include distributing metadata lookups, balancing data partitions, broadcasting joins to avoid skew, caching data in Tachyon and Jaws for a RESTful interface to Spark SQL.
Big Data Solutions in Azure - David Giard
Big Data Solutions in Azure - David GiardITCamp The document discusses Big Data on Azure and provides an overview of HDInsight, Microsoft's Apache Hadoop-based data platform on Azure. It describes HDInsight cluster types for Hadoop, HBase, Storm and Spark and how clusters can be automatically provisioned on Azure. It also summarizes tools like Hive, HBase and Spark that can be used to analyze data stored on HDInsight clusters.
Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena Edelson
Streaming Analytics with Spark, Kafka, Cassandra and Akka by Helena EdelsonSpark Summit Streaming Analytics with Spark, Kafka, Cassandra, and Akka discusses rethinking architectures for streaming analytics. The document discusses:
1) The need to build scalable, fault-tolerant systems to handle massive amounts of streaming data from different sources with varying structures.
2) An example use case of profiling cyber threat actors using streaming machine data to detect intrusions and security breaches.
3) Rethinking architectures by moving away from ETL pipelines and dual batch/stream systems like Lambda architecture toward unified stream processing with Spark Streaming, Kafka, Cassandra and Akka. This simplifies analytics and eliminates duplicate code and systems.
Devnexus 2018
Devnexus 2018Roy Russo Elasticsearch is a distributed, RESTful search and analytics engine that can be used for processing big data with Apache Spark. Data is ingested from Spark into Elasticsearch for features generation and predictive modeling. Elasticsearch allows for fast reads and writes of large volumes of time-series and other data through its use of inverted indexes and dynamic mapping. It is deployed on AWS for its elastic scalability, high availability, and integration with Spark via fast queries. Ongoing maintenance includes archiving old data, partitioning indices, and reindexing large datasets.
Jump Start with Apache Spark 2.0 on Databricks
Jump Start with Apache Spark 2.0 on DatabricksDatabricks The document provides an overview of Apache Spark 2.0 on Databricks, detailing its architecture, major new features, and improvements such as unified APIs and structured streaming capabilities. It emphasizes the role of Databricks in simplifying big data processing and introduces participants to various functionalities and workshops focused on Spark's dataframes, datasets, and performance optimizations through Project Tungsten. The presentation includes practical sessions to familiarize users with structured streaming and batch processing using Spark SQL.
Apache Spark Components
Apache Spark ComponentsGirish Khanzode Spark Streaming allows processing live data streams using small batch sizes to provide low latency results. It provides a simple API to implement complex stream processing algorithms across hundreds of nodes. Spark SQL allows querying structured data using SQL or the Hive query language and integrates with Spark's batch and interactive processing. MLlib provides machine learning algorithms and pipelines to easily apply ML to large datasets. GraphX extends Spark with an API for graph-parallel computation on property graphs.
Big Data Retrospective - STL Big Data IDEA Jan 2019
Big Data Retrospective - STL Big Data IDEA Jan 2019Adam Doyle The document outlines a retrospective meeting agenda for big data, highlighting the importance of assessing what to continue, stop, and start within the big data ecosystem. It discusses various technologies like Spark, Kafka, Hive, and NiFi, comparing their roles and advancements, while also noting the decline of tools like Pig and Oozie. Key takeaways include the introduction of Druid and JanusGraph for fast analytics and graph data processing, respectively, and the promotion of a future meeting focused on Docker in Hadoop.
Spark Under the Hood - Meetup @ Data Science London
Spark Under the Hood - Meetup @ Data Science LondonDatabricks The document summarizes a meetup on Apache Spark hosted by Data Science London. It introduces the speakers - Sameer Farooqui, Doug Bateman, and Jon Bates - and their backgrounds in data science and Spark training. The agenda includes talks on a power plant predictive modeling demo using Spark and different approaches to parallelizing machine learning algorithms in Spark like model, divide and conquer, and data parallelism. It also provides overviews of Spark's machine learning library MLlib and common algorithms. The goal is for attendees to learn about Spark's unified engine and how to apply different machine learning techniques at scale.
Spark to DocumentDB connector
Spark to DocumentDB connectorDenny Lee The document discusses the Spark to DocumentDB connector, highlighting its capabilities for scalable and low-latency data processing and analytics for various data science scenarios. It details advantages such as distributed aggregations, fast IoT support, and pushdown predicate filtering, while providing links to demos and test results. Additionally, the document outlines the architecture of the connector and encourages users to engage with it through GitHub for further improvements and information.
Real-Time Analytics with Apache Cassandra and Apache Spark,
Real-Time Analytics with Apache Cassandra and Apache Spark,Swiss Data Forum Swiss Data Forum This document provides an overview of real-time analytics with Apache Cassandra and Apache Spark. It discusses how Spark can be used for stream processing over Cassandra for storage. Spark Streaming ingests real-time data from sources like Kafka and processes it using DStreams that operate on microbatches. This allows joining streaming and batch data. Cassandra is optimized for high write throughput and scales horizontally. The combination of Spark and Cassandra enables transactional analytics over large datasets in real-time.
Real-Time Analytics with Apache Cassandra and Apache Spark
Real-Time Analytics with Apache Cassandra and Apache SparkGuido Schmutz This document provides an overview of real-time analytics with Apache Cassandra and Apache Spark. It discusses how Spark can be used for stream processing over Cassandra for storage. Spark Streaming ingests real-time data from sources like Kafka and processes it using Spark transformations and actions. The processed data can be stored in Cassandra for querying. Cassandra is well suited for high write throughput and storing large amounts of data, while Spark enables fast in-memory processing and machine learning capabilities. Together, Spark and Cassandra provide a scalable solution for real-time analytics and querying of large datasets.
Openstack meetup lyon_2017-09-28
Openstack meetup lyon_2017-09-28Xavier Lucas This document summarizes the key aspects of a public cloud archive storage solution. It offers affordable and unlimited storage using standard transfer protocols. Data is stored using erasure coding for redundancy and fault tolerance. Accessing archived data takes 10 minutes to 12 hours depending on previous access patterns, with faster access for inactive archives. The solution uses middleware to handle sealing and unsealing archives along with tracking access patterns to regulate retrieval times.
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Streaming Analytics with Spark, Kafka, Cassandra and AkkaHelena Edelson This document discusses a new approach to building scalable data processing systems using streaming analytics with Spark, Kafka, Cassandra, and Akka. It proposes moving away from architectures like Lambda and ETL that require duplicating data and logic. The new approach leverages Spark Streaming for a unified batch and stream processing runtime, Apache Kafka for scalable messaging, Apache Cassandra for distributed storage, and Akka for building fault tolerant distributed applications. This allows building real-time streaming applications that can join streaming and historical data with simplified architectures that remove the need for duplicating data extraction and loading.
Bullet: A Real Time Data Query Engine
Bullet: A Real Time Data Query EngineDataWorks Summit The document presents a real-time data query engine developed by Yahoo, detailing its architecture, functionality, and use cases. It addresses issues with conventional data querying methods, emphasizing the need for rapid instrumentation validation and programmatic access, while introducing a lightweight, scalable solution for querying un-persisted data. Future plans include expanding the system's capabilities to enhance data handling and querying efficiency.
Benchmarking Solr Performance at Scale
Benchmarking Solr Performance at Scalethelabdude The document provides an overview of the SolrCloud architecture and its features, emphasizing scalability and performance for distributed search indexing. It covers aspects like configuration management using Zookeeper, indexing and query performance, as well as the Solr Scale Toolkit for cluster deployment and management. Key takeaways include the importance of tuning garbage collection and the necessity of understanding server performance and query handling for optimal usage.
Typesafe spark- Zalando meetup
Typesafe spark- Zalando meetupStavros Kontopoulos This document discusses Typesafe's Reactive Platform and Apache Spark. It describes Typesafe's Fast Data strategy of using a microservices architecture with Spark, Kafka, HDFS and databases. It outlines contributions Typesafe has made to Spark, including backpressure support, dynamic resource allocation in Mesos, and integration tests. The document also discusses Typesafe's customer support and roadmap, including plans to introduce Kerberos security and evaluate Tachyon.
Ad
More from Carol McDonald (18)
Introduction to machine learning with GPUs
Introduction to machine learning with GPUsCarol McDonald The document provides an introduction to machine learning concepts including supervised and unsupervised learning. It discusses classification and regression as examples of supervised learning techniques and clustering as an example of unsupervised learning. It also provides an overview of deep learning using neural networks and examples of convolutional neural networks and recurrent neural networks. The document emphasizes how GPUs have accelerated machine learning by enabling parallel processing.
Streaming healthcare Data pipeline using Apache APIs: Kafka and Spark with Ma...
Streaming healthcare Data pipeline using Apache APIs: Kafka and Spark with Ma...Carol McDonald The document details a data pipeline using Spark, MapR Event Store for Kafka, and MapR Database, focusing on continuously reading payment data and analyzing it through Spark SQL. It provides examples of streaming data applications in healthcare, including monitoring patient health using machine learning. The document also outlines how to read, transform, and write streaming data to a database while applying SQL queries for insights on physician payments.
Analyzing Flight Delays with Apache Spark, DataFrames, GraphFrames, and MapR-DB
Analyzing Flight Delays with Apache Spark, DataFrames, GraphFrames, and MapR-DBCarol McDonald The document introduces the use of GraphFrames with Apache Spark to analyze flight delays, detailing concepts such as graphs, vertices, and edges. It discusses graph algorithms like PageRank, connected components, and motif queries, applying them to real-world examples including social media and flight data analysis. Finally, it illustrates how to create and query a flight dataset with GraphFrames and MapR-DB, demonstrating capabilities such as counting airports and filtering flight routes.
Analysis of Popular Uber Locations using Apache APIs: Spark Machine Learning...
Analysis of Popular Uber Locations using Apache APIs: Spark Machine Learning...Carol McDonald The document provides an overview of utilizing Apache technologies for real-time analysis of Uber locations, employing machine learning techniques such as k-means clustering and stream processing. It discusses the implementation of Spark, Kafka, and MapR for data ingestion, feature extraction, and model training to analyze geographical clustering of vehicles. Additionally, it presents how to handle streamed Uber data and model evaluation in a practical application context.
Predicting Flight Delays with Spark Machine Learning
Predicting Flight Delays with Spark Machine LearningCarol McDonald This document provides an overview of using Spark 2.x machine learning pipelines to predict flight delays, covering concepts in machine learning, particularly supervised learning and classification. It outlines the steps to build a machine learning solution, explores flight dataset handling with Spark, and demonstrates various operations on the data such as transformations and actions. Additionally, it discusses the use of algorithms like random forests and decision trees for prediction tasks.
Structured Streaming Data Pipeline Using Kafka, Spark, and MapR-DB
Structured Streaming Data Pipeline Using Kafka, Spark, and MapR-DBCarol McDonald This document discusses building a streaming data pipeline using Apache technologies like Kafka, Spark Streaming, and MapR-DB. It describes collecting streaming data with Kafka, organizing the data into topics, and processing the streams in Spark Streaming. The streaming data can then be stored in MapR-DB and queried using Spark SQL. An example uses a streaming payment dataset to demonstrate parsing the data, transforming it into a Dataset, and continuously aggregating values with Spark Streaming.
Streaming Machine learning Distributed Pipeline for Real-Time Uber Data Using...
Streaming Machine learning Distributed Pipeline for Real-Time Uber Data Using...Carol McDonald The document discusses the integration of machine learning with IoT and streaming data, specifically in the context of analyzing Uber trip data using Apache technologies such as Kafka and Spark. It covers key concepts of machine learning, including supervised and unsupervised learning, and provides an end-to-end architecture for processing real-time data. The document also illustrates practical use cases and offers examples of implementing Spark ML for monitoring and clustering Uber data.
Applying Machine Learning to IOT: End to End Distributed Pipeline for Real-Ti...
Applying Machine Learning to IOT: End to End Distributed Pipeline for Real-Ti...Carol McDonald This document discusses using Apache technologies like Kafka, Spark, and HBase to build an end-to-end machine learning pipeline for real-time analysis of Uber trip data. It provides an example of using K-means clustering on streaming Uber trip data to identify geographic patterns and visualize them in a dashboard. The document also provides background on machine learning, streaming data, Spark, and why combining IoT with machine learning is useful for applications like predictive maintenance, smart cities, healthcare, and more.
Applying Machine Learning to IOT: End to End Distributed Pipeline for Real- T...
Applying Machine Learning to IOT: End to End Distributed Pipeline for Real- T...Carol McDonald The document discusses the integration of machine learning with IoT for real-time analysis, focusing on a distributed pipeline for Uber trip data using Apache technologies like Kafka, Spark, and HBase. It highlights the importance of processing streaming data efficiently, various machine learning techniques, and the applicability of these technologies in sectors such as healthcare, automotive, and smart cities. Key topics include supervised and unsupervised learning, clustering, classification algorithms, and practical use cases for real-time event processing.
Demystifying AI, Machine Learning and Deep Learning
Demystifying AI, Machine Learning and Deep LearningCarol McDonald The document outlines the application of machine learning in IoT, focusing on real-time data processing using Apache technologies such as Spark and Kafka. It discusses the evolution of AI and machine learning, their advantages over traditional rule-based systems, and various types of machine learning including supervised and unsupervised learning. Additionally, it highlights the practical implementations of these technologies in areas such as fraud detection and predictive analytics in healthcare.
Spark graphx
Spark graphxCarol McDonald This document provides an introduction to GraphX, which is an Apache Spark component for graphs and graph-parallel computations. It describes different types of graphs like regular graphs, directed graphs, and property graphs. It shows how to create a property graph in GraphX by defining vertex and edge RDDs. It also demonstrates various graph operators that can be used to perform operations on graphs, such as finding the number of vertices/edges, degrees, longest paths, and top vertices by degree. The goal is to introduce the basics of representing and analyzing graph data with GraphX.
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...
Applying Machine learning to IOT: End to End Distributed Distributed Pipeline...Carol McDonald The document discusses the integration of IoT and machine learning for real-time data analysis, specifically using Uber trip data and various Apache technologies such as Kafka, Spark, and HBase. It covers concepts such as streaming events, machine learning techniques (supervised and unsupervised learning), and the importance of processing data as it arrives for applications like fraud detection and smart cities. Additionally, it provides practical examples of data processing architecture and methods for clustering and analyzing location data.
Streaming patterns revolutionary architectures
Streaming patterns revolutionary architectures Carol McDonald This document discusses streaming data architectures and patterns. It begins with an overview of streams, their core components, and why streaming is useful for real-time analytics on big data sources like sensor data. Common streaming patterns are then presented, including event sourcing, the duality of streams and databases, command query responsibility separation, and using streams to materialize multiple views of the data. Real-world examples of streaming architectures in retail and healthcare are also briefly described. The document concludes with a discussion of scalability, fault tolerance, and data recovery capabilities of streaming systems.
Spark machine learning predicting customer churn
Spark machine learning predicting customer churnCarol McDonald This document provides an introduction to machine learning techniques including classification and clustering. It discusses supervised learning algorithms like decision trees and how they can be used for classification problems like predicting customer churn. Unsupervised learning techniques like clustering are also introduced. The remainder of the document demonstrates how to use Spark ML and Spark SQL to build a machine learning pipeline to predict customer churn using decision trees on telecom customer data. Key steps discussed include data loading, feature extraction, model training, cross validation, and evaluation.
Apache Spark Machine Learning
Apache Spark Machine LearningCarol McDonald This document discusses machine learning techniques in Spark including classification, clustering, and collaborative filtering. It provides examples of building classification models with Spark including vectorizing data, training models, evaluating models, and making predictions. Clustering and collaborative filtering are also introduced. The document demonstrates collaborative filtering with Spark using alternating least squares to build a recommendation model from user ratings data.
Machine Learning Recommendations with Spark
Machine Learning Recommendations with SparkCarol McDonald Collaborative filtering algorithms recommend items to users based on the preferences of similar users. They work by building a model from user preference data on many items. The model can then be used to predict item preferences for new users based on similarities to other users with similar preferences. Alternating least squares (ALS) is an iterative collaborative filtering algorithm that approximates the user-item rating matrix as the product of two dense matrices to discover latent features of users and items.
CU9411MW.DOC
CU9411MW.DOCCarol McDonald The document discusses new TeMIP products for network management on Windows NT platforms. TeMIP Alarm Handling for Windows NT allows real-time alarm monitoring and analysis on Windows NT clients. The TeMIP Access Library Toolkit enables development of custom applications on Windows NT that can access TeMIP resources. Both products are part of the TeMIP V3.2A release and provide scalability, performance, standards compliance and other benefits for telecommunications network management.
Getting started with HBase
Getting started with HBaseCarol McDonald This document provides an overview and objectives of a session on getting started with HBase application development. It discusses why NoSQL and HBase are needed due to limitations of relational databases in scaling horizontally to handle big data. It provides an introduction to the HBase data model, architecture, and basic operations like put, get, scan, and delete. It explains how HBase stores data in a sorted map structure and how writes flow through the write ahead log, memstore, and are flushed to HFiles on disk.
Recently uploaded (20)
Raman Bhaumik - Passionate Tech Enthusiast
Raman Bhaumik - Passionate Tech EnthusiastRaman Bhaumik A Junior Software Developer with a flair for innovation, Raman Bhaumik excels in delivering scalable web solutions. With three years of experience and a solid foundation in Java, Python, JavaScript, and SQL, she has streamlined task tracking by 20% and improved application stability.
Database Benchmarking for Performance Masterclass: Session 2 - Data Modeling ...
Database Benchmarking for Performance Masterclass: Session 2 - Data Modeling ...ScyllaDB Specific best practices and pitfalls for benchmarking popular high-performance databases.
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdf
Oh, the Possibilities - Balancing Innovation and Risk with Generative AI.pdfPriyanka Aash Oh, the Possibilities - Balancing Innovation and Risk with Generative AI
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdfEarley Information Science As AI reshapes expectations in retail and B2B commerce, organizations are recognizing a critical reality: meaningful AI outcomes depend on well-structured, adaptable infrastructure. In this session, Seth Earley is joined by Phil Ryan - AI strategist, search technologist, and founder of Glass Leopard Technologies - for a candid conversation on what it truly means to engineer systems for scale, agility, and intelligence.
Phil draws on more than two decades of experience leading search and AI initiatives for enterprise organizations. Together, he and Seth explore the challenges businesses face when legacy architectures limit personalization, agility, and real-time decisioning - and what needs to change to support agentic technologies and next-best-action capabilities.
Key themes from the webinar include:
Composability as a prerequisite for AI - Why modular, loosely coupled systems are essential for adapting to rapid innovation and evolving business needs
Search and relevance as foundational to AI - How techniques honed-in enterprise search have laid the groundwork for more responsive and intelligent customer experiences
From MDM and CDP to agentic systems - How data platforms are evolving to support richer customer context and dynamic orchestration
Engineering for business alignment - Why successful AI programs require architectural decisions grounded in measurable outcomes
The conversation is practical and forward-looking, connecting deep technical understanding with real-world business needs. Whether you’re modernizing your commerce stack or exploring how AI can enhance product discovery, personalization, or customer journeys, this session provides a clear-eyed view of the capabilities, constraints, and priorities that matter most.
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FMESafe Software This session will cover a common use case for local and state/provincial governments who create and/or maintain their 9-1-1 addressing data, particularly address points and road centerlines. In this session, you'll learn how FME has helped Shelby County 9-1-1 (TN) automate the 9-1-1 addressing process; including automatically assigning attributes from disparate sources, on-the-fly QAQC of said data, and reporting. The FME logic that this presentation will cover includes: Table joins using attributes and geometry, Looping in custom transformers, Working with lists and Change detection.
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...
Wenn alles versagt - IBM Tape schützt, was zählt! Und besonders mit dem neust...Josef Weingand IBM LTO10
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025
CapCut Pro Crack For PC Latest Version {Fully Unlocked} 2025pcprocore 👉𝗡𝗼𝘁𝗲:𝗖𝗼𝗽𝘆 𝗹𝗶𝗻𝗸 & 𝗽𝗮𝘀𝘁𝗲 𝗶𝗻𝘁𝗼 𝗚𝗼𝗼𝗴𝗹𝗲 𝗻𝗲𝘄 𝘁𝗮𝗯> https://pcprocore.com/ 👈◀
CapCut Pro Crack is a powerful tool that has taken the digital world by storm, offering users a fully unlocked experience that unleashes their creativity. With its user-friendly interface and advanced features, it’s no wonder why aspiring videographers are turning to this software for their projects.
Security Tips for Enterprise Azure Solutions
Security Tips for Enterprise Azure SolutionsMichele Leroux Bustamante Delivering solutions to Azure may involve a variety of architecture patterns involving your applications, APIs data and associated Azure resources that comprise the solution. This session will use reference architectures to illustrate the security considerations to protect your Azure resources and data, how to achieve Zero Trust, and why it matters. Topics covered will include specific security recommendations for types Azure resources and related network security practices. The goal is to give you a breadth of understanding as to typical security requirements to meet compliance and security controls in an enterprise solution.
Mastering AI Workflows with FME by Mark Döring
Mastering AI Workflows with FME by Mark DöringSafe Software Harness the full potential of AI with FME: From creating high-quality training data to optimizing models and utilizing results, FME supports every step of your AI workflow. Seamlessly integrate a wide range of models, including those for data enhancement, forecasting, image and object recognition, and large language models. Customize AI models to meet your exact needs with FME’s powerful tools for training, optimization, and seamless integration
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?Shashi Sathyanarayana, Ph.D This slide illustrates a side-by-side comparison between human-written, AI-written, and ambiguous content. It highlights subtle cues that help readers assess authenticity, raising essential questions about the future of communication, trust, and thought leadership in the age of generative AI.
Agentic AI for Developers and Data Scientists Build an AI Agent in 10 Lines o...
Agentic AI for Developers and Data Scientists Build an AI Agent in 10 Lines o...All Things Open Presented at All Things Open RTP Meetup
Presented by William Hill - Developer Advocate, NVIDIA
Title: Agentic AI for Developers and Data Scientists
Build an AI Agent in 10 Lines of Code and the Concepts Behind the Code
Abstract: In this talk we will demonstrate building a working data science AI agent in 10 lines of basic Python code in a Colab notebook. Our AI Agent will perform LLM prompt-driven visual analysis using open-source libraries. In this session we will show how to develop an AI Agent using GPUs through NVIDIA’s developer program and Google Colab notebooks. After coding our AI Agent, we will break down the 10 lines of code. We will show the key components and open source library integrations that enable the agent's functionality, focusing on practical implementation details and then the theoretical concepts. The presentation concludes with a survey of current LLM technologies and the latest trends in developing AI applications for enthusiasts and enterprises
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/mpu-a-transformative-solution-for-next-gen-ai-at-the-edge-a-presentation-from-fotonation/
Petronel Bigioi, CEO of FotoNation, presents the “MPU+: A Transformative Solution for Next-Gen AI at the Edge” tutorial at the May 2025 Embedded Vision Summit.
In this talk, Bigioi introduces MPU+, a novel programmable, customizable low-power platform for real-time, localized intelligence at the edge. The platform includes an AI-augmented image signal processor that enables leading image and video quality.
In addition, it integrates ultra-low-power object and motion detection capabilities to enable always-on computer vision. A programmable neural processor provides flexibility to efficiently implement new neural networks. And additional specialized engines facilitate image stabilization and audio enhancements.
Connecting Data and Intelligence: The Role of FME in Machine Learning
Connecting Data and Intelligence: The Role of FME in Machine LearningSafe Software In this presentation, we want to explore powerful data integration and preparation for Machine Learning. FME is known for its ability to manipulate and transform geospatial data, connecting diverse data sources into efficient and automated workflows. By integrating FME with Machine Learning techniques, it is possible to transform raw data into valuable insights faster and more accurately, enabling intelligent analysis and data-driven decision making.
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptxFIDO Alliance Securing Account Lifecycles in the Age of Deepfakes
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdfyosra Saidani Salesforce Summer '25 Release Frenchgathering.pptx.pdf
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...
" How to survive with 1 billion vectors and not sell a kidney: our low-cost c...Fwdays Let's talk about our history. How we started the project with a small vector database of less than 2 million records. Later, we received a request for +100 million records, then another +100... And so gradually we reached almost 1 billion. Standard tools were quickly running out of steam - we were running into performance, index size, and very limited resources. After a long series of trials and errors, we built our own low-cost cluster, which today stably processes thousands of queries to more than 1B vectors.
Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes ProbableSaikat Basu Imagine combining the "brains" of Artificial Intelligence (AI) with the "super muscles" of Quantum Computing. That's Quantum AI!
It's a new field that uses the mind-bending rules of quantum physics to make AI even more powerful.
Cracking the Code - Unveiling Synergies Between Open Source Security and AI.pdf
Cracking the Code - Unveiling Synergies Between Open Source Security and AI.pdfPriyanka Aash Cracking the Code - Unveiling Synergies Between Open Source Security and AI
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance
Build a Time Series Application with Apache Spark and Apache HBase
- 1. @tgrall#Devoxx #sparkstreaming Build a Time Series Application with Spark and HBase Tugdual Grall @tgrall MapR Carol McDonald @caroljmcdonald MapR
- 2. @tgrall#Devoxx #sparkstreaming Agenda • Time Series • Apache Spark & Spark Streaming • Apache HBase • Lab
- 3. @tgrall#Devoxx #sparkstreaming About the Lab • Use Spark & HBase in MapR Cluster • Option 1: Use a SandBox (Virtual Box VM located on USB Key) • Option 2: Use Cloud Instance (SSH/SCP only) • Content: • Option 1: spark-streaming-hbase-workshop.zip on USB • Option 2: download zip from https://github.com/tgrall/spark-streaming-hbase-workshop
- 5. @tgrall#Devoxx #sparkstreaming What is a Time Series? • Stuff with timestamps • sensor measurements • system stats • log files • ….
- 6. @tgrall#Devoxx #sparkstreaming Got Some Examples?
- 17. @tgrall#Devoxx #sparkstreaming What do we need to do? • Acquire • Measurement, transmission, reception • Store • Individually, or grouped for some amount of time • Retrieve • Ad hoc, flexible, correlate and aggregate • Analyze and visualize • We facilitate this via retrieval
- 18. @tgrall#Devoxx #sparkstreaming Acquisition Not usually our problem • Sensors • Data collection – agents, raspberry pi • Transmission – via LAN/Wan, Mobile Network, Satellites • Receipt into system – listening daemon or queue, or depending on use case writing directly to the database
- 19. @tgrall#Devoxx #sparkstreaming Storage Choice • Flat files • Great for rapid ingest with massive data • Handles essentially any data type • Less good for data requiring frequent updates • Harder to find specific ranges • Traditional RDBMS • Ingests up to ~10,000/ sec; prefers well structured (numerical) data; expensive • NoSQL (such as MapR-DB or HBase) • Easily handle 10,000 rows / sec / node – True linear scaling • Handles wide variety of data • Good for frequent updates • Easily scanned in a range
- 20. @tgrall#Devoxx #sparkstreaming Specific Example Consider oil drilling rigs • When drilling wells, there are *lots* of moving parts • Typically a drilling rig makes about 10K samples/s • Temperatures, pressures, magnetics, machine vibration levels, salinity, voltage, currents, many others • Typical project has 100 rigs
- 21. @tgrall#Devoxx #sparkstreaming General Outline 10K samples / second / rig x 100 rigs = 1M samples / second • But wait, there’s more • Suppose you want to test your system • Perhaps with a year of data • And you want to load that data in << 1 year • 100x real-time = 100M samples / second
- 22. @tgrall#Devoxx #sparkstreaming Data Storage • Typical time window is one hour • Column names are offsets in time window • Find series-uid in separate table Key 13 43 73 103 … … series-uid.time-window 4.5 5.2 6.1 4.9 …
- 24. @tgrall#Devoxx #sparkstreaming Why do we need NoSQL / HBase? Relational Model bottleneck Key colB colC val val val xxx val val Key colB colC val val val xxx val val Key colB colC val val val xxx val val Storage ModelRDBMS HBase Distributed Joins, Transactions do not scale Data that is accessed together is stored together
- 25. @tgrall#Devoxx #sparkstreaming HBase is a ColumnFamily oriented Database • Data is accessed and stored together: • RowKey is the primary index • Column Families group similar data by row key CF_DATA colA colB colC Val val val CF_STATS colA colB colC val val val RowKey series-abc.time- window series-efg.time- window Customer id Raw Data Stats
- 26. @tgrall#Devoxx #sparkstreaming HBase is a Distributed Database Key Range xxxx xxxx CF1 colA colB colC val val val CF2 colA colB colC val val val Key Range xxxx xxxx CF1 colA colB colC val val val CF2 colA colB colC val val val Key Range xxxx xxxx CF1 colA colB colC val val val CF2 colA colB colC val val val Put, Get by Key Data is automatically distributed across the cluster • Key range is used for horizontal partitioning
- 27. @tgrall#Devoxx #sparkstreaming Basic Table Operations • Create Table, define Column Families before data is imported • but not the rows keys or number/names of columns • Low level API, technically more demanding • Basic data access operations (CRUD): put Inserts data into rows (both create and update) get Accesses data from one row scan Accesses data from a range of rows delete Delete a row or a range of rows or columns
- 28. @tgrall#Devoxx #sparkstreaming Learn More • Free Online Training: http://learn.mapr.com • DEV 320 - Apache HBase Data Model and Architecture • DEV 325 - Apache HBase Schema Design • DEV 330 - Developing Apache HBase Applications: Basics • DEV 335 - Developing Apache HBase Applications: Advanced
- 30. @tgrall#Devoxx #sparkstreaming What is Spark? • Cluster Computing Platform • Extends “MapReduce” with extensions • Streaming • Interactive Analytics • Run in Memory
- 31. @tgrall#Devoxx #sparkstreaming What is Spark? Fast • 100x faster than M/R Logistic regression in Hadoop and Spark
- 32. @tgrall#Devoxx #sparkstreaming What is Spark? Ease of Development • Write programs quickly • More Operators • Interactive Shell • Less Code
- 33. @tgrall#Devoxx #sparkstreaming What is Spark? Multi Language Support • Scala • Python • Java • SparkR
- 34. @tgrall#Devoxx #sparkstreaming What is Spark? Deployment Flexibility • Deployment • Local • Standalone • Storage • HDFS • MapR-FS • S3 • Cassandra • YARN • Mesos
- 35. @tgrall#Devoxx #sparkstreaming Unified Platform Spark SQL Spark Streaming (Streaming) MLlib (Machine Learning) Spark Core (General execution engine) GraphX (Graph Computation)
- 36. @tgrall#Devoxx #sparkstreaming Spark Components Driver Program (application) SparkContext Cluster Manager Worker Executor Task Task Worker Executor Task Task
- 37. @tgrall#Devoxx #sparkstreaming Spark Resilient Distributed Datasets Sensor RDD W Executor P4 W Executor P1 P3 W Executor P2 sc.textFile P1 8213034705, 95, 2.927373, jake7870, 0…… P2 8213034705, 115, 2.943484, Davidbresler2, 1…. P3 8213034705, 100, 2.951285, gladimacowgirl, 58… P4 8213034705, 117, 2.998947, daysrus, 95….
- 38. @tgrall#Devoxx #sparkstreaming Spark Resilient Distributed Datasets Transformation Filter() Action Count() RDD newRDD Value
- 39. @tgrall#Devoxx #sparkstreaming Spark Streaming Spark SQL Spark Streaming (Streaming) MLlib (Machine Learning) Spark Core (General execution engine) GraphX (Graph Computation)
- 40. @tgrall#Devoxx #sparkstreaming What is Streaming? • Data Stream: • Unbounded sequence of data arriving continuously • Stream processing: • Low latency processing, querying, and analyzing of real time streaming data
- 41. @tgrall#Devoxx #sparkstreaming Why Spark Streaming • Many applications must process streaming data • With the following Requirements: • Results in near-real-time • Handle large workloads • latencies of few seconds • Use Cases • Website statistics, monitoring • IoT • Fraud detection • Social network trends • Advertising click monetization put put put put Time stamped data data • Sensor, System Metrics, Events, log files • Stock Ticker, User Activity • Hi Volume, Velocity Data for real-time monitoring
- 42. @tgrall#Devoxx #sparkstreaming What is Spark Streaming? • Enables scalable, high-throughput, fault-tolerant stream processing of live data • Extension of the core Spark Data Sources Data Sinks
- 43. @tgrall#Devoxx #sparkstreaming Spark Streaming Architecture • Divide data stream into batches of X seconds • Called DStream = sequence of RDDs Spark Streaming input data stream DStream RDD batches Batch interval data from time 0 to 1 data from time 1 to 2 RDD @ time 2 data from time 2 to 3 RDD @ time 3RDD @ time 1
- 44. @tgrall#Devoxx #sparkstreaming Process DStream • Process using transformations • creates new RDDs transform Transform map reduceByValue count DStream RDDs Dstream RDDs transformtransform data from time 0 to 1 data from time 1 to 2 RDD @ time 2 data from time 2 to 3 RDD @ time 3RDD @ time 1 RDD @ time 1 RDD @ time 2 RDD @ time 3
- 45. @tgrall#Devoxx #sparkstreaming Time Series Data for real-time monitoring read Sensor Time stamped data HBase Processing data
- 47. @tgrall#Devoxx #sparkstreaming Convert Line of CSV data to Sensor Object case class Sensor(resid: String, date: String, time: String, hz: Double, disp: Double, flo: Double, sedPPM: Double, psi: Double, chlPPM: Double) def parseSensor(str: String): Sensor = { val p = str.split(",") Sensor(p(0), p(1), p(2), p(3).toDouble, p(4).toDouble, p(5).toDouble, p(6).toDouble, p(7).toDouble, p(8).toDouble) }
- 48. @tgrall#Devoxx #sparkstreaming Create a DStream val ssc = new StreamingContext(sparkConf, Seconds(2)) val linesDStream = ssc.textFileStream(“/mapr/stream") batch time 0-1 linesDStream batch time 1-2 batch time 1-2 DStream: a sequence of RDDs representing a stream of data stored in memory as an RDD
- 49. @tgrall#Devoxx #sparkstreaming Process DStream val linesDStream = ssc.textFileStream(”directory path") val sensorDStream = linesDStream.map(parseSensor) map new RDDs created for every batch batch time 0-1 linesDStream RDDs sensorDstream RDDs batch time 1-2 mapmap batch time 1-2
- 50. @tgrall#Devoxx #sparkstreaming Save to HBase rdd.map(Sensor.convertToPut).saveAsHadoopDataset(jobConfig) Put objects written To HBase batch time 0-1 linesRDD DStream sensorRDD Dstream batch time 1-2 map batch time 1-2 HBase save save save output operation: persist data to external storage map map
- 51. @tgrall#Devoxx #sparkstreaming Learn More • Free Spark Online Training: http://learn.mapr.com