IoT Applications and Patterns using Apache Spark & Apache Bahir
1 like1,013 views
The document discusses IoT applications and patterns using Apache Spark and Apache Bahir, focusing on integrating various data sources and streaming capabilities for real-time analytics. It highlights the core functionalities of Apache Spark, including Spark SQL, Spark Streaming, and the extensions provided by Apache Bahir for IoT connectivity such as MQTT and Cloudant. Additionally, it presents a live demo simulating an IoT environment, outlining the benefits of leveraging these technologies for processing IoT data.
1 of 43
Downloaded 25 times
![Apache Spark – Spark SQL
11
You can run SQL statement with SparkSession.sql(…) interface:
val spark = SparkSession.builder()
.appName(“Demo”)
.getOrCreate()
spark.sql(“create table T1 (c1 int, c2 int) stored as parquet”)
val ds = spark.sql(“select * from T1”)
You can further transform the resultant dataset:
val ds1 = ds.groupBy(“c1”).agg(“c2”-> “sum”)
val ds2 = ds.orderBy(“c1”)
The result is a DataFrame / Dataset[Row]
ds.show() displays the rows](https://image.slidesharecdn.com/apacheroadshow-iotapplicationsandpatternsusingapachesparkapachebahir-180712164137/85/IoT-Applications-and-Patterns-using-Apache-Spark-Apache-Bahir-10-320.jpg)
![Apache Spark – Spark SQL
You can read from data sources using SparkSession.read.format(…)
val spark = SparkSession.builder()
.appName(“Demo”)
.getOrCreate()
case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)
// loading csv data to a Dataset of Bank type
val bankFromCSV = spark.read.csv(“hdfs://localhost:9000/data/bank.csv").as[Bank]
// loading JSON data to a Dataset of Bank type
val bankFromJSON = spark.read.json(“hdfs://localhost:9000/data/bank.json").as[Bank]
// select a column value from the Dataset
bankFromCSV.select(‘age).show() will return all rows of column “age” from this dataset.
12](https://image.slidesharecdn.com/apacheroadshow-iotapplicationsandpatternsusingapachesparkapachebahir-180712164137/85/IoT-Applications-and-Patterns-using-Apache-Spark-Apache-Bahir-11-320.jpg)
![Apache Spark – Spark SQL
You can also configure a specific data source with specific options
val spark = SparkSession.builder()
.appName(“Demo”)
.getOrCreate()
case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)
// loading csv data to a Dataset of Bank type
val bankFromCSV = sparkSession.read
.option("header", ”true") // Use first line of all files as header
.option("inferSchema", ”true") // Automatically infer data types
.option("delimiter", " ")
.csv("/users/lresende/data.csv”)
.as[Bank]
bankFromCSV.select(‘age).show() // will return all rows of column “age” from this dataset.
13](https://image.slidesharecdn.com/apacheroadshow-iotapplicationsandpatternsusingapachesparkapachebahir-180712164137/85/IoT-Applications-and-Patterns-using-Apache-Spark-Apache-Bahir-12-320.jpg)
Ad
Recommended
Getting insights from IoT data with Apache Spark and Apache Bahir
Getting insights from IoT data with Apache Spark and Apache BahirLuciano Resende The document discusses the integration of IoT data insights using Apache Spark and Apache Bahir, highlighting their capabilities in processing real-time data and creating anomaly detection applications. It covers various IoT use cases and design patterns, such as lambda and kappa architectures, along with specific examples like smart homes and connected vehicles. Additionally, it emphasizes the importance of effective data handling and visualization in enterprise applications while showcasing live demos and contributions to the open-source community.
OGCE TeraGrid 2010 Science Gateway Tutorial Intro
OGCE TeraGrid 2010 Science Gateway Tutorial Intromarpierc The document summarizes a tutorial presentation about the Open Grid Computing Environments (OGCE) software tools for building science gateways. The OGCE tools include a gadget container, workflow composer called XBaya, and application factory service called GFac. The presentation demonstrates how these tools can be used to build portals and compose workflows to access resources like the TeraGrid.
Writing Apache Spark and Apache Flink Applications Using Apache Bahir
Writing Apache Spark and Apache Flink Applications Using Apache BahirLuciano Resende The document discusses the Apache Bahir project, established as a top-level Apache project for building applications using extensions to Apache Spark. It includes details about the project origins, benefits of using Apache Bahir, its relationship with Apache Spark, and various extension capabilities such as MQTT and Twitter data streaming. Additionally, it covers integration steps for using Bahir extensions in applications and showcases a demonstration environment for IoT simulations.
Apache Zeppelin Helium and Beyond
Apache Zeppelin Helium and BeyondDataWorks Summit/Hadoop Summit This document discusses Apache Zeppelin, an open-source web-based notebook that enables interactive data analytics. It describes Zeppelin's origins, key features like interpreters and notebook storage being pluggable, a proposed "Helium" project to extend Zeppelin into a full analytics application platform, the roadmap for further developing Zeppelin, and how to get involved with the Zeppelin community.
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)
Using the FLaNK Stack for edge ai (flink, nifi, kafka, kudu)Timothy Spann The document presents an overview of the Flink stack for edge AI, highlighting key components such as Apache NiFi and Kafka for data ingestion, management, and streaming analytics. It emphasizes the importance of a connected data lifecycle for real-time use cases and discusses Cloudera's commitment to community support and open-source contributions. Additionally, it includes various resources for further exploration of data streaming and Flink SQL capabilities.
Apache Zeppelin Meetup Christian Tzolov 1/21/16
Apache Zeppelin Meetup Christian Tzolov 1/21/16 PivotalOpenSourceHub The document outlines a unified data analytics platform integrating technologies such as Apache Zeppelin, Ambari, Geode, Spring XD, and Hawq, designed to manage and analyze large data streams. It highlights the capabilities of the platform to orchestrate processes, provide real-time analytics, and manage big data systems effectively. Key components include data lake management, in-memory data grids, and SQL compliance, facilitating high-performance applications for organizations like China Railway and Indian Railways.
Seattle spark-meetup-032317
Seattle spark-meetup-032317Nan Zhu The document discusses the design and implementation of Spark Streaming connectors for real-time data sources like Azure Event Hubs. It covers key aspects like connecting Event Hubs to Spark Streaming, designing the connector to minimize resource usage, ensuring fault tolerance through checkpointing and recovery, and managing message offsets and processing rates in a distributed manner. The connector design addresses challenges like long-running receivers, extra resource requirements, and data loss during failures. Lessons from the initial receiver-based approach informed the design of a more efficient solution.
NigthClazz Spark - Machine Learning / Introduction à Spark et Zeppelin
NigthClazz Spark - Machine Learning / Introduction à Spark et ZeppelinZenika The document introduces Apache Spark, a cluster computing framework that enables scalable data processing, and Apache Zeppelin, a web-based notebook for interactive data analytics. It covers Spark's history, core concepts like RDDs and DataFrames, and its capabilities for machine learning through Spark MLlib. Zeppelin facilitates data visualization and collaboration, supporting multiple programming languages for data ingestion and analysis.
Using Apache Spark with IBM SPSS Modeler
Using Apache Spark with IBM SPSS ModelerGlobal Knowledge Training The document presents an overview of using Apache Spark within IBM SPSS Modeler, highlighting its advantages over traditional Hadoop MapReduce in terms of speed and processing capabilities. It also details various machine learning techniques that can be applied using Spark's MLlib, such as gradient boosted trees, k-means clustering, and multinomial naive bayes. Additionally, the document provides installation guidelines and integration steps for leveraging these applications effectively within the SPSS environment.
Spark Summit EU talk by Steve Loughran
Spark Summit EU talk by Steve LoughranSpark Summit This document provides an overview of using Apache Spark with object stores like Amazon S3, Azure Blob Storage, and Google Cloud Storage. It discusses the key challenges of classpath configuration, credentials, code examples, and ensuring data consistency and durability. Specific tips are provided for configuring and working with S3 and Azure Blob Storage. The document emphasizes that object stores can be treated like any other URL, but some configuration is needed and performance/commitment challenges exist.
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Sparkfelixcss The document discusses using Spark for interactive analysis and visualization, detailing setup procedures, tools, and their functionalities such as PySpark, Jupyter, and Apache Zeppelin. It covers topics like k-means clustering, anomaly detection, and the streaming k-means algorithm, along with visualization techniques using libraries like Matplotlib and Lightning. Additionally, it highlights challenges related to package management and offers resources for implementation and further reading.
Apache Spark: Lightning Fast Cluster Computing
Apache Spark: Lightning Fast Cluster ComputingAll Things Open Apache Spark is an open-source cluster computing framework for fast and large-scale data processing. It uses an in-memory data abstraction called resilient distributed datasets (RDDs) that allow parallel operations on large datasets across a cluster. Spark also provides APIs in Java, Scala, Python and R for interactive data analysis through its core engine as well as high-level libraries for SQL, streaming, machine learning and graph processing.
Real-time Streaming Pipelines with FLaNK
Real-time Streaming Pipelines with FLaNKData Con LA The document details a presentation on real-time streaming pipelines using Apache Flink, Apache NiFi, and Apache Kafka, aimed at cloud data engineers. It highlights the functionalities of these tools for data ingestion, processing, and stream management, alongside sample data structures and SQL streaming capabilities. The content also touches upon practical applications, features of each tool, and links to related resources for further exploration.
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit This document discusses bringing multi-tenancy to Apache Zeppelin through the integration of Apache Livy. It introduces Zeppelin as an interactive web-based notebook for data analysis. It then describes how Livy allows Zeppelin to support multi-tenancy by enabling user authentication and session management for Spark jobs submitted through Zeppelin. The document provides an overview of Livy's architecture and security features like SPNEGO and impersonation that facilitate secure multi-tenant access to shared Spark clusters from Zeppelin.
Apache Zeppelin, Helium and Beyond
Apache Zeppelin, Helium and BeyondDataWorks Summit/Hadoop Summit Apache Zeppelin is an open-source web-based notebook that enables interactive data analytics. It began in 2012 as an internal tool called Zeppelin at NIF Labs and was later open-sourced. In 2014 it joined Apache Incubator and became an Apache project in 2016. Helium is a proposed next version of Zeppelin that aims to make visualizations and applications pluggable modules. This would allow users to more easily extend Zeppelin's capabilities.
S3Guard: What's in your consistency model?
S3Guard: What's in your consistency model?Hortonworks S3Guard provides a consistent metadata store for S3 using DynamoDB. It allows file system operations on S3, like listing and getting file status, to be consistent by checking results from S3 against metadata stored in DynamoDB. Mutating operations write to both S3 and DynamoDB, while read operations first check S3 results against DynamoDB to handle eventual consistency in S3. The goal is to improve performance of real workloads by providing consistent metadata operations on S3 objects written with S3Guard enabled.
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018Timothy Spann The document discusses IoT edge processing using Apache NiFi, MiNiFi, and Apache MXNet, highlighting their roles in ingesting and analyzing data from various sources like cameras and sensors. It outlines the capabilities of these technologies for real-time data flows, object recognition, and data enrichment using machine learning. Additionally, it presents the potential applications of these tools in enterprise settings, emphasizing their integration with different devices and protocols.
Intro to Big Data Analytics using Apache Spark and Apache Zeppelin
Intro to Big Data Analytics using Apache Spark and Apache ZeppelinAlex Zeltov The document is a comprehensive workshop presentation introducing Apache Spark and its components, including SparkSQL, MLlib, and Spark Streaming. It highlights Spark's capabilities in big data analytics, programming interfaces, and integration with Hadoop and Zeppelin for interactive data processing. In addition, it covers the architecture, benefits, and typical use cases of Apache Spark in handling large datasets efficiently.
Apache Spark Overview
Apache Spark OverviewairisData Tim Spann will present on learning Apache Spark. He is a senior solutions architect who previously worked as a senior field engineer and startup engineer. airis.DATA, where Spann works, specializes in machine learning and graph solutions using Spark, H20, Mahout, and Flink on petabyte datasets. The agenda includes an overview of Spark, an explanation of MapReduce, and hands-on exercises to install Spark, run a MapReduce job locally, and build a project with IntelliJ and SBT.
Migrating pipelines into Docker
Migrating pipelines into DockerDataWorks Summit/Hadoop Summit This document discusses Spotify's migration of data pipelines to Docker. It provides background on Spotify growing from 50 to 1000 engineers and the challenges of scaling their big data infrastructure. Spotify adopted Docker to help solve packaging and dependency issues, moving pipelines from cron jobs to a REST API and Docker images. Docker is allowing Spotify to transparently migrate their on-premise Hadoop cluster to Google Cloud, handling over 100 petabytes of data and growing.
Matt Franklin - Apache Software (Geekfest)
Matt Franklin - Apache Software (Geekfest)W2O Group The document discusses the potential benefits of container technologies like Docker. It notes that containers offer significantly higher density than virtual machines by avoiding hypervisor overhead. This density improvement can lead to major cost reductions by reducing infrastructure needs. Containers also improve developer efficiency by making development environments portable and disposable. This allows more rapid experimentation and innovation, potentially translating to increased revenue. Technologies like Amazon Lambda take the on-demand aspects of containers even further by abstracting compute resources. The document promotes StackEngine as a solution for managing containers at scale in production environments.
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT with Denis Magda
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT with Denis MagdaDatabricks The document discusses Apache Ignite and its relevance to the Internet of Things (IoT), outlining key components such as real-time data processing, data collection, and enrichment, as well as the Apache Mynewt RTOS. It highlights the smart metering use case with a focus on architecture, scalability, and security features. Additionally, the document provides resources for further learning and integration with Apache Spark.
Streamline Hadoop DevOps with Apache Ambari
Streamline Hadoop DevOps with Apache AmbariDataWorks Summit/Hadoop Summit Apache Ambari is an open-source tool for provisioning, managing, and monitoring Hadoop clusters. It allows users to deploy Hadoop clusters, install and manage services, configure settings, and perform rolling upgrades with minimal downtime. Ambari 2.4 includes new features like role-based access control, Grafana integration for visualization, log search capabilities, and improved upgrade workflows.
Spark,Hadoop,Presto Comparition
Spark,Hadoop,Presto ComparitionSandish Kumar H N The document discusses several big data frameworks: Spark, Presto, Cloudera Impala, and Apache Hadoop. Spark aims to make data analytics faster by loading data into memory for iterative querying. Presto extends R with distributed parallelism for scalable machine learning and graph algorithms. Hadoop uses MapReduce to distribute computations across large hardware clusters and handles failures automatically. While useful for batch processing, Hadoop has disadvantages for small files and online transactions.
August 2016 HUG: Recent development in Apache Oozie
August 2016 HUG: Recent development in Apache OozieYahoo Developer Network The document discusses the implementation and usage of Oozie at Yahoo, focusing on its capabilities for managing complex data pipelines and job dependencies across various platforms like Spark and Hive. It outlines the efficient handling of workflow jobs and coordinator jobs with insights on current limitations, enhancements, and future work in unit testing and dependency management. Additionally, it highlights Oozie's native support for Spark jobs and recent improvements in managing Spark actions within Oozie workflows.
Data Science with Spark & Zeppelin
Data Science with Spark & ZeppelinVinay Shukla This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It can be used for data exploration, visualization, collaboration and publishing. Zeppelin has deep integration with Apache Spark and supports multiple languages including Scala, Python, and SQL. It provides a Spark interpreter that allows users to analyze data using Spark without having to configure Spark themselves. The document demonstrates Zeppelin's functionality through examples and encourages readers to try it out and get involved in the community.
Apache Fink 1.0: A New Era for Real-World Streaming Analytics
Apache Fink 1.0: A New Era for Real-World Streaming AnalyticsSlim Baltagi The document discusses the evolution and capabilities of Apache Flink, a distributed dataflow streaming engine, with emphasis on its features that make it suitable for real-world streaming analytics. Key points include Flink's low latency, high throughput, stateful processing, fault tolerance, and support for various window semantics and time notions. Additionally, it outlines the roadmap for future enhancements in streaming capabilities post version 1.0.
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit The document provides a comprehensive overview of troubleshooting Kerberos in Hadoop, highlighting its significance as a universal authentication mechanism and its role across the Hadoop stack. It covers topics such as the workings of Kerberos, realms, principals, tools available in Hadoop, a systematic approach to diagnosing issues, and common error messages. The author, Vipin Rathor, emphasizes the importance of proper configuration and the integration of Kerberos with enterprise user management for secure Hadoop operations.
Building iot applications with Apache Spark and Apache Bahir
Building iot applications with Apache Spark and Apache BahirLuciano Resende The document discusses the IBM Spark Technology Center's contributions to open-source projects, particularly focusing on Apache Spark and Apache Bahir for building IoT applications. It includes technical details about the functionalities of Apache Spark, such as Spark SQL, streaming capabilities, and IoT connectivity protocols like MQTT. The document emphasizes the importance of community engagement and the potential of using these technologies for real-time IoT data processing.
Spark Concepts - Spark SQL, Graphx, Streaming
Spark Concepts - Spark SQL, Graphx, StreamingPetr Zapletal This document provides an overview of Apache Spark modules including Spark SQL, GraphX, and Spark Streaming. Spark SQL allows querying structured data using SQL, GraphX provides APIs for graph processing, and Spark Streaming enables scalable stream processing. The document discusses Resilient Distributed Datasets (RDDs), SchemaRDDs, querying data with SQLContext, GraphX property graphs and algorithms, StreamingContext, and input/output operations in Spark Streaming.
More Related Content
What's hot (20)
Using Apache Spark with IBM SPSS Modeler
Using Apache Spark with IBM SPSS ModelerGlobal Knowledge Training The document presents an overview of using Apache Spark within IBM SPSS Modeler, highlighting its advantages over traditional Hadoop MapReduce in terms of speed and processing capabilities. It also details various machine learning techniques that can be applied using Spark's MLlib, such as gradient boosted trees, k-means clustering, and multinomial naive bayes. Additionally, the document provides installation guidelines and integration steps for leveraging these applications effectively within the SPSS environment.
Spark Summit EU talk by Steve Loughran
Spark Summit EU talk by Steve LoughranSpark Summit This document provides an overview of using Apache Spark with object stores like Amazon S3, Azure Blob Storage, and Google Cloud Storage. It discusses the key challenges of classpath configuration, credentials, code examples, and ensuring data consistency and durability. Specific tips are provided for configuring and working with S3 and Azure Blob Storage. The document emphasizes that object stores can be treated like any other URL, but some configuration is needed and performance/commitment challenges exist.
Sparkly Notebook: Interactive Analysis and Visualization with Spark
Sparkly Notebook: Interactive Analysis and Visualization with Sparkfelixcss The document discusses using Spark for interactive analysis and visualization, detailing setup procedures, tools, and their functionalities such as PySpark, Jupyter, and Apache Zeppelin. It covers topics like k-means clustering, anomaly detection, and the streaming k-means algorithm, along with visualization techniques using libraries like Matplotlib and Lightning. Additionally, it highlights challenges related to package management and offers resources for implementation and further reading.
Apache Spark: Lightning Fast Cluster Computing
Apache Spark: Lightning Fast Cluster ComputingAll Things Open Apache Spark is an open-source cluster computing framework for fast and large-scale data processing. It uses an in-memory data abstraction called resilient distributed datasets (RDDs) that allow parallel operations on large datasets across a cluster. Spark also provides APIs in Java, Scala, Python and R for interactive data analysis through its core engine as well as high-level libraries for SQL, streaming, machine learning and graph processing.
Real-time Streaming Pipelines with FLaNK
Real-time Streaming Pipelines with FLaNKData Con LA The document details a presentation on real-time streaming pipelines using Apache Flink, Apache NiFi, and Apache Kafka, aimed at cloud data engineers. It highlights the functionalities of these tools for data ingestion, processing, and stream management, alongside sample data structures and SQL streaming capabilities. The content also touches upon practical applications, features of each tool, and links to related resources for further exploration.
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit This document discusses bringing multi-tenancy to Apache Zeppelin through the integration of Apache Livy. It introduces Zeppelin as an interactive web-based notebook for data analysis. It then describes how Livy allows Zeppelin to support multi-tenancy by enabling user authentication and session management for Spark jobs submitted through Zeppelin. The document provides an overview of Livy's architecture and security features like SPNEGO and impersonation that facilitate secure multi-tenant access to shared Spark clusters from Zeppelin.
Apache Zeppelin, Helium and Beyond
Apache Zeppelin, Helium and BeyondDataWorks Summit/Hadoop Summit Apache Zeppelin is an open-source web-based notebook that enables interactive data analytics. It began in 2012 as an internal tool called Zeppelin at NIF Labs and was later open-sourced. In 2014 it joined Apache Incubator and became an Apache project in 2016. Helium is a proposed next version of Zeppelin that aims to make visualizations and applications pluggable modules. This would allow users to more easily extend Zeppelin's capabilities.
S3Guard: What's in your consistency model?
S3Guard: What's in your consistency model?Hortonworks S3Guard provides a consistent metadata store for S3 using DynamoDB. It allows file system operations on S3, like listing and getting file status, to be consistent by checking results from S3 against metadata stored in DynamoDB. Mutating operations write to both S3 and DynamoDB, while read operations first check S3 results against DynamoDB to handle eventual consistency in S3. The goal is to improve performance of real workloads by providing consistent metadata operations on S3 objects written with S3Guard enabled.
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018Timothy Spann The document discusses IoT edge processing using Apache NiFi, MiNiFi, and Apache MXNet, highlighting their roles in ingesting and analyzing data from various sources like cameras and sensors. It outlines the capabilities of these technologies for real-time data flows, object recognition, and data enrichment using machine learning. Additionally, it presents the potential applications of these tools in enterprise settings, emphasizing their integration with different devices and protocols.
Intro to Big Data Analytics using Apache Spark and Apache Zeppelin
Intro to Big Data Analytics using Apache Spark and Apache ZeppelinAlex Zeltov The document is a comprehensive workshop presentation introducing Apache Spark and its components, including SparkSQL, MLlib, and Spark Streaming. It highlights Spark's capabilities in big data analytics, programming interfaces, and integration with Hadoop and Zeppelin for interactive data processing. In addition, it covers the architecture, benefits, and typical use cases of Apache Spark in handling large datasets efficiently.
Apache Spark Overview
Apache Spark OverviewairisData Tim Spann will present on learning Apache Spark. He is a senior solutions architect who previously worked as a senior field engineer and startup engineer. airis.DATA, where Spann works, specializes in machine learning and graph solutions using Spark, H20, Mahout, and Flink on petabyte datasets. The agenda includes an overview of Spark, an explanation of MapReduce, and hands-on exercises to install Spark, run a MapReduce job locally, and build a project with IntelliJ and SBT.
Migrating pipelines into Docker
Migrating pipelines into DockerDataWorks Summit/Hadoop Summit This document discusses Spotify's migration of data pipelines to Docker. It provides background on Spotify growing from 50 to 1000 engineers and the challenges of scaling their big data infrastructure. Spotify adopted Docker to help solve packaging and dependency issues, moving pipelines from cron jobs to a REST API and Docker images. Docker is allowing Spotify to transparently migrate their on-premise Hadoop cluster to Google Cloud, handling over 100 petabytes of data and growing.
Matt Franklin - Apache Software (Geekfest)
Matt Franklin - Apache Software (Geekfest)W2O Group The document discusses the potential benefits of container technologies like Docker. It notes that containers offer significantly higher density than virtual machines by avoiding hypervisor overhead. This density improvement can lead to major cost reductions by reducing infrastructure needs. Containers also improve developer efficiency by making development environments portable and disposable. This allows more rapid experimentation and innovation, potentially translating to increased revenue. Technologies like Amazon Lambda take the on-demand aspects of containers even further by abstracting compute resources. The document promotes StackEngine as a solution for managing containers at scale in production environments.
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT with Denis Magda
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT with Denis MagdaDatabricks The document discusses Apache Ignite and its relevance to the Internet of Things (IoT), outlining key components such as real-time data processing, data collection, and enrichment, as well as the Apache Mynewt RTOS. It highlights the smart metering use case with a focus on architecture, scalability, and security features. Additionally, the document provides resources for further learning and integration with Apache Spark.
Streamline Hadoop DevOps with Apache Ambari
Streamline Hadoop DevOps with Apache AmbariDataWorks Summit/Hadoop Summit Apache Ambari is an open-source tool for provisioning, managing, and monitoring Hadoop clusters. It allows users to deploy Hadoop clusters, install and manage services, configure settings, and perform rolling upgrades with minimal downtime. Ambari 2.4 includes new features like role-based access control, Grafana integration for visualization, log search capabilities, and improved upgrade workflows.
Spark,Hadoop,Presto Comparition
Spark,Hadoop,Presto ComparitionSandish Kumar H N The document discusses several big data frameworks: Spark, Presto, Cloudera Impala, and Apache Hadoop. Spark aims to make data analytics faster by loading data into memory for iterative querying. Presto extends R with distributed parallelism for scalable machine learning and graph algorithms. Hadoop uses MapReduce to distribute computations across large hardware clusters and handles failures automatically. While useful for batch processing, Hadoop has disadvantages for small files and online transactions.
August 2016 HUG: Recent development in Apache Oozie
August 2016 HUG: Recent development in Apache OozieYahoo Developer Network The document discusses the implementation and usage of Oozie at Yahoo, focusing on its capabilities for managing complex data pipelines and job dependencies across various platforms like Spark and Hive. It outlines the efficient handling of workflow jobs and coordinator jobs with insights on current limitations, enhancements, and future work in unit testing and dependency management. Additionally, it highlights Oozie's native support for Spark jobs and recent improvements in managing Spark actions within Oozie workflows.
Data Science with Spark & Zeppelin
Data Science with Spark & ZeppelinVinay Shukla This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It can be used for data exploration, visualization, collaboration and publishing. Zeppelin has deep integration with Apache Spark and supports multiple languages including Scala, Python, and SQL. It provides a Spark interpreter that allows users to analyze data using Spark without having to configure Spark themselves. The document demonstrates Zeppelin's functionality through examples and encourages readers to try it out and get involved in the community.
Apache Fink 1.0: A New Era for Real-World Streaming Analytics
Apache Fink 1.0: A New Era for Real-World Streaming AnalyticsSlim Baltagi The document discusses the evolution and capabilities of Apache Flink, a distributed dataflow streaming engine, with emphasis on its features that make it suitable for real-world streaming analytics. Key points include Flink's low latency, high throughput, stateful processing, fault tolerance, and support for various window semantics and time notions. Additionally, it outlines the roadmap for future enhancements in streaming capabilities post version 1.0.
Troubleshooting Kerberos in Hadoop: Taming the Beast
Troubleshooting Kerberos in Hadoop: Taming the BeastDataWorks Summit The document provides a comprehensive overview of troubleshooting Kerberos in Hadoop, highlighting its significance as a universal authentication mechanism and its role across the Hadoop stack. It covers topics such as the workings of Kerberos, realms, principals, tools available in Hadoop, a systematic approach to diagnosing issues, and common error messages. The author, Vipin Rathor, emphasizes the importance of proper configuration and the integration of Kerberos with enterprise user management for secure Hadoop operations.
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data Analysis
Apache Zeppelin + Livy: Bringing Multi Tenancy to Interactive Data AnalysisDataWorks Summit/Hadoop Summit
Similar to IoT Applications and Patterns using Apache Spark & Apache Bahir (20)
Building iot applications with Apache Spark and Apache Bahir
Building iot applications with Apache Spark and Apache BahirLuciano Resende The document discusses the IBM Spark Technology Center's contributions to open-source projects, particularly focusing on Apache Spark and Apache Bahir for building IoT applications. It includes technical details about the functionalities of Apache Spark, such as Spark SQL, streaming capabilities, and IoT connectivity protocols like MQTT. The document emphasizes the importance of community engagement and the potential of using these technologies for real-time IoT data processing.
Spark Concepts - Spark SQL, Graphx, Streaming
Spark Concepts - Spark SQL, Graphx, StreamingPetr Zapletal This document provides an overview of Apache Spark modules including Spark SQL, GraphX, and Spark Streaming. Spark SQL allows querying structured data using SQL, GraphX provides APIs for graph processing, and Spark Streaming enables scalable stream processing. The document discusses Resilient Distributed Datasets (RDDs), SchemaRDDs, querying data with SQLContext, GraphX property graphs and algorithms, StreamingContext, and input/output operations in Spark Streaming.
Started with-apache-spark
Started with-apache-sparkHappiest Minds Technologies The document discusses the evolution and significance of big data, highlighting Apache Spark as a pivotal open-source framework that addresses diverse use cases for data processing and analytics in a business context. It outlines features like machine learning capabilities, real-time data processing, and the integration of Spark with existing Hadoop environments, including various misconceptions about its operation. The author concludes that leveraging tools like Apache Spark can significantly enhance performance and insights for businesses managing large data sets.
Apache Spark in Industry
Apache Spark in IndustryDorian Beganovic 1. Apache Spark is an open source cluster computing framework for large-scale data processing. It is compatible with Hadoop and provides APIs for SQL, streaming, machine learning, and graph processing.
2. Over 3000 companies use Spark, including Microsoft, Uber, Pinterest, and Amazon. It can run on standalone clusters, EC2, YARN, and Mesos.
3. Spark SQL, Streaming, and MLlib allow for SQL queries, streaming analytics, and machine learning at scale using Spark's APIs which are inspired by Python/R data frames and scikit-learn.
Building highly scalable data pipelines with Apache Spark
Building highly scalable data pipelines with Apache SparkMartin Toshev This document provides a summary of Apache Spark, including:
- Spark is a framework for large-scale data processing across clusters that is faster than Hadoop by relying more on RAM and minimizing disk IO.
- Spark transformations operate on resilient distributed datasets (RDDs) to manipulate data, while actions return results to the driver program.
- Spark can receive data from various sources like files, databases, sockets through its datasource APIs and process both batch and streaming data.
- Spark streaming divides streaming data into micro-batches called DStreams and integrates with messaging systems like Kafka. Structured streaming is a newer API that works on DataFrames/Datasets.
Apache Spark PDF
Apache Spark PDFNaresh Rupareliya Spark is a cluster computing framework designed to be fast, general-purpose, and able to handle a wide range of workloads including batch processing, iterative algorithms, interactive queries, and streaming. It is faster than Hadoop for interactive queries and complex applications by running computations in-memory when possible. Spark also simplifies combining different processing types through a single engine. It offers APIs in Java, Python, Scala and SQL and integrates closely with other big data tools like Hadoop. Spark is commonly used for interactive queries on large datasets, streaming data processing, and machine learning tasks.
Apache Spark - A High Level overview
Apache Spark - A High Level overviewKaran Alang Apache Spark is an open-source unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. Some key components of Apache Spark include Resilient Distributed Datasets (RDDs), DataFrames, Datasets, and Spark SQL for structured data processing. Spark also supports streaming, machine learning via MLlib, and graph processing with GraphX.
Big data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle DatabaseMartin Toshev The document discusses Apache Spark, a powerful framework for distributed big data processing, emphasizing its capabilities, architecture, and comparison to other technologies like MapReduce and Hadoop. It details the use of Spark with Oracle RDBMS as a data source, including methods for both batch and real-time data processing. It highlights the flexibility of Spark's streaming capabilities, supported cluster managers, and practical examples of data transformation and manipulation.
In Memory Analytics with Apache Spark
In Memory Analytics with Apache SparkVenkata Naga Ravi The document provides an overview of Apache Spark, highlighting its in-memory analytics capabilities that drastically improve query response times, making it a powerful alternative to Hadoop. It explains Spark's architecture, its elements like Resilient Distributed Datasets (RDDs), and its stack extensions including Shark for SQL, Mlib for machine learning, and GraphX for graph processing. Additionally, it compares Spark with Hadoop, emphasizing Spark's speed and efficiency in handling large-scale data processing tasks.
Apache Spark Components
Apache Spark ComponentsGirish Khanzode Spark Streaming allows processing live data streams using small batch sizes to provide low latency results. It provides a simple API to implement complex stream processing algorithms across hundreds of nodes. Spark SQL allows querying structured data using SQL or the Hive query language and integrates with Spark's batch and interactive processing. MLlib provides machine learning algorithms and pipelines to easily apply ML to large datasets. GraphX extends Spark with an API for graph-parallel computation on property graphs.
Glint with Apache Spark
Glint with Apache SparkVenkata Naga Ravi The document outlines an agenda for a presentation on big data, focusing on Spark and its features, including RDDs and fault tolerance. It discusses the challenges of existing data processing frameworks and how Spark improves efficiency over Hadoop with capabilities for both batch and streaming data. Additionally, it introduces key components of the Spark stack, machine learning libraries, and the Berkeley Data Analytics Stack for managing big data.
Apache Spark Streaming
Apache Spark StreamingBartosz Jankiewicz The document discusses using Apache Spark for streaming analytics. It describes Spark as a fast, scalable, and fault-tolerant platform for real-time processing of streaming data. Some key points covered include using Spark Streaming to ingest data from various sources, process streaming data using Resilient Distributed Datasets (RDDs) and Distributed Streams (DStreams), and considerations for monitoring and optimizing Spark streaming jobs.
Not Your Father's Database by Databricks
Not Your Father's Database by DatabricksCaserta This document discusses appropriate and inappropriate use cases for Apache Spark based on the type of data and workload. It provides examples of good uses, such as batch processing, ETL, and machine learning/data science. It also gives examples of bad uses, such as random access queries, frequent incremental updates, and low latency stream processing. The document recommends using a database instead of Spark for random access, updates, and serving live queries. It suggests using message queues instead of files for low latency stream processing. The goal is to help users understand how to properly leverage Spark for big data workloads.
Real-Time Spark: From Interactive Queries to Streaming
Real-Time Spark: From Interactive Queries to StreamingDatabricks This document summarizes Michael Armbrust's presentation on real-time Spark. It discusses:
1. The goals of real-time analytics including having the freshest answers as fast as possible while keeping the answers up to date.
2. Spark 2.0 introduces unified APIs for SQL, DataFrames and Datasets to make developing real-time analytics simpler with powerful yet simple APIs.
3. Structured streaming allows running the same SQL queries on streaming data to continuously aggregate data and update outputs, unifying batch, interactive, and streaming queries into a single API.
New Analytics Toolbox DevNexus 2015
New Analytics Toolbox DevNexus 2015Robbie Strickland The document discusses the evolution and advantages of using Apache Spark over Hadoop for analytics, highlighting its speed, efficiency, and ease of integration with Apache Cassandra. It covers Spark's architecture, components, and applications, including real-time data processing through Spark Streaming and the advantages of using Spark SQL for structured data. The presentation emphasizes the shift in the analytics toolbox and provides a comprehensive overview of Spark's capabilities compared to traditional Hadoop frameworks.
Apache Spark Fundamentals
Apache Spark FundamentalsZahra Eskandari This document provides an overview of the Apache Spark framework. It covers Spark fundamentals including the Spark execution model using Resilient Distributed Datasets (RDDs), basic Spark programming, and common Spark libraries and use cases. Key topics include how Spark improves on MapReduce by operating in-memory and supporting general graphs through its directed acyclic graph execution model. The document also reviews Spark installation and provides examples of basic Spark programs in Scala.
APACHE SPARK.pptx
APACHE SPARK.pptxDeepaThirumurugan Spark is a fast, general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R for distributed tasks including SQL, streaming, and machine learning. Spark improves on MapReduce by keeping data in-memory, allowing iterative algorithms to run faster than disk-based approaches. Resilient Distributed Datasets (RDDs) are Spark's fundamental data structure, acting as a fault-tolerant collection of elements that can be operated on in parallel.
Boston Spark Meetup event Slides Update
Boston Spark Meetup event Slides Updatevithakur This document summarizes IBM's announcement of a major commitment to advance Apache Spark. It discusses IBM's investments in Spark capabilities, including log processing, graph analytics, stream processing, machine learning, and unified data access. Key reasons for interest in Spark include its performance (up to 100x faster than Hadoop for some tasks), productivity gains, ability to leverage existing Hadoop investments, and continuous community improvements. The document also provides an overview of Spark's architecture, programming model using resilient distributed datasets (RDDs), and common use cases like interactive querying, batch processing, analytics, and stream processing.
An introduction To Apache Spark
An introduction To Apache SparkAmir Sedighi Spark is a fast and general cluster computing system that improves on MapReduce by keeping data in-memory between jobs. It was developed in 2009 at UC Berkeley and open sourced in 2010. Spark core provides in-memory computing capabilities and a programming model that allows users to write programs as transformations on distributed datasets.
Ad
More from Luciano Resende (20)
A Jupyter kernel for Scala and Apache Spark.pdf
A Jupyter kernel for Scala and Apache Spark.pdfLuciano Resende Apache Toree is a Scala-based Jupyter kernel designed to execute Scala code within Jupyter notebooks and to connect with Apache Spark, enabling interactive applications. It supports over 50 programming languages and integrates with big data platforms like Apache Spark, providing a rich environment for data visualization and manipulation. The document outlines installation instructions, historical context, key features, and community contributions related to Apache Toree.
Using Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsLuciano Resende The document discusses Elyra, an open-source set of AI-focused extensions for JupyterLab designed to streamline the development and deployment of machine learning models, particularly in scenarios like COVID-19 analytics. Elyra enhances productivity through features like hybrid runtime support, visual pipeline editors, and Git integration, simplifying collaboration and resource management. It also highlights IBM's commitment to open-source contributions and innovation within the AI and data science community.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Luciano Resende The document introduces Elyra, a set of AI-centric extensions to JupyterLab designed to simplify the AI model development lifecycle for data scientists and machine learning engineers. It highlights JupyterLab's capabilities, including interactive notebooks and hybrid runtime support, as well as its integration with tools like Kubeflow for managing machine learning workflows. Additionally, it discusses IBM's significant participation in open source software and its contributions to AI technologies over the years.
From Data to AI - Silicon Valley Open Source projects come to you - Madrid me...
From Data to AI - Silicon Valley Open Source projects come to you - Madrid me...Luciano Resende The document outlines IBM's contributions to open source AI technologies, highlighting key initiatives like the Model Asset Exchange and Data Asset Exchange, which facilitate data sharing and model deployment. It details IBM's historical achievements in AI, from Deep Blue to contemporary AI frameworks, and emphasizes the importance of fairness, transparency, and accountability in AI systems. Additionally, it addresses advancements in tools and platforms, including Jupyter Notebooks for data science and the use of Kubernetes for machine learning workflows.
Ai pipelines powered by jupyter notebooks
Ai pipelines powered by jupyter notebooksLuciano Resende The document outlines the role of Jupyter Notebooks in AI pipelines, emphasizing their interactive capabilities for coding, data visualization, and multi-language support. It discusses IBM's contributions to open source initiatives, the management of AI workloads using platforms like Kubeflow and Apache Airflow, and tools like Papermill for notebook parameterization. Additionally, it highlights the importance of frameworks such as Fabric for Deep Learning in enabling scalable and accessible deep learning solutions.
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayLuciano Resende The document presents an overview of the Jupyter Enterprise Gateway, emphasizing its capabilities to enhance the performance and scalability of Jupyter notebooks in enterprise environments. It details the architecture, features, and benefits, including support for multiple users, improved resource allocation, and integration with Kubernetes for managing AI workloads. The document also includes information about the contributions of IBM to open-source technologies and their focus on collaborative innovation in data and AI solutions.
Scaling notebooks for Deep Learning workloads
Scaling notebooks for Deep Learning workloadsLuciano Resende The document discusses the use of Jupyter notebooks for deep learning workloads, emphasizing the importance of Kubernetes for resource management and scalability. It highlights the transition to using Jupyter enterprise gateway for kernel isolation and efficient resource utilization, enabling smoother deep learning application lifecycles. The presentation outlines the benefits of using lightweight containerized environments and the integration of various tools in the deep learning infrastructure to support extensive data science applications.
Jupyter Enterprise Gateway Overview
Jupyter Enterprise Gateway OverviewLuciano Resende This document discusses Jupyter notebooks and the Jupyter Enterprise Gateway. It provides an overview of Jupyter notebooks, their functionality, and architecture. It then describes the limitations of running Jupyter notebooks at an enterprise scale. The Jupyter Enterprise Gateway is introduced as a solution that enables sharing of resources across clusters in a scalable, secure, and multi-tenant way. The document outlines the key features and architecture of the Gateway when used with YARN and Kubernetes clusters. Finally, it provides information on deploying and connecting to the Jupyter Enterprise Gateway.
Inteligencia artificial, open source e IBM Call for Code
Inteligencia artificial, open source e IBM Call for CodeLuciano Resende The document outlines IBM's initiatives in open source and artificial intelligence, highlighting the contributions of Luciano Resende and the various projects like the Model Asset Exchange and the Fabric for Deep Learning (FFDL). It emphasizes the importance of open source in IBM's AI development, detailing how their platform supports deep learning across multiple frameworks and enhances enterprise applications through tools like the Jupyter Enterprise Gateway. The document also describes IBM's historical achievements in AI and the future direction of their open-source efforts.
Open Source AI - News and examples
Open Source AI - News and examplesLuciano Resende Luciano Resende, a data science platform architect at IBM, discusses the company's commitment to open source and AI over the years, including the establishment of various initiatives and projects like the Model Asset Exchange (MAX) and the Fabric for Deep Learning (FFDL). IBM's contributions to the open source community include substantial code contributions and the promotion of deep learning and AI technologies, making them accessible to developers. The presentation highlights the importance of collaboration and innovation in advancing AI solutions in the enterprise environment.
Building analytical microservices powered by jupyter kernels
Building analytical microservices powered by jupyter kernelsLuciano Resende This document discusses building analytical microservices powered by Jupyter kernels. It provides an overview of Jupyter notebooks and their architecture. It then introduces the Jupyter Enterprise Gateway, which allows running Jupyter kernels on a distributed cluster for improved scalability and security. Finally, it demonstrates a use case of a sentiment analysis microservice that leverages PySpark on a Hadoop cluster via Jupyter kernels.
An Enterprise Analytics Platform with Jupyter Notebooks and Apache Spark
An Enterprise Analytics Platform with Jupyter Notebooks and Apache SparkLuciano Resende The IBM Spark Technology Center's document outlines the features and functionalities of their data science platform, including the integration of Jupyter notebooks and Apache Spark for enhanced analytics. It highlights the capabilities of the IBM Analytics Engine, the use of Jupyter Enterprise Gateway for multi-user support and resource allocation, and provides deployment instructions for setting up these tools. Additionally, it emphasizes the center's contribution to open-source projects and its mission to assist in making Apache Spark technology ready for enterprise and cloud environments.
The Analytic Platform behind IBM’s Watson Data Platform - Big Data Spain 2017
The Analytic Platform behind IBM’s Watson Data Platform - Big Data Spain 2017Luciano Resende The document discusses the IBM Spark Technology Center's contributions to the Apache Spark community, focusing on the IBM Data Science Experience and the IBM Analytics Engine, which facilitate data science productivity and efficient data model building. It highlights the use of Jupyter notebooks and the development of the Jupyter Enterprise Gateway to enable multi-user support and enhance resource allocation, security, and administration across Apache Spark clusters. Additionally, it outlines the center's mission to make core technologies enterprise and cloud-ready while building data science skills.
What's new in Apache SystemML - Declarative Machine Learning
What's new in Apache SystemML - Declarative Machine LearningLuciano Resende The document discusses the IBM Spark Technology Center and its contributions to Apache Spark and SystemML, focusing on machine learning advancements and open source leadership. It outlines the history, achievements, and technical capabilities of the Spark Technology Center, highlighting projects such as SystemML and its effectiveness in optimizing machine learning algorithms. Additionally, it emphasizes the integration of various programming languages and APIs for broader applicability in data science tasks.
Big analytics meetup - Extended Jupyter Kernel Gateway
Big analytics meetup - Extended Jupyter Kernel GatewayLuciano Resende Luciano Resende from IBM's Spark Technology Center presented on building an enterprise/cloud analytics platform with Jupyter Notebooks and Apache Spark. The Spark Technology Center focuses on contributions to open source Apache Spark projects. Resende discussed limitations of the current Jupyter Notebook setup for multi-user shared clusters and demonstrated an Extended Jupyter Kernel Gateway that allows running kernels remotely in a cluster with enhanced security, resource optimization, and multi-user support through user impersonation. The Extended Jupyter Kernel Gateway is planned for open source release.
Jupyter con meetup extended jupyter kernel gateway
Jupyter con meetup extended jupyter kernel gatewayLuciano Resende The IBM Spark Technology Center focuses on developing an enterprise/cloud analytics platform utilizing Jupyter notebooks and Apache Spark, aiming to enhance data science skills and improve resource allocation. The center is characterized by a shared computing environment, multiuser support, and ongoing efforts to optimize security and manage resources more effectively. The extended Jupyter kernel gateway is being developed to provide better utilization of resources and support for additional resource managers, with plans for open-sourcing the technology soon.
How mentoring can help you start contributing to open source
How mentoring can help you start contributing to open sourceLuciano Resende This document discusses various mentoring programs that can help people start contributing to open source projects, including Google Summer of Code, Google Code-In, GNOME Outreach Program for Women, and the Apache Mentoring Programme. It provides details on the goals, eligibility requirements, and processes for these programs. It also discusses how to find mentors and project ideas within Apache projects and more generally.
SystemML - Declarative Machine Learning
SystemML - Declarative Machine LearningLuciano Resende The document discusses Apache SystemML, a declarative machine learning language that can run scalable machine learning algorithms on Apache Spark. It describes how SystemML allows data scientists to write algorithms in a simple R-like syntax and have them automatically compiled and run efficiently on big data in Spark. The speaker then demonstrates an alternating least squares algorithm written in R and how SystemML can compile and run the same algorithm at scale on Spark with no additional code.
Luciano Resende's keynote at Apache big data conference
Luciano Resende's keynote at Apache big data conferenceLuciano Resende This document discusses IBM's use of Apache Spark technology. It provides an overview of IBM's Spark Technology Center and examples of how customers are leveraging Apache Spark and open source analytics. Specifically, it describes how The Weather Company uses Spark for efficient batch and streaming analysis of large volumes of weather data, and how a healthcare enterprise uses Spark on IBM's SystemML for predictive risk modeling from clinical data lakes. It also outlines how one customer uses Spark to analyze customer experience data across multiple channels to improve satisfaction rates.
Asf icfoss-mentoring
Asf icfoss-mentoringLuciano Resende The document provides an overview of a 3-day open source workshop being conducted by Luciano Resende from the Apache Software Foundation. Day 1 will cover topics on open source, licenses, communities and how to get involved in Apache projects. Day 2 focuses on hands-on development, setting up environments and tools. Day 3 is about mentoring expectations and working on project proposals. The workshop aims to educate participants and help them get involved in open source.
Ad
Recently uploaded (20)
SQL-Demystified-A-Beginners-Guide-to-Database-Mastery.pptx
SQL-Demystified-A-Beginners-Guide-to-Database-Mastery.pptxbhavaniteacher99 Unlock the power of data with “SQL Demystified” — your perfect beginner-friendly guide to mastering Structured Query Language (SQL)! Whether you're a student, aspiring analyst, or a professional stepping into the world of data, this detailed and practical PowerPoint presentation simplifies the fundamentals and builds your SQL skills step-by-step.
📘 What’s Inside?
Introduction to SQL and its real-world importance
Understanding database structures: tables, columns & relationships
Core SQL queries: SELECT, FROM, WHERE, ORDER BY
Hands-on examples and practice exercises
Advanced SQL: JOIN, subqueries, and aggregate functions
Data manipulation: INSERT, UPDATE, DELETE
Performance tips and indexing basics
SQL security best practices and learning paths for future growth
🧠 Who is it for?
Beginners eager to learn SQL
Data science and computer students
Business professionals and analysts
Anyone curious about how databases work
✅ Features:
Clear visuals and examples
Easy-to-follow explanations
Real-life applications in banking, healthcare, e-commerce, and more
Reusable and editable presentation file (PPTX format)
Take your first step toward data mastery with confidence. Learn SQL the smart way — with structure, clarity, and purpose.
Power BI API Connectors - Best Practices for Scalable Data Connections
Power BI API Connectors - Best Practices for Scalable Data Connections Vidicorp Ltd This document explains how API connections work in Power BI, comparing different methods like custom connectors, push datasets, database pipelines, and third-party integration tools. It also gives a clear plan on how to use APIs for flexible, scalable, and smart business reporting.
2.5-DESPATCH-ORDINARY MAILS.pptxlminub7b7t6f7h7t6f6g7g6fg
2.5-DESPATCH-ORDINARY MAILS.pptxlminub7b7t6f7h7t6f6g7g6fgmk1227103 8hunubybub6v6vubug7gyvtvtvyb7g7h7hih8h8hij8jihih
FME Beyond Data Processing: Creating a Dartboard Accuracy App
FME Beyond Data Processing: Creating a Dartboard Accuracy Appjacoba18 Bij Nordend vroegen we ons af of FME ons kan helpen bepalen waar we moeten mikken voor de hoogste score, gebaseerd op onze nauwkeurigheid. We tonen hoe we met FME Flow, een zelfgemaakte WMS-dartbordlaag en geanalyseerde gooiresultaten optimale mikpunten berekenden.
apidays New York 2025 - The FINOS Common Domain Model for Capital Markets by ...
apidays New York 2025 - The FINOS Common Domain Model for Capital Markets by ...apidays The FINOS Common Domain Model for Capital Markets
Tom Healey, Founder & Director at FINXIS LLC
Daniel Schwartz, Managing Partner at FT Advisory LLC
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Media_Literacy_Index_of_Media_Sector_Employees.pdf
Media_Literacy_Index_of_Media_Sector_Employees.pdfOlhaTatokhina1 Report based on the findings of a quantitative research conducted by the research agency New Image Marketing Group, commissioned by the NGO Detector Media, compiled by PhD in Sociology Marta Naumova.
Managed Cloud services - Opsio Cloud Man
Managed Cloud services - Opsio Cloud ManOpsio Cloud At Opsio, we specialize in delivering advanced cloud services that enable businesses to scale, transform, and modernize with confidence. Our core offerings focus on cloud management, digital transformation, and cloud modernization — all designed to help organizations unlock the full potential of their technology infrastructure.We take a client-first approach, blending industry-leading hosted technologies with strategic expertise to create tailored, future-ready solutions. Leveraging AI, automation, and emerging technologies, our services simplify IT operations, enhance agility, and accelerate business outcomes. Whether you're migrating to the cloud or optimizing existing cloud environments, Opsio is your partner in achieving sustainable, measurable success.
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELSAmeya Patekar QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
MEDIA_LITERACY_INDEX_OF_EDUCATORS_ENG.pdf
MEDIA_LITERACY_INDEX_OF_EDUCATORS_ENG.pdfOlhaTatokhina1 Report based on the findings of a quantitative research conducted by the research agency New Image Marketing Group, commissioned by the NGO Detector Media, compiled by PhD in Sociology Marta Naumova
MICROSOFT POWERPOINT AND USES(BEST)..pdf
MICROSOFT POWERPOINT AND USES(BEST)..pdfbathyates Talks about presentation packages and their uses to man and how it functions being the best presentation package. Learn about presentation packages here with me
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...apidays Beyond Webhooks: The Future of Scalable API Event Delivery
Phil Leggetter, Head of Developer Experience at Hookdeck
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays Singapore 2025 - Enhancing Developer Productivity with UX (Government...
apidays Singapore 2025 - Enhancing Developer Productivity with UX (Government...apidays Enhancing Developer Productivity with UX
Petrine Tang, UX Designer at Government Technology Agency
Faith Ang, Product Manager at Government Technology Agency
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
unit- 5 Biostatistics and Research Methodology.pdf
unit- 5 Biostatistics and Research Methodology.pdfKRUTIKA CHANNE Design and Analysis of Experiments:
Factorial Design: Definition, 2, 2, 2' design. Advantage of factorial design
Response Surface methodology: Central composite design, Historical design, Optimization Techniques
apidays New York 2025 - The Future of Small Business Lending with Open Bankin...
apidays New York 2025 - The Future of Small Business Lending with Open Bankin...apidays The Future of Small Business Lending with Open Banking – Bridging the $750 Billion Funding Gap
Charles Groome, Vice President of Growth Strategy at Biz2Credit
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Unifying OpenAPI & AsyncAPI by Naresh Jain & Hari Kri...
apidays New York 2025 - Unifying OpenAPI & AsyncAPI by Naresh Jain & Hari Kri...apidays Unifying OpenAPI & AsyncAPI: Designing JSON Schemas+Examples for Reuse
Naresh Jain, Co-founder & CEO at Specmatic
Hari Krishnan, Co-founder & CTO at Specmatic
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - API Security and Observability at Scale in Kubernetes...
apidays New York 2025 - API Security and Observability at Scale in Kubernetes...apidays API Security and Observability at Scale in Kubernetes
Ben Urbanski, Product Manager for Layer7 at Broadcom
Geoffrey Duck, Solution Engineer and Service Lead for Layer7 at Broadcom
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays Singapore 2025 - What exactly are AI Agents by Aki Ranin (Earthshots ...
apidays Singapore 2025 - What exactly are AI Agents by Aki Ranin (Earthshots ...apidays What exactly are AI Agents?
Aki Ranin, Head of AI at Earthshots Collective | Deep Tech & AI Investor | 2x Founder | Published Author
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
IoT Applications and Patterns using Apache Spark & Apache Bahir
- 1. IoT Applications and Patterns using Apache Spark & Apache Bahir Luciano Resende June 14th, 2018 © 2018 IBM Corporation 1
- 2. About me - Luciano Resende 2 Data Science Platform Architect – IBM – CODAIT • Have been contributing to open source at ASF for over 10 years • Currently contributing to : Jupyter Notebook ecosystem, Apache Bahir, Apache Toree, Apache Spark among other projects related to AI/ML platforms [email protected] https://www.linkedin.com/in/lresende @lresende1975 https://github.com/lresende
- 3. Open Source Community Leadership C O D A I T Founding Partner 188+ Project Committers 77+ Projects Key Open source steering committee memberships OSS Advisory Board Open Source
- 4. Center for Open Source Data and AI Technologies CODAIT codait.org codait (French) = coder/coded https://m.interglot.com/fr/en/codait CODAIT aims to make AI solutions dramatically easier to create, deploy, and manage in the enterprise Relaunch of the Spark Technology Center (STC) to reflect expanded mission 5
- 5. Agenda 6 Introductions - Apache Spark - Apache Bahir IoT Applications Live Demo Summary References Q&A
- 7. Apache Spark Introduction 8 Spark Core Spark SQL Spark Streaming Spark ML Spark GraphX executes SQL statements performs streaming analytics using micro-batches common machine learning and statistical algorithms distributed graph processing framework general compute engine, handles distributed task dispatching, scheduling and basic I/O functions large variety of data sources and formats can be supported, both on-premise or cloud BigInsights (HDFS) Cloudant dashDB SQL DB
- 9. Apache Spark – Spark SQL 10 Spark SQL Unified data access APIS: Query structured data sets with SQL or Dataset/DataFrame APIs Fast, familiar query language across all of your enterprise data RDBMS Data Sources Structured Streaming Data Sources
- 10. Apache Spark – Spark SQL 11 You can run SQL statement with SparkSession.sql(…) interface: val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() spark.sql(“create table T1 (c1 int, c2 int) stored as parquet”) val ds = spark.sql(“select * from T1”) You can further transform the resultant dataset: val ds1 = ds.groupBy(“c1”).agg(“c2”-> “sum”) val ds2 = ds.orderBy(“c1”) The result is a DataFrame / Dataset[Row] ds.show() displays the rows
- 11. Apache Spark – Spark SQL You can read from data sources using SparkSession.read.format(…) val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer) // loading csv data to a Dataset of Bank type val bankFromCSV = spark.read.csv(“hdfs://localhost:9000/data/bank.csv").as[Bank] // loading JSON data to a Dataset of Bank type val bankFromJSON = spark.read.json(“hdfs://localhost:9000/data/bank.json").as[Bank] // select a column value from the Dataset bankFromCSV.select(‘age).show() will return all rows of column “age” from this dataset. 12
- 12. Apache Spark – Spark SQL You can also configure a specific data source with specific options val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer) // loading csv data to a Dataset of Bank type val bankFromCSV = sparkSession.read .option("header", ”true") // Use first line of all files as header .option("inferSchema", ”true") // Automatically infer data types .option("delimiter", " ") .csv("/users/lresende/data.csv”) .as[Bank] bankFromCSV.select(‘age).show() // will return all rows of column “age” from this dataset. 13
- 13. Apache Spark – Spark SQL – Data Sources Data Sources under the covers - Data source registration (e.g. spark.read.datasource) - Provide BaseRelation implementation • That implements support for table scans: – TableScans, PrunedScan, PrunedFilteredScan, CatalystScan - Detailed information available at • https://developer.ibm.com/code/2016/11/10/exploring-apache-spark-datasource-api/ 14
- 14. Apache Spark – Spark SQL – Data Sources Data Sources V1 Limitations - Leak upper-level API in the data source (DataFrame/SQLContext) - Hard to extend the Data Sources API for more optimizations - Zero transaction guarantee in the write APIs - Limited Extensibility 15
- 15. Apache Spark – Spark SQL – Data Sources Data Sources V2 - Support for row-based scan and columnar scan - Column pruning and filter push-down - Can report basic statistics and data partitioning - Transactional write API - Streaming source and sink support for micro-batch and continuous mode - Detailed information available at • https://developer.ibm.com/code/2018/04/16/introducing-apache-spark-data-sources-api-v2/ 16
- 16. Apache Spark – Spark SQL Structured Streaming Unified programming model for streaming, interactive and batch queries 17Image source: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html Considers the data stream as unbounded table
- 17. Apache Spark – Spark SQL Structured Streaming SQL regular APIs val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() val input = spark.read .schema(schema) .format(”csv") .load(”input-path") val result = input .select(”age”) .where(”age > 18”) result.write .format(”json”) . save(” dest-path”) 18 Structured Streaming APIs val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() val input = spark.readStream .schema(schema) .format(”csv") .load(”input-path") val result = input .select(”age”) .where(”age > 18”) result.write .format(”json”) . startStream(” dest-path”)
- 18. Apache Spark – Spark Streaming 19 Spark Streaming Micro-batch event processing for near-real time analytics e.g. Internet of Things (IoT) devices, Twitter feeds, Kafka (event hub), etc. No multi-threading or parallel process programming required
- 19. Apache Spark – Spark Streaming Also known as discretized stream or DStream Abstracts a continuous stream of data Based on micro-batching Based on RDDs 20
- 20. Apache Spark – Spark Streaming val sparkConf = new SparkConf() .setAppName("MQTTWordCount") val ssc = new StreamingContext(sparkConf, Seconds(2)) val lines = MQTTUtils.createStream(ssc, brokerUrl, topic, StorageLevel.MEMORY_ONLY_SER_2) val words = lines.flatMap(x => x.split(" ")) val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination() 21
- 21. Apache Bahir 22
- 22. Origins of the Apache Bahir Project MAY/2016: Established as a top-level Apache Project. - PMC formed by Apache Spark committers/pmc, Apache Members - Initial contributions imported from Apache Spark AUG/2016: Apache Flink community join Apache Bahir - Initial contributions of Flink extensions - In October 2016 Robert Metzger elected committer
- 23. Origins of the Bahir name Naming an Apache Project is a science !!! - We needed a name that wasn’t used yet - Needed to be related to Spark We ended up with : Bahir - A name of Arabian origin that means Sparkling, - Also associated with a guy who succeeds at everything
- 24. Why Apache Bahir It’s an Apache project - And if you are here, you know what it means Benefits of curating your extensions at Apache Bahir - Apache Governance - Apache License - Apache Community - Apache Brand 25
- 25. Why Apache Bahir Flexibility - Release flexibility • Bounded to platform or component release Shared infrastructure - Release, CI, etc Shared knowledge - Collaborate with experts on both platform and component areas 26
- 26. Bahir extensions for Apache Spark MQTT – Enables reading data from MQTT Servers using Spark Streaming or Structured streaming. • http://bahir.apache.org/docs/spark/current/spark-sql-streaming-mqtt/ • http://bahir.apache.org/docs/spark/current/spark-streaming-mqtt/ Couch DB/Cloudant – Enables reading data from CouchDB/Cloudant using Spark SQL and Spark Streaming. Twitter – Enables reading social data from twitter using Spark Streaming. • http://bahir.apache.org/docs/spark/current/spark-streaming-twitter/ Akka – Enables reading data from Akka Actors using Spark Streaming or Structured Streaming. • http://bahir.apache.org/docs/spark/current/spark-streaming-akka/ ZeroMQ – Enables reading data from ZeroMQ using Spark Streaming. • http://bahir.apache.org/docs/spark/current/spark-streaming-zeromq/ 27
- 27. Bahir extensions for Apache Spark Google Cloud Pub/Sub – Add spark streaming connector to Google Cloud Pub/Sub 28
- 28. Apache Spark extensions in Bahir Adding Bahir extensions into your application - Using SBT libraryDependencies += "org.apache.bahir" %% "spark-streaming-mqtt" % "2.2.0” - Using Maven <dependency> <groupId>org.apache.bahir</groupId> <artifactId>spark-streaming-mqtt_2.11 </artifactId> <version>2.2.0</version> </dependency> 29
- 29. Apache Spark extensions in Bahir Submitting applications with Bahir extensions to Spark - Spark-shell bin/spark-shell --packages org.apache.bahir:spark-streaming_mqtt_2.11:2.2.0 ….. - Spark-submit bin/spark-submit --packages org.apache.bahir:spark-streaming_mqtt_2.11:2.2.0 ….. 30
- 30. Internet of Things - IoT 31
- 31. IoT – Definition by Wikipedia The Internet of things (IoT) is the network of physical devices, vehicles, home appliances, and other items embedded with electronics, software, sensors, actuators, and network connectivity which enable these objects to connect and exchange data. 32
- 32. IoT – Interaction between multiple entities 33 Things Software People control observe inform command actuate inform
- 33. IoT Patterns – Some of them … 35 • Remote control • Security analysis • Edge analytics • Historical data analysis • Distributed Platforms • Real-time decisions
- 34. MQTT – M2M / IoT Connectivity Protocol 37 Connect + Publish + Subscribe ~1990 IBM / Eurotech 2010 Published 2011 Eclipse M2M / Paho 2014 OASIS Open spec + 40 client implementatio ns Minimal overhead Tiny Clients (Java 170KB) History Header 2-4 bytes (publish) 14 bytes (connect) V5 May 2018
- 35. MQTT – Quality of Service 38 MQTT Broker QoS0 QoS1 QoS2 At most once At least once Exactly once . No connection failover . Never duplicate . Has connection failover . Can duplicate . Has connection failover . Never duplicate
- 36. MQTT – World usage Smart Home Automation Messaging Notable Mentions: - IBM IoT Platform - AWS IoT - Microsoft IoT Hub - Facebook Messanger 39
- 37. Live Demo 40
- 38. IoT Simulator using MQTT The demo environment https://github.com/lresende/bahir-iot-demo 41 Node.js Web app Simulates Elevator IoT devices Elevator simulator Metrics: • Weight • Speed • Power • Temperature • System MQTT Mosquitto
- 39. Summary 42
- 40. Summary – Take away points Apache Spark - IoT Analytics Runtime with support for ”Continuous Applications” Apache Bahir - Bring access to IoT data via supported connectors (e.g. MQTT) IoT Applications - Using Spark and Bahir to start processing IoT data in near real time using Spark Streaming and Spark Structured Streaming 43
- 41. Join the Apache Bahir community 44
- 42. References Apache Bahir http://bahir.apache.org Documentation for Apache Spark extensions http://bahir.apache.org/docs/spark/current/documentation/ Source Repositories https://github.com/apache/bahir https://github.com/apache/bahir-website Demo Repository https://github.com/lresende/bahir-iot-demo 45Image source: http://az616578.vo.msecnd.net/files/2016/03/21/6359412499310138501557867529_thank-you-1400x800-c-default.gif
- 43. 46March 30 2018 / © 2018 IBM Corporation