Writing Apache Spark and Apache Flink Applications Using Apache Bahir
2 likes42,761 views
The document discusses the Apache Bahir project, established as a top-level Apache project for building applications using extensions to Apache Spark. It includes details about the project origins, benefits of using Apache Bahir, its relationship with Apache Spark, and various extension capabilities such as MQTT and Twitter data streaming. Additionally, it covers integration steps for using Bahir extensions in applications and showcases a demonstration environment for IoT simulations.
1 of 30
Downloaded 11 times
![IBM SparkTechnology Center
Apache Spark – Spark SQL
You can run SQL statement with SparkSession.sql(…) interface:
val spark = SparkSession.builder()
.appName(“Demo”)
.getOrCreate()
spark.sql(“create table T1 (c1 int, c2 int) stored as parquet”)
val ds = spark.sql(“select * from T1”)
You can further transform the resultant dataset:
val ds1 = ds.groupBy(“c1”).agg(“c2”-> “sum”)
val ds2 = ds.orderBy(“c1”)
The result is a DataFrame / Dataset[Row]
ds.show() displays the rows
10](https://image.slidesharecdn.com/apachebigdata-apachebahir-161116210757/85/Writing-Apache-Spark-and-Apache-Flink-Applications-Using-Apache-Bahir-10-320.jpg)
![IBM SparkTechnology Center
Apache Spark – Spark SQL
You can read from data sources using SparkSession.read.format(…)
val spark = SparkSession.builder()
.appName(“Demo”)
.getOrCreate()
case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)
// loading csv data to a Dataset of Bank type
val bankFromCSV = spark.read.csv(“hdfs://localhost:9000/data/bank.csv").as[Bank]
// loading JSON data to a Dataset of Bank type
val bankFromJSON = spark.read.json(“hdfs://localhost:9000/data/bank.json").as[Bank]
// select a column value from the Dataset
bankFromCSV.select(‘age).show() will return all rows of column “age” from this dataset.
11](https://image.slidesharecdn.com/apachebigdata-apachebahir-161116210757/85/Writing-Apache-Spark-and-Apache-Flink-Applications-Using-Apache-Bahir-11-320.jpg)
![IBM SparkTechnology Center
Apache Spark – Spark SQL
You can also configure a specific data source with specific options
val spark = SparkSession.builder()
.appName(“Demo”)
.getOrCreate()
case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer)
// loading csv data to a Dataset of Bank type
val bankFromCSV = sparkSession.read
.option("header", ”true") // Use first line of all files as header
.option("inferSchema", ”true") // Automatically infer data types
.option("delimiter", " ")
.csv("/users/lresende/data.csv”)
.as[Bank]
bankFromCSV.select(‘age).show() // will return all rows of column “age” from this dataset.
12](https://image.slidesharecdn.com/apachebigdata-apachebahir-161116210757/85/Writing-Apache-Spark-and-Apache-Flink-Applications-Using-Apache-Bahir-12-320.jpg)
Ad
Recommended
Luciano Resende's keynote at Apache big data conference
Luciano Resende's keynote at Apache big data conferenceLuciano Resende This document discusses IBM's use of Apache Spark technology. It provides an overview of IBM's Spark Technology Center and examples of how customers are leveraging Apache Spark and open source analytics. Specifically, it describes how The Weather Company uses Spark for efficient batch and streaming analysis of large volumes of weather data, and how a healthcare enterprise uses Spark on IBM's SystemML for predictive risk modeling from clinical data lakes. It also outlines how one customer uses Spark to analyze customer experience data across multiple channels to improve satisfaction rates.
SystemML - Declarative Machine Learning
SystemML - Declarative Machine LearningLuciano Resende The document discusses Apache SystemML, a declarative machine learning language that can run scalable machine learning algorithms on Apache Spark. It describes how SystemML allows data scientists to write algorithms in a simple R-like syntax and have them automatically compiled and run efficiently on big data in Spark. The speaker then demonstrates an alternating least squares algorithm written in R and how SystemML can compile and run the same algorithm at scale on Spark with no additional code.
How mentoring can help you start contributing to open source
How mentoring can help you start contributing to open sourceLuciano Resende This document discusses various mentoring programs that can help people start contributing to open source projects, including Google Summer of Code, Google Code-In, GNOME Outreach Program for Women, and the Apache Mentoring Programme. It provides details on the goals, eligibility requirements, and processes for these programs. It also discusses how to find mentors and project ideas within Apache projects and more generally.
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT with Denis Magda
Apache Spark and Apache Ignite: Where Fast Data Meets the IoT with Denis MagdaDatabricks The document discusses Apache Ignite and its relevance to the Internet of Things (IoT), outlining key components such as real-time data processing, data collection, and enrichment, as well as the Apache Mynewt RTOS. It highlights the smart metering use case with a focus on architecture, scalability, and security features. Additionally, the document provides resources for further learning and integration with Apache Spark.
Data Science with Spark & Zeppelin
Data Science with Spark & ZeppelinVinay Shukla This document discusses Apache Zeppelin, an open-source web-based notebook that allows for interactive data analytics. It can be used for data exploration, visualization, collaboration and publishing. Zeppelin has deep integration with Apache Spark and supports multiple languages including Scala, Python, and SQL. It provides a Spark interpreter that allows users to analyze data using Spark without having to configure Spark themselves. The document demonstrates Zeppelin's functionality through examples and encourages readers to try it out and get involved in the community.
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...
Introduction to Big Data Analytics using Apache Spark and Zeppelin on HDInsig...Alex Zeltov The document provides an introduction to Big Data analytics using Apache Spark on HDInsight and the Hortonworks Data Platform, outlining key components, programming languages, and functionalities such as data processing, ETL, and machine learning. It emphasizes the advantages of Spark, particularly its speed due to in-memory data processing, and discusses the resilience and parallelism of Resilient Distributed Datasets (RDDs). The document also highlights integration with various data sources, including Hadoop and Hive, as well as the use of DataFrames for efficient data manipulation.
Apache Spark Overview
Apache Spark OverviewairisData Tim Spann will present on learning Apache Spark. He is a senior solutions architect who previously worked as a senior field engineer and startup engineer. airis.DATA, where Spann works, specializes in machine learning and graph solutions using Spark, H20, Mahout, and Flink on petabyte datasets. The agenda includes an overview of Spark, an explanation of MapReduce, and hands-on exercises to install Spark, run a MapReduce job locally, and build a project with IntelliJ and SBT.
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...
SparkOscope: Enabling Apache Spark Optimization through Cross Stack Monitorin...Databricks The document discusses Sparkoscope, an enhancement to Apache Spark's monitoring framework that adds OS-level metrics and improves the Spark web UI for better visualization of these metrics. It includes various modules for data export and real-time monitoring, addressing limitations in current monitoring architectures. The presentation also outlines installation instructions and future roadmap plans for expanding the project.
Transitioning Compute Models: Hadoop MapReduce to Spark
Transitioning Compute Models: Hadoop MapReduce to SparkSlim Baltagi The document discusses the evolution and transition of compute models in big data, specifically from Hadoop MapReduce to Apache Spark, highlighting the differences in performance, ease of use, and compatibility with various tools. It outlines existing projects that are migrating or adapting to Spark and details integration capabilities with different data storage systems. Additionally, it emphasizes key alternatives like Apache Tez and Flink, and the continued evolution of big data processing frameworks.
An Insider’s Guide to Maximizing Spark SQL Performance
An Insider’s Guide to Maximizing Spark SQL PerformanceTakuya UESHIN This document provides an overview of optimizing Spark SQL performance. It begins with introducing the speaker and their background with Spark. It then discusses reading query plans, interpreting them to understand optimizations, and tuning plans by pushing down filters, avoiding implicit casts, and other techniques. It emphasizes tracking query execution through the Spark UI to analyze jobs, stages and tasks for bottlenecks. The document aims to help understand how to maximize Spark SQL performance.
Data science lifecycle with Apache Zeppelin
Data science lifecycle with Apache ZeppelinDataWorks Summit/Hadoop Summit This document discusses Apache Zeppelin, an open-source web-based notebook that enables interactive data analytics. It provides an overview of Zeppelin's history and architecture, including how interpreters and notebook storage are pluggable. The document also outlines Zeppelin's roadmap for improving enterprise support through features like multi-tenancy, impersonation, job management and frontend performance.
Introduction to Apache Spark
Introduction to Apache SparkRahul Jain The document details a September 2014 Meetup organized by the Big Data Hyderabad group focusing on Apache Spark, covering its advantages for in-memory data processing, especially in machine learning. It includes an introduction to Spark's architecture, deployment, and basic programming operations in various languages such as Scala, Python, and Java. The presentation also provides examples and practical information about using Spark, including code snippets and how to submit jobs.
Spark Summit EU talk by Yiannis Gkoufas
Spark Summit EU talk by Yiannis GkoufasSpark Summit This document introduces SparkOscope, a tool that enables optimization of Apache Spark through cross-stack monitoring and visualization. SparkOscope extends Spark's monitoring framework by collecting operating system-level metrics using Hyperic Sigar library. It enhances Spark's web UI to plot all available metrics, including the new OS metrics. SparkOscope provides an easy way to monitor Spark jobs and identify bottlenecks by associating metrics with source code locations. The author demonstrates SparkOscope and discusses plans to improve its storage mechanism and enable infrastructure recommendations based on job resource usage patterns.
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...Spark Summit The document discusses crucial insights into writing streaming applications, particularly focusing on monitoring, management, and preventing data loss. Key topics include the importance of orchestration, driver management, checkpointing, and using Spark Streaming for efficient data processing. It emphasizes strategies for graceful shutdowns and achieving exactly-once semantics in streaming applications.
Open Source Ingredients for Interactive Data Analysis in Spark by Maxim Lukiy...
Open Source Ingredients for Interactive Data Analysis in Spark by Maxim Lukiy...DataWorks Summit/Hadoop Summit The document discusses utilizing Yarn command line with Livy server and Thrift server for Jupyter notebooks. It highlights default and Thrift queues, as well as integration with IntelliJ and BI tools. Additionally, it provides links to relevant resources and documentation for further exploration.
Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan Volz
Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan VolzDatabricks This document discusses using Hadoop for archiving, e-discovery, and supervision. It outlines the key components of each task and highlights traditional shortcomings. Hadoop provides strengths like speed, ease of use, and security. An architectural overview shows how Hadoop can be used for ingestion, processing, analysis, and machine learning. Examples demonstrate surveillance use cases. While some obstacles remain, partners can help address areas like user interfaces and compliance storage.
Apache spark
Apache sparkTEJPAL GAUTAM This document provides an introduction and overview of Apache Spark, including:
- Spark is a lightning-fast cluster computing framework designed for fast computation on large datasets.
- It features in-memory cluster computing to increase processing speed and is used for fast data analytics like batch processing, iterative algorithms, and streaming.
- Spark evolved from a UC Berkeley research project and is now a top-level Apache project used by many large companies like IBM, Netflix, and Anthropic.
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...Edureka! The document provides an overview of PySpark, an open-source cluster-computing framework that integrates with Python for data science and machine learning applications. It covers the advantages of PySpark, installation steps, and fundamental concepts such as RDDs, DataFrames, and the Spark context. Additionally, it outlines functionalities like transformations, actions, and the MLlib machine learning API offered by Spark.
Spark mhug2
Spark mhug2Joseph Niemiec Spark is an open-source software framework for rapid calculations on in-memory datasets. It uses Resilient Distributed Datasets (RDDs) that can be recreated if lost and supports transformations and actions on RDDs. Spark is useful for batch, interactive, and real-time processing across various problem domains like SQL, streaming, and machine learning via MLlib.
Apache Spark At Apple with Sam Maclennan and Vishwanath Lakkundi
Apache Spark At Apple with Sam Maclennan and Vishwanath LakkundiDatabricks The document discusses the use of Apache Spark at Apple, highlighting the transition from traditional Hadoop and MapReduce to Spark, specifically in terms of scalability, resource optimization, and fault tolerance. It outlines the growth of Spark usage from 2016 to 2018, the challenges faced with streaming jobs, and the implementation of an elastic self-service infrastructure to enhance developer productivity and resource utilization. Additionally, it covers security measures, telemetry, and the integration of a multi-tenant history server for job management.
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016Anya Bida The document provides a detailed guide on tuning Apache Spark for enterprise system administrators, highlighting key parameters, memory management, and techniques for optimal application performance. It emphasizes the importance of configuring Spark settings beyond defaults and includes recommendations for managing memory issues, such as utilizing checkpoints and appropriate serialization. Additionally, it offers further reading resources and links to related presentations and cheat sheets.
Spark Summit EU talk by Tim Hunter
Spark Summit EU talk by Tim HunterSpark Summit This document summarizes Timothée Hunter's presentation on TensorFrames, which allows running Google TensorFlow models on Apache Spark. Some key points:
- TensorFrames embeds TensorFlow into Spark to enable distributed numerical computing on big data. This leverages GPUs to speed up computationally intensive machine learning algorithms.
- An example demonstrates speedups from using TensorFrames and GPUs for kernel density estimation, a non-parametric statistical technique.
- Future improvements include better integration with Tungsten in Spark for direct memory copying and columnar storage to reduce communication costs.
Securing Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise ContextDataWorks Summit/Hadoop Summit Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
Stream All Things—Patterns of Modern Data Integration with Gwen Shapira
Stream All Things—Patterns of Modern Data Integration with Gwen ShapiraDatabricks This document discusses patterns for modern data integration using streaming data. It outlines an evolution from data warehouses to data lakes to streaming data. It then describes four key patterns: 1) Stream all things (data) in one place, 2) Keep schemas compatible and process data on, 3) Enable ridiculously parallel single message transformations, and 4) Perform streaming data enrichment to add additional context to events. Examples are provided of using Apache Kafka and Kafka Connect to implement these patterns for a large hotel chain integrating various data sources and performing real-time analytics on customer events.
Apache Zeppelin Meetup Christian Tzolov 1/21/16
Apache Zeppelin Meetup Christian Tzolov 1/21/16 PivotalOpenSourceHub The document outlines a unified data analytics platform integrating technologies such as Apache Zeppelin, Ambari, Geode, Spring XD, and Hawq, designed to manage and analyze large data streams. It highlights the capabilities of the platform to orchestrate processes, provide real-time analytics, and manage big data systems effectively. Key components include data lake management, in-memory data grids, and SQL compliance, facilitating high-performance applications for organizations like China Railway and Indian Railways.
JEEConf 2015 - Introduction to real-time big data with Apache Spark
JEEConf 2015 - Introduction to real-time big data with Apache SparkTaras Matyashovsky The document provides an introduction to Apache Spark, highlighting its capabilities as a fast and general-purpose platform for large-scale data processing. It discusses key components such as the RDD API, data processing techniques, and integration use cases while emphasizing Spark's performance advantages over Hadoop. The document also covers project developments, resources for learning Spark, and lessons learned from practical applications.
Advanced Visualization of Spark jobs
Advanced Visualization of Spark jobsDataWorks Summit/Hadoop Summit This document discusses an advanced visualization tool for Spark and Flink jobs. It collects fine-grained data about task execution, including data characteristics and block fetch information. This information is exposed through a REST API and used to visualize the physical execution plan, detect issues like data skew, and help developers optimize their applications. The tool aims to help understand distributed data processing systems and guide testing of adaptive partitioning techniques. It has been extended to support Flink visualization as well. Future plans include open-sourcing the framework and adding more visualization features and metrics.
Jump Start with Apache Spark 2.0 on Databricks
Jump Start with Apache Spark 2.0 on DatabricksAnyscale This document provides an agenda for a 3+ hour workshop on Apache Spark 2.x on Databricks. It includes introductions to Databricks, Spark fundamentals and architecture, new features in Spark 2.0 like unified APIs, and workshops on DataFrames/Datasets, Spark SQL, and structured streaming concepts. The agenda covers lunch and breaks and is divided into hour and half hour segments.
Building iot applications with Apache Spark and Apache Bahir
Building iot applications with Apache Spark and Apache BahirLuciano Resende The document discusses the IBM Spark Technology Center's contributions to open-source projects, particularly focusing on Apache Spark and Apache Bahir for building IoT applications. It includes technical details about the functionalities of Apache Spark, such as Spark SQL, streaming capabilities, and IoT connectivity protocols like MQTT. The document emphasizes the importance of community engagement and the potential of using these technologies for real-time IoT data processing.
IoT Applications and Patterns using Apache Spark & Apache Bahir
IoT Applications and Patterns using Apache Spark & Apache BahirLuciano Resende The document discusses IoT applications and patterns using Apache Spark and Apache Bahir, focusing on integrating various data sources and streaming capabilities for real-time analytics. It highlights the core functionalities of Apache Spark, including Spark SQL, Spark Streaming, and the extensions provided by Apache Bahir for IoT connectivity such as MQTT and Cloudant. Additionally, it presents a live demo simulating an IoT environment, outlining the benefits of leveraging these technologies for processing IoT data.
More Related Content
What's hot (20)
Transitioning Compute Models: Hadoop MapReduce to Spark
Transitioning Compute Models: Hadoop MapReduce to SparkSlim Baltagi The document discusses the evolution and transition of compute models in big data, specifically from Hadoop MapReduce to Apache Spark, highlighting the differences in performance, ease of use, and compatibility with various tools. It outlines existing projects that are migrating or adapting to Spark and details integration capabilities with different data storage systems. Additionally, it emphasizes key alternatives like Apache Tez and Flink, and the continued evolution of big data processing frameworks.
An Insider’s Guide to Maximizing Spark SQL Performance
An Insider’s Guide to Maximizing Spark SQL PerformanceTakuya UESHIN This document provides an overview of optimizing Spark SQL performance. It begins with introducing the speaker and their background with Spark. It then discusses reading query plans, interpreting them to understand optimizations, and tuning plans by pushing down filters, avoiding implicit casts, and other techniques. It emphasizes tracking query execution through the Spark UI to analyze jobs, stages and tasks for bottlenecks. The document aims to help understand how to maximize Spark SQL performance.
Data science lifecycle with Apache Zeppelin
Data science lifecycle with Apache ZeppelinDataWorks Summit/Hadoop Summit This document discusses Apache Zeppelin, an open-source web-based notebook that enables interactive data analytics. It provides an overview of Zeppelin's history and architecture, including how interpreters and notebook storage are pluggable. The document also outlines Zeppelin's roadmap for improving enterprise support through features like multi-tenancy, impersonation, job management and frontend performance.
Introduction to Apache Spark
Introduction to Apache SparkRahul Jain The document details a September 2014 Meetup organized by the Big Data Hyderabad group focusing on Apache Spark, covering its advantages for in-memory data processing, especially in machine learning. It includes an introduction to Spark's architecture, deployment, and basic programming operations in various languages such as Scala, Python, and Java. The presentation also provides examples and practical information about using Spark, including code snippets and how to submit jobs.
Spark Summit EU talk by Yiannis Gkoufas
Spark Summit EU talk by Yiannis GkoufasSpark Summit This document introduces SparkOscope, a tool that enables optimization of Apache Spark through cross-stack monitoring and visualization. SparkOscope extends Spark's monitoring framework by collecting operating system-level metrics using Hyperic Sigar library. It enhances Spark's web UI to plot all available metrics, including the new OS metrics. SparkOscope provides an easy way to monitor Spark jobs and identify bottlenecks by associating metrics with source code locations. The author demonstrates SparkOscope and discusses plans to improve its storage mechanism and enable infrastructure recommendations based on job resource usage patterns.
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...
What No One Tells You About Writing a Streaming App: Spark Summit East talk b...Spark Summit The document discusses crucial insights into writing streaming applications, particularly focusing on monitoring, management, and preventing data loss. Key topics include the importance of orchestration, driver management, checkpointing, and using Spark Streaming for efficient data processing. It emphasizes strategies for graceful shutdowns and achieving exactly-once semantics in streaming applications.
Open Source Ingredients for Interactive Data Analysis in Spark by Maxim Lukiy...
Open Source Ingredients for Interactive Data Analysis in Spark by Maxim Lukiy...DataWorks Summit/Hadoop Summit The document discusses utilizing Yarn command line with Livy server and Thrift server for Jupyter notebooks. It highlights default and Thrift queues, as well as integration with IntelliJ and BI tools. Additionally, it provides links to relevant resources and documentation for further exploration.
Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan Volz
Archiving, E-Discovery, and Supervision with Spark and Hadoop with Jordan VolzDatabricks This document discusses using Hadoop for archiving, e-discovery, and supervision. It outlines the key components of each task and highlights traditional shortcomings. Hadoop provides strengths like speed, ease of use, and security. An architectural overview shows how Hadoop can be used for ingestion, processing, analysis, and machine learning. Examples demonstrate surveillance use cases. While some obstacles remain, partners can help address areas like user interfaces and compliance storage.
Apache spark
Apache sparkTEJPAL GAUTAM This document provides an introduction and overview of Apache Spark, including:
- Spark is a lightning-fast cluster computing framework designed for fast computation on large datasets.
- It features in-memory cluster computing to increase processing speed and is used for fast data analytics like batch processing, iterative algorithms, and streaming.
- Spark evolved from a UC Berkeley research project and is now a top-level Apache project used by many large companies like IBM, Netflix, and Anthropic.
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...Edureka! The document provides an overview of PySpark, an open-source cluster-computing framework that integrates with Python for data science and machine learning applications. It covers the advantages of PySpark, installation steps, and fundamental concepts such as RDDs, DataFrames, and the Spark context. Additionally, it outlines functionalities like transformations, actions, and the MLlib machine learning API offered by Spark.
Spark mhug2
Spark mhug2Joseph Niemiec Spark is an open-source software framework for rapid calculations on in-memory datasets. It uses Resilient Distributed Datasets (RDDs) that can be recreated if lost and supports transformations and actions on RDDs. Spark is useful for batch, interactive, and real-time processing across various problem domains like SQL, streaming, and machine learning via MLlib.
Apache Spark At Apple with Sam Maclennan and Vishwanath Lakkundi
Apache Spark At Apple with Sam Maclennan and Vishwanath LakkundiDatabricks The document discusses the use of Apache Spark at Apple, highlighting the transition from traditional Hadoop and MapReduce to Spark, specifically in terms of scalability, resource optimization, and fault tolerance. It outlines the growth of Spark usage from 2016 to 2018, the challenges faced with streaming jobs, and the implementation of an elastic self-service infrastructure to enhance developer productivity and resource utilization. Additionally, it covers security measures, telemetry, and the integration of a multi-tenant history server for job management.
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016
Spark Tuning For Enterprise System Administrators, Spark Summit East 2016Anya Bida The document provides a detailed guide on tuning Apache Spark for enterprise system administrators, highlighting key parameters, memory management, and techniques for optimal application performance. It emphasizes the importance of configuring Spark settings beyond defaults and includes recommendations for managing memory issues, such as utilizing checkpoints and appropriate serialization. Additionally, it offers further reading resources and links to related presentations and cheat sheets.
Spark Summit EU talk by Tim Hunter
Spark Summit EU talk by Tim HunterSpark Summit This document summarizes Timothée Hunter's presentation on TensorFrames, which allows running Google TensorFlow models on Apache Spark. Some key points:
- TensorFrames embeds TensorFlow into Spark to enable distributed numerical computing on big data. This leverages GPUs to speed up computationally intensive machine learning algorithms.
- An example demonstrates speedups from using TensorFrames and GPUs for kernel density estimation, a non-parametric statistical technique.
- Future improvements include better integration with Tungsten in Spark for direct memory copying and columnar storage to reduce communication costs.
Securing Hadoop in an Enterprise Context
Securing Hadoop in an Enterprise ContextDataWorks Summit/Hadoop Summit Hellmar Becker, a DevOps engineer, presented on securing Hadoop in an enterprise context at a summit in Dublin on April 14, 2016. The challenges of securing Hadoop include its default lack of security and risks of data loss, privacy breaches, and system intrusions. ING uses Hadoop for data storage, advanced analytics, real-time processing, and reporting. To secure Hadoop, ING implemented perimeter security, integrated Hadoop with its Active Directory for authentication and authorization using Ranger and Kerberos, and developed custom scripts to sync user groups efficiently with Ranger's limitations. Further improvements could include integrating OS and Hadoop security and using Identity and Policy Authentication for a centralized user database.
Stream All Things—Patterns of Modern Data Integration with Gwen Shapira
Stream All Things—Patterns of Modern Data Integration with Gwen ShapiraDatabricks This document discusses patterns for modern data integration using streaming data. It outlines an evolution from data warehouses to data lakes to streaming data. It then describes four key patterns: 1) Stream all things (data) in one place, 2) Keep schemas compatible and process data on, 3) Enable ridiculously parallel single message transformations, and 4) Perform streaming data enrichment to add additional context to events. Examples are provided of using Apache Kafka and Kafka Connect to implement these patterns for a large hotel chain integrating various data sources and performing real-time analytics on customer events.
Apache Zeppelin Meetup Christian Tzolov 1/21/16
Apache Zeppelin Meetup Christian Tzolov 1/21/16 PivotalOpenSourceHub The document outlines a unified data analytics platform integrating technologies such as Apache Zeppelin, Ambari, Geode, Spring XD, and Hawq, designed to manage and analyze large data streams. It highlights the capabilities of the platform to orchestrate processes, provide real-time analytics, and manage big data systems effectively. Key components include data lake management, in-memory data grids, and SQL compliance, facilitating high-performance applications for organizations like China Railway and Indian Railways.
JEEConf 2015 - Introduction to real-time big data with Apache Spark
JEEConf 2015 - Introduction to real-time big data with Apache SparkTaras Matyashovsky The document provides an introduction to Apache Spark, highlighting its capabilities as a fast and general-purpose platform for large-scale data processing. It discusses key components such as the RDD API, data processing techniques, and integration use cases while emphasizing Spark's performance advantages over Hadoop. The document also covers project developments, resources for learning Spark, and lessons learned from practical applications.
Advanced Visualization of Spark jobs
Advanced Visualization of Spark jobsDataWorks Summit/Hadoop Summit This document discusses an advanced visualization tool for Spark and Flink jobs. It collects fine-grained data about task execution, including data characteristics and block fetch information. This information is exposed through a REST API and used to visualize the physical execution plan, detect issues like data skew, and help developers optimize their applications. The tool aims to help understand distributed data processing systems and guide testing of adaptive partitioning techniques. It has been extended to support Flink visualization as well. Future plans include open-sourcing the framework and adding more visualization features and metrics.
Jump Start with Apache Spark 2.0 on Databricks
Jump Start with Apache Spark 2.0 on DatabricksAnyscale This document provides an agenda for a 3+ hour workshop on Apache Spark 2.x on Databricks. It includes introductions to Databricks, Spark fundamentals and architecture, new features in Spark 2.0 like unified APIs, and workshops on DataFrames/Datasets, Spark SQL, and structured streaming concepts. The agenda covers lunch and breaks and is divided into hour and half hour segments.
Open Source Ingredients for Interactive Data Analysis in Spark by Maxim Lukiy...
Open Source Ingredients for Interactive Data Analysis in Spark by Maxim Lukiy...DataWorks Summit/Hadoop Summit
Similar to Writing Apache Spark and Apache Flink Applications Using Apache Bahir (20)
Building iot applications with Apache Spark and Apache Bahir
Building iot applications with Apache Spark and Apache BahirLuciano Resende The document discusses the IBM Spark Technology Center's contributions to open-source projects, particularly focusing on Apache Spark and Apache Bahir for building IoT applications. It includes technical details about the functionalities of Apache Spark, such as Spark SQL, streaming capabilities, and IoT connectivity protocols like MQTT. The document emphasizes the importance of community engagement and the potential of using these technologies for real-time IoT data processing.
IoT Applications and Patterns using Apache Spark & Apache Bahir
IoT Applications and Patterns using Apache Spark & Apache BahirLuciano Resende The document discusses IoT applications and patterns using Apache Spark and Apache Bahir, focusing on integrating various data sources and streaming capabilities for real-time analytics. It highlights the core functionalities of Apache Spark, including Spark SQL, Spark Streaming, and the extensions provided by Apache Bahir for IoT connectivity such as MQTT and Cloudant. Additionally, it presents a live demo simulating an IoT environment, outlining the benefits of leveraging these technologies for processing IoT data.
Started with-apache-spark
Started with-apache-sparkHappiest Minds Technologies The document discusses the evolution and significance of big data, highlighting Apache Spark as a pivotal open-source framework that addresses diverse use cases for data processing and analytics in a business context. It outlines features like machine learning capabilities, real-time data processing, and the integration of Spark with existing Hadoop environments, including various misconceptions about its operation. The author concludes that leveraging tools like Apache Spark can significantly enhance performance and insights for businesses managing large data sets.
Apache Spark PDF
Apache Spark PDFNaresh Rupareliya Spark is a cluster computing framework designed to be fast, general-purpose, and able to handle a wide range of workloads including batch processing, iterative algorithms, interactive queries, and streaming. It is faster than Hadoop for interactive queries and complex applications by running computations in-memory when possible. Spark also simplifies combining different processing types through a single engine. It offers APIs in Java, Python, Scala and SQL and integrates closely with other big data tools like Hadoop. Spark is commonly used for interactive queries on large datasets, streaming data processing, and machine learning tasks.
In Memory Analytics with Apache Spark
In Memory Analytics with Apache SparkVenkata Naga Ravi The document provides an overview of Apache Spark, highlighting its in-memory analytics capabilities that drastically improve query response times, making it a powerful alternative to Hadoop. It explains Spark's architecture, its elements like Resilient Distributed Datasets (RDDs), and its stack extensions including Shark for SQL, Mlib for machine learning, and GraphX for graph processing. Additionally, it compares Spark with Hadoop, emphasizing Spark's speed and efficiency in handling large-scale data processing tasks.
Spark Concepts - Spark SQL, Graphx, Streaming
Spark Concepts - Spark SQL, Graphx, StreamingPetr Zapletal This document provides an overview of Apache Spark modules including Spark SQL, GraphX, and Spark Streaming. Spark SQL allows querying structured data using SQL, GraphX provides APIs for graph processing, and Spark Streaming enables scalable stream processing. The document discusses Resilient Distributed Datasets (RDDs), SchemaRDDs, querying data with SQLContext, GraphX property graphs and algorithms, StreamingContext, and input/output operations in Spark Streaming.
Evolution of spark framework for simplifying data analysis.
Evolution of spark framework for simplifying data analysis.Anirudh Gangwar This document provides an overview of Spark, a framework for simplifying big data analytics. It discusses the types of data used in big data, defines big data and big data analytics. It then describes Hadoop's traditional approach using HDFS for storage and MapReduce for processing. The document introduces Spark as a faster alternative to Hadoop and describes Spark's ecosystem including Spark SQL, Spark Streaming, MLib, and GraphX. It compares Hadoop and Spark and concludes that the choice depends on the specific use case.
Review on Apache Spark Technology
Review on Apache Spark TechnologyIRJET Journal 1. Apache Spark is an open-source cluster computing framework that is faster than Hadoop as it performs processing in main memory rather than on disk.
2. Spark was developed in 2009 at UC Berkeley and is now an Apache project. It can run on its own or on an existing Hadoop cluster, accessing Hadoop data.
3. Spark features include speed (100x faster than Hadoop for in-memory processing), ease of use through APIs in Java, Scala, Python, support for SQL queries, and the ability to run on Hadoop, standalone, or the cloud.
Apache Spark in Industry
Apache Spark in IndustryDorian Beganovic 1. Apache Spark is an open source cluster computing framework for large-scale data processing. It is compatible with Hadoop and provides APIs for SQL, streaming, machine learning, and graph processing.
2. Over 3000 companies use Spark, including Microsoft, Uber, Pinterest, and Amazon. It can run on standalone clusters, EC2, YARN, and Mesos.
3. Spark SQL, Streaming, and MLlib allow for SQL queries, streaming analytics, and machine learning at scale using Spark's APIs which are inspired by Python/R data frames and scikit-learn.
Hadoop or Spark: is it an either-or proposition? By Slim Baltagi
Hadoop or Spark: is it an either-or proposition? By Slim BaltagiSlim Baltagi The document discusses the relationship between Apache Spark and Hadoop, debating whether they complement each other or compete. It presents insights from various perspectives, including news, surveys, vendors, and analysts, asserting that Spark is gaining traction and redefining the big data landscape. Key takeaways emphasize that big data is no longer solely tied to Hadoop, and Spark is emerging as a vital player in big data processing.
Boston Spark Meetup event Slides Update
Boston Spark Meetup event Slides Updatevithakur This document summarizes IBM's announcement of a major commitment to advance Apache Spark. It discusses IBM's investments in Spark capabilities, including log processing, graph analytics, stream processing, machine learning, and unified data access. Key reasons for interest in Spark include its performance (up to 100x faster than Hadoop for some tasks), productivity gains, ability to leverage existing Hadoop investments, and continuous community improvements. The document also provides an overview of Spark's architecture, programming model using resilient distributed datasets (RDDs), and common use cases like interactive querying, batch processing, analytics, and stream processing.
Spark Summit Presentation by Anjul Bhambhri
Spark Summit Presentation by Anjul BhambhriSpark Summit Apache Spark is a framework for large-scale data processing. IBM fully supports Spark and is building it into many of its products and services. Spark can handle both batch and streaming analytics efficiently using techniques like the Lambda architecture. IBM discusses several use cases for Spark including weather data analytics, healthcare data lakes, and telecom customer experience analysis.
Spark Summit East Keynote by Anjul Bhambhri
Spark Summit East Keynote by Anjul BhambhriJen Aman Apache Spark is a framework for large-scale data processing. IBM fully supports Spark and is building it into many of its products and services. Spark can handle both batch and streaming analytics efficiently using techniques like the Lambda architecture. IBM discusses several use cases for Spark including weather data analytics, healthcare data lakes, and customer experience analysis in telecom.
Keynote at spark summit east anjul
Keynote at spark summit east anjulAnjul Bhambhri Apache Spark is a framework for large-scale data processing. IBM fully supports Spark and is building it into many of its products and services. Spark can handle both batch and streaming analytics efficiently and enables self-service data exploration through visualization. IBM discusses several use cases for Spark including weather data analytics, healthcare, and telecom customer analytics.
Apache Spark - A High Level overview
Apache Spark - A High Level overviewKaran Alang Apache Spark is an open-source unified analytics engine for large-scale data processing. It provides high-level APIs in Scala, Java, Python, and R, and an optimized engine that supports general computation graphs for data analysis. Some key components of Apache Spark include Resilient Distributed Datasets (RDDs), DataFrames, Datasets, and Spark SQL for structured data processing. Spark also supports streaming, machine learning via MLlib, and graph processing with GraphX.
Spark Summit EU: IBM Keynote
Spark Summit EU: IBM Keynotesparktc The document discusses IBM's investment in Apache Spark as a key analytics operating system, highlighting its performance, productivity, and support for multiple data solutions across various IBM products. It describes a scalable scan-sharing engine developed on Spark for financial reporting and innovative customer experience mapping using real-time data processing and advanced analytics techniques. The announcement also includes the availability of Spark-as-a-Service on IBM Bluemix and the introduction of the IBM Data Science Notebook.
Apache Spark and Python: unified Big Data analytics
Apache Spark and Python: unified Big Data analyticsJulien Anguenot The document provides an overview of Apache Spark, highlighting its unified computing engine for big data analytics, the support for multiple programming languages, and its community and ecosystem. It discusses Spark's evolution since its inception at UC Berkeley in 2009, its architecture, and APIs including PySpark for data analysis and machine learning. Additionally, it addresses the challenges posed by big data and emphasizes the benefits of in-memory processing and real-time analytics as part of Spark's capabilities.
Vitalii Bondarenko HDinsight: spark. advanced in memory big-data analytics wi...
Vitalii Bondarenko HDinsight: spark. advanced in memory big-data analytics wi...Аліна Шепшелей The document provides an overview of Spark, a fast and general-purpose cluster computing platform, detailing its components such as Spark Core, Spark SQL, and Spark ML, along with their functionalities and use cases. It covers features like resilient distributed datasets (RDDs), transformations, actions, and Spark Streaming for real-time data processing. Additionally, it highlights the benefits of using Spark on Microsoft Azure HDInsight, including ease of cluster creation, integration with various Azure services, and support for data exploration and analysis.
SE2016 BigData Vitalii Bondarenko "HD insight spark. Advanced in-memory Big D...
SE2016 BigData Vitalii Bondarenko "HD insight spark. Advanced in-memory Big D...Inhacking This document provides an overview of Spark and using Spark on HDInsight. It discusses Spark concepts like RDDs, transformations, and actions. It also covers Spark extensions like Spark SQL, Spark Streaming, and MLlib. Finally, it highlights benefits of using Spark on HDInsight like integration with Azure services, scalability, and support.
Ad
More from Luciano Resende (20)
A Jupyter kernel for Scala and Apache Spark.pdf
A Jupyter kernel for Scala and Apache Spark.pdfLuciano Resende Apache Toree is a Scala-based Jupyter kernel designed to execute Scala code within Jupyter notebooks and to connect with Apache Spark, enabling interactive applications. It supports over 50 programming languages and integrates with big data platforms like Apache Spark, providing a rich environment for data visualization and manipulation. The document outlines installation instructions, historical context, key features, and community contributions related to Apache Toree.
Using Elyra for COVID-19 Analytics
Using Elyra for COVID-19 AnalyticsLuciano Resende The document discusses Elyra, an open-source set of AI-focused extensions for JupyterLab designed to streamline the development and deployment of machine learning models, particularly in scenarios like COVID-19 analytics. Elyra enhances productivity through features like hybrid runtime support, visual pipeline editors, and Git integration, simplifying collaboration and resource management. It also highlights IBM's commitment to open-source contributions and innovation within the AI and data science community.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.
Elyra - a set of AI-centric extensions to JupyterLab Notebooks.Luciano Resende The document introduces Elyra, a set of AI-centric extensions to JupyterLab designed to simplify the AI model development lifecycle for data scientists and machine learning engineers. It highlights JupyterLab's capabilities, including interactive notebooks and hybrid runtime support, as well as its integration with tools like Kubeflow for managing machine learning workflows. Additionally, it discusses IBM's significant participation in open source software and its contributions to AI technologies over the years.
From Data to AI - Silicon Valley Open Source projects come to you - Madrid me...
From Data to AI - Silicon Valley Open Source projects come to you - Madrid me...Luciano Resende The document outlines IBM's contributions to open source AI technologies, highlighting key initiatives like the Model Asset Exchange and Data Asset Exchange, which facilitate data sharing and model deployment. It details IBM's historical achievements in AI, from Deep Blue to contemporary AI frameworks, and emphasizes the importance of fairness, transparency, and accountability in AI systems. Additionally, it addresses advancements in tools and platforms, including Jupyter Notebooks for data science and the use of Kubernetes for machine learning workflows.
Ai pipelines powered by jupyter notebooks
Ai pipelines powered by jupyter notebooksLuciano Resende The document outlines the role of Jupyter Notebooks in AI pipelines, emphasizing their interactive capabilities for coding, data visualization, and multi-language support. It discusses IBM's contributions to open source initiatives, the management of AI workloads using platforms like Kubeflow and Apache Airflow, and tools like Papermill for notebook parameterization. Additionally, it highlights the importance of frameworks such as Fabric for Deep Learning in enabling scalable and accessible deep learning solutions.
Strata - Scaling Jupyter with Jupyter Enterprise Gateway
Strata - Scaling Jupyter with Jupyter Enterprise GatewayLuciano Resende The document presents an overview of the Jupyter Enterprise Gateway, emphasizing its capabilities to enhance the performance and scalability of Jupyter notebooks in enterprise environments. It details the architecture, features, and benefits, including support for multiple users, improved resource allocation, and integration with Kubernetes for managing AI workloads. The document also includes information about the contributions of IBM to open-source technologies and their focus on collaborative innovation in data and AI solutions.
Scaling notebooks for Deep Learning workloads
Scaling notebooks for Deep Learning workloadsLuciano Resende The document discusses the use of Jupyter notebooks for deep learning workloads, emphasizing the importance of Kubernetes for resource management and scalability. It highlights the transition to using Jupyter enterprise gateway for kernel isolation and efficient resource utilization, enabling smoother deep learning application lifecycles. The presentation outlines the benefits of using lightweight containerized environments and the integration of various tools in the deep learning infrastructure to support extensive data science applications.
Jupyter Enterprise Gateway Overview
Jupyter Enterprise Gateway OverviewLuciano Resende This document discusses Jupyter notebooks and the Jupyter Enterprise Gateway. It provides an overview of Jupyter notebooks, their functionality, and architecture. It then describes the limitations of running Jupyter notebooks at an enterprise scale. The Jupyter Enterprise Gateway is introduced as a solution that enables sharing of resources across clusters in a scalable, secure, and multi-tenant way. The document outlines the key features and architecture of the Gateway when used with YARN and Kubernetes clusters. Finally, it provides information on deploying and connecting to the Jupyter Enterprise Gateway.
Inteligencia artificial, open source e IBM Call for Code
Inteligencia artificial, open source e IBM Call for CodeLuciano Resende The document outlines IBM's initiatives in open source and artificial intelligence, highlighting the contributions of Luciano Resende and the various projects like the Model Asset Exchange and the Fabric for Deep Learning (FFDL). It emphasizes the importance of open source in IBM's AI development, detailing how their platform supports deep learning across multiple frameworks and enhances enterprise applications through tools like the Jupyter Enterprise Gateway. The document also describes IBM's historical achievements in AI and the future direction of their open-source efforts.
Getting insights from IoT data with Apache Spark and Apache Bahir
Getting insights from IoT data with Apache Spark and Apache BahirLuciano Resende The document discusses the integration of IoT data insights using Apache Spark and Apache Bahir, highlighting their capabilities in processing real-time data and creating anomaly detection applications. It covers various IoT use cases and design patterns, such as lambda and kappa architectures, along with specific examples like smart homes and connected vehicles. Additionally, it emphasizes the importance of effective data handling and visualization in enterprise applications while showcasing live demos and contributions to the open-source community.
Open Source AI - News and examples
Open Source AI - News and examplesLuciano Resende Luciano Resende, a data science platform architect at IBM, discusses the company's commitment to open source and AI over the years, including the establishment of various initiatives and projects like the Model Asset Exchange (MAX) and the Fabric for Deep Learning (FFDL). IBM's contributions to the open source community include substantial code contributions and the promotion of deep learning and AI technologies, making them accessible to developers. The presentation highlights the importance of collaboration and innovation in advancing AI solutions in the enterprise environment.
Building analytical microservices powered by jupyter kernels
Building analytical microservices powered by jupyter kernelsLuciano Resende This document discusses building analytical microservices powered by Jupyter kernels. It provides an overview of Jupyter notebooks and their architecture. It then introduces the Jupyter Enterprise Gateway, which allows running Jupyter kernels on a distributed cluster for improved scalability and security. Finally, it demonstrates a use case of a sentiment analysis microservice that leverages PySpark on a Hadoop cluster via Jupyter kernels.
An Enterprise Analytics Platform with Jupyter Notebooks and Apache Spark
An Enterprise Analytics Platform with Jupyter Notebooks and Apache SparkLuciano Resende The IBM Spark Technology Center's document outlines the features and functionalities of their data science platform, including the integration of Jupyter notebooks and Apache Spark for enhanced analytics. It highlights the capabilities of the IBM Analytics Engine, the use of Jupyter Enterprise Gateway for multi-user support and resource allocation, and provides deployment instructions for setting up these tools. Additionally, it emphasizes the center's contribution to open-source projects and its mission to assist in making Apache Spark technology ready for enterprise and cloud environments.
The Analytic Platform behind IBM’s Watson Data Platform - Big Data Spain 2017
The Analytic Platform behind IBM’s Watson Data Platform - Big Data Spain 2017Luciano Resende The document discusses the IBM Spark Technology Center's contributions to the Apache Spark community, focusing on the IBM Data Science Experience and the IBM Analytics Engine, which facilitate data science productivity and efficient data model building. It highlights the use of Jupyter notebooks and the development of the Jupyter Enterprise Gateway to enable multi-user support and enhance resource allocation, security, and administration across Apache Spark clusters. Additionally, it outlines the center's mission to make core technologies enterprise and cloud-ready while building data science skills.
What's new in Apache SystemML - Declarative Machine Learning
What's new in Apache SystemML - Declarative Machine LearningLuciano Resende The document discusses the IBM Spark Technology Center and its contributions to Apache Spark and SystemML, focusing on machine learning advancements and open source leadership. It outlines the history, achievements, and technical capabilities of the Spark Technology Center, highlighting projects such as SystemML and its effectiveness in optimizing machine learning algorithms. Additionally, it emphasizes the integration of various programming languages and APIs for broader applicability in data science tasks.
Big analytics meetup - Extended Jupyter Kernel Gateway
Big analytics meetup - Extended Jupyter Kernel GatewayLuciano Resende Luciano Resende from IBM's Spark Technology Center presented on building an enterprise/cloud analytics platform with Jupyter Notebooks and Apache Spark. The Spark Technology Center focuses on contributions to open source Apache Spark projects. Resende discussed limitations of the current Jupyter Notebook setup for multi-user shared clusters and demonstrated an Extended Jupyter Kernel Gateway that allows running kernels remotely in a cluster with enhanced security, resource optimization, and multi-user support through user impersonation. The Extended Jupyter Kernel Gateway is planned for open source release.
Jupyter con meetup extended jupyter kernel gateway
Jupyter con meetup extended jupyter kernel gatewayLuciano Resende The IBM Spark Technology Center focuses on developing an enterprise/cloud analytics platform utilizing Jupyter notebooks and Apache Spark, aiming to enhance data science skills and improve resource allocation. The center is characterized by a shared computing environment, multiuser support, and ongoing efforts to optimize security and manage resources more effectively. The extended Jupyter kernel gateway is being developed to provide better utilization of resources and support for additional resource managers, with plans for open-sourcing the technology soon.
Asf icfoss-mentoring
Asf icfoss-mentoringLuciano Resende The document provides an overview of a 3-day open source workshop being conducted by Luciano Resende from the Apache Software Foundation. Day 1 will cover topics on open source, licenses, communities and how to get involved in Apache projects. Day 2 focuses on hands-on development, setting up environments and tools. Day 3 is about mentoring expectations and working on project proposals. The workshop aims to educate participants and help them get involved in open source.
Open Source tools overview
Open Source tools overviewLuciano Resende This document provides an overview of tools for software development within the Apache Software Foundation, focusing on Subversion, Git, and Maven. It includes installation instructions, key commands, and workflows for utilizing these version control and build management tools. Additionally, practical exercises and suggested homework are included to enhance understanding and collaboration using GitHub.
Data access layer and schema definitions
Data access layer and schema definitionsLuciano Resende Luciano Resende is a staff software engineer at Shutterfly and committer to Apache projects. He presented on using MongoDB and developing a data access layer and schema definition. The schema definition uses a textual domain-specific language to define models for code generation including persistence, DTOs, mapping, validation and more across platforms like Java, RDBMS and MongoDB.
Ad
Recently uploaded (20)
Report_Government Authorities_Index_ENG_FIN.pdf
Report_Government Authorities_Index_ENG_FIN.pdfOlhaTatokhina1 Report based on the findings of a quantitative research conducted by the research agency New Image Marketing Group, commissioned by the NGO Detector Media, compiled by PhD in Sociology Marta Naumova.
THE FRIEDMAN TEST ( Biostatics B. Pharm)
THE FRIEDMAN TEST ( Biostatics B. Pharm)JishuHaldar The Friedman Test is a valuable non-parametric alternative to the
Repeated Measures ANOVA, allowing for the comparison of three or
more related groups when data is ordinal or not normally distributed.
By ranking data instead of using raw values, the test overcomes the
limitations of parametric tests, making it ideal for small sample sizes and
real-world applications in medicine, psychology, pharmaceutical
sciences, and education. However, while it effectively detects differences
among groups, it does not indicate which specific groups differ, requiring
further post-hoc analysis.
Managed Cloud services - Opsio Cloud Man
Managed Cloud services - Opsio Cloud ManOpsio Cloud At Opsio, we specialize in delivering advanced cloud services that enable businesses to scale, transform, and modernize with confidence. Our core offerings focus on cloud management, digital transformation, and cloud modernization — all designed to help organizations unlock the full potential of their technology infrastructure.We take a client-first approach, blending industry-leading hosted technologies with strategic expertise to create tailored, future-ready solutions. Leveraging AI, automation, and emerging technologies, our services simplify IT operations, enhance agility, and accelerate business outcomes. Whether you're migrating to the cloud or optimizing existing cloud environments, Opsio is your partner in achieving sustainable, measurable success.
apidays New York 2025 - Life is But a (Data) Stream by Sandon Jacobs (Confluent)
apidays New York 2025 - Life is But a (Data) Stream by Sandon Jacobs (Confluent)apidays Life is But a (Data) Stream: Building Quality Data Pipelines
Sandon Jacobs, Senior Developer Advocate at Confluent
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
SAP_S4HANA_PPM_IT_Corporate_Services_Presentation.pptx
SAP_S4HANA_PPM_IT_Corporate_Services_Presentation.pptxvemulavenu484 The Corporate and IT Services industry is undergoing rapid transformation with growing demand for agility, innovation, and strategic project delivery. Enterprises are challenged to manage a wide variety of service-based projects, such as IT infrastructure upgrades, software development, digital transformation, regulatory compliance, and internal operational improvements.
SAP S/4HANA Portfolio and Project Management (PPM) is designed to help organizations align IT and corporate project portfolios with business goals, improve project visibility, enhance resource utilization, and ensure financial control. This document outlines key features of SAP PPM and its relevance for the Corporate and IT Services industry, emphasizing value realization and strategic alignment.
apidays New York 2025 - The Future of Small Business Lending with Open Bankin...
apidays New York 2025 - The Future of Small Business Lending with Open Bankin...apidays The Future of Small Business Lending with Open Banking – Bridging the $750 Billion Funding Gap
Charles Groome, Vice President of Growth Strategy at Biz2Credit
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
What is FinOps as a Service and why is it Trending?
What is FinOps as a Service and why is it Trending?Amnic The way we build and scale companies today has changed forever because of cloud adoption. However, this flexibility introduces unpredictability, which often results in overspending, inefficiencies, and a lack of cost accountability.
FinOps as a Service is a modern approach to cloud cost management that combines powerful tooling with expert advisory to bring financial visibility, governance, and optimization into the cloud operating model, without slowing down the engineering team. FinOps empowers the engineering team, finance, and leadership/management as they make data-informed decisions about cost, together.
In this presentation, we will break down what FinOps is, why it matters more than ever, and a little about how a managed FinOps service can help organizations:
- Optimize cloud spend - without slowing down dev
- Create visibility into the cost per team, service, or feature
- Set financial guardrails while allowing autonomy in engineering
- Drive cultural alignment between finance, engineering, and product
This will guide and help whether you are a cloud-native startup or a scaling enterprise, and convert cloud cost into a strategic advantage.
Media_Literacy_Index_of_Media_Sector_Employees.pdf
Media_Literacy_Index_of_Media_Sector_Employees.pdfOlhaTatokhina1 Report based on the findings of a quantitative research conducted by the research agency New Image Marketing Group, commissioned by the NGO Detector Media, compiled by PhD in Sociology Marta Naumova.
apidays Singapore 2025 - 4 Identity Essentials for Scaling SaaS in Large Orgs...
apidays Singapore 2025 - 4 Identity Essentials for Scaling SaaS in Large Orgs...apidays 4 identity factors you didn't know you needed to support large organizations in your SaaS
Daizen Ikehara, Principal Developer Advocate at Auth0
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Building Scalable AI Systems by Sai Prasad Veluru (Ap...
apidays New York 2025 - Building Scalable AI Systems by Sai Prasad Veluru (Ap...apidays Building Scalable AI Systems: Cloud Architecture for Performance
Sai Prasad Veluru, Software Engineer at Apple Inc
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Veilig en vlot fietsen in Oost-Vlaanderen: Fietssnelwegen geoptimaliseerd met...
Veilig en vlot fietsen in Oost-Vlaanderen: Fietssnelwegen geoptimaliseerd met...jacoba18 Dit project, in samenwerking met de provincie Oost-Vlaanderen, heeft als doel de veiligheid op de F45 tussen Oudenaarde en Gent te verbeteren door netwerkdata aan te vullen met snelheidslimieten, toegangsbeperkingen en voorrangsregels. Met mobile mapping en FME werden verkeersborden en kruispunten geanalyseerd, waarna de resultaten in ArcGIS inzicht boden in toegankelijke wegsegmenten en verkeersregels.
apidays Singapore 2025 - Enhancing Developer Productivity with UX (Government...
apidays Singapore 2025 - Enhancing Developer Productivity with UX (Government...apidays Enhancing Developer Productivity with UX
Petrine Tang, UX Designer at Government Technology Agency
Faith Ang, Product Manager at Government Technology Agency
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Using GraphQL SDL files as executable API Contracts b...
apidays New York 2025 - Using GraphQL SDL files as executable API Contracts b...apidays Using GraphQL SDL files as executable API Contracts
Hari Krishnan, Co-founder & CTO at Specmatic
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
SAP_S4HANA_EWM_Food_Processing_Industry.pptx
SAP_S4HANA_EWM_Food_Processing_Industry.pptxvemulavenu484 SAP Extended Warehouse Management (EWM) is a part of SAP S/4HANA offering advanced warehouse and logistics capabilities. It enables efficient handling of goods movement, storage, and inventory in real-time.
apidays New York 2025 - Why I Built Another Carbon Measurement Tool for LLMs ...
apidays New York 2025 - Why I Built Another Carbon Measurement Tool for LLMs ...apidays Why I Built Another Carbon Measurement Tool for LLMs (And What I Learned Along the Way)
Pascal Joly, Sustainability Consultant and Instructor at IT Climate Ed
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...apidays Beyond Webhooks: The Future of Scalable API Event Delivery
Phil Leggetter, Head of Developer Experience at Hookdeck
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELSAmeya Patekar QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
Writing Apache Spark and Apache Flink Applications Using Apache Bahir
- 1. IBM SparkTechnology Center Apache Big Data Seville 2016 Apache Bahir Writing Applications using Apache Bahir Luciano Resende IBM | Spark Technology Center
- 2. IBM SparkTechnology Center About Me Luciano Resende ([email protected]) • Architect and community liaison at IBM – Spark Technology Center • Have been contributing to open source at ASF for over 10 years • Currently contributing to : Apache Bahir, Apache Spark, Apache Zeppelin and Apache SystemML (incubating) projects 2 @lresende1975 http://lresende.blogspot.com/ https://www.linkedin.com/in/lresendehttp://slideshare.net/luckbr1975lresende
- 3. IBM SparkTechnology Center Origins of the Apache Bahir Project MAY/2016: Established as a top-level Apache Project. • PMC formed by Apache Spark committers/pmc, Apache Members • Initial contributions imported from Apache Spark AUG/2016: Flink community join Apache Bahir • Initial contributions of Flink extensions • In October 2016 Robert Metzger elected committer
- 4. IBM SparkTechnology Center The Apache Bahir name Naming an Apache Project is a science !!! • We needed a name that wasn’t used yet • Needed to be related to Spark We ended up with : Bahir • A name of Arabian origin that means Sparkling, • Also associated with a guy who succeeds at everything 4
- 5. IBM SparkTechnology Center Why Apache Bahir It’s an Apache project • And if you are here, you know what it means What are the benefits of curating your extensions at Apache Bahir • Apache Governance • Apache License • Apache Community • Apache Brand 5
- 6. IBM SparkTechnology Center Why Apache Bahir Flexibility • Release flexibility • Bounded to platform or component release Shared infrastructure • Release, CI, etc Shared knowledge • Collaborate with experts on both platform and component areas 6
- 7. IBM SparkTechnology Center Apache Spark 7
- 8. IBM SparkTechnology Center Apache Spark - Introduction What is Apache Spark ? 8 Spark Core Spark SQL Spark Streaming Spark ML Spark GraphX executes SQL statements performs streaming analytics using micro-batches common machine learning and statistical algorithms distributed graph processing framework general compute engine, handles distributed task dispatching, scheduling and basic I/O functions large variety of data sources and formats can be supported, both on- premise or cloud BigInsights (HDFS) Cloudant dashDB SQL DB
- 9. IBM SparkTechnology Center Apache Spark – Spark SQL 9 Spark Core Spark SQL Spark Streaming Spark ML Spark GraphX ▪Unified data access: Query structured data sets with SQL or Dataset/DataFrame APIs ▪Fast, familiar query language across all of your enterprise dataRDBMS Data Sources Structured Streaming Data Sources
- 10. IBM SparkTechnology Center Apache Spark – Spark SQL You can run SQL statement with SparkSession.sql(…) interface: val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() spark.sql(“create table T1 (c1 int, c2 int) stored as parquet”) val ds = spark.sql(“select * from T1”) You can further transform the resultant dataset: val ds1 = ds.groupBy(“c1”).agg(“c2”-> “sum”) val ds2 = ds.orderBy(“c1”) The result is a DataFrame / Dataset[Row] ds.show() displays the rows 10
- 11. IBM SparkTechnology Center Apache Spark – Spark SQL You can read from data sources using SparkSession.read.format(…) val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer) // loading csv data to a Dataset of Bank type val bankFromCSV = spark.read.csv(“hdfs://localhost:9000/data/bank.csv").as[Bank] // loading JSON data to a Dataset of Bank type val bankFromJSON = spark.read.json(“hdfs://localhost:9000/data/bank.json").as[Bank] // select a column value from the Dataset bankFromCSV.select(‘age).show() will return all rows of column “age” from this dataset. 11
- 12. IBM SparkTechnology Center Apache Spark – Spark SQL You can also configure a specific data source with specific options val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() case class Bank(age: Integer, job: String, marital: String, education: String, balance: Integer) // loading csv data to a Dataset of Bank type val bankFromCSV = sparkSession.read .option("header", ”true") // Use first line of all files as header .option("inferSchema", ”true") // Automatically infer data types .option("delimiter", " ") .csv("/users/lresende/data.csv”) .as[Bank] bankFromCSV.select(‘age).show() // will return all rows of column “age” from this dataset. 12
- 13. IBM SparkTechnology Center Apache Spark – Spark SQL Data Sources under the covers • Data source registration (e.g. spark.read.datasource) • Provide BaseRelation implementation • That implements support for table scans: • TableScans, PrunedScan, PrunedFilteredScan, CatalystScan • Detailed information available at • http://www.spark.tc/exploring-the-apache-spark-datasource-api/ 13
- 14. IBM SparkTechnology Center Apache Spark – Spark SQL Structured Streaming Unified programming model for streaming, interactive and batch queries 14 Image source: https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html Considers the data stream as unbounded table
- 15. IBM SparkTechnology Center Apache Spark – Spark SQL Structured Streaming SQL regular APIs val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() val input = spark.read .schema(schema) .format(”csv") .load(”input-path") val result = input .select(”age”) .where(”age > 18”) result.write .format(”json”) . save(” dest-path”) 15 Structured Streaming APIs val spark = SparkSession.builder() .appName(“Demo”) .getOrCreate() val input = spark.readStream .schema(schema) .format(”csv") .load(”input-path") val result = input .select(”age”) .where(”age > 18”) result.write .format(”json”) . startStream(” dest-path”)
- 16. IBM SparkTechnology Center Apache Spark – Spark SQL Structured Streaming 16 Structured Streaming is an ALPHA feature
- 17. IBM SparkTechnology Center Apache Spark – Spark Streaming 17 Spark Core Spark Streaming Spark SQL Spark ML Spark GraphX ▪Micro-batch event processing for near- real time analytics ▪e.g. Internet of Things (IoT) devices, Twitter feeds, Kafka (event hub), etc. ▪No multi-threading or parallel process programming required
- 18. IBM SparkTechnology Center Apache Spark – Spark Streaming Also known as discretized stream or Dstream Abstracts a continuous stream of data Based on micro-batching 18
- 19. IBM SparkTechnology Center Apache Spark – Spark Streaming val sparkConf = new SparkConf() .setAppName("MQTTWordCount") val ssc = new StreamingContext(sparkConf, Seconds(2)) val lines = MQTTUtils.createStream(ssc, brokerUrl, topic, StorageLevel.MEMORY_ONLY_SER_2) val words = lines.flatMap(x => x.split(" ")) val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _) wordCounts.print() ssc.start() ssc.awaitTermination() 19
- 20. IBM SparkTechnology Center Apache Spark extensions in Bahir MQTT – Enables reading data from MQTT Servers using Spark Streaming or Structured streaming. • http://bahir.apache.org/docs/spark/current/spark-sql-streaming-mqtt/ • http://bahir.apache.org/docs/spark/current/spark-streaming-mqtt/ Twitter – Enables reading social data from twitter using Spark Streaming. • http://bahir.apache.org/docs/spark/current/spark-streaming-twitter/ Akka – Enables reading data from Akka Actors using Spark Streaming. • http://bahir.apache.org/docs/spark/current/spark-streaming-akka/ ZeroMQ – Enables reading data from ZeroMQ using Spark Streaming. • http://bahir.apache.org/docs/spark/current/spark-streaming-zeromq/ 20
- 21. IBM SparkTechnology Center Apache Spark extensions coming soon to Bahir WebHDFS – Enables reading data from remote HDFS file system utilizing Spark SQL APIs • https://issues.apache.org/jira/browse/BAHIR-67 CounchDB / Cloudant– Enables reading data from CounchDB NoSQL document stores using Spark SQL APIs 21
- 22. IBM SparkTechnology Center Apache Spark extensions in Bahir Adding Bahir extensions into your application • Using SBT • libraryDependencies += "org.apache.bahir" %% "spark-streaming-mqtt" % "2.1.0-SNAPSHOT” • Using Maven • <dependency> <groupId>org.apache.bahir</groupId> <artifactId>spark-streaming-mqtt_2.11 </artifactId> <version>2.1.0-SNAPSHOT</version> </dependency> 22
- 23. IBM SparkTechnology Center Apache Spark extensions in Bahir Submitting applications with Bahir extensions to Spark • Spark-shell • bin/spark-shell --packages org.apache.bahir:spark-streaming_mqtt_2.11:2.1.0-SNAPSHOT ….. • Spark-submit • bin/spark-submit --packages org.apache.bahir:spark-streaming_mqtt_2.11:2.1.0-SNAPSHOT ….. 23
- 24. IBM SparkTechnology Center Apache Flink 24
- 25. IBM SparkTechnology Center Apache Flink extensions in Bahir Flink platform extensions added recently • https://github.com/apache/bahir-flink First release coming soon • Release discussions have started • Finishing up some basic documentation and examples • Should be available soon 25
- 26. IBM SparkTechnology Center Apache Flink extensions in Bahir ActiveMQ – Enables reading and publishing data from ActiveMQ servers • https://github.com/apache/bahir-flink/blob/master/flink-connector-activemq/README.md Flume– Enables publishing data to Apache Flume • https://github.com/apache/bahir-flink/tree/master/flink-connector-flume Redis – Enables reading data to Redis and publishing data to Redis PubSub • https://github.com/apache/bahir-flink/blob/master/flink-connector-redis/README.md 26
- 27. IBM SparkTechnology Center Live Demo 27
- 28. IBM SparkTechnology Center IoT Simulation using MQTT The demo environment https://github.com/lresende/bahir-iot-demo 28 Docker environment Mosquitto MQTT Server Node.js Webapplication Simulates Elevator IoT devices Elevator simulator Metrics: - Weight - Speed - Power - Temperature - System
- 29. IBM SparkTechnology Center Join the Apache Bahir community !!! 29
- 30. IBM SparkTechnology Center References Apache Bahir http://bahir.apache.org Documentation for Apache Spark extensions http://bahir.apache.org/docs/spark/current/documentation/ Source Repositories https://github.com/apache/bahir https://github.com/apache/bahir-flink https://github.com/apache/bahir-website 30 Image source: http://az616578.vo.msecnd.net/files/2016/03/21/6359412499310138501557867529_thank-you-1400x800-c-default.gif