Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners | Simplilearn
2 likes800 views
The document provides an overview of the Hadoop ecosystem, detailing various components such as HDFS for data storage, YARN for resource management, and MapReduce for data processing. It highlights tools like Sqoop for data transfer, Flume for log data ingestion, and Spark for real-time analytics, among others. Additionally, it discusses security and management features through frameworks like Ranger and Ambari, along with workflow management via Oozie.
![Oozie
Oozie is a workflow scheduler system to manage Hadoop jobs
Workflow
engine
Coordinator
engine
Consists of 2 parts
1 2
1
2
Directed Acyclic Graphs (DAGs) which
specifies a sequence of actions to be
executed
These consist of workflow jobs triggered by
time and data
availability
Start
MapReduce
Program [Action
Node]
Notify client of
success [Email
Action Node]
Notify Client of
Error [Email Action
Node]
Kill
(unsuccessful
termination)
Begin Success
Error
End
(successful
completion)](https://image.slidesharecdn.com/hadoopecosystem-190201064444/85/Hadoop-Ecosystem-Hadoop-Ecosystem-Tutorial-Hadoop-Tutorial-For-Beginners-Simplilearn-28-320.jpg)
Hadoop Ecosystem | Hadoop Ecosystem Tutorial | Hadoop Tutorial For Beginners | Simplilearn
- 1. What’s in it for you?
- 2. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 3. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 4. HDFS HDFS stands for Hadoop Distributed File System Stores different formats of data on various machines Namenode (Master) Datanode (Slave) 2 major components 128 MB 300 MB 128 MB 44 MB Splits the data into multiple blocks (128 MB by default)
- 5. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 6. YARN YARN stands for Yet Another Resource Negotiator ResourceManager (Master) NodeManager (Slave) 2 major components Allocates RAM, memory and other resources to different applications RAM Memory ResourcesHandles the cluster of nodes
- 7. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 8. MapReduce MapReduce processes large volumes of data in a parallelly distributed manner Big Data Map() Map() Map() Map() Map() Map() Shuffle and sort Reduce() Reduce() Output
- 9. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 10. Sqoop Sqoop is used to transfer data between Hadoop and external datastores such as relational databases and enterprise data warehouses Relational database and enterprise data warehouse Hadoop data It imports data from external datastores into HDFS, Hive and HBase Map Task HDFS/HBase/Hi ve Enterprise data warehouse Document based systems Relational Database Hadoop command
- 11. Flume Flume is distributed service for collecting, aggregating and moving large amounts of log data Unstructured and semi- structured data into HDFS Flume ingests Ingests online streaming data from social media, log files, web server into HDFS Web server/ Cloud/Social media data Source Sink HDFS Channel
- 12. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 13. Pig Pig is used to analyze data in Hadoop. It provides a high level data processing language to perform numerous operations on the data Pig Latin Scripts Parser Optimizer Compiler Execution Engine MapReduce HDFS Grunt Shell Pig Server Apache PigPig Latin Pig Latin Compiler Language for scripting Converts Pig Latin code to executable code Provides a platform for building data flow for ETL 10 lines of Pig Latin script is around 200 lines of MapReduce job
- 14. Hive Hive facilitates reading, writing and managing large datasets residing in the distributed storage using SQL (Hive Query Language) Hive Command Line JDBC/ODBC driver 2 major components JDBC/ODBC Hive Thrift Server Hive Web Interface CLI Driver (Compiler, Optimizer, Executor) Job Tracker Namenode Hive Datanode + Task Tracker Provides User Defined Functions (UDF) for data mining, document indexing, log processing, etc.
- 15. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 16. Spark Spark is an open-source distributed computing engine for processing and analyzing huge volumes of real time data Written in Driver Program SparkContext Cluster Manager Worker Node Task Cache Executor Task Cache Executor Worker Node Runs 100x times faster than MapReduce Provides in-memory computation of data Used to process and analyze real time streaming data such as stock market and banking data
- 17. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 18. Mahout Mahout is used to create scalable and distributed machine learning algorithms Machine learning applications Mahout environment builds Collaborative Filtering Classification Clustering Has a library that contains in- built algorithms for
- 19. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 20. Ambari Ambari is an open-source tool responsible for keeping track of running applications and their statuses Host Server Agent Agent Agent Ambari Web Database Host Server Host Server Ambari Server • Manages, monitors and provisions Hadoop clusters • Provides a central management service to start, stop and configure Hadoop services
- 21. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 22. Kafka Kafka is a distributed streaming platform to store and process streams of records Written in Builds real-time streaming data pipelines that reliably get data between applications Builds real-time streaming applications that transforms data into streams Kafka uses a messaging system for transferring data from one application to another Sender Receiver Message queue
- 23. Storm Storm is a processing engine that processes real-time streaming data at a very high speed Written in Clojure Ability to process over a million jobs in a fraction of seconds on a node It is integrated with Hadoop to harness higher throughputs
- 24. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 25. Ranger Ranger is a framework to enable, monitor and manage data securities across the Hadoop platform Provides centralized security administration to manage all security related tasks 1 Standardize authorization across all Hadoop components 2 Enhanced support for different authorization methods – Role based access control, attribute based access control, etc. 3
- 26. Knox Knox is an application gateway for interacting with the REST APIs and UIs of Hadoop deployments Knox delivers 3 groups of user facing services: Provides access to Hadoop via proxying the HTTP request Proxying Services1 Authentication for REST API access and WebSSO flow for user interfaces Authentication Services2 Client development can be done with the scripting through DSL or using the Knox shell classes Client Services3
- 27. Hadoop Ecosystem Data storage Cluster resource management Data processing Data collection and ingestion Scripting SQL queries Real time data analysis Machine Learning Management and monitoring Streaming Security Workflow system
- 28. Oozie Oozie is a workflow scheduler system to manage Hadoop jobs Workflow engine Coordinator engine Consists of 2 parts 1 2 1 2 Directed Acyclic Graphs (DAGs) which specifies a sequence of actions to be executed These consist of workflow jobs triggered by time and data availability Start MapReduce Program [Action Node] Notify client of success [Email Action Node] Notify Client of Error [Email Action Node] Kill (unsuccessful termination) Begin Success Error End (successful completion)
Editor's Notes
- #3: Style - 01
- #4: Style - 01
- #5: Style - 01
- #6: Style - 01
- #7: Style - 01
- #8: Style - 01
- #9: Style - 01
- #10: Style - 01
- #11: Style - 01
- #12: Style - 01
- #13: Style - 01
- #14: Style - 01
- #15: Style - 01
- #16: Style - 01
- #17: Style - 01
- #18: Style - 01
- #19: Style - 01
- #20: Style - 01
- #21: Style - 01
- #22: Style - 01
- #23: Style - 01
- #24: Style - 01
- #25: Style - 01
- #26: Style - 01
- #27: Style - 01
- #28: Style - 01
- #29: Style - 01