What Is Hadoop? | What Is Big Data & Hadoop | Introduction To Hadoop | Hadoop Tutorial | Simplilearn
14 likes1,918 views
The document discusses the challenges of managing big data using traditional methods, illustrated through a farming analogy where Jack hires help and modifies storage solutions for efficiency. It outlines how the paradigm of big data requires distributed storage and parallel processing techniques, exemplified by technologies like Hadoop, to manage vast volumes and varieties of data. Key components of Hadoop, such as HDFS and MapReduce, are explained alongside their applications in real-world scenarios, including combating fraudulent activities in banking.
What Is Hadoop? | What Is Big Data & Hadoop | Introduction To Hadoop | Hadoop Tutorial | Simplilearn
- 2. In a farm far away…
- 3. Jack harvests grapes and then sells it in the nearby town
- 4. After harvesting, he then stores the produce in a storage room
- 5. Soon there was a high demand for other fruits. So, he started harvesting apples and oranges as well
- 6. He then realizes that it is time consuming and difficult to harvest all the fruits by himself
- 7. So, he hires 2 more people to work with him. With this, harvesting is done simultaneously
- 8. Now, the storage room becomes a bottleneck to store and access all the fruits in a single storage area
- 9. Jack now decides to distribute the storage area and give each one of them a separate storage space
- 10. Hello, I want a fruit basket of 3 grapes, 2 apples and 3 oranges
- 11. To complete the order on time, all of them work parallelly with their own storage space Hello, I want a fruit basket of 3 grapes, 2 apples and 3 oranges
- 12. This solution helps them to complete the order on time without any hassles Fruit basket
- 13. All of them are happy and they are prepared for an increase in demand in the future
- 14. All of them are happy and they are prepared for an increase in demand in the future So, how does this story relate to Big Data?
- 15. The rise of Big Data Structured data Earlier with limited data, only one processor and one storage unit was needed
- 16. The rise of Big Data Structured data Semi structured data Unstructured data Soon, data generation increased leading to high volume of data along with different data formats
- 17. The rise of Big Data Structured data Semi structured data Unstructured data A single processor was not enough to process such high volume of different kinds of data as it was very time consuming
- 18. The rise of Big Data Structured data Semi structured data Unstructured data Hence, multiple processors were used to process high volume of data and this saved time
- 19. The rise of Big Data Structured data Semi structured data Unstructured data The single storage unit became the bottleneck due to which network overhead was generated
- 20. The rise of Big Data Structured data Semi structured data Unstructured data The solution was to use distributed storage for each processor. This enabled easy access to store and access data
- 21. The rise of Big Data Structured data Semi structured data Unstructured data This method worked and there was no network overhead generated
- 22. The rise of Big Data Structured data Semi structured data Unstructured data This is known as parallel processing with distributed storage
- 23. The rise of Big Data Structured data Semi structured data Unstructured data This is known as parallel processing with distributed storage Parallel processing
- 24. The rise of Big Data Structured data Semi structured data Unstructured data This is known as parallel processing with distributed storage Parallel processing Distributed storage
- 25. What’s in it for you?
- 26. What’s in it for you? 1. Big Data and it’s challenges1
- 27. What’s in it for you? 1. Big Data and it’s challenges1 1. Hadoop as a solution2
- 28. What’s in it for you? 1. Big Data and it’s challenges1 1. Hadoop as a solution2 1. What is Hadoop?3
- 29. What’s in it for you? 1. Big Data and it’s challenges1 1. Hadoop as a solution2 1. What is Hadoop?3 1. Components of Hadoop4
- 30. What’s in it for you? 1. Big Data and it’s challenges1 1. Hadoop as a solution2 1. What is Hadoop?3 1. Components of Hadoop4 1. Use case of Hadoop5
- 31. What is Big Data?
- 32. What is Big Data? Massive amount of data which cannot be stored, processed and analyzed using the traditional ways
- 33. What is Big Data? Massive amount of data which cannot be stored, processed and analyzed using the traditional ways VERACITY BIG DATA VELOCITY VOLUME VARIETYVALUE VERACITY
- 34. What is Big Data? Massive amount of data which cannot be stored, processed and analyzed using the traditional ways VERACITY BIG DATA VELOCITY VOLUME VARIETYVALUE VERACITY
- 35. What is Big Data? Massive amount of data which cannot be stored, processed and analyzed using the traditional ways VERACITY BIG DATA VELOCITY VOLUME VARIETYVALUE VERACITY
- 36. What is Big Data? Massive amount of data which cannot be stored, processed and analyzed using the traditional ways VERACITY BIG DATA VELOCITY VOLUME VARIETYVALUE VERACITY
- 37. What is Big Data? Massive amount of data which cannot be stored, processed and analyzed using the traditional ways VERACITY BIG DATA VELOCITY VOLUME VARIETYVALUE VERACITY
- 38. Big Data challenges and solution Single central storage Challenges
- 39. Big Data challenges and solution Distributed storagesSingle central storage Challenges Solutions Distributed storage
- 40. Big Data challenges and solution Serial processing OutputProcess Input A Distributed storagesSingle central storage Challenges Solutions Distributed storage
- 41. Big Data challenges and solution Serial processing OutputProcess Input A Distributed storagesSingle central storage Parallel processing Output B Inputs A Process Challenges Solutions Distributed storage
- 42. Big Data challenges and solution Serial processing OutputProcess Input A Distributed storagesSingle central storage Parallel processing Output B Inputs A Process Lack of ability to process unstructured data Challenges Solutions Distributed storage
- 43. Big Data challenges and solution Serial processing OutputProcess Input A Distributed storagesSingle central storage Parallel processing Output B Inputs A Process Lack of ability to process unstructured data Ability to process every type of data Challenges Solutions Distributed storage
- 44. Hadoop as a solution Serial processing OutputProcess Input A Distributed storagesSingle central storage Parallel processing Output B Inputs A Process Lack of ability to process unstructured data Ability to process every type of data Challenges Solutions Distributed storage
- 45. What is Hadoop?
- 46. What is Hadoop? Big Data VOLUME STORING Storing Processing Analyzing Hadoop is a framework that manages big data storage in a distributed way and processes it parallelly
- 48. Components of Hadoop Storage unit of Hadoop Processing unit of Hadoop
- 49. Components of Hadoop Storage unit of Hadoop Processing unit of Hadoop
- 50. What is HDFS?
- 51. What is HDFS? VOLUME STORING Hadoop Distributed File System (HDFS) is specially designed for storing huge datasets in commodity hardware Distributed storage
- 52. What is HDFS? VOLUME STORING Hadoop Distributed File System (HDFS) has two core components NameNode and DataNode NameNode DataNode
- 53. What is HDFS? VOLUME STORING Hadoop Distributed File System (HDFS) has two core components NameNode and DataNode NameNode DataNode There is only one NameNode
- 54. What is HDFS? VOLUME STORING Hadoop Distributed File System (HDFS) has two core components NameNode and DataNode NameNode DataNode There is only one NameNode DataNode DataNode There can be multiple DataNodes
- 55. What is HDFS? VOLUME STORING Master/slave nodes typically form the HDFS cluster
- 56. What is HDFS? VOLUME STORING Master/slave nodes typically form the HDFS cluster Master/NameNode Slave/DataNode Slave/DataNode Slave/DataNode
- 57. What is HDFS? VOLUME STORING Master/slave nodes typically form the HDFS cluster Master/NameNode Slave/DataNode Slave/DataNode Slave/DataNode NameNode maintains and manages the DataNode. It also stores the metadata
- 58. What is HDFS? VOLUME STORING Master/slave nodes typically form the HDFS cluster Master/NameNode Slave/DataNode Slave/DataNode Slave/DataNode NameNode maintains and manages the DataNode. It also stores the metadata DataNodes stores the actual data, does reading, writing and processing. Performs replication as well
- 59. What is HDFS? VOLUME STORING Master/slave nodes typically form the HDFS cluster Master/NameNode Slave/DataNode Slave/DataNode Slave/DataNode NameNode maintains and manages the DataNode. It also stores the metadata DataNodes stores the actual data, does reading, writing and processing. Performs replication as well
- 60. What is HDFS? VOLUME STORING Master/slave nodes typically form the HDFS cluster Master/NameNode Slave/DataNode Slave/DataNode Slave/DataNode NameNode maintains and manages the DataNode. It also stores the metadata DataNodes stores the actual data, does reading, writing and processing. Performs replication as well HeartBeat is the signal that DataNode continuously sends to the NameNode. This signal shows the status of the DataNode
- 61. What is HDFS? VOLUME STORING In HDFS, data is stored in a distributed manner 30 TB file
- 62. What is HDFS? VOLUME STORING In HDFS, data is stored in a distributed manner 30 TB file NameNode 30 TB of data is loaded
- 63. What is HDFS? VOLUME STORING In HDFS, data is stored in a distributed manner 30 TB file NameNode 30 TB of data is loaded . . . Data is divided into blocks of 128 MB each
- 64. What is HDFS? VOLUME STORING In HDFS, data is stored in a distributed manner 30 TB file NameNode 30 TB of data is loaded DataNodes . . . Data is divided into blocks of 128 MB each . . . . .
- 65. What is HDFS? VOLUME STORING In HDFS, data is stored in a distributed manner 30 TB file NameNode 30 TB of data is loaded DataNodes . . . Data is divided into blocks of 128 MB each Blocks are then replicated among the DataNodes . . . . .
- 66. What is HDFS? Provides distributed storage Features of HDFS
- 67. What is HDFS? Provides distributed storage Implemented on commodity hardware Features of HDFS
- 68. What is HDFS? Provides distributed storage Implemented on commodity hardware Provides data security Features of HDFS
- 69. What is HDFS? Provides distributed storage Implemented on commodity hardware Provides data security Highly fault tolerant Features of HDFS
- 70. Components of Hadoop Storage unit of Hadoop Processing unit of Hadoop
- 72. What is MapReduce? VOLUME STORING Hadoop MapReduce is a programming technique where huge data is processed in a parallel and distributed fashion
- 73. What is MapReduce? VOLUME STORING Hadoop MapReduce is a programming technique where huge data is processed in a parallel and distributed fashion Big Data
- 74. What is MapReduce? VOLUME STORING Hadoop MapReduce is a programming technique where huge data is processed in a parallel and distributed fashion Big Data Processor
- 75. What is MapReduce? VOLUME STORING Hadoop MapReduce is a programming technique where huge data is processed in a parallel and distributed fashion Big Data Processor MapReduce is used for parallel processing of the Big Data, which is stored in HDFS
- 76. What is MapReduce? VOLUME STORING Hadoop MapReduce is a programming technique where huge data is processed in a parallel and distributed fashion Big Data Output Processor MapReduce is used for parallel processing of the Big Data, which is stored in HDFS
- 77. What is MapReduce? VOLUME STORING In MapReduce approach, processing is done at the slave nodes and the final result is sent to the master node
- 78. What is MapReduce? VOLUME STORING In MapReduce approach, processing is done at the slave nodes and the final result is sent to the master node Master Slave Slave Slave Slave Traditional approach – Data is processed at the Master node
- 79. What is MapReduce? VOLUME STORING In MapReduce approach, processing is done at the slave nodes and the final result is sent to the master node Master Slave Slave Slave Slave Traditional approach – Data is processed at the Master node MapReduce approach – Data is processed at the Slave nodes Slave Slave Slave Slave Master
- 80. What is MapReduce? Input Bus Car Train Ship Ship Train Bus Ship Car
- 81. What is MapReduce? Input Split Ship Ship Train Bus Car Train Bus Car Train Ship Ship Train Bus Ship Car Bus Ship Car The input dataset is first split into chunks of data
- 82. What is MapReduce? Input Split Map phase Ship Ship Train Bus Car Train Bus Car Train Ship Ship Train Bus Ship Car Bus Ship Car Ship, 1 Ship, 1 Train, 1 Bus, 1 Ship, 1 Car, 1 Bus, 1 Car, 1 Train, 1 These chunks of data are then processed by map tasks parallelly
- 83. What is MapReduce? Input Split Map phase Reduce phase Ship Ship Train Bus Car Train Bus Car Train Ship Ship Train Bus Ship Car Bus Ship Car Ship, 1 Ship, 1 Train, 1 Bus, 1 Ship, 1 Car, 1 Bus, 1 Car, 1 Train, 1 Ship, 1 Ship, 1 Ship, 1 Bus, 1 Bus, 1 Car, 1 Car, 1 Train, 1 Train, 1
- 84. What is MapReduce? Input Split Map phase Shuffle and sortReduce phase Ship Ship Train Bus Car Train Bus Car Train Ship Ship Train Bus Ship Car Bus Ship Car Ship, 1 Ship, 1 Train, 1 Bus, 1 Ship, 1 Car, 1 Bus, 1 Car, 1 Train, 1 Ship, 1 Ship, 1 Ship, 1 Bus, 2 Car, 2 Ship, 3 Train, 2 Bus, 1 Bus, 1 Car, 1 Car, 1 Train, 1 Train, 1 At the reduce task, the aggregation takes place and the final output is obtained
- 85. Components of Hadoop version 2.0 Storage unit of Hadoop Processing unit of Hadoop Resource management unit of Hadoop
- 86. What is YARN?
- 87. YARN – Yet Another Resource Negotiator Acts like an OS to Hadoop 2 Does job scheduling Responsible for managing cluster resources What is YARN?
- 89. What is YARN? Client Client Client Client submits the job request
- 90. What is YARN? Resource Manager Client Client Client Client submits the job request
- 91. What is YARN? Resource Manager Responsible for resource allocation and management Client Client Client Client submits the job request
- 92. What is YARN? Resource Manager Responsible for resource allocation and management Node Manager Client Client Client Node Manager Node Manager Client submits the job request
- 93. What is YARN? Node Manager manages the nodes and monitors resource usage Resource Manager Responsible for resource allocation and management Node Manager Client Client Client Node Manager Node Manager Client submits the job request
- 94. What is YARN? Container is a collection of physical resources such as RAM, CPU Node Manager manages the nodes and monitors resource usage Resource Manager Responsible for resource allocation and management container Node Manager Client Client Client Node Manager Node Manager container container container Client submits the job request
- 95. What is YARN? Container is a collection of physical resources such as RAM, CPU Node Manager manages the nodes and monitors resource usage Resource Manager Responsible for resource allocation and management App Master container Node Manager Client Client Client Node Manager Node Manager App Master container container container App Master requests container from the NodeManager Client submits the job request
- 96. What is YARN? Container is a collection of physical resources such as RAM, CPU Node Manager manages the nodes and monitors resource usage Resource Manager Responsible for resource allocation and management App Master container Node Manager Client Client Client Node Manager Node Manager App Master container container container App Master requests container from the NodeManager Client submits the job request
- 97. Hadoop use case – Combating fraudulent activities
- 98. Hadoop use case – Combating fraudulent activities Fraud activities Detecting fraudulent transactions is one among the various problems any bank faces
- 99. Zions’ main challenge was to combat the fraudulent activities which were taking place Challenge Hadoop use case – Combating fraudulent activities
- 100. Approaches used by Zions’ security team to combat fraudulent activities Hadoop use case – Combating fraudulent activities
- 101. Approaches used by Zions’ security team to combat fraudulent activities Security information management – SIM Tools Problem It was based on RDBMS Unable to store huge data which needed to be analyzed Hadoop use case – Combating fraudulent activities
- 102. Approaches used by Zions’ security team to combat fraudulent activities Security information management – SIM Tools Problem It was based on RDBMS Unable to store huge data which needed to be analyzed Hadoop use case – Combating fraudulent activities Parallel processing system Problem Analyzing unstructured data was not possible
- 103. Approaches used by Zions’ security team to combat fraudulent activities Security information management – SIM Tools Problem It was based on RDBMS Unable to store huge data which needed to be analyzed Hadoop use case – Combating fraudulent activities Parallel processing system Problem Analyzing unstructured data was not possible
- 104. How Hadoop solved the problems Hadoop use case – Combating fraudulent activities Storing Zions could now store massive amount of data using Hadoop
- 105. How Hadoop solved the problems Hadoop use case – Combating fraudulent activities Storing Zions could now store massive amount of data using Hadoop Processing Processing of unstructured data (like server logs, customer data, customer transactions) was now possible
- 106. How Hadoop solved the problems Hadoop use case – Combating fraudulent activities Storing Zions could now store massive amount of data using Hadoop Processing Analyzing In-depth analysis of different data formats became easy and time efficient Processing of unstructured data (like server logs, customer data, customer transactions) was now possible
- 107. How Hadoop solved the problems Hadoop use case – Combating fraudulent activities Storing Zions could now store massive amount of data using Hadoop Processing Analyzing Detecting In-depth analysis of different data formats became easy and time efficient The team could now detect everything from malware, spear phishing attempts to account takeovers Processing of unstructured data (like server logs, customer data, customer transactions) was now possible
- 108. Key Takeaways
Editor's Notes
- #16: Style - 01
- #17: Style - 01
- #18: Style - 01
- #19: Style - 01
- #20: Style - 01
- #21: Style - 01
- #22: Style - 01
- #23: Style - 01
- #24: Style - 01
- #25: Style - 01
- #27: Style - 01
- #28: Style - 01
- #29: Style - 01
- #30: Style - 01
- #31: Style - 01
- #33: Style - 01
- #34: Style - 01
- #35: Style - 01
- #36: Style - 01
- #37: Style - 01
- #38: Style - 01
- #39: Style - 01
- #40: Style - 01
- #41: Style - 01
- #42: Style - 01
- #43: Style - 01
- #44: Style - 01
- #45: Style - 01