simple introduction to hadoop
0 likes1,857 views
The document discusses data storage and processing. Data can be stored in memory or on disk using file systems like local XFS/ZFS or distributed systems like HDFS, S3, or Ceph. Distributed file systems allow for parallel processing of data by moving computation to the data locations. This map-reduce framework involves mapping functions to distributed data segments followed by reducing the results. Hadoop uses HDFS for storage and the MapReduce framework for distributed computation on large datasets across clusters.
simple introduction to hadoop
- 3. Data Processing 1. Where is data stored ?
- 4. Data Processing 1. Where is data stored ? 2. Where does the compute run ?
- 5. Storage
- 7. Storage ● In Memory ● On Disk
- 8. Storage ● In Memory ● On Disk ○ File System
- 9. Storage ● In Memory ● On Disk ○ File System ■ Local file system - xfs , zfs , etc
- 10. Storage ● In Memory ● On Disk ○ File System ■ Local file system - xfs , zfs ■ Distributed File System
- 11. Storage ● In Memory ● On Disk ○ File System ■ Local file system - xfs , zfs ■ Distributed File System ● HDFS - Hadoop Distributed File System ● S3 ● Ceph etc
- 12. H-Distributed-FS
- 15. H-Distributed-FS Motivation: ● Parallel Processing of Data ○ When data is distributed, it can be processed in parallel * * some problem statements are not a fit for this.
- 16. H-Distributed-FS Motivation: ● Parallel Processing of Data ○ When data is distributed, it can be processed in parallel * ● Computation goes to data and not data to computation * some problem statements are not a fit for this.
- 17. Word Count Problem ● Single file
- 18. Word Count Problem ● Single file ○ O(n) time complexity
- 19. Word Count Problem ● Distributed data
- 20. Word Count Problem ● Distributed data O(m) time complexity ‘m’ = size of largest file
- 21. Parallel Compute
- 22. Parallel Compute ● Parallel Computation on the data
- 23. Parallel Compute ● Parallel Computation on the data ● Computation goes to data & not data to computation. Compute works over the data local to it.
- 24. Parallel Compute ● Parallel Computation on the data ● Computation goes to data & not data to computation. Compute works over the data local to it. ● A final aggregation happens.
- 25. Parallel Compute ● Parallel Computation on the data ● Computation goes to data & not data to computation. Compute works over the data local to it. ● A final aggregation happens. This Compute Paradigm is called “MAP REDUCE framework”
- 27. Hdfs + MapReduce
- 28. Hdfs + MapReduce = Hadoop
- 29. Some Notes ● Hadoop Version 1 - has Map Reduce (MR) framework ● Hadoop Version 2 - has Yarn (resource scheduler) ● Apache Spark ○ Alternative to Map Reduce Compute framework ○ Can use Apache Spark with data in HDFS ○ Can run in Yarn with data in s3 ! (we just use yarn features without using MR and HDFS features) :)
- 30. Hdfs + MapReduce = Hadoop By Vishnu Rao mash213.wordpress.com linkedin.com/in/213vishnu