Hadoop - Simple. Scalable.
1 like1,032 views
This document provides an overview of Hadoop, including: 1) Hadoop solves the problems of analyzing massively large datasets by distributing data storage and analysis across multiple machines to tolerate node failure. 2) Hadoop uses HDFS for distributed data storage, which shards massive files across data nodes with replication for fault tolerance, and MapReduce for distributed data analysis by sending code to the data. 3) The document demonstrates MapReduce concepts like map, reduce, and their composition with an example job.
Hadoop - Simple. Scalable.
- 3. Java. Clojure. Ruby. Cloudera Certified
- 4. posscon.org April 15, 16, and 17
- 5. Agenda Overview Massively Large Data Sets and the problems therein Distributed File System MapReduce Pig
- 6. Overview
- 7. Doug Cutting Genius
- 8. Favorite Hadoop Story New York Times
- 9. 4 Terabytes of Source Articles.
- 10. 24 Hours.
- 11. 5.5 Terabytes of PDFs.
- 12. Did it again.
- 13. $240.
- 14. Infoporn from Yahoo 73 hours 490 TB Shuffling 280 TB Output 4000 Nodes 16 PB Disk Space 32K Cores 64 TB RAM
- 15. Hadoop solves...
- 16. Analyzing Massively Large Datasets
- 17. Two Problems You have to distribute.
- 18. Data Storage Capacity has increased rapidly beyond read speeds. Datasets won't fit on one disk. Tolerate node failure.
- 19. Data Analysis Combine data from many machines. Tolerate node failure.
- 20. How Hadoop solves these problems.
- 21. Send Code to Data. Not Data to Code.
- 22. Data Storage HDFS
- 23. Name Node. Data Nodes. Master - Slave Relationship
- 24. Shard massive files across multiple machines. MB, GB, and TB
- 25. Tolerant of Node Failure Files replicated across at least 3 nodes.
- 26. HDFS behaves like a normal file system. No true appends yet.
- 27. Demonstration.
- 28. Data Analysis MapReduce
- 29. Job Tracker. Task Nodes. Master - Slave Relationship.
- 30. map
- 31. Demonstration
- 32. pmap
- 33. Demonstration
- 34. reduce
- 35. Demonstration
- 36. (reduce (pmap))
- 37. Demonstration.
- 38. MapReduce Java
- 39. Nobody likes it. :-)
- 40. MapReduce Ruby. Python. Unix Utilities.
- 42. Hadoop Ecosystem Pigkeeper. Hive. Cascading.
- 43. Pig
- 44. HBase