SlideShare a Scribd company logo

Hadoop: Distributed data processing

R
royans

The document summarizes a presentation given by Amr Awadallah of Cloudera on Hadoop. It discusses how current storage systems are unable to perform computation, and how Hadoop addresses this through its marriage of HDFS for scalable storage and MapReduce for distributed processing. It provides an overview of Hadoop's history and design principles such as managing itself, scaling performance linearly, and moving computation to data.

Technology
1 of 63
Downloaded 390 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Hadoop: Distributed Data Processing Amr Awadallah Founder/CTO, Cloudera, Inc. ACM Data Mining SIG Thursday, January 25th, 2010 Wednesday, January 27, 2010
Outline ▪Scaling for Large Data Processing ▪What is Hadoop? ▪HDFS and MapReduce ▪Hadoop Ecosystem ▪Hadoop vs RDBMSes ▪Conclusion Amr Awadallah, Cloudera Inc 2 Wednesday, January 27, 2010
Current Storage Systems Can’t Compute Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
Current Storage Systems Can’t Compute Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
Current Storage Systems Can’t Compute Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
Current Storage Systems Can’t Compute Interactive Apps RDBMS (200GB/day) ETL Grid Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
Current Storage Systems Can’t Compute Interactive Apps RDBMS (200GB/day) ETL Grid Filer heads are a bottleneck Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
Current Storage Systems Can’t Compute Interactive Apps Ad hoc Queries & Data Mining RDBMS (200GB/day) ETL Grid Non-Consumption Filer heads are a bottleneck Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
The Solution: A Store-Compute Grid Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
The Solution: A Store-Compute Grid Storage + Computation Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
The Solution: A Store-Compute Grid Interactive Apps RDBMS ETL and Aggregations Storage + Computation Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
The Solution: A Store-Compute Grid Interactive Apps “Batch” Apps RDBMS Ad hoc Queries ETL and & Data Mining Aggregations Storage + Computation Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
What is Hadoop? Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data ▪ A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …) Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data ▪ A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …) ▪ Open source under the friendly Apache License Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data ▪ A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …) ▪ Open source under the friendly Apache License ▪ http://wiki.apache.org/hadoop/ Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
Hadoop History Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes ▪ May 2009: ▪ Yahoo does fastest sort of a TB, 62secs over 1460 nodes ▪ Yahoo sorts a PB in 16.25hours over 3658 nodes Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes ▪ May 2009: ▪ Yahoo does fastest sort of a TB, 62secs over 1460 nodes ▪ Yahoo sorts a PB in 16.25hours over 3658 nodes ▪ June 2009, Oct 2009: Hadoop Summit (750), Hadoop World (500) Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes ▪ May 2009: ▪ Yahoo does fastest sort of a TB, 62secs over 1460 nodes ▪ Yahoo sorts a PB in 16.25hours over 3658 nodes ▪ June 2009, Oct 2009: Hadoop Summit (750), Hadoop World (500) Amr Awadallah, Cloudera Inc 6 ▪ September 2009: Doug Cutting joins Cloudera Wednesday, January 27, 2010
Hadoop Design Axioms Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
Hadoop Design Axioms 1. System Shall Manage and Heal Itself Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
Hadoop Design Axioms 1. System Shall Manage and Heal Itself 2. Performance Shall Scale Linearly Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
Hadoop Design Axioms 1. System Shall Manage and Heal Itself 2. Performance Shall Scale Linearly 3. Compute Should Move to Data Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
Hadoop Design Axioms 1. System Shall Manage and Heal Itself 2. Performance Shall Scale Linearly 3. Compute Should Move to Data 4. Simple Core, Modular and Extensible Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
HDFS: Hadoop Distributed File System Block Size = 64MB Replication Factor = 3 Cost/GB is a few ¢/month vs $/month Amr Awadallah, Cloudera Inc 8 Wednesday, January 27, 2010
HDFS: Hadoop Distributed File System Block Size = 64MB Replication Factor = 3 Cost/GB is a few ¢/month vs $/month Amr Awadallah, Cloudera Inc 8 Wednesday, January 27, 2010
MapReduce: Distributed Processing Amr Awadallah, Cloudera Inc 9 Wednesday, January 27, 2010
MapReduce: Distributed Processing Amr Awadallah, Cloudera Inc 9 Wednesday, January 27, 2010
MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt Split 1 Split i Split N Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt (words, counts) Split 1 (docid, text) Map 1 Be, 5 “To Be Or Not To Be?” Be, 12 Split i (docid, text) Map i Be, 7 Be, 6 Split N (docid, text) Map M (words, counts) Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt (words, counts) Split 1 (docid, text) Map 1 (sorted words, counts) Be, 5 Reduce 1 “To Be Or Not To Be?” Be, 12 Reduce i Split i (docid, text) Map i Be, 7 Be, 6 Shuffle Reduce R Split N (docid, text) Map M (words, counts) (sorted words, counts) Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt (words, counts) Split 1 (docid, text) Map 1 (sorted words, counts) Output File Be, 5 Reduce 1 (sorted words, sum of counts) 1 “To Be Or Not Be, 30 To Be?” Be, 12 Output File i (sorted words, Reduce i sum of counts) Split i (docid, text) Map i Be, 7 Be, 6 Shuffle Output File (sorted words, R Reduce R sum of counts) Split N (docid, text) Map M (words, counts) (sorted words, counts) Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
Hadoop High-Level Architecture Hadoop Client Contacts Name Node for data or Job Tracker to submit jobs Name Node Job Tracker Maintains mapping of file blocks Schedules jobs across to data node slaves task tracker slaves Data Node Task Tracker Stores and serves Runs tasks (work units) blocks of data within a job Share Physical Node Amr Awadallah, Cloudera Inc 11 Wednesday, January 27, 2010
Apache Hadoop Ecosystem MapReduce (Job Scheduling/Execution System) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
Apache Hadoop Ecosystem Zookeepr (Coordination) Avro (Serialization) MapReduce (Job Scheduling/Execution System) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
Apache Hadoop Ecosystem Zookeepr (Coordination) Avro (Serialization) MapReduce (Job Scheduling/Execution System) HBase (key-value store) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
Apache Hadoop Ecosystem ETL Tools BI Reporting RDBMS Pig (Data Flow) Hive (SQL) Sqoop Zookeepr (Coordination) Avro (Serialization) MapReduce (Job Scheduling/Execution System) HBase (key-value store) (Streaming/Pipes APIs) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
Use The Right Tool For The Right Job Hadoop: Relational Databases: Amr Awadallah, Cloudera Inc 13 Wednesday, January 27, 2010
Use The Right Tool For The Right Job Hadoop: Relational Databases: Amr Awadallah, Cloudera Inc 13 Wednesday, January 27, 2010
Use The Right Tool For The Right Job Hadoop: Relational Databases: When to use? When to use? • Affordable Storage/ • Interactive Reporting Compute (<1sec) • Structured or Not (Agility) • Multistep Transactions • Resilient Auto Scalability • Interoperability Amr Awadallah, Cloudera Inc 13 Wednesday, January 27, 2010
Economics of Hadoop Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node ▪ Effective HDFS Space: ▪ ¼ reserved for temp shuffle space, which leaves 9TB/node ▪ 3 way replication leads to 3TB effective HDFS space/node ▪ But assuming 7x compression that becomes ~ 20TB/node Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node ▪ Effective HDFS Space: ▪ ¼ reserved for temp shuffle space, which leaves 9TB/node ▪ 3 way replication leads to 3TB effective HDFS space/node ▪ But assuming 7x compression that becomes ~ 20TB/node Effective Cost per user TB: $250/TB Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node ▪ Effective HDFS Space: ▪ ¼ reserved for temp shuffle space, which leaves 9TB/node ▪ 3 way replication leads to 3TB effective HDFS space/node ▪ But assuming 7x compression that becomes ~ 20TB/node Effective Cost per user TB: $250/TB Other solutions cost in the range of $5K to $100K per user TB Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
Sample Talks from Hadoop World ‘09 ▪ VISA: Large Scale Transaction Analysis ▪ JP Morgan Chase: Data Processing for Financial Services ▪ China Mobile: Data Mining Platform for Telecom Industry ▪ Rackspace: Cross Data Center Log Processing ▪ Booz Allen Hamilton: Protein Alignment using Hadoop ▪ eHarmony: Matchmaking in the Hadoop Cloud ▪ General Sentiment: Understanding Natural Language ▪ Yahoo!: Social Graph Analysis ▪ Visible Technologies: Real-Time Business Intelligence ▪ Facebook: Rethinking the Data Warehouse with Hadoop and Hive Slides and Videos at http://www.cloudera.com/hadoop- Amr Awadallah, Cloudera Inc world-nyc 15 Wednesday, January 27, 2010
Cloudera Desktop Amr Awadallah, Cloudera Inc 16 Wednesday, January 27, 2010
Conclusion Amr Awadallah, Cloudera Inc 17 Wednesday, January 27, 2010
Conclusion Hadoop is a data grid operating system which provides an economically scalable solution for storing and processing large amounts of unstructured or structured data over long periods of time. Amr Awadallah, Cloudera Inc 17 Wednesday, January 27, 2010
Contact Information Amr Awadallah CTO, Cloudera Inc. aaa@cloudera.com http://twitter.com/awadallah Online Training Videos and Info: http://cloudera.com/hadoop- training http://cloudera.com/blog http://twitter.com/cloudera Amr Awadallah, Cloudera Inc 18 Wednesday, January 27, 2010
(c) 2008 Cloudera, Inc. or its licensors.  "Cloudera" is a registered trademark of Cloudera, Inc.. All rights reserved. 1.0 Wednesday, January 27, 2010

More Related Content

PPTX
Hadoop: Distributed Data Processing
Cloudera, Inc.
 
PPSX
Hadoop
Nishant Gandhi
 
PPTX
Hadoop technology
tipanagiriharika
 
PPTX
Introduction to Big Data & Hadoop Architecture - Module 1
Rohit Agrawal
 
DOCX
Hadoop Seminar Report
Atul Kushwaha
 
PPTX
Big data Hadoop presentation
Shivanee garg
 
PPTX
Introduction to Apache Hadoop
Christopher Pezza
 
PPTX
HADOOP TECHNOLOGY ppt
sravya raju
 
Hadoop: Distributed Data Processing
Cloudera, Inc.
 
Hadoop
Nishant Gandhi
 
Hadoop technology
tipanagiriharika
 
Introduction to Big Data & Hadoop Architecture - Module 1
Rohit Agrawal
 
Hadoop Seminar Report
Atul Kushwaha
 
Big data Hadoop presentation
Shivanee garg
 
Introduction to Apache Hadoop
Christopher Pezza
 
HADOOP TECHNOLOGY ppt
sravya raju
 

What's hot (20)

ODP
Hadoop seminar
KrishnenduKrishh
 
PPTX
Introduction to Hadoop Technology
Manish Borkar
 
PPT
Seminar Presentation Hadoop
Varun Narang
 
PPTX
Hadoop for beginners free course ppt
Njain85
 
PPTX
Hadoop and Big Data
Harshdeep Kaur
 
PDF
Hadoop Ecosystem Architecture Overview
Senthil Kumar
 
PPT
Hadoop Technology
Atul Kushwaha
 
PPTX
Hadoop and big data
Sharad Pandey
 
PPTX
Hadoop
Kartik Kalpande Patil
 
PPT
Presentation on Hadoop Technology
OpenDev
 
PDF
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
royans
 
PPTX
Big data and Hadoop
Rahul Agarwal
 
PPTX
Hadoop Presentation - PPT
Anand Pandey
 
PPTX
Apache Hadoop
Ajit Koti
 
PPTX
Big Data and Hadoop
Flavio Vit
 
PDF
Seminar_Report_hadoop
Varun Narang
 
PPTX
PPT on Hadoop
Shubham Parmar
 
PPTX
Hadoop Tutorial For Beginners
Dataflair Web Services Pvt Ltd
 
PPTX
Hadoop
Tuan Cuong Luu
 
PPT
Hadoop Technologies
Kannappan Sirchabesan
 
Hadoop seminar
KrishnenduKrishh
 
Introduction to Hadoop Technology
Manish Borkar
 
Seminar Presentation Hadoop
Varun Narang
 
Hadoop for beginners free course ppt
Njain85
 
Hadoop and Big Data
Harshdeep Kaur
 
Hadoop Ecosystem Architecture Overview
Senthil Kumar
 
Hadoop Technology
Atul Kushwaha
 
Hadoop and big data
Sharad Pandey
 
Hadoop
Kartik Kalpande Patil
 
Presentation on Hadoop Technology
OpenDev
 
Facebooks Petabyte Scale Data Warehouse using Hive and Hadoop
royans
 
Big data and Hadoop
Rahul Agarwal
 
Hadoop Presentation - PPT
Anand Pandey
 
Apache Hadoop
Ajit Koti
 
Big Data and Hadoop
Flavio Vit
 
Seminar_Report_hadoop
Varun Narang
 
PPT on Hadoop
Shubham Parmar
 
Hadoop Tutorial For Beginners
Dataflair Web Services Pvt Ltd
 
Hadoop
Tuan Cuong Luu
 
Hadoop Technologies
Kannappan Sirchabesan
 
Ad

Viewers also liked (9)

PPT
Distributed Processing
Imtiaz Hussain
 
PDF
Distributed processing
Neil Stein
 
PDF
Distributed Data Processing Workshop - SBU
Amir Sedighi
 
PPT
Distributed computing
Alokeparna Choudhury
 
PPT
Distributed Database Management System
Hardik Patil
 
PPT
Data Processing
Jhessie Abella RN,RM,MAN,CPSO
 
PPTX
Distributed database
ReachLocal Services India
 
PPT
Distributed Database System
Sulemang
 
PDF
Introduction to Distributed Computing Engines for Data Processing - Simone Ro...
Data Science Milan
 
Distributed Processing
Imtiaz Hussain
 
Distributed processing
Neil Stein
 
Distributed Data Processing Workshop - SBU
Amir Sedighi
 
Distributed computing
Alokeparna Choudhury
 
Distributed Database Management System
Hardik Patil
 
Data Processing
Jhessie Abella RN,RM,MAN,CPSO
 
Distributed database
ReachLocal Services India
 
Distributed Database System
Sulemang
 
Introduction to Distributed Computing Engines for Data Processing - Simone Ro...
Data Science Milan
 
Ad

Similar to Hadoop: Distributed data processing (20)

PPTX
Integrating hadoop - Big Data TechCon 2013
Jonathan Seidman
 
PPT
The Time Has Come for Big-Data-as-a-Service
BlueData, Inc.
 
ODP
The power of hadoop in cloud computing
Joey Echeverria
 
PDF
Semantic web meetup 14.november 2013
Jean-Pierre König
 
PPTX
Hadoop in a Nutshell
Anthony Thomas
 
PPTX
Hadoop World 2010: Productionizing Hadoop: Lessons Learned
Cloudera, Inc.
 
PPTX
Hadoop
Oded Rotter
 
PDF
P.Maharajothi,II-M.sc(computer science),Bon secours college for women,thanjavur.
MaharajothiP
 
PDF
Hadoop As The Platform For The Smartgrid At TVA
Cloudera, Inc.
 
PPTX
Hadoop Platforms - Introduction, Importance, Providers
Mrigendra Sharma
 
PDF
Big data and hadoop overvew
Kunal Khanna
 
PDF
How Apache Hadoop is Revolutionizing Business Intelligence and Data Analytics...
Amr Awadallah
 
PDF
Business Intelligence and Data Analytics Revolutionized with Apache Hadoop
Cloudera, Inc.
 
PPTX
Tera data
Naga Dinesh
 
PPTX
Hadoop Essentials -- The What, Why and How to Meet Agency Objectives
Cloudera, Inc.
 
PDF
Hadoop World 2011: How Hadoop Revolutionized Business Intelligence and Advanc...
Cloudera, Inc.
 
PPTX
Tera data
Naga Dinesh
 
PDF
Innovation in the Data Warehouse - StampedeCon 2016
StampedeCon
 
PPTX
Hadoop
ronit gaikwad
 
PPT
Data Science Day New York: The Platform for Big Data
Cloudera, Inc.
 
Integrating hadoop - Big Data TechCon 2013
Jonathan Seidman
 
The Time Has Come for Big-Data-as-a-Service
BlueData, Inc.
 
The power of hadoop in cloud computing
Joey Echeverria
 
Semantic web meetup 14.november 2013
Jean-Pierre König
 
Hadoop in a Nutshell
Anthony Thomas
 
Hadoop World 2010: Productionizing Hadoop: Lessons Learned
Cloudera, Inc.
 
Hadoop
Oded Rotter
 
P.Maharajothi,II-M.sc(computer science),Bon secours college for women,thanjavur.
MaharajothiP
 
Hadoop As The Platform For The Smartgrid At TVA
Cloudera, Inc.
 
Hadoop Platforms - Introduction, Importance, Providers
Mrigendra Sharma
 
Big data and hadoop overvew
Kunal Khanna
 
How Apache Hadoop is Revolutionizing Business Intelligence and Data Analytics...
Amr Awadallah
 
Business Intelligence and Data Analytics Revolutionized with Apache Hadoop
Cloudera, Inc.
 
Tera data
Naga Dinesh
 
Hadoop Essentials -- The What, Why and How to Meet Agency Objectives
Cloudera, Inc.
 
Hadoop World 2011: How Hadoop Revolutionized Business Intelligence and Advanc...
Cloudera, Inc.
 
Tera data
Naga Dinesh
 
Innovation in the Data Warehouse - StampedeCon 2016
StampedeCon
 
Hadoop
ronit gaikwad
 
Data Science Day New York: The Platform for Big Data
Cloudera, Inc.
 

More from royans (16)

PPS
Web20expo Filesystems
royans
 
PPS
Flickr Services
royans
 
PPS
Web20expo Scalable Web Arch
royans
 
PPS
Flickr Php
royans
 
PPT
Grid – Distributed Computing at Scale
royans
 
PPT
How Typepad changed their architecture without taking down the service
royans
 
PPT
Dmk Bo2 K7 Web
royans
 
PPT
21 Www Web Services
royans
 
PPS
Web20expo Filesystems
royans
 
PPS
Web Design World Flickr
royans
 
PPS
Flickr Services
royans
 
PPS
Filesystems
royans
 
PPS
Scalable Web Arch
royans
 
PPS
Web 2.0 Summit Flickr
royans
 
PPS
Web20expo Filesystems
royans
 
PPS
Etech2005
royans
 
Web20expo Filesystems
royans
 
Flickr Services
royans
 
Web20expo Scalable Web Arch
royans
 
Flickr Php
royans
 
Grid – Distributed Computing at Scale
royans
 
How Typepad changed their architecture without taking down the service
royans
 
Dmk Bo2 K7 Web
royans
 
21 Www Web Services
royans
 
Web20expo Filesystems
royans
 
Web Design World Flickr
royans
 
Flickr Services
royans
 
Filesystems
royans
 
Scalable Web Arch
royans
 
Web 2.0 Summit Flickr
royans
 
Web20expo Filesystems
royans
 
Etech2005
royans
 

Recently uploaded (20)

PPTX
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PPTX
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
PPTX
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
PDF
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
PDF
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
PPTX
Cloud computing and distributed systems.
kkkkkhan
 
PDF
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
PDF
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
PDF
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
PDF
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
PDF
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
PDF
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
PDF
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
PDF
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
PPT
Teaching material agriculture food technology
LiaRayya
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
PPTX
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
PDF
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 
Programs and apps: productivity, graphics, security and other tools
4mqw9zch22
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
Machine Learning_overview_presentation.pptx
kondavamsidharreddy2
 
20250228 LYD VKU AI Blended-Learning.pptx
DatVoNgoc
 
TokAI - TikTok AI Agent : The First AI Application That Analyzes 10,000+ Vira...
SOFTTECHHUB
 
Per capita expenditure prediction using model stacking based on satellite ima...
IAESIJAI
 
Cloud computing and distributed systems.
kkkkkhan
 
Encapsulation_ Review paper, used for researhc scholars
gurumoop
 
The Rise and Fall of 3GPP – Time for a Sabbatical?
3G4G
 
Advanced methodologies resolving dimensionality complications for autism neur...
IAESIJAI
 
cuic standard and advanced reporting.pdf
SimoneTitaniumTomase
 
Empathic Computing: Creating Shared Understanding
Mark Billinghurst
 
Dropbox Q2 2025 Financial Results & Investor Presentation
Dropbox
 
Architecting across the Boundaries of two Complex Domains - Healthcare & Tech...
Peter Tan
 
Optimiser vos workloads AI/ML sur Amazon EC2 et AWS Graviton
Julien SIMON
 
Teaching material agriculture food technology
LiaRayya
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
gpt5_lecture_notes_comprehensive_20250812015547.pdf
Chinchu40
 
VMware vSphere Foundation How to Sell Presentation-Ver1.4-2-14-2024.pptx
blackmambaettijean
 
7 ChatGPT Prompts to Help You Define Your Ideal Customer Profile.pdf
SOFTTECHHUB
 

Hadoop: Distributed data processing

  • 1. Hadoop: Distributed Data Processing Amr Awadallah Founder/CTO, Cloudera, Inc. ACM Data Mining SIG Thursday, January 25th, 2010 Wednesday, January 27, 2010
  • 2. Outline ▪Scaling for Large Data Processing ▪What is Hadoop? ▪HDFS and MapReduce ▪Hadoop Ecosystem ▪Hadoop vs RDBMSes ▪Conclusion Amr Awadallah, Cloudera Inc 2 Wednesday, January 27, 2010
  • 3. Current Storage Systems Can’t Compute Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
  • 4. Current Storage Systems Can’t Compute Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
  • 5. Current Storage Systems Can’t Compute Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
  • 6. Current Storage Systems Can’t Compute Interactive Apps RDBMS (200GB/day) ETL Grid Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
  • 7. Current Storage Systems Can’t Compute Interactive Apps RDBMS (200GB/day) ETL Grid Filer heads are a bottleneck Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
  • 8. Current Storage Systems Can’t Compute Interactive Apps Ad hoc Queries & Data Mining RDBMS (200GB/day) ETL Grid Non-Consumption Filer heads are a bottleneck Storage Farm for Unstructured Data (20TB/day) Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 3 Wednesday, January 27, 2010
  • 9. The Solution: A Store-Compute Grid Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
  • 10. The Solution: A Store-Compute Grid Storage + Computation Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
  • 11. The Solution: A Store-Compute Grid Interactive Apps RDBMS ETL and Aggregations Storage + Computation Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
  • 12. The Solution: A Store-Compute Grid Interactive Apps “Batch” Apps RDBMS Ad hoc Queries ETL and & Data Mining Aggregations Storage + Computation Mostly Append Collection Instrumentation Amr Awadallah, Cloudera Inc 4 Wednesday, January 27, 2010
  • 13. What is Hadoop? Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
  • 14. What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
  • 15. What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
  • 16. What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
  • 17. What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data ▪ A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …) Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
  • 18. What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data ▪ A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …) ▪ Open source under the friendly Apache License Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
  • 19. What is Hadoop? ▪A scalable fault-tolerant grid operating system for data storage and processing ▪ Its scalability comes from the marriage of: ▪ HDFS: Self-Healing High-Bandwidth Clustered Storage ▪ MapReduce: Fault-Tolerant Distributed Processing ▪ Operates on unstructured and structured data ▪ A large and active ecosystem (many developers and additions like HBase, Hive, Pig, …) ▪ Open source under the friendly Apache License ▪ http://wiki.apache.org/hadoop/ Amr Awadallah, Cloudera Inc 5 Wednesday, January 27, 2010
  • 20. Hadoop History Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 21. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 22. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 23. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 24. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 25. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 26. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 27. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 28. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes ▪ May 2009: ▪ Yahoo does fastest sort of a TB, 62secs over 1460 nodes ▪ Yahoo sorts a PB in 16.25hours over 3658 nodes Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 29. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes ▪ May 2009: ▪ Yahoo does fastest sort of a TB, 62secs over 1460 nodes ▪ Yahoo sorts a PB in 16.25hours over 3658 nodes ▪ June 2009, Oct 2009: Hadoop Summit (750), Hadoop World (500) Amr Awadallah, Cloudera Inc 6 Wednesday, January 27, 2010
  • 30. Hadoop History ▪ 2002-2004: Doug Cutting and Mike Cafarella started working on Nutch ▪ 2003-2004: Google publishes GFS and MapReduce papers ▪ 2004: Cutting adds DFS & MapReduce support to Nutch ▪ 2006: Yahoo! hires Cutting, Hadoop spins out of Nutch ▪ 2007: NY Times converts 4TB of archives over 100 EC2s ▪ 2008: Web-scale deployments at Y!, Facebook, Last.fm ▪ April 2008: Yahoo does fastest sort of a TB, 3.5mins over 910 nodes ▪ May 2009: ▪ Yahoo does fastest sort of a TB, 62secs over 1460 nodes ▪ Yahoo sorts a PB in 16.25hours over 3658 nodes ▪ June 2009, Oct 2009: Hadoop Summit (750), Hadoop World (500) Amr Awadallah, Cloudera Inc 6 ▪ September 2009: Doug Cutting joins Cloudera Wednesday, January 27, 2010
  • 31. Hadoop Design Axioms Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
  • 32. Hadoop Design Axioms 1. System Shall Manage and Heal Itself Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
  • 33. Hadoop Design Axioms 1. System Shall Manage and Heal Itself 2. Performance Shall Scale Linearly Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
  • 34. Hadoop Design Axioms 1. System Shall Manage and Heal Itself 2. Performance Shall Scale Linearly 3. Compute Should Move to Data Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
  • 35. Hadoop Design Axioms 1. System Shall Manage and Heal Itself 2. Performance Shall Scale Linearly 3. Compute Should Move to Data 4. Simple Core, Modular and Extensible Amr Awadallah, Cloudera Inc 7 Wednesday, January 27, 2010
  • 36. HDFS: Hadoop Distributed File System Block Size = 64MB Replication Factor = 3 Cost/GB is a few ¢/month vs $/month Amr Awadallah, Cloudera Inc 8 Wednesday, January 27, 2010
  • 37. HDFS: Hadoop Distributed File System Block Size = 64MB Replication Factor = 3 Cost/GB is a few ¢/month vs $/month Amr Awadallah, Cloudera Inc 8 Wednesday, January 27, 2010
  • 38. MapReduce: Distributed Processing Amr Awadallah, Cloudera Inc 9 Wednesday, January 27, 2010
  • 39. MapReduce: Distributed Processing Amr Awadallah, Cloudera Inc 9 Wednesday, January 27, 2010
  • 40. MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt Split 1 Split i Split N Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
  • 41. MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt (words, counts) Split 1 (docid, text) Map 1 Be, 5 “To Be Or Not To Be?” Be, 12 Split i (docid, text) Map i Be, 7 Be, 6 Split N (docid, text) Map M (words, counts) Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
  • 42. MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt (words, counts) Split 1 (docid, text) Map 1 (sorted words, counts) Be, 5 Reduce 1 “To Be Or Not To Be?” Be, 12 Reduce i Split i (docid, text) Map i Be, 7 Be, 6 Shuffle Reduce R Split N (docid, text) Map M (words, counts) (sorted words, counts) Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
  • 43. MapReduce Example for Word Count SELECT word, COUNT(1) FROM docs GROUP BY word; cat *.txt | mapper.pl | sort | reducer.pl > out.txt (words, counts) Split 1 (docid, text) Map 1 (sorted words, counts) Output File Be, 5 Reduce 1 (sorted words, sum of counts) 1 “To Be Or Not Be, 30 To Be?” Be, 12 Output File i (sorted words, Reduce i sum of counts) Split i (docid, text) Map i Be, 7 Be, 6 Shuffle Output File (sorted words, R Reduce R sum of counts) Split N (docid, text) Map M (words, counts) (sorted words, counts) Amr Awadallah, Cloudera Inc 10 Wednesday, January 27, 2010
  • 44. Hadoop High-Level Architecture Hadoop Client Contacts Name Node for data or Job Tracker to submit jobs Name Node Job Tracker Maintains mapping of file blocks Schedules jobs across to data node slaves task tracker slaves Data Node Task Tracker Stores and serves Runs tasks (work units) blocks of data within a job Share Physical Node Amr Awadallah, Cloudera Inc 11 Wednesday, January 27, 2010
  • 45. Apache Hadoop Ecosystem MapReduce (Job Scheduling/Execution System) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
  • 46. Apache Hadoop Ecosystem Zookeepr (Coordination) Avro (Serialization) MapReduce (Job Scheduling/Execution System) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
  • 47. Apache Hadoop Ecosystem Zookeepr (Coordination) Avro (Serialization) MapReduce (Job Scheduling/Execution System) HBase (key-value store) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
  • 48. Apache Hadoop Ecosystem ETL Tools BI Reporting RDBMS Pig (Data Flow) Hive (SQL) Sqoop Zookeepr (Coordination) Avro (Serialization) MapReduce (Job Scheduling/Execution System) HBase (key-value store) (Streaming/Pipes APIs) HDFS (Hadoop Distributed File System) Amr Awadallah, Cloudera Inc 12 Wednesday, January 27, 2010
  • 49. Use The Right Tool For The Right Job Hadoop: Relational Databases: Amr Awadallah, Cloudera Inc 13 Wednesday, January 27, 2010
  • 50. Use The Right Tool For The Right Job Hadoop: Relational Databases: Amr Awadallah, Cloudera Inc 13 Wednesday, January 27, 2010
  • 51. Use The Right Tool For The Right Job Hadoop: Relational Databases: When to use? When to use? • Affordable Storage/ • Interactive Reporting Compute (<1sec) • Structured or Not (Agility) • Multistep Transactions • Resilient Auto Scalability • Interoperability Amr Awadallah, Cloudera Inc 13 Wednesday, January 27, 2010
  • 52. Economics of Hadoop Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
  • 53. Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
  • 54. Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
  • 55. Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node ▪ Effective HDFS Space: ▪ ¼ reserved for temp shuffle space, which leaves 9TB/node ▪ 3 way replication leads to 3TB effective HDFS space/node ▪ But assuming 7x compression that becomes ~ 20TB/node Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
  • 56. Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node ▪ Effective HDFS Space: ▪ ¼ reserved for temp shuffle space, which leaves 9TB/node ▪ 3 way replication leads to 3TB effective HDFS space/node ▪ But assuming 7x compression that becomes ~ 20TB/node Effective Cost per user TB: $250/TB Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
  • 57. Economics of Hadoop ▪ Typical Hardware: ▪ Two Quad Core Nehalems ▪ 24GB RAM ▪ 12 * 1TB SATA disks (JBOD mode, no need for RAID) ▪ 1 Gigabit Ethernet card ▪ Cost/node: $5K/node ▪ Effective HDFS Space: ▪ ¼ reserved for temp shuffle space, which leaves 9TB/node ▪ 3 way replication leads to 3TB effective HDFS space/node ▪ But assuming 7x compression that becomes ~ 20TB/node Effective Cost per user TB: $250/TB Other solutions cost in the range of $5K to $100K per user TB Amr Awadallah, Cloudera Inc 14 Wednesday, January 27, 2010
  • 58. Sample Talks from Hadoop World ‘09 ▪ VISA: Large Scale Transaction Analysis ▪ JP Morgan Chase: Data Processing for Financial Services ▪ China Mobile: Data Mining Platform for Telecom Industry ▪ Rackspace: Cross Data Center Log Processing ▪ Booz Allen Hamilton: Protein Alignment using Hadoop ▪ eHarmony: Matchmaking in the Hadoop Cloud ▪ General Sentiment: Understanding Natural Language ▪ Yahoo!: Social Graph Analysis ▪ Visible Technologies: Real-Time Business Intelligence ▪ Facebook: Rethinking the Data Warehouse with Hadoop and Hive Slides and Videos at http://www.cloudera.com/hadoop- Amr Awadallah, Cloudera Inc world-nyc 15 Wednesday, January 27, 2010
  • 59. Cloudera Desktop Amr Awadallah, Cloudera Inc 16 Wednesday, January 27, 2010
  • 60. Conclusion Amr Awadallah, Cloudera Inc 17 Wednesday, January 27, 2010
  • 61. Conclusion Hadoop is a data grid operating system which provides an economically scalable solution for storing and processing large amounts of unstructured or structured data over long periods of time. Amr Awadallah, Cloudera Inc 17 Wednesday, January 27, 2010
  • 62. Contact Information Amr Awadallah CTO, Cloudera Inc. [email protected] http://twitter.com/awadallah Online Training Videos and Info: http://cloudera.com/hadoop- training http://cloudera.com/blog http://twitter.com/cloudera Amr Awadallah, Cloudera Inc 18 Wednesday, January 27, 2010
  • 63. (c) 2008 Cloudera, Inc. or its licensors.  "Cloudera" is a registered trademark of Cloudera, Inc.. All rights reserved. 1.0 Wednesday, January 27, 2010