2012 apache hadoop_map_reduce_windows_azure
Download as PPTX, PDF2 likes662 views
Hadoop is a system for processing large amounts of data using MapReduce and HDFS. HDFS is the storage component that splits files into blocks and stores multiple copies for reliability. MapReduce is the processing framework where mappers process key-value pairs in parallel and reducers aggregate the outputs. While Hadoop can process huge datasets, other systems like Pig, Hive, HBase, Accumulo, Avro, ZooKeeper, and Flume provide additional functionality for tasks like SQL queries, real-time processing, coordination, serialization, and data aggregation.
![HDFS
Hadoop Distributed File System
Yes it is file system written in Java
And you can do normal file system operations
like [ls, mkdir, ...].
Works best with large files. HDFS splits file into
blocks of 128 MB (can be configures)](https://image.slidesharecdn.com/2012apachehadoopmapreducewindowsazure-120722183909-phpapp01/85/2012-apache-hadoop_map_reduce_windows_azure-5-320.jpg)
![Map-Reduce JS example
var map = function ( key, value, context ) {
var words = value.split(/[^a-zA-Z]/);
for ( var i=0; i < words.length; i++ ) {
if ( words[i] !== „“ ) {
context.write( words[i].toLowerCase(), 1 );
}
}
}; var reduce = function ( key, values, context ) {
var sum = 0;
while ( values.hasNext() ) {
sum += parseInt( values.next() );
}
context.write( key, sum );
}](https://image.slidesharecdn.com/2012apachehadoopmapreducewindowsazure-120722183909-phpapp01/85/2012-apache-hadoop_map_reduce_windows_azure-8-320.jpg)
2012 apache hadoop_map_reduce_windows_azure
- 1. Apache Hadoop, MapReduce & Windows Azure Guðmundur Jón Halldórsson Five Degrees July 2012
- 2. Web crawler! „No this isn‘t about that“
- 3. What is Hadoop? System for processing mind-boggingly large amount of data
- 4. Hadoop Map-Reduce = Computation HDFS = Storage
- 5. HDFS Hadoop Distributed File System Yes it is file system written in Java And you can do normal file system operations like [ls, mkdir, ...]. Works best with large files. HDFS splits file into blocks of 128 MB (can be configures)
- 6. HDFS HDFS will keep 3 copies of each block The NameNode tracks blocks and datanodes DN1 DN2 DN3 NN DN4 DN5 DN5 Namenode DN1, DN4, DN7 DN3, DN5, DN8 DN5 DN8 DN9 DN3, DN4, DN5
- 7. Map-Reduce • Write a mapper that takes a key and value, emits zero or more new keys and values • Write a reducer all the values of one key and emits zero or more new keys and values
- 8. Map-Reduce JS example var map = function ( key, value, context ) { var words = value.split(/[^a-zA-Z]/); for ( var i=0; i < words.length; i++ ) { if ( words[i] !== „“ ) { context.write( words[i].toLowerCase(), 1 ); } } }; var reduce = function ( key, values, context ) { var sum = 0; while ( values.hasNext() ) { sum += parseInt( values.next() ); } context.write( key, sum ); }
- 9. MapReduce
- 10. Data Systems and Their Timeframes
- 11. Does hadoop solve all my DATA problems or is are there something else out there?
- 12. • PIG High-level MapReduce Language • HIVE SQL Like high-level MapReduce Language • HBase Realtime processing (based on google BigTable) • Accumulo NSA fork of Hbase • Avro Data Serialization • ZooKeeper Low level coordination • HCatalog Storage Management and interoperability between all systems • OOZIE Job scheduler • Flume Log and data aggregation • Whirr Automated cloud cluster on ec2, rackspace etc • Sqoop Relational data importer • MrUnit Unit testing job • Mahout Machine learning libraries • BigTop Interoperability • Crunch MapReduce pipelines in Java and Scala • Giraph Processing math on huge distribute graphs