Daniel Sikar: Hadoop MapReduce - 06/09/2010
1 like1,305 views
This document discusses using Hadoop MapReduce on Amazon Web Services for parallel processing of large datasets in a distributed manner. It provides an example of counting word frequencies from log files using Hadoop Streaming to run MapReduce jobs with scripts. The steps shown include launching an Elastic MapReduce cluster, uploading data and scripts to S3, running a MapReduce job to count word frequencies from Wikipedia log files, and then terminating the cluster.
Daniel Sikar: Hadoop MapReduce - 06/09/2010
- 1. QUICK AND DIRTY PARALLEL PROCESSING ON THE CLOUD Daniel Sikar
- 2. EC2 S3
- 4. Tools AWS Command line tools
- 5. Elastic MapReduce Ruby library
- 6. Hadoop
- 7. s3cmd
- 8. Hadoop MapReduce Job Tracker + Task Tracker + Slaves HDFS – Distributed file system
- 9. Hadoop MapReduce usage Data crunching in general Clicks Statistics etc
- 10. Hadoop Project Mgmt Committee
- 11. MapReduce ?
- 12. MapReduce Key Pairs <key,value>
- 13. MapReduce
- 14. HTTP Logs Log file A: (...) FreeTouchScreenNokia5230 (...) (...) GetRidofAllSpeedCameras(...) (...) USManWinsLottery (...) (...) BNPToLaunchElectionManifesto (...) Log file B: (...) FreeTouchScreenNokia5230 (...) (...) BodyLanguageTellsAll (...)
- 15. MapReduce <FreeTouchScreenNokia5230, 1> + <FreeTouchScreenNokia5230, 1> = <FreeTouchScreenNokia5230, 2>
- 16. Hadoop Streaming Running MapReduce jobs with .exe fiels and scripts $ <list> | mapper | reducer
- 17. Hadoop Streaming Running MapReduce jobs with .exe fiels and scripts $ <list> | mapper | reducer
- 18. Real life example of Hadoop Streaming usage
- 19. Wikipedia Page Access Logs
- 21. Wikipedia WGV Page Access Stats
- 23. Launching a virtual Hadoop Cluster $ elastic-mapreduce --create --name "Wiki log crunch" --alive --num-instances –instance-type c1.medium 20 Created job flow <job flow id> $ ec2din (...)
- 28. NameNode
- 29. JobTracker
- 34. NameNode
- 35. JobTracker
- 37. Add a step $ elastic-mapreduce --jobflow <jfid> --stream \ --step-name "Wiki log crunch" \ --input s3n://dsikar-wikilogs-2009/dec/ \ --output s3n://dsikar-wikilogs-output/21 \ --mapper s3n://dsikar-wiki-scripts/wikidictionarymap.pl \ --reducer s3n://dsikar-wiki-scripts/wikireduce.pl http://<instance public dns>:9100
- 38. s3cmd # make bucket $ s3cmd mb s3://dsikar-wikilogs # put log files $ s3cmd put pagecounts-200912*.gz s3://dsikar-wikilogs/dec $ s3cmd put pagecounts-201004*.gz s3://dsikar-wikilogs/apr # list log files $ s3cmd ls s3://dsikar-wikilogs/ # put scripts $ s3cmd put *.pl s3://dsikar-wiki-scripts/ # delete log files $ s3cmd del --recursive --force s3://dsikar-wikilogs/ # remove bucket $ s3cmd rb s3://dsikar-wikilogs/
- 39. Elastic MapReduce --create --list --jobflow --describe --stream --terminate
- 40. Output files part-00000 part-00001 part-00002 (...)
- 42. Conclusion Hadoop MapReduce provides out-of-the-box ready-to-go distributed computing.
Editor's Notes
- #21: So without further ado lets get this show on the road and run a job concurrently on a few virtual machines.