Cloud and Big Data trends
Download as PPTX, PDF7 likes1,718 views
The document discusses the evolution of big data and cloud computing, highlighting the development of distributed systems and the significance of tools like MapReduce and HDFS. It outlines the relationship between big data solutions and cloud infrastructure, emphasizing the integration of big data services with cloud platforms. The author concludes by stressing the ongoing need for improved management of data transfers and collaboration within the cloud and big data communities.
Cloud and Big Data trends
- 1. Cloud and Big Data Sebastien Goasguen, January 29th @sebgoa
- 2. A view on Big Data
- 4. SKA
- 8. How did we get there ?
- 10. New Distributed systems for: Large scale datasets • From scientific instruments • From Web apps logs Complex datasets • Not necessarily large. Object stores • S3 clones
- 11. BigData and map-reduce • While BigData is often associated with HDFS, Map-Reduce is the algorithm used to parallelize data processing. • BigData ≠ Map-Reduce ≠ HDFS • Map-reduce is a way to express embarrassingly parallel work easily. • You can do Map-Reduce without HDFS. • e.g Basho map-reduce on riackCS
- 12. A really quick view on Clouds
- 15. Open Source IaaS
- 16. Today
- 17. BigData at peak
- 18. History 2003 –Google File System 2005 – Hadoop 2006 – Hadoop enters ASF incubator (Feb) 2006 – S3 launched 2007 – Paper on Amazon Dynamo 2009 – EMR launched 2013 – CloudStack as a ASF TLP (March) 2013 – Spark/Mesos enters ASF incubator
- 19. The Apache Software Foundation
- 21. 35 projects in incubation: • • • 12 Hadoop related ~30% Big Data related Spark 117 top level projects: • • • • • ~16 cloud or bigdata +10% Deltacloud, Libcloud, Whirr, jclouds Hadoop, couchdb, cassandra, mesos Bigtop, accumulo, lucene, UIMA CloudStack
- 22. Hadoop Ecosystem + Up-coming next generation BD systems
- 23. Big Data and Cloud (Stack)s
- 24. Clouds and BigData • Object store + compute IaaS to build EC2+S3 clone • BigData solutions as storage backends for image catalogue and large scale instance storage. • BigData solutions as workloads to CloudStack based clouds.
- 25. EC2, S3 clone • An open source IaaS with an EC2 wrapper e.g Opennebula • Deploy a S3 compatible object store – separately- e.g riakCS • Two independent distributed systems deployed Cloud = EC2 + S3
- 26. Big Data as IaaS backend “Big Data” solutions can be used as secondary storage .
- 27. Example • Open source IaaS + EC2 wrapper, e.g CloudStack • Deploy S3 compatible object store, e.g riakCS or Ceph or glusterFS • Use S3 as image store • Your EC2 service is a customer to your S3 service • Logstash + elasticsearch for logs/monitoring
- 28. Even use Bare Metal
- 29. Big Data as a Workload to the Cloud
- 30. Mesos, Spark are EC2 native o ec2_deploy.py o ec2_deploy.sh o…
- 31. Tools
- 32. “PaaS”
- 33. Dev Pipeline
- 34. Conclusions • Big Data is “catching up” • Tackle the big three head on: • BigData, Cloud and DevOps • Add a big data backend to your cloud from the start • Provide Big Data services on your cloud
- 35. Still behind !
- 36. Final Thoughts Who manages my data transfers ?
- 37. Event ApacheCON + CloudStack Collaboration Conference Denver April 7-11th. Cloud and Big Data
- 38. Get Involved with Apache CloudStack Web: http://cloudstack.apache.org/ Mailing Lists: cloudstack.apache.org/mailing-lists.html IRC: irc.freenode.net: 6667 #cloudstack #cloudstack-dev Twitter: @cloudstack LinkedIn: www.linkedin.com/groups/CloudStack-Users-Group-3144859 If it didn’t happen on the mailing list, it didn’t happen.
Editor's Notes
- #4: Walmart, 1m customer transactions every hour, db of 2.5 PB in 2010 http://www.economist.com/node/15557443?story_id=15557443
- #5: Square Kilometer Array 10-500 TB per second ….1 exabyte per dayFacebook June 2012, 100 PB hadoop cluster, ½ PB per day = 180 PB per year -> ~350 PB now ?CERN ~20 PB EOS
- #6: 250k cables war and peace 450k words, 260M worlds in cable gate = 500x war and peace
- #7: 200 Million pages, 4 TB