Tachyon and Apache Spark
27 likes14,608 views
The document discusses the evolution of in-memory computing with a focus on Tachyon and Apache Spark, highlighting key innovations such as resilient distributed datasets (RDDs) that enhance data processing. It compares Hadoop's traditional MapReduce framework with Spark's capabilities and examines the role of Tachyon as an in-memory data-exchange layer, suggesting it is beneficial for cloud environments. Overall, it emphasizes the shift toward faster, iterative processing in big data applications.
Tachyon and Apache Spark
- 1. Tachyon and Apache Spark: heralds of in-memory computing era. Roman Shaposhnik Director of Open Source @Pivotal (Twitter: @rhatr)
- 2. Who’s this guy? • Director of Open Source @Pivotal • Apache Software Foundation guy (Member, VP of Apache Incubator, committer on Hadoop, Giraph, Sqoop, etc) • Used to be root@Cloudera • Used to be PHB@Yahoo! (original Hadoop team)
- 4. 20 minute to figure out Hadoop vs. Spark
- 5. 20 minute to figure out Hadoop++ == Spark
- 6. 20 minute to figure out Hadoop + Spark
- 7. But wait! There’s more! Tachyon
- 8. Long, long time ago… HDFS ASF Projects FLOSS Projects Pivotal Products MapReduce
- 9. In a blink of an eye MLib Shark GraphX Streaming HDFS Crunch Mahout Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Spark Impala HAWQ SpringXD MADlib Hamster PivotalR YARN Tachyon
- 10. A Spark view? HDFS MLib Shark YARN GraphX Streaming Tachyon Sqoop Flume Hadoop UI Hue Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie SolrCloud Phoenix HBase Spark SpringXD
- 11. BDAS
- 12. Long, long time ago…
- 13. This is 2014
- 14. What changed?
- 15. Your datacenter … server 1 server N
- 16. Hadoop’s view MapReduce server 1 server N HDFS
- 17. HDFS: decoupled storage … MR HDFS MR
- 19. Anatomy of MapReduce HDFS mappers reducers HDFS a b c d a c a 3 b 1 c 2 a 1 b 1 c 1 a 1 c 1 a 1 a 1 1 1 b 1 c 1 1
- 20. What’s wrong with MR? Source: UC Berkeley Spark project (just the image)
- 21. This looks familiar… $ grep –R | awk | sort …
- 22. Spark innovations • Resilient Distribtued Datasets (RDDs) • Distributed on a cluster • Manipulated via parallel operators (map, etc.) • Automatically rebuilt on failure • A parallel ecosystem • A solution to iterative and multi-stage apps
- 23. RDDs warnings = textFile(…).filter(_.contains(“warning”)) .map(_.split(‘ ‘)(1)) HadoopRDD path = hdfs:// FilteredRDD contains… MappedRDD split…
- 24. Parallel operators • map, reduce • sample, filter • groupBy, reduceByKey • join, leftOuterJoin, rightOuterJoin • union, cross
- 25. What is really happening? MLib Shark GraphX Streaming HDFS Crunch Mahout Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire XD Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Spark Impala HAWQ SpringXD MADlib Hamster PivotalR YARN Tachyon
- 26. May be its not so bad server 1 server N
- 27. But HDFS/YARN are safe? HDFS, Ceph, S3, NAS, etc. New HDFS New YARN
- 28. Tachyon • In-memory data-exchange layer • A set of evolving APIs: • filesystem • caching • RDDs • Materialized views
- 29. Tachyon
- 30. Spark is best for cloud
- 31. It will be called Hadoop MLib Shark GraphX Streaming HDFS Crunch Mahout Pig Sqoop Flume Coordination and workflow management Zookeeper Command Center ASF Projects FLOSS Projects Pivotal Products GemFire with Tachyon Oozie MapReduce Hive Tez Giraph Hadoop UI Hue SolrCloud Phoenix HBase Spark Impala HAWQ SpringXD MADlib Hamster PivotalR YARN
- 32. Spark/Tachyon recap • Is it “Big Data” (Yes) • Is it “Hadoop” (No) • It’s one of those “in memory” things, right (Yes) • JVM, Java, Scala (All) • Is it Real or just another shiny technology with a long, but ultimately small tail (Yes and ?)
- 33. A NEW PLATFORM FOR A NEW ERA
- 34. Questions ?