![Spark Internals
The Key IdeaāØ
āØ
Resilient Distributed Datasets
ā¢ Basic unit of abstraction of data
ā¢ Immutable
ā¢ Persistance
>>> data = [90, 14, 20, 86, 43, 55, 30, 94 ]
>>> distData = sc.parallelize(data)
ParallelCollectionRDD[13] at parallelize at
PythonRDD.scala:364](https://image.slidesharecdn.com/spark-150511063017-lva1-app6891/85/Python-and-Bigdata-An-Introduction-to-Spark-PySpark-13-320.jpg)
![Spark Internals
>>> increment_rdd = distData.map(mapFunc)āØ
>>> increment_rdd.collect()āØ
[91, 15, 21, 87, 44, 56, 31, 95]āØ
>>>āØ
āØ
>>> increment_rdd.filter(filterFunc).collect()āØ
[44, 56]āØ
āØ
ORāØ
āØ
>>> distData.map(mapFunc).filter(filterFunc).collect()āØ
[44, 56]](https://image.slidesharecdn.com/spark-150511063017-lva1-app6891/85/Python-and-Bigdata-An-Introduction-to-Spark-PySpark-20-320.jpg)
Python and Bigdata - An Introduction to Spark (PySpark)
- 1. Python and Big data - An Introduction to Spark (PySpark) Hitesh Dharmdasani
- 2. About me ā¢ Security Researcher, Malware Reversing Engineer, Developer ā¢ GIT > GMU > Berkeley āØ > FireEye > On Stage ā¢ Bootstrapping a few ideas ā¢ Hiring! InformationāØ Security BigāØ Data MachineāØ Learning Me
- 3. What we will talk about? ā¢ What is Spark? ā¢ How does spark do things ā¢ PySpark and data processing primitives ā¢ Example Demo - Playing with Network Logs ā¢ Streaming and Machine Learning in Spark ā¢ When to use Spark http://bit.do/PyBelgaumSpark http://tinyurl.com/PyBelgaumSpark
- 4. What will we NOT talk about ā¢ Writing production level jobs ā¢ Fine Tuning Spark ā¢ Integrating Spark with Kafka and the like ā¢ Nooks and Crooks of Spark ā¢ But glad to talk about it ofļ¬ine
- 5. The Common Scenario Some Data (NTFS, NFS, HDFS, Amazon S3 ā¦) Python Process 1 Process 2 Process 3 Process 4 Process 5 ā¦ You write 1 job. Then chunk,cut, slice and dice
- 6. Compute where the data is ā¢ Paradigm shift in computing ā¢ Don't load all the data into one place and do operations ā¢ State your operations and send code to the machine ā¢ Sending code to machine >>> Getting data over network
- 7. MapReduce public static MyFirstMapper { public void map { . . . } } public static MyFirstReducer { public void reduce { . . . } } public static MySecondMapper { public void map { . . . } } public static MySecondReducer { public void reduce { . . . } } Job job = new Job(conf, āFirst"); job.setMapperClass(MyFirstMapper .class); job.setReducerClass(MyFirstReduc er.class); /*Job 1 goes to Disk */ if(job.isSuccessful()) { Job job2 = new Job(conf,āSecondā); job2.setMapperClass(MySecondMap per.class); job2.setReducerClass(MySecondRe ducer.class); } This also looks ugly if you ask me!
- 8. What is Spark? ā¢ Open Source Lighting Fast Cluster Computing ā¢ Focus on Speed and Scale ā¢ Developed at AMP Lab, UC Berkeley by Matei Zaharia ā¢ Most active Apache Project in 2014 (Even more than Hadoop) ā¢ Recently beat MapReduce in sorting 100TB of data by being 3X faster and using 10X fewer machines
- 9. What is Spark? Spark Some Data (NTFS, NFS, HDFS, Amazon S3 ā¦) Java Python Scala MLLib Streaming ETL SQL ā¦.GraphX
- 10. What is Spark? Spark Some Data (NTFS, NFS, HDFS, Amazon S3 ā¦) ā¢ Inherently distributed ā¢ Computation happens where the data resides
- 11. What is different from MapReduce ā¢ Uses main memory for caching ā¢ Dataset is partitioned and stored in RAM/Disk for iterative queries ā¢ Large speedups for iterative operations when in- memory caching is used
- 12. Spark Internals The Init ā¢ Creating a SparkContext ā¢ It is Sparksā gateway to access the cluster ā¢ In interactive mode. SparkContext is created as āscā $ pyspark ... ... SparkContext available as sc. >>> sc <pyspark.context.SparkContext at 0xdeadbeef>
- 13. Spark Internals The Key IdeaāØ āØ Resilient Distributed Datasets ā¢ Basic unit of abstraction of data ā¢ Immutable ā¢ Persistance >>> data = [90, 14, 20, 86, 43, 55, 30, 94 ] >>> distData = sc.parallelize(data) ParallelCollectionRDD[13] at parallelize at PythonRDD.scala:364
- 14. Spark Internals Operations on RDDs - Transformations & Actions
- 16. Spark Internals Lazy Evaluation Now what?
- 18. Spark Internals Transformation Operations on RDDs Map def mapFunc(x): return x+1 rdd_2 = rdd_1.map(mapFunc) Filter āØ def filterFunc(x): if x % 2 == 0: return True else: return False rdd_2 = rdd_1.filter(filterFunc)
- 19. Spark Internals Transformation Operations on RDDs ā¢ map ā¢ filter ā¢ flatMap ā¢ mapPartitions ā¢ mapPartitionsWithIndex ā¢ sample ā¢ union ā¢ intersection ā¢ distinct ā¢ groupByKey
- 20. Spark Internals >>> increment_rdd = distData.map(mapFunc)āØ >>> increment_rdd.collect()āØ [91, 15, 21, 87, 44, 56, 31, 95]āØ >>>āØ āØ >>> increment_rdd.filter(filterFunc).collect()āØ [44, 56]āØ āØ ORāØ āØ >>> distData.map(mapFunc).filter(filterFunc).collect()āØ [44, 56]
- 21. Spark Internals Fault Tolerance and Lineage
- 22. Moving to the Terminal
- 24. ML Lib ā¢ Machine Learning Primitives in Spark ā¢ Provides training and classiļ¬cation at scale ā¢ Exploits Sparksā ability for iterative computation (Linear Regression, Random Forest) ā¢ Currently the most active area of work within Spark
- 25. How can I use all this? HDFS Spark + ML Lib Load Tweets Bad Tweets Model Live Tweets Good Bad Report to Twitter Spark Streaming
- 26. To Spark or not to Spark ā¢ Iterative computations ā¢ āDon't ļ¬x something that is not brokenā ā¢ Lesser learning barrier ā¢ Large one-time compute ā¢ Single Map Reduce Operation