20170927 pydata tokyo データサイエンスな皆様に送る分散処理の基礎の基礎、そしてPySparkの勘所
30 likes5,472 views
This document discusses PySpark and how it relates to Python, Spark, and big data frameworks. Some key points discussed include: - PySpark allows users to write Spark applications in Python, enabling Python users to leverage Spark's capabilities for large-scale data processing. - PySpark supports both the RDD API and DataFrame API for working with distributed datasets. It also integrates with Spark libraries like MLlib, GraphX, and Spark SQL. - The document discusses how PySpark fits into the broader Spark and Hadoop ecosystems. It also covers topics like Parquet and Apache Arrow for efficient data serialization between Python and Spark.
20170927 pydata tokyo データサイエンスな皆様に送る分散処理の基礎の基礎、そしてPySparkの勘所
- 1. PySpark @
- 2. ▸ facebook : Ryuji Tamagawa ▸ Twitter : tamagawa_ryuji ▸ FB pydata.tokyo ▸ Twitter
- 4. 8 11
- 5. Wes Mckinney blog ▸ http://qiita.com/tamagawa-ryuji
- 9. ▸ ▸ ▸ Spark Hadoop ▸ PySpark ▸ Spark/Hadoop PyData
- 11. ▸ ▸ ▸
- 15. ▸ ▸ ▸ groupby ▸
- 16. ▸ ▸
- 17. N ▸ N N ▸ …
- 18. … ▸
- 19. ▸ ▸ ▸ CPU/ ▸ CPU/ ▸ 1
- 20. Hadoop Spark ▸ ▸ ▸ n /n
- 21. ▸ ▸ ▸ Amazon EMR ▸ Microsoft Azure HDInsight ▸ Cloudera Altus ▸ Databricks Community Edition Spark ▸ PyData + Jupyter PySpark
- 22. Spark Hadoop
- 23. Spark Hadoop Hadoop0.x Spark OS HDFS MapReduce OS HDFS Hive e.t.c. HBase MapReduce OS HDFS Hive e.t.c. HBaseMapReduce YARN Spark Spark Streaming, MLlib, GraphX, Spark SQL) Impala SQL YARN Spark Spark Streaming, MLlib, GraphX, Spark SQL) Mesos Spark Spark Streaming, MLlib, GraphX, Spark SQL) Spark Spark Streaming, MLlib, GraphX, Spark SQL) Windows Hadoop 0.x Hadoop 1.x Hadoop 2.x + Spark
- 24. Spark Hadoop Hadoop Spark map JVM HDFS reduce JVM map JVM reduce JVM f1 RDD Executor JVM HDFS f2 f3 f4 f5 f6 f7 MapReduce Spark RDD
- 25. Spark Hadoop Spark ▸ Hadoop MapReduce ▸ Spark API MapReduce API ▸ Hadoop
- 26. PySpark (Py)Spark ▸ / Spark ▸ PyData ▸ Spark ▸ Spark Hadoop PyData PySpark
- 28. PySpark
- 29. PySpark RDD API DataFrame API ▸ RDD Resilient Distributed Dataset = Spark Java ▸ DataFrame RDD / R data.frame ▸ Python RDD API DataFrame API Scala / Java
- 30. PySpark DataFrame API RDD DataFrame / Dataset MLlib ML GraphX GraphFrame Spark Streaming Structured Streaming
- 31. Worker node PySpark Executer JVM Driver JVM Executer JVM Executer JVM Storage Python VM Worker node Worker node Python VM Python VM RDD API PySpark Worker node Executer JVM Driver JVM Executer JVM Executer JVM Storage Python VM Worker node Worker node Python VM Python VM DataFrame API PySpark
- 32. PySpark ▸ RDD API Executer JVM Python VM ▸ DataFrame API JVM ▸ UDF Python VM ▸ UDF Scala Java ▸ Spark 2.x DataFrame
- 33. Spark PyData
- 34. Spark PyData Spark PyData ▸ Spark ▸ Python PyData ▸ ▸ Parquet ▸ Apache Arrow
- 35. Spark PyData ▸ CSV JSON ▸Parquet Spark DataFrame API Python fastparquet pyarrow ▸ Performance comparison of different file formats and storage engines in the Hadoop ecosystem ▸ =
- 36. Spark PyData Parquet https://parquet.apache.org/documentation/latest/ zip CSV I/O ROW BLOCK COLUMN #0 ROW #0 COLUMN #0 ROW #1 COLUMN #0 ROW #N COLUMN #1 ROW #0 COLUMN #1 ROW #1 … … COLUMN #1 ROW #N COLUMN #2 ROW #0 COLUMN #2 ROW #1 … COLUMN #M ROW #N ROW BLOCK COLUMN #0 ROW #0 COLUMN #0 ROW #1 COLUMN #0 ROW #N COLUMN #1 ROW #0 COLUMN #1 ROW #1 … … COLUMN #1 ROW #N COLUMN #2 ROW #0 COLUMN #2 ROW #1 … COLUMN #M ROW #N ...
- 37. Spark PyData Spark df = spark.read.csv(csvFilename, header=True, schema = theSchema).coalesce(20) df.write.save(filename, compression = 'snappy') from fastparquet import write pdf = pd.read_csv(csvFilename) write(filename, pdf, compression='UNCOMPRESSED') fastparquet import pyarrow as pa import pyarrow.parquet as pq arrow_table = pa.Table.from_pandas(pdf) pq.write_table(arrow_table, filename, compression = 'GZIP') pyarrow
- 38. Spark PyData ▸ pandas CSV Spark Spark pandas … ▸ Spark - pandas ▸ pandas → Spark … ▸ Apache Arrow
- 39. Spark PyData Apache Arrow ▸ Apache Arrow ▸ PyData / OSS ▸ / https://arrow.apache.org
- 40. Spark PyData Wes blog ▸ pandas Apache Arrow ▸ Blog ▸ PyData Blog Wes OK ▸ Apache Arrow pandas 10 https://qiita.com/tamagawa-ryuji/items/3d8fc52406706ae0c144