20161004 データ処理のプラットフォームとしてのpythonとpandas 東京
8 likes3,233 views
This document discusses Python and the pandas library. It provides an overview of Python's history and advantages, such as being easy to learn and having a large standard library. It also discusses the major Python data analysis packages NumPy, SciPy, matplotlib, and pandas. Pandas allows importing data from various sources, manipulating datasets, and performing operations on labeled and indexed data. The document also covers using pandas with other tools like Spark, visualization with matplotlib, and IDEs and notebooks for Python development.
![1 1.2 1000000
‘abc’ ’ ’
[1, 2, 3, ‘foo’, ‘bar’, ‘foo’]
(1, 2, 3, ‘foo’, ‘bar’, ‘foo’)
{‘k1’: ‘value1’, ‘k2’: ‘value2’}
set(1, 2, 3, ‘foo’, ‘bar’)](https://image.slidesharecdn.com/20160708pythonpandas-161017135317/85/20161004-python-pandas-26-320.jpg)
![•
•
• split
s = ‘foo, bar, baz’
items = s.split(‘,’)
print items[0]
print items[-1]
print items[0][-2:]](https://image.slidesharecdn.com/20160708pythonpandas-161017135317/85/20161004-python-pandas-27-320.jpg)
![• ,
• lambda map, reduce, filter
sList = [‘foo’, ‘bar’, ‘baz’]
lList = [len(s) for s in sList]
lList = map(lambda s:len(s), sList)
lDict = {s:len(s) for s in sList}
lList = []
for s in sList:
lList.append(len(s))
lDict = {}
for s in sList:
lDict[s] = len(s)](https://image.slidesharecdn.com/20160708pythonpandas-161017135317/85/20161004-python-pandas-28-320.jpg)
![• http://sinhrks.hatenablog.com/entry/2015/01/28/073327
0 1
import pandas as pd
df[‘nValue’] = df[‘value’] / sum(df[‘value’])
id value color
sapporo 43 red
osaka 42 pink
matsumoto 40 green
id value color nValue
sapporo 43 red 0.344
osaka 42 pink 0.336
matsumoto 40 green 0.32
Python](https://image.slidesharecdn.com/20160708pythonpandas-161017135317/85/20161004-python-pandas-33-320.jpg)
20161004 データ処理のプラットフォームとしてのpythonとpandas 東京
- 3. • • Python 2000 (**) • db tech showcase MongoDB • • FB: Ryuji Tamagawa • Twitter : tamagawa_ryuji
- 5. 2015-2016
- 6. • Python / pandas • Python / pandas
- 7. • Python • • Python • NumPy, SciPy, matplotlib, pandas • Python • Python IPython, Jupyter notebook, Spyder, VisualStudio • Python / pandas • Python • pandas • Spark - PySpark DataFrame API • matplotlib
- 8. Part 1 : Python
- 9. Python • • Google Guido Google Google 1 • NumPy, SciPy, matplotlib → pandas • • -2000 Linux -2010 Web Trac Google
- 11. Python • 2.x 3.x 32bit 64bit 64bit • 2.x • 3.x 3 • 2.x 3.x
- 12. • Ruby? • R? • Java? • Scala?
- 13. Python • Python ’CPython’ JIT PyPy JVM Jython .Net IronPython • CPython • CPython 2 • C • processing PySpark
- 14. Python • Python Linux Mac OS Python Python Mac • Python pip 3.x 2.7.9 2.x Python pip Linux Python pip yum apt • Python Anaconda Python conda • python 2016 http://qiita.com/y__sama/items/5b62d31cb7e6ed50f02c
- 15. NumPy, SciPy, matplotlib, pandas • • NumPy SciPy • pandas pandas pandas NumPy • Anaconda Python
- 18. Python IPython Jupyter, … IDE Spyder, Rodeo Visual Studio, PyCharm, PyDev
- 20. • Jupyter Notebook • Python • IPython Notebook Python • Apache Zeppelin http:// zeppelin.apache.org
- 21. IDE • R RStudio • IDE • • 2 Spyder Rodeo • Spyder
- 22. • • Visual Studio • Eclipse PyDev • PyCharm •
- 23. Part 2 : Python / pandas
- 24. Python / pandas • pandas • / etc… • Spark • pandas
- 25. processing • • 64bit Python + GB • Python 1 1 CPU GIL • processing Jenkins CPU/ Jenkins
- 26. 1 1.2 1000000 ‘abc’ ’ ’ [1, 2, 3, ‘foo’, ‘bar’, ‘foo’] (1, 2, 3, ‘foo’, ‘bar’, ‘foo’) {‘k1’: ‘value1’, ‘k2’: ‘value2’} set(1, 2, 3, ‘foo’, ‘bar’)
- 27. • • • split s = ‘foo, bar, baz’ items = s.split(‘,’) print items[0] print items[-1] print items[0][-2:]
- 28. • , • lambda map, reduce, filter sList = [‘foo’, ‘bar’, ‘baz’] lList = [len(s) for s in sList] lList = map(lambda s:len(s), sList) lDict = {s:len(s) for s in sList} lList = [] for s in sList: lList.append(len(s)) lDict = {} for s in sList: lDict[s] = len(s)
- 29. pandas • pandas • matplotlib / seaborn • NumPy SciPy Python • pandas + matplotlib OK pandas NumPy NumPy / SciPy https://openbook4.me/projects/183
- 31. pandas • pandas DataFrame • R • RDB 2 • index Series Columns Columns Series Series SeriesIndex
- 32. IDE / • IDE • jupyter notebook
- 33. • http://sinhrks.hatenablog.com/entry/2015/01/28/073327 0 1 import pandas as pd df[‘nValue’] = df[‘value’] / sum(df[‘value’]) id value color sapporo 43 red osaka 42 pink matsumoto 40 green id value color nValue sapporo 43 red 0.344 osaka 42 pink 0.336 matsumoto 40 green 0.32 Python
- 34. pandas I/O • CSV JSON RDB Excel • column • RDB • import pandas as pd pd.read_csv(<filename>) pd.read_json(<filename>) pd.to_csv(<filename>) pd.to_excel(<filename>) # pd.to_clipboard()
- 35. pandas.read_csv • pandas CSV • • • usecols : • nrows : • na_values : na • parse_dates infer_datetime_format: • chunksize : • compression : zip CSV pandas.read_csv(filepath_or_buffer, sep=', ', delimiter=None, header='infer', names=None, index_col=None, usecols=None, squeeze=False, prefix=None, mangle_dupe_cols=True, dtype=None, engine=None, converters=None, true_values=None, false_values=None, skipinitialspace=False, skiprows=None, skipfooter=None, nrows=None, na_values=None, keep_default_na=True, na_filter=True, verbose=False, skip_blank_lines=True, parse_dates=False, infer_datetime_format=False, keep_date_col=False, date_parser=None, dayfirst=False, iterator=False, chunksize=None, compression='infer', thousands=None, decimal='.', lineterminator=None, quotechar='"', quoting=0, escapechar=None, comment=None, encoding=None, dialect=None, tupleize_cols=False, error_bad_lines=True, warn_bad_lines=True, skip_footer=0, doublequote=True, delim_whitespace=False, as_recarray=False, compact_ints=False, use_unsigned=False, low_memory=True, buffer_lines=None, memory_map=False, float_precision=None)
- 36. Spark - PySpark DataFrame API • Python • Spark PySpark findSpark Spark • Python Spark API DataFrame API • Spark pandas Spark PySpark Spark node Spark node Spark node Spark node driver
- 37. • • Apache Arrow • Python / R : feather • pandas 2.0, parquet for Python
- 38. Python / pandas
- 39. Questions ?