When big data meet python @ COSCUP 2012
22 likes4,823 views
The document presents a presentation by Jimmy Lai on the intersection of big data and Python, outlining the concepts, benefits, and technical issues associated with big data. It discusses various tools, techniques, and libraries available in Python for data collection, storage, processing, and visualization. Additionally, the document highlights initiatives from the White House to support big data education and technology development.
When big data meet python @ COSCUP 2012
- 1. When Big Data Meet Python Jimmy Lai (賴弘哲) [email protected] 2012/08/19 Slides: http://www.slideshare.net/jimmy_lai/when-big-data-meet-python 2012 When big data meet python by Jimmy Lai is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. 1
- 2. 自我介紹 • 賴弘哲 (Jimmy Lai) • Interests: Data mining, Machine Learning, Natural Language Processing, Distributed Computing, Python • LindedIn profile: http://goo.gl/XTEM5 • 現任職於引京聚點知識結構搜索公司， 從事大資料語意分析 2012 2
- 3. Outline 1. Big Data a. Concept b. Technical issues 2. Big Data + Python a. Related open source tools b. Example 2012 3
- 4. Benefits of Big Data 1. Creating transparency(透明度) e.g. http://www.data.gov/ 2. Enabling experimentation to discover needs, expose variability, and improve performance(發現需求及潛在威脅、改善產能) 3. Segmenting populations to customize(客製化) actions 4. Replacing/supporting human decision making with automated algorithms(自動決策) 5. Innovating new business models, products and services(創新的服務、產業) 深度資料分析人才的短缺 (May 2011). Big Data: The next frontier for innovation, competition, and productivity. 2012 McKinsey Global Institute. 4
- 5. Initiative from the White House • (Mar 2012) Big Data Research and Development Initiative, the White House. • National Science Foundation encourages education on Big Data. • Government invest on developing state-of- the-art technologies, harness those technologies, and expand the workforce for Big Data. 2012 5
- 6. Big Data Issues User Generated Content Machine Generated Data Collecting Storage Computing Analysis Visualization 2012 6
- 7. Big Data Techniques Machine User Generated Content Generated Data • Crawler – Collect raw data Collecting – E.g. Heritrix, Nutch • Scraping Storage – Parse information Computing from raw data – E.g. Yahoo! Pipes, Analysis Scrapy Visualization 2012 7
- 8. Big Data Techniques User Generated Machine Generated Data • Big Table Content – Distributed key-value storage Collecting – E.g.Hbase, Cassandra • NoSQL Storage – Not use SQL for manipulation Computing – Not use relational database model Analysis – E.g. MongoDB, Redis, CouchDB Visualization 2012 8
- 9. Big Data Techniques Machine User Generated Content Generated Data • Batch – MapReduce Collecting – E.g. Hadoop • Real-time Storage – Stream processing Computing – E.g. S4, Storm Analysis Visualization 2012 9
- 10. Big Data Techniques User Generated Machine • Data mining Content Generated Data – Weka • Machine learning Collecting – scikit-learn • Natural language Storage processing – NLTK, Stanford NLP Computing • Statistics –R Analysis Visualization 2012 10
- 11. Big Data Techniques Machine User Generated Content Generated Data • Abstract • Interactive Collecting • E.g. Processing, Gephi, D3.js Storage Computing Analysis Visualization 2012 11
- 12. Why Python? • Good code readability • Fast growing among for fast development. open source • Scripting language: the communities. less code, the more – Commits statistics from productivity. ohloh.net 2012 12
- 13. When Big Data meet Python User Generated Machine Content Generated Data Collecting Scrapy: scraping framework PyMongo: Python client for Mongodb Infrastructure Storage Hadoop streaming: Linux pipe interface Computing Disco: lightweight MapReduce in Python Pandas: data analysis/manipulation Analysis Statsmodels: statistics NLTK: natural language processing Scikit-learn: machine learning Visualization Matplotlib: plotting 2012 NetworkX: graph visualization 13
- 14. When Big Data meet Python User Generated Machine Generated Data http://scrapy.org/ Content web scraping framework • Simple and Extensible Collecting • Components: • Scheduler Storage • Downloader • Spider(Scraper) Computing • Item pipeline Analysis Visualization 2012 14
- 15. When Big Data meet Python User Generated Machine http://www.mongodb.org/ Content Generated Data NoSQL database • PyMongo: client for python Collecting • Document(JSON)-oriented • No schema Storage • Scalable • Auto-sharding Computing • Replica-set Analysis • File storage • MapReduce aggregation Visualization 2012 15
- 16. When Big Data meet Python Machine http://discoproject.org/ User Generated Content Generated Data • Distributed computing: – MapReduce Collecting – Disco distributed file system • Write code in Python Storage – Easy/fast to profiling – Easy/fast to debugging Computing Analysis Visualization 2012 16
- 17. When Big Data meet Python User Generated Machine Content Generated Data http://pandas.pydata.org/ • Data analysis library Collecting • Datastructure for fast data manipulation – Slicing Storage – Indexing – subsetting Computing • Handling missing data Analysis • Aggregation • Time series Visualization 2012 17
- 18. When Big Data meet Python User Generated Machine Statsmodels Content Generated Data http://statsmodels.sourceforge.net/ • Statistical analysis Collecting • Statistical models • Fit data with model Storage • Statistical tests • Data exploration Computing • Time series analysis Analysis Visualization 2012 18
- 19. When Big Data meet Python User Generated Machine scikit-learn Content Generated Data http://scikit-learn.org/ • Machine learning algorithms • Supervised learning Collecting • Unsupervised learning • Dataset Storage • Preprocessing Computing • feature extraction • Model Analysis • Selection • Pipeline Visualization 2012 19
- 20. When Big Data meet Python User Generated Machine Content Generated Data NLTK: Natural Language Toolkit http://scikit-learn.org/ • Natural language processing Collecting • Annotated corpora and resources Information Extraction Work Flow Storage Sentence Segmentation Tokenization POS tagging Computing Named Entity Relation Recognition Recognition Analysis Visualization 2012 20
- 21. When Big Data meet Python User Generated Machine Content Generated Data NL http://matplotlib.sourceforge.net/ • Plotting Collecting – Histograms – Power spectra Storage – Bar charts – Error charts Computing – Scatter plots • Full control to detail of plotting Analysis Visualization 2012 21
- 22. When Big Data meet Python User Generated Machine Content Generated Data NetworkX http://networkx.lanl.gov/ • Graph algorithms and visisualization Collecting • Draw graph with layout: – Circular Storage – Random – Spectural Computing – Spring – Shell Analysis – Graphviz Visualization 2012 22
- 23. 聚寶評 www.ezpao.com 美食搜尋引擎 搜尋各大部落格食記 2012 23
- 24. 聚寶評 www.ezpao.com 語意分析搜尋引擎 2012 24
- 25. 評論主題分析 網友分享菜分析 正評/負評分析 2012 25
- 26. Thank you for your attention. Q&A We are hiring! • 核心引擎演算法研發工程師 • 系統研發工程師 • 網路應用研發工程師 Oxygen Intelligence Taiwan Limited 引京聚點 知識結構搜索股份有限公司 • 公司簡介： http://www.ezpao.com/about/ • 職缺簡介： http://www.ezpao.com/join/ • 請將履歷寄到 [email protected] 2012 When big data meet python by Jimmy Lai is licensed under a Creative Commons Attribution-ShareAlike 3.0 Unported License. 26