SlideShare a Scribd company logo

Introduction to ML with Apache Spark MLlib

Taras Matyashovsky, profile picture
Taras Matyashovsky

The document provides an overview of using Apache Spark's MLlib for various machine learning tasks, including feature extraction, classification, and clustering. It discusses the process of cleansing and preparing datasets, particularly with song lyrics, and demonstrates using algorithms like logistic regression. Additionally, it highlights the importance of existing libraries and frameworks that simplify machine learning implementation.

Engineering
Related topics:
Apache Spark
1 of 88
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
Most read
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
Most read
61
62
63
64
65
66
67
68
69
70
71
72
73
Most read
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
with Apache Spark MLlib #javaone
https://ua.linkedin.com/in/tarasmatyashovsky 2
I am not a data science engineer 3
4
lyrics genre 5
“I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you” https://github.com/tmatyashovsky/spark-ml-samples 6
“I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you” https://github.com/tmatyashovsky/spark-ml-samples 7
8
 Look for particular words like “fear”, “fight”, “kill”, “devil”, ”death”, etc.?  Count length of a verse?  Count unique words in a verse? 9
10
15-20 11
is the study of computer algorithms that improve automatically through experience 12
Supervise d learning Unsupervise d learning Reinforcemen t learning 13
14
 Date & time  Conference name  Speaker  Talk name  Track  Duration  Type  Overall impression  Overall rating  Number of slides  Time spent on live coding  Number of jokes  Etc. 15
Learning algorithms Hypotheses: Сost function: Features: Target variable: Training example: Training set: 16
http://www.slideshare.net/liweiyang5/spark-mllib-training-material 17
Number of jokes during a talk Speaker’s rating 18
19
20
21
22
23
24
Positive Negative Impression Number of jokes during a talk 25
26
27
28
29
30
31
Numberofjokesduringa talk Time (min.) spent on live coding Number of clusters: K = 5K = 2 32
33  Initialize cluster centroids:  assign each example to the closest cluster centroid  Recalculate centroids as an average (mean) of examples assigned to a cluster
34
35
36
 Collect data set of lyrics:  Abba, Ace of base, Backstreet Boys, Britney Spears, Christina Aguilera, Madonna, etc.  Black Sabbath, In Flames, Iron Maiden, Metallica, Moonspell, Nightwish, Sentenced, etc.  Create training set, i.e. label (0|1) + features  Train logistic regression (or other classification algorithm) https://github.com/tmatyashovsky/spark-ml-samples 37
https://github.com/tmatyashovsky/spark-ml-samples 38
39
GloV e Bag of Words Word2VecTF- IDF http://spark.apache.org/docs/latest/ml-features.html#feature-extractors 40
 Produces unique fixed-size dense vectors  Captures semantic and morphologic similarity https://code.google.com/archive/p/word2vec/ 41
Similar scores (cos ~ 1) Opposite scores (cos ~ -1) Unrelated scores (cos ~ 0) http://bionlp-www.utu.fi/wv_demo/ http://blog.christianperone.com/wp-content/uploads/2013/09/cosinesimilarityfq1.png 42
43 Verse Cosine Distance baby one more time 0.482028 crazy for you 0.437875 show me the meaning of being lonely 0.258147 highway to hell -0.1120049 kill them all -0.231876 https://github.com/tmatyashovsky/spark-ml-samples
https://github.com/tmatyashovsky/spark-ml-samples 44
Introduction to ML with Apache Spark MLlib
Introduction to ML with Apache Spark MLlib
Under-fitting (high bias) Over-fitting (high variance) Appropriate fitting http://mlwiki.org/index.php/Overfitting 47
Training set (66,6%) Test set (33%) K = 3 48
Training set (66,6%) Test set (33%) K = 3 49
Training set (33,3%) Test set (33%) Training set (33,3%) K = 3 50
51
Java 52
Weka Encog AerosolveFlinkM L https://github.com/josephmisiti/awesome-machine-learning 53
Easy of use Cloud computing Spee d Generali ty Data processing 54
https://databricks.com/blog/2015/02/09/learning-spark-book-available-from-oreilly.html 55
Is a library of ML algorithms and utilities designed to run in parallel on Spark cluster 56
 Introduces a few new data types, e.g. vector (dense and sparse), labeled point, rating, etc.  Allows to invoke various algorithms on distributed datasets (RDD/Dataset) http://spark.apache.org/docs/latest/mllib-guide.html 57
http://spark.apache.org/docs/latest/mllib-guide.html Build on top of RDDs Build on top of Datasets spark.mll ib spark.ml 58
 Utilities: linear algebra, statistics, etc.  Features extraction, features transforming, etc.  Regression  Classification  Clustering  Collaborative filtering, e.g. alternating least squares  Dimensionality reduction  And many more http://spark.apache.org/docs/latest/mllib-guide.html 59
”All” spark.mllib features plus: • Pipelines • Persistence • Model selection and tuning: • Train validation split • K-folds cross validation http://spark.apache.org/docs/latest/ml-guide.html 60
Raw data Transformer Estimator [parameters] Transformer [parameters] Estimator [parameters] Dataset Dataset Dataset Dataset http://spark.apache.org/docs/latest/ml-pipeline.html Cross Validator [pipeline, evaluator, parameters] Dataset 61
Using Spark MLlib Pipeline
Lyrics https://github.com/tmatyashovsky/spark-ml-samples 63
I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you https://github.com/tmatyashovsky/spark-ml-samples 64
Lyrics Cleanser Dataset https://github.com/tmatyashovsky/spark-ml-samples 65
I'm a rolling thunder, a pouring rain I'm comin' on like a hurricane My lightning's flashing across the sky You're only young but you're gonna die I won't take no prisoners, won't spare no lives Nobody's putting up a fight I got my bell, I'm gonna take you to hell I'm gonna get you, Satan get you https://github.com/tmatyashovsky/spark-ml-samples 66
Lyrics Cleanser Dataset Numerator Dataset https://github.com/tmatyashovsky/spark-ml-samples 67
Im a rolling thunder a pouring rain Im comin on like a hurricane My lightnings flashing across the sky Youre only young but youre gonna die I wont take no prisoners wont spare no lives Nobodys putting up a fight I got my bell Im gonna take you to hell Im gonna get you Satan get you https://github.com/tmatyashovsky/spark-ml-samples 68 1 2 3 4 5 6 7 8
Lyrics Cleanser Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 69
im a rolling thunder a pouring rain im comin on like a hurricane My lightnings flashing across the sky youre only young but youre gonna die I wont take no prisoners wont spare no lives nobodys putting up a fight I got my bell im gonna take you to hell im gonna get you satan get you https://github.com/tmatyashovsky/spark-ml-samples 70 1 2 3 4 5 6 7 8
Lyrics Cleanser Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 71
im rolling thunder pouring rain im comin like hurricane lightnings flashing across sky youre young youre gonna die wont take prisoners wont spare lives nobodiys putting fight got bell im gonna take hell im gonna get satan get https://github.com/tmatyashovsky/spark-ml-samples 72 1 2 3 4 5 6 7 8
Lyrics Cleanser Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 73
4 im roll thunder pour rain im comin like hurrican lightn flash across sky your young your gonna die wont take prison wont spare live nobodi put fight got bell im gonna take hell im gonna get satan get https://github.com/tmatyashovsky/spark-ml-samples 74 1 2 3 4 5 6 7 8 verse1 verse2
8 im roll thunder pour rain im comin like hurrican Light n flash across sky your young your gonna die wont take prison wont spare live nobodi put fight got bell im gonna take hell im gonna get satan get https://github.com/tmatyashovsky/spark-ml-samples 75 1 2 3 4 5 6 7 8 verse1
Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 76
4 [0.036463763926011056, -0.013076733228398295, ... 0.03816963326281462] https://github.com/tmatyashovsky/spark-ml-samples 77 feature1 feature2 [-0.013962931134021625, 0.049275818325650804, ... -0.058982484615766086]
8 [0.036463763926011056, -0.013076733228398295, 0.044362547532774695, 0.03816963326281462, ... -0.013962931134021625, 0.049275818325650804, -0.058982484615766086] https://github.com/tmatyashovsky/spark-ml-samples 78 feature1
Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Logistic Regression [Max iterations, Reg parameter] Dataset Dataset Dataset https://github.com/tmatyashovsky/spark-ml-samples 79
Probability: [0.9212126972383768, 0.07878730276162313] Prediction: 0.0 https://github.com/tmatyashovsky/spark-ml-samples 80
Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Logistic Regression [Max iterations, Reg parameter] Dataset Dataset Cross Validator Model Dataset https://github.com/tmatyashovsky/spark-ml-samples 81
[0.8454839775240359, 0.9061236588248319, 0.9527128936788524, 0.9522790271664413, ... 0.9526248129757111, 0.9522790271664411] https://github.com/tmatyashovsky/spark-ml-samples 82
Lyrics Cleanser Word2Vec [Vector size] Dataset Dataset Numerator Tokenizer Stop Words Remover Dataset Dataset ExploderStemmer Dataset Uniter Dataset Verser [Sentences in verse] Dataset Logistic Regression [Max iterations, Reg parameter] Dataset Dataset Cross Validator Model Dataset https://github.com/tmatyashovsky/spark-ml-samples 83
84
85
86  ML is not as complex as it seems from an applied perspective  Existing libraries and frameworks reduce a lot of tedious work  For instance, Spark MLlib can help to build nice ML pipelines
Design by 87
 https://www.quora.com/What-is-the-difference-between-supervised-and-unsupervised-learning-algorithms  Learning Spark, by Holden Karau, Andy Konwinski, Patrick Wendell and Matei Zaharia  https://databricks.com/blog/2015/01/07/ml-pipelines-a-new-high-level-api-for-mllib.html  https://databricks.com/blog/2016/05/31/apache-spark-2-0-preview-machine-learning-model-persistence.html  https://en.wikipedia.org/wiki/List_of_datasets_for_machine_learning_research  https://www.kaggle.com/c/dogs-vs-cats/  http://yann.lecun.com/exdb/mnist/  http://www.bcl.hamilton.ie/~barak/teach/F98/ECE547/hw1/index.html  http://www.slideshare.net/jeykottalam/pipelines-ampcamp  https://github.com/master/spark-stemming  https://databricks.com/blog/2016/04/01/unreasonable-effectiveness-of-deep-learning-on-apache-spark.html  http://www.degeneratestate.org/posts/2016/Apr/20/heavy-metal-and-natural-language-processing-part-1/  https://spark.apache.org/docs/2.0.0/api/java/org/apache/spark/sql/functions.html  https://www.quora.com/What-is-the-difference-between-supervised-and-unsupervised-learning-algorithms  http://www.slideshare.net/liweiyang5/spark-mllib-training-material  https://databricks.com/blog/2016/01/25/deep-learning-with-apache-spark-and-tensorflow.htm  http://www.slideshare.net/databricks/combining-machine-learning-frameworks-with-apache-spark l  https://databricks.com/blog/2015/10/20/audience-modeling-with-apache-spark-ml-pipelines.html  https://github.com/deeplearning4j/deeplearning4j  http://deeplearning4j.org/spark  http://mlwiki.org/index.php/Overfitting  http://bionlp-www.utu.fi/wv_demo/  https://quomodocumque.wordpress.com/2016/01/15/messing-around-with-word2vec/ 88

More Related Content

PPTX
Introduction to Amazon S3
Ashay Shirwadkar
 
PPTX
Introduction to Hadoop and Hadoop component
rebeccatho
 
PDF
XGBoost & LightGBM
Gabriel Cypriano Saca
 
PPTX
Random forest
Musa Hawamdah
 
PDF
Data Visualization in Data Science
Maloy Manna, PMP®
 
PPTX
Delta lake and the delta architecture
Adam Doyle
 
PPTX
Gradient Boosted trees
Nihar Ranjan
 
PPTX
Phishing ppt
Sanjay Kumar
 
Introduction to Amazon S3
Ashay Shirwadkar
 
Introduction to Hadoop and Hadoop component
rebeccatho
 
XGBoost & LightGBM
Gabriel Cypriano Saca
 
Random forest
Musa Hawamdah
 
Data Visualization in Data Science
Maloy Manna, PMP®
 
Delta lake and the delta architecture
Adam Doyle
 
Gradient Boosted trees
Nihar Ranjan
 
Phishing ppt
Sanjay Kumar
 

What's hot (20)

PPTX
Spark architecture
GauravBiswas9
 
PDF
Machine Learning with Spark MLlib
Todd McGrath
 
PPTX
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
Simplilearn
 
PPTX
Spark streaming
Whiteklay
 
PDF
Introduction to MLflow
Databricks
 
PDF
What is in a Lucene index?
lucenerevolution
 
PDF
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
PPTX
Apache Spark Fundamentals
Zahra Eskandari
 
PDF
Introduction to elasticsearch
pmanvi
 
PPTX
Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spar...
Simplilearn
 
PDF
Apache Flume
Arinto Murdopo
 
PPTX
Apache Flink and what it is used for
Aljoscha Krettek
 
PDF
Introduction to Spark with Python
Gokhan Atil
 
PPTX
Apache Spark Architecture
Alexey Grishchenko
 
PPTX
Spark SQL Tutorial | Spark SQL Using Scala | Apache Spark Tutorial For Beginn...
Simplilearn
 
PPTX
Apache Spark Components
Girish Khanzode
 
PDF
Introduction to Spark Streaming
datamantra
 
PPTX
Apache Spark overview
DataArt
 
PDF
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
PDF
Apache spark
shima jafari
 
Spark architecture
GauravBiswas9
 
Machine Learning with Spark MLlib
Todd McGrath
 
What Is Apache Spark? | Introduction To Apache Spark | Apache Spark Tutorial ...
Simplilearn
 
Spark streaming
Whiteklay
 
Introduction to MLflow
Databricks
 
What is in a Lucene index?
lucenerevolution
 
Scaling your Data Pipelines with Apache Spark on Kubernetes
Databricks
 
Apache Spark Fundamentals
Zahra Eskandari
 
Introduction to elasticsearch
pmanvi
 
Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spar...
Simplilearn
 
Apache Flume
Arinto Murdopo
 
Apache Flink and what it is used for
Aljoscha Krettek
 
Introduction to Spark with Python
Gokhan Atil
 
Apache Spark Architecture
Alexey Grishchenko
 
Spark SQL Tutorial | Spark SQL Using Scala | Apache Spark Tutorial For Beginn...
Simplilearn
 
Apache Spark Components
Girish Khanzode
 
Introduction to Spark Streaming
datamantra
 
Apache Spark overview
DataArt
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
Noritaka Sekiyama
 
Apache spark
shima jafari
 
Ad

Viewers also liked (20)

PDF
Introduction to Machine Learning with Spark
datamantra
 
PDF
MLlib: Spark's Machine Learning Library
jeykottalam
 
PPTX
Yace 3.0
Atul Ashar
 
PPTX
MLlib and Machine Learning on Spark
Petr Zapletal
 
PDF
Large-Scale Machine Learning with Apache Spark
DB Tsai
 
PDF
Practical Machine Learning Pipelines with MLlib
Databricks
 
PPTX
Machine Learning With Spark
Shivaji Dutta
 
PDF
Spark DataFrames and ML Pipelines
Databricks
 
PDF
Ingesting Drone Data into Big Data Platforms
Timothy Spann
 
PDF
R, Scikit-Learn and Apache Spark ML - What difference does it make?
Villu Ruusmann
 
PDF
Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models
Anyscale
 
PDF
Yi Wang, Tech Lead of AI Platform, Baidu, at MLconf 2017
MLconf
 
PDF
Giraph+Gora in ApacheCon14
Renato Javier Marroquín Mogrovejo
 
PPTX
Introduction to Apache Spark and MLlib
pumaranikar
 
PDF
Reactive dashboard’s using apache spark
Rahul Kumar
 
PPTX
Seattle spark-meetup-032317
Nan Zhu
 
PDF
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
Databricks
 
PDF
Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017
Alluxio, Inc.
 
PPTX
Introduction to Apache Spark
Rahul Jain
 
PPTX
Machine learning com Apache Spark
Sandys Nunes
 
Introduction to Machine Learning with Spark
datamantra
 
MLlib: Spark's Machine Learning Library
jeykottalam
 
Yace 3.0
Atul Ashar
 
MLlib and Machine Learning on Spark
Petr Zapletal
 
Large-Scale Machine Learning with Apache Spark
DB Tsai
 
Practical Machine Learning Pipelines with MLlib
Databricks
 
Machine Learning With Spark
Shivaji Dutta
 
Spark DataFrames and ML Pipelines
Databricks
 
Ingesting Drone Data into Big Data Platforms
Timothy Spann
 
R, Scikit-Learn and Apache Spark ML - What difference does it make?
Villu Ruusmann
 
Apache ® Spark™ MLlib 2.x: How to Productionize your Machine Learning Models
Anyscale
 
Yi Wang, Tech Lead of AI Platform, Baidu, at MLconf 2017
MLconf
 
Giraph+Gora in ApacheCon14
Renato Javier Marroquín Mogrovejo
 
Introduction to Apache Spark and MLlib
pumaranikar
 
Reactive dashboard’s using apache spark
Rahul Kumar
 
Seattle spark-meetup-032317
Nan Zhu
 
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
Databricks
 
Effective Spark with Alluxio at Strata+Hadoop World San Jose 2017
Alluxio, Inc.
 
Introduction to Apache Spark
Rahul Jain
 
Machine learning com Apache Spark
Sandys Nunes
 
Ad

More from Taras Matyashovsky (12)

PPTX
Morning 3 anniversary
Taras Matyashovsky
 
PPTX
Distinguish Pop from Heavy Metal using Apache Spark MLlib
Taras Matyashovsky
 
PPTX
Morning at Lohika 2nd anniversary
Taras Matyashovsky
 
PPTX
Confession of an Engineer
Taras Matyashovsky
 
PPTX
Influence. The Psychology of Persuasion (in IT)
Taras Matyashovsky
 
PPTX
JEEConf 2015 - Introduction to real-time big data with Apache Spark
Taras Matyashovsky
 
PPTX
Morning at Lohika 1st anniversary
Taras Matyashovsky
 
PPTX
Introduction to real time big data with Apache Spark
Taras Matyashovsky
 
PPTX
New life inside monolithic application
Taras Matyashovsky
 
PDF
Distributed applications using Hazelcast
Taras Matyashovsky
 
PPTX
Morning at Lohika
Taras Matyashovsky
 
PPTX
From cache to in-memory data grid. Introduction to Hazelcast.
Taras Matyashovsky
 
Morning 3 anniversary
Taras Matyashovsky
 
Distinguish Pop from Heavy Metal using Apache Spark MLlib
Taras Matyashovsky
 
Morning at Lohika 2nd anniversary
Taras Matyashovsky
 
Confession of an Engineer
Taras Matyashovsky
 
Influence. The Psychology of Persuasion (in IT)
Taras Matyashovsky
 
JEEConf 2015 - Introduction to real-time big data with Apache Spark
Taras Matyashovsky
 
Morning at Lohika 1st anniversary
Taras Matyashovsky
 
Introduction to real time big data with Apache Spark
Taras Matyashovsky
 
New life inside monolithic application
Taras Matyashovsky
 
Distributed applications using Hazelcast
Taras Matyashovsky
 
Morning at Lohika
Taras Matyashovsky
 
From cache to in-memory data grid. Introduction to Hazelcast.
Taras Matyashovsky
 

Recently uploaded (20)

PPT
Drone Technology Electronics components_1
EswaramoorthyKV
 
PDF
오픈소스 LLM, vLLM으로 Production까지 (Instruct.KR Summer Meetup, 2025)
Hyogeun Oh
 
PPTX
web development for engineering and engineering
keshriji700
 
PPTX
Geodesy 1.pptx...............................................
abhi1361yadav
 
PDF
ETO & MEO Certificate of Competency Questions and Answers
Mahmoud Moghtaderi
 
PPTX
MCN 401 KTU-2019-PPE KITS-MODULE 2.pptx
VinayB68
 
PPTX
KTU 2019 -S7-MCN 401 MODULE 2-VINAY.pptx
VinayB68
 
PPTX
IOT PPTs Week 10 Lecture Material.pptx of NPTEL Smart Cities contd
ssusera400e8
 
PPTX
Internship_Presentation_Final engineering.pptx
AbdullahTahir790840
 
PPTX
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
PDF
Structs to JSON How Go Powers REST APIs.pdf
Emily Achieng
 
PPTX
FINAL REVIEW FOR COPD DIANOSIS FOR PULMONARY DISEASE.pptx
rubeshbme
 
PPTX
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
DOCX
573137875-Attendance-Management-System-original
AYUSHMANAV
 
PPTX
The-Looming-Shadow-How-AI-Poses-Dangers-to-Humanity.pptx
shravanidabhane8
 
PPTX
Internet of Things (IOT) - A guide to understanding
Davies Chacko
 
PDF
Evaluating the Democratization of the Turkish Armed Forces from a Normative P...
jpsjournal1
 
PDF
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
smceramu
 
PPT
Chapter 6 Design in software Engineeing.ppt
RahulBhole12
 
PDF
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 
Drone Technology Electronics components_1
EswaramoorthyKV
 
오픈소스 LLM, vLLM으로 Production까지 (Instruct.KR Summer Meetup, 2025)
Hyogeun Oh
 
web development for engineering and engineering
keshriji700
 
Geodesy 1.pptx...............................................
abhi1361yadav
 
ETO & MEO Certificate of Competency Questions and Answers
Mahmoud Moghtaderi
 
MCN 401 KTU-2019-PPE KITS-MODULE 2.pptx
VinayB68
 
KTU 2019 -S7-MCN 401 MODULE 2-VINAY.pptx
VinayB68
 
IOT PPTs Week 10 Lecture Material.pptx of NPTEL Smart Cities contd
ssusera400e8
 
Internship_Presentation_Final engineering.pptx
AbdullahTahir790840
 
CYBER-CRIMES AND SECURITY A guide to understanding
Davies Chacko
 
Structs to JSON How Go Powers REST APIs.pdf
Emily Achieng
 
FINAL REVIEW FOR COPD DIANOSIS FOR PULMONARY DISEASE.pptx
rubeshbme
 
Foundation to blockchain - A guide to Blockchain Tech
Davies Chacko
 
573137875-Attendance-Management-System-original
AYUSHMANAV
 
The-Looming-Shadow-How-AI-Poses-Dangers-to-Humanity.pptx
shravanidabhane8
 
Internet of Things (IOT) - A guide to understanding
Davies Chacko
 
Evaluating the Democratization of the Turkish Armed Forces from a Normative P...
jpsjournal1
 
SM_6th-Sem__Cse_Internet-of-Things.pdf IOT
smceramu
 
Chapter 6 Design in software Engineeing.ppt
RahulBhole12
 
Operating System & Kernel Study Guide-1 - converted.pdf
mranoninvisib16
 

Introduction to ML with Apache Spark MLlib

Editor's Notes

  • #19: Score of the speaker based on xxx.
  • #26: Quantity of jokes used. Liked or not liked the speaker.
  • #34: Assign or index each example to the cluster centroid closest to it Recalculate or move centroids as an average (mean) of examples assigned to a cluster Repeat until centroids not longer move
  • #41: Bag of words – a single word is a one hot encoding vector with the size of the dictionary. As a result – a lot of sparse vectors.
  • #42: Behind the scenes - a two-layer neural net that processes text. Captures semantic and morphologic similarity so similar words are close in the vector space Similar words would be clustered together in the high dimensional sphere. 
  • #43: If two words are very close to synonymous, you’d expect them to show up in similar contexts, and indeed synonymous words tend to be close. For two completely random words, the similarity is pretty close to 0. On an opposite side there is not an antonym, but usually just a noise. Used Google News Negative 300.
  • #44: My corpus - 8316 words
  • #53: Let’s finally go to the implementation using a library or framework that is going to help us to avoid tedious transformations and provide algorithms as well as feature extractors out-of-the-box.