Machine Learning using Apache Spark MLlib

thanachart@imcinstitute.com1
Hands-on: Exercise
Machine Learning using
Apache Spark MLlib
July 2016
Dr.Thanachart Numnonda
IMC Institute
thanachart@imcinstitute.com

What is MLlib?
Source: MapR Academy

MLlib is a Spark subproject providing machine
learning primitives:
– initial contribution from AMPLab, UC Berkeley
– shipped with Spark since version 0.8
– 33 contributors
What is MLlib?

Classification: logistic regression, linear support vector
machine(SVM), naive Bayes
Regression: generalized linear regression (GLM)
Collaborative filtering: alternating least squares (ALS)
Clustering: k-means
Decomposition: singular value decomposition (SVD),
principal component analysis (PCA)
Mllib Algorithms

What is in MLlib?
Source: Mllib:Spark's Machine Learning Library, A. Talwalkar

Part of Spark
Scalable
Support: Python, Scala, Java
Broad coverage of applications & algorithms
Rapid developments in speed & robustness
MLlib: Benefits

Thanachart Numnonda, thanachart@imcinstitute.com June 2016Apache Spark : Train the trainer
Machine Learning
Machine learning is a scientific discipline that
explores the construction and study of algorithms
that can learn from data.
[Wikipedia]

A point is just a set of numbers. This set of numbers or
coordinates defines the point's position in space.
Points and vectors are same thing.
Dimensions in vectors are called features
Hyperspace is a space with more than three dimensions.
Example: A person has the following dimensions:
– Weight
– Height
– Age
Thus, the interpretation of point (160,69,24) would be
160 lb weight, 69 inches height, and 24 years age.
Vectors
Source:Spark Cookbook

Spark has local vectors and matrices and also distributed
matrices.
– Distributed matrix is backed by one or more RDDs.
– A local vector has numeric indices and double values, and is
stored on a single machine.
Two types of local vectors in MLlib:
– Dense vector is backed by an array of its values.
– Sparse vector is backed by two parallel arrays, one for indices
and another for values.
Example
– Dense vector: [160.0,69.0,24.0]
– Sparse vector: (3,[0,1,2],[160.0,69.0,24.0])
Vectors in MLlib

Library
– import org.apache.spark.mllib.linalg.{Vectors,Vector}
Signature of Vectors.dense:
– def dense(values: Array[Double]): Vector
Signature of Vectors.sparse:
– def sparse(size: Int, indices: Array[Int], values: Array[Double]):
Vector
Vectors in Mllib (cont.)

Example

Labeled point is a local vector (sparse/dense), ), which
has an associated label with it.
Labeled data is used in supervised learning to help train
algorithms.
Label is stored as a double value in LabeledPoint.
Labeled point

Example
scala> import org.apache.spark.mllib.linalg.{Vectors,Vector}
scala> import org.apache.spark.mllib.regression.LabeledPoint
scala> val willBuySUV =
LabeledPoint(1.0,Vectors.dense(300.0,80,40))
scala> val willNotBuySUV =
LabeledPoint(0.0,Vectors.dense(150.0,60,25))
scala> val willBuySUV =
LabeledPoint(1.0,Vectors.sparse(3,Array(0,1,2),Array(300.0,80,
40)))
scala> val willNotBuySUV =
LabeledPoint(0.0,Vectors.sparse(3,Array(0,1,2),Array(150.0,60,
25)))

Example (cont)
# vi person_libsvm.txt
scala> import org.apache.spark.mllib.util.MLUtils
scala> import org.apache.spark.rdd.RDD
scala> val persons =
MLUtils.loadLibSVMFile(sc,"hdfs:///user/cloudera/person_libsvm
.txt")
scala> persons.first()

Spark has local matrices and also distributed matrices.
– Distributed matrix is backed by one or more RDDs.
– A local matrix stored on a single machine.
There are three types of distributed matrices in MLlib:
– RowMatrix: This has each row as a feature vector.
– IndexedRowMatrix: This also has row indices.
– CoordinateMatrix: This is simply a matrix of MatrixEntry. A
MatrixEntry represents an entry in the matrix represented by its
row and column index
Matrices in MLlib

Example
scala> import org.apache.spark.mllib.linalg.{Vectors,Matrix,
Matrices}
scala> val people = Matrices.dense(3,2,Array(150d,60d,25d,
300d,80d,40d))
scala> val personRDD =
sc.parallelize(List(Vectors.dense(150,60,25),
Vectors.dense(300,80,40)))
scala> import org.apache.spark.mllib.linalg.distributed.
{IndexedRow, IndexedRowMatrix,RowMatrix, CoordinateMatrix,
MatrixEntry}
scala> val personMat = new RowMatrix(personRDD)

Example
scala> print(personMat.numRows)
scala> val personRDD = sc.parallelize(List(IndexedRow(0L,
Vectors.dense(150,60,25)), IndexedRow(1L,
Vectors.dense(300,80,40))))
scala> val pirmat = new IndexedRowMatrix(personRDD)
scala> val personMat = pirmat.toRowMatrix
scala> val meRDD = sc.parallelize(List(
MatrixEntry(0,0,150), MatrixEntry(1,0,60),
MatrixEntry(2,0,25), MatrixEntry(0,1,300),
MatrixEntry(1,1,80),MatrixEntry(2,1,40) ))
scala> val pcmat = new CoordinateMatrix(meRDD)

Central tendency of data—mean, mode, median
Spread of data—variance, standard deviation
Boundary conditions—min, max
Statistic functions

Example
scala> import org.apache.spark.mllib.linalg.{Vectors,Vector}
scala> import org.apache.spark.mllib.stat.Statistics
scala> val personRDD =
sc.parallelize(List(Vectors.dense(150,60,25),
Vectors.dense(300,80,40)))
scala> val summary = Statistics.colStats(personRDD)

Hands-on
Movie Recommendation

Recommendation

Recommendation: Collaborative Filtering

Recommendation

Recommendation: ALS

Source: MLlib: Scalable Machine Learning on Spark, X. Meng
Alternating least squares (ALS)

numBlocks is the number of blocks used to parallelize
computation (set to -1 to autoconfigure)
rank is the number of latent factors in the model
iterations is the number of iterations to run
lambda specifies the regularization parameter in ALS
implicitPrefs specifies whether to use the explicit feedback
ALS variant or one adapted for an implicit feedback data
alpha is a parameter applicable to the implicit feedback
variant of ALS that governs the baseline confidence in
preference observations
MLlib: ALS Algorithm

MovieLen Dataset
1)Type command > wget
http://files.grouplens.org/datasets/movielens/ml-100k.zip
2)Type command > yum install unzip
3)Type command > unzip ml-100k.zip
4)Type command > more ml-100k/u.user

Moving dataset to HDFS
1)Type command > cd ml-100k
2)Type command > hadoop fs -mkdir /user/cloudera/movielens
3)Type command > hadoop fs -put u.user /user/cloudera/movielens
4)Type command > hadoop fs -put u.data /user/cloudera/movielens
4)Type command > hadoop fs -put u.genre /user/cloudera/movielens
5)Type command > hadoop fs -put u.item /user/cloudera/movielens
6)Type command > hadoop fs -ls /user/cloudera/movielens

Start Spark-shell with extra memory

Extracting features from the MovieLens dataset
scala> val rawData =
sc.textFile("hdfs:///user/cloudera/movielens/u.data")
scala> rawData.first()
scala> val rawRatings = rawData.map(_.split("t").take(3))
scala> rawRatings.first()
scala> import org.apache.spark.mllib.recommendation.Rating
scala> val ratings = rawRatings.map { case Array(user, movie,
rating) =>Rating(user.toInt, movie.toInt, rating.toDouble) }
scala> ratings.first()

Training the recommendation model
scala> import org.apache.spark.mllib.recommendation.ALS
scala> val model = ALS.train(ratings, 50, 10, 0.01)
Note: We'll use rank of 50, 10 iterations, and a lambda parameter of 0.01

Inspecting the recommendations
scala> val movies =
sc.textFile("hdfs:///user/cloudera/movielens/u.item")
scala> val titles = movies.map(line =>
line.split("|").take(2)).map(array
=>(array(0).toInt,array(1))).collectAsMap()

Inspecting the recommendations (cont.)
scala> val moviesForUser = ratings.keyBy(_.user).lookup(789)
scala> moviesForUser.sortBy(-_.rating).take(10).map(rating =>
(titles(rating.product), rating.rating)).foreach(println)

Top 10 Recommendation for userid 789
scala> val topKRecs = model.recommendProducts(789,10)
scala> topKRecs.map(rating => (titles(rating.product),
rating.rating)).foreach(println)

Evaluating Performance: Mean Squared Error
scala> val actualRating = moviesForUser.take(1)(0)
scala> val predictedRating = model.predict(789,
actualRating.product)
scala> val squaredError = math.pow(predictedRating -
actualRating.rating, 2.0)

Overall Mean Squared Error
scala> val usersProducts = ratings.map{ case Rating(user,
product, rating) => (user, product)}
scala> val predictions = model.predict(usersProducts).map{
case Rating(user, product, rating) => ((user, product),
rating)}
scala> val ratingsAndPredictions = ratings.map{
case Rating(user, product, rating) => ((user, product),
rating)
}.join(predictions)
scala> val MSE = ratingsAndPredictions.map{
case ((user, product), (actual, predicted)) =>
math.pow((actual - predicted), 2)
}.reduce(_ + _) / ratingsAndPredictions.count

Clustering using K-Means

Market segmentation
Social network analysis: Finding a coherent group of
people in the social network for ad targeting
Data center computing clusters
Real estate: Identifying neighborhoods based on similar
features
Text analysis: Dividing text documents, such as novels or
essays, into genres
Clustering use cases

thanachart@imcinstitute.com39Source: Mahout in Action

thanachart@imcinstitute.com40Source: Mahout in Action
Sample Data

Distance Measures
Source www.edureka.in/data-science

Euclidean distance measure
Squared Euclidean distance measure
Manhattan distance measure
Cosine distance measure
Distance Measures

Distance Measures

K-Means Clustering
Source: www.edureka.in/data-science

Example of K-Means Clustering

http://stanford.edu/class/ee103/visualizations/kmeans/kmeans.html

K-Means with different distance measures
Source: Mahout in Action

Choosing number of clusters

Dimensionality reduction
Process of reducing the number of dimensions or
features.
Dimensionality reduction serves several purposes
– Data compression
– Visualization
The most popular algorithm: Principal component
analysis (PCA).

Dimensionality reduction
Source: Spark Cookbook

Dimensionality reduction with SVD
Singular Value Decomposition (SVD): is based on a
theorem from linear algebra that a rectangular matrix A
can be broken down into a product of three matrices

Dimensionality reduction with SVD
The basic idea behind SVD
– Take a high dimension, a highly variable set of data
points
– Reduce it to a lower dimensional space that exposes
the structure of the original data more clearly and
orders it from the most variation to the least.
So we can simply ignore variation below a certain
threshold to massively reduce the original data, making
sure that the original relationship interests are retained.

Hands-on
Clustering on MovieLens Dataset

Extracting features from the MovieLens dataset
scala> val rawData =
sc.textFile("hdfs:///user/cloudera/movielens/u.item")
scala> println(movies.first)
scala> val genres =
sc.textFile("hdfs:///user/cloudera/movielens/u.genre")
scala> genres.take(5).foreach(println)

Extracting features from the MovieLens dataset (cont.)
scala> val genreMap = genres.filter(!_.isEmpty).map(line =>
line.split("|")).map(array=> (array(1),
array(0))).collectAsMap

Extracting features from the MovieLens dataset (cont.)
scala> val titlesAndGenres = movies.map(_.split("|")).map
{ array =>
val genres = array.toSeq.slice(5, array.size)
val genresAssigned = genres.zipWithIndex.filter { case (g,
idx) =>
g == "1"
}.map { case (g, idx) =>
genreMap(idx.toString)
}
(array(0).toInt, (array(1), genresAssigned))
}

Training the recommendation model
scala> :paste
import org.apache.spark.mllib.recommendation.ALS
import org.apache.spark.mllib.recommendation.Rating
val rawData =
sc.textFile("hdfs:///user/cloudera/movielens/u.data")
val rawRatings = rawData.map(_.split("t").take(3))
val ratings = rawRatings.map{ case Array(user, movie,
rating) => Rating(user.toInt, movie.toInt,
rating.toDouble) }
ratings.cache
val alsModel = ALS.train(ratings, 50, 10, 0.1)
import org.apache.spark.mllib.linalg.Vectors
val movieFactors = alsModel.productFeatures.map { case (id,
factor) => (id, Vectors.dense(factor)) }
val movieVectors = movieFactors.map(_._2)
val userFactors = alsModel.userFeatures.map { case (id,
factor) => (id, Vectors.dense(factor)) }
val userVectors = userFactors.map(_._2)

Normalization
scala> :paste
import org.apache.spark.mllib.linalg.distributed.RowMatrix
val movieMatrix = new RowMatrix(movieVectors)
val movieMatrixSummary =
movieMatrix.computeColumnSummaryStatistics()
val userMatrix = new RowMatrix(userVectors)
val userMatrixSummary =
userMatrix.computeColumnSummaryStatistics()
println("Movie factors mean: " + movieMatrixSummary.mean)
println("Movie factors variance: " +
movieMatrixSummary.variance)
println("User factors mean: " + userMatrixSummary.mean)
println("User factors variance: " +
userMatrixSummary.variance)

Output from Normalization

Training a clustering model
scala> import org.apache.spark.mllib.clustering.KMeans
scala> val numClusters = 5
scala> val numIterations = 10
scala> val numRuns = 3
scala> val movieClusterModel = KMeans.train(movieVectors,
numClusters, numIterations, numRuns)

Making predictions using a clustering model
scala> val movie1 = movieVectors.first
scala> val movieCluster = movieClusterModel.predict(movie1)
scala> val predictions =
movieClusterModel.predict(movieVectors)

Interpreting cluster predictions
scala> :paste
import breeze.linalg._
import breeze.numerics.pow
def computeDistance(v1: DenseVector[Double], v2:
DenseVector[Double]) = pow(v1 - v2, 2).sum
val titlesWithFactors = titlesAndGenres.join(movieFactors)
val moviesAssigned = titlesWithFactors.map { case (id,
((title, genres), vector)) =>
val pred = movieClusterModel.predict(vector)
val clusterCentre = movieClusterModel.clusterCenters(pred)
val dist =
computeDistance(DenseVector(clusterCentre.toArray),
DenseVector(vector.toArray))
(id, title, genres.mkString(" "), pred, dist)
}

Interpreting cluster predictions (cont.)
val clusterAssignments = moviesAssigned.groupBy { case (id,
title, genres, cluster, dist) => cluster }.collectAsMap
for ( (k, v) <- clusterAssignments.toSeq.sortBy(_._1)) {
println(s"Cluster $k:")
val m = v.toSeq.sortBy(_._5)
println(m.take(20).map { case (_, title, genres, _, d) =>
(title, genres, d) }.mkString("n"))
println("=====n")
}

Real-time Machine Learning
using Streaming K-Means

Online learning with Spark Streaming
Streaming regression
– trainOn: This takes DStream[LabeledPoint] as its
argument.
– predictOn: This also takes DStream[LabeledPoint].
Streaming KMeans
– An extension of the mini-batch K-means algorithm

Streaming K-Means Program

MovieLen Training Dataset
●
The rows of the training text files must be vector data
in the form
[x1,x2,x3,...,xn]
1)Type command > wget
https://s3.amazonaws.com/imcbucket/data/movietest.data
2)Type command > more movietest.data

Install & Start Kafka Server
# wget http://www-us.apache.org/dist/kafka/0.9.0.1/kafka_2.10-
0.9.0.1.tgz
# tar xzf kafka_2.10-0.9.0.1.tgz
# cd kafka_2.10-0.9.0.1
# bin/kafka-server-start.sh config/server.properties&

Streaming K-Means
$ scala> :paste
import org.apache.spark.mllib.linalg.Vectors
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.clustering.StreamingKMeans
import org.apache.spark.SparkConf
import org.apache.spark.streaming.{Seconds, StreamingContext}
import org.apache.spark.storage.StorageLevel
import StorageLevel._
import org.apache.spark._
import org.apache.spark.streaming._
import org.apache.spark.streaming.StreamingContext._
import org.apache.spark.streaming.kafka.KafkaUtils
val ssc = new StreamingContext(sc, Seconds(2))

val kafkaStream = KafkaUtils.createStream(ssc,
"localhost:2181","spark-streaming-consumer-group", Map("java-
topic" -> 5))
val lines = kafkaStream.map(_._2)
val ratings = lines.map(Vectors.parse)
val numDimensions = 3
val numClusters = 5
val model = new StreamingKMeans()
.setK(numClusters)
.setDecayFactor(1.0)
.setRandomCenters(numDimensions, 0.0)
model.trainOn(ratings)
model.predictOn(ratings).print()
ssc.start()
ssc.awaitTermination()

Running HelloKafkaProducer on another windows
●
Open a new ssh windows

Java Code: Kafka Producer
import java.util.Properties;
import kafka.producer.KeyedMessage;
import kafka.producer.ProducerConfig;
import java.io.*;
public class HelloKafkaProducer {
final static String TOPIC = "java-topic";
public static void main(String[] argv){
Properties properties = new Properties();
properties.put("metadata.broker.list","localhost:9092");
properties.put("serializer.class","kafka.serializer.StringEnco
der");

Java Code: Kafka Producer (cont.)
try(BufferedReader br = new BufferedReader(new
FileReader(argv[0]))) {
StringBuilder sb = new StringBuilder();
ProducerConfig producerConfig = new
ProducerConfig(properties);
kafka.javaapi.producer.Producer<String,String>
producer = new kafka.javaapi.producer.Producer<String,
String>(producerConfig);
String line = br.readLine();
while (line != null) {
KeyedMessage<String, String> message
=new KeyedMessage<String, String>(TOPIC,line);
producer.send(message);
line = br.readLine();
}

Java Code: Kafka Producer (cont.)
producer.close();
} catch (IOException ex) {
ex.printStackTrace();
}
}
}

Compile & Run the program
// Using a vi Editor to edit the sourcecode
# vi HelloKafkaProducer.java
// Alternatively
# wget
https://s3.amazonaws.com/imcbucket/apps/HelloKafkaProducer.java
// Compile progeram
# export CLASSPATH=".:/root/kafka_2.10-0.9.0.1/libs/*"
# javac HelloKafkaProducer.java
//prepare the data
# cd
# wget https://s3.amazonaws.com/imcbucket/input/pg2600.txt
# cd kafka_2.10-0.9.0.1
// Run the program
# java HelloKafkaProducer /root/movietest.data

Example Result

Recommended Books

Thank you
www.imcinstitute.com
www.facebook.com/imcinstitute

Machine Learning using Apache Spark MLlib

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Machine Learning using Apache Spark MLlib (20)

More from IMC Institute (20)

Recently uploaded (20)

Machine Learning using Apache Spark MLlib