Applying Machine learning to IOT: End to End Distributed Distributed Pipeline for Realtime Uber Montoring Dashboard using Apache APIs: Kafka Spark HBase

© 2017 MapR Technologies
Applying Machine Learning to IOT:
End to End Distributed Pipeline for Real-
Time Uber Data Using Apache APIs: Kafka,
Spark, HBase
Carol McDonald
@caroljmcdonald

Use Case: Real-Time Analysis of Geographically Clustered Vehicles
Uber trip data enrich with K-means
Cluster location
Stream
Topic
Stream
Topic
Spark
Streaming
Spark
Streaming
Write to
MapR-DB
SQL

Agenda
Using An End to End Distributed Pipeline for Real-Time Uber Data Using Apache
APIs: Kafka, Spark, Hbase we will discuss:
•  Why IOT and Streaming events?
•  Why combine Machine Learning with IOT?
•  What is Machine Learning? How do you do it with spark?
•  What is Streaming?
•  Why Kafka (-ish –esque) Distributed Immutable Log ?
•  How to do Spark Streaming?
•  Why Kafka + WebSockets?
•  Why NoSQL HBase?
Note: this code example is from me, only the data is from Uber

Why IOT? Lots of Things are Producing Streaming Data
Data Collection
Devices
Smart Machinery Phones and Tablets Home Automation
RFID Systems Digital Signage Security Systems Medical Devices

What’s a Stream ?
Producers ConsumersEvents_Stream
A stream is an unbounded sequence of events carried
from a set of producers to a set of consumers.
Events

Why Stream Processing?
6:05 P.M.: 90°
To
pic
Stream
Temperature
Turn on the
air
conditioning!
It’s becoming important to process events as they arrive

Why combine IOT with Machine Learning?
•  Audi and Daimler harness the power of
deep learning in order to achieve their
goal of building autonomous vehicles
–  Using MapR platform to scale deep learning
efforts https://mapr.com/company/press-releases/norcom-selects-
mapr-deep-learning/
•  Audi's new A8 takes us further down the
road to self-driving cars than ever before
–  https://www.cnet.com/roadshow/news/audis-new-a8-is-
designed-to-let-you-play-candy-crush-in-rush-hour-traffic-safely/

•  Cheaper sensors and machine learning are making it possible for doctors to
rapidly apply smart medicine to their patients’ cases
–  https://www.wsj.com/articles/the-smart-medicine-solution-to-the-health-care-
crisis-1499443449

•  A Stanford team has shown that a machine-learning model can identify heart
arrhythmias from an electrocardiogram (ECG) better than an expert
–  https://www.technologyreview.com/s/608234/the-machines-are-getting-ready-to-play-doctor/

Applying Machine Learning to Live Patient Data
•  https://www.slideshare.net/caroljmcdonald/applying-machine-learning-to-
live-patient-data

•  Connected care ensuring quicker Sepsis treatment:
–  Blood pressures, pulse rates and oxygen levels from monitoring devices
combined with algorithms to automatically calculate a score, and provide
alerts
–  http://www.computerweekly.com/news/450422258/Putting-sepsis-algorithms-into-electronic-
patient-records

•  Smart Cities will be using
1.39 billion connected
cars, IoT sensors, and
devices by 2020
•  http://www.cisco.com/c/en/us/solutions/
industries/smart-connected-communities.html

Why combine Streaming Events with Machine Learning?
Fraud detection Smart Machinery Utility Smart Meters Home Automation
Networks Manufacturing Security Systems Patient Monitoring

What if BP had detected problems before the oil hit the water ?
•  1M samples/sec
•  High performance at
scale is necessary!

•  Uber Near Realtime Price Surging
–  https://www.slideshare.net/ConfluentInc/kafka-uber-
the-worlds-realtime-transit-infrastructure-aaron-
schildkrout
•  machine learning & geolocation data is being
used in:
–  telecom, travel, marketing, and manufacturing
–  identify patterns and trends:
–  recommendations, anomaly detection, and fraud.
NEAR REALTIME
PRICE SURGING

Part 1: Spark Machine Learning
•  End to End Application for Monitoring Uber Data using Spark ML
•  https://mapr.com/blog/monitoring-real-time-uber-data-using-spark-machine-
learning-streaming-and-kafka-api-part-1/

What is Machine Learning?
Data Build ModelTrain Algorithm
Finds patterns
New Data Use Model
(prediction function)
Predictions
Contains patterns Recognizes patterns

ML Discovery Model Building
Model
Training/
Building
Training
Set
Test Model
Predictions
Test
Set
Evaluate Results
Historical
Data
Deployed
Model
Insights
Data
Discovery,
Model
Creation
Production
Feature Extraction
Feature
Extraction
●  Churn Modelling
Uber
trips
Stream
TopicUber
trips
New Data

End to End Application Architecture

What is Supervised Machine Learning?
Supervised
•  Classification
–  Naïve Bayes
–  SVM
–  Random Decision
Forests
•  Regression
–  Linear
–  Logistic
Machine Learning
Unsupervised
•  Clustering
–  K-means
•  Dimensionality reduction
–  Principal Component
Analysis
–  SVD

Supervised Algorithms use labeled data
Data
features
Build Model
New Data
features
Predict
Use Model
X1, X2
Y
f(X1, X2) =Y
X1, X2
Y

Supervised Machine Learning: Classification & Regression
Classification
Identifies
category for item

Classification: Definition
Form of ML that:
•  Identifies which category an item belongs to
•  Uses supervised learning algorithms
–  Data is labeled
Sentiment

If it Walks/Swims/Quacks Like a Duck …… Then It Must Be a Duck
swims
walks
quacks
Features:
walks
quacks
swims
Features:

Debit Card Fraud Example
•  What are we trying to predict?
–  This is the Label or Target outcome:
–  Fraud or Not Fraud
•  What are the “if questions” or properties we can use to predict?
–  These are the Features:
–  Is the amount spent today > historical average?
–  Are there new state purchases compared to the last 3 months?
–  Are there foreign purchases today compared to the last 3 months?
–  Are the number of transactions today > historical average?
–  Are the number of new merchant types today high compared to the last 3 months?
–  Are there multiple purchases today from merchants with a category code of risk?
–  Is there unusual signing activity today compared to historically using pin?

Decision Tree For Classification
•  Tree of decisions about features
•  IF THEN ELSE questions using
features at each tree node
•  Answers branch to child nodes
Is the amount spent in 24
hours > average
Is the number of
states used from > 2
Are there multiple
Purchases today from
risky merchants?
YES NO
NoYES
Fraud
90%
Not Fraud
50%
Fraud
90%
Not Fraud
30%
YES No

Real Time Credit Card Fraud Detection with Apache Spark Streaming
1.  Get event credit card
transaction data
2.  Read card holder profile
3.  Calculate history
features
4.  Publish Alerts for fraud
and enriched events
https://mapr.com/blog/real-time-credit-card-fraud-detection-apache-spark-and-event-streaming/

What is Unsupervised Machine Learning?
Machine Learning
Unsupervised
•  Clustering
–  K-means
•  Dimensionality reduction
–  Principal Component
Analysis
–  SVD
Supervised
•  Classification
–  Naïve Bayes
–  SVM
–  Random Decision
Forests
•  Regression
–  Linear
–  Logistic

Unsupervised Algorithms use Unlabeled data
Customer GroupsBuild ModelTrain Algorithm
Finds patterns
New Customer
Purchase Data
Use Model
Similar Customer Group
Contains patterns Recognizes patterns
Customer purchase
data

Unsupervised Machine Learning: Clustering
Clustering
group news articles into different categories

Clustering: Definition
•  Unsupervised learning task
•  Groups objects into clusters of high similarity

Clustering: Definition
•  Unsupervised learning task
•  Groups objects into clusters of high similarity
–  Search results grouping
–  Grouping of customers, patients
–  Text categorization
–  recommendations
•  Anomaly detection: find what’s not similar

Clustering: Example
•  Group similar objects

Clustering: Example
•  Use MLlib K-means algorithm
1.  Initialize coordinates to center
of clusters (centroid)
x
x
x
x
x

Clustering: Example
2.  Assign all points to nearest
centroid
x
x
x
x
x

Clustering: Example
centroid
3.  Update centroids to center of
points
x
x
x
x
x

Clustering: Example
centroid
3.  Update centroids to center of
points
4.  Repeat until conditions met
x
x
x
x
x

Cluster Uber Trip Locations

Uber Data
•  Date/Time: The date and time of the Uber pickup
•  Lat: The latitude of the Uber pickup
•  Lon: The longitude of the Uber pickup
•  Base: The TLC base company affiliated with the Uber pickup
The Data Records are in CSV format. An example line is shown below:
•  2014-08-01 00:00:00,40.729,-73.9422,B02598

Uber Example
•  What are the “if questions” or
properties we can use to group?
–  These are the Features:
–  We will group by Lattitude,
longitude
•  Use Spark SQL to analyze: Day of
the week, time, rush hour …
NEAR REALTIME
PRICE SURGING

Spark ML workflow

Zeppelin Notebook with Spark
Data
Engineer
Data
Scientist

Load the data into a Dataframe: Define the Schema
case class Uber(dt: String, lat: Double, lon: Double, base: String)
val schema = StructType(Array(
StructField("dt", TimestampType, true),
StructField("lat", DoubleType, true),
StructField("lon", DoubleType, true),
StructField("base", StringType, true)
))
Input Comma Separated Values:
datetime, lattitude, longitude, base
2014-08-01 00:00:00,40.729,-73.9422,B02598

Data
Frame
Load data
Load the data into a Dataset
val train: Dataset[Uber] = spark.read.option("inferSchema", "false")
.schema(schema).csv(”uber.csv").as[Uber]

Dataset merged with Dataframe
•  in Spark 2.0, DataFrame APIs merged with Datasets APIs
•  A Dataset is a collection of typed objects
•  A DataFrame is a Dataset of generic Row objects

Spark Distributed Datasets
Dataset
W
Executor
P4
W
Executor
P1 P3
W
Executor
P2
partitioned
Partition 1
8213034705, 95,
2.927373,
jake7870, 0……
Partition 2
8213034705,
115, 2.943484,
Davidbresler2,
1….
Partition 3
8213034705,
100, 2.951285,
gladimacowgirl,
58…
Partition 4
8213034705,
117, 2.998947,
daysrus, 95….
•  Read only collection of typed objects
•  Partitioned across a cluster
•  Operated on in parallel
•  in memory can be Cached

Spark Distributed Datasets
Spark Distributed Dataset
•  Read only collection of elements
•  Partitioned across a cluster
•  Operated on in parallel
•  Cached in memory

Extract the Features
Image reference O’Reilly Learning Spark
+
+
̶+
̶ ̶
Feature Vectors Model
Featurization Training
Model
Evaluation
Best Model
Training Data
+
+
̶+
̶ ̶
+
+
̶+
̶ ̶
+
+
̶+
̶ ̶
+
+
̶+
̶ ̶
Feature Vectors are vectors of numbers representing the value for each feature

Data
Frame
Load data Add column DataFrame +
Features
Use VectorAssembler to put features in vector column
val featureCols = Array("lat", "lon")
val assembler = new VectorAssembler()
.setInputCols(featureCols)
.setOutputCol("features")

Data
Frame
Load data transform
Estimator
val kmeans = new KMeans()
.setK(8)
.setFeaturesCol("features")
.setMaxIter(5)
Create Kmeans Estimator, Set Features
DataFrame +
Features

Data
Frame
Load data transform
Estimator
val Array(trainingData, testData) = df2.randomSplit(Array(0.7, 0.3), 5043)
val model = kmeans.fit(trainingData)
Fit the Model on the Training Data Features
DataFrame +
Features
fit fitted
model
input

Data
Frame
Load data transform
Estimator
model.clusterCenters.foreach(println)
[40.76930621976264,-73.96034885367698]
[40.67562793272868,-73.79810579052476]
[40.68848772848041,-73.9634449047477]
[40.78957777777776,-73.14270740740741]
[40.32418330308531,-74.18665245009073]
[40.732808848486286,-74.00150153727878]
[40.75396549974632,-73.57692359208531]
[40.901700842900674,-73.868760398198]
Clusters from fitted model
DataFrame +
Features
fit fitted
model
input

fitted
model
Transform test data, adds column with Clusters
transform
features
val clusters = model.transform(testdata)
prediction
DataFrame +
Features
DataFrame +
Features +
prediciton

Kafka API and Streaming Data

Part 2: MapR Event Streams with Kafka API and Spark Streaming
•  https://mapr.com/blog/monitoring-real-time-uber-data-using-spark-machine-
learning-streaming-and-kafka-api-part-2/

Serve DataStore DataCollect Data
What Do We Need to Do ?
Process DataData Sources
? ? ? ?

Collect the Data
Data Ingest
MapR-FS
Source
Stream
Topic
•  Data Ingest:
–  Network Based: MapR Event
Streams, Kafka, Kinesis, Twitter,
Sockets...
–  File Based: NFS with MapR-FS,
HDFS

Organize Data into Topics with MapR Streams
Topics Organize Events into Categories and Decouple Producers from Consumers
Consumers
MapR Cluster
Topic: Pressure
Topic: Temperature
Topic: Warnings
Consumers
Consumers
Kafka API Kafka API

Scalable Messaging with MapR Streams
Server 1
Partition1: Topic - Pressure
Partition1: Topic - Temperature
Partition1: Topic - Warning
Server 2
Server 3
Topics are
partitioned for
throughput and
scalability

Producers are load
balanced between partitions
Kafka API

Consumers
Consumers
Consumers
Consumer groups can read in parallel
Kafka API

Partition is like a Queue
Consumers
MapR Cluster
Topic: Admission / Server 1
Consumers
Consumers
Partition
1
New Messages are
appended to the end
Partition
2
Partition
3
6 5 4 3 2 1
3 2 1
5 4 3 2 1
Producers
Producers
Producers
New
Message
6 5 4 3 2 1
Old
Message

Events are delivered in the order they are received, like a queue
messages are delivered in the order they are received
MapR Cluster
6 5 4 3 2 1
Consumer
groupProducers
Read cursors
Consumer
group

Unlike a queue, events are persisted even after they’re delivered
Messages remain on the partition, available to other consumers
Minimizes Non-Sequential disk read-writes
MapR Cluster (1 Server)
Topic: Warning
Partition
1
3 2 1 Unread Events
Get Unread
3 2 1
Client Library ConsumerPoll

How do we do this with High Performance at Scale?
•  Parallel operations
•  minimize disk read/write time

Processing Same Message for Different Purposes
Consumers
Consumers
Consumers
Producers
Producers
Producers
MapR-FS
Kafka API Kafka API

Use the Model with Streaming Data

Collect Data
Process the Data with Spark Streaming and Spark Machine Learning
Process Data
Stream
Topic
•  Extension of the core Spark AP
•  Enables scalable, high-throughput,
fault-tolerant stream processing of
live data

ML Discovery Model Building
Model
Training/
Building
Training
Set
Test Model
Predictions
Test
Set
Evaluate Results
Historical
Data
Deployed
Model
Insights
Data
Discovery,
Model
Creation
Production
Feature Extraction
Feature
Extraction
Uber
trips
Stream
TopicUber
trips
New Data

Use Case: Time Series Data
Uber trip data
Stream
Topic
2014-08-01 00:00:00,
40.729,-73.9422,B02598
{"dt":"2014-08-01 00:00:00.0”,
"lat":40.3495,"lon":-74.0667,
"base":"B02682","cluster":5}
Enrich with
K-means cluster id
Spark
Streaming
read
Stream
Topic

Processing Spark DStreams
Data stream divided into batches of X milliseconds = DStreams

Function to Parse the Message Data to Uber Objects
2014-08-01 00:00:00, 40.729,-73.9422,B02598

Load the saved model
// load model for getting clusters
val model = KMeansModel.load(modelpath)

Create a DStream
DStream: a sequence of RDDs
representing a stream of data
val messagesDStream = KafkaUtils.createDirectStream[String,String]
(ssc, LocationStrategies.PreferConsistent,consumerStrategy)
// get message values from key,value and parse to Uber objects
val uDStream = linesDStream.map(_.value())
batch
time 0 to 1
batch
time 1 to 2
batch
time 2 to 3
dStream
Stored in memory
as an RDD

Parse message txt to Uber Object and convert to DataFrame
uDStream.foreachRDD{ rdd =>
val df = rdd.map(parseUber).toDF()
// get cluster centers and add to df
// send to Topic
}
ssc.start()
ssc.awaitTermination()

Enrich Data with Cluster

Convert to JSON send to Topic, Send the Enriched Message

Process Dstream Streaming Applicaton Output
dStream
batch
time 2 to 3
batch
time 1 to 2
batch
time 0 to 1
Result Dstream
Transformed RDDs
map map map
Stream
Topic

Real Time Dashboard

Part 3: Realtime Dashboard using Vert.x
•  https://mapr.com/blog/monitoring-uber-with-spark-streaming-kafka-and-
vertx/

Serve DataCollect Data
Serving the Data
MapR-FS
Stream
Topic

The Vert.x toolkit and Web Application Architecture
•  Event-driven
•  Event Bus
•  Verticles single threaded

Use Case Dashboard

Dashboard Architecture

Create a Vert.x Service
create a Router object, which routes HTTP request URLs to handlers

Route paths that match /eventbus/* to be associated with an
event bus bridge SockJSHandler

create an HttpServer object
tell the server to listen on the configured port for incoming
requests

Vert.x Service Kafka consumer

Create Kafka Consumer
Subscribe to Uber topic

Publish received messages to the Vert.x event bus address
“dashboard.”

The Dashboard Vert.x HTML5 Javascript Client

Javascript packages

Initializing the Heatmap

Creating the Vertx EventBus
•  create an instance of the vertx.EventBus object
•  add an onopen listener, which registers an event bus handler for the
address “dashboard.”
•  handler will receive all messages published to the “dashboard” address

Add Event Trip location points to Map

Parse JSON message

Add lattitude and longitude points to heatmap

If cluster center is new then add marker

Spark and HBase

Part 4: using MapR-DB with HBase API
•  https://mapr.com/blog/monitoring-uber-pt4/

Serve DataStore DataCollect Data
What Do We Need to Do ?
MapR-FS
MapR-FS
Stream
Topic

MapR-DB (HBase API) is Designed to Scale
Key
Range
xxxx
xxxx
Key
Range
xxxx
xxxx
Key
Range
xxxx
xxxx
Fast Reads and Writes by Key! Data is automatically partitioned
by Key Range!
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val

Store Lots of Data with NoSQL MapR-DB
bottleneck
Storage ModelRDBMS MapR-DB
Normalized schema à Joins for
queries can cause bottleneck De-Normalized schema à Data that
is read together is stored together
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val
Key colB colC
xxx val val
xxx val val

Spark Streaming writing to MapR-DB (HBase API)

Spark HBase and MapR-DB Binary Connector
•  HConnection object in every Spark Executor:
•  allowing for distributed parallel writes, reads, or scans

Spark Hbase streamBulkPut
•  HBaseContext streamBulkPut method parameters:
•  message value DStream, the TableName to write to, function to convert the Dstream
values to HBase put records.

Massively Parrallel writes to HBase
The Spark Streaming bulk put enables massively parallel sending of puts to HBase

HBase Schema
To use the Spark HBase Connector, you need to define the Catalog for the schema
mapping between the HBase and Spark

SparkSQL and DataFrames: Define the Schema
define the Catalog for the schema mapping between the HBase and Spark

Loading data from MapR-DB into a Spark DataFrame
Use Catalog defining schema

Spark Dataframes combine filters and select
filters rows for cluster ids (the beginning of the row key) >= 9. The select selects a
set of columns: key, lat, and lon.

To Learn More:
•  MapR Free ODT http://learn.mapr.com/

MapR Blog
• https://www.mapr.com/blog/

…helping you put data technology to work
●  Find answers
●  Ask technical questions
●  Join on-demand training course
discussions
●  Follow release announcements
●  Share and vote on product ideas
●  Find Meetup and event listings
Connect with fellow Apache
Hadoop and Spark professionals
community.mapr.com

Open Source Engines & Tools Commercial Engines & Applications
Enterprise-Grade Platform Services
DataProcessing
Web-Scale Storage
MapR-XD MapR-DB
Search
and
Others
Real Time Unified Security Multi-tenancy Disaster
Recovery
Global NamespaceHigh Availability
MapR Evemt Streams
Cloud
and
Managed
Services
Search and
Others
UnifiedManagementandMonitoring
Search
and
Others
Event StreamingDatabase
Custom
Apps
MapR Converged Data Platform
HDFS API POSIX, NFS Kakfa APIHBase API OJAI API

Stream Processing
Building a Complete Data Architecture
MapR File System
(MapR-XD)
MapR Converged Data Platform
MapR Database
(MapR-DB)
MapR Event Streams
Sources/Apps Bulk Processing

Q&A
ENGAGE WITH US

Applying Machine learning to IOT: End to End Distributed Distributed Pipeline for Realtime Uber Montoring Dashboard using Apache APIs: Kafka Spark HBase

Recommended

More Related Content

What's hot (20)

Similar to Applying Machine learning to IOT: End to End Distributed Distributed Pipeline for Realtime Uber Montoring Dashboard using Apache APIs: Kafka Spark HBase (20)

More from Carol McDonald (10)

Recently uploaded (20)

Applying Machine learning to IOT: End to End Distributed Distributed Pipeline for Realtime Uber Montoring Dashboard using Apache APIs: Kafka Spark HBase