Apache Spark Structured Streaming for Machine Learning - StrataConf 2016
5 likes1,171 views
The document discusses the integration of structured streaming with machine learning within Apache Spark, outlining the capabilities and experimental APIs being developed. It covers topics such as the use of datasets and continuous tables in structured streaming, as well as potential methods for building machine learning pipelines. Additionally, it highlights challenges and considerations for implementing streaming ML, alongside resources for further reading and community engagement.
![Using Datasets to mix functional & relational style:
val ds: Dataset[RawPanda] = ...
val happiness = ds.filter($"happy" === true).
select($"attributes"(0).as[Double]).
reduce((x, y) => x + y)](https://image.slidesharecdn.com/structuredstreamingformachinelearning-160930135414/85/Apache-Spark-Structured-Streaming-for-Machine-Learning-StrataConf-2016-8-320.jpg)
![So what was that?
ds.filter($"happy" === true).
select($"attributes"(0).as[Double]).
reduce((x, y) => x + y)
A typed query (specifies the
return type). Without the as[]
will return a DataFrame
(Dataset[Row])
Traditional functional
reduction:
arbitrary scala code :)
Robert Couse-Baker](https://image.slidesharecdn.com/structuredstreamingformachinelearning-160930135414/85/Apache-Spark-Structured-Streaming-for-Machine-Learning-StrataConf-2016-9-320.jpg)
![And functional style maps:
/**
* Functional map + Dataset, sums the positive attributes for the
pandas
*/
def funMap(ds: Dataset[RawPanda]): Dataset[Double] = {
ds.map{rp => rp.attributes.filter(_ > 0).sum}
}](https://image.slidesharecdn.com/structuredstreamingformachinelearning-160930135414/85/Apache-Spark-Structured-Streaming-for-Machine-Learning-StrataConf-2016-10-320.jpg)
![Streaming Estimator/Transformer (POC)
trait StreamingModel[S] extends Transformer {
def update(updates: S): Unit
}
trait StreamingEstimator[S] extends Estimator {
def model: StreamingModel[S]
def update(batch: Dataset[_]): Unit
}
Sufficient statistics for
model updates
Blinke
nArea](https://image.slidesharecdn.com/structuredstreamingformachinelearning-160930135414/85/Apache-Spark-Structured-Streaming-for-Machine-Learning-StrataConf-2016-25-320.jpg)
![And doing some ML with it:
def update(batch: Dataset[_]): Unit = {
val newCountsByClass = add(batch)
model.update(newCountsByClass)
} Aggregate new batch
Merge with previous aggregates](https://image.slidesharecdn.com/structuredstreamingformachinelearning-160930135414/85/Apache-Spark-Structured-Streaming-for-Machine-Learning-StrataConf-2016-28-320.jpg)
![And doing some ML with it*
(Algorithm specific)
def update(updates: Array[(Double, (Long, DenseVector))]): Unit = {
updates.foreach { case (label, (numDocs, termCounts)) =>
countsByClass.get(label) match {
case Some((n, c)) =>
axpy(1.0, termCounts, c)
countsByClass(label) = (n + numDocs, c)
case None =>
// new label encountered
countsByClass += (label -> (numDocs, termCounts))
}
}
}](https://image.slidesharecdn.com/structuredstreamingformachinelearning-160930135414/85/Apache-Spark-Structured-Streaming-for-Machine-Learning-StrataConf-2016-29-320.jpg)
![Working with the results - foreach (1 of 2)
val foreachWriter: ForeachWriter[T] =
new ForeachWriter[T] {
def open(partitionId: Long, version: Long): Boolean = {
True // always open
}
def close(errorOrNull: Throwable): Unit = {
// No close logic - if we wanted to copy updates per-batch
}
def process(record: T): Unit = {
db.update(record)
}
}](https://image.slidesharecdn.com/structuredstreamingformachinelearning-160930135414/85/Apache-Spark-Structured-Streaming-for-Machine-Learning-StrataConf-2016-31-320.jpg)
Apache Spark Structured Streaming for Machine Learning - StrataConf 2016
- 1. Structured Streaming For machine Learning! :) kroszk@ Built with experimental APIs :)
- 2. Holden: ● Prefered pronouns are she/her ● I’m a Principal Software Engineer at IBM’s Spark Technology Center ● previously Alpine, Databricks, Google, Foursquare & Amazon ● co-author of Learning Spark & Fast Data processing with Spark ○ co-author of a new book focused on Spark performance coming out this year* ● @holdenkarau ● Slide share http://www.slideshare.net/hkarau ● Linkedin https://www.linkedin.com/in/holdenkarau ● Github https://github.com/holdenk ● Spark Videos http://bit.ly/holdenSparkVideos
- 3. Seth: ● Machine learning engineer at IBM’s Spark Technology Center ○ Working on high-performance, distributed machine learning for Spark MLlib ○ Also, structured streaming! ● Previously electrical engineering ● @shendrickson16 ● Linkedin https://www.linkedin.com/in/sethah ● Github https://github.com/sethah ● SlideShare http://www.slideshare.net/SethHendrickson
- 4. What is going to be covered: ● Who we think y’all are ● Abridged Introduction to Datasets ● Abridged Introduction to Structured Streaming ● What Structured Streaming is and is not ● How to write simple structured streaming queries ● The exciting part: Building machine learning on top of structured streaming ● Possible future changes to make structured streaming & ML work together nicely Torsten Reuschling
- 5. Who we think you wonderful humans are? ● Nice* people ● Don’t mind pictures of cats ● Know some Apache Spark ● May or may not know the Dataset API ● Want to take advantage of Spark’s Structured Streaming ● May care about machine learning ● Possibly distracted with Pokemon GO
- 6. ALPHA =~ Please don’t use this Image by Mr Thinktank
- 7. What are Datasets? ● New in Spark 1.6 ● Provide templated compile time strongly typed version of DataFrames ● Make it easier to intermix functional & relational code ○ Do you hate writing UDFS? So do I! ● Still an experimental component (API will change in future versions) ● The basis of the Structured Streaming Houser Wolf
- 8. Using Datasets to mix functional & relational style: val ds: Dataset[RawPanda] = ... val happiness = ds.filter($"happy" === true). select($"attributes"(0).as[Double]). reduce((x, y) => x + y)
- 9. So what was that? ds.filter($"happy" === true). select($"attributes"(0).as[Double]). reduce((x, y) => x + y) A typed query (specifies the return type). Without the as[] will return a DataFrame (Dataset[Row]) Traditional functional reduction: arbitrary scala code :) Robert Couse-Baker
- 10. And functional style maps: /** * Functional map + Dataset, sums the positive attributes for the pandas */ def funMap(ds: Dataset[RawPanda]): Dataset[Double] = { ds.map{rp => rp.attributes.filter(_ > 0).sum} }
- 11. And now we can use it for streaming too! ● StructuredStreaming - new to Spark 2.0 ○ Emphasis on new - be cautious when using ● Extends the Dataset & DataFrame APIs to represent continuous tables ● Still very early stages - but lots of really cool optimizations possible now ● We can build a machine learning pipeline with it together :) ○ Well we have to use some hacks - but ssssssh don’t tell TD https://github.com/holdenk/spark-structured-streaming-ml
- 12. Get a streaming dataframe // Read a streaming dataframe val schema = new StructType() .add("happiness", "double") .add("coffees", "integer") val streamingDS = spark .readStream .schema(schema) .format(“parquet”) .load(path) Dataset isStreaming = true source scan
- 13. Build the recipe for each query val happinessByCoffee = streamingDS .groupBy($"coffees") .agg(avg($"happiness")) Dataset isStreaming = true source scan groupBy avg
- 14. Start a continuous query val query = happinessByCoffee .writeStream .format(“parquet”) .outputMode(“complete”) .trigger(ProcessingTime(5.seconds)) .start() StreamingQuery source scan groupBy avglogicalPlan =
- 15. Launch a new thread to listen for new data Source Available Offsets: Sink Committed Offsets: MicroBatch Thread StreamingQuery source scan groupBy avglogicalPlan = Listening
- 16. Write new offsets to WAL Source Available Offsets: 0, 1, 2 Sink Committed Offsets: MicroBatch Thread StreamingQuery source scan groupBy avglogicalPlan = Commit to WAL
- 17. Check the source for new offsets Source Available Offsets: 0, 1, 2 Sink Committed Offsets MicroBatch Thread StreamingQuery source scan groupBy avglogicalPlan = batchId= 42 getBatch()
- 18. Get the “recipe” for this micro batch Source Available Offsets: 0, 1, 2 Sink Committed Offsets MicroBatch Thread StreamingQuery source scan groupBy avglogicalPlan = batchId= 42 source scan groupBy avg batchId= 42
- 19. Send the micro batch Dataset to the sink Source Available Offsets: 0, 1, 2 Sink Committed Offsets MicroBatch Thread StreamingQuery source scan groupBy avglogicalPlan = groupBy avg MicroBatch Dataset isStreaming = false addBatch() batchId= 42 Backed by an incremental execution plan
- 20. Commit and listen again Source Available Offsets: 0, 1, 2 Sink Committed Offsets: 0, 1, 2 MicroBatch Thread StreamingQuery source scan groupBy avglogicalPlan = Listening
- 21. Execution Summary ● Each query has its own thread - asynchronous ● Sources must be replayable ● Use write-ahead-logs for durability ● Sinks must be idempotent ● Each batch is executed with an incremental execution plan ● Sinks get a micro batch view of the data ○ We exploit this for ML
- 22. Cool - lets build some ML with it! Lauren Coolman
- 23. Batch ML pipelines Tokenizer HashingTF String Indexer Naive Bayes Tokenizer HashingTF Streaming String Indexer Streaming Naive Bayes fit(df) Estimator Transformer ● In the batch setting, an estimator is trained on a dataset, and produces a static, immutable transformer. ● There is no communication between the two.
- 24. Streaming ML Pipelines (Proof of Concept) Tokenizer HashingTF Streaming String Indexer Streaming Naive Bayes Tokenizer HashingTF Streaming String Indexer Streaming Naive Bayes fit(df) Estimator Transformer (mutable) ● For streaming, an estimator is not fit once, but repeatedly on incoming data ● In one implementation, the estimator produces an initial transformer, and communicates updates to a specialized StreamingTransformer. ● Streaming transformers must provide a means of incorporating model updates into predictions. state state Lauren Coolman
- 25. Streaming Estimator/Transformer (POC) trait StreamingModel[S] extends Transformer { def update(updates: S): Unit } trait StreamingEstimator[S] extends Estimator { def model: StreamingModel[S] def update(batch: Dataset[_]): Unit } Sufficient statistics for model updates Blinke nArea
- 26. Getting a micro-batch view with distributed collection* case class ForeachDatasetSink(func: DataFrame => Unit) extends Sink { override def addBatch(batchId: Long, data: DataFrame): Unit = { func(data) } } https://github.com/holdenk/spark-structured-streaming-ml
- 27. And doing some ML with it: def evilTrain(df: DataFrame): StreamingQuery = { val sink = new ForeachDatasetSink({df: DataFrame => update(df)}) val sparkSession = df.sparkSession val evilStreamingQueryManager = EvilStreamingQueryManager(sparkSession.streams) evilStreamingQueryManager.startQuery( Some("snb-train"), None, df, sink, OutputMode.Append()) }
- 28. And doing some ML with it: def update(batch: Dataset[_]): Unit = { val newCountsByClass = add(batch) model.update(newCountsByClass) } Aggregate new batch Merge with previous aggregates
- 29. And doing some ML with it* (Algorithm specific) def update(updates: Array[(Double, (Long, DenseVector))]): Unit = { updates.foreach { case (label, (numDocs, termCounts)) => countsByClass.get(label) match { case Some((n, c)) => axpy(1.0, termCounts, c) countsByClass(label) = (n + numDocs, c) case None => // new label encountered countsByClass += (label -> (numDocs, termCounts)) } } }
- 30. Non-Evil alternatives to our Evil: ● ForeachWriter exists ● Since everything runs on the executors it's difficult to update the model ● You could: ○ Use accumulators ○ Write the updates to Kafka ○ Send the updates to a param server of some type with RPC ○ Or do the evil things we did instead :) ● Wait for the “future?”: https://github.com/apache/spark/pull/15178 _torne
- 31. Working with the results - foreach (1 of 2) val foreachWriter: ForeachWriter[T] = new ForeachWriter[T] { def open(partitionId: Long, version: Long): Boolean = { True // always open } def close(errorOrNull: Throwable): Unit = { // No close logic - if we wanted to copy updates per-batch } def process(record: T): Unit = { db.update(record) } }
- 32. Working with the results - foreach (2 of 2) // Apply foreach happinessByCoffee.writeStream.outputMode(OutputMode.Complete()) foreach(foreachWriter).start()
- 33. Structured Streaming in Review: ● Structured Streaming still uses Spark’s Microbatch approach ● JIRA discussion indicates an interest in swapping out the execution engine (but no public design document has emerged yet) ● One of the areas that Matei is researching ○ Researching ==~ future , research !~ today Windell Oskay
- 34. Ok but where can we not use it? ● A lot of random methods on DataFrames & Datasets won’t work ● They will fail at runtime rather than compile time - so have tests! ● Anything which roundtrips through an rdd() is going to be pretty sad (aka fail) ○ Lot’s of internals randomly do (like toJson) for historical reasons ● Need to run a query inside of a sink? That is not going to work ● Need a complex receiver type? Most receivers are not ported yet ● Also you will need distinct query names - even if you stop the previous query. ● Aggregations and Append output mode (and the only file sink requires Append) ● Need Kafka or other non-file based receiver ● DataFrame/Dataset transformations inside of a sink
- 35. Open questions for ML pipelines ● How to train and predict simultaneously, on the same data? ○ Transform thread should be executed first ○ Do we actually need to support this or is this just a common demo? ● How to ensure robustness to failures? ○ Store models in main memory for quick access ○ Back models by durable model store - write out every update ○ Similar to aggregate state store in structured streaming, but happens on the driver ● Model training must be idempotent - should not train on the same data twice ○ Leverage batch ID, similar to `FileStreamSink` ● How to extend MLWritable for streaming ○ Spark’s format isn’t really all that useful - maybe PMML or PFA Photo by bullet101
- 36. Structured Streaming ML vs DStreams ML What could be different for ML on structured streaming vs ML on DStreams? ● Structured streaming is built on the Spark SQL engine ○ Catalyst optimizer ○ Project tungsten ● Pipeline integration ○ ML pipelines have been improved and iterated across 5 releases, we can leverage their mature design for streaming pipelines ○ This will make adding and working with new algorithms much easier than in the past ● Event time handling ○ Streaming ML algorithms typically use a decay factor ○ Structured streaming provides native support for event time, which is more appropriate for decay
- 37. Batch vs Streaming Pipelines (Draft POC API) val df = spark .read .schema(schema) .parquet(path) val tokenizer = new RegexTokenizer() val htf = new HashingTF() val nb = new NaiveBayes() val pipeline = new Pipeline() .setStages(Array(tokenizer, htf, nb)) val pipelineModel = pipeline.fit(df) val df = spark .readStream .schema(schema) .parquet(path) val tokenizer = new RegexTokenizer() val htf = new HashingTF() val snb = new StreamingNaiveBayes() val pipeline = new StreamingPipeline() .setStages(Array(tokenizer, htf, snb)) .setCheckpointLocation(path) val query = pipeline.fitStreaming(df) query.awaitTermination() https://github.com/sethah/spark/tree/structured-streaming-fun
- 38. Additional Spark Resources ● Programming guide (along with JavaDoc, PyDoc, ScalaDoc, etc.) ○ http://spark.apache.org/docs/latest/ ● Books ● Videos ● Spark Office Hours ○ Normally in the bay area - will do Google Hangouts ones soon ○ follow me on twitter for future ones - https://twitter.com/holdenkarau
- 39. Structured Streaming Resources ● Programming guide (along with JavaDoc, PyDoc, ScalaDoc, etc.) ○ http://spark.apache.org/docs/latest/structured-streaming-programming -guide.html ● https://github.com/holdenk/spark-structured-streaming-ml ● TD https://spark-summit.org/2016/events/a-deep-dive-into-structured-st reaming/
- 40. Learning Spark Fast Data Processing with Spark (Out of Date) Fast Data Processing with Spark (2nd edition) Advanced Analytics with Spark Coming soon: Spark in Action Coming soon: High Performance Spark
- 41. And the next book….. First five chapters are available in “Early Release”*: ● Buy from O’Reilly - http://bit.ly/highPerfSpark ● Probably a book signing too! Get notified when updated & finished: ● http://www.highperformancespark.com ● https://twitter.com/highperfspark * Early Release means extra mistakes, but also a chance to help us make a more awesome book.
- 42. Surveys!!!!!!!! :D ● Interested in Structured Streaming? ○ http://bit.ly/structuredStreamingML - Let us know your thoughts ● Pssst: Care about Python DataFrame UDF Performance? ○ http://bit.ly/pySparkUDF ● Care about Spark Testing? ○ http://bit.ly/holdenTestingSpark Michael Himbeault
- 43. And some upcoming talks: ● September ○ This Talk (yay) ● October ○ PyData DC - Making Spark go fast in Python (vroom vroom) ○ Salt Lake City Spark Meetup - TBD ○ London - OSCON - Getting Started Contributing to Spark ● November ○ Possible London Office Hours if interest? ● December ○ Strata Singapore (Introduction to Datasets)
- 44. k thnx bye! If you care about Spark testing and don’t hate surveys: http://bit.ly/holdenTestingSpark Will tweet results “eventually” @holdenkarau Any PySpark Users: Have some simple UDFs you wish ran faster you are willing to share?: http://bit.ly/pySparkUDF Pssst: Have feedback on the presentation? Give me a shout ([email protected]) if you feel comfortable doing so :)