Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail Chernetsov
6 likes6,333 views
The document presents an intricate framework for building an analytics pipeline using Apache Spark, aimed at enhancing user understanding and engagement for a consumer product company. It details steps such as data capturing, enrichment, indexing, and querying, emphasizing the importance of data-driven decisions and the integration of various tools like Kafka, Cassandra, and S3. The discussion also touches on handling user attribution and system memory management for optimal performance.
rdd.saveToCassandra("raw")](https://image.slidesharecdn.com/074mikhailchernetsov-170614192109/85/Building-a-Versatile-Analytics-Pipeline-on-Top-of-Apache-Spark-with-Mikhail-Chernetsov-18-320.jpg)
![val firstUtmMedium: CalcProp[String] =
(E "url").as[Url]
.map(_.param("utm_source"))
.forEvent("page-visit")
.first
Enrichment Phase 2: Calculable Props Engine & DSL](https://image.slidesharecdn.com/074mikhailchernetsov-170614192109/85/Building-a-Versatile-Analytics-Pipeline-on-Top-of-Apache-Spark-with-Mikhail-Chernetsov-50-320.jpg)
![val rdd: RDD[T]
rdd.sendToKafka(“topic_x”)
rdd.saveToCassandra(“table_foo”)
rdd.saveToCassandra(“table_bar”)
Multiple Output Destinations: Try 1](https://image.slidesharecdn.com/074mikhailchernetsov-170614192109/85/Building-a-Versatile-Analytics-Pipeline-on-Top-of-Apache-Spark-with-Mikhail-Chernetsov-58-320.jpg)
![val rdd: RDD[T]
rdd.cache()
rdd.sendToKafka(“topic_x”)
rdd.saveToCassandra(“table_foo”)
rdd.saveToCassandra(“table_bar”)
Multiple Output Destinations: Try 2](https://image.slidesharecdn.com/074mikhailchernetsov-170614192109/85/Building-a-Versatile-Analytics-Pipeline-on-Top-of-Apache-Spark-with-Mikhail-Chernetsov-62-320.jpg)
![val rdd: RDD[T]
rdd.andSaveToCassandra(“table_foo”)
.andSaveToCassandra(“table_bar”)
.sendToKafka(“topic_x”)
Multiple Output Destinations: Try 3](https://image.slidesharecdn.com/074mikhailchernetsov-170614192109/85/Building-a-Versatile-Analytics-Pipeline-on-Top-of-Apache-Spark-with-Mikhail-Chernetsov-71-320.jpg)
Building a Versatile Analytics Pipeline on Top of Apache Spark with Mikhail Chernetsov
- 1. Building a Versatile Analytics Pipeline On Top Of Apache Spark Misha Chernetsov, Grammarly Spark Summit 2017 June 6, 2017
- 3. Data Team Lead @ Grammarly Building Analytics Pipelines (5 years) Coding on JVM (12 years), Scala + Spark (3 years) About Me: Misha Chernetsov @chernetsov
- 4. Tool that helps us better understand: ● Who are our users? ● How do they interact with the product? ● How do they get in, engage, pay, and how long do they stay? Analytics @ Consumer Product Company
- 5. We want our decisions to be data-driven Everyone: product managers, marketing, engineers, support... Analytics @ Consumer Product Company
- 6. data analytics report Analytics @ Consumer Product Company
- 7. Calendar Day Number of unique active users by day Example Report 1 – Daily Active Users dummy data!dummy data!
- 8. Example Report 2 – Comparison of Cohort Retention Over Time dummy data!dummy data!
- 9. Ads Email Social Number of users who bought a subscription. Split by traffic source type (where user came from) Calendar Day Example Report 3 – Payer Conversions By Traffic Source dummy data!dummy data!
- 10. ● Landing page visit ○ URL with UTM tags ○ Referrer ● Subscription purchased ○ Is first in subscription Example: Data
- 11. Everything is an Event Example: Data { "eventName": "page-visit", "url": "...?utm_medium=ad", … } { "eventName": "subscribe", "period": "12 months", … }
- 12. Enrich and/or Join Example: Data { "eventName": "page-visit", "url": "...?utm_medium=ad", … } { "eventName": "subscribe", "period": "12 months", … } Slice by Plot
- 13. capture enrich index query Analytics @ Consumer Product Company
- 14. capture enrich index query Use 3rd Party? 1. Integrated Event Analytics 2. UI over your DB Reports are not tailored for your needs, limited capability. Pre-aggregation / enriching is still on you. Hard to achieve accuracy and trust.
- 15. capture enrich index query Build Step 1: Capture
- 16. ● Always up, resilient ● Spikes / back pressure ● Buffer for delayed processing Kafka Capture REST { "eventName": "page-visit", "url": "...?utm_medium=paid", … }
- 17. Long-term Storage StreamKafka Save To Long-Term Storage
- 18. Cassandra micro-batch Kafka Save To Long-Term Storage val rdd = KafkaUtils.createRDD[K, V](...) rdd.saveToCassandra("raw")
- 19. capture enrich index query Build Step 2: Enrich
- 20. Enrichment 1: User Attribution
- 21. Enrichment 1: User Attribution { "eventName": "page-visit", "url": "...?utm_medium=ad", "fingerprint": "abc", … } { "eventName": "subscribe", "userId": 123, "fingerprint": "abc", … }
- 22. Enrichment 1: User Attribution { "eventName": "page-visit", "url": "...?utm_medium=ad", "fingerprint": "abc", … } { "eventName": "subscribe", "userId": 123, "fingerprint": "abc", … } "attributedUserId": 123,
- 23. t Non-authenticated: userId = null Authenticated: userId = 123 fingerprint = abc (All events from a given browser) Enrichment 1: User Attribution
- 24. t Non-authenticated: userId = null attributedUserId = 123 Authenticated: userId = 123 fingerprint = abc (All events from a given browser) Enrichment 1: User Attribution
- 25. Authenticated: userId = 756 Enrichment 1: User Attribution tfingerprint = abc (All events from a given browser) Authenticated: userId = 123 Heuristics to attribute those
- 26. Authenticated: userId = 756 Enrichment 1: User Attribution tfingerprint = abc (All events from a given browser) Authenticated: userId = 123 Heuristics to attribute those
- 27. rdd.mapPartitions { iterator => val buffer = new ArrayBuffer() iterator .takeWhile(_.userId.isEmpty) .foreach(buffer.append) val userId = iterator.head.userId buffer.map(_.setAttributedUserId(userId)) ++ iterator } Enrichment 1: User Attribution
- 28. rdd.mapPartitions { iterator => val buffer = new ArrayBuffer() iterator .takeWhile(_.userId.isEmpty) .foreach(buffer.append) val userId = iterator.head.userId buffer.map(_.setAttributedUserId(userId)) ++ iterator } Enrichment 1: User Attribution
- 29. rdd.mapPartitions { iterator => val buffer = new ArrayBuffer() iterator .takeWhile(_.userId.isEmpty) .foreach(buffer.append) val userId = iterator.head.userId buffer.map(_.setAttributedUserId(userId)) ++ iterator } Enrichment 1: User Attribution
- 30. rdd.mapPartitions { iterator => val buffer = new ArrayBuffer() iterator .takeWhile(_.userId.isEmpty) .foreach(buffer.append) val userId = iterator.head.userId buffer.map(_.setAttributedUserId(userId)) ++ iterator } Enrichment 1: User Attribution
- 31. rdd.mapPartitions { iterator => val buffer = new ArrayBuffer() iterator .takeWhile(_.userId.isEmpty) .foreach(buffer.append) val userId = iterator.head.userId buffer.map(_.setAttributedUserId(userId)) ++ iterator } Enrichment 1: User Attribution Can grow big and OOM your worker for outliers who use Grammarly without ever registering
- 32. By default we should operate in User Memory (small fraction). Spark & Memory User Memory 100% - spark.memory.fraction = 25% Spark Memory spark.memory.fraction = 75% Let’s get into Spark Memory and use its safety features.
- 33. rdd.mapPartitions { iterator => val buffer = new SpillableBuffer() iterator .takeWhile(_.userId.isEmpty) .foreach(buffer.append) val userId = iterator.head.userId buffer.map(_.setAttributedUserId(userId)) ++ iterator } Enrichment 1: User Attribution Can safely grow in mem while enough free Spark Mem. Spills to disk otherwise.
- 34. Spark Memory Manager & Spillable Collection Memory Disk
- 35. Spark Memory Manager & Spillable Collection Memory Disk
- 36. Spark Memory Manager & Spillable Collection Memory Disk
- 37. Spark Memory Manager & Spillable Collection Memory ×2 Disk
- 38. Spark Memory Manager & Spillable Collection Memory Disk
- 39. Spark Memory Manager & Spillable Collection Memory Disk
- 40. Spark Memory Manager & Spillable Collection Memory Spill to Disk Disk
- 41. Spark Memory Manager & Spillable Collection Memory Disk Spill to Disk
- 42. Spark Memory Manager & Spillable Collection Memory Disk
- 43. trait SizeTracker { def afterUpdate(): Unit = { … } def estimateSize(): Long = { … } } Call on every append. Periodically estimates size and saves samples. Extrapolates SizeTracker
- 44. trait Spillable { abstract def spill(inMemCollection: C): Unit def maybeSpill(currentMemory: Long, inMemCollection: C) { try x2 if needed } } Spillable call on every append to collection
- 45. public long acquireExecutionMemory(long required, …) public void releaseExecutionMemory(long size, …) TaskMemoryManager
- 46. ● Be safe with outliers ● Get outside User Memory (25%), use Spark Memory (75%) ● Spark APIs: Could be a bit friendlier and high level Custom Spillable Collection
- 47. Enrichment 2: Calculable Props
- 48. Enrichment Phase 2: Calculable Props { "eventName": "page-visit", "url": "...?utm_medium=ad", "fingerprint": "abc", "attributedUserId": 123, … } { "eventName": "subscribe", "userId": 123, "fingerprint": "abc", … }
- 49. Enrichment Phase 2: Calculable Props { "eventName": "page-visit", "url": "...?utm_medium=ad", "fingerprint": "abc", "attributedUserId": 123, … } { "eventName": "subscribe", "userId": 123, "fingerprint": "abc", "firstUtmMedium": "ad", … }
- 50. val firstUtmMedium: CalcProp[String] = (E "url").as[Url] .map(_.param("utm_source")) .forEvent("page-visit") .first Enrichment Phase 2: Calculable Props Engine & DSL
- 51. ● Type-safe, functional, composable ● Familiar: similar to Scala collections API ● Batch & Stream (incremental) Enrichment Phase 2: Calculable Props Engine & DSL
- 52. Enrichment Pipeline with Spark
- 53. Raw Kafka Spark Pipeline Stream: Save Raw Kafka Stream: User Attr. User-attributed Kafka Stream: Calc Props Enriched and Queryable Batch: User Attr. Batch: Calc Props
- 54. batch micro-batchmicro-batchmicro-batch Cassandra Kafka Spark Pipeline Kafka Cassandra Kafka Parquet on AWS S3 batch
- 55. ● Connectors for everything ● Great for batch ○ Shuffle with spilling ○ Failure recovery ● Great for streaming ○ Fast ○ Low overhead Spark Pipeline
- 56. batch micro-batchmicro-batchmicro-batch Cassandra Kafka Spark Pipeline Kafka Cassandra Kafka Parquet on AWS S3 batch
- 57. job Multiple Output Destinations Kafka Kafka CassandraCassandra
- 58. val rdd: RDD[T] rdd.sendToKafka(“topic_x”) rdd.saveToCassandra(“table_foo”) rdd.saveToCassandra(“table_bar”) Multiple Output Destinations: Try 1
- 59. rdd.saveToCassandra(...) rdd.forEachPartition(...) sc.runJob(...) Multiple Output Destinations: Try 1
- 60. job Multiple Output Destinations: Try 1 Kafka Kafka CassandraCassandra
- 61. job Multiple Output Destinations: Try 1 = 3 Jobs Kafka Kafka job Kafka Table 1 job Kafka Table 2
- 62. val rdd: RDD[T] rdd.cache() rdd.sendToKafka(“topic_x”) rdd.saveToCassandra(“table_foo”) rdd.saveToCassandra(“table_bar”) Multiple Output Destinations: Try 2
- 63. job Multiple Output Destinations: Try 2 = Read Once, 3 Jobs Kafka Kafka job Cache Table 1 job Cache Table 2
- 64. rdd.forEachPartition { iterator => val writer = new BufferedWriter( new OutputStreamWriter(new FileOutStream()) ) iterator.forEach { el => writer.writeln(el) } writer.close() // makes sure this writes } Writer
- 65. rdd.forEachPartition { iterator => val writer = new BufferedWriter( new OutputStreamWriter(new FileOutStream()) ) iterator.forEach { el => writer.writeln(el) } writer.close() // makes sure this writes } Writer
- 66. rdd.forEachPartition { iterator => val writer = new BufferedWriter( new OutputStreamWriter(new FileOutStream()) ) iterator.forEach { el => writer.writeln(el) } writer.close() // makes sure this writes } Writer
- 67. rdd.forEachPartition { iterator => val writer = new BufferedWriter( new OutputStreamWriter(new FileOutStream(...)) ) iterator.forEach { el => writer.writeln(el) } writer.close() // makes sure this writes } ● Buffer ● Non-blocking ● Idempotent / Dedupe Writer
- 68. andWriteToX = rdd.mapPartitions { iterator => val writer = new XWriter() val writingIterator = iterator.map { el => writer.write(el) }.closing(() => writer.close) } AndWriter
- 69. andWriteToX = rdd.mapPartitions { iterator => val writer = new XWriter() val writingIterator = iterator.map { el => writer.write(el) }.closing(() => writer.close) } AndWriter
- 70. andWriteToX = rdd.mapPartitions { iterator => val writer = new XWriter() val writingIterator = iterator.map { el => writer.write(el) }.closing(() => writer.close) } AndWriter
- 71. val rdd: RDD[T] rdd.andSaveToCassandra(“table_foo”) .andSaveToCassandra(“table_bar”) .sendToKafka(“topic_x”) Multiple Output Destinations: Try 3
- 72. job Multiple Output Destinations: Try 3 Kafka Kafka CassandraCassandra
- 73. ● Kafka ● Cassandra ● HDFS Important! Each andWriter will consume resources ● Memory (buffers) ● IO And Writer
- 74. capture enrich index query Build Step 3: Index
- 75. Index ● Parquet on AWS S3 ● Custom partitioning: By eventName and time interval ● Append changes, compact + merge on the fly when querying ● Randomized names to maximize S3 parallelism ● Use s3a for max performance and and tweak for S3 read-after-write consistency ● Support flexible schema, even with conflicts!
- 76. Some Stats ● Thousands of events per second ● Terabytes of compressed data
- 77. capture enrich index query Build Step 4: Query
- 78. ● DataFrames ● Spark SQL Scala dsl / Pure SQL ● Zeppelin Hardcore Query
- 79. ● Plot by day ● Unique visitors ● Filter by country ● Split by traffic source (top 20) ● Time from 2 weeks ago to today Casual Query
- 80. Option 1: SQL Quickly gets complex
- 81. Too expensive to build, extend and support Option 2: UI
- 82. SEGMENT “eventName” WHERE foo = “bar” AND x.y IN (“a”, “b”, “c”) UNIQUE BY m IS NOT NULL TIME from 2 months ago to today STEP 1 month SPAN 1 week Option 3: Custom Query Language
- 83. SEGMENT “eventName” WHERE foo = “bar” AND x.y IN (“a”, “b”, “c”) UNIQUE BY m IS NOT NULL TIME from 2 months ago to today STEP 1 month SPAN 1 week Option 3: Custom Query Language Expressions
- 84. ● Segment, Funnel, Retention ● UI & as DataFrame in Zeppelin ● Spark <= 1.6 – Scala Parser Combinators ● Reuse most complex part of expression parser ● Relatively extensible Option 3: Custom Query Language
- 85. Option 3: Custom Query Language
- 86. ● Custom versatile analytics is doable and enjoyable ● Spark is a great platform to build analytics on top of ○ Enrichment Pipeline: Batch / Streaming, Query, ML ● Would be cool to see even deep internals slightly more extensible Conclusion