SlideShare a Scribd company logo

Using apache spark for processing trillions of records each day at Datadog

V
Vadim Semenov

This document discusses Datadog's use of Apache Spark to process trillions of records daily. It describes their initial Spark setup using AWS EMR with large clusters. It then covers common out of memory errors, measuring memory usage, handling spot instances, and lessons learned around monitoring jobs and ensuring resilience.

Software
1 of 80
Downloaded 16 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
Using Apache Spark for processing trillions of records each day at Datadog Vadim Semenov Data Engineer @ Datadog vadim@datadoghq.com
Using apache spark for processing trillions of records each day at Datadog
Using apache spark for processing trillions of records each day at Datadog
Using apache spark for processing trillions of records each day at Datadog
Initial setup AWS EMR 100-200x r3.8xlarge 32cores, 244GiB, 640GB SSD, 10GbE 3200-6400 cores 24-48 TiB memory spot instances spark 1.6 in yarn-cluster mode scala + RDD API
Initial setup AWS EMR 100-200x r3.8xlarge 32cores, 244GiB, 640GB SSD, 10GbE 3200-6400 cores 23.5-47 TiB memory spot instances spark 1.6 in yarn-cluster mode scala + RDD API only 240.23 GiB available because of Xen
Some initial settings yarn.nodemanager.resource.memory-mb 240g yarn.scheduler.maximum-allocation-mb 240g spark.driver.memory 8g spark.yarn.driver.memoryOverhead 3g spark.executor.memory 201g spark.yarn.executor.memoryOverhead 28g spark.driver.cores 4 spark.executor.cores 32 spark.executor.extraJavaOptions -XX:+UseG1GC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:ErrorFile=/tmp/hs_err_pid%p.log
Trillion How big is a trillion? 2^40 = 1,099,511,627,776 2^31 = 2,147,483,648 = Int.MaxValue a trillion Integers = 4.3 TiB
OOMs
OOMs
OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size)
OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage)
OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO)
OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead)
OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead) - There is insufficient memory for the Java Runtime Environment to continue (Add more memory, reduce memory consumption)
The driver must survive spark.driver.memory 8g 83g spark.yarn.driver.memoryOverhead 3g 32g spark.driver.cores 4 15 spark.executor.memory 201g 166g spark.yarn.executor.memoryOverhead 28g 64g spark.executor.cores 32 30
IMAGE: TYNE & WEAR ARCHIVES & MUSEUMS
Measure memory usage https://github.com/etsy/statsd-jvm-profiler spark.files = /tmp/statsd-jvm-profiler.jar spark.executor.extraJavaOptions += -javaagent:statsd-jvm-profiler.jar=server=localhost,port=8125,profilers=Mem oryProfiler
Measure memory usage
Measure memory usage
Measure GC
OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead) - There is insufficient memory for the Java Runtime Environment to continue (Add more memory, reduce memory consumption)
Off-heap OOMs java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) … at parquet.hadoop.codec.…
Off-heap OOMs java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) … at parquet.hadoop.codec.…
Off-heap memory Direct Allocated Buffers (NIO): Parquet, MessagePack, … Java Native Interface (JNI): dynamically-linked native libraries like libhadoop.so, GZIP, ZLIB, LZ4 sun.misc.Unsafe: org.apache.hadoop.io.nativeio, org.apache.spark.unsafe
Process memory $ cat /proc/<spark driver/executor pid>/status VmPeak: 190317312 kB VmSize: 190268160 kB VmHWM: 187586408 kB VmRSS: 187586408 kB VmData: 190044492 kB
Process memory $ cat /proc/<spark driver/executor pid>/status VmPeak: 190317312 kB VmSize: 190268160 kB VmHWM: 187586408 kB VmRSS: 187586408 kB VmData: 190044492 kB
Process memory Solution: let the java-agent get the memory usage of its process right from the procfs https://github.com/DataDog/spark-jvm-profiler
Measure memory usage
Measure each executor
OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - There is insufficient memory for the Java Runtime Environment to continue (Add more memory, reduce memory consumption) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead)
Lessons - Give more resources than you think you would need, and then reduce
Lessons - Give more resources than you think you would need, and then reduce - Measure memory usage of each executor
Lessons - Give more resources than you think you would need, and then reduce - Measure memory usage of each executor - Keep an eye on your GC metrics
Measure slow parts val timer = MaxAndTotalTimeAccumulator rdd.map(key => { val startTime = System.nanoTime() ... val endTime = System.nanoTime() val millisecondsPassed = ((endTime - startTime) / 1000000).toInt timer.add(millisecondsPassed) })
Watch skewed parts .groupByKey().flatMap({ case (key, iter) => val size = iter.size maxAccumulator.add(key, size) if (size >= 100,000,000) { log.info(s"Key $key has $size values") None } else {
Report accumulators per partition sc.addSparkListener(new SparkListener { override def onTaskEnd( taskEnd: SparkListenerTaskEnd ): Unit = Option(taskEnd.taskMetrics) .foreach(taskMetrics => … ) })
Collect executors metrics
Lessons - Measure slowest parts of your job
Lessons - Measure slowest parts of your job - Count records in the most skewed parts
Lessons - Measure slowest parts of your job - Count records in the most skewed parts - Keep track of how much CPU time your job actually consumes
Lessons - Measure slowest parts of your job - Count records in the most skewed parts - Keep track of how much CPU time your job actually consumes - Have some alerting on these metrics, so you would know that your job gets slower
Spot instances
Spot instances mitigation - Break the job into smaller survivable pieces
Spot instances mitigation - Break the job into smaller survivable pieces - Use `rdd.checkpoint` instead of `rdd.persist` to save data to HDFS
Spot instances mitigation - Break the job into smaller survivable pieces - Use `rdd.checkpoint` instead of `rdd.persist` to save data to HDFS - Helps dynamic allocation since executors don't hold any data, so they can leave the job and join other jobs
Spot instances mitigation - Break the job into smaller survivable pieces - Use `rdd.checkpoint` instead of `rdd.persist` to save data to HDFS - Helps dynamic allocation since executors don't hold any data, so they can leave the job and join other jobs - Losing multiple executors won't result in recomputing partitions
ExternalShuffleService Ex1 1 2 3 4 Ex2 Ex3 Driver
ExternalShuffleService Ex1 1 2 3 4 Ex2 Ex3 Driver
ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver
ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver
ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver
ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver ERROR o.a.s.n.shuffle.RetryingBlockFetcher: Exception while beginning fetch of 13 outstanding blocks java.io.IOException: Failed to connect to ip-10-12-32-67.us-west-2.compute.internal/1 0.12.32.67:7337
ExternalShuffleService 2 3 4 Ex2 Ex3 Driver 1
ExternalShuffleService 2 3 4 Ex2 Ex3 Driver 1 ERROR o.a.s.n.shuffle.RetryingBlockFetcher: Exception while beginning fetch of 13 outstanding blocks java.io.IOException: Failed to connect to ip-10-12-32-67.us-west-2.compute.internal/1 0.12.32.67:7337
ExternalShuffleService 3 4 Ex2 Ex3 Driver 1 2
ExternalShuffleService SPARK-19753 Remove all shuffle files on a host in case of slave lost of fetch failure SPARK-20832 Standalone master should explicitly inform drivers of worker deaths and invalidate external shuffle service outputs
Other FetchFailures SPARK-20178 Improve Scheduler fetch failures
Keep an eye on failed tasks
Lessons - Keep all logs
Lessons - Keep all logs - Spark isn't super-resilient even when one node dies
Lessons - Keep all logs - Spark isn't super-resilient even when one node dies - Monitor the number of failed tasks/stages/lost nodes
Late arriving partitions rddA.cogroup(rddB, rddC).map({ case (k, iterA, iterB, iterC) => // We should always have one-to-one join, but who knows … if (iterA.toSet.size() > 1) throw new RuntimeException(s"Key $k received more than 1 A record") if (iterB.toSet.size() > 1) throw new RuntimeException(…) if (iterC.toSet.size() > 1) …
Late arriving partitions rddA.cogroup(rddB, rddC).map({ case (k, iterA, iterB, iterC) => // We should always have one-to-one join, but who knows … if (iterA.toSet.size() > 1) throw new RuntimeException(s"Key $k received more than 1 A record") if (iterB.toSet.size() > 1) throw new RuntimeException(…) if (iterC.toSet.size() > 1) …
Late arriving partitions rddA.cogroup(rddB, rddC).map({ case (k, iterA, iterB, iterC) => // We should always have one-to-one join, but who knows … if (iterA.toSet.size() > 1) throw new RuntimeException(s"Key $k received more than 1 A record") if (iterB.toSet.size() > 1) throw new RuntimeException(…) if (iterC.toSet.size() > 1) …
Late arriving partitions .map({ case (key, values: Iterator[(Long, Int)]) => values.toList.sortBy(_._1) // (1L, 10), (1L, 1), (2L, 1) // (1L, 1), (1L, 10), (2L, 1) }) SPARK-19263 DAGScheduler should avoid sending conflicting task set
Late arriving partitions .map({ case (key, values: Iterator[(Long, Int)]) => values.toList.sorted // (1L, 1), (1L, 10), (2L, 1) })
Lessons - Trust but put extra checks and log everything
Lessons - Trust but put extra checks and log everything - Add extra idempotency even if it should be there
Lessons - Trust but put extra checks and log everything - Add extra idempotency even if it should be there - Fail the job if some unexpected situation is encountered, but also think ahead of time if such situations can be overcome
Lessons - Trust but put extra checks and log everything - Add extra idempotency even if it should be there - Fail the job if some unexpected situation is encountered, but also think ahead of time if such situations can be overcome - Have retries on the pipeline scheduler level
Migration to Spark 2 SPARK-13850 TimSort Comparison method violates its general contract SPARK-14560 Cooperative Memory Management for Spillables SPARK-14363 Executor OOM due to a memory leak in Sorter SPARK-14560 Cooperative Memory Management for Spillables SPARK-22033 BufferHolder, other size checks should account for the specific VM array size limitations
Lessons - Check the bug tracker periodically
Lessons - Check the bug tracker periodically - Subscribe to mailing lists
Lessons - Check the bug tracker periodically - Subscribe to mailing lists - Participate in discussing issues
In conclusion - Log everything (driver/executors, NodeManagers, GC)
In conclusion - Log everything - Measure everything (heap/off-heap, GC, executors cpu, failed tasks/stages, slow parts, skewed parts)
In conclusion - Log everything - Measure everything - Trust but be ready
In conclusion - Log everything - Measure everything - Trust but be ready - Smaller survivable pieces
Thanks! Want to work with us on Spark, Kafka, ES, and more? Come to our booth! jobs.datadoghq.com twitter.com/@databuryat _@databuryat.com vadim@datadoghq.com

More Related Content

PPTX
From cache to in-memory data grid. Introduction to Hazelcast.
Taras Matyashovsky
 
PDF
The Parquet Format and Performance Optimization Opportunities
Databricks
 
PPTX
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
PDF
Performant Streaming in Production: Preventing Common Pitfalls when Productio...
Databricks
 
PPTX
APACHE KAFKA / Kafka Connect / Kafka Streams
Ketan Gote
 
PPTX
The Current State of Table API in 2022
Flink Forward
 
PDF
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
Databricks
 
PDF
Apache Spark At Scale in the Cloud
Databricks
 
From cache to in-memory data grid. Introduction to Hazelcast.
Taras Matyashovsky
 
The Parquet Format and Performance Optimization Opportunities
Databricks
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
Performant Streaming in Production: Preventing Common Pitfalls when Productio...
Databricks
 
APACHE KAFKA / Kafka Connect / Kafka Streams
Ketan Gote
 
The Current State of Table API in 2022
Flink Forward
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
Databricks
 
Apache Spark At Scale in the Cloud
Databricks
 

What's hot (20)

PDF
Efficient Data Storage for Analytics with Apache Parquet 2.0
Cloudera, Inc.
 
PDF
Running Apache Spark on Kubernetes: Best Practices and Pitfalls
Databricks
 
PDF
Hive tuning
Michael Zhang
 
PDF
Understanding Query Plans and Spark UIs
Databricks
 
PPTX
Demystifying flink memory allocation and tuning - Roshan Naik, Uber
Flink Forward
 
PPTX
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
 
PPTX
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward
 
PDF
Apache Jackrabbit Oak on MongoDB
MongoDB
 
PDF
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
Flink Forward
 
PDF
Advanced Cassandra Operations via JMX (Nate McCall, The Last Pickle) | C* Sum...
DataStax
 
PDF
Cassandra at Instagram 2016 (Dikang Gu, Facebook) | Cassandra Summit 2016
DataStax
 
PPTX
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks
 
PDF
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
The Hive
 
PPTX
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
Monica Beckwith
 
PDF
Presto on Apache Spark: A Tale of Two Computation Engines
Databricks
 
PPTX
Oracle Unified Directory. Lessons learnt. Is it ready for a move from OID? (O...
Andrejs Prokopjevs
 
PDF
Project Tungsten: Bringing Spark Closer to Bare Metal
Databricks
 
PDF
Top 5 Mistakes When Writing Spark Applications
Spark Summit
 
PDF
Battle of the Stream Processing Titans – Flink versus RisingWave
Yingjun Wu
 
PDF
Optimizing Delta/Parquet Data Lakes for Apache Spark
Databricks
 
Efficient Data Storage for Analytics with Apache Parquet 2.0
Cloudera, Inc.
 
Running Apache Spark on Kubernetes: Best Practices and Pitfalls
Databricks
 
Hive tuning
Michael Zhang
 
Understanding Query Plans and Spark UIs
Databricks
 
Demystifying flink memory allocation and tuning - Roshan Naik, Uber
Flink Forward
 
How to build a streaming Lakehouse with Flink, Kafka, and Hudi
Flink Forward
 
Exactly-Once Financial Data Processing at Scale with Flink and Pinot
Flink Forward
 
Apache Jackrabbit Oak on MongoDB
MongoDB
 
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
Flink Forward
 
Advanced Cassandra Operations via JMX (Nate McCall, The Last Pickle) | C* Sum...
DataStax
 
Cassandra at Instagram 2016 (Dikang Gu, Facebook) | Cassandra Summit 2016
DataStax
 
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks
 
Tech Talk: RocksDB Slides by Dhruba Borthakur & Haobo Xu of Facebook
The Hive
 
Garbage First Garbage Collector (G1 GC) - Migration to, Expectations and Adva...
Monica Beckwith
 
Presto on Apache Spark: A Tale of Two Computation Engines
Databricks
 
Oracle Unified Directory. Lessons learnt. Is it ready for a move from OID? (O...
Andrejs Prokopjevs
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Databricks
 
Top 5 Mistakes When Writing Spark Applications
Spark Summit
 
Battle of the Stream Processing Titans – Flink versus RisingWave
Yingjun Wu
 
Optimizing Delta/Parquet Data Lakes for Apache Spark
Databricks
 
Ad

Similar to Using apache spark for processing trillions of records each day at Datadog (20)

PDF
Spark 2.x Troubleshooting Guide
IBM
 
PDF
10 things i wish i'd known before using spark in production
Paris Data Engineers !
 
PDF
Advanced spark training advanced spark internals and tuning reynold xin
caidezhi655
 
PDF
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Yoshiyasu SAEKI
 
PDF
Debugging & Tuning in Spark
Shiao-An Yuan
 
PDF
Top 5 mistakes when writing Spark applications
hadooparchbook
 
PDF
TriHUG talk on Spark and Shark
trihug
 
PDF
Understanding Memory Management In Spark For Fun And Profit
Spark Summit
 
PDF
Top 5 mistakes when writing Spark applications
hadooparchbook
 
PDF
Apache Spark Internals - Part 2
Jéferson Machado
 
PDF
Top 5 mistakes when writing Spark applications
markgrover
 
PPTX
Apache Spark Workshop
Michael Spector
 
PDF
Top 5 mistakes when writing Spark applications
markgrover
 
PDF
Apache Spark in Depth: Core Concepts, Architecture & Internals
Anton Kirillov
 
PDF
Scaling Big Data with Hadoop and Mesos
Discover Pinterest
 
PDF
Sparklife - Life In The Trenches With Spark
Ian Pointer
 
PDF
Why your Spark job is failing
Sandy Ryza
 
PDF
Spark Gotchas and Lessons Learned (2/20/20)
Jen Waller
 
PPTX
Spark Gotchas and Lessons Learned
Jen Waller
 
PPTX
Tuning tips for Apache Spark Jobs
Samir Bessalah
 
Spark 2.x Troubleshooting Guide
IBM
 
10 things i wish i'd known before using spark in production
Paris Data Engineers !
 
Advanced spark training advanced spark internals and tuning reynold xin
caidezhi655
 
Apache Sparkにおけるメモリ - アプリケーションを落とさないメモリ設計手法 -
Yoshiyasu SAEKI
 
Debugging & Tuning in Spark
Shiao-An Yuan
 
Top 5 mistakes when writing Spark applications
hadooparchbook
 
TriHUG talk on Spark and Shark
trihug
 
Understanding Memory Management In Spark For Fun And Profit
Spark Summit
 
Top 5 mistakes when writing Spark applications
hadooparchbook
 
Apache Spark Internals - Part 2
Jéferson Machado
 
Top 5 mistakes when writing Spark applications
markgrover
 
Apache Spark Workshop
Michael Spector
 
Top 5 mistakes when writing Spark applications
markgrover
 
Apache Spark in Depth: Core Concepts, Architecture & Internals
Anton Kirillov
 
Scaling Big Data with Hadoop and Mesos
Discover Pinterest
 
Sparklife - Life In The Trenches With Spark
Ian Pointer
 
Why your Spark job is failing
Sandy Ryza
 
Spark Gotchas and Lessons Learned (2/20/20)
Jen Waller
 
Spark Gotchas and Lessons Learned
Jen Waller
 
Tuning tips for Apache Spark Jobs
Samir Bessalah
 
Ad

Recently uploaded (20)

PDF
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
PDF
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
PPTX
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
PDF
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
PDF
AI in Product Development-omnex systems
omnex systems
 
PPTX
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
PPTX
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
PPTX
history of c programming in notes for students .pptx
Vijayakumari829030
 
PDF
top salesforce developer skills in 2025.pdf
CloudMetic
 
PDF
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
PPTX
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
PDF
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
PDF
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
PPT
Introduction Database Management System for Course Database
ReinertYosua
 
PDF
medical staffing services at VALiNTRY
Tiyan Grayson
 
PDF
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 
PPTX
Transform Your Business with a Software ERP System
Canada Software Company
 
PDF
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
PDF
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
PDF
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 
Wondershare Filmora 15 Crack With Activation Key [2025
ghrom2211g
 
How to Migrate SBCGlobal Email to Yahoo Easily
raymondjones7273
 
Agentic AI Use Case- Contract Lifecycle Management (CLM).pptx
mhxyzzp
 
Softaken Excel to vCard Converter Software.pdf
markwillsonmw004
 
AI in Product Development-omnex systems
omnex systems
 
ai tools demonstartion for schools and inter college
harshavardhanreddyka11
 
Oracle E-Business Suite: A Comprehensive Guide for Modern Enterprises
Conacent Consulting
 
history of c programming in notes for students .pptx
Vijayakumari829030
 
top salesforce developer skills in 2025.pdf
CloudMetic
 
Understanding Forklifts - TECH EHS Solution
TECH EHS Solution
 
VVF-Customer-Presentation2025-Ver1.9.pptx
blackmambaettijean
 
Internet Downloader Manager (IDM) Crack 6.42 Build 42 Updates Latest 2025
itskinga12
 
Claude Code: Everyone is a 10x Developer - A Comprehensive AI-Powered CLI Tool
William Chong
 
Introduction Database Management System for Course Database
ReinertYosua
 
medical staffing services at VALiNTRY
Tiyan Grayson
 
SAP S4 Hana Brochure 3 (PTS SYSTEMS AND SOLUTIONS)
pts464036
 
Transform Your Business with a Software ERP System
Canada Software Company
 
Design an Analysis of Algorithms I-SECS-1021-03
DilanEscobar1
 
Audit Checklist Design Aligning with ISO, IATF, and Industry Standards — Omne...
rohnjim07
 
2025 Textile ERP Trends: SAP, Odoo & Oracle
Aetos Technologies
 

Using apache spark for processing trillions of records each day at Datadog

  • 1. Using Apache Spark for processing trillions of records each day at Datadog Vadim Semenov Data Engineer @ Datadog [email protected]
  • 5. Initial setup AWS EMR 100-200x r3.8xlarge 32cores, 244GiB, 640GB SSD, 10GbE 3200-6400 cores 24-48 TiB memory spot instances spark 1.6 in yarn-cluster mode scala + RDD API
  • 6. Initial setup AWS EMR 100-200x r3.8xlarge 32cores, 244GiB, 640GB SSD, 10GbE 3200-6400 cores 23.5-47 TiB memory spot instances spark 1.6 in yarn-cluster mode scala + RDD API only 240.23 GiB available because of Xen
  • 7. Some initial settings yarn.nodemanager.resource.memory-mb 240g yarn.scheduler.maximum-allocation-mb 240g spark.driver.memory 8g spark.yarn.driver.memoryOverhead 3g spark.executor.memory 201g spark.yarn.executor.memoryOverhead 28g spark.driver.cores 4 spark.executor.cores 32 spark.executor.extraJavaOptions -XX:+UseG1GC -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:OnOutOfMemoryError='kill -9 %p' -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -XX:ErrorFile=/tmp/hs_err_pid%p.log
  • 8. Trillion How big is a trillion? 2^40 = 1,099,511,627,776 2^31 = 2,147,483,648 = Int.MaxValue a trillion Integers = 4.3 TiB
  • 10. OOMs
  • 11. OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size)
  • 12. OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage)
  • 13. OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO)
  • 14. OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead)
  • 15. OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead) - There is insufficient memory for the Java Runtime Environment to continue (Add more memory, reduce memory consumption)
  • 16. The driver must survive spark.driver.memory 8g 83g spark.yarn.driver.memoryOverhead 3g 32g spark.driver.cores 4 15 spark.executor.memory 201g 166g spark.yarn.executor.memoryOverhead 28g 64g spark.executor.cores 32 30
  • 17. IMAGE: TYNE & WEAR ARCHIVES & MUSEUMS
  • 18. Measure memory usage https://github.com/etsy/statsd-jvm-profiler spark.files = /tmp/statsd-jvm-profiler.jar spark.executor.extraJavaOptions += -javaagent:statsd-jvm-profiler.jar=server=localhost,port=8125,profilers=Mem oryProfiler
  • 22. OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead) - There is insufficient memory for the Java Runtime Environment to continue (Add more memory, reduce memory consumption)
  • 23. Off-heap OOMs java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) … at parquet.hadoop.codec.…
  • 24. Off-heap OOMs java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:658) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) … at parquet.hadoop.codec.…
  • 25. Off-heap memory Direct Allocated Buffers (NIO): Parquet, MessagePack, … Java Native Interface (JNI): dynamically-linked native libraries like libhadoop.so, GZIP, ZLIB, LZ4 sun.misc.Unsafe: org.apache.hadoop.io.nativeio, org.apache.spark.unsafe
  • 26. Process memory $ cat /proc/<spark driver/executor pid>/status VmPeak: 190317312 kB VmSize: 190268160 kB VmHWM: 187586408 kB VmRSS: 187586408 kB VmData: 190044492 kB
  • 27. Process memory $ cat /proc/<spark driver/executor pid>/status VmPeak: 190317312 kB VmSize: 190268160 kB VmHWM: 187586408 kB VmRSS: 187586408 kB VmData: 190044492 kB
  • 28. Process memory Solution: let the java-agent get the memory usage of its process right from the procfs https://github.com/DataDog/spark-jvm-profiler
  • 31. OOMs - java.lang.OutOfMemoryError: Java heap space (Increase heap size) - java.lang.OutOfMemoryError: GC Overhead limit exceeded (Too much garbage) - java.lang.OutOfMemoryError: Direct buffer memory (NIO) - There is insufficient memory for the Java Runtime Environment to continue (Add more memory, reduce memory consumption) - YARN Container is running beyond physical memory limits. Killing container. (Increase memory overhead)
  • 32. Lessons - Give more resources than you think you would need, and then reduce
  • 33. Lessons - Give more resources than you think you would need, and then reduce - Measure memory usage of each executor
  • 34. Lessons - Give more resources than you think you would need, and then reduce - Measure memory usage of each executor - Keep an eye on your GC metrics
  • 35. Measure slow parts val timer = MaxAndTotalTimeAccumulator rdd.map(key => { val startTime = System.nanoTime() ... val endTime = System.nanoTime() val millisecondsPassed = ((endTime - startTime) / 1000000).toInt timer.add(millisecondsPassed) })
  • 36. Watch skewed parts .groupByKey().flatMap({ case (key, iter) => val size = iter.size maxAccumulator.add(key, size) if (size >= 100,000,000) { log.info(s"Key $key has $size values") None } else {
  • 37. Report accumulators per partition sc.addSparkListener(new SparkListener { override def onTaskEnd( taskEnd: SparkListenerTaskEnd ): Unit = Option(taskEnd.taskMetrics) .foreach(taskMetrics => … ) })
  • 39. Lessons - Measure slowest parts of your job
  • 40. Lessons - Measure slowest parts of your job - Count records in the most skewed parts
  • 41. Lessons - Measure slowest parts of your job - Count records in the most skewed parts - Keep track of how much CPU time your job actually consumes
  • 42. Lessons - Measure slowest parts of your job - Count records in the most skewed parts - Keep track of how much CPU time your job actually consumes - Have some alerting on these metrics, so you would know that your job gets slower
  • 44. Spot instances mitigation - Break the job into smaller survivable pieces
  • 45. Spot instances mitigation - Break the job into smaller survivable pieces - Use `rdd.checkpoint` instead of `rdd.persist` to save data to HDFS
  • 46. Spot instances mitigation - Break the job into smaller survivable pieces - Use `rdd.checkpoint` instead of `rdd.persist` to save data to HDFS - Helps dynamic allocation since executors don't hold any data, so they can leave the job and join other jobs
  • 47. Spot instances mitigation - Break the job into smaller survivable pieces - Use `rdd.checkpoint` instead of `rdd.persist` to save data to HDFS - Helps dynamic allocation since executors don't hold any data, so they can leave the job and join other jobs - Losing multiple executors won't result in recomputing partitions
  • 48. ExternalShuffleService Ex1 1 2 3 4 Ex2 Ex3 Driver
  • 49. ExternalShuffleService Ex1 1 2 3 4 Ex2 Ex3 Driver
  • 50. ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver
  • 51. ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver
  • 52. ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver
  • 53. ExternalShuffleService 1 2 3 4 Ex2 Ex3 Driver ERROR o.a.s.n.shuffle.RetryingBlockFetcher: Exception while beginning fetch of 13 outstanding blocks java.io.IOException: Failed to connect to ip-10-12-32-67.us-west-2.compute.internal/1 0.12.32.67:7337
  • 55. ExternalShuffleService 2 3 4 Ex2 Ex3 Driver 1 ERROR o.a.s.n.shuffle.RetryingBlockFetcher: Exception while beginning fetch of 13 outstanding blocks java.io.IOException: Failed to connect to ip-10-12-32-67.us-west-2.compute.internal/1 0.12.32.67:7337
  • 57. ExternalShuffleService SPARK-19753 Remove all shuffle files on a host in case of slave lost of fetch failure SPARK-20832 Standalone master should explicitly inform drivers of worker deaths and invalidate external shuffle service outputs
  • 58. Other FetchFailures SPARK-20178 Improve Scheduler fetch failures
  • 59. Keep an eye on failed tasks
  • 61. Lessons - Keep all logs - Spark isn't super-resilient even when one node dies
  • 62. Lessons - Keep all logs - Spark isn't super-resilient even when one node dies - Monitor the number of failed tasks/stages/lost nodes
  • 63. Late arriving partitions rddA.cogroup(rddB, rddC).map({ case (k, iterA, iterB, iterC) => // We should always have one-to-one join, but who knows … if (iterA.toSet.size() > 1) throw new RuntimeException(s"Key $k received more than 1 A record") if (iterB.toSet.size() > 1) throw new RuntimeException(…) if (iterC.toSet.size() > 1) …
  • 64. Late arriving partitions rddA.cogroup(rddB, rddC).map({ case (k, iterA, iterB, iterC) => // We should always have one-to-one join, but who knows … if (iterA.toSet.size() > 1) throw new RuntimeException(s"Key $k received more than 1 A record") if (iterB.toSet.size() > 1) throw new RuntimeException(…) if (iterC.toSet.size() > 1) …
  • 65. Late arriving partitions rddA.cogroup(rddB, rddC).map({ case (k, iterA, iterB, iterC) => // We should always have one-to-one join, but who knows … if (iterA.toSet.size() > 1) throw new RuntimeException(s"Key $k received more than 1 A record") if (iterB.toSet.size() > 1) throw new RuntimeException(…) if (iterC.toSet.size() > 1) …
  • 66. Late arriving partitions .map({ case (key, values: Iterator[(Long, Int)]) => values.toList.sortBy(_._1) // (1L, 10), (1L, 1), (2L, 1) // (1L, 1), (1L, 10), (2L, 1) }) SPARK-19263 DAGScheduler should avoid sending conflicting task set
  • 67. Late arriving partitions .map({ case (key, values: Iterator[(Long, Int)]) => values.toList.sorted // (1L, 1), (1L, 10), (2L, 1) })
  • 68. Lessons - Trust but put extra checks and log everything
  • 69. Lessons - Trust but put extra checks and log everything - Add extra idempotency even if it should be there
  • 70. Lessons - Trust but put extra checks and log everything - Add extra idempotency even if it should be there - Fail the job if some unexpected situation is encountered, but also think ahead of time if such situations can be overcome
  • 71. Lessons - Trust but put extra checks and log everything - Add extra idempotency even if it should be there - Fail the job if some unexpected situation is encountered, but also think ahead of time if such situations can be overcome - Have retries on the pipeline scheduler level
  • 72. Migration to Spark 2 SPARK-13850 TimSort Comparison method violates its general contract SPARK-14560 Cooperative Memory Management for Spillables SPARK-14363 Executor OOM due to a memory leak in Sorter SPARK-14560 Cooperative Memory Management for Spillables SPARK-22033 BufferHolder, other size checks should account for the specific VM array size limitations
  • 73. Lessons - Check the bug tracker periodically
  • 74. Lessons - Check the bug tracker periodically - Subscribe to mailing lists
  • 75. Lessons - Check the bug tracker periodically - Subscribe to mailing lists - Participate in discussing issues
  • 76. In conclusion - Log everything (driver/executors, NodeManagers, GC)
  • 77. In conclusion - Log everything - Measure everything (heap/off-heap, GC, executors cpu, failed tasks/stages, slow parts, skewed parts)
  • 78. In conclusion - Log everything - Measure everything - Trust but be ready
  • 79. In conclusion - Log everything - Measure everything - Trust but be ready - Smaller survivable pieces
  • 80. Thanks! Want to work with us on Spark, Kafka, ES, and more? Come to our booth! jobs.datadoghq.com twitter.com/@databuryat [email protected] [email protected]