Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!
4 likes1,794 views
The document discusses the lambda architecture as a robust framework for building scalable and reliable applications, particularly in big data scenarios. It emphasizes the architecture's fault-tolerance against hardware failures and human errors, as well as its capabilities for low latency querying and extensibility. Additionally, it covers key components such as the batch layer, speed layer, and serving layer, alongside the integration of Spark for real-time processing and analytics.
Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!
- 1. © 2015 MapR Technologies ‹#›© 2016 MapR Technologies Lambda Architecture: The Best Way to Build Scalable and Reliable Applications!
- 2. © 2016 MapR Technologies ‹#›@tgrall {“about” : “me”} Tugdual “Tug” Grall • MapR • Technical Evangelist • MongoDB • Technical Evangelist • Couchbase • Technical Evangelist • eXo • CTO • Oracle • Developer/Product Manager • Mainly Java/SOA • Developer in consulting firms • Web • @tgrall • http://tgrall.github.io • tgrall • NantesJUG co-founder • Pet Project : • http://www.resultri.com • [email protected] • [email protected]
- 3. © 2016 MapR Technologies@tgrall 3 Big Data & Hadoop In Production
- 4. © 2016 MapR Technologies 4 Data Warehouse Optimization
- 5. © 2016 MapR Technologies 5 Data Hub Choose the best “connector”: • File • Sqoop • ETL • … Use the aggregated data • In your applications • To update other systems • as an Open Data API • … Customer DB Customer DB Logs … Hadoop NoSQL
- 6. © 2016 MapR Technologies 6 Financial Services Fraud detection Personalized offers Fraud investigation tool Fraud investigator Fraud model Recommendations table Clickstream analysis Online transactions MapR Distribution for Hadoop Analytics Real-time Operational Applications Interactive marketer
- 7. © 2016 MapR Technologies@tgrall 7 Fault Tolerance
- 8. © 2016 MapR Technologies 8 Fault Tolerance hardware software developer ?
- 9. © 2016 MapR Technologies 9 Human fault tolerance
- 10. © 2014 MapR Technologies 10
- 11. © 2014 MapR Technologies 11
- 12. © 2014 MapR Technologies 12
- 13. © 2016 MapR Technologies@tgrall 13 Lambda Architecture To the rescue λ
- 14. © 2016 MapR Technologies 14 A little bit of history…. • Defined by Nathan Marz • ex BackType, Twitter • in a new Startup • Creator of … – Storm – Cascalog – ElephantDB
- 15. © 2016 MapR Technologies 15 Lambda Architecture Requirements • Fault-tolerant against both hardware failures & human errors • Support variety of use cases that include low latency querying as well as updates • Linear scale-out capabilities • Extensible, so that the system is manageable and can accommodate newer features easily
- 16. © 2016 MapR Technologies 16
- 17. © 2016 MapR Technologies 17 Lambda Architecture NEW DATA STREAM QUERY BATCH VIEWS √View 1 View 2 View N REAL-TIME VIEWS BATCH LAYER SERVINGLAYER SPEED LAYER MERGE IMMUTABLE MASTER DATA PRECOMPUTE VIEWSBATCH RECOMPUTE PROCESS STREAM INCREMENT VIEWS View 1 View 2 View N
- 18. © 2016 MapR Technologies 18 Data Ingestion All data entering the system are dispatched to both • the batch layer • the speed layer NEW DATA STREAM BATCH LAYER SPEED LAYER
- 19. © 2016 MapR Technologies Batch Layer • managing the master dataset, an immutable, append-only set of raw data • pre-computing arbitrary query functions, called batch views. BATCH VIEWS BATCH LAYER IMMUTABLE MASTER DATA PRECOMPUTE VIEWSBATCH RECOMPUTE View 1 View 2 View N
- 20. © 2016 MapR Technologies 20 Speed Layer √View 1 View 2 View N REAL-TIME VIEWS SPEED LAYER PROCESS STREAM INCREMENT VIEWS • Speed layer accommodates low latency requests that are subject to low latency requirements. • Using fast and incremental algorithms, deals with recent data only
- 21. © 2016 MapR Technologies 21 Serving Layer QUERY BATCH VIEWS √View 1 View 2 View N REAL-TIME VIEWS SERVINGLAYER MERGE View 1 View 2 View N • Serving layer indexes batch views so that they can be queried in ad hoc with low latency
- 22. © 2014 MapR Technologies 22 Lambda Architecture—Compensate Batch time not absorbed now
- 23. © 2016 MapR Technologies 23 Lambda Architecture—Immutable Data + Views http://openflights.org
- 24. © 2016 MapR Technologies 24 Lambda Architecture—Immutable Data + Views timestamp airport flight action 2016-02-04T10:00:00 MUC EY123 take-off 2016-02-04T10:05:00 BRU SAS45 take-off 2016-02-04T10:07:00 AMS BA99 take-off 2016-02-04T10:09:00 LHR LH17 landing 2016-02-04T10:10:00 CDG AF03 landing 2016-02-04T10:10:00 FCO AZ501 take-off immutable master dataset
- 25. © 2016 MapR Technologies 25 Lambda Architecture—Immutable Data + Views timestamp airport flight action 2016-02-04T10:00:00 MUC EY123 take-off 2016-02-04T10:05:00 BRU SAS45 take-off 2016-02-04T10:07:00 AMS BA99 take-off 2016-02-04T10:09:00 LHR LH17 landing 2016-02-04T10:10:00 CDG AF03 landing 2016-02-04T10:10:00 FCO AZ501 take-off air-borne: 2307 airline planes AF 59 AZ 23 BA 167 EY 19 LH 201 SAS 28 air-borne per airline: airport planes AMS 69 CDG 44 BRU 31 FCO 10 HEL 17 LHR 101 airport load:
- 26. © 2016 MapR Technologies@tgrall 26 Implementation
- 27. © 2016 MapR Technologies 27 Lambda Architecture NEW DATA STREAM QUERY BATCH VIEWS √View 1 View 2 View N REAL-TIME VIEWS BATCH LAYER SERVINGLAYER SPEED LAYER MERGE IMMUTABLE MASTER DATA PRECOMPUTE VIEWSBATCH RECOMPUTE PROCESS STREAM INCREMENT VIEWS View 1 View 2 View N
- 28. © 2016 MapR Technologies 28 Batch Layer: View Generation Master Data View 1 View 2 Master Data Master Data Master Data Events “Raw” Storage Processing Aggregated Data
- 29. © 2016 MapR Technologies 29
- 30. © 2016 MapR Technologies 30 • Cluster Computing Platform • Extends “MapReduce” with extensions – Streaming – Interactive Analytics • Run in Memory
- 31. © 2015 MapR Technologies ‹#›@tgrall Spark components Spark SQL Spark Streaming (Streaming) MLlib (Machine Learning) Spark Core (General execution engine) GraphX (Graph Computation) Mesos Distributed File System (HDFS, MapR-FS, S3, …) Hadoop YARN
- 32. © 2016 MapR Technologies 32 Spark Jobs Driver Program (application) sc=new SparkContext rDD=sc.textfile(“hdfs://…”) rDD.map Cluster Manager Worker Executor Task Task Worker Executor Task Task
- 33. © 2016 MapR Technologies 33 Spark Resilient Distributed Datasets “RDD” Sensor RDD W Executor P4 W Executor P1 P3 W Executor P2 sc.textFile P1 8213034705, 95, 2.927373, jake7870, 0…… P2 8213034705, 115, 2.943484, Davidbresler2, 1…. P3 8213034705, 100, 2.951285, gladimacowgirl, 58… P4 8213034705, 117, 2.998947, daysrus, 95….
- 34. © 2016 MapR Technologies 34 Spark Resilient Distributed Datasets Transformation Filter() Action Count() RDD newRDD Value
- 35. © 2015 MapR Technologies@tgrall Transformations • Process an RDD, returns an RDD • Examples : • map() : one value => another value • mapToPair() : one value => a tuple • filter() : filters values/tuples on a given condition • groupByKey() : groups values by key • reduceByKey() : aggregates values by key • join(), cogroup(), … : joins RDDs
- 36. © 2015 MapR Technologies@tgrall Actions • Process an RDD, returns a value • Examples : • count() : counts number of items in dataset • first() : returns first entry • take(n) : returns array of the n first elements • foreach() : applies a function on each element • collect() : returns all elements • saveAsTextFile() : saves in files each element
- 37. © 2016 MapR Technologies 37 Speed Layer Real Time View1 Real Time View 2 Events Processing NoSQL
- 38. © 2016 MapR Technologies 38 Serving Layer: Aggregated Data • Views are stored in a Read/Write database • Apache HBase • MapR DB Binary & JSON • Cassandra • MongoDB • Elasticsearch • …
- 39. © 2016 MapR Technologies 39 Serving Layer Real Time View Events Processing Aggregated Batch View Query-SQL Dataviz Query/Visualisation SQL
- 40. © 2016 MapR Technologies // Join MapR-DB Table, Parquet and MongoDB collection > SELECT u.name, b.category, count(1) nb_review FROM mongo.yelp.`user` u , dfs.yelp.`review.parquet` r, (select business_id, flatten(categories) category from maprdb.`business` ) b WHERE u.user_id = r.user_id AND b.business_id = r.business_id GROUP BY u.user_id, u.name, b.category ORDER BY nb_review DESC LIMIT 10; +-----------+--------------+------------+ | name | category | nb_review | +-----------+--------------+------------+ | Rand | Restaurants | 1086 | | J | Restaurants | 661 | | Aileen | Restaurants | 499 | | Michael | Restaurants | 496 | +-----------+--------------+------------+ 40
- 41. © 2016 MapR Technologies@tgrall 41 Events Capture?
- 42. © 2016 MapR Technologies 42 Events Capture Customer DB API Logs … Streaming Streams Files
- 43. © 2016 MapR Technologies 43 What is Spark Streaming? • Enables scalable, high-throughput, fault-tolerant stream processing of live data • Extension of the core Spark Data Sources Data Sinks
- 44. © 2016 MapR Technologies 44 Spark Streaming Architecture • Divide data stream into batches of X seconds (micro batching) • Called DStream = sequence of RDDs Spark Streaming input data stream DStream RDD batches Batch interval data from time 0 to 1 data from time 1 to 2 RDD @ time 2 data from time 2 to 3 RDD @ time 3RDD @ time 1
- 45. © 2016 MapR Technologies 45 What are Apache Kafka & MapR Streams? • Publish Subscribe Messaging • Fast • Scalable • Durable • Distributed
- 46. © 2016 MapR Technologies@tgrall 46 Summary
- 47. © 2016 MapR Technologies 47 Lambda Architecture NEW DATA STREAM QUERY BATCH VIEWS √View 1 View 2 View N REAL-TIME VIEWS BATCH LAYER SERVINGLAYER SPEED LAYER MERGE IMMUTABLE MASTER DATA PRECOMPUTE VIEWSBATCH RECOMPUTE PROCESS STREAM INCREMENT VIEWS View 1 View 2 View N NoSQL Distributed File System NoSQL Streams
- 48. © 2016 MapR Technologies 48 Lambda Architecture in Action Batch processing (MapReduce) Tax reduction reporting Shortest path graph algorithm (Titan on MapR-DB) Route optimization . . . Geolocation Geolocation Geolocation Geolocation Online alerts Real-time stream
- 49. © 2016 MapR Technologies 49 Lambda Architecture • Fault-tolerant • Use batch layer to pre compute complex/large data set queries • Use speed layer to deal with “near real time” use cases • Linear scale-out capabilities • Error Prone: • Recompute data from master data set when needed
- 50. © 2016 MapR Technologies 50
- 51. © 2016 MapR Technologies 51 Q&A @tgrall maprtech [email protected] Engage with us! MapR maprtech mapr-technologies
Editor's Notes
- #4: First of all since we will be talking about Big Data Applications… let’s see some use case that are very common… High level
- #8: New apps, big data or not, must be “fault tolerant”… and the lambda arch has been build for that.. and at which level…
- #9: Hardware, Commodity hardware : we know that it willl fail So we compensate for that using software : HDFS/MapR-FS, Hbase/MaprDB, Zookeeper, .. you have infrastructure to support failure What about the developer? human being becoming the weakest link
- #10: So infrastrcuture using Hadoop/MapR/Distributed software is Fault Tolerant but we still need to deal with HUMAN ERROR…. since some of us are making mistake the goal is to “recover from it” WE ALL DO MISTAKE… look at these big names
- #11: Facebook apologises after crash: Social network site went down for the third time in a month due to a 'configuration issue'
- #15: Storm is realtime computation system. Storm makes it easy to reliably process unbounded streams of data, doing for realtime processing what Hadoop did for batch processing. Cascalog Fully-featured data processing and querying library for Clojure or Java. ElephantDB Distributed database specialized in exporting key/value data from Hadoop
- #17: As you can guess, in application development when we talk about architecture it is all about LAYERS
- #19: So we can see that in this case the application generate “EVENTS” everything we do generate events: Credit Card Payiment, Commit toGit, WebPage Click, twet, …. The event are used to manipulate the “data”, but we can use the events as the main data
- #20: The event you generate are immutable they have “happened”, and they are time based
- #30: Data Locality
- #34: Resilient distributed datasets, or RDD, are the primary abstraction in Spark. They are a collection of objects that is distributed across nodes in a cluster, and data operations are performed on RDD. Once created, RDD are immutable. You can also persist, or cache, RDDs in memory or on disk. Spark RDDs are fault-tolerant. If a given node or task fails, the RDD can be reconstructed automatically on the remaining nodes and the job will complete.
- #35: There are two types of data operations you can perform on an RDD, transformations and actions. A transformation will return an RDD. Since RDD are immutable, the transformation will return a new RDD. An action will return a value.
- #44: ● Socket ● Kafka ● Flume ● HDFS ● MQ (ZeroMQ...) ● Twitter ● ... ● Or a custom implementation of Receiver
- #50: Store all events as raw data Create Intermediate Views Errors are fixed using re-computation Based on Scalable and Reliable Storage Distributed File System Optimized formats (Parquet, Avro, Protobuff, …) NoSQL Engines HBase, MapR-DB, Elasticsearch, Cassandra, MongoDB, … Distributed Processing Spark Drill (SQL)