The return of big iron?
1 like13,359 views
This document discusses the evolution of databases from traditional relational databases to NoSQL and big data technologies. It notes that NoSQL databases have emerged to solve problems related to large volumes of data and unreliable networks. Both relational and NoSQL solutions now have value. Most enterprise data is not highly structured and does not require the complexities of relational databases. A composite approach using both relational and NoSQL technologies in layers provides more flexibility to work with both structured and unstructured data.
The return of big iron?
- 1. Welcome
- 2. Remember when it looked like this. They were all pretty much alike?
- 3. It used to be easy…
- 6. But it’s also quite exciting!
- 7. We Lose: Joe Hellerstein (Berkeley) 2001 “Databases are commoditised and cornered to slow-moving, evolving, structure intensive, applications that require schema evolution.“ … “The internet companies are lost and we will remain in the doldrums of the enterprise space.” … “As databases are black boxes which require a lot of coaxing to get maximum performance”
- 9. The Web
- 10. With a lot of users.
- 11. we changed scale
- 12. tack ge d we ch an
- 13. New Approach to Data Access Simple Pragmatic Solved an insoluble problem Unencumbered by tradition (good & bad)
- 14. With this came a Different Focus Tradition No SQL • Global consistency • Local consistency • Schema driven • Schemaless • Reliable Network • Unreliable Network • Highly Structured • Semi-structured/ Unstructured NoSQL / Big Data technologies really focus on load and volume problems by avoiding the complexities associated with traditional transactional storage
- 15. The ‘Relational Camp’ had been busy too Realisation that the traditional architecture was insufficient for various modern workloads
- 16. End of an Era Paper - 2007 “Because RDBMSs can be beaten by more than an order of magnitude on the standard OLTP benchmark, then there is no market where they are competitive. As such, they should be considered as legacy technology more than a quarter of a century in age, for which a complete redesign and re-architecting is the appropriate next step.” – Michael Stonebraker
- 17. No Longer a One-Size-Fits-All
- 18. There is a new and impressive breed • Products ~ 5 years old • Shared nothing (sharded) • Designed for SSD’s & 10GE • Large address spaces (256GB+) • No indexes (column oriented) • Dropping traditional tenets (referential integrity etc) • Surprisingly quick for big queries when compared with incumbent technologies.
- 19. Both types of solution have clear value
- 20. ..and it’s not really a question of size TB 0 1 10 100 1000 10,000 The majority of us live in the overlap region
- 21. More a question of utility … which this tends to lead to composite offerings
- 23. So what does this mean for the enterprise?
- 24. 80% Enterprise Databases < 1TB This Reference is getting pretty old now, sorry (2009)
- 25. Yet we often have a lot of them
- 26. Communication is Store & Forward The outside world
- 27. Sometimes we’re a bit more organized!
- 28. But most of our data is not that accessible Core Operational Exposed Data
- 29. …and sharing is often an afterthought Core Operational Exposed Data
- 31. But as data is getting bigger and heavier
- 32. ..it can make it hard to join data together
- 33. So we often we turn to some form of Enterprise Data Warehouse (or maybe data virtualization)
- 34. Big data tech sometimes provides a composite solution (or ETL)
- 35. Ability to model data is much more of a gating factor than raw size Dave Campbell (Microsoft – VLDB Keynote 2012)
- 36. Importing data into a standard model is a slow and painful processs
- 37. An alternative is to use a Late Bound Schema
- 38. Combining structured & unstructured approaches in a layered fashion makes the process more nimble Structured Late Bound Standardisation Schema Layer Raw Data
- 39. We take this kind of approach • Grid of machines • Late bound schema • Sharded, immutable data • Low latency (real time) and high throughput (grid) use cases • All data is observable (an event) • Interfaces: Standardised (safe) or Raw (uayor)
- 40. Both Raw & Standardised data is available Operational Relational (real time / MR) Analytics Object/SQL Standardisation Raw Data
- 41. This helps to loosen the grip of the single schema, whilst also providing a more iterative approach to standardisation
- 42. Support for both one standardised and many bespoke models in the same technology Raw Facts from different systems Standardised Model
- 43. Next step: to centralise common processing tasks Standardised Risk Model Calculation
- 44. Are we back to the mainframe?