How to ensure Presto scalability in multi use case
Download as PPTX, PDF7 likes4,165 views
This document discusses how to ensure Presto scalability in multi-use case environments. It describes how Treasure Data uses Prestobase Proxy, a Finagle-based RPC proxy, to provide a scalable interface for BI tools. It also discusses Presto's node scheduler for distributing query stages across nodes and Treasure Data's use of resource groups to limit resource usage and isolate queries. The document advocates for approaches like dependency injection, VCR testing, and multi-dimensional resource scheduling to make Presto and its components reliable in distributed systems.
![Finagle
protected val service: Service[Request, Response] =
bind[SomeFilter] andThen
bind[AnotherHandler] andThen
LastFilter andThen
prestoClient
Build request pipeline by binding
filter, handlers with Airframe](https://image.slidesharecdn.com/stablelargescalepresto-170112142957/85/How-to-ensure-Presto-scalability-in-multi-use-case-11-320.jpg)
![Airframe
val design : Design =
newDesign
.bind[X].toInstance(new X) // Bind type X to a concrete instance
.bind[Y].toSingleton // Bind type Y to a singleton object
.bind[Z].to[ZImpl] // Bind type Z to an instance of ZImpl
import wvlet.airframe._
trait App {
val x = bind[X]
val y = bind[Y]
val z = bind[Z]
// Do something with X, Y, and Z
}
val session = design.newSession
val app : App = session.build[App]](https://image.slidesharecdn.com/stablelargescalepresto-170112142957/85/How-to-ensure-Presto-scalability-in-multi-use-case-14-320.jpg)
![VCR testing framework
protected val service: Service[Request, Response] =
bind[SomeFilter] andThen
bind[AnotherHandler] andThen
QueryRewriter andThen
bind[RequestVCR] andThen
prestClient
protected val service: Service[Request, Response] =
bind[SomeFilter] andThen
bind[AnotherHandler] andThen
QueryRewriter andThen
bind[NoRecording] andThen
prestClient
On CI
On Production](https://image.slidesharecdn.com/stablelargescalepresto-170112142957/85/How-to-ensure-Presto-scalability-in-multi-use-case-16-320.jpg)
![{
"runningQueryIds": ["query1", "query2"],
"accountId": 1,
"children": [{
"memoryUsage": 12345,
"runningQueryIds": [“query1"],
"children": [],
"runningQueries": 1,
"queuedQueries": 0,
"maxRunningQueries": 2,
"resourceId": "general"
}, {
"memoryUsage": 26296,
"runningQueryIds": ["query2"],
"children": [],
"runningQueries": 1,
"queuedQueries": 0,
"maxRunningQueries": 2,
"resourceId": "scheduled"
}],
"runningQueries": 2,
"maxRunningQueries": 30,
}
Queries in parent group
Running query in general
Running query in scheduled](https://image.slidesharecdn.com/stablelargescalepresto-170112142957/85/How-to-ensure-Presto-scalability-in-multi-use-case-36-320.jpg)
How to ensure Presto scalability in multi use case
- 1. How to ensure Presto scalability in multi use case Kai Sasaki Treasure Data Inc.
- 2. Kai Sasaki (@Lewuathe) Software Engineer at Treasure Data Inc. Hadoop/Presto/Spark
- 3. Presto In TD • 150000+ queries / day • 190+ TB processing / day • 10+ MB processing / query * sec • 100+ million processed records / query
- 4. Presto In TD Prestobase Proxy PerfectQueue query Plazma data Presto TD API BI Tool HTTP
- 5. How to make it scalable • Prestobase Proxy • Node scheduler • Resource Group
- 7. Prestobase proxy Prestobase proxy aims to provide the interface especially for BI tools through JDBC/ODBC and also to replace Prestogres.
- 8. Presto In TD Prestobase Proxy PerfectQueue query Plazma data Presto TD API BI Tool HTTP
- 9. Prestobase proxy • Written in Scala • Finagle base RPC proxy • Running as Docker container • A user of Airframe • VCR base light-weight test framework
- 10. Finagle Finagle is an extensible RPC system for the JVM, used to construct high-concurrency servers. Finagle implements uniform client and server APIs for several protocols, and is designed for high performance and concurrency. see: https://twitter.github.io/finagle/
- 11. Finagle protected val service: Service[Request, Response] = bind[SomeFilter] andThen bind[AnotherHandler] andThen LastFilter andThen prestoClient Build request pipeline by binding filter, handlers with Airframe
- 12. Airframe Airframe is a trait base dependency injection framework using Scala macro - https://github.com/wvlet/airframe
- 13. Airframe - Dependency injection tailored Scala - Tagged binding with wvlet https://github.com/wvlet/wvlet - Object lifecycle management
- 14. Airframe val design : Design = newDesign .bind[X].toInstance(new X) // Bind type X to a concrete instance .bind[Y].toSingleton // Bind type Y to a singleton object .bind[Z].to[ZImpl] // Bind type Z to an instance of ZImpl import wvlet.airframe._ trait App { val x = bind[X] val y = bind[Y] val z = bind[Z] // Do something with X, Y, and Z } val session = design.newSession val app : App = session.build[App]
- 15. VCR testing framework Record test suite HTTP interaction to make test stable and deterministic see more detail https://testing.googleblog.com/2016/11/what-test-engineers-do-at-google.html
- 16. VCR testing framework protected val service: Service[Request, Response] = bind[SomeFilter] andThen bind[AnotherHandler] andThen QueryRewriter andThen bind[RequestVCR] andThen prestClient protected val service: Service[Request, Response] = bind[SomeFilter] andThen bind[AnotherHandler] andThen QueryRewriter andThen bind[NoRecording] andThen prestClient On CI On Production
- 19. Prestobase proxy Will be open sourced soon
- 20. Node Scheduler
- 21. Node Scheduler Submitting query follows… - Analyze query AST - Make query logical/physical plan - Schedule each stage
- 22. Node Scheduler query stage2 stage1 stage0 task2-0 task2-1 task2-0 task1-0 task1-1 task0-0 Table Scan output
- 23. Node Scheduler NodeScheduler creates NodeSelector that selects worker nodes on which tasks are scheduled. NodeSelector picks up worker nodes when there is available splits.
- 24. Node Scheduler in TD Keeps worker node map that can be candidate for launching next tasks. - Ignore min candidates - Limit by available memory pool
- 25. Node Scheduler in TD Back to normal memory pool usage after task is completed.
- 26. Node Scheduler in TD Challenges - Smoothing CPU time metric - Split type awareness - Avoid problematic worker nodes
- 27. Resource Group
- 28. Resource Group Resource Group was introduced since 0.147 → https://prestodb.io/docs/current/admin/resource-groups.html Resource Group aims to limit the resource usage by account/group/query.
- 29. Resource Group rootGroup general adhoc softMemoryLimit: 100% maxQueued : 5000 maxRunning : 1000 softMemoryLimit: 100% maxQueued : 100 maxRunning : 200 softMemoryLimit: 100% maxRunning : 1000
- 30. Resource Group limits - maxQueued - maxRunning - softMemoryLimit Following queries will be queued - softCpuLimit Impose penalty against max running queries - hardCpuLimit Following queries will be queued
- 31. Resource Group scheduling - schedulingPolicy - fair : FIFO - weighted : Selected stochastically - query_priority : Selected according to priority - schedulingWeight
- 32. Resource Group Every query must be associated to a resource group. The matching can be done by configured selector. { "user": “bob", "group": "general" }, { "source": “.*adhoc.*", "group": "global.adhoc.adhoc_${USER}" }
- 33. Resource Group rootGroup general adhoc softMemoryLimit: 100% maxQueued : 5000 maxRunning : 1000 softMemoryLimit: 100% maxQueued : 100 maxRunning : 200 softMemoryLimit: 100% maxRunning : 1000 Bob’s query Bob’s query …
- 34. Resource Group DI Easily change resource group config behavior with Guice injection. - ResourceGroupConfigurationManager - configure(ResourceGroup, SelectionContext) - ResourceGroupSelector - match(Statement, SelectionContext)
- 35. SelectionContext SelectionContext holds the information for associating submitted query. - Authenticated - User - Source - Query Priority Currently available as default
- 36. { "runningQueryIds": ["query1", "query2"], "accountId": 1, "children": [{ "memoryUsage": 12345, "runningQueryIds": [“query1"], "children": [], "runningQueries": 1, "queuedQueries": 0, "maxRunningQueries": 2, "resourceId": "general" }, { "memoryUsage": 26296, "runningQueryIds": ["query2"], "children": [], "runningQueries": 1, "queuedQueries": 0, "maxRunningQueries": 2, "resourceId": "scheduled" }], "runningQueries": 2, "maxRunningQueries": 30, } Queries in parent group Running query in general Running query in scheduled
- 37. Recap Distributed system often requires each component to be stable and scalable. We can make Presto ecosystem reliable by doing… - Code modification reliability with DI - VCR testing - Multi dimensional resource scheduling - Resource isolation makes multi-tenant distributed SQL engine reliable