Automated Spark Deployment With Declarative Infrastructure
2 likes860 views
The document introduces Quilt, a system that allows for the automated deployment of distributed applications across multiple cloud infrastructures using a declarative specification language called Stitch. Stitch allows operators to specify applications, infrastructure resources, and their connections in a modular way. Quilt then deploys the application according to the Stitch specification across available infrastructure providers, providing portability. Future work with Quilt includes adding additional domain-specific primitives to Stitch and analyzing specifications to verify properties like reachability and availability.
![WordPress
wp-5: quay.io/netsys/di-wordpress
memcd-2: quay.io/netsys/di-memcached memcd-1: quay.io/netsys/di-memcacheddb-dbs-3: quay.io/netsys/di-wp-mysqldb-dbs-2: quay.io/netsys/di-wp-mysql
db-dbm-1: quay.io/netsys/di-wp-mysql
memcd-0: quay.io/netsys/di-memcached
spark-wk-0: quay.io/netsys/spark
spark-wk-2: quay.io/netsys/spark
spark-ms-0: quay.io/netsys/spark
spark-ms-1: quay.io/netsys/spark
spark-wk-3: quay.io/netsys/spark
spark-wk-1: quay.io/netsys/spark
k-2: quay.io/netsys/zookeeper
zk-1: quay.io/netsys/zookeeper
zk-0: quay.io/netsys/zookeeper
wp-0: quay.io/netsys/di-wordpress wp-3: quay.io/netsys/di-wordpresswp-7: quay.io/netsys/di-wordpress
hap-0: quay.io/netsys/di-wp-haproxy
wp-6: quay.io/netsys/di-wordpress wp-4: quay.io/netsys/di-wordpress wwp-1: quay.io/netsys/di-wordpress
hap-1: quay.io/netsys/di-wp-haproxy
public: [ ]
ZooKeeper
Spark
HAProxy
MySQL
WordPress
Memcached
Ethan J. Jackson Aurojit Panda Kevin Lin Johann Schleier-Smith
Nicholas Sun⇤
Luise Valentin⇤
Yuen Mei Wan⇤
Scott Shenker⇤†
⇤
UC Berkeley †
ICSI
Abstract
Despite the recent emergence of container orchestrators and
software defined networks, operators still face daunting chal-
lenges managing their distributed systems. In this paper we
present Stitch, a new language that specifies distributed system
policy directly, and Quilt, a system that automatically deploys
Stitch specifications on whatever infrastructure is available.
By disentangling application policy from application infras-
tructure, Quilt supports portable distributed applications and
automatically enforces strict network isolation.
1 Introduction
In recent years it has become easier to deploy distributed sys-
tems. Script-friendly cloud APIs [1, 50, 35, 20] and container
orchestrators [25, 33, 12, 52, 46, 7] allow administrators to
1 (import "haproxy")
2 (import "memcached")
3 (import "mysql")
4 (import "spark")
5 (import "wordpress")
6 (import "zookeeper")
7
8 (let ((db (mysql.New "db" 2))
9 (memcd (memcached.New "memcd" 3))
10 (wp (wordpress.New "wp" 8 db memcd))
11 (hap (haproxy.New "hap" 2 wp))
12 (zk (zookeeper.New "zk" 3))
13 (spark (spark.New "spark" 2 4 zk)))
14 (connect 7077 (hmapValues spark)
15 (hmapValues db))
16 (connect 80 "public" hap))
Figure 1: Stitch specification for a complex multi-t
WordPress deployment motivated in detail in §2.](https://image.slidesharecdn.com/8ycethanjackson-160615011322/85/Automated-Spark-Deployment-With-Declarative-Infrastructure-16-320.jpg)
![Stitch — Primitives
spark-master: quilt/spark
spark-worker: [ 10 quilt/spark ]](https://image.slidesharecdn.com/8ycethanjackson-160615011322/85/Automated-Spark-Deployment-With-Declarative-Infrastructure-26-320.jpg)
Automated Spark Deployment With Declarative Infrastructure
- 1. Quilt Ethan J. Jackson, Aurojit Panda, Kevin Lin, Johann Schleier-Smith, Nicholas Sun, Luise Valentin, Yuen Mei Wan, Scott Shenker quilt.io
- 2. Everything has an API
- 3. Compute
- 4. Network
- 5. DevOps 1. Choose a Compute API 2. Choose a Network API 3. Write a Deployment Script
- 7. spark-ec2.py • Official Spark Script • 1528 Lines of Code • Incomprehensible
- 8. Network Security • Status Quo – Secure the Perimeter • A Better Way – Tight East-West Firewall – Increased script complexity
- 11. Quilt DSL: Stitch • Declarative Application Specification • Lisp Dialect • Declaration Includes: – Application Network and Compute – Infrastructure
- 13. WordPress Yuen Mei Wan⇤ Scott Shenker⇤† rkeley † ICSI nd hal- we em oys ble. as- nd 1 (import "haproxy") 2 (import "memcached") 3 (import "mysql") 4 (import "spark") 5 (import "wordpress") 6 (import "zookeeper") 7 8 (let ((db (mysql.New "db" 2)) 9 (memcd (memcached.New "memcd" 3)) 10 (wp (wordpress.New "wp" 8 db memcd)) 11 (hap (haproxy.New "hap" 2 wp)) 12 (zk (zookeeper.New "zk" 3)) 13 (spark (spark.New "spark" 2 4 zk))) 14 (connect 7077 (hmapValues spark) 15 (hmapValues db)) 16 (connect 80 "public" hap)) ZooKeeper Spark HAProxy MySQL WordPress Memcached
- 14. wordpress.NewDeclarative Application Specification with Quilt . Jackson⇤ Aurojit Panda⇤ Kevin Lin⇤ Johann Schleier-Smith⇤ olas Sun⇤ Luise Valentin⇤ Yuen Mei Wan⇤ Scott Shenker⇤† ⇤ UC Berkeley † ICSI emergence of container orchestrators and etworks, operators still face daunting chal- heir distributed systems. In this paper we w language that specifies distributed system Quilt, a system that automatically deploys ns on whatever infrastructure is available. 1 (import "haproxy") 2 (import "memcached") 3 (import "mysql") 4 (import "spark") 5 (import "wordpress") 6 (import "zookeeper") 7 8 (let ((db (mysql.New "db" 2)) 9 (memcd (memcached.New "memcd" 3)) 10 (wp (wordpress.New "wp" 8 db memcd)) 11 (hap (haproxy.New "hap" 2 wp)) 1 (define (New name n db memcd) 2 (let ((dk (makeList n (docker image))) 3 (labelNames (strings.Range name n)) 4 (wp (map label labelNames dk))) 5 (configure wp db memcd) 6 (connect 3306 wp (hmapGet db "master")) 7 (connect 3306 wp (hmapGet db "slave")) 8 (connect 11211 wp memcd) 9 wp))
- 15. WordPress Yuen Mei Wan⇤ Scott Shenker⇤† rkeley † ICSI nd hal- we em oys ble. as- nd 1 (import "haproxy") 2 (import "memcached") 3 (import "mysql") 4 (import "spark") 5 (import "wordpress") 6 (import "zookeeper") 7 8 (let ((db (mysql.New "db" 2)) 9 (memcd (memcached.New "memcd" 3)) 10 (wp (wordpress.New "wp" 8 db memcd)) 11 (hap (haproxy.New "hap" 2 wp)) 12 (zk (zookeeper.New "zk" 3)) 13 (spark (spark.New "spark" 2 4 zk))) 14 (connect 7077 (hmapValues spark) 15 (hmapValues db)) 16 (connect 80 "public" hap)) ZooKeeper Spark HAProxy MySQL WordPress Memcached
- 16. WordPress wp-5: quay.io/netsys/di-wordpress memcd-2: quay.io/netsys/di-memcached memcd-1: quay.io/netsys/di-memcacheddb-dbs-3: quay.io/netsys/di-wp-mysqldb-dbs-2: quay.io/netsys/di-wp-mysql db-dbm-1: quay.io/netsys/di-wp-mysql memcd-0: quay.io/netsys/di-memcached spark-wk-0: quay.io/netsys/spark spark-wk-2: quay.io/netsys/spark spark-ms-0: quay.io/netsys/spark spark-ms-1: quay.io/netsys/spark spark-wk-3: quay.io/netsys/spark spark-wk-1: quay.io/netsys/spark k-2: quay.io/netsys/zookeeper zk-1: quay.io/netsys/zookeeper zk-0: quay.io/netsys/zookeeper wp-0: quay.io/netsys/di-wordpress wp-3: quay.io/netsys/di-wordpresswp-7: quay.io/netsys/di-wordpress hap-0: quay.io/netsys/di-wp-haproxy wp-6: quay.io/netsys/di-wordpress wp-4: quay.io/netsys/di-wordpress wwp-1: quay.io/netsys/di-wordpress hap-1: quay.io/netsys/di-wp-haproxy public: [ ] ZooKeeper Spark HAProxy MySQL WordPress Memcached Ethan J. Jackson Aurojit Panda Kevin Lin Johann Schleier-Smith Nicholas Sun⇤ Luise Valentin⇤ Yuen Mei Wan⇤ Scott Shenker⇤† ⇤ UC Berkeley † ICSI Abstract Despite the recent emergence of container orchestrators and software defined networks, operators still face daunting chal- lenges managing their distributed systems. In this paper we present Stitch, a new language that specifies distributed system policy directly, and Quilt, a system that automatically deploys Stitch specifications on whatever infrastructure is available. By disentangling application policy from application infras- tructure, Quilt supports portable distributed applications and automatically enforces strict network isolation. 1 Introduction In recent years it has become easier to deploy distributed sys- tems. Script-friendly cloud APIs [1, 50, 35, 20] and container orchestrators [25, 33, 12, 52, 46, 7] allow administrators to 1 (import "haproxy") 2 (import "memcached") 3 (import "mysql") 4 (import "spark") 5 (import "wordpress") 6 (import "zookeeper") 7 8 (let ((db (mysql.New "db" 2)) 9 (memcd (memcached.New "memcd" 3)) 10 (wp (wordpress.New "wp" 8 db memcd)) 11 (hap (haproxy.New "hap" 2 wp)) 12 (zk (zookeeper.New "zk" 3)) 13 (spark (spark.New "spark" 2 4 zk))) 14 (connect 7077 (hmapValues spark) 15 (hmapValues db)) 16 (connect 80 "public" hap)) Figure 1: Stitch specification for a complex multi-t WordPress deployment motivated in detail in §2.
- 17. Infrastructure 1 (define cfg 2 (list (provider "Amazon") (region "us-west-1") 3 (ram 32 64) (cpu 4 8) (sshkey "elided"))) 4 5 (makeList 3 (machine (role "Master") cfg)) 6 (makeList 32 (machine (role "Worker") cfg))
- 18. Infrastructure ? 1 (define cfg 2 (list (provider "Amazon") (region "us-west-1") 3 (ram 32 64) (cpu 4 8) (sshkey "elided"))) 4 5 (makeList 3 (machine (role "Master") cfg)) 6 (makeList 32 (machine (role "Worker") cfg))
- 19. 1 (define cfg 2 (list (provider "Amazon") (region "us-west-1") 3 (ram 32 64) (cpu 4 8) (sshkey "elided"))) 4 5 (makeList 3 (machine (role "Master") cfg)) 6 (makeList 32 (machine (role "Worker") cfg)) Infrastructure Azure Central US ?
- 21. Geographical Distribution Figure 4 shows a simple way an Stitch operator may instanti- ate our WordPress example. In addition to the application spec- 1 (define cfg (list (ram 32 64) (cpu 4 8) 2 (sshkey "<elided>"))) 3 4 (define db (mysql.New "db" 2)) 5 (define zk (zookeeper.New "zk" 3)) 6 (define spark (spark.New "spark" 2 4 zk)) 7 (connect 7077 (hmapValues spark) (hmapValues db)) 8 9 (define (makeLoc prvd rgn) 10 (list (provider prvd) (region rgn))) 11 12 (define (makePod name) 13 (let ((memcd (memcached.New (+ name "-mem") 1)) 14 (wp (wordpress.New (+ name "-wp") 15 2 db memcd)) 16 (hap (haproxy.New (+ name "-hap") 1 wp))) 17 (connect 80 "public" hap) 18 (list memcd wp hap))) 19 20 (define (deploy pod loc) 21 (makeList 16 (machine (role "Worker") cfg loc)) 22 (place (machineRule "on" loc) pod)) 23 24 (deploy (makePod "gce") 25 (makeLoc "Google" "europe-west1-b")) 26 27 (deploy (makePod "azure") 28 (makeLoc "Azure" "Central US")) 29 30 (let ((loc (makeLoc "Amazon" "ap-southeast-2")) 31 (nodes (append (makePod "aws") zk 32 (hmapValues db) 33 (hmapValues spark)))) 34 (machine (role "Master") cfg loc) 35 (deploy nodes loc))
- 22. Stitch
- 23. Stitch • Lisp (Scheme) – Variables – Arithmetic – Functions – Modules • Domain Specific Primitives
- 24. Stitch — Primitives • Application Primitives – “docker”, “label”, “connect”, “place”, “setEnv” • Infrastructure Primitives – “machine” – “role”, “provider”, “region”, “ram”, “cpu”, “size”
- 26. Stitch — Primitives spark-master: quilt/spark spark-worker: [ 10 quilt/spark ]
- 28. Goals • Simple • Robust • Portable
- 30. • Import Infrastructure Spec • Update Cluster • Cloud Provider Plugins – Amazon EC2 – Google Compute Engine – Microsoft Azure Infrastructure Controller Cluster AWS Azure Foreman DatabaseEngine VM AWS GCE VM VM Azure VM VM GCE VM
- 31. Cloud Provider • Boot, Stop, List • Network Reachability • Application Agnostic
- 32. • Virtual Machines Running … • Application Containers • Open Virtual Network – SDN Overlay • Infrastructure Agnostic Quilt Cluster
- 33. Unsolved Problems • Application Configuration • Container Security • State • External Services
- 34. Related Work
- 35. Related Work • Container Orchestrators – Kubernetes, Docker Swarm, Mesos, Nomad – No explicit application specification – No tight network firewall • Quilt is a policy layer above these systems
- 36. Related Work • Docker Compose / Kubernetes Helm – Declare Groups of Containers to Boot • Static Data Serialization Format – Poor modularity • Missing network graph
- 37. Future Work
- 38. Stitch: New Domains • Security policy – Key Management – User Management • Data • Application Configuration
- 39. Stitch Analysis • Verification – Stitch specifies app entirely – Simpler to verify than deployed systems • Reachability • Availability
- 40. Summary • Portable Application Deployment • Strict Network Security • Modular, Shareable, Reusable Specifications • In Future — Formal Analysis