Nelson: Rigorous Deployment for a Functional World
3 likes1,499 views
The document outlines the principles and goals of the Nelson automated multi-region container deployment system, emphasizing the importance of rigorous systems and principles of formal verification. It discusses challenges in current deployment strategies and highlights best practices for automating application lifecycles while ensuring security and efficient management of resources. Key features include a streamlined deployment process, active lifecycle management, and comprehensive auditing to improve overall platform complexity and team responsibility.
![Vertex.
vertex[N,L]](https://image.slidesharecdn.com/scale-by-the-bay-reduex2-171118185707/85/Nelson-Rigorous-Deployment-for-a-Functional-World-28-320.jpg)
![Directed Edge.
LEdge[N,A]
Edge[N]
A B](https://image.slidesharecdn.com/scale-by-the-bay-reduex2-171118185707/85/Nelson-Rigorous-Deployment-for-a-Functional-World-31-320.jpg)
![Workflow.
Coproduct[F[_], G[_], A](run: F[A] / G[A])](https://image.slidesharecdn.com/scale-by-the-bay-reduex2-171118185707/85/Nelson-Rigorous-Deployment-for-a-Functional-World-73-320.jpg)
![Workflow.
type Op0[A] = Coproduct[DockerOp, ConsulOp, A]
…
type WorkflowOp[A] = Coproduct[SchedulerOp, Op4, A]
type WorkflowF[A] = Free.FreeC[WorkflowOp, A]](https://image.slidesharecdn.com/scale-by-the-bay-reduex2-171118185707/85/Nelson-Rigorous-Deployment-for-a-Functional-World-74-320.jpg)
Nelson: Rigorous Deployment for a Functional World
- 1. Nelson automated, multi-region container deployment timperrett
- 3. Rig·or·ous. Extremely thorough, exhaustive, or accurate.
- 5. – Dr Tara Smith “Pragmatism is the opposition to principal, on principle.”
- 6. Rigorous Systems. • Theory is the foundation. • Category Theory, Combinatorics, Set Theory, Group Theory… • Principles of formal verification becoming more widespread. • Type systems act as proofs about your program. • Amazon S3 (among others) have TLA+ models. • Never fear practical application of rigorous systems. • Invest now to save your sanity tomorrow (i.e. 2am PagerDuty alert).
- 11. Deployment. sticky tape, bandages and bash scripts. antithesis of rigor
- 12. Problem • Provisioning applications is still too slow (bare metal or cloud). • Runtime traffic control systems are medieval at best. • Coupling CI and CD creates monolithic operational systems. • These systems do everything. This is a distinct problem. • Current market solutions limited or hard to adopt. • Most teams have brittle, painful automation nobody wants to use. • Many teams attempt CD ignorant of the side-effects.
- 13. Lessons • Automate every part of the system. • Testing a distributed system locally is a fable. • Emergent properties. Scaling issues etc. • Uniformity is highly desirable and wildly advantageous. • Beautiful, unique snowflakes are however, inevitable. • Automated lifecycle management is required.
- 14. Goals • Use the minimally powerful components. • System elements should be awesome at just one thing. • Reduce overall platform complexity. • Increase responsibility of engineering teams. Break it, you bought it. • Decentralize process gatekeepers. • No build team. No ticket filing for deployments. No configuration management.
- 15. Goals • All application specifications are checked in. • Build. Deployment. Alerting etc. • Reduce deployment time to 2 minutes or less. • Support multi-DC topologies from the get-go. • Automatic credential management and secure-introduction • Transparent, strong encryption for application I/O on the wire.
- 16. Nelson.
- 17. – Vice Admiral Horatio Nelson, 1758-1805 “Desperate affairs require desperate remedies.”
- 18. – Vice Admiral Horatio Nelson, 1758-1805 “Desperate affairs require desperate remedies.” #opslife
- 19. Overview • Provides “best practices” for your organization; focus on shipping. • Github driven developer workflow (.com or enterprise). • Choose whatever build / CI system you want. • State of the art runtime routing via Envoy. • Secure introduction for safe distribution of credentials from Vault. • Target any datacenter running a scheduler (Nomad, Mesos, etc).
- 22. Overview. users
- 24. Features. • Declarative, simplistic YAML manifest per-repository. • Active lifecycle management. • Management of edge ingress (e.g. public load balancers). • Optional routing control plane. • Comprehensive auditing. • Extensible with arbitrary scheduling, routing or artifact systems.
- 26. Mathematics. Combinatorics. Graph Theory. https://github.com/Verizon/quiver
- 27. Vertex.
- 29. A Edge. B
- 32. Graph.
- 33. Monoids. A B CD A F G H (A, A) => A A => A
- 35. A D C B E depends ondepends on depends on depends on
- 36. depends ondepends on depends on depends on
- 37. depends ondepends on depends on depends on [email protected] [email protected] [email protected] [email protected] [email protected]
- 38. Graphs. • Every application declares their dependencies a-priori. • No routing to disconnected vertices. • No Vault policy attributes for disconnected vertices. • Edge ingress points are the “root” of a graph. • Intersecting graphs with shared or discrete roots. • Provides anchor-point for lifecycle management. • Entire world view falls out of a union of all discrete graphs.
- 39. Lifecycle. deployment is the easy part.
- 40. Lifecycle. Amazon in 2008 https://twitter.com/Werner/status/741673514567143424
- 42. based on consul
- 43. typical state
- 44. pending GC
- 45. user activated
- 46. pluggable
- 47. borrowed time
- 49. Graph Pruning
- 50. X X Upgraded!
- 51. last two major revsXX X
- 53. Security. that thing you wish you did better. graphs help here too!
- 54. Security. • Derivation of security policy, as opposed to proclamation of policy. • Excellent audibility • All dependencies are known - internal and external. • Every single application instance has unique credentials. • Dynamic certificates, dynamic DB access etc. • Compromised containers have the minimal attack surface. • Joined with a service mesh, transparent mTLS for everyone is a breeze.
- 55. Namespaces.
- 56. machines
- 57. scheduler
- 58. namespaces
- 61. root namespace
- 62. qa/unstable
- 63. qa/staging/tim
- 66. Discovery. • Discovery protocol written to Consul KV for every stack • We call this Lighthouse protocol • Application dependencies are declared a-priori. • You cannot route to that which you do not tell Nelson about. • Makes for awesome auditing and security. • Language implementations need only consume the protocol.
- 67. Routing. • Non-prescriptive approach to routing tier implementation. • Provides a control plane protocol describe routing actions. • Typically implemented with Envoy, but you can choose. • Minor application changes required. • Incentivized these with tracing and context propagation. • Models traffic shifting as a time vs traffic policy curve.
- 69. Workflow. how the sausage is made.
- 70. Workflow. HTTPQueue Workflow X Workflow Y End End Executor Pool GitHub
- 71. Workflow. and thenand then start replicate containers create vault policies ??? and then
- 72. Workflow. Looks like a graph. Walks like a graph. Must be a… coproduct!
- 73. Workflow. Coproduct[F[_], G[_], A](run: F[A] / G[A])
- 74. Workflow. type Op0[A] = Coproduct[DockerOp, ConsulOp, A] … type WorkflowOp[A] = Coproduct[SchedulerOp, Op4, A] type WorkflowF[A] = Free.FreeC[WorkflowOp, A]
- 75. Workflow. for { _ <- status(id, Pending, "workflow about to start") i <- dockerOps(id, unit, dc.docker.registry) _ <- status(id, Deploying, s"writing alert definitions to ${dc.name}'s consul") _ <- writePolicyToVault(cfg = dc.policy, sn = sn, ns = ns.name, rs = rs) /////////////////////////////// // remove for space limitations /////////////////////////////// l <- launch(i, dc, ns.name, vunit, p, hash) _ <- debug(s"response from scheduler $l”) _ <- status(id, getStatus(unit, p), "======> workflow completed <======") } yield () Free monad
- 76. Challenges • Non-trivial level of investment and execution. • Tight integration with Hashistack is both pro or con. • Community adding k8s + Istio support % • Containerizing legacy applications can be “interesting”. • Migration can be a challenge if not collocated with “the new world”. • Small organizations better served by existing solutions.
- 77. Summary • Provide rigor to your application Death Star. • Fully automated application lifecycle: no manual housekeeping. • Choose whatever CI setup best fits your team. • Secure your deployments. • Transparent mTLS and rotating credentials. • Automatic Vault policy management.
- 79. Related Links. • Nelson documentation site: verizon.github.io/nelson/ • Nelson CLI: https://github.com/Verizon/nelson-cli • Quiver: Scala port of Haskell FGL https://github.com/Verizon/quiver • Slipway: integrate any CI system with Nelson https://github.com/Verizon/slipway