Managing computational resources with Apache Mesos
1 like702 views
The document discusses managing computational resources using Apache Mesos, emphasizing its ability to abstract resources for better efficiency in data centers. It outlines challenges such as orchestration complexity, resource isolation, and management in heterogeneous architectures. The presentation highlights the importance of automation and dynamic resource allocation, particularly for long-running jobs and frameworks like Marathon.
Managing computational resources with Apache Mesos
- 1. Managing Computational resources with Apache Mesos Jackson Oliveira @cyber_jso
- 2. Software Architect, more than 13 years working with I.T Currently Working at ilegra as consultant. SOA specialist. Believe on agile, devops principles, people transformation and open source community ideas behind. Like videogames and watch series. Football fun. Blog: http://jackson-s-oliveira.blogspot.com.br/ Linkedin: https://www.linkedin.com/in/jacksonsoliveira facebook:https://www.facebook.com/jackson.dossantos.5
- 3. Storymap traditional data centers resource usage …. 90s… A different scenario
- 4. Storymap traditional data centers resource usage …. 90s… Changes for datacenter adminstration
- 5. Storymap traditional data centers resource usage …. Virtualization, a bit game changing
- 6. Siloed cluster (static partition,low granularity ) no shared resource ... Brought a different issue… The siloed cluster
- 7. Siloed cluster (static partition,low granularity ) no shared resource ... Siloed cluster (static partitioning, low granularity) no shared resource Siloed data center Data center managed by mesos mesos
- 9. Management complexity (technology , hardware facts ) ... Management complexity (technology, hardware, facts) ..
- 10. Distributed systems added complexity
- 11. Releases demand more effort
- 13. Heterogeneous Archs on the same datacenter
- 14. Failures
- 16. Big Data - distribute is needed
- 17. Mesos - The datacenter operational system
- 26. Master nodes High Availability
- 28. Frameworks Ecosystem Long Running jobs Big Data Processing Batch Scheduling Data Storage
- 29. Frameworks can coexist on the same datacenter
- 35. Native isolation using Linux containers
- 37. CPU Share, Disk quotas and Bandwith limits CPU Disk Network - Core Isolation - Enforce maximum space usage limits - Limit I/O bandwidth usage - Ports
- 39. How to prevent frameworks starving? Before workload After workload
- 40. How to prevent frameworks starving?
- 41. Static reservation - Good for Stateful services
- 42. Dynamic reservation - Good for Stateful Scheduled tasks
- 46. Built to support long running jobs
- 47. Specificing contranstraints Good to ensure tasks are: ● Running all slaves ● Running At least one per datacenter ● Running on specific slaves ● Running at least one per cluster group
- 48. Scaling applications out New tasks can be allocated dinamically
- 52. Handling failures Reschedule the tasks!
- 53. ● Automation is heavily needed on this environment ● Troubleshooting can be tricky ● Monolithic systems may not take full advantage from this solution ● Ops: supporting the platform rather than specific products ● Applications that demands specific SO and hardware improvements may wont leverage the benefits from this approach Considerations
- 54. Managing Computational resources with Apache Mesos Jackson Oliveira @cyber_jso Cesar Mesquita @cmesquita00 Thank you!