Build your own clouds with Chef and MCollective
14 likes2,992 views
The document discusses building cloud infrastructure using tools like Chef and MCollective, highlighting their roles in configuration, deployment, and management. It covers the components involved in cloud computing, including automation, host configuration, and communication responsibilities. The content also provides insights into Chef's workflow, configuration elements, and MCollective's architecture for orchestrating server tasks.
Build your own clouds with Chef and MCollective
- 1. Build your own clouds with Chef & MCollective Jonathan Weiss
- 2. Who am I? Working for Peritor in Berlin, Germany Written, maintain, or involved in Webistrano Capistrano SimplyStored Happening The great fire of London http://github.com/jweiss @jweiss 2
- 3. Scalarium Amazon EC2 Cluster Management Auto-Config Self-Healing Auto-Scaling One-click-deployment www.scalarium.com 3
- 4. Cloud Computing Cloud Computing 4
- 5. 5
- 6. 6
- 7. What you get Open Solaris Oracle Suse Windows Linux Ubuntu CentOS Red IBM Hat Infrastructure as a Service
- 8. What you really want Load balancer App server Database Fully configured stack of servers, dependencies, firewalls, … + your own application deployed
- 9. Automation is needed for • Configuration • Healing • Scaling • Deployment flickr.com/photos/torek/3788181603
- 11. Reasons to build your own In-house capacity Governance: Security and regulations Flexibility (or lack of…) 11
- 12. Ingredients 12
- 13. Components Communication Host configuration Command & Control
- 14. Command & Control Responsibilities: Communication with IaaS API User Interface (Web, CLI, API): Query, deploy, start, stop Repository of projects &configuration Controls which hosts do what&when 14
- 15. Command & Control Responsibilities: Implementation: Communication with IaaS API Fog & other API implemenations User Interface (Web, CLI, API): Rails Query, deploy, start, stop Repository of projects &configuration CouchDB Controls which hosts do what&when Async wokers & agents 15
- 16. Communication Responsibilities: Distribute commands and responses in a scalable & fault tolerant way 16
- 17. Communication Responsibilities: Implementation: Distribute commands and responses MCollective / ActiveMQ in a scalable & fault tolerant way 17
- 18. Host Configuration Responsibilities: Actually executes commands Responds to cluster changes Ensures local host is configured according to commands: Install software, start/stop services, ... 18
- 19. Host Configuration Responsibilities: Implementation: Actually executes commands Chef Responds to cluster changes MCollective Agents Ensures local host is configured according to commands: Install software, start/stop services, ... 19
- 20. Host Configuration Bootstrapping new servers 20
- 21. Chef Open Source Provisioning Tool Configures your servers Cookbooks & recipes Ruby DSL 21
- 22. Chef 22
- 23. Chef Server 23
- 24. Chef Solo 24
- 25. Chef-Solo vs. Chef-Server Chef-Server Complex No concept of different environments, e.g. staging vs. production Didn‘t want to run one Chef server per project (and env) No lifecycle events, e.g. deployment vs. bootstrap Pull vs. push Chef-Solo MCollective agents to push cookbooks and Chef runs 25
- 26. Cookbooks Cookbooks apache2 memcached mysql rails 26
- 27. Memcached Cookbook Cookbooks memcachd attributes recipes templates 27
- 28. Memcached Cookbook memcached attributes default.rb recipes service.rb install.rb templates default memcached.conf.erb 28
- 29. Attributes cookbooks/memcached/attributes/default.rb Set default values for variables to use in recipes & templates 29
- 33. Chef Solo Run Configuration JSON: /etc/chef/solo.json Run Chef Solo 33
- 34. Available Resources Cookbook File Ifconfig SCM Cron Link Script Deploy Log Service Directory Mdadm Subversion Erlang Call Mount Template Execute Package User File Remote Directory Git Remote File Group Route HTTP Request Ruby Block 34
- 35. Deployment with Chef Deploy resource Compatible with Capistrano layout Supports Capistrano hooks via deploy/hook_name.rb Handles source checkout & callbacks Call custom script Arbitrary options, e.g: Download WAR file Compile Source … Automatically wrapped in Chef error handling & notifications 35
- 36. Deploy Resource 36
- 37. Communication Relaying messages to servers 37
- 38. Marionette Collective „a framework to build server orchestration or parallel job execution systems“ 38
- 39. MCollective Architecture 39
- 40. RPC Calls Service: chef-run Service: update-cookbooks Client Discover agents Service: chef-run Service: update-cookbooks Message Routing Agents
- 41. RPC Calls Service: chef-run Service: update-cookbooks Client Service: chef-run Service: update-cookbooks Call agents Message Routing Agents
- 42. MCollective Agent 42
- 43. Calling an Agent 43
- 44. Calling an Agent 44
- 45. Filters Facts Limit responding agents by filters of facts/classes: Classes Arbitary used defined settings role Can be set by userdata Facts Information about local machine Plugins for Chef/Ohai, Puppet/Facter Examples: Linux version, installed packages, .... 45
- 46. Filters on the Command Line 46
- 47. Filters in Code 47
- 48. MCollective Chef Call Chef action on remote agents/machines 48
- 49. MCollective Chef Agent running Chef 49
- 50. Security Clients Stomp / ActiveMQ user AES+RSA plugins: encrypt sign every message SSL plugin: sign every message TLS for connection encryption Middleware Topic permissions and subclusters Agents All client security measures available RPC authorization and auditing 50
- 51. Scalability Redundancy ActiveMQ offers different means Network of brokers Broker clusters Master/Slave 51
- 52. Command Control Controlling the cloud 52
- 53. Command Software General capabilities Stores cluster model Interacts with IaaS API (EC2, VMWare friends) Handles recovery presence Integrates communication solution Generates host configuration / Chef events Custom requirements User management authorization Business rules processes Deployment recipes scripts 53
- 54. Server Roles 54
- 55. Command Software 55
- 56. Experiences Chef Flexible and powerful Easy to learn first steps - easy to write spaghetti recipes Some annoyances: pseudo idempotent and 2 phases “Write once, test everywhere” MCollective Simple and robust Missing presences and events (Nanite has them but has different problems) Integrated with Puppet in the future? 56
- 57. QA Peritor GmbH Blücherstr. 22, Hof III Aufgang 6 10961 Berlin Tel.: +49 (0)30 69 20 09 84 0 Fax: +49 (0)30 69 20 09 84 9 Internet: www.peritor.com E-Mail: [email protected] © Peritor GmbH - Alle Rechte vorbehalten
- 58. Idempotent Bug or feature? 58
- 59. Will never happen, even if /data/logs/redhat will be created! 59
- 60. Two Phases of Chef 1) Compile Load all attributesrecipes and build list of actions/dependencies Compute attribute tree 2) Run Random Ruby Code runs here Actually execute the resources 60
- 61. Two Phases of Chef Use only_if and not_if Tell Chef to execute your ressource during compile phase 61