![{
"name": "hadoop_cluster_a",
"description": "A small hadoop cluster with hbase",
"version": "1.0",
"environment": "production",
"servers": [
{
"name": "zookeeper-a, zookeeper-b, zookeeper-c",
"description": "Zookeeper nodes",
"template": "rhel-5.6-base",
"service": "small",
"port_rules": "2181",
"run_list": "role[cluster_a], role[zookeeper_server]",
"actions": [
{ "knife_ssh": ["role:zookeeper_server", "sudo chef-client"] }
]
},
{
"name": "hadoop-master",
"description": "Hadoop master node",
"template": "rhel-5.6-base",
"service": "large",
"networks": "app-net, storage-net",
"port_rules": "50070, 50030, 60010",
"run_list": "role[cluster_a], role[hadoop_master], role[hbase_master]"
},
{
"name": "hadoop-worker-a hadoop-worker-b hadoop-worker-c",
"description": "Hadoop worker nodes",
"template": "rhel-5.6-base",
"service": "medium",
"port_rules": "50075, 50060, 60030",
"run_list": "role[cluster_a], role[hadoop_worker], role[hbase_regionserver]",
"actions": [
{ "knife_ssh": ["role:hadoop_master", "sudo chef-client"] },
{ "http_request": "http://${hadoop-master}:50070/index.jsp" }
]
}
}](https://image.slidesharecdn.com/devtest-130530021240-phpapp01/85/Building-a-Dev-Test-Cloud-with-Apache-CloudStack-12-320.jpg)
Building a Dev/Test Cloud with Apache CloudStack
- 1. Building a Test/Dev Cloud with Apache CloudStack David Nalley [email protected] @ke4qqq
- 2. #whoami ● Sysadmin of a decade ● Apache Software Foundation Member ● Apache CloudStack Committer & PMC Member ● Fedora Project Contributor ● Employed by Citrix in the Open Source Business Office
- 3. Why use cloud? From a dev POV the process normally looks like: ● Start new project ● File ticket for resources....wait....wait....wait ● Get resources, that aren't configured....wait... ● Get network access.....get permission....wait ● Get things done.
- 4. Why use cloud? ● What IT Operations provides is not what a developer wants.
- 5. Get rid of the waiting ● Remove the constraints - developers empowered to get things done. ● Agility ● Enforce automated process instead of manual ones
- 6. What does a dev/test cloud look like? ● Self-service - developers can provision their own environments ● Usage measurement - we worry about VM sprawl ● Isolated networks - must not let dev/test interfere with the real world. ● Commodity - as cheap as practical ● May also house production workloads
- 7. Self service ● Provisioning manually doesn't add value ● Can be completely automated ● Do they need full control or just pushing pre-configured environments?
- 8. Self-service ● UI? ● API? ● Some external tool?
- 10. API or Commandline? � cloudmonkey> deploy virtualmachine serviceofferingid=d8611d07-acf5-4cd4-a630-5c4d937ef043 templateid=081358ff-2427-44f8-adcc-1bb002fab361 zoneid=d06193b2-7980-4ad1-bd8-7b2f2eda63c3 curl 'http://localhost:8096/client/api? command=listUsers'
- 12. { "name": "hadoop_cluster_a", "description": "A small hadoop cluster with hbase", "version": "1.0", "environment": "production", "servers": [ { "name": "zookeeper-a, zookeeper-b, zookeeper-c", "description": "Zookeeper nodes", "template": "rhel-5.6-base", "service": "small", "port_rules": "2181", "run_list": "role[cluster_a], role[zookeeper_server]", "actions": [ { "knife_ssh": ["role:zookeeper_server", "sudo chef-client"] } ] }, { "name": "hadoop-master", "description": "Hadoop master node", "template": "rhel-5.6-base", "service": "large", "networks": "app-net, storage-net", "port_rules": "50070, 50030, 60010", "run_list": "role[cluster_a], role[hadoop_master], role[hbase_master]" }, { "name": "hadoop-worker-a hadoop-worker-b hadoop-worker-c", "description": "Hadoop worker nodes", "template": "rhel-5.6-base", "service": "medium", "port_rules": "50075, 50060, 60030", "run_list": "role[cluster_a], role[hadoop_worker], role[hbase_regionserver]", "actions": [ { "knife_ssh": ["role:hadoop_master", "sudo chef-client"] }, { "http_request": "http://${hadoop-master}:50070/index.jsp" } ] } }
- 13. Use a tool
- 14. Usage ● Jevons Paradox ● Plenty of waste possible as well - will developers always destroy a machine when they are done with it? ● Important to show what projects and groups are consuming resources as well as how they are using those resources
- 15. Commodity - Storage ● Commodity storage - this is dev/test environment - high performance, resilient storage isn't needed. ● Local storage tends to be the best mix of cheap and performant ● No failover, but it's dev/test - do you need it?
- 16. Commodity - Networking ● Layer 3 isolation - (aka Security Groups) ● VLANs - (not as commodity, but still relatively cheap on a small scale, but not at scale) ● Virtual routers (provide DHCP, DNS, LB, Firewall, PF, NAT, etc)
- 17. Commodity Hypervisor ● KVM is my personal choice in this space. ● Easiest to consume - completely open source
- 18. Limiting resources ● Limit the number of VMs, snapshots, IP addresses, etc. ● Use 'projects' to share resources ● This means most folks will never have problems, but heaviest users will not be able to interrupt service for others.
- 19. Questions?
- 20. Resources ● http://cloudstack.apache.org ● http://cloudstack.jp ● #cloudstack on irc.freenode.net ● Mailing lists/google group
Editor's Notes
- #5: Dev wants to work - IT has process, and tickets, etc.
- #7: Talk a lot about the general architecture of CloudStack
- #9: What does self service really mean?
- #10: IS this really your idea of enabling developers? It might be nice to have, but it shouldn't be the real interface - even with defaults this takes 6 clicks.....which is better than filing a ticket, but doesn't scale.
- #19: Elasticity is only imagined - look at default limits placed on AWS users.