![{
"name": "hadoop_cluster_a",
"description": "A small hadoop cluster with hbase",
"version": "1.0",
"environment": "production",
"servers": [
{
"name": "zookeeper-a, zookeeper-b, zookeeper-c",
"description": "Zookeeper nodes",
"template": "rhel-5.6-base",
"service": "small",
"port_rules": "2181",
"run_list": "role[cluster_a], role[zookeeper_server]",
"actions": [
{ "knife_ssh": ["role:zookeeper_server", "sudo chef-client"] }
]
},
{
"name": "hadoop-master",
"description": "Hadoop master node",
"template": "rhel-5.6-base",
"service": "large",
"networks": "app-net, storage-net",
"port_rules": "50070, 50030, 60010",
"run_list": "role[cluster_a], role[hadoop_master], role[hbase_master]"
},
{
"name": "hadoop-worker-a hadoop-worker-b hadoop-worker-c",
"description": "Hadoop worker nodes",
"template": "rhel-5.6-base",
"service": "medium",
"port_rules": "50075, 50060, 60030",
"run_list": "role[cluster_a], role[hadoop_worker], role[hbase_regionserver]",
"actions": [
{ "knife_ssh": ["role:hadoop_master", "sudo chef-client"] },
{ "http_request": "http://${hadoop-master}:50070/index.jsp" }
]
}
}](https://image.slidesharecdn.com/devtest3-130607131447-phpapp01/85/Building-a-Dev-Test-Cloud-with-Apache-CloudStack-12-320.jpg)
Building a Dev/Test Cloud with Apache CloudStack
- 1. Building a Dev/Test cloud with Apache CloudStack David Nalley PMC Member Apache CloudStack Member, Apache Software Foundation [email protected] Twitter: @ke4qqq
- 2. #whoami • Apache Software Foundation Member • Apache CloudStack PMC Member • Recovering Sysadmin • Fedora Project Contributor • Zenoss contributor • Employed by Citrix in the Open Source Business Office
- 3. Why use cloud? From a dev point of view the process looks like: • Start new project • File ticket for resources....wait....wait....wait • Get resources, that aren't configured....wait... • Get network access.....get permission....wait • Get things done.
- 4. Why use cloud? • What IT Ops provides is not what developers want.
- 5. Get rid of the waiting! ● Remove constraints - developers empowered to get things done. ● Agility ● Enforce automated process instead of manual ones
- 6. What does a dev/test cloud look like? ● Self-service - developers can provision their own environments ● Usage measurement - we worry about VM sprawl ● Isolated networks - must not let dev/test interfere with the real world. ● Commodity - as cheap as practical ● May also house production workloads
- 7. Self service ● Provisioning manually doesn't add value ● Can be completely automated ● Do they need full control or just pushing pre-configured environments?
- 10. API or Command-line � cloudmonkey> deploy virtualmachine serviceofferingid=d8611d07-acf5-4cd4-a630-5c4d937ef043 templateid=081358ff-2427-44f8-adcc-1bb002fab361 zoneid=d06193b2-7980-4ad1-bd8-7b2f2eda63c3 curl 'http://localhost:8096/client/api? command=listUsers'
- 12. { "name": "hadoop_cluster_a", "description": "A small hadoop cluster with hbase", "version": "1.0", "environment": "production", "servers": [ { "name": "zookeeper-a, zookeeper-b, zookeeper-c", "description": "Zookeeper nodes", "template": "rhel-5.6-base", "service": "small", "port_rules": "2181", "run_list": "role[cluster_a], role[zookeeper_server]", "actions": [ { "knife_ssh": ["role:zookeeper_server", "sudo chef-client"] } ] }, { "name": "hadoop-master", "description": "Hadoop master node", "template": "rhel-5.6-base", "service": "large", "networks": "app-net, storage-net", "port_rules": "50070, 50030, 60010", "run_list": "role[cluster_a], role[hadoop_master], role[hbase_master]" }, { "name": "hadoop-worker-a hadoop-worker-b hadoop-worker-c", "description": "Hadoop worker nodes", "template": "rhel-5.6-base", "service": "medium", "port_rules": "50075, 50060, 60030", "run_list": "role[cluster_a], role[hadoop_worker], role[hbase_regionserver]", "actions": [ { "knife_ssh": ["role:hadoop_master", "sudo chef-client"] }, { "http_request": "http://${hadoop-master}:50070/index.jsp" } ] } }
- 13. Use a tool
- 14. Usage ● Jevons Paradox ● Plenty of waste possible as well - will developers always destroy a machine when they are done with it? ● Important to show what projects and groups are consuming resources as well as how they are using those resources
- 15. Commodity Storage ● Commodity storage - this is dev/test environment - high performance, resilient storage isn't needed. ● Local storage tends to be the best mix of cheap and performant ● No failover, but it's dev/test - do you need it?
- 16. Commodity Networking ● Layer 3 isolation - (aka Security Groups) ● VLANs - (not as commodity, but still relatively cheap on a small scale, but not at a large scale) ● Virtual routers (provide DHCP, DNS, LB, Firewall, PF, NAT, etc)
- 17. Commodity Hypervisor ● KVM is my personal choice in this space. ● Easiest to consume - completely open source
- 18. Limiting Resources ● Limit the number of VMs, snapshots, IP addresses, etc. ● Use 'projects' to share resources ● This means most folks will never have problems, but heaviest users will not be able to interrupt service for others.
- 19. Questions