Scalable Object Storage with Apache CloudStack and Apache Hadoop
Download as pptx, pdf3 likes1,678 views
The document discusses the integration of scalable object storage with Apache CloudStack and Hadoop, focusing on its architecture, challenges, and potential solutions. It highlights the need for reliable, durable storage systems that can efficiently manage large volumes of data while considering performance and scalability. The future directions emphasize leveraging open-source technologies to enhance object storage capabilities in cloud environments.
1 of 32
Downloaded 39 times
![Object Storage characteristics
โข Highly reliable and durable
โ 99.9 % availability for AWS S3
โ 99.999999999 % durability
โข Massive scale
โ 1.3 trillion objects stored across 7 AWS regions [Nov 2012 figures]
โ Throughput: 830,000 requests per second
โข Immutable objects
โ Objects cannot be modified, only deleted
โข Simple API
โ PUT/POST objects, GET objects, DELETE objects
โ No seek / no mutation / no POSIX API
โข Flat namespace
โ Everything stored in buckets.
โ Bucket names are unique
โ Buckets can only contain objects, not other buckets
โข Cheap and getting cheaper](https://image.slidesharecdn.com/2-s3hdfsapachecon-130502175136-phpapp01/85/Scalable-Object-Storage-with-Apache-CloudStack-and-Apache-Hadoop-15-320.jpg)
![Why HDFS
โข ASF Project (Apache Hadoop)
โข Immutable objects, replication
โข Reliability, scale and performance
โ 200 million objects in 1 cluster [Facebook]
โ 100 PB in 1 cluster [Facebook]
โข Simple operation
โ Just add data nodes](https://image.slidesharecdn.com/2-s3hdfsapachecon-130502175136-phpapp01/85/Scalable-Object-Storage-with-Apache-CloudStack-and-Apache-Hadoop-22-320.jpg)
Ad
Recommended
Netflix running Presto in the AWS Cloud
Netflix running Presto in the AWS CloudZhenxiao Luo
ย
Netflix runs Presto in its AWS cloud environment to enable low-latency ad-hoc queries on petabyte-scale data stored in S3. Some key things Netflix did include optimizing Presto to read from and write directly to S3, fixing bugs, integrating Presto with its EMR and Ganglia monitoring, and deploying a 100+ node Presto cluster that handles over 1000 queries per day. Performance testing showed Presto was often 10x faster than Hive for various queries and joins. Netflix continues optimizing Presto for its needs like supporting Parquet, ODBC/JDBC drivers, and looking to address current limitations.Scaling Traffic from 0 to 139 Million Unique Visitors
Scaling Traffic from 0 to 139 Million Unique VisitorsYelp Engineering
ย
This document summarizes the traffic history and infrastructure changes at Yelp from 2005 to the present. It outlines the key milestones and technology changes over time as Yelp grew from handling around 200k searches per day with 1 database in 2005-2007 to serving traffic across 29 countries in 2014 with a distributed, scalable infrastructure utilizing technologies like Elasticsearch, Kafka, and Pyleus for real-time processing.Building a unified data pipeline in Apache Spark
Building a unified data pipeline in Apache SparkDataWorks Summit
ย
This document discusses Apache Spark, an open-source distributed data processing framework. It describes how Spark provides a unified platform for batch processing, streaming, SQL queries, machine learning and graph processing. The document demonstrates how in Spark these capabilities can be combined in a single application, without needing to move data between systems. It shows an example pipeline that performs SQL queries, machine learning clustering and streaming processing on Twitter data.MySQL in the Hosted Cloud
MySQL in the Hosted CloudColin Charles
ย
The document discusses MySQL as a Database as a Service (DBaaS) solution, detailing deployment options, service providers, costs, and features. It compares various cloud platforms such as Amazon RDS, Rackspace, Google Cloud SQL, and HP Cloud, emphasizing the ease of use and operational benefits of DBaaS. The presentation also covers topics like MySQL versions, access methods, support, monitoring, backup, and high availability strategies.Data warehouse solutions
Data warehouse solutionsTu Pham
ย
This document discusses data warehouse solutions using Amazon Web Services (AWS). It outlines the key components including data collection via a client tracker and log collector, storage options like S3, Glacier and Redshift, data ETL pipelines, and analytics using EMR and Kinesis. The document also compares pros and cons of the different AWS services and notes some challenges around using AWS for data warehousing across different regions.Amazon RedShift - Ianni Vamvadelis
Amazon RedShift - Ianni Vamvadelishuguk
ย
The document provides an overview of Amazon Redshift, a fully managed, petabyte-scale data warehouse service that integrates with other AWS services. It details features such as optimized hardware, security measures, data loading best practices, and workload management. Additionally, it discusses Redshift's pricing and use cases, emphasizing its efficiency for scalable data analysis and reporting.Qubole Overview at the Fifth Elephant Conference
Qubole Overview at the Fifth Elephant ConferenceJoydeep Sen Sarma
ย
The document discusses some of the challenges of managing Hadoop clusters in the cloud, including setting up infrastructure components like the Hive metastore and determining optimal cluster sizing. It then presents some solutions offered by Qubole's data platform, like auto-scaling clusters and running periodic jobs. The document also covers techniques for improving query performance, such as using HDFS as a cache layer and storing data in columnar format for faster access compared to JSON or CSV files stored in S3.HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...
HBaseConAsia2018 Track2-1: Kerberos-based Big Data Security Solution and Prac...Michael Stack
ย
This document discusses security practices for HBase clusters. It introduces the Hadoop Authentication Service (HAS) as a way to integrate enterprise identity management with Kerberos authentication in Hadoop. HAS supports plugins to authenticate with various systems. Alibaba uses HAS for authentication in ApsaraDB for HBase, along with optimizations like high availability for the backend storage and whitelists. The document outlines how HAS works and the benefits it provides for securing HBase deployments.Scaling Pinterest
Scaling PinterestC4Media
ย
The document outlines the evolution and scaling strategies employed by Pinterest from its inception in 2010 through various technology deployments and growth phases. It details the technical infrastructure choices, including the use of Amazon Web Services, Python, and MySQL, as well as challenges experienced during scalability efforts. The presentation emphasizes lessons learned, future goals, and seeks to refine operational methodologies for improved performance and user experience.HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...
HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...Michael Stack
ย
This document discusses AntsDB, an open source project that brings MySQL compatibility to HBase in order to address the need for relational database capabilities in NoSQL systems. It describes AntsDB's architecture, which uses caching and other techniques to provide low-latency transactions and joins on HBase. Performance tests show AntsDB can achieve high throughput for writes and OLTP workloads. AntsDB aims to be complementary to HBase by virtualizing MySQL atop HBase while simulating MySQL behaviors and allowing applications built for MySQL to run unchanged on HBase.Qubole @ AWS Meetup Bangalore - July 2015
Qubole @ AWS Meetup Bangalore - July 2015Joydeep Sen Sarma
ย
Qubole is a big data as a service platform that allows users to run analytics jobs on AWS infrastructure. It integrates tightly with various AWS services like EC2, S3, Redshift, and Kinesis. Qubole handles cluster provisioning and management, provides tools for interactive querying using Presto, and allows customers to access data across different AWS data platforms through a single interface. Some key benefits of Qubole include simplified management of AWS resources, optimized performance through techniques like auto-scaling and caching, and unified analytics platform for tools like Hive, Spark and Presto.Amazon Athena Hands-On Workshop
Amazon Athena Hands-On WorkshopDoiT International
ย
The Amazon Athena workshop held on January 26, 2017, covered various modules including an introduction to AWS Athena, practical labs on interacting with Athena, and understanding supported data formats like CSV, JSON, and Parquet. Athena is an interactive query service that allows users to analyze data stored in AWS S3 using standard SQL without the need for data loading or ETL processes. Key features discussed included its serverless architecture, security measures, and comparisons with services like Google BigQuery.HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...
HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...Michael Stack
ย
This document discusses the use of HBase in a vehicle monitoring system. It describes challenges including handling huge amounts of vehicle data from 100k vehicles generating 2TB of data daily. It outlines decisions around using Java, Kafka, HBase, and microservices. The system architecture is shown storing vehicle data in HBase with data backup. Challenges with HBase like query speed are discussed. Prospects include rewriting components in Go, splitting to microservices, and data analysis.An overview of Amazon Athena
An overview of Amazon AthenaJulien SIMON
ย
Amazon Athena is a serverless query service that allows users to run interactive SQL queries on data stored in Amazon S3 without having to load the data into a database. It uses Presto to allow ANSI SQL queries on data in formats like CSV, JSON, and columnar formats like Parquet and ORC. For a dataset of 1 billion rows of sales data stored in S3, Athena queries performed comparably to a basic Redshift cluster and much faster than loading and querying the data in Redshift, making Athena a cost-effective solution for ad-hoc queries on data in S3.Cloud Optimized Big Data
Cloud Optimized Big DataJoydeep Sen Sarma
ย
The document discusses cloud-optimized big data as a service, detailing the advantages of lower total cost of ownership (TCO), operational efficiency, and enhanced speed offered by Qubole. It highlights features like automated lifecycle management, collaboration through zero-copy data sharing, and rapidly scalable infrastructure utilizing both spot and regular instances. The service provides significant performance improvements, being faster than competitors while enabling seamless upgrade processes and operational support.Getting Started with EC2, S3 and EMR
Getting Started with EC2, S3 and EMRArun Sirimalla
ย
This document provides an overview of Amazon Web Services including EC2, S3, and EMR. It discusses regions and availability zones in EC2, how to set up VPCs, different EC2 instance types, AMIs, key pairs, and the differences between EBS and instance store. It also covers S3 concepts like buckets, objects, storage classes, and access controls. Finally, it briefly introduces EMR and how it provides a managed Hadoop framework on EC2 instances with integration to S3 for storage. The document includes demos of working with EC2 instances and EBS volumes, S3 buckets, and creating an EMR cluster.Oracle Databases on AWS - Getting the Best Out of RDS and EC2
Oracle Databases on AWS - Getting the Best Out of RDS and EC2Maris Elsins
ย
The document discusses the use of Oracle databases on AWS, detailing the distinctions between Amazon RDS and EC2, including management responsibilities, storage options, and performance considerations. It highlights the benefits and limitations of each service, emphasizing the need for careful planning and understanding of managed environments. Additionally, it covers aspects like database licensing, security features, and strategies for monitoring and managing resources effectively in the AWS cloud.Hadoop @ eBay: Past, Present, and Future
Hadoop @ eBay: Past, Present, and FutureRyan Hennig
ย
The document discusses the evolution and current use of Hadoop at eBay, detailing its growth, partnerships, and operational tools. It covers the transition from shared clusters to more specialized dedicated clusters along with various use cases such as search engine indexing and analytics dashboards. The future of Hadoop at eBay emphasizes advancements like HDFS federation, yarn for resource management, and a focus on operational efficiency and open source contributions.Autoscaling Spark on AWS EC2 - 11th Spark London meetup
Autoscaling Spark on AWS EC2 - 11th Spark London meetupRafal Kwasny
ย
This document discusses autoscaling Spark clusters on AWS for efficiency and cost-effectiveness. It presents a typical AWS architecture with Spark running on EC2 and data stored in S3. It describes how autoscaling works to dynamically adjust the number of EC2 instances based on demand metrics to match resource usage. The spark-cloud tool is introduced to simplify managing Spark clusters on AWS with features like building AMIs, starting and shutting down clusters, and using spot instances for lower costs compared to on-demand pricing. Autoscaling helps remove the need to manually scale clusters up and down.CosmosDB for DBAs & Developers
CosmosDB for DBAs & DevelopersNiko Neugebauer
ย
This document provides an overview and introduction to Cosmos DB. It discusses what Cosmos DB is, its data models, APIs, partitioning, and global distribution. It explains why Cosmos DB was created to address limitations of traditional databases. Key aspects covered include throughput and consistency levels, indexing, backups, failovers, and using Cosmos DB for developers and database administrators. The document also discusses migration tools, limitations, and integrations with PowerBI and geospatial data.Presto Fast SQL on Anything
Presto Fast SQL on AnythingAlluxio, Inc.
ย
This document discusses Presto, an open source distributed SQL query engine. It is used by many large companies like Facebook, Uber, and Netflix for querying large datasets across various data sources. Presto provides high performance through its columnar processing, runtime compilation, and new cost-based optimizer. The document also describes how Presto can be run on AWS and Azure cloud platforms through partnerships with Starburst, who contributed many features to Presto and provides commercial support for enterprises.CloudStack Object Storage Framework & Demo
CloudStack Object Storage Framework & DemoShapeBlue
ย
The document provides an overview of Kishan Kavala's experience and expertise in cloud storage, focusing on the CloudStack Object Storage Framework. It explains the difference between primary and secondary storage, details the characteristics and management of object storage, and lists various providers and open-source object stores. The document also mentions features like metadata management and future development of a user interface for managing object storage.OpenStack Swift In the Enterprise
OpenStack Swift In the EnterpriseHostway|HOSTING
ย
This document discusses object storage, specifically OpenStack Swift, within enterprise environments, highlighting its advantages over traditional storage types due to data growth and the need for scalability. It explains the architecture of Swift and its components, including how it handles metadata and object retrieval through HTTP APIs. Additionally, it outlines use cases, performance considerations, and the benefits of combining OpenStack Swift with private, public, or hybrid cloud infrastructure.Storage as a service OpenStack
Storage as a service OpenStackopenstackindia
ย
This document discusses storage as a service and how OpenStack provides storage options. It describes Storage as a Service and the types of storage available in OpenStack, including ephemeral, object, block, and file storage. It provides overviews of the OpenStack Object Storage (Swift) and Block Storage (Cinder) projects and how they work. The document also discusses filesystem storage in OpenStack using protocols like NFS and CIFS.Storing your data in the cloud: doing right reversim 2018
Storing your data in the cloud: doing right reversim 2018Orit Wasserman
ย
This document provides an overview of cloud object storage, including its key features like multipart uploads, tagging, versioning, lifecycles, prefixes, static websites, security measures, and geo-replication for disaster recovery. It discusses when object storage is suitable compared to block and file storage, how to properly use features like multipart uploads, encryption, and access control, and cautions about ensuring data security when using public cloud storage services.Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxJohn Burwell
ย
The document discusses the integration of Apache CloudStack and Riak CS for building a reliable and cost-effective private cloud infrastructure. It highlights the benefits of using commodity hardware, improved resource pooling, and dynamic scalability, while also detailing the architecture and components of both CloudStack and Riak CS. The presentation emphasizes agility, self-service provisioning, and customization to meet specific regulatory and operational needs.Apache CloudStack from API to UI
Apache CloudStack from API to UICloudStack - Open Source Cloud Computing Project
ย
CloudStack is an open source cloud computing platform that provides infrastructure as a service. It supports various hypervisors (KVM, Xen, VMware), has APIs for self-service provisioning, measures resource usage, and allows for rapid elasticity. CloudStack can be deployed as public, private or hybrid clouds and manages networks, storage, security and high availability of virtual machines.Hacking apache cloud stack
Hacking apache cloud stackNitin Mehta
ย
The document provides an agenda and overview of a session on hacking Apache CloudStack. The agenda includes introductions, a session on introducing CloudStack, and a hands-on session with DevCloud. The overview discusses what CloudStack is, how it works as an orchestration platform for IAAS clouds, its architecture and core components, and how users can consume and manage resources through it. Cloud storage: the right way OSS EU 2018
Cloud storage: the right way OSS EU 2018Orit Wasserman
ย
1. The document discusses cloud object storage, describing its features like multipart uploads, versioning, and lifecycles. It provides examples of using object storage for media and documents.
2. Key aspects of object storage security are covered, including signatures, encryption, access control lists, and policies. Disaster recovery options like geo-replication are also summarized.
3. In the conclusion, the document emphasizes using object storage APIs to access advanced features, ensuring data safety, testing disaster recovery plans, and using Ceph for private cloud object storage.Initial presentation of swift (for montreal user group)
Initial presentation of swift (for montreal user group)Marcos Garcรญa
ย
Swift is an open source object storage system that provides scalable storage and retrieval of any amount of unstructured data over HTTP. It is designed to be scalable, reliable, and inexpensive for storing large amounts of unstructured data. Some key uses of Swift include storing backups, web content like images, and large scientific data objects. Swift uses a ring architecture to distribute and replicate data across multiple servers for high availability.More Related Content
What's hot (13)
Scaling Pinterest
Scaling PinterestC4Media
ย
The document outlines the evolution and scaling strategies employed by Pinterest from its inception in 2010 through various technology deployments and growth phases. It details the technical infrastructure choices, including the use of Amazon Web Services, Python, and MySQL, as well as challenges experienced during scalability efforts. The presentation emphasizes lessons learned, future goals, and seeks to refine operational methodologies for improved performance and user experience.HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...
HBaseConAsia2018 Track2-3: Bringing MySQL Compatibility to HBase using Databa...Michael Stack
ย
This document discusses AntsDB, an open source project that brings MySQL compatibility to HBase in order to address the need for relational database capabilities in NoSQL systems. It describes AntsDB's architecture, which uses caching and other techniques to provide low-latency transactions and joins on HBase. Performance tests show AntsDB can achieve high throughput for writes and OLTP workloads. AntsDB aims to be complementary to HBase by virtualizing MySQL atop HBase while simulating MySQL behaviors and allowing applications built for MySQL to run unchanged on HBase.Qubole @ AWS Meetup Bangalore - July 2015
Qubole @ AWS Meetup Bangalore - July 2015Joydeep Sen Sarma
ย
Qubole is a big data as a service platform that allows users to run analytics jobs on AWS infrastructure. It integrates tightly with various AWS services like EC2, S3, Redshift, and Kinesis. Qubole handles cluster provisioning and management, provides tools for interactive querying using Presto, and allows customers to access data across different AWS data platforms through a single interface. Some key benefits of Qubole include simplified management of AWS resources, optimized performance through techniques like auto-scaling and caching, and unified analytics platform for tools like Hive, Spark and Presto.Amazon Athena Hands-On Workshop
Amazon Athena Hands-On WorkshopDoiT International
ย
The Amazon Athena workshop held on January 26, 2017, covered various modules including an introduction to AWS Athena, practical labs on interacting with Athena, and understanding supported data formats like CSV, JSON, and Parquet. Athena is an interactive query service that allows users to analyze data stored in AWS S3 using standard SQL without the need for data loading or ETL processes. Key features discussed included its serverless architecture, security measures, and comparisons with services like Google BigQuery.HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...
HBaseConAsia2018 Track3-7: The application of HBase in New Energy Vehicle Mon...Michael Stack
ย
This document discusses the use of HBase in a vehicle monitoring system. It describes challenges including handling huge amounts of vehicle data from 100k vehicles generating 2TB of data daily. It outlines decisions around using Java, Kafka, HBase, and microservices. The system architecture is shown storing vehicle data in HBase with data backup. Challenges with HBase like query speed are discussed. Prospects include rewriting components in Go, splitting to microservices, and data analysis.An overview of Amazon Athena
An overview of Amazon AthenaJulien SIMON
ย
Amazon Athena is a serverless query service that allows users to run interactive SQL queries on data stored in Amazon S3 without having to load the data into a database. It uses Presto to allow ANSI SQL queries on data in formats like CSV, JSON, and columnar formats like Parquet and ORC. For a dataset of 1 billion rows of sales data stored in S3, Athena queries performed comparably to a basic Redshift cluster and much faster than loading and querying the data in Redshift, making Athena a cost-effective solution for ad-hoc queries on data in S3.Cloud Optimized Big Data
Cloud Optimized Big DataJoydeep Sen Sarma
ย
The document discusses cloud-optimized big data as a service, detailing the advantages of lower total cost of ownership (TCO), operational efficiency, and enhanced speed offered by Qubole. It highlights features like automated lifecycle management, collaboration through zero-copy data sharing, and rapidly scalable infrastructure utilizing both spot and regular instances. The service provides significant performance improvements, being faster than competitors while enabling seamless upgrade processes and operational support.Getting Started with EC2, S3 and EMR
Getting Started with EC2, S3 and EMRArun Sirimalla
ย
This document provides an overview of Amazon Web Services including EC2, S3, and EMR. It discusses regions and availability zones in EC2, how to set up VPCs, different EC2 instance types, AMIs, key pairs, and the differences between EBS and instance store. It also covers S3 concepts like buckets, objects, storage classes, and access controls. Finally, it briefly introduces EMR and how it provides a managed Hadoop framework on EC2 instances with integration to S3 for storage. The document includes demos of working with EC2 instances and EBS volumes, S3 buckets, and creating an EMR cluster.Oracle Databases on AWS - Getting the Best Out of RDS and EC2
Oracle Databases on AWS - Getting the Best Out of RDS and EC2Maris Elsins
ย
The document discusses the use of Oracle databases on AWS, detailing the distinctions between Amazon RDS and EC2, including management responsibilities, storage options, and performance considerations. It highlights the benefits and limitations of each service, emphasizing the need for careful planning and understanding of managed environments. Additionally, it covers aspects like database licensing, security features, and strategies for monitoring and managing resources effectively in the AWS cloud.Hadoop @ eBay: Past, Present, and Future
Hadoop @ eBay: Past, Present, and FutureRyan Hennig
ย
The document discusses the evolution and current use of Hadoop at eBay, detailing its growth, partnerships, and operational tools. It covers the transition from shared clusters to more specialized dedicated clusters along with various use cases such as search engine indexing and analytics dashboards. The future of Hadoop at eBay emphasizes advancements like HDFS federation, yarn for resource management, and a focus on operational efficiency and open source contributions.Autoscaling Spark on AWS EC2 - 11th Spark London meetup
Autoscaling Spark on AWS EC2 - 11th Spark London meetupRafal Kwasny
ย
This document discusses autoscaling Spark clusters on AWS for efficiency and cost-effectiveness. It presents a typical AWS architecture with Spark running on EC2 and data stored in S3. It describes how autoscaling works to dynamically adjust the number of EC2 instances based on demand metrics to match resource usage. The spark-cloud tool is introduced to simplify managing Spark clusters on AWS with features like building AMIs, starting and shutting down clusters, and using spot instances for lower costs compared to on-demand pricing. Autoscaling helps remove the need to manually scale clusters up and down.CosmosDB for DBAs & Developers
CosmosDB for DBAs & DevelopersNiko Neugebauer
ย
This document provides an overview and introduction to Cosmos DB. It discusses what Cosmos DB is, its data models, APIs, partitioning, and global distribution. It explains why Cosmos DB was created to address limitations of traditional databases. Key aspects covered include throughput and consistency levels, indexing, backups, failovers, and using Cosmos DB for developers and database administrators. The document also discusses migration tools, limitations, and integrations with PowerBI and geospatial data.Presto Fast SQL on Anything
Presto Fast SQL on AnythingAlluxio, Inc.
ย
This document discusses Presto, an open source distributed SQL query engine. It is used by many large companies like Facebook, Uber, and Netflix for querying large datasets across various data sources. Presto provides high performance through its columnar processing, runtime compilation, and new cost-based optimizer. The document also describes how Presto can be run on AWS and Azure cloud platforms through partnerships with Starburst, who contributed many features to Presto and provides commercial support for enterprises.Similar to Scalable Object Storage with Apache CloudStack and Apache Hadoop (20)
CloudStack Object Storage Framework & Demo
CloudStack Object Storage Framework & DemoShapeBlue
ย
The document provides an overview of Kishan Kavala's experience and expertise in cloud storage, focusing on the CloudStack Object Storage Framework. It explains the difference between primary and secondary storage, details the characteristics and management of object storage, and lists various providers and open-source object stores. The document also mentions features like metadata management and future development of a user interface for managing object storage.OpenStack Swift In the Enterprise
OpenStack Swift In the EnterpriseHostway|HOSTING
ย
This document discusses object storage, specifically OpenStack Swift, within enterprise environments, highlighting its advantages over traditional storage types due to data growth and the need for scalability. It explains the architecture of Swift and its components, including how it handles metadata and object retrieval through HTTP APIs. Additionally, it outlines use cases, performance considerations, and the benefits of combining OpenStack Swift with private, public, or hybrid cloud infrastructure.Storage as a service OpenStack
Storage as a service OpenStackopenstackindia
ย
This document discusses storage as a service and how OpenStack provides storage options. It describes Storage as a Service and the types of storage available in OpenStack, including ephemeral, object, block, and file storage. It provides overviews of the OpenStack Object Storage (Swift) and Block Storage (Cinder) projects and how they work. The document also discusses filesystem storage in OpenStack using protocols like NFS and CIFS.Storing your data in the cloud: doing right reversim 2018
Storing your data in the cloud: doing right reversim 2018Orit Wasserman
ย
This document provides an overview of cloud object storage, including its key features like multipart uploads, tagging, versioning, lifecycles, prefixes, static websites, security measures, and geo-replication for disaster recovery. It discusses when object storage is suitable compared to block and file storage, how to properly use features like multipart uploads, encryption, and access control, and cautions about ensuring data security when using public cloud storage services.Better, faster, cheaper infrastructure with apache cloud stack and riak cs redux
Better, faster, cheaper infrastructure with apache cloud stack and riak cs reduxJohn Burwell
ย
The document discusses the integration of Apache CloudStack and Riak CS for building a reliable and cost-effective private cloud infrastructure. It highlights the benefits of using commodity hardware, improved resource pooling, and dynamic scalability, while also detailing the architecture and components of both CloudStack and Riak CS. The presentation emphasizes agility, self-service provisioning, and customization to meet specific regulatory and operational needs.Apache CloudStack from API to UI
Apache CloudStack from API to UICloudStack - Open Source Cloud Computing Project
ย
CloudStack is an open source cloud computing platform that provides infrastructure as a service. It supports various hypervisors (KVM, Xen, VMware), has APIs for self-service provisioning, measures resource usage, and allows for rapid elasticity. CloudStack can be deployed as public, private or hybrid clouds and manages networks, storage, security and high availability of virtual machines.Hacking apache cloud stack
Hacking apache cloud stackNitin Mehta
ย
The document provides an agenda and overview of a session on hacking Apache CloudStack. The agenda includes introductions, a session on introducing CloudStack, and a hands-on session with DevCloud. The overview discusses what CloudStack is, how it works as an orchestration platform for IAAS clouds, its architecture and core components, and how users can consume and manage resources through it. Cloud storage: the right way OSS EU 2018
Cloud storage: the right way OSS EU 2018Orit Wasserman
ย
1. The document discusses cloud object storage, describing its features like multipart uploads, versioning, and lifecycles. It provides examples of using object storage for media and documents.
2. Key aspects of object storage security are covered, including signatures, encryption, access control lists, and policies. Disaster recovery options like geo-replication are also summarized.
3. In the conclusion, the document emphasizes using object storage APIs to access advanced features, ensuring data safety, testing disaster recovery plans, and using Ceph for private cloud object storage.Initial presentation of swift (for montreal user group)
Initial presentation of swift (for montreal user group)Marcos Garcรญa
ย
Swift is an open source object storage system that provides scalable storage and retrieval of any amount of unstructured data over HTTP. It is designed to be scalable, reliable, and inexpensive for storing large amounts of unstructured data. Some key uses of Swift include storing backups, web content like images, and large scientific data objects. Swift uses a ring architecture to distribute and replicate data across multiple servers for high availability.Deploying Apache CloudStack from API to UI
Deploying Apache CloudStack from API to UIJoe Brockmeier
ย
This document provides an overview of Apache CloudStack, an open-source Infrastructure as a Service (IaaS) platform that supports various hypervisors, high availability, and resource management in a multi-tenant environment. It covers CloudStack's features, deployment models, architecture, and networking capabilities, as well as its history and the evolution of its API and UI. The talk is designed to inform attendees about CloudStack's functionalities and practical applications in cloud computing.Apache CloudStack: API to UI (STLLUG)
Apache CloudStack: API to UI (STLLUG)Joe Brockmeier
ย
Apache CloudStack is an open-source infrastructure-as-a-service (IaaS) platform designed for building and managing public, private, and hybrid clouds. It supports various hypervisors, integrates APIs for management, and provides features such as flexible networking, load balancing, and snapshot capabilities. The platform emphasizes community-driven governance, with continuous enhancements through regular releases.Architecting An Enterprise Storage Platform Using Object Stores
Architecting An Enterprise Storage Platform Using Object StoresNiraj Tolia
ย
This document discusses architecting an enterprise storage platform using object stores. It summarizes MagFS, a file system designed for the cloud that is layered on top of object storage. Key points include:
- MagFS provides a consistent, elastic, secure and mobile-enabled file system experience while leveraging low-cost object storage.
- The client architecture pushes intelligence to edges for heavy lifting like encryption, deduplication, and caching while coordinating with metadata servers.
- Metadata servers enforce strong consistency, authentication, and garbage collection while optimizing performance through virtualization and lease-based caching.
- Security is ensured through server-driven request signing and scrubbing writes to object storage after client acknowledgmentOpenStack 101: Introduction & Technical Overview
OpenStack 101: Introduction & Technical OverviewThang Man
ย
This document provides an introduction and technical overview of OpenStack presented by Mแบซn Thแบฏng. It discusses the motivation for OpenStack, what OpenStack is, key features of OpenStack Compute and Storage, use cases, and resources for learning more and getting involved in the OpenStack community. The presentation encourages participants to try OpenStack locally or on public clouds and provides tips on how to participate in the OpenStack community.Storage as a service and OpenStack Cinder
Storage as a service and OpenStack Cinderopenstackindia
ย
Storage as a Service provides scalable cloud storage through APIs that abstract the underlying implementation. OpenStack is an open source cloud platform that includes Cinder for block storage and Swift for object storage. Cinder provides persistent block storage volumes that can be attached to instances, while Swift stores scalable objects accessible through APIs.Txlf2012
Txlf2012Joe Brockmeier
ย
This document provides an overview of how to create your own cloud using Apache CloudStack. It discusses the key characteristics of clouds, different cloud service and deployment models supported by CloudStack, and the core components that make up a CloudStack deployment including zones, pods, clusters, primary and secondary storage, virtual routers, hypervisors, and the management server. The document also touches on CloudStack's networking, security, high availability, resource allocation, and usage accounting features.Power of OpenStack & Hadoop
Power of OpenStack & HadoopTuan Yang
ย
The document outlines the capabilities and architecture of OpenStack, an open-source cloud operating system that orchestrates cloud services through various components such as Nova for computing, Glance for imaging, Keystone for identity services, and Swift for storage. It describes how these components interact through APIs and message queues to provide a comprehensive cloud infrastructure solution. Additionally, it discusses integrating OpenStack with high performance computing and big data technologies, emphasizing the importance of collaborative development and scalability.HDFS Tiered Storage: Mounting Object Stores in HDFS
HDFS Tiered Storage: Mounting Object Stores in HDFSDataWorks Summit/Hadoop Summit
ย
The document discusses HDFS tiered storage and the integration of object stores such as Amazon S3 within the HDFS ecosystem, emphasizing improved handling of external storage without explicit data copying. Key features include the ability to mount various Hadoop-compatible filesystems, manage multiple clusters effectively, and maintain a unified namespace with transparent caching. The presentation outlines existing contributions, challenges faced with metadata synchronization, and future developments in write paths and dynamic mounting strategies.Building a Resilient, Scalable, Storage System with OpenStack
Building a Resilient, Scalable, Storage System with OpenStackCloudian
ย
Cloudian's S3 cloud storage platform offers a resilient and scalable storage solution integrated with OpenStack components. It features enhanced HyperStore technology for optimizing data management and performance, including erasure coding and support for a wide range of applications. The platform provides full compatibility with over 350 S3 applications, streamlined management, and cost-effective cloud storage options.Data storage in clouds
Data storage in cloudsThanassis Parathyras
ย
This document details the 12th meetup of the Athens OpenStack User Group, discussing various OpenStack storage services like Glance, Cinder, and Swift, along with their functionalities and architectures. It also highlights additional projects like Ceph and GlusterFS, emphasizing the importance of scalable, software-defined storage systems. The document includes information on community involvement, upcoming events, and resources for contributing to OpenStack development.Ad
More from buildacloud (20)
The Future of SDN in CloudStack by Chiradeep Vittal
The Future of SDN in CloudStack by Chiradeep Vittalbuildacloud
ย
The document outlines the features and architecture of Apache CloudStack, emphasizing its capabilities in cloud orchestration and multi-tenancy. It discusses networking principles, SDN integration, and the evolution of container management within CloudStack, highlighting future directions such as IPv6 support and NFV. The content targets a technical audience interested in building and optimizing cloud infrastructure by leveraging CloudStack's features and network services.Policy Based SDN Solution for DC and Branch Office by Suresh Boddapati
Policy Based SDN Solution for DC and Branch Office by Suresh Boddapatibuildacloud
ย
Nuage Networks provides an SDN solution called the Virtualized Services Platform that abstracts physical network infrastructure and automates network provisioning through centralized policy-based controls. Nuage has integrated its VSP with CloudStack to enhance CloudStack's networking capabilities with advanced virtual networking, improved scalability, and automated provisioning based on Nuage's policy engine. The Nuage-CloudStack plugin maps CloudStack networking constructs to corresponding constructs in the VSP to provide services like isolated networks, firewalls, and load balancing across multiple hypervisors.L4-L7 services for SDN and NVF by Youcef Laribi
L4-L7 services for SDN and NVF by Youcef Laribibuildacloud
ย
The document discusses the development and deployment of L4-L7 services in SDN and NFV environments, outlining typical services and their roles within enterprise and telco networks. It highlights the distinctions between SDN and NFV, emphasizing the importance of rapid service innovation through virtualized network functions. Additionally, it covers various techniques for accelerating data paths and managing network services, including service chaining and integration with emerging standards like NSH.Jenkins, jclouds, CloudStack, and CentOS by David Nalley
Jenkins, jclouds, CloudStack, and CentOS by David Nalleybuildacloud
ย
The document discusses the integration of Jenkins, JClouds, CentOS, and CloudStack for managing diverse testing needs across nearly 200 software projects. It outlines the transition from a monolithic Jenkins setup to a scalable solution that dynamically provisions VMs for builds based on demand. Key aspects include efficient configuration, the use of Packer for image building, and the installation of necessary plugins to streamline the process.Intro to Zenoss by Andrew Kirch
Intro to Zenoss by Andrew Kirchbuildacloud
ย
Zenoss is a unified monitoring solution that provides real-time insights into services using standard protocols and APIs for various infrastructures, including cloud services. It features a community-driven development approach with 394 available Zenpacks, and offers resources for support through forums, a wiki, and IRC. The installation process is straightforward, requiring minimal resources and community support for implementation.Guaranteeing Storage Performance by Mike Tutkowski
Guaranteeing Storage Performance by Mike Tutkowskibuildacloud
ย
The document outlines the development and features of SolidFire's clustered SAN technology, emphasizing its capability to offer guaranteed quality of service (QoS) for storage in CloudStack environments. It discusses the automation of provisioning primary storage, the creation of volumes, and the management of storage resources, especially highlighting its compatibility with various hypervisors. Additionally, it mentions the integration of unique vendor features within CloudStack to streamline storage allocation and orchestration processes.Cloud Application Blueprints with Apache Brooklyn by Alex Henevald
Cloud Application Blueprints with Apache Brooklyn by Alex Henevaldbuildacloud
ย
The document discusses Apache Brooklyn, an open source multi-cloud application management platform aimed at simplifying application deployment using composable blueprints. It outlines the framework's features, including push-button deployment, ongoing management, and benefits like agility and reliability. Additionally, the document presents various multi-tier application blueprints in YAML format demonstrating service configurations and policies for managing applications effectively.Introduction to Apache CloudStack by David Nalley
Introduction to Apache CloudStack by David Nalleybuildacloud
ย
The document provides an introduction to Apache CloudStack, detailing its history from stealth mode startup to its current status as a mature orchestration platform for Infrastructure as a Service (IaaS). It discusses key components such as compute, networking, storage, hypervisor support, and APIs, emphasizing the importance of orchestration in managing existing services. Additionally, the document highlights challenges in cloud networking, scaling issues, and the necessity for configuration management in cloud environments.Managing infrastructure with Application Policy by Mike Cohen
Managing infrastructure with Application Policy by Mike Cohenbuildacloud
ย
The document discusses the advantages of using declarative management and application-centric policies for managing network infrastructure. Key points include:
1) Declarative management allows defining policies based on application intent rather than low-level device configurations, improving scalability, resiliency, interoperability, and ease of use.
2) Application-centric policies define rules between endpoint groups (EGPs) using contracts that specify filters and actions applied to network traffic. This mirrors how applications are structured.
3) Open source initiatives aim to support application-centric policy models and APIs in projects like OpenStack, OpenDaylight, and hypervisors to provide an open approach for network automation aligned with application deployments.Intro to Zenoss by Andrew Kirch
Intro to Zenoss by Andrew Kirchbuildacloud
ย
This document summarizes an introduction presentation about the open source unified monitoring platform Zenoss given by Andrew Kirch on February 21, 2014 at SCaLE 12x. The presentation covered an elevator pitch describing Zenoss as using standard protocols like SNMP and SSH to monitor servers, networks, services, and clouds in a unified manner. It also demonstrated Zenoss' extensibility through ZenPacks and its ability to be installed with a small footprint using a community script. Questions from the audience were taken at the end.Monitoring CloudStack in context with Converged Infrastructure by Mike Turnlund
Monitoring CloudStack in context with Converged Infrastructure by Mike Turnlundbuildacloud
ย
CA Technologies offers CA Nimsoft Monitor for CloudStack, a solution designed to manage and monitor complex cloud environments. It provides end-to-end visibility, tracking SLAs and resource performance across physical and virtual components, ensuring efficient infrastructure management. The software aims to streamline monitoring with customizable dashboards and flexible reporting, catering to enterprises and managed service providers.Rest api design by george reese
Rest api design by george reesebuildacloud
ย
George Reese discusses best practices for REST API design, emphasizing the importance of treating the API as the primary interface and adhering strictly to HTTP specifications. He covers topics such as resource modeling, authentication, error handling, and documentation, while cautioning against the pitfalls of SOAP and poorly designed APIs. The presentation also highlights the flexibility that REST APIs offer for enabling diverse use cases beyond the original designer's intent.Enterprise grade firewall and ssl termination to ac by will stevens
Enterprise grade firewall and ssl termination to ac by will stevensbuildacloud
ย
The document outlines the integration of Palo Alto Networks firewall and SSL termination in ACS 4.3, emphasizing the benefits of enhanced security features for cloud infrastructures. It details the setup process for the Palo Alto device, the creation of service offerings, and the implementation of SSL termination workflows. Additional notes highlight limitations and the current status of the implementation's availability and features.State of the cloud by reuven cohen
State of the cloud by reuven cohenbuildacloud
ย
The document summarizes the state of cloud computing and its future direction. It discusses how (1) companies are embracing automation, data, design, and connectivity as core skills; (2) mobility and wearable computing will drive the next wave of disruption; and (3) computing will increasingly fuse with the physical world as more "things" become connected. It predicts surprising intersections of technologies will lead to world-changing breakthroughs in the coming age of innovation.Securing Your Cloud With the Xen Hypervisor by Russell Pavlicek
Securing Your Cloud With the Xen Hypervisor by Russell Pavlicekbuildacloud
ย
This document provides an overview of securing cloud infrastructure using the Xen hypervisor. It begins with introducing Xen security features like driver domains and PVgrub that isolate different components of the system. It then analyzes various attack surfaces like the network path, PyGrub bootloader, and Qemu device model. For each, it discusses what a successful exploit could achieve and how Xen features like driver domains and stub domains constrain the impact. The document concludes with recommendations to use the Xen security module FLASK to further restrict privileges and provides examples of applying it.DevCloud - Setup and Demo on Apache CloudStack
DevCloud - Setup and Demo on Apache CloudStack buildacloud
ย
The document discusses the setup and demonstration of DevCloud, an all-in-one-box solution for CloudStack development, highlighting its requirements and functionalities. It provides detailed instructions for configuring VirtualBox on both *nix and Windows systems, as well as commands for building and deploying applications in a DevCloud environment. The steps outlined allow users to set up a development environment, manage virtual machines, and deploy infrastructure efficiently.Cloud Network Virtualization with Juniper Contrail
Cloud Network Virtualization with Juniper Contrailbuildacloud
ย
The document discusses the fundamental shift in application evolution from standalone to dynamic virtual networks, emphasizing the need for automation and service orchestration in modern data centers. It outlines the architecture and operational models for integrating physical and virtual networks, highlighting the use of technologies like MPLS and VXLAN. Additionally, the document presents Juniper Networks' approach to enhancing cloud management through scalable, resilient federated controllers and service insertion capabilities.Ian rae panel cloud stack & cloud storage where are we at, and where do we ne...
Ian rae panel cloud stack & cloud storage where are we at, and where do we ne...buildacloud
ย
The document discusses Palo Alto Networks firewall orchestration using CloudStack. It provides an overview of the speakers, project objectives to integrate Palo Alto firewalls with CloudStack, and the approach taken. This includes pre-configuring the Palo Alto device, adding it as a service provider in CloudStack, creating network offerings, and demonstrating how firewall, NAT, and forwarding rules are configured on the Palo Alto when networks and VMs are deployed through CloudStack.Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski
Troubleshooting Strategies for CloudStack Installations by Kirk Kosinski buildacloud
ย
The document provides troubleshooting strategies for CloudStack installations, including network issues, security groups, host connectivity, virtual routers, templates, and log analysis. It discusses common problems such as VLAN misconfigurations, security group rules not being applied, hosts showing in the "avoid set", template preparation errors, and exceptions in the logs. It emphasizes analyzing logs at the management server, hypervisor, and job levels to find the root cause of failures.CloudStack University by Sebastien Goasguen
CloudStack University by Sebastien Goasguenbuildacloud
ย
This document discusses plans to create a MOOC (Massive Open Online Course) for teaching Apache CloudStack. Some ideas include creating real-time online webinars, self-paced courses with exercises and labs, and internship opportunities. It also lists some potential student project topics, such as improving the UI, adding hypervisor support, integrating Mesos and Hadoop, and improving LDAP integration. Finally, it demonstrates some CloudStack tools like CloudMonkey and provides resources for learning more about CloudStack.Ad
Recently uploaded (20)
FIDO Seminar: Evolving Landscape of Post-Quantum Cryptography.pptx
FIDO Seminar: Evolving Landscape of Post-Quantum Cryptography.pptxFIDO Alliance
ย
FIDO Seminar: Evolving Landscape of Post-Quantum CryptographyData Validation and System Interoperability
Data Validation and System InteroperabilitySafe Software
ย
A non-profit human services agency with specialized health record and billing systems. Challenges solved include access control integrations from employee electronic HR records, multiple regulations compliance, data migrations, benefits enrollments, payroll processing, and automated reporting for business intelligence and analysis.Supporting the NextGen 911 Digital Transformation with FME
Supporting the NextGen 911 Digital Transformation with FMESafe Software
ย
Next Generation 911 involves the transformation of our 911 system from an old analog one to the new digital internet based architecture. The evolution of NG911 opens up a host of new opportunities to improve the system. This includes everything from device based location, to real time text. This can improve location accuracy dramatically as well as provide live updates from the citizen in need along with real time sensor updates. There is also the opportunity to provide multi-media attachments and medical records if the end user approves. This digital transformation and enhancements all require the support of new NENA and CRTC standards, along with integration across a variety of data streams.
This presentation will focus on how FME has supported NG911 transformations to date, and how we are positioning FME to support the enhanced capabilities to come. This session will be of interest to emergency services, municipalities and anyone who may be interested to know more about how emergency services are being improved to provide more accurate, localized information in order to improve the speed and relevance of emergency response and ultimately save more lives and provide better outcomes for those in need.Security Tips for Enterprise Azure Solutions
Security Tips for Enterprise Azure SolutionsMichele Leroux Bustamante
ย
Delivering solutions to Azure may involve a variety of architecture patterns involving your applications, APIs data and associated Azure resources that comprise the solution. This session will use reference architectures to illustrate the security considerations to protect your Azure resources and data, how to achieve Zero Trust, and why it matters. Topics covered will include specific security recommendations for types Azure resources and related network security practices. The goal is to give you a breadth of understanding as to typical security requirements to meet compliance and security controls in an enterprise solution.Providing an OGC API Processes REST Interface for FME Flow
Providing an OGC API Processes REST Interface for FME FlowSafe Software
ย
This presentation will showcase an adapter for FME Flow that provides REST endpoints for FME Workspaces following the OGC API Processes specification. The implementation delivers robust, user-friendly API endpoints, including standardized methods for parameter provision. Additionally, it enhances security and user management by supporting OAuth2 authentication. Join us to discover how these advancements can elevate your enterprise integration workflows and ensure seamless, secure interactions with FME Flow.Kubernetes Security Act Now Before Itโs Too Late
Kubernetes Security Act Now Before Itโs Too LateMichael Furman
ย
In today's cloud-native landscape, Kubernetes has become the de facto standard for orchestrating containerized applications, but its inherent complexity introduces unique security challenges. Are you one YAML away from disaster?
This presentation, "Kubernetes Security: Act Now Before Itโs Too Late," is your essential guide to understanding and mitigating the critical security risks within your Kubernetes environments. This presentation dives deep into the OWASP Kubernetes Top Ten, providing actionable insights to harden your clusters.
We will cover:
The fundamental architecture of Kubernetes and why its security is paramount.
In-depth strategies for protecting your Kubernetes Control Plane, including kube-apiserver and etcd.
Crucial best practices for securing your workloads and nodes, covering topics like privileged containers, root filesystem security, and the essential role of Pod Security Admission.
Don't wait for a breach. Learn how to identify, prevent, and respond to Kubernetes security threats effectively.
It's time to act now before it's too late!Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019
ย
A presentation at Internetware 2025.Viral>Wondershare Filmora 14.5.18.12900 Crack Free Download
Viral>Wondershare Filmora 14.5.18.12900 Crack Free DownloadPuppy jhon
ย
โก ๐๐ฑ๐COPY & PASTE LINK๐๐๐ โค โคโค https://drfiles.net/
Wondershare Filmora Crack is a user-friendly video editing software designed for both beginners and experienced users.
Edge-banding-machines-edgeteq-s-200-en-.pdf
Edge-banding-machines-edgeteq-s-200-en-.pdfAmirStern2
ย
ืืืื ืช ืงื ืืื ืืืชืืืื ืื ืืจืืืช ืงืื ืืช ืื ืืืืืืช (ืืืืื ืช ืืืืื).
ืืืืืงื ืงื ืืื ืืืืื ืื ืคืกืื, ืขื ืขืืื ืงื ื โ 3 ื"ื ืืขืืื ืืืืจ ืขื 40 ื"ื. ืืงืจ ืืืืืฉื ืืืชืจืืข ืขื ืชืงืืืช, ืืื ืืขืื ืืืกืืืืื ืชืขืฉืืืชืืื ืืื ืืืืื ืืช ืืืืืืืช.Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation
ย
Can We Use Rust to Develop Extensions for PostgreSQL?
(POSETTE: An Event for Postgres 2025)
June 11, 2025
Shinya Kato
NTT DATA Japan CorporationArtificial Intelligence in the Nonprofit Boardroom.pdf
Artificial Intelligence in the Nonprofit Boardroom.pdfOnBoard
ย
OnBoard recently partnered with Microsoft Tech for Social Impact on the AI in the Nonprofit Boardroom Survey, an initiative designed to uncover the current and future role of artificial intelligence in nonprofit governance. June Patch Tuesday
June Patch TuesdayIvanti
ย
Ivantiโs Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There weโll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities. FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptxFIDO Alliance
ย
FIDO Seminar: New Data: Passkey Adoption in the WorkforceAI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?Shashi Sathyanarayana, Ph.D
ย
This slide illustrates a side-by-side comparison between human-written, AI-written, and ambiguous content. It highlights subtle cues that help readers assess authenticity, raising essential questions about the future of communication, trust, and thought leadership in the age of generative AI.FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce.pptx
FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce.pptxFIDO Alliance
ย
FIDO Seminar: Targeting Trust: The Future of Identity in the WorkforceOpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC
ย
The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025Safe Software
ย
Harness the full potential of AI with FME: From creating high-quality training data to optimizing models and utilizing results, FME supports every step of your AI workflow. Seamlessly integrate a wide range of models, including those for data enhancement, forecasting, image and object recognition, and large language models. Customize AI models to meet your exact needs with FMEโs powerful tools for training, optimization, and seamless integrationAI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc
ย
AI VIDEO MAGAZINE - r/aivideo community newsletter โ Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation
ย
Scalable Object Storage with Apache CloudStack and Apache Hadoop
- 1. Scalable Object Storage with Apache CloudStack and Apache Hadoop April 30 2013 Chiradeep Vittal @chiradeep
- 2. Agenda โข What is CloudStack โข Object Storage for IAAS โข Current Architecture and Limitations โข Requirements for Object Storage โข Object Storage integrations in CloudStack โข HDFS for Object Storage โข Future directions
- 3. โข History โข Incubating in the Apache Software Foundation since April 2012 โข Open Source since May 2010 โข In production since 2009 โ Turnkey platform for delivering IaaS clouds โ Full featured GUI, end-user API and admin API Apache CloudStack Build your cloud the way the worldโs most successful clouds are built
- 4. How did Amazon build its cloud? Commodity Servers Commodity Storage Networking Open Source Xen Hypervisor Amazon Orchestration Software AWS API (EC2, S3, โฆ) Amazon eCommerce Platform
- 5. How can YOU build a cloud? Servers StorageNetworking Open Source Xen Hypervisor Amazon Orchestration Software AWS API (EC2, S3, โฆ) Amazon eCommerce Platform Hypervisor (Xen/KVM/VMW/) CloudStack Orchestration Software Optional Portal CloudStack or AWS API
- 6. Secondary Storage Image L3/L2 core DC Edge End users Pod Pod Pod Pod Zone Architecture Pod Access Sw MySQL CloudStack Admin/User API Primary Storage NFS/ISCSI/FC Hypervisor (Xen /VMWare/KVM) VM VM Snapshot Snapshot Image Disk Disk VM
- 7. Cloud-Style Workloads โข Low cost โ Standardized, cookie cutter infrastructure โ Highly automated and efficient โข Application owns availability โ At scale everything breaks โ Focus on MTTR instead of MTBF
- 8. Secondary Storage Image L3/L2 core DC Edge Pod Pod Pod Pod At scaleโฆeverything breaks Pod Access Sw Primary Storage NFS/ISCSI/FC Hypervisor (Xen /VMWare/KVM) VM VM Snapshot Snapshot Image Disk Disk VM
- 9. Region โWestโ Zone โWest-Alphaโ Zone โWest-Betaโ Zone โWest-Gammaโ Zone โWest-Deltaโ Low Latency Backbone (e.g., SONET ring) Regions and zones
- 10. Region โEastโ Region โSouthโ Internet Geographic separation Region โWestโ Low Latency
- 11. Secondary Storage in CloudStack 4.0 โข NFS server default โ can be mounted by hypervisor โ Easy to obtain, set up and operate โข Problems with NFS: โ Scale: max limits of file systems โข Solution: CloudStack can manage multiple NFS stores (+ complexity) โ Performance โข N hypervisors : 1 storage CPU / 1 network link โ Wide area suitability for cross-region storage โข Chatty protocol โ Lack of replication
- 12. Object Storage Technology Region โWestโ Zone โWest-Alphaโ Zone โWest-Betaโ Zone โWest-Gammaโ Zone โWest-Deltaโ Object Storage in a region โข Replication โข Audit โข Repair โข Maintenance
- 13. Region โWestโ Object Storage enables reliability
- 14. Object Storage Technology Region โWestโ Object Storage also enables other applications Object Store API Servers โข DropBox โข Static Content โข Archival
- 15. Object Storage characteristics โข Highly reliable and durable โ 99.9 % availability for AWS S3 โ 99.999999999 % durability โข Massive scale โ 1.3 trillion objects stored across 7 AWS regions [Nov 2012 figures] โ Throughput: 830,000 requests per second โข Immutable objects โ Objects cannot be modified, only deleted โข Simple API โ PUT/POST objects, GET objects, DELETE objects โ No seek / no mutation / no POSIX API โข Flat namespace โ Everything stored in buckets. โ Bucket names are unique โ Buckets can only contain objects, not other buckets โข Cheap and getting cheaper
- 16. CloudStack S3 API Server Object Storage Technology S3 API Servers MySQL
- 17. CloudStack S3 API Server โข Understands AWS S3 REST-style and SOAP API โข Pluggable backend โ Backend storage needs to map simple calls to their API โข E.g., createContainer, saveObject, loadObject โ Default backend is a POSIX filesystem โ Backend with Caringo Object Store (commercial vendor) available โ HDFS backend also available โข MySQL storage โ Bucket -> object mapping โ ACLs, bucket policies
- 18. Object Store Integration into CloudStack โข For images and snapshots โข Replacement for NFS secondary storage Or Augmentation for NFS secondary storage โข Integrations available with โ Riak CS โ Openstack Swift โข New in 4.2 (upcoming): โ Framework for integrating storage providers
- 19. What do we want to build ? โข Open source, ASL licensed object storage โข Scales to at least 1 billion objects โข Reliability and durability on par with S3 โข S3 API (or similar, e.g., Google Storage) โข Tooling around maintenance and operation, specific to object storage
- 20. The following slides are a design discussion
- 21. Architecture of Scalable Object Storage API Servers Auth Servers Object Servers Replicators/Auditors Object Lookup Servers
- 22. Why HDFS โข ASF Project (Apache Hadoop) โข Immutable objects, replication โข Reliability, scale and performance โ 200 million objects in 1 cluster [Facebook] โ 100 PB in 1 cluster [Facebook] โข Simple operation โ Just add data nodes
- 23. HDFS-based Object Storage S3 API Servers S3 Auth Servers Data nodes Namenode pair HDFS API
- 24. BUT โข Name Node Scalability โ 150 bytes RAM / block โ GC issues โข Name Node SPOF โ Being addressed in the communityโ โข Cross-zone replication โ Rack-awareness placement โ โ What if the zones are spread a little further apart? โข Storage for object metadata โ ACLs, policies, timers
- 25. Name Node scalability โข 1 billion objects = 3 billion blocks (chunks) โ Average of 5 MB/object = 5 PB (actual), 15 PB (raw) โ 450 GB of RAM per Name Node โข 150b x 3 x 10^9 โ 16 TB / node => 1000 Data nodes โข Requires Name Node federation ? โข Or an approach like HAR files
- 26. Name Node Federation Extension: Federated NameNodes are HA pairs
- 27. Federation issues โข HA for name nodes โข Namespace shards โ Map object -> name node โข Requires another scalable key-value store โ HBase? โข Rebalancing between name nodes
- 28. Replication over lossy/slower links A. Asynchronous replication โ Use distcp to replicate between clusters โ 6 copies vs. 3 โ Master/Slave relationship โข Possibility of loss of data during failover โข Need coordination logic outside of HDFS B. Synchronous replication โ API server writes to 2 clusters and acks only when both writes are successful โ Availability compromised when one zone is down
- 29. CAP Theorem Consistency or Availability during partition Many nuances
- 30. Storage for object metadata A. Store it in HDFS along with the object โ Reads are expensive (e.g., to check ACL) โ Mutable data, needs layer over HDFS B. Use another storage system (e.g. HBase) โ Name node federation also requires this. C. Modify Name Node to store metadata โ High performance โ Not extensible
- 31. Object store on HDFS Future โข Viable for small-sized deployments โ Up to 100-200 million objects โ Datacenters close together โข Larger deployments needs development โ No effort ongoing at this time
- 32. Conclusion โข CloudStack needs object storage for โcloud-styleโ workloads โข Object Storage is not easy โข HDFS comes close but not close enough โข Join the community!
Editor's Notes
- #4: Need a better slide than this
- #5: Frequently require CCNA , Vmwareceritification, EMC training, etc etc. But they chose commondity systems. And simple networking.Can also sell cheaply since they use their own commerce platform.
- #6: The key here is the API on top of the infrastructure. This is the disruptive piece for the industry. Forget about CCNA, Vmware cert, now people can programmatically control their infrastructure as well as the VMs on top of it.