Bandwidth: Use Cases for Elastic Cloud on Kubernetes
0 likes1,889 views
The document outlines the evolution of logging infrastructure at Bandwidth, focusing on the integration of Kubernetes, Elastic Stack, and AWS services. It discusses various phases from 2016 to 2019, highlighting the implementation of Kafka for data management, OpenShift for streamlined operations, and Elastic Cloud for evolving use cases. Future plans include further integration of components and the development of operational capabilities within OpenShift.
1 of 47
Downloaded 19 times
Ad
Recommended
Search for All with Elastic Workplace Search
Search for All with Elastic Workplace Search Elasticsearch The document outlines Elastic's solutions for enterprise search, emphasizing the need for high-quality, purpose-built tools to manage increasing workplace applications and content. It introduces a comprehensive solution for deploying a personalized, secure search experience across various data sources with an extensive connector catalog. Additionally, it highlights future developments, including enhancements in search features, cloud deployment options, and simplified management functionalities.
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch This document discusses the partnership between Elastic and Microsoft Azure and highlights several products and services:
1. Elastic provides solutions for logs, metrics, application performance monitoring, uptime monitoring, security information and event management, and endpoints on the Elastic Stack that can be deployed on Azure in various ways.
2. The Elasticsearch Service on Azure is highlighted as the best way to deploy Elasticsearch, Kibana, and Elastic solutions on Azure with benefits like being hosted, secure, compliant, and always up-to-date.
3. Elastic Observability for Azure provides out-of-the-box support for Azure logs and metrics with integration for Azure Monitor and pre-built Kibana dashboards.
Deploying Cloud Use Cases
Deploying Cloud Use CasesJason Singh The document outlines Datapipe's expertise in cloud deployments, emphasizing its advanced security services, automation, orchestration, and proactive monitoring solutions. It highlights key use cases, including compliance enforcement in hybrid IT environments and the automation of large-scale deployments, showcasing the company's integration of security and efficiency. The lifecycle of security, automation, and monitoring is also briefly illustrated, indicating a comprehensive approach to managing cloud infrastructures.
Service Fabric and Azure Service Fabric Mesh introduction
Service Fabric and Azure Service Fabric Mesh introductionMikkel Mørk Hegnhøj Azure Service Fabric is an open-source distributed systems platform for packaging, deploying, and managing distributed applications and services at scale. Service Fabric provides capabilities for building microservices, managing application lifecycles, monitoring health and load balancing across infrastructure. Azure Service Fabric Mesh is a new serverless platform that builds upon Service Fabric to deploy and manage containerized microservices applications across any infrastructure.
DotnetConf - Cloud native and .Net5 announcements
DotnetConf - Cloud native and .Net5 announcementsSajeetharan - .NET 5 was recently released with improvements like smaller container images, web and cloud investments, and performance gains.
- The .NET ecosystem is growing rapidly, with over 5 million developers and top rankings in popularity.
- Cloud native development with .NET emphasizes microservices, containers, automation, and consuming backend services.
- Demonstrations showed deploying an application to Azure Kubernetes Service and using Project Tye for faster microservices development.
Monitoring Containerized Micro-Services In Azure
Monitoring Containerized Micro-Services In AzureAlex Bulankou This document discusses best practices for monitoring containerized microservices applications in Azure. It begins with an introduction to Application Insights and describes the agenda. It then discusses what is different about monitoring microservices compared to monolithic applications and some factors to consider when choosing a monitoring system. The document provides recommendations for setting up day-to-day monitoring operations, including maintaining a 15 minute daily triage process focusing on business metrics, application performance and health, and infrastructure and costs. It concludes with a demo of monitoring a sample microservices application using Application Insights and other tools.
Cloud Use Cases And Standards
Cloud Use Cases And StandardsGovCloud Network The document discusses cloud computing use cases and related standards requirements. It lists several general requirement categories for cloud computing including common virtual machine formats, data formats and APIs; cloud management; security; location awareness; identity; open clients; service level agreements; federated identity; metering and monitoring; and more. It also maps some of these requirements to specific use cases and discusses a phased approach and timeline for delivering different cloud computing models.
Azure Container Services
Azure Container ServicesPedro Sousa The document discusses Pedro Sousa's role as a systems engineer at DevScope and provides links to various resources and events related to Docker and container management. It mentions contributions from several key figures in the container ecosystem, including Microsoft program managers and Docker captains. The focus is on managing microservices and getting started with Windows Server containers.
Azure Container Services
Azure Container ServicesPedro Sousa The document summarizes a DevOps Porto Community meeting. It includes the name and contact information of a speaker, Pedro Sousa, as well as links to some of his past presentations. It also lists several Microsoft employees who were present at the meeting including Taylor Brown, Stefan Scherer, Ross Gardler, and Donovan Brown. The document poses a question and includes a diagram outlining the Azure container platform.
TugaIT 2016 - Docker and the world of “containerized" environments
TugaIT 2016 - Docker and the world of “containerized" environmentsPedro Sousa This document summarizes a presentation about Docker and containerized environments. It begins by thanking sponsors and the presentation team. The presentation introduces containers and Docker, explaining that Docker allows packaging applications and dependencies into standardized units. It demonstrates Docker through a DEMO and discusses container workloads and Docker orchestration with Swarm and other tools. It poses the question of whether containers are ready for production and enterprise use, citing a survey showing growing Docker adoption.
John Willis Cc Use Cases
John Willis Cc Use CasesGovCloud Network The document discusses cloud computing definitions, characteristics, delivery models, and deployment models as defined by NIST. It also discusses various cloud computing use cases including moving from an enterprise data center to public/hybrid clouds, using clouds for big data analytics, and changing between cloud vendors. Open source cloud platforms like Eucalyptus and OpenStack are also mentioned.
Opening Keynote
Opening KeynoteElasticsearch The presentation discusses Elastic's forward-looking statements and the potential risks associated with them. It highlights the significance of simplifying search functionality in everyday work and announces new integrations and cloud services for Elastic's offerings, including expanded availability across multiple cloud providers. The focus is on enhancing user experience with more options for deployment and services.
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityElasticsearch The document discusses Elastic's observability solutions for logs, metrics, and APM, highlighting the complexities of modern architectures and the transition to cloud-native environments. It emphasizes the need for a unified approach to monitoring through tools that correlate data from different sources and provides a streamlined licensing model. Additionally, the evolution of Elastic's stack for handling logs, metrics, and APM is outlined, showcasing its capabilities for high-cardinality aggregations and real-time monitoring.
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...Tom Kerkhove The document outlines Azure Data Factory's capabilities as a managed data orchestration service designed for creating data integration pipelines. It highlights features such as support for multiple data sources, various triggers, and activities involved in the data processing workflow. The document emphasizes Azure Data Factory's role in enabling serverless and hybrid data scenarios while maintaining compliance and security.
Empower Your Security Practitioners with Elastic SIEM
Empower Your Security Practitioners with Elastic SIEMElasticsearch The document presents an overview of Elastic SIEM (Security Information and Event Management) and its integration within the Elastic Stack, highlighting features like the Elastic Common Schema (ECS), automated detection using machine learning, and various data sources for enhanced security analytics. It emphasizes empowering security practitioners with tools for prevention, detection, and rapid response to incidents. Additionally, it notes the ongoing collaboration within a community of vendors and experts to improve the Elastic SIEM ecosystem.
Virtual Global Azure 2020 - Azure Monitor
Virtual Global Azure 2020 - Azure MonitorPedro Sousa The document details an Azure Monitor overview by Pedro Sousa, highlighting monitoring lifecycle aspects such as metrics, logs, application insights, and best practices. It includes a schedule for a session covering topics from product vision to practical applications. Additional resources like documentation and community support are provided for further learning.
Migrating SSIS to the cloud
Migrating SSIS to the cloudKoenVerbeeck This document discusses migrating SSIS packages to the cloud using the Azure-SSIS Integration Runtime (IR). It describes what the Azure-SSIS IR is, when it makes sense to migrate packages to it, and how to set up the Azure-SSIS IR. Setting up the IR involves choosing an Azure SQL database or managed instance for the SSIS catalog, configuring connections, deploying SSIS projects, and scheduling packages. Custom setups are also possible by loading external DLLs. Typical data flows in Azure Data Factory are then discussed for lifting and shifting SSIS packages to the cloud.
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...HostedbyConfluent The document discusses leveraging Apache Kafka for enhancing innovation within organizations, particularly focused on event-driven architecture. It outlines benefits such as faster development cycles, scalability, and improved data management through decoupled architecture. Additionally, it emphasizes the importance of a supportive team environment and strong partnerships with Kafka providers for successful implementation.
The Future of Enterprise Applications is Serverless
The Future of Enterprise Applications is ServerlessEficode The document discusses the evolution and benefits of serverless architecture within Amazon Web Services (AWS), highlighting the shift from monolithic systems to decoupled services managed by small teams. It emphasizes that serverless applications allow for greater agility, reduced overhead, and improved scalability without the need to manage servers. Key features such as AWS Lambda, API Gateway, and their applications in various domains like web development and data processing are also presented.
Nurturing a large GST ecosystem on AWS - Anil Sharma, Chicago
Nurturing a large GST ecosystem on AWS - Anil Sharma, ChicagoAWS Chicago Motherson Sumi is a large global automotive components manufacturer with over 235 facilities in 37 countries. In India, the government implemented a new Goods and Services Tax (GST) system to simplify taxation. Motherson Sumi was selected to develop a system to help businesses compute and file their GST taxes. They built a multi-tenant system on AWS that could scale to millions of users, integrate with other systems, and be quickly updated as tax policies changed. The system included a user portal and APIs to allow tax filing and integration with existing business systems. Motherson Sumi leveraged many AWS services to build this complex system securely, quickly and at scale to help modernize taxation in India.
Webinar Cloud Native Community.pptx
Webinar Cloud Native Community.pptxLibbySchulze This document provides an overview of making applications cloud-native including:
- Defining cloud-native and its impacts on applications
- The advantages of using containers for applications
- When and why to use Kubernetes for container orchestration
- A demonstration of containerizing a sample application
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch The presentation introduces the partnership between Microsoft and Elastic to offer various Elasticsearch services on Azure, highlighting new regions and integrations. It emphasizes the benefits of flexible service offerings, including managed and self-managed options. The document also includes a forward-looking statement regarding risks associated with the services.
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...Tom Kerkhove Kubernetes Event-driven Autoscaling (KEDA) provides application autoscaling on Kubernetes using a variety of metric sources. It automatically scales deployments, jobs, and other resources. KEDA supports over 30 built-in scalers for sources like Azure, AWS, Google Cloud, and more. It is cloud-agnostic and focuses on scaling applications without managing the scaling internals. The Azure Functions CLI makes it easy to deploy functions to Kubernetes and automatically configure KEDA for autoscaling. KEDA is an open source project with over 2,800 stars on GitHub and contributions from Microsoft, Red Hat, and other companies.
Application Development on Metapod
Application Development on MetapodCisco DevNet The document discusses various approaches to application development on OpenStack including doing it yourself with open source tools (DIY), using configuration management tools to gather automation tools into a management layer (Mantl), and adopting standards and components through Platform as a Service tools (Cloud Foundry). It provides examples of tools for infrastructure as code (Heat), container orchestration (Docker), and describes the basic flow of application deployment on Cloud Foundry including using its open source software plus commercial modules. The goal is to help development teams focus on application delivery rather than underlying OpenStack configuration and management.
Combinação de logs, métricas e rastreamentos para observabilidade unificada
Combinação de logs, métricas e rastreamentos para observabilidade unificadaElasticsearch The document discusses the evolution and capabilities of Elastic's observability solutions, highlighting the importance of logs, metrics, and APM for a unified observability strategy. It emphasizes the complexity of monitoring dynamic cloud-native architectures and the benefits of integrating various data sources for intelligent anomaly detection and unified monitoring. Additionally, it outlines the introduction of various Elastic tools like Metricbeat and Elastic APM for improved performance monitoring and analysis.
Application Autoscaling Made Easy with Kubernetes Event-Driven Autoscaling (K...
Application Autoscaling Made Easy with Kubernetes Event-Driven Autoscaling (K...Codit This document summarizes a presentation about Kubernetes Event-driven Autoscaling (KEDA). KEDA allows applications running on Kubernetes to automatically scale based on external events from services like Azure Event Hubs, Kafka, or Cosmos DB. It provides out-of-the-box and custom scalers to monitor event sources and scale deployments and jobs as needed. KEDA is open source, cloud agnostic, and aims to simplify autoscaling so developers can focus on their applications rather than scaling internals. The presenters demonstrate using KEDA to scale a .NET Core worker based on an Azure Service Bus queue depth.
Achieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactElasticsearch The document discusses the integration of Elastic and ECS within the Mission Assurance Decision Support System (MADSS) to enhance mission assurance capabilities through real-time data analysis. It outlines the evolution from traditional architectures to a modernized, microservices approach utilizing the Elastic stack for improved data handling, search capabilities, and observability. Key insights include the importance of understanding data dynamics and optimizing pipelines for effective ingestion and monitoring.
Grab: Building a Healthy Elasticsearch Ecosystem
Grab: Building a Healthy Elasticsearch EcosystemElasticsearch The document discusses how Grab uses Elasticsearch (ES) and cloud technologies to address diverse use cases across different teams, highlighting operational challenges like search and monitoring in a complex environment. It emphasizes the importance of customized solutions for various user groups, ensuring access control based on user demographics and roles. Additionally, the evolution of Grab's infrastructure from basic operations to a more integrated and scalable ES solution is outlined, showcasing enhanced performance and management through cloud technologies.
seven-ways-to-run-flink-on-aws.pdf
seven-ways-to-run-flink-on-aws.pdfSergioBruno21 This document provides an overview of Apache Flink and different ways to deploy Flink applications on cloud platforms. It discusses deploying Flink on EMR, EC2, ECS, EKS, Kinesis Data Analytics for Java, and Lambda. It recommends EMR, discusses advantages like managed Hadoop clusters, and disadvantages like needing to manage the cluster. It also provides an overview of Flink architecture, components like the JobManager and TaskManager, and the data flow API.
4 - Customer story: Telenet
4 - Customer story: TelenetKangaroot The document discusses the implementation and use cases of Elasticsearch for troubleshooting, log monitoring, and alerting within IT infrastructure. It outlines the setup process, including hardware migration strategies, data retention policies, and performance metrics for logs. Additionally, it addresses the incorporation of machine learning for improving alert accuracy and anomaly detection.
More Related Content
What's hot (18)
Azure Container Services
Azure Container ServicesPedro Sousa The document summarizes a DevOps Porto Community meeting. It includes the name and contact information of a speaker, Pedro Sousa, as well as links to some of his past presentations. It also lists several Microsoft employees who were present at the meeting including Taylor Brown, Stefan Scherer, Ross Gardler, and Donovan Brown. The document poses a question and includes a diagram outlining the Azure container platform.
TugaIT 2016 - Docker and the world of “containerized" environments
TugaIT 2016 - Docker and the world of “containerized" environmentsPedro Sousa This document summarizes a presentation about Docker and containerized environments. It begins by thanking sponsors and the presentation team. The presentation introduces containers and Docker, explaining that Docker allows packaging applications and dependencies into standardized units. It demonstrates Docker through a DEMO and discusses container workloads and Docker orchestration with Swarm and other tools. It poses the question of whether containers are ready for production and enterprise use, citing a survey showing growing Docker adoption.
John Willis Cc Use Cases
John Willis Cc Use CasesGovCloud Network The document discusses cloud computing definitions, characteristics, delivery models, and deployment models as defined by NIST. It also discusses various cloud computing use cases including moving from an enterprise data center to public/hybrid clouds, using clouds for big data analytics, and changing between cloud vendors. Open source cloud platforms like Eucalyptus and OpenStack are also mentioned.
Opening Keynote
Opening KeynoteElasticsearch The presentation discusses Elastic's forward-looking statements and the potential risks associated with them. It highlights the significance of simplifying search functionality in everyday work and announces new integrations and cloud services for Elastic's offerings, including expanded availability across multiple cloud providers. The focus is on enhancing user experience with more options for deployment and services.
Combining Logs, Metrics, and Traces for Unified Observability
Combining Logs, Metrics, and Traces for Unified ObservabilityElasticsearch The document discusses Elastic's observability solutions for logs, metrics, and APM, highlighting the complexities of modern architectures and the transition to cloud-native environments. It emphasizes the need for a unified approach to monitoring through tools that correlate data from different sources and provides a streamlined licensing model. Additionally, the evolution of Elastic's stack for handling logs, metrics, and APM is outlined, showcasing its capabilities for high-cardinality aggregations and real-time monitoring.
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...
Intelligent Cloud Conference 2018 - Next Generation of Data Integration with ...Tom Kerkhove The document outlines Azure Data Factory's capabilities as a managed data orchestration service designed for creating data integration pipelines. It highlights features such as support for multiple data sources, various triggers, and activities involved in the data processing workflow. The document emphasizes Azure Data Factory's role in enabling serverless and hybrid data scenarios while maintaining compliance and security.
Empower Your Security Practitioners with Elastic SIEM
Empower Your Security Practitioners with Elastic SIEMElasticsearch The document presents an overview of Elastic SIEM (Security Information and Event Management) and its integration within the Elastic Stack, highlighting features like the Elastic Common Schema (ECS), automated detection using machine learning, and various data sources for enhanced security analytics. It emphasizes empowering security practitioners with tools for prevention, detection, and rapid response to incidents. Additionally, it notes the ongoing collaboration within a community of vendors and experts to improve the Elastic SIEM ecosystem.
Virtual Global Azure 2020 - Azure Monitor
Virtual Global Azure 2020 - Azure MonitorPedro Sousa The document details an Azure Monitor overview by Pedro Sousa, highlighting monitoring lifecycle aspects such as metrics, logs, application insights, and best practices. It includes a schedule for a session covering topics from product vision to practical applications. Additional resources like documentation and community support are provided for further learning.
Migrating SSIS to the cloud
Migrating SSIS to the cloudKoenVerbeeck This document discusses migrating SSIS packages to the cloud using the Azure-SSIS Integration Runtime (IR). It describes what the Azure-SSIS IR is, when it makes sense to migrate packages to it, and how to set up the Azure-SSIS IR. Setting up the IR involves choosing an Azure SQL database or managed instance for the SSIS catalog, configuring connections, deploying SSIS projects, and scheduling packages. Custom setups are also possible by loading external DLLs. Typical data flows in Azure Data Factory are then discussed for lifting and shifting SSIS packages to the cloud.
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...
Accelerating Innovation with Apache Kafka, Heikki Nousiainen | Heikki Nousiai...HostedbyConfluent The document discusses leveraging Apache Kafka for enhancing innovation within organizations, particularly focused on event-driven architecture. It outlines benefits such as faster development cycles, scalability, and improved data management through decoupled architecture. Additionally, it emphasizes the importance of a supportive team environment and strong partnerships with Kafka providers for successful implementation.
The Future of Enterprise Applications is Serverless
The Future of Enterprise Applications is ServerlessEficode The document discusses the evolution and benefits of serverless architecture within Amazon Web Services (AWS), highlighting the shift from monolithic systems to decoupled services managed by small teams. It emphasizes that serverless applications allow for greater agility, reduced overhead, and improved scalability without the need to manage servers. Key features such as AWS Lambda, API Gateway, and their applications in various domains like web development and data processing are also presented.
Nurturing a large GST ecosystem on AWS - Anil Sharma, Chicago
Nurturing a large GST ecosystem on AWS - Anil Sharma, ChicagoAWS Chicago Motherson Sumi is a large global automotive components manufacturer with over 235 facilities in 37 countries. In India, the government implemented a new Goods and Services Tax (GST) system to simplify taxation. Motherson Sumi was selected to develop a system to help businesses compute and file their GST taxes. They built a multi-tenant system on AWS that could scale to millions of users, integrate with other systems, and be quickly updated as tax policies changed. The system included a user portal and APIs to allow tax filing and integration with existing business systems. Motherson Sumi leveraged many AWS services to build this complex system securely, quickly and at scale to help modernize taxation in India.
Webinar Cloud Native Community.pptx
Webinar Cloud Native Community.pptxLibbySchulze This document provides an overview of making applications cloud-native including:
- Defining cloud-native and its impacts on applications
- The advantages of using containers for applications
- When and why to use Kubernetes for container orchestration
- A demonstration of containerizing a sample application
Elasticsearch on Azure
Elasticsearch on AzureElasticsearch The presentation introduces the partnership between Microsoft and Elastic to offer various Elasticsearch services on Azure, highlighting new regions and integrations. It emphasizes the benefits of flexible service offerings, including managed and self-managed options. The document also includes a forward-looking statement regarding risks associated with the services.
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...
NDC London 2021 - Application Autoscaling Made Easy With Kubernetes Event-Dri...Tom Kerkhove Kubernetes Event-driven Autoscaling (KEDA) provides application autoscaling on Kubernetes using a variety of metric sources. It automatically scales deployments, jobs, and other resources. KEDA supports over 30 built-in scalers for sources like Azure, AWS, Google Cloud, and more. It is cloud-agnostic and focuses on scaling applications without managing the scaling internals. The Azure Functions CLI makes it easy to deploy functions to Kubernetes and automatically configure KEDA for autoscaling. KEDA is an open source project with over 2,800 stars on GitHub and contributions from Microsoft, Red Hat, and other companies.
Application Development on Metapod
Application Development on MetapodCisco DevNet The document discusses various approaches to application development on OpenStack including doing it yourself with open source tools (DIY), using configuration management tools to gather automation tools into a management layer (Mantl), and adopting standards and components through Platform as a Service tools (Cloud Foundry). It provides examples of tools for infrastructure as code (Heat), container orchestration (Docker), and describes the basic flow of application deployment on Cloud Foundry including using its open source software plus commercial modules. The goal is to help development teams focus on application delivery rather than underlying OpenStack configuration and management.
Combinação de logs, métricas e rastreamentos para observabilidade unificada
Combinação de logs, métricas e rastreamentos para observabilidade unificadaElasticsearch The document discusses the evolution and capabilities of Elastic's observability solutions, highlighting the importance of logs, metrics, and APM for a unified observability strategy. It emphasizes the complexity of monitoring dynamic cloud-native architectures and the benefits of integrating various data sources for intelligent anomaly detection and unified monitoring. Additionally, it outlines the introduction of various Elastic tools like Metricbeat and Elastic APM for improved performance monitoring and analysis.
Application Autoscaling Made Easy with Kubernetes Event-Driven Autoscaling (K...
Application Autoscaling Made Easy with Kubernetes Event-Driven Autoscaling (K...Codit This document summarizes a presentation about Kubernetes Event-driven Autoscaling (KEDA). KEDA allows applications running on Kubernetes to automatically scale based on external events from services like Azure Event Hubs, Kafka, or Cosmos DB. It provides out-of-the-box and custom scalers to monitor event sources and scale deployments and jobs as needed. KEDA is open source, cloud agnostic, and aims to simplify autoscaling so developers can focus on their applications rather than scaling internals. The presenters demonstrate using KEDA to scale a .NET Core worker based on an Azure Service Bus queue depth.
Similar to Bandwidth: Use Cases for Elastic Cloud on Kubernetes (20)
Achieving cyber mission assurance with near real-time impact
Achieving cyber mission assurance with near real-time impactElasticsearch The document discusses the integration of Elastic and ECS within the Mission Assurance Decision Support System (MADSS) to enhance mission assurance capabilities through real-time data analysis. It outlines the evolution from traditional architectures to a modernized, microservices approach utilizing the Elastic stack for improved data handling, search capabilities, and observability. Key insights include the importance of understanding data dynamics and optimizing pipelines for effective ingestion and monitoring.
Grab: Building a Healthy Elasticsearch Ecosystem
Grab: Building a Healthy Elasticsearch EcosystemElasticsearch The document discusses how Grab uses Elasticsearch (ES) and cloud technologies to address diverse use cases across different teams, highlighting operational challenges like search and monitoring in a complex environment. It emphasizes the importance of customized solutions for various user groups, ensuring access control based on user demographics and roles. Additionally, the evolution of Grab's infrastructure from basic operations to a more integrated and scalable ES solution is outlined, showcasing enhanced performance and management through cloud technologies.
seven-ways-to-run-flink-on-aws.pdf
seven-ways-to-run-flink-on-aws.pdfSergioBruno21 This document provides an overview of Apache Flink and different ways to deploy Flink applications on cloud platforms. It discusses deploying Flink on EMR, EC2, ECS, EKS, Kinesis Data Analytics for Java, and Lambda. It recommends EMR, discusses advantages like managed Hadoop clusters, and disadvantages like needing to manage the cluster. It also provides an overview of Flink architecture, components like the JobManager and TaskManager, and the data flow API.
4 - Customer story: Telenet
4 - Customer story: TelenetKangaroot The document discusses the implementation and use cases of Elasticsearch for troubleshooting, log monitoring, and alerting within IT infrastructure. It outlines the setup process, including hardware migration strategies, data retention policies, and performance metrics for logs. Additionally, it addresses the incorporation of machine learning for improving alert accuracy and anomaly detection.
Nine Publishing: Building a modern infrastructure with the Elastic Stack
Nine Publishing: Building a modern infrastructure with the Elastic StackElasticsearch The document discusses the implementation of the Elastic Stack to enhance infrastructure agility, flexibility, and efficiency. Key takeaways include the complexities of managing a self-hosted Elasticsearch cluster, the importance of automation in maintenance, and the benefits of centralized logging and monitoring. Additionally, it highlights the successful open-sourcing of their own Helm chart and discusses various cases for improved user experience and cost optimization.
Kubecon seattle 2018 recap - Application Deployment aspects
Kubecon seattle 2018 recap - Application Deployment aspectsKrishna-Kumar KubeCon Seattle 2018 featured over 10,000 attendees with numerous breakout sessions, keynote speakers, and significant announcements regarding cloud-native computing technologies and tools. Highlights included the introduction of various Kubernetes-related projects and a focus on diversity and community engagement within the tech sector. The event emphasized the growing complexity of Kubernetes and showcased new solutions like the operator framework, Helm, and the CNAB (Cloud Native Application Bundles) standard.
All about open stack
All about open stackDataCentred The document is a presentation by Nick Jones on OpenStack, covering its definition as a cloud operating system, its project history, key components such as compute (Nova), storage (Cinder), and networking (Neutron), as well as its installation and maintenance tools. It highlights notable users like CERN and Walmart, and outlines the need for automation in deployment through tools like Puppet. Additionally, it addresses various operational challenges encountered in the OpenStack ecosystem.
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets @ Cisco IntercloudStreamsets Inc. The document provides a detailed case study on the implementation of StreamSets Data Collector and Elasticsearch at Cisco Intercloud, highlighting the architecture, data flows, and benefits of the system. It discusses the challenges faced, including container management issues and performance optimization, while showcasing the adaptability and efficiency of the StreamSets pipelines for data processing. Additionally, the document emphasizes the collaborative efforts with the StreamSets team to resolve scalability and operational challenges, showcasing the system's capacity to handle high message throughput effectively.
Case Study: Elasticsearch Ingest Using StreamSets at Cisco Intercloud
Case Study: Elasticsearch Ingest Using StreamSets at Cisco IntercloudRick Bilodeau This document outlines a case study on the implementation of Elasticsearch and StreamSets at Cisco Intercloud, emphasizing the architecture, data flow, and performance management of data streams. It highlights the challenges faced in managing data efficiently, the adaptability of StreamSets for building ingest pipelines with minimal coding, and the integration of microservices within the framework. The study demonstrates the benefits of using the Elastic Stack for analytics and the successful processing of large volumes of data across various environments.
Xpdays: Kubernetes CI-CD Frameworks Case Study
Xpdays: Kubernetes CI-CD Frameworks Case StudyDenys Vasyliev The document outlines the professional background of Denys Vasyliev, highlighting over ten years of experience in the telecom industry and expertise in software engineering, with roles in startups and major telecom companies. It details technical requirements for cloud solutions, Kubernetes metrics, and various tools related to DevOps, infrastructure automation, and service management. The document also mentions specific technologies and platforms like Helm, Istio, and Knative, emphasizing their relevance in modern cloud-native application development.
X by orange; una telco en la nube
X by orange; una telco en la nubeJuan Vicente Herrera Ruiz de Alejo This document discusses how to setup a telco in the cloud using open source technologies. It describes how the company X by Orange uses infrastructure as code practices like Git, Packer, Terraform, and Ansible to provision their cloud infrastructure immutably. They deploy applications as containers using OpenShift and monitor services with Prometheus and Netdata. The goal is to provide flexible online solutions to customers faster than traditional telcos by embracing a cloud native approach.
The Netflix Way to deal with Big Data Problems
The Netflix Way to deal with Big Data ProblemsMonal Daxini The document discusses Netflix's approach to handling big data problems. It summarizes Netflix's data pipeline system called Keystone that was built in a year to replace a legacy system. Keystone ingests over 1 trillion events per day and processes them using technologies like Kafka, Samza and Spark Streaming. The document emphasizes Netflix's culture of freedom and responsibility and how it helped the small team replace the legacy system without disruption while achieving massive scale.
Big data on aws
Big data on awsSerkan Özal The Ankara Cloud Meetup in July 2016 covered Big Data on AWS, including insights into its concepts, storage, analytics, processing, and data flow capabilities. Key AWS services like S3, DynamoDB, Redshift, and EMR were discussed alongside scalability features and practical demonstrations for storing, searching, processing, and analyzing data. The session highlighted AWS’s managed services for big data solutions, showcasing how to effectively leverage them for various use cases.
Kubernetes at NU.nl (Kubernetes meetup 2019-09-05)
Kubernetes at NU.nl (Kubernetes meetup 2019-09-05)Tibo Beijen The document outlines the transition of nu.nl, the first Dutch digital news platform, to using Kubernetes to address several DevOps challenges they face, including slow feedback loops and the need for orchestration in application deployment. It emphasizes the benefits of containerization for application packaging, along with the adoption of Kubernetes for improved scalability, security, and deployment processes. The document also reflects on past incidents due to resource misconfigurations, illustrating the importance of setting limits and automating processes in operational practices.
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP
Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAPKrishna-Kumar The document recaps KubeCon 2019 held in San Diego, showcasing its scale with over 12,000 attendees and numerous announcements from major vendors. It highlights various case studies of companies like Slack, Uber, and Airbnb that have successfully transitioned to Kubernetes, implementing diverse use cases such as multi-tenancy, database migrations, and microservices deployment. Top announcements included the launch of Helm 3 and collaborative projects from AWS and Microsoft, among others.
Streaming Analytics with Spark, Kafka, Cassandra and Akka
Streaming Analytics with Spark, Kafka, Cassandra and AkkaHelena Edelson This document discusses a new approach to building scalable data processing systems using streaming analytics with Spark, Kafka, Cassandra, and Akka. It proposes moving away from architectures like Lambda and ETL that require duplicating data and logic. The new approach leverages Spark Streaming for a unified batch and stream processing runtime, Apache Kafka for scalable messaging, Apache Cassandra for distributed storage, and Akka for building fault tolerant distributed applications. This allows building real-time streaming applications that can join streaming and historical data with simplified architectures that remove the need for duplicating data extraction and loading.
Cloud Native Summit 2019 Summary
Cloud Native Summit 2019 SummaryEverett Toews The document summarizes key topics from the Cloud Native Summit conference, including:
- Distributed tracing and Zipkin, which allows visibility into request paths and troubleshooting of latency issues. Zipkin is an open source distributed tracing system.
- Production ready Kubernetes clusters on Catalyst Cloud, which provides security, high availability, and scalability for containerized applications.
- Building serverless applications at scale using services like AWS Lambda, and addressing concurrency bottlenecks when autoscaling.
- Istio service mesh, which provides control of traffic policies, authentication, and observability across distributed services through its control plane and sidecar proxy architecture.
- GitOps for infrastructure as code deployments on Open
Security Events Logging at Bell with the Elastic Stack
Security Events Logging at Bell with the Elastic StackElasticsearch The document outlines a comprehensive approach to security events logging, emphasizing the need for scalable log management solutions to handle increasing volumes of security data. It discusses the architecture for logging, data storage, and visualization, employing tools like Kafka, Logstash, and Elasticsearch to create a robust, automated pipeline. Additionally, it highlights the importance of secure data handling and smart detection through machine learning and correlation engines.
Kafka Summit SF 2017 - Running Kafka for Maximum Pain
Kafka Summit SF 2017 - Running Kafka for Maximum Painconfluent This document discusses some of the challenges of running Apache Kafka at scale at LinkedIn, including issues with multitenancy, infrastructure, and management. It describes how high volumes of data and many producers can complicate ownership and capacity planning when data is shared. It also explains the pain points of tools like Mirror Maker and the lack of topic configuration management across clusters. Finally, it outlines some of LinkedIn's open source efforts to improve Kafka operations through tools like Cruise Control, Kafka Monitor, and kafka-tools.
Elk stack @inbot
Elk stack @inbotJilles van Gurp Jilles van Gurp discusses logging and monitoring trends and introduces the ELK stack as a solution. The ELK stack consists of Elasticsearch for storage and search, Logstash for transport and processing, and Kibana for visualization. Proper logging is important - log enough but not too much. Logstash is used to ingest logs into Elasticsearch. An Inbot demo shows logging various services and visualizing logs in Kibana. Mapped diagnostic context and application metrics are discussed as ways to add useful context to logs.
Ad
More from Elasticsearch (20)
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch The document discusses advanced relevance ranking techniques for Elasticsearch, focusing on evaluation metrics and methods for improving search performance. Key topics include separating query types for better performance, various metrics like precision at k and discounted cumulative gain, and the use of vector similarity in ranking results. Additionally, it outlines planned improvements in ranking through machine learning and deep learning models for natural language processing and image search.
From MSP to MSSP using Elastic
From MSP to MSSP using ElasticElasticsearch The document outlines Eze Castle's transition from a Managed Service Provider (MSP) to a Managed Security Service Provider (MSSP) using Elastic technology, highlighting the challenges faced with log management and security visibility. Key milestones in their integration journey include partnerships with third-party vendors, onboarding systems, and product relaunches, emphasizing the benefits of using Elastic's cloud and support services. The initiative has led to significant improvements in event ingestion rates, threat feed integrations, and overall customer onboarding times.
Cómo crear excelentes experiencias de búsqueda en sitios web
Cómo crear excelentes experiencias de búsqueda en sitios webElasticsearch El documento describe cómo construir una experiencia de búsqueda efectiva en sitios web utilizando Elastic App Search, que ofrece herramientas y APIs para la ingesta y búsqueda de contenido. También aborda la flexibilidad en el despliegue, la afinación de relevancia y el control de acceso a través de su arquitectura basada en roles. Finalmente, menciona la importancia del rastreador web para optimizar el contenido y mejorar la experiencia del usuario.
Te damos la bienvenida a una nueva forma de realizar búsquedas
Te damos la bienvenida a una nueva forma de realizar búsquedas Elasticsearch 1) The document introduces ElasticON Solution Series, which provides out-of-the-box personalized, centralized, and secure organizational search across internal and external sources.
2) It discusses how Elastic Enterprise Search can improve productivity, satisfaction, collaboration, and decision making by connecting all applications and content with a single scalable search platform.
3) The solution achieves this through intuitive search features, powerful analytics and visualization tools, simplified administration, and security certifications to ensure data protection.
Tirez pleinement parti d'Elastic grâce à Elastic Cloud
Tirez pleinement parti d'Elastic grâce à Elastic CloudElasticsearch The document discusses the adoption of Elastic's cloud solutions, emphasizing its capacity to handle data growth and provide agility. It highlights the challenges organizations face with cloud data strategies, including visibility, control, and security. Additionally, it showcases the benefits of Elastic's managed services and compliance with various standards.
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesElasticsearch The document discusses the capabilities and features of the Elastic platform, emphasizing the transformation of data into insights through tools like Elasticsearch, Kibana, and Logstash. It highlights various use cases such as log analytics, threat detection, and business analytics, along with the importance of runtime fields for dynamic data management. Additionally, it covers the themes of choice, simplicity, and integrations to enhance user experience and data workflows.
Plongez au cœur de la recherche dans tous ses états.
Plongez au cœur de la recherche dans tous ses états.Elasticsearch The document outlines the features and benefits of the Elastic Enterprise Search Solutions, emphasizing the importance of a centralized search experience for employees and teams to enhance productivity and satisfaction. It highlights statistics showing the time and resources lost due to inefficient search processes, and positions Elastic's offerings as a solution to improve organizational search capabilities across various platforms. The document encourages engagement with Elastic's tools through free trials and training resources.
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]
Modernising One Legal Se@rch with Elastic Enterprise Search [Customer Story]Elasticsearch Linklaters has modernized its 'One Legal Search' platform by migrating to Elastic Cloud Enterprise to enhance efficiency and user experience after outgrowing a complex legacy system. The new platform allows for faster queries, better management of multilingual collections, and scalability for future expansions. In partnership with Kin + Carta, this move future-proofs their legal knowledge management system, ensuring reliable access to a comprehensive range of legal documents.
An introduction to Elasticsearch's advanced relevance ranking toolbox
An introduction to Elasticsearch's advanced relevance ranking toolboxElasticsearch This document provides an overview of Elasticsearch's advanced relevance ranking features that enhance search experiences, including tools such as dense vector fields and BM25F. It discusses performance optimization strategies, ranking evaluation metrics, and various query types that improve search result relevance. Additionally, the document outlines how these features are employed by users to create exceptional search experiences.
Welcome to a new state of find
Welcome to a new state of findElasticsearch The document presents the Elastic enterprise search solution, emphasizing its ability to provide a centralized, secure, and personalized search experience across various platforms to enhance productivity and collaboration. It highlights statistics on the time wasted by employees searching for information and the importance of a robust search tool in modern work environments. Additionally, the solution offers features such as intuitive user experience, flexibility in deployment, and machine learning-driven insights to empower organizations in managing their vast amounts of data effectively.
Building great website search experiences
Building great website search experiencesElasticsearch The document outlines the features and benefits of building effective website search experiences using Elastic's App Search and Workplace Search solutions. It emphasizes the core technology of content ingestion, UI/UX tools, and flexible deployment options, including self-managed and cloud deployments. Additionally, it highlights the importance of user analytics, relevance tuning, and customizable web crawler functionalities for enhancing search capabilities.
Keynote: Harnessing the power of Elasticsearch for simplified search
Keynote: Harnessing the power of Elasticsearch for simplified searchElasticsearch The document discusses the features and benefits of Elastic's enterprise search solutions, emphasizing ease-of-use, customizable analytics, and robust security. It outlines advancements in search technology, such as improved typo tolerance and machine learning capabilities, aimed at enhancing search relevance and user experience. Additionally, it highlights new features like the enterprise search web crawler and the integration of various applications to streamline information retrieval across platforms.
Cómo transformar los datos en análisis con los que tomar decisiones
Cómo transformar los datos en análisis con los que tomar decisionesElasticsearch The document discusses the Elastic platform and its solutions, including Elastic Enterprise Search, Elastic Security, and Elastic Observability, highlighting their use cases across various sectors such as log analytics and threat detection. It also emphasizes features like runtime fields, which allow for dynamic data analysis without the need for reindexing, thereby enhancing performance and adaptability. Forward-looking statements regarding the performance and risks associated with these solutions are included, along with details on data management and analytics workflows.
Explore relève les défis Big Data avec Elastic Cloud
Explore relève les défis Big Data avec Elastic Cloud Elasticsearch Expert en data intelligence, Explore propose des solutions de collecte et d'analyse de données pour divers secteurs, incluant immobilier et services. Avec une infrastructure technologique avancée, l'entreprise traite des données provenant de nombreuses sources pour optimiser la performance commerciale. Explore se concentre sur l'analyse des données comportementales et la transformation des moments de vie des clients en leviers de succès.
Comment transformer vos données en informations exploitables
Comment transformer vos données en informations exploitablesElasticsearch The document discusses the Elastic solution series, including the use cases and benefits of Elastic's tools such as Elasticsearch, Kibana, and Logstash for various applications like log analytics, threat detection, and business analytics. It highlights features like runtime fields for dynamic data management and analytics workflows to enhance user experience and reduce downtime. Additionally, it emphasizes themes of choice, simplicity, and integration within the Elastic platform.
Transforming data into actionable insights
Transforming data into actionable insightsElasticsearch The document discusses the Elastic platform, highlighting its components such as Elasticsearch, Kibana, and Logstash, which facilitate data analytics across various use cases like log analytics and security monitoring. It emphasizes the importance of runtime fields for dynamic data handling, enabling users to create fields on-the-fly and adapt to evolving data without reindexing. Additionally, it outlines key features for data management and collaboration in analytics workflows to enhance user experience.
Opening Keynote: Why Elastic?
Opening Keynote: Why Elastic?Elasticsearch The document presents the Elastic Search Platform, emphasizing its capacity to efficiently manage and analyze vast amounts of data while addressing business challenges related to search, security, and observability. It highlights various use cases across industries and mentions accolades from Gartner, recognizing Elastic as a valuable tool for enterprise search and application performance monitoring. Additionally, the document outlines the economic impacts of inefficiencies in data search and the importance of effective data management in modern businesses.
Empowering agencies using Elastic as a Service inside Government
Empowering agencies using Elastic as a Service inside GovernmentElasticsearch The document outlines ECS and Elastic's collaboration in providing cloud and cybersecurity solutions to government agencies. ECS delivers managed services including cloud architecture, cybersecurity solutions, and IT modernization, while Elastic offers a managed Elasticsearch service for scalable deployments. The partnership emphasizes centralized management, security, and automation within government missions.
The opportunities and challenges of data for public good
The opportunities and challenges of data for public goodElasticsearch The document discusses data for public good and the opportunities and challenges involved. It notes that data infrastructure is needed to deliver public good through data. There are almost endless opportunities to use data for public services, policy, and citizen benefits. However, challenges include legacy systems, data silos, unclear governance, and risk aversion. As a case study, it outlines how the UK Census 2021 addressed index faced challenges but showed progress on using data better, with lessons for continued public sector transformation.
Enterprise search and unstructured data with CGI and Elastic
Enterprise search and unstructured data with CGI and ElasticElasticsearch The document presents a case study on an elastic integration framework for the UK government aimed at better managing data risks and compliance. It highlights challenges such as aging systems, technical debt, and the need for operational improvements while proposing solutions like data classification and transformation processes. The approach emphasizes treating data as an asset and utilizing advanced tools for monitoring and remediation to mitigate risks effectively.
Ad
Recently uploaded (20)
Quantum AI: Where Impossible Becomes Probable
Quantum AI: Where Impossible Becomes ProbableSaikat Basu Imagine combining the "brains" of Artificial Intelligence (AI) with the "super muscles" of Quantum Computing. That's Quantum AI!
It's a new field that uses the mind-bending rules of quantum physics to make AI even more powerful.
Raman Bhaumik - Passionate Tech Enthusiast
Raman Bhaumik - Passionate Tech EnthusiastRaman Bhaumik A Junior Software Developer with a flair for innovation, Raman Bhaumik excels in delivering scalable web solutions. With three years of experience and a solid foundation in Java, Python, JavaScript, and SQL, she has streamlined task tracking by 20% and improved application stability.
10 Key Challenges for AI within the EU Data Protection Framework.pdf
10 Key Challenges for AI within the EU Data Protection Framework.pdfPriyanka Aash 10 Key Challenges for AI within the EU Data Protection Framework
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdf
A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy.pdfPriyanka Aash A Constitutional Quagmire - Ethical Minefields of AI, Cyber, and Privacy
The Growing Value and Application of FME & GenAI
The Growing Value and Application of FME & GenAISafe Software With the cost of using Generative AI services dropping exponentially and the array of available models continually expanding, integrating AI into FME workflows has become inexpensive, accessible and effective. This presentation explores how GenAI within FME can cost-effectively transform data workflows by automating data extraction, validation, classification and augmentation tasks. We’ll discuss how FME’s no-code flexibility enables users to combine Generative AI and Computer Vision tools that create efficient workflows tailored to specific challenges. Using recent practical examples, we’ll demonstrate how these integrations can simplify complex tasks, save time and enhance data quality.
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptxFIDO Alliance Securing Account Lifecycles in the Age of Deepfakes
Lessons Learned from Developing Secure AI Workflows.pdf
Lessons Learned from Developing Secure AI Workflows.pdfPriyanka Aash Lessons Learned from Developing Secure AI Workflows
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdfPriyanka Aash Coordinated Disclosure for ML - What's Different and What's the Same
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...Fwdays We will explore how e-commerce projects prepare for the busiest time of the year, which key aspects to focus on, and what to expect. We’ll share our experience in setting up auto-scaling, load balancing, and discuss the loads that Silpo handles, as well as the solutions that help us navigate this season without failures.
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdf
GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now.pdfPriyanka Aash GenAI Opportunities and Challenges - Where 370 Enterprises Are Focusing Now
Smarter Aviation Data Management: Lessons from Swedavia Airports and Sweco
Smarter Aviation Data Management: Lessons from Swedavia Airports and SwecoSafe Software Managing airport and airspace data is no small task, especially when you’re expected to deliver it in AIXM format without spending a fortune on specialized tools. But what if there was a smarter, more affordable way?
Join us for a behind-the-scenes look at how Sweco partnered with Swedavia, the Swedish airport operator, to solve this challenge using FME and Esri.
Learn how they built automated workflows to manage periodic updates, merge airspace data, and support data extracts – all while meeting strict government reporting requirements to the Civil Aviation Administration of Sweden.
Even better? Swedavia built custom services and applications that use the FME Flow REST API to trigger jobs and retrieve results – streamlining tasks like securing the quality of new surveyor data, creating permdelta and baseline representations in the AIS schema, and generating AIXM extracts from their AIS data.
To conclude, FME expert Dean Hintz will walk through a GeoBorders reading workflow and highlight recent enhancements to FME’s AIXM (Aeronautical Information Exchange Model) processing and interpretation capabilities.
Discover how airports like Swedavia are harnessing the power of FME to simplify aviation data management, and how you can too.
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdfMuhammad Yuga Nugraha Explore the ease of managing Firecracker microVM with the firecracker-python. In this session, I will introduce the basics of Firecracker microVM and demonstrate how this custom SDK facilitates microVM operations easily. We will delve into the design and development process behind the SDK, providing a behind-the-scenes look at its creation and features. While traditional Firecracker SDKs were primarily available in Go, this module brings a simplicity of Python to the table.
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FMESafe Software This session will cover a common use case for local and state/provincial governments who create and/or maintain their 9-1-1 addressing data, particularly address points and road centerlines. In this session, you'll learn how FME has helped Shelby County 9-1-1 (TN) automate the 9-1-1 addressing process; including automatically assigning attributes from disparate sources, on-the-fly QAQC of said data, and reporting. The FME logic that this presentation will cover includes: Table joins using attributes and geometry, Looping in custom transformers, Working with lists and Change detection.
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...Fwdays What can go wrong if you allow each service to access the database directly? In a startup, this seems like a quick and easy solution, but as the system scales, problems appear that no one could have guessed.
In my talk, I'll share Solidgate's experience in transforming its architecture: from the chaos of direct connections to a service-based data access model. I will talk about the transition stages, bottlenecks, and how isolation affected infrastructure support. I will honestly show what worked and what didn't. In short, we will analyze the controversy of this talk.
UserCon Belgium: Honey, VMware increased my bill
UserCon Belgium: Honey, VMware increased my billstijn40 VMware’s pricing changes have forced organizations to rethink their datacenter cost management strategies. While FinOps is commonly associated with cloud environments, the FinOps Foundation has recently expanded its framework to include Scopes—and Datacenter is now officially part of the equation. In this session, we’ll map the FinOps Framework to a VMware-based datacenter, focusing on cost visibility, optimization, and automation. You’ll learn how to track costs more effectively, rightsize workloads, optimize licensing, and drive efficiency—all without migrating to the cloud. We’ll also explore how to align IT teams, finance, and leadership around cost-aware decision-making for on-prem environments. If your VMware bill keeps increasing and you need a new approach to cost management, this session is for you!
Connecting Data and Intelligence: The Role of FME in Machine Learning
Connecting Data and Intelligence: The Role of FME in Machine LearningSafe Software In this presentation, we want to explore powerful data integration and preparation for Machine Learning. FME is known for its ability to manipulate and transform geospatial data, connecting diverse data sources into efficient and automated workflows. By integrating FME with Machine Learning techniques, it is possible to transform raw data into valuable insights faster and more accurately, enabling intelligent analysis and data-driven decision making.
AI vs Human Writing: Can You Tell the Difference?
AI vs Human Writing: Can You Tell the Difference?Shashi Sathyanarayana, Ph.D This slide illustrates a side-by-side comparison between human-written, AI-written, and ambiguous content. It highlights subtle cues that help readers assess authenticity, raising essential questions about the future of communication, trust, and thought leadership in the age of generative AI.
WebdriverIO & JavaScript: The Perfect Duo for Web Automation
WebdriverIO & JavaScript: The Perfect Duo for Web Automationdigitaljignect In today’s dynamic digital landscape, ensuring the quality and dependability of web applications is essential. While Selenium has been a longstanding solution for automating browser tasks, the integration of WebdriverIO (WDIO) with Selenium and JavaScript marks a significant advancement in automation testing. WDIO enhances the testing process by offering a robust interface that improves test creation, execution, and management. This amalgamation capitalizes on the strengths of both tools, leveraging Selenium’s broad browser support and WDIO’s modern, efficient approach to test automation. As automation testing becomes increasingly vital for faster development cycles and superior software releases, WDIO emerges as a versatile framework, particularly potent when paired with JavaScript, making it a preferred choice for contemporary testing teams.
Bandwidth: Use Cases for Elastic Cloud on Kubernetes
- 1. 1 Jeff Moore, Systems Engineer, Bandwidth Sam Mingolelli, Systems Engineer, Bandwidth 4/23/20 Use Cases for Elastic Cloud on Kubernetes (And how we got there)
- 2. 2 616 vs. 58
- 3. 3 Presenters… ● Tech stack: Kubernetes, Elastic Stack, AWS ● Focus: Openshift SRE, Elastic SRE ● Fun facts: Certified Kubernetes Administrator with plans to pursue Elastic Certified Engineer exam ● Tech stack: Kubernetes, Elastic Stack, AWS ● Focus: Openshift SRE, metrics & logging, hybrid cloud ● Fun facts: One of 5 moderators on the Stack Exchange site for Unix & Linux, father of 3 kids & 2 cats Jeff Moore is… Sam Mingolelli is…
- 5. 5 O U R S E R V I C E S
- 6. 6 Fun Facts About Bandwidth Incorporated 1999 Headquarters Raleigh, NC Number of Employees 700+ NASDAQ BAND
- 7. 7 Who's the Systems team at Bandwidth ● Systems team is separated into 2 sub teams ○ Infrastructure ○ Platform ● Platform Team manages (6 ppl): ○ Openshift ○ Elastic Stack ○ AWS services ● Infrastructure Team manages (5 ppl): ○ Storage (NetApp) ○ Virtualization (RHV) ○ Most everything else….
- 9. 9 Quick Snapshot of Today
- 11. 11 High Level Logging Needs ● Maintain 10+ Data Centers ● Guarantee integrity of log & metric data ● Perform maintenance of logging infrastructure without loss ● Retain logs & metrics for 30-60 days
- 12. 12 2016
- 13. 13 2016 - BELK + Kafka ● Kafka acts as a “queue” between the shippers & Elasticsearch ● Logstash performs the ETL of moving data from Kafka to Elasticsearch ● Kafka also a “queue” between Data Centers ● Kafka provides data rehydration in the event of data loss (Elasticsearch)
- 14. 14 2016 - BELK + Kafka - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 15. 15 2017
- 16. 16 2017 - OpenShift - take 1 ● OpenShift/k8s is our strategic “platform” play ● Start building out Elastic Stack infrastructure within OpenShift/k8s ● OpenShift/k8s provides: ○ self-service for end users ○ Hybrid cloud across our Data Centers and AWS ○ Standardizes how we scale capacity ● Move from complicated Ansible roles/playbooks to k8s deployments (IaaS)
- 17. 17 2017 - OpenShift - take 1 - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 18. 18 2018
- 19. 19 2018 - OpenShift - take 2 ● Convert monolith Logstash VMs into individualized Logstash per shipper type ● Now just a deployment ● Scaling is easy, bump replicas from X to Y ● Simplified Logstash configurations ● Removed complexity of multiple threads per Logstash, now 1 per CPU core
- 20. 20 2018 - OpenShift - take 2 - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 21. 21 2018 - OpenShift - take 2 - (Example)
- 22. 22 2018 - OpenShift - take 3 ● Take 2 worked so well, move rest of beats into deployments/daemonsets ● Strategically better aligns for OpenShift 4.x and CoreOS ● Everything is a deployment, no daemons local to nodes ● Simplifies architecture
- 23. 23 2018 - OpenShift - take 3 - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 24. 24 2018 - OpenShift - take 3 - (Example) Logstash Deployments Filebeat & Metricbeat Daemonsets
- 25. 25 2018 - Scaling the Consumption of Elastic Stack ● Self service to more efficiently scale our group's ability to support ES (5 ppl) ● Allow our group to not be in critical path of all ES consumption ● Act more as consultant to ES usage
- 26. 26 2018 - Elastic Cloud ● New use case with an unspecified amount of data appeared ● Use case required ML ● Use case was different enough from standard log & metrics ● Spin up Elastic Cloud (AWS) as PoC/PoT ● Self-Service
- 27. 27 2018 - Elastic Cloud - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 29. 29 2019
- 30. 30 What is ECK? (Architecture) DP K8s deployment KB Kibana CRD STS K8s statefulset NS K8s namespace ES Elasticsearch CRD
- 31. 31 2019 - ECK in our Infrastructure - (Architecture) D daemon DP/DS K8s deployment VM Virtual Machine ML Machine Learning
- 32. 32 2019 - Why ECK? ● Additional use cases continued to materialize for Elasticsearch ● Use cases were unique enough to justify their own clusters ● Continue to Strengthen OpenShift “platform” strategy by increasing features ● It fits well into the *aaS goal
- 33. 33 ECK Use Cases ● Replace InfluxDB with ES for APM style metrics ● Network FlowLogs ● Disposable ES for testing & upgrade PoC/PoT ● Self Service ES - (ownership)
- 34. 34 Replace InfluxDB with ES for APM style metrics ● Move away from InfluxDB supported stability at scale ● 12.5k events/sec ingested via logstash ● Mostly Java metrics
- 35. 35 Network FlowLogs ● Network Team needed visibility into their hardware ● Used ElastiFlow plugin ● ECK allows custom container images and init containers
- 36. 36 Disposable ES for testing & upgrade PoC/PoT ● Safe testing environment vs. testing on production clusters ● Accessible with production clusters with cross-cluster search and cross-cluster replication
- 37. 37 Self Service ES - (ownership) ● Allows for all other future use cases ● Strengthens our offerings as a team
- 38. 38 616 vs. 58
- 39. 39 Traditional Deployments vs. ECK - Deployment Ansible ECK Lines of Code 616 58 File count 9 2 Certificate management Manual Automagic Scaling Manual Automagic
- 40. 40 ECK Deployment Example - pt. 1
- 41. 41 ECK Deployment Example - pt. 2
- 42. 42 ECK Challenges... ● Becoming Kubernetes-native is hard ● Becoming Kubernetes-aware is much harder ● Operational benefits outweigh challenges by a very large margin
- 43. 43 2020+ - Future Plans ● Continue “refactoring” components into OpenShift ○ Kibana ○ ES Masters ○ ES Data Nodes? ECK? ● Build Operators within OpenShift to IaaS ES APIs (or maybe Elastic can…?) ○ Watchers ○ RBAC ● ES for… ○ SIEM ○ APM ○ ILM ● Elastic as a data source - Add what you want on top (Kibana, Grafana, etc.)
- 44. 44 How to Reach Us Jeff https://github.com/slmingol https://www.linkedin.com/in/sammingolelli/ Sam https://www.linkedin.com/in/jeff-moore-k8s https://github.com/geoffmore
- 45. 45 We'll answer questions in chat
- 46. 46 Thanks!