Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009
38 likes6,647 views
The document discusses distributed programming using RabbitMQ, emphasizing its role in concurrent computing by dividing tasks among multiple nodes. It covers challenges such as managing worker nodes, message acknowledgment, and control scripts, while promoting RabbitMQ's capabilities for high-performance messaging. Additionally, it highlights the use of Erlang and EventMachine to facilitate efficient, non-blocking operations in a distributed environment.
1 of 62
Downloaded 362 times
Ad
Recommended
%w(map reduce).first - A Tale About Rabbits, Latency, and Slim Crontabs
%w(map reduce).first - A Tale About Rabbits, Latency, and Slim CrontabsPaolo Negri The document discusses the MapReduce programming model for processing large data sets using distributed and concurrent systems, highlighting the use of RabbitMQ for managing queues. It emphasizes the architecture's flexibility and advantages, such as decoupling master/worker roles and support for multiple worker nodes. Additionally, it introduces EventMachine for non-blocking I/O and concurrency, alongside detailing task distribution and messaging systems.
RabbitMQ with python and ruby RuPy 2009
RabbitMQ with python and ruby RuPy 2009Paolo Negri The document discusses using RabbitMQ, an open-source message broker based on AMQP, for asynchronous messaging between Python and Ruby applications. It provides an overview of AMQP concepts like producers, consumers, exchanges and queues, and how RabbitMQ implements these using Erlang. Code examples are shown for sending and receiving messages asynchronously in Python and Ruby.
Improvements in RabbitMQ
Improvements in RabbitMQAlvaro Videla 1) RabbitMQ performance was improved by optimizing queue index flushing to disk, batching message publishes and deliveries, and reducing memory leaks.
2) Mirror queue synchronization was improved from 60 seconds to 6 seconds for 1 million messages by batching message publishes.
3) "Lazy queues" were implemented to reduce memory usage for queues with offline consumers by only loading messages into memory when consumers are present.
RabbitMQ Data Ingestion
RabbitMQ Data IngestionAlvaro Videla - RabbitMQ is an open-source message broker that can ingest data from various protocols and distribute it globally using federation.
- Federation allows replication of exchanges and queues across different RabbitMQ clusters. This can distribute data worldwide but only replicates necessary messages.
- To scale, queues can be sharded across nodes using consistent hashing and naming conventions. Load can also be balanced using federated queues that move messages between clusters as needed.
RabbitMQ Data Ingestion at Craft Conf
RabbitMQ Data Ingestion at Craft ConfAlvaro Videla This document discusses building a distributed data ingestion system using RabbitMQ. It begins with an introduction to RabbitMQ and its key features like being multi-protocol, open source, polyglot and written in Erlang. It then discusses the problem of distributing data across multiple servers. RabbitMQ federation is proposed as a solution, allowing replication across servers. Federation uses queues and exchanges to replicate data and can be configured using parameters and policies. The document also discusses scaling the system using sharded queues and federated queues to load balance consumers.
Scaling applications with RabbitMQ at SunshinePHP
Scaling applications with RabbitMQ at SunshinePHPAlvaro Videla This document discusses using RabbitMQ for scaling applications, highlighting its features as a multi-protocol messaging server that is open-source and polyglot. It includes examples of RabbitMQ use cases, code snippets for both publishing and consuming messages, and emphasizes its capability for handling requirements like message persistence and fault tolerance. The document serves as an exploratory talk rather than a step-by-step guide, encouraging innovative use of RabbitMQ in various scenarios.
RabbitMQ Model and Some Example Applications
RabbitMQ Model and Some Example ApplicationsHoucheng Lin RabbitMQ is used in several example applications and real case studies. Three example codes demonstrate: 1) the RabbitMQ tutorial using request/reply queues, 2) RPC using correlated messages, and 3) Celery's distributed task queue using polling. Case studies show RabbitMQ for: 1) message passing between Zenoss daemons, 2) Nova controller RPC, 3) Instagram's task queue with Celery, 4) MercadoLibre's service bus architecture, and 5) Indeed.com's distributed crawling. RabbitMQ provides reliability, scalability, and asynchronous messaging between distributed applications and microservices.
RabbitMQ
RabbitMQLenz Gschwendtner The document introduces RabbitMQ, an open source message broker that implements the Advanced Message Queuing Protocol (AMQP). It discusses why AMQP is an open industry standard that is not language dependent and supported by many major companies. It then provides an overview of messaging concepts like queues, exchanges, routing and pub/sub using RabbitMQ examples. It also mentions some advanced features of AMQP like authentication, load balancing and persistent/non-persistent messages. Finally, it provides information on how to get started with RabbitMQ.
Alvaro Videla, Building a Distributed Data Ingestion System with RabbitMQ
Alvaro Videla, Building a Distributed Data Ingestion System with RabbitMQTanya Denisyuk This document contains a summary of a presentation about using RabbitMQ for messaging. It discusses RabbitMQ's features like multi-protocol support, polyglot clients, and large users. It then describes challenges like distributing an application across servers. RabbitMQ solutions to these include federation to replicate across servers, sharding queues for scaling, and federating queues for load balancing consumers. The presentation argues RabbitMQ provides tools to ingest, distribute, and scale messaging applications globally.
Scaling application with RabbitMQ
Scaling application with RabbitMQNahidul Kibria The document discusses RabbitMQ, a widely used messaging broker, emphasizing its features, use cases, and implementation details. It covers topics such as scaling, clustering, high availability, and integration with various programming languages and tools. The document also highlights practical examples, installation steps, and performance considerations, illustrating RabbitMQ's suitability for enterprise messaging needs.
Dissecting the rabbit: RabbitMQ Internal Architecture
Dissecting the rabbit: RabbitMQ Internal ArchitectureAlvaro Videla The document presents an in-depth analysis of RabbitMQ's internal architecture and functionalities, including its messaging system, process management, and extensibility features. It explains concepts like message storage, supervision trees, and various RabbitMQ behaviors, as well as providing examples of Erlang code related to its operation. Additionally, it discusses practical implementations and configurations, highlighting RabbitMQ's adaptability for various user needs within different programming environments.
RabbitMQ Operations
RabbitMQ OperationsMichael Klishin The document provides insights and best practices for operating RabbitMQ, focusing on aspects such as provisioning, OS resources tuning, memory and disk usage, and cluster formation. It highlights the importance of configuring server settings for optimal performance, managing memory effectively, and handling network partitions. Additionally, it discusses upcoming features in RabbitMQ versions 3.6 and later, including pluggable cluster formation and improved management tools.
High powered messaging with RabbitMQ
High powered messaging with RabbitMQJames Carr Messaging can be used to integrate different systems and applications to work together by transferring data packets asynchronously using customizable formats. It allows for decoupling of publishers and subscribers. RabbitMQ is an open source message broker that implements the AMQP standard for high performance messaging. It can be used from Java applications using the RabbitMQ client library, Apache Camel AMQP component, or Spring AMQP framework.
Построение распределенной системы сбора данных с помощью RabbitMQ, Alvaro Vid...
Построение распределенной системы сбора данных с помощью RabbitMQ, Alvaro Vid...Ontico This document discusses building a distributed data ingestion system using RabbitMQ. It introduces RabbitMQ as a multi-protocol, polyglot messaging broker. The document then outlines some issues with a naïve ad-hoc solution to distributing data and proposes using RabbitMQ federation to address these issues. It provides an overview of how RabbitMQ federation works and how to configure it. Finally, it discusses additional RabbitMQ features like sharded queues and federated queues that can help scale the system.
Spring RabbitMQ
Spring RabbitMQMartin Toshev This document provides an overview of Spring RabbitMQ. It discusses messaging basics and RabbitMQ concepts like exchanges, queues, bindings. It then summarizes the Spring AMQP and Spring Integration frameworks for integrating RabbitMQ in Spring applications. Spring AMQP provides the RabbitAdmin, listener container and RabbitTemplate for declaring and interacting with RabbitMQ components. The document contains code examples for configuring RabbitMQ and consuming/producing messages using Spring AMQP.
Troubleshooting RabbitMQ and services that use it
Troubleshooting RabbitMQ and services that use itMichael Klishin This document discusses troubleshooting RabbitMQ and services that use it. It provides tips for troubleshooting publishers and consumers, including common issues like I/O exceptions, inadequate delivery quality of service, and lack of confirmations. It also recommends collecting logs, metrics, exceptions and traces system-wide to analyze issues. Distributed system debugging remains a challenging problem.
Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)
Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)James Titcumb The document provides a comprehensive guide on using RabbitMQ for practical message queueing, detailing installation, configuration, and various usage scenarios including producer-consumer setups. It explains different types of exchanges such as fanout, direct, and topic exchanges, as well as advanced concepts like clustering and high availability. The guide also presents real-world examples and solutions to common challenges, making it suitable for developers seeking to implement effective background processing solutions.
RabbitMQ fairly-indepth
RabbitMQ fairly-indepthWee Keat Chin The document provides a comprehensive overview of RabbitMQ, detailing its history, technical architecture, and components such as exchanges, queues, and the AMQP model. It also discusses advanced features like clustering, high availability through mirrored queues, and administrative tools for management. Key use cases and applications of RabbitMQ are highlighted, showcasing its role in various major projects.
Messaging Standards and Systems - AMQP & RabbitMQ
Messaging Standards and Systems - AMQP & RabbitMQAll Things Open The document discusses the Advanced Message Queuing Protocol (AMQP) and RabbitMQ, detailing their architecture, features, and usage. It highlights the evolution of AMQP, its various standards, and implementations, along with the benefits of using RabbitMQ for message queuing and system integration. Additionally, it covers common terms, performance considerations, and configurations relevant to RabbitMQ deployment.
Integrating PostgreSql with RabbitMQ
Integrating PostgreSql with RabbitMQGavin Roy This document discusses integrating RabbitMQ with PostgreSQL by using RabbitMQ as a messaging system to publish notifications from PostgreSQL. It describes pg_amqp, a PostgreSQL extension that allows publishing messages via AMQP from user-defined functions. It also describes a PostgreSQL Listen Exchange in RabbitMQ that issues LISTEN commands and publishes received notifications to queues based on routing keys matching the notification channel.
The RabbitMQ Message Broker
The RabbitMQ Message BrokerMartin Toshev RabbitMQ is an open source message broker that implements the AMQP protocol. It provides various messaging patterns using different exchange types and supports clustering for scalability and high availability. Administration of RabbitMQ includes managing queues, exchanges, bindings and other components. Integrations exist for protocols like STOMP, MQTT and frameworks like Spring, while security features include authentication, authorization, and SSL/TLS encryption.
PostgreSQL: meet your queue
PostgreSQL: meet your queueTheo Schlossnagle This document summarizes a presentation about using PostgreSQL with message queues. It discusses how PostgreSQL can integrate with other systems by using the pg_amqp extension to publish messages to an AMQP broker on transaction commit. This allows actions like cache purges or search index updates to happen automatically when related database records change. While the current implementation uses AMQP 0.8, future support for AMQP 1.0 would add two-phase commit to make the process safer.
The details of CI/CD environment for Ruby
The details of CI/CD environment for RubyHiroshi SHIBATA The document discusses the strategy for building and testing the programming language Hiroshi. It covers:
1. The Ruby Core team which maintains the language and includes over 80 volunteers.
2. The testing strategy for Ruby which involves testing at different levels from the interpreter to libraries. Extensive tests are run on Linux, Windows and macOS.
3. The CI environments used for Ruby development including GitHub Actions, Travis CI and AppVeyor, as well as internal VM clusters. Test results are collected on Ruby CI and discussed on Slack.
Introduction to AMQP Messaging with RabbitMQ
Introduction to AMQP Messaging with RabbitMQDmitriy Samovskiy This document introduces AMQP messaging using RabbitMQ as a broker. It explains that AMQP and RabbitMQ allow applications to communicate asynchronously by sending and receiving messages through a broker, providing decoupling, queueing, load balancing and scalability. It provides details on RabbitMQ as an open source AMQP broker developed by Rabbit Technologies and the AMQP Working Group which maintains the AMQP standard.
Cooking a rabbit pie
Cooking a rabbit pieTomas Doran This document discusses using AMQP and RabbitMQ for real-time system performance monitoring and job queueing. It describes key AMQP concepts like exchanges, queues, routing keys, and delivery modes. It then discusses how the speaker built a job queueing system called CatalystX::JobServer that uses these concepts and RabbitMQ. Demo examples are shown of using this system to queue and monitor the status of jobs. Potential next steps and other solutions like Gearman are also mentioned.
How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?Hiroshi SHIBATA The document discusses the future of the Ruby ecosystem, particularly the development of standard libraries and the introduction of gemification in Ruby 3.0 and subsequent versions. Key points include the categorization of standard libraries, the migration of certain gems to bundled status, and ongoing issues with Rubygems and default gems. It concludes with a call for community involvement in the development process and future plans for Ruby 3.1 and 4.0.
On Rabbits and Elephants
On Rabbits and ElephantsGavin Roy The document discusses using RabbitMQ and pg_amqp to distribute data from PostgreSQL databases. It describes setting up RabbitMQ and pg_amqp to publish data from triggers on tables in a PostgreSQL database. When data is inserted, updated, or deleted in a table, the trigger will publish a message on RabbitMQ. A Python script acts as a consumer that connects to RabbitMQ and another PostgreSQL database. It listens for messages, determines the SQL to execute based on the event type, and runs the SQL on the other database to keep the data in sync across the databases. The document provides an example using pgbench for the data and demonstrates publishing and consuming messages.
Ruby 1.9 Fibers
Ruby 1.9 FibersKevin Ball The document discusses Ruby 1.9 fibers, emphasizing their role in cooperative multi-processing and event-driven programming. It compares processes, threads, and fibers, highlighting the lightweight nature of fibers and their ability to simplify asynchronous code, while also presenting examples of event-driven programming. The document concludes by noting the advantages of event-based architectures and the growing popularity of fibers in Ruby applications.
EventMachine
EventMachineJonathan Weiss EventMachine is an event-driven I/O library for Ruby that aims for scalability and performance while simplifying low-level I/O programming. It supports various protocols and provides higher-level APIs for services like AMQP and Redis. The document also compares EventMachine to Node.js, highlighting their differences in blocking behavior and coding styles.
Mongrel2, a short introduction
Mongrel2, a short introductionPaolo Negri Mongrel2 is a language-agnostic web server that focuses on HTTP protocols and can be used with various programming languages. Its configuration is managed through a SQLite3 database and allows for dynamic management of server processes and handlers using the m2sh command-line tool. The server facilitates asynchronous request handling through ØMQ sockets, significantly enhancing performance and scalability.
More Related Content
What's hot (19)
Alvaro Videla, Building a Distributed Data Ingestion System with RabbitMQ
Alvaro Videla, Building a Distributed Data Ingestion System with RabbitMQTanya Denisyuk This document contains a summary of a presentation about using RabbitMQ for messaging. It discusses RabbitMQ's features like multi-protocol support, polyglot clients, and large users. It then describes challenges like distributing an application across servers. RabbitMQ solutions to these include federation to replicate across servers, sharding queues for scaling, and federating queues for load balancing consumers. The presentation argues RabbitMQ provides tools to ingest, distribute, and scale messaging applications globally.
Scaling application with RabbitMQ
Scaling application with RabbitMQNahidul Kibria The document discusses RabbitMQ, a widely used messaging broker, emphasizing its features, use cases, and implementation details. It covers topics such as scaling, clustering, high availability, and integration with various programming languages and tools. The document also highlights practical examples, installation steps, and performance considerations, illustrating RabbitMQ's suitability for enterprise messaging needs.
Dissecting the rabbit: RabbitMQ Internal Architecture
Dissecting the rabbit: RabbitMQ Internal ArchitectureAlvaro Videla The document presents an in-depth analysis of RabbitMQ's internal architecture and functionalities, including its messaging system, process management, and extensibility features. It explains concepts like message storage, supervision trees, and various RabbitMQ behaviors, as well as providing examples of Erlang code related to its operation. Additionally, it discusses practical implementations and configurations, highlighting RabbitMQ's adaptability for various user needs within different programming environments.
RabbitMQ Operations
RabbitMQ OperationsMichael Klishin The document provides insights and best practices for operating RabbitMQ, focusing on aspects such as provisioning, OS resources tuning, memory and disk usage, and cluster formation. It highlights the importance of configuring server settings for optimal performance, managing memory effectively, and handling network partitions. Additionally, it discusses upcoming features in RabbitMQ versions 3.6 and later, including pluggable cluster formation and improved management tools.
High powered messaging with RabbitMQ
High powered messaging with RabbitMQJames Carr Messaging can be used to integrate different systems and applications to work together by transferring data packets asynchronously using customizable formats. It allows for decoupling of publishers and subscribers. RabbitMQ is an open source message broker that implements the AMQP standard for high performance messaging. It can be used from Java applications using the RabbitMQ client library, Apache Camel AMQP component, or Spring AMQP framework.
Построение распределенной системы сбора данных с помощью RabbitMQ, Alvaro Vid...
Построение распределенной системы сбора данных с помощью RabbitMQ, Alvaro Vid...Ontico This document discusses building a distributed data ingestion system using RabbitMQ. It introduces RabbitMQ as a multi-protocol, polyglot messaging broker. The document then outlines some issues with a naïve ad-hoc solution to distributing data and proposes using RabbitMQ federation to address these issues. It provides an overview of how RabbitMQ federation works and how to configure it. Finally, it discusses additional RabbitMQ features like sharded queues and federated queues that can help scale the system.
Spring RabbitMQ
Spring RabbitMQMartin Toshev This document provides an overview of Spring RabbitMQ. It discusses messaging basics and RabbitMQ concepts like exchanges, queues, bindings. It then summarizes the Spring AMQP and Spring Integration frameworks for integrating RabbitMQ in Spring applications. Spring AMQP provides the RabbitAdmin, listener container and RabbitTemplate for declaring and interacting with RabbitMQ components. The document contains code examples for configuring RabbitMQ and consuming/producing messages using Spring AMQP.
Troubleshooting RabbitMQ and services that use it
Troubleshooting RabbitMQ and services that use itMichael Klishin This document discusses troubleshooting RabbitMQ and services that use it. It provides tips for troubleshooting publishers and consumers, including common issues like I/O exceptions, inadequate delivery quality of service, and lack of confirmations. It also recommends collecting logs, metrics, exceptions and traces system-wide to analyze issues. Distributed system debugging remains a challenging problem.
Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)
Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)James Titcumb The document provides a comprehensive guide on using RabbitMQ for practical message queueing, detailing installation, configuration, and various usage scenarios including producer-consumer setups. It explains different types of exchanges such as fanout, direct, and topic exchanges, as well as advanced concepts like clustering and high availability. The guide also presents real-world examples and solutions to common challenges, making it suitable for developers seeking to implement effective background processing solutions.
RabbitMQ fairly-indepth
RabbitMQ fairly-indepthWee Keat Chin The document provides a comprehensive overview of RabbitMQ, detailing its history, technical architecture, and components such as exchanges, queues, and the AMQP model. It also discusses advanced features like clustering, high availability through mirrored queues, and administrative tools for management. Key use cases and applications of RabbitMQ are highlighted, showcasing its role in various major projects.
Messaging Standards and Systems - AMQP & RabbitMQ
Messaging Standards and Systems - AMQP & RabbitMQAll Things Open The document discusses the Advanced Message Queuing Protocol (AMQP) and RabbitMQ, detailing their architecture, features, and usage. It highlights the evolution of AMQP, its various standards, and implementations, along with the benefits of using RabbitMQ for message queuing and system integration. Additionally, it covers common terms, performance considerations, and configurations relevant to RabbitMQ deployment.
Integrating PostgreSql with RabbitMQ
Integrating PostgreSql with RabbitMQGavin Roy This document discusses integrating RabbitMQ with PostgreSQL by using RabbitMQ as a messaging system to publish notifications from PostgreSQL. It describes pg_amqp, a PostgreSQL extension that allows publishing messages via AMQP from user-defined functions. It also describes a PostgreSQL Listen Exchange in RabbitMQ that issues LISTEN commands and publishes received notifications to queues based on routing keys matching the notification channel.
The RabbitMQ Message Broker
The RabbitMQ Message BrokerMartin Toshev RabbitMQ is an open source message broker that implements the AMQP protocol. It provides various messaging patterns using different exchange types and supports clustering for scalability and high availability. Administration of RabbitMQ includes managing queues, exchanges, bindings and other components. Integrations exist for protocols like STOMP, MQTT and frameworks like Spring, while security features include authentication, authorization, and SSL/TLS encryption.
PostgreSQL: meet your queue
PostgreSQL: meet your queueTheo Schlossnagle This document summarizes a presentation about using PostgreSQL with message queues. It discusses how PostgreSQL can integrate with other systems by using the pg_amqp extension to publish messages to an AMQP broker on transaction commit. This allows actions like cache purges or search index updates to happen automatically when related database records change. While the current implementation uses AMQP 0.8, future support for AMQP 1.0 would add two-phase commit to make the process safer.
The details of CI/CD environment for Ruby
The details of CI/CD environment for RubyHiroshi SHIBATA The document discusses the strategy for building and testing the programming language Hiroshi. It covers:
1. The Ruby Core team which maintains the language and includes over 80 volunteers.
2. The testing strategy for Ruby which involves testing at different levels from the interpreter to libraries. Extensive tests are run on Linux, Windows and macOS.
3. The CI environments used for Ruby development including GitHub Actions, Travis CI and AppVeyor, as well as internal VM clusters. Test results are collected on Ruby CI and discussed on Slack.
Introduction to AMQP Messaging with RabbitMQ
Introduction to AMQP Messaging with RabbitMQDmitriy Samovskiy This document introduces AMQP messaging using RabbitMQ as a broker. It explains that AMQP and RabbitMQ allow applications to communicate asynchronously by sending and receiving messages through a broker, providing decoupling, queueing, load balancing and scalability. It provides details on RabbitMQ as an open source AMQP broker developed by Rabbit Technologies and the AMQP Working Group which maintains the AMQP standard.
Cooking a rabbit pie
Cooking a rabbit pieTomas Doran This document discusses using AMQP and RabbitMQ for real-time system performance monitoring and job queueing. It describes key AMQP concepts like exchanges, queues, routing keys, and delivery modes. It then discusses how the speaker built a job queueing system called CatalystX::JobServer that uses these concepts and RabbitMQ. Demo examples are shown of using this system to queue and monitor the status of jobs. Potential next steps and other solutions like Gearman are also mentioned.
How to develop the Standard Libraries of Ruby?
How to develop the Standard Libraries of Ruby?Hiroshi SHIBATA The document discusses the future of the Ruby ecosystem, particularly the development of standard libraries and the introduction of gemification in Ruby 3.0 and subsequent versions. Key points include the categorization of standard libraries, the migration of certain gems to bundled status, and ongoing issues with Rubygems and default gems. It concludes with a call for community involvement in the development process and future plans for Ruby 3.1 and 4.0.
On Rabbits and Elephants
On Rabbits and ElephantsGavin Roy The document discusses using RabbitMQ and pg_amqp to distribute data from PostgreSQL databases. It describes setting up RabbitMQ and pg_amqp to publish data from triggers on tables in a PostgreSQL database. When data is inserted, updated, or deleted in a table, the trigger will publish a message on RabbitMQ. A Python script acts as a consumer that connects to RabbitMQ and another PostgreSQL database. It listens for messages, determines the SQL to execute based on the event type, and runs the SQL on the other database to keep the data in sync across the databases. The document provides an example using pgbench for the data and demonstrates publishing and consuming messages.
Viewers also liked (20)
Ruby 1.9 Fibers
Ruby 1.9 FibersKevin Ball The document discusses Ruby 1.9 fibers, emphasizing their role in cooperative multi-processing and event-driven programming. It compares processes, threads, and fibers, highlighting the lightweight nature of fibers and their ability to simplify asynchronous code, while also presenting examples of event-driven programming. The document concludes by noting the advantages of event-based architectures and the growing popularity of fibers in Ruby applications.
EventMachine
EventMachineJonathan Weiss EventMachine is an event-driven I/O library for Ruby that aims for scalability and performance while simplifying low-level I/O programming. It supports various protocols and provides higher-level APIs for services like AMQP and Redis. The document also compares EventMachine to Node.js, highlighting their differences in blocking behavior and coding styles.
Mongrel2, a short introduction
Mongrel2, a short introductionPaolo Negri Mongrel2 is a language-agnostic web server that focuses on HTTP protocols and can be used with various programming languages. Its configuration is managed through a SQLite3 database and allows for dynamic management of server processes and handlers using the m2sh command-line tool. The server facilitates asynchronous request handling through ØMQ sockets, significantly enhancing performance and scalability.
A Documentation Crash Course, LinuxCon 2016
A Documentation Crash Course, LinuxCon 2016Chris Ward The document presents a guide on effective documentation for developers, highlighting key questions to consider such as audience, purpose, and clarity. It emphasizes the importance of engaging language, active voice, and concise writing, drawing on notable quotes to stress brevity. Additionally, the document mentions tools for managing formatting, testing, and automating documentation processes.
Why you should come to DrupalSouth
Why you should come to DrupalSouthChris Ward Drupal South is a conference taking place February 14-16, 2014 in New Zealand that will feature around 40 sessions across training, business, and Birds of a Feather topics, as well as code sprints and social activities. The venue is at a museum and will have liquid refreshments available. While a few speakers had to cancel, their spots were filled, and attendees can expect the event to be an awesome experience as Drupal conferences always are.
Content Management Systems and Refactoring - Drupal, WordPress and eZ Publish
Content Management Systems and Refactoring - Drupal, WordPress and eZ PublishJani Tarvainen The document discusses the evolution of content management systems (CMS) with a focus on open source PHP options and the Nordic region’s CMS landscape. It highlights the shift towards central Europe in CMS momentum, the commoditization of content management, and the ongoing challenges of legacy systems, particularly with ez Publish. Refactoring efforts aimed at integrating modern frameworks like Symfony are outlined, emphasizing both improvements and challenges in developer experiences and customer satisfaction.
Entrez dans le mouvement Maker à l’aide des technologies Microsoft
Entrez dans le mouvement Maker à l’aide des technologies MicrosoftFabrice BARBIN Le document présente le mouvement maker et son intégration avec les technologies Microsoft, en mettant en avant des outils tels que l'impression 3D, le prototypage électronique avec Windows 10 IoT, et les services d'hébergement sur Azure. Il souligne la professionnalisation du mouvement maker et les interactions possibles grâce à des technologies comme Kinect. Enfin, des ressources et des contacts pour en savoir plus sur ces technologies et la communauté des makers sont fournis.
Electron - Solving our cross platform dreams?
Electron - Solving our cross platform dreams?Chris Ward The document discusses Electron, a framework developed by GitHub that allows developers to create cross-platform desktop applications using JavaScript. It provides installation instructions, a simple application structure, and sample code for setting up an Electron project. The document also touches on the challenges of using Electron, such as application size and performance concerns.
Scaling Social Games
Scaling Social GamesPaolo Negri The document discusses the challenges of scaling social games to millions of daily active users. It describes how the company scaled from 170,000 daily users to over 1,000,000 by:
1) Moving data and queries from MySQL databases to Redis to improve performance and handle higher volumes.
2) Sharding high-volume tables and migrating data to distribute load across multiple database servers.
3) Using load balancers and adding application servers to scale the architecture horizontally as more users were added.
SimpleDb, an introduction
SimpleDb, an introductionPaolo Negri The document provides an overview of Amazon SimpleDB, a NoSQL database designed for scalability and efficiency, allowing users to store and retrieve vast amounts of structured data in a schema-less manner. It highlights key features such as support for up to 1 billion attributes per domain and the use of a key-value pair structure, along with the associated costs for storage and data transfer. Additionally, it discusses the advantages of using SimpleDB, including easy indexing and distribution, while also noting some limitations such as eventual consistency and the requirement for string-only data types.
Offre développeur Javascript Back-end
Offre développeur Javascript Back-endSite Analyzer VSI Innovation, une start-up fondée en 2015, recherche un développeur back-end JavaScript pour travailler sur des applications innovantes et contribuer au développement de leur service d'analyse de sites web, Site Analyzer. La société, située dans un éco-système high-tech à Valenciennes, compte des clients dans plus de 60 pays et a établi un partenariat avec Google USA. Le candidat idéal doit maîtriser JavaScript et PHP, être passionné par le web, et posséder une forte motivation.
Contentful Berlin Offices
Contentful Berlin OfficesIrina Botea We're coffee aficionados and craft beer lovers. We play ping pong and kicker. We're music buffs, library rats or movie geeks. We work with passion and we believe in our product.
We are the Contentful team!
Automate your docs, automate yourself
Automate your docs, automate yourselfChris Ward This document discusses various techniques for automating and improving technical documentation processes, including spelling checking with Hunspell and markdown-spellcheck, writing better with write-good and LanguageTool, adding screenshots with Robot Framework and Selenium, testing with APIs like Dredd and Sphinx, using Pandoc for multiformat output, and placeholders with sed. It encourages automating documentation work and testing to ensure documentation quality and usability.
Erlang introduction geek2geek Berlin
Erlang introduction geek2geek BerlinPaolo Negri Erlang is a programming language well-suited for building reliable and concurrent systems. It uses lightweight processes that communicate asynchronously via message passing, allowing for easy development of concurrent and distributed applications. The key concepts in Erlang are processes and message passing - processes send and receive messages to communicate, and are isolated, fast to create and terminate. The language itself is functional with strict evaluation, single assignment and dynamic typing.
The Anatomy of Content Management (workshop by J Gollner at Intelligent Conte...
The Anatomy of Content Management (workshop by J Gollner at Intelligent Conte...Joe Gollner This document provides an overview of content management. It discusses core concepts like content, information, publishing, and intelligent content. It presents a content lifecycle model and analyzes the key components of acquisition, strategy, delivery, management, and evolution. It also maps the landscape of content solution technologies, categorizing tools for creation, analysis, publishing, asset management, and engagement. Overall, the document defines the anatomy of content management and its supporting concepts, models, and technologies.
AWS Lambda in infrastructure
AWS Lambda in infrastructurePaolo Negri AWS Lambda allows you to run code in response to events, extend AWS services with custom logic, and build serverless backends that operate at scale. With AWS Lambda, you write code, upload it to AWS Lambda, bind it to events from services like S3, DynamoDB, or SNS, and it will run automatically in response. At Contentful, AWS Lambda is used to automatically invalidate CloudFront caches when files in S3 buckets are updated, and to render templates when configuration files are updated in S3. These use cases require only 70 lines of code. AWS Lambda represents a shift from running programs in the cloud to programming the cloud.
Le futur de Drupal et des applications web
Le futur de Drupal et des applications webJulien Dubreuil Le document discute de l'évolution du web et des frameworks, notamment l'émergence de Drupal et des micro-frameworks, en soulignant leur impact sur le développement d'applications. Il aborde les transitions vers des architectures de microservices et l'intégration d'APIs dans des projets multi-langages. Enfin, il s'interroge sur l'avenir des applications web traditionnelles et la tendance vers la componentisation et les objets connectés.
Devoxx France 2015 - Se préparer à l'arrivée d'Angular 2
Devoxx France 2015 - Se préparer à l'arrivée d'Angular 2Romain Linsolas Le document présente une synthèse des évolutions d'Angular, notamment la transition vers Angular 2 avec une refonte complète des concepts précédents. Il met en avant l'importance d'apprendre TypeScript, de maîtriser les directives, et de suivre les actualités liées aux mises à jour d'Angular. Enfin, il propose des alternatives à Angular et des conseils pour ceux qui choisissent de rester sur la version 1.
Back to the future with static site generators
Back to the future with static site generatorsChris Ward The document discusses static site generators in the context of a presentation from Dev Week 2016 in London. It outlines the benefits and drawbacks of using static site generators and provides examples of various generators across different programming languages, highlighting their installation and configuration processes. The presentation concludes with insights into extending these generators and their possible applications.
ParisJS #10 : PhantomJs
ParisJS #10 : PhantomJs Maurice Svay Le document présente des informations sur PhantomJS, un outil de test et de capture d'écran pour les applications web, en détaillant ses fonctionnalités et API. Il mentionne également des technologies associées telles que TileMill et Routino, ainsi que des liens vers des démonstrations et des ressources. Enfin, il explore des aspects techniques comme le rendu et le traitement de données JSON pour des projets web.
Ad
Similar to Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009 (20)
Message Queues in Ruby - An Overview
Message Queues in Ruby - An OverviewPradeep Elankumaran This document provides an overview of message queueing in Ruby. It discusses several popular message queue options for Ruby including Delayed Job, Starling, ActiveMQ, RabbitMQ, and Resque. For each option, it provides a brief description and notes on usage. It emphasizes that the most important aspects are choosing the right queue for the job and load testing worker daemons. A new experimental project called Llama is also mentioned, which is a port of Apache Camel to Ruby using EventMachine.
Real time system_performance_mon
Real time system_performance_monTomas Doran This document discusses using message queues and AMQP for real-time system performance monitoring and job queuing. It describes how message queues allow producers and consumers to communicate asynchronously. The author details their experience building a job server using RabbitMQ, Catalyst, and Web::Hippie to queue and monitor long-running jobs. While the system is still a work in progress, all the major components are there and it is in use in production.
Scaling Ruby with Evented I/O - Ruby underground
Scaling Ruby with Evented I/O - Ruby undergroundOmer Gazit The document discusses scaling strategies in Ruby with evented I/O using EventMachine, highlighting non-blocking I/O operations, the reactor design pattern, and performance comparisons with Node.js. It includes code examples for asynchronous HTTP requests, RabbitMQ processing, and error handling, demonstrating how to efficiently manage concurrent tasks without blocking the event loop. The conclusion emphasizes that while evented I/O can effectively scale applications and offers advantages like speed and elegance, it may not suit every project due to limitations in debugging and community support.
The Art of Message Queues - TEKX
The Art of Message Queues - TEKXMike Willbanks The document discusses message queues and their uses. Message queues allow for asynchronous communication between applications and components. They decouple systems, allow for background processing, and improve scalability. Common use cases for message queues include processing email notifications, auto-scaling cloud applications, handling image/video processing, and interacting with services like Apple Push Notifications.
High scale flavour
High scale flavourTomas Doran This document discusses various approaches for implementing message queueing in web applications. It describes using a message queue to decouple producers and consumers, improving load handling and allowing more pages to be served quicker. Common solutions discussed include using Gearman, STOMP, and AMQP. Gearman is recommended for simple job queueing, while STOMP provides a simple tried and tested jobs solution. AMQP offers more flexibility but has fewer proven Perl implementations.
Introduction to Python Celery
Introduction to Python CeleryMahendra M Celery is a Python-based distributed task queue that uses message queues like AMQP to asynchronously execute tasks across multiple machines. It provides features like task scheduling, retries, error handling, and integration with Django. Tasks are defined as Python functions decorated with the "@task" decorator. Workers process tasks from the queue and results are sent back via messages. Celery simplifies building asynchronous workflows in Python applications.
Concurrency in ruby
Concurrency in rubyMarco Borromeo This document summarizes Marco Borromeo's presentation on concurrency in Ruby. It discusses various approaches to achieving concurrency like multiple processes/forking, threads, and fibers. However, it notes that the global interpreter lock (GIL) prevents true concurrency in Ruby with threads and fibers. EventMachine and the reactor pattern provide an asynchronous approach using blocking I/O. EM-Syncrony and Sinatra::Synchrony build on this to provide a synchronous style of programming with concurrency.
Presentation
Presentationzladuric This document discusses using RabbitMQ and message queues to process events from connected devices in a microservices architecture. RabbitMQ can be used to implement common messaging patterns like publish/subscribe and push/pull to distribute processing of device data across multiple services. It provides examples of using RabbitMQ exchanges, queues, and bindings to route messages based on topics and load balance work. Error handling is also addressed through retry queues and dedicated error services.
Lindsay distributed geventzmq
Lindsay distributed geventzmqRobin Xiao ZeroMQ provides a simple way to build distributed systems without relying on heavyweight message brokers. It uses a messaging model where messages are atomic and can be routed between applications. ZeroMQ avoids many of the complexities of traditional message queues and makes it easy to implement common distributed patterns like request-reply, publish-subscribe, and routing in a variety of programming languages using simple socket-like APIs.
Life in a Queue - Using Message Queue with django
Life in a Queue - Using Message Queue with djangoTareque Hossain This document provides a comprehensive overview of message queues, highlighting their role in decoupling sender and receiver processes within web applications. It discusses the functionality, scenarios for use, implementation with RabbitMQ and Celery in Python, and practical coding examples for task management. Additionally, it covers best practices for configuration and task routing to enhance application performance.
Microservices communication styles and event bus
Microservices communication styles and event busTouraj Ebrahimi The document discusses communication styles and technologies related to event sourcing and CQRS, focusing on messaging protocols and popular message brokers like RabbitMQ and Kafka. It provides insights into the features, performance, and comparison of various message brokers, highlighting their strengths and weaknesses. Additionally, it outlines principles of asynchronous communication, the role of event buses, and includes examples and trends within this domain.
Evented applications with RabbitMQ and CakePHP
Evented applications with RabbitMQ and CakePHPmarkstory The document discusses creating event-driven applications using RabbitMQ, Python, and CakePHP, focusing on offloading tasks for real-time capabilities outside the typical request/response cycle. It emphasizes the limitations of cron jobs for scalable applications and introduces queues as a solution for managing distributed workloads. The document also covers queue basics, producer/consumer patterns, and different tools and technologies to facilitate message processing and application scaling.
Message Queuing on a Large Scale: IMVUs stateful real-time message queue for ...
Message Queuing on a Large Scale: IMVUs stateful real-time message queue for ...Jon Watte The presentation discusses the challenges and solutions involved in low-latency game messaging and state distribution at IMVU, emphasizing the implementation of a messaging system using Erlang to handle large-scale requirements for up to 1,000,000 simultaneous users. Key topics include the design goals, different types of messaging (events vs. state), and the architecture for scaling across machines using consistent hashing. Future improvements and potential plans for open-sourcing elements of the system are also mentioned.
Queue Everything and Please Everyone
Queue Everything and Please EveryoneVaidik Kapoor Vaidik Kapoor presented on using message queues and task queues to improve the responsiveness of web applications. He discussed how queues can be used to offload processing tasks from the main request-response cycle to improve user experience. Some key points included:
- Message queues like RabbitMQ and Redis with libraries like HotQueue and PyRes can be used to asynchronously process tasks like data/media processing and cache updates.
- Celery is a popular task queue library that uses RabbitMQ and supports many frameworks. It allows distributing and managing cron-like tasks reliably.
- Proper integration of task processing with user interfaces is important using techniques like notifications, flashes, and polling to update users.
Messaging with amqp and rabbitmq
Messaging with amqp and rabbitmqSelasie Hanson This document discusses AMQP and RabbitMQ for messaging. AMQP is a networking protocol that enables client applications to communicate with messaging middleware brokers. Brokers receive messages from publishers and route them to consumers. RabbitMQ is an open source message broker that implements AMQP. It discusses using RabbitMQ with various languages like Ruby via libraries like Bunny. It provides examples of broker types, exchanges and queues.
RabbitMQ And Nanite
RabbitMQ And Nanitemattmatt RabbitMQ is an open source message broker that implements the AMQP protocol for facilitating asynchronous messaging. Nanite is a self-assembling system of Ruby daemons that uses RabbitMQ for messaging and is composed of mappers, agents, and actors to distribute tasks. Agents in Nanite register themselves with mappers and advertise their services to process messages. While Nanite provides a scalable solution, its documentation is limited and test suite is not robust.
Webinar: Queues with RabbitMQ - Lorna Mitchell
Webinar: Queues with RabbitMQ - Lorna MitchellCodemotion The document provides an overview of using RabbitMQ for managing queues in applications to handle asynchronous tasks, enabling scalability and loose coupling of system components. It discusses key RabbitMQ concepts such as brokers, exchanges, queues, and message processing, alongside example applications like webhooks. Various configuration options and practices for effective message handling, including handling retries and dead letter queues, are also highlighted.
The Future of Messaging: RabbitMQ and AMQP
The Future of Messaging: RabbitMQ and AMQP Eberhard Wolff RabbitMQ and AMQP provide a standardized messaging protocol that improves scalability and reliability over traditional RPC approaches. The document discusses the benefits of messaging including decoupling and asynchronous communication. It provides an overview of core AMQP concepts like exchanges, queues, bindings and routing. Spring AMQP and Spring's RabbitTemplate simplify working with RabbitMQ in Java by providing a common API and configuration support.
Servers with Event Machine - David Troy - RailsConf 2011
Servers with Event Machine - David Troy - RailsConf 2011David Troy EventMachine allows Ruby programs to handle many concurrent connections without blocking or threads. It uses an asynchronous model where callbacks are invoked in response to I/O events like incoming connections or data. EventMachine provides a high performance non-blocking TCP server that handles all I/O with callbacks rather than blocking. It can be used to build scalable I/O intensive applications like email servers that handle thousands of concurrent connections in a single Ruby process. The key is to never block the EventMachine reactor loop.
Ruby Microservices with RabbitMQ
Ruby Microservices with RabbitMQZoran Majstorovic Microservices with Zoran Majstorović discusses moving from monolithic applications to microservices architectures using RabbitMQ/AMQP for messaging. It covers refactoring Ruby code using callbacks to use a consumer microservice. It also demonstrates a simple way to deploy a microservice in a Docker container to AWS ECS. Key topics include decoupling services, different integration options like messaging, AMQP concepts, and using RabbitMQ with Ruby.
Ad
Recently uploaded (20)
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdf
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdfbiswajitbanerjee38 Russia is one of the most aggressive nations when it comes to state coordinated cyberattacks — and Ukraine has been at the center of their crosshairs for 3 years. This report, provided the State Service of Special Communications and Information Protection of Ukraine contains an incredible amount of cybersecurity insights, showcasing the coordinated aggressive cyberwarfare campaigns of Russia against Ukraine.
It brings to the forefront that understanding your adversary, especially an aggressive nation state, is important for cyber defense. Knowing their motivations, capabilities, and tactics becomes an advantage when allocating resources for maximum impact.
Intelligence shows Russia is on a cyber rampage, leveraging FSB, SVR, and GRU resources to professionally target Ukraine’s critical infrastructures, military, and international diplomacy support efforts.
The number of total incidents against Ukraine, originating from Russia, has steadily increased from 1350 in 2021 to 4315 in 2024, but the number of actual critical incidents has been managed down from a high of 1048 in 2022 to a mere 59 in 2024 — showcasing how the rapid detection and response to cyberattacks has been impacted by Ukraine’s improved cyber resilience.
Even against a much larger adversary, Ukraine is showcasing outstanding cybersecurity, enabled by strong strategies and sound tactics. There are lessons to learn for any enterprise that could potentially be targeted by aggressive nation states.
Definitely worth the read!
Lessons Learned from Developing Secure AI Workflows.pdf
Lessons Learned from Developing Secure AI Workflows.pdfPriyanka Aash Lessons Learned from Developing Secure AI Workflows
Creating Inclusive Digital Learning with AI: A Smarter, Fairer Future
Creating Inclusive Digital Learning with AI: A Smarter, Fairer FutureImpelsys Inc. Have you ever struggled to read a tiny label on a medicine box or tried to navigate a confusing website? Now imagine if every learning experience felt that way—every single day.
For millions of people living with disabilities, poorly designed content isn’t just frustrating. It’s a barrier to growth. Inclusive learning is about fixing that. And today, AI is helping us build digital learning that’s smarter, kinder, and accessible to everyone.
Accessible learning increases engagement, retention, performance, and inclusivity for everyone. Inclusive design is simply better design.
AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc AI VIDEO MAGAZINE - r/aivideo community newsletter – Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Enhance GitHub Copilot using MCP - Enterprise version.pdfNilesh Gule Slide deck related to the GitHub Copilot Bootcamp in Melbourne on 17 June 2025
FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce.pptx
FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce.pptxFIDO Alliance FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce
"Database isolation: how we deal with hundreds of direct connections to the d...
"Database isolation: how we deal with hundreds of direct connections to the d...Fwdays What can go wrong if you allow each service to access the database directly? In a startup, this seems like a quick and easy solution, but as the system scales, problems appear that no one could have guessed.
In my talk, I'll share Solidgate's experience in transforming its architecture: from the chaos of direct connections to a service-based data access model. I will talk about the transition stages, bottlenecks, and how isolation affected infrastructure support. I will honestly show what worked and what didn't. In short, we will analyze the controversy of this talk.
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft developmentshyamraj55 CurieTech AI in Action – Accelerate MuleSoft Development
Overview:
This presentation demonstrates how CurieTech AI’s purpose-built agents empower MuleSoft developers to create integration workflows faster, more accurately, and with less manual effort
linkedin.com
+12
curietech.ai
+12
meetups.mulesoft.com
+12
.
Key Highlights:
Dedicated AI agents for every stage: Coding, Testing (MUnit), Documentation, Code Review, and Migration
curietech.ai
+7
curietech.ai
+7
medium.com
+7
DataWeave automation: Generate mappings from tables or samples—95%+ complete within minutes
linkedin.com
+7
curietech.ai
+7
medium.com
+7
Integration flow generation: Auto-create Mule flows based on specifications—speeds up boilerplate development
curietech.ai
+1
medium.com
+1
Efficient code reviews: Gain intelligent feedback on flows, patterns, and error handling
youtube.com
+8
curietech.ai
+8
curietech.ai
+8
Test & documentation automation: Auto-generate MUnit test cases, sample data, and detailed docs from code
curietech.ai
+5
curietech.ai
+5
medium.com
+5
Why Now?
Achieve 10× productivity gains, slashing development time from hours to minutes
curietech.ai
+3
curietech.ai
+3
medium.com
+3
Maintain high accuracy with code quality matching or exceeding manual efforts
curietech.ai
+2
curietech.ai
+2
curietech.ai
+2
Ideal for developers, architects, and teams wanting to scale MuleSoft projects with AI efficiency
Conclusion:
CurieTech AI transforms MuleSoft development into an AI-accelerated workflow—letting you focus on innovation, not repetition.
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to productionMichele Leroux Bustamante We live in an ever evolving landscape for cyber threats creating security risk for your production systems. Mitigating these risks requires participation throughout all stages from development through production delivery - and by every role including architects, developers QA and DevOps engineers, product owners and leadership. No one is excused! This session will cover examples of common mistakes or missed opportunities that can lead to vulnerabilities in production - and ways to do better throughout the development lifecycle.
MuleSoft for AgentForce : Topic Center and API Catalog
MuleSoft for AgentForce : Topic Center and API Catalogshyamraj55 This presentation dives into how MuleSoft empowers AgentForce with organized API discovery and streamlined integration using Topic Center and the API Catalog. Learn how these tools help structure APIs around business needs, improve reusability, and simplify collaboration across teams. Ideal for developers, architects, and business stakeholders looking to build a connected and scalable API ecosystem within AgentForce.
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdf
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdfICT Frame Magazine Pvt. Ltd. Artificial Intelligence (AI) is rapidly changing the face of cybersecurity across the globe. In Nepal, the shift is already underway. Vice President of the Information Security Response Team Nepal (npCERT) and Information Security Consultant at One Cover Pvt. Ltd., Sudan Jha, recently presented an in-depth workshop on how AI can strengthen national security and digital defenses.
The Future of Technology: 2025-2125 by Saikat Basu.pdf
The Future of Technology: 2025-2125 by Saikat Basu.pdfSaikat Basu A peek into the next 100 years of technology. From Generative AI to Global AI networks to Martian Colonisation to Interstellar exploration to Industrial Nanotechnology to Artificial Consciousness, this is a journey you don't want to miss. Which ones excite you the most? Which ones are you apprehensive about? Feel free to comment! Let the conversation begin!
Using the SQLExecutor for Data Quality Management: aka One man's love for the...
Using the SQLExecutor for Data Quality Management: aka One man's love for the...Safe Software The SQLExecutor is one of FME’s most powerful and flexible transformers. Pivvot maintains a robust internal metadata hierarchy used to support ingestion and curation of thousands of external data sources that must be managed for quality before entering our platform. By using the SQLExecutor, Pivvot can efficiently detect problems and perform analysis before data is extracted from our staging environment, removing the need for rollbacks or cycles waisted on a failed job. This presentation will walk through three distinct examples of how Pivvot uses the SQLExecutor to engage its metadata hierarchy and integrate with its Data Quality Management workflows efficiently and within the source postgres database. Spatial Validation –Validating spatial prerequisites before entering a production environment. Reference Data Validation - Dynamically validate domain-ed columns across any table and multiple columns per table. Practical De-duplication - Removing identical or near-identical well point locations from two distinct source datasets in the same table.
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptxFIDO Alliance FIDO Seminar: New Data: Passkey Adoption in the Workforce
Improving Data Integrity: Synchronization between EAM and ArcGIS Utility Netw...
Improving Data Integrity: Synchronization between EAM and ArcGIS Utility Netw...Safe Software Utilities and water companies play a key role in the creation of clean drinking water. The creation and maintenance of clean drinking water is becoming a critical problem due to pollution and pressure on the environment. A lot of data is necessary to create clean drinking water. For fieldworkers, two types of data are key: Asset data in an asset management system (EAM for example) and Geographic data in a GIS (ArcGIS Utility Network ). Keeping this type of data up to date and in sync is a challenge for many organizations, leading to duplicating data and creating a bulk of extra attributes and data to keep everything in sync. Using FME, it is possible to synchronize Enterprise Asset Management (EAM) data with the ArcGIS Utility Network in real time. Changes (creation, modification, deletion) in ArcGIS Pro are relayed to EAM via FME, and vice versa. This ensures continuous synchronization of both systems without daily bulk updates, minimizes risks, and seamlessly integrates with ArcGIS Utility Network services. This presentation focuses on the use of FME at a Dutch water company, to create a sync between the asset management and GIS.
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019 A presentation at Internetware 2025.
Cluster-Based Multi-Objective Metamorphic Test Case Pair Selection for Deep N...
Cluster-Based Multi-Objective Metamorphic Test Case Pair Selection for Deep N...janeliewang985 the slides of the MP selection approach CMPS
Connecting Data and Intelligence: The Role of FME in Machine Learning
Connecting Data and Intelligence: The Role of FME in Machine LearningSafe Software In this presentation, we want to explore powerful data integration and preparation for Machine Learning. FME is known for its ability to manipulate and transform geospatial data, connecting diverse data sources into efficient and automated workflows. By integrating FME with Machine Learning techniques, it is possible to transform raw data into valuable insights faster and more accurately, enabling intelligent analysis and data-driven decision making.
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to productionMichele Leroux Bustamante
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdf
Information Security Response Team Nepal_npCERT_Vice_President_Sudan_Jha.pdfICT Frame Magazine Pvt. Ltd.
Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009
- 1. “Divide and conquer riding rabbits and trading gems” - a tale about distributed programming - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter Paolo Negri @hungryblank
- 3. About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
- 4. Summary:
- 7. Control
- 8. The problem Given a dictionary of 1.000.000 search phrases, compare for each one the first page of results on google.com and bing.com
- 9. Don’t do it. It’s a violation of terms of service! (Sorry, I needed an example that required little or no explanation)
- 10. Comparison
- 12. Great! but... • Fetching, parsing and analyzing 2.000.000 pages of results will take a long time • we want to collect the statistics in a limited time and this sequential approach is not quick enough
- 13. Distributed Computing Is a method of solving computational problem by dividing the problem into many tasks run simultaneously on many hardware or software systems (Wikipedia)
- 14. Map Reduce "Map" step the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)
- 15. Problems: • How many nodes? • How many workers? • Distribution mechanism to feed the workers?
- 16. What about queuing? • the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue
- 17. Here comes • RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging • It’s opensource • Written in Erlang
- 18. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
- 19. Erlang - is coming back • Projects • CouchDB - RESTful document storage • WebMachine - REST toolkit • Nitrogen - web framework • Mochiweb - web framework • Ruby shops using it • 37 Signal - Campfire • Heroku
- 20. + Erlang It’s messages all the way down
- 21. Install it • sudo apt-get install rabbitmq • sudo gem install tmm1-amqp Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4 to follow the code in the slides
- 22. Do it! - master node
- 23. Do it! - worker node
- 24. Get for free • Decoupling master/worker • Workers take care of feeding themselves • Flexible number of workers
- 25. Behind the scenes Worker1 Master msg A Exchange Queue Worker2 Worker3
- 26. Behind the scenes Worker1 Master Exchange Queue Worker2 msg A Worker3
- 27. Behind the scenes Worker1 Master Exchange Queue msg A Worker2 Worker3
- 28. Behind the scenes msg A Worker1 Master Exchange Queue Worker2 Worker3
- 29. RabbitMQ • Multiple exchanges (and multiple types of exchange) • Multiple queues • Queues are connected by bindings to exchanges • Exchanges route messages to queues
- 30. RabbitMQ • Exchanges and queues have names • BUT direct exchanges created implicitly are not public and don’t have name • Queues and messages are resident in RAM and can be persisted on disk (at a performance price)
- 31. What and where Master (ruby) RabbitMQ (Erlang) Worker TCP/IP (ruby) Worker (ruby) Exchange Queue Worker (ruby)
- 32. Problem #1 If a worker has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
- 33. Solution - ACK in worker
- 34. Acknoledgement • messages - 1 or more depending by prefetch settings - are passed to the client • the client processes and acknowledges the messages one by one • if the client connection closes => unacknowledged messages go back in the queue
- 35. http://www.flickr.com/photos/streetfly_jz/2770586821/ Problem #2 No convenient way to control the workers
- 36. System queue - worker
- 37. System queue - control • save it as system_command.rb • ruby system_command.rb halt
- 38. System queue Queue1 Worker1 Control script msg A Exchange Queue2 Worker2 Queue3 Worker3
- 39. System queue Queue1 Worker1 Control script Exchange Queue2 Worker2 msg A Queue3 Worker3
- 40. System queue msg A Queue1 Worker1 Control script Exchange msg A Queue2 Worker2 msg A Queue3 Worker3
- 41. EventMachine
- 42. EventMachine Is an implementation of Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
- 43. without EM with EM Ruby process Ruby process Time code code network network operation Free operation use network use network operation operation Callback result result Free Free
- 44. EventMachine amqp gem is built on top of EventMachine => you’re in a context where you can leverage concurrent programming
- 45. EM - Deferrables “The Deferrable pattern allows you to specify any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “
- 46. EM - Deferrables
- 47. EM - Deferrables
- 48. EM - Deferrables
- 49. EM - Deferrables
- 50. Deferrables without deferrables with deferrables GooglePage GooglePage BingPage BingPage Time
- 51. http://www.flickr.com/photos/philocrites/341850461/ Problem #3 How many of them? what are they doing?
- 54. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor Worker3 msg B
- 55. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor msg B Worker3
- 56. Heartbeat queue Worker1 Worker2 Exchange Queue msg B msg A Monitor Worker3
- 57. Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
- 58. async vs sync Syncronous clients on github • famoseagle/carrot • celldee/bunny
- 59. Rabbits on github • danielsdeleo/moqueue Tests/stubbing • Check out ezmobius/nanite “self assembling fabric of ruby daemons” • Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST) • auser/alice web app to monitor RabbitMQ
- 60. More rabbits on github • tmm1/em-spec • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin
- 61. Q&A ?
- 62. Thanks! (Thanks Mark!) Paolo Negri / hungryblank