JRuby with Java Code in Data Processing World
4 likes7,487 views
JRuby allows developers to write plugins for data processing systems like Norikra and Embulk in Ruby while taking advantage of Java libraries and the JVM. Norikra is a stream processing system that allows SQL queries over data streams. It is written in JRuby and uses the Java Esper library. Embulk is an open-source ETL tool that loads data between databases and file formats using plugins. Both systems use a plugin architecture where plugins can be written in JRuby or Java and are distributed as Ruby gems. This allows for a pluggable ecosystem that benefits from Ruby's productivity while utilizing Java libraries and the JVM's performance.
1 of 38
Downloaded 13 times
![InputPlugin
module Embulk
class InputExample < InputPlugin
Plugin.register_input('example', self)
def self.transaction(config, &control)
# read config
task = {
'message' =>
config.param('message', :string, default: nil)
}
threads = config.param('threads', :int, default:
2)
columns = [
Column.new(0, 'col0', :long),
Column.new(1, 'col1', :double),
Column.new(2, 'col2', :string),
]
# BEGIN here
commit_reports = yield(task, columns, threads)
# COMMIT here
puts "Example input finished"
return {}
end
def run(task, schema, index, page_builder)
puts "Example input thread #{@index}…"
10.times do |i|
@page_builder.add([i, 10.0, "example"])
end
@page_builder.finish
commit_report = { }
return commit_report
end
end
end](https://image.slidesharecdn.com/jrubyconfeu20150731-150731095955-lva1-app6892/85/JRuby-with-Java-Code-in-Data-Processing-World-32-320.jpg)
![OutputPlugin
module Embulk
class OutputExample < OutputPlugin
Plugin.register_output('example', self)
def self.transaction(
config, schema,
processor_count, &control)
# read config
task = {
'message' =>
config.param('message', :string, default: "record")
}
puts "Example output started."
commit_reports = yield(task)
puts "Example output finished. Commit
reports = #{commit_reports.to_json}"
return {}
end
def initialize(task, schema, index)
puts "Example output thread #{index}..."
super
@message = task.prop('message', :string)
@records = 0
end
def add(page)
page.each do |record|
hash = Hash[schema.names.zip(record)]
puts "#{@message}: #{hash.to_json}"
@records += 1
end
end
def finish
end
def abort
end
def commit
commit_report = {
"records" => @records
}
return commit_report
end
end
end](https://image.slidesharecdn.com/jrubyconfeu20150731-150731095955-lva1-app6892/85/JRuby-with-Java-Code-in-Data-Processing-World-33-320.jpg)
Ad
Recommended
Fluentd Overview, Now and Then
Fluentd Overview, Now and ThenSATOSHI TAGOMORI Fluentd is an open source data collector that allows flexible data collection, processing, and output. It supports streaming data from sources like logs and metrics to destinations like databases, search engines, and object stores. Fluentd's plugin-based architecture allows it to support a wide variety of use cases. Recent versions of Fluentd have added features like improved plugin APIs, nanosecond time resolution, and Windows support to make it more suitable for containerized environments and low-latency applications.
Fluentd at HKOScon
Fluentd at HKOSconN Masahiro Fluentd is a data collector for unified logging that provides a robust core and plugins. It allows for reliable data transfer through error handling and retries. The core handles common concerns like parsing, buffering, and writing data, while plugins handle input, output, and other use cases. Fluentd has a pluggable architecture and processes data through a pipeline of input, parser, filter, buffer, formatter, and output plugins.
Fluentd v0.14 Plugin API Details
Fluentd v0.14 Plugin API DetailsSATOSHI TAGOMORI The document summarizes the new plugin API in Fluentd v0.14. Key points include:
- The v0.12 plugin API was fragmented and difficult to write tests for. The v0.14 API provides a unified architecture.
- The main plugin classes are Input, Filter, Output, Buffer, and plugins must subclass Fluent::Plugin::Base.
- The Output plugin supports both buffered and non-buffered processing. Buffering can be configured by tags, time, or custom fields.
- "Owned" plugins like Buffer are instantiated by primary plugins and can access owner resources. Storage is a new owned plugin for persistent storage.
- New test drivers emulate plugin
Dive into Fluentd plugin v0.12
Dive into Fluentd plugin v0.12N Masahiro The document provides an overview of Fluentd and its plugin architecture, detailing input, filter, output, and buffer mechanisms. It includes examples of configuring Fluentd with plugins, particularly fluent-plugin-mongo, and discusses visualization tools, testing methods, and third-party plugins. The guide highlights the use of Ruby for development, the importance of message serialization, and best practices for plugin integration and data handling.
Fluentd meetup in japan
Fluentd meetup in japanTreasure Data, Inc. Fluentd is an open source data collector designed for log aggregation and processing. It has a pluggable architecture that allows for flexible input, filtering, and output of log data. It is useful for log collection, processing, and forwarding to storage or analysis services in real-time.
Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellN Masahiro This document summarizes the key features and changes between versions of Fluentd, an open source data collector.
The main points are:
1) Fluentd v1.0 will provide stable APIs and features while remaining compatible with v0.12 and v0.14. It will have no breaking API changes.
2) New features in v0.14 and v1.0 include nanosecond time resolution, multi-core processing, Windows support, improved buffering and plugins, and more.
3) The goals for v1.0 include migrating more plugins to the new APIs, addressing issues, and improving documentation. A release is planned for Q2 2017.
Fluentd - Set Up Once, Collect More
Fluentd - Set Up Once, Collect MoreSadayuki Furuhashi This document summarizes Sadayuki Furuhashi's background and open source projects, and provides an overview of Fluentd. Fluentd is an open source data collection tool that allows filtering, buffering, and routing logs and event data to various outputs such as databases, cloud services, and analysis systems. It has a simple core with plugins that provide extensibility and features like high availability, load balancing, and more.
Fluentd vs. Logstash for OpenStack Log Management
Fluentd vs. Logstash for OpenStack Log ManagementNTT Communications Technology Development The document compares two popular log collectors, Fluentd and Logstash, focusing on their features, configurations, performance, and transport protocols. Fluentd, written in cruby, uses tagged logs for routing, while Logstash, written in jruby, aggregates logs without tagging. The document also discusses integration with OpenStack, log handling, and some unique features of each log collector such as Fluentd's support for active-active load balancing.
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersSATOSHI TAGOMORI The document discusses the complexities of logging in environments utilizing microservices and containers, highlighting challenges such as the lack of permanent storage and fixed addresses. It presents various aggregation patterns for logging, including source-side and destination-side aggregation, as well as practices using tools like Fluentd and Elasticsearch for efficient log management. It emphasizes best practices for scalable logging that ensures service stability and business growth.
Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellN Masahiro This document summarizes Fluentd v1.0 and provides details about its new features and release plan. It notes that Fluentd v1.0 will provide stable APIs and compatibility with previous versions while improving plugin APIs, adding Windows and multicore support, and increasing event time resolution to nanoseconds. The release is planned for Q3 2017 to allow feedback on v0.14 before finalizing v1.0 features.
Docker and Fluentd
Docker and FluentdN Masahiro - The document discusses logging for containers using Fluentd, an open source data collector. It describes how Fluentd can provide a unified logging layer, reliably forwarding and aggregating logs from multiple containers and applications in a pluggable way.
- Key points covered include using Fluentd with the new Docker logging drivers to directly collect logs from containers, avoiding performance penalties from other approaches. A demo of Fluentd is also mentioned.
The basics of fluentd
The basics of fluentdTreasure Data, Inc. Fluentd is an open source data collector that allows flexible data collection, processing, and storage. It collects log data from various sources using input plugins and sends the data to various outputs like files, databases or forward to other Fluentd servers. It uses a pluggable architecture so new input/output plugins can be added through Ruby gems. It provides features like buffering, retries and reliability.
Fluentd 101
Fluentd 101SATOSHI TAGOMORI Fluentd is an open source data collector that provides a unified logging layer between data sources and backend systems. It decouples these systems by collecting and processing logs and events in a flexible and scalable way. Fluentd uses plugins and buffers to make data collection reliable even in the case of errors or failures. It can forward data between Fluentd nodes for high availability and load balancing.
Docker.io
Docker.ioLadislav Prskavec The document is a comprehensive guide on using Docker, an open-source platform for packaging applications in lightweight containers. It covers installation, command usage, Dockerfile best practices, and the lifecycle of containers and images, emphasizing efficient image creation and management. Additionally, it provides links to resources and projects that utilize Docker for further exploration.
Fluentd introduction at ipros
Fluentd introduction at iprosTreasure Data, Inc. Masahiro Nakagawa from Treasure Data gave a presentation on Fluentd, an open source log collector. Fluentd allows for reliable and structured logging, forwarding, and processing of data through its pluggable architecture. It can collect logs from various sources and output to different destinations using plugins. Common uses of Fluentd include log aggregation, monitoring, and analysis on large-scale architectures.
Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalkN Masahiro Fluentd v1 provides major improvements over v0.12 including nanosecond event time resolution, multi-core support, Windows support, and new plugin APIs. The new plugin APIs provide well-controlled lifecycles and integrate all output plugins. v1 also introduces a server engine based supervisor, dynamic buffering capabilities, and various plugin helpers. While maintaining compatibility with v0.12 plugins, v1 focuses on ease of use, stability, performance and flexibility.
The basics of fluentd
The basics of fluentdTreasure Data, Inc. This document discusses Fluentd, an open source log collector. It provides a pluggable architecture that allows data to be collected, filtered, and forwarded to various outputs. Fluentd uses JSON format for log messages and MessagePack internally. It is reliable, scalable, and extensible through plugins. Common use cases include log aggregation, monitoring, and analytics across multiple servers and applications.
Distributed Stream Processing on Fluentd / #fluentd
Distributed Stream Processing on Fluentd / #fluentdSATOSHI TAGOMORI This document discusses NHN Japan's use of Fluentd for log collection, conversion, and analysis. It summarizes their architecture which involves collecting logs from servers using Scribeline, delivering them to Fluentd processes, converting the logs to a structured format using Fluentd plugins, writing the structured data to HDFS, and performing analysis and reporting with Hive. Key aspects discussed include the scaling of Fluentd processes, performance improvements over their previous Hadoop Streaming-based approach, and implementation details of plugins used.
Async and Non-blocking IO w/ JRuby
Async and Non-blocking IO w/ JRubyJoe Kutner The document discusses asynchronous and non-blocking I/O with JRuby. It explains that asynchronous operations are better than synchronous operations because they use fewer resources and allow for parallelism. It provides an example of building a JRuby application with the Ratpack framework that makes asynchronous HTTP requests to eBay's API in a non-blocking way using promises. It also discusses using RxJava and Hystrix with Ratpack to build a book management application that handles data and API requests asynchronously.
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldSATOSHI TAGOMORI This document describes a presentation about introducing black magic programming patterns in Ruby and their pragmatic uses. It provides an overview of Fluentd, including what it is, its versions, and the changes between versions 0.12 and 0.14. Specifically, it discusses how the plugin API was updated in version 0.14 to address problems with the version 0.12 API. It also explains how a compatibility layer was implemented to allow most existing 0.12 plugins to work without modification in 0.14.
Fluentd meetup #2
Fluentd meetup #2Treasure Data, Inc. The document discusses the logging system Fluentd. It describes how Fluentd provides extensibility through plugins, uses a unified JSON log format, and offers simplicity. Major companies like NHN Japan and COOKPAD use Fluentd in production. The future of Fluentd includes new features like filters in sources, labels, faster MessagePack serialization, and a more efficient process model with live restart capabilities.
Fluentd unified logging layer
Fluentd unified logging layerKiyoto Tamura This document discusses Fluentd, an open source data collector. It provides an overview of Fluentd's architecture and components including input plugins, parser plugins, buffer plugins, output plugins, and formatter plugins. It also outlines Fluentd's roadmap, including plans to add filtering capabilities and improve the plugin API. Examples are given throughout to illustrate how Fluentd works and can be configured for use cases like log collection.
Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Yuta Iwama Yuta Iwama completed an internship where they added several new features and enhancements to Fluentd. Some of the key additions included implementing a counter API, adding data compression to buffers and forwards, creating a new simpler output plugin for secondary sections, developing a CLI tool to read dumped log data, and optimizing multiple filter calls. The internship provided valuable experience contributing to an open source middleware project and learning about aspects of design, development, and code quality.
Fluentd meetup
Fluentd meetupSadayuki Furuhashi This document provides an overview of Fluentd, an open source data collector for structured logging. Fluentd uses a pluggable architecture and JSON format for log messages, allowing logs to be filtered, buffered, and reliably forwarded to storage. It provides client libraries for integrating with applications in languages like Ruby, Perl, PHP, Python and Java. Fluentd is positioned as an alternative to other log collection systems like Scribe and Flume, with advantages of being easier to install, configure, extend with plugins, and smaller footprint.
On Centralizing Logs
On Centralizing LogsSematext Group, Inc. The document discusses the centralization of logs using tools like Elasticsearch, Logstash, and Kibana, emphasizing stability, performance, and configuration tips. It outlines best practices for log management, including preventing split-brain scenarios, optimizing indexing performance, and setting up syslog daemons. Additionally, it provides insights into the functionalities of tools like Rsyslog and the importance of different log formats.
Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)N Masahiro The document provides an overview of Fluentd, a log collector with a flexible plugin architecture, allowing customization through input, output, and buffer plugins. It details how to install Fluentd plugins, configure them, and develop new plugins using Ruby, alongside examples of using Fluentd with MongoDB. Additionally, it discusses plugin management, performance considerations, and testing within the Fluentd framework.
Plugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsSadayuki Furuhashi The document discusses plugin-based software architecture, specifically in the context of Ruby, RubyGems, and its applications in frameworks like Fluentd and Embulk. It outlines the benefits of plugin architecture, different design patterns, and the challenges faced in implementation, including version conflicts and performance issues. The presenter emphasizes the importance of modular design and community support in crafting effective plugin systems.
Dexador Rises
Dexador Risesfukamachi The document summarizes a presentation about HTTP clients in Common Lisp. Eitaro Fukamachi discusses several Common Lisp HTTP client libraries, including Drakma and his own library called Dexador. He notes some pitfalls of Drakma, such as forcing URL encoding and poor error handling. Dexador is presented as an alternative with simpler APIs, better language support, and improved error handling including automatic retrying. Benchmarks show that Dexador is faster than Drakma for local requests and comparable for remote requests, but connection pooling in Dexador can further improve performance for multiple requests.
Treasure Data and OSS
Treasure Data and OSSN Masahiro - Treasure Data is a cloud data service that provides data acquisition, storage, and analysis capabilities.
- It collects data from various sources using Fluentd and Embulk and stores it in its own columnar database called Plazma DB.
- It offers various computing frameworks like Hive, Pig, and Presto for analytics and visualization with tools like Tableau.
- Presto is an interactive SQL query engine that can query data in HDFS, Hive, Cassandra and other data stores.
Fluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At FossasiaN Masahiro Masahiro Nakagawa is a senior software engineer at Treasure Data and the main maintainer of Fluentd. Fluentd is a data collector for unified logging that provides a streaming data transfer based on JSON. It has a simple core with plugins written in Ruby to provide functionality like input/output, buffering, parsing, filtering and formatting of data.
More Related Content
What's hot (20)
The Patterns of Distributed Logging and Containers
The Patterns of Distributed Logging and ContainersSATOSHI TAGOMORI The document discusses the complexities of logging in environments utilizing microservices and containers, highlighting challenges such as the lack of permanent storage and fixed addresses. It presents various aggregation patterns for logging, including source-side and destination-side aggregation, as well as practices using tools like Fluentd and Elasticsearch for efficient log management. It emphasizes best practices for scalable logging that ensures service stability and business growth.
Fluentd v1.0 in a nutshell
Fluentd v1.0 in a nutshellN Masahiro This document summarizes Fluentd v1.0 and provides details about its new features and release plan. It notes that Fluentd v1.0 will provide stable APIs and compatibility with previous versions while improving plugin APIs, adding Windows and multicore support, and increasing event time resolution to nanoseconds. The release is planned for Q3 2017 to allow feedback on v0.14 before finalizing v1.0 features.
Docker and Fluentd
Docker and FluentdN Masahiro - The document discusses logging for containers using Fluentd, an open source data collector. It describes how Fluentd can provide a unified logging layer, reliably forwarding and aggregating logs from multiple containers and applications in a pluggable way.
- Key points covered include using Fluentd with the new Docker logging drivers to directly collect logs from containers, avoiding performance penalties from other approaches. A demo of Fluentd is also mentioned.
The basics of fluentd
The basics of fluentdTreasure Data, Inc. Fluentd is an open source data collector that allows flexible data collection, processing, and storage. It collects log data from various sources using input plugins and sends the data to various outputs like files, databases or forward to other Fluentd servers. It uses a pluggable architecture so new input/output plugins can be added through Ruby gems. It provides features like buffering, retries and reliability.
Fluentd 101
Fluentd 101SATOSHI TAGOMORI Fluentd is an open source data collector that provides a unified logging layer between data sources and backend systems. It decouples these systems by collecting and processing logs and events in a flexible and scalable way. Fluentd uses plugins and buffers to make data collection reliable even in the case of errors or failures. It can forward data between Fluentd nodes for high availability and load balancing.
Docker.io
Docker.ioLadislav Prskavec The document is a comprehensive guide on using Docker, an open-source platform for packaging applications in lightweight containers. It covers installation, command usage, Dockerfile best practices, and the lifecycle of containers and images, emphasizing efficient image creation and management. Additionally, it provides links to resources and projects that utilize Docker for further exploration.
Fluentd introduction at ipros
Fluentd introduction at iprosTreasure Data, Inc. Masahiro Nakagawa from Treasure Data gave a presentation on Fluentd, an open source log collector. Fluentd allows for reliable and structured logging, forwarding, and processing of data through its pluggable architecture. It can collect logs from various sources and output to different destinations using plugins. Common uses of Fluentd include log aggregation, monitoring, and analysis on large-scale architectures.
Fluentd v1 and future at techtalk
Fluentd v1 and future at techtalkN Masahiro Fluentd v1 provides major improvements over v0.12 including nanosecond event time resolution, multi-core support, Windows support, and new plugin APIs. The new plugin APIs provide well-controlled lifecycles and integrate all output plugins. v1 also introduces a server engine based supervisor, dynamic buffering capabilities, and various plugin helpers. While maintaining compatibility with v0.12 plugins, v1 focuses on ease of use, stability, performance and flexibility.
The basics of fluentd
The basics of fluentdTreasure Data, Inc. This document discusses Fluentd, an open source log collector. It provides a pluggable architecture that allows data to be collected, filtered, and forwarded to various outputs. Fluentd uses JSON format for log messages and MessagePack internally. It is reliable, scalable, and extensible through plugins. Common use cases include log aggregation, monitoring, and analytics across multiple servers and applications.
Distributed Stream Processing on Fluentd / #fluentd
Distributed Stream Processing on Fluentd / #fluentdSATOSHI TAGOMORI This document discusses NHN Japan's use of Fluentd for log collection, conversion, and analysis. It summarizes their architecture which involves collecting logs from servers using Scribeline, delivering them to Fluentd processes, converting the logs to a structured format using Fluentd plugins, writing the structured data to HDFS, and performing analysis and reporting with Hive. Key aspects discussed include the scaling of Fluentd processes, performance improvements over their previous Hadoop Streaming-based approach, and implementation details of plugins used.
Async and Non-blocking IO w/ JRuby
Async and Non-blocking IO w/ JRubyJoe Kutner The document discusses asynchronous and non-blocking I/O with JRuby. It explains that asynchronous operations are better than synchronous operations because they use fewer resources and allow for parallelism. It provides an example of building a JRuby application with the Ratpack framework that makes asynchronous HTTP requests to eBay's API in a non-blocking way using promises. It also discusses using RxJava and Hystrix with Ratpack to build a book management application that handles data and API requests asynchronously.
Modern Black Mages Fighting in the Real World
Modern Black Mages Fighting in the Real WorldSATOSHI TAGOMORI This document describes a presentation about introducing black magic programming patterns in Ruby and their pragmatic uses. It provides an overview of Fluentd, including what it is, its versions, and the changes between versions 0.12 and 0.14. Specifically, it discusses how the plugin API was updated in version 0.14 to address problems with the version 0.12 API. It also explains how a compatibility layer was implemented to allow most existing 0.12 plugins to work without modification in 0.14.
Fluentd meetup #2
Fluentd meetup #2Treasure Data, Inc. The document discusses the logging system Fluentd. It describes how Fluentd provides extensibility through plugins, uses a unified JSON log format, and offers simplicity. Major companies like NHN Japan and COOKPAD use Fluentd in production. The future of Fluentd includes new features like filters in sources, labels, faster MessagePack serialization, and a more efficient process model with live restart capabilities.
Fluentd unified logging layer
Fluentd unified logging layerKiyoto Tamura This document discusses Fluentd, an open source data collector. It provides an overview of Fluentd's architecture and components including input plugins, parser plugins, buffer plugins, output plugins, and formatter plugins. It also outlines Fluentd's roadmap, including plans to add filtering capabilities and improve the plugin API. Examples are given throughout to illustrate how Fluentd works and can be configured for use cases like log collection.
Treasure Data Summer Internship 2016
Treasure Data Summer Internship 2016Yuta Iwama Yuta Iwama completed an internship where they added several new features and enhancements to Fluentd. Some of the key additions included implementing a counter API, adding data compression to buffers and forwards, creating a new simpler output plugin for secondary sections, developing a CLI tool to read dumped log data, and optimizing multiple filter calls. The internship provided valuable experience contributing to an open source middleware project and learning about aspects of design, development, and code quality.
Fluentd meetup
Fluentd meetupSadayuki Furuhashi This document provides an overview of Fluentd, an open source data collector for structured logging. Fluentd uses a pluggable architecture and JSON format for log messages, allowing logs to be filtered, buffered, and reliably forwarded to storage. It provides client libraries for integrating with applications in languages like Ruby, Perl, PHP, Python and Java. Fluentd is positioned as an alternative to other log collection systems like Scribe and Flume, with advantages of being easier to install, configure, extend with plugins, and smaller footprint.
On Centralizing Logs
On Centralizing LogsSematext Group, Inc. The document discusses the centralization of logs using tools like Elasticsearch, Logstash, and Kibana, emphasizing stability, performance, and configuration tips. It outlines best practices for log management, including preventing split-brain scenarios, optimizing indexing performance, and setting up syslog daemons. Additionally, it provides insights into the functionalities of tools like Rsyslog and the importance of different log formats.
Fluentd meetup dive into fluent plugin (outdated)
Fluentd meetup dive into fluent plugin (outdated)N Masahiro The document provides an overview of Fluentd, a log collector with a flexible plugin architecture, allowing customization through input, output, and buffer plugins. It details how to install Fluentd plugins, configure them, and develop new plugins using Ruby, alongside examples of using Fluentd with MongoDB. Additionally, it discusses plugin management, performance considerations, and testing within the Fluentd framework.
Plugin-based software design with Ruby and RubyGems
Plugin-based software design with Ruby and RubyGemsSadayuki Furuhashi The document discusses plugin-based software architecture, specifically in the context of Ruby, RubyGems, and its applications in frameworks like Fluentd and Embulk. It outlines the benefits of plugin architecture, different design patterns, and the challenges faced in implementation, including version conflicts and performance issues. The presenter emphasizes the importance of modular design and community support in crafting effective plugin systems.
Dexador Rises
Dexador Risesfukamachi The document summarizes a presentation about HTTP clients in Common Lisp. Eitaro Fukamachi discusses several Common Lisp HTTP client libraries, including Drakma and his own library called Dexador. He notes some pitfalls of Drakma, such as forcing URL encoding and poor error handling. Dexador is presented as an alternative with simpler APIs, better language support, and improved error handling including automatic retrying. Benchmarks show that Dexador is faster than Drakma for local requests and comparable for remote requests, but connection pooling in Dexador can further improve performance for multiple requests.
Similar to JRuby with Java Code in Data Processing World (20)
Treasure Data and OSS
Treasure Data and OSSN Masahiro - Treasure Data is a cloud data service that provides data acquisition, storage, and analysis capabilities.
- It collects data from various sources using Fluentd and Embulk and stores it in its own columnar database called Plazma DB.
- It offers various computing frameworks like Hive, Pig, and Presto for analytics and visualization with tools like Tableau.
- Presto is an interactive SQL query engine that can query data in HDFS, Hive, Cassandra and other data stores.
Fluentd Unified Logging Layer At Fossasia
Fluentd Unified Logging Layer At FossasiaN Masahiro Masahiro Nakagawa is a senior software engineer at Treasure Data and the main maintainer of Fluentd. Fluentd is a data collector for unified logging that provides a streaming data transfer based on JSON. It has a simple core with plugins written in Ruby to provide functionality like input/output, buffering, parsing, filtering and formatting of data.
Fighting Against Chaotically Separated Values with Embulk
Fighting Against Chaotically Separated Values with EmbulkSadayuki Furuhashi Embulk is an open-source parallel bulk data loader designed to simplify and speed up the process of transferring data across various formats and storage systems. Developed by Sadayuki Furuhashi, it addresses common issues in data loading, such as handling broken records and improving performance through a plugin-based architecture that supports multiple input and output sources. The document intricately details Embulk's functionality, use cases, and enhancements since its initial release, highlighting its importance in managing big data efficiently.
Zero mq logs
Zero mq logsTomas Doran This document discusses using ZeroMQ and Elasticsearch for log aggregation. It proposes using ZeroMQ to transmit structured log data from application servers to a central Logstash server, which would then insert the logs into Elasticsearch for querying and analysis. This approach aims to provide a lightweight logging solution that doesn't block application servers like traditional logging to databases can. The document also provides background on tools like Logstash, Elasticsearch, and Splunk.
Fluentd and Embulk Game Server 4
Fluentd and Embulk Game Server 4N Masahiro This document summarizes Masahiro Nakagawa's presentation on Fluentd and Embulk. Fluentd is a data collector for unified logging that allows for streaming data transfer based on JSON. It is written in Ruby and uses plugins to collect, process, and output data. Embulk is a bulk loading tool that allows high performance parallel processing of data to load it into various databases and storage systems. Both tools use a pluggable architecture to provide flexibility in handling different data sources and targets.
Ruby - The Hard Bits
Ruby - The Hard BitsPaul Gallagher In his talk at the RedDotRubyConf 2013, Paul Gallagher reflects on his journey as a Rubyist and discusses the future of Ruby amid evolving technology trends like mobile applications, cloud computing, and big data. He highlights the importance of productivity, the rise of different programming paradigms, and how Ruby plays a role in these developments. Gallagher concludes with a positive outlook on the continued relevance of Ruby in the programming landscape over the next five years.
Norikra: SQL Stream Processing In Ruby
Norikra: SQL Stream Processing In RubySATOSHI TAGOMORI This document summarizes a presentation about Norikra, an open source SQL stream processing engine written in Ruby. Some key points:
- Norikra allows for schema-less stream processing using SQL queries without needing to restart for new queries. It supports windows, joins, UDFs.
- Example queries demonstrate counting events by field values over time windows. Nested fields can be queried directly.
- Norikra is used in production at LINE for analytics like API error reporting and a Lambda architecture with real-time and batch processing.
- It is built on JRuby to leverage Java libraries and Ruby gems, and can handle 10k-100k events/sec on typical hardware. The
The Enterprise Strikes Back
The Enterprise Strikes BackBurke Libbey The document discusses the integration of Ruby with Java using JRuby, highlighting its advantages and challenges. It covers topics such as invoking Java libraries in Ruby, handling impedance mismatches between Ruby and Java syntax, and practical applications like system provisioning with Puppet and Chef. Additionally, it emphasizes Ruby's role as a 'glue' language for connecting various codebases and its capabilities for handling database interactions through ORMs like Active Record and Sequel.
Fluentd - RubyKansai 65
Fluentd - RubyKansai 65N Masahiro Masahiro Nakagawa introduced Fluentd, an open source data collector. Fluentd provides a unified logging layer and collects data through a streaming data transfer based on JSON. It is written in Ruby and uses a plugin architecture to allow for various input and output functions. Fluentd is used in production environments for log aggregation, metrics collection, and data processing tasks.
Rollin onj Rubyv3
Rollin onj Rubyv3Oracle The document outlines the integration of Ruby on Rails with JRuby on the Oracle WebLogic Suite, emphasizing JRuby's capability for enterprise applications using Java libraries. It highlights key features of Ruby, the advantages of JRuby for performance and integration, and discusses tools and frameworks such as ActiveRecord and JPA for database access. Additionally, it covers the development tools and scalability solutions offered by Oracle to enhance web app deployments in enterprise environments.
First Day With J Ruby
First Day With J RubyPraveen Kumar Sinha This document provides an overview of getting started with JRuby on Rails. It covers JRuby 101 basics, how JRuby on Rails works similarly to standard Rails, key differences like database support and lack of native extensions, deployment options like WAR files and app servers, and migration considerations from Ruby to JRuby.
Embulk, an open-source plugin-based parallel bulk data loader
Embulk, an open-source plugin-based parallel bulk data loaderSadayuki Furuhashi The document discusses Embulk, an open-source parallel bulk data loader that uses plugins. Embulk loads records from various sources ("A") to various targets ("B") using plugins for different source and target types. This makes the painful process of data integration more relaxed. Embulk executes in parallel, validates data, handles errors, behaves deterministically, and allows for idempotent retries of bulk loads.
Insight Data Engineering: Open source data ingestion
Insight Data Engineering: Open source data ingestionTreasure Data, Inc. The document discusses the complexities of data collection and ingestion, comparing the two processes and outlining various tools for each. It highlights tools such as rsyslog, Fluentd, and Kafka, detailing their use cases and features. A case study on asynchronous application logging underscores the importance of effective logging practices in data handling.
Open source data ingestion
Open source data ingestionTreasure Data, Inc. This document discusses data collection and ingestion tools. It begins with an overview of data collection versus ingestion, with collection happening at the source and ingestion receiving the data. Examples of data collection tools include rsyslog, Scribe, Flume, Logstash, Heka, and Fluentd. Examples of ingestion tools include RabbitMQ, Kafka, and Fluentd. The document concludes with a case study of asynchronous application logging and challenges to consider.
20081022cca
20081022ccaJeff Hammerbacher 1. Jeff Hammerbacher managed Facebook's data team from 2006 to 2008 and helped build systems to handle the massive growth of user data and services at Facebook.
2. The data team developed distributed data platforms like Hadoop and Cassandra to process terabytes of user data across hundreds of servers and make it available for services and insights.
3. Key challenges included simplifying complex systems, achieving scalability to handle large data volumes, improving manageability with less human intervention, and using open source technologies to build developer communities.
An Open Source NoSQL solution for Internet Access Logs Analysis
An Open Source NoSQL solution for Internet Access Logs AnalysisJosé Manuel Ciges Regueiro The document discusses the advantages of using a NoSQL database for internet access log analysis, comparing it to traditional relational databases. It specifically highlights the author's choice of MongoDB for its flexibility, performance, and comprehensive support, as well as its ability to handle large volumes of data effectively. The work includes a detailed analysis of testing MongoDB's capabilities against MySQL in terms of speed and efficiency in processing various data queries.
Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!
Let's reconsider about collecting logs. Plus, visiting elastic@Moutain View!心 谷本 1. The document discusses reconsidering how logs are processed by examining use cases like visualizing logs in charts, watching logs for issues, and keeping logs for future use.
2. It suggests treating logs as a stream of events and using a log streaming hub to flexibly route logs to tools for viewing, watching, visualizing, and storage.
3. Examples of log streaming systems are shown using technologies like Fluentd, Redis, Amazon Kinesis, and AWS services to process logs in scalable and painless ways.
DataEngConf SF16 - Collecting and Moving Data at Scale
DataEngConf SF16 - Collecting and Moving Data at Scale Hakka Labs This document summarizes Sada Furuhashi's presentation on Fluentd, an open source data collector. Fluentd provides a centralized way to collect, filter, and output log data from various sources like applications, servers, and databases. It addresses challenges with typical log collection architectures that have high latency, complex parsing, and a combination explosion of connections. Fluentd uses a plugin-based architecture with input, filter, and output components to flexibly collect, transform, and deliver log data at scale to targets like files, databases and visualization tools. Many large companies like Microsoft, Atlassian and Amazon use Fluentd for log collection and analytics in production environments.
Gofer 200707
Gofer 200707oscon2007 Gofer is a scalable stateless proxy architecture for DBI that is transport independent, highly configurable, efficient, well tested, scalable, and simple. It consists of a simple request/response protocol, a DBI proxy driver called DBD::Gofer, a request executor module, pluggable transport modules like HTTP, SSH, and Gearman, and an extensible client configuration mechanism. It aims to minimize round trips and supports connection pooling to improve performance and scalability.
Ruby on Rails (RoR) as a back-end processor for Apex
Ruby on Rails (RoR) as a back-end processor for Apex Espen Brækken The document discusses using Ruby on Rails as a supplement to Oracle Application Express (Apex) for building applications. It provides an overview of why Rails may be useful when Apex has limitations, such as communicating with external systems, parsing emails, or SSH/FTP functionality. The document then covers why Ruby and Rails are good options, how ActiveRecord can be used to interface with databases, and examples of sample Rails applications and projects that interface with external systems or call Java code. It concludes with recommendations around learning Ruby first before frameworks, assessing needs, and using tools to complement rather than replace existing systems like Apex.
Ad
More from SATOSHI TAGOMORI (20)
Ractor's speed is not light-speed
Ractor's speed is not light-speedSATOSHI TAGOMORI Ractor is a new experimental feature in Ruby 3.0 that allows Ruby code to run in parallel on CPUs. It manages objects per Ractor and can move objects between Ractors, making moved objects invisible to the original Ractor. It can share certain "shareable" objects like modules, classes, and frozen objects between Ractors. For web applications to fully utilize Ractors, an experimental application server called Right Speed was created that uses Rack and runs processing workers on Ractors. However, there are still problems to address like exceptions when closing connections and accessing non-shareable constants and instance variables across Ractors before Ractors can be ready for production use in web applications.
Good Things and Hard Things of SaaS Development/Operations
Good Things and Hard Things of SaaS Development/OperationsSATOSHI TAGOMORI This document discusses the good and hard things about developing and operating a SaaS platform. It describes how the backend team at Treasure Data owns and manages various components of their distributed platform. It also discusses how they have modernized their deployment process from a periodic Chef-based approach to using CodeDeploy for more frequent deployments. This allows them to move faster by doing many small releases that minimize the number of affected components and customers.
Maccro Strikes Back
Maccro Strikes BackSATOSHI TAGOMORI The document is an invitation to a presentation about Maccro, a Ruby macro system that allows rewriting Ruby methods by registering rules. It summarizes Maccro's capabilities such as registering before/after matchers, applying rules to methods, matching method ASTs to rules, and rewriting method code by replacing placeholders. It also lists some limitations including only supporting Ruby 2.6+, not handling method call order dependencies, and not updating method visibility. The presentation aims to recruit more developers to Maccro to handle its remaining challenges.
Invitation to the dark side of Ruby
Invitation to the dark side of RubySATOSHI TAGOMORI The document outlines a presentation by Satoshi Tagomori at RubyKaigi 2019, introducing a new macro post-processor for Ruby called 'maccro'. It describes its functionality to automate routine code conversions and the various applications of macros within Ruby, including limitations and compatibility with Ruby versions. The presentation emphasizes the potential of macros in enhancing Ruby's capabilities while inviting the audience to experiment with the tool.
Hijacking Ruby Syntax in Ruby (RubyConf 2018)
Hijacking Ruby Syntax in Ruby (RubyConf 2018)SATOSHI TAGOMORI The document outlines a presentation on advanced Ruby features and techniques by @joker1007 and @tagomoris during RubyConf 2018. Key topics discussed include Ruby binding, tracepoint, method hooks, and various hacks such as method modifiers and resource management enhancements. The presentation emphasizes the creative use of these features to extend Ruby's functionality while cautioning about the complexities and potential pitfalls of these 'magic' techniques.
Make Your Ruby Script Confusing
Make Your Ruby Script ConfusingSATOSHI TAGOMORI The document discusses constants in Ruby programming. It notes that constant names start with capital letters and can be overwritten with warnings. It provides some constant name samples and encourages trying out different Ruby versions. It also discusses making Ruby scripts confusing by using Unicode characters and constant names that are difficult to understand for future maintainers.
Hijacking Ruby Syntax in Ruby
Hijacking Ruby Syntax in RubySATOSHI TAGOMORI This document summarizes a presentation given by @joker1007 and @tagomoris on hijacking Ruby syntax. It discusses various Ruby features like bindings, tracepoints, and refinements that can be used to modify Ruby's behavior. It then demonstrates several "hacks" that leverage these features, including finalist, overrider, and abstriker which add method modifiers, and binding_ninja which allows implicitly passing a binding. Other hacks discussed are with_resources and deferral which add "with" and "defer" statements to Ruby. The presentation emphasizes how these modifications are implemented using hooks, bindings, tracepoints and other Ruby internals.
Lock, Concurrency and Throughput of Exclusive Operations
Lock, Concurrency and Throughput of Exclusive OperationsSATOSHI TAGOMORI 1. The document discusses different patterns for implementing locks in a distributed key-value store to maximize concurrency and throughput of operations.
2. It describes a naive giant lock approach that locks the entire storage for any operation, resulting in poor concurrency and throughput.
3. Better approaches use metadata locks plus simple resource locks, and reference counting locks, to allow concurrent updates to different resources while minimizing critical sections.
Data Processing and Ruby in the World
Data Processing and Ruby in the WorldSATOSHI TAGOMORI This document discusses Ruby's role in data processing. It outlines the typical steps of data processing - collect, summarize, analyze, visualize. It then provides examples of open-source Ruby tools that can be used for each step, such as Fluentd for collection, and libraries for numerical analysis, bioinformatics, and machine learning. Services that use Ruby for collection and processing are also mentioned, like Log Analytics and Stackdriver Logging. The document encourages continuing to improve Ruby tools to make data processing better and celebrates Ruby's 25th anniversary.
Planet-scale Data Ingestion Pipeline: Bigdam
Planet-scale Data Ingestion Pipeline: BigdamSATOSHI TAGOMORI Bigdam is a planet-scale data ingestion pipeline designed for large-scale data ingestion. It addresses issues with the traditional pipeline such as imperfectqueue throughput limitations, latency in queries from event collectors, difficulty maintaining event collector code, many small temporary and imported files. The redesigned pipeline includes Bigdam-Gateway for HTTP endpoints, Bigdam-Pool for distributed buffer storage, Bigdam-Scheduler to schedule import tasks, Bigdam-Queue as a high throughput queue, and Bigdam-Import for data conversion and import. Consistency is ensured through at-least-once design and deduplication is performed at the end of the pipeline for simplicity and reliability. Components are designed to scale out horizontally.
Technologies, Data Analytics Service and Enterprise Business
Technologies, Data Analytics Service and Enterprise BusinessSATOSHI TAGOMORI This document discusses technologies for data analytics services for enterprise businesses. It begins by defining enterprise businesses as those "not about IT" and data analytics services as providing insights into business metrics like customer reach, ad views, purchases, and more using data. It then outlines some key technologies needed for such services, including data management systems, distributed processing systems, queues and schedulers, tools for connecting systems, and methods for controlling jobs and workflows with retries to handle failures. Specific challenges around deadlines, idempotent operations, and replay-able workflows are also addressed.
Ruby and Distributed Storage Systems
Ruby and Distributed Storage SystemsSATOSHI TAGOMORI This document discusses using Ruby for distributed storage systems. It describes components like Bigdam, which is Treasure Data's new data ingestion pipeline. Bigdam uses microservices and a distributed key-value store called Bigdam-pool to buffer data. The document discusses designing and testing Bigdam using mocking, interfaces, and integration tests in Ruby. It also explores porting Bigdam-pool from Java to Ruby and investigating Ruby's suitability for tasks like asynchronous I/O, threading, and serialization/deserialization.
Perfect Norikra 2nd Season
Perfect Norikra 2nd SeasonSATOSHI TAGOMORI This document discusses features that could make Norikra, an open source stream processing software, even more "perfect". It describes how Norikra currently works and highlights areas for improvement, such as enabling queries to resume processing from historical batch query results, sharing operators between queries to reduce memory usage, and developing a true lambda architecture with a single query language for both streaming and batch processing. The document envisions a "perfect stream processing engine" with these enhanced capabilities.
To Have Own Data Analytics Platform, Or NOT To
To Have Own Data Analytics Platform, Or NOT ToSATOSHI TAGOMORI This document discusses the pros and cons of building an in-house data analytics platform versus using cloud-based services. It notes that in startups it is generally better not to build your own platform and instead use cloud services from AWS, Google, or Treasure Data. However, the options have expanded in recent years to include on-premise or cloud-based platforms from vendors like Cloudera, Hortonworks, or cloud services from various providers. The document does not make a definitive conclusion, but discusses factors to consider around distributed processing, data management, process management, platform management, visualization, and connecting different data sources.
How To Write Middleware In Ruby
How To Write Middleware In RubySATOSHI TAGOMORI This document discusses middleware in Ruby and provides examples of considerations when writing middleware:
- Middleware should be a long-running daemon process that is compatible across platforms and environments and handles various data formats and traffic volumes.
- Tests must be run on all supported platforms to ensure compatibility as thread and process scheduling differs between operating systems.
- Memory usage and object leaks must be carefully managed in long-running processes to avoid consuming resources over time.
- Performance of JSON parsing/generation should be benchmarked and the most optimized library used to avoid unnecessary CPU usage.
Open Source Software, Distributed Systems, Database as a Cloud Service
Open Source Software, Distributed Systems, Database as a Cloud ServiceSATOSHI TAGOMORI - Treasure Data is a database as a cloud service company that collects and stores customer data beyond the cloud [1].
- It uses open source software like Fluentd and MessagePack to easily integrate and collect data from customers [2]. It also uses open source distributed systems software like Hadoop and Presto to store, process and query large amounts of customer data [3].
- As a database service, it needs to share computer resources securely for many customers. It contributes to open source to build and maintain the distributed systems software that powers its cloud database service [4].
How to Make Norikra Perfect
How to Make Norikra PerfectSATOSHI TAGOMORI This document discusses making the Norikra stream processing software more perfect. It outlines how Norikra currently works well for small to medium sites but has limitations for large deployments. The concept of a "Perfect Norikra" is introduced that would add distributed execution, high availability, and dynamic scaling capabilities. A rough design is sketched that involves a new query executor, dataflow manager, and strategies for dynamic scaling through intermediate results and merging across nodes. Challenges mentioned include resource monitoring, multi-tenancy, and supporting queries without aggregations.
Distributed Logging Architecture in Container Era
Distributed Logging Architecture in Container EraSATOSHI TAGOMORI Distributed Logging Architecture in Container Era
The document discusses distributed logging architecture in the container era. It covers: 1) The difficulties of logging with microservices and containers due to their ephemeral and distributed nature, 2) The need to redesign logging to push logs from containers to destinations quickly without fixed addresses or mappings; 3) Common patterns for distributed logging architectures including source aggregation, destination aggregation, and scaling; and 4) A case study using Docker and Fluentd to implement source aggregation and scaling for logging. Open source solutions are important to keep the logging layer transparent, interoperable, and able to scale independently of applications and infrastructure.
Fighting API Compatibility On Fluentd Using "Black Magic"
Fighting API Compatibility On Fluentd Using "Black Magic"SATOSHI TAGOMORI The document discusses Fluentd's changes to its plugin API between versions 0.12 and 0.14. In 0.14, the API was overhauled to separate entry points from implementations and introduce a plugin base class to control data and control flow. A compatibility layer was added to allow most 0.12 plugins to work unmodified in 0.14 by handling calls to overridden methods. However, plugins that override certain methods like #emit may cause errors due to changes in how buffering works.
Overview of data analytics service: Treasure Data Service
Overview of data analytics service: Treasure Data ServiceSATOSHI TAGOMORI Treasure Data provides a data analytics service with the following key components:
- Data is collected from various sources using Fluentd and loaded into PlazmaDB.
- PlazmaDB is the distributed time-series database that stores metadata and data.
- Jobs like queries, imports, and optimizations are executed on Hadoop and Presto clusters using queues, workers, and a scheduler.
- The console and APIs allow users to access the service and submit jobs for processing and analyzing their data.
Ad
Recently uploaded (20)
FME for Good: Integrating Multiple Data Sources with APIs to Support Local Ch...
FME for Good: Integrating Multiple Data Sources with APIs to Support Local Ch...Safe Software Have-a-skate-with-Bob (HASB-KC) is a local charity that holds two Hockey Tournaments every year to raise money in the fight against Pancreatic Cancer. The FME Form software is used to integrate and exchange data via API, between Google Forms, Google Sheets, Stripe payments, SmartWaiver, and the GoDaddy email marketing tools to build a grass-roots Customer Relationship Management (CRM) system for the charity. The CRM is used to communicate effectively and readily with the participants of the hockey events and most importantly the local area sponsors of the event. Communication consists of a BLOG used to inform participants of event details including, the ever-important team rosters. Funds raised by these events are used to support families in the local area to fight cancer and support PanCan research efforts to find a cure against this insidious disease. FME Form removes the tedium and error-prone manual ETL processes against these systems into 1 or 2 workbenches that put the data needed at the fingertips of the event organizers daily freeing them to work on outreach and marketing of the events in the community.
vertical-cnc-processing-centers-drillteq-v-200-en.pdf
vertical-cnc-processing-centers-drillteq-v-200-en.pdfAmirStern2 מכונות CNC קידוח אנכיות הן הבחירה הנכונה והטובה ביותר לקידוח ארונות וארגזים לייצור רהיטים. החלק נוסע לאורך ציר ה-x באמצעות ציר דיגיטלי מדויק, ותפוס ע"י צבת מכנית, כך שאין צורך לבצע setup (התאמות) לגדלים שונים של חלקים.
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...Safe Software The National Fuels Treatments Initiative (NFT) is transforming wildfire mitigation by creating a standardized map of nationwide fuels treatment locations across all land ownerships in the United States. While existing state and federal systems capture this data in diverse formats, NFT bridges these gaps, delivering the first truly integrated national view. This dataset will be used to measure the implementation of the National Cohesive Wildland Strategy and demonstrate the positive impact of collective investments in hazardous fuels reduction nationwide. In Phase 1, we developed an ETL pipeline template in FME Form, leveraging a schema-agnostic workflow with dynamic feature handling intended for fast roll-out and light maintenance. This was key as the initiative scaled from a few to over fifty contributors nationwide. By directly pulling from agency data stores, oftentimes ArcGIS Feature Services, NFT preserves existing structures, minimizing preparation needs. External mapping tables ensure consistent attribute and domain alignment, while robust change detection processes keep data current and actionable. Now in Phase 2, we’re migrating pipelines to FME Flow to take advantage of advanced scheduling, monitoring dashboards, and automated notifications to streamline operations. Join us to explore how this initiative exemplifies the power of technology, blending FME, ArcGIS Online, and AWS to solve a national business problem with a scalable, automated solution.
Security Tips for Enterprise Azure Solutions
Security Tips for Enterprise Azure SolutionsMichele Leroux Bustamante Delivering solutions to Azure may involve a variety of architecture patterns involving your applications, APIs data and associated Azure resources that comprise the solution. This session will use reference architectures to illustrate the security considerations to protect your Azure resources and data, how to achieve Zero Trust, and why it matters. Topics covered will include specific security recommendations for types Azure resources and related network security practices. The goal is to give you a breadth of understanding as to typical security requirements to meet compliance and security controls in an enterprise solution.
TrustArc Webinar - 2025 Global Privacy Survey
TrustArc Webinar - 2025 Global Privacy SurveyTrustArc How does your privacy program compare to your peers? What challenges are privacy teams tackling and prioritizing in 2025?
In the sixth annual Global Privacy Benchmarks Survey, we asked global privacy professionals and business executives to share their perspectives on privacy inside and outside their organizations. The annual report provides a 360-degree view of various industries' priorities, attitudes, and trends. See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar features an expert panel discussion and data-driven insights to help you navigate the shifting privacy landscape. Whether you are a privacy officer, legal professional, compliance specialist, or security expert, this session will provide actionable takeaways to strengthen your privacy strategy.
This webinar will review:
- The emerging trends in data protection, compliance, and risk
- The top challenges for privacy leaders, practitioners, and organizations in 2025
- The impact of evolving regulations and the crossroads with new technology, like AI
Predictions for the future of privacy in 2025 and beyond
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/addressing-evolving-ai-model-challenges-through-memory-and-storage-a-presentation-from-micron/
Wil Florentino, Senior Segment Marketing Manager at Micron, presents the “Addressing Evolving AI Model Challenges Through Memory and Storage” tutorial at the May 2025 Embedded Vision Summit.
In the fast-changing world of artificial intelligence, the industry is deploying more AI compute at the edge. But the growing diversity and data footprint of transformers and models such as large language models and large multimodal models puts a spotlight on memory performance and data storage capacity as key bottlenecks. Enabling the full potential of AI in industries such as manufacturing, automotive, robotics and transportation will require us to find efficient ways to deploy this new generation of complex models.
In this presentation, Florentino explores how memory and storage are responding to this need and solving complex issues in the AI market. He examines the storage capacity and memory bandwidth requirements of edge AI use cases ranging from tiny devices with severe cost and power constraints to edge servers, and he explains how new memory technologies such as LPDDR5, LPCAMM2 and multi-port SSDs are helping system developers to meet these challenges.
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdf
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdfbiswajitbanerjee38 Russia is one of the most aggressive nations when it comes to state coordinated cyberattacks — and Ukraine has been at the center of their crosshairs for 3 years. This report, provided the State Service of Special Communications and Information Protection of Ukraine contains an incredible amount of cybersecurity insights, showcasing the coordinated aggressive cyberwarfare campaigns of Russia against Ukraine.
It brings to the forefront that understanding your adversary, especially an aggressive nation state, is important for cyber defense. Knowing their motivations, capabilities, and tactics becomes an advantage when allocating resources for maximum impact.
Intelligence shows Russia is on a cyber rampage, leveraging FSB, SVR, and GRU resources to professionally target Ukraine’s critical infrastructures, military, and international diplomacy support efforts.
The number of total incidents against Ukraine, originating from Russia, has steadily increased from 1350 in 2021 to 4315 in 2024, but the number of actual critical incidents has been managed down from a high of 1048 in 2022 to a mere 59 in 2024 — showcasing how the rapid detection and response to cyberattacks has been impacted by Ukraine’s improved cyber resilience.
Even against a much larger adversary, Ukraine is showcasing outstanding cybersecurity, enabled by strong strategies and sound tactics. There are lessons to learn for any enterprise that could potentially be targeted by aggressive nation states.
Definitely worth the read!
FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce.pptx
FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce.pptxFIDO Alliance FIDO Seminar: Targeting Trust: The Future of Identity in the Workforce
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation Can We Use Rust to Develop Extensions for PostgreSQL?
(POSETTE: An Event for Postgres 2025)
June 11, 2025
Shinya Kato
NTT DATA Japan Corporation
Kubernetes Security Act Now Before It’s Too Late
Kubernetes Security Act Now Before It’s Too LateMichael Furman In today's cloud-native landscape, Kubernetes has become the de facto standard for orchestrating containerized applications, but its inherent complexity introduces unique security challenges. Are you one YAML away from disaster?
This presentation, "Kubernetes Security: Act Now Before It’s Too Late," is your essential guide to understanding and mitigating the critical security risks within your Kubernetes environments. This presentation dives deep into the OWASP Kubernetes Top Ten, providing actionable insights to harden your clusters.
We will cover:
The fundamental architecture of Kubernetes and why its security is paramount.
In-depth strategies for protecting your Kubernetes Control Plane, including kube-apiserver and etcd.
Crucial best practices for securing your workloads and nodes, covering topics like privileged containers, root filesystem security, and the essential role of Pod Security Admission.
Don't wait for a breach. Learn how to identify, prevent, and respond to Kubernetes security threats effectively.
It's time to act now before it's too late!
FIDO Seminar: Authentication for a Billion Consumers - Amazon.pptx
FIDO Seminar: Authentication for a Billion Consumers - Amazon.pptxFIDO Alliance FIDO Seminar: Authentication for a Billion Consumers - Amazon
MuleSoft for AgentForce : Topic Center and API Catalog
MuleSoft for AgentForce : Topic Center and API Catalogshyamraj55 This presentation dives into how MuleSoft empowers AgentForce with organized API discovery and streamlined integration using Topic Center and the API Catalog. Learn how these tools help structure APIs around business needs, improve reusability, and simplify collaboration across teams. Ideal for developers, architects, and business stakeholders looking to build a connected and scalable API ecosystem within AgentForce.
AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc AI VIDEO MAGAZINE - r/aivideo community newsletter – Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -
FME for Distribution & Transmission Integrity Management Program (DIMP & TIMP)
FME for Distribution & Transmission Integrity Management Program (DIMP & TIMP)Safe Software Peoples Gas in Chicago, IL has changed to a new Distribution & Transmission Integrity Management Program (DIMP & TIMP) software provider in recent years. In order to successfully deploy the new software we have created a series of ETL processes using FME Form to transform our gas facility data to meet the required DIMP & TIMP data specifications. This presentation will provide an overview of how we used FME to transform data from ESRI’s Utility Network and several other internal and external sources to meet the strict data specifications for the DIMP and TIMP software solutions.
High Availability On-Premises FME Flow.pdf
High Availability On-Premises FME Flow.pdfSafe Software FME Flow is a highly robust tool for transforming data both automatically and by user-initiated workflows. At the Finnish telecommunications company Elisa, FME Flow serves processes and internal stakeholders that require 24/7 availability from underlying systems, while imposing limitations on the use of cloud based systems. In response to these business requirements, Elisa has implemented a high-availability on-premises setup of FME Flow, where all components of the system have been duplicated or clustered. The goal of the presentation is to provide insights into the architecture behind the high-availability functionality. The presentation will show in basic technical terms how the different parts of the system work together. Basic level understanding of IT technologies is required to understand the technical portion of the presentation, namely understanding the purpose of the following components: load balancer, FME Flow host nodes, FME Flow worker nodes, network file storage drives, databases, and external authentication services. The presentation will also outline our lessons learned from the high-availability project, both benefits and challenges to consider.
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...Safe Software Jacobs has developed a 3D utility solids modelling workflow to improve the integration of utility data into 3D Building Information Modeling (BIM) environments. This workflow, a collaborative effort between the New Zealand Geospatial Team and the Australian Data Capture Team, employs FME to convert 2D utility data into detailed 3D representations, supporting enhanced spatial analysis and clash detection.
To enable the automation of this process, Jacobs has also developed a survey data standard that standardizes the capture of existing utilities. This standard ensures consistency in data collection, forming the foundation for the subsequent automated validation and modelling steps. The workflow begins with the acquisition of utility survey data, including attributes such as location, depth, diameter, and material of utility assets like pipes and manholes. This data is validated through a custom-built tool that ensures completeness and logical consistency, including checks for proper connectivity between network components. Following validation, the data is processed using an automated modelling tool to generate 3D solids from 2D geometric representations. These solids are then integrated into BIM models to facilitate compatibility with 3D workflows and enable detailed spatial analyses.
The workflow contributes to improved spatial understanding by visualizing the relationships between utilities and other infrastructure elements. The automation of validation and modeling processes ensures consistent and accurate outputs, minimizing errors and increasing workflow efficiency.
This methodology highlights the application of FME in addressing challenges associated with geospatial data transformation and demonstrates its utility in enhancing data integration within BIM frameworks. By enabling accurate 3D representation of utility networks, the workflow supports improved design collaboration and decision-making in complex infrastructure projects
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019 A presentation at Internetware 2025.
Creating Inclusive Digital Learning with AI: A Smarter, Fairer Future
Creating Inclusive Digital Learning with AI: A Smarter, Fairer FutureImpelsys Inc. Have you ever struggled to read a tiny label on a medicine box or tried to navigate a confusing website? Now imagine if every learning experience felt that way—every single day.
For millions of people living with disabilities, poorly designed content isn’t just frustrating. It’s a barrier to growth. Inclusive learning is about fixing that. And today, AI is helping us build digital learning that’s smarter, kinder, and accessible to everyone.
Accessible learning increases engagement, retention, performance, and inclusivity for everyone. Inclusive design is simply better design.
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...Edge AI and Vision Alliance
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation
JRuby with Java Code in Data Processing World
- 1. JRuby with Java Code in Data Processing World JRubyConf.EU at 31 Jul 2015 Satoshi Tagomori (@tagomoris)
- 2. Satoshi "Moris" Tagomori (@tagomoris) Fluentd, Norikra, MessagePack-Ruby,... Docker logging driver for Fluentd (docker v1.8) Treasure Data, Inc.
- 3. https://jobs.lever.co/treasure-data We're hiring! OSS team (developer / community manager) Distributed system engineer (Hadoop, queue/workers) Front-end engineer (RoR)
- 7. Data Processing World Hadoop, Spark, Tez, Flink, Storm, Kafka, ... Hive, Pig, Drill, Impala, Presto, ....
- 8. Java + Scala, Clojure + C++, .... Data Processing World on JVM
- 9. Data Processing World Many CPU cores, Large memory, High rate Disk I/O, ... High throughput data processing Hadoop YARN/MapReduce/HDFS API compatibility
- 10. Two OSS using Java&JRuby
- 11. Norikra: Stream Processing with SQL for everybody Server software, written in JRuby, runs on JVM Open source software (GPLv2) http://norikra.github.io/ https://github.com/norikra/norikra Distributed on rubygems.org "gem i norikra"
- 12. What Norikra does: SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC
- 13. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":300, "duration":0.03, "referer":"...", "user-agent":"...." path:"/", s:301 1
- 14. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/a", "status":200, "bytes":10240, "duration":0.53, "referer":"...", "user-agent":"...." path:"/", s:301 path:"/download/a", s:10240 2
- 15. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":404, "bytes":0, "duration":0.08, "referer":"...", "user-agent":"...." path:"/", s:301 path:"/download/a", s:10240 3
- 16. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.01, "referer":"...", "user-agent":"...." path:"/", s:602 path:"/download/a", s:10240 4
- 17. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/b", "status":200, "bytes":678, "duration":0.11, "referer":"...", "user-agent":"...." path:"/", s:602 path:"/download/a", s:10240 path:"/download/b", s:678 5
- 18. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/b", "status":200, "bytes":678, "duration":0.13, "referer":"...", "user-agent":"...." path:"/", s:602 path:"/download/a", s:10240 path:"/download/b", s:1356 6
- 19. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.02, "referer":"...", "user-agent":"...." path:"/", s:903 path:"/download/a", s:10240 path:"/download/b", s:1356 7
- 20. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.09, "referer":"...", "user-agent":"...." path:"/", s:1204 path:"/download/a", s:10240 path:"/download/b", s:1356 8
- 21. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/download/a", "status":200, "bytes":10240, "duration":1.1, "referer":"...", "user-agent":"...." path:"/", s:1204 path:"/download/a", s:20480 path:"/download/b", s:1356 9
- 22. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC {"path":"/", "status":200, "bytes":301, "duration":0.05, "referer":"...", "user-agent":"...." path:"/", s:1505 path:"/download/a", s:20480 path:"/download/b", s:1356 10
- 23. SELECT path, SUM(bytes) AS s FROM www_access_logs.win:length_batch(10) WHERE status=200 GROUP BY path ORDER BY s DESC 10 {"path":"/download/a", "s":20480} {"path":"/", "s":1505} {"path":"/download/b", "s":1356}
- 24. Norikra and Java Norikra is written in JRuby, and using Esper Key factor: productivity (33days until first release) Esper:Java library, provides Complex Event Processing SQL parser, executor Many features and good performance Licensed under GPLv2
- 25. Plugins as rubygems Norikra Server (on JVM) Esper (Query Engine) Type Definition Manager Output Event Pool Norikra Engine RPC Server mizuno (Jetty + Rack) Rack RPC Handler Listener UDF UDF User-Defined Functions "gem i norikra-udf-xxx" written in Java, or JRuby (compiled to Java) works in Esper instance: must be a Java class Listener handler for output data of queries, written in JRuby "gem i norikra-listener-xxx"
- 26. Embulk "Embulk is a open-source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services." http://www.embulk.org/docs/
- 27. Embulk: makes painful data integration work relaxed Plugin-based parallel bulk data loader Open source software (Apache License v2.0) http://www.embulk.org/ https://github.com/embulk/embulk Distributed as .jar or on rubygems.org Plugins are on rubygems.org http://www.slideshare.net/frsyuki/embuk-making-data-integration-works-relaxed http://www.slideshare.net/HiroshiNakamura/embulk-20150411
- 28. HDFS MySQL Amazon S3 Embulk CSV Files SequenceFile Salesforce.com Elasticsearch Cassandra Hive Redis ✓ Parallel execution ✓ Data validation ✓ Error recovery ✓ Deterministic behavior ✓ Idempotet retrying Plugins Plugins bulk load
- 29. #ccc_cd4 / #embulk InputPlugin OutputPlugin Executor plugin Filter plugin Filter plugin Filter plugins records Threads, MapReduce records convert, … input, … output. 29 records config
- 30. #ccc_cd4 / #embulk InputPlugin FileInput plugin OutputPlugin FileOutput plugin Encoder plugin Formatter plugin Decoder plugin Parser plugin HDFS, S3, Riak CS, … gzip, bzip2, aes, … CSV, JSON, pcap, … buffer buffer buffer buffer Filter plugin Filter plugin Filter plugins recordsrecords Executor plugin 30 records config
- 31. Embulk and Java Embulk core is written in Java mainly for performance Embulk plugins: are loaded over API based on JRuby are written in JRuby or Java JRuby for early release Java for performance
- 32. InputPlugin module Embulk class InputExample < InputPlugin Plugin.register_input('example', self) def self.transaction(config, &control) # read config task = { 'message' => config.param('message', :string, default: nil) } threads = config.param('threads', :int, default: 2) columns = [ Column.new(0, 'col0', :long), Column.new(1, 'col1', :double), Column.new(2, 'col2', :string), ] # BEGIN here commit_reports = yield(task, columns, threads) # COMMIT here puts "Example input finished" return {} end def run(task, schema, index, page_builder) puts "Example input thread #{@index}…" 10.times do |i| @page_builder.add([i, 10.0, "example"]) end @page_builder.finish commit_report = { } return commit_report end end end
- 33. OutputPlugin module Embulk class OutputExample < OutputPlugin Plugin.register_output('example', self) def self.transaction( config, schema, processor_count, &control) # read config task = { 'message' => config.param('message', :string, default: "record") } puts "Example output started." commit_reports = yield(task) puts "Example output finished. Commit reports = #{commit_reports.to_json}" return {} end def initialize(task, schema, index) puts "Example output thread #{index}..." super @message = task.prop('message', :string) @records = 0 end def add(page) page.each do |record| hash = Hash[schema.names.zip(record)] puts "#{@message}: #{hash.to_json}" @records += 1 end end def finish end def abort end def commit commit_report = { "records" => @records } return commit_report end end end
- 34. Plugin management: Norikra Esper instance Engine Plugin management UDF Listener plugins as gems plugin loader written in JRuby Java JRuby
- 35. Plugin management: Embulk Embulk core Plugin management input/output/filter parser/formatter Java JRuby decoder/encoder file-input/output executor plugins as gems plugin loader written in JRuby
- 36. Pluggable software on JVM & Java API Java? Scala? Clojure? JRuby?: JRuby Plugin packaging: jar? gem?: gem rubygem.org >>> maven central (or others) especially for plugin authors Plugin loader: Class Loader? "require"?: require
- 37. JRuby in Japan Not so many users :( CRuby is super major software in Japan Java -> Ruby -> Scala? Golang?
- 38. Make your software pluggable. Make eco-system&community. with JRuby! Thanks!