Making the case for write-optimized database algorithms / Mark Callaghan (Facebook)
4 likes736 views
This document details the advantages of write-optimized database algorithms, particularly focusing on the performance of RocksDB and its variants (MyRocks and MongoRocks) compared to traditional B-tree based systems. It emphasizes the importance of balancing read, write, and space efficiency in database operations and presents performance benchmarks demonstrating the higher throughput and lower space requirements of RocksDB. Additionally, it discusses the adaptability of algorithms to meet varying efficiency goals and highlights ongoing developments in open-source database technologies.
1 of 26
Downloaded 11 times
Ad
Recommended
Spilo, отказоустойчивый PostgreSQL кластер / Oleksii Kliukin (Zalando SE)
Spilo, отказоустойчивый PostgreSQL кластер / Oleksii Kliukin (Zalando SE)Ontico The document outlines Zalando's implementation of a highly available PostgreSQL cluster using Spilo and Patroni, managing over 150 production databases and addressing common database failures. It discusses the use of AWS for cloud services, microservices architecture, and the responsibilities of development teams regarding database infrastructure and failover management. Key features include automated failover, streaming replication, and the use of etcd for distributed consensus and state management.
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)
HighLoad Solutions On MySQL / Xiaobin Lin (Alibaba)Ontico The document summarizes several issues encountered with high load on Alibaba MySQL databases and solutions implemented:
1) Hotspot updating of a single row caused deadlocks; implementing queueing on the primary key resolved this.
2) Unexpected long transactions under high load led to clients waiting long periods; committing transactions early where possible addressed this.
3) More than 50,000 active threads overwhelmed MySQL's capabilities; implementing actions based on low and high thread thresholds helped.
MySQL Replication — Advanced Features / Петр Зайцев (Percona)
MySQL Replication — Advanced Features / Петр Зайцев (Percona)Ontico This presentation covers advanced MySQL replication features, emphasizing the importance of replication for availability, scalability, and performance. It outlines the evolution of MySQL replication through various versions, discusses different replication types such as synchronous and asynchronous, and introduces alternatives like MySQL Group Replication. Additionally, it addresses the complexities of multi-master setups and highlights traffic management solutions for better replication configuration and failover management.
Tarantool: как сэкономить миллион долларов на базе данных на высоконагруженно...
Tarantool: как сэкономить миллион долларов на базе данных на высоконагруженно...Ontico The document discusses strategies for optimizing database management systems (DBMS) to save costs, particularly through techniques like replication, sharding, and using smart caches. It highlights the importance of balancing data consistency and performance while managing workloads, particularly in high-traffic environments like mail.ru. A key case study demonstrates a significant cost reduction by switching to Tarantool, culminating in a savings of over $1 million.
Frontera распределенный робот для обхода веба в больших объемах / Александр С...
Frontera распределенный робот для обхода веба в больших объемах / Александр С...Ontico The document discusses the 'frontera' open-source web crawling framework developed by Alexander Sibiryakov at Scrapinghub, highlighting its capabilities in large-scale web data collection. It details various web crawling strategies, infrastructure solutions, and challenges encountered during operations, including handling large volumes of data and optimizing performance. The document also mentions practical applications such as news analysis, plagiarism detection, sentiment analysis, and due diligence tasks.
Understanding and tuning WiredTiger, the new high performance database engine...
Understanding and tuning WiredTiger, the new high performance database engine...Ontico The document discusses WiredTiger, MongoDB's high-performance database engine, covering its architecture, configuration options, and performance tuning strategies. It highlights MongoDB's evolution, storage engine choices, and features like multi-statement transactions and advanced compression methods. Performance optimization strategies with WiredTiger include fitting the working set in cache and leveraging SSDs for efficiency.
Как Web-акселератор акселерирует ваш сайт / Александр Крижановский (Tempesta ...
Как Web-акселератор акселерирует ваш сайт / Александр Крижановский (Tempesta ...Ontico The document discusses web-acceleration technologies developed by Tempesta Technologies, highlighting their hybrid HTTP accelerator and firewall for DDoS mitigation. It covers various caching strategies, SSL acceleration, and web application firewall capabilities, along with performance considerations of different technologies. Additionally, it provides insights into configuring and managing these systems for enhanced web performance and security.
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...
Как понять, что происходит на сервере? / Александр Крижановский (NatSys Lab.,...Ontico The document discusses monitoring server performance using various Linux tools and commands, focusing on an Nginx server's connection and resource utilization metrics. It highlights specific commands to check network traffic, CPU usage, and perform debugging with GDB, while analyzing bottlenecks and system calls. Key findings indicate the main time spent in kernel mode and the identification of performance issues through profilings, such as connection tracking and server response times.
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Tanya Denisyuk The document discusses the benefits of using Linux for running PostgreSQL databases, highlighting its active community support, performance features, and overall stability. It covers various technical aspects such as CPU scheduling, virtual memory management, and file systems, providing advice on optimizing configurations for PostgreSQL. The presentation emphasizes the importance of adapting Linux for specific workloads and suggests a testing methodology for performance improvements.
My Sql Performance In A Cloud
My Sql Performance In A CloudSky Jian The document discusses deploying MySQL in cloud environments and focuses on improving InnoDB performance for workloads with high IO demands. It provides benchmarks comparing the performance of MySQL using direct attached storage versus network attached storage, as well as the impact of virtualization. Several techniques are suggested for tuning MySQL and InnoDB for IO-bound workloads, including various configuration options and patches that improve concurrency and throughput.
Scylla Summit 2018: In-Memory Scylla - When Fast Storage is Not Fast Enough
Scylla Summit 2018: In-Memory Scylla - When Fast Storage is Not Fast EnoughScyllaDB Gleb Natapov presents on using in-memory storage in ScyllaDB to provide faster access times compared to disk-based storage. He discusses how ScyllaDB uses SSTables that are stored on disk by default, which can incur latency penalties during cache misses. Natapov introduces new in-memory and mirror file types that allow storing SSTable contents directly in RAM to avoid disk access latency. He demonstrates how to configure ScyllaDB to reserve RAM for in-memory storage and to create tables specifying the in-memory option to gain predictably low latency performance.
High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011Tim Y The document discusses building a high-performance Weibo system, emphasizing the role of various databases and caching mechanisms including MySQL, NoSQL options like MongoDB and Redis. It highlights the shift from traditional database systems to specialized solutions and the importance of efficient data handling through APIs and data formats like protocol buffers. Additionally, it touches on architectural strategies for managing relational and non-relational data alongside caching to enhance application performance.
My Sql Performance On Ec2
My Sql Performance On Ec2MySQLConference This document discusses optimizing MySQL performance when deployed in cloud environments. Some key points:
1) Deploying MySQL in the cloud presents new challenges like increased IO latency from network storage, but also opportunities to scale horizontally across multiple servers.
2) InnoDB performance is particularly important to optimize, through techniques like reducing mutex contention, increasing IO throughput, and using patches that improve handling of higher latency storage.
3) Benchmarks show that direct attached storage outperforms network storage, and optimizations like IO threads and capacity tuning can help mitigate higher latencies of network storage. Virtualization also incurs some performance overhead.
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)Ontico The document provides an overview of PostgreSQL configuration, including essential parameters and tuning advice for optimization. It emphasizes the importance of proper settings, such as memory management and connection limits, to improve database performance. Additionally, it introduces postgresqlco.nf, a web tool designed to assist users in creating and managing PostgreSQL configurations more efficiently.
Optimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at Localyticsandrew311 The document presents insights from Andrew Rollins on optimizing MongoDB, sharing practical tips on managing documents and indexes, handling fragmentation, and effective migrations, particularly in a cloud environment. Key strategies include using Bindata for identifiers, minimizing index entries, and optimizing shard keys for better locality. The discussion emphasizes the importance of hardware considerations and operational practices for maintaining high performance in MongoDB applications.
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...Ontico The document discusses Tarantool as a microservices platform, highlighting its performance capabilities, including handling over 100k requests per second per CPU core. It outlines various features such as SQL support, smart caching, complex queues, and its ability to handle OAuth tokens and user requests efficiently. Additionally, it emphasizes the importance of accessing data quickly and computing close to where the data is stored for optimal performance.
Redis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs TalksRedis Labs The document discusses challenges and advancements in managing Redis databases, highlighting issues like high database turnover, node failures, and the need for custom modifications to the platform. It details efforts to extend Redis functionality to flash storage, focusing on performance, resource management, and maintaining compatibility while addressing persistent storage solutions. Future enhancements aim to optimize memory management and data replication processes within clustered environments.
Scaling MongoDB in the cloud with Microsoft Azure
Scaling MongoDB in the cloud with Microsoft AzureIvan Fioravanti The document discusses the implementation of MongoDB in the cloud using Microsoft Azure, detailing the configuration of replica sets and scaling strategies. It highlights potential points of failure within a single data center and provides insights into automatic recovery setups. Additionally, it includes helpful resources and links for further reading and configuration tutorials.
No sql but even less security
No sql but even less securityiammutex The document discusses security issues related to NoSQL databases. It begins with an overview of eventual consistency and how it can allow reading stale data in some situations. It then covers how NoSQL databases often have open ports and lack authentication by default. It also discusses how REST APIs can enable cross-site request forgery attacks. Additionally, it shows how injection is possible in some NoSQL databases using $where queries or request parameters. The document aims to raise awareness of security challenges in NoSQL databases compared to traditional SQL databases.
Day 2 General Session Presentations RedisConf
Day 2 General Session Presentations RedisConfRedis Labs The document discusses new memory technologies like persistent memory and their implications. It provides latency and bandwidth numbers for different memory types and notes that heterogeneous memory systems using tiers of DRAM and NVM provide opportunities for better performance and cost. Examples are given of key-value stores and databases leveraging NVM to achieve high performance while reducing costs. The talk also discusses how new distributed data structures like CRDTs could be used across servers with shared memory.
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...Ontico Доклад о высокопроизводительном инференсе глубоких нейронных сетей на GPU с использованием TensorRT от NVIDIA освещает ключевые аспекты применения GPU и TensorRT для повышения эффективности инференса. В нем рассматриваются сценарии развертывания как в гипермасштабах, так и встроенных системах, с акцентом на оптимизацию производительности и использование технологий, таких как фузия слоев и автонастройка. Приводятся данные о производительности различных GPU и упоминается о поддержке INT8 для увеличения эффективности инференса.
Ужимай и властвуй алгоритмы компрессии в базах данных / Петр Зайцев (Percona)
Ужимай и властвуй алгоритмы компрессии в базах данных / Петр Зайцев (Percona)Ontico The document discusses data compression techniques in open-source databases, emphasizing its importance in managing the exponential growth of data. Various methods and algorithms for compression are analyzed, including lossless and lossy approaches, as well as practical implementations in MySQL, MongoDB, and PostgreSQL. It concludes that compression is crucial for improving storage efficiency and performance in modern databases.
微博cache设计谈
微博cache设计谈Tim Y The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs like followers and following lists. The caches aim to optimize for fast retrieval of recent home timelines and individual user feeds. Mutexes are used to synchronize cache updates between servers.
Ceph Day Beijing - Our journey to high performance large scale Ceph cluster a...
Ceph Day Beijing - Our journey to high performance large scale Ceph cluster a...Danielle Womboldt This document discusses optimizing performance in large scale CEPH clusters at Alibaba. It describes two use models for writing data in CEPH and improvements made to recovery performance by implementing partial and asynchronous recovery. It also details fixes made to bugs that caused data loss or inconsistency. Additionally, it proposes offloading transaction queueing from PG workers to improve performance by leveraging asynchronous transaction workers and evaluating this approach through bandwidth testing.
MongoDB World 2015 - A Technical Introduction to WiredTiger
MongoDB World 2015 - A Technical Introduction to WiredTigerWiredTiger This document provides a technical introduction to WiredTiger, a storage engine for MongoDB, highlighting its features like document-level concurrency, in-memory performance, and multi-core scalability. It discusses the architecture, including multi-version concurrency control, checksums, and compression methods, as well as the benefits of durability without journaling. The presentation outlines future plans for WiredTiger, including improved workload tuning, adding encryption, and enhancing transactional semantics.
[Pgday.Seoul 2018] PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개 apposha
[Pgday.Seoul 2018] PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개 apposhaPgDay.Seoul This document introduces AppOS, an operating system specialized for database performance. It discusses how AppOS improves on Linux by being more optimized for database workloads through techniques like specialized caching, I/O scheduling based on database priorities, and atomic writes. It also explains how AppOS is portable, high performing, and extensible to support different databases through its modular design. Future plans include improving cache management, parallel query optimization, and cooperative CPU scheduling.
Troubleshooting redis
Troubleshooting redisDaeMyung Kang This document discusses troubleshooting Redis. Some key points:
- Redis is single-threaded, so commands like KEYS, FlushAll, and deleting large collections can be slow. It's better to use SCAN instead of KEYS.
- Creating Redis database snapshots (RDB files) and rewriting the append-only file (AOF) can cause high disk I/O and CPU usage. It's best to disable automatic rewrites.
- Monitoring memory usage and fragmentation is important to avoid performance issues. The maxmemory setting also needs monitoring to prevent out-of-memory errors.
- Network and replication failures need solutions like DNS failover or using Zookeeper for coordination to maintain high availability of Redis
NoSQL внутри SQL: приземленные вопросы практического применения / Дмитрий До...
NoSQL внутри SQL: приземленные вопросы практического применения / Дмитрий До...Ontico Документ рассматривает вопросы интеграции и производительности при использовании NoSQL и SQL хранилищ данных, предлагая различные подходы к хранению данных в реляционных базах данных. Описываются примеры использования JSON и JSONB в PostgreSQL и MySQL, а также методы загрузки и тестирования производительности. Обсуждаются факторы, влияющие на производительность, такие как структура данных и индексы.
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...Ontico The document discusses Full Text Search (FTS) capabilities in PostgreSQL, highlighting its built-in features and flexibility, but also addressing limitations such as slow ranking and lack of phrase search. It outlines the evolution of text search functionalities in PostgreSQL from 2000 to 2016 and introduces enhancements with extensions like RUM that improve indexing and query performance. Overall, it portrays FTS as a customizable tool that integrates with other database functionalities, suitable for various types of documents and queries.
Life Of A Dirty Page Inno Db Disk Io
Life Of A Dirty Page Inno Db Disk IoSky Jian The document discusses the high-level and medium-level architecture of InnoDB and some of its performance issues for handling high IOPS workloads. Specifically, it notes that:
1) InnoDB was originally designed for single disk servers but now must handle servers with many cores and 10k IOPS, and some of its rate limits and designs are outdated for modern hardware.
2) Background threads in InnoDB perform IO work to flush dirty pages and merge insert buffer entries, but the rate limits for these tasks are too low for high IOPS servers.
3) InnoDB has problems enforcing its maximum dirty page and insert buffer limits, providing visibility into prefetching effectiveness, and handling large merged IO requests
More Related Content
What's hot (19)
Алексей Лесовский "Тюнинг Linux для баз данных. "
Алексей Лесовский "Тюнинг Linux для баз данных. "Tanya Denisyuk The document discusses the benefits of using Linux for running PostgreSQL databases, highlighting its active community support, performance features, and overall stability. It covers various technical aspects such as CPU scheduling, virtual memory management, and file systems, providing advice on optimizing configurations for PostgreSQL. The presentation emphasizes the importance of adapting Linux for specific workloads and suggests a testing methodology for performance improvements.
My Sql Performance In A Cloud
My Sql Performance In A CloudSky Jian The document discusses deploying MySQL in cloud environments and focuses on improving InnoDB performance for workloads with high IO demands. It provides benchmarks comparing the performance of MySQL using direct attached storage versus network attached storage, as well as the impact of virtualization. Several techniques are suggested for tuning MySQL and InnoDB for IO-bound workloads, including various configuration options and patches that improve concurrency and throughput.
Scylla Summit 2018: In-Memory Scylla - When Fast Storage is Not Fast Enough
Scylla Summit 2018: In-Memory Scylla - When Fast Storage is Not Fast EnoughScyllaDB Gleb Natapov presents on using in-memory storage in ScyllaDB to provide faster access times compared to disk-based storage. He discusses how ScyllaDB uses SSTables that are stored on disk by default, which can incur latency penalties during cache misses. Natapov introduces new in-memory and mirror file types that allow storing SSTable contents directly in RAM to avoid disk access latency. He demonstrates how to configure ScyllaDB to reserve RAM for in-memory storage and to create tables specifying the in-memory option to gain predictably low latency performance.
High Performance Weibo QCon Beijing 2011
High Performance Weibo QCon Beijing 2011Tim Y The document discusses building a high-performance Weibo system, emphasizing the role of various databases and caching mechanisms including MySQL, NoSQL options like MongoDB and Redis. It highlights the shift from traditional database systems to specialized solutions and the importance of efficient data handling through APIs and data formats like protocol buffers. Additionally, it touches on architectural strategies for managing relational and non-relational data alongside caching to enhance application performance.
My Sql Performance On Ec2
My Sql Performance On Ec2MySQLConference This document discusses optimizing MySQL performance when deployed in cloud environments. Some key points:
1) Deploying MySQL in the cloud presents new challenges like increased IO latency from network storage, but also opportunities to scale horizontally across multiple servers.
2) InnoDB performance is particularly important to optimize, through techniques like reducing mutex contention, increasing IO throughput, and using patches that improve handling of higher latency storage.
3) Benchmarks show that direct attached storage outperforms network storage, and optimizations like IO threads and capacity tuning can help mitigate higher latencies of network storage. Virtualization also incurs some performance overhead.
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)
PostgreSQL Configuration for Humans / Alvaro Hernandez (OnGres)Ontico The document provides an overview of PostgreSQL configuration, including essential parameters and tuning advice for optimization. It emphasizes the importance of proper settings, such as memory management and connection limits, to improve database performance. Additionally, it introduces postgresqlco.nf, a web tool designed to assist users in creating and managing PostgreSQL configurations more efficiently.
Optimizing MongoDB: Lessons Learned at Localytics
Optimizing MongoDB: Lessons Learned at Localyticsandrew311 The document presents insights from Andrew Rollins on optimizing MongoDB, sharing practical tips on managing documents and indexes, handling fragmentation, and effective migrations, particularly in a cloud environment. Key strategies include using Bindata for identifiers, minimizing index entries, and optimizing shard keys for better locality. The discussion emphasizes the importance of hardware considerations and operational practices for maintaining high performance in MongoDB applications.
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...
Tarantool как платформа для микросервисов / Антон Резников, Владимир Перепели...Ontico The document discusses Tarantool as a microservices platform, highlighting its performance capabilities, including handling over 100k requests per second per CPU core. It outlines various features such as SQL support, smart caching, complex queues, and its ability to handle OAuth tokens and user requests efficiently. Additionally, it emphasizes the importance of accessing data quickly and computing close to where the data is stored for optimal performance.
Redis Developers Day 2014 - Redis Labs Talks
Redis Developers Day 2014 - Redis Labs TalksRedis Labs The document discusses challenges and advancements in managing Redis databases, highlighting issues like high database turnover, node failures, and the need for custom modifications to the platform. It details efforts to extend Redis functionality to flash storage, focusing on performance, resource management, and maintaining compatibility while addressing persistent storage solutions. Future enhancements aim to optimize memory management and data replication processes within clustered environments.
Scaling MongoDB in the cloud with Microsoft Azure
Scaling MongoDB in the cloud with Microsoft AzureIvan Fioravanti The document discusses the implementation of MongoDB in the cloud using Microsoft Azure, detailing the configuration of replica sets and scaling strategies. It highlights potential points of failure within a single data center and provides insights into automatic recovery setups. Additionally, it includes helpful resources and links for further reading and configuration tutorials.
No sql but even less security
No sql but even less securityiammutex The document discusses security issues related to NoSQL databases. It begins with an overview of eventual consistency and how it can allow reading stale data in some situations. It then covers how NoSQL databases often have open ports and lack authentication by default. It also discusses how REST APIs can enable cross-site request forgery attacks. Additionally, it shows how injection is possible in some NoSQL databases using $where queries or request parameters. The document aims to raise awareness of security challenges in NoSQL databases compared to traditional SQL databases.
Day 2 General Session Presentations RedisConf
Day 2 General Session Presentations RedisConfRedis Labs The document discusses new memory technologies like persistent memory and their implications. It provides latency and bandwidth numbers for different memory types and notes that heterogeneous memory systems using tiers of DRAM and NVM provide opportunities for better performance and cost. Examples are given of key-value stores and databases leveraging NVM to achieve high performance while reducing costs. The talk also discusses how new distributed data structures like CRDTs could be used across servers with shared memory.
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...
Высокопроизводительный инференс глубоких сетей на GPU с помощью TensorRT / Ма...Ontico Доклад о высокопроизводительном инференсе глубоких нейронных сетей на GPU с использованием TensorRT от NVIDIA освещает ключевые аспекты применения GPU и TensorRT для повышения эффективности инференса. В нем рассматриваются сценарии развертывания как в гипермасштабах, так и встроенных системах, с акцентом на оптимизацию производительности и использование технологий, таких как фузия слоев и автонастройка. Приводятся данные о производительности различных GPU и упоминается о поддержке INT8 для увеличения эффективности инференса.
Ужимай и властвуй алгоритмы компрессии в базах данных / Петр Зайцев (Percona)
Ужимай и властвуй алгоритмы компрессии в базах данных / Петр Зайцев (Percona)Ontico The document discusses data compression techniques in open-source databases, emphasizing its importance in managing the exponential growth of data. Various methods and algorithms for compression are analyzed, including lossless and lossy approaches, as well as practical implementations in MySQL, MongoDB, and PostgreSQL. It concludes that compression is crucial for improving storage efficiency and performance in modern databases.
微博cache设计谈
微博cache设计谈Tim Y The document discusses caching strategies for social media feeds. It describes using different caches for hot, recent content (inbox cache), older content for followers (outbox vector), and archived historic content (archive cache). It also discusses caching social graphs like followers and following lists. The caches aim to optimize for fast retrieval of recent home timelines and individual user feeds. Mutexes are used to synchronize cache updates between servers.
Ceph Day Beijing - Our journey to high performance large scale Ceph cluster a...
Ceph Day Beijing - Our journey to high performance large scale Ceph cluster a...Danielle Womboldt This document discusses optimizing performance in large scale CEPH clusters at Alibaba. It describes two use models for writing data in CEPH and improvements made to recovery performance by implementing partial and asynchronous recovery. It also details fixes made to bugs that caused data loss or inconsistency. Additionally, it proposes offloading transaction queueing from PG workers to improve performance by leveraging asynchronous transaction workers and evaluating this approach through bandwidth testing.
MongoDB World 2015 - A Technical Introduction to WiredTiger
MongoDB World 2015 - A Technical Introduction to WiredTigerWiredTiger This document provides a technical introduction to WiredTiger, a storage engine for MongoDB, highlighting its features like document-level concurrency, in-memory performance, and multi-core scalability. It discusses the architecture, including multi-version concurrency control, checksums, and compression methods, as well as the benefits of durability without journaling. The presentation outlines future plans for WiredTiger, including improved workload tuning, adding encryption, and enhancing transactional semantics.
[Pgday.Seoul 2018] PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개 apposha
[Pgday.Seoul 2018] PostgreSQL 성능을 위해 개발된 라이브러리 OS 소개 apposhaPgDay.Seoul This document introduces AppOS, an operating system specialized for database performance. It discusses how AppOS improves on Linux by being more optimized for database workloads through techniques like specialized caching, I/O scheduling based on database priorities, and atomic writes. It also explains how AppOS is portable, high performing, and extensible to support different databases through its modular design. Future plans include improving cache management, parallel query optimization, and cooperative CPU scheduling.
Troubleshooting redis
Troubleshooting redisDaeMyung Kang This document discusses troubleshooting Redis. Some key points:
- Redis is single-threaded, so commands like KEYS, FlushAll, and deleting large collections can be slow. It's better to use SCAN instead of KEYS.
- Creating Redis database snapshots (RDB files) and rewriting the append-only file (AOF) can cause high disk I/O and CPU usage. It's best to disable automatic rewrites.
- Monitoring memory usage and fragmentation is important to avoid performance issues. The maxmemory setting also needs monitoring to prevent out-of-memory errors.
- Network and replication failures need solutions like DNS failover or using Zookeeper for coordination to maintain high availability of Redis
Viewers also liked (20)
NoSQL внутри SQL: приземленные вопросы практического применения / Дмитрий До...
NoSQL внутри SQL: приземленные вопросы практического применения / Дмитрий До...Ontico Документ рассматривает вопросы интеграции и производительности при использовании NoSQL и SQL хранилищ данных, предлагая различные подходы к хранению данных в реляционных базах данных. Описываются примеры использования JSON и JSONB в PostgreSQL и MySQL, а также методы загрузки и тестирования производительности. Обсуждаются факторы, влияющие на производительность, такие как структура данных и индексы.
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...Ontico The document discusses Full Text Search (FTS) capabilities in PostgreSQL, highlighting its built-in features and flexibility, but also addressing limitations such as slow ranking and lack of phrase search. It outlines the evolution of text search functionalities in PostgreSQL from 2000 to 2016 and introduces enhancements with extensions like RUM that improve indexing and query performance. Overall, it portrays FTS as a customizable tool that integrates with other database functionalities, suitable for various types of documents and queries.
Life Of A Dirty Page Inno Db Disk Io
Life Of A Dirty Page Inno Db Disk IoSky Jian The document discusses the high-level and medium-level architecture of InnoDB and some of its performance issues for handling high IOPS workloads. Specifically, it notes that:
1) InnoDB was originally designed for single disk servers but now must handle servers with many cores and 10k IOPS, and some of its rate limits and designs are outdated for modern hardware.
2) Background threads in InnoDB perform IO work to flush dirty pages and merge insert buffer entries, but the rate limits for these tasks are too low for high IOPS servers.
3) InnoDB has problems enforcing its maximum dirty page and insert buffer limits, providing visibility into prefetching effectiveness, and handling large merged IO requests
Open Source SQL-базы данных вступили в эру миллионов запросов в секунду / Фед...
Open Source SQL-базы данных вступили в эру миллионов запросов в секунду / Фед...Ontico Документ обсуждает производительность и оптимизацию различных реляционных баз данных, особенно MySQL и PostgreSQL, с акцентом на сравнительный анализ их возможностей. Проводятся тесты производительности, детально описываются проблемы и трудности, с которыми сталкиваются пользователи, а также предлагются различные конфигурационные решения для улучшения работы систем. Также упоминаются достижения и новшества в области разработки и поддержки данных СУБД с упором на использование ресурсов и управление транзакциями.
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)Ontico This document discusses the advantages and functionalities of non-relational storage within PostgreSQL, including support for various data types such as arrays, range types, geometry, XML, JSON, and JSONB. It highlights how these types can be utilized for better query performance, flexibility, and indexing, while comparing them to traditional relational storage methods. The presentation includes practical SQL examples to demonstrate data manipulation and querying techniques using these non-relational features.
Отладка производительности приложения на Erlang / Максим Лапшин (Erlyvideo)
Отладка производительности приложения на Erlang / Максим Лапшин (Erlyvideo)Ontico Документ описывает диагностику и оптимизацию производительности многопоточных приложений на Erlang. Автор делится опытом анализа bottleneck'ов с использованием различных инструментов, таких как lcnt, msacc и instrument, а также обсуждает проблемы с мьютексами и распределением задач между акторами. В результате были обнаружены недостатки в аллокации ресурсов и способы их исправления.
Хранение данных на виниле / Константин Осипов (tarantool.org)
Хранение данных на виниле / Константин Осипов (tarantool.org)Ontico Доклад рассматривает архитектуру и функциональные возможности системы хранения данных на виниле, включая структуру log structured merge tree и реализацию vinyl engine в Tarantool. Обсуждаются преимущества и недостатки LSM-деревьев, проблемы с производительностью, а также методы уменьшения задержки чтения и записи данных. В заключение представляются аспекты репликации и ограничения использования системы.
Профилирование кода на C/C++ в *nix-системах / Александр Алексеев (Postgres P...
Профилирование кода на C/C++ в *nix-системах / Александр Алексеев (Postgres P...Ontico Документ охватывает различные методы профилирования кода на C/C++ в системах *nix, включая инструменты, такие как gettimeofday, strace, gprof, и dtrace. Описываются распространенные ошибки при проведении бенчмарков, такие как неповторяемость и неправильные настройки. Также рассматриваются плюсы и минусы различных инструментов, а также рекомендации по их использованию.
Peeking into the Black Hole Called PL/PGSQL - the New PL Profiler / Jan Wieck...
Peeking into the Black Hole Called PL/PGSQL - the New PL Profiler / Jan Wieck...Ontico The document discusses profiling in PostgreSQL, explaining how profiling can help optimize program performance by collecting execution statistics and analyzing query behavior. It describes various tools and methods, including pg_stat_statements and plprofiler, for identifying performance issues, particularly in PL/pgSQL functions. The document also emphasizes the challenges of profiling in production environments and provides examples of how to generate and analyze performance reports.
Archival Disc на смену Blu-ray: построение архивного хранилища на оптических ...
Archival Disc на смену Blu-ray: построение архивного хранилища на оптических ...Ontico Сеть архивов русского искусства (RAAN) представляет собой онлайн-портал, который объединяет архивы русского современного искусства по всему миру и предоставляет исследователям доступ к информации о русском искусстве. Проект включает оцифровку и каталогизацию материалов из различных архивов, а также сотрудничество с международными музеями и университетами. Одним из ключевых компонентов является архив азиатского искусства, который содержит более 50,000 единиц хранения и продолжает развиваться.
Как смигрировать 50Пб в 32 без даунтайма? / Альберт Галимов, Андрей Сумин (Ma...
Как смигрировать 50Пб в 32 без даунтайма? / Альберт Галимов, Андрей Сумин (Ma...Ontico Документ описывает процесс миграции данных объемом 50 ПБ на платформе почтовых услуг Mail.ru, где освещаются детали структуры базы данных и управления файлами. В нем рассматриваются различные аспекты управления файлами, включая удаление, восстановление и проверку целостности данных. Основное внимание уделяется эффективному управлению метаданными и сохранению консистентности данных при проведении миграции.
Как мы сделали PHP 7 в два раза быстрее PHP 5 / Дмитрий Стогов (Zend Technolo...
Как мы сделали PHP 7 в два раза быстрее PHP 5 / Дмитрий Стогов (Zend Technolo...Ontico The document discusses the evolution and performance enhancements of PHP, particularly the transition from PHP 5 to PHP 7, led by Dmitry Stogov from Zend Technologies. It highlights the significant improvements in speed, memory usage, and new features like scalar type declarations and better error handling introduced in PHP 7. The development of Just-In-Time (JIT) compilation and the phpng project are also covered as pivotal efforts to enhance PHP's efficiency and versatility in modern applications.
Долгожданный релиз pg_pathman 1.0 / Александр Коротков, Дмитрий Иванов (Post...
Долгожданный релиз pg_pathman 1.0 / Александр Коротков, Дмитрий Иванов (Post...Ontico Документ описывает расширение pg_pathman для PostgreSQL, обеспечивающее эффективное управление секционированием данных. Оно поддерживает как хешевое, так и диапазонное секционирование, а также предлагает автоматическое и ручное управление секциями, что значительно ускоряет запросы. В нем также рассматриваются преимущества и недостатки, а также возможные решения для улучшения производительности при работе с большими объемами данных.
Performance Schema in MySQL (Danil Zburivsky)
Performance Schema in MySQL (Danil Zburivsky)Ontico The document discusses the Performance Schema feature in MySQL 5.5, which instruments and collects data about internal operations to help identify performance bottlenecks. It is implemented as a storage engine that collects data about events like query execution steps, locks, I/O, and threads into tables that provide visibility into where the server spends its time. This helps address the lack of good instrumentation previously available in MySQL for performance tuning.
MySQL Troubleshooting with the Performance Schema
MySQL Troubleshooting with the Performance SchemaSveta Smirnova This document discusses using the Performance Schema in MySQL to troubleshoot performance issues. It provides an overview of the Performance Schema and what information it collects. It then discusses how to use specific Performance Schema tables like events_statements_history_long, events_stages_history_long, and others to identify statements that examine too many rows, issues with index usage, and which internal operations are taking a long time. The document provides examples of queries to run and what to look for in the Performance Schema output to help troubleshoot and optimize SQL statements.
The MySQL Performance Schema & New SYS Schema
The MySQL Performance Schema & New SYS SchemaTed Wennmark The document discusses the Performance Schema and Sys Schema in MySQL, detailing their features and enhancements in versions 5.5 to 5.7, as well as their capabilities for profiling and monitoring server performance. It highlights the types of tables available, their uses in configuring and analyzing performance data, and improvements made in MySQL 5.7. Additionally, it provides an overview of MySQL Workbench and its integration with the Sys Schema to facilitate performance analysis.
MySQL Performance - SydPHP October 2011
MySQL Performance - SydPHP October 2011Graham Weldon The document discusses various strategies for optimizing MySQL performance beyond basic indexing and query optimization, such as tuning MySQL configurations and managing server resources. It advises on memory management, avoiding disk I/O bottlenecks, and the use of in-memory solutions and caching mechanisms to enhance efficiency. The document also highlights monitoring tools and best practices for assessing and improving MySQL and application performance.
Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...
Девять кругов ада или PostgreSQL Vacuum / Алексей Лесовский (PostgreSQL-Consu...Ontico Документ обсуждает важность процесса вакуума в PostgreSQL, указывая на его влияние на производительность системы. Приводятся детали настройки вакуума, включая параметры, которые влияют на эффективность команды и настройки автоматического вакуума. Описываются аспекты работы MVCC, фоновых процессов и стратегий обработки таблиц для оптимизации управления пространством.
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)
PostgreSQL @Alibaba Cloud / Xianming Dou (Alibaba Cloud)Ontico The document outlines the process of creating a PostgreSQL instance on Alibaba Cloud, focusing on aspects like high availability, monitoring, and management features. It details functionalities such as auto-repair, backup scheduling, privilege management, and how to handle out-of-memory (OOM) situations and I/O hangs. Additionally, it highlights the importance of maintaining connection control and replication across master and slave instances.
LuaJIT как основа для сервера приложений - проблемы и решения / Игорь Эрлих (...
LuaJIT как основа для сервера приложений - проблемы и решения / Игорь Эрлих (...Ontico Документ описывает Luajit, среду исполнения для Lua 5.1 с трассирующим JIT-компилятором, которая обеспечивает значительно более высокую скорость выполнения по сравнению с официальным интерпретатором. Обсуждаются проблемы с ограничениями по памяти и сборкой мусора, а также предлагаются возможные решения, включая снижение количества потоков и переход к процессам. В заключение подчеркивается высокое преимущество Luajit в производительности, хотя существуют реальные проблемы с управлением памятью и уязвимостями.
Ad
Similar to Making the case for write-optimized database algorithms / Mark Callaghan (Facebook) (20)
MyRocks Deep Dive
MyRocks Deep DiveYoshinori Matsunobu The document is a detailed presentation on MyRocks, a storage engine for MySQL developed by Facebook, focusing on its architecture, efficiency, and advantages over traditional InnoDB. It covers various topics including MyRocks' data structures, query optimizations, compression techniques, and the performance benefits for write-heavy applications. The tutorial is aimed at MySQL users who wish to integrate and leverage MyRocks capabilities.
Migrating from InnoDB and HBase to MyRocks at Facebook
Migrating from InnoDB and HBase to MyRocks at FacebookMariaDB plc Migrating large databases at Facebook from InnoDB to MyRocks and HBase to MyRocks resulted in significant space savings of 2-4x and improved write performance by up to 10x. Various techniques were used for the migrations such as creating new MyRocks instances without downtime, loading data efficiently, testing on shadow instances, and promoting MyRocks instances as masters. Ongoing work involves optimizations like direct I/O, dictionary compression, parallel compaction, and dynamic configuration changes to further improve performance and efficiency.
MyRocks introduction and production deployment
MyRocks introduction and production deploymentYoshinori Matsunobu The document presents an introduction to MyRocks, a MySQL storage engine developed by Facebook that integrates RocksDB, aiming to improve space efficiency and lower write amplification compared to traditional InnoDB databases. It discusses the architecture, advantages, and challenges faced during the migration and production deployment of MyRocks at Facebook, emphasizing the need for monitoring and maintaining performance under large-scale use. Technical aspects, including configuration, query execution, and data verification during migration, are also covered to illustrate how MyRocks addresses existing limitations in InnoDB.
MySQL Storage Engines - which do you use? TokuDB? MyRocks? InnoDB?
MySQL Storage Engines - which do you use? TokuDB? MyRocks? InnoDB?Sveta Smirnova The document discusses the evolution of MySQL storage engines, detailing the introduction of various engines like MyISAM, InnoDB, TokuDB, and MyRocks, and their respective features and optimizations. Key changes began with version 3.23 and continued to evolve, particularly notable in version 5.1 with the introduction of a pluggable storage engine API. It highlights the advantages and use cases of these engines, emphasizing flexibility and performance for different workloads.
MyRocks in MariaDB | M18
MyRocks in MariaDB | M18Sergey Petrunya This document discusses MyRocks, a storage engine for MariaDB that uses RocksDB as its backend. It begins by explaining the limitations of InnoDB that MyRocks aims to address, such as high write and space amplification. It then describes how RocksDB uses log-structured merge trees to reduce these issues. The document outlines how MyRocks implements the MySQL storage engine interface on top of RocksDB. It concludes by covering best practices for using MyRocks, including installation, migration, tuning for replication and backups.
M|18 How to use MyRocks with MariaDB Server
M|18 How to use MyRocks with MariaDB ServerMariaDB plc MyRocks in MariaDB summarizes MyRocks, a storage engine for MariaDB that is based on RocksDB. It discusses how MyRocks addresses some of the limitations of InnoDB such as high write and space amplification. It provides details on installing and using MyRocks, including data loading techniques, tuning considerations, and replication support. Parallel replication is supported, but the highest isolation level is repeatable-read and row-based replication must be used.
The Hive Think Tank: Rocking the Database World with RocksDB
The Hive Think Tank: Rocking the Database World with RocksDBThe Hive RocksDB is a new storage engine for MySQL that provides better storage efficiency than InnoDB. It achieves lower space amplification and write amplification than InnoDB through its use of compression and log-structured merge trees. While MyRocks (RocksDB integrated with MySQL) currently has some limitations like a lack of support for online DDL and spatial indexes, work is ongoing to address these limitations and integrate additional RocksDB features to fully support MySQL workloads. Testing at Facebook showed MyRocks uses less disk space and performs comparably to InnoDB for their queries.
When is MyRocks good?
When is MyRocks good? Alkin Tezuysal MyRocks is a storage engine for MySQL, optimized for write-heavy workloads and large datasets, featuring improved performance and reduced storage size compared to InnoDB. It is built on RocksDB technology and is used in production by Facebook since 2016, offering benefits like fast data loading and efficient replication, though it has limitations such as no foreign keys or support for certain transaction types. The document includes details on installation, configuration, architecture, and specific tuning suggestions for MyRocks.
Say Hello to MyRocks
Say Hello to MyRocksSergey Petrunya This document provides an overview of MyRocks, a storage engine for MySQL/MariaDB that uses the RocksDB key-value store. It discusses the write amplification issues with InnoDB, how LSM trees in RocksDB address these issues through log-structured merging, and benchmarks showing the size, write amplification, and performance improvements MyRocks provides over InnoDB. It also outlines the process of integrating MyRocks into MariaDB, current status as an alpha plugin, and plans to improve support and testing.
RocksDB Performance and Reliability Practices
RocksDB Performance and Reliability PracticesYoshinori Matsunobu The document provides an overview of RocksDB, an open-source log-structured merge (LSM) database optimized for performance and reliability in backend services. It details key differences between LSM and B+ tree architectures, performance practices, and reliability strategies including handling write stalls and managing tombstones. The insights drawn from years of production experience highlight the trade-offs between write efficiency and read performance, along with best practices for database operations.
When is Myrocks good? 2020 Webinar Series
When is Myrocks good? 2020 Webinar SeriesAlkin Tezuysal The document provides an overview of MyRocks, a storage engine for MySQL optimized for write-heavy workloads and designed to support large datasets. It outlines the architecture, features, and comparisons with other engines like InnoDB, highlighting its suitability for scenarios without foreign keys and serializable transactions. Additionally, it offers tuning suggestions and installation instructions for Percona Server and discusses backup and recovery methods.
M|18 How Facebook Migrated to MyRocks
M|18 How Facebook Migrated to MyRocksMariaDB plc MyRocks deployment at Facebook and Roadmaps
This document discusses Facebook's deployment of MyRocks, a MySQL storage engine that uses RocksDB. It summarizes Facebook's initial goals for MyRocks, the technical challenges of migrating to MyRocks, their production configuration, and monitoring. It also outlines Facebook's plans to help further develop MyRocks in MariaDB and Percona Server with a focus on read performance, mixed engines, better replication, and supporting larger instance sizes.
Best storage engine for MySQL
Best storage engine for MySQLtomflemingh2 The document discusses DeepDB, a storage engine plugin for MySQL that aims to address MySQL's performance and scaling limitations for large datasets and heavy indexing. It does this through techniques like a Cache Ahead Summary Index Tree, Segmented Column Store, Streaming I/O, Extreme Concurrency, and Intelligent Caching. The document provides examples showing DeepDB significantly outperforming MySQL's InnoDB storage engine for tasks like data loading, transactions, queries, backups and more. It positions DeepDB as a drop-in replacement for InnoDB that can scale MySQL to support billions of rows and queries 2x faster while reducing data footprint by 50%.
Myrocks in the wild wild west! FOSDEM 2020
Myrocks in the wild wild west! FOSDEM 2020Alkin Tezuysal The document presents an overview of MyRocks, a MySQL storage engine based on RocksDB, discussing its architecture, differences among distributions, and use cases. It highlights features like write optimization, compaction methods, and backup/recovery options, emphasizing its suitability for write-heavy workloads and large data sets. The conclusion suggests MyRocks is cost-effective and efficient for environments with high cloud resource utilization.
Using ScyllaDB for Real-Time Write-Heavy Workloads
Using ScyllaDB for Real-Time Write-Heavy WorkloadsScyllaDB ScyllaDB is designed for real-time, write-heavy workloads, offering over 5x higher throughput and 20x lower latency compared to traditional databases, making it suitable for data-intensive applications. It addresses challenges like scaling writes, locking, and I/O bottlenecks, while delivering significant cost savings and compatibility with other systems like Apache Cassandra. Success stories from various industries demonstrate ScyllaDB's ability to handle massive data flows, ensuring optimized performance in applications such as e-commerce, real-time analytics, and online gaming.
Mark Callaghan, Facebook
Mark Callaghan, FacebookOntico This document discusses various database technologies and their performance characteristics. It begins by outlining some key metrics for evaluating databases, including throughput, response time variance, and efficiency/cost. It then discusses storage strategies using flash and spinning disks to reduce costs while improving performance. The document goes on to examine technologies like B-trees, log structured merge trees, and their variants. It analyzes each in terms of read amplification, write amplification, space amplification, and how these factors influence hardware requirements and lifespan. Overall, the document provides a technical overview for choosing an appropriate database technology based on workload and performance goals.
TokuDB vs RocksDB
TokuDB vs RocksDBVlad Lesin TokuDB and RocksDB are two write-optimized database engines that can be used with Percona. The document compares their implementations of fractal trees and LSM trees, caching approaches, storage on disk, benchmarks showing their performance for different workloads, and features. Key differences include TokuDB using log and block files while RocksDB uses WAL and SST files, and their approaches to caching blocks, memtables, and indexes in memory. Benchmarks showed varying performance depending on the workload and tuning parameters.
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...Insight Technology, Inc. The document discusses the history and development of SQLite, particularly focusing on the differences between SQLite3 and the proposed SQLite4. It highlights the storage mechanisms used, such as b-trees and log structured merge (LSM) storage engines, and their respective advantages and disadvantages. The document also shares insights on SQLite's performance, API quirks, and lessons learned from both versions.
Write intensive workloads and lsm trees
Write intensive workloads and lsm treesTilak Patidar This document discusses LSM trees as an alternative to B+ trees for write-intensive workloads like time series and sensor data. LSM trees optimize for writes by sequentially appending data to sorted string tables, using bloom filters to index them, and periodically merging files in the background. This avoids issues with B+ trees like index locking during updates and excessive disk seeks from fragmentation over time. The document outlines the basic framework of LSM trees including memtables, sorted string tables, bloom filters, and compaction. It also describes how reads, writes, deletes, and compaction work in an LSM tree to provide fast, write-optimized performance for large volumes of time-series and sensor data.
Web scale MySQL at Facebook (Domas Mituzas)
Web scale MySQL at Facebook (Domas Mituzas)Ontico This document summarizes Facebook's use of MySQL at web scale. Key points include:
- Facebook has over 800 million monthly active users generating huge volumes of queries and data.
- MySQL is customized with patches and extra resiliency to handle these loads.
- Performance optimizations focus on avoiding stalls and fully utilizing hardware.
- Tools help identify stalls and bottlenecks like table extensions, purge contention, and I/O pressure.
- Memory and disk efficiency are improved through techniques like compact records and flash caching.
- Group commit and admission control were added to MySQL to further optimize high concurrency workloads.
- Ongoing work looks to leverage new hardware, improve replication, and incorporate compression.
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...
[db tech showcase Tokyo 2017] C23: Lessons from SQLite4 by SQLite.org - Richa...Insight Technology, Inc.
Ad
More from Ontico (20)
One-cloud — система управления дата-центром в Одноклассниках / Олег Анастасье...
One-cloud — система управления дата-центром в Одноклассниках / Олег Анастасье...Ontico Документ описывает систему управления датацентром one-cloud, включая детали о конфигурации серверов, распределении ресурсов и управлении сервисами. Он подчеркивает важность изоляции задач, квот на ресурсы и различий между приоритетами задач, таких как prod и batch. Также рассматриваются проблемы отказоустойчивости и необходимость в управлении слоем для координации сервисов внутри системы.
Масштабируя DNS / Артем Гавриченков (Qrator Labs)
Масштабируя DNS / Артем Гавриченков (Qrator Labs)Ontico Документ рассматривает масштабирование DNS и связанные с ним вызовы, такие как геобалансировка, динамическая конфигурация и уязвимости. Подчеркиваются преимущества использования управляемых DNS-сервисов по сравнению с собственными решениями, включая снижение задержек и повышенную защищенность. Кроме того, упоминаются актуальные инструменты и инициативы для автоматизации процессов управления DNS.
Создание BigData-платформы для ФГУП Почта России / Андрей Бащенко (Luxoft)
Создание BigData-платформы для ФГУП Почта России / Андрей Бащенко (Luxoft)Ontico Документ описывает разработку платформы Big Data Почты России, охватывая процессы обработки данных, архитектуру и команду разработчиков. Основное внимание уделяется важным технологиям, таким как Spark ETL, а также задачам по оптимизации и монетизации данных, включая борьбу с 'серой' почтой. Также рассматривается структура команды, плюсы и минусы различных организационных подходов к разработке.
Готовим тестовое окружение, или сколько тестовых инстансов вам нужно / Алекса...
Готовим тестовое окружение, или сколько тестовых инстансов вам нужно / Алекса...Ontico Документ описывает процесс подготовки тестового окружения для проекта в компании Superjob, включая использование Docker и автоматизацию развертывания тестовых стендов. Упоминаются преимущества создания независимых стендов и управление ими в контексте сложных проектов, требующих быстрой настройки сред для тестирования. В результате упрощается доступ к тестовым стендам и сокращается время их развертывания под конкретные задачи.
Новые технологии репликации данных в PostgreSQL / Александр Алексеев (Postgre...
Новые технологии репликации данных в PostgreSQL / Александр Алексеев (Postgre...Ontico Доклад Александра Алексеева рассказывает о новых технологиях репликации в PostgreSQL и направлен на аудиторию, не имеющую большого опыта в этой области. Он описывает физическую и логическую репликацию, а также возможности автоматического фейловера, детализируя их преимущества и ограничения. В конце предоставлены ссылки на дополнительные материалы для дальнейшего изучения.
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...
Inexpensive Datamasking for MySQL with ProxySQL — Data Anonymization for Deve...Ontico The document discusses the use of ProxySQL as an inexpensive solution for data masking in MySQL databases, focusing on features such as query rewriting and dynamic rule application to protect sensitive data. It highlights the motivations for using ProxySQL, its applications, and the process of implementing data masking and obfuscation techniques to remain compliant while allowing developers access to relevant data. Additionally, the document lists various rules and examples for managing data securely through ProxySQL.
Опыт разработки модуля межсетевого экранирования для MySQL / Олег Брославский...
Опыт разработки модуля межсетевого экранирования для MySQL / Олег Брославский...Ontico Документ обсуждает разработку парсера для SQL-запросов и защитные меры против SQL-инъекций, исследуя различные подходы, включая машинное обучение и лексический анализ. Он также предоставляет данные о производительности различных языков программирования и их реализации парсеров. В заключении подчеркивается важность инфраструктуры в обеспечении производительности парсера.
ProxySQL Use Case Scenarios / Alkin Tezuysal (Percona)
ProxySQL Use Case Scenarios / Alkin Tezuysal (Percona)Ontico The document outlines the use case scenarios and benefits of ProxySQL, a database proxy tool that enhances MySQL operations. Key benefits include improved scalability, high availability, and advanced query support, allowing for efficient connection pooling, read/write splitting, and seamless failover. It also discusses various advanced queries management features like query caching, throttling, and rewriting to optimize database performance and protect against unwanted traffic.
Внутренний open-source. Как разрабатывать мобильное приложение большим количе...
Внутренний open-source. Как разрабатывать мобильное приложение большим количе...Ontico Документ описывает процесс разработки мобильного приложения Сбербанка с использованием open source моделей и большого числа разработчиков, акцентируя внимание на проблемах, связанных с координацией и качеством продукта. Рассматриваются изменения в организационной структуре команд для победы над дублированием усилий и создания единого подхода к разработке. Главное внимание уделяется созданию сообщества разработчиков для улучшения совместной работы и обмена опытом.
Подробно о том, как Causal Consistency реализовано в MongoDB / Михаил Тюленев...
Подробно о том, как Causal Consistency реализовано в MongoDB / Михаил Тюленев...Ontico The document discusses the implementation of cluster-wide causal consistency in distributed systems, outlining its definition and academic perspectives. It covers the architectural components necessary for building secure, fast, and reliable causal consistency, as well as how it can be utilized by end-users through interaction with databases. The document includes technical details about timestamp management and operational mechanisms for maintaining causal relationships among events.
Балансировка на скорости проводов. Без ASIC, без ограничений. Решения NFWare ...
Балансировка на скорости проводов. Без ASIC, без ограничений. Решения NFWare ...Ontico Документ посвящен балансировке трафика 'на скорости проводов' без использования ASIC и описывает решения для борьбы с атаками типа SYN-флуд. Рассматриваются вопросы оптимизации работы серверов, архитектурные особенности, а также преимущества и функциональность системы. В заключении представлены контактные данные авторов для дальнейших вопросов.
Перехват трафика — мифы и реальность / Евгений Усков (Qrator Labs)
Перехват трафика — мифы и реальность / Евгений Усков (Qrator Labs)Ontico Документ обсуждает уязвимости систем безопасности, связанных с атаками типа MITM, и акцентирует внимание на проблемах при использовании протоколов DNS и BGP. Приводится ряд примеров инцидентов, связанных с перехватами трафика и утечками информации, подчеркивающих недостатки существующих методов сертификации и мониторинга. Автор настоятельно рекомендует улучшение безопасности и рассматривает альтернативы для более надежного контроля за состоянием сетевой инфраструктуры.
И тогда наверняка вдруг запляшут облака! / Алексей Сушков (ПЕТЕР-СЕРВИС)
И тогда наверняка вдруг запляшут облака! / Алексей Сушков (ПЕТЕР-СЕРВИС)Ontico Алексей Сушков из компании «Петер-Сервис» обсуждает технологии облачных решений для операторов связи, такие как SDN и NFV. Документ подчеркивает важность облаков и готовности приложений к работе в облачной среде, а также будущую роль искусственного интеллекта в управлении сетями. Компания «Петер-Сервис», с 25-летним опытом, обслуживает 90% трафика российских операторов связи и активно развивает свои облачные технологии.
Как мы заставили Druid работать в Одноклассниках / Юрий Невиницин (OK.RU)
Как мы заставили Druid работать в Одноклассниках / Юрий Невиницин (OK.RU)Ontico Документ описывает внедрение системы Druid в проекте 'Одноклассники', включая задачи, цели, и архитектурные решения. Представлены данные о масштабах обработки: 12 миллиардов событий в сутки на 350 таблиц и 50 ТБ информации. Обсуждаются проблемы, связанные с отказами и возможные решения для их предотвращения.
Разгоняем ASP.NET Core / Илья Вербицкий (WebStoating s.r.o.)
Разгоняем ASP.NET Core / Илья Вербицкий (WebStoating s.r.o.)Ontico Документ представляет собой экспертное руководство по оптимизации приложений на ASP.NET Core, охватывающее различные аспекты от управления памятью до кэширования и профилирования производительности. Автор делится своим опытом и предлагает инструменты для улучшения работы веб-приложений, такие как MiniProfiler и различные методы кэширования. Основное внимание уделяется важности применения рекомендаций по улучшению производительности, профилированию и использованию облачных технологий.
100500 способов кэширования в Oracle Database или как достичь максимальной ск...
100500 способов кэширования в Oracle Database или как достичь максимальной ск...Ontico The document outlines strategies for caching in Oracle Database to enhance query performance while minimizing costs, categorizing various caching techniques such as buffer, statement, and result caches. It includes case studies demonstrating the effective use of hand-made result caches and Oracle's built-in result cache, along with their pros and cons, performance metrics, and recommendations for implementation. Furthermore, it provides insights on cache management, invalidation scenarios, and detailed monitoring features to optimize performance in production environments.
Apache Ignite Persistence: зачем Persistence для In-Memory, и как он работает...
Apache Ignite Persistence: зачем Persistence для In-Memory, и как он работает...Ontico Документ обсуждает необходимость и преимущества обеспечения постоянства данных для in-memory технологий, таких как Apache Ignite, в условиях роста объемов данных. Описываются способы масштабирования вычислительных мощностей и эффективные подходы к хранению и обработке данных, включая использование распределенных SQL и архитектуры памяти. Также рассматриваются особенности работы Apache Ignite, в том числе наличие полной транзакционности и возможности обработки данных на разных уровнях хранения.
Механизмы мониторинга баз данных: взгляд изнутри / Дмитрий Еманов (Firebird P...
Механизмы мониторинга баз данных: взгляд изнутри / Дмитрий Еманов (Firebird P...Ontico Доклад посвящен механизмам мониторинга баз данных с акцентом на необходимость понимания информации, необходимой для администраторов и DevOps. Обсуждаются методы сбора и анализа метрик, включая снэпшоты и логирование в разных системах управления базами данных, таких как MySQL, PostgreSQL и Firebird. Особое внимание уделяется детализации метрик, доступности и целостности данных, а также рекомендациям по оптимизации мониторинга.
Как мы учились чинить самолеты в воздухе / Евгений Коломеец (Virtuozzo)
Как мы учились чинить самолеты в воздухе / Евгений Коломеец (Virtuozzo)Ontico Документ обсуждает концепцию 'live patching', позволяющую заменять часть логики процесса без его перезапуска, что снижает время простоя при критических уязвимостях. Это достигается посредством использования различных утилит для патчинга и требует тщательного анализа состояния процесса перед применением патча. В заключении подчеркивается важность ревью кода и упоминается использование библиотек для остановки и перезапуска процессов, а также для анализа стека.
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)
Java и Linux — особенности эксплуатации / Алексей Рагозин (Дойче Банк)Ontico The document discusses the relationship between Java and Linux, focusing on JVM memory management, garbage collection, and performance tuning. It covers various aspects such as the differences in memory metrics for Java and Linux, thread management, and best practices for resource usage. Additionally, it highlights tools for diagnostics and performance monitoring in a Java environment running on Linux.
Recently uploaded (20)
Impurities of Water and their Significance.pptx
Impurities of Water and their Significance.pptxdhanashree78 Impart Taste, Odour, Colour, and Turbidity to water.
Presence of organic matter or industrial wastes or microorganisms (algae) imparts taste and odour to water.
Presence of suspended and colloidal matter imparts turbidity to water.
SEW make Brake BE05 – BE30 Brake – Repair Kit
SEW make Brake BE05 – BE30 Brake – Repair Kitprojectultramechanix A SEW-EURODRIVE brake repair kit is needed for maintenance and repair of specific SEW-EURODRIVE brake models, like the BE series. It includes all necessary parts for preventative maintenance and repairs. This ensures proper brake functionality and extends the lifespan of the brake system
4th International Conference on Computer Science and Information Technology (...
4th International Conference on Computer Science and Information Technology (...ijait 4th International Conference on Computer Science and Information Technology
(COMSCI 2025) will act as a major forum for the presentation of innovative ideas,
approaches, developments, and research projects in the area computer Science and
Information Technology. It will also serve to facilitate the exchange of information
between researchers and industry professionals to discuss the latest issues and
advancement in the research area.
ACEP Magazine Fifth Edition on 5june2025
ACEP Magazine Fifth Edition on 5june2025Rahul This document provides information about the Fifth edition of the magazine "Sthapatya" published by the Association of Civil Engineers (Practicing) Aurangabad. It includes messages from current and past presidents of ACEP, memories and photos from past ACEP events, information on life time achievement awards given by ACEP, and a technical article on concrete maintenance, repairs and strengthening. The document highlights activities of ACEP and provides a technical educational article for members.
TEA2016AAT 160 W TV application design example
TEA2016AAT 160 W TV application design examplessuser1be9ce TEA2016AAT 160 W TV application design example (UM11553)
Center Enamel can Provide Aluminum Dome Roofs for diesel tank.docx
Center Enamel can Provide Aluminum Dome Roofs for diesel tank.docxCenterEnamel Center Enamel can Provide Aluminum Dome Roofs for diesel tank.docx
How Binning Affects LED Performance & Consistency.pdf
How Binning Affects LED Performance & Consistency.pdfMina Anis 🔍 What’s Inside:
📦 What Is LED Binning?
• The process of sorting LEDs by color temperature, brightness, voltage, and CRI
• Ensures visual and performance consistency across large installations
🎨 Why It Matters:
• Inconsistent binning leads to uneven color and brightness
• Impacts brand perception, customer satisfaction, and warranty claims
📊 Key Concepts Explained:
• SDCM (Standard Deviation of Color Matching)
• Recommended bin tolerances by application (e.g., 1–3 SDCM for retail/museums)
• How to read bin codes from LED datasheets
• The difference between ANSI/NEMA standards and proprietary bin maps
🧠 Advanced Practices:
• AI-assisted bin prediction
• Color blending and dynamic calibration
• Customized binning for high-end or global projects
3. What is the principles of Teamwork_Module_V1.0.ppt
3. What is the principles of Teamwork_Module_V1.0.pptengaash9 Demonstrate the role of teamwork in the execution of systems engineering.
Describe the principles of successful teams.
Development of Portable Biomass Briquetting Machine (S, A & D)-1.pptx
Development of Portable Biomass Briquetting Machine (S, A & D)-1.pptxaniket862935 Biomass briquetting Machine
Pavement and its types, Application of rigid and Flexible Pavements
Pavement and its types, Application of rigid and Flexible PavementsSakthivel M Pavement and its types, Application of rigid and Flexible Pavements
Water demand - Types , variations and WDS
Water demand - Types , variations and WDSdhanashree78 Water demand refers to the volume of water needed or requested by users for various purposes. It encompasses the water required for domestic, industrial, agricultural, public, and other uses. Essentially, it represents the overall need or quantity of water required to meet the demands of different sectors and activities.
Fundamentals of Digital Design_Class_12th April.pptx
Fundamentals of Digital Design_Class_12th April.pptxdrdebarshi1993 Boolean Algebra and Combinational Logic Circuit
Making the case for write-optimized database algorithms / Mark Callaghan (Facebook)
- 1. Making the case for write-optimized database algorithms Mark Callaghan Member of Technical Staff, Facebook
- 2. RocksDB • Embedded key-value storage engine MyRocks • RocksDB storage engine for MySQL • coming to Percona Server and MariaDB Server MongoRocks • RocksDB storage engine for MongoDB • In Percona Server today RocksDB, MyRocks & MongoRocks
- 3. • Good read efficiency • Better write efficiency • Best space efficiency RocksDB, MyRocks & MongoRocks • Use less SSD • Use lower endurance SSD In some cases, better read efficiency is possible Efficient performance is the goal
- 4. Benchmarketing? Sysbench read-write, in-memory 0 50000 100000 150000 200000 1 2 4 8 16 24 32 40 48 64 80 96 128 MyRocks InnoDB
- 5. Better is not just about throughput tpmC iostat rKB/t iostat wKB/t vmstat CPU/t Size (GB) p99 response (ms) MyRocks +zlib 95680 2.19 2.02 528 82 29.9 InnoDB 91981 2.67 7.49 400 222 18.4 tpcc-mysql, 1000 warehouses, IO-bound • QPS - average throughput • QoS - worst case throughput • Efficiency - hardware per query
- 6. Peak performance is overrated Performance Performance & Efficiency Meet performance goals Then optimize for efficiency
- 7. RUM or RWS? • RUM = Read, Update, Memory • RWS = Read, Write, Space Efficiency • Synonym for amplification • Related to, but not equivalent to, performance. An algorithm can’t be optimal for all of read, write & space amplification • See Designing Access Methods: The RUM Conjecture • daslab.seas.harvard.edu/rum-conjecture RUM Conjecture
- 8. For any useful database algorithm there exists another useful algorithm that has better read, write or space amplification. Define: optimal
- 9. Read, Write • physical work per logical request Space • sizeof(database files) / sizeof(data) Define: amplification
- 10. Storage • random operations • bytes read • bytes written Define: work CPU • blocks (un)compressed • memory/cache operations • hash searches • key comparisons • CPU seconds
- 11. Basic operations: • point read • range read • put • delete Complex operations: • query • transaction Define: operation Update is one of: • put • point read, put • point read, delete, put
- 12. Everything is relative tpmC iostat rKB/t iostat wKB/t vmstat CPU/t Size (GB) MyRocks+zlib 95680 2.19 2.02 528 82 InnoDB 91981 2.67 7.49 400 222 InnoDB/ MyRocks 0.96 1.22 3.71 0.76 2.71 tpcc-mysql, 1000 warehouses, IO-bound
- 13. Efficiency: B-Tree Example: InnoDB Read • logN key compares Write • sizeof(page) / sizeof(row) Space • 1.5X if leaf pages are 2/3 full Update-in-Place B-Tree • InnoDB Copy-on-Write B-Tree • WiredTiger • LMDB
- 14. Efficiency: leveled LSM Example: RocksDB, Cassandra Read • logN + log(N/10) + log(N/100) + log(N/1000) key compares • point reads can use bloom filter Write • rewrite previously written rows • worse than size-tiered LSM, better than B-Tree Space • 1.1X
- 15. Efficiency: size-tiered LSM Example: RocksDB, Cassandra, HBase Read • more than leveled LSM • point reads can use bloom filter Write • rewrite previously written rows • better than leveled LSM, better than B-Tree Space • ~2X, worse than leveled LSM
- 16. Efficiency: summary Read Write Space B-Tree best good leveled LSM good for point good best size-tiered LSM best
- 17. Theory meets practice Access distribution • LSM benefits from skew Cache • B-Tree - prefer to have index in cache • LSM - prefer to have all but largest level in cache IO costs are hard to predict
- 18. Linkbench, IO-bound TPS iostat r/t iostat wKB/t CPU usecs/t Size (GB) p99 update (ms) MyRocks+zlib 28965 1.03 1.25 999 374 1 InnoDB 21474 1.16 19.70 914 14xx 6 InnoDB+zlib 20734 1.07 14.59 1199 880 6 MyRocks: best throughput & QoS, most efficient
- 19. Space efficiency • Fragmentation • Fixed page size • More per-row metadata • No prefix encoding (InnoDB) Why did RocksDB beat a B-Tree? Write efficiency • Uses more space = more data to write • Working set larger than cache • sizeof(page) / sizeof(row) • Double write buffer (InnoDB)
- 20. Page size & write amplification Page size TPS iostat wKB/t MyRocks+zlib 16kb 28965 1.25 InnoDB 4kb 24845 6.13 InnoDB 8kb 24352 10.52 InnoDB 16kb 21414 19.70
- 21. Advantage B-Tree • Fewer key comparisons • Less IO for range queries Read efficiency: B-Tree vs LSM? Advantage LSM • Uses less space = more data in cache • Prefix key encoding when uncompressed • Efficient writes saves IO for reads • Read-free index maintenance • Bloom filter Performance is complex
- 22. Save on writes, spend more on reads MyRocks zlib TPS InnoDB TPS Ratio (MyRocks / InnoDB) Disk array 2195 414 5.3 Slow SSD 23484 10143 2.3 Fast SSD 28965 21414 1.4
- 23. Read versus Write efficiency • Indexes - more & wider Read versus Space efficiency • Bloom filters • Compression • Indexes Write versus Space efficiency • RocksDB fanout • Size-tiered vs leveled • GC & defragmentation frequency Trading between R, W and S efficiency
- 24. Write versus space amplification Space vs write amplification for a log-based algorithm WriteAmplification 0 2.5 5 7.5 10 Space Amplification 1.11 1.25 1.33 1.67 2 • space amplification = 100 / %full • write amplification = 100 / (100 - %full)
- 25. One size doesn’t fit all • B-Tree + LSM sharing one redo log Adaptive algorithms • DBA sets high-level goals • Algorithm adapts to achieve them More open source • MyRocks in MariaDB Server & Percona Server • MongoRocks in Percona Server • More features in RocksDB What comes next? More performance results are coming • YCSB • sysbench • time series • bulk load • tpcc-mysql