Automated Slow Query Analysis: Dex the Index Robot
Download as pptx, pdf4 likes2,064 views
This document discusses MongoDB indexing and the Dex automation tool. It provides an overview of MongoDB indexing fundamentals and best practices. It then introduces Dex, explaining that it analyzes query logs and profiles to recommend optimal indexes. The document demonstrates Dex's usage and capabilities, such as working with live query streams, filtering by namespace, and focusing on long-running queries. It outlines Dex's internal workings and future development plans.
1 of 34
Downloaded 23 times
![18
Identify the problematic queries
• Search the log file
– logs any query over 100ms
• Use the database profiler
① Turn it on
db.setProfilingLevel(1)logs slow queries
db.setProfilingLevel(2)logs all queries (helpful but noisy)
② Find the slow queries
.sort({millis: -1})
.find({ns: "mongoquest.adventurers"})
.find({op: {$in: ["query", "update", "command"]})
③ Cleanup
db.setProfilingLevel(0)
db.system.profile.dropCollection()](https://image.slidesharecdn.com/ericsedordex2013-06-130626091619-phpapp01/85/Automated-Slow-Query-Analysis-Dex-the-Index-Robot-19-320.jpg)
![The Heart of Dex
22
Dex understands that order of fields in an index
should be:
① Equivalency checks {a:1}
② Sorts .sort({b: 1})
③ Range checks {c: {$in: [1, 2]}}](https://image.slidesharecdn.com/ericsedordex2013-06-130626091619-phpapp01/85/Automated-Slow-Query-Analysis-Dex-the-Index-Robot-23-320.jpg)
![23
Using Dex is easy
Install using pip:
> sudo pip install dex
Usage: dex [<options>] uri
> dex –f my/mongod/data/path/mongodb.log
mongodb://myUser:myPass@myHost:12345/myDb
> dex –p mongodb://myUser:myPass@myHost:12345/myDb](https://image.slidesharecdn.com/ericsedordex2013-06-130626091619-phpapp01/85/Automated-Slow-Query-Analysis-Dex-the-Index-Robot-24-320.jpg)
![25
'runStats': {
'linesRecommended': 76,
'linesProcessed': 76,
'linesPassed': 93
},
'results': [
{
'index': '{"name": 1}',
'totalTimeMillis': 410041,
'namespace': 'mongoquest.adventurers',
'queryCount': 2161,
'avgTimeMillis': 189,
'queries': [
'{"q": {"name": "<name>"}}'
]
},
...
Example of Dex's output
(use –v for a shell command!)](https://image.slidesharecdn.com/ericsedordex2013-06-130626091619-phpapp01/85/Automated-Slow-Query-Analysis-Dex-the-Index-Robot-26-320.jpg)
![29
{parsed: ...,
namespace: db.adventurers,
queryAnalysis: {analyzedFields: [{fieldName: name,
fieldType: EQUIV},
{fieldName: class,
fieldType: EQUIV},
fieldCount: N,
supported: true|false},
indexAnalysis: {fullIndexes: [],
partialIndexes: [{name: 1}]
needsRecommendation: true|false },
recommendation: {namespace: mongoquest.adventurers
index: {name: 1, class: 1}
shellCommand: db.ensureIndex... } }
Dex's guts](https://image.slidesharecdn.com/ericsedordex2013-06-130626091619-phpapp01/85/Automated-Slow-Query-Analysis-Dex-the-Index-Robot-30-320.jpg)
Ad
Recommended
Log MongoDB slow query
Log MongoDB slow queryHadi Ariawan This document discusses logging slow queries in MongoDB. It provides instructions on enabling profiling in MongoDB to collect performance data on slow operations over 100 milliseconds by default. An example slow query document is shown from the system.profile collection, listing the command and time taken. Various solutions for slow queries are mentioned such as adding indexes, changing schema design, or scaling hardware.
Sekilas PHP + mongoDB
Sekilas PHP + mongoDBHadi Ariawan This document provides instructions on installing and configuring MongoDB to work with PHP applications. It discusses downloading and setting up MongoDB on Windows, and installing the PHP MongoDB driver. Code examples are given for connecting to MongoDB from PHP, selecting and creating databases and collections, performing insert, query, update and remove operations, and implementing geospatial indexing. Finally, several PHP frameworks and libraries that support MongoDB are listed.
MongoDB basics & Introduction
MongoDB basics & IntroductionJerwin Roy This document provides an overview and introduction to MongoDB including:
- MongoDB installation using yum or binary, roles in MongoDB, and how to create users
- Basic MongoDB commands like use, show dbs, and dropDatabase
- Tracing slow queries using the database profiler and viewing results in the system.profile collection
- Important monitoring commands like db.serverStatus(), currentOp(), db.stats(), and replica set commands like rs.status() and rs.printReplicationInfo()
Superficial mongo db
Superficial mongo dbDaeMyung Kang This document provides an overview of MongoDB, including its key features such as document-oriented storage, full index support, replication and high availability, auto sharding, querying capabilities, and fast in-place updates. It also discusses MongoDB's architecture for replication, sharding, and configuration servers.
MongoDB Schema Design by Examples
MongoDB Schema Design by ExamplesHadi Ariawan The document discusses MongoDB schema design, highlighting its advantages like dynamic schema, embedding documents to reduce joins, and high scalability with automatic sharding. Use cases are presented, demonstrating MongoDB's application in various scenarios, such as product specifications and logging systems. Additionally, it emphasizes the importance of understanding application requirements and designing schemas accordingly.
Mongo presentation conf
Mongo presentation confShridhar Joshi MongoDB is an open source, scalable, high-performance, document-oriented NoSQL database that uses a schema-less JSON-style document model, supports master-slave replication and horizontal sharding to scale out, and provides basic CRUD operations and rich queries on document collections. The document demonstrates MongoDB's data structures, basic CRUD operations, indexing, querying, optimization, architecture including replication and sharding, and use cases.
MongoDB
MongoDBMartin Lazarov This document provides an overview of MongoDB, including what it is, its features, and how to use it. MongoDB is an open-source, document-oriented database that bridges the gap between traditional relational databases and key-value stores. It features document-based storage, indexing, replication, querying, and scaling capabilities. The document also demonstrates how to easily install, start, and interact with MongoDB using simple commands.
MongoDB for Beginners
MongoDB for BeginnersEnoch Joshua MongoDB is an open-source, document-oriented, NoSQL database that provides scalability, performance, and high availability. It is written in C++ and stores data in flexible, JSON-like documents, allowing for easy querying and retrieval of data. MongoDB is commonly used for applications that require scalability and large datasets, and provides features like auto-sharding, replication, and rich queries.
Connecting NodeJS & MongoDB
Connecting NodeJS & MongoDBEnoch Joshua The document outlines steps for building a Node Express app with MongoDB including installing MongoDB NPM modules, creating and closing a database connection, querying, inserting, updating, and deleting documents from a collection called "Employee" with sample employee documents provided. It introduces the database and collection names and lists 7 commands for common MongoDB operations within a Node application.
Mongo db
Mongo dbRaghu nath MongoDB is a document-oriented database where data is stored in flexible JSON-like documents within collections, rather than rows in tables. Documents can contain various data types and dynamically vary in structure. The MongoDB database stores collections of documents and provides basic CRUD functions through its shell interface to create, read, update and delete documents through queries, indexes and special collection types.
10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators iammutex The document discusses 10 key performance indicators for MongoDB:
1) Slow operations using the profiler
2) Replication lag by checking oplog timestamps
3) High resident memory usage indicating paging is occurring
4) High page faults
5) High write lock percentage indicating concurrency issues
6) Large reader/writer queues indicating lock contention
7) Frequent background flushing indicating I/O issues
8) Too many connections
9) High network traffic
10) Collection fragmentation leading to increased storage size
It provides examples of how to check for each indicator using the db.serverStatus() command.
mongoDB Performance
mongoDB PerformanceMoshe Kaplan The document discusses MongoDB performance optimization strategies presented by Moshe Kaplan at a VP R&D Open Seminar. It covers topics like sharding, in-memory databases, MapReduce, profiling, indexes, server stats, schema design, and locking in MongoDB. Slides include information on tuning configuration parameters, analyzing profiling results, explain plans, index management, and database stats.
Indexing In MongoDB
Indexing In MongoDBKishor Parkhe This document discusses indexing and query optimization in MongoDB. It provides an overview of indexes and how they can improve query performance by allowing certain queries to use indexes rather than scanning all documents. It describes different types of indexes like compound, unique, multikey, covered and sparse indexes. It also covers topics like profiling queries, creating and managing indexes, and strategies for effective indexing.
Mongo indexes
Mongo indexesparadokslabs The document provides a comprehensive overview of MongoDB indexes, including types such as default, single, compound, multikey, geospatial, text, and hashed indexes. It discusses their creation, usage, and scenarios in query analysis, highlighting how indexes enhance database performance and efficiency. Additionally, it covers practical examples of index implementation and geospatial data handling within MongoDB.
11 schema design & crud
11 schema design & crudAhmed Elbassel This document discusses schema design and CRUD operations in MongoDB. It compares MongoDB and MySQL, and covers how to create databases and collections in MongoDB. It then describes various CRUD operations like create, read, update and delete. Finally, it discusses schema design considerations like one-to-one, one-to-many and many-to-many relationships, and recommends denormalizing data for better read performance.
Mongo db pefrormance optimization strategies
Mongo db pefrormance optimization strategiesronwarshawsky The document discusses MongoDB performance optimization strategies. It outlines various techniques such as using map-reduce operations, updating to the latest MongoDB version, implementing sharding, balancing shards, optimizing disk input/output, managing locks, using capped collections for fast writes, leveraging natural ordering for fast reads, considering query performance including indexes, minimizing document size through field name shortening, and keeping MongoDB updates. It also briefly describes Enteros' software for database problem root cause analysis across infrastructure tiers.
Introduction to MongoDB
Introduction to MongoDBNodeXperts The document provides an introduction to MongoDB, explaining it as an open-source document-oriented NoSQL database that contrasts with traditional SQL databases. It covers concepts such as the document data model, CRUD operations, and the benefits and drawbacks of using MongoDB, including its schema-less design and fast scalability. Additionally, it outlines when to use or not use MongoDB based on application requirements.
Mongo db
Mongo dbYuHsuan Chen This document provides an introduction to MongoDB, including:
- MongoDB is a cross-platform document-oriented database that uses documents with a dynamic schema, ad-hoc queries, indexing, replication and sharding.
- Key features include documents, collections, indexes, replication for redundancy and increased availability, and sharding to split data over multiple servers.
- The document discusses setting up a MongoDB instance, replication sets, and sharded clusters.
Mongo db nosql (1)
Mongo db nosql (1)Bhavesh Sarvaiya This document provides an overview of MongoDB, including:
- MongoDB is a non-SQL, document-oriented database that provides high performance, high availability, and easy scalability.
- In MongoDB, databases contain collections of documents rather than tables of rows. Documents can contain different fields and are equivalent to rows in a SQL database.
- Some key advantages of MongoDB over SQL databases include its schemaless nature, ease of horizontal scaling, and rich query capabilities.
Node js crash course session 6
Node js crash course session 6Abdul Rahman Masri Attal This document provides an overview and code snippets for building a Node.js application with user authentication functionality including registration and login. It discusses using Mongoose to define a user schema, hashing passwords on registration using bcrypt, and validating login credentials by comparing hashed passwords. Code examples show creating routes and controllers to handle the POST requests for registration and login.
Mongodb
MongodbScott Motte This document provides information about using MongoDB with Ruby. It discusses installing MongoDB on Mac OS X and Linux, running MongoDB, comparing MongoDB and CouchDB, using MongoDB ORMs like MongoMapper in Ruby applications, defining models and relationships, and additional features of MongoDB and MongoMapper. The conclusion recommends considering MongoDB as an alternative to MySQL for some web applications due to its speed, features, and schema-less flexibility.
Back to Basics Webinar 3: Introduction to Replica Sets
Back to Basics Webinar 3: Introduction to Replica SetsMongoDB This document provides an introduction to MongoDB replica sets, which allow for data redundancy and high availability. It discusses how replica sets work, including the replica set life cycle and how applications should handle writes and queries when using a replica set. Specifically, it explains that the MongoDB driver is responsible for server discovery and monitoring, retry logic, and handling topology changes in a replica set to provide a consistent view of the data to applications.
Mongo DB: Fundamentals & Basics/ An Overview of MongoDB/ Mongo DB tutorials
Mongo DB: Fundamentals & Basics/ An Overview of MongoDB/ Mongo DB tutorialsSpringPeople The document discusses MongoDB, an open-source document database. It provides an overview of MongoDB, including what it is, why it is used, its basic concepts like databases, collections, and documents, and how it compares to a relational database. It also covers MongoDB commands for creating and dropping collections, inserting, querying, and updating documents.
Dev Jumpstart: Build Your First App with MongoDB
Dev Jumpstart: Build Your First App with MongoDBMongoDB The document introduces MongoDB as a high-performance, open-source, document database that is scalable and supports a variety of programming languages. It covers setting up MongoDB, modeling documents, basic application requirements, and operations like data insertion and aggregation. Additionally, it explains scaling options such as replica sets and sharding, along with guidance on using MongoDB drivers for application development.
MongoDb and NoSQL
MongoDb and NoSQLTO THE NEW | Technology This document provides an overview of NoSQL databases, with a focus on MongoDB, including its installation, data models, querying methods, and indexing. It highlights MongoDB's schema-less nature, rich query capabilities, and the use of BSON for document storage. Additionally, it covers aggregation frameworks, indexing types, and schema design factors pertinent to efficient data management.
The Basics of MongoDB
The Basics of MongoDBvaluebound The document compares MongoDB, a NoSQL database, with traditional RDBMS, highlighting issues such as scalability, flexibility, and performance. It outlines the advantages of MongoDB, including its document-based structure and ease of use in REST APIs, along with basic MongoDB commands. The document also illustrates the difference in data representation and structures between SQL and MongoDB.
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]Dinesh Neupane This document is a presentation on the psycopg2 Python module, which is a PostgreSQL database adapter. It covers installation, connection methods, cursor operations, SQL transactions, and various Data Manipulation Language (DML) commands including insert, delete, and update. Key features discussed include handling connections, managing databases, and ensuring data integrity through transactions.
Getting Started with MongoDB
Getting Started with MongoDBAhasanul Kalam Akib The document provides an overview of MongoDB, highlighting its features such as being a cross-platform, document-oriented database with dynamic schema and high performance. It explains key concepts like collections, documents, and operations for inserting, querying, updating, and deleting data, all while emphasizing the advantages over traditional relational databases. Additionally, it includes command-line commands for interacting with MongoDB and sample code for .NET integration.
Fast querying indexing for performance (4)
Fast querying indexing for performance (4)MongoDB The document discusses strategies for optimizing performance in MongoDB through effective indexing techniques, including the creation and management of various types of indexes. It highlights examples of fast querying, the importance of index monitoring, and the latest features introduced in MongoDB 3.0, such as index compression and placement on separate storage devices. Additionally, it covers practical tips for using indexes and the query optimizer to enhance query performance.
Indexing Strategies to Help You Scale
Indexing Strategies to Help You ScaleMongoDB This document discusses various indexing strategies in MongoDB to help scale applications. It covers the basics of indexes, including creating and tuning indexes. It also discusses different index types like geospatial indexes, text indexes, and how to use explain plans and profiling to evaluate queries. The document concludes with a section on scaling strategies like sharding to scale beyond a single server's resources.
More Related Content
What's hot (20)
Connecting NodeJS & MongoDB
Connecting NodeJS & MongoDBEnoch Joshua The document outlines steps for building a Node Express app with MongoDB including installing MongoDB NPM modules, creating and closing a database connection, querying, inserting, updating, and deleting documents from a collection called "Employee" with sample employee documents provided. It introduces the database and collection names and lists 7 commands for common MongoDB operations within a Node application.
Mongo db
Mongo dbRaghu nath MongoDB is a document-oriented database where data is stored in flexible JSON-like documents within collections, rather than rows in tables. Documents can contain various data types and dynamically vary in structure. The MongoDB database stores collections of documents and provides basic CRUD functions through its shell interface to create, read, update and delete documents through queries, indexes and special collection types.
10 Key MongoDB Performance Indicators
10 Key MongoDB Performance Indicators iammutex The document discusses 10 key performance indicators for MongoDB:
1) Slow operations using the profiler
2) Replication lag by checking oplog timestamps
3) High resident memory usage indicating paging is occurring
4) High page faults
5) High write lock percentage indicating concurrency issues
6) Large reader/writer queues indicating lock contention
7) Frequent background flushing indicating I/O issues
8) Too many connections
9) High network traffic
10) Collection fragmentation leading to increased storage size
It provides examples of how to check for each indicator using the db.serverStatus() command.
mongoDB Performance
mongoDB PerformanceMoshe Kaplan The document discusses MongoDB performance optimization strategies presented by Moshe Kaplan at a VP R&D Open Seminar. It covers topics like sharding, in-memory databases, MapReduce, profiling, indexes, server stats, schema design, and locking in MongoDB. Slides include information on tuning configuration parameters, analyzing profiling results, explain plans, index management, and database stats.
Indexing In MongoDB
Indexing In MongoDBKishor Parkhe This document discusses indexing and query optimization in MongoDB. It provides an overview of indexes and how they can improve query performance by allowing certain queries to use indexes rather than scanning all documents. It describes different types of indexes like compound, unique, multikey, covered and sparse indexes. It also covers topics like profiling queries, creating and managing indexes, and strategies for effective indexing.
Mongo indexes
Mongo indexesparadokslabs The document provides a comprehensive overview of MongoDB indexes, including types such as default, single, compound, multikey, geospatial, text, and hashed indexes. It discusses their creation, usage, and scenarios in query analysis, highlighting how indexes enhance database performance and efficiency. Additionally, it covers practical examples of index implementation and geospatial data handling within MongoDB.
11 schema design & crud
11 schema design & crudAhmed Elbassel This document discusses schema design and CRUD operations in MongoDB. It compares MongoDB and MySQL, and covers how to create databases and collections in MongoDB. It then describes various CRUD operations like create, read, update and delete. Finally, it discusses schema design considerations like one-to-one, one-to-many and many-to-many relationships, and recommends denormalizing data for better read performance.
Mongo db pefrormance optimization strategies
Mongo db pefrormance optimization strategiesronwarshawsky The document discusses MongoDB performance optimization strategies. It outlines various techniques such as using map-reduce operations, updating to the latest MongoDB version, implementing sharding, balancing shards, optimizing disk input/output, managing locks, using capped collections for fast writes, leveraging natural ordering for fast reads, considering query performance including indexes, minimizing document size through field name shortening, and keeping MongoDB updates. It also briefly describes Enteros' software for database problem root cause analysis across infrastructure tiers.
Introduction to MongoDB
Introduction to MongoDBNodeXperts The document provides an introduction to MongoDB, explaining it as an open-source document-oriented NoSQL database that contrasts with traditional SQL databases. It covers concepts such as the document data model, CRUD operations, and the benefits and drawbacks of using MongoDB, including its schema-less design and fast scalability. Additionally, it outlines when to use or not use MongoDB based on application requirements.
Mongo db
Mongo dbYuHsuan Chen This document provides an introduction to MongoDB, including:
- MongoDB is a cross-platform document-oriented database that uses documents with a dynamic schema, ad-hoc queries, indexing, replication and sharding.
- Key features include documents, collections, indexes, replication for redundancy and increased availability, and sharding to split data over multiple servers.
- The document discusses setting up a MongoDB instance, replication sets, and sharded clusters.
Mongo db nosql (1)
Mongo db nosql (1)Bhavesh Sarvaiya This document provides an overview of MongoDB, including:
- MongoDB is a non-SQL, document-oriented database that provides high performance, high availability, and easy scalability.
- In MongoDB, databases contain collections of documents rather than tables of rows. Documents can contain different fields and are equivalent to rows in a SQL database.
- Some key advantages of MongoDB over SQL databases include its schemaless nature, ease of horizontal scaling, and rich query capabilities.
Node js crash course session 6
Node js crash course session 6Abdul Rahman Masri Attal This document provides an overview and code snippets for building a Node.js application with user authentication functionality including registration and login. It discusses using Mongoose to define a user schema, hashing passwords on registration using bcrypt, and validating login credentials by comparing hashed passwords. Code examples show creating routes and controllers to handle the POST requests for registration and login.
Mongodb
MongodbScott Motte This document provides information about using MongoDB with Ruby. It discusses installing MongoDB on Mac OS X and Linux, running MongoDB, comparing MongoDB and CouchDB, using MongoDB ORMs like MongoMapper in Ruby applications, defining models and relationships, and additional features of MongoDB and MongoMapper. The conclusion recommends considering MongoDB as an alternative to MySQL for some web applications due to its speed, features, and schema-less flexibility.
Back to Basics Webinar 3: Introduction to Replica Sets
Back to Basics Webinar 3: Introduction to Replica SetsMongoDB This document provides an introduction to MongoDB replica sets, which allow for data redundancy and high availability. It discusses how replica sets work, including the replica set life cycle and how applications should handle writes and queries when using a replica set. Specifically, it explains that the MongoDB driver is responsible for server discovery and monitoring, retry logic, and handling topology changes in a replica set to provide a consistent view of the data to applications.
Mongo DB: Fundamentals & Basics/ An Overview of MongoDB/ Mongo DB tutorials
Mongo DB: Fundamentals & Basics/ An Overview of MongoDB/ Mongo DB tutorialsSpringPeople The document discusses MongoDB, an open-source document database. It provides an overview of MongoDB, including what it is, why it is used, its basic concepts like databases, collections, and documents, and how it compares to a relational database. It also covers MongoDB commands for creating and dropping collections, inserting, querying, and updating documents.
Dev Jumpstart: Build Your First App with MongoDB
Dev Jumpstart: Build Your First App with MongoDBMongoDB The document introduces MongoDB as a high-performance, open-source, document database that is scalable and supports a variety of programming languages. It covers setting up MongoDB, modeling documents, basic application requirements, and operations like data insertion and aggregation. Additionally, it explains scaling options such as replica sets and sharding, along with guidance on using MongoDB drivers for application development.
MongoDb and NoSQL
MongoDb and NoSQLTO THE NEW | Technology This document provides an overview of NoSQL databases, with a focus on MongoDB, including its installation, data models, querying methods, and indexing. It highlights MongoDB's schema-less nature, rich query capabilities, and the use of BSON for document storage. Additionally, it covers aggregation frameworks, indexing types, and schema design factors pertinent to efficient data management.
The Basics of MongoDB
The Basics of MongoDBvaluebound The document compares MongoDB, a NoSQL database, with traditional RDBMS, highlighting issues such as scalability, flexibility, and performance. It outlines the advantages of MongoDB, including its document-based structure and ease of use in REST APIs, along with basic MongoDB commands. The document also illustrates the difference in data representation and structures between SQL and MongoDB.
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]
Connecting and using PostgreSQL database with psycopg2 [Python 2.7]Dinesh Neupane This document is a presentation on the psycopg2 Python module, which is a PostgreSQL database adapter. It covers installation, connection methods, cursor operations, SQL transactions, and various Data Manipulation Language (DML) commands including insert, delete, and update. Key features discussed include handling connections, managing databases, and ensuring data integrity through transactions.
Getting Started with MongoDB
Getting Started with MongoDBAhasanul Kalam Akib The document provides an overview of MongoDB, highlighting its features such as being a cross-platform, document-oriented database with dynamic schema and high performance. It explains key concepts like collections, documents, and operations for inserting, querying, updating, and deleting data, all while emphasizing the advantages over traditional relational databases. Additionally, it includes command-line commands for interacting with MongoDB and sample code for .NET integration.
Similar to Automated Slow Query Analysis: Dex the Index Robot (20)
Fast querying indexing for performance (4)
Fast querying indexing for performance (4)MongoDB The document discusses strategies for optimizing performance in MongoDB through effective indexing techniques, including the creation and management of various types of indexes. It highlights examples of fast querying, the importance of index monitoring, and the latest features introduced in MongoDB 3.0, such as index compression and placement on separate storage devices. Additionally, it covers practical tips for using indexes and the query optimizer to enhance query performance.
Indexing Strategies to Help You Scale
Indexing Strategies to Help You ScaleMongoDB This document discusses various indexing strategies in MongoDB to help scale applications. It covers the basics of indexes, including creating and tuning indexes. It also discusses different index types like geospatial indexes, text indexes, and how to use explain plans and profiling to evaluate queries. The document concludes with a section on scaling strategies like sharding to scale beyond a single server's resources.
How to use the new Domino Query Language
How to use the new Domino Query LanguageTim Davis The document provides a comprehensive overview of the new Domino Query Language (DQL) introduced in Domino 10, detailing its features, requirements, setup process, and limitations. It explains the syntax for queries, including examples, performance optimization tips, and how to utilize DQL in different programming environments such as LotusScript, Java, and Node.js. Additionally, it touches on future enhancements and best practices for avoiding code injection vulnerabilities in queries.
Rails Tips and Best Practices
Rails Tips and Best PracticesDavid Keener The document provides tips and best practices for developing with Ruby on Rails, emphasizing the importance of log files for debugging and optimizing code. It covers efficient database practices, model validations, and the use of foreign keys, illustrating database relationships and constraints. The author encourages developers to leverage Rails features wisely while maintaining data integrity and testing rigorously.
Scaling MongoDB
Scaling MongoDBMongoDB The document discusses strategies for scaling MongoDB, emphasizing schema design, indexing, and the use of the WiredTiger storage engine to improve performance and reduce storage needs. It provides tips for optimizing operations, such as vertical and horizontal scaling, and highlights the importance of monitoring and using the MongoDB Management Service (MMS) for efficient database management. Key points include schema patterns, indexing strategies, and best practices for selecting effective shard keys to enhance database performance.
Scaling MySQL Strategies for Developers
Scaling MySQL Strategies for DevelopersJonathan Levin Jonathan is a MySQL consultant who specializes in SQL, indexing, and reporting for big data. This tutorial will cover strategies for resolving 80% of performance problems, including indexes, partitioning, intensive table optimization, and finding and addressing bottlenecks. The strategies discussed will be common, established approaches based on the presenter's experience working with MySQL since 2007.
How to Achieve Scale with MongoDB
How to Achieve Scale with MongoDBMongoDB The document discusses various techniques for optimizing and scaling MongoDB deployments. It covers topics like schema design, indexing, monitoring workload, vertical scaling using resources like RAM and SSDs, and horizontal scaling using sharding. The key recommendations are to optimize the schema and indexes first before scaling, understand the workload, and ensure proper indexing when using sharding for horizontal scaling.
Automated product categorization
Automated product categorization Warply The document provides a comprehensive overview of the skroutz.gr marketplace, which simplifies online shopping by categorizing over 11 million products from 3,200 merchants. It details the limitations of the old categorization system and introduces a new solution using a supervised learning model called 'megatron', which enhances product classification with minimal human intervention. Performance improvements are highlighted, showing a significant reduction in error rates and increased efficiency in content management.
Automated product categorization
Automated product categorizationAndreas Loupasakis The document describes skroutz.gr, an online marketplace that categorizes over 11 million products from various e-shops using a machine learning-based classification system called 'megatron'. It outlines the architecture, training process, and evaluation of the model, highlighting significant improvements over the previous system, including a reduction in error rates and improved categorization efficiency. Additionally, future enhancements such as leveraging image features and entity recognition are discussed.
New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2
New Indexing and Aggregation Pipeline Capabilities in MongoDB 4.2Antonios Giannopoulos The document discusses new features in MongoDB 4.2, focusing on indexing, aggregation, and materialized views. Key improvements include hybrid indexing methods, the removal of index key limits, and the introduction of new aggregation operators such as trigonometric functions and regex processing. It also highlights the ability to use the aggregation framework for update operations, emphasizing the power and potential complexity of these enhancements.
MongoDB at Scale
MongoDB at ScaleMongoDB This document discusses how to achieve scale with MongoDB. It covers optimization tips like schema design, indexing, and monitoring. Vertical scaling involves upgrading hardware like RAM and SSDs. Horizontal scaling involves adding shards to distribute load. The document also discusses how MongoDB scales for large customers through examples of deployments handling high throughput and large datasets.
5 Pitfalls to Avoid with MongoDB
5 Pitfalls to Avoid with MongoDBTim Callaghan 1) MongoDB databases can grow very large due to flexible document schemas that allow large and denormalized data. This leads to increased storage requirements.
2) MongoDB replication can introduce lag on secondary nodes as they process write operations. This limits the ability to use secondary nodes for scaling reads.
3) MongoDB performance declines dramatically when indexes do not fit in memory, requiring more RAM, sharding, or reduced write performance.
4) MongoDB implements database-level locking, limiting write concurrency and the ability to run multiple shards on a single server.
5) MongoDB does not support ACID transactions, multi-version concurrency control, or consistent reads in the presence of concurrent writes. Workarounds
Probabilistic Data Structures (Edmonton Data Science Meetup, March 2018)
Probabilistic Data Structures (Edmonton Data Science Meetup, March 2018)Kyle Davis The document discusses probabilistic data structures and their applications within the Redis database, highlighting its performance and simplicity. It covers various data structures such as bloom filters, hyperloglog, and count min sketch, explaining their use cases and functionalities in terms of speed, compactness, and guarantees. Additionally, it emphasizes the extensibility of Redis through modules that enhance its capabilities as a multi-model database.
Nosql part 2
Nosql part 2Ruru Chowdhury [
{
name: "John",
age: 30
},
{
name: "Mary",
age: 25
}
]
}
]
}
{
_id: "address1",
street: "10 Simla Str"
}
}
{
_id: "user1",
name: "John",
age: 30
}
{
_id: "user2",
name: "Mary",
age: 25
}
Data Modelling in MongoDB..5
Embedded Documents vs References
Embedded Documents
References
- Faster reads since related data is
embedded and can be retrieved in
single query
Mongo db tutorials
Mongo db tutorialsAnuj Jain This document provides an introduction to MongoDB, a scalable, high-performance NoSQL database, emphasizing its document-oriented design and ease of use without a predefined schema. It discusses key features like replication, sharding, and indexing, while also highlighting MongoDB's strengths in real-time analytics and content management. Additionally, the document includes practical examples, schema design considerations, and CRUD operations relevant to MongoDB.
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails App
[Srijan Wednesday Webinar] Easy Performance Wins for Your Rails AppSrijan Technologies The document outlines various performance optimization strategies for Ruby on Rails applications, highlighting the importance of profiling, proper indexing, and efficient memory usage. It recommends tools such as New Relic for monitoring slow queries and suggests techniques like using includes to reduce database calls. The final tips include utilizing caching in Rails and minimizing unnecessary resources to enhance performance.
MongoDB: a gentle, friendly overview
MongoDB: a gentle, friendly overviewAntonio Pintus This document provides a gentle, friendly overview of MongoDB. It discusses that MongoDB is a document-oriented, non-relational database that uses JSON-like documents with dynamic schemas. It also covers MongoDB's query language, indexing, data storage and querying, drivers like PyMongo, and how MongoDB provides horizontal scalability through sharding and replication. The document concludes by noting some common use cases for MongoDB and areas it is less suited for.
Денис Резник "Зачем мне знать SQL и Базы Данных, ведь у меня есть ORM?"
Денис Резник "Зачем мне знать SQL и Базы Данных, ведь у меня есть ORM?"Fwdays The document discusses the importance of understanding SQL and database internals even when using ORMs for database interactions. It covers topics such as query performance, parameter sniffing, and the capabilities of various ORMs like Entity Framework and Dapper. The author encourages developers to familiarize themselves with generated SQL and query plans, and to invest time in learning about database operations for better performance.
MongoDB and Indexes - MUG Denver - 20160329
MongoDB and Indexes - MUG Denver - 20160329Douglas Duncan Indexes are data structures that store a subset of data to allow for efficient retrieval. MongoDB stores indexes using a b-tree format. There are several types of indexes including simple, compound, multikey, full-text, and geospatial indexes. Indexes improve performance by enabling efficient retrieval, sorting, and filtering of documents. Indexes are created using the createIndex command and their usage can be checked using explain plans.
Webinar: Indexing and Query Optimization
Webinar: Indexing and Query OptimizationMongoDB The document focuses on indexing and query optimization in MongoDB, explaining the importance of indexes for performance and how to create and manage them. It covers various types of indexes, including compound, multikey, geospatial, and sparse indexes, as well as common mistakes to avoid when using them. Additionally, the document highlights strategies for optimizing query performance and profiling slow operations.
Ad
More from MongoDB (20)
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB SoCal 2020: Migrate Anything* to MongoDB AtlasMongoDB This presentation discusses migrating data from other data stores to MongoDB Atlas. It begins by explaining why MongoDB and Atlas are good choices for data management. Several preparation steps are covered, including sizing the target Atlas cluster, increasing the source oplog, and testing connectivity. Live migration, mongomirror, and dump/restore options are presented for migrating between replicasets or sharded clusters. Post-migration steps like monitoring and backups are also discussed. Finally, migrating from other data stores like AWS DocumentDB, Azure CosmosDB, DynamoDB, and relational databases are briefly covered.
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!MongoDB The document outlines an event focused on utilizing MongoDB Charts for data analysis, highlighting the importance of insights in various roles beyond traditional analysts. It discusses the evolution of analytics and suggests questions to explore about film data, such as movie length trends and genre popularity. Additionally, it emphasizes sharing insights through dashboards and provides resources for learning more about MongoDB Charts.
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...MongoDB The document provides a comprehensive guide on using MongoDB services within Kubernetes for both development and production environments. It covers various topics including the setup of Kubernetes clusters, the use of MongoDB operators, Helm for package management, and considerations for database management within Kubernetes. Key components such as custom resource definitions, service catalogs, and operational tools like MongoDB Ops Manager and Cloud Manager are highlighted for efficient database deployment and management.
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDBMongoDB The document outlines a comprehensive methodology for data modeling in MongoDB, contrasting document-based with tabular modeling approaches. It details various modeling patterns, such as the use of fields, arrays, and sub-documents, through specific use cases like a coffee shop franchise and social networks. The document also emphasizes the importance of schema design patterns and practical considerations for schema evolution and workload management.
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...MongoDB The document presents David Munguia's insights on leveraging MongoDB for real-time alerting in healthcare, detailing Humana's journey and lessons learned since their kickoff in November 2017. Key lessons include understanding access patterns, utilizing resources effectively, and embracing a mindset shift away from SQL to MongoDB's JSON structure. The presentation culminates in the success of their deployment, which significantly improved ETL processing times and reporting efficiency.
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series DataMongoDB The document provides best practices for working with IoT and time-series data using MongoDB, discussing key concepts such as time-series data definitions, architecture design, and schema creation. It emphasizes efficient data management strategies, like data retention policies and archiving methods, while presenting case studies and practical examples of MongoDB's capabilities. Additionally, it offers insights on querying and visualizing time-series data and promotes resources for further engagement with MongoDB projects.
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB SoCal 2020: MongoDB Atlas Jump StartMongoDB The document outlines a presentation on MongoDB Atlas, a global cloud database that offers deployment, scaling, and operational automation. It highlights the service's key features, including security, multi-region capabilities, and analytics, with a focus on guiding users through the setup process. Attendees are encouraged to explore MongoDB Atlas and participate in related tutorials and initiatives.
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]MongoDB The document introduces key individuals involved in modernization practices at Infosys, focusing on the use of MongoDB to meet new data demands. It discusses real-world use cases, particularly for a major American technology conglomerate, detailing the design of an observability layer powered by MongoDB. The architecture addresses challenges related to data volume, velocity, and variety, enabling high availability and distributed query capabilities.
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2MongoDB The document provides a detailed overview of MongoDB's encryption methods, including client-side encryption and data handling practices for sensitive information such as Social Security Numbers (SSNs). It discusses various encryption techniques, schemas, potential attack scenarios, and compliance with regulations like GDPR. The implementation details cover the use of auto-encryption options and key management, emphasizing the importance of protecting user data throughout its lifecycle.
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...MongoDB The document outlines the setup and use of MongoDB with Kubernetes through the MongoDB Operator and Open Service Broker, highlighting the necessary configurations and resources for deployment. It includes commands for cluster creation, service configuration, and secret management, emphasizing the ease of connecting MongoDB with Kubernetes services. Additionally, considerations for maintenance and architectural options are discussed to aid in decision-making for deployment strategies.
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!MongoDB The document outlines a presentation by Munish Kapoor on utilizing MongoDB Charts for data analysis, emphasizing that insights from data are crucial for all roles in organizations. It discusses various analytical questions related to movies, such as trends in length and quality, and how to filter and share insights effectively. Key takeaways highlight the importance of making sense of increasing data volumes and leveraging MongoDB Charts for analysis and collaboration.
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your MindsetMongoDB The document discusses the transition from SQL databases to MongoDB, highlighting the mental shift required and providing a comparison of terms and concepts between the two. It outlines the advantages of MongoDB, such as the flexible schema and document-based storage, which allows for easier data manipulation and scaling. Additionally, it includes examples illustrating how to model data and perform operations in both types of databases.
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB .local San Francisco 2020: MongoDB Atlas JumpstartMongoDB The document presents an overview of MongoDB Atlas, a global cloud database service emphasizing its features such as automated maintenance, scalability, enterprise-grade security, and data locality. It highlights the importance of understanding forward-looking statements and risks associated with future developments. Additionally, the document offers insights on how to quickly set up and deploy a MongoDB cluster for users.
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...MongoDB The document discusses guidelines for ordering fields in compound indexes to optimize query performance. It recommends the E-S-R approach: placing equality fields first, followed by sort fields, and range fields last. This allows indexes to leverage equality matches, provide non-blocking sorts, and minimize scanning. Examples show how indexes ordered by these guidelines can support queries more efficiently by narrowing the search bounds.
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++MongoDB The document discusses the capabilities and improvements of the MongoDB 4.2 aggregation framework, highlighting its use in powering queries, updates, and views. It provides various examples of data pipelines, showcasing stages such as $match, $group, and $sort, and emphasizes the performance optimizations and additional features available in this version. Additionally, it covers how to leverage aggregation pipelines for updates and the unification of different languages in data handling.
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...MongoDB The document describes a methodology for data modeling with MongoDB. It begins by recognizing the differences between document and tabular databases, then outlines a three step methodology: 1) describe the workload by listing queries, 2) identify and model relationships between entities, and 3) apply relevant patterns when modeling for MongoDB. The document uses examples around modeling a coffee shop franchise to illustrate modeling approaches and techniques.
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep DiveMongoDB The document discusses MongoDB Atlas Data Lake, which aims to efficiently manage vast amounts of data in the cloud, addressing the challenges enterprises face with data storage and querying. It outlines the architecture, configuration options, and security measures for customer data management while emphasizing the importance of emulating MongoDB's functionalities. Future developments include support for more MongoDB operators, optimizations, new file formats, and integrations with major cloud providers.
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & GolangMongoDB The document provides a comprehensive overview of developing Alexa skills using MongoDB and Go, including connections to MongoDB, the use of AWS Lambda for serverless execution, and configurations for Alexa skills. It outlines the structure of intents and utterances necessary for skill functionality, as well as the importance of using unique and pronounceable invocation names. Additionally, the document offers resources such as a GitHub example and information on obtaining MongoDB Atlas credits.
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...MongoDB The document outlines the features and benefits of the Realm on-device object database, developed for seamless mobile app development, emphasizing offline capabilities and smooth synchronization with MongoDB. It highlights the importance of user experience, real-time data updates, and conflict resolution in app development. Additionally, it provides a small assignment example of a todo app to illustrate the practical application of these concepts.
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...MongoDB Le document aborde comment le machine learning transforme la supply chain en répondant à des défis tels que l'opacité des informations et les déséquilibres de l'offre et de la demande. Il décrit les opérations d'Upply, notamment son utilisation de données massives pour optimiser le matching entre chargeurs et transporteurs, ainsi que son infrastructure technique basée sur des modèles de prévisions avancés. Le document mentionne également l'adoption de MongoDB pour améliorer la gestion des données et les performances de la plateforme.
Ad
Recently uploaded (20)
Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025Safe Software Harness the full potential of AI with FME: From creating high-quality training data to optimizing models and utilizing results, FME supports every step of your AI workflow. Seamlessly integrate a wide range of models, including those for data enhancement, forecasting, image and object recognition, and large language models. Customize AI models to meet your exact needs with FME’s powerful tools for training, optimization, and seamless integration
Artificial Intelligence in the Nonprofit Boardroom.pdf
Artificial Intelligence in the Nonprofit Boardroom.pdfOnBoard OnBoard recently partnered with Microsoft Tech for Social Impact on the AI in the Nonprofit Boardroom Survey, an initiative designed to uncover the current and future role of artificial intelligence in nonprofit governance.
OWASP Barcelona 2025 Threat Model Library
OWASP Barcelona 2025 Threat Model LibraryPetraVukmirovic Threat Model Library Launch at OWASP Barcelona 2025
https://owasp.org/www-project-threat-model-library/
Creating Inclusive Digital Learning with AI: A Smarter, Fairer Future
Creating Inclusive Digital Learning with AI: A Smarter, Fairer FutureImpelsys Inc. Have you ever struggled to read a tiny label on a medicine box or tried to navigate a confusing website? Now imagine if every learning experience felt that way—every single day.
For millions of people living with disabilities, poorly designed content isn’t just frustrating. It’s a barrier to growth. Inclusive learning is about fixing that. And today, AI is helping us build digital learning that’s smarter, kinder, and accessible to everyone.
Accessible learning increases engagement, retention, performance, and inclusivity for everyone. Inclusive design is simply better design.
Data Validation and System Interoperability
Data Validation and System InteroperabilitySafe Software A non-profit human services agency with specialized health record and billing systems. Challenges solved include access control integrations from employee electronic HR records, multiple regulations compliance, data migrations, benefits enrollments, payroll processing, and automated reporting for business intelligence and analysis.
Securing Account Lifecycles in the Age of Deepfakes.pptx
Securing Account Lifecycles in the Age of Deepfakes.pptxFIDO Alliance Securing Account Lifecycles in the Age of Deepfakes
No-Code Workflows for CAD & 3D Data: Scaling AI-Driven Infrastructure
No-Code Workflows for CAD & 3D Data: Scaling AI-Driven InfrastructureSafe Software When projects depend on fast, reliable spatial data, every minute counts.
AI Clearing needed a faster way to handle complex spatial data from drone surveys, CAD designs and 3D project models across construction sites. With FME Form, they built no-code workflows to clean, convert, integrate, and validate dozens of data formats – cutting analysis time from 5 hours to just 30 minutes.
Join us, our partner Globema, and customer AI Clearing to see how they:
-Automate processing of 2D, 3D, drone, spatial, and non-spatial data
-Analyze construction progress 10x faster and with fewer errors
-Handle diverse formats like DWG, KML, SHP, and PDF with ease
-Scale their workflows for international projects in solar, roads, and pipelines
If you work with complex data, join us to learn how to optimize your own processes and transform your results with FME.
TrustArc Webinar - 2025 Global Privacy Survey
TrustArc Webinar - 2025 Global Privacy SurveyTrustArc How does your privacy program compare to your peers? What challenges are privacy teams tackling and prioritizing in 2025?
In the sixth annual Global Privacy Benchmarks Survey, we asked global privacy professionals and business executives to share their perspectives on privacy inside and outside their organizations. The annual report provides a 360-degree view of various industries' priorities, attitudes, and trends. See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar features an expert panel discussion and data-driven insights to help you navigate the shifting privacy landscape. Whether you are a privacy officer, legal professional, compliance specialist, or security expert, this session will provide actionable takeaways to strengthen your privacy strategy.
This webinar will review:
- The emerging trends in data protection, compliance, and risk
- The top challenges for privacy leaders, practitioners, and organizations in 2025
- The impact of evolving regulations and the crossroads with new technology, like AI
Predictions for the future of privacy in 2025 and beyond
AI VIDEO MAGAZINE - June 2025 - r/aivideo
AI VIDEO MAGAZINE - June 2025 - r/aivideo1pcity Studios, Inc AI VIDEO MAGAZINE - r/aivideo community newsletter – Exclusive Tutorials: How to make an AI VIDEO from scratch, PLUS: How to make AI MUSIC, Hottest ai videos of 2025, Exclusive Interviews, New Tools, Previews, and MORE - JUNE 2025 ISSUE -
Bridging the divide: A conversation on tariffs today in the book industry - T...
Bridging the divide: A conversation on tariffs today in the book industry - T...BookNet Canada A collaboration-focused conversation on the recently imposed US and Canadian tariffs where speakers shared insights into the current legislative landscape, ongoing advocacy efforts, and recommended next steps. This event was presented in partnership with the Book Industry Study Group.
Link to accompanying resource: https://bnctechforum.ca/sessions/bridging-the-divide-a-conversation-on-tariffs-today-in-the-book-industry/
Presented by BookNet Canada and the Book Industry Study Group on May 29, 2025 with support from the Department of Canadian Heritage.
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation Can We Use Rust to Develop Extensions for PostgreSQL?
(POSETTE: An Event for Postgres 2025)
June 11, 2025
Shinya Kato
NTT DATA Japan Corporation
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/from-enterprise-to-makers-driving-vision-ai-innovation-at-the-extreme-edge-a-presentation-from-sony-semiconductor-solutions/
Amir Servi, Edge Deep Learning Product Manager at Sony Semiconductor Solutions, presents the “From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge” tutorial at the May 2025 Embedded Vision Summit.
Sony’s unique integrated sensor-processor technology is enabling ultra-efficient intelligence directly at the image source, transforming vision AI for enterprises and developers alike. In this presentation, Servi showcases how the AITRIOS platform simplifies vision AI for enterprises with tools for large-scale deployments and model management.
Servi also highlights his company’s collaboration with Ultralytics and Raspberry Pi, which brings YOLO models to the developer community, empowering grassroots innovation. Whether you’re scaling vision AI for industry or experimenting with cutting-edge tools, this presentation will demonstrate how Sony is accelerating high-performance, energy-efficient vision AI for all.
Down the Rabbit Hole – Solving 5 Training Roadblocks
Down the Rabbit Hole – Solving 5 Training RoadblocksRustici Software Feeling stuck in the Matrix of your training technologies? You’re not alone. Managing your training catalog, wrangling LMSs and delivering content across different tools and audiences can feel like dodging digital bullets. At some point, you hit a fork in the road: Keep patching things up as issues pop up… or follow the rabbit hole to the root of the problems.
Good news, we’ve already been down that rabbit hole. Peter Overton and Cameron Gray of Rustici Software are here to share what we found. In this webinar, we’ll break down 5 training roadblocks in delivery and management and show you how they’re easier to fix than you might think.
Security Tips for Enterprise Azure Solutions
Security Tips for Enterprise Azure SolutionsMichele Leroux Bustamante Delivering solutions to Azure may involve a variety of architecture patterns involving your applications, APIs data and associated Azure resources that comprise the solution. This session will use reference architectures to illustrate the security considerations to protect your Azure resources and data, how to achieve Zero Trust, and why it matters. Topics covered will include specific security recommendations for types Azure resources and related network security practices. The goal is to give you a breadth of understanding as to typical security requirements to meet compliance and security controls in an enterprise solution.
FIDO Seminar: Evolving Landscape of Post-Quantum Cryptography.pptx
FIDO Seminar: Evolving Landscape of Post-Quantum Cryptography.pptxFIDO Alliance FIDO Seminar: Evolving Landscape of Post-Quantum Cryptography
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptx
FIDO Seminar: New Data: Passkey Adoption in the Workforce.pptxFIDO Alliance FIDO Seminar: New Data: Passkey Adoption in the Workforce
Crypto Super 500 - 14th Report - June2025.pdf
Crypto Super 500 - 14th Report - June2025.pdfStephen Perrenod This OrionX's 14th semi-annual report on the state of the cryptocurrency mining market. The report focuses on Proof-of-Work cryptocurrencies since those use substantial supercomputer power to mint new coins and encode transactions on their blockchains. Only two make the cut this time, Bitcoin with $18 billion of annual economic value produced and Dogecoin with $1 billion. Bitcoin has now reached the Zettascale with typical hash rates of 0.9 Zettahashes per second. Bitcoin is powered by the world's largest decentralized supercomputer in a continuous winner take all lottery incentive network.
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/addressing-evolving-ai-model-challenges-through-memory-and-storage-a-presentation-from-micron/
Wil Florentino, Senior Segment Marketing Manager at Micron, presents the “Addressing Evolving AI Model Challenges Through Memory and Storage” tutorial at the May 2025 Embedded Vision Summit.
In the fast-changing world of artificial intelligence, the industry is deploying more AI compute at the edge. But the growing diversity and data footprint of transformers and models such as large language models and large multimodal models puts a spotlight on memory performance and data storage capacity as key bottlenecks. Enabling the full potential of AI in industries such as manufacturing, automotive, robotics and transportation will require us to find efficient ways to deploy this new generation of complex models.
In this presentation, Florentino explores how memory and storage are responding to this need and solving complex issues in the AI market. He examines the storage capacity and memory bandwidth requirements of edge AI use cases ranging from tiny devices with severe cost and power constraints to edge servers, and he explains how new memory technologies such as LPDDR5, LPCAMM2 and multi-port SSDs are helping system developers to meet these challenges.
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdf
War_And_Cyber_3_Years_Of_Struggle_And_Lessons_For_Global_Security.pdfbiswajitbanerjee38 Russia is one of the most aggressive nations when it comes to state coordinated cyberattacks — and Ukraine has been at the center of their crosshairs for 3 years. This report, provided the State Service of Special Communications and Information Protection of Ukraine contains an incredible amount of cybersecurity insights, showcasing the coordinated aggressive cyberwarfare campaigns of Russia against Ukraine.
It brings to the forefront that understanding your adversary, especially an aggressive nation state, is important for cyber defense. Knowing their motivations, capabilities, and tactics becomes an advantage when allocating resources for maximum impact.
Intelligence shows Russia is on a cyber rampage, leveraging FSB, SVR, and GRU resources to professionally target Ukraine’s critical infrastructures, military, and international diplomacy support efforts.
The number of total incidents against Ukraine, originating from Russia, has steadily increased from 1350 in 2021 to 4315 in 2024, but the number of actual critical incidents has been managed down from a high of 1048 in 2022 to a mere 59 in 2024 — showcasing how the rapid detection and response to cyberattacks has been impacted by Ukraine’s improved cyber resilience.
Even against a much larger adversary, Ukraine is showcasing outstanding cybersecurity, enabled by strong strategies and sound tactics. There are lessons to learn for any enterprise that could potentially be targeted by aggressive nation states.
Definitely worth the read!
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...
Can We Use Rust to Develop Extensions for PostgreSQL? (POSETTE: An Event for ...NTT DATA Technology & Innovation
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...
“From Enterprise to Makers: Driving Vision AI Innovation at the Extreme Edge,...Edge AI and Vision Alliance
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...
“Addressing Evolving AI Model Challenges Through Memory and Storage,” a Prese...Edge AI and Vision Alliance
Automated Slow Query Analysis: Dex the Index Robot
- 1. 0 Eric Sedor Index Automation and Dex June 2013
- 3. 2 Agenda • MongoDB index basics • Indexing tips and tricks • Dex automation • Dex details and demo • Extras
- 4. 3 Some notable MongoDB fundamentals • Good performance starts with indexes – you create them; they don’t just happen • Each query uses at most one index – so index accordingly • The query optimizer is empirical – every so often (~1k writes) MongoDB runs a race between query plans. The first query plan to complete wins. – query plans are also re-run after certain changes to a collection (such as adding an index).
- 5. 4 Proper indexing is critical • Indexes can improve query performance by 2 to 3 orders of magnitude – 1000ms query down to <1ms! • Bad queries don’t just get in their own way, they get in the way of other things too: – write lock, queued operations, page faults • Bad indexing → Memory Apocalypse – without warning, large portions of your working data topple out of memory and must be page-faulted back
- 6. 5 Five key commands db.adventurers.find( {"name" : "Eric", "class": "Wizard"}).explain() db.adventurers.getIndexKeys() db.adventurers.getIndexes() db.adventurers.ensureIndex({"name": 1, "class": 1}, {"background": true}) db.adventurers.dropIndex({"name": 1, "class": 1})
- 7. 6 explain()will reveal a scanAndOrder • scanAndOrder is almost always bad! • If MongoDB is re-ordering to satisfy a sort clause, explain() includes: { scanAndOrder: true } • MongoDB sorts documents in-memory! (very expensive!) – without an index, large result sets are rejected with an error
- 9. 8 An index is a b-tree that maps a sequence of key values to a list of document pointers * “Ben” “Fighter” “Noble” “Eric” “Engineer” “Wizard” { "name": 1, "class": 1 } name-> class-> the order of the keys really matters!
- 10. 9 Index key order determines how the b-tree is constructed This ordering of keys influences how: • applicable an index is to a given query – a query that doesn't include the first field(s) in the index cannot use the index • quickly the scope of possible results is pruned – here is where your data's cardinality weighs in • documents are sorted in result sets – did I mention scanAndOrder was bad?
- 11. 10 Ordering is tricky and especially important with range operators The order of fields in an index should be the: ① fields on which you will query for exact values ② fields on which you will sort ③ fields on which you will query for a range of values ($in, $gt, $lt, etc.) Article explaining this topic in detail: bit.ly/mongoindex
- 12. 11 Put the range field value last in your index diagram at bit.ly/mongoindex
- 13. 12 Put the range field value last in your index diagram at bit.ly/mongoindex
- 14. 13 Put the range field value last in your index diagram at bit.ly/mongoindex
- 15. 14 Put the range field value last in your index diagram at bit.ly/mongoindex
- 16. 15 Slow Hell (like normal hell only slower) What do we do?
- 17. 16 Be warned if you... • Use a variety of query patterns • Give the app user control over queries • Use MongoDB like a relational database • Have many indexes in each collection
- 18. 17 Don’t die the death of a thousand cuts • The most expensive queries are not always the slowest queries. – 50 queries * 20 ms == 1 s That’s 1 second other queries can't use! • Profile your queries and check the <100ms range for a high volume of expensive but relatively fast queries • Remember... bad queries don't just get into their own way!
- 19. 18 Identify the problematic queries • Search the log file – logs any query over 100ms • Use the database profiler ① Turn it on db.setProfilingLevel(1)logs slow queries db.setProfilingLevel(2)logs all queries (helpful but noisy) ② Find the slow queries .sort({millis: -1}) .find({ns: "mongoquest.adventurers"}) .find({op: {$in: ["query", "update", "command"]}) ③ Cleanup db.setProfilingLevel(0) db.system.profile.dropCollection()
- 20. 19 Here’s a hint() if you have too many indexes • The query optimizer might choose a suboptimal index – It’s empirical, so it is vulnerable to poor conditions at query time, especially in high-page-fault environments • Hint your queries to the better index – db.adventurers.find(…).hint({“myIndex”: 1})
- 22. 21 How Dex Works ① Dex iterates over the input (log or profile collection) ② A LogParser or ProfileParser extracts queries from each line of input. ③ Dex passes the query to a QueryAnalyzer. ④ The QueryAnalyzer compares the query to existing indexes (from left to right) ⑤ If an index meeting Dex's criteria does not already exist, Dex suggests the best index for that query
- 23. The Heart of Dex 22 Dex understands that order of fields in an index should be: ① Equivalency checks {a:1} ② Sorts .sort({b: 1}) ③ Range checks {c: {$in: [1, 2]}}
- 24. 23 Using Dex is easy Install using pip: > sudo pip install dex Usage: dex [<options>] uri > dex –f my/mongod/data/path/mongodb.log mongodb://myUser:myPass@myHost:12345/myDb > dex –p mongodb://myUser:myPass@myHost:12345/myDb
- 25. 24 Demo
- 26. 25 'runStats': { 'linesRecommended': 76, 'linesProcessed': 76, 'linesPassed': 93 }, 'results': [ { 'index': '{"name": 1}', 'totalTimeMillis': 410041, 'namespace': 'mongoquest.adventurers', 'queryCount': 2161, 'avgTimeMillis': 189, 'queries': [ '{"q": {"name": "<name>"}}' ] }, ... Example of Dex's output (use –v for a shell command!)
- 27. 26 > dex -f my/mongod/data/path/mongodb.log -n "myFirstDb.collectionOne" mongodb://myUser:myPass@myHost:12345/myFirst Db > dex -f my/mongod/data/path/mongodb.log -n "*.collectionOne" mongodb://myUser:myPass@myHost:12345/admin > dex -f my/mongod/data/path/mongodb.log -n "myFirstDb.*" -n "mySecondDb.*" mongodb://myUser:myPass@myHost:12345/admin Note the auth to the admin db to run against more than one db! The namespace filter (-n)
- 28. 27 For when you want current results, not prior results. > dex –w -f my/mongod/data/path/mongodb.log mongodb://myUser:myPass@myHost:12345/myFirst Db > dex –w –p –n "dbname.*" mongodb://myUser:myPass@myHost:12345/admin Watch mode (-w)
- 29. 28 Focus on longer-running queries > dex –w -f my/mongod/data/path/mongodb.log mongodb://myUser:myPass@myHost:12345/myFirst Db –s 1000 > dex –w –p mongodb://myUser:myPass@myHost:12345/admin - -slowms 5000 SlowMS (-s/--slowms)
- 30. 29 {parsed: ..., namespace: db.adventurers, queryAnalysis: {analyzedFields: [{fieldName: name, fieldType: EQUIV}, {fieldName: class, fieldType: EQUIV}, fieldCount: N, supported: true|false}, indexAnalysis: {fullIndexes: [], partialIndexes: [{name: 1}] needsRecommendation: true|false }, recommendation: {namespace: mongoquest.adventurers index: {name: 1, class: 1} shellCommand: db.ensureIndex... } } Dex's guts
- 31. 30 Future plans for Dex Dev/Testing now: – Aggregation framework, geospatial queries, map/reduce – min/max/average nscanned and nreturned – scanAndOrder true/false Soon: • Renovation of internals • Improved index recommendations – set-wise optimization of index fields • minimize the number of indexes required to cover all of your queries – order-wise optimization of index fields • measure cardinality for key ordering
- 34. 33 Questions? Thank you and good luck out there! [email protected] www.github.com/mongolab/dex http://mongolab.org http://blog.mongolab.com/2012/06/introducing-dex-the-index-bot/ http://blog.mongolab.com/2012/07/remote-dex/ http://blog.mongolab.com/2012/06/cardinal-ins/ http://blog.mongolab.com/2013/04/thinking-about-arrays-in-mongodb/