How Kafka Powers the World's Most Popular Vector Database System with Charles Xie and Frank Liu | Current 2022
0 likes1,612 views
The document discusses how Kafka functions as a crucial messaging backbone for Milvus, a vector database designed to manage large volumes of unstructured data using embeddings. It outlines the architecture of Milvus, highlighting its features such as support for various hardware accelerators and cloud-native deployments. Key points include the use of Kafka for centralized logging, enabling efficient stream and batch processing while providing significant horizontal scalability.
How Kafka Powers the World's Most Popular Vector Database System with Charles Xie and Frank Liu | Current 2022
- 1. How Kafka Powers the World’s Most Popular Vector Database Frank Liu
- 2. Speaker Frank Liu Director of Operations & ML Architect [email protected] https://linkedin.com/in/fzliu https://twitter.com/frankzliu
- 3. 01 Unstructured Data and Embeddings CONTENTS 02 Vector Database Overview 03 Milvus Architecture 04 Kafka as a Messaging Backbone
- 4. 01 Unstructured Data and Embeddings
- 6. What is Unstructured Data? Any data that does not conform to a pre-de fi ned data model.
- 7. The Evolution of Data
- 8. Using Vectors to Represent Data Embeddings!
- 10. Vector Database Overview A database purpose-built to store, index, and query large quantities of embeddings.
- 11. Vector Databases in Production
- 12. Milvus Features • Supports hardware accelerators • SIMD support on CPUs • GPU support for faster querying & indexing • Supports key database functions • Data partitioning and data sharing • Filtered queries and searches • Multiple options for indices and similarity metrics • FAISS (HNSW, Flat, PQ), ANNOY, DiskANN, ScANN, etc… • Euclidean, dot product (cosine), boolean • A number of SDKs • Python • Go • Node • Java
- 13. Cloud Nativity • Kubernetes native • Deployment through Helm • Native S3 support • MinIO-based design • Azure Blob and GCS support • Easy on-prem to cloud conversion • Fully distributed • Highly elastic and horizontally scalable • Disaggregated storage and compute (shared storage) • Separate read, write, and background (indexing) services
- 17. Access Layer • Data-related languages (SQL context) • Data de fi nition language: modify/de fi ne database schema • Data management language: store, modify, and retrieve data • Data control language: de fi ne user rights and permissions • Access layer = multiple proxy nodes • Proxy node functions • Manage message ingestion and routing • Points DDL and DCL instructions to coordinators • Point DML to log for for worker consumption
- 18. Coordinator Layer • Root coordinator node • Handles DDL and DCL requests • Data coordinator node • Triggers background data operations ( fl ush, compact, etc) • Manages data node cluster • Maintains metadata of inserted data • Query coordinator node • Manages query node cluster • Index coordinator node • Manages index node cluster • Determines when indexes are built • Maintains index metadata
- 19. Worker Layer • Worker overview • All workers are stateless • All DML requests are handled by workers • Data node • Retrieves incremental log data from log • Packs and stores log data into log snapshots • Processes mutation requests • Query node • Loads indexes and data from object storage • Runs searches and queries • Index node • Builds indexes on inserted data
- 20. Storage Layer • Log broker - Kafka • Streaming data persistence • Execution of reliable asynchronous queries • Event noti fi cation • Metadata storage - etcd • Service registration and health checks • Message consumption checkpoints • Object storage - S3/MinIO • Stores snapshot fi les of logs • Stores index fi les for scalar and vector data • Stores intermediate query results
- 21. Key Takeaways • Single coordinator instance per service type • Coordinators manage corresponding worker node cluster • Data is stored in Collections • Akin to collections in MongoDB or tables in relational databases • Disaggregation of query, indexing, and data • Signi fi cant horizontal scalability • Support for a wide range of application requirements • Message streams are core to Milvus • All data passes through message queue • Kafka innate cloud nativity allows Milvus to easily scale
- 22. 04 Kafka as a Messasging Backbone
- 24. Milvus’ Messaging Backbone • Log as data • Operations are centralized around the log broker • CRUD operations by subscribing to and consuming logs • Pub/sub scheme allows for stream & batch processing • Decoupling of read and write components • Coordinators manage corresponding worker node cluster • Support for both streaming and batched execution • Data nodes read from streams and write to binlog • Streaming uses WAL, batching uses binlog • All requests that change system state go through WAL • Create collection, delete collection • Insert, update, delete vector
- 26. Example: Vector Insert • Loggers are organized into a hash ring • Time Stamp Oracle (TSO) ensures logger consistency • Different channels for different requests • Prevents request type interference • Data nodes subscribe to speci fi c channels • Inserts hashed across multiple channels (+ef fi ciency) • Data nodes can be freely expanded to increase throughput • Convert row-based WAL to column-based binlogs • Kafka’s cloud nativity powers Milvus’ scalability
- 28. THANK YOU FOR LISTENING https://github.com/milvus-io/milvus https://zilliz.com