SeaweedFS introduction
- 2. SeaweedFS Intro ● Overview ● Internal Architecture ○ Object/Blob store ○ Filer Store ○ S3/Hadoop ○ Notification/Cross-Region Replication
- 5. Overview: What is special? ● Distributed ● Handles large and small files ● Optimized for large amount of small files ● Random access any file ● Low-latency access any file ● Parallel processing
- 6. Overview: APIs ● REST API for object storage ● REST/gRPC API for file system storage ● Hadoop Compatible ● FUSE client to mount file system locally ● S3 API
- 7. Architecture ● Object Storage ● File Storage ● Interface/Client Layer
- 8. Volume Store ● Based on Facebook Haystack paper
- 9. Object Storage Object Storage Master Volume Server Volume Server Volume Server Client Write 1. Http request file id 3. Http upload file with file id 2. Get file id
- 10. Object Storage Object Storage Master Volume Server Volume Server Volume Server Client Write 1. Http request file id 3. Http upload file with file id 2. Get file id Example file id, 3,01637037d6 ● 3 : a volume id ● 01: file key ● 637037d6: file cookie
- 11. Object Storage Object Storage Master Volume Server Volume Server Volume Server Client Read 1. Lookup volume id 3. Http get file with file id 2. Get volume location ● Volume locations can be cached. ● Clients can also subscribe to volume location changes.
- 12. Object Storage File Storage Master Volume Server Volume Server Volume Server Filer Client Upload a file to a directory File Storage Filer Filer Store Local MySql Postgres Redis Cassandra Metadata Blobs S3 API Gateway S3 Clients
- 13. Filer Store Data Layout /a/b/c/ Attr /a/b/c/def.txt Attr FileChunks
- 14. Volume-Aware Clients Object Storage Master Volume Server Volume Server Volume Server Other SeaweedFS Volume-Aware Clients Metadata File Storage Filer Filer Store Local MySql Postgres Redis Cassandra Metadata Blobs Hadoop Client Mounted FUSE Client
- 15. Volume-based data placement ● Volumes are organized with different settings: ○ Collection ■ TTL ■ Replication ● Master randomly assigns a write request to one of the writable volumes. ● Strong consistent writes to all replicas. ● If one replica fails heartbeat, the master marks the volume id as read-only. ● Writes should be assigned to other writable volumes.
- 16. Object Storage Security: per object access control with JWT Master Volume Server Volume Server Volume Server Client 1. Request FileId 3. Upload File with FileId + JWT 2. Get FileId + JWT ● A Json Web Token (JWT) has permission to create/update/delete a file. ● Expires after 10 seconds.
- 17. Secure Volume Server ● Mutual TLS ○ Secure master to volume server admin operations ● JWT ○ Secure object changes Volume servers can be placed anywhere. Any server with some free space can be a volume server. Master Volume Server Volume Server Volume Server Mutual TLS gRPC calls JWT authorized changes
- 18. High Availability: Master Server Object Storage Master Volume Server Volume Server Volume Server Master Master ● Multi-Master cluster ● Leader election with Raft consensus algorithm
- 19. High Availability: Filer Server ● Multiple stateless filer servers ● Shared filer store could be any HA storage solution. File Storage Filer Filer Store MySql Postgres Redis Cassandra Filer Filer
- 20. Scalability: Filer ● Direct blob access. ● Filer store can be any proven store, and simple to add new store: ○ Redis ○ MySql/Postgres ○ Cassandra ○ Interface for any key-value store ● Unlimited files under one directory. ● Blob storage supports multiple filers.
- 21. File Change Notification ● All filer change notifications can be sent to a message queue. ● Protobuf encoded notification. ● Cross-Region replication is built on top of this. File Storage Filer Filer Store Local MySql Postgres Redis Cassandra Metadata Message Queue notifications Kafka AWS SNS/SQS, Azure Service Bus, Google Pub/Sub, NATS and RabbitMQ
- 23. Atomicity Operation Atomicity Note Creating a file yes Deleting a file yes Renaming a file Yes with mysql/postgres. No with redis/leveldb/cassandra. Implemented via database transactions. Renaming a directory Yes with mysql/postgres. No with redis/leveldb/cassandra. Implemented via database transactions. Creating a single directory with mkdir() yes Recursive directory deletion No
- 24. Comparing to HDFS HDFS SeaweedFS File Metadata Storage Single namenode Multiple stateless filers with proven scalable filer store, redis/cassandra/etc. Storing small files Not recommended. Optimized for small files. Parallel data access Yes Yes Hadoop Compatible Yes Yes. (Atomic rename via database transactions.)
- 25. Comparing to CEPH CEPH SeaweedFS Data Placement CRUSH maps of the whole cluster, rather complicated, especially when adding storage. Calculated for each object. Volume level placement, amortized for each object. Storing small files Not optimized. Optimized for small files. Scaling file system metadata MDS dynamically partition subtree Flat and linearly scalable. Easy to set up Mixed reviews Yes
- 26. Design Philosophy ● Scale up each layer independently. ● Batch small files ○ Data placement (CEPH file-level, SeaweedFS volume-level) ○ Tracking (HDFS namenode track blocks, SeaweedFS track volume locations) ○ Easy move/delete/replicate operation.
- 27. Open APIs ● gRPC APIs for admin operations ● HTTP APIs for uploading and serving blobs ● gRPC for filer metadata operations ● Protocol buffer defined metadata
- 28. Future Plan ● Volume Server ○ Async Replica ○ Erasure Coding ○ Tiered Storage ● Integration ○ CSI, docker volume plugin ○ Kerberos ● Tools ○ Auto Balance
- 29. Open APIs for possible extensions ● Build a different filer with striping. ● Build a different replication ● Admin tools ● Custom Encryption ● Async Operations ○ Search ○ Secondary index ● Local cache for cloud files ● CDN