High Concurrency Architecture at TIKI
7 likes3,676 views
This document discusses high concurrency architectures at TIKI. It describes Pegasus, the highest throughput API, which uses caching, compression, and a non-blocking architecture to handle over 200k requests per minute with sub-2ms latency. It also describes Arcturus, the high concurrency inventory API, which uses an in-memory ring buffer, Kafka for ordering, and asynchronous database flushing to handle millions of inventory transactions per second with eventual consistency. Key techniques discussed include non-blocking designs, caching, compression, ordering queues, and asynchronous data replication.
![Arcturus - Batching marshaller
vBatching: use multi thread marshaller
● Pros
- Fast (400k-600k ops/s*)
- Avoid multi key locking
- Write only most updated value
● Cons
- Inconsistent transaction offset
→ use when you just want to make your code as fast
as possible.
* Macbook pro 2016 15’’ 2.7GHz Core i7, 16G ram. Without I/O, single update per
transaction
*** Test without I/O: [Single ops vbatch] Total time running for 1000000 transactions: 1085 ms; at rate: 921,658.99 ops/s](https://image.slidesharecdn.com/highconcurrencyarchitecturetiki-190908005606/85/High-Concurrency-Architecture-at-TIKI-19-320.jpg)
![Arcturus - Batching marshaller
hBatching: use single marshaller thread
● Pros
- Ensure transaction offset
- Ordered db updating
● Cons
- A little bit slower than vBatching.
→ use when you need to re-create application state
after fail.
*** Test without I/O: [Single ops hbatch] Total time running for 1000000 transactions: 1056 ms; at rate: 946,969.70 ops/s](https://image.slidesharecdn.com/highconcurrencyarchitecturetiki-190908005606/85/High-Concurrency-Architecture-at-TIKI-20-320.jpg)
High Concurrency Architecture at TIKI
- 2. Contributors Bùi Anh Dũng Principal Engineer Nguyễn Hoàng Bách Senior Principal Engineer Phan Công Huân Senior Software Engineer Lê Minh Nghĩa Senior Architect Trần Nguyên Bản Principal Engineer
- 3. Agenda - Principles - Pegasus - Highest throughput API - Arcturus - High concurrency inventory API - Conclusion
- 4. Principles - Use local memory to handle high concurrency transaction - Non blocking architecture - Trade-offs consistency vs eventual consistency - Reliable replication - Authority: each service owns its problems.
- 5. Pegasus - Highest Throughput API at TIKI Problems: - Handle the most of traffic to fetch product information - Has to handle at least 10k request/s, tp95 < 5ms
- 7. Pegasus - Architecture Practises: - Cache product data in local memory - Subscribe product change event to invalid cache - Non-Blocking http web server - Compress to reduce payload size Technology: - Java - Guava - In Memory Cache - Gridgo - Async IO & Event driven framework
- 8. Pegasus - Compression Two cases: - Get single product by id - Get multiple product by a list id Solutions: - Using gzip to compress product data to store in cache. Reduce from 200kb text to 3Kb - Handle single product request: return compressed gzip bytes to client directly - Handle multiple product request: store plain product data and merge them to build a list of products, then use Snappy format to compress data and return to client - Gzip format compress better than Snappy and is supported natively by most http client, but Snappy compress faster and use less CPU - Output cache can be enabled in hot-deal situation
- 9. Pegasus - Technology - Java - Gridgo - Guava - Kafka
- 10. Pegasus - Benchmark Benchmark use WRK tools, 4VM, L4 LB, GCP: - 90%<6.6ms - 96k request/s Production - 200k request/minutes (all platforms) - <2ms Cache hit: 85%. Increased to 95% with Consistent Hashing With Bounded Load. Average payload: Single (3KB), Multi (60KB)
- 12. Arcturus - High concurrency Inventory API Problems: - Handle inventory transaction when customers place orders - Handle high concurrency transaction for extreme hot deals with very cheap and low quantity. Exp: Only ten 1đ iPhone XS, only in ten minutes from 9AM to 9h10 AM. - Guarantee eventual consistency of inventory data between many systems
- 13. Arcturus - Context Diagram
- 15. Arcturus - Problems and Solutions Problems: - Many customers may place order at the same time, especially for hot skus. - System has to be deterministic and can be recovered after crashing Solutions: - All write requests are published to Kafka with only one partition to guarantee the ordering of message and recover if system crashes. - Non Blocking In Memory Cache for both read and write - All changes are flushed to database asynchronously - Recovery based on the offset of write command
- 16. Arcturus - In Memory Cache Data Structure Problem: - There will be a race condition when many customer place the same skus at the same time - Building an in memory cache structure that can buffer data changes and flush to database asynchronously Approach: - Each record have a key (string, bytes…) and a value (8 bytes long number). - A transaction is a set of record updates. - Transactions must be ordered (e.g. key_1 must +3 before can be -1). - It can be millions of keys and process upto 10k TPS. - Using ring buffer data structure to guarantee the ordering of changes * The hardest thing is to keep everything ordered when make it fast enough.
- 17. Arcturus - Old solution - Multi threading application - DB transaction, row locking Pros - Strong consistency - Simple implementation Cons - Slow - Deadlock/timeout implicit risk
- 18. Arcturus - LMax Architecture - Non Blocking architecture using single thread business logic processor - Use Ring Buffer data structure to communicate between processors - Handle million transaction per seconds
- 19. Arcturus - Batching marshaller vBatching: use multi thread marshaller ● Pros - Fast (400k-600k ops/s*) - Avoid multi key locking - Write only most updated value ● Cons - Inconsistent transaction offset → use when you just want to make your code as fast as possible. * Macbook pro 2016 15’’ 2.7GHz Core i7, 16G ram. Without I/O, single update per transaction *** Test without I/O: [Single ops vbatch] Total time running for 1000000 transactions: 1085 ms; at rate: 921,658.99 ops/s
- 20. Arcturus - Batching marshaller hBatching: use single marshaller thread ● Pros - Ensure transaction offset - Ordered db updating ● Cons - A little bit slower than vBatching. → use when you need to re-create application state after fail. *** Test without I/O: [Single ops hbatch] Total time running for 1000000 transactions: 1056 ms; at rate: 946,969.70 ops/s
- 22. Arcturus - Technology - Java - Kafka - ZeroMQ - Gridgo - MySQL - LMax/Ring Buffer
- 23. Conclusion - Has to trade-off between consistency and eventual consistency - In Memory is a great way to improve the performance - Reliable replication is the key to split and scale system - Non Blocking architecture is a great way to utilize hardware resources efficiently.