A Scalable Architecture For Distributed Retrieval System In High Concurrency Environment
- 1. A Scalable Architecture for Distributed Retrieval System in High Concurrency Environment Concepts and Implementations of ClusCom and FeDrs Liu Liu
- 2. Motivation Scale retrieval system to billions; Accuracy depends on size of database; High concurrency in real world.
- 3. What is ClusCom A C++ lib for distributed computing in high concurrency environment; Easy to use and employ;
- 4. Structure Tree-based structure: ........................................................ Node 1 Node 3 Node 2
- 5. Lib Interface virtual int handle ( char * to_request, char * to_reply); virtual int synthesize (apr_uint32_t &node, char ** to_synthe, char * to_reply); virtual int manipulate ( char * to_mani, apr_uint32_t& return_size, char * to_reply);
- 6. Some Facts of ClusCom Based on Apache Portable Runtime Library; Self-managed memory allocate; Persistent socket connection pipe with backups; epoll and multithreading hybird server socket.
- 7. How ClusCom Works 7 seperated threads ; Request Pre-handler Dispatcher Handler Classifier Synthesizer Repeater Reply Pre-handler
- 8. Why Seperate network I/O handling and actual data process; 7 threads are not only 7 threads; Request Pre-handler Dispatcher Handler Classifier Synthesizer Repeater Reply Pre-handler
- 9. Look into ClusCom Network I/O handler using persistent socket pipe; Handler and synthesizer using leader-follower thread pool; Request Pre-handler Dispatcher Handler Classifier Synthesizer Repeater Reply Pre-handler
- 10. Deal with High Concurrence REST Connection Persistent Connection ClusCom Server ClusCom Proxy Application Servers
- 11. What is ClusCom Proxy establish non-state connection with application; Maintain a persistent connection with ClusCom server; ClusCom Proxy REST Connection Persistent Connection
- 12. FeDrs Based on ClusCom; Design for retrieval system; Easy to use and employ.
- 13. FeDrs Interface Expose only one function and can be changed on fly; double evaluate_func( const char * input_a, const char * input_b, const char * config);
- 14. Some Facts of FeDrs Data saved with radix tree and 128-bit key; A PHP extension to access FeDrs; Retrieval within groups;