Phpconf2008 Sphinx En
- 1. The riddles of the Sphinx Full-text engine anatomy atlas
- 2. Who are you ? Sphinx – FOSS full-text search engine
- 3. Who are you ? Sphinx – FOSS full-text search engine Good at playing ball
- 4. Who are you ? Sphinx – FOSS full-text search engine Good at playing ball Good at not playing ball
- 5. Who are you ? Sphinx – FOSS full-text search engine Good at playing ball Good at not playing ball Good at passing the ball to a team-mate
- 6. Who are you ? Sphinx – FOSS full-text search engine Good at playing ball Good at not playing ball Good at passing the ball to a team-mate Good at many other “inferior” games “ faceted” search , geosearch , snippet extraction, multi-queries, IO throttling, and 1 0-20 other interesting directives
- 7. What are you here for? What will not be covered? No entry-level “what’s that Sphinx and what’s in it for me” overview No long quotes from the documentation No C++ architecture details
- 8. What are you here for? What will not be covered? No entry-level “what’s that Sphinx and what’s in it for me” overview No long quotes from the documentation No C++ architecture details What will be? How does it generally work inside How things can be optimized How things can be parallelized
- 9. Chapter 1. Engine insides
- 10. Total workflow Indexing first Searching second
- 11. Total workflow Indexing first Searching second There are data sources (what to fetch, where from) There are indexes What data sources to index How to process the incoming text Where to put the results
- 12. How indexing works In two acts , with an intermission Phase 1 – collecting documents Fetch the documents ( loop over the sources ) Split the documents into words Process the words ( morphology , * fixes ) Replace the words with their wordid’s (CRC32/64) Emit a number of temp files
- 13. How indexing works Phase 2 – sorting hits Hit ( occurrence) is a (docid,wordid,wordpos) record Input is a number of partially sorted (by wordid) hit lists The incoming lists are merge-sorted Output is essentially a single fully sorted hit list Intermezzo Collect and sort MVA values Sort ordinals Sort extern attributes
- 14. Dumb & dumber The index format is… simple Several sorted lists Dictionary ( the complete list of wordid’s ) Attributes ( only if docinfo=extern) Document lists ( for each keyword ) Hit lists ( for each keyword ) Everything is laid out linearly, good for IO
- 15. How searching works For each local index Build a list of candidates ( documents that satisfy the full-text query ) Filter ( the analogy is WHERE) Rank ( compute the documents’ relevance values ) Sort ( the analogy is ORDER BY) Group ( the analogy is GROUP BY) Merge the results from all the local indexes
- 16. 1. Searching cost Building the candidates list 1 keyword = 1 + IO (document list) Boolean operations on document lists Cost is proportional (~) to the lists lengths That is , to the sum of all the keyword frequencies In case of phrase/proximity/etc search , there also will be operations on hit lists – approx. 2x IO/CPU
- 17. 1. Searching cost Building the candidates list 1 keyword = 1 + IO (document list) Boolean operations on document lists Cost is proportional (~) to the lists lengths That is , to the sum of all the keyword frequencies In case of phrase/proximity/etc search , there also are operations on hit lists – approx. 2x IO/CPU Bottom line – “The Who” are really bad
- 18. 2. Filtering cost docinfo=inline Attributes are inlined in the document lists ALL the values are duplicated MANY times! Immediately accessible after disk read docinfo=extern Attributes are stored in a separate list ( file ) Fully cached in RAM Hashed by docid + binary search Simple loop over all filters Cost ~ number of candidates and filters
- 19. 3. Ranking cost Direct – depends on the ranker To account for keyword positions – Helps the relevancy But costs extra resources – double impact ! Cost ~ number of results Most expensive – phrase proximity + BM25 Most cheap – none (weight=1) Indirect – induced in the sorting
- 20. 4. Sorting cost Cost ~ number of results Also depends on the sorting criteria (documents will be supplied in @id asc order) Also depends on max_matches The more the max, the worse the server feels 1-10K is acceptable , 100K is way too much 10-20 is not enough (makes little sense)
- 21. 5. Grouping cost Grouping is internally a kind of sorting Cost affected by the number of results, too Cost affected by max_matches, too Additionally , max_matches setting affects @count and @distinct precision
- 22. Chapter 2. Optimizing things
- 23. How to optimize queries Partitioning the data Choosing ranking vs. sorting mode Filters vs. keywords Filters vs. manual MTF Multi queries
- 24. How to optimize queries Partitioning the data Choosing ranking vs. sorting mode Filters vs. keywords Filters vs. manual MTF Multi queries Last line of defense – Three Big Buttons
- 25. 1. Partitioning the data Swiss army knife, for different tasks Bound by indexing time? Partition , re-index the recent changes only Bound by filtering ? Partition , search the needed indexes only Bound by CPU/HDD? Partition, move out to different cores/HDDs/boxes
- 26. 1a. Partitioning vs. indexing Vital to keep the balance right Under-partition – and indexing will be slow Over-partition – and searching will be slow 1-10 indexes – work reasonably well Some users are fine with 50+ (30+24...) Some users are fine with 2000+ (!!!)
- 27. 1b. Partitioning vs. filtering Totally, 100% dependent on production query statistics Analyze your very own production logs Add comments if needed (3 rd arg to Query()) Justified only if the amount of processed data is going to decrease significantly Move out last week’s documents – yes Move out English-only documents – no (!)
- 28. 1c. Partitioning vs. CPU/HDD Use a distributed index , explicitly map the chunks to physical devices Point searchd “at itself” – index dist1 { type = distributed local = chunk01 agent = localhost:3312:chunk02 agent = localhost:3312:chunk03 agent = localhost:3312:chunk04 }
- 29. 1 c. How to find CPU/HDD bottlenecks Three standard tools vmstat – what’s the CPU busy with? how busy is it? oprofile – specifically who eats the CPU ? iostat – how busy is the HDD? Also use logs , also use searchd --iostats option Normally everything is clear (us/sy/bi/bo…), but! Caveat – HDD might be iops bound Caveat – CPU load from Sphinx might be induced and “hidden” in sy
- 30. 2. Ranking Can now be very different ( so called rankers in extended2 mode ) Default ranker – phrase+BM25, accounts for keyword positions – not for free Sometimes it’s ok to use simpler ranker Sometimes @weight is ignored at all совсем ( searching for ipod , sorting by price ) Sometimes you can save on ranker
- 31. 3. Filters vs. keywords Well-known trick When indexing , add a special, fake keyword to the document (_authorid123) When searching, add it to the query Obvious questions What’s faster, what’s better? Simple answer Count the change before moving away from the cashier
- 32. 3. Filters vs. keywords Cost of searching ~ keyword frequencies Cost of filtering ~ number of candidates Searching – CPU+IO , filtering – CPU only Fake keyword frequency = filter value selectivity Frequent value + few candidates -> bad! Rare value + many candidates -> good!
- 33. 4. Filters vs. manual MTF Filters are looped over sequentially In the order specified by the app! Narrowest filter – better at the start Widest filter – better at the end Does not matter if you use fake keywords Exercise to the reader – why?
- 34. 5. Multi-queries Any queries can be sent together in a batch Always saves on network roundtrip Sometimes allows the optimizer to trigger Especially important and frequent case – different sorting/grouping modes 2x+ optimization for “faceted” searches
- 35. 5. Multi-queries $client = new SphinxClient (); $q = “laptop”; // coming from website user $client->SetSortMode ( SPH_SORT_EXTENDED, “@weight desc”); $client->AddQuery ( $q, “products” ); $client->SetGroupBy ( SPH_GROUPBY_ATTR, “vendor_id” ); $client->AddQuery ( $q, “products” ); $client->ResetGroupBy (); $client->SetSortMode ( SPH_SORT_EXTENDED, “price asc” ); $client->SetLimit ( 0, 10 ); $result = $client->RunQueries ();
- 36. 6. Three Big Buttons If nothing else helps … Cutoff ( см . SetLimits()) Forcibly stops searching after first N matches Per-index, not overall MaxQueryTime ( см. SetMaxQueryTime()) Forcibly stops searching after M milli-seconds Per-index, not overall
- 37. 6. Three Big Buttons If nothing else helps … Consulting We can notice the unnoticed We can implement the unimplemented
- 38. Chapter 3. Parallelization sample
- 39. Combat mission Got ~160M cross-links Needed misc reports (by domains -> groupby) *************************** 1. row *************************** domain_id: 440682 link_id: 15 url_from: http://www.insidegamer.nl/forum/viewtopic.php?t=40750 url_to: http://xbox360achievements.org/content/view/101/114/ anchor: NULL from_site_id: 9835 from_forum_id: 1818 from_author_id: 282 from_message_id: 2586 message_published: 2006-09-30 00:00:00 ...
- 40. Tackling – one Partitioned the data 8 boxes , 4x CPU, ~5M links per CPU Used Sphinx In theory, we could had used MySQL It practice, way too complicated Would had resulted in 15-20M+ rows/CPU Would had resulted in “manual” aggregation code
- 41. Tackling – two Extracted “interesting parts” of the URL when indexing, using an UDF Replaced the SELECT with full-text query *************************** 1. row *************************** url_from: http://www.insidegamer.nl/forum/viewtopic.php?t=40750 urlize(url_from,0): www$insidegamer$nl insidegamer$nl insidegamer$nl$forum insidegamer$nl$forum$viewtopic.php insidegamer$nl$forum$viewtopic.php$t=40750 urlize(url_from,1): www$insidegamer$nl insidegamer$nl insidegamer$nl$forum insidegamer$nl$forum$viewtopic.php
- 42. Tackling – three 64 indexes 4 searchd instances per box , by CPU/HDD count 2 indexes (main+delta) per CPU All searched in parallel Web box queries the main instance at each box Main instance queries itself and other 3 copies Using 4 instances, because of startup/update Using plain HDDs , because of IO stepping
- 43. Results The precision is acceptable “ Rare” domains – precise results “ Frequent” domains – precision within 0.5% Average query time – 0.125 sec 90% queries – under 0.227 sec 95% queries – under 0.352 sec 99% queries – under 2.888 sec
- 44. The end