Introduction to Lucene & Solr and Usecases
Download as PPTX, PDF10 likes10,261 views
Rahul Jain gave a presentation on Lucene and Solr. He began with an overview of information retrieval and the inverted index. He then discussed Lucene, describing it as an open source information retrieval library for indexing and searching. He discussed Solr, describing it as an enterprise search platform built on Lucene that provides distributed indexing, replication, and load balancing. He provided examples of how Solr is used for search, analytics, auto-suggest, and more by companies like eBay, Netflix, and Twitter.
![Query Types
• Single and multi term queries
• ex fieldname:value or title: software engineer
• +, -, AND, OR NOT operators.
• ex. title: (software AND engineer)
• Range queries on date or numeric fields,
• ex: timestamp: [ * TO NOW ] or price: [ 1 TO 100 ]
• Boost queries:
• e.g. title:Engineer ^1.5 OR text:Engineer
• Fuzzy search : is a search for words that are similar in
spelling
• e.g. roam~0.8 => noam
• Proximity Search : with a sloppy phrase query. The close
together the two terms appear, higher the score.
• ex “apache lucene”~20 : will look for all documents where “apache”
word occurs within 20 words of “lucene”
20](https://image.slidesharecdn.com/introductiontolucenesolrandusecases-131030063115-phpapp01/85/Introduction-to-Lucene-Solr-and-Usecases-20-320.jpg)
Introduction to Lucene & Solr and Usecases
- 1. Introduction to Lucene & Solr and Use-cases October Solr/Lucene Meetup Rahul Jain @rahuldausa
- 2. Who am I? Software Engineer 7 years of programming experience Areas of expertise/interest High traffic web applications JAVA/J2EE Big data, NoSQL Information-Retrieval, Machine learning 2
- 3. Agenda • • • • • • • Overview Information Retrieval Lucene Solr Use-cases Solr In Action (demo) Q&A 3
- 4. Information Retrieval (IR) ”Information retrieval is the activity of obtaining information resources (in the form of documents) relevant to an information need from a collection of information resources. Searches can be based on metadata or on full-text (or other content-based) indexing” - Wikipedia 4
- 6. Basic Concepts • tf (t in d) : term frequency in a document • measure of how often a term appears in the document • the number of times term t appears in the currently scored document d • idf (t) : inverse document frequency • measure of whether the term is common or rare across all documents, i.e. how often the term appears across the index • obtained by dividing the total number of documents by the number of documents containing the term, and then taking the logarithm of that quotient. • coord : coordinate-level matching • number of terms in the query that were found in the document, • e.g. term ‘x’ and ‘y’ found in doc1 but only term ‘x’ is found in doc2 so for a query of ‘x’ OR ‘y’ doc1 will receive a higher score. • boost (index) : boost of the field at index-time • boost (query) : boost of the field at query-time 6
- 8. Apache Lucene • Information Retrieval library • Open source • Initially developed by Doug Cutting (Also author of Hadoop) • Indexing and Searching • Inverted Index of documents • High performance, scalable • Provides advanced Search options like synonyms, stopwords, based on similarity, proximity. 8
- 10. Apache Solr • Initially Developed by Yonik Seeley • Enterprise Search platform for Apache Lucene • Open source • Highly reliable, scalable, fault tolerant • Support distributed Indexing (SolrCloud), Replication, and load balanced querying 10
- 11. Apache Solr - Features • • • • • full-text search hit highlighting faceted search (similar to GroupBy clause in RDBMS) near real-time indexing dynamic clustering (e.g. Cluster of most frequent words, tagCloud) • database integration • rich document (e.g., Word, PDF) handling • geospatial search 11
- 12. Solr – schema.xml • Types with index and query Analyzers - similar to data type • Fields with name, type and options • Unique Key • Dynamic Fields • Copy Fields 12
- 13. Solr – Content Analysis • • • • Defines documents Model Index contains documents. Each document consists of fields. Each Field has attributes. – What is the data type (FieldType) – How to handle the content (Analyzers, Filters) – Is it a stored field (stored="true") or Index field (indexed="true") 13
- 14. Solr – Content Analysis • Field Attributes Name : Name of the field Type : Data-type (FieldType) of the field Indexed : Should it be indexed (indexed="true/false") Stored : Should it be stored (stored="true/false") Required : is it a mandatory field (required="true/false") Multi-Valued : Would it will contains multiple values e.g. text: pizza, food (multiValued="true/false") e.g. <field name="id" type="string" indexed="true" stored="true" required="true" multiValued="false" /> 14
- 15. Solr – Content Analysis • FieldType can be – – – – – StrField : String Field TextField : Similar to StrField but can be analyzed BoolField : Boolean Field IntField : Integer Field Trie Based • • • • TrieIntField TrieLongField TrieDateField TrieDoubleField – Few more…. e.g. <fieldType name="string" class="solr.StrField" sortMissingLast="true" omitNorms="true"/> <fieldType name="boolean" class="solr.BoolField" sortMissingLast="true" omitNorms="true"/> <fieldType name="tint" class="solr.TrieIntField" precisionStep="8" positionIncrementGap="0" omitNorms="true"/> <fieldType name="tfloat" class="solr.TrieFloatField" precisionStep="8" positionIncrementGap="0" omitNorms="true"/> <fieldType name="tlong" class="solr.TrieLongField" precisionStep="8" positionIncrementGap="0" omitNorms="true"/> <fieldType name="tdouble" class="solr.TrieDoubleField" precisionStep="8" positionIncrementGap="0" omitNorms="true"/> <fieldType name="tdate" class="solr.TrieDateField" precisionStep="6" positionIncrementGap="0" omitNorms="true"/> Check for more Field Types @ https://cwiki.apache.org/confluence/display/solr/Field+Types+Included+with+Solr 15
- 16. Indexing Pipeline • Analyzer : create tokens using a Tokenizer and/or applying Filters (Token Filters) • Each field can define an Analyzer at index time/query time or the both at same time. Credit : http://www.slideshare.net/otisg/lucene-introduction 16
- 17. Solr – Content Analysis • Commonly used tokenizers: • • • • • • • • StandardTokenizerFactory WhitespaceTokenizerFactory KeywordTokenizerFactory LowerCaseTokenizerFactory PatternTokenizerFactory LetterTokenizerFactory ClassicTokenizerFactory UAX29URLEmailTokenizerFactory 17
- 18. Solr – Content Analysis • Commonly used filters: • • • • • • • • • ClassicFilterFactory LowerCaseFilterFactory CommonGramsFilterFactory EdgeNGramFilterFactory TrimFilterFactory StopFilterFactory TypeTokenFilterFactory PatternCaptureGroupFilterFactory PatternReplaceFilterFactory 18
- 19. Solr – solrconfig.xml • Data dir: where all index data will be stored • Index configuration: ramBufferSize, mergePolicy etc. • Cache configurations: document, query result, filter, field value cache • Query Component • Spell checker component 19
- 20. Query Types • Single and multi term queries • ex fieldname:value or title: software engineer • +, -, AND, OR NOT operators. • ex. title: (software AND engineer) • Range queries on date or numeric fields, • ex: timestamp: [ * TO NOW ] or price: [ 1 TO 100 ] • Boost queries: • e.g. title:Engineer ^1.5 OR text:Engineer • Fuzzy search : is a search for words that are similar in spelling • e.g. roam~0.8 => noam • Proximity Search : with a sloppy phrase query. The close together the two terms appear, higher the score. • ex “apache lucene”~20 : will look for all documents where “apache” word occurs within 20 words of “lucene” 20
- 22. Solr/Lucene Use-cases • • • • • • • • Search Analytics NoSQL datastore Auto-suggestion / Auto-correction Recommendation Engine (MoreLikeThis) Relevancy Engine Solr as a White-List Spatial based Search 22
- 23. Search • Application – Eclipse, Hibernate search • E-Commerce : – Flipkart.com, Infibeam.com, Buy.com, Netflix.com, ebay.com • Jobs – Indeed.com, Simplyhired.com, Naukri.com, Shine.com, • Auto – AOL.com • Travel – Cleartrip.com • Social Network – Twitter.com, LinkedIn.com, mylife.com 23
- 24. Search (Contd.) • Search Engine – Yandex.ru, DuckDuckGo.com • News Paper – Guardian.co.uk • Music/Movies – Apple.com, Netflix.com • Events – Stubhub.com, Eventbrite.com • Cloud Log Management – Loggly.com • Others – Whitehouse.gov 24
- 25. Results Grouping (using facet) Source: www.career9.com, www.indeed.com 25
- 26. Analytics Analytics source : Kibana.org based on ElasticSearch and Logstash Image Source : http://semicomplete.com/presentations/logstash-monitorama-2013/#/8 26
- 27. Autosuggestion Source: www.drupal.org , www.yelp.com 27
- 28. Integration • • • • • Clustering (Solr – Carrot2) Named Entity extraction (Solr-UIMA) SolrCloud (Solr-Zookeeper) Stanbol EntityHub Parsing of many Different File Formats (SolrTika) 28
- 30. Thanks! @rahuldausa on twitter and slideshare http://www.linkedin.com/in/rahuldausa Found Interesting ? Join us @ http://www.meetup.com/Hyderabad-Apache-Solr-Lucene-Group/ 30