![{ tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Document:](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-10-320.jpg)
![{ tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Fields:](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-11-320.jpg)
![{ tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Values:](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-12-320.jpg)
![{ tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org" }, tags => [" perl" ,"opinion"], posts => 2, } Field types: # object # string # date # nested object # string # string # array of enums # integer](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-13-320.jpg)
![{ tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Nested objects flattened:](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-14-320.jpg)
![{ tweet => "Perl is GREAT!", posted => "2011-08-15", user.name => "Clinton Gormley", user.email => "drtech@cpan.org", tags => [" perl" ,"opinion"], posts => 2, } Nested objects flattened](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-15-320.jpg)
![{ tweet => "Perl is GREAT!", posted => "2011-08-15", user.name => "Clinton Gormley", user.email => "drtech@cpan.org", tags => [" perl" ,"opinion"], posts => 2, } Values analyzed into terms](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-16-320.jpg)
![{ tweet => ['perl','great'], posted => [Date(2011-08-15)], user.name => ['clinton','gormley'], user.email => ['drtech','cpan.org'], tags => [' perl' ,'opinion'], posts => [2], } Values analyzed into terms](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-17-320.jpg)
![Core field types { type => 'string', index => ' analyzed ', # 'Foo Bar' ⇒ [ 'foo', 'bar' ] }](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-26-320.jpg)
![Core field types { type => 'string', index => ' not_analyzed ', # 'Foo Bar' ⇒ [ 'Foo Bar' ] }](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-27-320.jpg)
![Core field types { type => 'string', index => ' no ', # 'Foo Bar' ⇒ [ ] }](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-28-320.jpg)
![Custom Analyzer $c->create_index( index => 'twitter', settings => { analysis => { analyzer => { ascii_html => { type => 'custom', tokenizer => 'standard', filter => [ qw( standard lowercase asciifolding stop ) ], char_filter => ['html_strip'] } } }} );](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-61-320.jpg)
![Searching $result = $es->search( index => ['twitter','facebook'] , type => ['tweet','post'] , );](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-63-320.jpg)
![Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, sort => [{ '_score': 'desc' }] );](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-67-320.jpg)
![Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, sort => [{ '_score': 'desc' }] from => 0, size => 10, );](https://image.slidesharecdn.com/termsofendearment-110817092504-phpapp01/85/Terms-of-endearment-the-ElasticSearch-Query-DSL-explained-68-320.jpg)
Terms of endearment - the ElasticSearch Query DSL explained
- 1. “ Terms of Endearment” The ElasticSearch query language explained Clinton Gormley, YAPC::EU 2011 DRTECH @clintongormley
- 2. search for : “ DELETE QUERY ” We can
- 3. search for : “ DELETE QUERY ” and find : “ deleteByQuery ” We can
- 4. but you can only find what is stored in the database
- 5. Normalise values “ deleteByQuery” 'delete' 'by' 'query' 'deletebyquery'
- 6. Normalise values and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
- 7. Normalise values and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
- 8. Analyse values and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
- 9. What is stored in ElasticSearch?
- 10. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "[email protected]", }, tags => [" perl" ,"opinion"], posts => 2, } Document:
- 11. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "[email protected]", }, tags => [" perl" ,"opinion"], posts => 2, } Fields:
- 12. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "[email protected]", }, tags => [" perl" ,"opinion"], posts => 2, } Values:
- 13. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "[email protected]" }, tags => [" perl" ,"opinion"], posts => 2, } Field types: # object # string # date # nested object # string # string # array of enums # integer
- 14. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "[email protected]", }, tags => [" perl" ,"opinion"], posts => 2, } Nested objects flattened:
- 15. { tweet => "Perl is GREAT!", posted => "2011-08-15", user.name => "Clinton Gormley", user.email => "[email protected]", tags => [" perl" ,"opinion"], posts => 2, } Nested objects flattened
- 16. { tweet => "Perl is GREAT!", posted => "2011-08-15", user.name => "Clinton Gormley", user.email => "[email protected]", tags => [" perl" ,"opinion"], posts => 2, } Values analyzed into terms
- 17. { tweet => ['perl','great'], posted => [Date(2011-08-15)], user.name => ['clinton','gormley'], user.email => ['drtech','cpan.org'], tags => [' perl' ,'opinion'], posts => [2], } Values analyzed into terms
- 18. database table row ⇒ many tables ⇒ many rows ⇒ one schema ⇒ many columns In MySQL
- 19. index type document ⇒ many types ⇒ many documents ⇒ one mapping ⇒ many fields In ElasticSearch
- 20. Create index with mappings $es-> create_index ( index => 'twitter', mappings => { tweet => { properties => { title => { type => 'string' }, created => { type => 'date' } } } } );
- 21. Add a mapping $es-> put_mapping ( index => 'twitter', type => ' user ', mapping => { properties => { name => { type => 'string' }, created => { type => 'date' }, } } );
- 22. Can add to existing mapping
- 23. Can add to existing mapping Cannot change mapping for field
- 24. Core field types { type => 'string', }
- 25. Core field types { type => 'string', # byte|short|integer|long|double|float # date, ip addr, geolocation # boolean # binary (as base 64) }
- 26. Core field types { type => 'string', index => ' analyzed ', # 'Foo Bar' ⇒ [ 'foo', 'bar' ] }
- 27. Core field types { type => 'string', index => ' not_analyzed ', # 'Foo Bar' ⇒ [ 'Foo Bar' ] }
- 28. Core field types { type => 'string', index => ' no ', # 'Foo Bar' ⇒ [ ] }
- 29. Core field types { type => 'string', index => 'analyzed', analyzer => 'default', }
- 30. Core field types { type => 'string', index => 'analyzed', index_ analyzer => 'default', search_ analyzer => 'default', }
- 31. Core field types { type => 'string', index => 'analyzed', analyzer => 'default', boost => 2, }
- 32. Core field types { type => 'string', index => 'analyzed', analyzer => 'default', boost => 2, include_in_all => 1 |0 }
- 33. Standard
- 34. Simple
- 35. Whitespace
- 36. Stop
- 37. Keyword Built in analyzers Pattern
- 38. Language
- 39. Snowball
- 40. Custom
- 41. The Brown-Cow's Part_No. #A.BC123-456 [email protected] keyword: The Brown-Cow's Part_No. #A.BC123-456 [email protected] whitespace: The, Brown-Cow's, Part_No., #A.BC123-456, [email protected] simple: the, brown, cow, s, part, no, a, bc, joe, bloggs, com standard: brown, cow's, part_no, a.bc123, 456, joe, bloggs.com snowball (English): brown, cow, part_no, a.bc123, 456, joe, bloggs.com
- 43. ASCII Folding
- 44. Length
- 45. Lowercase
- 46. NGram
- 47. Edge NGram
- 48. Porter Stem
- 49. Shingle
- 50. Stop
- 52. KStem
- 53. Snowball
- 54. Phonetic
- 55. Synonym
- 56. Compound Word
- 57. Reverse
- 58. Elision
- 59. Truncate
- 60. Unique
- 61. Custom Analyzer $c->create_index( index => 'twitter', settings => { analysis => { analyzer => { ascii_html => { type => 'custom', tokenizer => 'standard', filter => [ qw( standard lowercase asciifolding stop ) ], char_filter => ['html_strip'] } } }} );
- 62. Searching $result = $es->search( index => 'twitter', type => 'tweet', );
- 63. Searching $result = $es->search( index => ['twitter','facebook'] , type => ['tweet','post'] , );
- 64. Searching $result = $es->search( # all indices # all types );
- 65. Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, );
- 66. Searching $result = $es->search( index => 'twitter', type => 'tweet', query b => 'foo' , # b == ElasticSearch::SearchBuilder );
- 67. Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, sort => [{ '_score': 'desc' }] );
- 68. Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, sort => [{ '_score': 'desc' }] from => 0, size => 10, );
- 69. Query DSL
- 70. Queries vs Filters
- 71. Queries vs Filters full text & terms terms only
- 72. Queries vs Filters full text & terms
- 73. relevance scoring terms only
- 74. no scoring
- 75. Queries vs Filters full text & terms
- 78. no scoring
- 79. faster
- 80. Queries vs Filters full text & terms
- 82. slower
- 83. no caching terms only
- 84. no scoring
- 85. faster
- 86. cacheable
- 87. Queries vs Filters full text & terms
- 89. slower
- 90. no caching terms only
- 91. no scoring
- 92. faster
- 93. cacheable Use filters for anything that doesn't affect the relevance score!
- 94. Query only Query DSL: $es->search( query => { text => { title => 'perl' } } ); SearchBuilder: $es->search( query b => { title => 'perl' } );
- 95. Filter only Query DSL: $es->search( query => { constant_score => { filter => { term => { tag => 'perl } } } }); SearchBuilder: $es->search( query b => { -filter => { tag => 'perl' } });
- 96. Query and filter Query DSL: $es->search( query => { filtered => { query => { text => { title => 'perl' } }, filter =>{ term => { tag => 'perl' } } } }); SearchBuilder: $es->search( query b => { title => 'perl', -filter => { tag => 'perl' } });
- 97. Filters
- 98. Filters : equality Query DSL: { term => { tags => 'perl' }} { terms => { tags => ['perl','ruby'] }} SearchBuilder: { tags => 'perl' } { tags => ['perl','ruby'] }
- 99. Filters : range Query DSL: { range => { date => { gte => '2010-11-01', lt => '2010-12-01' }} SearchBuilder: { date => { gte => '2010-11-01', lt => '2011-12-01' }}
- 100. Filters : range (many values) Query DSL: { numeric_range => { date => { gte => '2010-11-01', lt => '2010-12-01 }} SearchBuilder: { date => { ' >= ' => '2010-11-01', ' < ' => '2011-12-01' }}
- 101. Filters : and | or | not Query DSL: { and => [ {term=>{X=>1}}, {term=>{Y=>2}} ]} { or => [ {term=>{X=>1}}, {term=>{Y=>2}} ]} { not => { or => [ {term=>{X=>1}}, {term=>{Y=>2}} ] }} SearchBuilder: { X => 1, Y => 2 } [ X => 1, Y => 2 ] { -not => { X => 1, Y => 2 } } # and { -not => [ X => 1, Y => 2 ] } # or
- 102. Filters : exists | missing Query DSL: { exists => { field => 'title' }} { missing => { field => 'title' }} SearchBuilder: { -exists => 'title' } { -missing => 'title' }
- 103. Filter example SearchBuilder: { -filter => [ featured => 1, { created_at => { gt => '2011-08-01' }, status => { '!=' => 'pending' }, }, ] }
- 104. Filter example Query DSL: { constant_score => { filter => { or => [ { term => { featured => 1 }}, { and => [ { not => { term => { status => 'pending' }}, { range => { created_at => { gt => '2011-08-01' }}}, ] } ] } } }
- 105. Filters : others script
- 106. nested
- 107. has_child
- 108. query
- 109. match_all
- 110. prefix
- 111. limit ids
- 112. type
- 113. geo_distance
- 114. geo_distance_range
- 115. geo_bbox
- 116. geo_polygon
- 117. Text / Analyzed: text
- 118. query_string / field
- 119. flt / flt_field
- 120. mlt / mlt_field Term / Not analyzed: term / terms
- 121. range
- 122. prefix
- 123. fuzzy
- 124. wildcard
- 125. ids
- 126. span queries Combining: bool
- 127. dis_max
- 130. constant_score
- 131. filtered “ Joins”: nested
- 132. has_child
- 133. top_children Queries
- 134. Text / Analyzed: text
- 135. query_string / field
- 136. flt / flt_field
- 137. mlt / mlt_field Term / Not analyzed: term / terms
- 138. range
- 139. prefix
- 140. fuzzy
- 141. wildcard
- 142. ids
- 143. span queries Combining: bool
- 144. dis_max
- 147. constant_score
- 148. filtered “ Joins”: nested
- 149. has_child
- 150. top_children Queries
- 151. Text/Analyzed Queries mapping aware
- 152. Text/Analyzed Queries not_analyzed ⇒ term query
- 153. Text/Analyzed Queries analyzed ⇒ text query using search_analyzer
- 154. Text-Query Family Query DSL: { text => { title => 'great perl' }} Search Builder: { title => 'great perl' }
- 155. Text-Query Family Query DSL: { text => { title => { query => 'great perl' }}} Search Builder: { title => { '=' => { query => 'great perl' }}}
- 156. Text-Query Family Query DSL: { text => { title => { query => 'great perl' , operator => 'and' }}} Search Builder: { title => { '=' => { query => 'great perl', operator => 'and' }}}
- 157. Text-Query Family Query DSL: { text => { title => { query => 'great perl' , fuzziness => 0.5 }}} Search Builder: { title => { '=' => { query => 'great perl', fuzziness => 0.5 }}}
- 158. Text-Query Family Query DSL: { text => { title => { query => 'great perl', type => 'phrase' }}} Search Builder: { title => { '==' => { query => 'great perl', }}}
- 159. Text-Query Family Query DSL: { text => { title => { query => ' great perl ', type => 'phrase' }}} Search Builder: { title => { '==' => { query => ' great perl ', }}}
- 160. Text-Query Family Query DSL: { text => { title => { query => ' perl is great ', type => 'phrase' }}} Search Builder: { title => { '==' => { query => ' perl is great ', }}}
- 161. Text-Query Family Query DSL: { text => { title => { query => ' perl great ', type => 'phrase', slop => 3 }}} Search Builder: { title => { '==' => { query => ' perl great ', slop => 3 }}}
- 162. Text-Query Family Query DSL: { text => { title => { query => ' perl is gr ', type => ' phrase_prefix ', }}} Search Builder: { title => { '^' => { query => ' perl is gr ', }}}
- 163. Query string / Field Lucene Query Syntax aware “ perl is great”~5 AND author:clint* -deleted
- 164. Query string / Field Syntax errors: AND perl is great ” author : clint* -
- 165. Query string / Field Syntax errors: AND perl is great ” author : clint* - ElasticSearch::QueryParser
- 166. Combining: Bool Query DSL: { bool => { must => [ { term => { foo => 1}}, ... ], must_not => [ { term => { bar => 1}}, ... ], should => [ { term => { X => 2}}, { term => { Y => 2}},... ], minimum_number_should_match => 1, }}
- 167. Combining: Bool SearchBuilder: { foo => 1, bar => { '!=' => 1}, -or => [ X => 2, Y => 2], } { -bool => { must => { foo => 1 }, must_not => { bar => 1 }, should => [{ X => 2}, { Y => 2 }], minimum_number_should_match => 1, }}
- 168. Combining: DisMax Query DSL: { dis_max => { queries => [ { term => { foo => 1}}, { term => { bar => 1}}, ] }} SearchBuilder: { -dis_max => [ { term => { foo => 1}}, { term => { bar => 1}}, ], }
- 169. Bool: combines scores DisMax: uses highest score from all matching clauses
- 170. Tweaking relevance:
- 172. Boosting: at index time { properties => { content => { type => “string” }, title => { type => “string” }, }
- 173. Boosting: at index time { properties => { content => { type => “string” }, title => { type => “string”, boost => 2, }, }, }
- 174. Boosting: at index time { properties => { content => { type => “string” }, title => { type => “string”, boost => 2, }, rank => { type => “integer” }, }, _boost => { name => 'rank', null_value => 1.0 }, }
- 175. Boosting: at search time Query DSL: { bool => { should => [ { text => { content => 'perl' }}, { text => { title => 'perl' }}, ] }} SearchBuilder: { content => 'perl', title => 'perl' }
- 176. Boosting: at search time Query DSL: { bool => { should => [ { text => { content => 'perl' }}, { text => { title => { query => 'perl', }}, ] }} SearchBuilder: { content => 'perl', title => { '=' => { query => 'perl' }} }
- 177. Boosting: at search time Query DSL: { bool => { should => [ { text => { content => 'perl' }}, { text => { title => { query => 'perl', boost => 2 }}, ] }} SearchBuilder: { content => 'perl', title => { '=' => { query => 'perl', boost=> 2 }} }
- 178. Boosting: custom_score Query DSL: { custom_score => { query => { text => { title => 'perl' }}, script => “_score * foo /doc['rank'].value”, }} SearchBuilder: { -custom_score => { query => { title => 'perl' }, script => “_score * foo /doc['rank'].value”, }}
- 179. Query example SearchBuilder: { -or => [ title => { '=' => { query => 'custom score', boost => 2 }}, content => 'custom score', ], -filter => { repo => 'elasticsearch/elasticsearch', created_at => { '>=' => '2011-07-01', '<' => '2011-08-01'}, -or => [ creator_id => 123, assignee_id => 123, ], labels => ['bug','breaking'] } }
- 180. Query example Query DSL: { query => { filtered => { query => { bool => { should => [ { text => { content => "custom score" } }, { text => { title => { boost => 2, query => "custom score" } } }, ], }, }, filter => { and => [ { or => [ { term => { creator_id => 123 } }, { term => { assignee_id => 123 } }, ]}, { terms => { labels => ["bug", "breaking"] } }, { term => { repo => "elasticsearch/elasticsearch" } }, { numeric_range => { created_at => { gte => "2011-07-01", lt => "2011-08-01" }}}, ]}, }}