Elasticsearch And Ruby [RuPy2012]

Elasticsearch
And Ruby

Karel Minařík

http://karmi.cz

Elasticsearch and Ruby

{elasticsearch in a nutshell}

Built on top of Apache Lucene
Searching and analyzing big data
Scalability
REST API, JSON DSL

Great fit for dynamic languages and web-oriented workflows / architectures

http://www.elasticsearch.org Elasticsearch and Ruby

{ }

{ }
It all started in this gist… (< 200 LOC)


Example

class Results
include Enumerable
attr_reader :query, :curl, :time, :total, :results, :facets

def initialize(search)
response = JSON.parse( Slingshot.http.post("http://localhost:9200/#{search.indices}/_search", search.to_json) )
@query = search.to_json
@curl = %Q|curl -X POST "http://localhost:9200/#{search.indices}/_search?pretty" -d '#{@query}'|
@time = response['took']
@total = response['hits']['total']
@results = response['hits']['hits']
@facets = response['facets']
end

def each(&block)
@results.each(&block)
end
end

Elasticsearch plays nicely with Ruby…


elasticsearch’s Query DSL
curl -‐X POST "http://localhost:9200/articles/_search?pretty=true" -‐d '
{
"query" : {
"filtered" : {
"filter" : {
"range" : {
"date" : {
"from" : "2012-‐01-‐01",
"to" : "2012-‐12-‐31"
}
}
},
"query" : {
"bool" : {
"must" : {
"terms" : {
"tags" : [ "ruby", "python" ]
}
},
"must" : {
"match" : {
"title" : {
"query" : "conference",
"boost" : 10.0
}
}
}
}
}
}
}
}

Example

Tire.search('articles') do
query do
boolean do
must { terms :tags, ['ruby', 'python'] }
must { string 'published_on:[2011-01-01 TO 2011-01-02]' }
end
end
end


Example

tags_query = lambda do |boolean|
boolean.must { terms :tags, ['ruby', 'python'] }
end

published_on_query = lambda do |boolean|
boolean.must { string 'published_on:[2011-01-01 TO 2011-01-02]' }
end

query { boolean &tags_query }
end

query do
boolean &tags_query
boolean &published_on_query
end
end


Example

search = Tire.search 'articles' do
query do
string 'title:T*'
end
filter :terms, tags: ['ruby']
facet 'tags', terms: tags
sort { by :title, 'desc' }
end

search = Tire::Search::Search.new('articles')
search.query { string('title:T*') }
search.filter :terms, :tags => ['ruby']
search.facet('tags') { terms :tags }
search.sort { by :title, 'desc' }


TEH PROBLEM

Designing the Tire library as domain-specific language, from the higher level, and
consequently doing a lot of mistakes in the lower levels.

‣ Class level settings (Tire.configure); cannot connect to two elasticsearch clusters in one codebase
‣ Inconsistent access (methods vs Hashes)
‣ Not enough abstraction and separation of concerns


”Blocks with arguments”
(alternative DSL syntax)

Tire.search do
query do
text :name, params[:q]
end
end

Tire.search do |search|
search.query do |query|
query.text :name, params[:q]
end
end Elasticsearch and Ruby

The Git(Hub) (r)evolution
‣ Lots of contributions... but less feedback
‣ Many contributions focus on specific use case
‣ Many contributions don’t take the bigger picture and codebase conventions into account

‣ Almost every patch needs to be processed, polished, amended
‣ Maintainer: lots of curation, less development — even on this small scale (2K LOC, 7K LOT)

‣ Contributors very eager to code, but a bit afraid to talk

Tire’s Ruby on Rails integration

$ rails new myapp
-‐m "https://raw.github.com/karmi/tire/master/examples/rails-‐application-‐template.rb"

‣ Generate a fully working Rails application with a single command
‣ Downloads elasticsearch if not running, creates the application, commits
every step, seeds the example data, launches the application on a free port, …
‣ Tire::Results::Item fully compatible with Rails view / URL helpers

‣ Any ActiveModel compatible OxM supported
‣ Rake task for importing data (using pagination libraries)


Rails integration baked in
‣ No proper separation of concerns / layers
‣ People expect everything to be as easy as that
‣ Tire::Results::Item baked in, not opt-in, masquerades as models

‣ People consider ActiveRecord the only OxM in the world

…

Persistence extension

Rails extensions

ActiveRecord extensions

ActiveModel integration

The Ruby DSL

Base library (HTTP, JSON, API)

https://rubygems.org
https://github.com/rubygems/rubygems.org/pull/455

“Search”
class Rubygem < ActiveRecord::Base
# ...

def self.search(query)
conditions = <<-SQL
versions.indexed and
(upper(name) like upper(:query) or
upper(translate(name, '#{SPECIAL_CHARACTERS}', '#{' ' *
SPECIAL_CHARACTERS.length}')) like upper(:query))
SQL

where(conditions, {:query => "%#{query.strip}%"}).
includes(:versions).
by_downloads
end
end

https://github.com/rubygems/rubygems.org/blob/master/app/models/rubygem.rb Elasticsearch and Ruby

1

2

3

4
5

6

Adding search to an existing application

https://github.com/karmi/rubygems.org/compare/search-steps

“Hello Cloud” with Chef Server

http://git.io/chef-hello-cloud
‣ Deploy Rubygems.org on EC2 (or locally with Vagrant) from a “zero state”
‣ 1 load balancer (HAproxy), 3 application servers (Thin+Nginx)
‣ 1 database node (PostgreSQL, Redis)
‣ 2 elasticsearch nodes
‣ Install Ruby 1.9.3 via RVM
‣ Clone the application from GitHub repository
‣ init.d scripts and full configuration for every component
‣ Restore data from backup (database dump) and import into search index
‣ Monitor every part of the stack


Elasticsearch And Ruby [RuPy2012]

More Related Content

What's hot (20)

Similar to Elasticsearch And Ruby [RuPy2012] (20)

More from Karel Minarik (20)

Recently uploaded (20)

Elasticsearch And Ruby [RuPy2012]