Service-Oriented Design and Implement with Rails3

Service-Oriented
Design and Implement
with Rails 3
ihower @ Ruby Tuesday
2010/12/15

About Me
• a.k.a. ihower
• http://ihower.tw

• http://twitter.com/ihower

• Rails Developer since 2006
• The Organizer of Ruby Taiwan Community
• http://ruby.tw
• http://rubyconf.tw

Agenda
• What’s SOA
• Why SOA
• Considerations
• The tool set overview
• Service side implement
• Client side implement
• Library packaging
• Caching

What’s SOA
Service oriented architectures

• “monolithic” approach is not enough
• SOA is a way to design complex applications
by splitting out major components into
individual services and communicating via
APIs.
• a service is a vertical slice of functionality:
database, application code and caching layer

a monolithic web app example
request

Load
Balancer

WebApps

Database

a SOA example
request

Load
request
Balancer

WebApp WebApps
for Administration for User

Services A Services B

Database Database

Why SOA? Isolation
• Shared Resources
• Encapsulation
• Scalability
• Interoperability
• Reuse
• Testability
• Reduce Local Complexity

Shared Resources
• Different front-web website use the same
resource.
• SOA help you avoiding duplication databases
and code.
• Why not only shared database?
• code is not DRY WebApp
for Administration
WebApps
for User

• caching will be problematic
Database

Encapsulation
• you can change underly implementation in
services without affect other parts of system
• upgrade library
• upgrade to Ruby 1.9
• upgrade to Rails 3
• you can provide API versioning

Scalability1: Partitioned
Data Provides
• Database is the ﬁrst bottleneck, a single DB server
can not scale. SOA help you reduce database load
• Anti-pattern: only split the database
• model relationship is broken WebApps

• referential integrity
• increase code complexity Database
A
Database
B

• Myth: database replication can not help you speed
and consistency

Scalability 2: Caching

• SOA help you design caching system easier
• Cache data at the right place and expire
at the right times
• Cache logical model, not physical
• You do not need cache view everywhere

Scalability 3: Efﬁcient
• Different components have different task
loading, SOA can scale by service.

WebApps

Load
Balancer Load
Balancer

Services A Services A Services B Services B Services B Services B

Security

• Different services can be inside different
ﬁrewall
• You can only open public web and
services, others are inside ﬁrewall.

Interoperability
• HTTP is the most common interface, SOA
help you integrate them:
• Multiple languages
• Internal system e.g. Full-text searching engine
• Legacy database, system
• External vendors

Reuse

• Reuse across multiple applications
• Reuse for public APIs
• Example: Amazon Web Services (AWS)

Testability

• Isolate problem
• Mocking API calls
• Reduce the time to run test suite

Reduce Local
Complexity
• Team modularity along the same module
splits as your software
• Understandability: The amount of code is
minimized to a quantity understandable by
a small team
• Source code control

Design considerations

• Partition into Separate Services
• API Design
• Which Protocol

How to partition into
Separate Services
• Partitioning on Logical Function
• Partitioning on Read/Write Frequencies
• Partitioning on Minimizing Joins
• Partitioning on Iteration Speed

on Iteration Speed
• Which parts of the app have clear deﬁned
requirements and design?
• Identify the parts of the application which
are unlikely to change.
• For example:
The ﬁrst version data storage is using
MySQL, but may change to NoSQL in the
future without affecting front-app.

on Logical Function

• Higher-level feature services
• articles, photos, bookmarks...etc
• Low-level infrastructure services
• a shared key-value store, queue system

On Read/Write
Frequencies
• Ideally, a service will have to work only with
a single data store
• High read and low write: the service should
optimize a caching strategy.
• High write and low read: don’t bother with
caching

On Join Frequency

• Minimize cross-service joins.
• But almost all data in an app is joined to
something else.
• How often particular joins occur? by read/
write frequency and logical separation.
• Replicate data across services
(For example: a activity stream by using messaging)

API Design Guideline
• Send Everything you need
• Unlike OOP has lots of ﬁnely grained method calls

• Parallel HTTP requests
• for multiple service requests

• Send as Little as Possible
• Avoid expensive XML

Versioning
• Be able run multiple versions in parallel:
Clients have time to upgrade rather than having to
upgrade both client and server in locks step.
• Ideally, you won’t have to run multiple versions for
very long
• Two solutions:
• Including a Version in URIs
• Using Accept Headers for Versioning
(disadvantage: HTTP caching)

Physical Models &
Logical Models
• Physical models are mapped to database
tables through ORM. (It’s 3NF)
• Logical models are mapped to your
business problem. (External API use it)
• Logical models are mapped to physical
models by you.

Logical Models
• Not relational or normalized
• Maintainability
• can change with no change to data store
• can stay the same while the data store
changes
• Better ﬁt for REST interfaces
• Better caching

Which Protocol?

• SOAP
• XML-RPC
• REST

RESTful Web services
• Rails way
• Easy to use and implement
• REST is about resources
• URI
• HTTP Verbs: GET/PUT/POST/DELETE
• Representations: HTML, XML, JSON...etc

The tool set

• Web framework
• XML Parser
• JSON Parser
• HTTP Client
• Model library

Web framework

• Ruby on Rails, but we don’t need afull features.
(Rails3 can be customized because it’s lot more
modular. We will discuss it later)

• Sinatra: a lightweight framework
• Rack: a minimal Ruby webserver interface
library

ActiveResource

• Mapping RESTful resources as models in a
Rails application.
• Use XML by default
• But not useful in practice, why?

XML parser

• http://nokogiri.org/
• Nokogiri ( ) is an HTML, XML, SAX, and
Reader parser. Among Nokogiri’s many
features is the ability to search documents
via XPath or CSS3 selectors.

JSON Parser

• http://github.com/brianmario/yajl-ruby/
• An extremely efﬁcient streaming JSON
parsing and encoding library. Ruby C
bindings to Yajl

HTTP Client

• How to run requests in parallel?
• Asynchronous I/O
• Reactor pattern (EventMachine)
• Multi-threading
• JRuby

Typhoeus
http://github.com/pauldix/typhoeus/

• A Ruby library with native C extensions to
libcurl and libcurl-multi.
• Typhoeus runs HTTP requests in parallel
while cleanly encapsulating handling logic

Typhoeus: Quick
example
response = Typhoeus::Request.get("http://www.pauldix.net")
response = Typhoeus::Request.head("http://www.pauldix.net")
response = Typhoeus::Request.put("http://localhost:3000/posts/1",
:body => "whoo, a body")
response = Typhoeus::Request.post("http://localhost:3000/posts",
:params => {:title => "test post", :content => "this is my test"})
response = Typhoeus::Request.delete("http://localhost:3000/posts/1")

Hydra handles requests
but not guaranteed to run in any particular order
HYDRA = Typhoeus::HYDRA.new

a = nil
request1 = Typhoeus::Request.new("http://example1")
request1.on_complete do |response|
a = response.body
end
HYDRA.queue(request1)

b = nil
request2 = Typhoeus::Request.new("http://example1")
request2.on_complete do |response|
b = response.body
end
HYDRA.queue(request2)

HYDRA.run # a, b are set from here

a asynchronous method
def foo_asynchronously
request = Typhoeus::Request.new( "http://example" )
request.on_complete do |response|
result_value = ActiveSupport::JSON.decode(response.body)
# do something
yield result_value
end

self.hydra.queue(request)
end

Usage
result = nil
foo_asynchronously do |i|
result = i
end

foo_asynchronously do |i|
# Do something for i
end

HYDRA.run
# Now you can use result1 and result2

a synchronous method

def foo
result = nil
foo_asynchronously { |i| result = i }
self.hydra.run
result
end

Physical Models
mapping to database directly

• ActiveRecord
• DataMapper
• MongoMapper, MongoId

Logical Models

• ActiveModel: an interface and modules can
be integrated with ActionPack helpers.
• http://ihower.tw/blog/archives/4940

integrated with helper?

• For example:
• link_to post_path(@post)
• form_for @post
• @post.errors

A basic model
class YourModel
extend ActiveModel::Naming
include ActiveModel::Conversion
include ActiveModel::Validations

def persisted?
false
end
end

without validations
class YourModel

def persisted?
false
end

def valid?() true end

def errors
@errors ||= ActiveModel::Errors.new(self)
end
end

Many useful modules
• MassAssignmentSecurity
• Serialization
• Callback
• AttributeMethods
• Dirty
• Observing
• Translation

Serializers
class Person

include ActiveModel::Serializers::JSON
include ActiveModel::Serializers::Xml

attr_accessor :name

def attributes
@attributes ||= {'name' => 'nil'}
end

end

person = Person.new
person.serializable_hash # => {"name"=>nil}
person.as_json # => {"name"=>nil}
person.to_json # => "{"name":null}"
person.to_xml # => "<?xml version="1.0" encoding="UTF-8"?>
n<serial-person...

Mass Assignment

class YourModel
# ...
def initialize(attributes = {})
if attributes.present?
attributes.each { |k, v| send("#{k}=", v) if respond_to?("#{k}=") }
end
end
end

YourModel.new( :a => 1, :b => 2, :c => 3 )

MassAssignmentSecurity
class YourModel
# ...
include ActiveModel::MassAssignmentSecurity

attr_accessible :first_name, :last_name

def initialize(attributes = {})
if attributes.present?
sanitize_for_mass_assignment(attributes).each { |k, v| send("#{k}=", v) if
respond_to?("#{k}=") }
end
end
end

Scenario we want to
implement
• an Users web service, which provide basic
CRUD functions.
• an web application with the Users client
library

Customized Rails3

• We don’t need some components.
• We can customize ActionController
• Building a fast, lightweight REST service
with Rails 3
http://pivotallabs.com/users/jdean/blog/articles/1419-building-a-fast-
lightweight-rest-service-with-rails-3

# config/appliction.rb

%w(
active_record
action_controller
action_mailer
).each do |framework|
begin
require "#{framework}/railtie"
rescue LoadError
end
end

# config/application.rb
[
Rack::Sendfile,
ActionDispatch::Flash,
ActionDispatch::Session::CookieStore,
ActionDispatch::Cookies,
ActionDispatch::BestStandardsSupport,
Rack::MethodOverride,
ActionDispatch::ShowExceptions,
ActionDispatch::Static,
ActionDispatch::RemoteIp,
ActionDispatch::ParamsParser,
Rack::Lock,
ActionDispatch::Head
].each do |klass|
config.middleware.delete klass
end

# config/environments/production.rb
config.middleware.delete
ActiveRecord::ConnectionAdapters::ConnectionManagement

# /app/controllers/application_controller.rb
class ApplicationController < ActionController::Base
class ApplicationController < ActionController::Metal
include AbstractController::Logger
include Rails.application.routes.url_helpers
include ActionController::UrlFor
include ActionController::Rendering
include ActionController::Renderers::All
include ActionController::MimeResponds

if Rails.env.test?
include ActionController::Testing
# Rails 2.x compatibility
include ActionController::Compatibility
end

end

http://ihower.tw/blog/archives/4561

APIs design best practices (1)
• Routing doesn't need Rails resources
mechanism , but APIs design should follow
RESTful.
(This is because we don't have view in service and we don't need URL
helpers. So use resources mechanism is too overkill)

• RESTful APIs is stateless, each APIs should
be independent. So, requests which have
dependency relationship should be
combined into one API request. (atomic)

• The best format in most case is JSON.
( one disadvantage is we can’t return binary data directly. )

• Use Yajl as parser.
ActiveSupport::JSON.backend = "Yajl"

• Don't convert data to JSON in Model, the
converting process to JSON should be
place in Controller.

• I suggest it shouldn't include_root_in_json
ActiveRecord::Base.include_root_in_json = false

• Please notice “the key is JSON must be string”.
whether you use symbol or string in Ruby, after JSON
encode should all be string.
• related key format should be xxx_id or xxx_ids for
example:
{ "user_id" => 4, "product_ids" => [1,2,5] }.to_json

• return user_uri ﬁeld in addition to the user_id ﬁeld if
need

a return data example
model.to_json and model.to_xml is easy to use, but not useful in practice.

# one record
{ :name => "a" }.to_json

# collection
{ :collection => [ { :name => "a" } , { :name =>
"b" } ], :total => 123 }.to_json

If you want to have pagination, you
need total number.

• except return collection, we can also
provide Multi-Gets API. through params :
ids. ex. /users?ids=2,5,11,23
• client should sort ID ﬁrst, so we can design
cache mechanism much easier.
• another topic need to concern is the URL
length of GET. So this API can also use
POST.

an error message
return example
{ :message => "faild", :error_codes => [1,2,3],
:errors => ["k1" => "v1", "k2" => "v2" ] }.to_json


• error_codes & errors is optional, you can
deﬁne it if you need.
• errors is used to put model's validation
error : model.errors.to_json

HTTP status code
We should return suitable HTTP status code

• 200 OK
• 201 Created ( add success)
• 202 Accepted ( receive success but not
process yet, in queue now )
• 400 Bad Request ( ex. Model Validation
Error or wrong parameters )
• 401 Unauthorized

class PeopleController < ApplicationController

def index
@people = Person.paginate(:per_page => params[:per_page] || 20, :page => params[:page])
render :json => { :collection => @people, :total => @people.total_entries }.to_json
end

def show
@person = Person.find( params[:id] )
render :json => @person.to_json
end

def create
@person = Person.new( :name => params[:name], :bio => params[:bio],
:user_id => params[:user_id] )

@person.save!
render :json => { :id => @person.id }.to_json, :status => 201
end

def update
@person = user_Person.find( params[:id] )
@person.attributes = { :name => params[:name], :bio => params[:bio],
:user_id => params[:user_id] }

@person.save!
render :status => 200, :text => "OK"
end

def destroy
@person = Person.find( params[:id] )
@person.destroy

render :status => 200, :text => "OK"
end

end

Note
• No active_record, we get data from service
through HTTP client (typhoeus)
• Model can include some ActiveModel,
modules so we can develop more efﬁciently.
• This model is logical model, mapping to the
data from API, not database table. It's
different to service's physical model ( ORM-
based)

# config/appliction.rb

%w(
action_controller
action_mailer
).each do |framework|
begin
require "#{framework}/railtie"
rescue LoadError
end
end

Setup a global Hydry
# config/initializers/setup_hydra.rb
HYDRA = Typhoeus::Hydra.new

An example you can
inherited from (1)
class LogicalModel

include ActiveModel::Serializers::JSON
include ActiveModel::Validations
include ActiveModel::MassAssignmentSecurity

self.include_root_in_json = false

# continued...
end

class LogicalModel An example you can
# continued... inherited from (2)
def self.attribute_keys=(keys)
@attribute_keys = keys
attr_accessor *keys
end

def self.attribute_keys
@attribute_keys
end

class << self
attr_accessor :host, :hydra
end

def persisted?
!!self.id
end

def initialize(attributes={})
self.attributes = attributes
end

def attributes
self.class.attribute_keys.inject(ActiveSupport::HashWithIndifferentAccess.new) do |result, key|
result[key] = read_attribute_for_validation(key)
result
end
end

def attributes=(attrs)
sanitize_for_mass_assignment(attrs).each { |k, v| send("#{k}=", v) if respond_to?("#{k}=") }
end

Model usage example

class Person < LogicalModel

self.attribute_keys = [:id, :name, :bio, :user_id, :created_at, :updated_at]
self.host = PEOPLE_SERVICE_HOST
self.hydra = HYDRA

validates_presence_of :title, :url, :user_id

# ...
end

# ...
paginate
def self.people_uri
"http://#{self.host}/apis/v1/people.json"
end

def self.async_paginate(options={})
options[:page] ||= 1
options[:per_page] ||= 20
request = Typhoeus::Request.new(people_uri, :params => options)
if response.code >= 200 && response.code < 400
log_ok(response)

result_set = self.from_json(response.body)
collection = result_set[:collection].paginate( :total_entries => result_set[:total] )
collection.current_page = options[:page]
yield collection
else
log_failed(response)
end
end

end

def self.paginate(options={})
result = nil
async_paginate(options) { |i| result = i }
self.hydra.run
result
end

end

will_paginate hack!
• in order to use will_paginate's helper, we
must set current_page manually, so we hack
this way:

# /config/initializers/hack_will_paginate.rb
# This is because our search result via HTTP API is an array and need be paginated.
# So we need assign current_page, unless it will be always 1.

module WillPaginate
class Collection
def current_page=(s)
@current_page = s.to_i
end
end
end

from_json & logging
class LogicalModel
# ...

def self.from_json(json_string)
parsed = ActiveSupport::JSON.decode(json_string)
collection = parsed["collection"].map { |i| self.new(i) }
return { :collection => collection, :total => parsed["total"].to_i }
end

def self.log_ok(response)
Rails.logger.info("#{response.code} #{response.request.url} in #{response.time}s")
end

def self.log_failed(response)
msg = "#{response.code} #{response.request.url} in #{response.time}s FAILED: #{ActiveSupport::JSON.decode
(response.body)["message"]}"
Rails.logger.warn(msg)
end

def log_ok(response)
self.class.log_ok(response)
end

def log_failed(response)
self.class.log_failed(response)
end

end

# ...
ﬁnd
def self.person_uri(id)
"http://#{self.host}/apis/v1/people/#{id}.json"
end

def self.async_find(id)

request = Typhoeus::Request.new( person_uri(id) )
if response.code >= 200 && response.code < 400
log_ok(response)
yield self.new.from_json(response.body)
else
end
end
This from_json is deﬁned by
ActiveModel::Serializers::JSON
end

def self.find(id)
result = nil
async_find(id) { |i| result = i }
self.hydra.run
result
end

end

create&update
# ...

def create
return false unless valid?

response = Typhoeus::Request.post( self.class.people_uri, :params => self.attributes )
if response.code == 201
log_ok(response)
self.id = ActiveSupport::JSON.decode(response.body)["id"]
return self
else
return nil
end
end

def update(attributes)
self.attributes = attributes

return false unless valid?
response = Typhoeus::Request.put( self.class.person_uri(id), :params => self.attributes )
log_ok(response)
return self
else
log_failed(response) Normally data writes do not
return nil need to occur in parallel
end
end
end Or write to a messaging
system asynchronously

delete&destroy
# ...

def self.delete(id)
response = Typhoeus::Request.delete( self.person_uri(id) )
log_ok(response)
return self
else
return nil
end
end

def destroy
self.class.delete(self.id)
end

end

Service client Library
packaging
• Write users.gemspec ﬁle
• gem build users.gemspec
• distribution
• http://rubygems.org
• build your local gem server
• http://ihower.tw/blog/archives/4496

About caching

• Internally
• Memcached
• Externally: HTTP Caching
• Rack-Cache,Varnish, Squid

Service-Oriented Design and Implement with Rails3

References
• Books&Articles:
• Service-Oriented Design with Ruby and Rails, Paul Dix (Addison Wesley)
• Enterprise Rails, Dan Chak (O’Reilly)
• RESTful Web Services, Richardson&Ruby (O’Reilly)
• RESTful WEb Services Cookbook, Allamaraju&Amundsen (O’Reilly)
• Blog:
• Rails3: ActiveModel http://ihower.tw/blog/archives/4940
• Rubygems http://ihower.tw/blog/archives/4496
• Rails3: Railtie Plugins http://ihower.tw/blog/archives/4873
• Slides:
• Distributed Ruby and Rails
http://ihower.tw/blog/archives/3589

Service-Oriented Design and Implement with Rails3

More Related Content

What's hot (20)

Similar to Service-Oriented Design and Implement with Rails3 (20)

More from Wen-Tien Chang (20)

Recently uploaded (20)

Service-Oriented Design and Implement with Rails3