Optimize drupal using mongo db

optimize Drupal using
mongoDB

agenda
let’s get started

what is problem that we face (or what we need/want)

sql - relational databases

shift of technologies - noSQL databases

mongoDB - what is mongo

how to install mongo

finally - drupal & mongo

mongoDB module - what we can do with it

benefits

q&a

about me

name: Vladimir Ilic
employed at: OWFG
email: v_ilic@hotmail.com
t witter: burgerboydaddy

Who should be here instead of me

what is problem that we face
or what we want to achieve
with Drupal based solutions

speed

speed

scale

speed

scale
simplicity

how we can do that?
or how to speedup Drupal
server side
install Varnish for caching pages for
anonymous users.
install a persistent cache system
(Memcached, APC, Memcache).
use a CDN such as Akamai to ser ve
static files (JavaScript, CSS, images).

how to speedup drupal -
code side
use Pressflow, it allows Varnish to serve cached page for
anonymous users.
clean Drupal's watchdog table. Every time a watchdog
error gets logged, it consumes CPU resources on the web
ser ver and database server. It also increases load time
significantly.
implement static and persistent cache strategies until the
slow query log comes up clean.
avoid PHP errors that occur within nested foreach loops at
all costs.
uninstall unused modules.
turn on caching for Drupal core blocks and Views.

how to speedup drupal -
database side
Drupal sites live or die by their database.
make sure the tables are properly
indexed for faster searching.
do not store unnecessary records, a 100
node database will be always accessed
faster than a 3 million node database.
change how/where you store your data

speed -
shift of technologies
from Pentium 100MHz, 16MB RAM,
200 MB HDD to
my cell phone: dual core 1GHz, 1GB RAM,
32GB storage
or maybe some cloud ser ver with 32+
CPU’s, few TB RAM, ...

our database
technology
1974 - The
relational
database is created

sql - relational dbs

1979 1982-1996

sql - relational dbs

1979 1982-1996 1995

simplicity?

relational database model for eCommerce app

simplicity?

* joins
* joins
* joins

relational database model for eCommerce app

real life example
sell groceries
Product {
General Product
id: attributes
UPC:
brand:
description:
MSRP:
price:
in-stock:
...
PLU: 4011
unit of measure: lb
Item Speciﬁc
origin: BC attributes
seasonal: yes
...

real life example -
but we also sell books
Product {

id: General Product
UPC: attributes stay
brand:
description:
the same
MSRP:
price:
in-stock:
...
author: Isaac Asimov Book Speciﬁc
title: Foudation’s Edge attributes are
binding: Paperback
publication date: 1982 different
publisher name: Spectra
number of pages: 480
ISBN: 0553293389

real life example -
ops, we want more
Product {
General Product
id: attributes stay the
UPC: same
brand:
description:
MSRP:
price:
in-stock:
... Clothing speciﬁc
brand: Lee attributes are
gender: Mens totally different ...
make: Vintage
style: Straight Cut and not consistent
length: 34 across brands &
width: 42 make
color: Black
material: Cotton
...

shift of technologies -
db solutions
from well-established relational databases to NoSQL
technologies.
NoSQL was first introduced as concept in 1998, it really
wasn’t until 2009 that it emerged as a real trend.
NoSQL solutions aren't replacements traditional
solutions, but rather address a specific need in addition
to what one might get from traditional offerings.
In 90% of the time you’ll probably implement a hybrid
system.
put simply, NoSQL is about being open and aware of
alternative, existing and additional patterns and tools
for managing your data.

shift is happening now
few years ago (and still now) MySQL was
the undisputed king of the open-source
database hill.
It is still growing with great speed (40%
compound annual grow rate)
but
There are some competitors that are
emerging and grow even faster.


451 Research notes:

"NoSQL database technologies are largely being adopted for
new projects that require additional scalability,
performance, relaxed consistency and agility."


451 Research notes:

"NoSQL database technologies are largely being adopted for
new projects that require additional scalability, in other words:
performance, relaxed consistency and agility." web :-)

what is Not Only SQL?
•Non-Relational •Schema-Free
•Distributed •Replication Support
•Open-Source •Simple API
•Horizontally •Eventually
Scalable Consistent

mongoDB
agile and scalable

DB designed for today
MongoDB (from "humongous") is a
scalable, high-performance, open source
NoSQL database.

what is mongo
Application Document
Oriented
High
{ author: “vladimir”,
date: new Date(),
text: “drupal-MongoDB...”,
Performance tags: [“tech”, “database”]}

Fully
Consistent

Horizontally Scalable

other features of MongoDB
Document-based queries
Flexible document queries expressed in JSON/Javascript.
Map Reduce
Flexible aggregation and data processing.
Queries run in parallel on all shards.
GridFS
Store files of any size easily.
Geospatial Indexing
Find object based on location. (i.e. find closest n items to x)
Many Production Deployments

what is mongo
Document oriented storage
JSON-style documents with dynamic schemas offer simplicity and power.

Full index support (+geo)
Index on any attribute, just like you're used to.

Replication and High Availability
Mirror across LANs and WANs for scale and peace of mind.

Querying
Rich, document-based queries

Fast In-Place Updates
Map/Reduce

mongoDB philosophy
Keep functionality when we can (key/
value stores are great, but we need
more)
Non-relational (no joins) makes scaling
horizontally practical
Document data models are good
Database technology should run
anywhere VMs, cloud, etc

6 mongoDB concepts
MongoDB has the same concept of a 'd atabase' with
which you are likely already familiar (or a schema
for you Oracle folks). Within a MongoDB instance
you can have zero or more databases, each acting
as high-level containers for everything else.
A database can have zero or more 'c ollections'. A
collection shares enough in common with a
traditional 'table' that you can safely think of the
t wo as the same thing.
Collections are made up of zero or more 'documents'.
Again, a document can safely be thought of as a
'row'.

6 mongoDB concepts
A document is made up of one or more 'fields', which you
can probably guess are a lot like 'c olumns'.
'Indexes' in MongoDB function much like their RDBMS
counterparts.
'Cursors' are different than the other five concepts but
they are important enough, and often overlooked, that
I think they are worthy of their own discussion. The
important thing to understand about cursors is that
when you ask MongoDB for data, it returns a cursor,
which we can do things to, such as counting or skipping
ahead, without actually pulling down data.

our eCommerce problem again
{ "_id" : ObjectId("4fc19d5e0ddc54e49b84928c"),
‘customer_id’: ObjectId("4fc19d5e5eec78a34f24653d"),
‘state’: ‘cart’,

‘line_items’: [
{‘UPC’:885909377275’,
‘name’: ‘Tide HE’,
‘quantity’: 2,
‘tax’: ‘HST’,
‘retail_price’: 10.99
},

{‘UPC’: ‘2348751987’,
‘name’: ‘bananas’,
‘weight’: 3.5,
‘UOM’: kg,
},
],

‘shipping_address’: {
‘street’: ‘1245 76 Ave.’,
‘city’: ‘Surrey’,
‘province’: ‘BC’,
‘postal code’: ‘V1Q 1K8‘
}

‘subtotal’: 13.74
}

Document ‘state’: ‘cart’,

- Analogous to a row in RDBMS ‘line_items’: [
- Represented as JSON (BSON) {‘UPC’:885909377275’,
‘quantity’: 2,
},

{‘UPC’: ‘2348751987’,
‘weight’: 3.5,
‘UOM’: kg,
},
],

‘street’: ‘1245 76 Ave.’,
}

}


‘quantity’: 2,
},

Embedding {‘UPC’: ‘2348751987’,
- Analogous to a foreign key ‘name’: ‘bananas’,
‘weight’: 3.5,
- Can be ‘UOM’: kg,
- sub objects },

- collections ],

‘street’: ‘1245 76 Ave.’,
}

}


‘quantity’: 2,
},

Embedding {‘UPC’: ‘2348751987’,
- Analogous to a foreign key ‘name’: ‘bananas’,
‘weight’: 3.5,
- Can be ‘UOM’: kg,
- sub objects },

- collections ],

‘street’: ‘1245 76 Ave.’,
References ‘city’: ‘Surrey’,
- Analogous to a foreign key }
- Think “relationship”
}

and some basic operations
Querying
All the following queries will find the document.
‘state’: ‘cart’,
By property value
‘line_items’: [
db.orders.find({‘state’: ‘cart’}) {‘UPC’:885909377275’,
‘quantity’: 2,
},

By embedded object property value
{‘UPC’: ‘2348751987’,
db.orders.find({‘shipping_address.province’: ‘BC’}) ‘weight’: 3.5,
‘UOM’: kg,
},
],
With comparison operators
‘street’: ‘12345 76 Ave.’,
db.orders.find({‘subtotal’: {$gt: 10}) ‘city’: ‘Surrey’,
}

Values in collections (implicit “in”) ‘subtotal’: 13.74
}

db.orders.find({‘line_items.UPC’: ‘885909377275’})

how to install mongo
you can have mongo on virtually any platform
(Windows, OS X, Linux, Solaris, FreeBSD)
mongo on Ubuntu in less than 5min
add the 10gen GPG key
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

Open document /etc/apt/sources.list (sudo vim /etc/
apt/sources.list). Add at the bottom
deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist
10gen

Update source list and install package
sudo apt-get update
sudo apt-get install mongodb-10gen

basic mongo commands
db.help() -- also db.help (no parentheses)
show dbs -- show databases
use -- for selecting database
db.getCollectionNames()
Since collections are schema-less, we don't
explicitly need to create them. We can
simply insert a document into a new
collection. To do so, use the insert command,
supplying it with the document to insert.

use learn <--- switched to db learn

> db.getCollectionNames()

[ ]

> db.starwars.insert({name: 'C-3PO', gender:
'robot', position: 'Protocol droid', homeworld:
'Tatooine'})

> db.getCollectionNames()

[ "starwars", "system.indexes" ]

What you're seeing is the name of the index, the
database and collection it was created against
and the fields included in the index


> db.starwars.find()

"name" : "C-3PO", "gender" : "robot",
"position" : "Protocol droid", "homeworld" :
"Tatooine" }

> db.system.indexes.find()

{ "v" : 1, "key" : { "_id" : 1 }, "ns" :
"learn.starwars", "name" : "_id_" }

Drupal & mongo
In Drupal 7 if you use field API you want your fields inside
mongodb. If you store them inside sql you cannot query them
efficiently.
Storing data into SQL will crete cases where you will run into
denormalization issues, and with mongo that will be solved.
In Drupal 7 everything is entity; nodes are entity, users are
entity, comments are entity.
We only need few system tables in MySQL that are hard-
wired with core drupal; but those things are cached
(memcache)
everything else can go to mongo. and they (mongo &
memcache) are easy to scale.

mongoDB module
http://drupal.org/project/mongodb

mongoDB module

mongodb: support library for the other modules

mongoDB module

mongodb_cache: Store cache items in mongodb.

mongoDB module

mongodb_field_storage: Store the fields in mongodb.

mongoDB module

mongodb_session: Store sessions in mongodb.

mongoDB module

mongodb_watchdog: Store watchdog messages in
mongodb

mongoDB module

mongodb_watchdog: Store watchdog messages in
mongodb
mongodb_block: Store block information in
mongodb.
Very close to the core block API.

EntityFieldQuery Views
Backend
http://drupal.org/
project/efq_views
This module enables Views to use
EntityFieldQuery as the query
backend, allowing you to query all
defined entity types and their fields,
even the ones stored in non-sql
storage such as mongodb.

Load into nodes into mongo
<?php
// Connect
$mongo = new Mongo();

// Get the database (it is created automatically)
$db = $mongo->testDatabase;

// Get the collection for nodes (it is created automatically)
$collection = $db->nodes;

// Get a listing of all of the node IDs
$r = db_query('SELECT nid FROM {node}');

// Loop through all of the nodes...

foreach($r as $row) {
print "Writing node $row->nidn";

// Load each node and convert it to an array.
$node = (array)node_load($row->nid);

// Store the node in MongoDB
$collection->save($node);
}
?>

code sample
# drush php-script mongoimport.php

# use testDatabase;
# db.nodes.find( {title: /Distineo/i} , {title: true}).limit(4);

code sample
# drush php-script mongoimport.php

# use testDatabase;
# db.nodes.find( {title: /Distineo/i} , {title: true}).limit(4);

// how to use mongo in php code
<?php
// Connect
$mongo = new Mongo();

// Write our search filter (same as shell example above)
$filter = array(
'title' => new MongoRegex('/Distineo/i'),
);

// Run the query, getting only 5 results.
$res = $mongo->testDatabase->nodes->find($filter)->limit(5);

// Loop through and print the title of each article.
foreach ($res as $row) {
print $row['title'] . PHP_EOL;
}
?>

Where mongo won’t
work

Joining across Entities
ex. return birthday from profile
belonging to author of current node

Thanks for you patience
enjoy rest of the day!
:-)

Optimize drupal using mongo db

Recommended

More Related Content

What's hot (20)

Similar to Optimize drupal using mongo db (20)

Recently uploaded (20)

Optimize drupal using mongo db

Editor's Notes