Data Processing and Aggregation with MongoDB

Data Processing and Aggregation
Senior Solutions Architect, MongoDB Inc
massimo@mongodb.com.
Massimo Brignoli
@massimobrignoli

Chi sono?
•  Solutions Architect/Evangelist in MongoDB Inc.
•  24 anni di esperienza nel mondo dei database e dello
sviluppo software
•  Ex dipendente di MySQL e MariaDB
•  In precedenza: web,web,web

Understanding Big Data – It’s Not Very “Big”
from Big Data Executive Summary – 50+ top executives from Government and F500 firms
64% - Ingest diverse,
new data in real-time
15% - More than 100TB
of data
20% - Less than 100TB
(average of all? <20TB)

“I have not failed. I've just found 10,000 ways that won't work.”
― Thomas A. Edison

Tante grandi innovazioni dal 1970…

Ma usereste una di queste
tecnologie per lanciare un
nuovo business oggi?

Incluso il modello relazionale dei dati!

Per quali computer è stato
pensato il modello
relazionale?

E come si sviluppava il software?
pio, il LISP (LISt Processing language) [24].
A quel tempo, i problemi significativi non ri-
denti con interfacce chiare e componibili. Si
diffusero concetti quali la programmazione
1
ei
gi
Processo Bisogno Linguaggio
1950
1960
1970
1980
1990
2000
Primi tentativi di “ordine”
nello sviluppo
Comprensibilità e portabilità del codice,
per sostenere la sua evoluzione
Organizzazione “industriale”
dello sviluppo dei sistemi software
Impossibilità di definire in modo
preciso il sistema da sviluppare
Sviluppo e distribuzione molto
rapidi e orientati ai sistemi
di comunicazione
Waterfall, a “V”, ...
Incrementale, Spirale, ...
Metodologie agili
Linguaggi assemblativi
Linguaggi di alto livello
Linguaggi strutturati
Linguaggi orientati agli oggetti
Linguaggi per lo sviluppo
dinamico

RDBMS Rende lo Sviluppo Difﬁcile
Relational
Database
Object Relational
Mapping
Application
Code XML Conﬁg DB Schema

E Ancora Più Difﬁcile Evolverlo…
New
Table
New
Table
New
Column
Name Pet Phone Email
New
Column
3 months later…

RDBMS
Dalla Complessità alla Semplicità..
MongoDB
{
_id : ObjectId("4c4ba5e5e8aabf3"),
employee_name: "Dunham, Justin",
department : "Marketing",
title : "Product Manager, Web",
report_up: "Neray, Graham",
pay_band: “C",
benefits : [
{ type : "Health",
plan : "PPO Plus" },
{ type : "Dental",
plan : "Standard" }
]
}

Chiave → Valore
•  Storage mono-dimensionale
•  Il singolo valore e’un blob
•  Le query sono solo per chiave
•  Nessuno schema
•  I valore non può essere aggiornato ma solamente
sovrascritto
Key Blob

Relazionale
•  Storage bi-dimensionale (tuple)
•  Ogni campo contiene solo un valore
•  Query sono su ogni campo
•  Schema molto strutturato (tabelle)
•  Update sul posto
•  Il processo di normalizzazione richiede molte tabelle,
indici e con una pessima localizzazione dei dati.
Primary
Key

Documento
•  Storage N-dimensionale
•  Ogni campo può contenere 0,1,
tanti o valori incapsulati
•  Query su tutti i campi e livelli
•  Schema dinamico
•  Update in linea
•  Incapsulare i dati migliora la localizzazione dei dati,
richiede meno indici e ha migliori performance
_id

For over a decade
Big Data == Custom Software

In the past few years
Open source software has
emerged enabling the rest of
us to handle Big Data

How MongoDB Meets Our Requirements
•  MongoDB is an operational database
•  MongoDB provides high performance for storage and
retrieval at large scale
•  MongoDB has a robust query interface permitting
intelligent operations
•  MongoDB is not a data processing engine,but provides
processing functionality

http://www.ﬂickr.com/photos/torek/4444673930/
MongoDB data processing options

The“hello world”of
MapReduce is counting words
in a paragraph of text.
Let’s try something a little more
interesting…

What is the most popular pub name?

#!/usr/bin/env python

# Data Source
# http://www.overpass-api.de/api/xapi?*[amenity=pub][bbox=-10.5,49.78,1.78,59]

import re
import sys

from imposm.parser import OSMParser
import pymongo

class Handler(object):
def nodes(self, nodes):
if not nodes:
return
docs = []
for node in nodes:
osm_id, doc, (lon, lat) = node
if "name" not in doc:
node_points[osm_id] = (lon, lat)
continue
doc["name"] = doc["name"].title().lstrip("The ").replace("And", "&")
doc["_id"] = osm_id
doc["location"] = {"type": "Point", "coordinates": [lon, lat]}
docs.append(doc)
collection.insert(docs)

Open Street Map Data

{
"_id" : 451152,
"amenity" : "pub",
"name" : "The Dignity",
"addr:housenumber" : "363",
"addr:street" : "Regents Park Road",
"addr:city" : "London",
"addr:postcode" : "N3 1DH",
"toilets" : "yes",
"toilets:access" : "customers",
"location" : {
"type" : "Point",
"coordinates" : [-0.1945732, 51.6008172]
}
}

Example Pub Data

MongoDB MapReduce• 
MongoDB
map
reduce
ﬁnalize

Map Function
> var map = function() {
emit(this.name, 1);
MongoDB
map
reduce
ﬁnalize

Reduce Function
> var reduce = function (key, values) {
var sum = 0;
values.forEach( function (val) {sum +=
val;} );
return sum;
}
MongoDB
map
reduce
ﬁnalize

Results
> db.pubs.mapReduce(map, reduce, { out: "pub_names",
query: { } } )
> db.pub_names.find().sort({value: -1}).limit(10)

{ "_id" : "The Red Lion", "value" : 407 }
{ "_id" : "The Royal Oak", "value" : 328 }
{ "_id" : "The Crown", "value" : 242 }
{ "_id" : "The White Hart", "value" : 214 }
{ "_id" : "The White Horse", "value" : 200 }
{ "_id" : "The New Inn", "value" : 187 }
{ "_id" : "The Plough", "value" : 185 }
{ "_id" : "The Rose & Crown", "value" : 164 }
{ "_id" : "The Wheatsheaf", "value" : 147 }
{ "_id" : "The Swan", "value" : 140 }

Data Processing and Aggregation with MongoDB

> db.pubs.mapReduce(map, reduce, { out: "pub_names",
query: {
location: {
$within: { $centerSphere: [[-0.12, 51.516], 2 / 3959] }
}}
})

{
"result" : "pub_names",
"timeMillis" : 116,
"counts" : {
"input" : 643,
"emit" : 643,
"reduce" : 54,
"output" : 537
},
"ok" : 1,
}

Pub Names in the Center of London


{ "_id" : "All Bar One", "value" : 11 }
{ "_id" : "The Slug & Lettuce", "value" : 7 }
{ "_id" : "The Coach & Horses", "value" : 6 }
{ "_id" : "The Green Man", "value" : 5 }
{ "_id" : "The Kings Arms", "value" : 5 }
{ "_id" : "Corney & Barrow", "value" : 4 }
{ "_id" : "O'Neills", "value" : 4 }
{ "_id" : "Pitcher & Piano", "value" : 4 }

Results

MongoDB MapReduce
•  Real-time
•  Output directly to document or collection
•  Runs inside MongoDB on local data
− Adds load to your DB
− In Javascript–debugging can be a challenge
− Translating in and out of C++

Aggregation Framework• 
MongoDB
op1
op2
opN

Aggregation Framework in 60 Seconds

Aggregation Framework Operators
•  $project
•  $match
•  $limit
•  $skip
•  $sort
•  $unwind
•  $group

$match
•  Filter documents
•  Uses existing query syntax
•  If using $geoNear it has to be ﬁrst in pipeline
•  $where is not supported

Matching Field Values
{
"_id" : 271421,
"amenity" : "pub",
"name" : "Sir Walter Tyrrell",
"location" : {
"type" : "Point",
"coordinates" : [
-1.6192422,
50.9131996
]
}
}

{
"_id" : 271466,
"amenity" : "pub",
"name" : "The Red Lion",
"location" : {
"type" : "Point",
"coordinates" : [
-1.5494749,
50.7837119
]
}
Matching Field Values
{ "$match": {
"name": "The Red Lion"
}}
{
"_id" : 271466,
"amenity" : "pub",
"name" : "The Red Lion",
"location" : {
"type" : "Point",
"coordinates" : [
-1.5494749,
50.7837119
]}
}

$project
•  Reshape documents
•  Include,exclude or rename fields
•  Inject computed fields
•  Create sub-document fields

Including and Excluding Fields
{
"_id" : 271466,
"amenity" : "pub",
"name" : "The Red
Lion",
"location" : {
"type" :
"Point",
"coordinates" :
[
-1.5494749,
50.7837119
]
}
}
{ “$project”: {

“_id”: 0,
“amenity”: 1,
“name”: 1,

}}
{
“amenity” : “pub”,
“name” : “The Red Lion”
}

Reformatting Documents
{
"_id" : 271466,
"amenity" : "pub",
"name" : "The Red
Lion",
"location" : {
"type" :
"Point",
"coordinates" :
[
-1.5494749,
50.7837119
]
}
}
{ “$project”: {

“_id”: 0,
“name”: 1,
“meta”: {

“type”: “$amenity”}
}}

{
“name” : “The Red Lion”
“meta” : {
“type” : “pub”
}}

$group
•  Group documents by an ID
•  Field reference,object,constant
•  Other output ﬁelds are computed
$max,$min,$avg,$sum
$addToSet,$push $ﬁrst,$last
•  Processes all data in memory

Back to the pub!
•  http://www.offwestend.com/index.php/theatres/pastshows/71

Popular Pub Names
>var popular_pub_names = [
{ $match : location:

{ $within: { $centerSphere:

[[-0.12, 51.516], 2 / 3959]}}}
},
{ $group :

{ _id: “$name”

value: {$sum: 1} }
},
{ $sort : {value: -1} },
{ $limit : 10 }

> db.pubs.aggregate(popular_pub_names)
{
"result" : [
],
"ok" : 1
}

Results

Aggregation Framework Beneﬁts
•  Real-time
•  Simple yet powerful interface
•  Declared in JSON,executes in C++
•  Runs inside MongoDB on local data
− Adds load to your DB
− Limited Operators
− Data output is limited

Analyzing MongoDB Data in
External Systems

MongoDB with Hadoop• 
MongoDB

MongoDB with Hadoop• 
MongoDB warehouse

MongoDB with Hadoop
• 
MongoDBETL

from pymongo_hadoop import BSONMapper

def mapper(documents):
bounds = get_bounds() # ~2 mile polygon
for doc in documents:
geo = get_geo(doc["location"]) # Convert the geo
type
if not geo:
continue
if bounds.intersects(geo):
yield {'_id': doc['name'], 'count': 1}

BSONMapper(mapper)
print >> sys.stderr, "Done Mapping."

Map Pub Names in Python


from pymongo_hadoop import BSONReducer

def reducer(key, values):
_count = 0
for v in values:
_count += v['count']
return {'_id': key, 'value': _count}

BSONReducer(reducer)

Reduce Pub Names in Python

hadoop jar target/mongo-hadoop-streaming-
assembly-1.1.0-rc0.jar
-mapper examples/pub/map.py
-reducer examples/pub/reduce.py
-mongo mongodb://127.0.0.1/demo.pubs
-outputURI mongodb://127.0.0.1/demo.pub_names

Execute MapReduce


{ "_id" : "The George", "value" : 4 }

Popular Pub Names Nearby

MongoDB and Hadoop
•  Away from data store
•  Can leverage existing data processing infrastructure
•  Can horizontally scale your data processing
-  Ofﬂine batch processing
-  Requires synchronisation between store & processor
-  Infrastructure is much more complex

The Future of Big Data and
MongoDB

What is Big Data?
Big Data today will be normal
tomorrow

Exponential Data Growth
0
1000
2000
3000
4000
5000
6000
7000
8000
9000
10000
2000 2002 2004 2006 2008 2010 2012
Billions of URLs indexed by Google

MongoDB enables you to
scale big

MongoDB is evolving
so you can process the big

Data Processing with MongoDB
•  Process in MongoDB using Map/Reduce
•  Process in MongoDB using Aggregation Framework
•  Process outside MongoDB using Hadoop and other
external tools

MongoDB Integration
•  Hadoop
https://github.com/mongodb/mongo-hadoop
•  Storm
https://github.com/christkv/mongo-storm
•  Disco
https://github.com/mongodb/mongo-disco
•  Spark
Coming soon!

Thanks!
massimo@mongodb.com
Massimo Brignoli
@massimobrignoli

Data Processing and Aggregation with MongoDB

Recommended

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Data Processing and Aggregation with MongoDB (20)

More from MongoDB (20)

Recently uploaded (20)

Data Processing and Aggregation with MongoDB