Logging rails application behavior to MongoDB

Logging rails application behavior to MongoDB

Vasiliev Alexey Yuriyovich
leopard.in.ua

2011

Vasiliev A.Y. Logging rails application behavior to MongoDB

MongoDB

Introduction
MongoDB is an open source document-oriented database.

Features
Consistent UTF-8 encoding;
Cross-platform support;
Eﬃcient storage for large binary data, for example photos and
video;
Replication and failover;
MapReduce.


Why application should log to DB?

Centralized application
logging;
Easy remote access,
compared to ﬁles;
Can be fast and scalable if
you go with database
partitioning and/or multi-log
databases.


What are the advantages appear logging in MongoDB?

Flexible schema;
Asynchronous inserts;
Capped collection;
Fast enough compared to
ﬁles;
Indexed and searched;
Analyze data in-place.


Gem for Rails
MongoDB as Rails logger

Central Logger (https://github.com/
customink/central_logger). You should do
only several steps:
gem ’central_logger’;
include CentralLogger::Filter;
Conﬁg the connection to mongodb;
That’s all!


Rails log structure

{
’ action ’ : action_name ,
’ application_name ’ : application_name ( r a i l s root ) ,
’ controller ’ : controller_name ,
’ ip ’ : ip_address ,
’ messages ’ : {
’ info ’ : [ ],
’ debug ’ : [ ] ,
’ error ’ : [ ] ,
’ warn ’ : [ ],
’ fatal ’ : [ ]
},
’ params ’ : { },
’ path ’ : path ,
’ request_time ’ : date_of_request ,
’ runtime ’ : execution_time_in_milliseconds ,
’ url ’ : full_url
}


Add data to log

Add additional data to log
# make s u r e we ’ r e u s i n g t h e C e n t r a l L o g g e r i n t h i s e n v i r o n m e n t
i f R a i l s . l o g g e r . r e s p o n d _ t o ? ( : add_metadata )
R a i l s . l o g g e r . add_metadata ( : u s e r _ g u i d => @ u s e r . i d )
end


Querying via the Rails console

Handle on the MongoDB
>> db = R a i l s . l o g g e r . mongo_connection
=> #&l t ; Mongo : : DB: 0 x 1 0 2 f 1 9 a c 0 @ s l a v e _ o k= n i l ...

>> c o l l e c t i o n = db [ R a i l s . l o g g e r . m o n g o _ c o l l e c t i o n _ n a m e ]
=> #&l t ; Mongo : : C o l l e c t i o n : 0 x 1 0 3 1 b 3 e e 8 . . .

Find all requests for a speciﬁc user (with guid)
>> c u r s o r = c o l l e c t i o n . f i n d ( : u s e r _ g u i d => 1 2 3 5 5 )
=> #&l t ; Mongo : : C u r s o r : 0 x 1 0 3 1 a 3 e 3 0 . . . &g t ;
>> c u r s o r . count
=> 255



Find all requests that took more that two second to complete
>> c o l l e c t i o n . f i n d ( { : r u n t i m e => { ’ $gt ’ => 2 0 0 0 } } ) . c o u n t
=> 3

Find all requests with an exception
>> c o l l e c t i o n . f i n d ( { " m e s s a g e s . e r r o r " => / R o u t i n g E r r o r / } )



Find by params values
>> c o l l e c t i o n . f i n d (
" params . a c t i v i t y " => { ’ $ e x i s t s ’ => t r u e } ,
" params . a c t i v i t y . s t o r i e s " => { ’ $ e x i s t s ’ => t r u e } ,
" r e q u e s t _ t i m e " => {
’ $ g t e ’ => s t a r t _ d a t e . u t c ,
’ $ l t e ’ => end_date . u t c
}
) . s o r t ( " params . a c t i v i t y . o c c u r r e d _ a t " , 1 ) . c o u n t

=> 542



Find current status of story using logs
>> c o l l e c t i o n . f i n d (
" params . a c t i v i t y . s t o r i e s . i d " => s t o r y _ i d . to_i ,
" params . a c t i v i t y . s t o r i e s . c u r r e n t _ s t a t e " =>
{ ’ $ e x i s t s ’ => t r u e }
) . s o r t ( " params . a c t i v i t y . o c c u r r e d _ a t " , −1). f i r s t

=> {" _id"=>BSON : : O b j e c t I d ( ’ 4 e 7 c 6 2 e d 7 6 1 2 6 8 0 4 d 4 0 0 0 0 0 f ’ ) ,
" a c t i o n "=>"hook " ,
" c o n t r o l l e r "=>" s e r v i c e s " , " p a t h "=>"/ a p i / v1 / hook " ,
" params"=>{" a c t i v i t y "=>{ . . . ,
" s t o r i e s " = > [ { . . . , " c u r r e n t _ s t a t e "=>" f i n i s h e d " } ] } }



Tailable Cursors
v a r c o l l = db . some . c a p p e d . c o l l e c t i o n ;
var l a s t V a l = c o l l . f i n d ( ) . s o r t ({ ’ $ n a t u r a l ’ : 1 })
. l i m i t ( 1 ) . next ( ) [ ’ increasing ’ ];
while (1){

c u r s o r = c o l l . f i n d ({ ’ i n c r e a s i n g ’ : {
’ $gte ’ : l a s t V a l
} });

// t a i l a b l e
c u r s o r . addOption ( 2 ) ;
// a w a i t d a t a
c u r s o r . a d d O p t i o n ( 32 ) ;

// W a i t s s e v e r a l s e c f o r more d a t a
w h i l e ( c u r s o r . hasNext ( ) ){
v a r doc = c u r s o r . n e x t ( ) ;
l a s t V a l = doc [ ’ i n c r e a s i n g ’ ] ;
p r i n t j s o n ( doc ) ;
}

} Vasiliev A.Y. Logging rails application behavior to MongoDB

Using MapReduce

Find average time of page loading
>> c o l l e c t i o n . g r o u p ( " f u n c t i o n ( x ) {
r e t u r n { month : x . r e q u e s t _ t i m e . getMonth ( ) ,
year : x . request_time . g e t F u l l Y e a r ( ) } ; }" ,
{ : r e q u e s t _ t i m e => {
’ $ g t e ’ => Time . u t c ( 2 0 1 1 , 0 1 , 0 1 , 0 0 , 0 0 , 0 0 ) ,
" $ l t " => Time . u t c ( 2 0 1 1 , 8 , 0 1 , 0 0 , 0 0 , 0 0 ) }
} , { : c o u n t => 0 , : t o t a l _ t i m e => 0 } ,
" f u n c t i o n ( doc , o u t ) {
o u t . c o u n t ++; o u t . t o t a l _ t i m e+=doc . r u n t i m e } " ,
" f u n c t i o n ( out ){
o u t . avg_time = o u t . t o t a l _ t i m e / o u t . c o u n t } " )

=> [
{" month "= >4.0 ,
" y e a r "= >2011.0 ,
" c o u n t "= >5385.0 ,
" t o t a l _ t i m e "= >1027015.0 ,
" avg_time "= >190.717734447539} ,
...
]


Restrictions

The database doesn’t allow update the
existing objects in the collection (but
there is possibility).
The database doesn’t allow deleting
objects from a capped collection.
Capped collection are not shardable.
More indexes - slower inserts.


Q&A

Thank you for attention!


Logging rails application behavior to MongoDB

Recommended

More Related Content

What's hot (19)

Similar to Logging rails application behavior to MongoDB (20)

Recently uploaded (20)