Work Queues

Work Queues
with Gearman and CodeIgniter

CI Conf ’12 - San Francisco

Erik Giberti - JD Supra
Senior Developer
@giberti
erik@jdsupra.com

What’s a Work Queue Anyway?

“A sequence of stored data or programs awaiting
processing.”
~ American Heritage Dictionary


processing.”

“[F]or storing messages as they travel between
computers.”
~ Amazon SQS site


processing.”

“[F]or storing messages as they travel between
computers.”
~ Amazon SQS site

“[I]t’s the nervous system for how distributed
processing communicates”
~ Gearman site

You might be using them already...

You might be using them already...
<?php

// filename: do_some_work.php
$conn = mysqli_connect($server, $user, $pass, $database);

$select = “SELECT * FROM things WHERE status = ‘not done’ ORDER BY timestamp ASC”;
$res = $conn->query($select);

while($work = $res->fetch_assoc()){
!
//
! // do work
! //

$update = “UPDATE things SET status = ‘done’ WHERE id = {$things[‘id’]}”;
! $conn->query($update);
}

It works!
It’s simple to understand
Get work -> do work -> done.

It works!
Can be implemented in other languages
This job/worker could have been created in Python, Java... whatever!

It works!
Can be deployed on a different server
So long as it can talk to the DB

It works!
It’s persistent
MySQL is pretty good at keeping data

What’s wrong with that?
Runs at the frequency that cron ﬁres it off
Can only run once a minute

Single threaded
Race condition if you start two overlapping threads
Mitigation strategies
Have workers kill themselves every minute
Use modulus to only do certain jobs in certain threads
Create different DB pools for each worker

Single threaded
Race condition if you start two overlapping threads
Mitigation strategies
Have workers kill themselves every minute
Use modulus to only do certain jobs in certain threads
Create different DB pools for each worker

Hits the database at the predeﬁned frequency
Wasting DB resources
Hardest tech in most stacks to scale...

Worker Take 2

// filename: do_some_work_two.php
$conn = mysqli_connect($server, $user, $pass, $database);

$select = “SELECT * FROM things WHERE status = ‘not done’ ORDER BY timestamp ASC LIMIT 1”;
$done = false;

while(!$done){
$res = $conn->query($select);
if($res->num_rows == 1){
! $work = $res->fetch_assoc();

! //
! // do some stuff
! //

$update = “UPDATE things SET status = ‘done’ WHERE id = {$things[‘id’]}”;
! $conn->query($update);

} else {
! sleep(1);
}
}

A little better?
Get work -> do work -> repeat

A little better?

A little better?
It’s persistent

A little better?
It’s persistent
It’s near real time
Maximum delay of 1 second between jobs

Runs at the frequency of the sleep() delay
1 second waits are pretty good for offloaded tasks
Can tweak the timing delay to our level of tolerance
usleep() for ﬁner control of the interval and less than 1 second intervals are
possible!

possible!

Less likely to hit a race condition
Same mitigation strategies apply

possible!

Less likely to hit a race condition
Same mitigation strategies apply
Hits the database even more!
MySQL can cache queries pretty well with MyISAM tables...
... but your using InnoDB ... right?

- a quick history lesson

Originally developed by Danga Interactive to solve speciﬁc issues
in the building and hosting of LiveJournal.com. It was originally
announced in 2005 and was written in Perl.



It was ported to C by a number of big companies, like Google
and enjoys wide support from the community; not that much
really goes wrong with it.




Incidentally, Danga Interactive also created Memcached




Incidentally, Danga Interactive also created Memcached

Danga Interactive exists no more, it was sold to Six Apart then
LiveJournal was sold to SUP. Six Apart was acquired by SAY
Media.

How Gearman Work Queues Stack Up

They’re simple to understand

They can work across languages
Jobs are simple strings, so they can pass whatever you want
Additionally, tools exist for things like MySQL to trigger new jobs

They can work across servers
Workers don’t need to live where clients do

Persistent
This requires additional conﬁguration

Persistent
Real Time

Persistent
Real Time
Bonus: Can be asynchronous and synchronous!

But I don’t have this sort of workload...

But I don’t have this sort of workload...
O’RLY?

Have you seen this pattern?
class Upload extends CI_Controller {

! function index(){

! ! $this->load->helper(array(‘url’));
! ! $ul_config = array( /* ... */ );
! ! $this->load->library(‘upload’, $ul_config);
! ! if($this->upload->do_upload()){

! ! ! $file_info = $this->upload->data();
! ! ! $img_config = array( /* ... */);
! ! ! $this->load->library(‘image_lib’, $img_config);
! ! ! $this->image_lib->resize();

! ! ! // watermark too?
! ! ! // write to the database
! ! ! // move to web accessible directory

! ! ! redirect(site_url(‘controller/next’));

! ! } else { /* error handling code */ }
! }
}

It Works

Reasonably fast in many circumstances

It Works

Reasonably fast in many circumstances

Most modern servers can handle a few of these calls at the
same time.

But there are limitations...
Cameras take big images
The iPhone 4 is a 5 megapixel camera, most ﬁles are about 2Mb
Entry level DSLR cameras are 14 megapixel with 5Mb ﬁles

What about multiple uploads
Present the form, upload one image, process, repeat?
Provide a block of upload ﬁelds and hope it’s enough?
Use HTML5 / Flash plugin to send multiple ﬁles?
Hope you have enough CPU & RAM to crunch them in parallel.

What about multiple uploads
Present the form, upload one image, process, repeat?
Provide a block of upload fields and hope it’s enough?
Use HTML5 / Flash plugin to send multiple files?
Hope you have enough CPU & RAM to crunch them in parallel.

Processing may delay the client longer than the default
timeout.
What happens to your user’s confidence in your product if things are
slow?

Okay, I’m interested
how do I add it?

Gearmand runs as another daemon/service in your stack.
Can be a different server or run along side everything else

Gearman’s only job is to facilitate the handling of these
messages
It can optionally store them in a persistent store to recover from
system reboots, service restarts, etc.

messages
PHP talks to the Gearmand server via an API extension
Just like MySQL, Memcached, Postgresql, APC and other tools

messages
PHP talks to the Gearmand server via an API extension
Just like MySQL, Memcached, Postgresql, APC and other tools
Your client and worker code pass messages back and forth
to Gearman
You’ll use a predeﬁned set of PHP function calls to do this

Getting Gearmand running - installers for many systems

Linux: apt-get, yum etc
sudo apt-get install gearmand
sudo yum install gearmand

Windows: cygwin ...

Windows: cygwin ...
OS X: ...
Install a package manager: MacPorts, Homebrew
sudo port install gearmand
brew install gearmand

Add a Virtual Machine that has an installer...

Getting Gearmand running - compile your own

Get your dependencies
sudo yum install gpp gcc-c++ boost boost-devel libevent libevent-devel libuuid libuuid-
devel

devel
Get the latest stable source from gearman.org (Launchpad)
wget https://launchpad.net/gearmand/trunk/0.34/+download/gearmand-0.34.tar.gz

devel
Unpack, Compile and Install
tar -xzf gearmand-0.34.tar.gz
cd gearmand-0.34.tar.gz
./conﬁgure
make
sudo make install

devel
Unpack, Compile and Install
tar -xzf gearmand-0.34.tar.gz
cd gearmand-0.34.tar.gz
./conﬁgure
make
sudo make install
Add appropriate system hooks to start the service on reboot
chkconﬁg
systemctl
?

Conﬁguring Gearmand
Default install is pretty good
Load gearmand as a background service
No persistence
Listens on all available interfaces at port 4730

Default install is pretty good
Load gearmand as a background service
No persistence
Listens on all available interfaces at port 4730
Persistence
Easily enabled using
MySQL based databases (Drizzle, MySQL, MariaDB etc)
Postgresql
SQL Lite
Memcached
Add the appropriate ﬂags to your init script for details on each
Docs: http://gearman.org/index.php?id=manual:job_server#persistent_queues
Example: MySQL
/sbin/gearmand -q libdrizzle --libdrizzle-host=127.0.0.1 --libdrizzle-user=gearman
--libdrizzle-password=secret --libdrizzle-db=queue --libdrizzle-table=gearman
--libdrizzle-mysql

Defaults to single thread (mostly)
Some versions default to use more than one...
Non-blocking I/O which works very fast with a single thread
To give each of the internals a dedicated thread use
/sbin/gearmand -d -t 3
Additional threads are then used for client worker connections

Security
Lock gearmand to a single IP
/sbin/gearmand -d -L 127.0.0.1
Change the port from the default 4730
/sbin/gearmand -d -L 127.0.0.1 -p 7003

Security
Lock gearmand to a single IP
/sbin/gearmand -d -L 127.0.0.1
Change the port from the default 4730
/sbin/gearmand -d -L 127.0.0.1 -p 7003

HTTP Access
Gearmand supports a pluggable interface architecture and can use HTTP for
communication. Requests are sent using GET and POST data.

Conﬁguring Gearmand for high availability

Gearmand doesn’t replicate data
May be a weak point depending on your use case

Gearmand doesn’t replicate data
May be a weak point depending on your use case
Redundancy without a load balancer
In the client logic, add multiple servers and let the driver sort it out
Workers register themselves with each gearmand server
Done.

Easy to load balance
Put the two servers behind a load balancer
Each client connects to the load balancer
Workers register themselves with each gearmand server
Done.

Adding Gearman API to PHP
Package managers to the rescue again!
sudo apt-get install php-pecl-gearman
sudo yum install php-pecl-gearman

Pecl
sudo pecl install gearman
sudo echo “extension=gearman.so” > /etc/php.d/gearman.ini

Pecl
Quick test
php -i | grep “gearman support”
gearman support => enabled

Pecl
Quick test
php -i | grep “gearman support”
gearman support => enabled

Restart any services with PHP resident in memory
Apache, Nginx, etc

Can we get to the code already?

Gearman in PHP - The Client
Clients create the jobs and tasks for the workers to do

Create a client object
$gm = new GearmanClient();

Deﬁne the server(s) to use
$gm->addServer(); // defaults to localhost:4730

Create the job and wait for a response
$jobdata = “/* any valid string */”;
do {
$res = $gm->do(‘image_resize’, $jobdata);
} while ($gm->returnCode() != GEARMAN_SUCCESS);

Create the job and wait for a response
$jobdata = “/* any valid string */”;
do {
$res = $gm->do(‘image_resize’, $jobdata);
} while ($gm->returnCode() != GEARMAN_SUCCESS);
Or don’t
$jobid = $gm->doBackground(‘image_resize’, $jobdata);

Gearman in PHP - The Worker
Workers do the jobs and tasks the clients request

Deﬁne the callbacks
function callback_resize($job){ /* do stuff */ }
function callback_watermark($job){ /* do stuff */ }

Create a worker object
$gm = new GearmanWorker();

Deﬁne the server(s) to respond to

Tell the server which callbacks to use for each task
$gm->addFunction(‘resize’, ‘callback_resize’);
$gm->addFunction(‘watermark’, ‘callback_watermark’);

Wait for work to do
while($gm->work());

Wait for work to do
while($gm->work());
Callback functions can post status updates on their progress back to the client
$job->setStatus($numerator/$denominator);

Example Clients in CodeIgniter
class Photos extends CI_Model {

! public function resize_sync($filename){
! ! $gm = new GearmanClient();
! ! $gm->addServer('127.0.0.1', 4730);
! ! do {
! ! ! $res = $gm->do('image_resize', $filename);
! ! ! switch($gm->returnCode()){
! ! ! ! case GEARMAN_WORK_FAIL:
! ! ! ! ! return FALSE;
! ! ! ! case GEARMAN_SUCCESS:
! ! ! ! ! return TRUE;
! ! ! }
! ! } while ($gm->returnCode() != GEARMAN_SUCCESS);!
! return TRUE;
! }

! public function resize_async($filename){
! ! $gm = new GearmanClient();
! ! $gm->addServer('127.0.0.1', 4730);
! ! $res = $gm->doBackground('image_resize', $filename);
if(!$res){ return FALSE; }
! return TRUE;
! }

}

The Callback Handlers

! public function resize_sync($filename){ /* ... */ }
! public function resize_async($filename){ /* ... */ }

! static public function image_resize($job){
! ! $filename = $job->workload();
! ! $CI =& get_instance();
! ! $config = array(
! ! ! 'source_image' => $filename,
! ! ! 'create_thumbnail' => TRUE,
! ! ! 'maintain_ratio' => TRUE,
! ! ! 'width' => 800,
! ! ! 'height' => 600,
! ! );
! ! $CI->image_lib->initialize($config);
! ! $CI->image_lib->resize();
! ! $CI->image_lib->clear();
! ! return true;
! }
! static public function image_watermark($job){ /* ... */ }

}

The Worker

! public function resize_sync($filename){ /* ... */ }
! public function resize_async($filename){ /* ... */ }

! static public function image_resize($job){ /* ... */ }
! static public function image_watermark($job){ /* ... */ }

! public function gearman_worker()
! {! !
! ! $gm = new GearmanWorker();
! ! $gm->addServer('127.0.0.1', 4730);
! ! $gm->addFunction('image_resize', 'Photos::image_resize');
! ! $gm->addFunction('image_watermark', 'Photos::image_watermark');
! ! while($gm->work());
! }

}

Bringing it all together

class Upload extends CI_Controller {

! function index(){
! ! $this->load->model('photos');
! ! $this->load->helper(array(‘url’));
! ! $ul_config = array( /* ... */ );
! ! $this->load->library(‘upload’, $ul_config);
! ! if($this->upload->do_upload()){

! ! ! $file_info = $this->upload->data();
! ! ! $this->photos->resize_async($file_info['full_path']);

! ! ! redirect(site_url(‘controller/next’));

! ! } else { /* error handling code */ }
! }

! function worker(){
! ! $this->load->model('photos');
! ! $this->photos->gearman_worker();
! }

}

Running your workers

#!/bin/bash
SCRIPT="index.php upload gearman_worker"! # this is your CI controller/method
WORKDIR=/var/www/html/ ! ! ! # this is your CI app root
MAX_WORKERS=5! ! ! ! # number of workers
PHP=/usr/bin/php! ! ! ! # location of PHP on your system
COUNT=0! ! ! ! ! # internal use variable

for i in `ps -afe | grep "$SCRIPT" | grep -v grep | awk '{print $2}'`
do
! COUNT=$((COUNT+1))
done

if test $COUNT -lt $MAX_WORKERS
then
! cd $WORKDIR
! $PHP $SCRIPT
else
! echo There are enough workers running already.
fi

About the job data
Job data is always a string

About the job data
You can serialize data to pass more complex objects
$data = array(
‘filename’ => $filename,
‘methods’ => array(‘resize’, ‘watermark’),
‘user_id’ => $user_id,
);
$jobdata = json_encode($data);
$gm->doBackground(‘processor’, $jobdata);

About the job data
You can serialize data to pass more complex objects
$data = array(
‘filename’ => $filename,
‘methods’ => array(‘resize’, ‘watermark’),
‘user_id’ => $user_id,
);
$jobdata = json_encode($data);
$gm->doBackground(‘processor’, $jobdata);
And then deserialize it in the worker
$jobdata = $job->workload();
$data = json_decode($jobdata, true);

About the job data
In theory, you can pass about 2Gb per message
Limited by the server protocol which uses a 32 bit integer to deﬁne
message size.

About the job data
message size.
You can pass binary data by encoding it
$binary_base64 = base64_encode($binary);

About the job data
message size.
You can pass binary data by encoding it
$binary_base64 = base64_encode($binary);
But use it sparingly
Any data passed is stored in memory
and your persistent store (if enabled)
and is likely on disk already
Base 64 encoded data is roughly 1 1/2 times the size of the source
data too
Compressing the binary data ﬁrst can help

About the job data
Don’t pass 2Gb objects

About the job data
Keep your message data small, less than 64K is perfect
Amazon’s SQS has a hard limit of 64K

About the job data
Pass pointers instead
Pass paths and ﬁlenames instead of ﬁles
Pass cache keys instead of complex class objects
that won’t deserialize right anyway

About the job data
Pass pointers instead
Pass paths and ﬁlenames instead of ﬁles
Pass cache keys instead of complex class objects
that won’t deserialize right anyway

Gearman is not a data store!

Analytics: Problem
We wanted to provide near realtime analytics to our clients

Analytics: Problem
We liked the data we could scrape out of Google

Analytics: Problem
Networks Referrers Location

Analytics: Problem
so we did... for a while... and it was good!

Analytics: Problem
But Google has API limits for getting data out of Analytics

Analytics: Problem
Only a few thousand requests per day

Analytics: Problem
Batching requests was good - but not ideal

Analytics: Problem
Data was only monthly, no daily, weekly breakdowns

Analytics: Problem
Still ended up taking weeks to gather a full month of data

Analytics: Problem
Still ended up taking weeks to gather a full month of data
We already had scraped a signiﬁcant amount of Google data so any
future collection would need to play nice with the existing data

Analytics: Solution
Build our own platform that gathered exactly what we wanted
modeled after Google Analytics

Analytics: Solution
How we did our tracking pixel

Analytics: Solution
The page has a small javascript ﬁle included that

Analytics: Solution
captures the current url, user agent, referrer and so on

Analytics: Solution
and embeds it as an image tag with a carefully crafted URL for the 1x1
tracking pixel /statistics/pixel/?ua=Mozilla%2F...

Analytics: Solution
The pixel request is handled by CodeIgniter, which returns the image,

Analytics: Solution
but not before we lookup the country, state, city of the originating IP and
pass that data into Gearman for future processing.

Analytics: Solution
The Gearman worker then parses and normalizes the data,

Analytics: Solution
The Gearman worker then parses and normalizes the data,
performs a host lookup on the source IP (which is often very slow) all
before recording the result in our datastore.

Patterns and Recipes
One to one
Workers can run in optimized environments for speciﬁc tasks
Example: Using R, Matlab etc to run mathematic analysis and still use CI to for
the front end
Example: Run code on a different server to avoid CPU/disk/memory contention
with Apache etc.
Example: Run tools on Windows platforms like .NET components for generating
word ﬁles.

One to one
Workers can run in optimized environments for specific tasks
Example: Using R, Matlab etc to run mathematic analysis and still use CI to for
the front end
Example: Run code on a different server to avoid CPU/disk/memory contention
with Apache etc.
Example: Run tools on Windows platforms like .NET components for generating
word files.

One to many
A single client can part out jobs to multiple workers
Example: Performing TF*IDF analysis on documents to find keywords
Example: Handling image manipulations in parallel, resizing 2 or 3 new
thumbnails at a time.

Many to one
Multiple clients utilizing a single worker thread
Share memory across jobs. Arrays are faster than APC, Memcached, MySQL
Example: Database write buffering (for non critical data only)
Example: Perform database writes across shards. MySQL UDF inserts a record into
gearman that’s then re-written out to appropriate user shards

Many to one
Multiple clients utilizing a single worker thread
Share memory across jobs. Arrays are faster than APC, Memcached, MySQL
Example: Database write buffering (for non critical data only)
Example: Perform database writes across shards. MySQL UDF inserts a record into
gearman that’s then re-written out to appropriate user shards

Optimize front end displays
Example: Pagination optimization
User requests the ﬁrst page of data
Kick off a background task to pre-cache the next page before it’s requested
Example: Dashboards
User logs into your site, you ﬁre off background tasks to generate dashboard information
that user will need
Workers begin crunching the data and store the results in a cache
Meanwhile, the user is served a page with spaces allocated for each widget, which are then
loaded via AJAX

Delayed/Deferred Processing
Use whenever tasks would run long or are potentially unreliable
Preparing images
Preparing video
Remote service calls
Sending content to Twitter, Facebook, LinkedIn...
Triggering other remote services faxes, emails etc
Prefetching data

Other Solutions
Most alternative solutions have APIs for PHP
Some interface over other protocols like HTTP(s) or memcache,
which can ease deploying new servers

Other Solutions
Most alternative solutions have APIs for PHP
Some interface over other protocols like HTTP(s) or memcache,
which can ease deploying new servers
Alternatives
Active MQ
Amazon’s SQS
Beanstalkd
Microsoft Message Queuing
RabbitMQ
Others - check for activity before adopting

Some helpful tips
Not everything belongs in a work queue

Some helpful tips
Queue it if...

Some helpful tips
Queue it if...
the process is intensive on any subsystem, CPU, RAM etc

Some helpful tips
Queue it if...
the process is slow, taking 1/2 second or longer to run, proﬁle your
app with CodeIgniter’s built-in proﬁler.

Some helpful tips
Queue it if...
you want the beneﬁt of running in parallel

Some helpful tips
Queue it if...
Don’t use your database server as your work queue

Some helpful tips
Queue it if...
It might work short term, but it’s not scaleable.

Some helpful tips
Queue it if...
Persistence comes with a cost

Some helpful tips
Queue it if...
Writes to the datastore are never free and will slow down the queue

Some helpful tips
Queue it if...
Writes to the datastore are never free and will slow down the queue
Adds an additional layer of stuff to maintain

(this slide intentionally left blank)

Work Queues

More Related Content

What's hot (11)

Viewers also liked (20)

Similar to Work Queues (20)

More from ciconf (7)

Recently uploaded (20)

Work Queues

Editor's Notes