TX/RX 101: Transfer data efficiently

Tx/Rx 101: Transfer data efficiently

SAPO Codebits 2011
Lisbon - http://codebits.eu
Lourens Naudé
Wildfire Interactive, Inc. | 1600 Seaport Boulevard, Suite 500, Redwood City, CA 94063 | (888) 274-0929

Thursday, November 10, 11

Bio

Operations
@ WildfireApp.com



A few months ago at WildﬁreApp.com I made the transition from a developer background and role to the operations team.
With an Engineering mind it’s easy to get distracted by lower level details and loose sight of the bigger picture. During this
time we’ve released several products and this changed my perspective on operational complexity, moving parts and the
importance of getting and keeping stable software out in front in customers. Less is more. There will be some notes on this
during today’s talk as well.

Communication

All modern software should
represent a connected
world.



Lots of trending products are actually platforms, spread across multiple devices. API and integration driven, like Twitter. Even
this event exposes various APIs for projects to consume in the coming days. Software is social.

Communication

Most social problems are
communication problems.

Systems aren’t any different



Two outstanding individuals can engage in a relationship, but it’s bound to fail if there’s little or no communication.

All systems GO !


http://www.ﬂickr.com/photos/23968709@N03/5452362745/

Efficiency
Performant individual requests
Scalability
Operational complexity
Minimal / no developer on boarding
Cost of ownership (today and forward
looking)



Different things to different people. It’s not just about performant individual requests, but also scaling those out to multiple
clients. Operational complexity and moving parts should be kept to a minimum. Often overlooked, especially in startup
environments, is on boarding costs of new developers, especially in the presence of lower level / funky protocols etc. that
require a bootstrap period. Rule of thumb: your cowboy should get it. All of this directly affects the bottom line.

Efficient Scalability

To maintain acceptable
response times for more clients,
with the same or less
infrastructure



This is representative of most public facing production deployments. Remember, not about the speed of a single
request, but being able to easily handle increased throughput.

Agenda
System calls and file descriptors
Blocking, nonblocking and async I/O
Multiplexed I/O
There’s a better way ...
Optimizations



We’ll allocate time towards system calls, ﬁle descriptors - the canonical reference used by user space apps for I/O and how all
of that ﬁts into common I/O models: blocking, nonblocking and async. The next challenge is applying this knowledge at scale
using multiplexed I/O, however apparently the world is coming to an end in 2012 and there’s much better alternatives. We’ll
then phase out with a few common optimizations.

Head count ?



Anyone currently consuming or exposing APIs ? Messing around with or using any of these technologies in
production ?

The kernel

Mediates access to system
resources :
I/O, memory and CPU



Since the 60s, nothings really changed much with how we access system resources. First line of communication:
between the Kernel and User space is a negotiation for resources.

User mode

Sandboxed execution for
security and stability
requirements



The vast majority of processes, especially scripting languages, will spend most of their on CPU time in user mode.
How does user processes access system resources ?

System calls

Ruby
process

Kernel disk

Ruby
process
read(5, &buf, 4000)


Ask the Kernel to do work on behalf of a user process.

Syscall definition

A protected function call
from user to kernel space
with the intent to interact
with a system resource



Characteristics
Uniform API on POSIX systems
Ditto for behavior. LIES !
Slow emulations

ssize_t read(int d, void *buf,
size_t nbytes)

ssize_t write(int d, const void
*buf, size_t nbytes)



It's also guaranteed to have a uniform API and behavior on POSIX compliant (read: most UNIX) systems. Just like
HTML and CSS compliance in modern browsers, most operating system implementations do not conform 100% to
spec. Conﬁgure checks at compile time may conﬁrm the presence of a performant system call when in fact it’s just
an emulation.

Syscall performance
MUCH slower than function calls
20x and more
Definite context switch
Calls can block - slow upstream peers



An important performance characteristic of system calls is that they're an order of magnitude more expensive
than eg. a libc function
call in the same process. In other words, there's a context switch that affects several subsystems eg. swapping out
CPU registers and memory references.

They may also block indeﬁnitely or until a condition's met eg. a full buffer on a read syscall.

Who doesn't know what file
descriptors or file handles
are ?



open syscall

fd = open(“/path/file”, O_READ)
read(fd, &buf, 4000)



OPEN system call to open a ﬁle. Give it a path and it returns a resource, which is used in subsequent READ and
WRITE calls for I/O.

Examples
local file
socket
directory
pipe
Consistent API, semantics differ



A ﬁle descriptor may represent a local ﬁle, a socket, a directory or a pipe etc. Semantics for these resources differ
immensely, even
though the canonical reference is a numeric handle and the basic read and write system call APIs are equivalent.
Only ever a device to the kernel.

Definition
Numeric handle
References a kernel allocated resource
Kernel and user space buffer

/path/serv.sock
( fd 5 )

User Kernel
buffer buffer



A ﬁle descriptor (or handle) is a numeric handle representing an I/O resource allocated by the Kernel, with
metadata in the Kernel and buffers in both Kernel and User space. SIMPLE and EFFICIENT contract.

Blocking I/O

A request to read a chunk of
data from a descriptor may
block depending on
buffer state



File descriptors are blocking by DEFAULT. Easiest I/O model, but also unpredictable with the worst performance
guarantees.

Blocked buffer state

read(5, &b, 4000)

User Kernel
buffer buffer
2000
1000



Kernel puts the caller process in a waiting state. The application ( execution thread ) sleeps until the I/O operation
has completed (or has generated an error) at which point it is scheduled to run again. Simple to use, but many
downsides. Domino effect on connected peers.

Nonblocking I/O

fd = socket (PF_INET, SOCK_STREAM, 0);
fcntl (fd, F_SETFL, O_NONBLOCK)



The POSIX spec allows for changing this behavior by setting the O_NONBLOCK ﬂag on a descriptor.

Nonblocking I/O

A read request would only
be initiated if the system
call won’t block



read(5, &b, 4000)

User Kernel
buffer EAGAIN buffer
2000
1000



In this scenario, we’ll partial read the 1000 bytes, but errno would be set to EAGAIN, indicating that we need to
issue the read request again.

Lots of system call overhead – constantly making system calls to see if I/O is ready. Can be high latency if I/O
arrives and a system call is not made for a while. Efficiency is very poor due to the constant polling with system
calls from user space

Don’t drink the kool aid
Guideline only
Not free - invokes a syscall
Not supported by all devices



Do note that this behavior is thus just a guideline for the caller. We’re told to retry, just not when. It's not free
either - there's the context switch as well as system call arguments validation in the Kernel as well.

There's a considerable grey area, as per POSIX spec, that ignores O_NONBLOCK semantics on any ﬁle handle that
references a ﬁle on a
block device. This has an important performance impact on systems that access block devices with variable
performance characteristics, like EBS @ Amazon AWS. All system calls don’t respect non-blocking I/O either
( libeio / libuv )

Async I/O myth

Nonblocking I/O IS NOT
asynchronous I/O



File descriptors change state ( read / write ) asynchronously at the Kernel layer, we handle those events in User
space. A NIC receives data which is buffered in the kernel. We still need to ask for and copy it. The takeaway, and
what you MUST remember here is that this IS NOT asynchronous I/O.

An event occurs asynchronously, but is handled synchronously.

Async I/O
Callbacks and notifications
Issues I/O in parallel
Direct I/O (no double buffering)
Popular with database systems
Tell the Kernel everything it needs to
complete an I/O job



Helps maximize I/O throughput by allowing lots of I/O to issued in parallel. Network IO is frequently not
supported.

fd : 10
op : read
notify : signal
buffer : 0x100431
submit job
size : 4

callback() data, notify

"data"

Application Kernel



POSIX Realtime Extension
aio_read, aio_write etc.
Supports file I/O only



OS X Lion - AIO support

case SIGEV_NONE:
! ! break;

case SIGEV_THREAD:
! /* Unsupported [RTS] */

default:
! return (EINVAL);



Preferred notiﬁcation mechanism is through signal ( SIGIO ), but doesn’t support passing any metadata either.
Thus if there’s 10 jobs submitted, we still need to ﬁgure out which of those jobs are in a completed state.

Windows I/O Completion Ports
Supports all descriptor types
Very mature and stable technology



Redmond has given us malware, but has done some things right as well.

Recap
O_NONBLOCK is a guideline only
Invoked in user space
Data transfer in user space
Blocking and nonblocking I/O terms



Moving forward, we shall prefer the terms blocking and non-blocking I/O as the work done transferring data from
Kernel to User space
buffers is still done in user space. Even if we do change descriptor behavior with the O_NONBLOCK ﬂag, it's merely
a guideline for the
caller to choose a strategy as the intent and additional context is known there.

Challenges of scale
Large number of file descriptors
Respond to descriptor state changes
Execute domain logic



SINGLE FDs until now. Remember, the primary uses case is increased throughput. The main challenge here is
handling a large number of ﬁle descriptors (representing client connections) efficiently in a single process.
Previous examples represented single descriptors only.

Multiplexed I/O

Who's familiar with select,
poll, epoll or kqueue ?



Multiplexed I/O
Nonblocking I/O is inefficient
Retry on EAGAIN is polling
Multiplexed I/O: concurrent blocking
Notified of state changes on any
registered fd



Nonblocking I/O is inefficient - we continuously need to retry on EAGAIN error conditions and as such waste
resources polling. Multiplexed I/O allows an application to concurrently block on a set of ﬁle descriptors
(thousands with epoll and kqueue) and receive notiﬁcation when any of them are in a readable or writable state.
We aim to block on a set of descriptors, not just 1.

Multiplexed I/O
fd 5
fd 6
fd 7
fd 8
I/O bound
App Multiplexer

fd 9
fd 10
fd 11
fd 12

User space Kernel space



At any given time a small subset of the descriptors registered with the multiplexer would be in a readable or
writable state.

Multiplexed I/O

1. Tell me when fds 1..200
are ready for a read or a
write



Multiplexed I/O

2. Do other work until one
of them's ready



Multiplexed I/O

3. Get woken up by the
multiplexer on state
changes



Multiplexed I/O

4. Which of fd set 1..200
are in a ready state ?



Multiplexed I/O

5. Handle all fds ready for
I/O



Repeat. There will almost always just be a small subset of fds ready to drive work from. The more we have, the
smaller this total : ready ratio generally becomes.

Checkpoint.

Questions ?



ETOOCOMPLEX



You don’t need a UNIX beard in your department. And you should be building apps.

http://www.ﬂickr.com/photos/taniwha/2237514055/

There’s a better way

http://zeromq.org



Recently I’ve spent some free time on a new set of Ruby bindings for ZeroMQ - a communications and
concurrency framework. This framework is going to change the way we do I/O moving forward - plans for Linux
Kernel integration.

Super Sockets

Scripting for sockets



We spend copious amounts of time implementing business logic, refactoring and often don’t think of how
individual components will communicate. Requirements also change over time - sockets should be “scriptable” as
well. Not an out the box technology - provides the building blocks for solutions. Like legos. Especially important
in service integration - connect with Facebook, Twitter etc. Social software. Crappy communication patterns with
third party integrations is much more likely to affect your performance proﬁles than inefficient code paths
deployed from within your organization.

BSD sockets + Messaging patterns

REQ PUB PULL PULL

REP SUB SUB PUSH



Maps to messaging patterns
Request-Reply, Publish-Subscribe, Pipeline
Prefers mailboxes to buffers and is asynchronous by design ( Actor Pattern )
Build out topologies just in time.
Allows us to focus on interfaces and patterns ﬁrst

Transport agnostic
inproc (Threads)
IPC
TCP/IP
Multicast
Uniform API



Scales to X cores: threads in a process, processes on a box, boxes on a network
Same API regardless of transport

Operations can deploy a single process on ﬁrst service iteration, using threads to utilize cores. If the engineering
team’s weak with multithreaded programming ( or the codebase is too complex ), can move to IPC transport, which
requires managing more processes but allows developers to code for a single thread of execution. If and when
capacity is required, a TCP transport allows for scaling to multiple boxes.

Brokerless



Broker is always a SPOF
Less admin overhead to not rely on message broker.
Any intermediate is slow - most brokers touch disk extensively for durability
Sometimes platforms are built around the capabilities of a particular broker implementation
http://www.ﬂickr.com/photos/epsos/5591761716/

FAST
methodmissing:rbczmq lourens$ ruby perf/local_thr.rb tcp://
127.0.0.1:5020 10 1000000
message size: 10 [B]
message count: 1000000
mean throughput: 410276 [msg/s]
mean throughput: 32.822 [Mb/s]

methodmissing:rbczmq lourens$ ruby perf/local_thr.rb tcp://
127.0.0.1:5020 1024 1000000
message size: 1024 [B]
message count: 1000000
mean throughput: 176342 [msg/s]
mean throughput: 1444.595 [Mb/s]

methodmissing:rbczmq lourens$ ruby -v
ruby 1.9.2p290 (2011-07-09 revision 32553) [x86_64-
darwin11.0.1]



Minimal latency - higher than TCP (which is normal), BUT higher throuhgput than TCP ( batching )
Communications and concurrency framework - can load balance between threads and processes as well
Socket options allow for tuning individual sockets - not system wide as per raw sockets

Resilient
Auto-reconnects
Durable sockets
Socket options: backlog, HWM, SWAP
Clients can come up before servers
Atomic message delivery



Manage a socket’s ability to handle variable throughput and not drop messages with the backlog, HWM and SWAP
socket options.
Easy for upgrades if we don’t need to always coerce things in sequence.
Fault tolerant message passing - atomic - you either get the message (blob), or you don’t

Interjection principle

REQ REQ REQ

ROUTER
DEALER

REP REP REP



Interjection principle states that inserting an intermediary node into the topology should not change the
behavior at the endpoints.
DEALER / ROUTER pair allows for non-blocking request/response. Great for slow / fast clients alike.

X Platform
Multiple OS’s
Multiple languages - clients ecosystem
No message contracts - BLOBS



A good clients ecosystem is so often overlooked. There’s a Syslog module for easily tapping into event streams as
needed as well.

ZMQ efficiency
Brokerless (or roll your own)
Easy to re-architect
Swap transport to scale to more cores
Resiliency
Specialize as needed with socket
options



Working with multiple buffers

Write data from multiple
buffers to a single stream ...
b1

b2

b3


... or read data from a single
stream to multiple buffers.
b1

b2

b3


Vectored I/O

ssize_t writev(int fildes,
const struct iovec *iov,
int iovcnt);

ssize_t readv(int fildes,
const struct iovec *iov,
int iovcnt);



Benefits
Atomic - reads / writes all or nothing
Reduces the amount of syscalls
Fixed sized headers
Database records
Well supported



Reduces the amount of system calls required to read / write data from non-contiguous buffers

Write example

b1

b2


Call write for each buffer

write
b1
write
b2


Allocate buffer large enough for both
buffers, copy data and write once

copy
b1 write
b3
copy
b2


Call writev once to output both buffers

b1 writev

b2


ZeroMQ message batching

Higher latency than TCP,
but also higher throughput.



Message batching

Sending two or more
messages down the
networking stack is faster
than individual messages.



Characteristics
Opportunistic batching
NIC asks 0mq for messages
Sends all available at any given time
Low latency for single VS multiple
messages use case



Zero-Copy sends
Saves on memcpy()
Delegate ownership of a buffer to omq
Scripting language strings
Callback function on send
Tricky with Garbage Collection



void free_msg(void *data, void *s);

zmq_msg_init_data(&m, RSTRING_PTR(s),
RSTRING_LEN(s), free_msg, (void*)s);

rb_gc_register_address(&s);
zmq_send(s, &m);

void free_msg(void *data, void *s){
rb_gc_unregister_address(&s);
}


"message"

"message" "message"

callback() "message"

"message"

Ruby ZeroMQ



Message in Ruby VM VS message in ZeroMQ

Passing thoughts
System calls
Blocking, nonblocking and async I/O
Messaging patterns ftw!
Efficiency is context dependent
Read http://zguide.zeromq.org
Reread http://zguide.zeromq.org



Questions ?



Wildfire Interactive, Inc. is hiring



Thanks !

http://www.jobscore.com/jobs/
wildfireapp
follow @methodmissing
fork github.com/methodmissing



TX/RX 101: Transfer data efficiently

More Related Content

Similar to TX/RX 101: Transfer data efficiently (20)

More from Lourens Naudé (7)

Recently uploaded (20)

TX/RX 101: Transfer data efficiently