BINARY DATA ADVENTURES IN BROWSER JAVASCRIPT

BINARY DATA ADVENTURES
IN BROWSER JAVASCRIPT
OR HILTCH / CTO @ STREAMRAIL

INTRO
• Background
• Evolution in the browser
• The Binary Arsenal in JS (type system,
inputs, outputs)
• Solving Cool, Practical Problems
00110110011100100101010100100
10101011101000001010100101010
01010010111010101001010101010
010101011111001010100010101110
0101010111101000000101010100
01011001010101010111100001010

TRADITIONALLY
• Outside of JS:  
Low level stuff: Compression (gzip/deflate), Bitfields (ACLs),
Networks (CRC32), Graphics (algorithms, positioning), ...
• Inside of JS:  
Solving weird issues

$http.post('/someUrl', {score:42}). 
success(function(data, status, headers, conﬁg) { 
// not today :-( 
}). 
error(function(data, status, headers, conﬁg) { 
// data=Object {‘error’: ‘Float64 is not int’} 
}

In JS, all numbers are of type Float64
But all bitwise ops are Int32!
Bitwise OR: number | 0
Bitwise NOT: ~~number
Shift: number >> 0
parseInt: parseInt(number, 10)
ﬂoor: Math.ﬂoor(number)
modulo: number - number % 1

BINARY DATA ADVENTURES IN BROWSER JAVASCRIPT

BROWSERS, EVOLVED
• Type system: Blob, ArrayBuﬀer, TypedArray, DataView
• Inputs: 
Input methods: XHR 2, File API, Canvas, WebSockets, WebRTC, ...
• Outputs: 
MSE, Canvas, WebSockets, WebRTC, WebGL, ...

WHAT MAKES
TYPED ARRAYS FAST
• Passed to native interfaces completely AS IS (direct memory access)
• Native “endianness” - watch out!
• DataView adapter
Byte order
0x12345678 Hex inside a Uint32Array of 4
Little-endian: 78 56 34 12
12 34 56 78Big-endian:

STREAMING VIDEO
WITH JS
• No plugins (Netﬂix: silverlight, Facebook: ﬂash)
• MSE: Pure JS DASH/HLS (YouTube)
• DRM Support (EME)
• MSE + WebRTC: Bittorrent, in the browser! (WebTorrent)

‘USE ASM’
sys/libkern/strlen.c
 
size_t strlen(const char *str) {
const char *s;
for (s = str; *s; ++s);
return(s - str);
}
asm.js
 
function strlen(ptr) { 
ptr = ptr|0; 
var curr = 0; 
curr = ptr; 
while (MEM8[curr]|0 != 0) { 
curr = (curr + 1)|0; 
} 
return (curr - ptr)|0; 
}

a subset of JS for handling numbers faster
• OdinMonkey - AOT in FF (fallback to IonMonkey JIT)
• Support by all major browsers (to an extent)
• Typed Array used as “memory”
• All add/sub are 32 bit (number | 0)
• DI: function(stdlib, foreign, heap) 
‘USE ASM’

KRIPKEN TO THE RESCUE
C/C++ > LLVM > Emscripten > asm.js!
• Huge code bases (SQLite, BananaBread,J2ME VM, …) 
- close to impossible by hand
• Enjoy Clang/LLVM optimizations 
- decades of work done to optimize compiled code
• OpenGL > WebGL “for free”
- directly map a lot of OpenGL ES 2.0 command to WebGL
- not only for graphics rendering, also for GPU oﬄoading

• Use Docker to run Emscripten
- lots of software (emscripten, emscripten-fastcomp, emscripten-
fastcomp-clang, llvm clang)
- build your own or use one from DockerHub
KRIPKEN TO THE RESCUE

• Obtain bytes of video (and audio) by downloading them
• Decode the video (and audio) using native decoders in the browser/OS
• Animate the frames to create the video experience
 
VIDEO TAG EMULATION

• Obtain bytes of video (and audio) by downloading them
> XHR 2.0, response type “arraybuﬀer”
• Decode the video (and audio) using native decoders in the browser/OS
> Cross compile a native decoder to JS using Emscripten
• Animate the frames to create the video experience
> For each decoded pixel, create a 2D Texture, transform it to RGBA using
a fragment shader, and render it with WebGL using canvas
(“experimental-webgl”). 
VIDEO TAG EMULATION

YUV -> RGB
• Luma + (2 * Chroma) vs  
3 * (Chroma + Luma)
• Different needs, different inputs
• Color space conversion heavily
computational (floating point coefficients
matrix multiplication)

function yuv2canvas(buffer, width, height) {
var lumaSize = width * height;
var chromaSize = lumaSize >> 2;
webGLCanvas.YTexture
.fill(buffer.subarray(0, lumaSize));
webGLCanvas.UTexture
.fill(buffer.subarray(lumaSize, lumaSize + chromaSize));
webGLCanvas.VTexture
.fill(buffer.subarray(lumaSize + chromaSize,
lumaSize + 2 * chromaSize));
webGLCanvas.drawScene();
}
4:2:0
=
+

precision highp ﬂoat;
varying highp vec2 vTextureCoord;
uniform sampler2D YTexture;
uniform sampler2D UTexture;
uniform sampler2D VTexture;
const mat4 YUV2RGB = mat4 (
1.1643828125, 0 , 1.59602734375, -.87078515625,
1.1643828125, -0.39176171875, -0.81296875 , .52959375,
1.1643828125, 2.017234375 , 0 , -1.081390625,
0 , 0 , 0 , 1
);
void main(void) {
gl_FragColor = vec4(texture2D(YTexture, vTextureCoord).x,
texture2D(UTexture, vTextureCoord).x,
texture2D(VTexture, vTextureCoord).x,
1)
* YUV2RGB;
}
<SCRIPT TYPE="X-SHADER/X-FRAGMENT">

THANK YOU!
OR HILTCH / CTO @ STREAMRAIL

BINARY DATA ADVENTURES IN BROWSER JAVASCRIPT

Recommended

More Related Content

What's hot (19)

Viewers also liked (20)

Similar to BINARY DATA ADVENTURES IN BROWSER JAVASCRIPT (20)

Recently uploaded (20)

BINARY DATA ADVENTURES IN BROWSER JAVASCRIPT