ML in the Browser: Interactive Experiences with Tensorflow.js
0 likes1,783 views
This document discusses machine learning in the browser using Tensorflow.js. It begins with an introduction and overview of Tensorflow.js, including how it can be used for both authoring models and importing pre-trained models for inference. Examples are provided of using the Ops API to fit a polynomial function and the Layers API to build and train an autoencoder in the browser. Challenges of developing machine learning applications in the browser are also discussed.
![64
# Keras, Python
# Encoder
inputs =
Input(shape=(num_feature
s,))
# define hidden layers
// Tensorflow JS
// Encoder
const input = tf.input({
shape: [numFeatures] })
# define hidden layers](https://image.slidesharecdn.com/untitled-200109133729/85/ML-in-the-Browser-Interactive-Experiences-with-Tensorflow-js-66-320.jpg)
![79
[{
bbox: [x, y, width, height],
class: "hand",
score: 0.8380282521247864
},
{ bbox: [x, y, width, height],
class: "hand",
score: 0.74644153267145157
}]](https://image.slidesharecdn.com/untitled-200109133729/85/ML-in-the-Browser-Interactive-Experiences-with-Tensorflow-js-81-320.jpg)
ML in the Browser: Interactive Experiences with Tensorflow.js
- 1. 1 ML in the Browser Victor Dibia, PhD Cloudera Fast Forward Labs @vykthur Interactive Experiences with Tensorflow.js
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ ml-browser-tensorflow-js/
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. 2 - Background in CS, Human Computer Interaction (HCI), Applied AI. - Previously Research Scientist at IBM Research New York. - Community Contributions - GDE for Machine Learning - Research Engineer (ML) at Cloudera Fast Forward Labs Research Engineer Cloudera Fast Forward Labs, Brooklyn. @vykthur | @victordibia
- 6. 4
- 7. 5
- 9. 7 Agenda • Why Machine Learning in the Browser? • Overview of the Tensorflow.js library API with examples • The Ops API • The Layers API • Building Handtrack.js - a javascript library for prototyping gesture based interactions in the browser. •
- 10. 8 Why Machine Learning in the Browser?
- 11. Artificial Intelligence Machine Learning Deep Learning The broad field that describes efforts to make machines intelligent. Algorithms that enable machines independently learn from data and make predictions. Decision trees, random forests, reinforcement learning, neural networks, Neural Networks (NN): a set of stacked computation units. - High accuracy, scales with data - Advances in NN algorithms - Amenable to GPU computation But What is Machine Learning? Let's review some terminology ..
- 12. Artificial Intelligence Machine Learning Deep Learning The broad field that describes efforts to make machines intelligent. Algorithms that enable machines independently learn from data and make predictions. Decision trees, random forests, reinforcement learning, neural networks, Neural Networks (NN): a set of stacked computation units. - High accuracy, scales with data - Advances in NN algorithms - Amenable to GPU computation
- 13. Artificial Intelligence Machine Learning Deep Learning The broad field that describes efforts to make machines intelligent. Algorithms that enable machines independently learn from data and make predictions. Decision trees, random forests, reinforcement learning, neural networks, Neural Networks (NN): a set of stacked computation units. - High accuracy, scales with data - Advances in NN algorithms - Amenable to GPU computation For today, each time I refer to Machine Learning or AI, what I am referring to is Deep Learning
- 15. Dense layer (nodes in one layer connected to all nodes in the following layer). Fast Forward Labs Multilayer Perceptron
- 16. 14 The Languages of Machine Learning
- 17. 15 The Languages of Machine Learning
- 18. 16 The Languages of Machine Learning
- 19. 17 The Languages of Machine Learning Backend programming languages .. and they work well! .. but ..
- 20. 18 • Cloudera Fast Forward Labs • Intro to Transfer Learning for NLP • Applications • Appendix: Positioning & FAQ (internal audience only) 2019 Stackoverflow survey of 90k developers What programming language do developers use?
- 21. 19 2019 Stackoverflow survey of 90k developers What programming language do developers want to learn?
- 22. 20 Github 2018 Top Programming languages between 2008 and 2018.
- 24. 22 Backend Node.js + Front End Vanilla js, Frameworks
- 25. 23 • Wait .. thats a bad idea Tenor Bad Idea
- 26. 24 Can I really train a neural network in Javascript? Orienta Boy thinking
- 27. 25 Yes!! Tensorflow.js - a library for building/training ML models in Javascript.
- 28. 26 Will it meet my speed requirements?
- 29. 27 Yes!! Tensorflow.js, is accelerated in the browser using WebGL (CPU + GPU) and in Node.js using Tensorflow C API (CPU + GPU).
- 30. 28 How much effort is this? Can I re-use my existing models/pipelines?
- 31. 29 Not much! - Tensorflowjs API is Similar to Keras - Tensorflow.js converter allows you import pretrained Keras (hd5, savedModel), Tensorflow SavedModel, Tensorflow Hub Module
- 32. 30 Oprah “You get a car”
- 33. 31 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity
- 34. 32 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • Strong Notion of Privacy We do not see your data, it never gets to our servers. • Example use case: sensitive documents identification, domain specific text autocomplete for chat systems etc
- 35. 33 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • No installs, drivers and dependency issues. • Navigate to a URL and your application is available to end users. • Free model asset hosting/versioning/distribution (NPM/jsdelvr), excellent for end user applications.
- 36. 34 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • ML models can be optimized for speed and can run fast on mobile and in the browser • In some cases, it is faster to run your model in the browser, rather than send make round trip requests to a server. • Avoid reliability issues with intermittent connectivity environments
- 37. 35 Why Machine Learning in the Browser? • Privacy • Distribution • Latency • Interactivity • The Browser is designed for interactive experiences. ML can supercharge this! • Build models on the fly using rich user data available in the browser (camera, file uploads), run inference, retrain existing models, enable dynamic behaviour. • ML education, retail, advertising, arts, entertainment, gaming.
- 39. 37 Example Use Cases Client-side indecent content checking nsfw.js.com
- 40. 38 Handtrack.js
- 41. 39
- 42. 40 Tensorflow.js A library for machine learning in JavaScript
- 43. 41 What can you do with Tensorflow.js • Author models • Import pre-trained models for inference • Re-train imported models Online Flow • Design/Train/Test/ Inference, all in your javascript code.
- 44. 42 What can you do with Tensorflow.js • Author models • Import pre-trained models for inference • Re-train imported models Offline Flow • Design/Train/Test model on your preferred hardware (GPU, TPU clusters) • Import into javascript for inference
- 45. 43 What can you do with Tensorflow.js • Author models • Import pre-trained models for inference • Re-train imported models Hybrid Flow • Design/Train/Test model on your preferred hardware (GPU, TPU clusters) • Import into javascript application • Retrain/Run inference
- 46. 44 Tensorflow.js - Backend and Frontend Browser (Front end) Server (Back end, Node.js)
- 47. 45 Tensorflow.js - Backend and Frontend Browser (Front end) • Usage • Vanilla js script tag import (like jquery) • Import - NPM install ( build tools, frameworks such as React, Vue, etc). • Acceleration • WebGL • Tensors implemented as Shader programs (will use GPUs where available)
- 48. 46 Tensorflow.js - Backend and Frontend Server (Back end, Node.js) • Usage • NPM install • (tfjs-node, tfjs-node-gpu) • Acceleration • Tensorflow C binary
- 49. WebGL Browser TF CPUTF GPU Node.js Runtime API Ops API (Eager) Layers API TF TPU Keras Model TensorFlow SavedModel
- 50. 48 Ops API (Eager) Layers API Fit data to a polynomial function Build and train a two layer autoencoder in the browser Import a trained model for inference in the browser (Handtrack.js) API
- 51. 49 Tensorflow.js OPS API Use with caution
- 52. 50 The Ops API - Fit a Polynomial f(x) = ax2 + bx + c
- 53. 51 import * as tf from '@tensorflow/tfjs';
- 54. 52 import * as tf from '@tensorflow/tfjs'; const a = tf.tensor(0.1).variable(); const b = tf.tensor(0.1).variable(); const c = tf.tensor(0.1).variable();
- 55. 53 import * as tf from '@tensorflow/tfjs'; const a = tf.tensor(0.1).variable(); const b = tf.tensor(0.1).variable(); const c = tf.tensor(0.1).variable(); // f(x) = a*x^2 + b^x + c const f = x => a.mul(x.square()).add(b.mul(x)).add(c); // ...
- 56. 54 // ... // Mean-squared error const loss = (preds, label) => preds.sub(label).square().mean();
- 57. 55 // ... // Mean-squared error const loss = (preds, label) => preds.sub(label).square().mean(); const optimizer = tf.train.sgd(learningRate);
- 58. 56 // ... // Mean-squared error const loss = (preds, label) => preds.sub(label).square().mean(); const optimizer = tf.train.sgd(learningRate); for (let i = 0; i < EPOCHS; i++) { optimizer.minimize(() => loss(f(data.xs), data.ys)); }
- 59. 57 Tensorflow.js Layers API Very much like keras
- 60. 58 The Layers API • Recommended API for neural networks • Similar in spirit to the Keras Library. Keras is a well designed api standard for composing and training neural networks.
- 61. 59 An Autoencoder Dimensionality reduction • Neural Network with two parts. • Given some high dimension data (e.g. tabular data with many fields), find a smaller representation. • This representation should support reconstruction.15 7 2 15 7
- 62. 60 An Autoencoder Encoder • Take a high dimensional input. • Learn to compress it to a meaningful smaller representation (bottleneck) 15 7 2 15 7
- 63. 61 An Autoencoder Decoder • Takes a bottleneck vector • Learn to reconstruct original input from z. 15 7 2 15 7
- 64. 62 An Autoencoder for Detecting Anomalies Anomaly Detection • Train on Normal Data • Learns a bottleneck vector suitable for reconstructing normal signals • High reconstruction error (MSE) for abnormal images. Flag high error as anomalies. 15 7 2 15 7
- 65. 63 An Autoencoder for Detecting Anomalies Example, ECG data 15 7 2 15 7 0.8 0.2 Error
- 66. 64 # Keras, Python # Encoder inputs = Input(shape=(num_feature s,)) # define hidden layers // Tensorflow JS // Encoder const input = tf.input({ shape: [numFeatures] }) # define hidden layers
- 67. 65 # Keras, Python # define hidden layers enc_hidden = Dense(15, activation='relu')(inpu ts) // Tensorflow JS # define hidden layers let encHidden = tf.layers.dense({ units: 15, activation: "relu" }).apply(input);
- 68. 66 # Keras, Python # define hidden layers enc_hidden = Dense(7, activation='relu') (enc_hidden) // Tensorflow JS # define hidden layers let encHidden = encHidden = tf.layers.dense({ units: 7, activation: "relu" }).apply(encHidden);
- 69. 67 # Keras, Python # Encoder z_ = Dense(latent_dim)(enc_hidden) encoder = Model(inputs, z_, name='encoder') // Tensorflow JS // Encoder const z_ = tf.layers.dense({ units: latentDim }).apply(encHidden); const encoder = tf.model({ inputs: input, outputs: z_, name:"encoder" })
- 71. 69 Train - Data
- 72. 70 Test - Data
- 73. 71 Test - Data
- 74. 72 Interactive Visualizations of Predictions
- 75. 73 Interactive Visualizations of Bottleneck Predictions
- 76. 74 Interactive Visualizations of Bottleneck Predictions 2600 data points
- 77. 75 Interactive Visualizations of Bottleneck Predictions
- 78. 76 Building Handtrack.js, https://github.com/victordibia/handtrack.js/ A library for prototyping hand gestures in the browser
- 79. 77 Handtrack.js • Given an image, predict the location (bounding box) of all hands. • Uses a Convolutional Neural Network (Object Detection) • Bundled into a JS library to enable developers prototype interactive. • Open source.
- 80. 78 // Load the model. handTrack.load().then(model => { // detect objects in the image. model.detect(img).then(predictions => { console.log('Predictions: ', predictions); }); }); <script src="https://cdn.jsdelivr.net/npm/handtrackjs/dist/handtrack.min.js"> </script>
- 81. 79 [{ bbox: [x, y, width, height], class: "hand", score: 0.8380282521247864 }, { bbox: [x, y, width, height], class: "hand", score: 0.74644153267145157 }]
- 82. 80 How was this built?
- 83. 81 How was this built?
- 84. 82 Data Assembly (Egohands Dataset)
- 85. 83 Model Training • Convert images to Tensorflow Records format – For each image, list of bbox coordinates, and labels. • Train using Tensorflow Object Detection API (python) – Framework and sample code for training various object detection models. – Mobilenet, SSD (fast, optimized for mobile). • Export Trained Model as savedModel from checkpoints. •
- 86. 84 Model Conversion • Use tensorflow.js converter • Identify model output node variable name • Specify model format • Specify input/output directories • Supports • savedModel (default for Tensorflow) • Keras Model • Tensorflow Hub $ pip install tensorflowjs $ tensorflowjs_converter --input_format tf_saved_model --output_node_names 'Predictions/Reshape_1' path/to/tf_saved_model path/to/tfjs_model Exported model can be imported in javascript.
- 87. 85 Exported Model • 18.5mb .. final model size • Sharded into 5 files with a maximum size of 4.2mb • • Bundled into an NPM library, hosted on NPM (with benefits!) • Model artifact versioning • CDN serving (jsdelivr) •
- 88. 86
- 89. 87
- 90. 88
- 91. 89 Demo! • So .. does it work live? • https://victordibia.github.io/handtrack.js/#/
- 92. 90 Some Challenges • Manual Memory Management • Tensorflow.js implements tensors using WebGLTextures. • Copied to GPU and cached to enable fast computation • Need to be explicitly deleted using the tensor.dispose(). • Use tf.memory() to see how many tensors your application is holding onto
- 93. 91 Some Challenges • General limitations of the Browser. • The browser is single threaded • Compute heavy operations will block the UI thread.
- 94. 92 Some Challenges • Device Fragmentation • Differences in devices can lead to wildly varying performance • CPU (Mobile, Desktop, Embedded) • GPU availability • Device power mode
- 95. 93 Some Good Practices • Optimize Your Model • Replace full Convolutional Operations with Depthwise separable convolutions (less parameters). • Remove optional post processing from model graph during model export. • Explore model compression methods (quantization, distillation, pruning). • Explore NAS/AutoML. .
- 96. 94 Some Good Practices • Design and Test Considerations • Asynchronous methods (async await, promises) to avoid blocking the UI thread. • Visually communicate model load/inference latency. • Communicate model uncertainty where possible. • Test and benchmark across multiple devices!! • Chrome Profiler is your friend! • See Google PAIR (People + AI Guidebook) document on how to design human-centered AI products.
- 97. 95 Conclusion • ML in the browser is valuable when your objectives are focused on privacy, ease of distribution, low latency and interactivity. • Tensorflow.js provides an API for ML in Javascript (online, offline, hybrid workflow). Fast, expressive, integrated with the rest of the TF ecosystem. • Research advances in ML (compression, distillation, pruning) + Advances in acceleration in the browser (WebAssembly, WebGPU), positions ML in the browser for faster and wider integration.
- 98. 96 Thank you! Research Engineer Cloudera Fast Forward Labs, Brooklyn. @vykthur | @victordibia Source Code: Handtrack.js https://github.com/victordibia/handtrack.js
- 99. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ ml-browser-tensorflow-js/