Introduction to theano, case study of Word Embeddings
Download as PPTX, PDF3 likes499 views
Theano is a Python library that allows defining, optimizing, and evaluating mathematical expressions involving multi-dimensional arrays efficiently. It can compile expressions into optimized C code for fast CPU and GPU execution. Theano uses symbolic differentiation to automatically compute gradients for neural network training via backpropagation. It represents computations as a graph with variable nodes and operation nodes. This graph can be optimized before generating efficient C code. Theano is useful for machine learning algorithms that require large-scale numeric optimization like neural networks. The document discusses implementing word embedding models in Theano including autoencoder, GloVe, and skip-gram negative sampling models. Code examples are provided in GitHub links.
Introduction to theano, case study of Word Embeddings
- 1. Introduction to theano Case study of word embedding models Shashank Gupta MS, SIEL IIIT-H
- 2. Numpy is fast, why theano? Bound by CPU Bound by Python interpreter (less scope of runtime optimization like lazy evaluation etc.) Lack of symbolic differentiation Finite difference method exists, but prone to numerical errors SymPy supports symbolic differentiation but expression graphs not optimized as theano
- 3. What’s theano? Math expression compiler Compiles a math expression to optimized C code Targeted for GPU (CPU if GPU not supported) Strongly typed as compared to python’s dynamically typed system Works as GPU metaprogrammer (abstraction on top of GPU programming)
- 4. Alternative to theano CUDA (PyCUDA it’s python wrapper) Asynchronous Stochastic Gradient Descent - smart way to parallelize SGD Main idea - Don’t create lock for one update, let threads run in async. Could speed up ML algos more than matrix-vector parallelization
- 5. Automatic Differentiation Crude way - Finite difference method df / dx = (f(x + h) - f(x - h)) / 2*h Prone to coding bugs Prone to numerical error
- 6. Symbolic differentiation Calculate gradient analytically For a given math expression construct symbolic computation graph Ref: http://cs231n.github.io/optimization-2/
- 7. Backprop using symbolic differentiation Once graph is computed forward pass is just flow of input to compute output Backprop is going backwards from last node to compute gradients locally at each node and accumulate those local gradients to compute global gradient (using chain rule) Ex :
- 8. Cont. Each entity in the computation graph is a symbolic variable (a node in computational graph) Each operation is an ‘op’ node (theano specific) ‘Op’ node takes some inputs and produces some output ‘Apply’ node which applies op to inputs ‘Type’ nodes with associated type information of symbolic variable
- 9. Cont. This is kind of abstract representation of math expression Can think as intermediate representation in standard compiler phase This is how theano represents the computational graph internally
- 11. Optimization Once this ‘intermediate representation’ is generated theano optimizes this graph for efficient computation Can think of it as compiler optimization phase It reorders, remove redundant expression etc. to generate an ‘equivalent’ graph which gives same output as input
- 12. Shared variables Symbolic variables with predefined values In Machine learning used to define parameters of the models with pre-defined values (W, b matrices in NN) Sends these values to host GPU with optimised storage
- 13. Theano functions Compiles the ‘abstract’ computation graph to optimised C code Compiles it targeting GPU C code is highly optimized for numeric computation Sort of interface between theano code and it’s calling python code
- 14. Case study : Word embedding models 1st Model : Autoencoder based word embedding Ref : http://arxiv.org/pdf/1412.4930v2.pdf
- 15. Cont. U is the embedding matrix which is learned by optimizing this objective
- 16. Code Refer to github gist : https://gist.github.com/shashankg7/aec2303803e7b39b150a9f78cb59db09 Only theano part included, I/O and preprocessing omitted
- 17. Model 2 - GloVe word embedding Optimizes squared loss function Challenges in practical implementation in theano Main thing to remember - VECTORIZED implementation How to handle large matrices Refer to github gist : https://gist.github.com/shashankg7/aec2303803e7b39b150a9f78cb59db09
- 18. Model 3 : Skip-gram negative sampling Alpha stage: Not able to figure out vectorization of loss function Refer to github gist : https://gist.github.com/shashankg7/aec2303803e7b39b150a9f78cb59db09
- 19. THANKS !!!