Introduction to Deep learning
Download as PPTX, PDF1 like3,472 views
The document provides an introduction to deep learning, covering its history, key concepts like convolutional neural networks, and research focuses in image segmentation. It discusses various applications of deep learning across fields such as computer vision, natural language processing, and robotics, as well as current activities at the University of Leeds. The text emphasizes the evolution of deep learning, addressing challenges and advancements including containerization for high-performance computing.
![Resurgence as Deep learning (Mid 2000-Present)
Vanishing gradient problem : ReLUs [1]
[1] Nair, Vinod, and Geoffrey E. Hinton. "Rectified linear units improve restricted boltzmann machines." Proceedings of the 27th international conference on machine learning (ICML-10). 2010](https://image.slidesharecdn.com/deeplearningintrobyleopauly-170618123413/85/Introduction-to-Deep-learning-10-320.jpg)
![Computer Vision
Object detection [2]
Image classification [1]
Image segmentation [3]
Edge detection [4]
[1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012.
[2] Girshick, Ross. "Fast r-cnn." Proceedings of the IEEE International Conference on Computer Vision. 2015.
[3] Zheng, Shuai, et al. "Conditional random fields as recurrent neural networks." Proceedings of the IEEE International Conference on Computer Vision. 2015.
[4] Xie, Saining, and Zhuowen Tu. "Holistically-nested edge detection." Proceedings of the IEEE International Conference on Computer Vision. 2015.](https://image.slidesharecdn.com/deeplearningintrobyleopauly-170618123413/85/Introduction-to-Deep-learning-40-320.jpg)
![[1]Kafle, Kushal, and Christopher Kanan. "Visual Question Answering: Datasets, Algorithms, and Future Challenges." arXiv preprint arXiv:1610.01465 (2016).
Natural Language processing
Visual question answering [1]](https://image.slidesharecdn.com/deeplearningintrobyleopauly-170618123413/85/Introduction-to-Deep-learning-41-320.jpg)
![[1] Levine, Sergey, et al. "Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection." arXiv preprint arXiv:1603.02199 (2016).
[2] Chen, Chenyi, et al. "Deepdriving: Learning affordance for direct perception in autonomous driving." Proceedings of the IEEE International Conference on Computer
Vision. 2015.
Robotics
Grasping objects[1]
Autonomous driving[2]](https://image.slidesharecdn.com/deeplearningintrobyleopauly-170618123413/85/Introduction-to-Deep-learning-42-320.jpg)
![[1] Esteva, Andre, et al. "Dermatologist-level classification of skin cancer with deep neural networks." Nature 542.7639 (2017): 115-118.
[2] Maninis, Kevis-Kokitsi, et al. "Deep retinal image understanding." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International
Publishing, 2016.
[3] Ramsundar, Bharath, et al. "Massively multitask networks for drug discovery." arXiv preprint arXiv:1502.02072 (2015).
Medicine
Skin cancer classification[1]
Retinal vessel segmentation[2]
Drug Discovery[3]](https://image.slidesharecdn.com/deeplearningintrobyleopauly-170618123413/85/Introduction-to-Deep-learning-43-320.jpg)
Introduction to Deep learning
- 1. Introduction to Deep Learning Leo Pauly 1st year PhD Researcher in Computer vision & Deep Learning University of Leeds
- 2. Contents • What is deep learning ? • Convolutional neural networks : explained • My research : Deep learning based Image segmentation • Applications in different domains • Deep learning with HPC Leeds • Future activities of Deep Learning @ Leeds
- 3. TLU : threshold logic unit 1943: Warren McCulloch and Walter Pitts create a computational model for neural networks based on mathematics and algorithms called threshold logic.
- 4. Perceptron 1958: Frank Rosenblatt creates the perceptron, an algorithm for pattern recognition based on a two-layer computer neural network using simple addition and subtraction Activation function
- 5. Similarity with biological neurons
- 6. A simple neural network
- 7. How neural networks work Optimisation algorithm : Mini batch stochastic gradient descendant algorithm
- 8. AI winter (90’s to early 2000) Computational power Difficulty to train larger network : Vanishing gradient problem Lack of large dataset
- 9. Resurgence as Deep learning (Mid 2000-Present)
- 10. Resurgence as Deep learning (Mid 2000-Present) Vanishing gradient problem : ReLUs [1] [1] Nair, Vinod, and Geoffrey E. Hinton. "Rectified linear units improve restricted boltzmann machines." Proceedings of the 27th international conference on machine learning (ICML-10). 2010
- 11. Resurgence as Deep learning (Mid 2000-Present) Computational powerVanishing gradient problem : ReLUs
- 12. Resurgence as Deep learning (Mid 2000-Present) Computational powerVanishing gradient problem : ReLUs Lack of large dataset
- 13. Resurgence as Deep learning (Mid 2000-Present) Articles to read: - A brief historyof neural nets and deep learning - Welcome to the AI Conspiracy: The 'Canadian Mafia' Yann LeCun, New York University & Facebook Yoshua Bengio, Universite de Montreal Geoffrey Hinton, Google & University of Toronto Jurgen Schmidhuber, Dalle Molle Institutefor ArtificialIntelligence Research
- 14. Deeper neural networks rebranded as deep learning
- 15. Deeper neural networks rebranded as deep learning
- 16. Deeper neural networks rebranded as deep learning
- 18. Amount of data vs performance
- 19. Types of networks used for deep learning • Convolutional neural networks
- 20. Types of networks used for deep learning • Convolutional neural networks • Recurrent neural networks
- 21. Types of networks used for deep learning • Convolutional neural networks • Recurrent neural networks • Long Short term memory (LSTM) networks
- 22. Types of networks used for deep learning • Convolutional neural networks • Recurrent neural networks • Long Short term memory (LSTM) networks • Deep Boltzmann machines
- 23. Types of networks used for deep learning • Convolutional neural networks • Recurrent neural networks • Long Short term memory (LSTM) networks • Deep Boltzmann machines • Deep Q-networks
- 24. Types of networks used for deep learning • Convolutional neural networks • Recurrent neural networks • Long Short term memory (LSTM) networks • Deep Boltzmann machines • Deep Q-networks • Deep belief networks
- 25. Types of networks used for deep learning • Convolutional neural networks • Recurrent neural networks • Long Short term memory (LSTM) networks • Deep Boltzmann machines • Deep Q-networks • Deep belief networks • Deep stacking networks
- 27. 1. Convolutional operation in convolutional layer
- 28. Link: chttps://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/ 3.Pooling operation in pooling layer2. ReLU operating in activation layer
- 30. My research My research interests: Intersection of Computer vision, Machine Learning & Robotics My current research focus for PhD: Image segmentation & Deep learning Theoretical level: - Conditional/ Parametric CNNs for segmentation - Transfer learning for segmentation Application level: - Pavement crack segmentation
- 31. Parametric / Conditional CNN Exploring the possibility of using extra parameter to steer the output of the network in one direction Additional control parameter Input image Output image
- 32. Transfer learning Training machine learning models in one domain and deploying it in another domain: Training: Deploying: Deep learning model Deep learning model
- 33. Applications of Deep learning
- 34. Applications of Deep learning
- 35. Applications of Deep learning
- 36. Applications of Deep learning
- 37. Applications of Deep learning
- 38. Applications of Deep learning
- 39. Applications of Deep learning
- 40. Computer Vision Object detection [2] Image classification [1] Image segmentation [3] Edge detection [4] [1] Krizhevsky, Alex, Ilya Sutskever, and Geoffrey E. Hinton. "Imagenet classification with deep convolutional neural networks." Advances in neural information processing systems. 2012. [2] Girshick, Ross. "Fast r-cnn." Proceedings of the IEEE International Conference on Computer Vision. 2015. [3] Zheng, Shuai, et al. "Conditional random fields as recurrent neural networks." Proceedings of the IEEE International Conference on Computer Vision. 2015. [4] Xie, Saining, and Zhuowen Tu. "Holistically-nested edge detection." Proceedings of the IEEE International Conference on Computer Vision. 2015.
- 41. [1]Kafle, Kushal, and Christopher Kanan. "Visual Question Answering: Datasets, Algorithms, and Future Challenges." arXiv preprint arXiv:1610.01465 (2016). Natural Language processing Visual question answering [1]
- 42. [1] Levine, Sergey, et al. "Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection." arXiv preprint arXiv:1603.02199 (2016). [2] Chen, Chenyi, et al. "Deepdriving: Learning affordance for direct perception in autonomous driving." Proceedings of the IEEE International Conference on Computer Vision. 2015. Robotics Grasping objects[1] Autonomous driving[2]
- 43. [1] Esteva, Andre, et al. "Dermatologist-level classification of skin cancer with deep neural networks." Nature 542.7639 (2017): 115-118. [2] Maninis, Kevis-Kokitsi, et al. "Deep retinal image understanding." International Conference on Medical Image Computing and Computer-Assisted Intervention. Springer International Publishing, 2016. [3] Ramsundar, Bharath, et al. "Massively multitask networks for drug discovery." arXiv preprint arXiv:1502.02072 (2015). Medicine Skin cancer classification[1] Retinal vessel segmentation[2] Drug Discovery[3]
- 44. Several other applications - Agriculture - Game playing systems : AlphaGo - Language-Language translation - Synthetic sound generation - Deep reinforcement learning in robotics - So on……..
- 45. Deep learning using containers at HPC, Leeds What & Why containers ? + + + OpenCv + NLTK+ Numpy + Anaconda = DS1 Container in HPC What it means to user ? - Load container engine : Singularity / Docker - Load container image - Start writing code Courtesy : Martin Callaghan, HPC, University of Leeds
- 46. Deep learning using containers at HPC, Leeds Courtesy : Martin Callaghan, HPC, University of Leeds - Demo…!! - Guide - Course on June 26th : Dockers & Containers - P100s coming up…..!!!!!
- 47. Discussion : Deep learning @ Leeds • Regular meetings : - Start with monthly meetings ( July- Oct) - Followed by biweekly meetings (1 research group meeting + 1 talk by a speaker) • Mailing list • Lightening talks @ Departments • Robotics away day • General discussion…!!
Editor's Notes
- #2: Intoduction;
- #3: Tried to in cooperate everyone's interest
- #4: Pardon me biologists;
- #5: Remember activation functions
- #6: The interactions of neurons is not merely electrical, though, but electro-chemical. Each axon terminal contains thousands of membrane-bound sacs called vesicles, which in turn contain thousands of neurotransmitter molecules each. Neurotransmitters are chemical messengers which relay, amplify and modulate signals between neurons and other cells. The two most common neurotransmitters in the brain are the amino acids glutamate and GABA
- #8: Loss function : need not be a convex fucntion
- #9: Vanishing gradient problem : 1992 with his student; Sepp Hochreiter ; Jurgen Schmidhuber
- #12: ReLU : even a small idea can bring a large change HGX-1 with 8 tesla V-100; DGX-1 with 8 tesla P100 with $129k ; P100s in HPC
- #14: Inspiring story for young researchers : Never give up what you believe in; Yann Lecunn – CVPR story
- #19: How many layers ? Bengio’s answer
- #28: chttps://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/
- #29: Min-max pooling; No rule of thumb
- #30: Computer vision : works perfectly well on images
- #32: Not just segmentation : Image – image translation ; Continuous parameter input & continuously output : rgb-gray ; ask for suggestions
- #33: DRUI : explain with authors, ETHZ
- #43: Autonomous driving : Black box : lack of surety about decision : Bayesian deep learning
- #44: Nature; Deploy in third world countries ; humans are far supperior
- #46: New computer analogy ; Martin thanks ; shan