Backpropagation And Gradient Descent In Neural Networks | Neural Network Tutorial | Simplilearn
4 likes2,526 views
The document explains the process of training a simple neural network to recognize handwritten alphabets 'a', 'b', and 'c' through a series of inputs and initial predictions, where the network's weights are randomly assigned. It details how backpropagation and gradient descent are utilized to minimize prediction error through the adjustment of weights based on calculated losses from actual versus predicted outputs. The aim is to achieve accurate predictions after multiple training iterations.
![Loss function
The loss function is a measurement of error which defines the
precision lost on comparing the predicted output to the actual output
loss = [(actual output) – (predicted output)]2](https://image.slidesharecdn.com/backpropagationgradientdescentppt-190606071046/85/Backpropagation-And-Gradient-Descent-In-Neural-Networks-Neural-Network-Tutorial-Simplilearn-28-320.jpg)
Backpropagation And Gradient Descent In Neural Networks | Neural Network Tutorial | Simplilearn
- 2. y 1 y2 y3 x 1 x2 xn This simple neural network must be trained to recognize handwritten alphabets ‘a’, ‘b’ and ‘c’ a b c Neural Network
- 3. The handwritten alphabets are present as images of 28*28 pixels y 1 y2 y3 x 1 x2 xn a b c Neural Network 28 28
- 4. The 784 pixels are fed as input to the first layer of our neural network y 1 y2 y3 x 1 x2 xn a b c 28*28=784 Neural Network 784 neurons 28 28
- 5. The initial prediction is made using the random weights assigned to each channel 0.3 0.5 0.2 x 1 x2 xn a b c 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.3 28 28*28=784 Neural Network 28 28
- 6. Our network predicts the input to be ‘b’ with a probability of 0.5 0.3 0.5 0.2 x 1 x2 xn a b c 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.328*28=784 Neural Network 28
- 7. The predicted probabilities are compared against the actual probabilities and the error is calculated 0.3 0.5 0.2 x 1 x2 xn a b c 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.328*28=784 0.9 0.0 0.0 actual probabilities +0.6 -0.5 -0.2 error = actual - prediction Neural Network 28
- 8. The magnitude indicates the amount of change while the sign indicates an increase or decrease in the weights 0.3 0.5 0.2 x 1 x2 xn a b c 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.328*28=784 actual probabilities error = actual - prediction Neural Network 0.0 0.0 -0.5 -0.2 28 0.9 +0.6
- 9. The information is transmitted back through the network 0.3 0.5 0.2 x 1 x2 xn a b c 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.328*28=784 actual probabilities error = actual - prediction Neural Network 0.0 0.0 -0.5 -0.2 28 0.9 +0.6
- 10. Weights through out the network are adjusted in order to reduce the loss in prediction 0.6 0.2 0.0 x 1 x2 xn a b c 28*28=784 actual probabilities +0.3 -0.2 0.0 error = actual - prediction 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.3 Neural Network 0.0 0.0 28 0.9
- 11. a b c 28 28 28*28=784 actual probabilities error = actual - prediction Neural Network +0.2 -0.1 0.0 In this manner, we keep training the network with multiple inputs until it is able to predict with high accuracy 0.7 0. 1 0.0 x 1 x2 xn 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.3 0.0 0.0 0.9
- 12. a b c 28 28 28*28=784 actual probabilities error = actual - prediction Neural Network 0.0 0.0 0.0 In this manner, we keep training the network with multiple inputs until it is able to predict with high accuracy 0.9 0. 1 0.0 x 1 x2 xn 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.3 0.0 0.0 0.9
- 13. 28 28 1.0 0.0 1. 0 0.0 0.0 Neural Network 0.0 x 1 x2 xn a b c 28*28=784 actual probabilities 0.0 0.0 0.0 error = actual - prediction 0.4 0. 8 0. 20. 3 0.2 1 0.2 6 0.6 1.0 0.9 1.3 Similarly, our network is trained with the images for ‘b’ and ‘c’ too
- 14. Here’s a straightforward dataset. Let’s build a neural network to predict the outputs, given the inputs Input Output 0 1 2 3 4 0 6 1 2 1 8 2 4 Example
- 15. Input Output Neural Network x y This box represents our neural network x*w Example
- 16. Input Output Neural Network x y ‘w’ is the weight x*w Example
- 17. Input Output Neural Network x y The network starts training itself by choosing a random value for w x*w Example
- 18. Input Output Neural Network x y x*wW=3 Example
- 19. Example Input Output Neural Network x y x*wW=3
- 20. Input Output Neural Network x y x*wW=3 Example
- 21. Input Output Neural Network x y x*wW=6 Our second model has w=6 Example
- 22. Example Input Output Neural Network x y x*wW=6 Our second model has w=6
- 23. Input Output Neural Network x y x*wW=6 Example
- 24. Input Output Neural Network x y x*wW=9 And finally, our third model has w=9 Example
- 25. Example Input Output Neural Network x y x*wW=9 And finally, our third model has w=9
- 26. Input Output Neural Network x y x*wW=9 Example
- 27. Example Input Output Neural Network x y x*wW=9 We, as humans, can know just by a look at the data that our weight should be 6. But how does the machine come to this conclusion?
- 28. Loss function The loss function is a measurement of error which defines the precision lost on comparing the predicted output to the actual output loss = [(actual output) – (predicted output)]2
- 29. Loss function Let’s apply the loss function to input value “2” Input Actual Output W=3 W=6 W=9 2 12 6 12 18 ---loss (12-6)2 = 36 (12-12)2 = 0 (12-18)2 = 36Los s ---
- 30. Loss function We now plot a graph for weight versus loss.
- 31. Loss function This graphical method of finding the minimum of a function is called gradient descent
- 32. Gradient descent
- 33. A random point on this curve is chosen and the slope at this point is calculated Gradient descent
- 34. A random point on this curve is chosen and the slope at this point is calculated A positive slope indicates an increase in weight Gradient descent
- 35. This time the slope is negative. Hence, another random point towards its left is chosen A positive slope indicates an increase in weight A negative slope indicates a decrease in weight Gradient descent
- 36. Gradient descent loss This time the slope is negative. Hence, another random point towards its left is chosen A positive slope indicates an increase in weight A negative slope indicates a decrease in weight We continue checking slopes at various points in this manner
- 37. Our aim is to reach a point where the slope is zero A positive slope indicates an increase in weight A negative slope indicates a decrease in weight A zero slope indicates the appropriate weight Gradient descent
- 38. Our aim is to reach a point where the slope is zero A positive slope indicates an increase in weight A negative slope indicates a decrease in weight A zero slope indicates the appropriate weight Gradient descent
- 39. Backpropagation Backpropagation is the process of updating the weights of the network in order to reduce the error in prediction
- 40. Backpropagation The magnitude of loss at any point on our graph, combined with the slope is fed back to the network backpropagation
- 41. Backpropagation A random point on the graph gives a loss value of 36 with a positive slope backpropagation
- 42. Backpropagation A random point on the graph gives a loss value of 36 with a positive slope We continue checking slopes at various points in this manner A random point on the graph gives a loss value of 36 with a positive slope 36 is quite a large number. This means our current weight needs to change by a large number A positive slope indicates that the change in weight must be positive
- 43. Backpropagation A random point on the graph gives a loss value of 36 with a positive slope We continue checking slopes at various points in this manner Similarly, another random point on the graph gives a loss value of 10 with a negative slope 10 is a small number. Hence, the weight requires to be tuned quite less A negative slope indicates that the weight needs to be reduced rather than increased
- 44. Backpropagation After multiple iterations of backpropagation, our weights are assigned the appropriate value Input Output x y x*6
- 45. Backpropagation After multiple iterations of backpropagation, our weights are assigned the appropriate value Input Output x y x*6 At this point, our network is trained and can be used to make predictions
- 46. Backpropagation After multiple iterations of backpropagation, our weights are assigned the appropriate value Input Output x y x*6 Let’s now get back to our first example and see where backpropagation and gradient descent fall into place
- 47. As mentioned earlier, our predicted output is compared against the actual output 0.3 0.5 0.2 x 1 x2 xn a b c 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.3 28 28 28*28=784 1. 0 0.0 0.0 actual probabilities +0.7 -0.5 -0.2 error = actual - prediction Neural Network
- 48. As mentioned earlier, our predicted output is compared against the actual output 0.3 0.5 0.2 x 1 x2 xn a b c 0.2 0. 8 1. 20. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.3 28 28 28*28=784 1. 0 0.0 0.0 actual probabilities error = actual - prediction Neural Network +0.7 -0.5 -0.2 loss(a) 0.72 = 0.49 loss(b) 0.52 = 0.25 loss(c) 0.22 = 0.04 1st iteration
- 49. Weights through out the network are adjusted in order to reduce the loss in prediction 0.6 0.2 0.1 x 1 x2 xn a b c 28 28 28*28=784 actual probabilities +0.4 -0.2 -0.1 error = actual - prediction 0.2 0. 8 1. 30. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.7 Neural Network 1. 0 0.0 0.0
- 50. Weights through out the network are adjusted in order to reduce the loss in prediction 0.6 0.2 0.1 x 1 x2 xn a b c 28 28 28*28=784 actual probabilities error = actual - prediction 0.2 0. 8 1. 30. 3 0. 2 0.3 6 0.3 6 1.4 0.9 0.7 Neural Network 1. 0 0.0 0.0 loss(a) 0.72 = 0.49 loss(b) 0.52 = 0.25 loss(c) 0.22 = 0.04 1st iteration 2nd iteration loss(a) 0.42 = 0.16 loss(b) 0.22 = 0.04 loss(c) 0.12 = 0.01 +0.4 -0.2 -0.1
- 51. Weights through out the network are adjusted in order to reduce the loss in prediction 0.8 0.1 0.1 x 1 x2 xn a b c 28 28 28*28=784 actual probabilities +0.2 -0.1 -0.1 error = actual - prediction 0.2 0. 2 1. 20. 3 1. 2 0.6 0.3 6 0.4 0.9 0.3 Neural Network 1. 0 0.0 0.0
- 52. Weights through out the network are adjusted in order to reduce the loss in prediction 0.8 0.1 0.1 x 1 x2 xn a b c 28 28 28*28=784 actual probabilities error = actual - prediction 0.2 0. 2 1. 20. 3 1. 2 0.6 0.3 6 0.4 0.9 0.3 Neural Network 1. 0 0.0 0.0 loss(a) 0.72 = 0.49 loss(b) 0.52 = 0.25 loss(c) 0.22 = 0.04 1st iteration 2nd iteration loss(a) 0.42 = 0.16 loss(b) 0.22 = 0.04 loss(c) 0.12 = 0.01 +0.2 -0.1 -0.1 3rd iteration loss(a) 0.22 = 0.04 loss(b) 0.12 = 0.01 loss(c) 0.12 = 0.01
- 53. Weights through out the network are adjusted in order to reduce the loss in prediction 0.8 0.1 0.1 x 1 x2 xn a b c 28 28 28*28=784 actual probabilities error = actual - prediction 0.2 0. 2 1. 20. 3 1. 2 0.6 0.3 6 0.4 0.9 0.3 Neural Network 1. 0 0.0 0.0 loss(a) 0.72 = 0.49 loss(b) 0.52 = 0.25 loss(c) 0.22 = 0.04 1st iteration 2nd iteration loss(a) 0.42 = 0.16 loss(b) 0.22 = 0.04 loss(c) 0.12 = 0.01 +0.2 -0.1 -0.1 3rd iteration loss(a) 0.22 = 0.04 loss(b) 0.12 = 0.01 loss(c) 0.12 = 0.01 Let’s focus on finding the minimum loss for our variable ‘a’
- 54. Weights through out the network are adjusted in order to reduce the loss in prediction 0.8 0.1 0.1 x 1 x2 xn a b c 28 28 28*28=784 actual probabilities error = actual - prediction 0.2 0. 2 1. 20. 3 1. 2 0.6 0.3 6 0.4 0.9 0.3 Neural Network 1. 0 0.0 0.0 loss(a) 0.72 = 0.49 loss(b) 0.52 = 0.25 loss(c) 0.22 = 0.04 1st iteration 2nd iteration loss(a) 0.42 = 0.16 loss(b) 0.22 = 0.04 loss(c) 0.12 = 0.01 +0.2 -0.1 -0.1 3rd iteration loss(a) 0.22 = 0.04 loss(b) 0.12 = 0.01 loss(c) 0.12 = 0.01 Let’s focus on finding the minimum loss for our variable ‘a’
- 55. Weights through out the network are adjusted in order to reduce the loss in prediction 0.8 0.1 0.1 x 1 x2 xn a b c 28 28 28*28=784 actual probabilities error = actual - prediction 0.2 0. 2 1. 20. 3 1. 2 0.6 0.3 6 0.4 0.9 0.3 Neural Network 1. 0 0.0 0.0 loss(a) 0.72 = 0.49 loss(b) 0.52 = 0.25 loss(c) 0.22 = 0.04 1st iteration 2nd iteration loss(a) 0.42 = 0.16 loss(b) 0.22 = 0.04 loss(c) 0.12 = 0.01 +0.2 -0.1 -0.1 3rd iteration loss(a) 0.22 = 0.04 loss(b) 0.12 = 0.01 loss(c) 0.12 = 0.01 And here is where gradient descent comes into the picture
- 56. Neural Network weight loss 0.5 1 0.1 7 0.34 w1 w2 w3 Let’s assume the below to be our graph for the loss of prediction with variable a as compared to the weights contributing to it from the second last layer
- 57. Neural Network weight loss 0.5 1 0.1 7 0.34 w1 w2 w3 Random points chosen on the graph is now backpropagated through the network in order to adjust the weights 0.8 0.1 0.1 x 1 x2 xn a b c 0.4 0. 8 0. 20. 3 0.2 1 0.2 6 0.6 1.0 0.9 1.3
- 58. Neural Network weight loss 0.5 1 0.1 7 0.34 w1 w2 w3 The network is run once again with the new weights 1.0 0.1 0.1 x 1 x2 xn a b c 0.4 0. 8 0. 20. 3 0.2 1 0.2 6 0.6 1.0 0.9 1.3
- 59. Neural Network weight loss 0.5 1 0.1 7 0.34 w1 w2 w3 This process is repeated multiple times till it provides accurate predictions 1.0 0.1 0.1 x 1 x2 xn a b c 0.4 0. 8 0. 20. 3 0.2 1 0.2 6 0.6 1.0 0.9 1.3
- 60. Neural Network weight loss 0.5 1 0.1 7 0.34 w1 w2 w3 This process is repeated multiple times till it provides accurate predictions 1.0 0.0 0.0 x 1 x2 xn a b c 0.4 0. 8 0. 20. 3 0.2 1 0.2 6 0.6 1.0 0.9 1.3
- 61. Neural Network weight loss 0.5 1 0.1 7 0.34 w1 w2 w3 The weights are further adjust to identify ‘b’ and ‘c’ too 0.0 1.0 0.0 x 1 x2 xn a b c 0.4 0. 8 0. 20. 3 0.2 1 0.2 6 0.6 1.0 0.9 1.3
- 62. Neural Network 0.0 1.0 0.0 x 1 x2 xn a b c 0.5 0. 8 0. 20. 3 0.2 1 0.2 0.6 1.0 1.9 0.3 The weights are further adjust to identify ‘b’ and ‘c’ too
- 63. Neural Network 0.0 00 1.0 x 1 x2 xn a b c 0.4 0. 3 0. 20. 3 0.2 1 0.2 0.7 0.5 0.9 1.3 The weights are further adjust to identify ‘b’ and ‘c’ too
- 64. Neural Network 0.0 00 1.0 x 1 x2 xn a b c 0.4 0. 3 0. 20. 3 0.2 1 0.7 0.5 0.9 1.3 Thus, through gradient descent and backpropagation, our network is completely trained 0.2
Editor's Notes
- #3: Style - 01
- #4: Style - 01
- #5: Style - 01
- #6: Style - 01
- #7: Style - 01
- #8: Style - 01
- #9: Style - 01
- #10: Style - 01
- #11: Style - 01
- #12: Style - 01
- #13: Style - 01
- #14: Style - 01
- #15: Style - 01
- #16: Style - 01
- #17: Style - 01
- #18: Style - 01
- #19: Style - 01
- #20: Style - 01
- #21: Style - 01
- #22: Style - 01
- #23: Style - 01
- #24: Style - 01
- #25: Style - 01
- #26: Style - 01
- #27: Style - 01
- #28: Style - 01
- #29: Style - 01
- #30: Style - 01
- #31: Style - 01
- #32: Style - 01
- #33: Style - 01
- #34: Style - 01
- #35: Style - 01
- #36: Style - 01
- #37: Style - 01
- #38: Style - 01
- #39: Style - 01
- #40: Style - 01
- #41: Style - 01
- #42: Style - 01
- #43: Style - 01
- #44: Style - 01
- #45: Style - 01
- #46: Style - 01
- #47: Style - 01
- #48: Style - 01
- #49: Style - 01
- #50: Style - 01
- #51: Style - 01
- #52: Style - 01
- #53: Style - 01
- #54: Style - 01
- #55: Style - 01
- #56: Style - 01
- #57: Style - 01
- #58: Style - 01
- #59: Style - 01
- #60: Style - 01
- #61: Style - 01
- #62: Style - 01
- #63: Style - 01
- #64: Style - 01
- #65: Style - 01