Derivation of Convolutional Neural Network from Fully Connected Network Step-by-Step
18 likes3,630 views
The document presents a step-by-step derivation of a convolutional neural network from a fully connected network, emphasizing key components such as input images and vector representations. It details the calculations of weights and parameters of the network based on the architecture, illustrating the transition from fully connected layers to convolutional structures. The total number of parameters is calculated, highlighting the network's complexity and the effect of biases in the model.
Derivation of Convolutional Neural Network from Fully Connected Network Step-by-Step
- 1. Derivation of Convolutional Neural Network from Fully Connected Network Step-by-Step Ahmed Fawzy Gad [email protected] MENOUFIA UNIVERSITY FACULTY OF COMPUTERS AND INFORMATION المنوفية جامعة الحاسبات كليةوالمعلومات المنوفية جامعة Ahmed F. Gad 18-May-2018
- 2. 15 8 9 10 17 22 20 3015 Input Image 3x3 Ahmed F. Gad
- 3. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 4. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 5. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 6. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 7. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 8. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 9. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 10. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 11. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 12. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 13. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 Ahmed F. Gad
- 14. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Hidden Layer Ahmed F. Gad
- 15. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Fully Connected Network Ahmed F. Gad
- 16. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Fully Connected Network Ahmed F. Gad
- 17. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝒚 𝒙 𝒙: Pixel Index 𝒚: Hidden Neuron Index Fully Connected Network Ahmed F. Gad
- 18. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝒚 𝒙 𝒙: Pixel Index 𝒚: Hidden Neuron Index 𝒘 𝟎 𝟎 Fully Connected Network Ahmed F. Gad
- 19. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝒚 𝒙 𝒙: Pixel Index 𝒚: Hidden Neuron Index 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 Fully Connected Network Ahmed F. Gad
- 20. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝒚 𝒙 𝒙: Pixel Index 𝒚: Hidden Neuron Index 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 Fully Connected Network Ahmed F. Gad
- 21. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 𝒘 𝟒 𝟎 𝒘 𝟓 𝟎 𝒘 𝟔 𝟎 𝒘 𝟕 𝟎 𝒘 𝟖 𝟎 𝒘 𝟗 𝟎 𝒘 𝟏𝟎 𝟎 𝒘 𝟏𝟏 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 𝒘 𝒚 𝒙 𝒙: Pixel Index 𝒚: Hidden Neuron Index Fully Connected Network Ahmed F. Gad
- 22. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 𝒘 𝟒 𝟎 𝒘 𝟓 𝟎 𝒘 𝟔 𝟎 𝒘 𝟕 𝟎 𝒘 𝟖 𝟎 𝒘 𝟗 𝟎 𝒘 𝟏𝟎 𝟎 𝒘 𝟏𝟏 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 Weight 16 𝒘 𝟎 𝟎 𝒘 𝟏𝟓 𝟎 Total Number of Weights For First Pixel Ahmed F. Gad
- 23. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 24. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟏 𝒘 𝟏 𝟏 𝒘 𝟐 𝟏 𝒘 𝟑 𝟏 𝒘 𝟒 𝟏 𝒘 𝟓 𝟏 𝒘 𝟔 𝟏 𝒘 𝟕 𝟏 𝒘 𝟖 𝟏 𝒘 𝟗 𝟏 𝒘 𝟏𝟎 𝟏 𝒘 𝟏𝟏 𝟏 𝒘 𝟏𝟐 𝟏 𝒘 𝟏𝟑 𝟏 𝒘 𝟏𝟒 𝟏 𝒘 𝟏𝟓 𝟏 Ahmed F. Gad
- 25. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟏 𝒘 𝟏 𝟏 𝒘 𝟐 𝟏 𝒘 𝟑 𝟏 𝒘 𝟒 𝟏 𝒘 𝟓 𝟏 𝒘 𝟔 𝟏 𝒘 𝟕 𝟏 𝒘 𝟖 𝟏 𝒘 𝟗 𝟏 𝒘 𝟏𝟎 𝟏 𝒘 𝟏𝟏 𝟏 𝒘 𝟏𝟐 𝟏 𝒘 𝟏𝟑 𝟏 𝒘 𝟏𝟒 𝟏 𝒘 𝟏𝟓 𝟏 Weight 16 𝒘 𝟎 𝟏 𝒘 𝟏𝟓 𝟏 Total Number of Weights For Second Pixel Ahmed F. Gad
- 26. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟐 𝒘 𝟏 𝟐 𝒘 𝟐 𝟐 𝒘 𝟑 𝟐 𝒘 𝟒 𝟐 𝒘 𝟓 𝟐 𝒘 𝟔 𝟐 𝒘 𝟕 𝟐 𝒘 𝟖 𝟐 𝒘 𝟗 𝟐 𝒘 𝟏𝟎 𝟐 𝒘 𝟏𝟏 𝟐 𝒘 𝟏𝟐 𝟐 𝒘 𝟏𝟑 𝟐 𝒘 𝟏𝟒 𝟐 𝒘 𝟏𝟓 𝟐 Ahmed F. Gad
- 27. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟑 𝒘 𝟏 𝟑 𝒘 𝟐 𝟑 𝒘 𝟑 𝟑 𝒘 𝟒 𝟑 𝒘 𝟓 𝟑 𝒘 𝟔 𝟑 𝒘 𝟕 𝟑 𝒘 𝟖 𝟑 𝒘 𝟗 𝟑 𝒘 𝟏𝟎 𝟑 𝒘 𝟏𝟏 𝟑 𝒘 𝟏𝟐 𝟑 𝒘 𝟏𝟑 𝟑 𝒘 𝟏𝟒 𝟑 𝒘 𝟏𝟓 𝟑 Ahmed F. Gad
- 28. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟒 𝒘 𝟏 𝟒 𝒘 𝟐 𝟒 𝒘 𝟑 𝟒 𝒘 𝟒 𝟒 𝒘 𝟓 𝟒 𝒘 𝟔 𝟒 𝒘 𝟕 𝟒 𝒘 𝟖 𝟒 𝒘 𝟗 𝟒 𝒘 𝟏𝟎 𝟒 𝒘 𝟏𝟏 𝟒 𝒘 𝟏𝟐 𝟒 𝒘 𝟏𝟑 𝟒 𝒘 𝟏𝟒 𝟒 𝒘 𝟏𝟓 𝟒 Ahmed F. Gad
- 29. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟓 𝒘 𝟏 𝟓 𝒘 𝟐 𝟓 𝒘 𝟑 𝟓 𝒘 𝟒 𝟓 𝒘 𝟓 𝟓 𝒘 𝟔 𝟓 𝒘 𝟕 𝟓 𝒘 𝟖 𝟓 𝒘 𝟗 𝟓 𝒘 𝟏𝟎 𝟓 𝒘 𝟏𝟏 𝟓 𝒘 𝟏𝟐 𝟓 𝒘 𝟏𝟑 𝟓 𝒘 𝟏𝟒 𝟓 𝒘 𝟏𝟓 𝟓 Ahmed F. Gad
- 30. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟔 𝒘 𝟏 𝟔 𝒘 𝟐 𝟔 𝒘 𝟑 𝟔 𝒘 𝟒 𝟔 𝒘 𝟓 𝟔 𝒘 𝟔 𝟔 𝒘 𝟕 𝟔 𝒘 𝟖 𝟔 𝒘 𝟗 𝟔 𝒘 𝟏𝟎 𝟔 𝒘 𝟏𝟏 𝟔 𝒘 𝟏𝟐 𝟔 𝒘 𝟏𝟑 𝟔 𝒘 𝟏𝟒 𝟔 𝒘 𝟏𝟓 𝟔 Ahmed F. Gad
- 31. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟕 𝒘 𝟏 𝟕 𝒘 𝟐 𝟕 𝒘 𝟑 𝟕 𝒘 𝟒 𝟕 𝒘 𝟓 𝟕 𝒘 𝟔 𝟕 𝒘 𝟕 𝟕 𝒘 𝟖 𝟕 𝒘 𝟗 𝟕 𝒘 𝟏𝟎 𝟕 𝒘 𝟏𝟏 𝟕 𝒘 𝟏𝟐 𝟕 𝒘 𝟏𝟑 𝟕 𝒘 𝟏𝟒 𝟕 𝒘 𝟏𝟓 𝟕 Ahmed F. Gad
- 32. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟖 𝒘 𝟏 𝟖 𝒘 𝟐 𝟖 𝒘 𝟑 𝟖 𝒘 𝟒 𝟖 𝒘 𝟓 𝟖 𝒘 𝟔 𝟖 𝒘 𝟕 𝟖 𝒘 𝟖 𝟖 𝒘 𝟗 𝟖 𝒘 𝟏𝟎 𝟖 𝒘 𝟏𝟏 𝟖 𝒘 𝟏𝟐 𝟖 𝒘 𝟏𝟑 𝟖 𝒘 𝟏𝟒 𝟖 𝒘 𝟏𝟓 𝟖 Ahmed F. Gad
- 33. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 34. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Total Number of Parameters For the Entire Network 9 Pixels 16 Weight Ahmed F. Gad
- 35. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Total Number of Parameters For the Entire Network 9 Pixels 16 Weight Total Parameters = 9*16=144 Ahmed F. Gad
- 36. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Total Number of Parameters For the Entire Network 9 Pixels 16 Weight Total Parameters = 9*16=144 Bias is Neglected. Ahmed F. Gad
- 37. Hidden Layer 1 90 Neuron Input Layer 9 Neuron Hidden Layer 2 50 Neuron Too Many Parameters Ahmed F. Gad
- 38. 9*90=810 Hidden Layer 1 90 Neuron Input Layer 9 Neuron Hidden Layer 2 50 Neuron Too Many Parameters Ahmed F. Gad
- 39. 9*90=810 Hidden Layer 1 90 Neuron Input Layer 9 Neuron Hidden Layer 2 50 Neuron 90*50=4,500 Too Many Parameters Ahmed F. Gad
- 40. 9*90=810 Hidden Layer 1 90 Neuron Input Layer 9 Neuron Hidden Layer 2 50 Neuron 90*50=4,500+ 810+4,500=5,310= Too Many Parameters Ahmed F. Gad
- 41. Input Image 32x32 Too Many Parameters Ahmed F. Gad
- 42. Hidden Layer 1 500 Neuron Input Layer 1,024 Neuron Input Image 32x32 Too Many Parameters Ahmed F. Gad
- 43. 1,024*500 Hidden Layer 1 500 Neuron Input Layer 1,024 Neuron 512,000= Input Image 32x32 Too Many Parameters Ahmed F. Gad
- 44. 1,024*500 Hidden Layer 1 500 Neuron Input Layer 1,024 Neuron 512,000= Input Image 32x32 CNN can create a large network but with less number of parameters than FC networks Too Many Parameters Ahmed F. Gad
- 45. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 𝒘 𝟒 𝟎 𝒘 𝟓 𝟎 𝒘 𝟔 𝟎 𝒘 𝟕 𝟎 𝒘 𝟖 𝟎 𝒘 𝟗 𝟎 𝒘 𝟏𝟎 𝟎 𝒘 𝟏𝟏 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 Ahmed F. Gad
- 46. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 𝒘 𝟒 𝟎 𝒘 𝟓 𝟎 𝒘 𝟔 𝟎 𝒘 𝟕 𝟎 𝒘 𝟖 𝟎 𝒘 𝟗 𝟎 𝒘 𝟏𝟎 𝟎 𝒘 𝟏𝟏 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 Ahmed F. Gad
- 47. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 𝒘 𝟒 𝟎 𝒘 𝟓 𝟎 𝒘 𝟔 𝟎 𝒘 𝟕 𝟎 𝒘 𝟖 𝟎 𝒘 𝟗 𝟎 𝒘 𝟏𝟎 𝟎 𝒘 𝟏𝟏 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 Ahmed F. Gad
- 48. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟒 𝟎 𝒘 𝟓 𝟎 𝒘 𝟔 𝟎 𝒘 𝟕 𝟎 𝒘 𝟖 𝟎 𝒘 𝟗 𝟎 𝒘 𝟏𝟎 𝟎 𝒘 𝟏𝟏 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 Ahmed F. Gad
- 49. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟖 𝟎 𝒘 𝟗 𝟎 𝒘 𝟏𝟎 𝟎 𝒘 𝟏𝟏 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 Ahmed F. Gad
- 50. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟏𝟐 𝟎 𝒘 𝟏𝟑 𝟎 𝒘 𝟏𝟒 𝟎 𝒘 𝟏𝟓 𝟎 Ahmed F. Gad
- 51. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 Ahmed F. Gad
- 52. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 Ahmed F. Gad
- 53. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 Ahmed F. Gad
- 54. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟎 𝒘 𝟏 𝟎 𝒘 𝟐 𝟎 𝒘 𝟑 𝟎 Number of Parameters for First Pixel After Grouping Neurons 16/4=4 Weights Ahmed F. Gad
- 55. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟏 𝒘 𝟏 𝟏 𝒘 𝟐 𝟏 𝒘 𝟑 𝟏 Ahmed F. Gad
- 56. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟏 𝒘 𝟏 𝟏 𝒘 𝟐 𝟏 𝒘 𝟑 𝟏 Number of Parameters for Second Pixel After Grouping Neurons 16/4=4 Weights Ahmed F. Gad
- 57. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟐 𝒘 𝟏 𝟐 𝒘 𝟐 𝟐 𝒘 𝟑 𝟐 Continue Ahmed F. Gad
- 58. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟑 𝒘 𝟏 𝟑 𝒘 𝟐 𝟑 𝒘 𝟑 𝟑 Continue Ahmed F. Gad
- 59. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟒 𝒘 𝟏 𝟒 𝒘 𝟐 𝟒 𝒘 𝟑 𝟒 Continue Ahmed F. Gad
- 60. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟓 𝒘 𝟏 𝟓 𝒘 𝟐 𝟓 𝒘 𝟑 𝟓 Continue Ahmed F. Gad
- 61. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟔 𝒘 𝟏 𝟔 𝒘 𝟐 𝟔 𝒘 𝟑 𝟔 Continue Ahmed F. Gad
- 62. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟕 𝒘 𝟏 𝟕 𝒘 𝟐 𝟕 𝒘 𝟑 𝟕 Continue Ahmed F. Gad
- 63. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘 𝟎 𝟖 𝒘 𝟏 𝟖 𝒘 𝟐 𝟖 𝒘 𝟑 𝟖 Continue Ahmed F. Gad
- 64. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 All Connections Ahmed F. Gad
- 65. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Total Parameters Before Grouping Neurons 9 Pixels 16 Weight Ahmed F. Gad
- 66. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Total Parameters Before Grouping Hidden Neurons 9 Pixels 16 Weight Total Parameters = 9*16=144 Ahmed F. Gad
- 67. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Total Parameters Before Grouping Neurons 9 Pixels 16 Weight Total Parameters = 9*16=144 Total Parameters After Grouping Neurons 9 Pixels 4 Weight Total Parameters = 9*4=36Ahmed F. Gad
- 68. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Before 144 After 36 Ahmed F. Gad
- 69. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Before 144 After 36 Saved Parameters 144-36=108 Reduction % (108/144)*100 =75% Ahmed F. Gad
- 70. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Before 144 After 36 Saved Parameters 144-36=108 Reduction % (108/144)*100 =75% More Reduction Ahmed F. Gad
- 71. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 72. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 73. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 74. In image analysis, each pixel is highly correlated to pixels surrounding it (i.e. neighbors) Ahmed F. Gad
- 75. In image analysis, each pixel is highly correlated to pixels surrounding it (i.e. neighbors) Ahmed F. Gad
- 79. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Hidden Layer 4 Groups 4 Filters 4 Feature Maps Ahmed F. Gad
- 80. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Hidden Layer 4 Groups 4 Filters 4 Feature Maps Ahmed F. Gad 4 Neurons
- 81. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Hidden Layer 4 Groups 4 Filters 4 Feature Maps Ahmed F. Gad 4 Neurons Each neuron will process an image region of a specific size. In this example, region size is 2x2.
- 82. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 83. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 84. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 85. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 86. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 87. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad 𝒘` 𝒚 𝒙 𝒙: Group Index 𝒚: Index of Group Input
- 88. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 Ahmed F. Gad 𝒘` 𝒚 𝒙 𝒙: Group Index 𝒚: Index of Group Input
- 89. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 Ahmed F. Gad
- 90. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 Ahmed F. Gad
- 91. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 Ahmed F. Gad
- 92. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟏 𝟎 Ahmed F. Gad
- 93. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟏 𝟎 Ahmed F. Gad
- 94. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 𝒘` 𝟏 𝟎 Ahmed F. Gad
- 95. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 96. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 97. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 98. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 99. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 Ahmed F. Gad
- 100. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 101. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 102. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 103. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 104. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 Ahmed F. Gad
- 105. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 106. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 107. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 108. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 Ahmed F. Gad
- 109. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15Ahmed F. Gad
- 110. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Correlated Pixels 4 Weights Parameters for First Hidden Neurons Group Ahmed F. Gad
- 111. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Correlated Pixels Parameters for First Hidden Neurons Group 4*4=16 Weights Parameters for Entire Network 4 Weights Ahmed F. Gad
- 112. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Before 144 After 16 Ahmed F. Gad
- 113. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 Before 144 After 16 Saved Parameters 144-16=128 Reduction % (128/144)*100 =88.89% Ahmed F. Gad
- 114. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 𝒘` 𝟏 𝟎 15*𝒘` 𝟎 𝟎 +8*𝒘` 𝟏 𝟎 +10*𝒘` 𝟐 𝟎 +17𝒘` 𝟑 𝟎 Ahmed F. Gad
- 115. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15 8 9 10 17 22 20 3015 15*𝒘` 𝟎 𝟎 +8*𝒘` 𝟏 𝟎 +10*𝒘` 𝟐 𝟎 +17𝒘` 𝟑 𝟎 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* 𝒘` 𝟎 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 𝒘` 𝟏 𝟎 Ahmed F. Gad
- 116. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15 8 9 10 17 22 20 3015 15*𝒘` 𝟎 𝟎 +8*𝒘` 𝟏 𝟎 +10*𝒘` 𝟐 𝟎 +17𝒘 𝟑 `𝟎 * 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 𝒘` 𝟎 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 𝒘` 𝟏 𝟎 Ahmed F. Gad
- 117. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 15 8 9 10 17 22 20 3015 8*𝒘` 𝟎 𝟎 +9*𝒘` 𝟏 𝟎 +17*𝒘` 𝟐 𝟎 +22𝒘` 𝟑 𝟎 * 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 Ahmed F. Gad
- 118. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 15 8 9 10 17 22 20 3015 10*𝒘` 𝟎 𝟎 +17*𝒘` 𝟏 𝟎 +20*𝒘` 𝟐 𝟎 +15𝒘` 𝟑 𝟎 * 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 Ahmed F. Gad
- 119. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 15 8 9 10 17 22 20 3015 17*𝒘 𝟎 𝟎 +22*𝒘 𝟏 𝟎 +15*𝒘 𝟐 𝟎 +30𝒘 𝟑 𝟎 * 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎 Ahmed F. Gad
- 120. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* Ahmed F. Gad
- 121. 15 8 9 10 17 22 20 3015 15 8 9 10 17 22 20 30 15 Input Image 3x3 Vector 9x1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* 15 8 9 10 17 22 20 3015 𝒘` 𝟎 𝟎 𝒘` 𝟏 𝟎 𝒘` 𝟐 𝟎 𝒘` 𝟑 𝟎* Ahmed F. Gad
- 122. References • Aghdam, Hamed Habibi, and Elnaz Jahani Heravi. Guide to Convolutional Neural Networks: A Practical Application to Traffic- Sign Detection and Classification. Springer, 2017. • Derivation of Convolutional Neural Network from Fully Connected Network Step-By-Step • https://www.linkedin.com/pulse/derivation-convolutional-neural-network- from-fully-connected-gad • https://www.slideshare.net/AhmedGadFCIT/derivation-of-convolutional- neural-network-convnet-from-fully-connected-network • https://www.kdnuggets.com/2018/04/derivation-convolutional-neural- network-fully-connected-step-by-step.html Ahmed F. Gad