Machine Learning - Convolutional Neural Network

Editor's Notes

• #2: Convolution Neural Network for Visual Recognition(捲積神經網絡用於視覺識別)
• #7: Max-Pooling 最大池化 Use 6 filters size = 5 x 5 x 3 3072 x 3072 = 9.43m vs 156 x 4704 = 733824 Stride 步長
• #13: 9 + 1 + (-2) + 1 (bias) = 9 Hyper-Parameters: Accepts a volume of size W1×H1×D1 Requires four hyper-parameters: Number of filters K, their spatial extent F, the stride S, the amount of zero padding P. Produces a volume of size W2×H2×D2 where: W2=(W1−F+2P)/S+1 H2=(H1−F+2P)/S+1 (i.e. width and height are computed equally by symmetry) D2=K With parameter sharing, it introduces F⋅F⋅D1 weights per filter, for a total of (F⋅F⋅D1)⋅K weights and K biases. In the output volume, the d-th depth slice (of size W2×H2) is the result of performing a valid convolution of the d-th filter over the input volume with a stride of S, and then offset by d-th bias. A common setting of the hyper-parameters is F=3,S=1,P=1.
• #14: For consistency, function f should be g
• #16: Max-Pooling 最大池化 http://www.ais.uni-bonn.de/papers/icann2010_maxpool.pdf show max-pooling is effective.
• #17: Source cs231n: Example Architecture: Overview: We will go into more details below, but a simple ConvNet for CIFAR-10 classification could have the architecture [INPUT - CONV - RELU - POOL - FC]. In more detail: INPUT [32x32x3] will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B. CONV layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as [32x32x12] if we decided to use 12 filters. Use 6 here. RELU layer will apply an elementwise activation function, such as the max(0,x) thresholding at zero. This leaves the size of the volume unchanged ([32x32x12]). POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as [16x16x12]. FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size [1x1x10], where each of the 10 numbers correspond to a class score, such as among the 10 categories of CIFAR-10. As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume.
• #19: Each Filter Generates One Feature Map
• #21: In particular, pooling makes the input representations (feature dimension) smaller and more manageable reduces the number of parameters and computations in the network, therefore, controlling overfitting [4] makes the network invariant to small transformations, distortions and translations in the input image (a small distortion in input will not change the output of Pooling – since we take the maximum / average value in a local neighborhood). helps us arrive at an almost scale invariant representation of our image (the exact term is “equivariant”). This is very powerful since we can detect objects in an image no matter where they are located (read [18] and [19] for details).
• #22: [INPUT – [CONV – RELU]*2 – POOL]*3 – [FC]*2 - SoftMax
• #23: Alexnet - https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf We trained a large, deep convolutional neural network to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes. On the test data, we achieved top-1 and top-5 error rates of 37.5% and 17.0% which is considerably better than the previous state-of-the-art. The neural network, which has 60 million parameters and 650,000 neurons, consists of five convolutional layers, some of which are followed by max-pooling layers, and three fully-connected layers with a final 1000-way softmax
• #24: Concept: Find a set of filters (function-g, matrix with weights) and parameters which can create proper feature maps, and cause various activation functions to be fired at different (layers) that leads to correct class has highest probability. f*g*a*p*fc -> max-y This should include the option of DROPOUT. Give a image function f, find a filter g, and activation function a, and pooling function p that leads to max y value (associate with f). Use red color glass filter to look a red letter-A written on a white paper, we will see a write letter-A written on a black paper.
• #25: Source cs231n: Example Architecture: Overview: We will go into more details below, but a simple ConvNet for CIFAR-10 classification could have the architecture [INPUT - CONV - RELU - POOL - FC]. In more detail: INPUT [32x32x3] will hold the raw pixel values of the image, in this case an image of width 32, height 32, and with three color channels R,G,B. CONV layer will compute the output of neurons that are connected to local regions in the input, each computing a dot product between their weights and a small region they are connected to in the input volume. This may result in volume such as [32x32x12] if we decided to use 12 filters. RELU layer will apply an elementwise activation function, such as the max(0,x) thresholding at zero. This leaves the size of the volume unchanged ([32x32x12]). POOL layer will perform a downsampling operation along the spatial dimensions (width, height), resulting in volume such as [16x16x12]. FC (i.e. fully-connected) layer will compute the class scores, resulting in volume of size [1x1x10], where each of the 10 numbers correspond to a class score, such as among the 10 categories of CIFAR-10. As with ordinary Neural Networks and as the name implies, each neuron in this layer will be connected to all the numbers in the previous volume.
• #34: Demo: http://cs231n.stanford.edu/
• #35: Max-Pooling 最大池化 Use 6 filters size = 5 x 5 x 3 3072 x 3072 = 9.43m vs 156 x 4704 = 733824 Stride 步長
• #37: []()