AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)

Editor's Notes

• #5: make 5 guesses about the image label
• #8: The advantage of the ReLu over sigmoid is that it trains much faster than the latter because the derivative of sigmoid becomes very small in the saturating region and therefore the updates to the weights almost vanish(Figure 4). This is called vanishing gradient problem. In the network, ReLu layer is put after each and every convolutional and fully-connected layers(FC).
• #9: نکته دیگه اینکه امروزه دیگه مثل قبل از لایه نرمالیزاسیون! استفاده نمیشه بجاش از لایه بچ نورمالیزیشن استفاده میشه علاوه بر اون تعریف یک معماری الان بر اساس قانون خاصی نیست . بیشتر یک هنره . و تست و تجربه . نکته ای که طی این سالها بهش رسیدن اینه که هرچقدر شبکه شما عمیق تر باشه موفقیت شبکه شما هم بیشتره . اینم باید بدونید که هرچقدر عمق شبکه بیشتر باشه مشکلات اموزش اون خیلی خیلی بیشتر میشه . حالا از طرف دیگه اینکه چرا همه اش از لایه های تماما متصل استفاده نمیشه و بخاطرش از لایه کانولوشن استفاده میشه بخاطر این هست که به غیر از بحثهای پردازشی و سربار زیاد این لایه ها اورفیتینگ زیادی رو هم بخاطر تعداد زیاد پارامترها باعث میشن و از طرف دیگه ماهیت 2 بعدی تصاویر در لایه کانولوشن بخوبی قابل استفاده اس. برای همین برای اینکه بشه خصائیص غیر خطی بیشتری رو بدست اورد میان تعداد مختلفی از لایه های کانولوشن رو پشت سر هم قرار میدن و بعدش در لایه اخر از حداقل یه لایه تماما متصل برای دسته بندی استفاده میکنن . این وسط برای ایجاد یک translation invariance و همینطور کاهش ابعاد در توده های موجود در شبکه میان از pooling استفاده میکنن چون پولینگ هر بار کاهش ابعاد رو باعث میشه به همین خاطر اگر اندازه تصویر شما نسبتا کوچیک باشه زیاد و به کرار ازش استفاده نمیکنن بجاش سعی میکنن از طریق خود لایه کانولوشن با اندازه فیلتر بزرگتر و یا گام متفاوت تر کاهش ابعاد رو داشته باشن. پس خط مشی کلی امروزه اینه سعی کنید تعدادی لایه کانولوشن پشت سر هم داشته باشید بعد بسته به اندازه تصویر سعی کنید بین اینها از pooling استفاده کنید (Max نشون داده که بعضا بهتر جواب میده اما شما بسته به کارتون ممکنه از توابع دیگه نتایج بهتری بگیرید) . بعد در لایه های اخر هم از لایه تماما متصل استفاده کنید . البته تکنیک های جدیدی هم اومدن که با استفاده از اونها میتونید باعث بهتر شدن نتیجه بیشید (مثل استفاده از ELU وPrelu بجای Relu یا استفاده از dropout و dropconnect برای مقابله با اورفیتینگ و یا spatial pyramid pooling یا stochastic pooling (که این نمونه اخری رو من تست کردم نتیجه ام خیلی بد شد!) و... در آخر هم باید بگم هرکدوم از این روشها و تکنیک هایی که معرفی میشن باید تست بشن و ممکنه شما یکی از این روشها رو تست کنید و نتیجه اتون بدتر بشه وقتی این اتفاق افتاد یا نیاز به تست بیشتر دارید یا در حالت شما اصلا نیازی به استفاده از اون ویژگی خاص نیست . مثلا من بدون stochastic pooling به دقت 99درصد در ام نیست رسیده بودم که وقتی فعالش کردم دقتم شد 43 درصد! اینکه اندازه تعداد فیچرمپهای شبکه کانولوشن رو هم چقدر بگیریم این هم یه پارامتر هست و قائده خاصی نداره . چیزی که من میتونم بگم اینه که از کم شروع کنید بعد کم کم برید بالا. یه نکته خیلی مهم دیگه هم بخش optimiziation و مسائلی مثل انتخاب نرخ یادگیری و مومنتوم و... هست . ممکنه شما یک مدل /معماری خیلی خوب از لایه ها رو تعریف کنید و بعد فقط بخاطر درست انتخاب نکردن پارامترهای optimization نتایج بدی بگیرید . دوباره بعنوان نمونه من عینا همون معماری که با اون به دقت 99درصد رسیده بودم با تغییر پارامترهای بهینه سازی نتونستم بیشتر از 86درصد برم . کاری که من شخصا میکنم اینه اول سعی میکنم با یه پیکربندی اولیه شروع کنم و انقدر با پارامترهای solver کار کنم تا مطمئن بشم بهترین نتیجه رو گرفتم بعد شروع میکنم تعداد لایه ها و یا خروجی اونها رو تغییر دادن و دوباره این مسیر رو ادامه میدم تا به نتیجه برسم . البته این مربوط به زمانی هست که من بخوام بنا به دلایلی خاص از هیچ مدلی دیگه ای استفاده نکنم . ولی معمولا در 90 درصد اوقات اکثر محققا یک مدل مثل الکس نت یا گوگل نت یا Vggnet رو انتخاب میکنن و بعد شروع میکنن با تغییر دادن تو اوون از اون در کار مورد نظر خودشون استفاده کردن . نکته دیگه ای هم در مورد انتخاب مدل حائز اهمیت هست مشخصات سخت افزاری مورد نیاز اونها هست که باید بهش دقت کنید. در پناه حق موفق و سربلند باشید
• #10: INPUT => [CONV => RELU => POOL] * 2 => [CONV => RELU] * 3 => POOL => [FC => RELU => DO] * 2 => SOFTMAX There are two methods to reduce the size of an input volume – CONV layers with a stride >1 (which we’ve already seen) and POOL layers. It is common to insert POOL layers in-between consecutive
• #11: It contains 5 convolutional layers and 3 fully connected layers. Relu is applied after very convolutional and fully connected layer. Dropout is applied before the first and the second fully connected year. The image size in the following architecutre chart should be 227 * 227 instead of 224 * 224, as it is pointed out by Andrei Karpathy in his famous CS231n Course. More insterestingly, the input size is 224 * 224 with 2 padding in the pytorch torch vision. The output width and height should be (224–11+4)/4 + 1=55.25! The explanation here is pytorch Conv2d apply floor operator to the above result, and therefore the last one padding is ignored. It is worth noting that the only difference between FC and CONV layers is that the neurons in the CONV layer are connected only to a local region in the input, and that many of the neurons in a CONV volume share parameters.
• #12: Actvity of a neuron computed by applying kernel i at positon (x,y) and then applying the ReLU nonlinearity after each nomramalition What exactly is Local Response Normalization? Local Response Normalization (LRN) layer implements the lateral inhibition we were talking about in the previous section. This layer is useful when we are dealing with ReLU neurons. Why is that? Because ReLU neurons have unbounded activations and we need LRN to normalize that. We want to detect high frequency features with a large response. If we normalize around the local neighborhood of the excited neuron, it becomes even more sensitive as compared to its neighbors. At the same time, it will dampen the responses that are uniformly large in any given local neighborhood. If all the values are large, then normalizing those values will diminish all of them. So basically we want to encourage some kind of inhibition and boost the neurons with relatively larger activations. This has been discussed nicely in Section 3.3 of the original paper by Krizhevsky et al. How is it done in practice? There are two types of normalizations available in Caffe. You can either normalize within the same channel or you can normalize across channels. Both these methods tend to amplify the excited neuron while dampening the surrounding neurons. When you are normalizing within the same channel, it’s just like considering a 2D neighborhood of dimension N x N, where N is the size of the normalization window. You normalize this window using the values in this neighborhood. If you are normalizing across channels, you will consider a neighborhood along the third dimension but at a single location. You need to consider an area of shape N x 1 x 1. Here 1 x 1 refers to a single value in a 2D matrix and N refers to the normalization size. normalization layer carries out channel-wise normalization. =================================== ReLUs have the desirable property that they do not require input normalization to prevent them from saturating. If at least some training examples produce a positive input to a ReLU, learning will happen in that neuron. However, we still find that the following local normalization scheme aids generalization. Denoting by ai x,y the activity of a neuron computed by applying kernel i at position (x, y) and then applying the ReLU nonlinearity, the response-normalized activity bi x,y is given by the expression where the sum runs over n “adjacent” kernel maps at the same spatial position, and N is the total number of kernels in the layer. The ordering of the kernel maps is of course arbitrary and determined before training begins. This sort of response normalization implements a form of lateral inhibition inspired by the type found in real neurons, creating competition for big activities amongst neuron outputs computed using different kernels. The constants k, n,α, and β are hyper-parameters whose values are determined using a validation set; we used k = 2, n = 5, α = 10−4, and β = 0.75. We applied this normalization after applying the ReLU nonlinearity in certain layers (see Section 3.5). This scheme bears some resemblance to the local contrast normalization scheme of Jarrett et al. [11], but ours would be more correctly termed “brightness normalization”, since we do not subtract the mean activity. Response normalization reduces our top-1 and top-5 error rates by 1.4% and 1.2%, respectively. We also verified the effectiveness of this scheme on the CIFAR-10 dataset: a four-layer CNN achieved a 13% test error rate without normalization and 11% with normalization3. 3.4
• #13: d averaging the predictions made by the network’s softmax layer on the ten patches. The with magnitudes proportional to the corresponding eigenvalues times a random variable drawn from a Gaussian with mean zero and standard deviation 0.1 where pi and λi are ith eigenvector and eigenvalue of the 3 × 3 covariance matrix of RGB pixel values,
• #14: Why does DropOut work? The idea behind the dropout is similar to the model ensembles. Due to the dropout layer, different sets of neurons which are switched off, represent a different architecture and all these different architectures are trained in parallel with weight given to each subset and the summation of weights being one. For n neurons attached to DropOut, the number of subset architectures formed is 2^n. So it amounts to prediction being averaged over these ensembles of models. This provides a structured model regularization which helps in avoiding the over-fitting. Another view of DropOut being helpful is that since neurons are randomly chosen, they tend to avoid developing co-adaptations among themselves thereby enabling them to develop meaningful features, independent of others. . Dropout is applied before the first and the second fully connected year.
• #15: We trained our models using stochastic gradient descent Gradient of Loss Accelearteing the early stage of neuron
• #16: were “pre-trained” to classify the entire ImageNet 2011 Fall Our top-1 and top-5 error rates release. See
• #18: In the left panel of Figure 4 we qualitatively assess what the network has learned by computing its top-5 predictions on eight test images. Notice that even off-center objects, such as the mite in the top-left, can be recognized by the net. Most of the top-5 labels appear reasonable. For example, only other types of cat are considered plausible labels for the leopard. In some cases (grille, cherry) there is genuine ambiguity about the intended focus of the photograph. Another way to probe the network’s visual knowledge is to consider the feature activations induced by an image at the last, 4096-dimensional hidden layer. If two images produce feature activation vectors with a small Euclidean separation, we can say that the higher levels of the neural network consider them to be similar. Figure 4 shows five images from the test set and the six images from the training set that are most similar to each of them according to this measure. Notice that at the pixel level, the retrieved training images are generally not close in L2 to the query images in the first column. For example, the retrieved dogs and elephants appear in a variety of poses. We present the results for many more test images in the supplementary material. Computing similarity by using Euclidean distance between two 4096-dimensional, real-valued vec- tors is inefficient, but it could be made efficient by training an auto-encoder to compress these vectors to short binary codes. This should produce a much better image retrieval method than applying auto- encoders to the raw pixels [14], which does not make use of image labels and hence has a tendency to retrieve images with similar patterns of edges, whether or not they are semantically similar. 7
• #19: Example filters learned by Krizhevsky et al. Each of the 96 filters shown here is of size [11x11x3], and each one is shared by the 55*55 neurons in one depth slice. Notice that the parameter sharing assumption is relatively reasonable: If detecting a horizontal edge is important at some location in the image, it should intuitively be useful at some other location as well due to the translationally-invariant structure of images. There is therefore no need to relearn to detect a horizontal edge at every one of the 55*55 distinct locations in the Conv layer output volume.