Setting Artificial Neural Networks parameters
3 likes4,629 views
The document discusses the essential parameters for setting up artificial neural networks, focusing on aspects such as learning rates, weight initialization, training types (supervised and unsupervised), and momentum in training algorithms. It emphasizes the importance of a well-defined training set for generalizability, outlining methods to avoid overfitting and ensure networks can perform well on unseen data. Various strategies for determining the number of hidden layers and nodes are also presented, along with termination criteria for training processes.
![INITIALISATION OF WEIGHTS
ď PROBLEM ITH THIS CHOISE:
ď If some of input parameters are very high, they
will predominate the output.
ď e. g. x = [ 10 2 0.2 1]
ď SOLUTION:
ď Weights are initialized as inversely proportional
to input.
ď Output will not depend on any individual
parameters, but total input as a whole.](https://image.slidesharecdn.com/nn-parameters-160218114453/85/Setting-Artificial-Neural-Networks-parameters-6-320.jpg)
Setting Artificial Neural Networks parameters
- 1. ARTIFICIAL NEURAL NETWORKS PARAMETERS Setting ARTIFICIAL NEURAL NETWORKS PARAMETERS
- 2. NEED FOR SETTING PARAMETER VALUES 1. LOCAL MINIMA w1 â Global Minima w2, w3 â Local minima W1 W2 W3 1 32 Erms Erms min
- 3. NEED FOR SETTING PARAMETER VALUES 2. LEARNING RATE ď Small learning rate â Slow and lengthy learning ď Large learning rate â ď Output may saturate ď or may swing across desired. ď May take too long to train. 3. Learning will improve and network training will converge if inputs and outputs are statistical i.e. numeric.
- 4. TYPES OF TRAINING Supervised Training â˘Supplies the neural network with inputs and the desired Outputs ⢠Response of the network to the inputs is measured â˘The weights are modified to reduce the difference between the actual and desired outputs Unsupervised Training â˘Only supplies inputs â˘The neural network adjusts its own weights so that similar inputs cause similar outputs â˘The network identifies the patterns and differences in the inputs without any external assistance
- 5. I.INITIALISATION OF WEIGHTS ď Larger weights will drive the output of layer 1 to saturation. ď Network will require larger training time to emerge out of saturation. ď Weights chosen as :- Small weights between -1 and 1 Or between -0.5 and 0.5
- 6. INITIALISATION OF WEIGHTS ď PROBLEM ITH THIS CHOISE: ď If some of input parameters are very high, they will predominate the output. ď e. g. x = [ 10 2 0.2 1] ď SOLUTION: ď Weights are initialized as inversely proportional to input. ď Output will not depend on any individual parameters, but total input as a whole.
- 7. RULE FOR INITIALISATION OF WEIGHTS ď Weight between input and 1st layer: P ď vij = (1/2P) âp=1 (1/|xj|) ď P is total no of input patterns. ď Weight between 1st layer and output layer: P ď wij = (1/2P) âp=1 (1/ f(â vij xj )
- 8. II. FREQUENCY OF WEIGHT UPDATES ď Per pattern training: Weight changes after every input is applied. ď Input set repeated if NN is not trained yet. ď Per epoch training: Epoch is one iteration through the process of providing the network with an input and updating the network's weights ď Many epochs are required to train the neural network ď weight changes as suggested by every input are accumulated together into a single change at the end of each epoch i.e. set of patterns. ď No change in weight at end of each input. ď Also called BATCH MODE training
- 9. FREQUENCY OF WEIGHT UPDATES ď Advantages / Disadvantages ď Batch mode training not possible for on- line training. ď For large applications, with large training time, parallel processing may reduce time in batch mode training. ď Per pattern training is more expensive as weight changes more often. ď Per pattern suitable for small NN and small data set
- 10. III. LEARNING RATE ď FOR PERCEPTRON TRAINING ALGORITHM ď Too small Ρ â Very slow learning ď Too large Ρ â Output may saturate on one direction. ď Ρ = 0 --- no weight change ď Ρ = 1 --- Common Choice
- 11. PROBLEM WITH Ρ = 1 ď If Ρ = 1 âw = Âą x ď New Output = (w + âw)t x ď Output = wtx Âą xtx ď Here if wtx > xtx output will always be positive and grows in one direction only. ď Should be - wtx < âwtx âw = Âą x ď Ρ |xtx| >| wtx | ď Ρ > | wtx | / |xtx| ď Ρ is normally between 0 and 1.
- 12. III. LEARNING RATE ď FOR BACK PROPAGATION ALGORITHM ď Large Ρ in early iterations and steadily decrease it when NN is converging. ď Increase Ρ at every iteration that improves performance by significant amount and vise versa. ď Steadily double the Ρ untill error value worsens. ď If Second derivative of E, âź2E is constant and low, Ρ can be large. ď If Second derivative of E, âź2E is large, Ρ can be small. ď For above, more computation required.
- 13. MOMENTUM â˘Training done to reduce this error. â˘Training may stop at local minima instead global minima.
- 14. MOMENTUM ď Can be prevented if weight changes depend on average gradient of Error, rather than gradient at a point. ď Averaging δE/ δw in a small neighborhood leads the network in general direction of MSE decrease without getting stuck at local minima. ď May become complex.
- 15. MOMENTUM ď Shortcut method: ď Weight change at ith iteration of back propagation algorithm also depends on immediately preceding weight changes. ď This has an averaging effect. ď This diminishes drastic fluctuations in weight changes over consecutive iterations. ď Achieved by adding momentum to weight update rule.
- 16. MOMENTUM ď Îwkj(t+1) = Ρδkxi + Îąâwkj(t) ď âwkj(t) is weight change required at time t . ď Îą is a constant . Îą ⤠1. ď Disadvantage: ď Past training trend can strongly bias current training. ď Îą depends on application. ď Îą = 0, no effect of past value. ď Îą = 1, no effect of current value.
- 17. What constitutes a âgoodâ training set? ď Samples must represent the general population ď Samples must contain members of each class ď Samples in each class must contain a wide range of variations or noise effect
- 18. GENERALIZABILITY ď Occurs more in large NN with less inputs. ď Inputs are repeated while training till error reduces. ď This leads to network memorizing the inputs samples. ď Such trained NN may behave correctly with training data but fail with any unknown data. ď Also called over training.
- 19. GENERALIZABILITY- SOLUTION ď The set of all known samples is broken into two orthogonal (independent) sets: ď Training set - A group of samples used to train the neural network ď Testing set - A group of samples used to test the performance of the neural network ⌠Used to estimate the error rate ď Training continues as long as error to test data gradually reduces. ď Training terminates as soon as error on test data increases.
- 20. GENERALIZABILITY E time Error on test data Error on training data Time when error on test data starts to increase â˘Performance over test data is monitored over several iterations, not just one iteration.
- 21. GENERALIZABILITY ď Weight will NOT change on test data. ď Overtraining can be avoided by using small number of parameters (hidden nodes and weights). ď If size of training set is small, multiple sets can be created by adding small randomly generated noise or displacement. ď X = { x1, x2, x3âŚ..xn} then ď Xâ = { x1+Ă1, x2+Ă2, x3+Ă3⌠xn + Ăn}
- 22. NO. OF HIDDEN LAYERS AND NODES ď Mostly obtained by trial and error. ď Too few nodes â NW may not be efficient. ď Too large nodes â ď Computation is tedious and expensive. ď NW may memorize the inputs and perform poorly on test data. ď NW is called well trained if performs well on data not used for testing. ď Hence NN should be capable of generalizing from input, rather than memorizing the inputs.
- 23. NO. OF HIDDEN LAYERS AND NODES ď Methods: ď Adaptive algorithm- ⌠Choose large number of nodes and train. ⌠Gradually discard nodes one by one during training. ⌠Train till performance reduces below unacceptable level. ⌠NN to be retrained at each change in nodes. ⌠Or vice versa ⌠Choose small number of nodes and increase nodes till performance is satisfactory.
- 24. Letâs see how NN size advances: ď Linear Classification: L1 ax1+bx2+c>0 ax1+bx2+c<0 L1
- 25. Letâs see how NN size advances: ď Two class problem - Nonlinear L1 L2 L11 L1 L2
- 26. Letâs see how NN size advances: ď Two class problem - Nonlinear L1 L2 L11 L3 L4 L1 L2 L3 L4 P
- 27. Letâs see how NN size advances: ď Two class problem - Nonlinear L22 PP1 P2 P3 P4 P1 P4 P2 P3
- 29. NUMBER OF INPUT SAMPLES ď As a thumb rule: 5 to 10 times as many samples as the number of weights to be trained. ď Baum and Haussler suggest: ⌠P > |w| /(1-a) ⌠P is number of samples, ⌠|w| is number of weights to be trained, ⌠a expected accuracy on test set.
- 30. Non-numeric inputs ď Nonnumeric inputs like colours have no inherent order. ď Can not be depicted on an axis e.g. red-blue- green-yellow. ď Colour becomes position sensitive. Results in Erroneous training. ď Hence assign binary vector with component corresponding to each colour. e.g. ď Green â 0 0 1 0 red â 1 0 0 0 ď Blue â 0 1 0 0 yellow â 0 0 0 1 ď But dimension increases drastically
- 31. Termination criteria ď âHalt when goal is achieved.â ď Perceptron training of linearly separable patterns â ⌠Correct classification of all samples. ⌠Termination is assured if Ć is sufficiently small. ⌠Program may run indefinitely if Ć is not appropriate. ⌠Different choice of if Ć may yield classification. ď Back propagation algorithm using delta ruleâ ⌠Termination can never be achieved with above criteria as output can never be +1 or -1. ⌠Will have to fix Emin , the minimum error acceptable. Terminates as error goes below Emin.
- 32. Termination criteria ď Perceptron training of linearly non-separable patterns â ⌠Above criteria will allow procedure to run indefinitely. ⌠Compare amount of progress in recent past. ⌠If number of misclassification has not changed in large step, samples are not linearly separable. ⌠Can fix limit of minimum % of correct classification for termination.