H046014853

Hong Li Int. Journal of Engineering Research and Applications www.ijera.com
ISSN : 2248-9622, Vol. 4, Issue 6( Version 1), June 2014, pp.48-53
www.ijera.com 48 | P a g e
Convergence Analysis of Adaptive Recurrent Neural Network
Hong Li*, Ali Setoodehnia**
*(Department of Computer Systems Technology, New York City College of Technology, New York, USA)
** (Department of Electronics Engineering Technology, ECPI University, USA)
ABSTRACT
This paper presents analysis of a modified Feed Forward Multilayer Perceptron (FMP) by inserting an ARMA
(Auto Regressive Moving Average) model at each neuron (processor node) with the Backp ropagation learning
algorithm. The stability analysis is presented to establish the convergence theory of the Back propagation
algorithm based on the Lyapunov function. Furthermore, the analysis extends the Back propagation learning
rule by introducing the adaptive learning factors. A range of possible learning factors is derived from the
stability analysis. Performance of such network learning with adaptive learning factors is presented and
demonstrates that the adaptive learning factor enhance the performance of training while avoiding oscillation
phenomenon.
Keywords–Adaptive learning, Back propagation, Neural networks, Stability analysis
I. INTRODUCTION
In the last few decades, Artificial Neural
Networks (ANNs) have been successfully applied to
many real world applicationsand proven to be useful
in various tasks of modeling nonlinear systems, such
as signal processing, pattern recognition,
optimization, weather forecasting, to name a few.
ANN is a set of processing elements (neurons or
perceptrons) with a specific topology of weighted
=interconnections between these elements and a
learning law for updating the weights of
interconnection between two neurons. To respond to
the increased demand of system identification and
forecasting with large set of data, many different
ANNs structures and learning rules, supervised, or
unsupervised, have been proposed to meet various
needs as robustness and stability. The FMP networks
have been shown to obtain successful results in
system identification and control [1, 2]. The
Lyapunov function approach was used to obtain
stability analysis of Backpropagation training
algorithm of such network in [3]. The major drawback
of the FMP is that it requires large number of input
datafor training to achieve sufficient performance.
Recurrent neural networks have been shown
successful in identification of time varying systems
along with the stability analysis in [4, 5]. However,
the training process can be very sensitive to initial
conditions such as number of neurons, the number of
layers, and value of weights, and learning factors
which are often chosen by trial and error. This paper
presents a modified FMP architecture which inserts a
dynamic filtering capability, ARMA local feedback at
each neuron in the FMP structure. The
Backpropagation algorithm is used for learning –
weight adjusting. Stability analysis will be derived
using Lyapunov function. It turns out that the
learning factor must be within a range of values in
order to guarantee the convergence of the algorithm.
In the simulation, instead of selecting a learning factor
by trial and error, authors define an adaptive learning
factor which satisfies the convergence condition and
adjust connection weight accordingly. The simulation
results are presented to demonstrate the performance.
II. BASIC PRINCIPLE OF ARMA-LFMP
NETWORK
An identification problem can be outlined as it is
shown in Figure 1. A set of data is collected from the
PLANT: input data and corresponding output data
observed, or measured as target output of the
identification problem. The set is often called
“training set”. A neural network model with
parameters, called weights, is designed to simulate
the PLANT. When the output from the neural
network is calculated, an error between the network
output and the target output is generated. The
learning process of neural network is to modify the
network to minimize the error.
Consider a system with n inputs and
m output unitsY = . In a typical neuron,
called perceptron, the output Y is expressed as:
(1)
Fig.1 Outline of ANN identification problem
RESEARCH ARTICLE OPEN ACCESS

Fig.2Feedforward multi-layer network
where is called connection weight from input ; F
is a nonlinear sigmoid function with a constant
number θ, call it slope.
A FMP network combines number of neurons,
called nodes, feed forward to next layer of nodes,
illustrated in Figure 2. Suppose is the number of
nodes in the lth layer, each output from thel-1th layer
will be used as input for the next layer, that can be
expressed as:
(2)
, where ; L is the number of layers in the
network and is the connection weight from the ith
node in l-1 layer to the jth node in l layer.
In this structure in Figure 3, an ARMA model is
inserted at each node in the Local feedback FMP
network. The outputs from ARMA model are used as
inputs to the FMP neural network node. The output at
the j-th node in the l-th layer with an ARMA model is
expressed as:
and
(4)
where
andT is the number of
patterns of the data set. X represents the output of
nonlinear sigmoid function for the hidden layer and
output layer, and is also used as an input to ARMA
model; represents the output of ARMA model; a
and b are connection weights of the ARMA, ν is
weight of local feedback at each node and DA and DB
are number of delays for AR and MA processes
respectively.
The back-propagation algorithm has become the
standard algorithm used for training feed-forward
multilayer perceptrons. It is a generalized the Least
Mean Square algorithm that minimizes the mean
squared error between the target output and the
network output with respect to the weights. The
algorithm looks for the minimum of the error
Fig. 3 One node of ARMA-LFMP model
function in the weight space using the method of
gradient descent. The combination of weights which
minimizes the error function is considered to be a
solution of the learning problem. A proof of the
Backpropagation algorithm was presented in [6] based
on a graphical approach in which the algorithm
reduces to a graph labeling problem.
The total error E of the network over all training
set is defined as
where is the error associated with pth pattern at
the kth node of output layer,
where is the target at kth node and is the
output of network at the kth node.
The network connection weights
between neurons i in layer l-1 and neuron j in Layer l
( l = 1, …, L) are updated iteratively by the Gradient
Descent Rule
where µ is the learning factor. Substituting (7) into
(8), (9) and (10), the above updating equation can be
expressed as follow:
The rate of change of an output from k-th node of
l-th layer with respect to connection weights a, b,and ν
in lth layer can be expressed as:

j(11)
(12)
Where
(13)
(14)
where
(15)
Further calculation leads to the expressions of ,
, and , rate of change of an output from k-
th node of l-thlayer with respect to connection weights
a, b, and ν in layer l-n for n < l:
(16)
(17)
(18)
where
(19)
(20)
(21)
III. STABILITY ANAYLSIS
Stability for nonlinear systems refers to the
stability of a particular solution. There may be one
solution which is stable and another which is not
stable. There are no inclusive general concepts of
stability for nonlinear systems. The behavior of a
system may depend drastically on the inputs and the
disturbances. However, Lyapunov developed a theory
to examine the stability of nonlinear systems.
The definition of Lyapunov function and
Lyapunov theorem are quoted below [7]:
Definition 1 (Lyapunov function): A scalar function
V(x) is a Lyapunov function for the system
(22)
if the following conditions hold:
1. and is continuous in x
2. is positive definite, that
is, with only if
3. is negative
definite, that
is, with onl
y if ;
Theorem 1(Lyapunov Theorem): The solution
for the system given by (11) is
asymptotically stable if there exists a Lyapunov
function in x.
The stability of the learning process in an
identification approach leads to a better modeling and
a guaranteed reached performance. According to
Lyapunov theorem, the determination of stability
depends on the selection and verification of a positive
definite function. For the systems defined in (3) – (5),
assume that the Backpropagation learning rule is
applied and the error function and weights updating
rule are defined in (6) – (10), then define
The proof is given in the following theorem that
V(t) satisfies the Lyapunov condition.
Theorem 2:Assume that the nonlinear function F( ) is
continuous and differentiable, the ARMA-LFMP is
defined in (3)-(5), rewrite the learning rule (8) - (10)
in weights vector form as:

(25)
(26)
where , are weight vectors in lth layer,
then the system is stable under the condition:
Proof: Assume that the Lyapunov function is defined
in (23), calculation of leads to:
(28)
Apply the first order Taylor expansion of
with respect to weight vectors ,
(29)
Substitute (24), (25), and (26) into (29),
(30)
Then, substitute the (30) into (28),
(31)
Apply the algebraic rule
into the second
term of (31)
(32)
For simplicity, let
(33)
Then apply it into (32) and consider that
,
, and
, then
(34)
To ensure that the function V satisfies the Lyapunov
condition,
Let the right-hand side of (34) be less than zero, and
consider that
, , and
,
then we obtain the condition
(35)
Therefore, the ARMA-LFMP system defined in (3)-
(5) is stable when the learning factor in
Backpropagation learning rule described in (8) –(10)
satisfies the condition (35).
For purpose of simplifying the simulation, instead of
calculating all , , and for l = 1, … L; j = 1,
… , the following corollary will identify an upper
bound of
, then
replace the denominator of the (35) by the upper
bound that provides a more restrictive but an easier
calculated condition.
Consider the infinite norm notation for any vector
that = , for
simplicity, we use notation in this paper
representing )
First, for the output layer L, apply infinite norm in
(11) and the notation , calculation
leads to
and then
(36)
From (5),

(37)
let , ,and
then from (13)
,
(38)
Apply(38) into (12)
(39)
From (15),
(40)
Apply(40) into (14)
(41)
Without loss of generality, for change of rate of layer l
with respect to l-n,we derive for n = 1 first. From (19)
(42)
(43)
Apply (43) into (16),
(44)
Similarly, apply norm in (20), and then apply into
(17),
(45)
Apply norm in (21), and then apply into (18),
(46)
Inductively, for any layer L-k< L, let
Δ = ,
(47)
(48)
(49)
Apply (47), (48) and (49) into condition (35), we
obtain a more restrictive condition as follows:
(50)
Corollary 1: The ARMA-LFMP system converges if
the following conditions are satisfied:
(51)
(52)
and .
IV. SIMULATION
In this section, an example of chaotic system
known as Henon system is considered to demonstrate
the effectiveness of developed methods in this paper.
The Hénon map is a discrete-time dynamical system
described as:
It is one of the most studied examples of
dynamical systems that exhibit chaotic behavior. The
Hénon map takes a point (x(k), y(k)) in the plane and
maps it to a new point.The map depends on two
parameters, a and b, which for the classical Hénon
map have values of a = 1.4 and b = 0.3. For the
classical values the Hénon map is chaotic.
In this simulation, consider the system
and a three-layer neural network structure was
selected for two inputs and two output with number
of nodes as 5, 5 and 2 in layer 1, 2 and 3
respectively. 100 patterns of data were generated and
used for learning. After number of trial and error
attempts, with slope set as 0.6 and learning factor set
as constant .01, and random generated initial weights,
the system reached to absolute error 0.0899999 after
2177311 iterations.
With adaptive learning factor, it took 373317
number of iteration to reach the same threshold of
0.089999. It is also observed that the error decreases
steadily while the adaptive learning factor is applied
and the oscillation of error was observed while a
predefined constant learning factor is applied.
The Figure 4 demonstrated 100 patterns of data
generated from Henon system comparing with
simulated data from the neural network described
above.
The adaptive learning factor guarantees that the
errors will steadily decrease. The drawback is that it
increased calculation since an updated learning factor
needs to be calculated at every weights update based
on Backpropagation algorithm. Applying the constant
learning factor avoids the calculation, but a proper
constant learning factor has to be identified through

trial and error. In the Figure 5, the plot demonstrated
comparison of
Figure 4. ANN Simulation of Henon System
errors from learning process with constant learning
factor and the adaptive learning factor. The constant
learning factor was selected with value of 0.1. To
compare the effectiveness of adaptive learning
method, the 0.1 was used as initial learning factor in
the adaptive method. Points for the plot were taken
from the errors of every 1000th of iteration.
Figure 5. Comparison of errors from learning with
adaptive and constant learning factor
REFERENCES
[1] R. Raeisi & A. Kaur, Artificial Neural
Networks for Realization and Verification of
Digital Logic Circuits. International Journal
of Modern Engineering, Spring/Summer
12(2), 5-9, 2012.
[2] W. Yu, A. S. Poznyak, and X. Li, Multilayer
dynamic neural networks for no-linear system
on-line identification,International Journal of
Control, 7(18). 1858-1864, 2001.
[3] T. Korkobi, M. Djemel, and M. Ctourou,
Stability analysis of neural netorks-based
system identification, Modeling and
Simulation in Engineering, Volume 2008,
Article ID 343940
[4] E. Zhu. Stability Analysis of Recurrent Neural
Networks with Random Delay and Markovian
Switching, Journal of Inequalities and
Applications 2010
[5] H. Akça, R. Alassar, and V. Covachev,
Stability of neural networks with time varying
delays in the presence of impulses, Advances
in Dynamical Systems and Applications 1(1),
pp. 1-15, 2006
[6] R. Rojas, Neural Networks, Springer-Verlag,
1996
[7] Sontag, E.D., Mathematical Control Theory,
Springer-Verlag, New York
[8] M. Young, The Technical Writer’s Handbook.
Mill Valley, CA: University Science, 1989.

H046014853

More Related Content

What's hot (18)

Viewers also liked (20)

Similar to H046014853 (20)

Recently uploaded (20)

H046014853