11.digital image processing for camera application in mobile devices using artificial neural networks
0 likes417 views
This document discusses using artificial neural networks for digital image processing on mobile devices. It proposes training a neural network using sample input and output images to generate a "function matrix" that can then process other images in real-time on mobile devices. The neural network has 9 input nodes, 9 hidden nodes, and 9 output nodes arranged to process 3x3 pixel sections of images. It is trained using backpropagation to modify weights to match sample output images. This allows mobile devices to perform effects like edge detection without large pre-defined processing matrices.
![Control Theory and Informatics www.iiste.org
ISSN 2224-5774 (print) ISSN 2225-0492 (online)
Vol 2, No.1, 2012
This section describes the existing methods involved in the processing of the image data.
The conventional methods of processing the image involve the formation of a matrix (usually a [3x3] or
[9x9] matrix) called the function matrix. The elements of this matrix depend on the type of processing
required. For example, for edge detection, the matrix is one which is determined by the type of algorithm
used for edge detection namely, Sobel, Prewitt, Roberts, log, zero cross, etc. Hence, this matrix is not
unique and depends on the output required. In addition, the processing method is meant to determine this
matrix and as such varies depending on the application. One needs to write a different program for different
application to determine the matrix. Once the function matrix is determined, the next step is to convolve the
above matrix with the given input matrix to obtain the required output image matrix. The disadvantage of
this method is that one can modify the images to certain predetermined format only. If multiple effects are
to be given, then the image needs to be modified several times. This makes the methods computationally
intensive and not suitable for limited resource devices.
3. Artificial Neural Network Method
The processor is trained to form the function matrix (weight matrix) by comparing the sample input matrix
and sample output matrix. Hence, the essential requirement of this technique is a sample input image
matrix and a desired sample output image matrix. Once these two images are available, the system is
trained to generate the function matrix by adaptive processing and once the function matrix is generated,
any number of input images can be converted to the required format. The program is very much generic
and can be used for any type of processing like edge detection, smoothening, filtering, etc. without any
modifications. Hence, the only requirement of this method is that of an initial input and output sample
images. The function matrix can also be generated on a separate high capacity system and used by the
mobile device for processing the final images. Depending upon the type of effects needed, several such
matrices can be stored and appropriately loaded when the desired effect is selected from the camera
application.
For the implementation of image processor, a 3-layer feed forward neural network model has been made
use of. There are 9 input nodes in layer 1, 9 hidden nodes in the 2nd layer and 9 nodes in the output layer.
The 9 inputs are the 9 pixel values of the image data obtained in [3x3] format. The neural network structure
used in this technique is shown in figure 1.
The input values range from 0 to 1, corresponding to the colour of the pixel. Moreover, since the output of
the network is the corresponding [3x3] matrix of the output image, the output will also range from 0 to 1.
The activation function for the neuron is a sigmoid as shown in figure 2.
The activation function is described as follows.
f ( net ) = 1 /(1 + exp( −net ))
(1)
The network is trained using the error back propagation algorithm. Since it is a universal approximator, the
network can perform any linear or non-linear point-by-point operation.
For training the network, a sample image and the desired output image needs to be given in a very simple
format. These images are then loaded into the system. The neural network maps the sample image through
it, and if it finds that its output and the expected output do not match then it modifies itself according to the
learning rule. The network takes a [3x3] matrix of the input and the output and scans over the whole image,
similar to the convolution process. In order to perform a good and effective training process, most of the
possibilities must be presented to the network.
Hence, the training set may not be able to follow all the desired properties, unless given enough experience.
Once the training is over, the modified network parameters can be used to work on any other image to
produce the desired output. The input image is fed to the neurons in the form of a [3x3] matrix, and the
output of the neurons is transformed into a [3x3] matrix and stored as the output image. This process is
carried out repeatedly, until the whole image is scanned. If the output that we get does not have a particular
property, then the same neural network can be further trained for that specific property, while the earlier
12](https://image.slidesharecdn.com/11-digitalimageprocessingforcameraapplicationinmobiledevicesusingartificialneuralnetworks-120513002007-phpapp02/85/11-digital-image-processing-for-camera-application-in-mobile-devices-using-artificial-neural-networks-2-320.jpg)
![Control Theory and Informatics www.iiste.org
ISSN 2224-5774 (print) ISSN 2225-0492 (online)
Vol 2, No.1, 2012
properties still remain embedded in the network. Hence, we can modify and train the network for more and
more properties at a time.
3.1 Formulation
The mathematical formulation required to implement the image processing algorithm using neural
networks is presented below.
Let the vector x = [Xi]9x1 be the input to the neural network, where i stands for the ith input or pixel of the
[3x3] matrix.
v = [Vij]9X9 and w = [Wij]9x9 are the weight matrices, where ij stands for the weight between the ith input and
the jth neuron. [netj] is the net input to the jth neuron, and is given by
[net ]
j 9 x1 = [vij ]9 x 9 ∗ [xi ]9 x1 (2)
Or,
[net ] = [w ]∗ [y ]
j ij i
(3)
Then the output of the neuron is given by the vector [Oj] as,
[O ]
j 9 x1 = [ f ( net j )]9 x1 (4
)
3.2 Processing
If a network is already trained then the output of the network is obtained by feed forward method as
[ y] = f ([v] ∗[ x]) (5)
[ z ] = f ([ w] ∗ [ y ]) (6)
Where [y] is the output of the hidden layer and [z] is the output of the network.
3.3 Training
To train the network, weight matrices w and v are initialized with some random values. Next, the error back
propagation algorithm is used to train the network as follows
[ ∆w] = ([ d ] − [ z ]) • (1 − [ z ]).[ z ] (7)
[ ∆v ] = [ y ] • (1 − [ y ]) • ([ w]T ∗ [ ∆w]) (8)
Where,
[d] is the desired output matrix
[∆w] is the change in w
[∆v] is the change in v
The weights of the system are updated as
13](https://image.slidesharecdn.com/11-digitalimageprocessingforcameraapplicationinmobiledevicesusingartificialneuralnetworks-120513002007-phpapp02/85/11-digital-image-processing-for-camera-application-in-mobile-devices-using-artificial-neural-networks-3-320.jpg)
![Control Theory and Informatics www.iiste.org
ISSN 2224-5774 (print) ISSN 2225-0492 (online)
Vol 2, No.1, 2012
[ w] = [ w] + c([ ∆w] ∗ [ y ]T ) (9)
[v ] = [v ] + c ([ ∆v ] ∗ [ x ]T ) (10)
In (9) and (10), c is called the learning constant and its value is positive.
3.4 Implementation
The image to be processed and the desired output images are split into [3x3] matrices, then rearranged in a
[9x1] format and stored in x and d respectively. They are used to train the network. After each update, the
new output is calculated. If the error between the output image and the desired image is smaller than ∆,
next training sample is taken, else the training is repeated.
Once the training is complete, the weight matrices can be used to perform the trained operation on any
other input images. To do that, first the given image is split into [3x3] blocks and processed upon in the
feed forward mode. The output [z](9x1) is then rearranged into a [3x3] matrix and is used to replace the input
matrix in its respective position. After all the blocks of the image are processed, we get the desired output
image.
The camera sensors generally produce output in RGB as well as YCbCr formats. If the processing does not
involve modifying the colour components and YCbCr format is used as input, then only the Y component
can be used for operating with the function matrix and later re-composed with the Cb and Cr components.
This reduces the overall computations involved in the processing.
4. Experimental Results
In this section the simulation results of the algorithm described in section 3 for detecting the edges of the
input image are presented. Two simple sample images were created using a software drawing tool on a
computer. One of them, the input, had shaded geometric objects (Figure 3(a)), and the other, the output
image, had only the edges of the input image (Figure 3(b)). The two images were loaded into an array and
[3x3] matrices were derived from the input and the output arrays. These were then processed using the
formulation mentioned in section 3.1 and the resulting weight matrix stored for further usage as the edge
detector.
The edge detector matrix is then retrieved into v and w. The new image to be processed is loaded onto an
array and processed according to the implementation formulation. The output is obtained in the form of
edges of the input image. Figures 4(a) and 4(b) show the input and the result.
5. Conclusion
This technique of image processing allows the user to add any desired effect to the image without any prior
database. Hence the users themselves can program the image processor. Since the matrix once computed
can be used on any other image, the time and complexity of it is very less. Thus this is highly suitable for
mobile devices. This technique can be used for the edge detection, digital filters, colour detection, colour
transformation, colour edge detection, etc. Thus this method can be an efficient substitute for hardware
ISPs and advanced image sensors.
References
Castleman, K. R. (1998), “Digital Image Processing”, Prentice Hall.
Etemad, K. & Chellappa, R. (1993), “A Neural Network Based Edge Detector”, IEEE International
Conference on Neural Networks, 1, 132-137.
Kamat, S. P. & Jayakumar, A. (2004), “A Novel Method for Colour Edge Detection in an Image Using
Neural Networks Approach”, Proc. of Global Signal Processing Expo and Conference, GSPx 2004.
14](https://image.slidesharecdn.com/11-digitalimageprocessingforcameraapplicationinmobiledevicesusingartificialneuralnetworks-120513002007-phpapp02/85/11-digital-image-processing-for-camera-application-in-mobile-devices-using-artificial-neural-networks-4-320.jpg)
11.digital image processing for camera application in mobile devices using artificial neural networks
- 1. Control Theory and Informatics www.iiste.org ISSN 2224-5774 (print) ISSN 2225-0492 (online) Vol 2, No.1, 2012 Digital Image Processing for Camera Application in Mobile Devices Using Artificial Neural Networks Sachin P. Kamat Samsung India Software Operations Pvt. Ltd., Bangalore, 560052, India * E-mail: [email protected] Abstract This paper utilizes artificial neural networks based image processing techniques for low capacity and resource constrained devices like mobile phones for camera applications. The system is trained to develop the operating matrix, called the function matrix, by using artificial neural network theory, from a sample input and output image matrix. This is done when the system is in idle mode. After having obtained the function matrix, it can be very conveniently operated upon any other input image matrix by simple multiplication to obtain the desired modification in the input image in real time. Computer simulation results are provided to prove the concept. Keywords: image processing, artificial neural networks, mobile devices, camera application 1. Introduction Camera is an integral part of most mobile devices coming to market in recent times. High end mobile devices come with high resolution cameras and powerful sensors that can process large size images instantly through the on-chip image signal processors (ISP) and digital signal processors (DSP). They support several kinds of image effects like edge detection, smoothening, filtering, embossing, pencil sketch, etc. These operations are computationally intensive and require high speed CPUs and digital signal processors to perform them in real time. Thus these operations are not generally supported by low-end mobile devices which have a basic and low resolution sensor that can cater to only very basic image and colour effects. It is also not possible to perform all these image processing operations in software on the mobile device without loss of real time constraints and hence most of the operations need to be carried out on desktop computers using the bundled software tools. This paper describes the concept of using artificial neural networks for image processing whereby the embedded system is trained to develop the operating matrix, called the function matrix, by using artificial neural network theory using a sample input and output image matrix. Having obtained the function matrix, it can be very conveniently operated upon any other input image matrix by simple multiplication to obtain the desired modification in the input image. The conventional methods of image processing operations involve convolving the given image matrix with a predetermined function matrix depending upon the effect required. This is a computationally intensive operation and it becomes necessary to know the function matrix before hand. In addition, the matrix required for convolution varies with the type of processing required. Image processing using artificial neural networks does not involve the conventional convolution techniques. On the other hand, the system is trained to generate the function matrix depending upon the type of output required. Several methods and techniques of achieving the same are already available and this paper attempts to utilize the results of artificial neural network based image processing techniques for a real life use case scenario. The rest of the paper is organized as follows. Section 2 describes the existing methods of image processing, artificial neural networks based method is described in section 3. Experimental results are elaborated in section 4 and conclusions are given in section 5. 2. Conventional Methods 11
- 2. Control Theory and Informatics www.iiste.org ISSN 2224-5774 (print) ISSN 2225-0492 (online) Vol 2, No.1, 2012 This section describes the existing methods involved in the processing of the image data. The conventional methods of processing the image involve the formation of a matrix (usually a [3x3] or [9x9] matrix) called the function matrix. The elements of this matrix depend on the type of processing required. For example, for edge detection, the matrix is one which is determined by the type of algorithm used for edge detection namely, Sobel, Prewitt, Roberts, log, zero cross, etc. Hence, this matrix is not unique and depends on the output required. In addition, the processing method is meant to determine this matrix and as such varies depending on the application. One needs to write a different program for different application to determine the matrix. Once the function matrix is determined, the next step is to convolve the above matrix with the given input matrix to obtain the required output image matrix. The disadvantage of this method is that one can modify the images to certain predetermined format only. If multiple effects are to be given, then the image needs to be modified several times. This makes the methods computationally intensive and not suitable for limited resource devices. 3. Artificial Neural Network Method The processor is trained to form the function matrix (weight matrix) by comparing the sample input matrix and sample output matrix. Hence, the essential requirement of this technique is a sample input image matrix and a desired sample output image matrix. Once these two images are available, the system is trained to generate the function matrix by adaptive processing and once the function matrix is generated, any number of input images can be converted to the required format. The program is very much generic and can be used for any type of processing like edge detection, smoothening, filtering, etc. without any modifications. Hence, the only requirement of this method is that of an initial input and output sample images. The function matrix can also be generated on a separate high capacity system and used by the mobile device for processing the final images. Depending upon the type of effects needed, several such matrices can be stored and appropriately loaded when the desired effect is selected from the camera application. For the implementation of image processor, a 3-layer feed forward neural network model has been made use of. There are 9 input nodes in layer 1, 9 hidden nodes in the 2nd layer and 9 nodes in the output layer. The 9 inputs are the 9 pixel values of the image data obtained in [3x3] format. The neural network structure used in this technique is shown in figure 1. The input values range from 0 to 1, corresponding to the colour of the pixel. Moreover, since the output of the network is the corresponding [3x3] matrix of the output image, the output will also range from 0 to 1. The activation function for the neuron is a sigmoid as shown in figure 2. The activation function is described as follows. f ( net ) = 1 /(1 + exp( −net )) (1) The network is trained using the error back propagation algorithm. Since it is a universal approximator, the network can perform any linear or non-linear point-by-point operation. For training the network, a sample image and the desired output image needs to be given in a very simple format. These images are then loaded into the system. The neural network maps the sample image through it, and if it finds that its output and the expected output do not match then it modifies itself according to the learning rule. The network takes a [3x3] matrix of the input and the output and scans over the whole image, similar to the convolution process. In order to perform a good and effective training process, most of the possibilities must be presented to the network. Hence, the training set may not be able to follow all the desired properties, unless given enough experience. Once the training is over, the modified network parameters can be used to work on any other image to produce the desired output. The input image is fed to the neurons in the form of a [3x3] matrix, and the output of the neurons is transformed into a [3x3] matrix and stored as the output image. This process is carried out repeatedly, until the whole image is scanned. If the output that we get does not have a particular property, then the same neural network can be further trained for that specific property, while the earlier 12
- 3. Control Theory and Informatics www.iiste.org ISSN 2224-5774 (print) ISSN 2225-0492 (online) Vol 2, No.1, 2012 properties still remain embedded in the network. Hence, we can modify and train the network for more and more properties at a time. 3.1 Formulation The mathematical formulation required to implement the image processing algorithm using neural networks is presented below. Let the vector x = [Xi]9x1 be the input to the neural network, where i stands for the ith input or pixel of the [3x3] matrix. v = [Vij]9X9 and w = [Wij]9x9 are the weight matrices, where ij stands for the weight between the ith input and the jth neuron. [netj] is the net input to the jth neuron, and is given by [net ] j 9 x1 = [vij ]9 x 9 ∗ [xi ]9 x1 (2) Or, [net ] = [w ]∗ [y ] j ij i (3) Then the output of the neuron is given by the vector [Oj] as, [O ] j 9 x1 = [ f ( net j )]9 x1 (4 ) 3.2 Processing If a network is already trained then the output of the network is obtained by feed forward method as [ y] = f ([v] ∗[ x]) (5) [ z ] = f ([ w] ∗ [ y ]) (6) Where [y] is the output of the hidden layer and [z] is the output of the network. 3.3 Training To train the network, weight matrices w and v are initialized with some random values. Next, the error back propagation algorithm is used to train the network as follows [ ∆w] = ([ d ] − [ z ]) • (1 − [ z ]).[ z ] (7) [ ∆v ] = [ y ] • (1 − [ y ]) • ([ w]T ∗ [ ∆w]) (8) Where, [d] is the desired output matrix [∆w] is the change in w [∆v] is the change in v The weights of the system are updated as 13
- 4. Control Theory and Informatics www.iiste.org ISSN 2224-5774 (print) ISSN 2225-0492 (online) Vol 2, No.1, 2012 [ w] = [ w] + c([ ∆w] ∗ [ y ]T ) (9) [v ] = [v ] + c ([ ∆v ] ∗ [ x ]T ) (10) In (9) and (10), c is called the learning constant and its value is positive. 3.4 Implementation The image to be processed and the desired output images are split into [3x3] matrices, then rearranged in a [9x1] format and stored in x and d respectively. They are used to train the network. After each update, the new output is calculated. If the error between the output image and the desired image is smaller than ∆, next training sample is taken, else the training is repeated. Once the training is complete, the weight matrices can be used to perform the trained operation on any other input images. To do that, first the given image is split into [3x3] blocks and processed upon in the feed forward mode. The output [z](9x1) is then rearranged into a [3x3] matrix and is used to replace the input matrix in its respective position. After all the blocks of the image are processed, we get the desired output image. The camera sensors generally produce output in RGB as well as YCbCr formats. If the processing does not involve modifying the colour components and YCbCr format is used as input, then only the Y component can be used for operating with the function matrix and later re-composed with the Cb and Cr components. This reduces the overall computations involved in the processing. 4. Experimental Results In this section the simulation results of the algorithm described in section 3 for detecting the edges of the input image are presented. Two simple sample images were created using a software drawing tool on a computer. One of them, the input, had shaded geometric objects (Figure 3(a)), and the other, the output image, had only the edges of the input image (Figure 3(b)). The two images were loaded into an array and [3x3] matrices were derived from the input and the output arrays. These were then processed using the formulation mentioned in section 3.1 and the resulting weight matrix stored for further usage as the edge detector. The edge detector matrix is then retrieved into v and w. The new image to be processed is loaded onto an array and processed according to the implementation formulation. The output is obtained in the form of edges of the input image. Figures 4(a) and 4(b) show the input and the result. 5. Conclusion This technique of image processing allows the user to add any desired effect to the image without any prior database. Hence the users themselves can program the image processor. Since the matrix once computed can be used on any other image, the time and complexity of it is very less. Thus this is highly suitable for mobile devices. This technique can be used for the edge detection, digital filters, colour detection, colour transformation, colour edge detection, etc. Thus this method can be an efficient substitute for hardware ISPs and advanced image sensors. References Castleman, K. R. (1998), “Digital Image Processing”, Prentice Hall. Etemad, K. & Chellappa, R. (1993), “A Neural Network Based Edge Detector”, IEEE International Conference on Neural Networks, 1, 132-137. Kamat, S. P. & Jayakumar, A. (2004), “A Novel Method for Colour Edge Detection in an Image Using Neural Networks Approach”, Proc. of Global Signal Processing Expo and Conference, GSPx 2004. 14
- 5. Control Theory and Informatics www.iiste.org ISSN 2224-5774 (print) ISSN 2225-0492 (online) Vol 2, No.1, 2012 Terry, P. J. & Vu, D. (1993), “Edge Detection Using Neural Networks”, 1993 Conference Record of The Twenty-Seventh Asilomar Conference on Signals, Systems and Computers, 1, 391-395. Zurada, J. M. (1994), “Introduction to Artificial Neural Systems”, Jaico publishing house. Figure 1. Neural Network Structure Figure 2. Sigmoid Activation Function for the Neuron 15
- 6. Control Theory and Informatics www.iiste.org ISSN 2224-5774 (print) ISSN 2225-0492 (online) Vol 2, No.1, 2012 Figure 3. (a) Input Image for Training the Neural Network (b) Output Image for Training the Neural Network Set of input and output images used for training the neural network. The output image consists of the edges of the image in the input. Figure 4. (a) Input Image for Processing (b) Output Image after Processing Output image obtained from the neural network after feeding the input image. The network has been trained to produce edges of the input image. 16
- 7. International Journals Call for Paper The IISTE, a U.S. publisher, is currently hosting the academic journals listed below. The peer review process of the following journals usually takes LESS THAN 14 business days and IISTE usually publishes a qualified article within 30 days. Authors should send their full paper to the following email address. More information can be found in the IISTE website : www.iiste.org Business, Economics, Finance and Management PAPER SUBMISSION EMAIL European Journal of Business and Management [email protected] Research Journal of Finance and Accounting [email protected] Journal of Economics and Sustainable Development [email protected] Information and Knowledge Management [email protected] Developing Country Studies [email protected] Industrial Engineering Letters [email protected] Physical Sciences, Mathematics and Chemistry PAPER SUBMISSION EMAIL Journal of Natural Sciences Research [email protected] Chemistry and Materials Research [email protected] Mathematical Theory and Modeling [email protected] Advances in Physics Theories and Applications [email protected] Chemical and Process Engineering Research [email protected] Engineering, Technology and Systems PAPER SUBMISSION EMAIL Computer Engineering and Intelligent Systems [email protected] Innovative Systems Design and Engineering [email protected] Journal of Energy Technologies and Policy [email protected] Information and Knowledge Management [email protected] Control Theory and Informatics [email protected] Journal of Information Engineering and Applications [email protected] Industrial Engineering Letters [email protected] Network and Complex Systems [email protected] Environment, Civil, Materials Sciences PAPER SUBMISSION EMAIL Journal of Environment and Earth Science [email protected] Civil and Environmental Research [email protected] Journal of Natural Sciences Research [email protected] Civil and Environmental Research [email protected] Life Science, Food and Medical Sciences PAPER SUBMISSION EMAIL Journal of Natural Sciences Research [email protected] Journal of Biology, Agriculture and Healthcare [email protected] Food Science and Quality Management [email protected] Chemistry and Materials Research [email protected] Education, and other Social Sciences PAPER SUBMISSION EMAIL Journal of Education and Practice [email protected] Journal of Law, Policy and Globalization [email protected] Global knowledge sharing: New Media and Mass Communication [email protected] EBSCO, Index Copernicus, Ulrich's Journal of Energy Technologies and Policy [email protected] Periodicals Directory, JournalTOCS, PKP Historical Research Letter [email protected] Open Archives Harvester, Bielefeld Academic Search Engine, Elektronische Public Policy and Administration Research [email protected] Zeitschriftenbibliothek EZB, Open J-Gate, International Affairs and Global Strategy [email protected] OCLC WorldCat, Universe Digtial Library , Research on Humanities and Social Sciences [email protected] NewJour, Google Scholar. Developing Country Studies [email protected] IISTE is member of CrossRef. All journals Arts and Design Studies [email protected] have high IC Impact Factor Values (ICV).