Robot Arm Utilized Having Meal Support System Based on Computer Input by Human Eyes Only

Kohei Arai & Kenro Yajima
International Journal of Human Computer Interaction (IJHCI), Volume (2) : Issue (1) : 2011 1
Robot Arm Utilized Having Meal Support System Based on
Computer Input by Human Eyes Only
Kohei Arai arai@is.saga-u.ac.jp
Department of Information Science
Saga University
1 Honjo, Saga 840-8502 Japan
Kenro Yajima yaji@ip.is.saga-u.ac.jp
Department of Information Science
Saga University
1 Honjo, Saga 840-8502 Japan
Abstract
A robot arm utilized having meal support system based on computer input by human eyes only is
proposed. The proposed system is developed for handicap/disabled persons as well as elderly
persons and tested with able persons with several shapes and size of eyes under a variety of
illumination conditions. The test results with normal persons show the proposed system does
work well for selection of the desired foods and for retrieve the foods as appropriate as users’
requirements. It is found that the proposed system is 21% much faster than the manually
controlled robotics.
Keywords: Computer Input by Human Eyes Only, Line of Sight Estimation, Gaze Estimation,
Robot Arm control, Image Segmentation With Clustering.
1. INTRODUCTION
There are some systems which allow computer input by human eyes only [1]. Just by sight, any
desired key can be selected and determined with the system composed with a camera mounted
computer display and the methods for gaze estimation with the acquired user face images [2].
The system allows users’ head movements using head pose estimation by using extracted
feature points from the acquired face images and also estimate line of sight vector which defined
a s the line which connects between pupil center and eyeball center [3]. Thus computer input by
human eyes only can be done.
There are 2.1 million of handicap persons (about 6%) in Japan. They can use their eyes; they
cannot use their body below their neck though. By using the computer input system by human
eyes only, they can take a meal through a selection of foods of which they want to have if they
guide the robot arm by their eyes only. Using the system, users can control robot arms then users
can bring foods of which they would like to have by their own choice. The current system allows
to support having meal by retrieving not too hot drinkable foods and drinks with tube because it is
not easy to feed the foods with folk (it is dangerous to get close the folk to user’s face) for disable
persons.
There are some meal feeding support systems using robotics [4],[5]. Most of the existing systems
use joysticks for user interface and brain-machine interface [6]. There is commercially available
meal feeding support system so called "My Spoon"1
by Secom Co., Ltd.[7]. It uses joystick and or
button for operation of robot arm. Y.Kohya et al., proposed ultrasonic motor featured robot for
1
http://www.secom.co.jp/personal/medical/myspoon.html

meal feeding support system [8] while a meal feeding robot system with collaborative motions
between robot arm and meal tray is also proposed [9].
Assuming disable persons in this research cannot use their hands and arms so that joystick
interface does not work. Brain-machine interface, meanwhile, insists users a psychological impact
(Sensor heads have to be attached on their head and/or face). On the other hand, there is no
such impact for the proposed computer input system by human eyes only. Robot arm utilized
having meal support system with computer input system human eyes only has to be robust
against illumination condition changes and a various types of users eye shapes as well as seeing
capabilities. Experiments with a variety of conditions have to be conducted.
In the next section, the proposed system is described followed by some experimental results with
a variety of conditions. Concluding remarks and some discussions together with future works are
described in the final section.
2. PROPOSED SYSTEM
The proposed system is composed with visible camera and HMD: Head Mount Display mounted
glass and camera mounted robot arm. While user wears the glass, user can look at the outer
world which is acquired with the camera mounted at the tip of the robot arm and also user’s eye
and the surrounding images are acquired with the camera mounted on the glass. System
configuration is shown in Fig.1. The only thing user has to do is to look at the desired foods on
the tray of the meal displayed onto the screen in the HMD of computer screen. The specification
of HMD, camera and robot arm are shown in Table 1, 2 and 3, respectively. Also outlook of the
HMD and camera mounted glass and the camera mounted robot arm are shown in Fig.2 and 3,
respectively.
FIGURE 1: System Configuration of the Proposed System

FIGURE 2: Outlook of the HMD and Camera Mounted Glass
FIGURE 3: Outlook of the camera Mounted Robot Arm
Pixel size SVGA(800 × 600)
Assumed display distance 2m
Assumed display size 60 inch
Field of view 42 deg.
Input RGB
Working temperature 0 to 40 deg.C
Size 28mm(W) × 35.24mm(H) × 56mm(D)
Weight 20g
TABLE 1: The Specification of HMD
Sensor element 0.3 million of elements of CMOS
Maximum pixel size 1280 × 960
Maximum viewing angle 78 deg.
Frame rate 320 × 240 for 30fps
Frame rate 640 × 480 for 15fps
TABLE 2: The Specification of Camera

Power supply Dry battery× 4
Supply voltage ±3
Maximum load weight 130g
Maximum rotation radius 360mm
Maximum height 510mm
Base plate W180 × D180mm
Total weight 1050g
Controller weight 150g
The number of joints 5
Base rotation angle 350 °
Shoulder rotation angle 120 °
Elbow rotation angle 135 °
Wrist rotation angle 340 °
Robot finger 50mm
TABLE 3: The Specification of Robot Arm
In accordance with the estimated gaze location on the HMD through an analysis of acquired
user’s face images, the desired foods on the tray can be selected and determined. Tray image is
acquired with the camera mounted on the tip of the robot arm. The acquired image can be used
to get close to the desired foods with robot arm controls. If the user looks at the desired foods,
line of sight of user’s eye is pointing to the foods displayed onto HMD screen and both eyes
moves simultaneously so that the desired foods can be selected and determined if user’s gaze is
estimated. In order to extract eye and its surrounding from the acquired face image, adaptive
threshold method [10] is used.
As is shown in Fig.4, gaze location on HMD display can be estimated through the line of sight
estimation with an analysis of the acquired user’s face images. The line of sight is defined as the
line between eyeball center and pupil center. Eyeball is assumed sphere with the diameter of
24mm (Typical eyeball size). The cornea can be detected by using the reflectance difference
between cornea and sclera. Then pupil center is extracted in the cornea then it can be estimated
as is shown in Fig.5. Red point in the image shows pupil center. Thus the line (line of sight)
between pupil center and the eyeball center can be estimated [8]. Therefore, if user looks at the
desired foods on the tray, the robot arm gets close to the desired foods automatically.
FIGURE 4444: Process flow for gaze location estimation from the acquired face images
FIGURE 5: Process flow for estimation of cornea center

FIGURE 6: Definition of eyeball
Eyeball is assumed as is shown in Fig.6. Pupil center is estimated based on the aforementioned
manner so that the eyeball rotation angle and the line of sight is estimated as is shown in Fig.7
(a). R is assumed to be 12mm and r is estimated through pupil center movement estimations.
Thus gaze location on the HMD display is estimated as is shown in Fig.7 (b).
r = Rsinθ (1)
θ= arcsin(r/R) (2)
(a) Line of sight estimation (b) Gaze location on HMD display estimation
FIGURE 7: Process flow for line of sight estimation and gaze location estimation
Fig.8 (a) shows the outlook of the robot arm and Fig.8 (b) also shows one of examples of camera
image which is acquired with the camera that is mounted on the top of the robot arm. Objective
target of the desired food on the tray can be selected and determined with the camera acquired
image. When user is looking at the desired food, the robot arm gets close to the food
automatically as is shown in Fig.9. During the food retrieval process, the desired food has to be
detected and tracked because the objective target of the desired food is moving in the acquired
image. Object tracking is done with optical flow [12] while the robot arm is controlled by its base
location movement as well as up and down movements of its elbow and shoulder. These robot

arm movements can be controlled automatically in accordance with user’s gaze location (ideally
the center of the desired food) estimated with the camera acquired image. Fig.10 shows the
definitions of the coordinate system and base, shoulder and elbow angles.
(a) Outlook of the robot arm (b) Food on the tray
FIGURE 8: Outlook of the robot arm and an example image of food.
FIGURE 9: Process flow for the desired food retrievals with robot arm which is controlled by human eyes
only.
θ1: Base angle, θ2: Shoulder angle, θ3: Elbow angle
FIGURE 10: Definition of base, shoulder and elbow angles and coordinate system
Blue circle portion in Fig.10 denotes the camera mounted at the tip of the robot arm while yellow
dot shows the location of the origin of the robot arm coordinate system.
3. EXPERIMENTS
The proposed system can be operated manually and automatically (based on human eyes only).

In the manual mode, the robot arm can be controlled by computer key in of selection processes of
base, shoulder and elbow angles control as well as right, left, up and down movements using
camera acquired image. On the other hand, the robot arm can be controlled by human eyes only
in the automatic mode (the proposed method). Table 4 shows the experimental results for the
automatic mode (the proposed method) while Table 5 shows those for the manual mode. There
are two types of persons for the test, the persons who have typical eye shape and size eyes and
the persons who have narrow eye shape and relatively small size of eyes. Illumination condition
was changed during the day time and the night time. Illumination was measured in unit of Lux
with radiance measuring instrument from the just above of the surface of the desired food. The
time required for retrievals of the desired food is also measured.
There was only one failure that was happened for the person with narrow eye with a week seeing
capability of eyes in the night time for relatively high illumination condition. Due to the fact that
optical flow was failed so that B(1) gave up to retrieve the desired food. He tried 12 times then he
gave up. Only the possible reason for failure is caused by the shape of the eye. It is rather difficult
to detecting and tracking the narrow and small eye than the typical normal size of eyes. A(1), A(2),
B(2) and C(1) made a success to retrieve the desired food in the automatic mode. Success rate
and the required time for retrievals do not depend on illumination condition. Success rate and the
required time for retrievals are not influenced by the illumination condition, if the illumination
ranges from 165 to 641 Lux. The number of trials for the manual mode is greater than that in the
automatic mode with human eyes only so that the required time for retrievals takes much longer.
It is rather difficult to retrieve the desired food in the manual mode. Six degree of freedom is not
easy to control in the manual mode. Meanwhile it is rather easy to retrieve the desired food in the
automatic mode because only thing user has to do is just looking at the desired food then the
feedback system control the location of the tip of the robot arm appropriately without
consideration of six degree of freedom.
Person
Eye feature (Seeing
capability)
Illumination
condition[Lux｣
Success or
Fail
The number of
trials
Required time for
retrievals[ms]
A(1) typical(1.5) Night time(169) Success 6 42085
A(2) typical(1.5) Day time(343) Success 4 57579
B(1) narrow(0.05) Night time(336) Fail 2
B(2) narrow(0.05) Night time(338) Success 12 29908
C(1) typical(1.5) Night time(186) Success 2 21921
Average 3.2 37873.25
TABLE 4: Experimental results for the automatic mode (the proposed method).

Person
Eye feature (Seeing
capability)
Illumination
condition[Lux｣
Success or
Fail
The number of
trials
Required time for
retrievals[ms]
A(1) typical(1.5) Day time(241) Success 12 39623
A(2) typical(1.5) Night time(165) Success 7 32913
B(1) narrow(0.05) Day time(207) Success 9 57048
C(1) typical(1.5) Day time(641) Success 12 59535
C(2) typical(1.5) Day time(211) Success 9 39989
Average 9.8 45821.6
TABLE 5: Experimental results for the manual mode.
4. CONSLUSION & FUTURE WORK
The proposed having meal support system with automatically controlled robot arm by human
eyes only is approximately 21% much faster than that with manually controlled robot arm. The
number of trials of the proposed system is about one third in comparison to the manually
controlled robot arm. The proposed system does work in a variety of illumination condition in the
day time and the night time and also work for several types of normal person with different eye
shapes and seeing capabilities. It is obvious that the proposed system has to be tested with
handicap persons in the near future.
5. REFERENCES
1. Arai K., Uwataki H., Computer input system by human eyes only with cornea center detection which
allows users’ movements, Journal of Institute of Electric and Electronics Society of Japan, 127-C, 7,
1107- 1114, 2007
2. Arai K., Yamaura M., Improvements of blink detection with morphologic filter in the computer input
system by human eyes only, Journal of Image and Electronics Society of Japan, 37, 5, 601-609, 2008
3. Arai K., Yajima K., Communication aids based on computer input system by human eyes only, Journal
of Institute of Electric and Electronics Society of Japan, 128-C, 11, 1679-1686, 2008
4. Yamaguchi A, Ito A., Kuroiwa S., Oikawa N., Matsuda T., Ishii S., The clinical application of meal
support robots. The 1 : Role of OT on the introduction to Guillain-Barre syndrome patients. The 2 :
Application of a meal robot to neuromuscular disease, Annual Report of the Research on Nervous and
Mental Disorders, 131-132, 2003
5. Martens, C., Prenzel, O., Gräser, A., "The Rehabilitation Robots FRIEND-I & II: Daily Life Independency
through Semi-Autonomous Task-Execution". I-Tech Education and Publishing (Vienna, Austria): 137–
162. ISBN 978-3-902613-04-2. http://intechweb.org/downloadpdf.php?id=556. 2007.
6. Lüth, T., Graimann, B., Valbuena, D., Cyriacks, M., Ojdanic, D., Prenzel, O., Gräser, A., A Brain-
Computer Interface for Controlling a Rehabilitation Robot". In BCI Meets Robotics: Challenging Issues
in Brain-Computer Interaction and Shared Control (Leuven, Belgium): 19–20, 2003.
7. http://www.secom.co.jp/personal/medical/myspoon.html
8. Koya Y., M.Oka, T.Akashi, Y.Wakasa, M.Tanaka, Meal feeding robot system with ultrasonic motor,
Proceedings of the SICE Chugoku-brunch of general assembly, 16, 116-117, 2007.
9. Kobayashi H., K.Konishi, K.Kikuchi, Development of feeding support system and its quantitative
estimation, Proceedings of the Robotics and Mecha-tronics of the Society of Machinery of Japan, 1A1-
H-50, 2004.

10. Paul Viola and Michael J. Jones. Rapid Object Detection using a Boosted Cascade of Simple Features.
IEEE CVPR, 2001.
11. Kohei Arai, Ronny Mardiyanto: Improvement of Gaze Estimation Robustness Using Pupil Knowledge.
Proceedings of the ICCSA (2) 2010: 336-350, (2010)
12. Intel Corp.: OpenCV (Intel Open. Computer Vision Library）,
http://www.intel.com/technology/computing/opencv/

Robot Arm Utilized Having Meal Support System Based on Computer Input by Human Eyes Only

More Related Content

Viewers also liked (20)

Similar to Robot Arm Utilized Having Meal Support System Based on Computer Input by Human Eyes Only (20)

More from Waqas Tariq (11)

Recently uploaded (20)

Robot Arm Utilized Having Meal Support System Based on Computer Input by Human Eyes Only