Human Activity Recognition using Smartphone's sensor

HUMAN ACTIVITY RECOGNITION WITH SMARTPHONE
1
HUMAN ACTIVITY RECOGNITION
WITH SMARTPHONE
Submitted in partial fulfillment of the requirements for
the award degree of
Summer internship program
in
INDIAN INSTITUTE OF TECHNOLOGY (BHU)
Department of computer science
Guide: Submitted by:
Dr. Hari Prabhat Gupta Pankaj Kumar Mishra
Assistant professor
Computer Sc. Dept.

2
ACKNOWLEDGEMENT
The success and final outcome of this project “HUMAN ACTIVITY RECOGNITION
WITH SMARTPHONE” required a lot of guidance and assistance from many people and
I am extremely privileged to have got this all along the completion of my project. All that
I have done is only due to such supervision and assistance and I would not forget to thank
them.
I respect and thank Mr. Hari Prabhat Gupta, assistant professor department of Computer
Science for providing me an opportunity to do the project work in IIT BHU and giving us
all support and guidance which made me complete the project duly. I am extremely
thankful to him for providing such a nice support and guidance, although he had busy
schedule managing the college affairs. I owe my deep gratitude to our project guide, who
took keen interest on our project work and guided us all along, till the completion of our
project work by providing all the necessary information for developing a good system.
I am thankful to and fortunate enough to get constant encouragement, support and
guidance from all Teaching staffs of [Computer Science] which helped us in successfully
completing our project work. Also, I would like to extend our sincere esteems to all staff
in laboratory for their timely support.

3
Contents
Acknowledgement ………………………………………… ………………………….2
Abstract ………………………………………………………………………………...4
Objective ……………………………………………………………………………….5
Introduction …………………………………………………………………………….6
Motivation ………………………………………………………………………………7
Literature Review ……………………………………………………………………....8
Background …………………………………………………………………………… 9-12
i.Sensor Activity …………………………………………………………………….9
ii.Accelerometer …………………………………………………………………….10
iii.Gyroscope ………………………………………………………………………….11
iv.Wearable Sensor ……………………………………………………………………12-14
Machine Learning …………………………………………………………………….15
Contribution ………………………………………………………………………….16-21
Decision Tree …………………………………………………………………………22-23
Confusion Matrix ………………………………………………………………………24
K-Nearest neighbor …………………………………………………………………….25-27
Conclusion ……………………………………………………………………………..28
References ……………………………………………………………………………..29

4
1. ABSTRACT
Human activity recognition plays significant role in medical field and in security system. In this
project we have design a model which recognize a person’s activity based on Smartphone.
A 3- dimensional Smartphone sensor named accelerometer and gyroscope is used to collect time
series signal, from which 26 features are generated in time and frequency domain. The activities
are classified using 2 different dormant learning method i.e. k-nearest neighbor algorithm,
decision tree algorithm.
We also applied various active machine learning algorithms in order to reduce data labeling
expenses. To make a model successful its working accuracy must be much better and in this
experiment the classification rate of dormant learning is 90.3% which is substantial for the
common position and state of Smartphone, in the first classification rate which was taken by the
decision tree we have calculated the accuracy about 85.67% which further improved by the next
classifier K- nearest neighbor.
To achieve a comparable performance with dormant learning, reduction in labeling labor is
needed. And this reduction demonstrates by result of active learning on real data.

5
2. OBJECTIVE
Since there are many sensors are inbuilt in our smart phone to measure our position, movements
and orientation and because of these sensors the improvements in daily life of human increases.
Main objective of our project is to recognize the human’s activities by analyzing the mobile
phone’s sensor data. More specifically, we have to make a model which can predict or accurately
classifies whether a person is performing the action of laying, walking, walking upstairs, walking
downstairs, sitting or standing only on the basis of mobile phone sensor data.
Using sensor data obtained from study participants performing six different activities (walking,
walking upstairs, walking downstairs, sitting, standing and laying), our objective is to build a
model that accurately classifies which of these activities is being performed.
This human activity recognition proposes many application and several benefits. This mobile
based health application can be beneficial for the elderly or senior assistance. Also we can use
this application for the personal health monitoring because mobile will be attached with us and
the application tracks our activity overtime.
Our project falls into the scope of Activity Recognition, a field that offers many benefits and
enables many new applications, for example step counters on your Smartphone, as well as
applications for elderly assistance and personal health monitoring.

6
3. INTRODUCTION
the demand of recognizing human activities have grown basically in the domain of health,
basically in taking care of senior assistances and cognitive disorders persons. We can save a lot
of resources and time if we are analyzing the activity of any patients or his abnormal behavior
through these mobile sensors.
Instead of health domain there are so many other applications which are also performing by IoT
devices like security issue. But these applications could be performed in surveillance camera. In
this scenario we make a camera much advance so that it can define a restricted zone and trace the
objects under its focus. These objects can be anything like an animal, human or box etc. and if
any stranger or any box remains it’s a fixed position for certain time then we will get informed or
it will inform security guards.
In many studies it is found that the wearable sensors have predicted activity at very low error
rate. This project uses only easily available and low cost sensors to recognize the human activity.
In today’s teens mobile phone is ubiquitous device and its computational quality is much better
which make it ideal for non-intrusive body attached sensors.
Today about 95% mobile phones have in built sensors like accelerometer and gyroscope. In
research found that gyroscope can help in activity recognition, but contribution in alone for it is
not as good as accelerometer. Because any Smartphone can easily accessed but gyroscope can’t.
In our design Smartphone can be placed anywhere around waist such as jacket pocket or pant
pocket.
Whenever any new activity is added to the system we need to train entire system. Because of
variance in sensors, if algorithms run on different device, the parameters of algorithms need to
get trained. We propose active learning process to accelerate the training process because
labeling a time series data is time consuming process and it is not possible to give label for all
the training data. Given a classifier, active learning intelligently queries the unlabelled samples
and learns parameters from the correct label. In this manner user do label only the samples that
the algorithm asks for the total amount of required training samples reduced.
The goal of this project is to make a model on Smartphone that can easily recognize the human
activity. Moreover active learning models are developed in order to reduce labeling time and
burden. Through testing and comparing with different learning algorithms, we find one best fit
model for our system.

7
4. MOTIVATION
By understanding human action and their interaction with the surrounding is a key element for
the development of aforementioned intelligent system. Human activity recognition is field that
deals with the problems generates in the integration of sensing and reasoning, to provide context-
aware data that can confer the personalized support across an application.
For example, if we imagine a smart home which is equipped with various sensors and devices
and which is able to detect the working of all the appliances and presence of people in home. It is
possible to deduce the various activities performed by a person inside the home based on the
sensors signal with other relevant factors like time domain and date ( like a person in morning is
supposed to be in kitchen and coffee machine suggests that person is making breakfast). That’s
why the collected HAR can be absorbed to anticipate future people demands and can be
responsive for their purpose (example- automatic temperature set, automatic controlling of light
etc).
In the Human Activity Recognition system, still there are various issues which need to be
addressed like as battery limitation of wearable sensors, privacy concern regarding continuous
monitoring of activities, offhandedness of the wearable sensors, difficulty in performing HAR
(HUMAN ACTIVITY RECOGNITION) in real time and lack of fully ambient systems able to
reach users at any time.

8
5. LITERATURE REVIEW
Human activity recognition been studied by many researchers and each have proposed different
solution to tackle the problems regarding this. To tackle with this problem typically they used
vision sensors, inertial sensors or both. Machine learning and threshold learning are often used.
The basic difference in machine learning and threshold learning is that- machine learning
produces result more accurate and reliable while threshold learning is faster and simpler.
To recognize the activity or to accomplish the HAR task, one or multiple cameras have been
used to capture the body postures or multiple accelerometer and gyroscope sensors have been
attached in body are the most common solution. Also various approaches came in scene in which
both camera and sensors have been used. After getting dataset by using these IoT devices another
essential part is data preprocessing. Because quality of input have more impact on the desired
result. To get a good and high accuracy we need to have a good and quality input data.
Various machine learning problems that are time consuming and labor- expensive are being
solved by active learning technique. There are some applications which include speech
recognition, information extraction or hand writing recognition etc. This technique have also
been used in human activity recognition before face recognition points used by FBI in
surveillance.

9
6. BACKGROUND
6.1 Sensing Activity:
Although there are so many sensors which are generally used in HAR and they
measure different attributes including vital signs (e.g. heart rate, body temperature
and blood pressure), motion (e.g. acceleration, speed) and environmental signal
(light intensity and environment) but to choose a right sensor we need to have
consider first element for the design of HAR system.
Figure 1
Sensor mechanism is classified on the basis of sensor placement with respect to the user.
If the sensor is located in environment then it is ambient and if the sensor is attached
with user’s body then it is wearable.

10
6.2 Accelerometer
The accelerometer is an instrument that measures the experienced physical
acceleration of an object. It is generally used for the measurement in application like
as vibration in machinery, acceleration in high speed vehicle and high loaded bridges
etc.
Figure 2
Its principle of operation generally consists of a seismic mass which is displaced in
relation to the acceleration it is subject to. The displacement can then be transuded into a
measurable electrical signal. The phenomenon has been applied for the development of
micro electromechanical system (MEMS) sensors. Their technology allows creating
nano-sacle devices fabricated with semiconductors. Most common MEMS

11
accelerometers works as capacitive sensors composed of a cantilever beam with a proof
mass whose deflection is correlated with the acceleration experienced by the sensors.
 Accelerometer data plot:
Figure 3
6.3 Gyroscope
Gyroscope is a sensor that can provide orientation information as well, but with
greater precision. It is generally used in many applications like as inertial navigation
systems, aerial vehicle for stability augmentation and now-a-days it has been used in
electronic device (e.g. Smartphone, game console etc). For HAR, this sensor has been

12
used for various tasks such as for the detection of various transitions between various
postures (e.g. in sitting and standing) and for the detection of activities.
Figure 4
Gyroscopes have also been manufactured with MEMS technologies. However we can
measure only orientation by this sensor. However it is required first to have a reference
initial angular position to achieve this.
 Gyroscope scatter plot
Figure 5
6.4 Smartphone as wearable sensor

13
A wearable technology suggests all the devices which can bind with our body in
order to process information from the users and their interaction with our
environment. In this project we have selected Smartphone as a wearable device
because there are many sensors are inbuilt into it. And we are using accelerometer
and gyroscope sensor to detect the activity of a body which is also inbuilt in a
Smartphone.
Figure 6
Using a Smartphone as a wearable device we easily get information of a user’s linear
acceleration and angular velocity. The information will not be highly affected by the
accelerometer and gyroscope by the bad indoor signal of GPS and electromagnetic noise
in compass. However, accelerometer measurements are always influenced by the gravity
factor in the detection of acceleration of a moving body.

14
Figure 7

15
6.5 Machine Learning
Machine learning is an application of artificial intelligence (AI) that provides systems
the ability to automatically learn and improve from experience without being
explicitly programmed. Machine learning focuses on the development of computer
programs that can access data and use it learn for themselves.
In the area of machine learning we study about the design, development and
evaluation of system capable to learn from the data. In this field, to complete a task or
to complete a prediction we need to have old observation. So machine learning is
capable to predict future observation on the basis of past experiences.
Machine learning algorithms have been categorized according to the type of input
used for training and its expected outcome.
 Supervised learning
Supervised learning algorithm generates a function which maps input in desired
outputs. It is called supervised learning because the process of algorithm learning
from the training dataset can be thought of as a teacher supervising the learning
process.
Supervised learning is where you have input variables (x) and an output variable
(Y) and you use an algorithm to learn the mapping function from the input to the
output.
Y = f(X)
The goal is to approximate the mapping function so well that when you have new
input data (x) that you can predict the output variables (Y) for that data.
 Unsupervised Learning
Unsupervised learning which model a set of inputs; labeled examples are not
available.
 Semisupervised learning

16
Semi- supervised learning ---which combines both to generate an appropriate
function or classifier.
 Reinforcement Learning
Reinforcement learning is an important type of Machine Learning where an agent
learns how to behave in a environment by performing actions and seeing the
results.
7.CONTRIBUTION
7.1 Sensing and data collection
The main aspect in HAR is the experimental set up for the data acquisition. It’s totally
depending on the vision means how the subject is observed without any expedient by the
observer. For any experiment naturalistic environment is ideal but it also get failed in
many circumstances. So the controlled experiment should be performed in laboratory
condition.
Failure of the design of the HAR system can be because of the lack of real life experience
like the activities which has been not counted in the experiment and sensor calibration or
positioning etc.

17
Figure 8
This latter is for the instances linked perfectly to the system in which many sensors are
evaluated to determine the perfect position to accomplish HAR experiment through the
use of wearable sensors (accelerometer and gyroscope).
7.2 Experimental data collection
Data from the accelerometer has the following attributes: time, acceleration along x-axis,
acceleration along y-axis and acceleration along z-axis. This data is collected from 30
users. Collected accelerometer data every 20 second, so there will be 20 samples per
second.
We used decision tree and K-nearest neighbor algorithm to analyze the raw dataset which
has been recorded by the mobile phone. To compare the activity pattern with training data

18
we used sequential comparing in every calculating the distance to record the acceleration
data points.
 Importing data
Figure 9
 Raw data
Figure 10

19
7.3 Feature generation
For the collection of accelerometer data, a group of 30 members performed some activities
(standing, sitting, laying, walking, walking downstairs, and walking upstairs). The
position of the phone can be anywhere close to the waist.
Due to instability of phone sensor which may drop samples accidently, interpolation
method is applied to fill the gaps.
Also there were total 256 features were provided to us in dataset but we select only few of
them. We have selected 31features.
These features were selected by the two methods:
1. Embedded method
Embedded method finds the optimal feature subset during model training.
2. RFECV (wrapper method)
Also no scaling is needed since all the values are already normalized within the range
between -1 and 1.

20
Figure 11
To complete this activity, we group 200 samples in a window, which corresponds
to 10 second data which is power of two, is a preferred size when applying fast
Fourier transformation.

21
Total 31 features are extracted in both time and frequency domain. These features
are generated on the basis of 200 raw readings. Each reading contains X, Y and Z
values corresponding to the three axes.
For this purpose,
Z-axis - capture forward movement of the leg
Y-axis – capture upward and downward motion
X-axis – capture horizontal movement of the leg
7.4 Feature selection and extraction
In order to prevent from high complexity and over fitting problem we need to choose only
some of the given features. So the meaning of feature selection is to select a significant set
of features which impact largely on the learning ability of machine learning algorithms.
And feature extraction refers to the process through which we diminish the dimensionality
of the available set of feature. For this we perform inter- feature transform to get a new
dimensionally reduced representation.
There are several feature selection methods,
(1) Feature subset selection,
(2) PCA,
And among feature subset selection methods are filter method, wrapper method, and
embedded method.
 Feature importance:
Figure 12

22
7.5 Induce machine learning models
Once the dataset is prepared following two classification techniques were used to induce
to model to predict user’s activity.
 Decision tree
 K- nearest neighbor
7.5.1 Decision tree
Decision tree is an important algorithm for predictive modeling and can be used
to visually and explicitly represent decisions. It is a graphical representation that
makes use of branching methodology to exemplify all possible outcomes based
on certain conditions. In decision tree internal node represents a test on the
attribute, branch depicts the outcome and leaf represents decision made after
computing attribute.
It can be classified into two types, Classification trees which are used to separate
a dataset into different classes based on the particular basis and generally used
when we expect response variable in categorical nature. The other type is called
Regression Trees which are used when the response variable is continuous or
numerical.
Decision Tree Classifier is a simple and widely used classification technique. It
applies a straightforward idea to solve the classification problem.
Figure 13

23
7.5.2 Extension to multiclass decision tree
In machine learning, multiclass or multinomial classification is the problem
of classifying instances into one of three or more classes. Decision trees are a
powerful classification technique. The tree tries to infer a split of the training data
based on the values of the available features to produce a good generalization.
The algorithm can naturally handle binary or multiclass classification problems.
The leaf nodes can refer to either of the K classes concerned. There are several
method are proposed to solve multiclass problems from binary formulations.
Generally two methods are used:
1. OVA (one-vs.-all), 2. OVO (one-vs.-one)
The OVA approach consists on constructing a set of m binary decision tree,
each one existing to a class c. They are built from positive training sample
coming from the class of interest (labeled as +1) and negative samples
(labeled as -1). Once the classifier learned, it is possible to compare them to
determine which class is most likely to represent a test sample.
Figure 14

24
7.5.3 Performance evaluation
The complete evaluation of machine learning model is mainly a statistical analysis
using the analysis of available data. And confusion matrix is the main common
method through which performance of an algorithm can be represent by clearly
identifying the type errors (false positive and negative). Through the decision tree
various other matrices can also be extracted like as F-1 score, precision, sensitivity
and accuracy etc.
7.5.4 Confusion matrix
Through this classifier we got accuracy 85.74 %. To visualize performance of a
classifier we need to represent a confusion matrix. Assuming there are m classes
available, a typical confusion matrix consists of a squared matrix of , where
misclassification is outside of diagonal.
Figure 15

25
True Positive (TP): actual samples of class a correctly predicted as class a.
True Negative (TN): actual samples of class b correctly predicted as class b.
False Positive (FP): actual samples of class b incorrectly predicted as class a.
False Negative (FN): actual samples of class a incorrectly predicted as class b.
Note: vertical classes are true classes and horizontal classes are predicted classes.
7.5.5 K-Nearest neighbor (K-NN)
The KNN algorithm is a robust and versatile classifier that is often used as a
benchmark for more complex classifiers such as Artificial Neural Networks (ANN)
and Support Vector Machines (SVM). Since KNN algorithm is very simple but it is
used in variety applications such as in data compression and economic forecasting
because this algorithm is capable in outperforming more complex classifiers.
KNN falls in the supervised learning family of algorithms. Informally, this means
that we are given a labeled dataset consisting of training observations (x, y) and
would like to capture the relationship between x and y. More formally, our goal is to
learn a function h: X→Y so that given an unseen observation x, h(x) can confidently
predict the corresponding output y.
HowdoesKNNwork?
In the classification setting, the K-nearest neighbor algorithm essentially boils down to
forming a majority vote between the K most similar instances to a given “unseen”
observation. Similarity is defined according to a distance metric between two data points.
A popular choice is the Euclidean distance given by:
 MoreonK
At this point, you’re probably wondering how to pick the variable K and what its effects
are on your classifier. Well, like most machine learning algorithms, the K in KNN is a
hyper parameter that you, as a designer, must pick in order to get the best possible fit for
the data set. Intuitively, you can think of K as controlling the shape of the decision
boundary we talked about earlier.

26
When K is small, we are restraining the region of a given prediction and forcing our classifier to
be “more blind” to the overall distribution. A small value for K provides the most flexible fit..
Larger values of K will have smoother decision boundaries increased.

27
7.5.6 Performance evaluation
The complete evaluation of machine learning model is mainly a statistical analysis
using the analysis of available data. And confusion matrix is the main common
method through which performance of an algorithm can be represent by clearly
identifying the type errors (false positive and negative). Through the decision tree
various other matrices can also be extracted like as F-1 score, precision, sensitivity
and accuracy etc.
7.5.7 Confusion matrix
Through this classifier we got accuracy 90.77 %. To visualize performance of a
classifier we need to represent a confusion matrix. Assuming there are m classes
available, a typical confusion matrix consists of a squared matrix of , where
misclassification is outside of diagonal.

28
Conclusions
Human activity recognition has broad application in medical research and human survey system.
In this project, we designed a Smartphone based recognition system. Through this project we can
recognize overall 6 activities (standing, sitting, walking, walking upstairs, walking downstairs
and laying). The system collected time series signals using a built-in accelerometer, generated 26
features in both time and frequency domain, and the features are get selected in order to improve
the efficiency (performance). The activity data were trained and tested with the two classifiers
which are: Decision tree and K-nearest neighbor algorithm.
The best classification rate in our experiment is 90.77% attained by the K-nearest neighbor
algorithm. This classification attained by the K-nearest neighbor is robust in orientation and the
position of Smartphone. Better accuracy can be achieved anywhere near the waist.
Among the classifier K-NN and Decision tree improve most applying active learning. The result
demonstrates entropy and distance to the boundary are robust to the uncertainty measures when
performing queries on Decision tree and K- nearest neighbor. K-nearest neighbor is our optimal
solution for our problem.
In future work we can consider more activities and can develop a real time system on
Smartphone. Also we can take more features in order to have better accuracy in our model and
we can develop density-weighted methods and variance reduction to solve other query strategies.

29
References
[1]https://www.google.co.in/search?rlz=1C1CHBF_enIN779IN779&biw=1366&bih=662&tbm=i
sch&sa=1&ei=sYebW7DaNomA8QXU9KawCg&q=nearest+neighbor+k%3D20&oq=nearest+ne
ighbor+k%3D20&gs_l=img.3...4019.4209.0.5302.2.2.0.0.0.0.201.201.2-
1.1.0....0...1c.1.64.img..1.0.0....0.JTp8h1O65-8#imgrc=eCJUhalfq1IvqM:
[2] https://en.wikipedia.org/wiki/Euclidean_distance
[3] https://kevinzakka.github.io/2016/07/13/k-nearest-neighbor/
[4] https://www.geeksforgeeks.org/decision-tree-introduction-example/
[5]https://www.google.co.in/search?q=decision+tree+algorithm&rlz=1C1CHBF_enIN779IN779
&source=lnms&tbm=isch&sa=X&ved=0ahUKEwiFxf6rqbrdAhWLvLwKHQM7DhcQ_AUIDig
B&biw=1366&bih=613
[6] https://www.kaggle.com/uciml/human-activity-recognition-with-smartphones
[7] https://www.solver.com/k-nearest-neighbors-k-nn-prediction

Human Activity Recognition using Smartphone's sensor

More Related Content

What's hot (20)

Similar to Human Activity Recognition using Smartphone's sensor (20)

Recently uploaded (20)

Human Activity Recognition using Smartphone's sensor