Decision Tree Algorithm With Example | Decision Tree In Machine Learning | Data Science |Simplilearn

Decision Tree Tutorial
I think I have to buy a car

How do I decide which
one to buy?

Is mileage > 20?
Is Price < 15 Lakhs?
Will it be sufficient for 6
people?
Number of airbags = 4
Anti-lock brakes?

This seems good

What’s in it for you?
What is Machine Learning?
What is Decision Tree?
Problems in Machine Learning
What are the problems a Decision Tree solves?
Advantages of Decision Tree
Disadvantages of Decision Tree
How does Decision Tree work?
Use Case – Loan repayment prediction
Types of Machine Learning

I wish I was smarter

Artificial Intelligence

I can think in
newer ways now

Learn
Analyze
Decide
Remember
Machine Learning
Recognize
Predict

Machine Learning is an application of Artificial Intelligence wherein the system gets the ability
to automatically learn and improve based on experience
Ordinary system

Ordinary system
With Artificial
Intelligence

Ordinary system
Ability to learn and improve on
its own

Ordinary system
Ability to learn and improve on
its own
Machine Learning

Supervised Learning

Supervised Learning Unsupervised Learning

Supervised Learning Unsupervised Learning Reinforcement Learning

Classification
Problems with categorical
solutions like ‘Yes’ or ‘No’,
‘True’ or ‘False’,’1’ or ‘0’

Classification Regression
Problems wherein
continuous value needs to
be predicted like ‘Product
Prices’, ‘Profit’

Classification
Regression Clustering
Problems wherein
Problems wherein the data
needs to be organized to
find specific patterns like
in the case of ‘Product
Recommendation’

Regression Clustering
Problems wherein
Problems wherein the data
needs to be organized to
find specific patterns like
in the case of ‘Product
Recommendation’
Classification

Classification
Decision Tree

Classification
Decision Tree
Random Forest
Logistic Regression
Naïve Bayes

Decision Tree is a tree shaped diagram used to determine a course of action. Each branch of the tree represents a
possible decision, occurrence or reaction

How do I Identify a random
vegetable from a shopping
bag?

Which Vegetable?
Is color = red?
False True
diameter > 2
False True

Which Vegetable?
So it’s a
capsicum
Is color = red?
False True
diameter > 2
False True

Problems that Decision Tree can solve

RegressionClassification

A classification tree will determine a set
of logical if-then conditions to classify
problems.
For example, discriminating between
three types of flowers based on certain
features

Regression tree is used when the
target variable is numerical or
continuous in nature. We fit a
regression model to the target
variable using each of the
independent variables. Each split
is made based on the sum of
squared error.
A classification tree will determine a set
of logical if-then conditions to classify
problems.
For example, discriminating between
three types of flowers based on certain
features

Simple to
understand,
interpret and
visualize

Little effort required
for data preparation

Can handle both
numerical and
categorical data

Non linear
parameters don’t
effect its
performance

Overfitting
Map
Overfitting occurs when
the algorithm captures
noise in the data

noise in the data
Overfitting
High
Variance
Map
The model can get
unstable due to small
variation in data

noise in the data
The model can get
unstable due to small
variation in data
Overfitting
High
Variance
Low biased
Tree
Map
A highly complicated Decision
tree tends to have a low bias
which makes it difficult for the
model to work with new data

Decision Tree – Important Terms

ImportantTerms

Entropy Example
Entropy is the measure of
randomness or
unpredictability in the
dataset
This Dataset has a very
high entropy
High entropy

Entropy Example
Entropy is the measure of
randomness or
unpredictability in the
dataset
High entropy(E1)
Color == yellow?
True False
Height=10? Height<10?
True TrueFalse False
Lower entropy(E2)
zero entropy
After split

Information gain Example
High entropy(E1)
Color == yellow?
True False
Lower entropy(E2)
Gain = E1 - E2
After split
It is the measure of decrease
in entropy after the dataset
is split

Leaf Node Example
Leaf Node
Color == yellow?
True False
Leaf node carries the
classification or the decision

Decision Node Example
decision Node
Color == yellow?
True False
Decision node has two or
more branches

Root Node Example
Root Node
Color == yellow?
True False
The top most Decision node
is known as the Root node

Wonder what kind of animals I’ll get
in the jungle today
How does a Decision Tree work?

Let’s try to classify different types of
animals based on their features using a
DecisionTree

Problem statement
To classify the different types of animals
based on their features using decision tree

The dataset is looking quite messy and
the entropy is high in this case
Problem statement
To classify the different types of animals
based on their features using decision tree

The dataset is looking quite messy and
the entropy is high in this case
Training Dataset
Color Height Label
grey
Yellow
brown
grey
Yellow
10
3
10
10
4
elephant
elephant
giraffe
Tiger
Monkey

How to split the data
We have to frame the conditions that split
the data in such a way that the
information gain is the highest

How to split the data
We have to frame the conditions that split
the data in such a way that the
information gain is the highest
Note
Gain is the measure of
decrease in entropy after
splitting

𝑖=1
𝑘
𝑃(𝑣𝑎𝑙𝑢𝑒𝑖). 𝑙𝑜𝑔2(𝑃 𝑣𝑎𝑙𝑢𝑒𝑖
Formula for entropy
Let’s try to calculate the entropy
for the current dataset

total
3
2
1
2
8

𝑖=1
𝑘
Let’s use the formula

𝑖=1
𝑘
Entropy = (
3
8
) 𝑙𝑜𝑔2(
3
8
) + (
2
8
) 𝑙𝑜𝑔2(
2
8
) + (
1
8
) 𝑙𝑜𝑔2(
1
8
)+(
2
8
) 𝑙𝑜𝑔2(
2
8
) Entropy=0.571

𝑖=1
𝑘
Entropy = (
3
8
) 𝑙𝑜𝑔2(
3
8
) + (
2
8
) 𝑙𝑜𝑔2(
2
8
) + (
1
8
) 𝑙𝑜𝑔2(
1
8
)+(
2
8
) 𝑙𝑜𝑔2(
2
8
) Entropy=0.571
We will calculate the entropy
of the dataset similarly after
every split to calculate the
gain

Gain can be calculated by
finding the difference of the
subsequent entropy values
after split

Now we will try to choose a
condition that gives us the
highest gain

Now we will try to choose a
condition that gives us the
highest gain
We will do that by splitting
the data using each condition
and checking the gain that
we get out them.

We will do that by splitting
the data using each condition
and checking the gain that
we get out them.
The condition that gives us
the highest gain will be used
to make the first split

Training Dataset
Color Height Label
grey
Yellow
brown
grey
Yellow
10
3
10
10
4
elephant
elephant
giraffe
Tiger
Monkey
Conditions
Color==Yellow?
Height>=10
Color== Brown?
Color==Grey
Diameter<10

Let’s say this condition gives us the
maximum gain
Conditions
Color==Yellow?
Height>=10
Color== Brown?
Color==Grey
Diameter<10
Training Dataset
Color Height Label
grey
Yellow
brown
grey
Yellow
10
3
10
10
4
elephant
elephant
giraffe
Tiger
Monkey

We split the data
Color == yellow?
True False

The entropy after splitting has
decreased considerably
Color == yellow?
True False

The entropy after splitting has
decreased considerablyhowever we still need
some splitting at both
the branches to attain an
entropy value equal to
zero
Color == yellow?
True False

So, we decide to split
both the nodes using
‘height’ as the condition
Color == yellow?
True False
Height>=10? Height<10?

since every branch now contains
single label type, we can say that
the entropy in this case has
reached the least value
Color == yellow?
True False

Color == yellow?
True False
ThisTree can now predict all the
classes of animals present in the
dataset with 100% accuracy

Color == yellow?
True False
ThisTree can now predict all the
classes of animals present in the
dataset with 100% accuracy
That was easy

Use Case – Loan Repayment Prediction

Use Case – Loan Repayment prediction
I need to find out if my customers are
going to return the loan they took
from my bank or not

Use Case – Problem Statement
Problem statement
To predict if a customer will repay loan
amount or not using DecisionTree
algorithm in python

Use Case – Implementation
#import the necessary packages
import numpy as np
import pandas as pd
from sklearn.cross_validation import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from sklearn import tree
#Loading data file
balance_data =pd.read_csv('C:/Users/anirban.dey/Desktop/data_2.csv',
sep= ',', header= 0)

#import the necessary packages
print ("Dataset Lenght:: "), len(balance_data)
print ("Dataset Shape:: "), balance_data.shape

print ("Dataset:: ")
balance_data.head()

#Seperating the Target variable
X = balance_data.values[:, 1:5]
Y = balance_data.values[:,0]
#Spliting Dataset into Test and Train
X_train, X_test, y_train, y_test = train_test_split( X, Y, test_size = 0.3,
random_state = 100)
#Function to perform training with Entropy
clf_entropy = DecisionTreeClassifier(criterion = "entropy", random_state = 100,
max_depth=3, min_samples_leaf=5)
clf_entropy.fit(X_train, y_train)

#Function to make Predictions
y_pred_en = clf_entropy.predict(X_test)
y_pred_en

#Checking Accuracy
print ("Accuracy is "), accuracy_score(y_test,y_pred)*100

Use Case
So, we have created a model that uses
decision tree algorithm to predict
whether a customer will repay the loan
or not

Use Case
The Accuracy of the model is 94.6%

Use Case
The bank can use this model to decide
whether it should approve loan request
from a particular customer or not

Decision Tree Algorithm With Example | Decision Tree In Machine Learning | Data Science |Simplilearn

Recommended

More Related Content

What's hot (20)

Similar to Decision Tree Algorithm With Example | Decision Tree In Machine Learning | Data Science |Simplilearn (20)

More from Simplilearn (20)

Recently uploaded (20)

Decision Tree Algorithm With Example | Decision Tree In Machine Learning | Data Science |Simplilearn

Editor's Notes