![Analysis of
Datasets
Rafsanjani
Muhammod
References Analysis of Datasets
[ Using Machine Learning Algorithms ]
Rafsanjani Muhammod
Undergrad Student, Department of Computer Science & Engineering
United International University, Bangladesh
April 16, 2017](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-1-320.jpg)
![Analysis of
Datasets
Rafsanjani
Muhammod
References
Dataset # 1 : KDD Cup 1999 Datasets III
Snapshot of Dataset # 1 :
This datasets have some problems :
Redundancy :
K. Leung et al.[5] observed that threre’re :
Around 78% “train data” are duplicant &
Around 75% “test data” are duplicant.](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-18-320.jpg)
![Analysis of
Datasets
Rafsanjani
Muhammod
References
Dataset # 1 : KDD Cup 1999 Datasets IV
Data Partitioning :
Portnoy et al.[3] divede this big data into 10 sub data.](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-19-320.jpg)
![Analysis of
Datasets
Rafsanjani
Muhammod
References
Dataset # 1 : KDD Cup 1999 Datasets V
Key Issues :
M. Tavallaee[4] observed :
1 Data redundancy
2 High accuracy rate
3 Highly imbalanced
Costly Cross-Validation :
Portnoy et al.[3] also observed that the distribution of this
data set(s) are very uneven which made cross-validation
difficult.
For avoid those kind of problem [4] proposed a
modified version of KDD Cup’99 datasets, is
known as NSL-KDD datasets.](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-20-320.jpg)
![Analysis of
Datasets
Rafsanjani
Muhammod
References
Dataset # 2 : NSL-KDD Datasets II
Brief Description :
Data Set Characteristics : Multivariate
Attribute Characteristics : Categorical,
Integer
Associated Tasks : Classification
Area : Computer Networking
Donar (with Date) : [4] (2009)
Instances : 125, 973
Features : 41 ( & class values : 23)
Missing Values : None](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-22-320.jpg)
![Analysis of
Datasets
Rafsanjani
Muhammod
References
Dataset # 2 : NSL-KDD Datasets IV
Details Results (70% train & 30%test) [using WEKA API] :](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-24-320.jpg)
![Analysis of
Datasets
Rafsanjani
Muhammod
References
Dataset # 2 : NSL-KDD Datasets VI
Details Results (10 Folds Cross-Validation) [using WEKA API] :](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-26-320.jpg)
![Analysis of
Datasets
Rafsanjani
Muhammod
References
Dataset # 2 : NSL-KDD Datasets IX
Data Visualization : [1][2] [ Using Orange3 ]](https://image.slidesharecdn.com/latex-170604051848/85/Analysis-of-the-Datasets-29-320.jpg)
Analysis of the Datasets
- 1. Analysis of Datasets Rafsanjani Muhammod References Analysis of Datasets [ Using Machine Learning Algorithms ] Rafsanjani Muhammod Undergrad Student, Department of Computer Science & Engineering United International University, Bangladesh April 16, 2017
- 2. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 3. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 4. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 5. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 6. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 7. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 8. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 9. Analysis of Datasets Rafsanjani Muhammod References Outlines Data Analysis Data Visualization Data Preprocessing Working Datasets Datasets Description Results Comparison between classifiers References
- 10. Analysis of Datasets Rafsanjani Muhammod References Name of Datasets KDD Cup 1999 Datasets NSL-KDD Datasets Hypothyroid Diseases Dataset Chronic Kidney Diseases Datasets Leaf Dataset
- 11. Analysis of Datasets Rafsanjani Muhammod References Name of Datasets KDD Cup 1999 Datasets NSL-KDD Datasets Hypothyroid Diseases Dataset Chronic Kidney Diseases Datasets Leaf Dataset
- 12. Analysis of Datasets Rafsanjani Muhammod References Name of Datasets KDD Cup 1999 Datasets NSL-KDD Datasets Hypothyroid Diseases Dataset Chronic Kidney Diseases Datasets Leaf Dataset
- 13. Analysis of Datasets Rafsanjani Muhammod References Name of Datasets KDD Cup 1999 Datasets NSL-KDD Datasets Hypothyroid Diseases Dataset Chronic Kidney Diseases Datasets Leaf Dataset
- 14. Analysis of Datasets Rafsanjani Muhammod References Name of Datasets KDD Cup 1999 Datasets NSL-KDD Datasets Hypothyroid Diseases Dataset Chronic Kidney Diseases Datasets Leaf Dataset
- 15. Analysis of Datasets Rafsanjani Muhammod References Hawk eyes view : Datasets Description Name of Dataset Instances Features Tasks KDD Cup ’99 4, 898, 431 41 (& 23) Classification NSL-KDD 125, 973 41 (& 23) Classification Hypothyroid 3772 29 (& 4) Classification Chronic Kidney 400 24 (& 2) Classification Leaf 340 16 (& 36) Classification
- 16. Analysis of Datasets Rafsanjani Muhammod References Dataset # 1 : KDD Cup 1999 Datasets I Problem Description : This is a “Computer Networks Intrusion Detection” problem. Brief Description : Data Set Characteristics : Multivariate Attribute Characteristics : Categorical, Integer Associated Tasks : Classification Area : Computer Networking Donar (with Date) : Unknown (January 1, 1991) Instances : 4, 898, 431 Features : 41 ( & class values : 23) Missing Values : None
- 17. Analysis of Datasets Rafsanjani Muhammod References Dataset # 1 : KDD Cup 1999 Datasets II There’re ‘4’ types of attacks : 1 Denial of Service Attack (DoS) 2 User to Root Attack (U2R) 3 Remote to Local Attack (R2L) 4 Probing Attack
- 18. Analysis of Datasets Rafsanjani Muhammod References Dataset # 1 : KDD Cup 1999 Datasets III Snapshot of Dataset # 1 : This datasets have some problems : Redundancy : K. Leung et al.[5] observed that threre’re : Around 78% “train data” are duplicant & Around 75% “test data” are duplicant.
- 19. Analysis of Datasets Rafsanjani Muhammod References Dataset # 1 : KDD Cup 1999 Datasets IV Data Partitioning : Portnoy et al.[3] divede this big data into 10 sub data.
- 20. Analysis of Datasets Rafsanjani Muhammod References Dataset # 1 : KDD Cup 1999 Datasets V Key Issues : M. Tavallaee[4] observed : 1 Data redundancy 2 High accuracy rate 3 Highly imbalanced Costly Cross-Validation : Portnoy et al.[3] also observed that the distribution of this data set(s) are very uneven which made cross-validation difficult. For avoid those kind of problem [4] proposed a modified version of KDD Cup’99 datasets, is known as NSL-KDD datasets.
- 21. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets I “NSL-KDD” datasets is a subset of “KDD Cup’99” datasets. Problem Description : This is also a “Computer Networks Intrusion Detection” problem.
- 22. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets II Brief Description : Data Set Characteristics : Multivariate Attribute Characteristics : Categorical, Integer Associated Tasks : Classification Area : Computer Networking Donar (with Date) : [4] (2009) Instances : 125, 973 Features : 41 ( & class values : 23) Missing Values : None
- 23. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets III Experiment Results (70% train & 30%test) : Classifiers vs Accuracy Graph (in R) Plotting with R : http://goo.gl/OpkU7e
- 24. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets IV Details Results (70% train & 30%test) [using WEKA API] :
- 25. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets V Experiment Results (10 Folds Cross-Validation) : Classifiers vs Accuracy Graph (in R) Plotting with R : https://goo.gl/4MVxwl
- 26. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets VI Details Results (10 Folds Cross-Validation) [using WEKA API] :
- 27. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets VII Now the question : which classifier is better & why ? OneR, use feature : “A5 src byte” which can classify 90% accurate. Important feature : “A5 src byte” is most important feature. Why “Naive Bayes” is poor : Beacuse : 1 Lack of independence variables. 2 “Naive Bayes classifier” since it is much more robust to overfitting. Ensure : I’m confident that DT related classifiers ensure more that 90% accuracy.
- 28. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets VIII ROC Curve (with Python) :
- 29. Analysis of Datasets Rafsanjani Muhammod References Dataset # 2 : NSL-KDD Datasets IX Data Visualization : [1][2] [ Using Orange3 ]
- 30. Analysis of Datasets Rafsanjani Muhammod References Analysis of Confusion Matrix :
- 31. Analysis of Datasets Rafsanjani Muhammod References References DM Farid et al. “Hybrid decision tree and “Naive Bayes” classifiers for multi-class classification tasks”. In: (2014). J. Han et al. “Data Mining : Concepts and Techniques”. In: (2012). L. Portnoy et al. “Intrusion detection with unbalanced data using clusters”. In: (). M. Tavallaee et al. “A Detailed Analysis of the KDD Cup ’99 data Set”. In: (2009). K. Leung and C. Leckie. “Unsupervised Anomaly Detection in Network Intrusion Detection Using Clusters”. In: ().
- 32. Analysis of Datasets Rafsanjani Muhammod References Questions & Answers Ask me.
- 33. Analysis of Datasets Rafsanjani Muhammod References Thankyou ! My LaTeX Template : https://goo.gl/tzFlD1