Machine Learning Algorithm & Anomaly detection 2021
0 likes160 views
This document discusses various techniques for anomaly detection using machine learning, including: 1. It introduces machine learning and anomaly detection. 2. It covers different machine learning techniques for anomaly detection like genetic algorithms, Monte Carlo simulation, reinforcement learning, and generative adversarial networks. 3. It describes different types of anomalies like point, contextual, and collective anomalies. It also discusses techniques for identifying outliers. 4. Specific anomaly detection techniques are discussed, including recurrent neural networks, historical analysis with DBSCAN clustering, time shift detection, and text pattern anomaly detection using techniques like cosine similarity and autoencoders.
Machine Learning Algorithm & Anomaly detection 2021
- 7. What is Machine Learning? https://www.seas.upenn.edu/~cis519/fall2017/lectures/01_introduction.pdf Computer Computer Data Program Output Data Output Algorithm
- 9. Agenda • Part I Machine learning technique • Introduction to Machine Learning • Genetic Algorithm • Monte Calo • Reinforcement Learning • Generative Adversarial Networks • Part II Anomaly Detection technique • Type of Anomaly • RNN • Historical • DB-SCAN • Time Shift Detection • Text Pattern Anomaly Detection
- 11. Charles Darwin
- 15. Monte Calo https://colab.research.google.com/drive/11gw_ccZp8hGE09AUHuW4Zg_k3w7jDpmb#scrollTo=mQrx1Xs96kp7 บริษัทต้องขนส่งอย่างไรเพื5อให้เกิด คชจ. น้อยที5สุด กําลังการผลิต (Supply) โรงงานมีนบุรี 4,000 ชิ,น โรงงานบางปู 9,000 ชิ,น โรงงานอยุธยา 2,000 ชิ,น ปริมาณความต้องการของลูกค้า (Demand) คลังภาคเหนือ 4,000 ชิ,น คลังภาคตะวันออกเฉียงเหนือ 6,000 ชิ,น คลังภาคใต้ 2,000 ชิ,น คลังภาคตะวันออก 3,000 ชิ,น ปลายทาง (บาทต่อชิ:น) ภาคเหนือ ภาคตะวันออกเฉียงเหนือ ภาคใต้ ภาคตะวันออก โรงงานมีนบุรี 70 50 50 10 โรงงานบางปู 90 70 50 20 โรงงานอยุธยา 40 40 60 40
- 16. Monte Calo ปลายทาง (บาทต่อเครื>อง) ภาคเหนือ ภาคตะวันออกเฉียงเหนือ ภาคใต้ ภาคตะวันออก โรงงานมีนบุรี 1,000x70 3,000x50 0x50 0x10 โรงงานบางปู 1,000x90 3,000x70 2,000x50 3,000x20 โรงงาน อยุธยา 2,000x40 0x40 0x60 0x40 รวม 240,000 360,000 100,000 60,000 760,000 Optimize Value
- 18. Optimization Technique üDynamic Programming üMonti Calo üReinforcement Learning
- 21. Gym Cartpole with Deep Q-learning
- 25. Generative adversarial network https://arxiv.org/pdf/1708.05509.pdf Towards the Automatic Anime Characters Creation
- 26. Generative adversarial network https://arxiv.org/abs/1703.10593 Unpaired Image-to-Image Translation
- 27. Generative adversarial network Face Frontal View Generation Rui Huang, et al. in their 2017 paper titled “Beyond Face Rotation: Global and Local Perception GAN for Photorealistic and Identity Preserving Frontal View Synthesis”
- 28. Generative adversarial network DeblurGAN helps object detection. YOLO detections on the blurred image (top), DeblurGAN restored image (middle) and the sharp ground truth image from the GoPro dataset (bottom) Super Resolution using SRGANs taken from the paper, “Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network”
- 31. Question ?
- 33. Type Of Anomaly
- 35. Anomalies can be broadly categorized as 39 1. Point Anomalies 2. Contextual Anomalies 3. Collective Anomalies If an individual data instance can be considered as anomalous with respect to the rest of data, then the instance is termed as a point anomaly. Chandola, V., Banerjee, A. & Kumar, V., 2009. Anomaly Detection: A Survey. ACM Computing Surveys, July. 41(3).
- 36. Anomalies can be broadly categorized as 40 1. Point Anomalies 2. Contextual Anomalies 3. Collective Anomalies If a data instance is anomalous in a specific context (but not otherwise), then it is termed as a contextual anomaly Chandola, V., Banerjee, A. & Kumar, V., 2009. Anomaly Detection: A Survey. ACM Computing Surveys, July. 41(3).
- 37. Type Of Anomaly
- 38. Anomalies can be broadly categorized as 1. Point Anomalies 2. Contextual Anomalies 3. Collective Anomalies Chandola, V., Banerjee, A. & Kumar, V., 2009. Anomaly Detection: A Survey. ACM Computing Surveys, July. 41(3). If a collection of related data instances is anomalous with respect to the entire data set, it is termed as a collective anomaly
- 39. How to Identify Outliers in your Data 43 1. Extreme Value Analysis 2. Proximity Methods 3. Projection Methods 4. Methods Robust to Outliers https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/
- 40. How to Identify Outliers in your Data 1. Extreme Value Analysis 2. Proximity Methods 3. Projection Methods 4. Methods Robust to Outliers https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/ 1. Focus on univariate methods 2. Visualize the data using scatterplots, histograms and box and whisker plots and look for extreme values 3. Assume a distribution (Gaussian) and look for values more than 2 or 3 standard deviations from the mean or 1.5 times from the first or third quartile 4. Filter out outliers candidate from training dataset and assess your models performance https://en.wikipedia.org/wiki/Multivariate_normal_distribution
- 41. How to Identify Outliers in your Data 1. Extreme Value Analysis 2. Proximity Methods 3. Projection Methods 4. Methods Robust to Outliers https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/
- 42. How to Identify Outliers in your Data 1. Extreme Value Analysis 2. Proximity Methods 3. Projection Methods 4. Methods Robust to Outliers https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/
- 43. How to Identify Outliers in your Data 47 1. Extreme Value Analysis 2. Proximity Methods 3. Projection Methods 4. Methods Robust to Outliers https://machinelearningmastery.com/how-to-identify-outliers-in-your-data/
- 44. Anomaly based IDS using Backpropagation Neural Network https://pdfs.semanticscholar.org/3ac9/37cb50bad238091fba6d1076a78136fe8691.pdf • Vrushali D. Mane ME (Electronics) JNEC, Aurangabad • S.N. Pawar Associate Professor JNEC, Aurangabad • Neural Network detect 98% accuracy
- 45. Recurrent neural network (RNN)
- 46. Recurrent neural network (GRU) n=24
- 48. Historical
- 49. Historical
- 50. DB-SCAN
- 51. Historical with DB- Scan https://en.wikipedia.org/wiki/DBSCAN https://www.mathworks.com/matlabcentral/fileexchange/53842-dbscan
- 54. RNN with DB-Scan
- 57. Create new graph with shift to 1 step
- 58. Zoom your graph and shift first 3 step
- 59. Zoom your graph and shift first 3 step
- 60. Zoom your graph and shift first 3 step
- 61. Zoom your graph and shift first 3 step
- 63. Rotate graph next 3 step and compare with cosine similarity Remark when value lower than threshold
- 64. Compare with anomaly detection (left) and correlation values (right)
- 67. Machine Learning Concept type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/.unknow" hostname=? addr=? terminal=cron res=success' Yes No
- 68. Challenge We don’t have many anomaly dataset.
- 69. Machine Learning Concept type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' Yes Yes Yes type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' Hello world Yes
- 70. Machine Learning Concept type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' Hello World!!! type=Hello…
- 71. Autoencoder Autoencoders are a specific type of feedforward neural networks where the input is the same as the output. https://www.researchgate.net/figure/Autoencoder-architecture_fig1_318204554
- 72. Autoencoder type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_op en acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_op en acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success' Similarity = 100%
- 73. Autoencoder type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/.unknow" hostname=? addr=? terminal=cron res=success' type=USER_START msg=audit(1600272421): user pid=3401 uid=0 auid=0 ses=140812 msg='op=PAM:session_open acct="root" exe="/usr/sbin/?????" hostname=? addr=? terminal=cron res=success' ? Similarity < 100%
- 75. Question?