Data Analysis: Evaluation Metrics for Supervised Learning Models of Machine Learning
Download as PPTX, PDF5 likes5,190 views
This document discusses various machine learning evaluation metrics for supervised learning models. It covers classification, regression, and ranking metrics. For classification, it describes accuracy, confusion matrix, log-loss, and AUC. For regression, it discusses RMSE and quantiles of errors. For ranking, it explains precision-recall, precision-recall curves, F1 score, and NDCG. The document provides examples and visualizations to illustrate how these metrics are calculated and used to evaluate model performance.
Data Analysis: Evaluation Metrics for Supervised Learning Models of Machine Learning
- 1. DATA ANALYTICS Evaluation Metrics for Supervised Learning Models of Machine Learning Md. Main Uddin Rony Software Developer, Infolytx,Inc.
- 2. Machine Learning Evaluation Metrics
- 3. ML Evaluation Metrics Are….. ● tied to Machine Learning Tasks ● methods which determine an algorithm’s performance and behavior ● helpful to decide the best model to meet the target performance ● helpful to parameterize the model in such a way that can offer best performing algorithm
- 4. Evaluation Metrics Types... ● Various types of ML Algorithms (classification, regression, ranking, clustering) ● Different types of evaluation metrics for different types of algorithm ● Some metrics can be useful for more than one type of algorithm (Precision - Recall) ● Will cover Evaluation Metrics for Supervised learning models only ( Classification, Regression, Ranking)
- 6. Classification Model Does... Predict class labels given input data In Binary classification, there are two possible output classes ( 0 or 1, True or False, Positive or Negative, Yes or No etc.) Spam detection of email is a good example of Binary classification.
- 7. Some Popular Classification Metrics... Accuracy Confusion Matrix Log-Loss AUC
- 8. Accuracy ● Ratio between the number of correct predictions and total number of predictions ● Example: Suppose we have 100 examples in the positive class and 200 examples in the negative class. Our model declares 80 out of 100 positives as positive correctly and 195 out of 200 negatives as negative correctly. ● So, accuracy is = (80 + 195)/(100 + 200) = 91.7%
- 9. Confusion Matrix ● Shows a more detailed breakdown of correct and incorrect classifications for each class. ● Think about our previous example and then the confusion matrix looks like: ● What is the accuracy that positive class has ? And Negative class? ● Clearly, positive class has lower accuracy than the negative class ● And that information is lost if we calculate overall accuracy only. Predicted as positive Predicted as negative Labeled as positive 80 20 Labeled as negative 5 195
- 10. Per-Class Accuracy ● Average per class accuracy of previous example: (80% + 97.5%)/2 = 88.75 %, different from accuracy Why important? - Can show different scenario when there are different numbers of examples per class - Class with more examples than other will dominate the statistic of accuracy, hence produced a distorted picture
- 11. Log-Loss Very much useful when the raw output of classifier is a numeric probability instead of a class label 0 or 1 Mathematically , log-loss for a binary classifier: Minimum is 0 when prediction and true label match up Calculate for a data point predicted by classifier to belong to class 1 with probability .51 and with probability 1 Minimizing this value, maximizing the accuracy of the classifier
- 12. AUC (Area Under Curve) ● The curve is receiver operating characteristic curve or in short ROC curve ● Provides nuanced details about the behavior of the classifier ● Bad ROC curve covers very little area ● Good ROC curve has a lot of space under it ● But, how?
- 13. AUC (contd..)
- 14. AUC (contd..)
- 15. AUC (contd..)
- 16. AUC (contd..)
- 17. AUC (contd..)
- 18. AUC (contd..)
- 19. AUC (contd..) ● So, what’s the advantage of using of ROC curve over a simpler metric? ROC curve visualizes all possible classification thresholds, whereas other metrics only represents your error rate for a single threshold
- 20. Ranking Metrics
- 21. Ranking ... Is related to binary classification Internet Search can be a good example which acts as a ranker. During a query, it returns ranked list of web pages relevant to that query So, here ranking can be a binary classification of “relevant query” or “irrelevant query” It also ordering the results so that the most relevant result should be on top So, what can be done in underlying implementation considering both?? Can we predict what will ranking metrics evaluate and how?
- 22. Some Ranking Metrics.. Precision - Recall Precision - Recall Curve and F1 Score NDCG
- 23. Precision - Recall Considering the scenario of web search result, Precision answers this question: “Out of the items that the ranker/classifier predicted to be relevant, how many are truly relevant?” Whereas, Recall answers this: “Out of all the items that are truly relevant, how many are found by the ranker/classifier?”
- 24. Precision - Recall (Contd..)
- 25. Calculation Example Of Precision- Recall Total Negative = 9760 + 140 = 9900 Total Positive = 40 + 60 = 100 Total Negative prediction = 9760 + 40 = 9800 Total Positive prediction = 140 + 60 = 200 Precision = TP / (TP+FP) = 60 / (60 + 140) = 30% Recall = TP / (TP+FN) = 60 / (60+40) = 60% Predicted as Negative Predicted as Positive Actual Negative 9760 (TN) 140 (FP) Actual Positive 40 (FN) 60 (TP)
- 26. Precision - Recall Curve When the numbers of answers returned by the ranker will change, the precision and recall score will also be changed By plotting precision versus recall over a range of k values which denotes numbers of results returned, we get the precision - recall curve
- 28. Interpolating a Recall/Precision Curve
- 29. Trade-off between Recall and Precision
- 30. F-Measure One measure of performance that takes into account both recall and precision Harmonic mean of recall and precision: Compared to arithmetic mean, both need to be high for harmonic mean to be high
- 31. NDCG ● Precision and recall treat all retrieved items equally. ● But, a relevant item in position 1 and a relevant item in position 5 bear same significance? ● Think about a web search result ● NDCG tries to take this scenario into account.
- 32. What? ● NDCG stands for Normalized Discounted Cumulative Gain ● First just focus on DCG (Discounted Cumulative Gain)
- 33. Discounted Cumulative Gain ● Popular measure for evaluating web search and related tasks. ● Discounts items that are further down the search result list ● Two assumptions: - Highly relevant documents are more useful than marginally relevant document - the lower the ranked position of a relevant document, the less useful it is for the user, since it is less likely to be examined
- 34. Discounted Cumulative Gain ● Uses graded relevance as a measure of the usefulness, or gain, from examining a document ● Gain is accumulated starting at the top of the ranking and may be reduced, or discounted, at lower ranks ● Typical discount is 1/log (rank) - With base 2, the discount at rank 4 is ½, and at rank 8 it is 1/3
- 35. Discounted Cumulative Gain ● DCG is the total gain accumulated at a particular rank p: ● Alternative formulation: - used by some web search companies - emphasis on retrieving highly relevant documents * Equation used from Addison Wesley’s
- 36. DCG Example ● 10 ranked documents judged on 0-3 relevance scale: 3, 2, 3, 0, 0, 1, 2, 2, 3, 0 ● discounted gain: 3, 2/1, 3/1.59, 0, 0, 1/ 2.59, 2/2.81, 2/3 , 3/3.17, 0 = 3, 2, 1.89, 0, 0, 0.39, 0.71, 0.67, 0.95, 0 ● DCG: 3, 5, 6.89, 6.89, 6.89, 7.28, 7.99, 8.66, 9.61, 9.61 * Example used from Addison Wesley’s presentation
- 37. Normalized DCG ● Normalized version of discounted cumulative gain ● Often normalized by comparing the DCG at each rank with the DCG value for the perfect ranking ● Normalized score always lies between 0.0 and 1.0
- 38. NDCG Example ● Let’s look back the list of ranked document judged on relevance scale: 3, 2, 3, 0, 0, 1, 2, 2, 3, 0 ● Perfect ranking: 3, 3, 3, 2, 2, 2, 1, 0, 0, 0 ● Perfect discounted gain: 3, 3/1, 3/1.59, 2/2, 2/2.32, 2/ 2.59, 1/2.81, 0 , 0, 0 = 3, 3, 1.89, 1, 0.86, 0.77, 0.36, 0, 0, 0
- 39. NDCG Example ● Ideal DCG values: 3, 6, 7.89, 8.89, 9.75, 10.52, 10.88, 10.88, 10.88, 10.88 NDCG values( divide actual by ideal): 3/3, 5/6, 6.89/7.89, 6.89/8.89, 6.89/9.75, 7.28/10.52, 7.99/10.88, 8.66/10.88, 9.61/10.88, 9.61/10.88 = 1, 0.83, 0.87, 0.76, 0.71, 0.69, 0.73, 0.8, 0.88, 0.88 3, 2, 3, 0, 0, 1, 2, 2, 3, 0
- 41. What Regression Tasks do? Model learns to predict numeric scores. For example, we try to predict the price of a stock on future days given past price history and other useful information
- 42. Some Regression Metrics.. RMSE (Root Mean Square Error) Quantiles of Errors
- 43. RMSE The most commonly used metric for regression tasks Also known as RMSD ( root-mean-square deviation) This is defined as the square root of the average squared distance between the actual score and the predicted score:
- 44. Quantiles of Errors RMSE is an average, so it is sensitive to large outliers. If the regressor performs really badly on a single data point, the average error could be big, not robust Quantiles (or percentiles) are much more robust Because it is not affected by large outliers It’s important to look at the median absolute percentage: It gives us a relative measure of the typical error.
- 45. Acknowledgement Evaluating Machine Learning Models by Alice Zheng Many slides in this section are adapted from Prof. Joydeep Ghosh (UT ECE) who in turn adapted them from Prof. Dik Lee (Univ. of Science and Tech, Hong Kong) Tutorial of Data School on ROC Curves and AUC by Kevin Markham
- 46. Questions???
- 47. Thank You