Automated Clustering Project - 12th CONTECSI 34th WCARS
0 likes362 views
This document describes an automated clustering and outlier detection program. The program normalizes data, performs principal component analysis to select important components, compares clustering algorithms, selects the best model using silhouette values, and produces outputs labeling clusters and outliers. It is demonstrated on a sample of 5,000 credit card customer records, identifying a small cluster of 3 accounts as outliers based on features like new status and high late payments.
![Data normalization
Data normalization
Converts each numerically represented dimension to be clustered into the range [0,1].
A desirable procedure for preparing numeric attributes for clustering](https://image.slidesharecdn.com/12contecsi-34thwcarsautomatedclusteringprojectpresentationv22-150602043124-lva1-app6892/85/Automated-Clustering-Project-12th-CONTECSI-34th-WCARS-4-320.jpg)
Automated Clustering Project - 12th CONTECSI 34th WCARS
- 1. Automated Clustering Project MiklosVasarhelyi, Paul Byrnes, andYunsenWang Presented by DenizAppelbaum
- 2. Motivation Motivation entails the development of a program that automatically performs clustering and outlier detection for a wide variety of numerically represented data.
- 3. Outline of program features Normalizes all data to be clustered Creates normalized principal components from the normalized data Automatically selects the necessary normalized principal components for use in actual clustering and outlier detection Compares a variety of algorithms based upon the selected set of normalized principal components Adopts the top performing model based upon silhouette coefficient values to perform the final clustering and outlier detection procedures Produces relevant information and outputs throughout the process
- 4. Data normalization Data normalization Converts each numerically represented dimension to be clustered into the range [0,1]. A desirable procedure for preparing numeric attributes for clustering
- 5. Principal component analysis Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. In this way, PCA can both reduce dimensionality as well as eliminate inherent problems associated with clustering data whose attributes are correlated In the following slides, a random sample of 5,000 credit card customers is used to demonstrate the automated clustering and outlier detection program
- 6. Principal component analysis PCA initially results in four principal components being generated from the original data Using a cumulative data variability threshold of 80% (default specification), three principal components are automatically selected for analysis – they explain the vast majority of data variability
- 7. Principal component analysis Scatter plot of PC1 and PC2 In this view, the top 2 principal components are plotted for each object in two-dimensional space. As can be seen, a small subset of records appear significantly more distant/different from the vast majority of objects.
- 8. Clustering exploration/simulation process - examples Ward method Ward suggested a general agglomerative hierarchical clustering procedure, where the criterion for choosing the pair of clusters to merge at each step is based on the optimal value of an objective function. Complete link method This method is also known as farthest neighbor clustering.The result of the clustering can be visualized as a dendrogram, which shows the sequence of cluster fusion and the distance at which each fusion took place. PAM (partitioning around medoids) The k-medoids algorithm is a clustering method related to the k-means algorithm and the medoids shift algorithm; It is considered more stable than k-means, because it uses the median rather than mean K-means k-means clustering aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean, serving as a prototype of the cluster.
- 9. Clustering exploration results The result shown below is based upon a simulation exercise, whereby all four algorithms are automatically compared on the data set (i.e., a random sample of 5,000 records from the credit card customer data). In this particular case, the best model is found to be a two-cluster solution using the complete link hierarchical method. This is the final model and is used for subsequent clustering and outlier detection. Best clustering result: The silhouette value can theoretically range from -1 to +1, with higher values indicative of better cluster quality in terms of both cohesion and separation. Best Method Number Of Clusters SilhouetteValue complete link hierarchical 2 0.753754205720575
- 10. Complete-link Hierarchical clustering (1/2) The 5,000 instances are on the x-axis. In moving vertically from the x-axis, one can begin to see how the actual clusters are formed.
- 11. Plot of PCs with cluster assignment labels (1/3) In this view, the top two principal components (i.e., PC1 and PC2) are plotted for each object in two- dimensional space. In the graph, there are two clusters, one dark blue and the other light blue. The small subset of three records appears substantially more different from the majority of objects.
- 12. Plot of PCs with cluster assignment labels (2/3) In this view, PC1 and PC3 are plotted for each object in two-dimensional space. In the graph, the two clusters are again shown. It is once again evident that the small subset of three records appears more different from the majority of other objects.
- 13. Plot of PCs with cluster assignment labels (3/3) In this view, PC2 and PC3 are plotted for each object in two- dimensional space. Cluster differences appear less prominent from this perspective.
- 14. Principal components 3D scatterplot Cluster one represents the majority class (black) while cluster two represents the rare class (red). In this view, one can clearly see the subset of three records (in red) appearing more isolated from the other objects.
- 15. Cluster 1 outlier plot In this view, an arbitrary cutoff is inserted at the 99.9th percentile (red horizontal line) so as to provide for efficient identification of very irregular records. Objects further from the x-axis are more questionable. While all objects distant from the x- axis might be worth investigating, points above the cutoff should be viewed as particularly suspicious.
- 16. Conclusion of Process At the conclusion of outlier detection, an output file for each cluster containing the unique record identifier, original variables, normalized variables, principal components, normalized principal components, cluster assignments, and mahalanobis distance information can be exported to facilitate further analyses and investigations. Cluster 2 – final output file of a subset of fields: Distinguishing features of cluster 2 records: 1) New accounts (age = 1 month), 2) Very high incidence of late payments, and 3) Relatively high credit limits, particularly given the account age and late payment issues. Record AccountAge CreditLimit AdditionalAssets LatePayments model.cluster md 32430 1 2500 1 3 2 5.83E-05 65470 1 8500 1 4 2 0.002371778 78772 1 2200 0 3 2 0.000442305