![Jan Jantzen, DTU
3
Vehicle Example
Vehicle Top speed
km/h
Colour Air
resistance
Weight
Kg
V1 220 red 0.30 1300
V2 230 black 0.32 1400
V3 260 red 0.29 1500
V4 140 gray 0.35 800
V5 155 blue 0.33 950
V6 130 white 0.40 600
V7 100 black 0.50 3000
V8 105 red 0.60 2500
V9 110 gray 0.55 3500
Vehicle Clusters
100 150 200 250 300
500
1000
1500
2000
2500
3000
3500
Top speed [km/h]
Weight[kg]
Sports cars
Medium market cars
Lorries](https://image.slidesharecdn.com/clustering-tutorial-130701165659-phpapp01/85/Clustering-tutorial-3-320.jpg)
![Jan Jantzen, DTU
4
Terminology
100 150 200 250 300
500
1000
1500
2000
2500
3000
3500
Top speed [km/h]
Weight[kg]
Sports cars
Medium market cars
Lorries
Object or data point
feature
feature space
cluster
feature
label
Example: Classify cracked tiles](https://image.slidesharecdn.com/clustering-tutorial-130701165659-phpapp01/85/Clustering-tutorial-4-320.jpg)
Clustering tutorial
- 1. Jan Jantzen, DTU 1 Tutorial On Fuzzy Clustering Jan Jantzen Technical University of Denmark [email protected] Abstract uProblem: To extract rules from data uMethod: Fuzzy c-means uResults: e.g., finding cancer cells
- 2. Jan Jantzen, DTU 2 Cluster (www.m-w.com) uA number of similar individuals that occur together as a: two or more consecutive consonants or vowels in a segment of speech b: a group of houses (...) c: an aggregation of stars or galaxies that appear close together in the sky and are gravitationally associated. Cluster analysis (www.m-w.com) uA statistical classification technique for discovering whether the individuals of a population fall into different groups by making quantitative comparisons of multiple characteristics.
- 3. Jan Jantzen, DTU 3 Vehicle Example Vehicle Top speed km/h Colour Air resistance Weight Kg V1 220 red 0.30 1300 V2 230 black 0.32 1400 V3 260 red 0.29 1500 V4 140 gray 0.35 800 V5 155 blue 0.33 950 V6 130 white 0.40 600 V7 100 black 0.50 3000 V8 105 red 0.60 2500 V9 110 gray 0.55 3500 Vehicle Clusters 100 150 200 250 300 500 1000 1500 2000 2500 3000 3500 Top speed [km/h] Weight[kg] Sports cars Medium market cars Lorries
- 4. Jan Jantzen, DTU 4 Terminology 100 150 200 250 300 500 1000 1500 2000 2500 3000 3500 Top speed [km/h] Weight[kg] Sports cars Medium market cars Lorries Object or data point feature feature space cluster feature label Example: Classify cracked tiles
- 5. Jan Jantzen, DTU 5 475Hz 557Hz Ok? -----+-----+--- 0.958 0.003 Yes 1.043 0.001 Yes 1.907 0.003 Yes 0.780 0.002 Yes 0.579 0.001 Yes 0.003 0.105 No 0.001 1.748 No 0.014 1.839 No 0.007 1.021 No 0.004 0.214 No Table 1: frequency intensities for ten tiles. Tiles are made from clay moulded into the right shape, brushed, glazed, and baked. Unfortunately, the baking may produce invisible cracks. Operators can detect the cracks by hitting the tiles with a hammer, and in an automated system the response is recorded with a microphone, filtered, Fourier transformed, and normalised. A small set of data is given in TABLE 1 (adapted from MIT, 1997). Algorithm: hard c-means (HCM) (also known as k means)
- 6. Jan Jantzen, DTU 6 Plot of tiles by frequencies (logarithms). The whole tiles (o) seem well separated from the cracked tiles (*). The objective is to find the two clusters. -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres 1. Place two cluster centres (x) at random. 2. Assign each data point (* and o) to the nearest cluster centre (x) -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres
- 7. Jan Jantzen, DTU 7 -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres 1. Compute the new centre of each class 2. Move the crosses (x) Iteration 2 -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres
- 8. Jan Jantzen, DTU 8 Iteration 3 -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres Iteration 4 (then stop, because no visible change) Each data point belongs to the cluster defined by the nearest centre -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres
- 9. Jan Jantzen, DTU 9 The membership matrix M: 1. The last five data points (rows) belong to the first cluster (column) 2. The first five data points (rows) belong to the second cluster (column) M = 0.0000 1.0000 0.0000 1.0000 0.0000 1.0000 0.0000 1.0000 0.0000 1.0000 1.0000 0.0000 1.0000 0.0000 1.0000 0.0000 1.0000 0.0000 1.0000 0.0000 Membership matrix M −≤− = otherwise if m jkik ik 0 1 22 cucu data point k cluster centre i distance cluster centre j
- 10. Jan Jantzen, DTU 10 c-partition Kc iallforUCØ jiallforØCC UC i ji c i i ≤≤ ⊂⊂ ≠=∩ = = 2 1 U All clusters C together fills the whole universe U Clusters do not overlap A cluster C is never empty and it is smaller than the whole universe U There must be at least 2 clusters in a c-partition and at most as many as the number of data points K Objective function ∑ ∑∑ = ∈= −== c i Ck ik c i i ik JJ 1 2 ,1 u cu Minimise the total sum of all distances
- 11. Jan Jantzen, DTU 11 Algorithm: fuzzy c-means (FCM) Each data point belongs to two clusters to different degrees -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres
- 12. Jan Jantzen, DTU 12 1. Place two cluster centres 2. Assign a fuzzy membership to each data point depending on distance -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres 1. Compute the new centre of each class 2. Move the crosses (x) -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres
- 13. Jan Jantzen, DTU 13 Iteration 2 -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres Iteration 5 -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres
- 14. Jan Jantzen, DTU 14 Iteration 10 -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres Iteration 13 (then stop, because no visible change) Each data point belongs to the two clusters to a degree -8 -6 -4 -2 0 2 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 log(intensity) 475 Hz log(intensity)557Hz Tiles data: o = whole tiles, * = cracked tiles, x = centres
- 15. Jan Jantzen, DTU 15 The membership matrix M: 1. The last five data points (rows) belong mostly to the first cluster (column) 2. The first five data points (rows) belong mostly to the second cluster (column) M = 0.0025 0.9975 0.0091 0.9909 0.0129 0.9871 0.0001 0.9999 0.0107 0.9893 0.9393 0.0607 0.9638 0.0362 0.9574 0.0426 0.9906 0.0094 0.9807 0.0193 Fuzzy membership matrix M ( ) ∑ = − = c j q jk ik ik d d m 1 1/2 1 ikikd cu −= Distance from point k to current cluster centre i Distance from point k to other cluster centres j Point k’s membership of cluster i Fuzziness exponent
- 16. Jan Jantzen, DTU 16 Fuzzy membership matrix M ikm ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )1/21/2 2 1/2 1 1/2 1/21/2 2 1/2 1 1 1/2 111 1 1 1 −−− − −−− = − +++ = ++ + = = ∑ q ck q k q k q ik q ck ik q k ik q k ik c j q jk ik ddd d d d d d d d d d L L Gravitation to cluster i relative to total gravitation Electrical Analogy R1 R2 i1 i2 U I I i i UI U R R RRR R R R RRR R RIU i i i c i i c == +++ = +++ = = 11 111 1 1 111 1 21 21 L L Same form as mik
- 17. Jan Jantzen, DTU 17 Fuzzy Membership 1 2 3 4 5 0 0.5 1 Cluster centres Membershipoftestpoint o is with q = 1.1, * is with q = 2 Data point Fuzzy c-partition Kc iallforUCØ jiallforØCC UC i ji c i i ≤≤ ⊂⊂ ≠=∩ = = 2 1 U All clusters C together fill the whole universe U. Remark: The sum of memberships for a data point is 1, and the total for all points is K Not valid: Clusters do overlap A cluster C is never empty and it is smaller than the whole universe U There must be at least 2 clusters in a c-partition and at most as many as the number of data points K
- 18. Jan Jantzen, DTU 18 Example: Classify cancer cells Normal smear Severely dysplastic smear Using a small brush, cotton stick, or wooden stick, a specimen is taken from the uterincervix and smeared onto a thin, rectangular glass plate, a slide. The purpose of the smear screening is to diagnose pre-malignant cell changes before they progress to cancer. The smear is stained using the Papanicolau method, hence the name Pap smear. Different characteristics have different colours, easy to distinguish in a microscope. A cyto-technician performs the screening in a microscope. It is time consuming and prone to error, as each slide may contain up to 300.000 cells. Dysplastic cells have undergone precancerous changes. They generally have longer and darker nuclei, and they have a tendency to cling together in large clusters. Mildly dysplastic cels have enlarged and bright nuclei. Moderately dysplastic cells have larger and darker nuclei. Severely dysplastic cells have large, dark, and often oddly shaped nuclei. The cytoplasm is dark, and it is relatively small. Possible Features uNucleus and cytoplasm area uNucleus and cyto brightness uNucleus shortest and longest diameter uCyto shortest and longest diameter uNucleus and cyto perimeter uNucleus and cyto no of maxima u(...)
- 19. Jan Jantzen, DTU 19 Classes are nonseparable Hard Classifier (HCM) Ok light moderate severeOk A cell is either one or the other class defined by a colour.
- 20. Jan Jantzen, DTU 20 Fuzzy Classifier (FCM) Ok light moderate severeOk A cell can belong to several classes to a Degree, i.e., one column may have several colours. Function approximation 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 -1.5 -1 -0.5 0 0.5 1 1.5 Input Output1 Curve fitting in a multi-dimensional space is also called function approximation. Learning is equivalent to finding a function that best fits the training data.
- 21. Jan Jantzen, DTU 21 Approximation by fuzzy sets 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 -2 -1 0 1 2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 0.2 0.4 0.6 0.8 1 Procedure to find a model 1. Acquire data 2. Select structure 3. Find clusters, generate model 4. Validate model
- 22. Jan Jantzen, DTU 22 Conclusions uCompared to neural networks, fuzzy models can be interpreted by human beings uApplications: system identification, adaptive systems Links u J. Jantzen: Neurofuzzy Modelling. Technical University of Denmark: Oersted-DTU, Tech report no 98-H-874 (nfmod), 1998. URL http://fuzzy.iau.dtu.dk/download/nfmod.pdf u PapSmear tutorial. URL http://fuzzy.iau.dtu.dk/smear/ u U. Kaymak: Data Driven Fuzzy Modelling. PowerPoint, URL http://fuzzy.iau.dtu.dk/tutor/ddfm.htm
- 23. Jan Jantzen, DTU 23 Exercise: fuzzy clustering (Matlab) u Download and follow the instructions in this text file: http://fuzzy.iau.dtu.dk/tutor/fcm/exerF5.txt u The exercise requires Matlab (no special toolboxes are required)