Nyc open-data-2015-andvanced-sklearn-expanded

1
Advanced Scikit-Learn
Andreas Mueller (NYU Center for Data Science, scikit-learn)

3
Classification
Regression
Clustering
Semi-Supervised Learning
Feature Selection
Feature Extraction
Manifold Learning
Dimensionality Reduction
Kernel Approximation
Hyperparameter Optimization
Evaluation Metrics
Out-of-core learning
…...

5
Overview
● Reminder: Basic sklearn concepts
●
Model building and evaluation:
– Pipelines and Feature Unions
– Randomized Parameter Search
– Scoring Interface
● Out of Core learning
– Feature Hashing
– Kernel Approximation
● New stuff in 0.16.0
– Overview
– Calibration

6
Training Data
Training Labels
Model
Supervised Machine Learning
clf = RandomForestClassifier()
clf.fit(X_train, y_train)

7
Training Data
Test Data
Training Labels
Model
Prediction
y_pred = clf.predict(X_test)

8
clf.score(X_test, y_test)
Training Data
Test Data
Training Labels
Model
Prediction
Test Labels Evaluation
y_pred = clf.predict(X_test)

9
pca = PCA(n_components=3)
pca.fit(X_train) Training Data Model
Unsupervised Transformations

10
pca = PCA(n_components=3)
pca.fit(X_train)
X_new = pca.transform(X_test)
Training Data
Test Data
Model
Transformation
Unsupervised Transformations

11
Basic API
estimator.fit(X, [y])
estimator.predict estimator.transform
Classification Preprocessing
Regression Dimensionality reduction
Clustering Feature selection
Feature extraction

12
Cross-Validation
from sklearn.cross_validation import cross_val_score
scores = cross_val_score(SVC(), X, y, cv=5)
print(scores)
>> [ 0.92 1. 1. 1. 1. ]

13
Cross-Validation
print(scores)
>> [ 0.92 1. 1. 1. 1. ]
cv_ss = ShuffleSplit(len(X_train), test_size=.3,
n_iter=10)
scores_shuffle_split = cross_val_score(SVC(), X, y,
cv=cv_ss)

14
Cross-Validation
print(scores)
>> [ 0.92 1. 1. 1. 1. ]
cv_ss = ShuffleSplit(len(X_train), test_size=.3,
n_iter=10)
scores_shuffle_split = cross_val_score(SVC(), X, y,
cv=cv_ss)
cv_labels = LeaveOneLabelOut(labels)
scores_pout = cross_val_score(SVC(), X, y, cv=cv_labels)

15
Cross -Validated Grid Search

16
Cross -Validated Grid Search
from sklearn.grid_search import GridSearchCV
from sklearn.cross_validation import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y)
param_grid = {'C': 10. ** np.arange(-3, 3),
'gamma': 10. ** np.arange(-3, 3)}
grid = GridSearchCV(SVC(), param_grid=param_grid)
grid.fit(X_train, y_train)
grid.predict(X_test)
grid.score(X_test, y_test)

17
Training DataTraining Labels
Model

18
Model

19
Model
Feature
Extraction
Scaling
Feature
Selection

20
Model
Feature
Extraction
Scaling
Feature
Selection
Cross Validation

21
Model
Feature
Extraction
Scaling
Feature
Selection
Cross Validation

22
Pipelines
from sklearn.pipeline import make_pipeline
pipe = make_pipeline(StandardScaler(), SVC())
pipe.fit(X_train, y_train)
pipe.predict(X_test)

23
Combining Pipelines and
Grid Search
Proper cross-validation
param_grid = {'svc__C': 10. ** np.arange(-3, 3),
'svc__gamma': 10. ** np.arange(-3, 3)}
scaler_pipe = make_pipeline(StandardScaler(), SVC())
grid = GridSearchCV(scaler_pipe, param_grid=param_grid, cv=5)

24
Combining Pipelines and
Grid Search II
Searching over parameters of the preprocessing step
param_grid = {'selectkbest__k': [1, 2, 3, 4],
'svc__C': 10. ** np.arange(-3, 3),
'svc__gamma': 10. ** np.arange(-3, 3)}
scaler_pipe = make_pipeline(SelectKBest(), SVC())
grid = GridSearchCV(scaler_pipe, param_grid=param_grid, cv=5)

25
Feature Union
Model
Feature
Extraction I
Feature
Extraction II

26
Feature Union
char_and_word = make_union(CountVectorizer(analyzer="char"),
CountVectorizer(analyzer="word"))
text_pipe = make_pipeline(char_and_word, LinearSVC(dual=False))
param_grid = {'linearsvc__C': 10. ** np.arange(-3, 3)}
grid = GridSearchCV(text_pipe, param_grid=param_grid, cv=5)

27
Feature Union
char_and_word = make_union(CountVectorizer(analyzer="char"),
CountVectorizer(analyzer="word"))
text_pipe = make_pipeline(char_and_word, LinearSVC(dual=False))
param_grid = {'linearsvc__C': 10. ** np.arange(-3, 3)}
grid = GridSearchCV(text_pipe, param_grid=param_grid, cv=5)
param_grid2 = {'featureunion__countvectorizer-1__ngram_range': [(1, 3), (1, 5), (2, 5)],
'featureunion__countvectorizer-2__ngram_range': [(1, 1), (1, 2), (2, 2)],
'linearsvc__C': 10. ** np.arange(-3, 3)}

28
Randomized Parameter Search

29
Source: Bergstra and Bengio

30
Source: Bergstra and Bengio
Step-size free for continuous parameters
Decouples runtime from search-space size
Robust against irrelevant parameters

31
params = {'featureunion__countvectorizer-1__ngram_range':
[(1, 3), (1, 5), (2, 5)],
'featureunion__countvectorizer-2__ngram_range':
[(1, 1), (1, 2), (2, 2)],
'linearsvc__C': 10. ** np.arange(-3, 3)}

32
[(1, 3), (1, 5), (2, 5)],
[(1, 1), (1, 2), (2, 2)],
'linearsvc__C': expon()}

33
rs = RandomizedSearchCV(text_pipe,
param_distributions=param_distributins, n_iter=50)
[(1, 3), (1, 5), (2, 5)],
[(1, 1), (1, 2), (2, 2)],
'linearsvc__C': expon()}

34
● Always use distributions for continuous
variables.
● Don't use for low dimensional spaces.
● Future: Bayesian optimization based search.

35
Generalized Cross-Validation and Path Algorithms

36
rfe = RFE(LogisticRegression())

37
param_grid = {'n_features_to_select': range(1, n_features)}
gridsearch = GridSearchCV(rfe, param_grid)
grid.fit(X, y)

38
grid.fit(X, y)

39
grid.fit(X, y)
rfecv = RFECV(LogisticRegression())

40
grid.fit(X, y)
rfecv = RFECV(LogisticRegression())
rfecv.fit(X, y)

42
Linear Models Feature Selection Tree-Based models [possible]
LogisticRegressionCV [new] RFECV [DecisionTreeCV]
RidgeCV [RandomForestClassifierCV]
RidgeClassifierCV [GradientBoostingClassifierCV]
LarsCV
ElasticNetCV
...

44
Default:
Accuracy (classification)
R2 (regression)
GridSeachCV
RandomizedSearchCV
cross_val_score
...CV

45
Scoring with imbalanced data
cross_val_score(SVC(), X_train, y_train)
>>> array([ 0.9, 0.9, 0.9])

46
>>> array([ 0.9, 0.9, 0.9])
cross_val_score(DummyClassifier("most_frequent"), X_train, y_train)
>>> array([ 0.9, 0.9, 0.9])

47
>>> array([ 0.9, 0.9, 0.9])
>>> array([ 0.9, 0.9, 0.9])
cross_val_score(SVC(), X_train, y_train, scoring="roc_auc")
array([ 0.99961591, 0.99983498, 0.99966247])

48
>>> array([ 0.9, 0.9, 0.9])
>>> array([ 0.9, 0.9, 0.9])
cross_val_score(SVC(), X_train, y_train, scoring="roc_auc")
array([ 0.99961591, 0.99983498, 0.99966247])

49
Available metrics
print(SCORERS.keys())
>> ['adjusted_rand_score',
'f1',
'mean_absolute_error',
'r2',
'recall',
'median_absolute_error',
'precision',
'log_loss',
'mean_squared_error',
'roc_auc',
'average_precision',
'accuracy']

50
Defining your own scoring
def my_super_scoring(est, X, y):
return accuracy_scorer(est, X, y) - np.sum(est.coef_ != 0)

52
Or: save ourself the effort

53
Think twice!
● Old laptop: 4GB Ram
● 1073741824 float32
● Or 1mio data points with 1000 features
● EC2 : 256 GB Ram
● 68719476736 float32
● Or 68mio data points with 1000 features

54
Supported Algorithms
● All SGDClassifier derivatives
● Naive Bayes
● MinibatchKMeans
● IncrementalPCA
● MiniBatchDictionaryLearning

55
Out of Core Learning
sgd = SGDClassifier()
for i in range(9):
X_batch, y_batch = cPickle.load(open("batch_%02d" % i))
sgd.partial_fit(X_batch, y_batch, classes=range(10))
Possibly go over the data multiple times.

56
Stateless Transformers
● Normalizer
● HashingVectorizer
● RBFSampler (and other kernel approx)

57
Text data and the hashing trick

58
Bag Of Word Representations
“You better call Kenny Loggins”
CountVectorizer / TfidfVectorizer

59
['you', 'better', 'call', 'kenny', 'loggins']
tokenizer

60
[0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, …., 0 ]
better call youaardvak zyxst
tokenizer
Sparse matrix encoding

61
Hashing Trick
HashingVectorizer
tokenizer
hashing
[hash('you'), hash('better'), hash('call'), hash('kenny'), hash('loggins')]
= [832412, 223788, 366226, 81185, 835749]
[0, …, 0, 1, 0, … , 0, 1 , 0, …, 0, 1, 0, ... 0 ]
Sparse matrix encoding

62
Out of Core Text Classification
hashing_vectorizer = HashingVectorizer()
for i in range(9):
text_batch, y_batch = cPickle.load(open("text_%02d" % I))
X_batch = hashing_vectorizer.transform(text_batch)
sgd.partial_fit(X_batch, y_batch, classes=range(10))

66
Reminder: Kernel Trick
Classifier linear → need only

67
Reminder: Kernel Trick
Classifier linear → need only
Linear:
Polynomial:
RBF:
Sigmoid:

68
Complexity
● Solving kernelized SVM:
~O(n_samples ** 3)
● Solving linear (primal) SVM:
~O(n_samples * n_features)
n_samples large? Go primal!

69
Undoing the Kernel Trick
● Kernel approximation:
● k =
= RBFSampler

70
Usage
kernel_approximation = RBFSampler(gamma=.001, n_components=400)
for i in range(9):
X_batch, y_batch = cPickle.load(open("batch_%02d" % i))
if i == 0:
kernel_approximation.fit(X_batch)
X_transformed = kernel_approximation.transform(X_batch)
sgd.partial_fit(X_transformed, y_batch, classes=range(10))

72
Highlights from 0.16.0
● Multinomial Logistic Regression,
LogisticRegressionCV.
● IncrementalPCA.
● Probability callibration of classifiers.
● Birch clustering.
● LSHForest.
● More robust integration with pandas.

73
Probability Calibration
SVC().decision_function()
→ CalibratedClassifierCV(SVC()).predict_proba()
RandomForestClassifier().predict_proba()
→ CalibratedClassifierCV(RandomForestClassifier()).predict_proba()

75
CDS is hiring Research Engineers

76
Thank you for your attention.
@t3kcit
@amueller
t3kcit@gmail.comx

77
Bias Variance Tradeoff
(why we do cross validation and grid searches)

78
Overfitting and Underfitting
Model complexity
Accuracy
Training

79
Model complexity
Accuracy
Training
Generalization

80
Model complexity
Accuracy
Training
Generalization
Underfitting Overfitting
Sweet spot

96
Know where you are on the bias-variance tradeoff

97
Validation Curves
train_scores, test_scores = validation_curve(SVC(), X, y,
param_name="gamma", param_range=param_range)

98
train_sizes, train_scores, test_scores = learning_curve(
estimator, X, y,train_sizes=train_sizes)
Learning Curves

Nyc open-data-2015-andvanced-sklearn-expanded

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Nyc open-data-2015-andvanced-sklearn-expanded (20)

More from Vivian S. Zhang (12)

Recently uploaded (20)

Nyc open-data-2015-andvanced-sklearn-expanded