3.2. Grid search:searching for Estimator parameters

3.2. Grid search:searching for Estimator parameters

Parameters that is not directly learnt within estimators can is set by searching a parameter space for the best cross -validation:evaluating Estimator Performance score. Typical examples include C, kernel and Gamma for support Vector Classifier, Alpha for Lasso, etc.

Any parameter provided if constructing an estimator is optimized in this manner. Specifically, to find the names and current values for all parameters for a given estimator, use:

Estimator.  Get_params()  

Such parameters is often referred to as Hyperparameters (particularly in Bayesian learning), distinguishing them From the parameters optimised-a machine learning procedure.

A Search consists of:

• An estimator (regressor or classifier such as Sklearn.svm.SVC ());
• A parameter space;
• A method for searching or sampling candidates;
• a cross-validation scheme; and
• A score function.

Some models allow for specialized, efficient parameter search strategies, outlined below.  Generic approaches to sampling search candidates is provided in scikit-learn:for given values, GRIDSEARCHCV Exhaustively considers all parameter combinations, while RANDOMIZEDSEARCHCV can sample a given number of Candida TES from a parameter space with a specified distribution. After describing these tools we detail best practice applicable to both approaches.

3.2.1. Exhaustive Grid Search

The grid search provided by GRIDSEARCHCV exhaustively generates candidates from a grid of parameter values Specif IED with the param_grid parameter. For instance, the following Param_grid:

Param_grid=[{C:[110100 Span class= "Mi" >1000 " ' kernel ' : [ ' Linear ' { ' C ' : [ 110100 1000: [0.001 0.0001 ' kernel ' : [ ' RBF ' }, ]        

Specifies that grids should is explored:one with a linear kernel and C values in [1, ten, +, +], and the second O NE with a RBF kernel, and the cross-product of C values ranging in [1, ten, 0.001, +] and gamma values in [, 0.0001] .

The GRIDSEARCHCV instance implements the usual estimator Api:when "fitting" it on a datasets all the possible COM Binations of parameter values are evaluated and the most combination is retained.

Examples:

• See Parameter Estimation using grid search with Cross-validation for a example of grid search computation on the Digits dataset.
• See Sample Pipeline for Text feature extraction and evaluation for an example of Grid Search coupling parameters From a text documents feature extractor (N-gram count Vectorizer and TF-IDF transformer) with a classifier (here a linear SVM trained with SGD with either elastic net or L2 penalty) using a pipeline. Pipeline instance.

3.2.2. Randomized Parameter optimization

While using a grid of parameter settings are currently the most widely used method for parameter optimization, and other search Methods has more favourable properties. RANDOMIZEDSEARCHCV implements a randomized search over parameters, where each setting are sampled from a distribut Ion over possible parameter values. This have both main benefits over an exhaustive search:

• A budget can be chosen independent of the number of parameters and possible values.
• Adding parameters that does not influence the performance does not decrease efficiency.

Specifying how parameters should be sampled are done using a dictionary, very similar to specifying parameters FORGRIDSEARCHCV . Additionally, a computation budget, being the number of sampled candidates or sampling iterations, is specified using the& nbsp n_iter  parameter. For each parameter, either a distribution over possible values or a list of discrete choices (which'll be sampled Unifor Mly) can be specified:

 [{ ' C ' : scipy. Stats. Expon (scale=100),  ' gamma ' : scipy. Stats. Expon (scale=. 1),  ' kernel ' : [ ' RBF '  " ' class_weight ' :[ ' auto '  none]}]            

This example uses THE scipy.stats  module, which contains many useful distributions for sampling parameters, such as expon , gamma ,  uniform  or  randint . In principle, any function can is passed that provides A rvs   (Random variate sample) method to sample a value. A call to THE rvs  function should provide Independent random samples from possible parameter values on consecutive calls.

Warning

The distributions in Scipy.stats does not allow specifying a random state. Instead, they use the global NumPy random state, which can be seeded via np.random.seed or set using np.random . set_state.

For continuous parameters, such as C above, it's important to specify a continuous distribution to take full adv Antage of the randomization. This, increasing n_iter 'll always leads to a finer search.

Examples:

• comparing randomized search and grid search for Hyperparameter estimation compares the usage and efficiency of RA Ndomized Search and Grid search.

References:

• Bergstra, J. and Bengio, Y., Random Search for hyper-parameter optimization, the Journal of machine learning (201 2)

3.2.3. Tips for parameter search3.2.3.1. Specifying an objective metric

By default, parameter search uses thescorefunction of the estimator to evaluate a parameter setting. These is theSklearn.metrics.accuracy_scoreFor classification andSklearn.metrics.r2_scoreFor regression. For some applications, other scoring functions is better suited (for example in unbalanced classification, the accuracy s Core is often uninformative). A alternative scoring function can be specified via thescoringparameter toGRIDSEARCHCV,RANDOMIZEDSEARCHCVAnd many of the specialized cross-validation tools described below. SeeThe scoring parameter:defining model evaluation rulesFor more details.

3.2.3.2. Composite estimators and parameter spaces

pipeline:chaining estimators describes building composite estimators whose parameter space can be searched with These tools.

3.2.3.3. Model Selection:development and evaluation

Model selection by evaluating various parameter settings can be seen as a a-use-the-labeled data to "train" the Param Eters of the grid.

When evaluating the resulting model it's important to does it on held-out samples this were not seen during the grid search Process:it is recommended-to-split the data into a development set (to being fed to the GRIDSEARCHCV Inst ance) and anevaluation set to compute performance metrics.

This can is done by using the cross_validation.train_test_split utility function.

3.2.3.4. Parallelism

GRIDSEARCHCV and RANDOMIZEDSEARCHCV evaluate each parameter setting independently. Computations can is run in parallel if your OS supports it, by using the keyword n_jobs=-1. See function signature for more details.

3.2.3.5. Robustness to failure

Some parameter settings may result in a failure to fit One or more folds of the data. By default, this would cause the entire search to fail, even if some parameter settings could be fully evaluated. Setting error_score=0 (or =np. NaN) would make the procedure robust to such failure, issuing a warning and setting the score for that fold to 0 (or C5>nan), but completing the search.

3.2.4. Alternatives to brute force parameter search3.2.4.1. Model specific Cross-validation

Some models can fit data for a range of value of Some parameter almost as efficiently as fitting the estimator for a Singl E value of the parameter. This feature can is leveraged to perform a further efficient cross-validation used for model selection of this parameter.

The most common parameter amenable-strategy is the parameter encoding the strength. In this case we say we compute the regularization path of the The estimator.

Here is the list of such models:

Linear_model. ELASTICNETCV([L1_ratio, EPS, ...])	Elastic Net model with iterative fitting along a regularization path
Linear_model. LARSCV([Fit_intercept, ...])	cross-validated Least Angle Regression model
Linear_model. LASSOCV([EPs, N_alphas, ...])	Lasso linear model with iterative fitting along a regularization path
Linear_model. LASSOLARSCV([Fit_intercept, ...])	Cross-validated Lasso, using the LARS algorithm
Linear_model. Logisticregressioncv([Cs, ...])	Logistic Regression CV (aka Logit, MaxEnt) classifier.
Linear_model. MULTITASKELASTICNETCV([...])	Multi-task l1/l2 elasticnet with built-in cross-validation.
Linear_model. MULTITASKLASSOCV([EPs, ...])	Multi-task l1/l2 Lasso with built-in cross-validation.
Linear_model. ORTHOGONALMATCHINGPURSUITCV([...])	cross-validated orthogonal Matching Pursuit model (OMP)
Linear_model. Ridgecv([Alphas, ...])	Ridge regression with built-in cross-validation.
Linear_model. RIDGECLASSIFIERCV([Alphas, ...])	Ridge classifier with built-in cross-validation.

3.2.4.2. Information Criterion

Some models can offer an information-theoretic closed-form formula of the optimal estimate of the regularization parameter By computing A, regularization path (instead of several when using cross-validation).

Here are the list of models benefitting from the Aikike information Criterion (AIC) or the Bayesian information Criterion ( BIC) for automated model selection:

Linear_model. Lassolarsic([criterion, ...])

Lasso model fit with Lars using BIC or AIC for model selection

3.2.4.3. Out of Bag estimates

When using the ensemble methods base upon bagging, i.e. generating new training sets using sampling with replacement, part of The training set remains unused. For each classifier in the ensemble, a different part of the of the training set are left out.

This portion can is used to estimate the generalization error without have to rely on a separate validation set . This estimate comes ' for free ' as no additional data is needed and can being used for model selection.

This was currently implemented in the following classes:

Ensemble. Randomforestclassifier([...])	A Random forest classifier.
Ensemble. Randomforestregressor([...])	A Random Forest Regressor.
Ensemble. Extratreesclassifier([...])	An extra-trees classifier.
Ensemble. Extratreesregressor([N_estimators, ...])	An extra-trees regressor.
Ensemble. Gradientboostingclassifier([loss, ...])	Gradient boosting for classification.
Ensemble. Gradientboostingregressor([loss, ...])	Gradient boosting for regression.

3.2. Grid search:searching for Estimator parameters