Chapter 1.3-1.4:model Selection & the Curse of dimensionality
Chapter 1.3-1.4:model Selection & the Curse of dimensionality
Christopher M. Bishop, PRML, Chapter 1 introdcution
1. Model Selection
In our example of polynomial curve fitting using least squares:
1.1 Parameters and Model complexity:
- Order of polynomial: There was a optimal Order of polynomial that gave the best generalization. The order of the polynomial controls the number of free parameters in the model and thereby governs the model complexity.
- regularization coefficient : With regularized least squares, the regularization coefficientλalso controls the effective complexity of the Model.
- More complex models: such as mixture distributions or neural networks there could be multiple parameters Gover Ning complexity.
1.2 Model Selection and why?
The principal objective in determining the values of such parameters, is
- Usually to achieve the best predictive performance on new data.
- Finding the appropriate values for complexity parameters within a given model.
- To consider a range of different types of model in order to find the best one for our particular application.
1.3 Category 1-a Training set, a validation set, a third Test set
If data is plentiful, then one approach are simply to use some of the available data
- To train 1) a range of models, or 2) a given model with a range of values for its complexity parameters,
- And then to compare them on independent data (called a validation set), and select the one has the best predic tive performance.
- If the model design is iterated many times using a limited size data set and then some over-fitting to the Validatio n data can occur and so it is necessary to keep aside a third Test set on which the performance of the Select Ed model is finally evaluated.
1.4 Category 2-cross Validation
Problem:what if the supply of data for training and testing are limited?
- In order to build good models, we wish to use as much of the available data as possible for training.
- However, if the validation set is small, it'll give a relatively noisy estimate of predictive performance.
- One solution to this dilemma are to use cross-validation, which are illustrated in Figure 1.18.
cross-validation allows a proportion of the available data to being used for training while Makin G use of any of the data to assess performance. When data was particularly scarce, it may be appropriate to consider the case , where was the total Numbe R of data points, which gives the leave-one-out technique.
1.5 Drawbacks of Cross-validation
- One major drawback of cross-validation:is that the number of training runs this must be performed are increased by a facto R of, and this can prove problematic-models in which the training is itself computationally expensive 4>.
- Exploring combinations of settings for multiple complexity parameters for a single model could, in the worst case, require A number of training runs that's exponential in the number of parameters.
- Clearly, we need a better approach. Ideally, this should rely only on the training data and should allow multiple hyperparameters and model types to be compar Ed in a single training run.
2. The curse of Dimensionality2.1 A simplistic classification approach
One very simple approach would is to divide the input space into regular cells, as indicated in figure 1.20.
2.2 Problem with this Naive approach
The origin of the problem is illustrated in Figure 1.21, which shows so, if we divide a region of a space into regular C Ells, then the number of such cells grows exponentially with the dimensionalityof the space. The problem with a exponentially large number of cells is so we would need an exponentially large quantity of training The data in order to ensure the cells is not empty.
- The Curse of dimensionality: The severe difficulty that can arise in spaces of many dimensions is sometimes calle D The Curse of dimensionality.
- The reader should be warned this not all intuitions developed in spaces of a low dimensionality would generalize to High-dimensional spaces.
CCJ prml Study note-chapter 1.3-1.4:model Selection & the Curse of dimensionality
