R in action Reading Notes (11)-Chapter 8: Regression -- Select the "best" regression model and action Reading Notes

R in action Reading Notes (11)-Chapter 8: Regression -- Select the "best" regression model and action Reading Notes

8.6 select the "best" Regression Model

8.6.1 model comparison

The anova () function in the Basic installation can be used to compare the goodness of fit of the two nested models. The so-called nested model, that is, its 1

Some items are completely included in another model

Comparison Using anova () Functions

> States <-as. data. frame (state. x77 [, c ("Murder", "Population", "Illiteracy", "Income", "Frost")])

> Fit1 <-lm (Murder ~ Population + Illiteracy + Income + Frost, data = states)

> Fit2 <-lm (Murder ~ Population + Illiteracy, data = states)

> Anova (fit2, fit1)

Analysis of Variance Table

 

Model 1: Murder ~ Population + Illiteracy

Model 2: Murder ~ Population + Illiteracy + Income + Frost

Res. Df RSS Df Sum of Sq F Pr (> F)

1 47289.25

2 45289.17 2 0.078505 0.0061 0.9939

AIC (AkaikeInformation Criterion) can also be used to compare models.

Measure the fit degree and the number of parameters used for fitting. A model with a smaller AIC value should be selected first, which indicates that the model uses fewer parameters.

Sufficient fit is obtained.

> AIC (fit1, fit2)

Df AIC

Fit1 6 241.6429

Fit2 4 237.6565

8.6.2 Variable Selection

1. gradually return to stepwise method

In a regression model, a variable is added or deleted at a time until a criterion is reached. Forward

A prediction variable is added to the model each time, until the model is not changed.

. Backward stepwise deletes a variable at a time starting from the model containing all prediction variables.

Until the quality of the model is reduced. Stepwise

), Combined with the forward and backward regression methods, each variable enters one, but each step

Variables will be reevaluated, variables that do not contribute to the model will be deleted, and prediction variables may be added or deleted

Several times until the optimal model is obtained .. The stepAIC () function in the MASS package can be implemented.

The progressive regression model (forward, backward, and backward) is based on the precise AIC criterion.

> Library (MASS)

> Fit1 <-lm (Murder ~ Population + Illiteracy + Income + Frost, data = states)

> StepAIC (fit1, direction = "backward ")

Start: AIC = 97.75

Murder ~ Population + Illiteracy + Income + Frost

Df Sum of Sq RSS AIC

-Frost 1 0.021 289.19 95.753

-Income 1 0.057 289.22 95.759

<None> 289.17 97.749

-Population: 1 39.238 328.41 102.111

-Illiteracy 1 144.264 433.43 115.986

Step: AIC = 95.75.

Murder ~ Population + Illiteracy + Income

Df Sum of Sq RSS AIC

-Income 1 0.057 289.25 93.763

<None> 289.19 95.753

-Population 1 43.658332.85 100.783

-Illiteracy 1 236.196 525.38 123.605

Step: AIC = 93.76.

Murder ~ Population + Illiteracy

Df Sum of Sq RSS AIC

<None> 289.25 93.763

-Population: 1 48.517 337.76 99.516

-Illiteracy 1 299.646588.89 127.311

Call:

Lm (formula = Murder ~ Population + Illiteracy, data = states)

Coefficients:

(Intercept) Population Illiteracy

1.6515497 0.0002242 4.0807366

2. Full subset regression

Full-subset regression can be implemented using the regsubsets () function in the leaps package. You can adjust the R square or

Mallows Cp Statistics and other criteria to select the "best" Model

> Library ("leaps", lib. loc = "d:/ProgramFiles/R/R-3.1.3/library ")

> Leaps <-regsubsets (Murder ~ Population + Illiteracy + Income + Frost, data = states, nbest = 4)

> Plot (leaps, scal = "adjr2 ")

> Library (car)

> Subsets (leaps, statistic = "cp", main = "cpplot for all subsets regression ")

> Abline (1, 1, lty = 2, col = "red ")

 

 

8.7 deep analysis

8.7.1 cross-validation

The so-called cross-validation means that a certain percentage of data is selected as a training sample, and another sample is reserved as a sample.

Obtain the regression equation on the training sample, and then make a prediction on the reserved sample. Because the reserved samples do not involve the selection of model parameters

Samples can obtain more accurate estimates than new data. In k-crossover verification, the sample is divided into k sub-samples, and k1 sub-samples are combined in turn as the training set, and the other one sub-sample is used as the retention set. In this way, k prediction equations are obtained, the prediction results of k retention samples are recorded, and the average value is obtained. [When n is the total number of observations and k is n, this method is also called the jackknifing method] The crossval () function in the bootstrap package can implement k-crossover verification.

Fit <-lm (mpg ~ Hp + wt + hp: wt, data = mtcars)

Shrinkage <-function (fit, k = 10 ){

Require (bootstrap)

Theta. fit <-function (x, y) {lsfit (x, y )}

Theta. predict <-function (fit, x) {cbind (1, x) % * % fit $ coef}

X <-fit $ model [, 2: ncol (fit $ model)]

Y <-fit $ model [, 1]

Results <-crossval (x, y, theta. fit, theta. predict, ngroup = k)

R2 <-cor (y, fit $ fitted. values) ^ 2

R2cv <-cor (y, results $ cv. fit) ^ 2

Cat ("original r-square =", r2, "\ n ")

Cat (k, "fold cross-validated r-square =", r2cv, "\ n ")

Cat ("change =", r2-r2cv), "\ n ")

}