2.5 ESL Chapter II Exercise 2.5

Topic

Describe

• $y _i=x_i^t\beta+\epsilon_i$
  $\epsilon_i\sim N (0,\sigma^2) $

• There are training sets $\tau$, where $x:n\times p,y:n\times 1,\epsilon:n\times 1$
  Get $\hat{\beta}=\left (x^tx\right) using least squares ^{-1}x^ty$
  $y =x\beta+\epsilon$

• Need to predict the $x_0$ of points $y_0$

Question 2.7

• Get ready
  • $E (Y_0) =e (x_0^t\beta+\epsilon_0) =e (X_0^t\beta) +e (\EPSILON_0) =x_0^t\beta+0$
  • $E [\left (Y_0-e (y_0) \right) ^2]=e[\left (x_0^t\beta+\epsilon_0-x_0^t\beta\right) ^2]=e[\epsilon_0^2]=\sigma^2$
  • $\hat{y_0}=x_0^t\hat{\beta}=x_0^t\left (x^tx\right) ^{-1}x^t\left (x\beta+\epsilon\right) \ \ =x_0^T\left (X^TX\ right) ^{-1}x^tx\beta+x_0^t\left (x^tx\right) ^{-1}x^t\epsilon\\ \ =x_0^t\beta+x_0^t\left (X^TX\right) ^{-1}X^T\ epsilon\\ \ =x_0^t\beta+\sum_i a_i\epsilon_i$ <br> where $a_i=\left[x_0^t\left (x^tx\right) ^{-1}x^t\right]_i$
  • $E (\hat{y_0}) =e (x_0^t\beta+\sum_i a_i\epsilon_i) =x_0^t\beta+\sum_i E (a_i) e (\epsilon_i) =x_0^t\beta$
    This is because $x$ is generated by a distribution, so $e (a_i) $ is not a simple constant
• Exercises

  $EPE (X_0) =\int\int\left (y_0-\hat{y_0}\right) ^2p (Y_0) p (\hat{y_0}) \mathrm{d} y_0\mathrm{d}\hat{y_0}\\ \ =\int\int\ LEFT[\HAT{Y_0}-E (\hat{y_0}) +e (\hat{y_0})-y_0\right]^2p (Y_0) p (\hat{y_0}) \mathrm{d} y_0\mathrm{d}\hat{y_0}\\ \ =\ INT\LEFT[\HAT{Y_0}-E (\hat{y_0}) \right]^2p (\hat{y_0}) \mathrm{d}\hat{y_0}+\int\int\left[e (\hat{y_0})-y_0\right]^ 2p (Y_0) p (\hat{y_0}) \mathrm{d} y_0\mathrm{d}\hat{y_0}+2\times 0\\ \ ={var}_\tau (\hat{y_0}) +\int\int\left[y_0-e (Y_0) +e (Y_0)-E (\hat{y_0}) \right]^2p (Y_0) p (\hat{y_0}) \mathrm{d} y_0\mathrm{d}\hat{y_0}\\ \ ={var}_\tau (\hat{y_0}) +\int\ LEFT[Y_0-E (Y_0) \right]^2p (y_0) \mathrm{d} y_0+\left[e (Y_0)-E (\hat{y_0}\right]^2+2\times 0\\ \ ={Var}_\tau (\hat{y_0} ) +\sigma^2+0^2$
  ${var}_\tau (\hat{y_0}) =e\left[\hat{y_0}-e (\hat{y_0}) \right]^2\\ \ =e\left[x_0^t\beta+\sum_i a_i\epsilon_i-x_0^T\ Beta\right]^2=e\left[\sum_i\sum_j a_ia_j\epsilon_i\epsilon_j\right]\\ \ =e\left[\sum_i a_i^2\epsilon_i^2 \right]+E\ Left[\sum_i\sum_{j:j\neq i} a_ia_j\epsilon_i\epsilon_j\right]\\ \ =\sum_ie (a_i^2) E (\epsilon_i^2) +\sum_i\sum_{j:j\ Neq i} e (A_ia_j) e (\epsilon_i) e (\epsilon_j) \ \ =\sigma^2e (\sum_i a_i^2) +0=\sigma^2e\left (X_0^t\left (x^tx\right) ^{-1 }x^tx\left (x^tx\right) ^{-1}x_0\right) \ \ =\sigma^2e\left (X_0^t\left (x^tx\right) ^{-1}x_0\right) $

Question 2.8

• Get ready
  • Suppose $e (x^{(i)}) =0,i=1...p$, which is the expectation of 0 for each dimension
    $X ^tx$ Get the matrix of $p\times p$
    $X _{:i}$ represents the $x$ column, which is the first dimension of the input portion of the training set $i$
    $X _{:i}^tx_{:i}=\sum_j^n {x_j^{(i)}}^2=n\ \frac{1}{n}\sum_j^n (x_j^{(i)}-e (x^{(i)})) ^2=n\hat{var} (x^{(i)}) $ Get diagonal elements
    $X _{:i}^tx_{:j}=\sum_t^n x_t^{(i)}x_t^{(j)} = n\ \frac{1}{n} (x_t^{(i)}-e (x^{(i)})) (x_t^{(j)}-e (x^{(j)})) =n\hat{cov } (x^{(i)},x^{(j)}) $
    When $n\to \infty $, $X ^tx \to Ncov (X) $
  • $K:p \times p,b:p\times 1$
    $trace kbb^t=\sum_i {[kbb^t]}_{ii}=\sum_i \sum_j k_{ij}{[bb^t]}_{ji}=\sum_i \sum_j k_{ij}b_ib_j$
    $b ^tkb=\sum_i {b^t}_{1i}{[kb]}_{i1}=\sum_i \sum_j b_ik_{ij}b_j$
    $trace kbb^t=b^tkb$
• Exercises

  $E \left (X_0^t\left (x^tx\right) ^{-1}x_0\right) \sim E\left (X_0^t{cov (x)}^{-1}x_0\right)/n\\ \ =E\left (Trace {Cov (x)} ^{-1}x_0x_0^t\right)/n\\ \ =trace {Cov (x)}^{-1}e (x_0x_0^t)/n=trace {Cov (x)}^{-1}cov (x)/n\\ \ =trace i/n=p/n$
  $EPE (X_0) = (p/n+1) \sigma^2$

2.5 ESL Chapter II Exercise 2.5