Some cutting points in statistical learning-starting from basic concepts

1. Non-probability model vs Probability Model

A non-probability model is a model represented by a decision function, and a probability model is a model represented by a conditional probability.

2. loss function (loss fun) VS risk function (cost fun)

The loss function measures the quality of a model's prediction at a time. The risk function measures the quality of a model's prediction at an average value, which is assumed the expected loss of the model under the joint distribution.

The risk calculation function must know the joint distribution, and which is unknown.

3. Empirical loss or empirical risk: average loss of the model about the training set. According to the law of large numbers, we use empirical loss to estimate risk functions.

4. empirical risk mininmization (ERM) vs structural risk minimization (SRM)

Common basic correction strategies for estimating expected losses using empirical risks.

Erm: applicable when the sample size is large enough. When the sample size is smallOverfitting(Over-fitting.

SRM: This is equivalent to regularization ).

Structural Risk = empirical risk + regularization item