This article can basically be considered as a PRML Reading Note, mainly from reading the content in [1]. If you have any questions, please correct me. Thank you very much. The method of describing probability by the frequency of occurrence of a random event is generally referred to as classical probability or frequency school. In addition, a more comprehensive view is the Bayesian School, where probability represents the uncertainty of events. Probability is used to indicate uncertainty. Although it is not the only choice, it is inevitable, because if you want to use a natural feeling for reasonable and comprehensive inference. In the field of pattern recognition, a more comprehensive understanding of probability will be very helpful. For example, in the process of polynomial curve fitting, it seems appropriate to understand the observed target variable using the viewpoint of frequency school. However, we want to determine the uncertainty of the parameter W of the optimal model, so we can see that in bayesian theory, we can not only describe the uncertainty of the parameter, but also choose the model itself. In the Bayesian perspective, we usually need to model a Prior Distribution in the model. For example, in the fitting process of polynomial curves, we should not only choose to determine the parameters of the model, we also need to establish a prior parameter, so it is easy to combine the Bayesian formula :. In formula (1.43), P (d | W) on the right is a function under W, indicating the data occurrence under W, so we call it a likelihood function. After the likelihood is defined, we can use (1.44) for bayesian theory. In Bayesian and frequency theories, the likelihood function p (d | W) plays a very important role. In the frequency perspective, W is regarded as a definite parameter, which is determined by some form of estimation, which is obtained based on the distribution of possible datasets. In the Bayesian view, the parameter is derived from a modeling of W distribution. The advantage of Bayes's view is that it is natural to include a prior knowledge in the model. For example, in a coin-throwing experiment, if three coins are displayed on the front, the likelihood of positive appearance is 1 if the maximum likelihood is used for estimation based on frequency, this means that in the future, there will be positive values with a probability of 1. On the contrary, introducing a rational prior in bayesian theory will avoid such extreme conclusion. Although there is a lot of debate between the frequency School and the Bayesian School, there is actually no pure frequency viewpoint or Bayesian viewpoint. However, in practical application, there is a great criticism on the application of bayesian theory. That is to say, the prior choice is usually based on the convenience of mathematical theory, rather than reflecting any prior belief. Although Bayesian frameworks have been proposed since the 18th century, the application of bayesian theory is limited by the whole process of Bayesian method calculation, the entire parameter space of marginalize is required, especially when predicting or comparing models. However, with the development of the sampling method, such as the Markov Chain Monte Carlo method, it can be applied to small-scale problems. In addition, the arc direction of deterministic approximation schemes (variational Bayes and expectation Propagation) is an optional alternative to the sampling method, which allows Bayesian methods to be applied to large-scale applications. In fact, in terms of the metaphor in [2], in the process of playing mahjong, It is the frequency school to determine which card you play only based on the card on the desktop; if we consider who played the cards and the cards on the desktop, we can be understood as the Bayesian School.
References: [1]. Pattern Recongnition and machine learning, author Christopher M. Bishop, section 1.2.3 Bayesian probabilities. [2]. [Machine Learning] frequency School and Bayesian School, http://blog.csdn.net/zhuangxiaobin/article/details/26166599.
Analysis of frequency school viewpoint and Bayesian school viewpoint in Probability
