Exponential family: Exponential distribution family

Exponential family (exponential distribution family) is a common concept, but its definition is not particularly clear, today look at the wiki content, have a general understanding, first and share with you. This article is basically the translation of some content on the wiki.

1. Several questions

What is an exponential distribution family?

Since it is a "family", what are the common characteristics in the clan?

Why are exponential distribution families widely used? Are the exponential distribution families choosing us, or are we choosing an exponential distribution family? (This question does not answer, need to combine specific case analysis)

2. Reference

Exponential family. (February, 26). in Wikipedia, the free encyclopedia. Retrieved 05:00, April 3, from http://en.wikipedia.org/w/index.php?title=Exponential_family&oldid= 648989632

3. Exponential distribution family: definition

Exponential distribution family refers to the distribution of probability distributions that meet the following forms

Where ($\theta$, $x $ can also be scalar)

Exponential family, also known as exponential Class, includes a number of common distributions. Such as

Normal, exponential, gamma, chi-squared, beta, Dirichlet, Bernoulli, categorical, Poisson, Wishart, inverse Wishart.

The T (x ) in the distribution function, η (θ ) and A(η ) are not arbitrarily defined, and each part has its special meaning.

T (x ) is a sufficient statistic of the distribution (sufficient statistic )

η is a natural parameter. For a finite function, the set of η is called the Natural parameter space.

A (η ) is called a logarithmicpartition function, which is actually a logarithmic form of a normalized factor. It satisfies the condition that the probability distribution integral is 1.

The above formula can be seen, by the derivation of A(η ), it is easy to get sufficient statistics T (x ) mean, variance and other properties. (How to beg?) ）

4. Exponential distribution family: nature

The exponential distribution family has many properties, which make the exponential distribution family play an important role in statistical analysis. And in many cases, only exponential distribution families have those properties. These include

Do not understand, afraid of a mistake, or to the original good.

exponential families has Sufficient Statistics That can summarize arbitrary amounts of Independent identically distributed data using a fixed number of values. exponential families has conjugate priors , an important property in Bayesian Statistics . the posterior predictive distribution of an exponential-family random variable with a conjugate Prior can always is written in closed form (provided that normalizing factor of the Exponential-fam ily distribution can itself is written in closed form). Note that these distributions is often not themselves exponential families. Common examples of non-exponential families arising from exponential ones is the Student ' s t-distribution , beta-binomial distribution and dirichlet-multinomial distribution . In the Mean-field approximation in variational Bayes (used for approximating the posterior distribution in large Bayesian networks), The best approximating posterior distribution of a exponential-family node (a node was a random variable in the context of Bayesian networks) with a conjugate prior are in the same family as the node.

The specific explanation is as follows (what you don't see is not explained ...). ）：

(1) The sufficient statistics of the exponential function can be attributed to a large number of I.I.D. The estimated number of values (that is, T (x )), as described in sufficient statistics

according to the Pitman–koopman–darmois theorem, among famil ies of probability distributions whose domain does not vary with the parameter being estimated, only in Exponenti Al families is there a sufficient statistic whose dimension remains bounded as sample size increases. Less tersely, suppose is independent identically distributed random variables Whose distribution is known to being in some family of probability distributions. Only if the family is an exponential family is there a (possibly vector-valued) sufficient statistic W Hose number of scalar components does not increase as the sample size n increases. This theorem shows that sufficiency (or rather, the existence of a scalar or vector-valued of bounded dim Ension sufficient statistic) sharply restricts the possible forms of the distribution.

(2) The exponential distribution family has a conjugate priori characteristic. Refer to the article "Terminology interpretation".

5. Exponential Distribution Family: example

Normal

This is a single-parameter exponential distribution family, which can be written in the following standard form.

Refer to Wikipedia for more information.

6. Terminology Interpretation

independent identically distributed Independent distribution (I.I.D.)

If, in a set of random variables, any random variable has the same probability distribution and is independent of each other, then the set of variables is said to be distributed independently.

Sufficient statistic sufficient statistical capacity

The statistic T(x ) is sufficient for a given parameter θ to refer to the given statistic T (x ) for the conditional probability of x , is not dependent on the parameter θ.

A more understandable approach is (see Steven M.kay, "Fundamentals of Statistical signal Processing"), where sufficient statistics are more than one. For the parameter θ to be estimated, the observed data set is clearly a sufficient statistic. It is well represented that once sufficient statistics are given, the conditional probabilities of the parameters are independent of the other statistics.

The above two formulas are equivalent. The Bayesian formula can be deduced from each other.

partition function Distribution Function

It's too long to see. is a special case of normalized parameters, it is simple to understand that the normalization of parameters is good.

conjugate priors conjugate priori (do not know whether this should be translated)

In Bayesian probability theory, if the posterior probability distribution p(θ| x ) and the prior probability distribution p(θ) belong to a probability distribution family, then the posterior and transcendental are called conjugate distributions. At the same time this transcendental is called a conjugate priori under this likelihood function.

Bayes theorem has

Given the likelihood function (which is usually very good by observing the result), the difficulty of the integral solution on the equation is related to the prior knowledge. Under certain selection, the posterior probability distribution and the prior probability distribution have similar algebraic structures.

The conjugate Transcendental property gives the closed section of the posterior probability distribution, otherwise we need to solve the complex integrals. Furthermore, the conjugate priori allows us to see clearly the effect of the likelihood function on the probability distribution.

7. To be Continued

Exponential family: Exponential distribution family