Machine Learning & Data Mining note _ 10 (simple understanding of Gaussian process)

 

　　Preface:

GP (Gaussian process) is a common and important random process in nature. It is also called a normal random process and widely used in ml and other fields. The purpose of this experiment is to give a simple understanding of GP. In particular, a sample of GP is no longer a common point, but a function. The experiment completes the display of one and two dimensional samples of common GP. Common GP include linear GP, Brown motion, exponential GP, Ornstein-Uhlenbeck Process, symmetric and periodic GP. Reference all video http://www.youtube.com/playlist? List = pld0z06aa0d2e8zzba.

 

　　Lab basics:

First, let's take a look at the GP definition:

GP is the probability distribution of Gaussian Functions. Given any independent variable Si, SJ,… in the s set ,... SK, satisfying Z (SI), Z (SJ ),..., Z (SK) is a Gaussian random variable, and Z (SI), Z (SJ ),..., Z (SK) is a multi-dimensional Gaussian function (can have any number of), which is recorded as Z. It is called a random variable set Z (SI), Z (SJ ),..., Z (SK) is the Gaussian process on the s set.

If the number of elements in the s set is limited, whether Z is GP can be used to determine whether it is a multi-dimensional Gaussian function through the exhaustive query. If the number of elements in the s set is infinite, Z cannot be obtained through the exhaustive query. However, if the variable in Z is directly related to a Gaussian variable, then Z may also be a GP on S.

Let's take a look at the existence theorem of GP and how to construct GP:

The existence theorem indicates that an mean function exists for a single element in S of any set, and a kernel function (covariance function) exists for two elements in S of any set ), then there must be a Gaussian process Z (t) in S, and its elements have the mean and variance similar to the S-form. Therefore, after the given set S, we only need to give a mean function of one dollar, a binary kernel function expression, and a Gaussian process can be constructed. Common Gaussian processes include:

Random planes, Brownian motion, squared exponential GP, Ornstein-Uhlenbeck, a periodic GP, and a regular Ric GP.

Their mean functions and covariance functions are defined as follows:

　　

Here we will talk about the common method of sampling Gaussian Functions: First, we know that any Gaussian function can be written as a linear combination of standard Gaussian Functions, therefore, it is okay to sample the standard Gaussian function. Method

To: Calculate the distribution function of the standard Gaussian function. Use the [0, 1] uniform distribution random generator to select the random value y as the function value of the standard Gaussian function, then find the corresponding s under the distribution function, which is the sample we need.

The theory of refactoring Z through the variable decomposed by SVD in the program is not understood yet.

 

　　Experiment results:

First, let's look at the GP sample in 1d, and its linear GP is as follows (the image shows that a sample in the GP that meets the condition is a straight line ):

The result of the Brown motion is as follows (a bit of random swimming ):

The exponential GP result is (characteristic is very smooth ):

The Ornstein-Uhlenbeck process is as follows (similar to the Brown motion ):

Cyclical GP:

Symmetric GP:

The following figure shows the result in 2D mode. Linear GP (a plane ):

Squared exponential:

Ornstein-Uhlenbeck Process:

The processing time and memory consumption of 2D are relatively large, so the number of displayed sample points is reduced here.

 

　　Code and comments of the experiment:

Gp_1d.m:

% Gp_1d.m % select the kernel function (covariance function), and its mean function is constant 0 by default. kernel = 6; Switch kernel case 1; k = @ (x, y) 1 * x' * Y; % linear case 2; k = @ (x, y) 1 * min (x, y); % Brownian case 3; k = @ (x, y) exp (-100 * (x-y) '* (x-y); % squared exponential case 4; k = @ (x, y) exp (-1 * SQRT (x-y) '* (x-y); % Ornstein-Uhlenbeck case 5; k = @ (x, y) exp (-1 * sin (5 * pI * (x-y )). ^ 2); % A periodic GP case 6; k = @ (x, y) exp (-100 * min (ABS (x-y ), ABS (x + y )). ^ 2); % A Random Ric g Pend % select the point X to be displayed, that is, part of the set S x = (-1:0. 005); n = length (x); % returns the covariance matrix C = zeros (n, n); for I = 1: N for j = 1: n C (I, j) = k (x (I), x (j); endend % samples GP Rn = randn (n, 1); % generates N 0 ~ Random number between 1, satisfying the normal distribution [U, S, V] = SVD (c); % SVD decomposition rn matrix, S is the singular value matrix, U is the singular vector. C = usv 'z = u * SQRT (s) * rn; % Z why is this representation ?? % Draw a sample figure (1) of GP; Hold on; clfplot (x, z, '.-'); % axis ([,-]);

 

Gp_2d.m:

% Gp_2d.m % select the kernel function (covariance function), and its mean function is constant 0 by default. kernel = 1; Switch kernel case 1; k = @ (x, y) 1 * x' * Y; % linear case 2; k = @ (x, y) exp (-100 * (x-y) '* (x-y); % squared exponential case 3; k = @ (x, y) exp (-1 * SQRT (x-y) '* (x-y); % Ornstein-uhlenbeckend % select the point to be displayed points, two-dimensional points = (0: 0. 02:1) '; [U, V] = meshgrid (points, points); X = [U (:) V (:)]'; n = size (x, 2 ); % construct covariance matrix C = zeros (n, n); for I = 1: N for j = 1: n C (I, j) = k (x (:, i), x (: , J); endend % samples GP Rn = randn (n, 1); % generates N 0 ~ Random number between 1, satisfying the normal distribution [U, S, V] = SVD (c); % SVD decomposition rn matrix, S is the singular value matrix, U is the singular vector. C = usv 'z = u * SQRT (s) * rn; % Z why is this representation ?? % Draw a sample figure (2) of GP; clfz = reshape (z, SQRT (N), SQRT (n); SURF (u, v, Z );

 

 

　　References:

Http://www.youtube.com/playlist? List = pld0z06aa0d2e8zzba