Bayesian generation Confrontation Network (GAN) _ Bayesian

Transferred to the rehabilitation of intellectual Yuan: http://www.sohu.com/a/144843442_473283

Original title: Bayesian Generation Confrontation Network (GAN): The best performance end-to-end half Supervision/unsupervised Learning _ Sohu Technology _ Sohu

New Intellectual Yuan Report

Author: Alex Ferguson

"New wisdom Yuan Guidance" Cornell University researcher combined with Bayesian and confrontation Generation Network, in the 6 open benchmark data set to achieve the best performance of semi-supervised learning, at the same time, this is a major step towards the ultimate unsupervised learning. In this paper, a practical Bayesian formula is proposed, which uses Gan for unsupervised learning and semi supervised learning. This new approach, simplicity is its greatest advantage--reasoning is directly, can be explained and stable. All experimental results are obtained without the need for parameter matching, regularization or any special (AD-HOC) techniques.

Yunus Saatchi, Andrew Gordon Wilson of Cornell University and Permutation venture, recently published a study on unsupervised and unsupervised learning called Bayesian generation (Bayesian GAN).

The dependence of deep learning on a large number of tag data is obvious, which is also one of the potential factors inhibiting the development of deep learning. Scientists have long explored the use of as little label data as possible, hoping to transform from supervised learning to semi supervised learning to the final unsupervised learning.

The authors also mentioned in the article that "effective semi-supervised learning of natural high-dimensional data is essential for reducing the dependence of depth learning on a large number of tagged Datasets." ”

In general, we do not have tagged data unless it is achieved at high cost or through human labor or through expensive instruments such as the laser radar for autonomous driving.

At present, unsupervised learning is still some distance away from us, but semi-supervised learning has become the newest research hotspot. In particular, since the 2017, the Fight Generation Network (GAN) and automatic coding technology have made continuous progress, supporting the development of the semi-supervised learning field.

For one of the ultimate goals of artificial intelligence-unsupervised learning, semi-supervised learning also provides a practical and quantifiable mechanism to evaluate the latest developments in unsupervised learning.

Bayesian Warfare Generation Network: The common benchmark can provide the best semi-supervised learning quantization results

Let's take a look at the summary of the article:

The author mentions that generating a confrontation network can unconsciously learn the richness of images, sounds and data. These distributions are often difficult to model because of their explicit similarity.

They put forward a practical Bayesian formula in the study, in the practice of Gan for unsupervised learning and semi supervised learning. Under this framework, the dynamic gradient Hamilton Monte Carlo (Hamiltonian Monte Carlo) is used to maximize the weights of the generation networks and the discriminant networks. The method of obtaining results is very straightforward and is well behaved without the need for any standard intervention, such as feature matching or mini-batch discrimination.

Deploy an expressive posterior mechanism (posteriors) to the parameters in the builder. Bayesian gan avoids pattern collisions (mode-collapse), produces a variety of predictable and diverse candidate samples, and provides the best quantifiable results of semi supervised learning in some of the existing benchmark tests. For example, Svhn, Celeba and CIFAR-10. The effect is far more than Dcgan, Wasserstein Gans and Dcgan and so on.

Key milestones in machine learning: the establishment of high-dimensional natural signal generation model

By learning to high dimensional natural signals, such as images, video and audio, and then creating a good model for generation, it has long been one of the key milestones in machine learning. Under the enabling of the deep neural network learning ability, the generation of the Antagonism Network (GAN) (Goodfellow et al, 2014) and the Variational Automatic encoder (Kingma and welling,2013) make the AI domain closer to achieving this goal.

Gan transforms white noise through a deep neural network to generate candidate samples from the distribution of noise data. A discriminant will learn how to adjust its parameters in a supervised way to correctly distinguish a particular sample from a generator or real data distribution. At the same time, the generator updates its parameters to better "cheat" the discriminant. Once the generator has enough capacity, it can extract the CDF, the anti-CDF combination, approximately from the data distribution of interest.

Since the Designed convolution neural network provides a reasonable index for the image (unlike, for example, Gauss-likelihood, Gaussian likelihoods), the use of the convolution neural network of Gan in turn provides convincing, implicit distribution on the image.

Although Gan has great influence, their learning goals can lead to pattern collisions (mode

Collapse), that is, the generator only stores a small number of training samples to cheat the discriminant. This methodology is a "nostalgia" for the estimation of the maximum likelihood density in the past Gaussian mixture: through the collision of each component's change, we can obtain some permanent similarities, and then store these similarities in the dataset, but these similarities are useless for the generated density estimates.

In addition, there is a great deal of intervention in the process of Gan training, including feature matching, label grooming and mini-batch discrimination. In order to alleviate these difficulties in practice, many recent studies have focused on replacing jensen-shannon differences with convertible metrics, such as f-fivergences and Wasserstein differences, in standard GAN training.

Many of these studies have chosen to introduce variable regularization matrices to maximize similarity density estimates. However, just as it is very difficult to choose the regularization matrix to fight for, it is equally difficult to determine the "disagreement" that you want to use in the practice of Gan.

The author's idea is that GAN can be promoted through full probability inference. Indeed, a posteriori distribution of the parameters on the generator can be broad and highly multimodal. In general, the training of Gan is based on the minimization-maximization optimization, which usually measures the weight of the posterior mechanism in the whole network as a focal point on a single node.

In this way, even if the generator does not store the training samples, we can still expect that the samples in the generator are completely related to the samples obtained from the data distribution.

In addition, each of the models in the backend (posterior) in the network weights echoes with a wider variety of generators, each with its own very meaningful interpretation. By fully rendering the posterior distribution in the parameters of the generator and the discriminant, we are able to model the real data more accurately. Subsequently, the inferred data distribution can be used for accurate and high data efficiency and semi supervised learning.

This new approach, simplicity is its greatest advantage--reasoning is directly, can be explained and stable. Indeed, all experimental results are obtained without the need for parameter matching, regularization or any particular (AD-HOC) technique.

The relevant code will be exposed soon.

Effect: Validation on 6 large datasets

In this study, the authors propose a simple Bayesian formula for end-to-end unsupervised and unsupervised learning in Gan. Within this framework, the dynamic gradient Hamiltonian Monte Carlo is used to marginalize the weight posteriors of the generator and the classifier. The author analyzes the data sample obtained from the generator, and shows the exploration of spanning several unique models in the weight of the generator. It also shows the validity of data and loops in the process of learning real distributions.

The authors say the best half-supervised learning performance has been demonstrated in several well-known benchmark tests, such as Svhn, Mnist, CIFAR-10 and Celeba.

The above graph is the performance comparison of Bayesian gan with Dcgan, W-dcgan and other models on several datasets.

Minst is a well-known benchmark for evaluating the new machine learning model, containing a label image of the 60k (50k training and 10k test) handwritten digits.

http://yann.lecun.com/exdb/mnist/

Minst's earliest author was Chris Burges, Corinna Cortes, which was completed by Yann LeCun, Corinna Cortes and Christopher j.c. Burges.

The authors used a total of 6 well-known public datasets to test Bayesian warfare generation Network models: Synthetic, mnist, CIFAR-10, Svhn, and Celeba. Each dataset has four different sets of label samples. These are the sample images of CIFAR-10, Svhn and Celeba respectively.

Future development Direction: Continuous exploration of Bayesian deep learning

Bayesian Gan can capture a wide variety of complementary and interpretive data expressions through the enrichment of multimode distributions in the weighted parameters of the generators. The author's research has shown that such expression can use simple inference program to realize the superiority of the semi-supervised problem.

Figure 4: Function test precision about the number of iterations. We can see that after approximately 1000 SG-HMC iterations, the sampler is well blended. We also see that each iteration, the SG-HMC sampler is more efficient at learning data distribution than other scenarios.

In the future, it can be combined by estimating the marginal similarity of the probability Gan (marginal likelihood) according to the distribution of parameters. Marginal similarity provides a natural utility function (natural utility functions) that automatically learns the parameters, as well as a model comparison between different Gan schemas. It is also possible to study different metrics (such as the alpha-divided family) for deterministic similarity inference to promote entropy in the sample. It is also interesting to combine the Bayesian gan with the nonparametric Bayesian deep learning framework (such as deep core learning). We hope our work will help inspire continuous exploration of Bayesian depth learning.

About Gan

New intelligence Yuan previous report: "The most detailed gan introduction" Wang Leap, etc.: The research progress and prospect of generation against network GAN the introduction of:

Generation against network GAN (generative adversarial networks) is a generation model proposed by Goodfellow in 2014. GAN is inspired by the two-person zero-sum game in the game theory (that is, the sum of the interests of two people is zero, the income of one side is the loss of the other), and the system is composed of a generator and a discriminant. The generator captures the potential distribution of real data samples and generates new data samples; The discriminant is a two classifier to determine whether the input is a real or a generated sample. Both the generator and the discriminant can be used to study the hot deep neural network at present. The optimization process of GAN is a minimax game (Minimax game) problem, the optimization goal is to achieve Nash equilibrium, so that the generator estimates the distribution of data samples.

Since the development of Gan in 2014, various models based on Gan have been proposed, including model structure improvement, theory expansion and application.

Under the current artificial intelligence upsurge, GAN's proposal satisfies many fields research and the application demand, simultaneously has injected the new development impetus to these domains. GAN has become a popular research direction in the field of artificial intelligence, and renowned scholar LeCun even called it "the most exciting idea in machine learning in the past ten years". Currently, the field of image and vision is one of the most widely used fields of GAN research and application, already can generate a number, face and other objects, constitute a variety of lifelike indoor and outdoor scenes, from the image of the original image restoration, color black and white images, from the object contour Restore object image, from low-resolution images to generate high-resolution images. In addition, GAN has been applied to the research of speech and language processing, computer virus monitoring, chess game program and so on.

Bayesian Confrontation Generation Network paper address: https://arxiv.org/pdf/1705.09558.pdf